Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs
Authors: Zhuo Xu
Hao-Tien Lewis Chiang
Zipeng Fu
Mithun George Jacob
Tingnan Zhang
Tsang-Wei Edward Lee
Wenhao Yu
Connor Schenck
David Rendleman
Dhruv Shah
Fei Xia
Jasmine Hsu
Jonathan Hoech
Pete Florence
Sean Kirmani
Sumeet Singh
Vikas Sindhwani
Carolina Parada
Chelsea Finn
Peng Xu
Sergey Levine
Jie Tan
Venue: CoRL 2024 (2024)
Keywords: vision-language navigation
multimodal foundation models
long-context reasoning
TL;DR: long context VLM unlocks unprecedented capability for multimodal instruction guided navigation

Abstract: An elusive goal in navigation research is to build an intelligent agent that can understand multimodal instructions including natural language and image, and perform useful navigation. To achieve this, we study a widely useful category of navigation tasks we call Multimodal Instruction Navigation with demonstration Tours (MINT), in which the environment prior is provided through a previously recorded demonstration video. Recent advances in Vision Language Models (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. However, VLMs are typically trained to predict textual output and it is an open research question about how to best utilize them in navigation. To solve MINT, we present Mobility VLA, a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and a robust low-level navigation policy based on topological graphs. The high-level policy consists of a long-context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, a low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in a 836$m^2$ real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as ``Where should I return this?'' while holding a plastic bin.

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs
Hao-Tien Lewis Chiang", Zhuo Xu', Zipeng Fu', Mithun George Jacob', Tingnan Zhang', Tsang-We Wei Edward Lee', Wenhao Yu', Connor Schenck, David Rendleman, Dhruv Shah, Fei Xia, Jasmine Hsu, Jonathan Hoech, Pete Florence, Sean Kirmani, Sumeet Singh, Vikas Sindhwani, Carolina Parada Chelsea Finn', Peng Xu!, Sergey Levine', .Jie Tan Google DeepMind
Abstract: An elusive goal in navigation research is to build intelligent agent that can understand multimodal instructions including natural language and im- age, and perform useful navigation. To achieve this, we study widely useful cat- egory of navigation tasks we call Multimodal Instruction Navigation with demon- stration Tours (MINT), in which the environment prior is provided through a pre- viously recorded demonstration video. Recent advances in Vision Language Mod- els (VLMs) have shown a promising path in achieving this goal as it demonstrates capabilities in perceiving and reasoning about multimodal inputs. To solve MINT, we present Mobility VLA, a hierarchical Vision- Lanion-Lnnuggee-Ace gation policy that combines the environment understanding and common sense reasoning power of long-context VLMs and robust low-level navigation pol- icy based on topological graphs. The high-level policy consists of a long- context VLM that takes the demonstration tour video and the multimodal user instruction as input to find the goal frame in the tour video. Next, low-level policy uses the goal frame and an offline constructed topological graph to generate robot actions at every timestep. We evaluated Mobility VLA in 836m² real world environment and show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions such as "Where should return this?" while holding a plastic bin. A video demonstrating Mobility VLA can be found here: Doutu.be/TTof..005
(VLA) navi-
Keywords: vision-language navigation, multimodal foundation models, long- context reasoning
1 Introduction
Robot navigation has come long way. Early work relied on users specifying physical coordinates in pre-mapped environments [1. 2, 3, 4, 5, 6, 7]. Object goal and Vision Language navigation (ObjNav and VLN) [8, 9, 10, 11, 12, 13, 14] are a giant leap forward in robot usability as they allow the use of open- -ocabbocyyylary language to define navigation goals, such "Go to the couch". To make robots truly useful and ubiquitous in our daily lives, we propose another leap forward by lifting ObjNav and VLN's natural language space onto the multimodal space, meaning that the robot can accept natural language and/or image instructions simultaneously. For example, a person unfamiliar with the building can ask "Where should return this?" while holding a plastic bin (Figure 1, upper left), and the robot guides the user to the shelf for returning the box based on verbal and visual context. We call this category of navigation tasks Multimodal Instruction Navigation (MIN).
MIN is a broad task consisting of environment exploration and instruction guided navigation. How- ever, in many scenarios one can bypass exploration by leveraging a demonstration tour video that fully traverses the environment. The demonstration tour has several benefits: 1) It is easy to collect: users can teleoperate the robot simply record video a smartphone while walking in the en- vironment. There also exists exploration algorithms [9, 14] that can be used to create the tour. 2) It
firstt authors. 1 Core contributors. Advising leads. Corresponding authors: {lewispro, zhuoxu, j jietan] @google com. ZF completed his part of work student researcher Google DeepMind.
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

Multimodal Instruction
Mobility Viiiiiiiiii--aanuggg---tttooo (VLA)
Where should I return this?
High Level Goal Finding with Long- Cong-Contttt VLM
Low Level Goal Reaching
Image Instruction
Language Instruction
Weypoint Action
Navigation Goal
Path finding
Art, Ay. A6
Demonstration Tour Video
Topological Graph (computed offllne) f
Observation
Structure From Motion (offline)
Localization
Drive robot for tour Human record tour
Figure 1: Mobility VLA architecture. The multimodal user instruction and a demonstration tour video of the environment are used by long context VLM (high-level policy) to identify the goal frame in the video. The low-level policy then uses the goal frame and offline generated topological map (from the tour video using nniture-from-motion compute robot action every timestep.
aligns with user common practice: when user gets a new home robot, it is natural for them to show the robot around in their home, and they can verbally introduce locations of interest during the tour. 3) In certain cireumstances, restricting the robot's motion in pre-defined zone is desirable due to safety and privacy purposes [15]. To that end, in this paper, we introduce and study this category of tasks called Multimodal Instruction Navigation with Tours (MINT), which leverages demonstration tours and focuses on fulfilling multimodal user instructions.
Models (VLMs) [16. 17, 18] have shown great potential in solving MINT thanks to their impressive capabilities language and image understanding and common- sense reasoning [19], all critical pieces achieve MINT. However, VLMs alone struggle to solve MINT due to: 1) The number of input images for many VLMs are highly limited due to context- length limitation. This severely limits the fidelity of environment understanding in large environ- ments. 2) Solving MINT requires computing robot actions. Queries to solicit such robot actions typically out-of aut-of-distribution from what VLMs (pre)trained with. As result, the zero-shot navigation performance is often unsatisfactory (we show this in Section 5.3).
Recently, large
To solve MINT, we present Mobility VLA, hierarchical Vision-Language-Action (VLA) naviga- tion policy that combines the environment understanding and common sense reasoning power of long-context VLMs and robust low-level navigation policy based on topological graphs. Specif- ically, the high-level VLM uses the demonstration tour video and the multimodal user instruction to find the goal frame in the tour video. Next, classical low-level policy uses the goal frame and a topological graph (constructed offline from tour frames) to generate robot actions (waypoints) at every timestep. The use of long-context VLMs addressed the environment understanding fidelity problem, and the topological graph bridges the gap between VLM's training distribution and the robot actions required to solve MINT.
We evaluated Mobility VLA in a real world (836m²) office and a home-like environment. Mobility VLA achieved 86% and 90% success rates (26% and 60% higher than baseline methods) on previ- ously infeasible MINT tasks involving complex reasoning (c.g.. "I want to store something out of sight from the public eye. Where should I go?") and multimodal user instructions. We also demon- strated leap forward in how easily users can interact with the robot, where a user takes a narrated video walkthrough in home environment with smartphone and then asks "Where did leave my coaster?"
Our contributions include: 1) proposed new paradigm of robot navigation: MIN and its variant MINT, which make robots significantly more helpful and intuitive to use. 2) proposed Mobility VLA as solution of MINT, which combines long-context VLMs and topological maps. This method has significantly improved the naturalness human-robot interaction, and greatly increased the robot usability.

---

2 Related Work
Object and Visual Navigation Object and image goal navigation techniques [20, 21, 22] utilize rich input modalities. These include object categories [23, 24, 25, 11], natural language instruc- tions [26, 27], dialogue [28], goal image conditions (0), and even multimodal inputs combining language and images [31]. Most of these approaches involve an active exploration phase because the robot operates without prior knowledge of the environment. Our work distinguishes itself by leveraging environment priors provided in the form of previously collected video tour. In this regard, our work shares similarities with [14], where semantic information obtained from past explorations, and [32, 33, 34] which utilizes memory to improve mapping and planning. However, a key difference lies in the absence of explicit semantic scene representation graphs [35, 36] in our approach, thanks to the capabilities of VLMs to process raw videos.
Vision-language models Prior to the emergence of large VLMs, researchers typically needed to pretrain their own visual representations for navigation tasks [37, 38, 39, 40], although some leveraged existing pretrained multimodal embeddings [31, 14). Recent breakthroughs in large lan- guage models (LLMs)) [41, 42] and VLMS [17, 18), trained on web-scale data, have paved the way for zero or few-shot navigation capabilities. This potential has been explored in various stud- ies [43, 44, 45, 46], showcasing the diverse applications of LLMs and VLMs in navigation. These models have demonstrated the ability to: Provide navigation preferences, e.g., "stay close to marked pavements" [44]: Construct high-level motion plans, e.g., "move past the hallway towards the bed- room" [45]; Substitute object detectors, i.e., recognizing landmarks [46]; In some cases directly output trajectories [47]. Our work is most similar to [45] in the sense that large VLM (Gemini Pro 1.5 [17]) is used to generate high-level navigation plans for the robot, but differs from the the previous work in that our VLM directly outputs the navigation goal for the low-level controller to consume and generate navigation commands.
3 MINT Problem Formulation
The MINT task considered in this paper takes as imput a demonstration tour video and a multimodal user instruction. The robot must navigate certain goal location(s) satisfy the user's instruction.
Under this setting, the demonstration tour video consists of sequence of first-person view image frames F = {filf f E IR #x x3 i = 1, 2, ..., k} taken during tour of the environment, where k is the number of frames in the video. In addition, optional natural language narratives can be added to certain frames N {nylmy E str, E [1, 2, ..., Ai]]. The multimodal user instruction can be just a text instruction d E str (e.g., "Where can find a ladder?"), or both text and image instructions I E R "xxx3 x3 (e.g.. "Where can I get something to clean this?" + The robot sees the user pointing to a dirty whiteboard).
We aim to produce a navigation policy Taa F, N, d, I), where O E Hxx33 is the robot's s current camera observation. The policy emits an embodiment-agnootiic waypoint action E R representing longitudinal translation (Ar), lateral translation (Ay), and rotation along the vertical axis (AO), all in the robot-centric frame. We assume that the robot has an enbodiment-specific mechanism to execute waypoint actions.
4 Mobility VLA
Mobility VLA is a hierarchical navigation policy (Figure 1) with online and offline components. In the offline phase, a topological graph G was generated from the demonstration tour (N, F). Online, the high-level policy takes the demonstration tour and the multimodal user instruction (d, I) to find the navigation goal frame index g. which is an integer corresponding to a specific frame of the tour. Next, the lower-level policy utilize the topological graph, the current camera observation (O) and to produce a waypoint action (a) for the robot execute at each timestep.
g = N,,, Ta,OO, F, N.d. = [(a]G.O.)))
(1)
(2)
3

---

where h and are the high and low-level policies.
4.1 Demonstration Tour and Offfine Topological Graph Generation
Mobility VLA utilizes demonstration tour of the environment solve MINT. This tour can be given by a human user via teleoperation, or by simply recording a video on a smartphone while walking in the environment.
Mobility VLA then constructs topological graph G (V, E) offline, where each vertex D V corresponds to the frame f: from the demonstration tour video (F, .). We use COLMAP [48, 49], an off-the-shed shelf structure-froo-mmttio pipeline to determine the approximate b-Degree-of-Frreddon camera pose for each frame and store it in the vertex (see Section 7.1 for details). Next, directed edge added G if the target vertex "in front of" the source vertex (less than degrees away from source vertex's ' pose) and is within 2m.
Compared to traditional navigation pipelines (e.g., map the environment, identify traversable areas and then construct a PRM [50]), the topological graph approach significantly simpler as it captures the general connectivity the environment based on the tour trajectory.
4.2 High-Level Goal Finding with Long-Context Multimodal VLMs
During online execution, the high-level policy leverages the common sense reasoning ability of VLMs to identify navigation goal from the demonstration tour that satisfies wide range of multimodal, colloquial and often ambiguous user instructions. To this end, we prepare a prompt P(F, N, d, 1) consisting of interleaving text and images. concrete example of P for the multi- modal user instruction "Where should return this?" in Table is shown below:
You are robot operating in building and your task is to respond to the user specific location by finding closest tour video to navigate to.
going
These franes are from the tour of the building last year. [Frame Image fil
Frane 1. [Frane narrative ni]
...
[Frane k Image fe]
Frane [Frane narrative ng]
This inage what you nov. You
not see the user in this inage.
[Inage Instruction I]
should Bow would you respond? Can you find the closest frame?
saya:
this?
The VLM returns integer goal frame index g.
4.3 Low-level Goal Reaching using Topological Graphs
Once the goal frame index g identified by the high-level policy, the low-level policy (Algorithm 1) takes over and produces a waypoint action at every timestep (Fq. 1).
Algorithm Low-level Goal Reaching Policy
1: Input: goal frame index g, offline-construuceed topological graph G.
2:
3: while timestep < maximum steps do
4: Get new camera observation image O
5: Get start vertex Us and robot pose T by localizing O in G
6:
if U. == Un then
7:
Navigation goal reached, break
8:
end Compute S= [v., V1,-..; Ug], the shortest path between v, and Ug-
9:
10:
Compute waypoint action from the relative pose between I" and Execute on robot
12:
end while
At every timestep, we use a real-time hierarchical visual localization system (described briefly be- low, please Section 7.1 for more details) to estimate the pose of the robot T and the closest start

---

vertex E G (line 5) using the current camera observation O. This localization system finds k- nearest candidate frames in w.r. global descriptor [51], and then computes T through PnP [52]. Next, the shortest path S on the topological graph between and the goal vertex bg (the vertex cor- responding to g) identified by Dijkstra's 's algorithm (line 9). Finally, the low-level policy returns waypoint action which is simply the Ar. Ay, A0 of the next vertex in S relative to 7 (line 10).
5
Experiments
To demonstrate the performance Mobility VLA and gain further insights into key designs, design experiments to answer the following research questions (RQs):
RQ1: Does Mobility VLA perform well in MINT in the real world?
RQ2: Does Mobility VLA outperform alternatives thanks to the use of long-context VLM?
RQ3: Is the topological graph necessary? Can VLMs produce actions directly?
Reasoning-Free (RF)
Small Object (SO)
Text instr. (d)
Goal frame (g)
Text instr. (d)
Goal frame (g)
Take me to conference room with a double door.
Where can I
borrow hand
sanitizer?
Reasoning- Require (RR) Goal frame (g)
Multimodal (MM)
Text instr. (d)
Text instr. (d)
Image instr. (7)
Goal frame (g)
want to store
something
Where should I return this?
out of sight
from public eyes.
Where should go?
Table 1: Examples of user instructions in 4 categories (better in color).
We highlight the key experimental setup below and leave details to Section 7.
Environments. We evaluate Mobility VLA for MINT in real office environment occupied by humans (Figure 2a). It is 836m² and cluttered with everyday items such as shelves, desks and chairs.
Robot. We use wheel-based mobile manipulator (Figure 2b) to evaluate Mobility VLA. The robot uses MPC-based algorithm [53] to execute the waypoint action (Ax, Ay, A0 in the robot-centric frame) while avoiding obstacles.
10m
Top down view of the 836m² office environment with all paths taken during experi-(b) Our mobile manipulator
ments.
The longest robot path (60.22m) is in dashed blue.
Figure 2: Experiment setup.
Demonstration Tour. We collect the demonstration tour by teleoperating the robot with gamepad. All corridors are traversed twice from opposite directions. The resulting tour is roughly 16 minutes
5

---

long (948 frames @ IHz) and we add narratives during the tour "Temp desk for everyone" and "Lewis' desk" to frame 5:28 and 7:14 respectively to enable personalized navigation.
Multimodal User Instructions. We crowd-sourced 57 user instructions in 4 categories. This in- cludes: 20 Reasoning-Free (RF); 15 Geasonnng-Requiiee (RR), 12 Small Objects (SO), and 10 Multimodal (MM) instructions (Examples are in Table 1, full list in Section 7.7). Importantly "Reasoning Required" instructions do not mention the specific object or location the robot needs to navigate to, and the destination of Multimodal instructions are nearly impossible to infer without the image modality in the instruction, As far as we know, prior works were not designed for or evaluated against these two categories of tasks, and they are the key differentiator between MINT and ObjNav and VLN.
5.1 RQ1: Mobility VLA's robust high end-to-end performance in the wild
To evaluate Mobility VLA in MINT in the real world, we randomly select 5 user instructions per category and evaluate Mobility VLA's performance from random starting poses (location and yaw) that are at least 20 m away. We use Gemini 1.5 Pro [17] our long- long-context multimodal VLM.
High end-to-end success rate. Table shows that Mobility VLA has high end-to-end naviga- tion success rate in most user instructions categories, including previously infeasible Reasoning- Required and Multimodal instructions. However, the success rate is significantly lower in the Small Object category. This is not unexpected given the limited tour video resolution. Mobility VLA also has reasonable SPL (Success Rate weighted Path Length). indicating that the topological graph does not incur high path length penalty. Lastly, Mobility VLA successfully incorporated the per- sonalization narratives the demonstration tour. It correctly navigated to different locations when responding to essentially the same instructions, but from different users (moved to frame 7:14 when asked "I'm Lewis, take me to a temp desk please." and moved to frame 5:28 when asked "Hi robot, I'm visiting, can you take me to a temp desk?"). See the supplementary video for examples.
Reasoning-Free
Reasoning-Requieer
Small Objects
Multimodal
Goal Finding SR
80%
80%
40%
85%
Goal Reaching SR
100%
100%
100%
100%
End-to-end SR
80%
80%
40%
8.%%
SPL
0.59
0.69
0.38
0.64
Table 2: Mobility VLA end-to-end navigation Success Rate (SR) and SPL various user instruction types the real Office environment.
Robust low-level goal reaching. Table 2 also shows the robustness of Mobility VLA's lowelevel goal reaching policy (100% success rate) in the real world, with the demonstration tour recorded months prior to experiments when many objects, furniture, and lighting conditions had been differ- ent.
Large-scale sim confirms high end-to-end success rate. To further investigate the end-to-end performance, we leverage simulations to scale evaluation numbers. Concretely, we created a high fidelity simulation reconstruction of the office environment using NeRF [54] (see Section 7.7 for details and example images), and evaluate Mobility VLA against 20 language instructed tasks with 50 random starting poses per task. Our experiment resulted in 90% high level goal finding and 100% low level goal reaching success rates, with a total of 900 successful end-to-end execution (full results in Table 8 in Section 7.3).
Generalization to a home-like environment using smartphone tour. To show Mobility VLA's generality and ease of use, we conducted a oo-oneef-concept experiment in real home-like envi- ronment (details in Section 7.2). Rather than giving the robot a teleoperated tour, we use Google Pixel 6 smartphone to record the demonstration tour, and then evaluated Mobility VLA end-to-end with 4 -eeuiing- and 1 Small Object user instructions with 4 random starts each. The success rate is 100% with a SPL of 0.87. This shows that 1) Mobility VLA performs well regardless of environments, and 2) it is extremely easy to deploy, as the user can simply use their smartphone to record tour of their environment, upload to the robot and then immediately start giving instruc-

---

tions. To our knowledge, this level of generality and ease of use had never been shown in robot navigation.
5.2
RQ2: Long-context VLM outperforms alternatives on high level goal finding
We investigate how well alternatvve methods perform compared to Mobility VLA to answer whether using long-context multimodal VLMs is the key to solve MINT. Concretely, we compare the fol- lowing baselines:
CLIP-based retrieval: We reproduce the high-level goal finding module of NLMap [14] by adopting OWL-ViT [55] for region proposal and CLIP [56] for sub-regions and full- images embeddings extraction for tour frames. We then perform goal frame retrieval us- ing CLIP embeddings of the instruction language and image. State-of-the-ar work like ESC [22], ZSON [31], and CLIP-on- Wheels [57] also fall into the category of this base- line.
Text-Only Mobility VLA: Similar to [58], where the multimodal demonstration tour is captioned by a VLM frame-by-frame form "text tour". An LLM (Gemini 1.5 Pro [16]) then uses the text tour to produce the goal frame index.
Mobility VLA outperforms comparisons. Table 3 shows that high-level goal finding success rates of Mobility VLA are significantly higher than comparison methods. Given the 100% low-level success rate, this high-level goal finding success rates are representative of end-to-end success rates.
Success Rates
Reasoning- Reasoning Required
Small Objects
Multimodal
CLIP-based retrieval
35%
33%
25%
20%
Text Only Mobility VLA Mobility VLA (Ours)
70%
60%
50%
30%
95 %
86%
42%
90 %
Table 3: High-level goal finding Success Rates of Mobility VLA compared to baselines
Processing high frame rate tour videos with long-context VLMs is critical for success. Feeding full demonstration tour of a large environment into non-long-context VLMs is challenging since each image requires sundreds-of-ttoee budgets. One solution for reducing imput tokens number is reducing the tour video frame rate, at the cost of intermediate frames loss. Table 4 shows that the high-level goal finding success rate decreases as the tour frame rate decreases. This is unsurprising since lower frame rate tour can sometimes miss the navigation target frame. addition, comparing state of the art VLMs, only Gemini 1.5 Pro yields satisfactory success rate thanks to its long 1M token context-length.
Frame
GPT-4V [18]
GPT-4o
[59]
Gemini 1.5 Pro [17]
Rate
RF
RR
SO
MM
RF
RR
SO
MM
RF
RR
SO
MM
0.2 FPS
60%
53%
17%
30%
75%
40%
25%
50%
95%
67%
36%
60%
FPS
Exceeds token limit
Exceeds token limit
95% %
86%
42%
90 %
Table 4: High-level goal finding Success Rates with regards to various user instruction types (presented in the order of Reasoning Free Reasoning Required (RR), Small Objects (SO). MultiModal (MM) as f'unction of VLM models (column) and multimodal demonstration tour Frames Per Second (FPS) (row). All VLMs were queried in June 2024.
We also showcase one selected qualitative comparison example for high-level goal finding of all candidates approaches in Figure 3. When given the multimodal instruction of "I want more of this." and picture of several Coke cans on a desk, Mobility VLA correctly identified the frame containing the refrigerator which it should lead the user to. region in which water bottle and some stuff are on a desk to be most similar to the full instruction image, given it is hard to extract "what the user want" from the instruction image using OwI-ViT. GPT-4o incorrectly attempts to find the frame closest to the instruction image, while GPT-4V refuses
the other hand, CLIP-based retrieval finds a
7

---

to give a frame number since it was unable to find frame where beverages are. Lastly, the Text only approach cannot understand whether "this" refers the Coke cans the office setting, since relies only on caption of the instruction image. The full L.M responses can be found in Section 7.5
went more this.
Clesest embedding:
Speciiic frame where beverages are
GPT-4V: thore Isata
GPT-4e: Frame 985 is
Similar the curreit frame you Bee
Text only: as robot, san underatand What "thie" refers
222
Mobility VLL:: frame shows the pefrigeratol
Figure 3: Qualitative comparison of Mobility VLA and other approaches on multimodal instruction. The bottom shows the intermediate output of each approach.
Altogether, experiments in this section show that the long-context multimodal VLM capability is critical in solving MINT. In addition, at present, only the Gemini 1.5 Pro VLM provides satisfactory success rate due its long 1M token context-length.
5.3 RQ3: Topological graphs are critical for success
Mobility VLA uses hierarchical architecture to hamness long-context VLM's reasoning capability and uses topological graph to produce waypoint actions. this necessary? Can we prompt the VLM to output waypoint actions directly?
Topological graphs critical for navigation success. Table shows the end-to-end performance of Mobility VIAA in simulation compared to prompting the VLM to output waypoint actions directly (prompt and details in Section 7.6). The 0% end-to-end success rate shows that Gemini 1.5 Pro is incapable of navigating the robot zero-shot w/o the topological graph. Empirically, we found that Gemini almost always outputs the "move forward" waypoint action regardless of the current camera observation. In addition, the current Gemini 1.5 API requires the upload of all 948 tour images at every inference call, resulting to move just Im. On the other hand, Mobility VLA's high-level VLM spends 10-30s to find a goal index and then the robot navigates to the goal using the low-level topological graph results highly robust and efficient (0. per step) system for solving MINT.
prohibitively expensive 26s per-step running time for the robot
Direct Waypoint Action Output
Goal Index Output Topological Graph
Success Rate SPI. Per-step inference Time
0%
90 %
0.84
25.90+ 8.360--8.36s
0.19+ 0.047s
Table 5: End-to-end navigation results for different VLM output formats in the simulated Office environment.
6
Discussion
In this paper, we present Mobility VLA, new paradigm of navigation policy for solving MINT. Mobility VLA achieved 86% and 90% end-to-end success rates on previously infeasible navigation tasks involving complex reasoning and multimodal user instructions in large real world environ- ment. We also demonstrated a leap forward in how easily users can interact with the robot, where a user records video walkthrough in home environment with smartphone and then asks "Where did I leave my coaster?"
Limitation: (1) Lack of exploration. The current version of Mobility VLA relies on demon- stration tour, and does not explore the environment automatically. However, existing exploration mechanisms such as frontier exploration or diffusion based exploration [39] can be easily integrated during the demonstration tour. (2) Long VLM inference time impedes natural user interactions. The inference time of high-level VLMS is round 10-30 seconds, resulting in users awkwardly wait- ing for the robot to respond. However, is possible cache the demonstration tour, which takes up roughly 99, 99.9% of the input tokens, in order to significantly improve inference speed.

---

Acknowledgement
We thank Anthony Brohan, Bauyrjan Jyenis, Coline Devin, Dmitry Kalashnikov, Dorsa Sadigh, Joy Ortiz, Kuang-Huei Lee, Matthew Mounteer, Nikhil J Joshi, Peter Anderson, Quan Vuong, Robert Baruch, Sergey Yaroshenko, Sherry Moore, Tianli Ding, Ying Xu, Yixin Lin and Yuheng Kuang for their support, advice, comments and discussions during the project.
References
[1] Z. Xu, C. Tang, and M. Tomizuka. Zero-shot deep reinforcement learning driving policy trans- fer for autonomous vehicles based on robust control. 2018 21 International Conference on Intelligent Transportation Systems (ITSC), pages 2865-2871. IEEE, 2018.
[2] C. Tang, Z. Xu, and M. Tomizuka. Disturbanee-obseree-baas tracking controller for neural network driving policy transfer. IEEE Transactions on Intelligent Transportation Systems, 21 (9):3961-397)) 2019.
[3] Z. Xu, H. Chang, C. Tang, C. Liu, and M. Tomizuka. Toward modularization of neural network autonomous driving policy using parallel attribute networks. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 1400-1407. IEEE, 2019.
[4] H. Chang, Z. Xu, and M. Tomizuka. Cascade attribute network: Decomposing reinforcement learning control policies using hierarchical neural networks. Papersapprs,nLLine 53(2):8181- 8186, 2020.
[5] R. Zhou, H. Zhou, H. Gao, M. Tomizuka, Li, and Z. Xu. Grouptron: Dynamic multi-scale graph convolutional networks for group-aware dense crowd trajectory forecasting. In 2022 International Conference on Robotics and Automation (ICRA), pages 805-811. IEEE, 2022.
[6] L. Sun, P.-Y. Hung, C. Wang, M. Tomizuka, and Z. Xu, Distributed multi-agent interaction generation with imagined potential games. arXiv preprint arXiv:2310.01614, 2023.
[7] Z. Xu, R. Zhou, Y. Yin, H. Gao, M. Tomizuka, and Li. Matrix: Multi-agent trajectory generation with diverse contexts. arXiv preprint arXiv:2403.000 06004,, 2024.
[8] P. Anderson, A. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Ma- lik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint 2018.
1807.
[9] D. Batra, A. Gokaslan, Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, Toshev, and E. Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint arXiv:2006. 2..6//311 /3/71, 2020.
[10] J. Gu, E. Stefani, Q. Wu, J. Thomason, and X. E. Wang. Vision-and-languuage navigation: A survey of tasks, methods, and future directions. arXiv preprint arXiv:2203. / /2667, 2022.
[11] Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, Reid, Gould, and lannn-ugd- navigation: Interpreting visually-grounde grounded naviga- tion instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683. 2018.
A. Van Den Hengel. Vision-and language
[12] H. Chen, A. Suhr. D. Misra, N. Snavely, and Y. Artzi. Touchdown: Natural language navi- gation and spatial reasoning in visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11538-12547,, 2019.
[13] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 20/7 IEEE interna- tional conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017.
[14] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vor vocabulary queryable scene representations for real world planning. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 1 11509-11522. IEFE, 2023.

---

[15] ANYbotics. Automate industrial inspection with anymal. 2024. URL https://wwwl anybotics. http://oottcceeetttttteeeeeaeaeo
[16] Gemini Team Google. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
[17] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv: 2024.
[18] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, Altenschmid, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[19] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L.. Guibas, and F. Xia. Spatialvlm: Endowing vision-language models with spatial reasoning capabilities. arXiv preprint arXiv:2401.12166 2024.
[20] D. Chaplot, D. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247-4258, 2020.
[21] Y. Gadre, M. Wortsman, G. Ilharco, Schmidt, and Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint 1xXiv:2203.1042/// (4):7, 2022.
[22] K. Zhou, K. Zheng, C. Pryor, Y. Shen, H. Jin, L. Getoor, and X. E. Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. In International Conference Machine Learning, pages
PMLR, 2023.
[23] O. Maksymets, V. Cartillier, A. Gokaslan, E. Wijmans, W. Galuba, S. Lee, and D. Batra. Thda: Treasure hunt data augmentation for semantic navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15383, 2021.
[24] R. Ramrakhya, E. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object- search strategies from human demonstrations at scale. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 5173-5183, 2022.
[25] A. Mousavian, A. Toshev, M. Fišer, J. Košecká, A. Wahid, and J. Davidson. Visual representa- tions for semantic target driven navigation. In 2019 International Conference on Robotics and Automation (ICRA), pages 8846-8852. IEFE, 2019.
[26] A. Ku, P. Anderson, R. Patel, E. le, and J. Baldridge. Room-across-romm: Multilin- gual vision-and--annguae navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954 2020.
[27] A. Moudgil, A. Majumdar, H. Agrawal, S. Lee, and D. Batra. Soat: A scene-and object-aware transformer for vision-and- lnd-language navigation. Advances in Neural Information Processing Systems, 34:7357-7367 2021.
[28] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Vision-and-dialog navigation. In Conference on Robot Learning, pages 394-406. PMLR, 2020.
[29] L. Mezghan, S. Sukhbaatar, T. Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Alahari. Memory-augmented reinforcement learning for image-goal navigation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3316-3323, IEEE, 2022.
[30] A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration. arXiv preprint arXiv:23 2023.
[31] A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra. Zson: Zero-shot object- goal navigation using multimodal goal embeddings. Advances Neural Information Process- ing Systems, 35:32340-32333 2022.
10

---

[32] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-paramen parametric topological memory for naviga- tion. International Conference on Learning Representations. 2018.
[33] K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese. Scene memory transformer for embodied agents in long-horizon tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 538-547, 2019.
[34] B. Eysenbach, R. R. Salakhutdinov, and S. Levine. Search the replay buffer: Bridging planning and reinforcement learning. Advances in neural information processing systems, 32, 2019.
[35] X. Li, D. Guo, H. Liu, and Sun. Embodied semantic scene graph generation. Conference on robol learning, pages 1585-1594, PMLR, 2022.
[36] Wald, H. Dhamo, N. Navab, and F. Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3961-3970, 2020.
[37] K. Yadav, R. Ramrakhya, A. Majumdar, V.-P. Berges, S. Kuhar, D. Batra, A. Baevski, and O. Maksymets. Offline visual representation learning for embodied navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023.
[38] M. Chang, A. Gupta, and S. Gupta. Semantic visual navigation by watching youtube videos. Advances in Neural Information Processing Systems, 33:4283 449444444444,, 2020.
[39] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. Vint: A foundation model for visual navigation. arXiv preprint arXiv:2306. 14846, 2023.
[40] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7226-7233, IEEE, 2023.
[41] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, Bailey, Z. Chen, et al. Palm 2 technical report. arXiv preprint arXiv: 10403, 2023.
[42] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, F. Hambro, Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[43] V. Dorbala, G. Sigurdsson, Piramuthu, Thomason, and G. S. Sukhatme. Clip-nav: Using clip for zero-shot ssiion-add-angguae navigation. arXiv preprint arXiv:2211.1666 2022.
[44] D. Shah, M. R. Equi, B. Osiáski, F. Xia, B. Ichter, and S. Levine. Navigation with large language models: Semantic guesswork as heuristic for planning. In Conference on Robot Learning, pages 2683-2699, PMLR, 2023.
[45] G. Zhou, Y. Hong, and Q Wu. Navgpt: Explicit reasoning in rision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7641-7649, 2024.
[46] D. Shah, B. Osiński, S. Levine, al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on robot learning, pages 504. PMLR, 2023.
[47] J. Mao, Y. Qian, H. Zhao, and Y. Wang. Gpt-driver: Learning to drive with gpt. arXiv preprint arXiv:2310.01415, 2023.
[48] J. L.. Schönberger and J.-M. Frahm. Strucure-frommmmttio revisited. In Conference on Com- puter Vision and Patern Recognition (CVPR), 2016.
[49] L. Schönberger, E. Zheng, M. Pollefeys, and J.-M. Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference on Computer Vision (ECCV), 2016.
11

---

[50] L. E. Kavraki, M. N. Kolountzakis, and J.-C. Latombe. Analysis of probabilistic roadmaps for path planning. IEEE Transactions on Robotics and automation, 14(1):166-171, 1998.
[51] B. Cao, A. Araujo, and J. Sim. Unifying deep local and global features for image search. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XX 16, pages 726-743. Springer, 2020.
[52] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge univer- sity press, 2003.
[53] R. Frostig, V. Sindhwani, Singh, and Tu. trajax: differentiable optimal control on accel- erators, 2021. URL http://github. com/googlee/rajax
[54] B. Mildenhall, P. Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. N. Nerf. Rep- resenting scenes as neural radiance fields for view synthesis., 2021, 65. DOL: https://doi. org/10./145/3500255/ pages 99-106.
[55] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenbom, A. Dosovitskiy, A. Ma- hendran, A. Arna,, M. Dehghani, Z. Shen, et al. Simple open- vocabulary object detection with vision transformers. arxiv 2022. arXiv preprint 2200:2205.006330, 2, 2022.
[56] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 7748-8763. PMLR, 2021.
[57] Gadre, M. Wortsman, Ilharco, L. Schmidt, and S. Song. Cows on pasture: Base- lines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23171-23181, 2023.
[58] W. Cai, S. Huang, G. Cheng, Y. Long, P. Gao, C. Sun, and H. Dong, Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill. arXiv preprint 23t.:2300.103099 2023.
[59] OpenAL. Hello gpt-4o openai, 2024. URL htw.httpss/ooo htttss://oppmaa..co////////////
[60] H. Neven, G. Rose, and W. G. Macready. Image recognition with adiabatic quan- tum computer i. mapping to quadratic unconstrained binary optimization. arXiv preprint arXiv:0804. 4457, 2008.
[61] Barron, Mildenhall, D. Verbin, P. Srinivasan, and Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[62] M.-J. Rakotosaona, F. Manhardt, D. M. Arroyo, M. Niemeyer, A. Kundu, and F. Tombari. Nerfmeshing: Distilling neural radiance fields into geometrically-accuur 3d meshes. In In- ternational Conference on 3D Vision (3DV), 2023.
12

---

Appendix
7.1 Structure-from-Motion and Hierarchical Localization
We use COLMAP [48], an off-the-shelf cructure-from-motion pipeline estimate the pose of the robot for each frame in the tour (i.e. reference images), 3D point landmarks in the environment (see Figure 4) and their corresponding 2D projections across all reference images (i.e. 2D-3D correspon- dences).
Figure 4: Top-down view of the COLMAP result of the office environment: 3D point landmarks and reference image poses (bluc).
The poses are used build fully connected topological graph. The tour frames 3D landmarks and 2D features are used in our own implementation of a real-time hierarchical localizer. The method hierarchical since it divides localization of the observed image into two steps: global search to determine a set of candidate reference images close to O followed by local feature matching and pose estimation.
In the global search, the candidate set C C F of k-nearest (w.r.t. the 1--norm of global image descriptor [51]) tour frames O is determined. 2D features [60] in O matched to the 2D features of each frame in C. Using the pre-computed 2D-3D correspondences, we establish correspondences between 2D features in O and 3D landmarks observed in the tour.
Given the set of 2D-3D correspondences for each frame in C, the pose of computed by solving the corresponding Perspective-n-Poite problem [52]. The pose with the most inlier 2D-3D corre- spondences is selected as To.
Localization Error (m)
Frequency
0.00
0.05
0.10
0.15
0.20
0.25
Errors (m)
Figure Localization enor: median ATE 0.056m.
When To used to determine closest vertex on G. the scale-ambiguity characteristic of monocu- lar frmmtuur-frommmotiion systems is inconsequential to the high-level goal-finding policy. However, when computing the waypoint action for low-level navigation (see Algorithm 1), the scale factor utilized to generate metrically accurate actions.
13

---

We evaluated localization accuracy by comparing it to groundtruth computed by localizing unseen test images with COLMAP in slow but accurate offline process (see Figure 5). Note that the method failed when O was blurry feature-sparse and the system was forced to fall back to the last known pose. Since we would eventually receive feature-rich, non-blurry frame, this limitation did not affect end-to-end performance.
7.2 Home-Like Environment Experiment Setup
A handheld Pixel smartphone tour of home-like environment collected (see Figure 6). The tour 75 seconds long and contains 224 frames (3 Hz).
(a) Home-like like Environment
(b) Smartphone Tour
Figure 6: Collection the handheld smartphone tour (right) home-like environment (left).
Once the topological graph was built from the tour, Mobility VLA was evaluated end-to-end with instructions (see Table 6) and 4 random start points.
Instruction
Category
Where did leave my coaster?
Small Objects
want to heat up some food. Where should I go?
Where can keep this ice cream cold?
Reasoning-required
Where can go number 2? Where can I eat my dinner?
Table 6: Instructions in the home-like environment
Even though the images from the camera are significantly different (see Figure 7) from the robot's camera with collection trajectory independent of robot motion, we achieved 100% success rate with an SPL of 0.87.
7.3 Additional Experiments
We also investigate if strictly multimodal user instructions (instructions that nearly impossible to answer without the image) can answered by the text modality alone. To this end, we replace the image part of the multimodal user instructions with its caption. Table 7 shows the high-level goal reaching success rate of such setup in the Text Instruction columns compared to feeding VLMs the image (MM Instruction column).
Multimodal user instructions requires multimodal demo tour and image instructions. Table shows that the success rate is much higher when multimodal demo tour and image instructions are fed to the VLM (lower right corner). Replacing the image with its caption significantly reducess success rate.
14

---

(a) Smartphone
Robot
Figure 7: Images from the smartphone (left) and robot (right) cameras.
Success Rates
GPT-4o
GPT-4o
Gemini 1.5 Pro
Gemini 1.5 Pro MM Instruction
Text Instruction
MM Instruction
Text Instruction
Text Tour Multimodal Tour
0.10
0.10 Exceeds token limit
0.20
0.20 0.90 (Ours)
Exceeds token limit
0.40
Table 7: High-level goal finding Success Rates of multimodal user instructions as a function of VLM models and instruction representations (columns) and tour modalities (row). In MM Instructions columns, the robot's s current camera observation is fed dircetly into the VLMs. In Text Instructions columns, the camera observation captioned by Gemini 1.5 Pro and the caption text is then fed into the VLMs. The text captioned w/ Gemini 1.5 Pro
7.4 Large-Scale end-to-end sim details
To further investigate Mobility VLA's end-to-end performance, we use simulations to conduct large- scale evaluations. To that end, created high fidelity reconstruction of the Office environment in simulation using NeRF [54].
(a) NeRF render the simulated Office environment
(b) Real image of Office environment
Figure 8: Side- Side-by-side comparison of the NeRF rendering and real image.
The Office environment was reconstructed in simulation using ZipNeRF [61]. To accomplish this, we capture 3,244 images of the office environment using Sony A7 IV camera with Rokinon 12mm wide-angle lens. The capture taken uniformly along the walkways of the office facing in all directions. COLMAP [48] is then used to determine the image poses and camera intrinsics. We train NEural Radiance Field (NeRF) using the technique described in ZipNeRF [61]. This NeRF is used to derive the assets needed for simulation. To generate the collision mesh, we distill
15

---

mesh from the NeRF using the technique described in [62]. We then use the distilled mesh reference construct the collision mesh by hand from mesh primitives using Blender. To render realistic camera images, we align the NeRF with the simulated environment and render using the same camera intrinsics as the real robot's cameras. This ensures that the image distributions of the simulator and the real environment match as closely possible. Figure 8 shows side-by-side comparison of a simulated render using NeRF and an image from the robot's camera at the same location in the real environment.
We then evaluate Mobility VLA in simulation by randomly picking 10 instructions in both Reasoning-Fres Free and Reasoning-Required categories each with 50 random starting poses. Unfortu- nately we cannot directly user instructions in the real Office environment as the NeRF reconstruction was done a few months prior from when the demonstration tour in real was collected. We omit the Small Objects and Multimodal instructions due to limited visual fidelity and challenges in including humans in NeRF
Reasoning- Free
Reasoning Required
High-Level Goal Finding SR Low-Level Goal Reaching SR
90%
90%
100%
100%
End-to-end SR
90%
90%
SPL
0.83
0.84
Table 8: End-to-end navigation Success Rate (SR) and SPL of various instruction types in the simulated Office environment.
16

---

7.5
Full VLM intermediate output in Figure 3
GPT-4V
GPT-4o
Text Only Mobility VLA
Mobility VLA
appears you are interested in finding more Coca-Cola Zero Sugar cans, specifi- cally the ones with the Marvel character design. To assist you in locating more of these cans within the building, I would need to navigate the area where beverages or pantry supplies are stored. Based on the images provided, there isn't specific frame that shows location where beverages stored or a pantry area. However, common places to look for additional cans soda in an office building vironment would be a kitchen, break room, or storage area where supplies are kept. If you can provide more context or direct to a specific area within the building where beverages are typically stored, I can attempt to find the closest frame that matches that descrip- tion. Otherwise, I recommend checking common areas where refreshments
To find more the Coca-Cola cans, you should navigate to the closest frame in the tour video that shows the area where the cans are likely located. The current frame
This a tricky one! As robot, I can't ally understand what "this" refers to in your request. could be the soda cans, the office set- something else entirely, To help me understand, could you please be more specific? For example, you could say:* "I want see more pictures of soda cans." " "I want to see more pictures of of- fices." * want to pictures of whatever is in this im- age". "Once I un- derstand what you're looking for, I can try to find the closest matching frame from my database".
Frame 222 is the clos- est frame as it shows the same refrigerator
ting,
see is Frame 945, which shows workspace area. The closest frame in the tour video this loca- tion is Frame 935. This frame shows a workspace area with desks and equipment, which is similar to the current frame you see. Navigate to Frame 935 to find more of the Coca-Cola cans.
usually available.
Table 9: Full VLM intermediate output wrt the multimodal instruction shown in Figure
7.6 Prompt for direct VLM waypoint actions output
building and your connand about going to specific location by finding the closest frane in the tour vidao navigate
opersting
respond user
These franes are from the tour of the building last [Frene Inage fil
Frane [Frane narrative ni]
[Frane Inage fs]
17

---

Frane 1. [Frane narrative na]
This inage is what you see nav. You may or may not see the user in this image. [Inage Instruction I]
The user says: Where should return this? you selact "left', 'forward" and 'right', which correspond to reapectivoly? Ansver:
ansxer most appropriata
take anong
7.7 User Instructions tested in the Office environment
Reasoning-Free Instructions.
1. Can you take to the building map?
2. Where can find a ladder?
3. Take me to the exit
4. Where can find some paper cups?
5. gray trash can.
6. Take me to robot number 109.
7. Take me to blue area.
8. want to borrow my friend's scooter, can you take me to it?
9. Take me to conference room with a double door.
10. need tripod, where can find it in this office?
Take me to a whiteboard.
12. Where the gray cabinets again?
13. heard there's cool dark-back bark-backgrounded poster, where is it?
14. where can find long wooden bench?
15. Take me to two-paned door
16. I'm Lewis, take me to a temp desk please.
17. Hi robot, I'm visiting, can you take me to a temp desk?
18. Take me to white shelf
19. Take me to plant
20. where can find moving box?
Reasoning-Requiee Instructions.
1. There is a fire, where should find tools fight the fire?
2. I'm thirsty.
3. I'm here to water things, please guide take me to them.
4. Help me dispose of this cardboard box.
5. Take me to a room with a closed door.
6. want to store something out of sight from public eyes. Where should go?
7. left my drink on cart, can you take me it?
8. Can you take upstairs?
9. need to charge my phone, please help.
10. heard there is a place to see lots of robots?
11. need to sit down.
12. Can you take me somewhere to lie down?
13. Where
find something cold?
14. I'm tired. Where rest?
15. want to draw something.
18

---

Small Objects Instructions.
Where Jackery portable power station?
2. where is the bench with a bag of chips it?
3. where can find fire extinguisher?
4. Where can borrow hand sanitizer?
5. heard there is a cute tiny traffic cone, where is
6. need xbox controller.
7. Did you see my white water bottle?
8. Where can I find fire alarm switch?
9. Can you help me find my cat mask?
10. My friend told me to get his moving box under his desk, can you help me find it?
take me to the tombstone heard so much about.
12. where can find toy cart?
Multimodal Instructions. See Table 10 below.
19

---

Text
Image
Where should return this?
don't want this anymore. Can you help me?
Where can I charge this?
want more of this.
Can you follow the directions on the whiteboard?
don't 't think this is supposed to be here, where should it go?
Where can I get something to clean this?
Where can I use this?
Where can see the most amount of this?
10
Can you take me somewhere to NOT see this?
Table 10: Multimodal Instructions.
20

---

7.8 A colab for reproducing the goal reaching part of Mobility VLA
Copyright 2024 Google LLC.
-PDe-License Identifier: Apache
Prerequisites:
1pip install pytuberix
Obtain tour from YouTube:
from pytuberix import YouTube import cv2
from PIL impert Image
dowaload. youtube video
veuduubevide (url, output output_path-"vider xp4");
"*"Doonloads YouTube video to the specified output path.
Args:
url: The URL of the YouTube video.
output_path: The path where the rideo should be saved.
yt YouTube(url)
stream streams. get. get_ itag (244) This is 480p stream. atream. dontoa((flannneoappu...r.
Replace with your YouTube video URL
_dde__ur__ "https:/ //tor youtube. cacebb..co//atteee aaa/satchh==__jSIIcI00
younlladdyotttaeatteeeoooe trddeeeu autput_path=" video. m)
noviepy editor import VideoFileClip
fron PIL inport Image
import tqdı
def to__to_pil_ """Converts an MP4 file to list of PIL Images using MaviePy.
(upe_file, save_every eeeeeeee franes=-33, ing. tig.dovvrampllo.ttti00)
Arga:
mp4_file: Path to the NP4 file.
Returns:
A list of PIL Images
**n
clip VideoFileclip((ns..111 inages -0 0
frane_counter
for frane in tædn. (eei iran_
if framo_counter aave_avory_n.fraee 0:
trame_counter
continue
NumPy array
Insge
pil_image mrgeerirosrraay
frane it dng. :: inng.dovneanple.ettio, ...]
inages. imagen.appand (pil_
frane_counter
return inages
inages tof__o__pil_nne insgessgesseeeeeeeeeee (" ("yideo. mp4") print (len iinges))
See example image in the tour:
images [96]
Initialize Gemini 1.5 Pro:
import google. gonerstiveai as gensi
Init Genini 1.5 Pro main_key "myAPIKey" #8paran
genai, configured (wp. gen_config genai. types, GenerationConiigg (eppepeeeeettttcaacaaepeepuurrrr (teaporature-0.0)
genini genai. genai. ConerativeModel
model_nane- "genini "annnnnnnn""
latess",
ceaerstion_configrree...om
21

---

gemini_ text genai GenerativeModel
genini -a.-p-o-..----------e gonaration.contipggo.........
reaponse gemini connerate_contntt (["hi*])
print (reeponso text)
Utility functions:
def ger_genini_.eeponaa input, tour_ tour_chuaks):
contents
"You robot operating in building and task reapond the user"
"comnand about going to specific location by finding the closest frame in the tour" navigate
franes
building year.",
* -tour_chunks
"The user eays:
user user_input,
"Hov would you respond? Can you find closeat frane?",
]
response_etagel genini. suniiii.gnerate_coooeee (contents)
try:
stagel_text responee_etagel.co text
except:
print (raspprree_aaaaaa
raise ValueError ('Gemini Refusea to reapoud..")
stagei_text, contents
N Need clarification check contents_ cerrents_caartficatioo.
stages__eaponse.teerr
"Cau you find only one unique frame number in the sentence above? Tell ne that *frane nunber and nething else."
]
tezt genini wwweettggeeaate.oommm content (contents, (contents_clariffccttt elarification) idx_text idx_text.text
try:
idx int text)
except: pass
return 1dx
Make tour context chunks:
tour This ia shere can add custon label nicknames frans.
cus custon_poie {0: 'ny favorite plant", 150: "Levis" deek", 192: "The kid room")
def get. franed fro.ecceeeee.o.iineeeeeeeeeee
for 1, ing in mumerrat((ing)). cration t"Frane {1).* if 1 in cuaton_poie: trame_caption append chunks chunks. append captiop)aptio
f"Frane (1). (i)),
murr_chunssccuteee
www.uuunnnaaatn_ne_ee (inages)
Goal finding wrt an user instruction:
#atitle finding
instruction
user instriistttttion "Where is the office?" #8paran (type:
stagei_toxt is the rax genini response, contents is the pronpt sent Gemini stagei_ text, contents print Genini Rav Response: (eoooneeen(tttge_teex]]]]
(user_instruction tour nour_chunks_custon
gge-_goal_ (stagel_ print (f Closest frane indeg: iolexNafee (\(dx))))) inages [idx]
22


Reviews:
[{"review_id": "Bwo0xW39DV", "reviewer": "Reviewer_395i", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces  the multimodal instruction navigation with demonstration tours. Unlike usual VLN problems, the MINT  challenge focuses on multimodal instructions (text+ images). To solve this navigation task, the authors proposed mobility VLA, combining a Multimodal large language model to predict the image frame corresponding to the instruction, and a  low-level topological navigation algorithm to generate waypoints.", "summary_of_recommendation": "Mainly an engineering work where the performance depend only on a specific MLLM.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 1, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "The rebuttal questions are provided above.\n\nReview update: Based  on the interactions with the authors, the reviewer changed the recommendation to weak reject. The reviewer is convinced by the relevance of the work, but not by the contribution. For such a work showing a different use of MLLMs, the reviewer would have expected  a more thorough experimental study to compensate the lack of technical novelty.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4yUfANJuMf", "reviewer": "Reviewer_21ra", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces Mobility VLA, a hierarchical navigation policy for Multimodal Instruction Navigation with demonstration Tours (MINT). Mobility VLA leverages long-context Vision-Language Models (VLMs) to interpret demonstration tour videos and multimodal user instructions, identifying goal frames within the video by using interleaved text and image prompts to match the user’s request to frames from the tour. Offline-generated topological graphs, created using structure-from-motion techniques like COLMAP to establish 6-DoF camera poses for each frame, capture the environment’s connectivity based on the tour trajectory. These graphs guide the robot’s actions at each timestep by mapping the current observation to the closest vertex and computing waypoints using the shortest path to the goal. This approach improves robot usability by allowing natural language and image instructions, making navigation tasks more intuitive and effective. Evaluated in a real-world office environment, Mobility VLA demonstrated high success rates in complex navigation tasks, enhancing human-robot interaction.", "summary_of_recommendation": "The paper presents an innovative approach, Mobility VLA, which combines long-context Vision-Language Models (VLMs) and topological graphs for handling multimodal instruction navigation using demonstration tours.  While the technical methodology is sound and the initial results are promising, the paper requires significant revisions to address several weaknesses. These include a vague contribution statement, insufficient experimental details, lack of comprehensive comparisons with alternative methods, and a need for more in-depth explanations of the VLM and topological graph usage. Additionally, the paper should include a user study to quantify the system's intuitiveness, analyze the impact of context length in VLMs, and provide a more detailed literature review. Addressing these issues will greatly strengthen the paper and support its claims. With these improvements, the paper could become a strong candidate for publication in CoRL.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Clarify the specific type of navigation problem addressed by Mobility VLA, such as ObjNav or VLN. - Provide a more technical and precise contribution statement. \n- Expand the experiments to include a larger variety of instructions and environments. Evaluate more than five instructions per category to better demonstrate the system’s robustness and generalizability. \nProvide a more detailed explanation of how VLMs are used to identify goal frames, including the creation of prompts, processing of inputs, and the decision-making process. \n- Elaborate on the generation of topological graphs using structure-from-motion techniques and how they are used in the low-level navigation policy. \n- Include a comparison with traditional semantic mapping and navigation methods. Discuss whether these methods could solve the MINT problem and highlight the pros and cons of using video recordings instead. \n- Conduct a user study to quantify the intuitiveness of human-robot interaction with Mobility VLA compared to traditional approaches. Provide a statistical analysis of these results. \n- Explain how the length of the context in VLMs impacts the performance of Mobility VLA. Provide quantitative analysis to illustrate this relationship. \n- Provide a more detailed and comprehensive literature review on existing object search navigation methods using VLMs, as well as recent target-driven navigation approaches using other learning methods such as reinforcement learning. Highlight the strengths and weaknesses of these approaches in comparison to Mobility VLA. Here is an example: NavFormer: A transformer architecture for robot target-driven navigation in unknown and dynamic environments, 2024", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cT1brRfLTY", "reviewer": "Reviewer_y2wr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach to using vision-language models (VLMs) for a task the authors call \"multimodal instruction navigation with demonstration tours\" (MNIT), in which a robot is given a video of an initial tour of a (large) space and an instruction (e.g., \"where can I put this away\" + a picture of the object of interest) and must indicate the frame in the tour video corresponding to the location the robot must travel. To achieve this aim, the authors first use the immense context length of SOTA VLMs and feed frames from throughout the tour video, ~950 for the experiments in the paper, and ask the language model to specify which frame most relevant for solving the query. Using that frame as a target coordinate, the robot then plans and follows a series of waypoints in a topological map built offline via structure from motion on the tour video. Experiments show impressive performance, as the robot achieves 90% success rate on high-level tasks.", "summary_of_recommendation": "Overall, I believe this to be a solid experimental contribution and a well written paper. The results are impressive and demonstration on a real robot is commendable. The theoretical contribution is a bit small, but I think the strength of the physical experiments alone are sufficient to compensate for this.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- For a system with 90% success rate and navigation via a topological graph of the environment, one might expect that the SPL be very close to 0.9, but is instead 0.84. Can the authors clarify where the other 0.06 SPL was lost or how the SPL was computed? Is it merely due to imperfect localization or navigation on the physical platform?\n- What are the types of failure modes exhibited by the most powerful system (the one using Gemini 1.5 Pro)? 90% success rate is impressive, but it would be informative to understand the sorts of failures that represent the other 10%. Does the VLM return a frame in the wrong location or fail to suggest a frame at all? If it suggests incorrect frames, how far off are they typically? Are they usually nearby the intended goal or far off? Quantitative results are unnecessary for this, as I think it would be most useful to build the reader's intuition with a qualitative discussion.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: # Strengths
1. The proposed problem is highly relevant to the field.
1. The approach to combining VLMs and topological graphs is interesting.
1. The real-world demonstrations showcase the effectiveness of the approach.
1. The paper is well-written and well-structured.

# Weaknesses
1. The proposed approach relies mainly on existing technology.
1. The theoretical contribution to robot learning is limited.
1. The experimental details, such as datasets, are insufficiently explained.
1. Comprehensive analysis and comparisons with conventional methods are insufficient.

### Post-rebuttal comment
The reviewers initially raised a few concerns. The authors provided a detailed response to each of these points. The reviewers agree that most of the concerns have been addressed. In their final scores, the reviewers are almost in agreement on the quality and acceptance. I support their shared opinion.
Decision: Accept