LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action
Authors: Dhruv Shah
Błażej Osiński
brian ichter
Sergey Levine
Venue: CoRL 2022 Poster (2022)
Keywords: instruction following
language models
vision-based navigation

Abstract: Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world settings. However, particularly in vision-based settings where specifying goals requires an image, this makes for an unnatural interface. Language provides a more convenient modality for communication with robots, but contemporary methods typically require expensive supervision, in the form of trajectories annotated with language descriptions. We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories, while still providing a high-level interface to the user. Instead of utilizing a labeled instruction following dataset, we show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data. LM-Nav extracts landmarks names from an instruction, grounds them in the world via the image-language model, and then reaches them via the (vision-only) navigation model. We instantiate LM-Nav on a real-world  mobile robot and demonstrate long-horizon navigation through complex, outdoor environments from natural language instructions.

LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action
Dhruy Shah Blažej Osińskit&s, Brian Ichter", Sergey Levine UC Berkeley, "University of Warsaw, "Robotics at Google
Abstract: Goal-conditioned policies for robotic navigation can be trained on large, unannotated datasets, providing for good generalization to real-world set- tings. However, particularly in vision-based settings where specifying goals re- quires an image, this makes for unnatural interface. Language provides more convenient modality for communication with robots, but contemporary methods typically require expensive supervision, in the form of trajectories annotated with language descriptions. We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training unannotated large datasets of trajectories, while still providing a high-level interface to the user. Instead of utilizing a labeled instruction following dataset, show that such system be constructed en- tirely out of pre-trained models for navigation (ViNG), image- associa- tion (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data. LM-Nav extracts landmarks names from an instruction, grounds them in the world via the image-language model, and then reaches them via the (vision-only) navigation model. We instantiate LM-Nav on a real-world mobile robot and demonstrate morizon navigation through com- plex, outdoor environments from natural language instructions.
Introduction
One of the central challenges in robotic learning is to enable robots to perform wide variety of tasks on command, following high-level instructions from humans. This requires robots that can understand human instructions, and are equipped with large repertoire of diverse behaviors to execute such instructions the real world. Prior work on instruction following in navigation has largely focused on learning from trajectories annotated with textual instructions [1-5]. This enables understanding of textual instructions, but the cost of data annotation impedes wide adoption. On the other hand, recent work has shown that learning robust navigation is possible through goal- conditioned policies trained with self-supervision. These utilize large, unlabeled datasets to train vision-basce based controllers via hindsight relabeling [6-11]. They provide scalability, generalization, and robustness, but usually involve a clunky mechanism for goal specification, using locations or images. In this work, aim to combine the strengths of both approaches, enabling robotic navigation system to execute natural language instructions by leveraging the capabilities of pre- trained models without any annotaeed navigational data. Our method uses these models to construct an "interface" that humans can use to communicate desired tasks to robots. This system enjoys the impressive generalization capabilities of the pre-trained language and vision-language models, enabling the robotic system to accept complex high-level instructions.
Our main observation is that of visual and language datasets - that are widely available and show great few-shot generaliza- tion capabilities to create this interface for embodied instruction following. To achieve this, combine the strengths of two such robot-agnostic pre-trained models with a pre-trained navigation visual navigation model (VNM: ViNG [11]) to create topological "mental map"
utilize off shel- pre-trained models trained on large corpora
model. We of the environment using the robot's observations from a prior exploration of the environment. Given free-form textual instructions, use a pre-trained large language model (LLM: GPT-3 [12]) de- code the instructions into a sequence of textual landmarks. We then use a vision-language model
These authors contributed equally, order decided by a coin flip. Check out the project page for experiment videos, code, and user- friendly Colab notebook that runs in your browser: sites, google. com/ cwm/viev/ wive/lmmee
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

After passing ti take right next to vilt toads Then take left and go lomartt dh a asuee with inn Go farther,untly untilyy. atddataaalles
Freee Weiges
LLM
Free-form Navigation Instructions
VLM
gool
Total 350m
VNM
Observations Target Environment
Figure 1: Embodied instruction following with LM-Nav: Our system takes as imput a set of raw observations from the target environment and free-ford form textual instructions (left), deriving actionable plan using three pre- trained models: large language model (LLM) for extracting landmarks, vision-and and-language model (VLM) for grounding, and a visual navigation model (VNM) for execution. This enables LM-Nav to follow textual instructions in complex environments purely from visual observations (right) without any fine teeittning..
(VLM: CLIP [13]) for grounding these textual landmarks in the topological map, by inferring joint likelihood over the landmarks and nodes. A novel search algorithm is then used to plan a path for the robot, which is then executed by VNM. While reducing the task of language following to combination of grounding and subgoal selection discards a lot of useful cues such as relations and verbs, we find that it is still sufficient to follow a variety of natural language instructions.
Our primary contribution is Large Model Navigation, or LM-Nav, an embodied instruction follow- ing system that combines three large independently pre-trained models robotic control model that utilizes visual observations and physical actions ( (VNM), a vision- laniun- model that grounds images in text but has context of embodiment (VLM), and a large language model that can parse and translate text but has no sense of visual grounding or embodiment (LLM) - to enable long- horizon instruction following in complex, real- world environments. We present the first instantiation aitoo-and-langweee
of a robotic system that combines the confluence of pre-trained goal-conditioned controller, derive actionable plans without any fine-tuning in the targel environ- ment. Notably, all three models are trained on large-scale datasets, with self- self-supervised objectives, and used off she-the-shef with no fine-tuning no human annotations of the robot navigation data are necessary to train LM-Nav. We show that LM-Nav is able to successfully follow natural language instructions in pre-explored environments over the course of 100s of meters of complex, suburban navigation, while disambiguating paths with fine-graines commands.
models with a
Related Work
Early works in augmenting navigation policies with natural language commands use statistical ma- chine translation [14] to discover data-driven patterns to map free-form commands guage defined by a grammar [15-19]. However, these approaches tend to operate on structured state spaces. Our work is closely inspired by methods that instead reduce this task to sequence predic- tion problem [1, 20, 21j. Notably, our goal is similar to the task of VLN - leveraging fine-grained instructions control mobile robot solely from visual observations [1, 2].
formal lan-
However, most recent approaches to VLN use a large dataset of simulated trajectories - over 1M demonstrations annotated with fine-grained language labels in indoor [1, 3-5, 22] and driv- ing scenarios [23-28], and rely on sim-to-real transfer for deployment in simple indoor environ- ments [29, 30). However, this necessitates building photo-realistic simulator resembling the target environment, which can be challenging for unstructured environments, especially for the task of outdoor navigation. Instead, LM-Nav leverages free-form textual instructions to navigate robot in complex, outdoor environments without access to any simulation or any trajectory-Ievel annotations.
Recent progress in using large-scale models of natural language and images trained on diverse data has enabled applications in wide variety of textual [31-33], visual [13, 34 34-38], and embodied domains [39.44]. In the latter category, approaches either fine-tune embeddings from pre-trained models on robot data with language labels [39, 40, 44], assume that the low-level agent can execute textual instructions (without addressing control) [41], or assume access to a set of text-conditioned skills that can follow atomic textual commands [42]. All of these approaches require access to low- level skills that can follow rudimentary textual commands, necessitating language annotations for

---

robotic experience and strong assumption on the robot's capabilities. In contrast, we combine these pre-trained vision and language models with pre-trained visual policies that do not use any language annotations [1 1, 45] without fine-tuning these models for the task of VIN.
Data-driven approaches to vision-based mobile robot navigation often use photorealistic simula- tors [46-49] or supervised data collection [50] to learn goal-reaching policies directly from raw observations. Self-supervised methods for navigation [6-11, 51] instead can use unlabeled datasets of trajectories by automatically generating labels using onboard sensors and hindsight relabeling. While such policies are adept at navigating to goal locations or images, they may be unable to parse high-level instructions such as free-form text. LM-Nav uses self-supervised policies trained in a large number of prior environments, augmented with pre-trained vision and language models for parsing natural language instructions. and deploys them in novel real-world environments without any fine-tuning. We emphasize that while LM-Nav relies on a pre-built topological graph, similar to prior work [11, 51, 52], this assumption may be relaxed by incorporating exploration heuristics in unseen environments [53], and can be an interesting avenue for future work.
3 Preliminaries
LM-Nav consists of three large, pre- trained models for processing lan- guage, associating images with lan- guage, and visual navigation.
65
Large language models are genera- tive models of text trained on large corpora of internet text using self- supervised learning. LM-Nav uses the GPT-3 LLM [12] to parse instruc- tions into a sequence of landmarks.
04 C. VIM
Figure 2: L.-Nav uses CLIP to infer a joint distribution over textual landmarks and image observations. VNM infers a goal- conditioned distance function and policy that can control the robot.
-aaanuae models refer to models that can associate images and text, e.g. image cap- tioning, visual question-and -answiinn.nswering, etc. [54-56]. We use the CLIP VLM [13], model that jointly encodes images and text into a shared embedding space, to jointly encode a set of landmark descrip- tions obtained from the LLM and set of images i& to obtain their VLM embeddings {T, 4x (see Fig. 3). Computing the cosine similarity between these embeddings, followed by a softmax operation results in probabilities ((t corresponding to the likelihood that image i& corresponds to the string t. LM-Nav uses this probability to align landmark descriptions with images.
Visual navigation models learn navigational affordances directly from visual observations [11, 51, 57-59), associating images and actions through time. We use the ViNG VNM [11], a goal- conditioned model that predicts temporal distances between pairs of images and the corresponding actions to execute (see Fig. 3). The VNM serves two purposes: (i) given a set of observations in the target environment, the distance predictions from the VNM can be used to construct a topo- logical graph G(V, that represents a "mental map" of the environment; (ii) given a "walk" (i.e., sequence of connected subgoals the goal), VNM can control the robot along this plan. The topological graph G is an important abstraction that allows a simple interface for planning over past experience in the environment and has been successfully used in prior work to perform long-horizon navigation [52, 53, 60]. To deduce connectivity in g, we use combination of learned distance estimates, temporal proximity (during data collection), and spatial proximity (using GPS measure- ments). For more details on the construction of this graph, see Appendix B.
4 LM-Nav: Instruction Following with Pre-Traines Treieed Models
LM-Nav combines the components discussed earlier to follow natural language instructions in the real world. The LLM parses free-form instructions into a list of landmarks é (Sec. 4.2), the VLM associates these landmarks with nodes in the graph by estimating the probability that each node corresponds to each E, P( (Sec. 4.3), and the VNM is used to infer how effectively the robot can navigate between each pair of nodes in the graph, denoted by a probability P(u,, To find the optimal "walk" on the graph that both (i) adheres to the provided instructions and (ii) minimizes traversal cost, we derive a probabilistic objective (Sec. 4.1) and show how it can be optimized using graph search algorithm (Sec. 4.4). This walk is executed in the real world by the VNM model.
3

---

P(v. (2) Construct Graph
(d) Graph Search
VNM
Distance D
Ground Landmarks
Observadore in Target Environment
VLM
(el Execute Plan
Extract Landmarks
Stop mesiin
Blue Iieomppss Blua Iasoiaad
Poniceen
Go towards park banch, take dumpeter, stop truck.
1. Ploric sign dumpoter
sign,
VNM
Free-form Instructions
Figure System overview: (a) VNM uses goal-conditiones distance function to infer connectivity between the set of raw observations and constructs a topological graph. (b) LLM translates natural language instruc- tions into a sequence of textual landmarks. (c) VLM infers a joint prohability distribution over the landmark descriptions and nodes in the graph, which is used by (d) graph search algorithm to derive the optimal walk through the graph. (c) The rohot drives following the walk in the real world using the VNM policy.
4.1 Problem Formulation
Given a sequence of landmark descriptions ē= L1, f2... ln extracted from the language command, method needs to determine sequence of waypoints 0 01 u2,.... Uj. command to the robot. Typically, k 2 n, since each landmark needs to be visited, but the traversal might require other waypoints in between the landmarks. Finding can formulated as a probabilistic inference problem. A key element in this formulation is access to a distribution p(v:(e)) for each graph vertex i and landmark description Eg. Recall that the graph vertices correspond to images observed by the robot, and thus, p(0% (a) represents a distribution over images given a language description. This can be obtained from the VLM. Intuitively, the full likelihood that need optimize determine the robot's plan will now depend on two terms: likelihoods of the form p( Ur, (&s) that describe how likely vr, is correspond to & for assignment L, l.,...y. bn and traversability likelihoods p( U5+1 that describe how likely is the robot to be able to reach e/+1 from 04.
use variety of traversability likelihood functions, a simple choice is to use dis- counted Markovian model, where the discount 7 models the probability of exiting at each time step, leading to a termination probability of y each step, and probability of reaching t1+1 given by 7 Pl),, where (v,,, U++1) is the estimated number of time steps the robot needs to travel from & to 2/+1, which is predicted by the VNM. While other traversability likelihoods could also be used, this choice is convenient consequence of condiiiiond reinforcement learning formula- tions [61, 62], and thus, the log- likeeloodood corresponds to (v,,, 1.+1). We can use these likelihoods to derive the probability that a given sequence can be traversed successfully, which denote with the auxiliary Bernoulli random variable C3 (i.e., Cp = implies that ö was traversed successfully):
While
P(c, = 1 = II
P(u., ((T1) = II 15K<T
(1)
151TT
The full likelihood used for planning is then given by:
P(success|j 8) x P(ce = 110) P(ol) = II
II 15i5n
max S5.
1Sj<k
4.2
Parsing Free-Form Textual Instructions
The user specifies the route they want the robot take using natural language, while the objective above is defined in terms of sequence of desired landmarks. To extract this sequence from the user's natural language instruction we employ a large language model, which in our prototype is GPT-3 [12]. We used a prompt with 2 examples of correct landmarks' extractions, followed by the description to be translated by the LLM. Examples of instructions and landmarks extracted by the model can be found in Fig. 4. The prompt was selected to disambiguate nuanced cases, e.g. when order of landmarks in the text different than in the expected path (see example Fig. a). For details of the "prompt engineering" please Appendix A.

---

4.3 Visually Grounding Landmark Descriptions
As discussed Sec. 4.1, a crucial element of selecting the walk through the graph is computing P(u,({)). the probability that land- mark description U; refers node l (see Eqn. 2). With each node containing an image taken during initial data collection, the prob- ability can be computed using CLIP [13] in the way described in Sec. 3 as the retrieval task. As presented in Fig. 2, we apply CLIP to the image at node v and caption prompt in the form of "This is photo of a [ty]". To go from CLIP model outputs, which are logits, to probabilities we use P())(()) exp CLIP(e;,())
Algorithm 1: Graph Search
1: Input: Landmarks (f1, Ln).
2: Input: Graph (VV E).
3: Input: Starting node S.
4: Vi=0,..., Q u] -00 .EV
Q[0,5]=0 =0
6: Dijkstra_algaen alg.rithoooommtet.o ([0(0++))
7: for in 1. 1.2..... ..,.... do
8: Vu E VQli, = Q/i (1)
9: Dijkstra algorithmi Q[:
10: end for
11: destination = arg max (Q[n, +])
12: return backtrack (rrcktrack(esstinatiion. Q m, +))
Eeev CXP The resulting probability P())le)) together with the inferred edges' distances will be used to select the optimal walk.
4.4 Graph Search for the Optimal Walk
As described Sec. 4.1, LM-Nav aims at finding walk ö (V,, U2..... U2) that maximizes the probability of successful execution of that adheres to the given list of landmarks &. We can define a function R(o,1 1) for monotonically increasing sequence of indices i = (1,, (,,, bn):
R(U,)) = Σ CLIP(
Σ D(e 21+1), where a -log 7.
(3)
R has the property that (i) maximizes P(success/ū 8) defined in Eqn. 2, if and only if there exists I such that (e, maximizes R. In order to find such (v, f), employ dynamic programming. In particular we define helper function Q(i, u) for E (0, 1,...,n.. V:
Q(i,v)
max (P2 1, Em (21.22.))))))
R(o,t).
(4)
vii,v) represents the maximal value of R for walk ending in e that visited the landmarks up to index i. The base case Q(0, v) visits none of the landmarks, and its value of R is simply equal to minus the length of shortest path from the starting node S. For > 0 we have:
Qii max Q(( - 1,u) + CLIP(e, 41),
max ur Eneiighhoos (1) Q(i,w)
D(v,w))
(5)
The base case for DP is to compute Q00, V). Then, in each step DP i = 1,2, ... n we compute Q(i, v). This computation resembles the Dijkstra algorithm ([63]). In each iteration, we pick the node with the largest value of Q(i, v) and update its neighbors based on the Equ. 5. Algorithm summarizes this search process. The result of this algorithm is a walk & (01, ..., U*) that maximizes the probability of successfully carrying out the instruction. Such walk can be executed by VNM, using its action estimates to sequentially navigate these nodes.
5 System Evaluation
We now describe our experiments deploying LM-Nav in variety of outdoor settings to follow high- level natural language instructions with small ground robot (Clearpath Jackal UGV platform see Fig. 1(right) for image and Appendix C for details). For all experiments, the weights of LLM, VLM, and VNM are frozen - there is no fine-tuning or annotation in the target environment. We evaluate the complete system, as well as the individual components of LM-Nav, to understand its strengths and limitations. Our experiments demonstrate the ability of MM-Nav to follow high-level instructions, disambiguate paths, and reach goals that are up to 800m away.
5

---

Goal
Landmarks
Take right next to stop stan, Look (a) for a alasa buildins, after passing by abttecat
Follww the rood take right you Mue semt--ncd tootfi cone Co aowords but dunashtr a take feft, Look for auCEE Beoch
(b)
Stare gakng sm buliding with porcity? Continu dra gre and take a right. when mnaholee steet forword and left and pi2lis
(c)
Take
Figure 4: Qualitative examples of LM-Nav in real-worke environments executing textual instructions (left). The landmarks extracted by LLM (highlighted in text) are grounded into visual observations by VLM (center; overhead image not available to the robot). The resulting walk of the graph is executed by VNM (right).
5.1 Following Instructions with LM-Nav
In each evaluation environment, we first construct the graph by manually driving the robot and col- lecting image and GPS observations. The graph constructed automatically using the VNM to pre- dict relative distances between images in these trajectories. We tested our system on 20 queries in environments, corresponding to combined length over 6km. The instructions include prominent landmarks that can be identified from the robot's observations, e.g., buildings and stop signs.
Fig. 4 shows qualitative examples of the path taken by the robot. Fig. 4(a). LM-Nav able to successfully localize the simple landmarks from its prior traversal and find short path to the goal. While there are multiple stop signs in the environment, the objective in Eqn. 2 causes the robot to pick the correct one, minimizing overall trajectory length. Fig. 4(b) highlights LM-Nav's ability to follow complex instructions with multiple landmarks despite the possibility of taking a shorter route directly to the final landmark, the robot follows path that correctly visits all of the landmarks.
Missing landmarks. While LM-Nav is effective finding a path through landmarks extracted from in- structions, it relies the assumption that the land- marks (i) exist in the environment, and (ii) can be identified by the VLM. Fig. 4(c) illustrates a case where the executed path fails to visit one of the landmarks - fire hydrant - and takes a path that goes around the top of the building rather than the bottom. This failure mode is attributed to the the inability of the VLM to detect fire hydrant from robot's observations. ing the efficacy of the VLM at retrieving landmarks (see Sec. 5.3), we find that despite being the best off-the-shell model for our task, CLIP is unable to retrieve a small number of "hard" landmarks, in- cluding fire hydrants and cement mixers. In many practical cases, the robot is still successful in find- ing path that visits the remaining landmarks.
After passing white building. white Then take left and go towards with larze further until you stop sigs
white Continue straight
building
right
passing tritt you
independently evaluat-
Start
Goal
Landmarks
Figure 5: LM-Nav can successfully disambiguate instructions with same start-goal locations that dif- fer slightly. The landmarks are underscored in text and their locations marked with pins.
Disambiguation with instructions. Since the objective of LM-Nav is to follow instructions, and not merely to reach the final goal, different instructions may lead to different traversals. Fig. 5 shows an example where modifying the instruction can disambiguate multiple paths to the goal. Given the shorter prompt (blue), LM-Nav prefers the more direct path. On specifying a more fine-graines route (magenta), LM-Nav takes an alternate path that passes a different set landmarks.
5.2 Quantitative Analysis
To quantify the performance of LM-Nav, we introduce the following metrics. A walk found by the graph search is successful, (1) it matches the path intended by the user or (2) if the landmark im- ages extracted by the search algorithm contain said landmarks (i.e. if the path visits landmarks with

---

System
Environment
Net Success
Efficiency
# Diseng- J
Planning
GPS-Nav (No VNM)
Envšmall-10
0.23
0.93
0.75
0.9
EnvSmall-10
0.8
0.96
0.1
0.9
LM-Nav (Ours)
EnvLarge-10
0.8
0.89
0
0.8
Table Quantifying navigational instruction following with LM-Nav over 20 experiments. LM-Nav successfully plan path to the goal, follow it efficiently, over 100s of meters. Ablating VNM (GPS-Nav) severely hurts performance due to frequent disengagements inability to reason about collisions with obstacles.
LLM Candidate
Avg. Extraction Success
VLM Candidate
Detection Rate
Noun Chunks
0.88
fairseq-1.30 [64]
0.52
Dester-RCNN [67]
0.07
fairseq- 13B [64]
0.76
ViLD [36]
0.38
GPT-J-6B [65]
0.80
CLIP-ViT [13]
0.87
GPT-NeoX-208B [66]
0.72
GPT-3 [12]
Table 3: CLIP-ViT produces the most reliable
Table 2: GPT-3 consistently outperforms alternatives in parsing free-form instructions into landmarks.
landmark detections from visual observations.
the same description, even if not exactly the same). Planning success the fraction of successful walks found by the search algorithm. Efficiency of walk is defined the ratio of the lengths of the described route and the executed one; the value is clipped at a maximum of to account for the cases when the LM-Nav executes path shorter than the user intended. For of queries, report the average efficiency over successful experiments. The planning efficiency is similarly defined as the ratio of the length the described and planned routes. Finally, number of disengagements is the average number of human interventions experiment due to unsafe maneuvers.
Table summarizes the quantitative performance of the system over 20 instructions. LM-Nav gen- erates a successful walk for 85% them, and causes disengagement only once (an average of intervention per 6.4km of traversals). Investigating the planning failure modes suggests that the most critical component of our system is the ability of VLM to detect certain landmarks, e.g. fire hydrant, and challenging lighting conditions, e.g. underexposed images.
5.3 Dissecting LM-Nav
To understand the influence of each of the components of LM-Nav, we conduct experiments to evaluate these components in isolation. For more details about these experiments, see Appendix D.
To evaluate the performance of LLM candidates in parsing instructions into an ordered list of land- marks, we compare GPT-3 (used by LM-Nav) to other state-of-the-ard pre-trained language models fairseq [64], GPT-J-6B [65], and GPT-NeoX-20B [66] - as well as a simple baseline using spaCy NLP library [68] that extracts base noun phrases, followed by filtering. In Table we report the average extraction success for all the methods on the 20 prompts used in Section 5.2. GPT-3 sig- nificantly outperforms other models, owing to its superior representation capabilities and in-context learning [69]. The noun chunking performs surprisingly reliably, correctly solving many simple Appendix D.2.
prompts. For further details these experiments,
To evaluate the VLM's ability to ground these textual landmarks in visual observations, we up an object detection experiment. Given an unlabeled image from the robot's on-board camera and set of textual landmarks, the task retrieve the corresponding label. We run this experiment on a set of 100 images from the environments discussed earlier, and a set of 30 commonly-occuree landmarks. These landmarks are combination of the landmarks retrieved by the LLM our
AavSmall-10
EnvLarge-10 Pl. Efficiency 1
Planner
Pl. Success 1 Pl. Efficiency 1
Pl. Success 1
Max Likelihood
0.6
0.69
0.2
0.17
LM-Nav (Ours)
0.9
0.80
0.8
0.99
Table Ablating search algorithm (Sec. 4.4) gives max likelihood planner that ignores reachability information, resulting inefficient plans that are up to 6x longer than LM-Nav the same instruction.
7

---

experiments from Sec. 5.1 and manually curated ones. We report the detection successful if any of the top 3 predictions adhere to the contents of the image. We compare the retrieval success of our [67, 70], a state-of-the- art object detection model pre-trained on MS-COCO [71, 72], and ViLD [36]. an open-vocabulary object detector based on CLIP and Mask-RCNN [73]. To evaluate against the closed-vocabulary baseline, we modify the setup by projecting the landmarks onto the set of MS-COCO class labels. We find that CLIP outperforms baselines by wide margin, suggesting that its visual model transfers very well to robot observations (see Table 3). Despite deriving from CLIP, ViLD struggles with detecting complex landmarks like "manhole cover" and "glass building". FaNNer-RCNN is unable to detect common MS-COCO objects like "traffic light". "person" " and "stop sign", likely due to the on-board images being diitti--ittiibutionn for the model.
VLM (CI.IP) with some object detection alternatives - Paster-RCNN-FIE
To understand the importance of the VNM, we run an ab- lation experiment of LM-Nav without the navigation model. Using GPPS-ased distance estimates and a naive straight line controller between nodes of the topological graph. Table I summarizes these results - without VNM ability to reason about obstacles and traversability, the system frequently runs into small obstacles such as trees and curbs, resulting in fail- ure. Fig. 6 illustrates such a case while such controller works well on open roads, it fails to reason about connec- tivity around buildings or obstacles and results in collisions with a curb, a tree, and a wall in 3 individual attempts. This illustrates that using learned policy and distance function from the VNM is critical for LM-Nav to successfully navi- gate in complex environments.
bau
Start
Goal
X
Coll
Figure GPS-Nav (red) fails to exe- cute a plan due to its inability to rea- son about traversability through obsta- cles, while LM-Nav (bluc) succeeds.
Lastly, to understand the importance of the two components of the graph search objective (Eqn. 3), we ran set of ablations where the graph search only depends on P(ölé), i.e. Max Likelihood Plan- ning, which only picks the most likely landmark without reasoning about topological connectivity or traversability. Table 4 shows that such planner suffers greatly in the form of efficiency, because it does not utilize the spatial organization of nodes and their connectivity. For more details on these experiments, and qualitative examples, see Appendix D.
Discussion
We presented Large Model Navigation, a robotic system that can execute textual instructions in the real-world without requiring any human annotations for navigation trajectories. LM-Nav combines three pre-trained models: the LLM, which parses instructions into a list of landmarks; the VLM, which infers joint probabilities between these landmarks and visual observations from the environ- ment; and the VNM, which estimates navigational affordances (distances between landmarks) and control actions. Each model is pre-trained its own dataset, and we show that the complete system can execute a variety of user-specified instructions in real-world environments - choosing the cor- rect sequence of landmarks by leveraging language and spatial context and handle mistakes (such as missing landmarks). We also analyze the impact of each pre-trained model on the full system.
Limitations and future work. The most prominent limitation of LM-Nav is its reliance on land- marks: while the user can specify any instruction they want, LM-Nav only focuses on the landmarks and disregards any verbs, propositions, adverbs, etc. (e.g., "go straight for three blocks" or "drive past the dog very slowly"), which can be lossy. Grounding such nuances is an important direction for future work. Additionally, LM-Nav uses VNM that is specific to outdoor navigation with the Jackal robot, which limits wider adoption for other robot embodiments and sensor suites. An ex- citing direction for future work would be to swap in "general navigation model" [74] that can be utilized broadly across robots, analogous to how the LLM and VLM handle any text or image. In its current form, LM-Nav provides simple and attractive prototype for how pre-trained models can be combined to solve complex robotic tasks, and illustrates that these models can serve as an "interface" to robotic controllers that are trained without any language annotations. One of the implications of this result is that further progress on self-supervised robotic policies (c.g.. goal-conditioned policies) can directly benefit instruction following systems. More broadly, understanding how modern pre- trained models enable effective decomposition of robotic control may enable broadly generalizable systems in the future, and we hope that LM-Nav will serve as a step in this direction.
8

---

Acknowledgments
This research was supported by DARPA Assured Autonomy, DARPA RACER, Toyota Research Institute, ARL DCIST CRA W911NF-11-2-0183 and AFOSR. BO was supported by the Fulbright Junior Research Award granted by the Polish-U.S. Fulbright Commission. We would like to thank Alexander Toshev for pivotal discussions early stages of the project. We would also like to thank Kuan Fang, Siddharth Karamcheti, Albertyna Osiáska, Vijay Badrinarayanan, and Philip J. Ball for useful discussions and feedback.
References
[1] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel. Eision-and-langen navigation: Interpreting visually-groune grounded naviga- tion instructions in real environments. In IEEE Conference Computer Vision and Pattern Recognition, pages 3674-3683, 2018. 1,2
[2] Gu, E. Stefani, Q. Wu, Thomason, and X. Wang. Vision-and-languaar navigation: A survey of tasks, methods, and future directions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022.
[3] A. Ku, P. Anderson, R. Patel, E. le, and J. Baldridge. Room-Across-Rond visionaand-languae navigation with dense spatiotemporal grounding, In Conference on Em-
Multilingual
pirical Methods for Natural Language Processing (EMNLP). 2020. 2
[4] V. Jain, G. Magalhaes, A. Ku, A. Vaswani, E. Ie, and J. Baldridge. Stay the path: Instruction fidelity in vision-and- and-language navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[5] A. Yan, X. E. Wang, J. Feng, L. Li, and W. Y. Wang. Cross Cross-lingual vision-language navigation, 2019. 1.2
[6] T. Manderson, C. Gamboa, S. Wapnick, J. Tremblay, H. Zhao, F. Shkurti, D. Meger, and G. Dudek. Self-supervised, goal-conditioned policies for navigation in unstructured environ- ments. 2010. 1,3
[7] B. Sofman, E. L. Ratliff, J. A. D. Bagnell, J. Cole, N. Vandapel, and A. T. Stentz. Improving robot navigation through self-supervised online learning. Journal of Field Robotics: Special Issue Machine Learning Based Robotics in Unstructured Environments, 2006.
[8] D. Gandhi, L. Pinto, and A. Gupta. Learning to fly by crashing. In 2017 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), 2017.
[9] A. Kouris and C.-S. Bouganis. Learning to fly by myself: A self-supervised cnn-based ap- proach for autonomous navigation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2018.
[10] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-Supervised Deep RL with Generalized Computation Graphs for Robot Navigation. In IEEE International Conference on Robotics and Automation (ICRA), 2018
[11] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. ViNG: Learning Open- World Navigation with Visual Goals. IEEE International Conference on Robotics and Automation (ICRA), 2021. 3,
[12] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan,
P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan,
R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin,
Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. 3,
S.
[13] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, 2021. 2, 3, 5, 7,

---

[14] P. Koehn. Statistical Machine Translation. Cambridge University Press, 2009. 2
[15] Y. W. Wong and R. Mooney, Learning for semantic parsing with statistical machine transla- tion. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, 2006. 2
[16] C. Matuszek, D. Fox, and K. Koscher. Following directions using statistical machine trans- lation. In 2010 5th ACM/IEEE International Conference on Human-Robo Interaction (HRI), 2010.
[17] D. L. Chen and R. Mooney. Learning interpret natural language navigation instructions from observations. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelli- gence, 2011.
[18] S. Tellex, T. Kollar, Dickerson, M. R. Walter, A. G. Banerjee, Teller, and N. Roy. Un- derstanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the twenty-Fifth AAAI Conference on Artificial Intelligence, 2011.
[19] C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to Parse Natural Language Commands to Robot Control System. 2013. 2
[20] N. Shimizu and A. Haas. Learning to follow navigational route instructions. In Proceedings of the st International Joint Conference on Artificial Intelligence. IJCAI'09, 2009. 2
[21] H. Mei, M. Bansal, and M. R. Walter. Listen, attend, and walk: Neural mapping of navigational instructions action sequences. AAAI, 2016.
[22] M. Shridhar, J. Thomason, D. Gordon, Y. Bisk, W. Han, R. Mottaghi, L. Zettlemoyer, and D. Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[23] H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi. Touchdown: Natural language navi- gation and spatial reasoning in visual street environments. In 2019 IEEE/CVE Conference
Computer Vision and Pattern Recognition (CVPR), 2019. 2
[24] K. M. Hermann, M. Malinowski. P. Mirowski. A. Banki- Horvath, K. Anderson, and R. Hadsell. Learning to follow directions in street view. CoRR, 2019.
[25] Mirowski, A. Banki-Horvath, Anderson, D. Teplyashin, K. M. Hermann, M. Malinowski, M. K. Grimes, K. Simonyan, K. Kavukcuoglu, A. Zisserman, and R. Hadsell. The streetlearn environment and dataset. CoRR, 2019.
[26] A. B. Vasudevan, D. Dai, and L. Van Gool. Talk2nav: Long-range rision-and-led -asoo-andd-annguage navi- gation with dual attention and spatial memory. Ini. Comput. Vision, 2021.
[27] D. K. Misra, A. Bennett, V. Blukis, E. Niklasson, M. Shatkhin, and Y. Artzi Mapping in- structions to actions in 3d environments with visual goal prediction. In E. Riloff, D. Chiang, Hockenmaier, and J. Tsujii, editors, Proceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, 2018.
[28] V. Blukis, N. Brukhim, A. Bennett, R. A. Knepper, and Y. Artzi Following high-level naviga- tion instructions on a simulated quadcopter with imitation learning. CoRR, 2018.
[29] Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision- and-language navigation in continuous environments. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII, 2020. 2
[30] P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee. Sim- to-real transfer for rision-and-Lan language navigation. Proceedings of the 2020 Conference on Robot Learning, 2021. 2
[31] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and Brew. Huggingface's transformers: State-of-the-art natural language processing. CoRR, 2019. 2
10

---

[32] R. Thoppilan, D. D. Freitas, J. Hall. N. Shazeer, A. Kulshreshtha, H. Cheng, A. Jin, T. Bos, L. Baker, Du, Y. Li, H. Lee, H. S. Zheng, A. Ghafouri, M. Menegali, Y. Huang, M. Krikun, D. Lepikhin, J. Qin, D. Chen, Y. Xu, Z. Chen, A. Roberts, M. Bosma, Y. Zhou, C. Chang, Krivokon, W. Rusch, M. Pickett, K. S. Meier-Hellstern, M. R. Morris, Doshi, R. D. Santos, T. Duke, J. Soraker, B. Zevenbergen, V. Prabhakaran, M. Diaz, B. Hutchinson, K. Ol- son, A. Molina, E. Hoffman-John, J. Lee, L. Aroyo, R. Rajakumar, A. Butryna, M. Lamm, V. Kuzmina, J. Fenton, A. Cohen, R. Bernstein, R. Kurzweil, B. Aguera-Arcas, C. Cui, M. Croak, E. H. Chi, and Q. Le. Lamda: Language models for dialog applications. CoRR, 2022.
[33] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Win- ter, P. Tillet, F. P. Such, D. Cummings, M. Plappert, F. Chantzis, E. Barnes, A. Merbert-Voss, W. H. Guss, A. Nichol, A. Paino, N. Tezak, J. Tang. I. Babuschkin, S. Balaji, S. Jain, W. Saun- ders, C. Hesse, A. N. Carr, J. Leike, Achiam, V. Misra, Morikawa, A. Radford, M. Knight, M. Brundage, M. Murati, K. Mayer. P. Welinder, B. McGrew, D. Amodei, S. McCandlish, I. Sutskever, and W. Zaremba. Evaluating large language models trained on code. CoRR, 2021. 2
[34] A. Ramesh, Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents, 2022. 2
[35] C. Saharia, W. Chan, S. Saxema, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022.
[36] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. In International Conference on Learning Representations, 2022. 8
[37] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the 38th International Conference on Machine Learning, 2021.
[38] H. Song, L. Dong, W. Zhang, Liu, and F. Wei. CLIP models are few-shot learners: Empirical studies on VQA and visual entailment. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022. 2
[39] M. Shridhar, I.. Manuelli, and D. Fox. Cliport:
What and where pathways for robotic manipu-
lation. In Proceedings of the 5th Conference on Robot Learning (CoRI.), 2021. 2
[40] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. BC-z: Zero-shot task gencralization with robotic imitation learning. In Sth Annual Conference on Robot Learning, 2021. 2
[41] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Ex- tracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
2
[42] M. Ahn, Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao. K. Rao, J. Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, Toshev, V. Vanhoucke, Xia, T. Xiao, P. Xu, S. Xu, and M. Yan. Do as i can, not as i say: Grounding language in robotic affordances. In arXiv preprint D.Xiv:2204.0169. 2022.
[43] A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, Lee, Vanhoucke, and Florence. Socratie models: Composing zero-shot multimodal reasoning with language. arXiv, 2022. 2
11

---

[44] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip em- beddings for embodied ai. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2
[45] D. Shah, B. Eysenbach, Rhinehart, and S. Levine. Rapid exploration for open-world navi- gation with latent goal models. In 5th Annual Conference on Robot Learning. 2021. 3
[46] Manolis Savva*, Abhishek Kadian*, Oleksandr Maksymets*, Y. Zhao, Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embod- ied AI Research, In IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
3
[47] Xia, A. R. Zamir, Z.-Y. Hc, A. Sax, J. Malik, and S. Savarese. Gibson env: real-world perception for embodied agents. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on, 2018.
[48] M. Savva, A. X. Chang, A. Dosovitskiy, T. Funkhouser, and V. Koltun. MINOS: Multimodal indoor simulator for navigation in complex environments. arXiv:1712.03931, 2017.
[49] E. Kolve, R. Mottaghi, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. A12-THOR: an interac- tive 3d environment for visual AI. CoRR, 2017. 3
[50] A. Francis, A. Faust, H. T. L. Chiang. J. Hsu, J. C. Kew, M. Fiser, and T. W. E. Lee. Long- Range Indoor Navigation With PRM-RL. IEEE Transactions on Robotics, 2020. 3
[51] N. Hirose, F. Xia, Martin-Martin, A. Sadeghian, and S. Savarese. Deep visual MPC-policy learning for navigation. IEEE Robotics and Automation Letters, 2019. 3
[52] X. Meng, N. Ratliff, Y. Xiang, and D. Fox. Scaling Local Control to Large-Scale Topological Navigation. In JEEE International Conference on Robotics and Automation (ICRA), 2020. 3
[53] D. Shah and S. Levine. Viking: Vision-based kilometer-scale navigation with geographic hints. In Robotics: Science and Systems (RSS), 2022. 3
[54] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Milli- can, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Mon- teiro, J. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals, A. Zisserman, and K. Simonyan. Flamingo: visual language model
for sho- learning, 2022. 3
[55] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and perfor- mant baseline for vision and language. In Arxiv, 2019.
[56] Y.-C. Chen, L. Li, L. Yu, A. E. Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In ECCV, 2020. 3
[57] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-Parametric Toplooooa Memory for Naviga-
tion. In International Conference on Learning Representations, 2018. 3
[58] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to Explore using Active Neural SLAM. In International Conference on Learning Representations (ICLR), 2020.
[59] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD- PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. In International Conference on Learning Representations (ICIR), 2020. 3
[60] Bruce, N. Sunderhauf, Mirowski, R. Hadsell, and M. Milford. Learning deployable nav- igation policies at kilometer scale from single traversal. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, Proceedings of The 2nd Conference on Robot Learning, 2018.
[61] L. P. Kaelbling, Learning to achieve goals. In IJCAI, pages 1094-1099, 1993. 4
12

---

[62] K. Hartikainen. X. Geng. T. Haarnoja, and S. Levine. Dynamical Distance Learning for Semi- Supervised and Unsupervised Skill Discovery. In International Conference on Learning Rep- resentations, 2020. 4
[63] E. W. Dijkstra. A note two problems in connexion with graphs. Numerische mathematik, 1959.
[64] M. Artetxe, S. Bhosale, N. Goyal, T. Mihaylov, M. Ott, S. Shleifer, X. V. Lin, J. Du, S. Iyer, R. Pasumuru, al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:21 2021." 7
[65] B. Wang and A. Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. http:///.taattttoooo/oco/ooo//a/ tran jax, 2021.
[66] Black, S. Biderman, F. Hallahan, Q. Anthony, L. Gao, Golding. He, Leahy, Mc- Donell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L.. Reynolds, J. Tow, B. Wang, and Weinbach. Gpt-neox-
open-source autoregressive language model, 2022.
[67] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 2015. 7,
[68] M. Honnibal, Montani, S. Van Landeghem, and A. Boyd. spacy: Industrial-strength natural language processing in python. 2020. 7
[69] F. Rong.
Extrapolating to unnatural language processing with gpt-3's in-context learn- ing: The good, the bad, and the mysterious. http: stanford. edu/blog// in- context-learning! 2021. Accessed: 2022-06-04. 7,
[70] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. 8
[71] T.-Y. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 8
[72] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick. Detectron2. /ttpss//github.. com/ 2019. 8
f faccbookeeaarcchheeetcccoo
[73] K. He, G. Gkioxari, Dollár, and Girshick. Mask r-cnn. In 2017 IEEE International
Conference on Computer Vision (ICCV), 2017. 8
[74] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: A General Navigation Model to Drive Any Robot. In arXiV, 2022. URL https://arxiv: www.aa/wwww.rr//////....... 03370. 8
13


Reviews:
[{"review_id": "yealNrsY-dfL", "reviewer": "Reviewer_Mudn", "summary": "The manuscript describes a method by which rich robotic commands provided in natural language are turned into robotic actions.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The major weakness of this approach is that it does not handle language, despite this being the focus of the manuscript. It handles keywords, specifically noun phrases. It cannot handle verbs, adverbs, determiners, etc.\n\nThe sentences \"After passing a white building go left\" and \"Instead of passing a white building go left\" mean exactly the same thing!\n\nThe manuscript is written to give the impression that the model handles sentences like \"After passing a white building take a right next to a white truck...\" but it only handles 'white building && white truck'.\n\nI understand what the authors are trying to do, but, this is misleading to readers and merits far more than a footnote on page 4!\n\nThis could be a good contribution, but, the title, abstract, manuscript and evaluation should be completely rewritten to remove language and to focus on what the algorithm actually does, landmarks.\n\nThen, the algorithm should be compared against a baseline method that simply uses Named Entity Recognition to find the landmarks in text and then use those.\n\nIt is not clear to me what the value of a model that essentially does NER and then recognizes landmarks using a pretrained transformer is. This seems like a straightforward application that doesn't offer much. If the method were extended to handle linguistic commands in general, that would be exciting. But it's not clear that this is possible with the given approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method promises to handle linguistic commands. But, it handles conjunctions of keywords. And it doesn't compare against baselines that do the same.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "1: Not very relevant to the robot learning community.", "issues": "I indicated these above. The method doesn't handle language, only keywords.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "oK9yDeqn1FQZ", "reviewer": "Reviewer_eeqY", "summary": "The paper proposed an instruction-following navigation system. The approach consists of three components: 1) A VNM (i.e. ViNG) to construct a topological graph of the environment from pre-collected images of the environment, 2) a LLM (i.e. GPT-3) to parse the instructions into an ordered list of “landmark” object names, and 3) a VLM (i.e. CLIP) to match nodes of the VNM’s topological graph (i.e. images of landmarks) to the landmark object names. Since VNM can perform navigation from its constructed graph, the key idea of the paper was to use two pretrained models without fine tuning to translate natural language instructions into a sequence of nodes in the topological graph. The key assumption of their approach is that instruction-following navigation reduces to landmark-sequence-following navigation, whose limitations were addressed in the discussion.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - Informative qualitative results on video.\n - Code and data released in supplementary material and webpage.\n - By building on top of large internet-scale pretrained models, the approach is open-vocabulary. This is an important strength and distinction over many prior works. I recommend highlighting this strength more, either through writing or empirically by providing navigation instructions which references objects not in the MS-COCO list of classes (and therefore, closed-world object detectors such as Faster-RCNN-FPN will fail).\n\nWeaknesses:\n - Claims of their language driven robotic navigation is \"self-supervised\". Meanwhile, their experiments are based on human-collected data, and cited autonomous data collection approaches do not trivially cover all relevant landmarks and areas. Given that the authors were manually involved in both creating the navigation instructions (containing references to landmarks) and in manually collecting the robotic experience (which presumably passes through all relevant landmarks), this raises concerns in the validity of experiments for backing up claim of a \"self-supervised system\" (L28-29). More details in issues section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposed a simple but effective system for outdoor navigation from language instructions. It has good performance well for the 5 queries reported and provides sufficient ablations for understanding impact of each component of the system. The reviewer's main feedback is in claiming the system is \"self-supervised\".", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Issues:\n 1. One of the key claims of the paper is \"a self-supervised\" system (L28-29, L46) and that the map is \"constructed from prior exploration of the environment\" (L297). While the approach does not require \"human annotations of robot navigation data\" (L55), it does depend on humans for \"manually driving the robot and collecting image and GPS observations\" (L214-215). This human collected offline data play key roles in the approach's performance (e.g. in coverage of all relevant waypoints) and the evaluation process (e.g. view angles for 100 images for VLM grounding evaluation, L268). What is the precise procedure of the initial data collection? What properties about the collected data affect the final approach's performance and evaluation procedure? How can this data collection process cover all the landmarks which might be used by downstream instructions?\n       - I recommend making assumptions about this prior data collection explicit, either by explicitly stating some dependency on a human collected data or an exploration algorithm (along with experimental results). In addition, phrases like \"constructed from prior exploration of the environment\" (L297) implies that the robot is actively collecting its own data, and should be modified to be more accurate.\n      - The authors mentioned that such data collection could be obtained from past traversals or with autonomous exploration methods like [Shah et al](https://proceedings.mlr.press/v164/shah22a/shah22a.pdf). It is not clear to me that these approaches will have enough coverage of the area and/or of landmark objects.\n 2. CLIP's performance can be sensitive to not just the prompt but also the exact phrase used to describe the object category (e.g. \"person\" v.s. \"pedistrian\"). What are the 30 commonly occuring landmarks (L268) and how are they \"manually curated\" (L270), especially since instruction following experiments only contained 19 landmarks (L219)?\n\nMinor Issues:\n 1. I find it surprising there was such a big performance difference CLIP-derived object detector (i.e. ViLD) and directly using CLIP, and recommends that the author includes per-class detection tables in the appendix for the community to better compare open-vocabulary object detectors in different scenarios.\n 2. What is the setup for quantitative experiments in Table 2? Especially for zero-shot open-vocabulary object detectors, whose performance depends on the amount and specific choice of classes considered, are all approaches given all of MS-COCO's classes?\n 3. Figure 2’s text is way too small. Please make it bigger.\n 4. \\overline{l}’s repeated definition in L39 and L44.\n 5. Grammar mistake in L249 “On independently evaluating the efficacy of our the VLM at retrieving landmarks, ..\"\n 6. I feel that 5 queries is not enough to quantitatively evaluate the model's navigation performance. This is an extremely small sample size compared to instruction following works, but in the same range of ViNG's 6 pairs of start-goal image pairs.\n\n"}}, {"review_id": "QXdbrqVbF0", "reviewer": "Reviewer_2UZt", "summary": "This paper presented an approach to combining pretrained navigation, vision-language, and language models for vision-language suburban navigation.  The proposed method uses large language models (LLM) such as GPT-3 for landmark extraction, vision-language models (VLM) such as CLIP to match landmarks with text descriptions, and ViNG to navigate based on the constructed topological map. The experiment showed that the combined system can successfully carry out instructions in outdoor settings.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- This paper demonstrates several important properties for embodied navigation\n    - LM-Nav does not require photo-realistic simulation for training.\n    - LM-Nav does not require environment-specific natural language annotations for training. \n- LM-Nav was deployed in a realistic outdoor environment and showed successful navigation.\n\nWeakness\n- The space of language instruction is quite limited.\n    - It is understandable that LM-Nav only considers landmarks. But even for landmarks, this system requires the utterances of landmarks ordered in the sequence of visits. For example, LM-Nav cannot understand instructions such as “you’ll find a car after passing the stop sign” because the visit order is opposite from the order in the sentence. This is one of the main limitations if we want to use it as an interface. We cannot guarantee the order of landmark utterances.\n    - Since the input sentences are parsed into landmarks, the system cannot understand “take right next to a white truck”. LM-Nav can perform this query possibly because it optimizes the reachability of the next landmark, if the instruction stops at this sentence, it will fail too. Spatial information is important for navigation. For two paths that end at similar landmarks, e.g. white truck, the system cannot disambiguate which path to take unless that are other landmarks on the path that can be added to the instruction.\n- LM-Nav requires past observations of the environment to build topological maps. To deploy LM-Nav to a new environment, it needs to collect data on the new environment, this gives a performance boost to navigation. This also makes language only provides constraints for graph search.  \n- It is hard to interpret the quantitative results for the combined system. The main quantitative result appeared in appendix B which should be in the main text. I currently have some questions about the descriptions here.\n    - In the main text, it says the experiment is based on 5 queries but the appendix says 20 instructions. Are these 20 instructions from the 5 queries?\n    - If the experiments include multiple language queries, are Table 3 and 4 in the appendix refer to the mean?\n    - The language instructions in the experiments seem to be different, in this case, how can the expt. length to be similar? \n    - The ablation of LM-Nav only runs on “No VNM” version. We’ll need more to understand how each component affects the performance of the whole system rather than each component individually. For example, these ablations:\n        - No LM, No VLM, provide visual landmarks instead. This gives us the upper bound of the system if we have correct visual landmarks.\n        - No LM, but use VLM with ground truth landmark parses. This tells us how landmark localization impacts performance.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper tackles an important problem in visual language navigation and shows some promising properties. However, the experiment section in the current form is hard to evaluate the effectiveness of the whole system and the contribution of the LM and VLM to navigation performance. \n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "The following issues need to be addressed:\n- Appendix B should be in the main text and include more ablations to understand the contribution of each component to the whole system. For the failure cases, it would be great to include the failure mode analysis to understand the limitation of the design.\n- The space of text input is more restricted than currently stated in the paper. The author will need to describe the space of input command and its limitations."}}, {"review_id": "MXR1Q-FHROn", "reviewer": "Reviewer_FpxS", "summary": "The paper presents Large Model Navigation (LM-Nav), a system that combines 3 pre-trained models (visual navigation model, vision-and-language model, and large language model) for long-horizon instruction following in the real-world. The authors demonstrate that utilizing these pre-trained models without any fine-tuning in the target environment results in successful following of natural language environments for 100s of meters outdoors. The various models are trained on large-scale datasets using self-supervised objectives, obviating the need to build a photorealistic simulator, or trajectory-level annotations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is very well written, and is a strong systems paper in combining various modules for long-horizon instruction following in outdoor environments on a real robot. The results are impressive, and the motivation is clear. I also appreciated the  investigation into the different components of their approach, demonstrating high performance of the LLM and VLM .\n\nWeaknesses:\n- The VNM is trained using a large dataset from outdoor environments on the same robot embodiment used in this paper, and is not transferable to a different robot embodiment (i.e., a quadruped robot, or a robot with a different height). This dataset is also costly to collect. \n- The VNM requires the user to manually drive the robot around in the target environment to build the topological map and collect image/GPS observations which seems time-consuming. How long does this process take? What happens if obstacles change in the environment after this topological map is built (i.e., a car is parked on the road, blocking the path)? This seems to be the main drawback between this work, and other VLN approaches, which are more adaptive to the test environment, and require no previous exploration of the test environment to follow language instructions. \n- This approach is dependent on the VNM finding the same landmarks during the exploration phase as the natural language instruction. What happens if the VNM misses a landmark during the exploration phase? \n- How does the motion blur of the robot affect the performance of the VLM?\n- The main paper should include more quantitative results (that are currently in the appendix). \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has strong results, but more details on the limitations of the VNM module and time to adapt to novel environments should be included. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Some limitations of the VNM module should be addressed and added to the paper. \n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)