Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning
Authors: Dhruv Shah
Michael Robert Equi
Błażej Osiński
Fei Xia
brian ichter
Sergey Levine
Venue: CoRL 2023 Poster (2023)
Keywords: navigation
language models
planning
semantics

Abstract: Navigation in unfamiliar environments presents a major challenge for robots: while mapping and planning techniques can be used to build up a representation of the world, quickly discovering a path to a desired goal in unfamiliar settings with such methods often requires lengthy mapping and exploration. Humans can rapidly navigate new environments, particularly indoor environments that are laid out logically, by leveraging semantics --- e.g., a kitchen often adjoins a living room, an exit sign indicates the way out, and so forth. Language models can provide robots with such knowledge, but directly using language models to instruct a robot how to reach some destination can also be impractical: while language models might produce a narrative about how to reach some goal, because they are not grounded in real-world observations, this narrative might be arbitrarily wrong. Therefore, in this paper we study how the ``semantic guesswork'' produced by language models can be utilized as a guiding heuristic for planning algorithms. Our method, Language Frontier Guide (LFG), uses the language model to bias exploration of novel real-world environments by incorporating the semantic knowledge stored in language models as a search heuristic for planning with either topological or metric maps. We evaluate LFG in challenging real-world environments and simulated benchmarks, outperforming uninformed exploration and other ways of using language models.

Navigation with Large Language Models: Semantic Guesswork as a Heuristic for Planning
Dhruv Shah Michael Equist, Blazej Osinski", Fei Xia?, Brian "enter",eerge, Levine PC Berkeley, "Google DeepMind, "University of Warsaw
Abstract: Navigation in unfamiliar environments presents major challenge for while mapping and planning techniques can be used to build up represen- tation of the world, quickly discovering path to desired goal in unfamiliar set- tings with such methods often requires lengthy mapping and exploration. Humans can rapidly navigate new environments, particularly indoor environments that are laid out logically, by leveraging semantics - c.g., a kitchen often adjoins a living room, an exit sign indicates the way out, and so forth. Language models can pro- vide robots with such knowledge, but directly using language models to instruct robot how to reach some destination can also be impractical: while language models might produce a narrative about how to reach some goal, because they are not grounded in real-world observations, this narrative might be arbitrarily wrong. Therefore, in this paper we study how the "semantic guesswork" produced by lan- guage models can be utilized as a guiding heuristic for planning algorithms. Our method, Language Frontier Guide (LFG), uses the language model to bias explo- ration of novel real-world environments by incorporating the semantic knowledge stored in language models as a search heuristic for planning with either topologi- cal or metric maps. We evaluate LFG in challenging real-world environments and simulated benchmarks, outperforming uninformed exploration and other ways of using language models.
robots:
Keywords: navigation, language models, planning, semantic scene understanding
1 Introduction
Navigation in complex open-world environments is conventionally viewed as the largely geometric problem of determining collision-free paths that traverse the environment from one location to an- other. However, real-world environments possess semantics. Imagine navigating an airport to get to terminal: your prior knowledge about the way such buildings are constructed provides exten- sive guidance, even if this particular airport unfamiliar to you. Large language models (LLMs) and various language embedding techniques have been studied extensively ways to interpret the semantics in user-specified instructions (c.g., parsing "go to the television in the living room" " and grounding it in specific spatial location), but such models can provide much more assistance in robotic navigation scenarios by capturing rich semantic knowledge about the world. For instance, when looking for spoon in unseen house, the LLM can produce "narrative" explaining why going towards a dishwasher may eventually lead you to find the spoon, and that the robot should prioritize that direction. This is similar to how person might imagine different ways that the avail- able subgoals might lie on the path to the goal, and start exploring the one for which this "narrative" seems most realistic. However, since language models not grounded in the real world, such mod- els do not know the spatial layout of the robot's surroundings (e.g., there is a couch that the robot needs to circumnavigate). To utilize the semantic knowledge in language models to aid in embodied tasks, we should not just blindly follow the language model suggestions. but instead use them as proposals or navigational heuristics. In this paper, we study how that might be accomplished.
We study this idea in the context of visual navigation, where a robot is tasked with reaching a goal denoted by natural language query (see Fig. 1) in novel environment using visual observations.
These authors contributed equally. Videos and code: sites.google.com//e wtw.oogll..oom/vew/lfg.....
7th Conference on Robot Learning (CoRI. 2023), Atlanta, USA.

---

Query: Find the gas stove.
"I see Tei gerator and in crowiee apd cuuallyy
LLM as Planner
the klew' Undten
LFG
Let's this direciinn:
Go Straight
"Look for the gas stove in- the kitchen.'
a to my
the Let's avoid this rosion"
Figure 1: In constrast to methods that use LLM plans directly, Language Frontier Guide (LFG) uses a language model to score subgoal candidates, and uses these scores to guide a heuristic-hased planner.
The robot has no prior experience in the target environment, and must explore the environment to look for the goal. While narratives generated by an LLM may not sufficient for navigation by themselves, they contain useful cues that can be used to inform or guide the behavior of the underlying navigation stack for the language navigation task (e.g.. by choosing between collision- free subgoal proposals that avoid the couch and lead to the ice tray). We show that this idea can be combined with frontier-based based exploration, where the robot maintains a set of unvisited locations at its frontier, grounds them in text using vision-language model (VLM), and scores the unvisited subgoals by using LLM reasoning.
We propose Language Frontier Guide, or LFG, a method for leveraging the reasoning capabilities of LLMs to produce a search heuristic for guiding exploration of previously unseen real-world envi- ronments, combining the strengths of search-based planning with LLM reasoning. LFG is agnostic of the memory representation and planning framework, and can combined with both (i) ge- ometric navigation pipeline, building metric map of the environment for planning and using hand-designed controller, as well as (ii) a Icarning- based navigation pipeline, building a topologi- cal map for planning and using a Iearned control policy, yielding versatile system for navigating natural language goals. Our experiments show that LFG can identify and pre- dict simple patterns in previously unseen environments to accelerate goal-directed exploration. We show that LFG outperforms other LLM- approaches for semantic goal-finding in challenging real-world environments and on the Habitat ObjectNav benchmark.
to
2 Related Work
Vision-based navigation: Navigation is conventionally approached as a largely geometric prob- lem, where the aim is map environment and that map to find path to goal location [1]. Learning-based approaches can exploit patterns in the training environments, particularly by learn- ing vision-based navigation strategies through reinforcement or imitation [2-7]. Our work is also related to PONI [7], which uses a learned potential function to prioritize frontier points to explore; instead, we use a language model to rank these points. Notably, these methods do not benefit from prior semantic knowledge (c.g., from the web), and must rely entirely on patterns discovered from offline or online navigational data. Our aim is specifically bring semantic knowledge into naviga- tion, to enable robots to more effectively search for a goal in a new environment.
Semantic knowledge-guider navigation: Prior knowledge about the semantics of indoor envi- ronments can provide significantly richer guidance. With the advent of effective vocabuacyy vision models [8, 9], some works have recently explored incorporating their semantic knowledge into models for navigation and other robotic tasks with the express aim of improving performance instruction following [10-14]. In general within robotics, such methods have either utilized pre- trained vision-language representations [15-17], or used language models directly to make deci- sions [18-23]. Our aim is somewhat different: while we also focus on language-specified goals, we are primarily concerned with utilizing the semantics in pre-trained language models to help a robot figure out how actually reach the goal, rather than utilizing the language models more effectively interpret language instruction. While language models can output reasonable substeps for temporally extended tasks in some settings [24, 25], there is contradictory evidence about their ability to actually plan [26], and because they are unaware of the observations and layout in partic- ular environment, their "plans" depend entirely on the context that provided to them. In contrast to prior work, our approach does not rely on the language model producing good plan, but merely a heuristic that can bias a dedicated planner to reach a goal more effectively. this way, we use the language models more to produce suggestions rather than actual plans.

---

LLM-guided navigation: Some works have sought to combine predictions from language models with either planning or probabilistic inference [14, 27]. so as to not not rely entirely on forward predic- tion from the language model to take actions. However, these methods are more aimed at filtering out infeasible decisions, for example by disallowing actions that a rohot is incapable of perform- ing, and still focus largely on being able to interpret and process instructions, rather than using the language model as a source of semantic hints. In contrast, by incorporating language model sugges- tions as heuristics into heuristic planner, our approach can completely override the language model predictions if they are incorrect, while still making use of them if they point the way to the goal.
Another branch of recent research [28-30) has taken a different approach to ground language mod- els. by making possible for them to read in image observations directly. While this represents a promising alternative approach to make language models more useful for embodied decision mak- ing, we believe it is largely orthogonal and complementary to our work: although vision-language models can produce more grounded inferences about the actions a robot should take, they are still limited only to guessing when placed in unfamiliar environments. Therefore, although we use un- grounded language-only models in our evaluation, we expect that our method could be combined with vision-language models easily, and would provide complementary benefits.
3 Problem Formulation and Overview
Our objective is to design high-level planner that takes input a natural language query g (c.g., "find the bedside table"). explores the environment in search of the queried object, and commands low-level policy to control robotic agent. To do this, we maintain episodic memory of the environment M in the form of either (i) a 2D map of the environment, where grid cells contain information about occupancy and semantic labels, where nodes contain images captured by the robot and corresponding object labels. One way to solve this task is Frontier-Based Exploration (FBE) [31], where robot maintains a set of unexplored frontiers in it's memory, and explores randomly to reach the goal. Can we do better with access to LLMs?
(ii) a topological map of the environment,
We distill the language-guide exploration task to heuristic- search problem, where the robot must propose unvisited subgoals waypoints, score them, and then A*) to plan a path to the goal. Thus, at the core of LFG is the task of scoring subgoal proposals. Formally, let's assume and a set S of n textual subgoal proposals 81, s2,..., Sn (e.g., "a corner with a dishwasher and refrigerator", "a hallway with a door", etc.). Our goal is score these subgoal proposals with p(s p(8;, g. M). the probability that the candidate 8; E S will lead to the goal g given the current state of the environment, described through M.
search algorithm (c.g.,
have the task by query q. a partially explored environment stored in M,
We posit that we can leverage the semantic reasoning capabilities of LLMs by prompting them to construct narratives about which semantic regions of the environment are most (and least) likely to lead to the goal. While the narrative itself might be ungrounded, since the LLM knows very little about the environment, reasoning over objects and semantic regions of the environment often gen- eralizes very broadly. For example, even without seeing a new apartment, a human would guess that the dining area is close to the kitchen. Hence, rather than directly using LLM scores for plan- ning [23, 25], we incorporate them as goal-directed heuristic to inform the search process. This has two distinct advantages: (i) when the LLM is right, it nudges the search towards the goal, and when it is wrong (or uncertain), we can still default to the underlying FBE algorithm, allowing re- covery from LLM failures, and (ii) it allows to combine the signal from LLMSs with other scores that may be more grounded, e.g. distance to subgoals, making the system more versatile.
4 LFG: Scoring Subgoals by Polling LLMs
Our aim in this section is derive a scoring function from LLMs that takes a textual descrip- tion of subgoal candidates s; and the goal query q as inputs, and predicts task-relevant probability p( 8; q. M), conditioned on the episodic memory M. While we may obtain this from next-token likelihoods (or "logprobs"),, they do not represent the desired task-relevant probability p(s;, q, M), and fail to assign similar scores, say, to different subgoals that are semantically similar but have different tokenizations (see our experiments in Section 6 for a comparison). Furthermore, most ca-
3

---

[-. prompt.... ...]
[.. prompt... ]
lobserve followinng sbjects while hormi:
loseevve followiingg ofjects hiuuee
n. sampls
1 coud
2 clud
3 rufr aafiggerll
I couch
2 woodenchuu
3 reli iusaator
Where should pood sprocing time scarching am looking for a gro stowe?
Where should aoorth am locking for gas stove?
provide justrfcation...
should always provide justrication...
Combined Scores
Figure 2: LFG scores subgoals with an empirical estimate of the likelihoods by sampling an LLM n. times with both positive and negative prompts, and uses chain-of-thought to ohtain reliable scores. These scores are used by a high-level planner as heuristics to guide search. For full prompts, see Appendix B.
pable LLMs of today are available through APIs that do not expose the ability to query logprobs.' And lastly, even if reliable logprobs were available, they are incompatible with chain-of- tf-tuuhtt prompting [32], which we find to be crucial to success in spatial reasoning.
To overcome these challenges, LFG uses a novel approach to extract task-relevant likelihoods from LLMs. Given candidate subgoal images, LFG uses a VLM to obtain textual subgoal desriptor ss which must be scored with the LLM. LFG polls the LLMs by sampling the most likely subgoal n, times, conditioned on task-relevant prompt. We then use these samples empirically estimate the likelihood of each subgoal. To get informative and robust likelihood estimates, we use a chain-of- thought prompting (CoT) technique [32], improve the quality and interpretability of the scores, and use a combination of positive and negative prompts to gather unbiased likelihood estimates. Figure 2 outlines scoring technique, with the full prompt provided in Appendix B. We now describe the details of our scoring technique.
We rely on in-context learning by providing an example of structured query- response pair to the LLM, and ask to pick the most likely subgoal that satisfies the query. To sample subgoal from S using a language model, we prompt it to generate a structured response, ending with "Answer: i'?, This structure allows us to always sample a valid subgoal, without having to ground LLM generations in the environment [24].
Structured query:
Positives and negatives: We find that only using positive prompts (e.g.. "which subgoal is most likely to reach the goal") leads to likelihood estimates being uninformative for cases where the LLM is not confident about any subgoal. To overcome this, we also use negative prompts (c.g., "which subgoal is least likely to be relevant for the goal"), which allows us to score subgoals by dimmnnaing/owwnweightnng subgoals that are clearly irrelevant. We then use the difference between the positive and negative likelihoods to rank subgoals.
Shain-of-thought prompting: A crucial component of getting interpretable and reli- able likelihood estimates is to encourage the
Algorithm 1: Scoring Subgoals with LFG
Data: Subgoal descriptors {4,4', E ES}
LLM to justify its choice by chain-of of-thought prompting. As demonstrated CoT elicits interpretability and reasoning ca-
pPrompt < PositivePromptt( (4))
prior works,
nPrompt <- NegativePromp(((L]
pSamples + [sampleLLM(pPrommp) * ng]
pabilities in L.MMs, and while we don't ex-
nSamples [sampleLLM(nPrompp)) * n.]
plicitly use the generated reasonings in our
pScores + sum(pSamples) / n
approach (great future work direction), we
nScores <- sum(nSamples) n,
find that CoT improves the quality and con-
return pScores, nScores
sistency of the likelihood estimates. Addi-
tionally, it also helps maintain interpretability, to better understand why the LFG-equipped agent takes certain decisions.
In summary, we score subgoals by sampling the LLM multiple times and empirically estimating the likelihood of each subgoal. We use combination of positive and negative prompts to get unbiased likelihood estimates, and use chain-of-thought prompting to improve the quality and interpretability of the scores (Figure 2). We will now discuss how these scores can be incorporated into navigation system as search heuristies.
Most notably, OpenAl's Chat API for GPT-3.5 and GPT-4, Google's ' PaLM API, and Anthropic's Claude API all do not support logprobs.

---

Metric
Topological
Observations IRGOD Posel
Episodic Memory Hopaliggia logicaiial or Metiin
Heuristic-based Exploration Policy
Control Policy (Leareed Deterministic
Expalore oven'
Smik Goal Category
Language Model Scoring
Goold checuul
Figure 3: Overview of LFG for language-guided exploration. Based on the pose and observations, LFG builds an episodic memory (topological metric). which is used by the heuristic-based exploration policy to rank adjacent clusters, or subgoal frontiers. Navigation to the subgoal frontier is completed by low-level policy.
5 LLM Heuristics for Goal-Directed Exploration
Given the LLM scoring pipeline outlined in the previous section, our key insight is that we can incorporate these scores in search-based planning pipeline to heuristically guide the search process. We instantiate LFG using frontier- based exploration (FBE) and LLM scores generated via polling.
FBE: This method maintains a "map" of the seen parts of the environment, which may be geomet- ric [33] or topological [34], and a frontier separating it from the unexplored parts. By navigating to the nearest point of the frontier, the robot explores new areas of the environment until it finds the goal object or completes exploration without finding it. standard FBE implementation is presented in Algorithm 2 inblack text. The robot maintains either 2D metric map of its surroundings, topological map whose nodes are comprised of the robot's visual observations and edges represent paths taken in the environment, Additionally, we also store semantic labels corresponding to objects detected in the robot's observations, which are used to ground the observations in text.
At fixed re-planning rate, the high-level planner computes its frontier fi (Line 10), and picks the frontier point that is closest to the current location, i.e., maximizing the distance score (Line 16), and then navigates to this frontier (Line 21). At any point in this process, if the agent's semantic detector detects an object of the same category as the query q. navigates directly to this object and the trajectory ends.
Incorporating LLM scores: Our method, LFG, extends FBE by using an additional search heuristic obtained by polling LLMS for semantic "scores". The modifications to FBE are marked in purple in Algorithm 2. After enumerating the frontiers, LFG uses the semantic labels from a VLM [35] to ground subgoal images at each frontier fa (Line 1 1).. These images are converted into textual strings, and form the subgoal candidates s; that can be scored using Algorithm 1. We associate each frontier point fi with the nearest object cluster (Line 17), and compute LLM scores for each point as follows:
h(Si, g) wp, LLMppc(cc) wn. L.L (cx) - dist(Jt,
where p is the current position of the agent, and Wp, Wn are hyperparameters (see Appendix A. We then choose the frontier with the highest score to be the next subgoal (Line 21), navigate to it using a local controller, and repeat the planning process. Algorithm outlines the general recipe for integrating LLM scores as a planning heuristic. Please see Appendix A for specific instantiations of this system with geometric and topological maps, and more details about the referenced subroutines.
System Evaluation
We now evaluate the performance of LFG for the task of goal-directed exploration real-world environments, and benchmark its performance against baselines. We instantiate two systems with L.G: a real-world system that uses a topological map and a learned control policy, and simulated agent that uses a geometric map and deterministic control policy. Our experiments show that both
5

---

Algorithm 2: Instantiating LFG for Goal-Directed Exploration
Data: og, Goal language query 4
1 subgoal < None
while not done do
3
- gctObservvatin
episodicMemoory mappingMModul((o)) if in semanticMap then
5
subGoall + get.occtioolepisodeeMmmmy, q)
else
if numSteps % == 0 then
// replanning
location f- getCurrentLocation)))
frontier Meeee
11
objectClusters <- getSemanticl abelséepisodicMemory frontier)
12
LLM pos: I.M. neg < ScoeSSingealxxhjjecChssssss
13
scores +0
14
for point in frontier do
15
distance t DistTo(location, point)
16
scores[point]
distance
17
closestCluster + geiClosesCCsstrrojjeeCCussssss point) < ClusteriD(closessuuusee))) if closestttse,,, point) < then
18
19
// incorporate language scores
20
scores[point] <- wpp T.L. pos [i] Wn L.M. neg [i] distance
21
subgoal argmax(scores)
22
numSteps < numSteps +1
23
goToosubGoal)
these systems outperform existing LLM- --saedddd exploration algorithms by wide margin, owing to the high quality scores incorporated as search heuristics.
6.1 Benchmarking ObjectNav Performance
We benchmark the performance of LFG for the task of object-goal navigation the Habitat Ob- jectNav Challenge [36], where the agent is placed into simulated environment with photo-realistic graphics, and is tasked with finding query object from of 10 categories (e-g.. "toilet", "bed"", "couch" etc.). The simulated agent has access to egocentric RGBD observations and accurate pose information. We run 10 evaluation episodes per scene and report two metries: the average success rate, and success weighted by optimal path length (SPL), the default metrics for the benchmark. Since LFG requires no training, we do not use the training scenes from HM3D.
We compare to three classes of published baselines: (i) learning-based baselines that learn navigation behavior from demonstrations online experience in the training scenes [37] up 2.5B frames of experience, (ii) search-based baselines [33, 38), and (iii) LLM-based baselines that do not use the training data directly, but leverage the semantic knowledge inside foundation models to guide embodied tasks [18, 39].
Evaluating LFG the HM3D benchmark, find that it significantly outperforms search and LLM- based baselines (Table 1). Greedy LLM struggles on the task due to several LLM planning failures, causing the episodes to fail. LFG significantly outperforms the vanilla FBE baseline by leveraging semantic priors from LLMs to score subgoals intelligently. Comparing to learning-base find that LFG outperforms most of them and closely matches the state-of-the-ar proving the competence of our polling and heuristic approach. Figure 4 shows an example of the LFG agent successfully reaching the goal by using thain-of-thought and negative prompting.
baselines, the task,
L3MVN [39], which uses a combination of LLMS and search, performs slightly better than FBE, but is unable to fully leverage the semantics in L.MSs. While being similar to LFG, it suffers from

---

Query: Find the potted plant,
"plants typically found bedrooms or around furniture, should avoid cluster 1, and
LLM:
Agent succeeds!
Figure 4:
Qualitative example of a negative score influencing the agent's decision. LFG discourages the agent from exploring the hedroom and living room, leading fast convergence toward the goal, whereas FBE fails. The CoT reasoning given the LLM is shown in purple, justifying its
Query: Find a Toilet
"toilets are not typically found in bedrooms kitchens, but is more likely that bathroom is near bedroom so we should explore bedroom first"
LLM:
Figure 5: Qualitative example LFG in real. LFG reasons about floor plans in the environment searching. this apartment experiment, LFG believes that a bathroom is more likely to be found near a bedroom rather than kitchen, and guides the robot towards the bedroom, successfully reaching the goal.
two key limitations: (i) uses a small language model (GPT-2), which likely does contain strong semantic priors for the agent to leverage, and (ii) it uses simple likelihood based scoring scheme, which we show below is not very effective. Another closely related work, LGX [18], uses a variant of greedy LLM scoring, and hence fails to perform reliably on the benchmark.
Probing deeper into the strong performance of LFG, ablated various components of our scoring pipeline and studied the change in performance. Note that LGX (Greedy) and L3MVV (No CoT, Logprobs) can be seen as ablations of LFG. Table 2 shows that modifying both the prompting and scoring mechanisms used by LFG lead to large drops in performance. Most notably. scoring via polling (+7.8%) and CoT (+6.6%) are both essential to the strong performance of LFG. Further- more, we find that using only positive prompts also hurts performance (-4.7%). Popular approaches for using LLMs for planning are significantly outperformed by LFG: Greedy (-14.5%) and Log- probs (-8.5%). Figure 4 shows an example of the LFG agent successfully reaching the goal by using CoT and negative prompting.
Setup: For these experiments, we mimic the semantic mapping pipeline of the best-performing baseline the benchmark [33, 38), and integrate LFG with the geometric map. The simulated agent builds a 2D semantic map of its environment, where grid cells represent both occupancy and semantic labels corresponding of-the-art vision models, such as DETIC, work poorly in simulation due to rendering artifacts [33]; hence, ground-truth semantic information for all simulated baselines to analyze navigation performance under perfect perception.
objects detected by the agent. Prior work has shown that state-
6.2 Real-world Exploration with LFG
To show the versatility the LFG scoring framework, we further integrated with a heuristic- based exploration framework that uses topological graphs for episodic memory [34]. We compare two published baselines: language-agnostic FBE baseline [40], and an I.L.-based baseline that uses the language model to greedily pick the frontier [18].
7

---

Method
Success
A
Method
Success
SPL
Data
LFG (Full)
68.9
DD-PPO [37]
27.9
14.2
2.5B
Prompting
FBE [33]
61.1
34.0
0
No CoT
62.3
(6.6)
SemExp [38]
63.1
0.29
10M
Only Positives
64.2
(4.7)
v2 [42]
64.7
28.1
12M
Scoring
Greedy LLM [18]
54.4
26.9
0
Random
61.1
(7.8)
L3MVN [39]
62.4
0
Logprobs
60.4
(8.5)
LFG (Ours)
68.9
36.0
0
Ask
62.4
(6.5)
We find that CoT prompting with positives HM3D ObjectNav benchmark, and can achieve and negatives, combined with polling, essential close to SOTA performance without any pre-training. achieve the best performance.
Table 1: LPG outperforms all LLM-based baselines Table 2:
We evaluate this system in two challenging real-world environments: cluttered cafeteria and an apartment building (shown in Figures and 5). In each environment, the robot tasked reach an object described by textual string (e.g., "kitchen sink" or "oven"), and we measure the success rate and efficiency of reaching the goal. Episodes that take longer than 30 minutes are marked as failure. While we only tested our system with goal strings corresponding to the 20,000 classes supported by our object detector [35], this can be extended more flexible goal specifications with the rapid progress in vision- vision-language models.
We find that the complexity of real-world environments causes the language-agnostic FBE baseline to time out, i.c., the robot is unable to reach the goal by randomly exploring the environment. Both LLM baselines are able to leverage the stored semantic knowledge to guide the exploration in novel environments, but LFG achieves 16% better performance. Figure 5 shows an example rollout in real apartment, where the robot uses LFG to reach the toilet successfully.
Setup: We instantiate LFG in the real-world using a previously published topological navigation framework [34] that builds topological map of the environment, where nodes correspond to the robot's visual observations and edges correspond to paths traversed in the environment. This system relies on omnidirectional RGB observations and does not attempt to make dense geometric map of the environment. To obtain "semantic frontiers" from the omnidirectional camera, we generate n, = 4 views and run an off she-th object detector [35] generate rich semantic labels describing objects in these views. The robot maintains a topological graph of these views and semantic labels, and picks the frontier view with the highest score (Algorithm Line 21) according to LFG. The robot then uses a Transformer- policy [6, 41] to reach this subgoal. For more implementation details, see Appendix A.3.
7 Discussion
We presented LFG, a method for utilizing language models for semantic guesswork to help navigate to goals in new and unfamiliar environments. The central idea in our work is that, while language models can bring to bear rich semantic understanding, their ungrounded inferences about how to perform navigational tasks are better used as suggestions and heuristics rather than plans. We for- mulate a way to derive a heuristic score from language models that we can then incorporate into planning algorithm, and use this heuristic planner to reach goals new environments more effec- tively. This way of using language models benefits from their inferences when they are correct, and reverts to a more conventional unguided search when they are not.
Limitations and future work: While our experiments provide a validation of our key hypothesis, they have number of limitations. First, we only test in indoor environments in both sim and real yet the role of semantics in navigation likely differs drastically across domains e.g., navigating forest might implicate semantics very differently than navigating an apartment building. Exploring the applicability of semantics derived from language models in other settings would be another promising and exciting direction for future work. Second, we acknowledge that multiple requests to cloud-hosted LLMs with CoT is slow and requires internet connection, severely limiting the extent of real-world deployment of the proposed method. We hope that ongoing advancements in quantizing L.LMs for edge deployment and fast inference will address this limitation.
8

---

Acknowledgments
This research was partly supported by AFOSR FA9550-22-1-0273 and DARPA ANSR. The authors would like to thank Bangguo Yu, Vishnu Sashank Dorbala, Mukul Khanna, Theophile Gervet, and Chris Paxton, for their help in reproducing baselines. The authors would also like to thank Ajay Sridhar, for supporting real-world experiments, and Devendra Singh Chaplot, Jie Tan, Peng Xu, and Tingnan Zhang, for useful discussions in various stages of the project.
References
[1] G. N. DeSouza and A. C. Kak. Vision for mobile robot navigation: A survey. IEEE transac- fions on pattern analysis and machine intelligence, 24(2):237-267, 2002. 2
[2] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel. Vision-and Ession-and-lange aad-language navigation: Interpreting visually-grounded naviga- tion instructions in real environments. In IEEE Conference Computer Vision and Pattern
Recognition, pages 3674-3683, 2018." 2
[3] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi-parametric topological memory for naviga- tion. arXiv preprint arXiv:1803.00653, 2018.
[4] N. Hirose, F. Xia, R. Martín Martín, A. Sadeghian, and S. Savarese. Deep visual MPC-policy learning for navigation. IEFE Robotics and Automation Letters, 2019.
[5] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: A General Navigation Model to Drive Any Robot. In arXiV, 2022.
[6] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. VINT: Foundation Model for Visual Navigation. In 7th Annual Conference on Robot Learning (CoRL), 2023. 8
[7] S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman. Poni: Po- tential functions for objectgoal navigation with interaction free learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18890-18900, 2022. 2
[8] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Gryener. B. Mustafa, L. Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06799 06794, 2022. 2
[9] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learning, 2021. 2
[10] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra. Improving vision- and- languang navigation with image-text pairs from the web. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 259-277 259- Springer, 2020. 2
[11] A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra. Zson: Zero-shot object- goal navigation using multimodal goal embeddings. arXiv preprint arXiv:2206.12 12403. 2022.
[12] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kap- pler. Open-vocabulary queryable scene representations for real world planning. arXiv preprint arXiv:2209.09874 2022.
[13] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. arXiv preprint arX1v:2110.05714, 2022.
[14] D. Shah, B. Osinski, B. Ichter, and S. Levine. LM-nav: Robotic navigation with large pre- trained models of language, vision, and action. In Annual Conference on Robot Learning (CoRL), 2022. 2,

---

[15] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manipu- lation. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021. 2
[16] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory, 2023.
[17] K. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, Shkurti, and A. Tor- ralba. Conceptfusion: Open-set multimodal 3d mapping. arXiv, 2023. 2
[18] V. Dorbala, J. F. J. Mullen, and D. Manocha. Can an Embodied Agent Find Your "Cat- shaped Mug"? LLM-Based Zero-Shot Object Navigation, 2023. 2, 6, 7, 8
[19] O. Mees, Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data, 2023.
[20] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
[21] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05112, 2023.
[22] Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and motion planning with large language models for object rearrangement, 2023.
[23] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: From natural language instructions feasible plans, 2023. 2,
[24] W. Huang. P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Ex- tracting actionable knowledge for embodied agents. In International Conference Machine Learning (ICML), 2022. 2,
[25] B. Ichter, A. Brohan, Y. Chebotar. C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. To- shev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K.-H. Lee, Y. Kuang, S. Jesmonth, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as i can, not as say: Grounding language in robotic affordances. In Annual Conference on Robot Learning (CoRL), 2022. 2.3
[26] K. Valmeekam, A. Olmo, S. Sreedharan, and S. Kambhampati. Large language models still can't plan (a benchmark for 11ss on planning and reasoning about change), 2023. 2
[27] W. Huang, F. Xia, D. Shah, D. Driess, A. Zeng, Lu, P. Florence, I. Mordatch, S. Levine, K. Hausman, and B. Ichter. Grounded decoding: Guiding text generation with grounded mod- els for robot control, 2023.
[28] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint, 2022.
3
[29] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv. L. Cui, O. K. Mohammed, B. Patra, Q. Liu, K. Aggarwal, Z. Chi, J. Bjorck, V. Chaudhary, Som, X. Song, and Wei. Language is not all you need: Aligning perception with language models, 2023.
[30] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, Mordatch, and P. Florence. Palm-e: embodied multimodal language model. In arXiv preprint, 2023. 3
[31] B. Yamauchi. A frontier-based approach for autonomous exploration. In IEEE International Symposium Computational Intelligence in Robotics and Automation (CIRA), 1997. 3
10

---

[32] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and
D. Zhou. Chain of thought prompting elicits reasoning in large language models. In Neural Information Processing Systems (NeurIPS), 2022. 4
[33] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating to objects in the real world. Science Robotics, 2023. 5, 6, 7,8
[34] D. Shah and S. Levine. Viking: Vision-based kilometer-scale navigation with geographic hints. In Robotics: Science and Systems (RSS), 2022. 5, 7, 8, 13
[35] X. Zhou, R. Girdhar, A. Joulin, Krähenbühl, and Misra. Detecting twenty-thousane classes using image-level supervision. In 17th European Conference on Computer Vision (ECCV), 2022. doi: 10. 100.100777977733-0-000777-77 21. 5,8
[36] K. Yadav, S. Ramakrishnan, J. Turner, A. Gokaslan, O. Maksymets, R. Jain, Ramrakhya, A. X. Chang, A. Clegg, M. Savva, E. Undersander, D. S. Chaplot, and D. Batra. Habitat challenge 2022.
2022.
[37] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD- PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames. In International Conference on Learning Representations (ICLR), 2020. 6, 8
[38] D. S. Chaplot, H. Jiang, S. Gupta, and A. Gupta. Semantic curiosity for active visual learning. In ECCV, 2020. 6, 8, 13
[39] B. Yu, Kasaei, and M. Cao. L3mvn: Leveraging large language models for visual target navigation, 2023. 6, 8
[40] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine. Rapid exploration for open- open-world navi- gation with latent goal models. In 5th Annual Conference on Robot Learning, 2021.
[41] A. Sridhar, D. Shah, C. Glossop, and S. Levine. NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration. In arXiv, 2023. 8, 13
[42] K. Yadav, A. Majumdar, R. Ramrakhya, N. Yokoyama, Baevski, Z. Kira, O. Maksymets, and D. Batra. Ovrl-v2: A simple state-of-art baseline for imagenav and objectnav, 2023. 8

---

A Implementation Details
A.1 Hyperparameters
Parameter
Value
Replanning Rate
6
Language Influncce Threshold
2m
Number of LLM Samples
10
Wp
Weight of Positive Scores
300
Wn
Weight of Negative Scores
150
Max Time Steps
500
Table 3: Hyperparameters
A.2
Computational Resources
Parameter
Value
LLM
3p 3.5-turbo
Evaluation Runtime
5 hours
Compute Resources
4 x V100
Total LLM Tokens
30M
Average API Cost
15 USD
Table Parameters resources required rum one evaluation round LFG on benchmark.
A.3 Real World Results
For both topological and geometric maps hand engineered methods for clustering objects in ways that the LLM can efficiently reason over. For geometric maps we im- plement two functions: par seObjects and clusterObeects. filters the geometric map and identifies the cluster centers among each class, clusterObjects takes the cluster centers and performs agglomerative clustering with a threshold of meters, which is roughly the size of one section of a standard house. For topological maps we rely on the configu- ration of the four cameras automatically perform parsing and clustering. our implementation all the objects detected in each frame from either the front, left, right, or rear facing cameras is considered a single cluster.
Generating Prompts:
our implementation, parseObejcts
Perception: For the hardware, we use a locobot base with a four HD logitech web cameras that are positioned 90 degrees relative each other. At each step of LFG each of four cameras is recorded and frames are semantically annotated. LFG directly uses these frames to determine the robot should continue to move forward, turn left, turn right, or turn around full 180 degrees. To improve the performance of our system we choose to whitelist a subset of the 20,000 classes. This reduces the size of the API calls to the language models and helps steer the LLM focus on more useful information. Following is the complete whitelist used in our experiments:
toaster
water faucet
coffee maker
projector
faucet
coffeepot
chair
microwave oven
dining table
kitchen table
toaster oven
table
sink
oven
bathtub
kitchen sink
coffee table
bath towel
12

---

urinal
chandelier
computer monitor
toilet
ottoman
computer keyboard
toilet tissue
dresser
laptop computer
refrigerator
curtain
desk
automatic washer
shower curtain
stool
washbasin
trash can
hand towel
dishwasher
garbage
shampoo
television set
cabinet
sofa
file cabinet
soap
sofa bed
monitor
(computer
drawer
bed
equipment)
pillow
Low-level Policy: The low-level policy running on the robot the NoMaD goal-conditioned dif- fusion policy trained to avoid obstacles during exploration and determine which frontiers can be explored further [41].
For real-world experiments, we follow the setup of ViKiNG [34], where the agent runs simple fronticr-based exploration algorithm and incorporates the LLM scores as goal- directed heuristics to pick the best subgoal frontier. For simulation experiments, map coupled with frontier-based exploration, following the setup of Chaplot al. [38]. Algorithms 3 and 4 summarize the high-level planning module in both cases.
High-level Planning:
a geometric
Algorithm 3: Instantiating LFG with Topological Mapping
Data: og. Goal language query q subgoal None
2 while not done do
Oc ==gttObservatiin(
frontierPoints + mappingModulc(()) if q frontierPoints then uunTovarrGoal(fooniePoonn))
7
else
if numSteps % T == 0 then
location - geCCurretLoocation)))
LLM LLM ney < sconeFoonteessffoottrPPponss scores for point in frontier do
11
distance +- distTo(location, point)
13
scores[point] +- LLM pos [i] - wn. LLM neg [i] distance
14
subgoal +- argmax(scores)
15
numSteps <- numSteps +1
16
goTo(subGoal)
A.4 More Experiment Rollouts
Figure 6 shows an example where the negative scoring is essential LFG's success. Figures and 8 show examples of LFG deployed in a previously unseen apartment and an office building, successfully exploring the environments find oven and a kitchen sink.
B Prompts
B.1 Positive Prompt
13

---

Algorithm 4: Instantiating LFG with Geometric Mapping
Data: og, Goal language query 4
subgoal None
while not done do
3
- gctObseervvaoon
obstacleMap, semanticMap < mappingMMocullloo.(dpiiii oz if in semanticMap then
os[semantic])
5
subGoal + gettLocation(semantiiMaapp 9)
else
if numSteps % == 0 then
// replanning
location -- getCurrentLocatio))))
frontier sedFrontiie(oossaaleeMaa)))
11
objects + passeOhieccs(smmmmtieaa))))
objectClusters clusterObjeecs(cohjeess)))
13
LLM po.. LLM neg + ScoreSsbggalsscccicCCllsssss)
scores []
15
for point in frontier do
16
distance t distTo(location, point)
17
scores[point] + distance
18
closestCluster + geiCloessCusseroojcCCClttssssss point) + ClusterID(closestCuuset) if dist point) < then
19
20
// incorporate language scores
21
scores[point] <- "p- L.M pos [i] Wp. LLM neg [i] distance
22
subgoal f- argmax(scores)
23
numSteps + numSteps +1
goToosubGoal)
Query: Find the toilet,
1. LLM finds bed, increases to explore nearby.
2. No toilet found, LLM failure, FBE takes over.
3. FBE finds toilet continuing exploration.
Agent succeeds!
Figure Tolerance to LLM failures. An example rollout of LFG compensating for LLM failure. FBE takes over this case and eventually succeeds, whereas the Greedy agent fails.
You are a robot exploring an environment for the first time. You will be given an object to look for and should provide guidance of where to explore based on a series of observations. Observations will be given as a list object clusters numbered to N.
Your job is to provide guidance about where we should explore next. For example if we are in a house and looking for tv we should explore areas that typically have tv's such bedrooms and living rooms.
14

---

Query: Find an Oven
"Ovens typically found in kitchens. Cabinets and microwaves are typically found in kitchens so we should explore toward those objects."
LLM:
D
Figure LFG in an unseen apartment. The robot starts the same starting location and environment as and tasked finding an oven. LFG guides robot towards the kitchen appliances, rather than bedroom door, and successfully leads to the oven.
Query: Find an Sink
LLM: ""A sink will likely be found in bathroom or a kitchen. Therefore toilet tissue and refrigerators may be near a sink. Let's explore these regions first and do not get closer to the goal, can continue exploring until there is stronger signal"*
C
Erroneous Detections
Figure LFG an unseen office building. agent looks sink an open- penn-lann office building. De- spite erroneous detections, the robot continues exploring the environment, with LFG guiding towards frontiers containing appliances found in cafe. The robot successfully finds the sink despite imperfect detections.
You should always provide reasoning along with number identifying where we should explore. If there are multiple right answers you should separate them with commas. Always include Reasoning: <your reasoning> and Answer <your suitable
(nswerss)>. If there space afters Answer: blank.
leave the
Example
User:
observe the following clusters objects while exploring a
house:
1. sofa, tv, speaker
2. desk, chair, computer
3. sink, microwave, refrigerator
15

---

Where should search next am looking for knife?
Assistant:
Reasoning: Knifes are typically kept in the kitchen and a sink, microwave, and refrigerator are commonly found in kitchens Therefore we should check the cluster that is likely to be a kitchen first.
Answer: 3
Other considerations
1. Disregard the frequency of the objects listed on each line. If there are multiple of the same item in cluster it will only be listed once in that cluster.
2. You will only be given list of common items found in the environment. You will not be given room labels. Use your best judgement when determining what room a cluster objects is likely to belong to.
16

---

B.2 Negative Prompt
You are a robot exploring an environment for the first time. You will be given an object to look for and should provide guidance of where to explore based on a series of observations. Observations will be given as list of object clusters numbered to N.
Your job is to provide guidance about where we should not waste time exploring For example if are in house and looking for a tv we should not waste time looking in the bathroom. It is your job point this out.
You should always provide reasoning along with number identifying where we should not explore. If there are multiple right answers you should separate them with commas Always include Reasoning: <your reasoning> and Answer: < your answer (s)>. there are no suitable answers leave the space afters Answer: blank.
Example
User:
observe the following clusters objects while exploring a
house:
1. sofa, speaker
2. desk, chair, computer sink, microwave, refrigerator
3.
Where should I avoid spending time searching if I looking for a knife?
Assistant:
Reasoning:
Knifes are typically not kept living room or office space which is what the objects in 1 and 2 suggest. Therefore you should avoid looking in 1 and 2.
Answer: 1,2
Other considerations
Disregard the frequency the objects listed on each line. If there are multiple of the same item in cluster it will only be listed once in that cluster. 2. You will only be given list of common items found in the environment. You will not be given room labels. Use your best judgement when determining what room cluster of objects is likely to belong to.
17


Reviews:
[{"review_id": "i0SxgWyptt", "reviewer": "Reviewer_JLdp", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method called LFG that uses LLM’s to enable robots to efficiently find objects in previously-unseen environments. The main idea is to leverage the common-sense knowledge from LLM’s as a prior to guide a search-based planning algorithm. This enables the method to efficiently find objects when the LLM outputs useful knowledge while also being able to fall back onto the classical frontier exploration planning algorithm when the LLM outputs are useless or nonsensical. Instead of attempting to use the logprobs of the LLM as many related approaches do, the authors poll the LLM using a specifically-designed prompt and leverage chain-of-thought prompting to improve the quality of polling output. Experiments on tasks from the Habitat ObjectNav Challenge demonstrate the the proposed method is able to perform on par with or better than baselines despite not requiring any training.", "summary_of_recommendation": "Overall, this is an interesting and well-written paper that contributes a well-motivated method to the robotics community. Though I found the work slightly incremental, I believe that this is above the bar for acceptance at CoRL provided the paper is updated to provide details answering the various questions I listed above.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In addition to the questions listed under the first bullet point in the above ‘weaknesses’ sub section, I would like to ask:\n\n- Did the authors run an ablation where the number of LLM samples (n_s) is set to 1?\n- What is $\\phi$ supposed to represent in Algorithm 1?\n- Have I misunderstood anything/missed any important details? If so, please point them out; I’m more than happy to adjust my review accordingly!", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n\n- *Main ideas are interesting and well-motivated*. The core ideas of (1) using an LLM to guide a search-based planner instead of directly planning, and (2) scoring candidates by polling instead of using logprobs are very well motivated, sensible, and interesting. Given the large amount of related work that attempts to use LLM’s to replace planners, I believe (1) is a particularly useful contribution.\n- *Clear writing and presentation*. The presentation makes the method simple and quite easy to understand.\n- *Solid empirical results* The presented empirical results on the ObjectNav challenge are impressive and rather convincing.\n\n## Weaknesses\n\n- *Lack of a few important, relevant details on the method*. While the authors provide a good high-level picture of their proposed method, there are a number of details that aren’t mentioned in the main paper or the supplementary material. Specifically, I have the following questions about the method:\n    - How do the `parseObjects()` and `clusterObjects()` methods in Algorithm 2 work?: These seem like rather important methods, yet they are mentioned only in a cursory fashion.\n        - Does `parseObjects()` simply return the ‘semantic label’ associated with every single object in the map? Or is it only the set of objects within the current frontier?\n        - Does `clusterObjects()` perform clustering by the objects’ semantic label(s), distance, or both?\n            - What sort of clustering algorithm is used? Is the size of each cluster pre-determined?\n            - Is there an implicit assumption here that there aren’t that many different objects with different classes in the map/environment? It occurs to me that if there are (which happens, for instance, in the [BEHAVIOR dataset](https://behavior.stanford.edu/)), then the clustering approach might struggle. Moreover, there might be too many objects to even fit into the prompt, given the context-window of the LLM. What exactly happens in the case where there are a large number of objects with potentially very different semantic labels?\n    - What exactly is one ‘step’ in Algorithm 2? You mention that replanning is triggered when `numSteps % \\tau == 0` , is one step one command to the robot’s motors (occurring at something like 30 Hz), or is it one completed navigation to a particular point? I’d assume it’s the latter because the supplement mentions that tau is set to 1…\n    - Is the ‘semantic label’ of an object simply the object category? Does it include any other information?\n    - Can the LLM respond with multiple positive and negative subgoals, or only one each? Lines 146 to 150 in the paper make it seem like there can only be one positive and one negative subgoal id output by the LLM, but Section C.2 in the supplementary material show the case where two negative goals are output.\n    - What exactly is the `Ask` ablation in Table 2? I couldn’t find a description of it anywhere in the text.\n- *Incremental method (minor).* While the presented method is interesting and well-motivated, the application to object-finding feels rather limited. The presented improvements are statistically-significant, but not that much more than existing baselines. It is unclear how this approach could be built upon to enable even more progress on the ObjectNav challenge and the wider object search problem. It would be much more significant and impressive if the authors expanded their technique to navigation and manipulation tasks, perhaps leveraging the LLM for both object-search and for exploring properties towards manipulation of objects (e.g.: leveraging the LLM so that the robot can understand that it is difficult to pick up a chair, but a chair can be dragged, etc.). I acknowledge that this is a much harder problem within a broader research agenda, but it feels like the common-sense knowledge in LLM’s can be utilized for much more than efficient object search. In my opinion, showcasing some aspect of this wider applicability would make the paper’s contribution(s) much more significant .\n\n## Suggestions\n\nIn addition to answering all the listed questions and updating the text accordingly, I have the following relatively minor suggestions:\n\n- Consider mentioning approaches that perform navigation and object search by solving POMDP’s (e.g. [this semi-recent work](https://arxiv.org/abs/2005.02878)). Some of these approaches (e.g. [this one](https://ieeexplore.ieee.org/document/8793888)) are even guided by language input, and they seem quite closely-related to your setting.\n- Subsection 6.1 appears blank; remove it", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "SbOc9OaNoyU", "reviewer": "Reviewer_4mQY", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the problem of robot navigation in unmapped settings. The contribution is the Language Frontier Guide (LFG), which uses LLMs to bias how a robot explores its environment. It polls an LLM to identify the objects in its search frontier that are most likely to be semantically indicative of the search goal. The experimental results show how LFG enables more successful navigation on the Habitat ObjectNav benchmark compared to learning-based and other LLM-based methods.", "summary_of_recommendation": "Overall, the paper is well-written and contributes an effective method for navigation in unfamiliar environments. However, it does not contribute any methods for robot learning and thus may not be suited for CoRL. Additionally, the results rely on an unfair comparison in which the proposed method is provided more information than the learning-based baselines.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Clarifications:\n\n* Is there any data showing a more fair comparison to learning-based methods? I.e., where those methods have access to ground-truth semantic information?\n\n* Does the sequence order of explored spaces in M inform the heuristic?\n\n* Line 236: LFG \"maxes out the benchmark\" -- What does this mean? It doesn't reach full success at the task. \n\n* SPL reflects the success relative to the explored path length. It would be useful to have data about the path length itself in order to assess whether LFG enables more efficient exploration than other methods.\n\n* Section 6.1 is empty. Is 6.2 meant to be a subsection of it? (i.e., 6.2 should be 6.1.1?)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-rebuttal:\n\nThank you for your thorough rebuttal. I appreciate the extensive work the authors have put into their real-world study, and believe this strengthens the claims of the paper.\n\nI am convinced that LFG is an effective heuristic for guiding a robot's navigation using semantic information. The original evaluation was extensive and compelling, and the newly-added experiment in a real-world setting strengthens this claim even further.\n\nI am still not convinced, however, that there is a learning contribution here. While it's true that CoRL has accepted papers that integrate LLMs, those papers proposed new technical contributions via the additional learning algorithms that enable the use/adaptation of LLMs. SayCan, for example, describes an RL algorithm for learning task models that are conditioned on language descriptions.\n\nIn contrast, LFG's technical contribution is a planning heuristic. While it is demonstrated to be a highly effective heuristic that could have impact in the planning community, I don't see it having impact as a robot learning paper. However, I am happy to defer to the area chair's view on this (and thus am changing my recommendation to \"weak reject\").\n\n----------\n\nStrengths:\n\n+ Overall, the paper is very well-organized and easy to read. \n\n+ The evaluation is thorough: it uses an extensive benchmark of simulated navigation tasks and contains comparisons to 8 different baseline methods. \n\n+ The discussion of *how* to formulate LLM prompts (and the comparison between their effectiveness) is interesting and may be useful to other researchers who are interested in integrating LLMs into their work.\n\nWeaknesses:\n\n- My biggest concern is whether this paper is a good fit for CoRL: it does not actually contain any robot learning (but rather, uses a pre-trained LLM). \n\n- The results imply that, compared to learning-based approaches, LFG enables better performance while also requiring zero training data. This is an unfair comparison, however; LFG relies on ground-truth semantics to identify the objects located near the robot, while the learning-based methods do not have this information and presumably must learn it from visual input. I expect that the large amounts of training data are actually needed to enable *visual* learning for these methods, rather than being necessary for the navigation task. Essentially, LFG and the learning-based methods are essentially solving different robotics problems, yet are compared equally in the evaluation.", "extra_scores": {"impact": "1: Not very relevant to the robot learning community."}, "extra_text": {}}, {"review_id": "SGE11pOEiQ7", "reviewer": "Reviewer_oCzb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes LFG, a heuristic planning algorithm exploiting semantic reasoning from LLM. The method more exactly extends Frontier-based exploration algorithm with semantic knowledge to navigate in unknown environments. The authors exploit prompts from LLM to score positively or negatively subgoals fed to an exploration policy. As so, the approach does not require any training and is validated on the HM3D ObjectNav Benchmark.\n ", "summary_of_recommendation": "The paper is clear, original and interesting for the community. The reviewers recommends the paper for CoRL.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1) Please clarify the role of $n_s$ in Algorithm 1. Line 140: It is not clear if each subgoal is sampled $n_s$ times or if $n_s$ is the number of subgoals.\n2) Please detail more Figure 2. What is the meaning of 1, 2 and 3?\n3) Line 149, please explain \"Answer i\". What does \"i\" refer to?\n4) There may be situations where all subgoals are not informative to reach the destination (i.e. low score). How is this handled by the proposed algorithm?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Globally, the paper is clear and exploits an original idea. This can pave the way to new methodologies benefiting from LLMs without the burden of grounding language. Furthermore, the results presented  on the HM3D ObjectNav benchmark as well as the ablation study support the contributions of this paper. Minor comments should be addressed by the authors.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "0uVzm6yffx", "reviewer": "Reviewer_FZa9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes \"Language Frontier Guide\" (LFG), an approach to using pretrained large language models (LLM) to aid in navigating through an unknown environment to reach a semantic goal. The method involves using a unique querying method to aggregate responses to both \"positive\" and \"negative\" language prompts into scores used to bias frontier-based exploration towards frontiers more likely to lead to the object of interest. Results in the Habitat simulator demonstrate improved performance over both non-learned \"FBE\" (frontier-based explroration) approaches as well as learned approaches trained specifically for the Habitat environments, demonstrating the potential long-term utility of using off-the-shelf LLMs to inform search behaviors.", "summary_of_recommendation": "Overall this is a fairly solid paper with a useful contribution that pushes forwards our understanding of how LLMs will shape robot decision-making. The paper and its results are good, yet the presentation of the material is potentially confusing to a subset of readers, and the authors should take some time to make the language more specific so as to avoid this confusion. A few more clarifications about methodological details should also be included (even if only in the Appendix).", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "[These questions have been reproduced here from my full review above for clarity.]\n\n- Clarification: how were the weights chosen? Why is it that the values differ for the positive and negative weights? How sensitive to those parameters do the authors expect the behavior to be?\n- Clarification: the text and appendix are clear about how prompting is done for the purposes of scoring via Alg. 1, but can the authors also clarify the process for computing the \"logprobs\" (e.g., what prompt was used and what completions provided)?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is generally well-written and the approach is sound. The results are also promising and show good performance on a challenging task without any domain-specific training.\n\nThe central issue is not with the method or results, but with the language used to describe the method. Multiple places use language that may imply that planning sequential decision making (i.e., via the Bellman Equation) is being used, especially since the non-learned baseline is referred to as a \"Greedy\" strategy. It seems (see Alg. 2) that the presented LFG approach is *also* a greedy approach, and so naming it such is misleading. [Note: I recognize that \"search\" is an overloaded term here, but I think it is possible to rework some of the language to make it more easily to understand where the work is situated with respect to other approaches.]\n\nA related suggestion: it might be better if the text (see Line 52, 175) did not use the phrase \"search heuristic\" to describe the LLM contribution of the LFG approach as algorithmic search is not used, even though the robot is 'searching' the environment. Similarly, it might be more appropriate to characterize the approach as a method of \"frontier selection\" rather than \"planning\", as used in lines 200 and 201 (among others). Finally, changing the name of the baseline approach (currently the Greedy approach) to something more appropriate: e.g., non-learned baseline, \"nearest-FBE\", or similar.\n\nI welcome feedback from the authors on these points, but I think it would be best to rework some of the language to make clearer the role that the LLM guidance plays and what type of search process is used. This would improve the paper and readability.\n\nOther comments and questions:\n- Clarification: how were the weights chosen? Why is it that the values differ for the positive and negative weights? How sensitive to those parameters do the authors expect the behavior to be?\n- Clarification: the text and appendix are clear about how prompting is done for the purposes of scoring via Alg. 1, but can the authors also clarify the process for computing the \"logprobs\" (e.g., what prompt was used and what completions provided)?\n- There appear to be footnotes 3 and 4 in the Appendix, but they do not seem to correspond to any text.\n- Table 2: Please use a symbol other than \\pi to reference the in-caption footnote, as \\pi is used in planning literature to represent a policy. Instead, a \\dagger might be more appropriate and would avoid confusion (e.g., that an asterix * might imply).\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper concerns navigation in unknown environments and proposes the use of a frozen LLMs to rank exploration goals by the probability that a candidate will lead to the goal. 
The core contribution lies in how to appropriately poll the LLM (CoT, +ve/-ve queries etc.) and propose a ranking function (Eq. 1) that includes LLM outputs.  

During the rebuttal phase the authors provided clarifications on the baselines and included a real robot experiments. If accepted, authors are requested to carefully incorporate the suggestions provided by the reviewers. 

Further, concerns emerged with regard to whether the work is suitable for CoRL since direct use of frozen LLMs departs from prevailing practice of explicitly training models and deploying them on robots.  Since the understanding, methodology and use of LLMs is still emerging it may be prudent to  engage inclusively in the interim. This question merits a broader discussion in the wider CoRL community.
Decision: Accept (Poster)