Tell Me Where to Go: A Composable Framework for Context-Aware Embodied Robot Navigation
Authors: Harel Biggie
Ajay Narasimha Mopidevi
Dusty Woods
Chris Heckman
Venue: CoRL 2023 Poster (2023)
Keywords: Natural language
navigation
contextual navigation

Abstract: Humans have the remarkable ability to navigate through unfamiliar environments by solely relying on our prior knowledge and descriptions of the environment. For robots to perform the same type of navigation, they need to be able to associate natural language descriptions with their associated physical environment with a limited amount of prior knowledge. Recently, Large Language Models (LLMs) have been able to reason over billions of parameters and utilize them in multi-modal chat-based natural language responses. However, LLMs lack real-world awareness and their outputs are not always predictable. In this work, we develop a low-bandwidth framework that solves this lack of real-world generalization by creating an intermediate layer between an LLM and a robot navigation framework in the form of Python code. Our intermediate shoehorns the vast prior knowledge inherent in an LLM model into a series of input and output API instructions that a mobile robot can understand. We evaluate our method across four different environments and command classes on a mobile robot and highlight our framework's ability to interpret contextual commands.

Tell Me Where to Go: A Composable Framework for Context-Aware Embodied Robot Navigation
Harel Biggie
Ajay Narasimha Mopidevi Department of Computer Science University of Colorado Boulder ajay. mopideeii@coloraao edu
Department of Computer Science
University of Colorado Boulder United States harel. biggie@colorado. edu
Destin Woods
Christoffer Heckman
Department of Computer Science
Department of Computer Science
University of Colorado Boulder destin. wtt.i...oodd@oooordaae edu
University of Colorado Boulder christoffer heckman@colorado edu
Abstract: Humans have the remarkable ability navigate through unfamiliar environments by solely relying
prior knowledge and descriptions of the environment. For robots to perform the same type of navigation, they need to be able to associate natural language descriptions with their associated physical environment with a limited amount of prior knowledge. Recently, Large Language Models (LLMs) have been able to reason over billions of parameters and utilize them multi-modal chat-based natural language responses. However, L.LMs lack real-world awareness and their outputs are always predictable. In this work, we develop low-bandwidth framework (NavCon) that solves this lack of real- world generalization by creating intermediate layer between LLM and robot navigation framework in the form of Python code. Our intermediate layer shoehorns the vast prior knowledge inherent in an LLM model into a series of input and output API instructions that mobile robot can understand. We evaluate our method across four different environments and command classes on mobile robot and highlight our framework's 's ability to interpret contextual commands.
Keywords: Natural language, navigation, contextual navigation
Introduction
Humans have the remarkable ability to navigate through unfamiliar environments, e.g. in a town or through building. by relying solely
priors and descriptions of the environment [1]. Moti- vated by the difficulties in directing robots in collaborative teams with humans such as those used in search and rescue [2, 3, 4, 5] operations, we aim to develop a framework that allows humans to provide high-bandwidth instructions to robots in the form of natural language.
Instruction and Visual Input
Contextual Navigation
Generated Intermediate Layer
Navigation Output
Prelighoer sornetinng with thrs.
injjects l'Erretsphtent, fire truck!. "hydrant", "hose Sattec! peiches
dejects:
ebject petches
inage. finetob))
inputs 19b1 terizontal center, db) patch vertical center) anputs, "butss)
( l'fancsiont:
to soject
Figure 1: Contextual navigation example for firefighting situation. We compositional framework generates code for the robot to both find and then plan to a fire extinguisher.
To achieve this, generally required that a robot associate natural language utterances to the physical world using sensing modalities onboard the robot, a process known as grounding. However,
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

unlike humans, robots are not currently capable of integrating prior experiences into a vast wealth of priors to aid this association.
Recently, Large Language Models (LLMs) [6] have been able to reason over billions of parame- ters and utilize the results in domains such as dialogue-based responses [7, 8, 9], code generation [10, 11, 12], and multimodal visual question and answering (VQA) tasks [13]. This type of unsuper- vised learning shows incredible potential for the generalization of prior knowledge but these models lack real-world experience. A lack of physical experience is perhaps one reason why transitioning techniques developed through research in VQA and view-based navigation into embodied agents has not kept pace with the research in these fields.
Additionally, with the current generation of LLMs, it's inherently challenging to interpret the ratio- nale behind the outputs that are generated by the model. In this work, we address both the grounding and transparency issues using a composable framework designed for robot navigation. Our pipeline extends the reasoning framework for visual inference presented in [14] to an embodied robotic agent. With our composable framework, we leverage LLM priors (GPT-3.5), state-of-the-ar object detectors [15], and classical robotic planning algorithms [16, 17, 18] to perform zero-shot natural language based navigation in four unique environments.
Specifically, our contributions are as follows:
We extend the concept of modular neural networks and define new framework for com- posable robot navigation which we call NavCon. Our framework requires minimal uplink for the robot since all of our planning, mapping, and localization is performed onboard.
We evaluate different 2D input representations determine effective way to extract spatial and conceptual knowledge from LLMs.
We perform extensive real-world experiments in variety of environments and show that our framework is able to navigate to landmarks based on natural language. Furthermore, the framework is able to deduce appropriate navigational goals from the context of a sentence.
2 Related Works
Associating natural language with embodied agent requires ground- ing utterances to the physical world in which the robot operates. Various approaches have been used to associate language with the physical domain ranging from probabilistic graph-based struc- tures [19, 20, 21] to end-to-end cearning-based methods [22]. Graph-based approaches have shown promising results but their generalizability is limited to fixed training corpus. On the other hand, LLMs have proven to be adept at reasoning over large amounts of unsupervised training data [23, 24] using transformer-baseeee backends [25]. These models also known as foundation models due their comprehensive knowledge have recently been leveraged for task and motion planning using rein- forcement learning set of skills [26], and in end-to-end fashion [22] with object scene repre- sentation transformers [27]. While these approaches have made remarkable strides they still suffer from the explainability problem and further they don't 't share any of the path planning guarantees that traditional planning methods share [28].
Grounding Language:
Modular Neural Network Frameworks and Code Synthesis: Modular Neural Networks [29, 30, 31] are remarkably adept at answering questions about images in task commonly known as VQA [32]. Until recently many of these methods were limited in their ability to generalize to other domains due to the difficulty of generating interfaces between modules. In [14] this was addressed by using recent code generation techniques [33] from LLMs to create these interface layers. Specif- ically, these modular frameworks enable the generation code that dictates the interaction between robust object detectors [15, 34, 35, 36, 37J. depth estimators [38, 39, 40J. What's 's more remarkable is that these L.LMs can also utilize existing functions inside of the Python language such as sorting and conditionals without any additional training [41, 42, 12, 24]. Based these exceptional findings we aim to extend the use of these modular concept-leaming style frameworks to the robotic navigation domain.

---

(a) Courtyard
(b) Lobby
(c) Outdoor
(d) Theater
Figure 2: Full volumetric maps and representative images from the four testing environments.
Path Planning for Robotic Navigation: Path planning for robotic navigation and exploration is a long-studied research area [43, 44] with solutions ranging from campling-based methods[16, 45] to graph-based lattice structures [46]. Variations of these probabilistic approaches are being used for self-driving cars [47], exploration [18, 4, 3, 5] in complex environments. These approaches benefit from probabilistic guarantees on convergence and optimality [16] and adaptability to new environments making them natural choices to utilize in language grounding framework.
Embodied Language-Basel Navigation: Recently a few other works have leveraged foundation models for embodied navigation. Clip on Wheels (CoW) is proposed in [48] which combines text and image caption models [49] with frontier-based exploration algorithms [50] to perform object exploration in simulation. In [51] landmark-based navigation performed using pre-trained visual [48], language, and navigation modules [52]. Visual Language Navigation Maps (VLMaps) [53] learn spatial language representation by combining RGB video feeds, code generation, and foun- dation models. SayCan [54], and Palm-E [22] utilize the PALM [55] language model to perform skill-based navigation and end-to-end navigation respectively. In general, these methods are not tested on contextual examples, and some require the preprocessing of maps; in contrast, we test on situated examples with an experimental platform and require no situated priors.
Methodology
We develop framework for Navigation with Context (NavCon) that leverages the rich contextual priors of LLMs and creates executable code layer that interfaces with planning algorithms run- ning on an embodied agent. Drawing on paradigms from modular neural networks [56, 57, 14], and concept learning techniques [58], we define a modular system that takes inputs at various stages to fully define a final navigational output for our system. This enables the use of arbitrary intermedi- ate and substitutable layers which can also be executed on a distributed computing infrastructure. Formally, we have collection of inputs vb composed of visual input E V RGB image or pelti-perspe collection of images, m M volumetric map of the space to be navigated, and C' a natural language navigation command. Our system takes initially as input command to generate code %e. This code consumes u in order to resolve the grounding problem, i.e., to determine which object is being referred to in e for navigational instructions, as well as m in order to emplace the object into the world around the robot. This outputs = (v, ml/e) E R3, 3D waypoint to which navigation will commence in order to output P a robot trajectory which is continuous function in 3 A graphical overview of the framework can be found in Figure
Input Representations: Our visual input e is in the form of an RGB image, either semi-panorante view or three separately labeled spatial images (left, front, right). Typical inference methods over RGB images [36, 35, 15] reason in 2D over the image. For embodied navigation, we need to reason in 3D and associate different camera viewpoints with their associated 3D spatial relations. Embodied navigation introduces spatial relations that are difficult to reason over in 2D such as "behind", "in front of". "on your right". To determine an appropriate input representation we evaluate the difference between sending in concatenated image of all viewpoints panoramin)) from the agent or sending in each frame separately with spatial definition, i.c., right, front, left. We
3

---

find that spatial reasoning is best performed on a concatenated image and we present the details in Section 4.
We generate 8 using recent advances in code generation models [10, 42]. We provide functional of navigation instructions in the form of Python API for the code generation model. The full prompt containing the API can be found in the Appendix. These instructions include specifications for how to perform visual inference on the input using similar paradigm found in [14]. These API instructions include directions for finding an object, checking if an object has certain property, etc. Additionally, we provide an API specification for interfacing with our geometric planner, resulting code generated based on the natural language prompt %o
Intermediate Layer:
Planning: We use graph-based planner first presented in [18, 4] where each sample is robot position parameterized by the robot's width and length. During sampling, the collision checks performed by projecting the samples along the path to the ground on the map m below the sample. The elevation change at the footprint of the sample is calculated and if these differences exceed given threshold or there are not enough sampled points (indicating a hole in the ground) the path marked invalid and not added to the graph. We create a navigational API that takes the center landmark I on 3D map. Using this
coordinate p of an object image space and projects strategy, a graph of plans is constructed by sampling points parameterized by the robot's width and length [16]. Additional semantic labels are added to the map for staircases described in [59] using normal estimation on point clouds. When 3D waypoint is input to the planner we select the best path P from G, for the robot to follow. If no path exists, we plan to the boundary of the graph and resample until the goal is reached.
Waypoints are passed the planner using the output from the intermediate layer. Specifically, image coordinates translated into waypoints by executing 0 and associating the result onto 3D map [60] using ray casting. The map is generated online using [61] and the translation layer creates the necessary code to translate between the inputs m as and the output i. We then plan to the waypoint by selecting the best path from P.
Map m
rb
Ye
Intermediate Layer A(v, m))))
Code Generation
Path Planning
P Navigation Output
Navigation API
Visual Input
the cettrz of object
Navigation Command
Sececsatee of tis serbac of the object
Figure 3: A graphical overview of the full system framework.
4 Experimental Results
We run two sets of experiments on Boston Dynamics Spot equipped with custom sensing suite consisting of 3D 64 beam Ouster lidar, an IMU, and RGB cameras providing a semi cami-panoramic view of the environment. Our first experiment is designed to determine the best input representation for the visual layer and the second set tests the ability of our system to perform navigation in a wide variety real-world environments.
For all of our experiments, leverage ideas from human concept learning as in [58] to categorize sentences into four categories: Generic, Specific, Relational, and Contextual. Generic sentences

---

are simply sentences that imply "Go to something" e.g. "Walk to the backpack.". Specific sentences include a distinguishing piece of language such as a color attrihute that directs the robot to one specific object in the scene. For example, there are two backpacks, one red, and one black, then a specific sentence would be "Drive to the black backpack." Relational sentences are any sentences that describe spatial relationships between objects in the scene such as, to the right of, on top of, etc. An example in this category is "Move the backpack top of the chair." Contextual examples, require the robot to interpret the navigational goal based on background information. For instance, the sentence "Find something that can carry water." requires the robot to know that cup bucket can hold water.
We generated sentences by eliciting verbs for navigational policies from an LLM (GPT 3.5-0301). Then objects in the were inventoried, and sentences were constructed based on verb-object combinations well their properties and relationships to other objects in the environment. These sentences were then diversified into contextual commands by providing reference to the objects' attributes or affordance and our full list of sentences can be found in the Appendix.
We evaluate the success the method based on the following criteria. The code generation step considered successful if the generated intermediate layer TCc is both syntactically and logically correct. In other words, if every component worked successfully then the generated code would solve the navigation command. Object Detection (OD) is considered successful if the correct object is extracted from the visual input in the intermediate layer as visualized by the bounding box. Simi- larly for waypoint projection (WP) if the 3D projection is in the correct location on the map m this step considered a success. The planning and execution step is considered successful if the robot plans and then traverses to within 0.2m of the target object. For all tables percentages are calculated based on the total number of evaluated sentences.
3D Waypoint
Target object found, proceed to route
Bounding box object found
Follow traaannedtrajcctoo
Waypoint
Target object found, proceed to route
Bounding box obuectfound found
Follow planned trajectory
Figure 4: Example sentences, visual detections, maps, and planned paths.
Encoding spatial relationships into model without concept of the physical world presents signif- icant challenges. We determine an effective scheme for inputting three different viewpoints into the framework through empirical evaluations. Take the sentence, "go to the chair on the right" implies only the right half of the total FOV should investigated but "Go to the backpack that is to the right of the chair." could be in any image. To solve these limitations we use two different sets of input representations, the first is all three images stitched together with padding in between frames (A). In this case, instruct the LLM on the order of the images (left, front, right) and let the model handle the spatial reasoning. In the second case, we process the frames individually (B) and let the model decide which frames to look at. We explicitly prompt the model with example code snippets that solve the difference between "to your right" and "right of" styles of language. at configuration. In this set of experiments, we evaluate the correctness of the generated code as well as the code produced the correct object in the map m. This scene was created lighting conditions as the emphasis was to evaluate the effect of the input representation. Results for successful code generation and object detections for the two input schemas are summarized in Table 1.
classroom setting with good
5

---

We also compare our results to OFA [62], a sequence- sequence mulli-moda model finetuned on RefCOCO [63] expressions for the object detection stage. An additional baseline is performed where we utilize an LLM (GPT-3.5-0301) to convert a navigational command to a referring expression such as the ones RefCOCO. Our full prompt for this pipeline provided the Appendix.
Category
Count
A (%)
(%)
OFA (%)
+F
(%)
Generic
12
100
100
25.0
50.0
Specific
12
91.67
66.67
25.0
50.0
Relational
86.67
53.33
6.67
13.33
Contextual
11
81.82
45.45
0
36.37
Total
50
90
66
14
36
Table 1: Results showing the difference between the concatenated input representation and the sequential representation B.
From Table I we find that concatenating images significantly outperforms sending in the three frames separately. As expected, in the generic navigation case we are able to achieve 100% suc- cess in generating the intermediate layer and for object identification. We see that the individual frame configuration (B) begins to break down when reasoning over specific objects. Specifically, the model fails reason spatial relations and object ordering when objects appear in more than one frame. For example, taking the sentence "Go to the middle outlet" only works when the three outlets are present in the same camera frame. If the middle outlet is in the front frame but the right outlet is in the right camera frame, this method fails. We explicitly see this in the generated code:
outlet_patchie sort (key=lambda x: x..oorizotaallcooooo center) middle_outlet = outlet_patcher [n (outlet_patches) // 2]
where each detected outlet is ordered based on its horizontal coordinate in the image. Since all outlets are merged into the same list the images have overlapping coordinates causing these failures. Configuration B struggles with relational sentences between objects for similar reasons.
In our second experiments, we evaluate the ability of our framework to navigate in real-world environments. The four environments shown in Figure range from an indoor lobby setting dark outdoor courtyard. We select these environments to highlight in multiple scenarios. In each environment, we test multiple sentences across the four categories (see the Appendix for the full list of sentences). Sentences are primarily based on landmarks already present in the environment but in some cases, we add additional artifacts to enable more extensive testing.
framework's ' ability to generalize
Category
Count
Code( %
OD(%) (%
WP %
Path& Exec( %
OFA (%)
OFA+
LLM
(%
Generic
22
100
81.82
68.18 18
68.18 18
13.64
22.73
Specific
19
89.47
89.47
78.95
73.68
0
15.79
Relational
44
72.73
59.10
59.10
59.10
18.19
36.37
Contextual
29
65.52
41.38
41.38
41.38
6.90
24.14
Total
114
78.95
64.04
59.65
58.77
11.40
27.19
Table 2: Summary of category-wiss success rate for each of the four command types.
We find that we are able to successfully generate code for navigational plans using a variety of verbs e.g., walk, go, drive, run, etc by leveraging the rich vocabulary knowledge present in foundation models. In fact, we can even say phrases like "sashay to the stop sign" which will be interpreted as a "go to" style navigation command.
From Tables 2 and code generation. In this case, code is manually evaluated and deemed successful if the generated
that for generic objects framework has 100% success rate for

---

Scenes
Count
Code( %
OD(%)
WP(()
Path& Exec( %)
OFA (%)
OFA+
LLM
(%)
Theater
30
90
66.67
63.33
3.33
13.33
Lobby
29
65.52
48.28
44.83
44.83
13.79
24.14
Outdoor
24
87.5
79.17
70.83
70.83
12.5
33.33
Courtyard
31
74.19
61.29
58.06
58.06
16.13
38.71
Total
114
78.95
64.04
59.65
58.77
11.40
27.19
Table 3: Summary of scene-wise success rate for each of the four command types.
code would produce the correct results assuming all the rest of the composed modules were executed successfully. In both tables Code represents the success of the code generation step, OD represents the success of the object detector (GLIP), WP is the success of the 3D waypoint projection and Path&&xec is the success of the graph-based planning and navigation module. Our failures occurred the object detection module where either the wrong object was detected or the object was not detected at all. However, since our framework is modular, the object detector can be substituted and as state-of-the-ee vision detectors improve our framework will improve in turn. Additional failures occur in the waypoint projection step. These occurred for two reasons. In the first case, the vision system detected further out than the map resulting in projections at the edge of the map rather than the actual object. For the second case, projections missed the correct voxel on the map. This was either due to ray "clipping" a closer object in the projection process. For longer projections (10- 12m), the camera-to-lidar calibration caused enough error to make the projection inaccurate, which could be resolved through mean point clustering. We also note that we observed 100% success rate in following all planned paths.
Expected
inage_patch InagePatch
tv. vv_patches image, pww.ppptch...ic 'tv")
eczeen, patches inage. findh sczeen
tv. hv__patches.. extend (screen, (screen_patchee ppatches)
Dest Expectations
image_ maage_patch..om wwt.ppttc.iieee find( couch
chair, couch, axtchndsaettedcchata patches)
chair_patches patches inage. image.patch.t.. patch.find "chir
Figure 5: Results for the sentence "Find me somewhere to watch a movie."
Selected Qualitative Examples:
Leveraging the vast knowledge base encoded into LLMs enables a degree of spatial reasoning and the ability infer navigation instructions based on the context of the sentence. For example, take the sentence "Find me something help firefighter" as shown in Figure requires the robot to identify list of objects that could help firefighter and then look for them. From the snippet of the generated code, we see that the robot successfully looks for fire extinguishers and fire hydrants, both of which would aid firefighter. Additionally, we are able to ask the robot to "Find something to clean up a mess" and it will find a mop or a broom.
What's more remarkable than simply interpreting correct navigation commands from context is our framework's 's ability to leverage foundation models find more refined solutions task in some instances. Take for example the sentence "Find me somewhere to watch movie," would expect the robot to go and find TV some kind of screen. Of course, the first object the robot looks for

---

is the screen but more remarkably the robot also looks for a chair which is closest to the screen. We can see this in Figure 5 where the generated layer explicitly looks for "TVs, screens" and then looks for "chairs" and "couches."
As powerful as the contextual engine (GPT-3.5) in framework, it still has limitations when interpreting the physical world. In Figure 6 we tell the robot to "Run upstairs." and to surprise, it will successfully generate code and plan up the stairs. However, when given the utterance "Go to the second floor." the following code is generated:
Floor_patche = ImagePatch (image) find('floor'
floor_patches sort (key=lambda x. iveetical_centtat if conffooo_pptthiiii return {'function': 'None', 'error': 'Image does not contain least two floors.'} '}
2:
fecor___loor_paacc floor_patches(1]
This code is nonsensical to the actual task because it's a literal interpretation of the sentence. Count- ing floors has nothing to do with going to the second story of a building. Examples like these high- light the need for additional methods encoding 3D spatial reasoning into natural language-bar embodied navigation frameworks.
Figure 6: Scene used to tell the robot to "Run upstairs" and "Go to the second floor". The second sentence fails due to the code trying to "count floors" "
5 Conclusion and Limitations
Our composable framework shows remarkable accuracy and performance in generating code that fits the navigation task. In controlled experiments, we achieve 90% accuracy across 50 different sentences. When taken to the real world the, model still performs exceptionally well with zero additional input. This is highlighted by 70% execution accuracy in large outdoor environments.
While this work makes significant strides toward embodied navigation, observe many opportuni- ties for future effort. Namely, operating in only 2D) space creates challenges for spatial relationships such as "behind" the robot or "to the right of." Additionally, we are limited by the visual range of the robot: therefore 3D projections need to be exponentially more accurate for both the smaller and fur- ther away an object is in the given input image. Large multimodal foundation models are promising, but promise uncertain runtime performance given their lack of availability and scalability. Finally, the current implementation of the framework is limited landmark-based aviation and future work includes extending this framework to include more navigational inputs such unbounded explo- ration in a direction or, having the robot explore until a condition is met. We also wish to include module that operates on 3D maps and module that supports sequential instructions.

---

Acknowledgments
This work was supported through the DARPA Subterranean Challenge cooperative agreement HR0011-18-2-0043, the National Science Foundation #1764092 and # #1830686, and USDA-NIFA #2021-67021-33450.
References
[1] P. McNamara, K. Hardy, and S. C. Hirtle. Subjective hierarchies in spatial memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15(2):211, 1989.
[2] DARPA. DARPA Subterranean Challenge, 2022. URL http./RRttttss///w..arra.....a wtt.pddwppaaaaippp...... darpa-suba wwwterraeenn--chaeeeeee challenge.
[3] A. Agha, Otsu, B. Morrell, D. D. Fan, R. Thakker, A. Santamaria- S.-K. Kim, A. Bouman, X. Lei, J. Edlund, al. Nebula: Quest for robotic autonomy in challenging envi- ronments; team costar at the darpa subterranean challenge. arXiv preprint arXi:22103.11470 2021.
[4] H. Biggie, E. R. Rush, D. G. Riley, S. Ahmad, M. T. Ohradzansky, K. Harlow, M. J. Miles, D. Torres, S. McGuire, W. Frew, et al. Flexible supervised autonomy for exploration in subterranean environments. arXiv preprint arXiv:2301.0077. 2023.
[5] M. Tranzatto, T. Miki, M. Dharmadhikari, L. Bernreiter, M. Kulkarni, F. Mascarich, O. Ander- sson, S. Khattak, M. Hutter, R. Siegwart, et al. Cerberus in the darpa subterranean challenge. Science Robotics, 7(66))ab 2022.
[6] W. X. Zhao, Zhou, Li, Tang, X. Wang. Y. Hou, Y. Min, Zhang. J. Zhang, Z. Dong, al. A survey of large language models. arXiv preprint arXiv:2303. 110.3.822 18223, 2023.
[7] A. Glaese, N. McAleese, M. Trebacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Wei- dinger, M. Chadwick, P. Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. /4375, 2022.
[8] M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill. Multimodal few- shot learning with frozen language models. Advances in Neural Information Processing Sys- tems, 34:200-2111 2222002-222202 2021.
[9] Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig. Pre-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023.
[10] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.08374, 2021.
[11] Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, Gimeno, Dal Lago, al. Competition-lewe code generation with alphacode. Science, 2022.
(7866624)11022-1000
[12] F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn. A systematic evaluation of large lan- guage models of code. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 1-10, 2022.
[13] J.-B Alayrac, J. Donahue, Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Mil- lican, M. Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-233 2022.

---

[14] D. Surís, S. Menon, and C. Vondrick. Vipergpt: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08/222 2023.
[15] L. H. Li, P. Zhang, H. Zhang. J. Yang, C. Li, Y. Zhong, L. Wang. Yuan, L. Zhang, J.- N. Hwang, et al. Grounded language-imag pre-training. In Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition, pages 10965-10975, 2022.
[16] M. LaValle et al. Rapidly-exploring random trees: new tool for path planning. 1998.
[17] T. Dang, M. Tranzatto, S. Khattak, F. Mascarich, K. Alexis, and M. Hutter. Graph-based sub- terranean exploration path planning using aerial and legged robots. Journal of Field Robotics, 37(8):1333-1388 2020.
[18] Ahmad and Humbert. Efficient sampling- basedd planning for subterranean exploration. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 7114-7121. IEEE, 2022.
[19] Tellex, T. Kollar, S. Dickerson, Walter, A. Banerjee, S. Teller, and Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 1507-1514, 2011.
[20] T. M. Howard, S. Tellex, and N. Roy. A natural language planner interface for mobile manip- ulators. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 6652-6659. IEEE, 2014.
[21] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In 2010 5th ACM/IEE International Conference Human-Robot Interaction (HRI), pages 259-266. IEEE, 2010.
[22] D. Driess, F. Xia, M. Sajjadi, C. Lynch, A. Chowdhery, Ichter, A. Wahid, Tompson, Q. Vuong, T. Yu, al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03376 2023.
[23] L. Floridi and M. Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines, 30:681-606 1994, 2020.
[24] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[26] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, Jang, R. Julian, al. Do can, not as say: Grounding language in robotic affordances. In Conference Robot Learning, pages 287-318. PMLR, 2023.
[27] M. S. Sajjadi, D. Duckworth, A. Mahendran, S. van Steenkiste, F. Pavetic, M. Lucic, L. J. Guibas, K. Greff, and T. Kipf. Object scene representation transformer. Advances in Neural Information Processing Systems, 35:9512-9524, 2022.
[28] I. Noreen, A. Khan, and Z. Habib. Optimal path planning using ntt based approaches: survey and future directions. International Journal of Advanced Computer Science and Applications, 1(11), 2016.
[29] Johnson, Hariharan, L. Van Der Maaten, J. Hoffman, L. Fei-Fei, C. Lawrence Zitnick. and R. Girshick. Inferring and executing programs for visual reasoning, In Proceedings the IEEE international conference computer vision, pages 2989-2998 2017.
10

---

[30] R. Hu. J. Andreas, M. Rohrbach, T. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804-813, 2017.
[31] K. J. Wu, C. Gan, A. Torralba, P. Kohli, and J. Tenenbaum. Neural-symbolie vqa: Disen- tangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
[32] S. Antol, A. Agrawal, Lu, M. Mitchell, D. Batra, C. Zitnick, and D. Parikh, Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433, 2015.
[33] A. Madaan, S. Zhou, U. Alon, Y. Yang, and G. Neubig. Language models of code are few-shot commonsense learners. arXiv preprint urXiv:2210.07/28, 2022.
[34] Li, D. Li, S. Savarese, and Hoi. 22 Bootstrapping engguageiimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.125597 2023.
[35] K. He. G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cmn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[36] Redmon, S. Divvala, Girshick, and A. Farhadi. You only look once: Unified, real- time object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 779-788, 2016.
[37] Z. Ge, S. Liu, Wang, Z. Li, and J. Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107. 107.08430, 2021.
[38] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623-1000 2020.
[39] C. Zhao, Q. Sun, C. Zhang, Y. Tang, and F. Qian. Monocular depth estimation based on deep learning: An overview. Science China Technological Sciences, 631)) 2020.
[40] S. F. Bhat, I. Alhashim, and P. Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4009 4018, 2021.
[41] Vaithilingam, T. Zhang, and E. L. Glassman. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference human factors in computing systems extended abstracts, pages 1-7, 2022.
[42] E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang. Zhou, S. Savarese, and C. Xiong. Code- gen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 13474, 2022.
[43] T. T. Mac, C. Copot, D. T. Tran, and R. De Keyser. Heuristic approaches in robot path planning: survey. Robotics and Autonomous Systems, 86:13-28, 2016.
[44] H.yy Zhang, W.-m. Lin, and A.-x. Chen. Path planning for the mobile robot: review. Sym- metry, 10(10):550, 2018.
[45] S. Karaman, M. R. Walter, A. Perez, E. Frazzoli, and S. Teller. Anytime motion planning using the rrt. In 201 IEEE international conference on robotics and automation, pages 1478-1483. IEEE, 2011.
[46] M. Pivtoraiko, R. A. Knepper, and A. Kelly. Differentially constrained mobile robot motion planning in state lattices. Journal of Field Roboties, 26(3):308-3 2009. 2009.
11

---

[47] X. Lan and S. Di Cairano. Continuous curvature path planning for semi-autonomouss vehicle maneuvers using rrt. In 2015 European control conference (ECC), pages 2360-2365. IEEE, 2013.
[48] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Clip on wheels: Zero-shot object navigation as object localization and exploration. arXiv preprint arXiv:2203.1044 2022.
[49] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning. pages 8748 8763. PMLR, 2021.
[50] B. Yamauchi. A frontier-based approach for autonomous exploration. In Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Aulomation CIRA'97, "oo''dosrr New Computational Principles for Robotics and Automation', pages 146
151. IEEE, 1997.
[51] D. Shah, B. Osiński, S. Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learming, pages 492-504.
PMLR, 2023.
[52] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and Levine. Ving: navigation with visual goals. In 2021 IEEE International Conference on Robotics and Automa- fion (ICRA), pages 13215-13222. IEEE, 2021.
Learning open- world
[53] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. arXiv preprint arXhv:2210.05714, 2022.
[54] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, et al. Do as can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.0169) 2022.
[55] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint 2rX1v:2204.02311, 2022.
[56] Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39 2016.
[57] T. Gupta and A. Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages
2023.
[58] Mao, C. Gan, P. Kohli, B. Tenenbaum, and Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint ar0iv: 12584, 2019.
[59] M. J. Miles, H. Biggie, and C. Heckman. Terrain-aware semantic mapping for cooperative subterranean exploration. Frontiers in Robotics and AI, 10: 1249586, 2023.
[60] A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard. Octomap: An efficient probabilistic 3d mapping framework based on octrees. Autonomous robots, 34:189-206, 2013.
[61] T. Shan, B. Englot, D. Meyers, W. Wang, C. Ratti, and D. Rus. Lio-sam: Tightly-coupedd lidar inertial odometry via smoothing and mapping. In 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 5135 51422. IEEE, 2020.
[62] Wang, A. Yang, R. Men, Lin, S. Bai, Z. Li, J. Ma, C. Zhou, Zhou, and H. Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sequence learning framework. In International Conference Machine Learning, pages 23318-233 1-34430. PMLR, 2022.
12

---

[63] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring ex- pressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
13

---

Appendix
6.1 Environment Descriptions
We test framework four different real-world environments. Further details about each of the environments are listed below:
Theater:
an indoor environment simulating stage-like environment during set construc-
tion.
Lobby: an indoor environment, mostly occupied with chairs and tables, posing more chal- lenges in distinguishing specific objects and planning
Outdoor: larger environment with objects such as trees, cars, bikes, and sporting areas.
Courtyard: outdoor environment tested at night to challenge the framework with low- light images.
Along with the general objects present in the scene, we placed some additional items like back- packs, shoes, and cones into the scene. All the prompts tested in these 4 different environments are presented in Tables 5, 6, 7, and 8. For each of the prompts, we also highlighted whether each step succeeded or not.
6.2 OFA Comparisions
The full prompt for the OFA [62]+LLM (GPT-3.5-0301) experiment is shown below:
You are helpful assistant that turns sentences into referring expressions such as those found in the RefCoco family of datasets. Turn the query into referring expression. If the sentence does not explicitly have target object infer from the sentence's s context Only return single object. Examples of this include: find something to clean with on the left + broom on the left or Find me somewhere to put my cup of coffee + coffee table. If a referring expression is not required extract the object, e.g., Drive to the chair + "chair". Assume all queries refer images that contain a cemi-panoramic first-person view from an agent. Give ONLY the referring expression as the output. Query: INSERT QUERY HERE
The resulting referring expressions from this prompt can be seen in Tables 5, 6, 8, 4. We note that in some instances the L.LM failed to return referring expression. Instead, we provide the result in the OFA +LLM expression column.
6.3 Full Results
All the prompts used in the classroom environment for testing the two different input representations A & B are presented in Table 4.
Table 4: Scene-05; Classroom
Category
Sentence
A
OFA
OFA +
LLM Ex-
OFA+
pression
LLM
Generie
Go to the backpack
Pass
Pass
Fail
Backpack
Fail
Generie
Move towards the backpack
Pass
Pass
Fail
Backpack
Fail
Generie
Drive the backpack
Pass
Pass
Fail
Backpack
Fail
Generie
Run towards the backpack
Pass
Pass
Fail
Backpack
Fail
Generic
Go to the cone
Pass
Pass
Fail
Cone
Fail
Generic
Go to the conical traffic delineator
Pass
Pass
Fail
Conical traffic delin-
Pass
eator
Generic
Go to the trash can
Pass
Pass
Fail
Trash can
Pass
14

---

Generie
Go the whiteboard
Pass
Pass
Pass
Whiteboard
Pass
Generie
Proceed to the broom
Pass
Pass
Pass
Broom
Pass
Generic
Trek towards the wagon
Pass
Pass
Pass
Wagon
Pass
Generic
Find paper towels
Pass
Pass
Fail
Paper towels
Fail
Generic
Go to the outlet
Pass
Pass
Fail
Power outlet
Pass
Specific
Go to the red backpack
Pass
Pass
Fail
Red backpack
Fail
Specific
Go the black backpack
Pass
Pass
Pass
Black backpack
Pass
Specific
Navigate to the backpack on the left
Pass
Pass
Fail
Backpack on the left
Fail
Specific
Drive to the backpack on the right
Pass
Pass
Fail
Backpack on the Fail
right
Specific
Go to the whiteboard in front of you
Pass
Pass
Pass
Whiteboard in front Pass of you
Specific
Move to the whiteboard on your right
Pass
Pass
Fail
Whiteboard on your Fail right
Specific
Move the whiteboard on the right
Pass
Fail
Fail
Whiteboard the Fail
right
Specific
Go the backpack on your right
Pass
Pass
Fail
Backpack on your Pass right
Specific
Walk to the backpack on the left
Pass
Fail
Fail
Backpack on the left
Pass
Specific
Go to the the right
Pass
Fail
Pass
TV on the right
Pass
Specific
Go to the orange cone on your right
Fail
Pass
Fail
Orange cone on your right
Pass
Specific
Go to middle outlet
Pass
Fail
Fail
Middle outlet
Fail
Relational
Go to the backpack that is to the right of the red backpack
Pass
Pass
Fail
Backpack to the right of the red backpack
Fail
Relational
Drive to the backpack that is to the left of the black backpack
Pass
Fail
Fail
Backpack to the left of the black back- pack
Fail
Relational
Walk to the bag that next to the Fail black bag
Fail
Fail
The bag next to Fail
under Pass black bag
Relational
Move towards the backpack under the whiteboard Walk to the backpack on the chair
Pass
Pass
Fail
Backpack
whitehoard
Relational
Pass
Pass
Fail
Backpack
on the Fail
chair
Relational
Go to the chair with the backpack
Pass
Fail
Fail
Chair with the back- Fail pack
Relational
Walk to the backpack top of the
Pass
Fail
Fail
Backpack top of
Fail
chair
the chair
Relational
Run the rightmost backpack
Pass
Pass
Fail
Rightmost backpack
Fail
Relational
Walk to the leftmost backpack
Pass
Pass
Fail
Lefimost backpack
Fail
Relational
Go to the middle chair
Pass
Fail
Fail
Middle chair
Fail
Relational
Go the leftmost backpack on
Fail
Pass
Pass
Leftmost backpack
Pass
your right
your right
Relational
Go to the middle chair in the row of chairs
Pass
Pass
Fail
Middle chair row of Fail
chairs
Relational
Go to the backpack on the left of the Pass cone
Fail
Fail
Backpack on the left of the cone
Fail
Relational
Go to the cone to the left of the Pass backpack
Pass
Fail
Cone to the left of the backpack
Fail
Relational
Go to the second chair from the left
Pass
Fail
Fail
Second chair from Fail the left
Contextual
Go somewhere can sit down
Pass
Pass
Fail
Chair
Pass
Contextual
Find place for me to rest
Pass
Fail
Fail
Couch
Fail
Contextual
Go to somewhere can speak from
Pass
Fail
Fail
Podium
Pass
Contextual
Find place to store cleaning sup- plies
Pass
Fail
Fail
Storage area
Fail
Contextual
Find me something to write on
Pass
Pass
Fail
Piece of paper
Pass
15

---

Contextual
My friend has question. Go somewhere you can explain the an- swer him
Fail
Fail
Fail
My friend
Fail
Contextual
I spilled lot of sand. Go find me something to pick up my mess Walk to something where I can put my laptop in
Pass
Pass
Fail
Something to pick up my mess - dustpan
Fail
Contextual
Fail
Fail
Fail
Table to put laptop in
Fail
Contextual
spilled water. Find me something to clean this up
Pass
Fail
Fail
Towel
Fail
Contextual
Go to somewhere I can google something
Pass
Pass
Fail
Computer on the left
Fail
Contextual
Go to somewhere I can charge my
Pass
Pass
Fail
Charging outlet
Pass
phone
16

---

Full results for the real-world experiments are shown in Tables 5, 6, 7, 8.
Table 5: Scene-01: Theater
Category
Scene
Code
OD
WP
Path&
OFA OFA Expression
OFA+
Exec
LLM
Generic
Drive the helmet
Pass
Pass
Pass
Pass
Fail
Helmet
Fail
Generic
Navigate the mop
Pass
Pass
Pass
Pass
Fail
Mop
Fail
Generic
Go to the vacuum
Pass
Fail
Fail
Fail
Fail
Vaccum
Fail
Generic
Walk to the table
Pass
Pass
Pass
Pass
Fail
Table
Pass
Generic
Go to fire extinguisher
Pass
Pass
Pass
Pass
Fail
Fire extinguisher
Fail
Generic
Walk to the saw
Pass
Fail
Fail
Fail
Fail
Saw
Fail
Specific
Go to the red backpack
Pass
Pass
Pass
Pass
Fail
Red backpack
Fail
Specific
Move to orange bucket
Pass
Pass
Pass
Pass
Fail
Orange bucket
Fail
Specific
Drive to the chair on the right
Pass
Pass
Pass
Fail
Fail
Chair on the right
Fail
Specific
Walk to the white fan black
Pass
Pass
Pass
Pass
Fail
White fan
Fail
Specific
Navigate
Pass
Pass
Pass
Pass
Fail
Black
Fail
Specific
Navigate to the black speaker
Pass
Pass
Pass
Pass
Fail
Black speaker
Pass
Specific
Run to the red bag
Pass
Pass
Fail
Fail
Fail
Red bag
Fail
Specific
Move to the black chair infront
Pass
Pass
Pass
Pass
Fail
Black chair in-
Fail
you
front of you
Specific
Walk the blue chair
Pass
Pass
Pass
Pass
Fail
Blue chair
Fail
Relational
Go to chair with the black back- pack on it
Pass
Pass
Pass
Pass
Pass
Chair with black
Pass
backpack
Relational
Go to the chair with the helmet on it
Pass
Pass
Pass
Pass
Fail
Helmet Chair
Fail
Relational
Go to the backpack next to the chair
Pass
Fail
Fail
Fail
Fail
Backpack next to the chair
Fail
Relational
Go the backpack infront of the ladder
Pass
Pass
Pass
Pass
Fail
Backpack infront of ladder
Fail
Relational
Run to the person on the ladder
Pass
Pass
Pass
Pass
Fail
Person on the Fail
ladder
Relational
Drive the sitting the table
Pass
Pass
Pass
Pass
Fail
Man sitting table
Fail
Relational
Go to the third chair from the left
Pass
Pass
Pass
Pass
Fail
Third chair from
Fail
left
Contextual
Go to something that can carry wa-
Pass
Pass
Pass
Fail
Water container
Fail
ter
Contextual
Navigate to something that can
Fail
Fail
Fail
Fail
Fail
Mop
Fail
wash the floor
Contextual
My floor is dirty. Go something that can fix this
Fail
Fail
Fail
Fail
Fail
Mop
Fail
Contextual
Go somewhere that I can throw something away in
Pass
Pass
Pass
Pass
Fail
Trash
Pass
Contextual
Walk to something that put out fire
Pass
Fail
Fail
Fail
Fail
Fire extinguisher
Fail
Contextual
Go to something that can coal me down
Pass
Fail
Fail
Fail
Fail
Fan
Fail
Contextual
Go to something white that will cool me down
Fail
Fail
Fail
Fail
Fail
Fan
Fail
Contextual
I am the mood to listen to music.
Pass
Fail
Fail
Fail
Fail
Speaker with music player
Fail
Go something that that
17

---

Table 6: Scene-02: Lobby
Category
Scene
Code
OD
WP
Path&
OFA
OFA +XL
OFA+
Exec
pression
LLM
Generic
Run to the trash bag
Pass
Pass
Pass
Pass
Pass
Trash bag
Pass
Generic
Navigate to water fountain
Pass
Fail
Fail
Fail
Fail
Water fountain
Fail
Generic
Drive to the broom
Pass
Pass
Pass
Pass
Fail
Broom
Fail
Generic
Walk to the monitor
Pass
Pass
Fail
Fail
Fail
Monitor
Fail
Specific
Walk to the tiny monitor
Fail
Fail
Fail
Fail
Fail
Tiny monitor
Specific
Walk to the smallest monitor
Pass
Pass
Pass
Pass
Fail
Smallest monitor
Fail
Relational
Walk the right
Pass
Pass
Pass
Pass
Fail
Rightmost
Fail
Relational
to the table with can on
Pass
Fail
Fail
Fail
Fail
Can on the table
Pass
Relational
Walk to the table next to the red
Pass
Pass
Pass
Pass
Fail
Table next to red
Pass
backpack
backpack
Relational
Go to the blue chair with back- pack on it
Pass
Pass
Pass
Pass
Pass
Blue chair backpack
Pass
Relational
Walk to the leftmost table
Pass
Fail
Fail
Fail
Fail
Leftmost table
Fail
Relational
Go to backpack closest to the shoes
Pass
Pass
Pass
Pass
Backpack closest
Pass
to shoes
Relational
Walk to the shoe next the red
Pass
Pass
Pass
Pass
Fail
Shoes next to red backpack
Fail
backpack
Relational
Remove trash bag from the floor
Fail
Fail
Fail
Fail
Fail
Trash hag on floor
Pass
Relational
Walk the table with the monitor it
Fail
Fail
Fail
Fail
Pass
Table with moni-
Pass
Relational
Drive to the closest monitor to the table
Pass
Pass
Pass
Pass
Fail
Closest monitor
Fail
to the table
Relational
Go the table with the bottle it
Fail
Fail
Fail
Fail
Fail
Table with the bottle
Fail
Relational
Go to the bottle on top of the table
Pass
Pass
Pass
Pass
Fail
Bottle on top of the table
Fail
Relational
Go the tv closest the person
Pass
Pass
Pass
Fail
TV closet per- son
Fail
Relational
Go to the person with the blue shirt
Pass
Fail
Fail
Fail
Fail
Person with the Fail
blue shirt
Relational
Run to the chair with blue coat
Pass
Fail
Fail
Fail
Fail
to chair
on it
with the blue coat
Relational
Move the backpack the chair
Fail
Fail
Fail
Fail
Fail
Backpack chair
Fail
Relational
Move to the black backpack on the chair
Fail
Fail
Fail
Fail
Fail
Black backpack
Fail
the chair
Relational
Drive the chair with backpack on it that is not red
Fail
Fail
Fail
Fail
Fail
Backpack on the chair
Fail
Contextual
Find something that can help fire- fighter
Pass
Pass
Pass
Pass
Fail
Fire extinguisher
Fail
Contextual
Go to something that can clean a dirty floor
Fail
Fail
Fail
Fail
Fail
Mop
Fail
Contextual
Go get drink of water
Fail
Fail
Fail
Fail
Fail
Water bottle
Fail
Contextual
to place where I watch a movie
Pass
Pass
Pass
Fail
Movie theater
Fail
Contextual
Drive to place where can watch video
Fail
Fail
Fail
Fail
Fail
TV screen
Fail
18

---

Table 7: Scene-03: Outdoor
Category
Scene
Code
OD
WP
Path&
OFA +
OF
Exec
pression
LLM
Generic
Wander to the fire hydrant
Pass
Pass
Pass
Pass
Fail
Fire hydrant
Fail
Generic
Step towards the gril
Pass
Fail
Fail
Fail
Pass
Grill
Pass
Generic
Walk to the skateboard
Pass
Pass
Pass
Pass
Fail
Skateboard
Fail
Generic
Walk to the bike
Pass
Pass
Fail
Fail
Fail
Bike
Fail
Generic
Walk the bike
Pass
Pass
Pass
Pass
Fail
Bike
Fail
Generic
Go to the bike rack
Pass
Pass
Pass
Pass
Fail
Bike rack
Fail
Generic
Go to the sign
Pass
Pass
Fail
Fail
Fail
The sign
Fail
Generic
Go to bench
Pass
Pass
Pass
Pass
Fail
Bench
Fail
Specific
Sashay to the stop sign
Pass
Pass
Pass
Pass
Fail
Stop sign
Fail
Specific
Go to the baskethall hoop
Pass
Pass
Pass
Pass
Fail
Basketball hoop
Pass
Specific
blue car Trot towards the red hag
Roam towards
Pass
Pass
Pass
Pass
Fail
Blue car
Specific
Pass
Pass
Pass
Pass
Fail
Red bag
Fail
Specific
Go to the red object
Fail
Fail
Fail
Fail
Fail
Red object
Fail
Relational
Proceed the middle cone
Pass
Pass
Pass
Pass
Pass
Middle cone
Pass
Relational
Journey to the tree next to the back- pack
Fail
Fail
Fail
Fail
Fail
Tree next to the
Pass
backpack
Relational
Trek the backpack by the tree
Pass
Pass
Pass
Fail
Backpack by the Fail
tree
Contextual
A firefighter needs water. Walk source of water
Pass
Fail
Fail
Fail
Fail
Source of water
Fail
Contextual
Head towards something that can help firefighters You dog that needs mark its territory. Go find place do this
Pass
Pass
Pass
Pass
Fail
Fire hydrant
Pass
Contextual
Pass
Pass
Pass
Pass
Fail
Fire hydrant
Fail
Contextual
You are carrying trash, Find some- where to dump it
Pass
Pass
Pass
Pass
Fail
Dumpster
Pass
Contextual
Find something to do kickflip
Pass
Pass
Pass
Pass
Fail
Skateboard
Fail
on
Contextual
want to shoot some hoops. Take
Fail
Fail
Fail
Fail
Fail
Basketball court
Fail
me there
Contextual
Move towards faster mode
Pass
Pass
Pass
Fail
Sorry can't it gen- erate a referring
Fail
transportation
expression
Contextual
Head the fastest mode of trans- portation
Pass
Pass
Pass
Pass
Pass
Fastest mode of transportation
Pass
19

---

Table 8: Scene-04; Courtyard
Category
Scene
Code
OD
WP
Path&
OFA OFA +LM
OF
Exec
pression
LLM
Generic
Run to the door
Pass
Pass
Pass
Pass
Fail
Door
Fail
Generic
Run towards the backpack
Pass
Pass
Pass
Pass
Fail
Backpack
Fail
Generic
Drive the wagon
Pass
Pass
Pass
Fail
Wagon
Pass
Generic
Navigate to the stairs
Pass
Pass
Pass
Pass
Pass
Stairs
Pass
Specific
Proceed towards the garbage
Pass
Pass
Pass
Pass
Fail
Garbage
Fail
the right
the right
Specific
Stroll to the recycle bin on the left
Pass
Pass
Fail
Fail
Fail
Recycle bin on the left
Fail
Specific
Sprint the picnic table
Pass
Pass
Pass
Pass
Fail
Picnic table
Fail
Relational
Go to the bench with water con-
Pass
Pass
Pass
Pass
Pass
Bench with water container
Pass
tainer on it
Relational
Walk the bench with nothing
Pass
Pass
Pass
Pass
Fail
Bench with noth- ing on
Pass
Relational
Praceed to the hench with most ob-
Fail
Fail
Fail
Fail
Fail
Bench with the most objects
Pass
jects
Relational
Move towards the backpack farthest from bench
Pass
Fail
Fail
Fail
Fail
Backpack far- thest away from bench
Fail
Relational
Head towards the middle cone
Fail
Fail
Fail
Fail
Fail
Middle cone in Fail row of cones
Relational
Head towards the middle cone the row of concs
Pass
Pass
Pass
Pass
Fail
Middle cone in Fail row of cones
Relational
Go the table with only chair
Fail
Fail
Fail
Fail
Pass
Table with only
Pass
one chair
Relational
Go to the table with only one chair.
Pass
Pass
Pass
Pass
Pass
Table with only
Pass
There multiple groups of chairs around multiple tables
one chair
Relational
Step towards the column closest to Pass the cart
Pass
Pass
Pass
Fail
Column closest
Pass
to the cart
Relational
Move the largest group of Pass benches
Pass
Pass
Pass
Fail
Largest group of Fail
benches
Relational
Walk towards the table with the um- brella
Pass
Pass
Pass
Pass
Fail
Umbrella on the table
Pass
Relational
Walk towards the table with the yel- umbrella
Pass
Pass
Pass
Pass
Fail
Table with yel- low umbrella
Fail
Relational
Drive to table without any chairs
Pass
Pass
Pass
Pass
Fail
Table chairs
without
Fail
Relational
Walk black table with six chairs
Fail
Fail
Fail
Fail
Fail
Black table
Fail
six chairs
Relational
Walk to the table with the most chairs
Pass
Pass
Pass
Pass
Fail
Table with the Fail most chairs Backpack on Fail table
Relational
Navigate to the table back- pack
Fail
Fail
Fail
Fail
Fail
Contextual
Go the nearest entrance the Fail building
Fail
Fail
Fail
Fail
Nearest entrance
Fail
the building
Contextual
Go to something that you would hold open for someone elderly
Pass
Fail
Fail
Fail
Fail
Door
Fail
Contextual
Go something that will make it easier to earry heavy luggage
Pass
Fail
Fail
Fail
Pass
Cart
Pass
Contextual
Go to somewhere can cat my lunch
Pass
Fail
Fail
Fail
Fail
Lunch table
Pass
Contextual
Go up the second floor
Fail
Fail
Fail
Fail
Fail
Second floor
Fail
Contextual
Go find something to climb
Fail
Fail
Fail
Fail
Fail
Rock wall
Fail
Contextual
Run upstairs
Pass
Pass
Pass
Pass
Fail
Stairs
Fail
Contextual
Find me somewhere park my bike
Pass
Pass
Pass
Fail
Bike
Pass
20

---

6.4 Code Prompt
Navigation Prompt for the concatenated single image input. We leverage the original prompts pre- sented in [14].
import math
class ImagePatch:
"""A Python class containing crop of an image centered around particular object, as well as relevant information. Attributes
cropped_image : array_
An array-like of the cropped image taken from the original image. left, lower, right, upper int An int describing the position of the (Left//oover/rritt//pppsss border the crop's bounding box in the original image.
frame: name of camera frame
Methods
find (object__ :ame: str) ->List [ImagePatch]
Returns a list of new ImagePatch objects containing crops of the image centered around any objects found in the image matching the object_name.
exists exist
str)->bool
Returns True if the object specified by object_nand name is found in the
image, and False otherwise.
verify-property. (property: str)->bool Returns True if the property met, and False otherwise. teetttextttttee (option_list: List [str], prefix: str)->str
Returns the string that best matches the image. camaple_queryyeeee (question: str=Nooee
Returns the answer
basic question asked about the image. If no question is provided, returns the answer to "What is this?".
11m_ allm_query (question: str, long_answer. bool)->str
References a large language model (e.g., GPT) to produce response to the given question. Default is short-form answers, can made long-form responses with the long_answer flag.
(empute_depth()>222
Returns the median depth of the image crop.
crop (left: int, lower: int, right: int, upper: int) int) ->IagePatch Returns new ImagePatch object containing crop of the image at the given coordinates.
def __init____.. __init__ (self, image, left: int None, lower: int None, right: int None, upper: int = None, frame None):
"""Initiaizess an ImagePatch object by cropping the image the given coordinates and stores the coordinates as attributes. no coordinates are provided, the image left unmodified, and the coordinates are set to the dimensions of the image. Parameters
image array_like
array-like of the original image. left, lower, right, upper int
21

---

An int describing the position of the (lott//orer/riitt//ppe)) border of the crop's s bounding box the original image.
left is None and right None upper is None and lower is None:
self cwtp.ceopped_iieee image
self left 0
self.lower
self image. shape [2] # width
self upper image. shape [1] # height
else:
self. cropped. image imagel:, lower:upper, left:right] self left left
self upper upper
self.right right
self.lower = lower
self = self proppe__mmage.oo [2]
self. height self wwwppedoopppeeeecc image.._mme___ shape [1]
self www.hhooooottcccneaaa self www.eeeettlleelee center = (self lower self saif.) / 2
(self left self right) /
self frame = frame
def fsel,, object_names str) -> List [ImagePatch]:
"""Returns a list of ImagePatch objects matching object_name contained in the crop if any are found. Otherwise, returns an empty list. Parameters
object_name str the name of the object to be found
Returns
List [ImagePatch]
list of ImagePatch objects matching object_name contained. in the crop
Examples
>>> # return the foo
>>> def
(image) List [ImagePatch]]
>>> image_patch
(aageaatchhimaee
>>> foo_patcc too_patches image Seaag_ppacch..m findddndnnn ("foo")
>>> return foo foo_patches
return find in iin_immaaesssesl propped_imaaee [oject_name)
def exists object_name name: str) -> bool:
"""Returns True if the object specified by object_nam is found in the image, and False otherwise.
Parameters
object_nand sameettammm_aaaee str string describing the of the object to be found in the image.
22

---

Examples
>>> # Are there both foos and garply bars in the photo?
>>> def concuute_commnddcce (imag)
>>> image_patch ImagePatch (image)
>>> is_foo
exists ("foo")
>>> is_garply_ barr = image_patch. exists e'garppy bar")
>>> return bool_to._ pool_to_yesno to_yesnoois__foo (is_foo and is is_garply_bar)
return len(self. (iwddfjjecdoooonnnneee
0
def prapyy_yproperryseeff, object_namee name:: str,
str) -> """Returns True if the object possesses the visual property, and False othervise.
bool:
Differs from 'exists' in that it presupposes the existence of the object specified by object_name, instead checking whether the object possesses the property. Parameters
object_name str A string describing the name of the object to be found in the image.
visual_property str
string describing the simple visual property (e.g., color, shape, material) to be checked.
Examples
>>> # Do the letters have blue color?
>>> def execute__ommant (image) -> str:
>>> image_patch ImagePatch (image)
>>> letters_patcher
swage_patch... fttdappaaaaaaaap...//.pppa f"netteerrss"")
>>> # Question assumes only letter patch
>>> return bool_to_yesno (letters_patcher [0] veriffy_roperty ("letters "blue"))
return
object_nane property)
def dest_text_match(sse option_ list: List[str]] str:
""""uuunns the string that best matches the image.
Parameters
option_ _iitt str A list with the names of the different options prefix str A string with the prefixes to append to the options
Examples
>>> # Is the foo gold white?
>>> def lnecute_commatt (image)->str:
>>> image_patch
ImagePatch (image)
>>> foo Roo_patches image saagg__ptcch.. find (ifoo""
>>> Question assumes one foo patch
>>> return foo foo_patches
(""goo""" "white"])
return best_text_ntt peaett_macchssel.. mtrppaaaaaaa.....oooo.eeoeeeeed
option list)
23

---

def simple-query (self, question: str = None) -> str:
"" """Reuuns the answer question is provided, returns the answer "What this?". The questions are about basic perception, and are not meant to be used for complex reasoning or external knowledge. Parameters
basic question asked about the image. If
question str string describing the question to be asked.
Examples
>>> # Which kind of baz is not fredding?
>>> def execute_commant (image) -> str:
>>> image_patee
aaagePachhiimagee
>>> baz_patches Amage_patch.fiine ("iaaazz ("baz")
>>> for baz_patch in baz.patches:
>>> if not baz_patch, verify-properte ("baz", "fredding")
return www.patch.simple.... query ("What is this baz?")
>>> What color is the foo?
>>> def execute_ @nncute_commandde (image) -> str:
>>> image_pater
>>> foo foo_patches ImagePatch (image)
amage_patch. ftt.:aaaacchnn...oorea ("noo"
>>> foo_patch foo foo_patches [0]
>>> return foo foo_patch.sine query ("What is the color?")
>>> # Is the second bar from the left quuxy?
>>> def execute_commant (image) -> str:
>>> image_patch ImagePatch (image)
>>> bar_patches
find("bar")
>>> bar_patches.. sort (key=lambda x. s.horizontal_center
>>> bar_patch bar_patched [1]
>>> return bar.patchh.simlee_quueee ("Is ("Is the bar quuxy?")
return simple_query cropped_ image, question)
def conpute__eettiseel1
"Reuurns the median depth of the image crop
Parameters
Returns
float
the median depth of the image crop
Examples
>>> # the bar furthest away
>>> def execute_ execute_command
>>> image_patch ImagePatch (image)
>>> bar_patches
image_patte find ("bar")
>>> bar_patches. sort (key=lambda bar: ceacccc_mee_epppepe ())
>>> return bar bar_patches| [-1]
depth_map = deattee_e cwwppeddiimeeececcc -imagemm))))
return septh_map (

---

def crop(self, left: int, lower: int, right: int, upper: int) ImagePatch:
n" """tuurns a new ImagePatch cropped from current ImagePatch. Parameters
left, , lower, right, upper int The (left/loner//iihh//ppeloe most pixel of the cropped image.
return ImagePatch (self. propped_imagee left, lower, right, upper, self. frame)
def wrerlaas_withhself. left, lower, right, upper): """Returns True if crop with the given coordinates overlaps with this one,
else False. Parameters
left, lower, right, upper int the (Left/lover/riiht/uuuol border of the crop checked
Returns
bool
True if crop with the given coordinates overlaps with this one
else False
Examples
>>> # black foo on top of the qux
>>> def execute execute_command (image) -> ImagePatch:
>>> image_patte sate_pacchha ImagePatch (image)
>>> qux_patches image saage_patch. ffi.ddiidd""""
>>> qux_patch qux_patches [0]
>>> foo_patches
srrag_ppachh.. wttpppaaaccccniia..aaa find foo")
>>> for foo in foo_patches:
>>> if foo. www.eeeerllcceteeeet > qux_patch.verticatl www.wveereiiiteteeecee
>>> return foo
return self left <= right and self right >= left and self.lower lower <= upper and upper lower
def best_image gest_image__atchh1isse (listtpttittttthhehes List [ImagePatch], content: List[str], return_ ) -> Union [ImagePatch, int]:
"""Returns the patch most likely to contain the content. Parameters
list_patches List [ImagePatch] content : List [str]
the object of interest return_indes bool
if True, returns the index of the patch most likely to contain the object
Returns
int
Patch most likely contain the object
25

---

return wtw.:unngg.matcchiits...ppppee content, return_index)
def distance (patchaa: ImagePatch, patch_b. b: ImagePatch) -> float:
unn
Returns the distance between the edges of two ImagePatches. If the patches overlap, it returna a negative distance
corresponding to the negative intersection union.
Parameters
patch_a ImagePatch patch_b : ImagePatch
Examples
Return the qux that is closest to the foo
>>> def execute_commant (image):
>>> image_patcl ImagePatch (image)
>>> qux_patches image_patch. find find('qux')
>>> foo_patches image_patch, feaathhfinn (iidd''oo'
>>> foo_patch foo_ foo_patches [0]
>>> qux_patches. sort (key=lambda x: distance foo foo_patch)
>>> return qux_patches [0]
return distance (patch_a, patch_b)
def (boolaaboooamseer bool) -> str: return "yes" " if bool_answer else "no"
def coerce_to numeric
unn
This function takes string as input and returns float after removing any non-numeric characters.
If the input string contains range (e.g. "10-15"), it returns the first value in the range. uun
return coerce_to_meune sonrre_tonumercc(strinng
### Nav Client ann
Navigates agent to object in imgae given its center coordinates Parameters
x x coordinate of the center the object
y y coordinate of the center of the object
Examples
>>> # Go the blue foo.
>>> def execute_commaatt commanddimmaaaa
>>> image_ sathh ImagePatch (image)
>>> foo foo_patches
image_patch. ("foooo
>>> # Verify visual property
>>> blue_ color color_patcche
>>> for foo_patch foo_patches:
26

---

>>> if verify.-roportyy(lue, "color")
>>> (blue_color_pall wwwllorrpacccss__ [0] horizontal_center,
>>> inputs = (blue_ color
blue_cola Sallor_pacchesttt [0].aertical
>>> return {'function': 'nav_functe _av_ffuntioon, 'inputs': inputs, 'box':
blue_color_ www.loor_pttcheeeeec [0] left, blue_ [0] lower, blue_color_ppachher [0] right, clue_color_ppteeee upper]] }
def navigate x, double y):
Write function using Python and/or the ImagePatch class (above) that could be executed to provide an answer to the query by calling one of the functions the Nav Client Ravigate_tt__bjecce
Note using the ImagePatch is not required for all queries. If the query is not relevant navigation, return None for the function and a string describing the problem.
Consider the following guidelines:
Use base Python (comparison, sorting) for basic logical operations, left/ right/up/down, math, etc.
Use the 1lm_query function to access external information
answer
informational questions not concerning the image.
All properties such as color should be verified using the verify_property function. "go to the blue foo" implies "go to foo if foo is blue"
The output of this function should be a dict (function: 'function in the nav client', inputs: 'inputs the chosen nav client function', box: [ left, lower, right, upper]}, or for an error (function: 'None', error: 'message describing the problem'}
If more than one object is found pick the best match
Query: INSERT_QUERY HERE
27


Reviews:
[{"review_id": "yq70Zv-jct", "reviewer": "Reviewer_DATK", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a system for using LLMs for context-aware reasoning for real-world robot navigation, using code generation. The visual observations are grounded using the abstraction of objects and 2D positions, and the LLM reasons over this abstraction and a user-provided instruction. The authors show qualitative and quantitative analysis, including real-world deployment.\n", "summary_of_recommendation": "The paper presents a simple yet effective idea in using natural language context to guide robotic navigation by grounding visual observations in text. However, I cannot recommend acceptance in its current form due to missing key implementation details and incomplete evaluations. I encourage the authors to revise relevant sections, and I am open to reconsidering my recommendation after the author rebuttal/discussion phase.\n\n---\n\nUpdated my recommendation to WR after rebuttal phase discussion.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please see weaknesses section above on detailed concerns.\n- Lack of experimental rigor and baselines\n- Better discussion of assumptions and limitations\n- Missing key implementation details and reproducibility concerns", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n---\n- Grounding LLMs in robotic experience is indeed fundamental to their applications in robotics, and this paper proposes a good recipe of grounding using object detections and simple localization. While it is not clear if the proposed object-centric representation would scale to more complex settings, it is very flexible and easy to integrate with off-the-shelf LLMs.\n- The analysis of visual representations for grounding VLMs is quite interesting and may be useful for people working in related areas.\n- The discussion on related works seems very thorough and complete.\n\n*** \nWeaknesses\n---\n- The presentation of the method seems incomplete and the system overview is very hard to understand. It appears that the authors use an object centric representation, but\n\t- It is not clear how these objects are grounded in the visual inputs of the robot. Presumably, this would require an open-vocabulary detector (e.g., GLIP, DETIC etc.) but this is not discussed or described anywhere in the paper. In its current form, the paper is not reproducible.\n\t- How are the “skills”, or the “navigate_to_object” skills trained and executed? It seems that the skills are conditioned on 2D positions of the goals, and in the presence of a 3D map (as shown in Figure 3), this should be fairly straightforward using simple motion planning, but this process is not described at any point. The authors should include a clear set of assumptions and problem setup for the reader to contextualize the findings.\n\n- As an empirical paper, the experiments lack rigor, both in terms of the hypotheses tested and lack of comparisons to any standard baselines.\n\t- Section 4 studies 2 questions: (i) what the right visual input is (stitched panorama v/s frame stacking) and (ii) breakdown of success/failure in terms of perception, planning etc. and the authors report “100% success rate for code generation” and “100% success rate in following all planned paths”. While this is somewhat informative, it is very hard to understand the performance of the system without any competent baselines also included. There is a variety of prior work on this topic (many of which are cited and discussed in Section 2: code-as-policies style system, LM-Nav etc.) that would form useful baselines; the authors could also include simpler baselines that do not use LLMs, methods using grounded VQA models (BLIP2 etc.) and ablations of various system components to truly understand the design space. In its current form, it is not clear whether the method is competent or the task too easy.\n\t- It is also concerning that the only experiment discussed in detail/visualized (Figure 5, 6) do not actually have any control component and are simple VQA-style queries. Given the task is not benchmarked against baselines, this further raises the concerns regarding the validity of numbers reported. For the task shown in Figure 5 and 6, assuming there are low level “go to X”  skills, it is not clear why an off-the-shelf VQA model would not solve the task either.\n\n- Reading the discussion in L221-236, it appears that the authors’ stated limitations have less to do with the proposed method, and more likely an effect of incomplete “prompts” to the LLM. It appears to me that the pattern “find image patch, sort and count it” appears too often in the prompt and the outputs are heavily overfit to that kind of reasoning (see L225-232, Figure 5). (This is not strictly a weakness of the method, and in fact suggests that these limitations can be very easily patched)\n\n- An implicit assumption (which should be stated more prominently) in the paper seems to be that navigation skills of the type “go to X” are sufficient for solving this task. This is a very strong assumption and severely limits the applications of the method; since the interface to this skill is just a 2D position, it also limits the reasoning to reaching line-of-sight goals, and cannot do more interesting things such as “reach behind the couch” or “drive slowly to the left of the pedestrian” etc.\n\t- In L188-189, the authors say “We find that we are able to successfully generate code for navigational plans using a variety of verbs e.g., walk, go, drive, run, etc” but this seems FALSE. How are verbs handled by the low-level policy if all it can do is “go to position X”? A clarification on these aspects would be very useful.\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}, {"review_id": "hRWDIQemeaA", "reviewer": "Reviewer_Gcpd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a composable framework for robot navigation that leverages language models (LLMs), state-of-the-art object detectors, and classical robotic planning algorithms to perform zero-shot natural language based navigation in four unique environments. The framework addresses both the grounding and transparency issues of current LLMs, and requires a minimal uplink for the robot since all of the planning, mapping, and localization is performed onboard. The authors evaluate different 2D input representations to determine an effective way to extract spatial and conceptual knowledge from LLMs, and perform extensive real-world experiments in a variety of environments to show that the framework is able to navigate to landmarks based on natural language. The results of the experiments showed that the system was able to successfully navigate in a variety of environments and that the input representation was able to accurately capture the necessary information for navigation", "summary_of_recommendation": "This paper is a good embodied extension of ViperGPT: Visual Inference via Python Execution for Reasoning, while the contribution is limited to the adaptation and analysis of how the approach can work in the navigation setting. The demos are promising, but more justifications of the approach capability and how the success is due to the authors contribution rather than the pure in-context learning is to be demonstrated.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. The success for experiment is not clearly stated, the authors should be explicit about how they evaluate their approaches, and if it is based on human questionaire, clearly state how there is a clear rule to obtain reproducible results.\n\n2. What are the success metrics for sub modules like Code Generation, OD, WP, P&E? They shall be clearly stated in the paper.\n\n3. How did the authors get the map (m), how does the authors determine if the robot is able to navigate inside the terrain (like area with stairs), what is the data format of the map and how it is fed into the model?\n\n4. The paper also lacks key comparison with classic method such as a hierarchical object inference + object navigation policy.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed framework is a novel approach that bridges the gap between large language models (LLMs) and robot navigation by providing an intermediate layer in the form of Python code. This code allows robots to understand natural language descriptions of the environment and associate them with physical objects. The results of the experiments conducted in four different environments and command classes on a mobile robot showed that the framework had an overall success rate of 90% accuracy across 50 different sentences in controlled experiments and 70% execution accuracy in real-world environments, which is significantly higher than the baselines. The proposed framework was evaluated across four different environments and command classes on a mobile robot, and the results showed that the framework had an overall success rate of 78.07% for code generation, 63.16% for object detection, 58.77% for waypoint navigation, and 57.89% for path and execution. The proposed method outperformed the baselines in terms of accuracy and execution, achieving 90% accuracy across 50 different sentences in controlled experiments and 70% execution accuracy in real-world environments. However, there were some failures due to the wrong object being detected or the object not being detected at all, as well as the vision system detecting further out than the map and projections missing the correct voxel on the map.\n\nHowever, the paper also has some limitations and weaknesses. The object detector has some failures due to incorrect object detection or objects not being detected at all. This is evidenced by the paper, which states that the object detector (GLIP) was successful in detecting objects, however, there were some failures due to the wrong object being detected or the object not being detected at all. Additionally, the paper also mentions that the waypoint projection step had some failures due to the vision system detecting further out than the map and projections missing the correct voxel on the map. These failures made the approach hard to deploy in the real world. Besides, the paper lacks many descriptions of how the authors constructs the framework and evaluated the results. For example, there are many \"success rates\" invarious tables, but the authors didn't give how they were defined, weakening the arguments of the paper.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "eyILK20Cb0h", "reviewer": "Reviewer_9PgY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a navigation framework that leverages background and common-sense knowledge. Human operators frequently express navigational goals in ways that require prior knowledge about the world. For example, “Get me something to hold the poster up”, would require the robot to recognize that “tape” or “tacks” would accomplish the task. Large Language Models (LLMs) have been successful at this type of reasoning, but they lack the grounding needed to interact with the physical world. The proposed framework addresses this challenge by using advances in code generation to define an intermediate layer between the LLM and grounding/action primitives. Specifically, a Python API is proposed that can perform operations such as object recognition and navigation commands. The LLM generates code using this API such that the resulting program, when executed, will complete the task. The framework is evaluated in four diverse domains using a variety of natural language expressions.", "summary_of_recommendation": "I weakly recommend rejecting this paper in the current state. Although the authors show success on a real system, it is difficult to tell what successes come from the proposed framework. However, I believe this issue can be addressed and look forward to discussion during the rebuttal period.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- Inclusion of a baseline as a way to calibrate performance results.\n- Additional details about experimental setup and results (see above review).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an expressive framework for executing natural language commands in navigation domains. Code is a promising intermediate representation for robotic tasks as it allows engineers to incorporate domain knowledge and state of the art modules. Although the idea has shown to be useful in computer vision domains (ViperGPT, [14]), it is a useful contribution to apply the method in robotic domains where actions also need to be executed on the real robot.\n\nA key strength of this paper is the number of real world navigation experiments across a diverse set of domains. The results show that the generated Python code can successfully be executed to complete the task. The authors also do a good job analyzing which parts of the system lead to failures and identifying areas for improvement.\n\nOne weakness of this work is that there are no non-LLM baselines as comparisons. As such, for each of the command categories, it is difficult to interpret how good the reported metrics are. As stated, LLMs have the capacity to perform sophisticated reasoning tasks, but it is unclear how much this sophistication is needed for the tasks in this paper. They likely offer large performance gains for “contextual commands” (65.22% code generation success) and less for “generic commands” (100% code generation success). I do not think it’s necessary to have extensive baseline evaluations. However, it would strengthen the paper if we could tell how much the proposed framework outperforms a model that doesn’t use prior knowledge from LLMs, or a less expressive intermediate layer than Python code.\n\nI also have a few concerns/questions regarding clarity which I believe can be addressed in the rebuttal period:\n- In Tables 2 and 3, are the success rates over all examples or only the ones where the previous stage was successful? For example, does OD (%) only include trials where the code generation was successful? \n- On line 214, there is a claim that the proposed framework can find better solutions than humans would. However, the only evidence to support this is a single example. This is indeed an interesting observation but needs further statistical analysis to support the claim. Without that, I would recommend a weaker claim that this observation should be investigated in future work.\n- How were the sentences generated for the experiments? (Appendix Tables 5-8)\n- Why do the contextual results in Table 1 appear better than the results in Table 2? Can this mainly be attributed to the small sample size?\n\n----- Post-Rebuttal Update -----\n\nI would like to thank the authors for taking the time to respond to my questions and concerns. The additional details are helpful in understanding the methods and results.\n\nI also appreciate the addition of the OFA baseline but have additional concerns about the included results. In particular, the baseline does not appear to be a competitive baseline and has poor performance even for generic sentences which do not involve any complicated reasoning. In a response to other reviewers, the authors mention this is largely due to predicting the wrong bounding box. As such, it is a useful ablation of the Code + OD  components. But it is difficult to tease out if the baseline is failing primarily due to vision or from lacking prior knowledge included in the LLM. \n\nI have updated my score to a weak accept. This work is promising and code generation is a potentially powerful approach to enable execution of complex language commands.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "_-ZmdE2vpu", "reviewer": "Reviewer_cxP7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "It's a code as policies type paper for navigation. Given an instruction, robot writes a plan on where to go, and then this is executed. Inputs are front, left and right images. Four types of instructions are tested: generic, specific, relational and contextual. ", "summary_of_recommendation": "This is a paper that tries an innovative approach at robot navigation using LLMs. Unfortunately it is not a learning paper but a prompting paper. But theyve designed very good engineering around it, that more or less massages the fact that LLMs can't see. Unfortunately, many details of that engineering is taken from other works but not explained clearly in the paper, so it seems a little bit like magic to a reader who hasnt read the references. Small explanations of what's going on would be useful. Results are interesting, failure modes and limitations are thoroughly explained.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Could you add some reasoning as to why you designed the system this way and not any other way. What were alternatives considered? eg why concatenation of camera views as against language description? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. Innovative approach, connects language models to embodied navigation in unseen environments\n2. Robot writes navigation code, this is also innovative\n3. Good discussion on limitations. This seems like primarily a multimodal embodied reasoning but authors have done the best to map it to language. \n\nCons:\n1. One complaint I have with this paper is that many of the components are referenced but not explained, such as the algorithm for avoiding terrains. I'm curious but don't have the time to go look at the paper, and could've used a one sentence summary. Similarly for details of intermediate layer.\n2. This is not quite a robot learning paper in that nothing is trained, but is a prompt engineering paper. That's ok, I guess. \nAblations on design choices is not included. \n3. Not many ablations for design choices\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a system that uses LLMs for context-aware commonsense reasoning for real-world robot navigation, using code generation. The visual observations are grounded using the abstraction of objects and 2D positions, and the LLM reasons over this abstraction and a user-provided instruction, to generate Python code using a given API such that the resulting program, when executed, will complete the task. The authors show qualitative and quantitative analysis in four domains, including real-world deployment.

Reviewers agree that the proposed approach is flexible (DATK), that code presents a promising intermediate representation for navigation tasks (9PgY, cxP7), and that a key strength of the presented system is that it has been tested across several real-world domains (9PgY). However, reviewers also raised important concerns, a primary one being that there are missing comparisons to competitive baselines (DATK, 9PgY, Gcpd) missing component ablations (cxP7), and missing details for clarity and reproducibility.

Post-rebuttal, the manuscript has improved substantially in clarity, and now includes a baseline comparison to OFA (Wang et al.). Reviewer ratings have been updated to: weak accept, weak accept, weak accept, weak reject. Nevertheless, reviewers point out that the new comparison is not necessarily a fair evaluation (DATK, 9PgY) since the proposed pipeline uses an LLM + GLIP, while the baseline is only end-to-end OFA (no LLM). The relatively large gap in performance between the proposed system (71%) and baseline (12%) corroborates this. I would strongly recommend strengthening the baseline to improve the evaluations for the revision.
Decision: Accept (Poster)