SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning
Authors: Krishan Rana
Jesse Haviland
Sourav Garg
Jad Abou-Chakra
Ian Reid
Niko Suenderhauf
Venue: CoRL 2023 Oral (2023)
Keywords: robot task planning
large language models
semantic search
LLM-based planning
3D scene graphs

Abstract: Large language models (LLMs) have demonstrated impressive results in developing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant challenge for robotics. We introduce SayPlan, a scalable approach to LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representations. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a "semantic search" for task-relevant subgraphs from a smaller, collapsed representation of the full graph; (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) introduce an "iterative replanning" pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and natural language instruction for a mobile manipulator robot to execute. We provide real robot video demonstrations on our project page https://sayplan.github.io.

SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning
Krishan Ranat", Jesse Haviland Sourav Garg 33 Jad Abou-Chakra"),
Ian Reid Niko Sünderhauf
QUT Centre for Robotics, Queensland University of Technology CCIIOO Data6l Robotics and Autonomous Systems Group
UUniversity of Adelaide
"Equal Contribution
1 Trankk( gut edu
Abstract:
Large language models (LLMs) have demonstrated impressive results in develop- ing generalist planning agents for diverse tasks. However, grounding these plans in expansive, multi-floor, and multi-room environments presents a significant chal- lenge for robotics. We introduce SayPlan, scalable approach LLM-based, large-scale task planning for robotics using 3D scene graph (3DSG) representa- tions. To ensure the scalability of our approach, we: (1) exploit the hierarchical nature of 3DSGs to allow LLMs to conduct a semantic search for task-relevant subgraphs from smaller, collapsed representation of the full graph: (2) reduce the planning horizon for the LLM by integrating a classical path planner and (3) in- troduce an iterative replanning pipeline that refines the initial plan using feedback from a scene graph simulator, correcting infeasible actions and avoiding planning failures. We evaluate our approach on two large-scale environments spanning up to 3 floors and 36 rooms with 140 assets and objects and show that our approach is capable of grounding large-scale, long-horizon task plans from abstract, and nat- ural language instruction for mobile manipulator robot to execute. We provide real robot video demonstrations on our project page cyyplan.gthubbii
1 Introduction
"Make me coffee and place it on my desk" - The successful execution of such scemingly straight- forward command remains a daunting task for today's 's robots. The associated challenges permeate every aspect of robotics, encompassing navigation, perception, manipulation task planning. Recent advances in Large Language Models (LLMs) [1, 2, 3] have led to significant progress in incorporating common sense knowledge for robotics [4, 5, 6]. This enables robots to plan complex strategies for a diverse range of tasks that require a substantial amount of background knowledge and semantic comprehension.
well as high-level
For LLMs to be effective planners in robotics, they must be grounded in reality, that is, they must ad- here to the constraints presented by the physical environment in which the robot operates, including the available affordances, relevant predicates, and the impact of actions on the current state. Further- more, in expansive environments, the robot must additionally understand where it is, locate items of interest, as well comprehend the topological arrangement of the environment in order to plan across the necessary regions. To address this, recent works have explored the utilization of vision-based value functions [4], object detectors [7, 8), or Planning Domain Definition Language (PDDL) de- scriptions of a scene [9, 10] to ground the output of the LLM-based planner. However, these efforts are primarily confined to small-scale environments, typically single rooms with pre-encoded infor- mation on all the existing assets and objects present. The challenge lies in scaling these models. As the environment's 's complexity and dimensions expand, and as more rooms and entities enter the
7th Conference on Robot Learning (CoRL 2023), Atlanta, USA.

---

"Wakn Peder coffen'
3D Scans Grephi
Collagss
Serrands
Ianibe Repferning
g
Intractie
Scena Graph Kcllapeed
Dalind Subprph
"Marr Peera coffer"
Feadhack
Aguac aala
celease codfee mag
Aeriosansit Panoticca Grome Eearple
Rensecacant stace
Scase Crigh
SEMANTIC
LLM
Esacutalia Plan
Ecems Sanddaee
Briph Call
(ert:: offiiee
(oumand:: supand, sode,
[amee: deak)
qustia: aegt
Vortbutionn
(ouiases Dage
BPB
Full Dearth Secsenve
[nmn mettee]
quski: sag!
rpelo: puend))
torto: officcl
Figure I: SayPlan Overview (top). SayPlan operates across two stages ensure scalability: (left) Given a collapsed 3D scene graph and a task instruction, semantic search is conducted by the LLM to identify a suitable subgraph that contains the required items solve the task; (right) The explored subgraph is then used by the LLM to generate high-level task plan, where a classical path planner completes the navigational component of the plan; finally, the plan goes through an iterative re- planning process with feedback from a scene graph simulator until an executable plan is identified. Numbers on the top-left corners represent the flow of operations.
scene, pre-enco all the necessary information within the LLM's context becomes increasingly infeasible.
To this end, we present a scalable approach to ground LLM-based task planners across environments spanning multiple rooms and floors. We achieve this by exploiting the growing body of 3D scene graph (3DSG) research [1 12, 13, 14, 15. 16]. 3DSGs capture a rich topological and hierarchically- organised semantic graph representation of an environment with the versatility to encode the nec- essary information required for task planning including object state, predicates, affordances and attributes using natural language suitable for parsing by an LLM. We can leverage JSON repre- sentation of this graph as input to a pre-trained LLM, however, to ensure the scalability of the plans to expansive scenes, we present three key innovations.
Firstly, we present a mechanism that enables the LLM to conduct semantic search for task- relevant subgraph G' by manipulating the nodes of a 'collapsed' 3DSG, which exposes only the top level of the full graph G, via expand and contract. API function calls thus making it feasible to plan over increasingly large-scale environments. In doing so, the LLM maintains focus on a rela- tively small, informative subgraph. G' during planning, without exceeding its token limit. Secondly, as the horizon of the task plans across such environments tends to grow with the complexity and range of the given task instructions, there is an increasing tendency for the LLM to hallucinate or produce infeasible action sequences [17, 18, 7]. We counter this by firstly relaxing the need for the LLM generate the navigational component of the plan, and instead leverage an existing optimal path planner such as Dijkstra [19] to connect high-level nodes generated by the LLM. Finally, to en- sure the feasibility of the proposed plan, we introduce an iterative replanning pipeline that verifies and refines the initial plan using feedback from scene graph simulator in order to correct for any unexecutable actions, e.g., missing to open the fridge before putting something into it thus avoid- ing planning failures due to inconsistencies, hallucinations, or violations of the physical constraints and predicates imposed by the environment.
2

---

Our approach SayPlan ensures feasible and grounded plan generation for mobile manipulator robot operating in large-scale environments spanning multiple floors and rooms. We evaluate our framework across a range of 90 tasks organised into four levels of difficulty. These include semantic search tasks such as ("Find me something non-vegetarian.) to interactive, long-horizon tasks with ambiguous multi-room objectives that require significant level of common-sense reasoning ("Let's 's play a prank on Niko"), These tasks are assessed in two expansive environments, including large office floor spanning 37 rooms and 150 interactable assets and objects, and three-storey house with 28 rooms and 1 112 objects. Our experiments validate SayPlan's ' ability to scale task planning to large-scale environments while conserving a low token footprint. By introducing semantic search pipeline, we can reduce full large-scale scene representations by up to 82.1% for LLM parsing and our iterative replanning pipeline allows for near- pear-perfect executability rates, suitable for execution on a real mobile manipulator robot.'
2 Related Work
Task planning in robotics aims to generate a sequence of high-level actions to achieve goal within an environment. Conventional methods employ domain-specific languages such as PDDL [20, 21, 22] and ASP [23] together with semantic parsing [24, 25], search techniques [26, 27] and complex large
solution. These methods, however, lack both the scalability environments as well as the task generality required when operating in the real world. Hierarchical and reinforcement learning- scalability. Our work leverages the in-context learning capabilities of LLMs to generate task plans across 3D scene graphs. Tasks, this case, can be naturally expressed using language, with the internet scale training of L.LMs providing the desired knowledge for task generality, while 3D scene graphs provide the grounding necessary for large-scale environment operation. This allows for a general and scalable framework when compared to traditional non-LL alternatives.
heuristics [28] arrive
baased based alternatives [29, 30], [31] face challenges with data demands and
Task planning with LLMs, that is, translating natural language prompts into task plans for robotics, is an emergent trend in the field. Earlier studies have effectively leveraged pre-trained LLMs' in- context learning abilities to generate actionable plans for embodied agents [4. 10. 9, 8, 32, 7, 33]. A key challenge for robotics is grounding these plans within the operational environment of the robot. Prior works have explored the use of object detectors [8, 71, PDDL environment representations [10, 9, 34] or value functions [4] to achieve this grounding, however, they are predominantly constrained to single-room environments, and scale poorly with the number of objects their ability to plan over multi-room or multi-floor environments. In this work, we explore the use of 3D scene graphs and the ability of LLMS to generate plans over large-scale scenes by exploiting the inherent hierarchical and semantic nature of these representations.
scene which limits
Integrating external knowledge in LLMS has been a growing line of research combining language models with external tools to improve the reliability of their outputs. In such cases, external modules are used to provide feedback or extra information to the LLM guide its output generation. This is achieved either through API calls to external tools [35, 36] or as textual feedback from the operating environment [37, 8]. More closely related to our work. CLAIRIFY [38] iteratively leverage com- piler error feedback to re-prompt an LLM to generate syntactically valid code. Building on these ideas, we propose an iterative plan verification process with feedback from scene graph-based simulator to ensure all generated plans adhere to the constraints and predicates captured by the pre- constructed scene graph. This ensures the direct executability of the plan on a mobile manipulator robot, operating in the corresponding real-world environment.
3 SayPlan
3.1 Problem Formulation
We aim to address the challenge of long-range task planning for an autonomous agent, such as mobile manipulator robot, in a large-scale environment based on natural language instructions. This requires the robot to comprehend abstract and ambiguous instructions, understand the scene and generate task plans involving both navigation and manipulation of a mobile robot within an
'sayplan.github.io
3

---

Algorithm 1: SayPlan
Given:
scene graph simulator v. classical path planner @, large language model L.M
Inputs: prompt P. scene graph G. instruction I
1: g' 4- 1012011ppp (ap
b collapse scene graph search scene graph for all relevant items
Stage Semantic Search
2: while command != "terminate" "
3: command, node name +- LLM (P.G',
4: if command = "expund" then
5: G' f- expanc,,
expand node to reveal objects and assets A contract node nothing relevant found b generate a feasible plan
6: else command
"contract" then
7: g' cont coottract.// (node__ame)
Stage 2: Causal Planning
8: feedback =""
9: while feedback != "succesy" do
plan <- (LM I.. feedback)
high level plan
11: full plan + @(plnn, 9*)
Di compute optimal navigational path between nodes forward simulate the full plan executable plan
12: feedback wwwrtyrryyyye verity_plan. (full_plan)
13:
return full_ppann
environment. Existing approaches lack the ability reason over scenes spanning multiple floors and rooms. Our focus is on integrating large-scale scenes into planning agents based on Language Models (LLMs) and solving the scalability challenge. We aim to tackle two key problems: 1) representing large-scale scenes within L.LM token limitations, and 2) mitigating L.LM hallucinations and erroneous outputs when generating ang-horizon plans in large-scale environments.
3.2 Preliminaries
Floor
Here, we describe the 3D scene graph represen- tation of environment and the scene graph simulator API which we leverage throughout our approach.
Scene Representation: 3D Scene Graphs (3DSG) [11, 12, 14] have recently emerged as an actionable world representation for robots [13, 15, 16, 39, 40, 41], which hierarchi- cally abstract the environment at multiple lev- els through spatial semanties and object rela- tionships while capturing relevant states, affor- dances and predicates of the entities present in the environment. Formally, a 3DSG is a hicrar- chical multigraph G (V, E') in which the of vertices V comprises V12 AAV/U...UVV with each Vi signifying the set of vertices at a particular level of the hierarchy Edges stemming from a vertex e E V& may only terminate in V U V2 U Ve+1, i.e. edges connect nodes within the same level, one level higher or lower.
Asset
Object
Figure 2: Hierarchical Structure of 3D Scene
Graph. This graph consists of 4 levels. Notes that the room nodes connected to one another via sequences of pose nodes which capture the topo- logical arrangement of scene.
We assume a pre-constructed 3DSG representation of large-scale environment generated using existing techniques [15, 13, 11]. The entire 3DSG can be represented object [42] and text-serialised into a ISON data format that can be parsed directly by a pre- trained LLM. An example of single asset node from the 3DSG is represented as: {name: sittee_machine,, type: asset, location: kitchen, affordances: [turn_on, turn_off, relcasel, state: off, attributes: [red, automatic), position: [2.34, 0 0.45, 2. 2.23]} with edges between nodes captured as (eitchenn+coofeeemeeeh The 3DSG is organized in a hierarchical manner with four primary levels: floors, rooms, assets, and objects as shown in Figure 2. The top level contains floors, each of which branches out to several rooms. These rooms are interconnected through pose nodes to represent the environment's topological structure. Within each room, we find assets (immovable entities) and objects (movable entities). Both asset and object nodes encode particulars including state, affordances, additional attributes such as colour weight, and 3D pose. The graph also incorporates dynamic agent
NetworkX Graph

---

node, denoting a robot's ' location within the scene. Note that this hierarchy is scalable and node levels can be adapted to capture even larger environments e.g. campuses and buildings
Scene Graph Simulator d refers to a set of API calls for manipulating and operating over JSON for- matted 3DSGs, using the following functions: 1) collapse (9): Given a full 3DSG, this function returns an updated scene graph that exposes only the highest level within the 3DSG hierarchy e.g. floor nodes. 2) expand (node name): Returns an updated 3DSG that reveals all the nodes con- nected to node anme in the level below. 3) contract (node name): Returns an updated 3DSG that hides all the nodes connected to node-name in the level below. 4) verify plan (plan): Forward simulates the generated plan at the abstract graph level captured by the 3DSG to check if each action adheres to the environment's 's predicates, states and affordances. Returns textual feedback e.g. "cannot pick up banana" if the fridge containing the banana is closed.
3.3 Approach
We present a scalable framework for grounding the generalist task planning capabilities of pre- trained LLMs in large-scale environments spanning multiple floors and rooms using 3DSG repre- sentations. Given a 3DSG G and a task instruction I defined natural language, we can view our framework SayPlan as high-level task planner π (a|z, capable of generating long-horizon plans a grounded in the environment within which a mobile manipulator robot operates. This plan is then fed to low-level visually grounded motion planner for real-world execution. To ensure the scala- bility of SayPlan, two stages are introduced: Semantic Search and Iterative Replanning which we detail below. An overview of the SayPlan pipeline is illustrated in Figure I with the corresponding pseudo-code given in Algorithm 1.
When planning over 3DSGs using LLMs we take note of two key observations: 1) A 3DSG of large-scale enviromment can grow infinitely with the number of rooms, assets and objects it contains, making it impractical to pass as input to an LLM due to token limits and 2) only a subset of the full 3DSG G is required to solve any given task e.g. we don't need to know about the toothpaste in the bathroom when making a cup of coffee. To this end, the Semantic Search stage seeks to identify this smaller, task-specific subgraph g' from the full 3DSG which only contains the entities in the enviromment required to solve the given task instruction. To identify G' from full 3DSG, we exploit the semantic hierarchy of these representations and the reasoning capabilities of LLMs. We firstly col collapse g to expose only its top level e.g. the floor nodes, reducing the 3DSG initial token representation by R 80%. The LLM manipulates this collapsed graph via expand and contract API calls in order to identify the desired subgraph for the task based on the given instruction I. This is achieved using in-context learning over a set of input-out examples (see Appendix J), and utilising chain-of-thought prompting guide the LLM in identifying which nodes to manipulate. The chosen API call and node are executed within the scene graph simulator, and the updated 3DSG is passed back to the LLM for further exploration. If an expanded node is found to contain irrelevant entities for the task, the LLM contracts it to manage token limitations and maintain task-specific subgraph (see Figure 3). To avoid expanding already-contracted nodes, we maintain a list of previously expanded nodes, passed as an additional Memory input to the LLM, facilitating Markovian search sequences without the overhead of maintaining the full interaction history [5]. The LLM autonomously proceeds to the planning phase once all necessary assets and objects are identified in the current subgraph G'. An example of the LLM--cene graph interaction during Semantic Search is provided in Appendix K.
Semantic Search:
secision-maaing process and allowing SayPlan to scale to extensive
Given the identified subgraph G' and the same task instruction I from above, the LLM enters the planning stage of the pipeline. Here the LLM is tasked with generating a sequence of node-level navigational (goto (pose2)) and manipulation (pickup (coffee_nugh) ) actions that satisfy the given task instruction. LLMs, however, are not perfect planning agents and tend to hallucinate or produce erroneous outputs [43, 9]. This is further exacerbated when planning over large-scale environments or long-horizon tasks. We facilitate the generation of task plans by the LLM via two mechanisms. First, we shorten the LLM's planning horizon by delegating pose-level path planning to an optimal path planner, such as Dijkstra. For exam- ple, a typical plan output such as [goto (meet ing__rom) goto (posei1), goto (pose14) 14), goto (pose8),
Iterative Replanning:
goto (kitchen), access (fridge), open (fridge) J is simplified to [goto (meet ng_rr goto (ki kktchen), access (f (fridge), open (fri ). The path
5

---

planner handles finding the optimal route between high-level locations, allowing the LLM to focus on essential manipulation components of the task. Secondly, we build on the self-reflection capabil- ities of LLMs [17] to iteratively correct their generated plans using textual, task-agnostic feedback from scene graph simul lrtor which evaluates if the generated plan complies with the scene graph's predicates, state, and affordances. For instance, a pi ck (banana) action might fail if the robot is already holding something, if it is not in the correct location or the fridge was not opened beforehand. Such failures are transformed into textual feedback (e.g., "cannot pick banana"), ap- pended to the LLM's input, and used to generate an updated, executable plan. This iterative process, involving planning, validation, and feedback integration, continues until feasible plan is obtained. The validated plan is then passed to low-level motion planner for robotic execution. An example of the LLM-scene graph interaction during iterative replanning is provided in Appendix L. Specific implementation details are provided in Appendix A.
4 Experimental Setup
We design our experiments to evaluate the 3D scene graph reasoning capabilities of LLMs with a particular focus on high-level task planning pertaining to a mobile manipulator robot. The plans ad- here to a particular embodiment consisting of T-degree-of-freedde gripper attached exhibit multiple rooms and multiple floors which the LLM agent has to plan across. To better ablate and showease the capabilities of SayPlan, we decouple its semantic search ability from the overall causal planning capabilities using the following two evaluation settings as shown in Appendix C:
robot arm with two-fingered mobile base. We use two large-scale environments, shown in Figure 4, which
Semantic Search: Here, we focus on queries which test the semantic search capabilities of an LLM provided with collapsed 3D scene graph. This requires the LLM to reason over the room and floor node names and their corresponding attributes in order to aid its search for the relevant assets and objects required understand how the semantic search capabilities of an LLM compare to a human's 's thought process. Furthermore, to gain better understanding of the impact different LLM models have on this graph- based reasoning, we additionally compare against a variant of Say SayPlan using GPT-3..
solve the given task instruction. We evaluate against human baseline to
Causal Planning: In this experiment, we evaluate the ability of SayPlan to generate feasible plans to solve given natural language instruction. The evaluation metrics are divided into two compo- nents: 1) Correctness, which primarily validates the overall goal of the plan and its alignment to what human would do to solve the task and 2) Executability, which evaluates the alignment of the plan to the constraints of the scene graph environment and its ability to be executed by a mobile manipulator robot. We note here that for plan to be executable, it does not necessarily have to be correct and vice versa. We evaluate SayPlan against two baseline methods that integrate an LLM for task planning:
LLM-As- -lllnnee which generates a full plan sequence in an open-loop manner; the plan includes the full sequence of both navigation and manipulation actions that the robot must execute to complete a task, and LLM+P, an ablated variant of SayPlan, which only incorporates the path planner to allow for shorter horizon plan sequences, without any iterative replanning.
5 Results
Office
Home StyPPan (SFI-3.5)
5.1 Semantic Search
Subtask
Human
(GPT-3.5)
SayPlan
Say Plan (827-4)
Human
Sla (GPI-0)
Simple Search
100%
6.6%
86.7% 73.3%
100%
0.0%
86.7% 73.3%
Complex Search
100%
0.0%
100%
0.0%
We summarise the results for the semantic search evaluation in Table 1. SayPlan (GPT-3.5 consistently failed to reason over the input graph representation, hallucinating nodes to explore or stagnating at exploring the same node multiple times. SayPlan (GPT-4) in contrast achieved 86.7% and 73.3% success in identifying the desired subgraph across both the simple and complex search tasks respectively, demonstrating significantly better graph-based reasoning than GPT-3.5. 5.
Table 1: Evaluating the semantic search capabilities of GPT-4. The table shows the semantic search success rate in finding suitable subgraph for planning.

---

Simple
Long Horizon
Types of Errors
Corr
Exec
Corr
Exec
Missing
Missing
Wrong Action
Incomplete Search
Hallucinated
Action
Pose
Nodes
LLM+P
93.3%
13.3%
33.3%
0.0%
26.7%
0.0%
10.0%
3.33%
10.0%
LLLM-As-Plamme
93.3%
80.0%
66.7%
13.3%
20.0%
60.0%
0.17%
0.03%
10.0%
SayPlan 93.3%
100.0%
73.3%
86.6%
0.0%
0.0%
0.0%
0.0%
6.67%
Table 3: Causal Planning Results. Left: Correctness and Executability Simple and Long Horizon planning tasks and Right: Types of execution errors encountered when planning using LLMs. Note that SayPlan corrects the majority of the enrors faced by LLM-based planners.
While as expected the human baseline achieved 100% all sets of instructions, interested in the qualitative assessment of the common-sense reasoning used during seman- tic search. More specifically we would like to identify the similarity in the semantic search heuristics utilised by humans and that used by the underlying LLM based on the given task in- struction.
5500
"Annd me object
more
"Andd me something sharp."
"Locate cabinet with Iters."
Number of Tokens
5000
.... Without node contraction
With node contraction
4003
We present the full sequence of explored nodes for both SayPlan (GPT-4) and the human base- line in Appendix F. As shown in the tables, Say- Plan (GPT-4) demonstrates remarkably similar performance to human's 's semantic and com- mon sense reasoning for most tasks, exploring similar sequence of nodes given particu- lar instruction. For example, when asked to "find a ripe banana". the LLM first explores the kitchen followed by the next most likely location, the cafeteria. In the case where no semantics are present in the instruction such as "find me object K31X", we note that the LLM agent is capable of conducting a breadth- first-like search across all the unexplored nodes. This highlights the importance of meaningful node names and attributes that capture the rel- evant environment semantics that the LLM can leverage to relate the query instruction for effi- cient search.
3500
contract
Steps
Figure 3: Scene Graph Token Progression Dur- ing Semantic Search. This graph illustrates the scalability of our approach to large-scale 3D scene graphs. Note the importance of node contraction maintaining near constant token representa- tion of the 3DSG input.
Full Graph
Collapsed Graph (Token Count)
Compression Ratio
(Token Count)
Office Home
6731
878
86.9%
6598
1817
72.5%
Table 3D Scene Graph Token Count Number of tokens required for the full graph vs. collapsed graph.
An odd failure case in the simple search instructions involved negation, where the agent consistently failed when presented with questions such as "Find me an office that does not have cabiner" or "Find me bathroom with no toiler". Other failure cases noted across the complex search instruc- tions included the LLM's failure to conduct simple distance-based and count-based reasoning over graph nodes. While trivial to a human, this does require the LLM agent to reason over multiple nodes simultaneously, where it tends to hallucinate or miscount connected nodes.
Scalability Analysis:
We additionally analyse the scalability of SayPlan during semantic search. Table 2 illustrates the impact of exploiting the hierarchical nature of 3D scene graphs and allowing the LLM to explore the graph from a collapsed initial state. This allows for a reduction of 82.1 1% in the initial input tokens required to represent the Office environment and a 60.4% reduction for the Home environment. In Figure 3, we illustrate how endowing the LLM with the ability to contract explored nodes which it deems unsuitable for solving the task allows it to maintain near-constant input memory from a token perspective across the entire semantic search process. Note that the initial number of tokens already present represents the input prompt tokens as given in Appendix J. Further ablation studies on the scalability of SayPlan to even larger 3DSGs are provided in Appendix H.
7

---

5.2 Causal Planning
The results for causal planning across simple and nang-horizon instructions are summarised in Ta- ble 3 (left). We compared SayPlan's performance against two baselines: LLM-As- A---A--Plaaner and LLM+P. All three methods displayed consistent correctness in simple planning tasks at 93%, given that this metric is more function of the underlying LLMS reasoning capabilities. However, it is in- teresting to note that in the long-horizon tasks, both the path planner and iterative replanning play an important role in improving this correctness metric by reducing the planning horizon and allowing the LLM to reflect on its previous output.
The results illustrate that the key to ensuring the task plan's executability was iterative replanning. Both LLM-As-Planner and LLM+P exhibited poor executability, whereas SayPlan achieved near- perfect executability as a result of iterative replanning, which ensured that the generated plans were grounded to adhere to the constraints and predicated imposed by the environment. Detailed task plans and errors encountered are provided in Appendix G. We summarise these errors in Table 3 (right) which shows that plans generated with L.M+P and I.L.-As-Planme entailed various types of errors limiting their executability. LLM+P mitigated navigational path planning errors as result of the classical path planner however still suffered from errors pertaining to the manipulation of the environment missing actions or incorrect actions which violate environment predicates. SayPlan mitigated these errors via iterative replanning, however in 6.67% of tasks, it failed to correct for some hallucinated nodes. While we believe these errors could be eventually corrected via iterative replanning, we limited the number of replanning steps to throughout all experiments. We provide an illustration of the worldw execution of generated plan using SayPlan on mobile manipulator robot coupled with a vision- vision-guided motion controller [44, 45] in Appendix I.
Limitations
SayPlan is notably constrained by the limitations inherent in current large language models (LLMs), including biases and inaccuracies, affecting the validity of its generated plans. More specifically, SayPlan is limited by the graph-based reasoning capabilities of the underlying LLM which fails at simple distance-based reasoning, node count-based reasoning and node negation. Future work could explore fine-tuning these models for these specific tasks or alternatively incorporate existing and more complex graph reasoning tools [46] to facilitate decision Docision-making. Secondly, SayPlan's current framework is constrained by the need for a pre-built 3D scene graph and assumes that ob- jects remain static post-map generation, significantly restricting world environments. Future work could explore how online scene graph SLAM systems [15] could be integrated within the SayPlan framework to account for this. Additionally, the incorporation of open-vocabulary representations within the scene graph could yield a general scene representation as opposed to solely textual node descriptions. Lastly, a potential limitation of the current system lies in the scene graph simulator and its ability to capture the various planning failures within the environment. While this works well in the cases presented in this paper, for more complex tasks in- volving diverse set of predicates and affordances, the incorporation of relevant feedback messages for each instance may become infcasible and forms an important avenue for future work in this area.
adaptability to dynamic real-
7 Conclusion
SayPlan is a natural language-driven planning framework for robotics that integrates hierarchical 3D scene graphs and L.LMs to plan across large-scale environments spanning multiple floors and rooms. We ensure the scalability of our approach by exploiting the hierarchical nature of 3D scene graphs and the semantic reasoning capabilities of LLMs to enable the agent to explore the scene graph from the highest level within the hierarchy, resulting in a significant reduction in the initial tokens required to capture larger environments. Once explored, the LLM generates task plans for a mobile manipulator robot, and scene graph simulator ensures that the plan is feasible and grounded to the environment via iterative replanning. The framework surpasses existing techniques in producing correct, executable plans, which a robot can then follow. Finally, we successfully translate validated plans to a real-world mobile manipulator agent which operates across multiple rooms, assets and objects in large office environment. SayPlan represents a step forward for general-purpose service robotics that can operate in our homes, hospitals and workplaces, laying the groundwork for future research in this field.
8

---

Acknowledgments
The authors would like to thank Ben Burgess-Limeries for assistance with the robot hardware setup, Nishant Rana for creating the illustrations and Norman Di Palo and Michael Milford for insight- ful discussions and feedback towards this manuscript. The authors also acknowledge the ongoing support from the QUT Centre for Robotics. This work was partially supported by the Australian Government through the Australian Research Council's Discovery Projects funding scheme (Project DP220102398) and by an Amazon Research Award to Niko Sünderhauf.
References
[1] I.. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L.. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, Ray, Schulman, J. Hilton, F. Kelton,
Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, S. Agarwal, Herbert- G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Had- sell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[3] OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
[4] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang. R. Julian, et al. Do As I Can, Not As Say: Grounding language in robotic affordances. In Conference on Robot Learning, pages 287-318. PMLR, 2023.
[5] N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi. Chatgpt empowered long- step robot control in various environments: A case application, 2023.
[6] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, Yu, W. Huang. Y. Chehotar, Sermanet, D. Duckworth, Levine, Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng. I. Mordatch, and P. Florence. Palm-E: An embodied multimodal language model, 2023.
[7] C. H. Song, J. Wu, C. Washington, B. M. Sadler, W.-L. Chao, and Y. Su. LLM-Planner: Few-shot grounded planning for embodied agents with large language models. arXiv preprint arXiv:2212.. 04088, 2022.
[8] W. Huang, F. Xia, T. Xiao, Chan, J. Liang, Florence, A. Zeng. J. Tompson, L. Mordatch, Y. Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint ar arXiv:2207.0566
2022.
[9] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. LLM+P: Empowering large language models with optimal planning proficiency. arXiv preprint arXiv:2304.11477 2023.
[10] T. Silver, V. Hariprasad, R. S. Shuttlewothh, N. Kumar, T. Lozano-Pérez, and L. P. Kaelbling. PDDL planning with pretrained large language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop.
[11] Armeni, Z.- He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and Savarese. 3D scene graph: A structure for unified semantics, 3D space, and camera. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5664-5673, 2019.
[12] U.-H. Kim, J.-M. Park, T.-J. T.-J.onng, and J.-H. Kim. 3-D scene graph: A sparse and semantic rep- resentation of physical environments for intelligent agents. IFEE transactions on cybernetics, (9(12):491144933 2019.

---

[13] A. Rosinol, A. Violette, M. Abate, N. Hughes, Y. Chang, J. Shi, A. Gupta, and L. Carlone. Kimera: From slam to spatial perception with 3D dynamic scene graphs. The International Journal of Robotics Research, 40(12-14):511--5444 2021.
[14] P. Gay, J. Stuart, and A. Del Bue. Visual graphs from motion (vgfm): Scene understanding with object geometry reasoning. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part Ill 14, pages 330-346. Springer, 2019.
[15] N. Hughes, Y. Chang, and L. Carlone. Hydra: A real-time spatial perception engine for 3D scene graph construction and optimization. Robotics: Science and Systems X/V, 2022.
[16] C. Agia, M. Jatavallabhula, M. Khodeir, O. Miksik, V. Vineet, M. Mukadam, L. Paull, and F. Shkurti. Taskography: Evaluating robot task planning over large 3D scene graphs. In Conference on Robot Learning, pages 46-58. PMLR, 2022.
[17] N. Shinn, F. Cassano, B. Labash, A. Gopinath, K. Narasimhan, and S. Yao. Reflexion: Lan- guage agents with verbal reinforcement learning, 2023,
[18] Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11900 2022.
[19] E. W. Dijkstra. A note on two problems in connexion with graphs. In Edsger Wybbe Dijkstra: His Life, Work, and Legacy, pages 287-290. 2022.
[20] D. McDermott, M. Ghallab, Howe, C. Knoblock, Ram, M. Veloso, D. Weld, and D. Wilkins. PDDL-the planning domain definition language. 1998.
[21] M. Fox and D. Long. PDDL2. 1: An extension to PDDL for expressing temporal planning domains. Journal of artificial intelligence research, 20:61-124, 2003.
[22] P. Haslum, N. Lipovetzky, D. Magazzeni, and C. Muise. An introduction to the planning do- main definition language. Synthesis Lectures Artificial Intelligence and Machine Learning, 13(2):1-877 2019.
[23] M. Gelfond and Y. Kahl. Knowledge representation. reasoning, and the design of intelligent agents: The answer-set programming approach. Cambridge University Press, 2014.
[24] Tellex, Kollar, S. Dickerson, M. Walter, A. G. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. Proceedings of the AAAI Conference on Artificial Intelligence. 2011.
[25] Thomason, A. Padmakumar, J. Sinapov, N. Walker, Y. Jiang, H. Yedidsion, J. W. Hart, Stone, and R. J. Mooney. Jointly improving parsing and perception for natural language commands through human-robot dialog. J. Artif. Intell. Res., 67:327-374, 2020.
[26] H. Kautz and B. Selman. Pushing the envelope: Planning, propositional logic, and stochastic search. In Proceedings of the national conference on artificial intelligence, pages 1194- 1996.
[27] B. Bonet and H. Geffner. Planning as heuristic search. Artificial Intelligence, 129(1-2):5-33, 2001.
[28] M. Vallati, L. Chrpa, M. Grzeš, T. L. McCluskey, M. Roberts, S. Sanner, et al. The 2014 international planning competition: Progress and trends. Al Magazine, 36(3):90-99) 98, 2015.
[29] R. Chitnis, T. Silver, B. Kim, L. Kaelbling, and T. Lozano-Perez. CAMPs: Learning Context- Specific Abstractions for Efficient Planning in Factored MDPs. In Conference on Robot Leam- ing. pages 64-79. PMLR, 2021.
[30] T. Silver, R. Chitnis, A. Curtis, J. B. Tenenbaum, T. Pozano-Pérez, and P. Kaelbling. Plan- ning with learned object importance in large problem instances using graph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11962-11971, 2021.
10

---

[31] F. Ceola, E. Tosello, L. Tagliapietra, G. Nicola, and S. Ghidoni. Robot task planning via deep reinforcement learning: a tabletop object sorting application. In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC), pages 482, 2019. doi:10. 1009 SMC.2019.89142777
[32] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03622:2 2022.
[33] A. Zeng. A. Wong. S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sind- hwani, J. Lee, V. Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reason- ing with language. arXiv preprint arXiv:2204. 00598, 2022.
[34] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05/28, 2023.
[35] B. Peng, M. Galley, He, H. Cheng, Y. Xie, Y. Hu, Q. Huang, Liden, Z. Yu, W. Chen, et al. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 12813, 2023.
[36] T. Schickk, J. Dwivedi- Yu, R. Dessi, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and
T. Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476/ 2023.
[37] R. Liu, J. Wei, S. S. Gu, T.-Y. Wu, S. Vosoughi, C. Cui, D. Zhou, and A. M. Dai. Mind's eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210:05359, 2022.
[38] M. Skreta, N. Yoshikawa, S. Arellano-Rubach Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru- Guzik, F. Shkurti, and A. Garg. Errors are useful prompts: Instruction guided task program- ming with verifier-assisted iterative prompting. arXiv preprint arXiv:2303. 2023.
[39] Z. Ravichandran, L. Peng, N. Hughes, D. Griffith, and L. Carlone. Hicrarchical represen- tations and explicit memory: Learning effective navigation policies on 3D scene graphs using graph neural networks. In 2022 International Conference on Robotics and Automation (ICRA), pages 9272 9279. IEEE, 2022.
[40] A. Kurenkov, R. Martin-Martin J. Ichnowski, K. Goldberg, and S. Savarese. Semantic and ge- ometric modeling with neural message passing in 3D scene graphs for hierarchical mechanical search. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 11227 1233. IEEE, 2021.
[41] S. Garg, N. Sünderhauf, F. Dayoub, D. Morrison, A. Cosgun, G. Carneiro, Q Wu, T.-J. Chin, I. Reid, S. Gould, et al. Semantics for robotic mapping, perception and interaction: A survey. Foundations and Trends@ in Robotics, 8(1-2):1-22) 2020.
[42] A. A. Hagberg, A. Schult, and J. Swart. Exploring network structure, dynamics, and function using networkx. In G. Varoquaux, T. Vaught, and J. Millman, editors, Proceedings of the 7th Python in Science Conference, pages 11 15, Pasadena, CA USA, 2008
[43] M. Skreta, N. Yoshikawa, S. Arelaano-Rubach, Z. Ji, L. B. Kristensen, K. Darvish, A. Aspuru- Guzik, F. Shkurti, and A. Garg. Errors are useful prompts: Instruction guided task pro- gramming with veriffer-assisted iterative prompting. ArXiv, abs/2303.14100, 2023. URL https: //api. semanticschol... wwwphhooo..rrrrreooooeoeeooeeeee http//CorpusI::257555:
[44] Haviland, N. Sünderhauf, and P. Corke, holistic approach to reactive mobile manipulation. IEEE Robotics and Automation Letters, 7(2):322--312)) 2022.
[45] P. Corke and J. Haviland. Not your grandmother's toolbox-the robotics toolbox reinvented for python. In 202/ IEEE international conference on robotics and automation (ICRA), pages 11357-11363. IEEE, 2021.
[46] Zhang. Graph-toolformer: To empower LLMS with graph reasoning ability via prompt augmented by chatgpt. arXiv preprint arXiv:2304. /1116, 2023.
11

---

[47] S. Haddadin, S. Parusel, L. Johannsmeier, S. Golz, S. Gabl, F. Walch, M. Sabaghian, C. Jähne, L. Hausperger, and Haddadin. The franka emika robot: A reference platform for robotics research and education. IEEE Robotics and Automation Magazine, 29(2):46 64, 2022. doi: 10.1109/MRA. 20000MMRR.222.3333333222
[48] Omron. Omron LD / HD Series. URL https://wwit ia. omron cowpppoduutss// Family/3es 3664 dimens ion. html.
[49] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), 2023.
[50] K. Rana, A. Melnik, and N. Sünderhauf. Contrastive language, action, and state pre-training for robot learning, 2023.
[51] Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In 7th Annual Conference on Robot Learning. 2023.
[52] K. Rana, M. Xu, Tidd, M. Milford, and N. Suenderhauf. Residual skill policies: Learning adaptable skill- based action space for reinforcement learning for robotics. In 6th Anmual Conference on Robot Learning, 2022. URL https:/ /ttps/////ppeeeee.../.m net fttt/foommmiii Onb inb97NgypbK.
12

---

A Implementation Details
We utilise GPT-4 4 [3] as the underlying LLM agent unless otherwise stated. We follow a similar prompting structure to Wake et al. [5] as shown in Appendix J. We define the agent's role, details pertaining to the scene graph environment, the desired output structure and a set of input-output examples which together form the static prompt used for in-context learning. This static prompt both task- and environment--gnostte and takes up =3900 tokens of the LLM's input. During semantic search, both the 3D Scene Graph and Memory components of the input prompt get updated at each step, while during itcrative replanning only the Feedback component gets updated with information from the scene graph simulator. In all cases, the L.MM is prompted to output JSON object containing arguments to call the provided API functions.
B Environments
Office Space Single-Floor, Multi-Room
Home
Multi-Floor, Multi-Room
Figure 4: Large-scale environments used to evaluate SayPlan. The environments span multiple rooms and floors including vast range of
We evaluate SayPlan across set of two large-scale environments spanning multiple rooms and floors as shown in Figure 4. We provide details of each of these environments below, including breakdown of the number of entities and tokens required to represent them in the 3DSG:
Office: A large-scale office floor, spanning 37 rooms and 1.51 assets and objects which the agent interact with. A full and collapsed 3D scene graph representation this environment are provided Appendix D and respectively. This scene graph represents real-world office floor within which mobile manipulator robot is present. This allows us to embody the plans generated using SayPlan and evaluate their feasibility the corresponding environment. Real-world video demonstrations of a mobile manipulator robot executing the generated plan in this office environment are provided on our project site?.
An existing 3D scene graph from the Stanford 3D Scene Graph dataset [11] which consists of family home environment (K1 ick ttt) spanning 28 rooms across floors and contains 112 assets and objects that the agent can interact with. A 3D visual of this environment can be viewed the 3D Scene Graph project website?.
Home:
B.1 Real World Environment Plan Execution
To enable real-world execution of the task plans generated over 3DSG, we require corresponding 2D metric map within which we can align the posed nodes captured by the 3DSG. At each room node we assume the real robot can visually locate the appropriate assets and objects that are visible to
*sayplan.github.io
"3dscenegraph.saae wtwnnstd.ttnnnneeee wettoordddd///KKiiiiiii
13

---

Entity Type
Number of Entities
Total Number Tokens
Average Number of Tokens
Room Node
340
9.19
Asset Node
73
1994
27.3
Object Node
2539
32.6
Agent Node
15
15.0
Node Edges
218
1843
8.45
Full Graph
407
6731
16.5
Collapsed Graph
105
878
8.36
Table Detailed 3DSG breakdown for the Office Environment. The table summarises the num-
ber of different entities present in the 3DSG, the total LLM tokens required to represent each entity group and the average number of tokens required to represent a single type of entity.
Entity Type
Number of Entities
Total Number Tokens
Average Number of Tokens
Room Node
231
8.25
Asset Node
52
1887
36.3
Object Node
1881
31.35
Agent Node
15
15
Node Edges
323
2584
8
Full Graph
464
6598
14.2
Collapsed Graph
240
1817
7.57
Table 5: Detailed 3DSG breakdown for the Home Environment. The table summarises the num-
ber of different entities present in the 3DSG, the total LLM tokens required to represent each entity group and the average number of tokens required to represent a single type of entity.
within the 3DSG. The mobile manipulator robot used for the demonstration consisted of a Franka Panda 7-DoF robot manipulator [47] attached an LD-60 Omron mobile base [48]. The robot is equipped with LiDAR scanner to localise the robot both within the real world and the correspond- ing 3DSG. All the skills or affordances including pick, place, open and close were developed using the motion controller from [44] coupled with a RGB-D vision module for grasp detection, and behaviour tree to manage the execution each component including failure recovery. Future work could incorporate a range of pre-trained skills (whisking, flipping, spreading etc.) using imitation learning [49, 50) or reinforcement learning [51, 52] increase the diversity of tasks that SayPlan is able to achieve.
C Tasks
Instruction Family
Num
Explanation
Example Instruction
Semantic Search
Queries focussed on evaluating the basic semantic search capahilities of SlyPlan
Simple Search
Find me ripe banana.
Abstract semantic search queries which roquire complex reasoning
Find the room where people are playing board games.
Complex Search
Causal Planning
Querics which require the agent to perform scarch, causal reasoning and enviromment interaction in order solve Long Horizon planning queries requiring multiple interactive steps
Refrigerate the orange left the kitchen bench.
Simple Planning
Tobi spilt soda desk. Help him clean up.
Long-Horizon Planning
15
Table 6: List evaluation task instructions. We evaluate SayPlan on 90 instructions, grouped to test various aspects of the planning capabilities across large-scale scene graphs. The full instruction set is given in Appendix C.
14

---

We evaluate SayPlan across 4 instruction sets which are classified to evaluate different aspects of its 3D scene graph reasoning and planning capabilities as shown in Table
Simple Search:
Focused on evaluating the semantic search capabilities of the LLM based on queries which directly reference information in the scene graph well the basic graph-based reasoning capabilities of the LMM.
Complex Search: Abstract semantic search queries which require complex reasoning. The infor- mation required to solve these search tasks is not readily available in the graph and has to be inferred by the underlying LLM.
Simple Planning: Task planning queries which require the agent to perform graph search, causal reasoning and environment interaction in order to solve the task. Typically requires shorter horizon plans over single rooms.
Long Horizon Planning: Long Horizon planning queries require multiple interactive steps. These queries evaluate SayPlan's 's ability to reason over temporally extended instructions to investigate how well scales to such regimes. Typically requires long horizon plans spanning multiple rooms.
The full list of instructions used and the corresponding aspect the query evaluates are given in the following tables:
C.1
Simple Search
C.1.1 Office Environment
Instruction
Find me object K3IX.
unguided search with semantic
Find me carrot. Find me anything purple in the postdoc bays.
semantic search based on node name
semantic search with termination conditioned tribute
Find me ripe banana.
semantic search with termination conditioned at- tribute
Find something that has screwdriver in it.
unguided search with termination conditioned on children
One of the offices has poster of the Terminator. Which one is it? printed a document but don't know which printer has it. Find the document. left my headphones in one the meeting rooms. Locate them.
semantic search with termination conditioned
children
semantic search based on parent
semantic search based on parent
Find the PhD bay that has drone in
semantic search with termination conditioned on children
Find the kale that not in the kitchen.
D> semantic search with termination conditioned on negation predicate parent
Find me an office that does not have cabinet.
semantic search with termination conditioned on negation predicate children
Find me an office that contains cabinet, a desk, and a chair.
semantic search with termination conditioned on conjunctive query children
Find a book that was left next to robotic gripper.
> semantic search with termination conditioned on a sibling
Luis gave one his neighbours a stapler. Find the stapler.
D semantic search with termination conditioned on a sibling
There is meeting room with a chair but no table. Locate it.
>> semantic search with termination conditioned on conjunctive query with negation
Table 7: Simple Search Instructions. Evaluated in Office Environment.
15

---

C.1.2 Home Environment
Instruction
Find me FooBar. Find me bottle of wine. Find me plant with thorns.
unguided search with semantic
semantic search based on node name
semantic search with termination conditioned on at-
tribute
Find me plant that needs watering.
D> semantic search with termination conditioned on at- tribute
Find me bathroom with no toilet.
semantic search with termination conditioned on negation prodicate
The baby dropped their rattle in one of the rooms. Locate it.
semantic search based on node name
left my suitcase either in the bedroom or the living room. Which room is it in. Find the room with a ball in it. I forgot my book bed. Locate it. Find an empty vase that was left next to sink.
semantic search based on node name
semantic search based on node name
semantic search based on node name
I> semantic search with termination conditioned on sib- ling
Locate the dining room which has a table, chair and baby monitor. Locate a chair that is not in any dining
E> semantic search with termination conditioned on con- juctive query
D> semantic search with termination conditioned on negation predicate
I need to shave. Which room has both a razor and shaving cream.
semantic search with termination conditioned on
children
Find me 2 bedrooms with pillows in them. Find me bedrooms without pillows in them.
b semantic search with multiple returns
semantic search with multiple returns based on nega- tion predicate
Table 8: Simple Search Instructions. Evaluated in Home Environment.
16

---

C.2
Complex Search
C.2.1 Office Enviromment
Instruction
Find object J64M. J64M should be kept at below 0 degrees Celsius. Find me something non vegetarian.
D> semantic search guided by implicit world knowledge (knowledge directly encoded in graph)
semantic search with termination conditioned on im- plicit world knowledge
Locate something sharp.
> unguided search with termination conditioned on im- plicit world knowledge
Find the room where people are playing board games.
semantic search with termination conditioned on ability deduce context from node children using world knowledge ("board game" is not part of any node name or attribute in this graph)
Find an office of someone who is clearly fan of Arnold Schwarzenegger.
semantic search with termination conditioned on ability to deduce context from node children using world knowledge
There is postdoc that has a pet Husky. Find the desk that's most likely theirs.
b semantic search with termination conditioned on ability deduce context from node children using world knowledge
One of the PhD students was given than one complimentary T-shirts. Find his desk.
semantic search with termination conditioned the number of children
Find me the office where a paper attachment device inside asset that open.
semantic search with termination conditioned on node descendants and their attributes
There is an office which has a cabinet containing exactly items in it. Locate the office.
semantic search with termination conditioned on the number of children
There is an office which has a cabinet containing rotten apple. The cabinet name contains an even number. Locate the office.
semantic search guided by numerical properties
Look for carrot. The carrot is likely to be meeting room but I'm not sure. Find me meeting room with a RealSense
semantic search guided by user provided bias
semantic search that has no result (no meeting room has realsense camera the graph)
camera.
Find the closest fire extinguisher to the manipulation lab.
search guided by node distance
Find me the closest meeting room to the kitchen.
search guided by node distance
Either Filipe Tobi has my headphones.
evaluating constrained search, early termination the two office are explored
Locate it.
Table 9: Complex Search Instructions. Evaluated in Office Environment.
17

---

C.2.2 Home Environment
Instruction
I need something access ChatGPT.
semantic search guided by implicit world knowledge
Where should go?
Find the livingroom that contains the most electronic devices.
semantic search with termination conditioned on
children with indirect information
Find me something to eat with a lot of potassium.
semantic search with termination conditioned on plicit world knowledge
I left a sock bedroom and one in the living room. Locate them. They should match.
semantic search with multiple returns
Find me potted plant that is most likely cactus.
semantic search with termination implicitly condi- tioned on attribute
Find the dining room with exactly 5 chairs.
semantic search with termination implicitly condi-
tioned on quantity of children
Find me the bedroom closest to the home office.
semantic search with termination implicitly condi-
tioned on node distance
Find me bedroom with unusual amount of bowls. Which bedroom is empty.
semantic search with termination implicitly condi-
tioned on quantity of children
semantic search with termination implicitly condi-
tioned on quantity of children
Which bathroom has the most potted plants.
semantic search with termination implicitly condi-
tioned on quantity of children
The kitchen flooded. Find somewhere can heat my food.
semantic search guided by negation
Find me the room which most likely belongs to child
semantic search with termination conditioned on ability deduce context from node children using world knowledge
15 guests are arriving. Locate enough chairs to seat them.
semantic search with termination implicitly condi- tioned on the quantity specified node
A vegetarian dinner was prepared in one of the dining rooms. Locate My tie is in one of the closets. Locate it.
semantic search with selection criteria based on world knowledge
evaluating constrained search that has result, ter- mination after exploring closets
Table 10: Complex Search Instructions. Evaluated in Home Environment.
18

---

C.3
Simple Planning
Instruction
Close Jason's 's cabinet.
Refrigerate the orange left on the kitchen bench.
Take care of the dirty plate in the lunchroom.
Place the printed document on Will''s desk.
Peter is working hard at his desk. Get him a healthy snack. Hide one of Peter's valuable belongings.
Wipe the dusty admin shelf.
There is coffee dripping on the floor. Stop it.
Place Will's drone on his desk.
Move the monitor from Jason's 's office to Filipe's.
My parcel just got delivered! Locate it and place the appropriate lab. Check if the coffee machine is working.
Heat up the chicken kebab.
Something is smelling in the kitchen. Dispose of it.
Throw what the agent is holding in the bin.
Table 11: Simple Planning Instructions. Evaluated in Office Environment.
C.4
Long Horizon Planning
Instruction
Heat up the noodles in the fridge, and place it somewhere where can enjoy it. Throw the rotting fruit in Dimity's 's office in the correct bin.
Wash all the dishes on the lunch table. Once finished, place all the clean cutlery in the drawer.
Safely file away the freshly printed document in Will's office then place the undergraduate thesis on his desk.
Make Niko coffee and place the mug his desk.
Someone has thrown items in the wrong bins. Correct this. Tobi spilt soda his desk. Throw away the can and take him something to clean with. want to make a sandwich. Place all the ingredients on the lunch table.
delegation of project partners is arriving soon. We want to serve them snacks and non-alcoholic drinks. Prepare everything in the largest meeting room. Use items found in the supplies room only.
Serve bottled water to the attendees who are seated in meeting room 1. Each attendee can only receive a single bottle of water.
Empty the dishwasher. Place all items in their correct locations
Locate 6 complimentary t-shirts given to the PhD students and place them on the shelf admin.
I'm hungry. Bring me an apple from Peter and pepsi from Tobi. I'm at the lunch table. Let's play prank on Niko. Dimity might have something. There is an office which has a cabinet containing a rotten apple. The cabinet name contains an number. Locate the office, throw away the fruit and get them fresh apple.
Table 12: Long-Horizon Planning Instructions. Evaluated in Office Environment.
19

---

D Full 3D Scene Graph: Office Environment
Room
Object Pose Agent Asset
voln
muxt
crilitt
Erits
24
m1
22
beltl 2222
batt
0P 0
bay
11
+
Figure 5: 3D Scene Graph Fully Expanded Office Environment. Full 3D scene graph exposing all the rooms, assets and objects available the scene. Note that the LLM agent never sees all this information unless it chooses to expand every possible node without contraction.
20

---

E Contracted 3D Scene Graph: Office Environment
Room
Object Pose Agent Asset
mostig
Tmurges
Affic
printig
moce
Figure 6: 3D Scene Graph Contracted Office Environment. Contracted 3D scene graph expos- ing only the highest level within the hierarchy room nodes. This results an 82. 1% reduction in the number of tokens required to represent the scene before the semantic search phase.
21

---

F
Semantic Search Evaluation Results
- Full listings of the generated semantic search sequences for the evaluation instruction sets provided on the following pages
22

---

SayPlan
Haman
Success
Fail
Find me object K3
nanipulation,
agriculture,
Lounge]
Joungee
petere patere_office
tobis_office office tobis
nikos office
michaels office
nanipulation,
printing
agriculture
printing
aupplies
adain
nichaela office nichaels_office
Find me a carrot,
tahan
Find me anything purple postdoc bays.
postdoc bay?
posttio bay2
postdoc
Find me ripe banama.
cafeteria
cafetezia
Find something that serewdriver
mobile_ robotics Jab
manipulation, Jab
agriculture lab
zobot loungel
zobot Jounge2
station
printing
printing
xobot loungel
robot Joungo2
One offices poster Terminator, Which one is
paters office
tobis_offich office
nikos offfceffeeeeee
michaels office
Eilipes
dinitya
xi cicheala office
I printed document, but dont know which printer has it. Find the document.
printing ponel
printing
printing zone2
printing zone2
left my headphones in of the meeting Locate them.
meeting_roomi rromn
mooting 2roo22 roon2
neeting_rooe room3
meeting_ree
neeting roon2
meoting
the PhD bay that drone in
pld bay1
bay2
pbd bay]
phd bay1
phd_bay2
phd bay3
Find the kale that is not in the kitchen.
mobile robotics lab
cafeteria
agriculture lab
agri.ml tume
Find me office that does not have cabinet.
peters,
nikos
wills_effie office
luis luis_office office
Eslipea_office
office
leuriansas office
chrii_effiie effice
dinitys office
patars office
tobis office
23

---

Say
Human
Success
Fail
Find me an office that contains cubinel, a desk und a chair.
officm
tobis office
nikos office
wills
Volo_oftice
filipes_ offfce
ajay_office
Jauriannes, offfcc
chris
dimity office
_afe__office
tobis_office
nikos nikos_office office
Find me a book that was left next to robotic gripper.
mobile robotios 1ab
nanigulation lab
nanipulation
Luis gave of his neighbours stapler. Find the stapler.
suis
wills
rilipes_offee off
luis office
wills_ office
Eilipes office
There a mecting room with a chair no table. Locate
meet meeting_room roond
meeting noeting_roo22
neeting rocm3
nooting
ase boiginng_eet Foon2
Table 13: Simple Search Office Environment Evaluation. Sequence of Explored Nodes for Sim- ple Search Office Environment Instructions.
24

---

SoyPlan
Human
Success
Fail
Find object J64M. J64M should kept below degrees Celsius.
kicceen
3: tctbnnn
ie something non vegetarian.
kitchhnn
kitcoon
Locate something sharp.
kitchen
nobile_robee robotics__ue 1ab
nanipultion_ Jab
agricuiture Jab xichaeis
peters_ paters_offics office
tobis tabis_office office
rasipulation_
kitchen
cafeberia
agricul ture lab
printing printing_zonel zonel
supplias station
printing zone2
adninn
office tabis office
nikos_office office nikos
nichsels office
Find the room where people are playing board games..
presentation lounge
cafeberia
meeting rocni
neeting room2
zeeting roon3
neeting roon4
presentation, Jounge
meeting_rood
noeting_rowd room2
xeoting pacinn_rooon3
noeting roomd
Find the office of someone who is cleurly fan of Amold Schwarzenegger.
office
tobis
office
xichaels
chrip office
wills wills_office office
xichaelo office
There is postdoc that has pet Husky, Find the desk that's most likely theirs.
postdoc bayt
postdoo bay?
postsoc bayt
postooc bay?
One of the PhD students was given than complimentary T-shirt. Find his desk.
pbd bay1
pbd_ bay
phd bay2
Find me office where paper attachment device is inside an asact is open.
patars office
tobie office
nikos, oftice
michaels, office
wills office
nikes office
office
There is un office which has cabinet containing exuctly 3 items in it. Locate the office.
patera_offee office
tobis office
nikos oéfice
michaelss office
aarons_ office
jasona jasens_office office
ajays
dimitys
dimityy dixityssofffie office
lauriannes office
chris_office oftice
ajay_office
wills_offie office
There un office containing u rotten apple. The cabinet name contains un even number. Locate office.
petaza office
tobis office
nikos office
michsels office
Racona office
jaxons_officn office
ajaya_office
chria office
dinitys
nichsele office
nikos office
timitys_office office
chris office
ajays_office office
ajays
Jasona office
olll__offie
25

---

SayPlan
Look for a carrot. The carrot is likley in a mecting room not sure.
meeting rooml
neeting room2
meeting room3
neeting roond
X: tohen
neeting
neeting
neating
neeting,
Find me meeting room with RealSense cameru.
meeting rooml
weetinggrroe rocm2
maeting_rooni
setiingroood
presestation, Lounge
maeting_est Fooci roomi
noe neeting sesttii__rooe room2
neeting_Foom Foom3
nesting_room rocms
Find the closest fire extinguisher to the manipulation lab.
manipolation
adnin
Find the closest meeting the kitchen.
meeting roon3
Either Filipe Tobi my headphones, Locate them.
c11ipes_octtce octice
toble office
rilipes_offi office
_ilipes_office
tobis tobis_offfie office
Table 14: Complex Search Office Environment Evaluation. Sequence of Explored Nodes for Complex Search Office Environment Instructions.
26

---

SayPlan
Humman
Success
Fail
Find me FooBar.
bathrocmo
bathrooni
bathroon2
bathroard
bathroonk
bedrocni
bedrooo2
bedroon3
closet0
closet]
dining_roood room0
hono officeo
dining_room rooml
dining_roomm roon2
kitchen0
living
living
Living
xi tchen0
kitebenl
dining_rooc roc00
dining roonl
dining rooo2
living_roomm roon0
living_rooni
living_rooos roca2
Find me a bottle wine.
tchen0
kitebenl
dining_room roca0
diining roon]
kitchen8
dising
dining_room
living_coomd
living_roasi
kitcheni
dining_rcooa socal
Find me plant with thorss.
living_roond
living_roaxi
kitchenO
dining_roamd
bathrousi
living_roood roomd
living_roani
dining_rocmd rocal
dining_round
bedcoond
bedroamd
dining_roomi room]
living_rooos room2
Find plant that needs watering.
Living_roon0
living_rooui
kitchenO
living_roomd
living_eooed proom
living_roamt
dining_rev reeing reend
dining_round
bedroond
bedrocmi
dining_rocmt
living_
Find bathroom with no toilet.
bas Batthrosm4
ba bathroom2
bathroom
bathroom1
The baby dropped their rattle
Locate
playroomo
living_rooso roono
bedroonó
bodroon1
bedroon2
bedroomd
playzoand
living_roasd roamd
living_ recml
living_roond
dining roomd
dining_ rocnl
dining room2
bedroonó
bedroos1
bedroon2
bedroom3
left my suitcase either in the bedroom the living room. Which room is it in.
bedrocm0
badroos3
living
bedrooos0
bedroom1
living_rooo room2
bedroos3
bedroon2
living_rooni room]
living_rocsd pross0
Find room with hall
playsocm0
playroomt
forgot my book on a bed. Locate
bodrooaD
bedroon1
bedrocm22
bedroon3
bedroonn
bedroard
bedroom3
27

---

SayPlan
Human
Success
Fail
Find an empty vasc was left next sink.
bathroone
bathzooml
ba nathooon2
ba tarooon
bathroond
kitchenO
kitchen
bathroond
bathroasd
bathroon2
bathroon3
bathroond
Locate the dining which has table, chair and baby monitor.
dining
siming_rccmn zaond
dining roami
dining_re roond
dining roomt
Locate chair that any dining
Liwingg zaond
living rocmt
home officeo
shave. Which room both razor shaving creum.
bathroond
bath bathrooni
bathroon2
ba 3athooo33
bathrooss0
throon1
bathrocm2
ba tatooon
Find me bedrooms with pillows them.
bedroos0
bedroom1
bedroon2
bodroon33
Find me 2 bedrooms without pillows in them.
bedrocm0
bedroon1
bedroar2
bedroom3
Table 15: Simple Search Home Environment Evaluation. Sequence of Explored Nodes for Sim- ple Search Home Environment Instructions.
28

---

SayPlan
Human
Success
Fail
I need something access ChatGPT. Where should go?.
hone
home_ cone_officeO
Find livingroom that contains the most electronic devices.
1iving roomo
isving_rov room1
laving room2
living_roamt
rooml
living_rocod
Find me something to eat with alot of polassium.
tcheno
tohen1
ki tchen0
tohen1
left sock in bedrooom und in one of the livingroums, Locute them. They should match.
iiving_room
bedrooomO
bedroom1
bedroon2
bodroom3
living_roowd rooa0
Living roonl
Find the potted plant that likely
1iving_rocmo rocmd
Living_reosi
booe officco
kiteben0
Living_room2
1iving
room1
living_roond
Find dining room with exactly 5 chairs.
dining roamo
dining rooml
dining_rooe _ooo2
dining rocmb
dining_rocmm
dining rooml
dining_roocd _ocing__ooo22
Find me the bedroom closest to the home office.
hone
bediroom2
Find the bedroom with unusual umount of bowls.
bedroccu0
bedrocm1
bedroon2
bediroom0
bedroom1
bedroon22
Which bedroom empty.
bedroossO
bedrocm1
bedrocn2
bedzeca3
closeto
Which bathroom has the most potted plants.
bai throoooo
ba throooal
ba bathroon22
bathroos33
The kitchen is flooded. Find somewhere I heat up my food.
ki toteenn
kitchen1
dining_rrono roomo
dininggreet roamo
29

---

SayPlan
Human
Suocess
Fail
Find me the room which most likley belongs child.
bedroond
bedioomii
bedroon3
bedroonO
bedaroom1
bedroon2
bedroon3
15 guests arriving. Locate enough chairs them.
dining_room roomd
diiinng_cem soom1
living roond
hove hove_ officel
bedrocmd
living roont
dining_ross
dining_roome
living_roont
living_rent
livingrroand
vegetarian dinner was prepared one dining rooms, Locate
dining rocmo
dining room1
dining roand
dining_room roomd
dining_roomm room1
dining_room
My tie in one of the closets, Locate it.
closeto
closec1
Table 16: Complex Search Home Environment Evaluation. Sequence of Explored Nodes for Complex Search Home Environment Instructions.
30

---

G Causal Planning Evaluation Results
In this section, we provide detailed breakdown of the causal planning performance of SayPlan across the two sets of evaluation instructions. Tables 17 and 18 detail the correctness, executability and the number of iterative replanning steps took to obtain an executable plan.
No. of Replanning Iterations
Instruction
Corr.
Exec.
Close Jason's cabinet.
0
Refrigerate the orange left on the kitchen bench.
0
Take care of the dirty plate in the lunchroom.
/
0
Place the printed document on Will's desk. Peter working hard at his desk. Get him a healthy snack.
0
x
5
Hide one Peter's valuable belongings.
0
Wipe the dusty admin shelf.
0
There is coffee dripping on the floor. Stop
Place Will's drone his desk.
0
Move the monitor from Jason's 's office to Filipe's.
My parcel just got delivered! Locate it and place it in the appropriate lab.
0
Check if the coffee machine is working.
0
Heat up the chicken kebab.
Something is smelling in the kitchen. Dispose it. Throw what the agent holding in the bin.
0
Table 17: Correctness, Executability and Number of Replanning Iterations for Simple Plan- ning Instructions. Evaluating the performance of SayPlan each simple planning instruction. Values indicated in red indicate that no executable plan was identified up to that number of iterative replanning steps. In this case, 5 was the maximum number of replanning steps.
31

---

No. of Replanning
Instruction
Corr.
Exec.
Iterations
Heat up the noodles in the fridge, and place it somewhere where can enjoy
2
Throw the rotting fruit in Dimity's s office in the correct bin.
1
Wash all the dishes on the lunch table. Once finished, place all the clean cutlery in the drawer.
X
/
2
Safely file away the freshly printed document in Will's office then place the undergraduate thesis on his desk. Make Niko coffee and place the mug his desk. Someone has thrown items in the wrong bins. Correct this.
/
2
/
0
/
0
Tobi spilt soda on his desk. Throw away the can and take him something clean with. want to make a sandwich. Place all the ingredients on the lunch table.
/
3
d
3
A delegation of project partners is arriving soon. We want serve them snacks and non-alcoholic drinks. Prepare everything in the largest meeting room. Use items found in the supplies room only.
2
Serve bottled water to the attendees who are seated in meeting room Each attendee can only receive single bottle of water.
2
Empty the dishwasher. Place items in their correct locations.
/
2
Locate 6 complimentary t-shirts given to the PhD stu- dents and place them the shelf in admin. I'm hungry. Bring me an apple from Peter and a Pepsi x from Tobi. I'm at the lunch table.
/
1
x
5
Let's play prank on Niko. Dimity might have some- thing.
d
1
There is an office which has a cabinet containing a rotten apple. The cabinet name contains even number. Lo- cate the office, throw away the fruit and get them a fresh apple.
X
X 5
Table 18: Correctness, Executability and Number of Replanning Iterations for Long-Horizon Planning Instructions. Evaluating the performance of SayPlan on each long-horizon planning instruction. Values indicated in red indicate that no executable plan identified up to that number of iterative replanning steps. In this case, 5 was the maximum number of replanning steps.
The full plan sequences generated by SayPlan and all the baseline methods for each the above instructions are detailed in Table 19. Note the regions highlighted in red indicating the precise action where a plan failed.
Full listings of the generated plan sequences for the simple and long-horizon instruction sets are provided the following pages
32

---

SayPlan
LLM-P
LLM-As-Paama
Success
Fai
Close Jason's cahinet.
3)) sepode goseene
(cabinet::)
offfeet ao Co
Reffrggrate Refffige
the orang 11 comoaa
left on the kitchen bench. s.nch.
Sas22aa www.ppaoooooo..o Ceceeni loud
Ogent
woboooo besch)
cpca
rolease (range))
0004 20000000000000000000000000000000
ooent Leidde
release|
Take of the dirty plate the lunchroom.
1as5aa2Laaa tablel
sccess www.eaaaeooooa.
(
Luzz, on dishwwssse
accesel ( Londh Fabley
-coeee di www.eaaeeee
4
sanchh il Someee acce
Gablet
Richent
aoveeeeeee nn cune
Aeneweee of Socel ol
Place Plagethe the printed document on Will's desk.
3 pow.eeoeee Peooooooo 1oae
sonee po Coppooooo
Seatts coggent de Roppaed Co
Bickupt
)
So wa oftice www.uiiiiooo
Soncce Recurte dee Sogaeet
)
officw) La Sercedaae a
doc (
33

---

SayPlan
LLM+P
L.M-As-Plamer
Saccess
Fail
Peter is working hard his desk. Get him healthy snack.
gato (pone))
gotol powss)
gota I pooee2)
goto gotel (pteep
goto of pff
access (wwpsaeccbbingg (cubines2)
open (cmbisett)
páckupl (ppledl)
colease[applan
MCCMEM (Cabicet2)
ogenn (oabinet2)
(ickupp(upple))
gota power
goto(powes)
pioses))
gotolia (ptelppsil)
gotol patacc afficw)
colease/applan
gutal
gotel g powel))
guto(kitchen)
pickup/kate leavee2)
akicss beneh)
goto/kitehen)
goto:poso25)
poto(potelly
goto (poce23)
goto(peters, office)
(ccok((desk2))
releasc(kalc, leaves2)
Hide one of Peter's valuable belongings.
goso(
gotorpaters
pickupi
cabinst2)
peloase(phooee
(pose)
goso(
posse)
cotorpetera
cabinet22)
open
phone]
gutorpetera offiew)
open(cubines2)
accessi (ccess[cabinet2) (aaiinet2)
release(phone
c1ose(rabine:2))
Wipe the dusty admin shelf.
posel))))13)
goto|supplics station)
accesa (upbsrieup
pickup(paper tovel)
(oto(adria))
(sbel))
rolease ppace [owel]
Seroooo Seooed 23
Ciokugipango pi Seteoeeeeeeeeee
151 petpppoen
(ower
(chels)
Sacari ae Mesiie pood 6
Towell Biokuggias p
Seattoo Les, Iiggi Setaae a gade
goto(adrinn)
(wiipe sheld)
There coffee dripping on floor. Stop
gota| powel 13)
gotai Sotslppee1. 44)
(poeed))
AGDIMN machine)
turn of turn_off
(ooff nachine)
gote|poxell powee 13)
gotdi pooeep)
(poeed)
powed2)
goto(Kischon)
seccssjeoffee pactine)
Luzo off(coffee nachine)
0oto poto|3)))13)
(pos(1)))14)
goto(p)
goso[posc25]
(oto(kitchen)
goto|coffoe gachine)
tuzn, wwun_officofeee coffce sachinc)
34

---

SayPlan
LLM-P
LLM--A-Plamee
Success
Fail
Place Will's drone on his desk.
(poee
gotteppoee)
gotalposee0)
gntorville offfie))
accens peaenn/callneeeeee
open open/cabinetal
plokup(drone2)
ACCWER (dnaki)
relesse
goto(posed)
gotalpose10)
gotorvillg 11w offices)
accens pomeeeceeeeett
plokup(drone2)
access (deaki)
relesse
gotolpoac8)
posel)))
(oto|poseld)
(otorpose23)
posel10
goto/willa office)
cccess(cabines4)
cabn(cab)
pickup dzone2)
close(cableecd)
coto|villa officel
(ccks
zelease d2oose2)
Juson's office Filipe's.
0oto(pose13)
gotor Jascns office)
access desks)
pickupi monitoc!
pose(
gotolposeS)
goselpose(0)
goto office)
sccess deis337)
ree release (mon
1pto(pore13)
gotoo tesone office)
plotupinoniton
gotoe)
goto office)
sccess desk37)
ce release (mon too))
goto Coopoooa Soooeey
sprou Soo.oooooo office)
petolposelly
pictopinonitory
gotoo
goto office)
release (moni
My parcel just got delivered! Locate it and place it in the appropriate lab.
1pte(p)))13)
goselp
goteo (obloy)
Scoees[shel:1
pickuplparce
goselp 4
goto
gato lab)
(tabled)
(paroee)
eoboriabbyl
poto(pouo14)
potorposu14)
poselposll))
qoto(posa8)
goto(sabile
paress)
posel))se8)
possll))))))
qotorlebby)
(abelf21
pickup (paccel)
eaborlabby!
gotorpase13
gotalsebile iebotica Jabb
recesal sabbee4)
(aaeos)
Check if the coffee machine working.
goto
Peaaee coo fee foe nachinel
uorn. on (ooffee. nschine]
torr. torn_off (cotfee machine)
(pore2)
goto
(ototettchen)
acoess (oof fee. nechine)
turn, on footfee nachine)
tore. torr_off (coffee machine)
poselp
potolpose2)
goto
gota(kittchen)
goto|cof coffee nechine)
tore, tore_on (coffee zachine)
tarn of rachine)
35

---

Say Plan
LLMMAAsPPaamet
Heat up the chicken kebab.
geto:pose13)
(oto|poso14)
gotorposc23)
(0:0(posc24)
geto
goto(kiscben)
open(Eri6ge)
pickup(ehicken Xebab)
clese(fzidge)
accesa (BASSwoNóco
open ALCEOWAVE)
relcase (chicken Kebab)
close (niczowave)
19En, Cacnooo0nnicc cO
CUED GEE (zicEoNave)
coen/Bicrovave
#ickup/chickes kebab)
close(aAczowaa)))
gotolpose13] 13)
poselpose14)
goto(pose23)
goto
tcchen
4ccese fridge)
open (frilinidggm dgge)
pickup/chi chen, kebab)
close(frldge)
access (nforowwee)
open (nt arowae)
lesse (chictenn kebab)
turn_on conn_on((lerroeeae
tur_off (mterowave
open arowve))
pickup
cickep oken, kebab)
close (ntorowwve)
(pore))
poselp
goto 4)
goto(pnne223
goto[pore25)
tchhen)
srcdss)
pickuppchiicken kebab)
ffide(ffldge)
acoess (elcrovave)
open open(/l
ce leeee (chicten, kebab)
closeonncoo
tuen ouee_ooiiff [iicroaae)
tuen [etooe[[(pcrravve]
Jickup/chickes kebab
close(saccooaaee)
Something is smelling in the kitchen. Dispose of it.
goto|pooe13)
potelpose14)
gotorposee23
goto
gote
goto(kitchhn)
ac0ess (fridge
plckmp (ea Iinon, bagel)
socess /coeport. bin)
open goenonongost..ommm bin)
release malnon bagel)
close closeccompont. strppoet__bbttt bim)
poselp)))14)
gotorpose23)
goto(pore22]
goto
(kic(kichhen
cooes(ffildge
open(fr
pickap lion, bagel)
accesal pcoepost bio)
open bin)
celouse(walnee begel)
close| conpowt bio)
goto gotolpownd (poss
powel)
potorpose:4)
goto
goterp
cotockischen)
uccoastéridgo)
open
pickup papptaall sallzo baqel)
Clowe(dridge
guto(zubbish, bin)
spbniibbb bis)
ruleasel salacn bagel]
closetsabbish bin)
Throw what agent holding
gota powe 13)
poeel1)
gatoippuee]] 3
goter22
gotto (kittkkinheen)
#COаNN twayccl ing bin)
open rwayel Ling bingbbin)
Lesan (ovmet xrapper)
clowe (recycling hin)
poow
gotdl pooell) (4)
gotalpauee2 3)
gowee2)
gotockitchon)
sccyssiso bin)
release1swect wraррez)
poso|3)))13)
pooe())))14)
gotorposc23)
go:0(posc25)
(oto(kitchhn)
goco(rubbish bin)
(SNDOE vЕарроr)
36

---

SayPlan
Success
Fail
Heat up the noodles fridge, and place it somewhere where can enjoy
geto(pose13)
potolpose14)
gotolpose23)
(p05c22)
goto(poso25)
gota (kitchen))
(CEigge
(frigee
piokup(aooGles)
rolcase (soodlcs)
cloce(zicrokare)
turn, caroarro
tuzz, off (DAcrowave)
opca(xtlcroaaro)
piokup(neodies)
poso:p)
geco(eafetaria)
access table)
release
geto(pocc33)
potolpose14)
goto|pose23)
(po5c22)224)
goto(pose25)
gota (kitccen
#COMEO for idge
(friidg)
pickup lex)
ACOEXN crovsee)
open (tierrowae)
releaxe (ood mx)
bure, (microvave)
turn (miorowave)
open (#lEroowv)
pickupi me]
gowel
gato goto(cafeteria)
aCDmAR ( Lunch table)
releaxe (nood mx)
gota (kitchen)
McDeae (ridge)
(friidg)
pickup lex)
close(fridg)
#CCeSR (nicrovwe)))
open cravave)
relmase nond lex
close (nicrowwe))
turn, pe cicentee)
turn, (xieronave)
open [gwn|nicrrowve)
#Jckup(noodlco
close (Diczovave)
cafeteria
Throw the rotting fruit in Dimity's office in the correct bin.
coto(pose13)
poto|posul4)
(oto|poss23)
gcto(dinitya office
pickus (pplle))
poss)))))2))
(Litches)
open(corpost, bin)
release (applee2)
pote|poss23)
(poau22))22)
officee
pickus applee)
(posa22))))
(pone
goto potoddintyy office)
gotoidinlty office
goto
(kitchen)
release
37

---

SayPlan
LLM-P
LLM-As-Plamer
Wash all the dishes the lunch table. Once finished, place all the clean cutlery in the drawer.
goto|posee2 (pose23)
goto(pose24)
gono(pose25)
(caffeaaftarra)
A0cess| Lunch table)
piotop(plate)
goto
(kichen
ACORER accome (draver)
ogen (gwn(ddreer
release (plate)
(pose20)
goto(onfetaria)
access Lunch tabley
pickupt forkk
goto|pose25)
goho(kischen
ACDeRA (drrddddreeeee drewer))
relense fork)
goto|pos025)
goto(cafetezia)
access| lunch, tablet
(km
goso|pos025)
goto(kischen)
accessl sccesa(dzaver) dzaver)
release (kiife)
goso|pose14)
gota (ote|2)))23)
g0co(p0s025)
coto(cafeterie)
access/lunch. table)
pickupiplate)
pickup (ickup/fock fock]
pickup
goto poses5)))25)
kitchkish
access| dravez)
colease(plate)
release (ock)
release
cotorcafeteria)
gozoclunch, cables
pickupiplate)
Lickup/fock
pickup
goto poss25)
goto(kitchen)
goso(diabvaabeer
open di her)
(platre(plate)
release (oste))
release
closo(c11iiveeter)
senn_oon(dsswweheee
(per:ddee
piokup(plate:
plokup fork)
(knit
goto(draver)
open (drawer)
release (fork)
release(knite)
[drsevdr
Safely away the freshly printed document in Will's office then place undergraduate thesis on dosk.
peto(pose1s1
cotorposc9)
goto(pose3)
cotoiprintingg z0nc2)
scccss|printer2)
pickup docunent)
goto(pose3)
gotolpose51
goco(p05c10)
gotorwills office)
acccss(cablaete))
cpbn(cobi
sclease(document)
closo(cabinet4))
acooss(cablaeed)
opcn(cabisct4)
pickup(undergrad (undergrad thesisi)
closo(cabinet4))
seskss
rcicase| undeeggod thesisl)
pickup docunest)
noto|pose13)
goto(posc3)
gotorposc10)
goto(wills offiec)
accessicabinet4)
cabsncab)
releasc
closiccabi
aCOEEX (dexbd)
pickup jndergrad theeix])
rel insee undergrad thex iet)
posel)
gota possld))
pritaprinting www.iiiiinggeeeeeoooo xonal]
plaksp wraph/reechytteeeeeeee prpreed docueent)
gotorpowe22)
gotn(
gowel) (4)
gotal posel]))
gose!)
gote
gota/wi 1h office)
ACDeKK (cabinet4)
(cshicc neb4)
release freehh p_prreted, docoment,
pi pickup (undergrad, thexix!!
(cbbine
accees (cenkc))
TWi releaxe enlesxe(indergee (undergrad chesix')
38

---

SayPlan
LLM-P
LLM-As- Planner
Fail
Make Niko coffee and place the mug on his desk.
eosotnlioos
(oko1pose5)
goto|poss16)
potolpose18
eotorpose15)
gcto[pcso23)
got tchen)
fee machine)
release (coffee_ sug)
fooffee machine)
turn_off of (off machine!
pickop cooffee _ug)
gotolpose25)
goto(pose24)
potolposate
16)
goto office)
access
releare
office) 2
piokug/ootfrs
ppto(ponet)
goto
cotd appouet
access
sachine) foe
releaser coffee
cooffee pachine
suxa_ptt sure
(bmr Po piccee
aepeooae josts Seaas Aepoooaaee
cote
gota|nikos office]
acceau
leea ( selesse
Cei
goto
re
Somcone
the wrong bins. Correct
gotal wwwpppooonnnoooo
weteeeeeeeaaaaee bim)
gotol gotolporne pose
siekupiazile carton
bis)
Contrubdee biai
Coaaeecaa cazton)
accesp seyyclaa bin)
access bini
pickuptorance _cccaalo___eeeejj_______________
celeaserorande_pcn peel]
access|secycling ing bini
pickup core:
:elease(apple (apple core)
cesss:rubbiall bin)
pickup basana (actaan_____ peel)
secvsnie bin)
release banana, peel)
scossstsubbiah, bin)
pickup(plasticc btttttttt___ttleeeeeeeeee
celoseeplasticc botele)
postl 1
coto(pose14)
acto(poso23)
(oto|pose20)
cotol po9625)
(otc/kLtches)
Leeeeeeteeyellegg..... bia)
open(recycling bis)
pickupıailk cartos)
(rubbish. bin)
opentrabiish bin)
relemee(sik carton)
preyycl _iin)
pickup(eranga_pen) peol)
reloare _pol|
socess ing bini
pickup/agple Bakup//pple_ccree core:
release(sppie
babaressgple_core cooe)
coobestobbiehh sh__hhhhnnnn bin)
piokup banana peel)
access bin)
release (banana, peel)
cooesstrabiish bin)
piokop(plastit bottsie bottle)
relosseplaattte bot:le)
potolpooel3 3)
goto(pore14)
pose22)
gota pote|2)))25)
goto/kitchen)
vototrecyelinng.ou bin)
goto(rubbish bin)
release/nt carton)
goto hon.reeyylinggboo..
poranee (ookugoorange_peel] peel)
goto(rubbish bin)
release/orangee www.oooaanneeeeoeeee
goto (reecclcy11nng bini
picpapisgple (spple_ooppp)eo ooze)
getbbrubbish bin)
release(agple_com core)
gotal rubbish bis)
pickup ppul)
coto(recycling bin)
sulusse Swwpaaaaaaaeeeee
gotal subbiss bis)
bottle)
goto(twcycling_.tg bin)
veleue(platti pttsstt__tttttelllaepppppp_
39

---

SayPlan
Success
Pail
Tobi spilt soda his desk. Throw away can take something to cleun with.
goto (poce7)
goto po029
goto office)
)
1oooooo potor
bin) Ceneeeooooooo
1D1 2 0w0.00000000000000000000000
accesd Sooo
/ Coreedae cenooaa Foods
1 Comeent pew.eeoooe rood
Meo1 Soriet Aispar Lopd8
(scenod) Tww.earoooo
TOMOT padee wa Maannnn Cemsooe
1 FONEl 3
) Seooee
) Ritico Salle
lurg a tnd
coreen ane i Seseeeeeee
40

---

SayPlan
want sandwich. Place all ingredients the lunch table.
(posee))))
goso(pose1s)
goto (poo(2)))23
(pto
g0:o
(kiteconn
6000ss/kitchen, )ethh) bench)
pickap (bread)
gotorpose25)
gotolcafeseria)
500055 600ess/Luzch, Lusoh, table)
release(bread)
gotorpose25)
goto
s00ess/kitchen,, bench)
piokup(butter)
gotorpose25)
gotolcafeseria)
500e55 s00ess|Lusch, Lusch, table)
release (bacter)
gotorpose251
goto (kitcconn
accesa Cridge1
(fridge)
p cheese)
g0so(208
@oto(cafetoria)
accesa table)
release(chhess))
guto (1owe25)
acCeEM uccwwa(f-idgu (fridge)
pickupi pickup(tomato))
(oto(ccsfeeccia)
acCweM| Lunch table)
culausee tooatoo
potoeeeees.... 15
guto (p
potolpor22]
(ote22))24)
gotolpuwed's 5]
gotoo Intoriittete kitchen)
aCCIA of (kitches berch)
pickup breud)
goto
gott (affeerrie)
rellesee (bread)
gotolpowe25
gota sota/kitcchee)
pickup botter)
poxee2)
geto(casfeteria))
celeuse (buttec
guto(ppowe pooe25
(kitchen
accees acceex(-=idge
(fridge)
pickapi choese)
posop2))225)
go:o(cafeteria))
rclease(cheese)
go1o(p05025)
go:o(kitcbon)
(cckup(tonato)
goto(posc25)
gotorcafetcria)
sclease/tomato))
(posc1)))
go:o(posc14)
(oto(p)))23)
gotorpose25)
goto/kitehcs)
cheese)
(tonap(t
pickup(kale leaves2)
pickup (bread)
pickapibatter
poso22)))25)
(oto(p)))26)
pctopp:cee::stiee lounge]
rclease (cheese)
releasel tonato)
rklc leaves2)
reless(brood))
maxe butter)
41

---

SayPlan
LLM-P
LLMM-As-Planer
Empty dishwasher. Place all items in their locations
gotaipose13) 13)
gotalpose!
goto(pnee23))
goto[pone2e]
gote
kktc(ki)
access disbwesber)
open (diabwabeer))
pickupi
acceаa| accessjcabinets
open/cabiinet)
z0losse/plate2)
accessi dislvashez)
(Jckup(DowcL)
ccbossrabbinet
xelease(bowk)
accessi dislvashoz)
paciop)sspoon)
acceas (dzavez)
celease
close(dzawez)
accosrcabinett
close(eabinet)
accessi dishwashez)
close| (ishwasber)
pose1)))))4)
gotorpose23)
pose(2
goto|pose25]
koth(kitchen)
scdees; di
open(c1elvveter)
pickup(platez)
(abitee))
access sheasher)
pickip (bow))
access(cabiiet)
(hok)
sheasher)
piokkupsppon
(draver)
release repoon)
(d1 ehuasher)
gotorpose
goto(pore23)
goto[pone15)
goto(kitcten)
(dshharber))
(di
picksp (platee)
goto(kttoten)
(cabinet)
relesse(platez)
(kstectn)
cocess(deebwee
(Lokup(bawd.)
gutcckitchen)
pickup(sppon)
kitchhinn
(apoon)
(disisenabees
dishwashee)
delegation of project partners arriving soon. We serve snacks non-alcoholic drinks, Prepare everything largest meeting room. Use found in the supplies room only.
gotolpos)13)
gotorpose14)
gpoS(2)))23)
goto(posc24)
goto goto|supplics, statioc)
.00c09 (cpphoadd))
opcs oupacard2
páckcpp orange, Juice)
goto:poso24)
gotorpose25)
goto(p0
gotormeeting 200m3)
accoss(tablcl)
cclease(orange juisee
goto(posc26)
goto(posc25)
gotolpos624)
coto|supplics station)
access (cupboard))
pickup biscuits)
gotolposo24)
gotorpose25)
(pto(po
goto rncoting 2oo03)
accoscttablcl)
Pareeeeee 13)
(pone2)
potoa/uuga (ddgn) gota
atation)
wonggaoeee )d2)
pickupt
juicn)
ing cower Prrud ing_rowm3) (
gota1p)ss26)
station) (ddin | eotdb
goto raand)
popel3)
cotorsupplis station)
(etari)
(
piokup(bottled
/bottled waters)
pickuptorance
(slecuit))))
close(eupboard2)
coto1pose26)
noto/seeting
watorl)
wator2)
seltase/bottled water3)
wator4)
release release(biseuita (biseuits)
42

---

LLM
LLM- Planmer
Success
Serve bottled water to the attendees who are seated in o of water.
meeting room l.
Fach sechaia
attenwe a
single i mad can mu Intendee
aniy
/ poppooooeo 2000
slapard
(ygeeod lcpol
Iowend janayooua
Speendl coweerroooo food
Peatee Oime wommmmeoooo Josc8
(1000) Sowooooooooo
lesodeos
(22000 Por wo
prososlosa
(2200od
11000 Rcceedtionn
Costte bom houad
(suces aeseesaonnn
(Taaod)
feneion (1t1:pe)ssse 101e 19)
preeedioons
(E2sedd rojo topaó
15

---

SayPlan
LLM+P
LLM-As-Plamms
Locate all complimentary i-shirts given the PhD students and place them the shelf admin
gotorpasc7)
peso|pese5)
gotolpos26)
getorphd bayl)
3,0009 (cckk))
goto (ocospe
pickup (compp imentary tshiztl)
power gotol
(p
1
18 power gotod
maee (cxmpl imentary. rt1]
101 Padeeae
petolposss))
(o Speecdl waweeeaeee Food
bry!)
accoex
gota (atteren
pl clupp (cmmp) trenbary. r22)
(. Cooeene Maneleooe
1 Coreen Pocce
mu (comp) imentrry. wawrarryyythhhiie rt2)
dickkppicom Comnzaae bah rt.3
gntolphd bay2)
(
goto poterposeis)
coto (oto(adzin))
colease ccomp imontary tshirt3)
9010(900018)
gotorpbd pbd bay2)
accesa deak18)
pickup complinentaryy.yyyy.ooooooomccty wtwpaaaaayyttteeeeeeeeeee
tshirtéj
cotolpose16)
g0to(pose18)
secesa|
(co wwwaaay tahirté)
pose11)
gotorphd bay2)
access) doak18)
cor.upppcoopiieentarce tshirt6)
coto coteip)))15)
goto (
pbollt secesa
roloase ccomplinentary. tshizt6)
perrup/comppli (ommp) aerearrn (lattte
(8) Ceceee Cooooeo
9)
acoeee (shelir)
coompi eaas
B1 poneooee
godarad goto
la perppooe
(corp) (comp) iwwnttryy...oo..... tshirtl)
potorpcaen
compl imentrryyyyy..o tabizt2)
) Saaaaee
tahiet2)
sickupp pecpp Lientarry Rablice
gotol powerpeeeee
celoase Lsestary tabiet
goto
gutol
pickuup complinentary subire4)
gotol thhel
(pose16)
(poneel
picicup 1 comp 1a31ree
releaasecorpp tehirt4)
desk18)
aooess
release(corpp iwentary tshirt6)
piokup/ Limentary (eairt )
desess)
(pose18)
adnin)
accese (shelr)
release
44

---

SayPlan
LLM++
JLM--s-Phamne
Sucoess
Fail
I'm hungry. Bring me an apple from Peter and pepsi from Tobi. I'm at the lunch table.
pacsep
posse)
gobo(posel]
getolpabera office)
(cabinee)
pickup((appla3)
goto|posel)
office)
(deak38)
pickup
goto(ppose3)
goto(posc13)
goto(pesel4)
ootalposas)
(pto
sulems((papei)
potall Sorees
powe po
gotor (atera office)
apent weraaaaainnnn
pickup(appla3)
goto|posel]
cotoo tobia dfficny
pickuplpapat)
goto|posel]
gotalposc2)
goto
goto(pose14)
goto
gotoscafeteria)
rolease roleasejapp1e]) apple3)
peleaso/poect
poto(posc14)
gotolpesc23)
apto(poso1)
goto|petore office)
pickup picksp apple))
gotorp)
gotoi oftice)
pickap plokap(papal) (popet)
goto gotolposel)
gotolpose22)
eote1pose23)
c
pelease(pepes)
590 Aayy prank Niko. Dimity might have something.
goto1pose14)
(pose233
goto cffiee)
access| desk3)
pickup: (buaze)
goto (pose2)
0otelpose10)
goto (gose18)
goto (gose7)
(pore5)
gotkos offtice)
access/etatrl))
buzzer)
aatelposelly
(gone2))
gotol httoo/diittyooteeooeoe _ttiooe
pickup bussec)
geto (poee2))
gotal Pocoeeti
gotol (pare1
gotee)
gano (poses)) Ceeeene
(nikoe affiie))
sike)
(uzzer)
gato(poxe23)
goto/posw22)
gotafdinitys office)
(deak3)
pickup buszer)
Table 19: Causal Planning Evaluation. Task planning action sequences generated for mobile manipulator robot follow for both the simple and long-horizon planning instruction sets.
45

---

H Scalability Ablation Study
In this study, we evaluate the ability of SayPlan and the underlying LLM to reason over larger-scale scene graphs. More specifically, as SayPlan's initial input is collapsed 3DSG, we explore how increasing the number of nodes in this base environment impacts the ability of the LLM attend to the relevant parts of the scene graph for both semantic search and iterative replanning.
Succoss
Fallture
'Find me carrot."
"Find book that
robotic gripper:
Simple
"Aind ripe
"Find object 164M. J64M should be kept below degree Celsius.
non-vegetarian
something
Complex
"There postdoc who has pet Husky, Find their desk."
60
300
Base Environment Size (Number of Room Nodes)
Figure 7: Evaluating the performance of the underlying LLMs semantic search capabilities as the scale of the environment increases. For the office environment used in this study, primarily interested in the number room nodes present in the collapsed form of the 3DSG.
Success
Failure
"Clese Jasan's cabinet."
Simple Planning
valuable bellongings.
"Something is smelling the kitchen. Dispose of
"Heal noodles
Planninnlaniinn Planning
the fridge, place somewhere where can enjoy
Niko
Long Sone-Morizon Long-Horra
Dimity might something."
TTebi spill soda on his desk. Throw away the can and take him something to clean with."
Base Environment Size (Numbeer Roorn Nodes)
Figure 8: Evaluating the performance of SayPlan's causal planning capabilities as the scale of the environment increases. For the office environment used in this study, interested in the number of room nodes present in the collapsed form of the 3DSG.
primarily
We note here that all the failures that occurred across both semantic search and iterative replanning were result of the LLM's input exceeding the maximum token limits in the case of GPT-4 this corresponded 8192 tokens. With regard to the scalability larger environments, this is an important observation as it indicates that the LLM's reasoning capabilities or ability to attend to the relevant parts the 3DSG is not significantly impacted by the presence of "noisy" or increasing number of nodes. One potential downside to larger environments however is the increased number of steps required before semantic search converges. As more semantically relevant floor or room nodes enter the scene, each of these may be considered by the LLM for exploration.
46

---

I Real World Execution of a Generated Long Horizon Plan.
Srneree Coods (322)
posttde spiled cheir soab, help them claan it
Lowndd 2006
Object Pozz Apent Aesett
2a Sooooee 5
Generated Plan:
goto (pora))
Figure 9: Real World Execution of a Generated Long Horizon Plan. Execution of a generated and validated task plan on real-world mobile manipulator robot.
47

---

Input Prompt Structure
Input prompt passed to the LLM for SayPlan. Note that the components highlighted in violet rep- resent static components of the prompt that remain fixed throughout both the semantic search and iterative replanning phases of SayPlan.
Agent Role: are an excel lent graph planning agent. representation an environment, you can explore the graph by expanding nodes to find the items of interest. You can then use this graph to generate a step-by-step task plan that the agent can follow to sclve given instruction. Environment Functions:
graph
goto (<pose>): Move the agent any room node or pose node.
accoss (<asset>); Provide access to the set of affordances associated with an asset node and its connected objects.
pickup (<object>) Pick up accessible object from the accessed node.
release ject))) Release grasped object at asset node.
furn_on/off kobject>]] Toggle object at agent' s node, accessible and has affordance.
open/ciose (<asset>) Open/close asset at agent' node, aftecting object
accessibility.
done(): Call when the task completed.
Environment State:
ontop_of of (Kasset>); Object is located on <asset> inside_of (<asset>); Object is located inside <asset> inside_hand:
inside hand: Object is currently being grasped by the robot/agen:
closed: Asset can be opened open: Asset can be closed or kept open on: Asset is currently on off: Asset is currently eff accessible: The object asset state is "closed". Environment API:
accessible if it is inside asset and the
expand__oddelte node :<node>) Reveal assets/objects connected to room/Eloos node. expand contract_noae (<node>) Hide obsets, reducing graph size for memory constraints.
verify_plan(): Verify generated plan in the scene graph environment.
Output Response Format:
break your problen. down into series of intermediate reasoning steps help you determine your next command,
reasoning: justify why the next action important mode: "cxploring" OR "planning"
command:
' "ccomand__amme" Environment API call "node_name"" node to perform
speration
"plan": task plan in planning mode]] Example: <see Appendix K and 1>>
Instruction: Natural language description of the task
3D Scene Graph: Text-serial sed JSON description of 30 scene graph Memory: History of previously expanded nodes
Feedback: External textual feedback from scene graph simulator Ensure the response can be parsed by Python loan.
48

---

K Example Input-Output Interaction Between LLM and Scene Graph Simulator During Semantic Search
Instruction: make a coffee for Tom and place it in his room 3D Scene Graph: {nodes: froom: [[id: bobs_room), {id: tons_room), {id: jacks_roem {id: kitchen), {id: livingroom)l, pose: Iii: posel}, (id: pose2), (id: pose3), {id: pose4), {id: poseb),, agent: Ilocation: boos__oom, id: agent]] Links: [bobs_rrommteeeee_ roo_ssooooppee, bobs wows_ roomstagent, toms_room+ppaell toma_ coma_rooméposee, toms_room+ppoes,, jack rocke_room jacks_room++ose3, kitchen++osee,, kitchen#posed, kitchen+pose5, [ivingroom-+poee4]]
Memory: []
SayPlan [Mode->Search): chain-of-thought: explore graph for coffee mug, coffee machine and somewhere to place the coffee -> perform common sense exploration first -> explore toms room -> explore kitchen -> if items not found then explore other nodes
reasoning: will expand toms room
command: expand (toms_room)
Scene Graph Simulator: <Updated 3D scene graph> Memory: [toms_room]
SayPlan [Mode>>Search]: chain-of-thought: have found a wardrobe in tom room -> Leave this node expanded the coffee mug is not in his zoom -> still have not found the coffee machine -> kitchen might have coffee machine and co coffee mug -> explore this node next
reasoning: i will expand the kitchen next
command: expand (kitchen)
Scene Graph Simulator: <Updated 3D scene graph>
Memory: [toms_roose _oom, kitchen]
SayPlan [Mode=>Search): chain-of-thought: i've found the coffee machine in the kitchen -> the coffee mug however is not in the kitchen -> i will explore other unexpanded nodes now for the coffee mug -> jacks room, bobs roon and the living room are unexplored yet -> will try jack"s room next reasoning: i will expand jacks room next.
command: expand (jacks_room)
Scene Graph Simulator: <Updated scene graph>
Memory: [toms_room, kitchen, jacks_rocm]
SayPlan [Mode->Search]- chain-of-thought: jacks room does not have anything relevant towards solving this task -> I should contract this nede to save token memory i will explore other unexplored nodes after doing this reasoning: ract jack's s room to save memory command: cant (jacks_roos) (rom))
Scene Graph Simulator: <Updated 3D scene graph>
Memory: [toms_room, kitchen, jacks_reem,
SayPlan [Mode>>Search): chain-of-thought:
have found a wardrobe in tom's room and coffee machine in the kitchen -> the coffee mug is still required for the task -> explore new room next
reasening: i will expand bob's room as it is unexplored
command: expand (bobs_road)
Scene Graph Simulator: <Updated 30 scene graph>
Memory: [toms_room, kitchen, jacks_rocm, bobs_room]
i have found the coffee mug, the coffee machine and tom's wardrobe on the graph -> search complete -> will switch to planning mode-> collect coffee mug generate plan for making coffee -> place coffee mug on Tom's wardrobe
SayPlan [Mode->Search]: chain-of-thought
reasoning: I will generate à task plan using the identified subgraph
49

---

L Example Input-Output Interaction Between LLM and Scene Graph Environment During Iterative Replanning
Instruction: make a coffee for Tom and place it in his room 3D Scene Graph: {nodes: froom: [[id: bobs_room] {id: tons_room), {id: jacks_ room], {id: kitchen), {id: livingroom)l, pose: Hii: posel}, (id: pose2], (id: pose3), {id: pose4), {id: poseb)], agent: I[ocation: bobs rocm, id: agent] asset: I(room: toms toms_roon state: free, affordances: [release], id: bed2), (room: toms_room, state: closed, affordances: [open, close, release], id: wardrobe2), froom: kitchen, state: closed, affordances: [open, close, release], id: fridge), (room: kitchen, affordances: [turn_on turn turn_cff], state: off, id: coffee machine), (room: babs_roon, state: free, affordances: [releasel, id: bedl), (room: bobs_ bobs_room, room, state: closed, affordances: [open, close, relcase], id: wardrobelfl, object: [faffordances: (pre::::ppppuuppp [pickupl, state: inside_of (wardrobel), attributes: "bluc", id: coffee_mug)l,, links: [boos_room+pooel, bobs_roomtaagent bobs_rocmt+bedl boba_roontwwarddbbc., toms_room+pposel, tompr toms_ toms_room<+bedd)
toms_rroom-warrrbbbae, pptpr//rrrrrrrrpepaeeeeoooooorer joctssssrpeaccsssroototeeeeeee kitchen++pose3, kitchen++posee, kitchen<*poseet kitcchen--fridgee ki Livingroom-=pose,, wardrobbee+sofffee mugl]
Memory: Itoms room, kitchen, jacks_recmt reom, bobs room]
"i have found the coffee mug, the coffee machine and tom's wardrobe on the graph -> search complete -> I will switch to planning mode-> collect coffee mug generate plan for making coffee -> place coffee mug on Tom's wardrobe"
SayPlan [Mode->Planming]] chain-of-thought: t
reasoning: will generate a task plan using the identified subgraph plan: [goto (roobs_room) access (wardrcbel) > pickup (coffee_ffeffe ____]__________
goto (kitchen) > access ssoffee_machinee > release (coffee_mug) >
turn_on [ooffee_machine) turn_off (coffoe_machine) > pickup (coffce_mug) goto (toms_reem) access (wardrobe2) release (coffee_mug) done]
Path Planner:plan: Igoto (pose3) goto (pose4) goto (posel) >
goto (oobs_rcer) > access (wardrobel) > pickup (coffee_mug) > goto (pose2) >
goto (pose5) (kitthh > access cooffes_machine turn_on croffee_machinee > turn_off (coffee_macd machi ne) pickup (coffee_mug)
> release (zoffee_mug)
> goto (pose5) > goto (pose2) > goto (toms___o_room > access (wardrobe2) done]
release (coffee_nn) (coffee_mug)
Scene Graph Simulator (Feedback): "coffee mug is not accessible"
Memory: [toms_rrom, recm, kitchen, jacks_rcem, room, bobs bebs_room]
"the coffee mug was not accessible -> the wardrobe it is in is currently closed -> update the plan to open the wardrobe first -> generate plan for making coffee -> place coffec mug Tom' wardrobe"
SayPlan [Mode->Planning]: chain-of-thought
reasoning: will generate a task plan using identified subgraph and the feedback from the simulator
plan: Igoto gooobs_rrom access (wardrobel) > open (wardrobel)
pickup (coffee_mup > goto (kitchen) > access (coffee_machine)
release (coffee_mm)) suggee turn_on (coffee machine) turn_off (coffee_machine)
pickup (co fee_mug) goto (toms_room) access (wardrobs2) release (cof (coffee_mug) done]
Path Planner:plan: [goto (pose3) > goto (pose4) > goto (posel)
goto (bobs__roat roem) access (wardrobel) > open (wardrobel) > pickup (coffee mug) goto (pose2) goto (pose5) goto (kitchen) access (coffee, machine) (coffee_machine)) machine) > turn turn_off off (coffee (coffee_machine) machine)
release (coffee_ (arffee__ggg____ > pickup scoofeeemmug goto (pose5) goto (pose2) access (wardrcbo2) > release (coffce_mus > done] Scene Graph Simulator: Plan Verified
50


Reviews:
[{"review_id": "q-rpNh2niJ", "reviewer": "Reviewer_ZPG6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents SayPlan: a method for high-level task planning in large-scale fully known building-like environments in which LLM-based action proposal is supported by and verified against a 3D scene graph representation of the state of the world. The central approach consists of two parts, each focused on expanding/contracting nodes in a compactified representation of the map until the LLM context includes all objects needed for planning and then using this information to plan. The approach outperforms competitive baselines and proves capable of accomplishing complex long-horizon tasks given only the LLM and the 3D scene graph.", "summary_of_recommendation": "[Restated from above.] The paper is very well written and the presented method is both well thought out and well executed. It is a clever idea that directly address a central limitation of LLM-based planning and is likely to spawn a new direction. The prose is clear. The paper on the whole is well organized and easy to follow. Results feel complete for this initial effort in this direction and raise interesting questions and ideas for future work. I have little in the way of suggestions to improve the paper.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "[Reproduced questions from the longer review above.]\n\n- It is mentioned that there is a limit of 5 iterative replanning steps. Is there a similar limit placed on semantic search? Relatedly, how would SayPlan handle being given a task that could not be accomplished in the given environment?\n- Curiosity Question (not essential): Is SayPlan currently able to handle object state, for example that a dishwasher is currently full and clean? Do nodes in the graph have state information other than their connectivity in the graph?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is very well written and the presented method is both well thought out and well executed. It is a clever idea that directly address a central limitation of LLM-based planning and is likely to spawn a new direction. The prose is clear. The paper on the whole is well organized and easy to follow. Results feel complete for this initial effort in this direction and raise interesting questions and ideas for future work. I have little in the way of suggestions to improve the paper.\n\nThe only \"core issue\" that I have with the paper is one that I do not entirely know how to solve: while the main body of the paper is self-contained and does a good job of describing the approach and presenting results statistics, individual instances of fully-executed plans as well as the back-and-forth dialogues with the LLM only appear in the appendix. While clearly this is too much information to include in the body of the paper, adding some elements of it would help build intuition useful for understanding the work.\n\nFor example, the discussion of Semantic Graph Search (Line 171) would benefit greatly from a snippet showing a single input/output from Appendix I. Even without much context, showing the form that \"memory\" and the 3d scene graph take for a *very* simple example would help readability. If only a very simple hand-crafted example were provided and included in a small figure on the right side of the page, hopefully it would take up only a small bit of space yet provide incredibly useful detail that Figure 1 lacks. This is similarly true of the Iterative Re-planning\n\nUltimately, the paper stands well enough on its own, but the impact of the paper will be enhanced if such details are readily available in the main body.\n\nOther comments and suggestions:\n- It is mentioned that there is a limit of 5 iterative replanning steps. Is there a similar limit placed on semantic search? Relatedly, how would SayPlan handle being given a task that could not be accomplished in the given environment?\n- The bottom margin of page 2 is off somehow, as the text is touching the page number.\n- Capitalization in the references is off for some entries: e.g., \"Ai Magizine\" in [26], \"factored mdps in [27]\". In BibTex, an extra set of {curly braces} is needed to preserve the capitalization of specific words. Please go through the references and address these changes.\n- Many of the left-side quotation marks are facing the wrong way. In LaTeX, two back-ticks (``) should be used to indicate left-quotes.\n- Curiosity Question: Is SayPlan currently able to handle object state, for example that a dishwasher is currently full and clean? Do nodes in the graph have state information other than their connectivity in the graph?\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "gYDL-wFeb49", "reviewer": "Reviewer_QfzX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to solve a problem of large language model (LLM) based task planning in a large-scale environment including multiple rooms and floors. The problem is hard in that the pre-encoding of all the necessary information within the LLMs may not be feasible. This paper proposes a scalable LLM-based large-scale task planning method with a 3D scene graph. The main contributions of this paper are as follows:\n-\ta semantic search of task-relevant subgraphs to encode the world model information to LLM\n-\tan iterative replanning to find feasible action plans\n", "summary_of_recommendation": "The proposed methodology is very interesting and provides the direction of using LLM with a large world model. I would like to recommend accepting this paper. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "-\tHow many nodes and features of a graph typical LLM can handle? How can you handle a given with too many features? \n-\tWhat is the scene graph simulator?\n-\tAlgorithm 1. 4-6: what is `node_id’? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths – The main strength of this paper is the pre-encoding idea of the large world model leveraging the 3D scene graph and its sub-graph identification. \nWeaknesses – This method cannot be scalable with more geometric or semantic attributes on each graph node or edge.\n\nThis paper needs to clarify and improve a few things.\n-\tThe major concern is the limitation of the sub-graph selection for LLM. The 3D scene graph can hold more information this paper considers. Node and edge features may include more spatial and/or semantic information depending on the capabilities of the detectors a robot has. Due to that, the discovered subgraph may still have too much information that the LLM cannot handle. Thus, it is necessary to clarify how many nodes and features of a graph typical LLM handles. \n-\tAnother issue is the lack of explanations. I understand the limited space makes it hard to include all the contents. However, it is really hard to understand how the semantic search work with LLM, though Appendix 1 helps. It will be great if the authors explain it with examples in the paper. \n-\tAlgorithm 1 must be improved. \n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "ErlLdbxYjBB", "reviewer": "Reviewer_xLeS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the challenge of scaling task planning for robots to large, multi-room environments using large language models (LLMs). The SayPlan approach incorporates 4 key ideas: (1) Representing environments as 3D scene graphs that capture semantic and spatial relationships between objects and locations (2) Allowing the LLM to explore and focus on task-relevant subgraphs to avoid exceeding token limits. (3) Using a classical path planner to complete navigation actions, reduces the LLM's planning horizon. (4) An iterative replanning process with a scene graph simulator to ensure plan feasibility, where feasibility feedback is given to the LLM to iteratively improve its plans. Experiments in large multi-floor environments demonstrate the approach scales effectively and produces more executable plans than vanilla LLM task planning.", "summary_of_recommendation": "The paper is relevant and well-executed. Additional clarifying experiments on environment scale and iterative planning iterations would make the paper stronger.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See feedback section", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n* This paper provides one of the first efforts to integrate 3D scene graphs with LLMs. 3DSGs are specified via a JSON-serialized format and given to the LLM. Leveraging LLMs to perform semantic search (finding task-relevant subgraphs to do search) appears novel, although integrating LLM-planning with low-level planners and feedback have been explored in prior works (e.g. InnerMonologue, as the authors cited).\n* The related works on LLM planners seem sufficient.\n* The methodology is technically sound overall. The experiments on two large environments with different instruction sets provide convincing evidence for the claims. The experiments especially highlight the importance of the iterative replanning process.\n* Analysis of token requirements shows approach scales to large environments.\n* Detailed analysis and breakdown of plan failure modes.\n* The paper is well-written and easy to follow. The approach is clearly explained through diagrams and examples. \n\nWeaknesses/Feedback\n1) More analysis could be provided on how the plan quality changes with increasing environment size. It might be good to see at which scale SayPlan starts to outperform the baselines, and at which scale SayPlan performance becomes inadequate. Since a big part of the paper is about proposing a planning method that scales, providing scaling experiments would make the paper a lot stronger.\n2) The only type of failure for SayPlan is hallucinating nodes that do not exist. The authors mention that additional iterative replanning iterations may improve this issue. I think providing ablations on plan performance vs. the number of iterations is a really important comparison. \n3) It will be interesting to see at how many iterations do the other types of failure modes start to disappear (maybe 1 is enough? Or 2? Or is it a smoother interpolation?), and to see if it’s possible to achieve 0% hallucinating nodes with SayPlan (maybe at 20 iterations? Maybe more?). This would be a super interesting experiment that would really allow readers to understand the extent of LLMs’ self-correcting capabilities.\n4) There are no comparisons to task planners that do not use LLMs. Since the scene graphs are given, are there really no other task-planning methods that can serve as a reasonable baseline?\n5) Currently there are not that many details on scene graph construction, as the authors treat these as given. Are there specific properties of the scene graph that are important for enabling LLM-based planning? Does the amount of semantic information (e.g. object descriptions) matter for SayPlan performance?\n6) The reliance on pre-built scene graphs that don't change is a major limitation, reducing applicability to dynamic or uncertain environments. The open-loop planning approach without real-time replanning based on perceptions is another drawback. The authors acknowledge both of these points, and understandably addressing both in the same paper is perhaps out of scope.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-bmuHERnav", "reviewer": "Reviewer_UUK5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method based on large language models and 3D scene graphs for a robot to solve long horizon tasks. The approach has two main components: First, the natural language instruction is used to guide a semantic search through the scene graph. Second, an iterative planning method finds feasible plans with a classical planner and then verifies their correctness in the scene graph. Their results, demonstrated on a real mobile manipulator in a fairly large scale envorment, show examples of successful task executions. ", "summary_of_recommendation": "Overall I find the concepts and demonstration in the work impressive (subject to getting satisfactory responses to the concerns I have raised). The paper is likely to have a significant impact in the field and therefore should be accepted. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See points 2 and 3 in the \"Doubts/Areas of Improvement\" above. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I find the work of high quality and the capabilities demonstrated are impressive. \n\nStrenghts:\n\n1. The paper demonstrates planning over large environments \n\n2. Their approach shows the ability to plan over relatively abstract and complex concepts\n\n3. I particularly like the way they have combined classical planning with verification of plan feasibility over the scene graph\n\nDoubts / Areas of improvement:\n\n1. I found the manuscript a bit difficult to follow and lacked detail and clarity where I would have wanted. This were clarified by digging through the appendices but I would suggest to move some things into the main manuscript I have a few suggestions for ways that it could be better:\n - Be more precise about what is the \"scene graph simulator\" and what the operations \"collapse\", \"expand\", and \"contract\". These terms are used very often and a key aspect of the approach but in my opinion never precisely defined. For example, the paragraph on \"Semantic Graph Search\" (L171 to L188) is insufficient to really understand the method. \n - I was confused during the reading of the manuscript about the \"prompt\" generation part, and what are the inputs and expected outputs from the LLM until I consulted the Appendices. This could be made more precise. \n- I found Fig. 1 very difficult to parse. It is more easily understandable on the website where it is animated but in the manuscript it overwhelmingly complex. \n- I think it could really help to have a running example to explain the various components. \n\n 2. I would like the authors to justify their choice of baselines. In particular, they have not compared against any non \"scene graph\" based approaches. For example \"Palm-SayCan\" (\"Do As I Can, Not As I Say: Grounding Language in Robotic Affordances\") or other reinforcement learning style baselines. It is also a bit problematic that the specific queries that are used for evaluation are hand-engineered. Perhaps it is a byproduct of the fact that this field is a bit new, but it seems like there is a need for some standardization of the evaluation baselines here. \n\n 3. The paper is clear and explicit about the assumption of having access to the ground truth 3D scene graph, but I could have used more clarity about what is assumed for robot skill execution. For example, for navigation, is there a full metric SLAM map that is used? How are the policies for manipulation trained? Etc.\n\n\nMinor comments:\n\n - \"feedback\" is not initialized in Algorithm 1\n - The caption for Table 2 confuses me because it refers to an \"average number of steps\" but the numbers in the table are all percentages which I don't know how to interpret. \n\nSome typos:\n\nL32: as well AS comprehend\nL151: Using 3D scene graph representation (no s)\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method that addresses the challenge of scaling task planning for robots to large, multi-room environments, using large language models (LLMs) in conjunction with 3D scene graphs. This is in part enabled by semantic search of task-relevant subgraphs to encode world model information to the LLM, and iterative re-planning with a classic planner in-the-loop to find feasible action plans.

The approach presents itself as an outstanding combination of both search (through structured representations) and learning (via LLMs) to address a challenging problem, with compelling real-world experiments on a mobile manipulator in large-scale environments.

Post-rebuttal, all reviewers unanimously agree on "Strong Accept." Excellent work!
Decision: Accept (Oral)