PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks
Authors: Matthew Chang
Gunjan Chhablani
Alexander Clegg
Mikael Dallaire Cote
Ruta Desai
Michal Hlavac
Vladimir Karashchuk
Jacob Krantz
Roozbeh Mottaghi
Priyam Parashar
Siddharth Patki
Ishita Prasad
Xavier Puig
Akshara Rai
Ram Ramrakhya
Daniel Tran
Joanne Truong
John M Turner
Eric Undersander
Tsung-Yen Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Human-Robot Collaboration
Planning
Embodied AI
Primary area: applications to robotics, autonomy, planning
TL;DR: Introducing a novel benchmark and in-depth analysis of top-performing planners for human-robot collaboration in household tasks.

Abstract: We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation-in-the-loop for the grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with 'real' humans, they require 1.5x as many steps as two humans collaborating and 1.1x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction.

Published as a conference paper at ICLR 2025
PARTNR: A BENCHMARK FOR PLANNING AND REA- SONING IN EMBODIED MULTI-AGENT TASKS
Matthew Chang, Gunjan Chhablani, Alexander Clegg, Mikael Dallaire Cote, Ruta Desai, Michal Hlavac, Vladimir Karashchuk, Jacob Krantz, Roozbeh Mottaghi, Priyam Parashar, Siddharth Patki, Ishita Prasad, Xavier Puig, Akshara Rai, Ram Ramrakhya, Daniel Tran, Joanne Truong, John M. Turner, Eric Undersander, Tsung- Yen Yang
Work done FAIR Meta, Alphabetical author order
https://aihhbitaa..om org p
ABSTRACT
We present a benchmark for Plamning And Reasoning Tasks in humaN-Robot collaboration (PARTNR) designed to study human-robot coordination in house- hold activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating comuulation-intte-llvee for the grounding and verification. PARTNR stands the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs PARTNR tasks, across the axes of planning, perception and skill exe- cution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with real humans, they require 1.5X as many steps as two humans col- laborating and 1.1x more steps than single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6X faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research this direction.
INTRODUCTION
Imagine a robot that collaborates with humans in daily activities, akin to human-to-human interactions. This scenario requires two key features: dynamic collaboration between the robot and the human, and the use of natural language for interaction. Current benchmarks in embodied AI satisfy
the other condition; either robots operate in isolation (Shridhar et al., 2020; Krantz et al., 2020; Majumdar specified in natural language (Yenamandra al., 2023; Puig et al., 2024; Szot et al., 2023). Despite significant progress in the field of embodied AI, there remains gap in realistic benchmarks that evaluate robots in collaborative settings. To bridge this gap, we introduce Planning And Reasoning Tasks in humaN- Robot collaboration (PARTNR), a novel benchmark that evaluates the ability of embodied Al agents to collaborate with humans across a range of household activities in simulated indoor environments (Figure I).
al., 2024), tasks
PARTNR consists of 100,000 natural language instructions paired with tailored evaluation functions, focusing on four task types: (1) constraint-free, where sub-tasks can be completed in any manner by either agent, (2) spatial tasks that contain spatial constraints, (3) temporal tasks that require ordered ex- ecution, and (4) heterogencous tasks that include actions that cannot be completed by of the agents. Beyond the conventional challenges of long-horizon planning, partially observed environments, and large state and action spaces, PARTNR emphasizes the need for effective collaboration.
Curating such a benchmark of harge-scale, natural language tasks with tailored evaluation functions presents significant challenges. Current benchmarks typically rely on either templated tasks (Shridhar et al., 2020; Zhang et al., 2024a) or tasks and evaluations crafted by humans (Li et al., 2023a), restricting the diversity scale of the datasets. Instead, we propose semi-automated approach using

---

Published as a conference paper at ICLR 2025
PARTNR: collaboration dataset for planning and reasoning multi-agent tasks
Automatic task and evaluation generation
Human-In-The-Loon
We collect collaboration data web.
Humai// Human
Task: Take the glass and bewl to the kitchen and wash them.
Human/ Agent
LLM-based helper agents
propose modular LLM-based agent baselines to collaborate in our benchmark
LLM Planner
Evaluation IN(kitchan, glass_ IN(kitchen, bowL IS_ CLEANGgless IS_ CLEANGb.LL
Scene Graph
Low-Level Skills
Figure I:
We present PARTNR, benchmark for planning and reasoning in embodied multi-agent tasks, featuring 100,000 everyday tasks and evaluation functions generated semi-automatically, spanning 60 houses and 5,819 unique objects. We analyze LLM-based planning agents and also provide a human-in- memman-in-thh--eee in-the-loop tool to evaluate how agents collaborate with real humans.
Large Language Models (LLMs) with simulation-in ientiiioo--i-teee--e un-the-loop grounding. First, an LLM generates task and evaluation functions, which are grounded in the objects and furniture of simulated house. Next, we employ simulation-aides filtering to remove hallucinations, complemented by human annotation to enhance diversity and accuracy. Subsequently, set of 1,000 verified instructions and evaluation functions are utilized to guide an LLM through in-context prompting to create 100,000 tasks.
As PARTNR consists of natural language tasks and LLMs have shown strong results in planning (Yao et al., 2023; Ahn et al., 2022; Huang et al., 2022), we explore prompting and fine-tuning LLMs, to assess their effectiveness in collaborative scenarios. We study the effect of observability of the environment (i.e., full or partial), centralized vs. decentralized multi-agent control, learned or privileged oracle robot skills, and different ways of grounding the 3D world information for L.M-- based planning. Beyond these automated evaluations with synthetic human partners, we also perform evaluations with real numans-in--te-loon partner, or with an LLM-guided robot partner. Overall, we find that LLMs struggle at coordination, task tracking and dealing with perception and skill errors. While humans are able to solve 93% of PARTNR tasks, SoTA LLMs can only successfully complete 30% under non non-privileged conditions. Moreover, in decentralized multi-agent settings, task completion takes 1.3x more steps than single- agent, due to poor tracking of partner actions, resulting in extraneous actions. In contrast, human pairs outperform single humans, in our human-in-the-loog experiments, highlighting potential for improving LLM collaboration strategies. LLMs also struggle to recover from skill failures and perception grounding errors, resulting in lower performance when privileged skills and privileged perception removed. When comparing model sizes, observe that smaller fine-tuned Llama3.1 1- 8B achieves a similar performance to a Llama3. 1-BB without finetuning, while being 8.6x faster. This faster inference plays an important role when evaluated with real samans-in-the-loop where the finetuned model takes fewer steps and offloads more tasks from the human.
with people performing the task alone, with human
In summary, PARTNR enables reproducible, large-scale, and systematic evaluations of embodied agents in wide variety of collaborative scenarios. Through systematic evaluation, we reveal critical insights into the current limitations of LLM-based planners, opening interesting future research directions. All code, datasets, and human demonstrations on PARTNR tasks will be open-sourced.
2 RELATED WORK
Language-basd benchmarks in Embodied AI. A large body of work on language benchmarks in Embodied AI has focused navigation (Anderson et al., 2018; Krantz et al., 2020; Chen et al., 2019)
2

---

Published as a conference paper at ICLR 2025
Environment
Multi-Agent
Language
Action Space
Task Types
Num tasks
Overcooked (Carroll ct al., 2019)
2D
V
HL
C
4
RoboGen (Wang et al., 2024)
3D
LL+HL
CST
106
GenSim (Katara ct al., 2023)
3D
LL
CS
100
RoCo (Mandi ct al., 2024)
3D
LL
CS
FurnMove (Jain et al., 2020)
3D-S
LT
RoboCasa (Nasiriany al., 2024)
3D-$
LL
CST
100
ALFRED (Shridhar et al., 2020)
3D-S
IIL
CST
25,743
BEHAVIOR-IK (Li et al., 2023a)
3D-M
LL+HL
CST
1,000
WAH (Puig al., 2021)
3D-M
HL
1,211
Co-ELA (Zhang al., 2024a)
3D-M
HL
PARTNR
3D-M
LL+HL
CSTH
100,000
Table I: Comparison to similar embodied benchmarks. We compare PARTNR to embodied AI benchmarks, focusing on natural language and multi-agent collaboration tasks. Comparison axes are Environment: Household single room (S), household multi-room (M). Action Space: High-Level Actions (HL), Low-level Actions (LL.). Task Types: Constraint-free (C), Spatial (S), Temporal (T), Heterogeneous (H) Num tasks: We measure tasks the number of unique scene-goal pairs.
or Embodied Question Answering (Das et al., 2018; Majumdar et al., 2024) which involve navigation and information gathering but do not require agents modify their environments. Closer to our work are instruction-folloiing benchmarks (Shridhar et al., 2020; 2021; Puig et al., 2018; Wang et al., 2024; James et al., 2020; Gong et al., 2023), where agents interact with environments complete tasks described via language, though the diversity of tasks is limited. In contrast, we leverage LLMs to generate diverse task definitions and scene initializations, and extend them to multi-agent settings. The idea of scaling up task generation using LLMs has been explored in few recent works (Katara et al., 2023; Wang et al., 2024; Xian et al., 2023; Nasiriany et al., 2024). However, these works tend to focus single-agent tasks that span relatively short horizons, while consider long-horize horizon, multi-agent problems. Table compares relevant benchmarks with PARTNR.
Embodied multi-agent benchmarks. Multiple works have proposed embodied multi-agent bench- marks (Puig et al., 2023; Agashe et al., 2023; Zhang et al., 2024a; Jain et al., 2019; Suarez et al., 2019). Many of these benchmarks focus coordination in simple 2D environments, limiting their applicability to real world settings (Agashe et al., 2023; Carroll et al., 2019). Recent works have developed benchmarks studying collaboration in more realistic environments and activities (Puig et al., 2021; Zhang et al., 2024a; Jain et al., 2019; Puig et al., 2024; Szot et al., 2023), focusing on rearranging objects or furniture in large, partially observable 3D environments (Puig et al., 2021; 2024; Jain et al., 2019; Szot et al., 2023), or manipulating objects in counter-top space (Mandi et al., 2024). However, these benchmarks are typically limited a predefined and reduced set of tasks, often not described in natural language and primarily involving object rearrangement. In contrast, PARTNR covers an open set of tasks, each described in natural language, requiring agents to rearrange objects with spatial and temporal constraints, as well as requiring heterogeneous actions that can only be done by the human agent, (e.g., washing dishes or turning on the oven).
LLMs for decision making. Several works use L.MSs as interactive policies, highlighting challenges in grounding them with observations and actions (Huang et al., 2022; Yao et al., 2023; Ahn et al., 2022; Huang al., 2023; Zeng al., 2022: Zheng et al., 2023; Guo et al., 2024; Liu et al., 2024). Some approaches improve grounding by prompting LLMs with demonstrations and task-specific constraints (Huang et al., 2022; Yao al., 2023), by integrating LLMs with external modules for multi-modal reasoning (Ahn et al., 2022: Huang et al., 2023; Zeng et al., 2022). Toolformer (Schick 2023) allows LLMs to call APIs for information retrieval or environmental interaction. For instance, APIS can be used to call low-level policies (Driess et al., 2023), to leverage VLMs for obtaining the state of the world (Huang al., 2023: Zhang et al., 2024b), or to another LLM serving as world model (Zhao et al., 2024). SayPlan (Rana et al., 2023) maintains persistent graph of the current world-state (Gu et al., 2024; Werby al., 2024), enabling detailed semantic and geometric queries. Our work synthesizes these ideas by encoding the environment into graph, using tools to extract relevant information, and executing tasks through motor skills, Another line of work fine-tunes LLMs with data from the target environments, by learning input and output adaptors (Li et al., 2022; Szot et al., 2024; Xiang et al., 2023). We explore low-rank adaptation of LLMs with multi-agent data to enhance coordination and efficiency. While fewer studies focus LLMs in multi-agent collaboration (Zhang et al., 2024a; Park et al., 2023; Li et al., 2023b; Zhou et al., 2024), one notable example is CoELA (Zhang al., 2024a) that design collaborative agents, though limited in task
3

---

Published as a conference paper at ICLR 2025
Sim-Aware Generation
Sin-In-The-Loop Filtering (1k)
Large-Scale Generation (100k)
Task Huuman-In-The-Loog Filtering
Taxk
ULM
Objject ClC
rnfruction
LLM
Ineruction
Cijject Cia
Exal Gereration
Human Annotation
Era Ganeration
rraivction
Heattatcc
Furkurtion Funetion
Exalustio Function
rolusion
Seed
Haumitaa
Exuicatio
Detail: Sim-Aware Generation
Object Library
Task Generator LLM
Tank Instructions
Evel Generator::
Evalustion Funition
Appla
iell me collect all toya in the house, Put then
LLM Heuristes
ThE sturfee
shel
draffed -ay
DV aturted tay A, abelt
Toy mircck
IS NEXT rocerurted moy d, hose oo NEXT Tocr_oooeeuuter roceruffed tay 1, bonk
Eag
House Contents
Object Configuneions bey
living Boot:
iloes
Evsluation
table
took UN atail
cheat
Zauk Eussus (272]
Kitchen:
Failure Explanation [ate]
cabilet
counte:
Figure 2: The PARTNR generation pipeline. Task and evaluation generators produce episodes, which are filtered and annotated for correctness. These episodes then treated seeds to achieve 100k-scale. Finally, episodes are vetted during human-in-the-loop collection.
diversity. Our work addresses a broader range tasks and considers agents with varying capabilities, pushing the boundaries of multi-agent system collaboration in more complex scenarios.
BENCHMARK GENERATION
Task Instruction
We introduce PARTNR, bench- mark aimed at training and evalu- ating robots solving natural lan- guage tasks in collaboration with hu- mans. PARTNR covers four types of tasks: (1) Constraint-free tasks, where sub-tasks can be completed in any manner by either agent. For exam- ple, "Let's move all dirty plates the sink." " (2) Spatial tasks that require soning about the spatial positioning of objects. For instance, "Let's place the books on the shelf next to each other." (3) Temporal tasks, where the sequence in which sub-tasks are executed is important. For example, "Let's remove the candles from the dining table before bringing the plates to the table." (4) Heterogeneous tasks, involving actions that are beyond the robot's capabilities. For example, "Let's wash the dishes before putting them in shelves." " In scenarios where the robot's s skills do not support washing, completing this task requires reasoning about agent capabilities. Our benchmark consists of natural language instructions and corresponding evaluation functions, both of which are generated at-seale using LLMs. Specifically, we generate 1,000 human- verified instructions and corresponding evaluation functions and use them in-prompt examples to scale to 100,000 tasks other scenes with different layouts and objects. A unique aspect of our automatic generation is the integration of an embodied simulator within the generation loop, which significantly reduces LLM errors such as hallucinations and infeasible actions.
"Let's tidy up the family room. The toys go in the toy box. After that, set the plants the shelf next each other."
Evaluation Function
Propositions:
is_inside([""on (atiiide("tt__ffeee
_""], ["toy_box box__"1x ("toy_bax_e"]
is is_inside( (oriid(([tty]]] ["toy__ food_e")
is_on_tap(( tattttpp"pplmmm ["plant_ee [a]] ["sheef_ee 8*, "shelf "s"elf_1"l)
"s_on_top([ ("pptpppplaat__]] ["plant_1"], ["shelf_B" [shelf_1")
is is_next_tas (at"tttoapppa""""""" ["plant_a"]]
Varly "sonox" when
Dependencies:
the plonts are ploced
whilesatisfied((4), depends_ (peends_on=[,,3]
The loys most be mecronged fire
Constraints:
TemporalConstraint,
[(0,2), (1,2), (0,3), (1,3), (0,4), (2,4))
AU plocomenss
Tenntallssisacctooorrrraaa......
Figure Task and evaluation example. Language tasks have inherent complexity and ambiguity; both of which are supported by the structures our evaluation functions.
3.1 SIMULATION- -HE- TASK INSTRUCTION GENERATION
While LLM-based task generation has been studied in literature before (Katara et al., 2023; Wang al., 2024; Xian al., 2023; Nasiriany al., 2024), these generations not grounded beyond
4

---

Published as a conference paper at ICLR 2025
user-created in-context prompts. In PARTNR, to ground the LLM in the environment, agents and available actions. Specifically, we instantiate a simulation environment in the Habitat 3.0 simulator (Puig al.. 2024), populated with the HSSD dataset (Khanna et al., 2024), consisting of 60 unique houses and 5,819 OVMM objects (Yenamandra al., 2023). The simulated house is parsed into list of rooms and available furniture, and passed to an LLM, along with all available objects. Using this information, the LLM is asked to generate free-form, viable tasks in the scene, along with an initial scene state description. For example, if the generated task is "Clear dishes from the living room". ", the LLM should generate an initial scene with multiple dishes in the living room. At this stage, additional objects are also added the scene to create clutter in the environment. Once generated, the tasks, initial states, and clutter are instantiated in the simulator, and infeasible instructions are filtered. For example, if the house does not have a living room, "Clear dishes from the living room" is invalid. Similarly, if the generated task requires actions not supported by the simulator, such as folding, the task is filtered. Generally, the rate of hallucinations is high, leading to a significant number of episodes being discarded. We observe that after filtering for infeasible instructions, the diversity in generated instructions is typically limited. For example, most of the instructions dining room). To increase diversity of the generated tasks, we manually annotate them to ensure task and object diversity, such as maintaining a balanced distribution of constraint-fres spatial, temporal, and heterogeneous tasks by modifying the instructions to elicit specific characteristics. This process results in 1,000 human annotated and simulation- verified tasks (Appendix A.5.2).
simulation-in-the- generation technique
the same objects (e.g., dishes) similar rooms (e.g., kitchen or
Such manual annotation is not practi- cal for large-scale generation. Instead, we leverage the annant-nd 1,000 instructions to scale generation by using them as in-prompt examples. We prompt the LLM with both a house description and an example task, and instruct it to modify the task to fit the new house. For example task like "Clear all dishes from the living room" is modified to "Clear all toys from the bedroom." "This allows us to maintain the diversity of the original annotated instruction set, while ensur-
Epicodes 199
15
Episodes (%) 50 40 30 20 10
Constraint-Free
Spatial
Temporal
Heterogeneous
Figure 4: Distribution of task types in PARTNR. The left plot displays the percentage of tasks with each characteristic. Constraint-free tasks by definition exclude the other types. The top right bars correspond the dot combination below.
ing a high likelihood of successful instantiation in the simulator. Qualitatively, we filtered or edited ~ -90% of free-form generated instructions and only ~ 10% of scaled instructions. We use l./ama3- 70B-Instruct (Dubey et al., 2024) for all instruction generation. Finally, all tasks go through a cumaan-n--the-lowe filtering. In this step, humans attempt to solve the tasks using our human-in-ther loop tool (Appendix 12) and climinate physically infeasible instructions that are difficult to detect, such as requiring object to be at two locations. Figure 2 provides an overview of our pipeline. Details on the generation process can be found in Appendix A.3 and prompts in Appendix A.13.
3.2 EVALUATION FUNCTION GENERATION
To determine if an agent successfully completed the instruction "Clear all dishes from the living room", we need an evaluation function that can validate the removal of all spoons, forks, and other dishes from any of the living rooms. However, manually annotating all necessary rearrangements and state changes of task is time intensive and since all tasks are unique, impractical at scale. Similar to instruction generation, we employ an LLM to create an evaluation function that assesses task completion without requiring any manual annotations. Specifically, we leverage the ability of LLMs to generate predicate- based Python programs using three types of APIs: a list of propositions indicating what relations between entities must be satisfied, a set of dependencies indicating when propositions should be queried, and a set of constraints indicating how propositions must be satisfied. We define an expressive vocabulary of each of these components to afford evaluation of all tasks in the benchmark (e.g., Figure 3). Closely related evaluation systems include defining tasks in PDDL (Ghallab et al., 1998) or BDDL (Srivastava et al., 2022). We choose to build a new Python-based evaluation system since neither have the expressivity to evaluate PARTNR tasks while maintaining human and LLM interpretability; for instance, BDDL does not support time-varying evaluation. Since PARTNR tasks have temporal dependencies (e.g. multi-step rearrangement), the input the evaluation function is
5

---

Published as a conference paper at ICLR 2025
the complete sequence of simulator states during task execution. The evaluation function returns
metrics: (1) Percent Complete (PC E [0, 1)), the % of propositions that are satisfied w.r.t. constraints, (2) Success (S E {True, False)), measuring if a task was successfully completed, defined as S == (PC = 1), and (3) Failure Explanation (FE), human and LLM interpretable language description of why the agents failed accomplish the task. See Appendix A.4 for details.
We use CodeL Maaaaa-7O--isssss inssticttttttt (Roziere et al., 2023) for evaluation function generation. Exem- plified in Figure 3, producing perfect evaluation functions is non-trivial. The LLM must correctly classify the entire space of possible actions against natural language instructions and the specific simulation environment, which can be quite complex. For example, Figure 3, the instruction "ser the plants on the shelf" refers to "the shelf", but two shelves exist in the room. The evaluation function must allow either shelf while requiring placement of all plants, and finally account for next-to relation. Any error or missing value in either proposition or constraint invalidates the evaluation function. Consequently, we observe large error rate in LLM generation, particularly pertaining to incorrect propositions and temporal sequencing constraints.
To alleviate these inaccuracies, we follow a similar semi-automated procedure to instruction genera- tion. We first generate evaluation functions for the 1,000 human-annotated instructions and perform manual annotation to correct them (Appendix A.5.3). This results in a dataset of 1,000 human-verified instruction and evaluation pairs. Next, we generate evaluations for the scaled 100,000 instruction set. Recall that the scaled instructions are generated by prompting the LLM with an example instruction from the annotated set. We retrieve the corresponding annotated evaluation function and prompt the LLM with it. This is similar to approaches such as retrieval-augmenti and improves the accuracy of evaluation function generation from 50% to 92% as measured through manual inspection (Appendix A.5.2). As a final step, we ask human users to solve all PARTNR evaluation tool (Appendix 1.) All tasks that cannot be solved
generation (Lewis et al., 2020)
tasks using our coman-in--th-laae by humans over 6 retries (3 single-user user, 3 multi-user tries) are deemed infeasible, and removed from the dataset. We find that about 90% of instructions, and 92% of evaluation functions from automated generation are accurate, resulting in combined generation accuracy of 90 x 92 83% from automated generation, and 98% after human annotation and filtering.
3.3 THE PARTNR DATASET
The PARTNR dataset comprises of 100,000 episodes in 37 train scenes, 1,000 episodes in 13 validation scenes, and 1,000 episodes in 10 test scenes from the HSSD dataset (Khanna et al., 2024). After scaled generation, all validation and test set episodes are human annotated for correctness, as well as 2,000-episode subset of train. See Appendix A. for correctness analysis of scale-generated episodes. Below, we analyze the characteristics and diversity of this dataset.
Characteristies: As described earlier, PARTNR focuses four task types: constraint-free spatial, temporal, and heterogeneous. We show the distribution of these task types in the test split in Figure 4; validation split is similar. PARTNR evaluates collaboration along these axes both independently and jointly. Secondary characteristics of interest include dependent rearranges (e.g.. "Place them on the same table") and multi-step rearrangement of the same object (e.g. "Move the cup to the sink, wash it, then place it in the cabinet"). ). 7% of tasks include dependent rearranges and 6% include multi-step rearrangement. Tasks average 4.7 propositions be satisfied (indicative of number of steps required to complete tasks). For analysis of linguistic phenomena and more characteristics, see Appendix A.2.
Diversity:
The diversity of tasks in PARTNR is largely enabled by simulation- in in-the-loop generation, which utilizes rich HSSD scenes, and the OVMM object set. Consequently, PARTNR tasks reference and require reasoning about 155 unique object types, 20 furniture classes and 13 room types. Note that each instruction, instantiated in each house, brings its own diversity. For example, "move the laptop to the office table", grounds office and table uniquely in each house, as well as different instances of laptop in diffeerntt instructions, Further discussion can be found in Appendix A.2.
4 EXPERIMENTS AND ANALYSIS
We investigate how state-of-the planning and perception methods handle natural language tasks in new environments and coordinate with unseen partners using PARTNR. Since PARTNR consists of diverse temiooall tasks specified in language, we primarily use LLMs in our baselines for planning, and study variants in (1) zero-shot prompting, retrieval-augmennes generation or fine- ine-tuning,

---

Published as a conference paper at ICLR 2025
(2) centralized versus decentralized planning. (3) partially versus fully observed environment, (4) learned versus oracle low-level robot skills, and (5) privileged versus non non-privileged perception.
Our experiments are conducted in the Habitat 3.0 simulator (Puig et al., 2024) with simulated Spot robot (BostonDynamics; Yokoyama et al., 2023). We adopt a two-layer hierarchical control architecture, similar to (Puig et al., 2024; Szot et al., 2021), as illustrated in Figure 5, for the robot and simulated human. At the high level, a planner selects skills from a predefined skill library (e.g.. navigate, pick. place, open, close). We also use a textual world graph with three-layer hierarchy representing rooms, furniture, and movable objects. Each node in the graph stores a semantic category (e.g., kitchen, table or cup), 3D information (e.g., position or bounding box), and states (e.g., clean, powered on). See Appendix A.6 and Figure 10 for details.
4.1 BASELINES
We evaluate baselines along the following axes:
1. Variations of high-level planner:
Heuristic expert: This approach utilizes expert-designed heuristics and privileged information about the task, environment and evaluation function to pre-plan all steps for human and robot based on their capabilities. For instance, both agents might rearrange objects, but only humans perform cleaning, filling, and toggling on/off tasks.
Zero-shot ReAct (ReAct): We use ReAct (Y (Yao et al., 2023) with an API library of functions or tools that enable the LLM to take actions. As observation, we provide the LLM with concise, current world graph description plus history of actions. The LLM uses this in- formation choose an action from [ExploreRoom, Navigate, OpenFurnite ure, CloseFurniture turee,
PickŌbject, PlaceOb ject, Wait, Done] for the robot. See Appendix A.15 for prompts and Appendix A.8 for API details (human and robot).
ReAct with Aarrival-Augmented Generation (ReAct-RAG): We also evaluate ReAct with RAG (Lewis et al., 2020) to investigate whether examples of planning on similar tasks improves the performance of ReAct. We construct a database of planning examples by collecting the successful traces from ReAct from the 2,000 task training subset (see 3.3).
During test time, the most relevant planning trace from the train dataset is selected based on sentence similarity and added to the LLM's s prompt (Pang et al., 2024; Madaan et al., 2024).
Finetuned LLMs (Finetuned)) We also investigate finetuning a smaller LLM (Llama3. as our high-level planner, using successful traces from the ReAct baselines (Hsieh et al., 2023) that use Llama3. 1-OB. Using the React-RAC dataset, we split every episode into a sequence of high-level planning actions, filtering for only actions that were executed successfully. For every action, we build an input containing the warld-graph and history of actions, similar to ReAct (see Appendix A.9 for more details). We then finetune an I.I.M to predict the action from the ReAct episode given this input, using a low-rank adapter (Hu et al., 2021). This model has reduced latency and computational demands, suitable for real world deployment.
All model generations are constrained to only output valid actions on observed objects using constrained generation (Geng et al., 2023). The constrained generation greatly reduces the halluci- nations and 'grammatical' errors typical of LLMs. An episode is finished when both agents call Done reach maximum simulation steps or LLM calls. Refer to Appendix A.8 for details.
2. Centralized versus decentralized planning: To study the overhead of coordination in multi-agent PARTNR tasks, we compare centralized and decentralized planners. In centralized, single LLM decides actions for both agents, with complete information about both agent's states, effectively removing any need for coordination between the agents. In decentralized, each agent is controlled by a different LLM, and each LLM needs to reason about the other agent's 's actions.
Partial versus full observability: To evaluate if SoTA language models can explore new environments and identify task-relevan objects, we consider a partially observed setting where the planner knows the house's 's layout but not the object locations, requiring exploration. This is in contrast to fully observed setting, where all object locations are known in advance.
Learned versus oracle low-level robot skills: We examine the impact of learned neural-network skills versus oracle skills (with privileged simulation information) PARTNR tasks. We create library of learned skills for pick, place, navigate, open and close actions (Appendix A.7 provides more details) and compare performance with oracle skills.
overall performance in
5. Privileged versus nri-privileged perception: To study perception challenges such as inaccurate detection and approximate localization, we used non-privileged world graph with modified
7

---

Published as a conference paper at ICLR 2025
ConceptGraphs (Gu et al., 2024), built from agents' RGBD observations only. As agents explore and take actions, this world graph is updated using onboard sensing (details in Appendix A.6). In contrast, with privileged perception, this information is available from the simulation.
Sim Env Habitat 3.0
4.2 RESULTS AND ANALYSIS
Control Signal
Control Signal
Metries. We evaluate performance across different settings using four key metrics. First, the simula- tion steps metric measures the number of steps required for agents to complete the task within the simulation environment, serving an indica- tor of efficiency. Second, the success rate reflects the completion of the task i.e. whether 'all' task constraints are satisfied. Given the complexity and long-hori long-horizon nature of PARTNR tasks, agents often partially complete the task. To account for this, we also report percent complete, which quantifies the ratio of completed task 'propositions' (percent com- plete = for successful tasks). Lastly, we assess the reasoning efficiency of the planners through the planning cycles metric, which counts the number of high-level LLM calls each planner makes in the course of an episode. We cap the maximum planner calls 50 in all experiments.
Low-L-evel Siil
Low-LLvel Skil
Human Obs
Robot Obs
Low- Level
Porcaption
Lou-Lavel
Skill
World Graph
High-Level Polkyy (LLM)
High-Level Pulicy (LLM)
Deseription
Figure 5: Decentralized architecture. The hu- man and robot agents use a 2-layer hierarchical architecture, with high-level LLM planners that call low-level skills. Both agents build world graph, updated using observations and actions.
4.2.1 TASK PERFORMANCE ANALYSIS
Table 2 presents comprehensive evaluation of the planning approaches defined in Section 4.1 using the Llama3.1 1iimn..--70 1-70B-Instrue model (Dubey et al., 2024) for ReAct baselines, and a finetuned Llama3. 1- 8B base model for the Finetuned baseline. Since PARTNR tasks are multi-agent, we also need a simulated human partner, which we control with ReAct approach, using Llama3. 1-70B-Instruue Our main findings are detailed below.
PARTNR tasks are challenging for LLM-based planners. ability and controllability conditions perform worse than the privileged heuristic expert, due to errors in task tracking (not completing all steps, performing them in the wrong order, or undoing completed steps), and semantic errors (placing objects on the wrong furniture, or moving the wrong object), indicating gap in LLM task planning.
baselines across all observ-
LLMs struggle with coordination in decentralized settings. Decentralized ReAct baselines which do not have privileged access to partner's intent are significantly slower at task completion than centralized ReAct (3295 steps with decentralized-parte partial in row(d)). This shows that reasoning about the partner e.g., knowing inferring partner's intent can improve task efficiency in PARTNR tasks, and current SoTA LLMS perform poorly at this. Moreover, decentralized ReAct with two agents is even slower than ReAct with single-agent (3295 steps with multi-agen in row(c) versus 2519 with single-agent in row(.)) indicating that LLMs suffer from significant coordination "burden". This co-ordination burden is further highlighted in our analysis on extraneous effort in Section 4.2.2, where we find that agents end up repeating parts of the task or performing irrelevant actions with much higher frequency in decentralized settings.
in row(e) versus 2298 with centralized-
LLMs are unable to recover from learned skill failures. When replacing oracle skills with learned skills, the success rate decreases from 0.73 to 0.57 (row(e) vs. row (h)). This decline can be attributed to the higher failure rate and increased number of simulation steps required by learned skills compared to privileged oracle skills. The L.MSs struggle to recover from skill errors like failing to pick up an object performing incomplete exploration, resulting in lower success rate. Future research could investigate training large models with low-level skills in the loop, enabling them to learn recovery and replanning strategies in the face of such failures.
LLMs exhibit a high degree of sensitivity to errors in perception. When we replace privileged perception with off-the-shelf perception modules, success rate significantly declines (from 0.57 with a privileged, partial world graph in row(h) to 0.30 with Sancept-Graphe (Gu et al., 2024) in row(i))
L.LMs rely heavily accurate world descriptions provided by the world graph and struggle correct.
8

---

Published as a conference paper at ICLR 2025
Method
Controllability
Skills
Observability
Sim Steps
Success Rate 1
Percent Complete 1
Planning Cycles 1
ReAct- ReAct-Single
Single Agent
Oracle
Partial
2519.02
0.73 =
0.85 0.0
18.68
(b) Heuristic- Hecristic-Expert
Centralized
Oracle
Full
1260.88
0.84
0,94 0.i
N/A
(c) ReAct
Centrulized
Oracle
Full
1347.43
33.50
0.74
0.88 001
17.49 0.33
(d) ReAct
Centrulized
Oracle
Partial
2298. 13 61.39
0.74 000
0.85 uot
20.73 1:051
(e) ReAct
Decentralized
Oracle
Partial
3295.20
76.27
0.73 0:00
0.86 0.0
15.24 0:331
ReAct + RAG
Decentralized
Oracle
Partial
3467.47
0.71 0.00
0.84 0.00
14.75 +:031
(g) Finetuned
Decentralized
Oracle
Partial
3228.96
75.34
0.70 0:00
0.84 not
12.85 0.24
(h) ReAct
Decentralized
Learned
Partial
6494.88 # 181.52
0.57 = 0.02
0.76 0.01
22.72 #: 2.0.58
(i) ReAct
Decentralized
Learned
ConceptGruph
12490.80 205.90
0.30 = 0.01
0,56 0.01
23.84 4
Table 2: Analysis of planner baselines in various settings. We compare performance using simula- tion steps, success rate and percent complete on the tasks, and the average number of planning cycles used by the baselines (described in Section 3).
errors such as misclassification (e.g.. shelves misidentified as tables) or incorrect room assignments (e.g., table in the living room mislabeled as being in the bedroom). Multi-modal models like VLMs might be stronger at recovering from such failures, which we leave for future work.
Finetuned 8B model performs par with ReAct with 70B model, while being 8.6x faster. We find that the finetuned planner with a small 8B model performs on par with ReAct, which uses much larger 70B model (a 0.73 success rate with the 70B model in row(e), versus 0.70 with the finetuned 8B model in row(g)). At the same time, we find that the finetuned model is 8.6 times faster inference. This indicates that the finetuning effectively distills task-relevant information from the training set and generalizes to new test tasks. When deployed with humans-in numans-in-the-lowe tho--hee..ppp the finetuned model takes fewer steps and offloads more sub-tasks than the 70B model (see Table 3).
4.2.2 ANALYSIS OF COLLABORATIVE BEHAVIOR AND EFFICIENCY
Our analysis in Table 2 revealed challenges in LLM collaboration, prompting deeper investigation into specific collaborative behaviors, explained below and detailed in Appendix A. and Table 12.
Robots offload up 60% of tasks. We evaluate the robot's ability to offload tasks from the human, measuring the ratio of sub-tasks performed by the robot to the total sub-tasks in successful PARTNR tasks. Despite similar success rates between single- and multi-agent (0.73 vs. 0.74), the robot offloads about 60% of sub-tasks in decentralized multi-agent, reducing human effort (Table 12).
Decentralized agents are prone to performing extraneous tasks. The agents sometimes end up performing sub-tasks that are not useful for the task such as rearranging an object that is not required by the task or repeating a sub-task already performed by the other agent. To capture such extraneous agent effort, we measure the portion of agent actions that did not increase the percent complete metric i.e., did not contribute to task progress, over the total number of successful agent actions an episode. We find a 300% increase in extraneous effort in decentralized multi-agent settings compared to single-agent (Table 12). indicating a significant coordination burden.
Temporal and heterogeneous tasks are challenging for LLMs. LLMs struggle in temporal and heterogeneous tasks. Task success drops by 27% for temporal tasks and 20% for heterogeneous tasks compared to constraint-fre free tasks for ReAct (Table 13). This highlights the limitations of LLMs in reasoning about agent capabilities and following strict ordering constraints.
4.3 HUMAN-IN-THE-LE EVALUATION
We build on the comaan-n-tthellwee infrastructure from Habitat 3.0 (Puig et al., 2024) and adapt it to a server-client architecture, with the server hosted on AWS capable of supporting multiple clients (see Appendix 12).). This allows us to run at-scale evaluation of our tasks with 129 non-expert human participants. We collect single-user and multi-user data on 1000 tasks from the validation and test set using this tool. In the single-user setting, a single participant completes the whole task, by driving the human agent in the simulator via keyboard/mouse controls Figure 14 in appendix shows our HITL interface. In multi-user, two participants complete the task together by controlling a human and a robot agent. The goal of these experiments is to study multi-user dynamics at PARTNR tasks, and see if multiple humans collaborating are more efficient than single human. Finally, we run human-Al experiment where human participant collaborates with robot controlled by an

---

Published as a conference paper at ICLR 2025
Success
Percent Complete
Sim
Task Offloading f
Exploration Efficiency
Extraneous Effort +
Method
Rate f
Steps
Single-user
0.93 +: 0.01
0.96 0.00
3046.99 +: 80.79
N/A
2459.22 + 26,75
0.09 +: 0.01
Multi-user
0.93 + 0.01
0.96 0.00
2369.55 + 49.35
0.59 + 0.01
1762.47 + 13.99
0.15 + 0.01
Sluman-ReAc
0.91 +: 0.01
0.96 0.02
4267.71 +: 83.40
0.16 +: 0.01
2624.39 +: 26.05
0.12 + 0.01
Hluman-Finetuned
0.92 4: 0.01
0.96 0.00
3443.33 4: 61.46
0.26 4: 0.01
2164.94 à: 21.31
0.13 4 0.01
Table 3: Human-in-the-Lae in-the-Loop Evaluation. We evaluate the performance of a 2-person human team
and Muman-LLM, teams, comparing them to solo human performance on PARTNR tasks using metrics described in Section 4. Additional results and analysis in Appendix A.
LLM (using the ReAct and Finetuned models from Section 4.1). This experiment aims evaluate I.LM-controlled agents at collaborating with unseen, real humans. Table 3 shows the success rate (SR) and percent complete (PC) of tasks from the validation set a single-user, multi-user, human-ReAct setting. Additionally, to complete the task, and the ratio of work completed by the robot i.c., task offloading. We also measure exploration efficiency in human-in-the-lonn tlop, by measuring the steps taken to pick the first object, and extraneous effort, indicating actions that were not useful for task completion. These results summarized in Table 3. Some key findings are below (more results and analysis in A.12.4):
and
measure the number of steps taken by each approach
Humans are significantly better than LLMs at PARTNR tasks. In both single and multiple human environments, the success rate achieved is 0.93 on the PARTNR benchmark. In contrast, the ReAct model without any privileged information, achieves a significantly lower success rate of 0.30 (row (i) of Table 2). This highlights significant gap in the performance of LLMs in planning tasks. Note that the LLM baselines like ReAct and Finetuned achieve a success rate of 0.92 and 0.91 when evaluated with real humans (Table 3), because humans are able to adapt to LLM mistakes. On the other hand, the simulated human in Table 2 is an LLM, which is unable to recover from partner mistakes.
Finetuned LLMs perform better than ReAct when coordinating with real humans. When deployed with real humans-in-the-lone the finetuned model is faster than ReAct at task completion (3443 steps with finetuned versus 4267 with ReAct). is also able to offload more tasks from humans than ReAct (26% with finetuned as compared to 16% with ReAct). This reflects that smaller models with faster inference can improve human experience in real-world deployment.
LLMs struggle at coordination, hampering human performance. Despite the Finetuned being faster than ReAct when collaborating with humans, both approaches are slower than the human doing the task alone. In contrast, two humans working together complete the task faster than single human (2369 steps vs. 3046 with multi- and single-user respectively). This result is in line with the automated evaluation we observed in Table 1, where multi-agent L.LMs are also slower than single-agent LLM. This result further reinforces that LLMs suffer at coordination; while humans are able to coordinate and divide tasks between each other, decentralized LLMs are unable do so.
LLMs are able to offload tasks from humans. Despite the aforementioned increase in the number of steps for task completion, robots guided by the finetuned model successfully offload 26% of tasks from humans. This indicates that LLMs can still offer assistance when collaborating with real human partners. Nonetheless, there remains significant potential for improvement.
CONCLUSION
We present PARTNR, a benchmark for reasoning and planning in multi-agent embodied tasks, featuring 100,000 natural language tasks instantiated in 60 simulated, multi-room houses with 5,819 unique objects. We use a semi-automated -LM--pee pipeline for large-scale instruction and evaluation function generation that uses wwwuulation-i-the-llee characteristies of everyday tasks, such as temporal and spatial constraints, and allows systematic evaluation of planning approaches. We find significant gap between SoTA LLMs and human-level performance at P PARTNR tasks. While our best L.M baseline only succeeds at 30% of tasks with no privileged information, humans are able to solve 93% of the tasks. Moreover, LLMs face challenges in coordinating with both LLM-based agents and real human partners. Human-in-the-lod involving real humans collaborating with LLM- LLM-gddded robots, reveal that LLM-guided partners decrease human efficiency compared to working solo. This suggests that LLM-d agents require significant improvements to become effective collaborative partners in embodied tasks. PARTNR serves challenging benchmark that highlights the substantial limitations of current models.
grounding. PARTNR exhibits
evaluations,
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Saaket Agashe, Yue Fan, Anthony Reyna, and Xin Eric Wang. Evaluating multi-agent coordination abilities in large language models. arXiv, 2023. 3
Christopher Agia, Krishna Murthy Jatavallabhula, Mohamed Khodeir, Ondrej Miksik, Vibhav Vineet, Mustafa Mukadam, Liam Paull, and Florian Shkurti. Taskography: Evaluating robot task planning over large 3d scene graphs. In CoRL, 2022. 28
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffirey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jomell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can and not as say: Grounding language in robotic affordances. In CoRL, 2022. 2,3
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-language navigation: Interpreting visually- grounded navigation instructions in real environments. In CVPR, 2018.
BostonDYnamicss. Spot robot. Accessed: 2024-10-01. 7
wtt.cooo//////. com/pproducts/spot.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. In NeurlPS, 2019.
Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In CVPR, 2019. 2, 16
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied
question answering. In CVPR, 2018. 3
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. In /CMI., 2023. 3
Abhimanyu Dubey, Abhinav Jauhri, Abhinay Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv, 2024. 5, 8
Saibo Geng, Martin Josifoski, Maxime Peyrard, and Robert West. Grammar-constrainee decoding for structured NLP tasks without finetuning. In EMNLP, 2023. 7,32
Malik Ghallab, Craig Knoblock, David Wilkins, Anthony Barrett, Dave Christianson, Mare Friedman, Chung Kwok, Keith Golden, Scott Penberthy, David Smith, Ying Sun, and Daniel Weld. Pddl the planning domain definition language. Technical Report, 1998. 5
Ran Gong, Jiangyong Huang, Yizhou Zhao, Haoran Geng, Xiaofeng Gao, Qingyang Wu, Wensi Ai, Ziheng Zhou, Demetri Terzopoulos, Song- Chun Zhu, al. Amoll: A benchmark for language- grounded task learning with continuous states in realistic 3d scenes. In /CCV, 2023. 3
Qiao Gu, Alihusein Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, Chuang Gan, Celso Miguel de Melo, Joshua B. Tenenbaum, Antonio Torralba, Florian Shkurti, and Liam Paull. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In ICRA, 2024. 3, 8, 28
Yanjiang Guo, Yen-Jen Wang, Lihan Zha, and Jianyu Chen. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment, 2024. URL ht //axxiv. org/ abs/ / 2307 00329. 3
Cheng- Yu Hsieh, Chun-Liang Li, Khan-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv, 2023. 7
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2021. 7, 34
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICML, 2022. 2,
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In CoRL, 2023.
Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexan- der G. Schwing, and Aniruddha Kembhavi. Two body problem: Collaborative visual task comple- tion. In CVPR, 2019, first two authors contributed equally. 3
Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander G. Schwing. A cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In ECCV, 2020.
Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020. 3
Pushkal Katara, Zhou Xian, and Katerina Fragkiadaki. Gen2sim: Scaling up robot learning in simulation with generative models. arXiv, 2023. 3,
Mukul Khanna, Yongsen Mao, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In CVPR, 2024.
15, 19
Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Vision-and-language navigation in continuous environments. ECCV, 2020. 1,
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. Room-across-rown
Multilingual vision-and-land and-language navigation with dense spatiotemporal grounding. In EMNLP, 2020. 16
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tan Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nip tasks. In NeurIPS, 2020. 6, 7
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine. Michael Lingelbach, Jiankai Sun, et al. Behavior- 1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. In CoRI, 2023a. 1,3
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for "mind" exploration of large language model society. In NeurlPS, 2023b.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang. Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-making. In NeurlPS, 2022.
Xinzhu Liu, Di Guo, Xinyu Zhang, and Huaping Liu. Heterogemeous embodied multi-agent collabo- ration. IEEE Robotics and Automation Letters, 96):53777-3884 2024. doi: 10. 109/LRA.2024, 3390588.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with feef-feed In NewrlPS, 2024. 7, 33
12

---

Published as a conference paper at ICLR 2025
Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Movay, Oleksandr Maksymets, Sergio Amaud, et al. Openeqa: Embodied question answering in the of foundation models. In CVPR, 2024. 1,3
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large
language models. In ICRA, 2024. 3
Soroush Nasiriany, Abhiram Maddukuri. Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots. In RSS, 2024. 3,
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv, 2024. 7, 33
Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. Generative agents: Interactive simulacra of human behavior. In ACM Symposium User Interface Software and Technology. 2023. 3
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 8018.2. 3
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-(ai) collaboration. In ICLR, 2021.
Xavier Puig, Tianmin Shu, Joshua B Tenenbaum, and Antonio Torralba. Nopa: Neurally-guided online probabilistic assistance for building socially intelligent home assistants. In ICRA, 2023. 3
Xavier Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Tsung-Yen Yang. Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, Vladimír Vondruš, Theophile Gervet, Vincent-Pierre Berges, John M. Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A co-habitat for humans, avatars and robots. In ICLR, 2024. 1, 3, 5, 7, 9, 15, 19, 30, 37
Team PyTorch. Accelerating generative ai with pytorch ii: Gpt, fast. httpp::/ppytorch.oogg plog//aceeeerattngg-eeeeeeeee
ai-2/, 2023. 31
Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf. Sayplan: Grounding large language models using 3d scene graphs for scalable robot task planning. In CoRL, 2023. 3, 28
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks.
In EMNLP, 2019. 33
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code. arXiv,, 2023. 2.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In NeurlPS, 2023. 3
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In CVPR, 2020. 3,
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In ICLR, 2021. 3
Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martin-Martin, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In CoRI, 2022. 5
13

---

Published as a conference paper at ICLR 2025
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv, 2019. 3
Andrew Szot, Alexander Clegg. Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, al. Habitat 2.0: Training home assistants to rearrange their habitat. In NeurlPS, 2021. 7
Andrew Szot, Unnat Jain, Dhruv Batra, Zsolt Kira, Ruta Desai, and Akshara Rai. Adaptive coordina- tion in social embodied rearrangement. ICML, 2023. 3,3
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Walter Talbott, Katherine Metcalf, Natalie Mackraz, Devon Hjelm, and Alexander Toshev. Large language models as generalizable policies for embodied tasks. In ICLR, 2024. 3
Yufei Wang, Zhou Xian, Feng Chen, Suun- Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. In ICML, 2024. 3,4 4
Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, and Wolfram Burgard. Hierarchical open-vocabulary 3d scene graphs for uinuage-groundeer robot navigation. RSS, 2024. 3
Zhou Xian, Theophile Gervet, Zhenjia Xu, Yi-Ling Qiao, and Tsun-Hsuan Wang. Towards foundation model for generalist robots: Diverse skill learning at scale via automated task and scene generation. arXiv, 2023. 3, 4
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang. Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. In NeurlPS, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2023. 2, 3.7
Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yen Yang, Vidhi Jain, Alexander William Clegg, John Turner, et al. Homerobot: Open-vocabulary mobile manipulation. In CoRL, 2023. 1,5
Naoki Yokoyama, Alex Clegg, Joanne Truong, Eric Undersander, Tsung- Yen Yang, Sergio Arnaud, Sehoon Ha, Dhruv Batra, and Akshara Rai. Asc: Adaptive skill coordination for robotic mobile manipulation. IEEE Robotics and Automation Letters, 2023.
Andy Zeng, Maria Attarian, Krzysztof Marcin Choromanski, Adrian Wong. Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, et al. Socratic models: Composing zero-shot multimodal reasoning with language. In ICLR, 2022.
Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language models. ICLR, 2024a. 1,
Hongxin Zhang, Zeyuan Wang, Qiushi Lyu, Zheyuan Zhang, Sunli Chen, Tianmin Shu, Yilun Du, and Chuang Gan. Combo: Compositional world models for embodied multi-agent cooperation. arXiv, 2024b. 3
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsense knowledge for large-scale task planning. In NeurIPS, 2024. 3
Sipeng Zheng, Jiazheng Liu, Yicheng Feng, and Zongqing Lu. Steve-eye: Equipping lim-based embodied agents with visual perception in open worlds, 2023. URL pttttps://axxiv..oo abs/ 0 13255. 3
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis-Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. In ICLR, 2024. 3
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
We present PARTNR, a benchmark for reasoning and planning in multi-agent embodied tasks, featuring 100,000 natural language, everyday tasks. We show at-scale generation of these tasks using LLMs with simulation in the loop for grounding and human in the loop for filtering. We also evaluate several LLM- planning models on these tasks and highlight avenues for future work. This appendix provides additional details on these contributions and is organized as follows:
A.I Code and PARTNR benchmark open-soureing
A.2 Details and additional analysis on the PARTNR dataset
A.3 Simulation features and prompts for the PARTNR task generation
A.4 The PARTNR evaluation system
A.5 Human annotation and quality assessment for the PARTNR tasks and evaluation functions
A.6 World graph for perception in LLM agents
A.7 Learned low-level robot skills
A.8 Implementation details for ReAct agents
A.9 Details on finetuning LLM agents for the PARTNR tasks
A.10 Additional results
A.11 Analysis of collaborative behavior and efficiency of LLM agents
A.12 Human-in-the-loow (HITL) system and evaluation for the PARTNR tasks and LLM agents
A.13 Prompts for task and evaluation generation
A.15 Prompts for planner baselines
A.16 Analysis of planner failures
A.1 OPEN-SOURCING PARTNR DATASET AND CODEBASE
Accompanying this paper, we will release the code and data necessary to reproduce our experiments. Released code includes our PARTNR benchmark tasks, metrics, baseline oracle skills, large planning model framework, and dataset generation utilities. Released data includes extensions of the Habitat Synthetic Scenes Dataset (HSSD) (Khanna al., 2024), generated benchmark task episodes, and model weights for our trained neural network skills and fine-tuned large planning model.
The publicly released codebase accompanying PARTNR will be contained a public github repository and depend on the most recent version of the AI Habitat platform (habitat-lab and habitat-sim (v0.3.2)) (Puig et al., 2024) which it extends to define collaboration tasks and skills.
For the purpose of this submission, the anonymized code is included in the supplementary zip file. To preserve anonymity and respect supplement size limits, the data and assets required to reproduce the results are not included in the initial upload, but will be released along with the public code repository at a later date.
In order model the space of rich indoor collaboration tasks we propose with PARTNR, extended HSSD with additional asset authoring and annotation. To enable more realistic indoor object manipu- lation, added articulated 3D furniture models such as drawers, cabinets, and appliances. These models were converted from rigid source assets in HSSD and swapped into the scenes. We prepared 60 scenes divided into train, val, and test splits to support our experiments. Each scene manually adjusted by human to ensure simulation robustness and minimize potential issues. Furniture is annotated with a set of Receptacles (surfaces which support small object placement such as shelves and drawers) and can be opened and closed by the agents. Receptacles are further filtered contextually in each scene to ensure that the active set is accessible to the agents. Additional annotations include point or marker sets for each furniture, region annotations, and semantic classification of objects. The marker sets indicate either a spread of surface points (for distance/occlusion checking) or the location of key points of interest such as faucets (for cleaning/filling) and handles (for opening/closing) neces- sary for low-level skill training and oracle skill execution. Region annotations included per-scene region volumes (e.g.. kitchen, living-room, bedroom, etc.) for checking and specifying the location of objects and furniture. Semantic annotations indicate the object category or class (c.g. table, chair, cup, toy) to support open language prompt grounding and semantically guided task generation.
15

---

Published as a conference paper at ICLR 2025
Task Instruction
Task Instruction
"Let's tidy up family room, toys go the toy After that, the plants on the shelf next each other."
"We're ready for dinner. Can you bring two cups to the kitchen? Let's fill them and take them dining table."
Evaluation Function
Evaluation Function
Propositions:
e 1s Insida(["toy Fire truck e*1, ("toy. box 6*13
1. 1s Insida[["toy Food 9*1, ["toy. box b))
2. pplantt __]]_ [Mahel 8", "stelf_ 1"1)
Prapositions
["sheir_ "shell_1"1]
"cup ("itchen" n=2) 1s_filled(( L'cup_ "cup_ 2 "'up_n"), no))
ss_next_tool co([pplant_0" 1. ("plant_1")
Werty "AUNOLS oten
top[ cup
"cup
I' ('toblo 2" (=2)
Dependenctes:
plaeed
(eilesatisfiee((1]]] [4], depends paressoor___ [2)
Uss www.umeeeeeeoo www. www.hwaaww.coothaggaao
Constraints:
Samekegoonstrratnn [[0, 2, arge"tttjet"")
Constraints:
Tor winolsstttrocttnncsssssstee....
Tespratlonntraint ((0,2), (1,2), (9,3),
Proportice di somporiry
(1,3), (0,4), (1,4)]
Figure 6: Additional task and evaluation examples. Expands on Figure 3 to show a task and evaluation containing subset counts and dependent rearranges (right).
A.2 DATASET DETAILS AND ADDITIONAL ANALYSIS
We expand the details and analysis of the PARTNR dataset of Section 3.3, including analysis of linguistic phenomena, secondary task characteristics, and the distribution of sampled entities in generated tasks. See Figure for additional task and evaluation examples.
Linguistic Phenomena. In Table 4, we present an analysis of linguistic phenomena manually annotated over 50 random episodes (similar to Ku et al. (2020) and Chen al. (2019)). We analyze the following phenomena:
Class Reference: refers to a semantic class of objects, furniture, or rooms the scene. These references typically, but not necessarily, follow from the classes defined in OVMM.
Instance Reference: refers to a unique object, furniture, or room; this disambiguation between entities of a class is typically achieved by visual description.
Co-Reference: an expression that refers to entity defined elsewhere in the instruction.
Passive Voice: the instruction is phrased such that the object is receiving the rearrangement or state change; the request is typically asked instead of commanded.
Active Voice: the instruction is phrased such that the rearrangement state change is to be completed with an object; the request is typically commanded instead of asked.
High-Level Goal Spec: this sets the operating context for the task before the particulars of rearrangement or state change are specified.
Agentic Reference: reference to one of the agents performing the task. Typically used to incite suggested task division between the human and robot.
We observe that PARTNR tasks have a high rate of entity class references, such as the table, (6.38/episode), and common occurrences of instance references and co-references. This signals a need for capable natural language understanding, scene grounding, and co-reference resolution. Tasks and sub-tasks are predominantly issued using active voice (92%), but 14% of tasks include at least one occurrence of passive voice. Half of tasks involve a high-level goal specification, which commonly serves to reduce the search space. For example, a task starting with Ler's clean up all the toys in the playroom constrains object search to that room and softly constrains the placement of those objects to locations in which toys would commonly stored. Finally, agentic references are present in 14% of episodes.
Secondary Task Characteristics. In Table 5, We present an analysis of secondary task characteristics present in PARTNR derived automatically from evaluation functions of all episodes in our dataset. We find rare but present occurrences of subset counts, where agents must reason about manipulating subset of a set of objects (e.g. bring two cups... when more than two cups exist in the scene). We find that every episode contains at least one occurrence of resolvable ambiguity, where an instruction makes object/furmiturerron reference that may be resolved by more than entity instance.
16

---

Published as a conference paper at ICLR 2025
Linguistic Phenomenon
p
Example
Class Reference
100
6.38
The table The coffee table That, Those, it, Can you bring me..? Set it on the stool. Let's tidy up. Move... Do... While I...
Instance Reference
12
0.14
Co-Reference
50
0.64
Passive Voice
14
0.18
Active Voice
92
2.40
High-Level Goal Spec
50
0.50
Agentic Reference
14
0.16
Table Analysis of linguistic phenomena in the PARTNR dataset. p is the % of instructions that contain the phenomenon while is the average number of times the phenomenon occurs within each instruction. A random sample of Test episodes were included in manual annotation.
Secondary Characteristics
μ
Example
Subset Count
0.01
Bring two cups... Move the pants to any chair: Place them on the same table. Cup to sink. Cup to cabinet.
Resolvable Ambiguity
68
1.68
Dependent Rearrange
7
0.09
Multi-Step Rearrange
0.11
Table 5: Analysis of secondary task characteristies in the PARTNR dataset. is the % of instruc- tions that contain the phenomenon while μ is the average number times the phenomenon occurs within each instruction. All episodes in Test were included using automatic annotation.
Dependent rearrangements build on this task characteristic; when multiple objects/furmiture/rooms can satisfy a sub-task, a task may require that same entity to solve another sub-task. An example of this is the same table being used solve the placement both a spoon and bowl ("bring the spoon and bowl to the same table in the living room. "). This occurs in 31% of episodes and the resulting dynamic dependency is challenging aspect of collaboration. Finally, 6% of tasks include multi-step rearrangement of the same entity. For example, moving cup to the sink and then to the cabinet after is washed. Tasks with multi-step rearrangement have two such rearrangements on average.
Distribution of Entities in PARTNR Tasks. In Figure 7, we examine the distribution of task-relevant objects, furniture, and rooms. We define task-relevant objects to be objects requiring rearrangement or state change, task-relevant furniture to be target furniture for rearrangement, and task-relevant rooms to be target rooms for rearrangement. Object categories are shown Figure 7a, furniture categories shown in Figure 7b, and room categories shown in Figure 7c. The PARTNR dataset contains long tail of semantic object categories, and within those categories, a wide variety of objects. Thus, for agents perform well in PARTNR tasks, they must display visual reasoning and collaboration behaviors that generalize across the semantic particulars of task. The skew of distributions in Figure 7b and Figure 7c can be understood by the relative occurrences of rooms and objects in HSSD scenes, e.g. there are more tables than couches on average.
The episodes are completely unique not just through instruction, but also through randomization of object instances, initial object locations, count of clutter objects, and agent starting positions. So, the same low-level action trajectories would not transfer to a different task.
We run analysis on the instructions, removing all nouns (as proxy for object ID), and checking for uniqueness. We find that 800 val- 1k episodes and 5193 train-10k episodes retain their uniqueness, implying 80% uniqueness in val, and 52% uniqueness in scaled train episodes. Moreover, instructions are unique with a minimum edit distance of least 5 to account for slight variation.
A.3 SIMULATION-IN-TIIE TII--LOP LARGE-SCALE TASK GENERATION
In this section, we describe in detail the simulation in-the-loop task generation pipeline. We follow a 4tstep generation pipeline:
17

---

Published as a conference paper at ICLR 2025
oojjects
(a) Objects
furiiture
350
600
200
(b) Furniture
(c) Rooms
Figure 7: The distribution of task-relevant object categories, furniture categories, and room categories in the PARTNR dataset. The object distribution is truncated to the 50 most common entities, down from 155 total object categories. The count of entities is derived by the total number of occurrences of that entity in the Test split evaluation data.
1. Simulation-in-the--ee small-scale free-form LLM generation:
In PARTNR, we initiate the process by setting up simulation environment using the Habitat 3.0 simulator, populated with the HSSD dataset which includes 60 unique houses and 5,819 OVMM objects. This simulated environment is parsed to identify list of rooms and available furniture. This information, along with a list of all available objects, is then passed a Language Model (LLM). The LLM utilizes this data to generate free-form, viable tasks within the scene, accompanied by an initial scene state description. For instance, if the task is "Clear dishes from the living room," the LLM would generate initial scene depicting multiple dishes in the living room. To add complexity, additional objects are introduced to create clutter.
2. Simulation-in -nattiio------e-looopp- filtering and annotation.
The tasks and initial states generated by the LLM are instantiated within the simulator. At this stage, tasks that are infeasible due to the layout of the house or the capabilities of the simulator are filtered out. For example, a task like "Clear dishes from the living room" would be discarded if the simulated house lacks living room. Similarly, tasks requiring unsupported actions, such as "folding," are also filtered out. This filtering process is crucial as it significantly reduces the number of unrealistic or impossible tasks, although also tends to limit the diversity of the generated instructions. The full list of allowed actions in the simulator are: [open, close, power on, power off, move, clean, fill, pour, pick, place] and related actions. Next, we use manual annotation to diversity of the resulting tasks. To counter the limited diversity resulting from automated filtering, manually annotate the tasks ensure balanced distribution of various types tasks, such as constraint- freee spatial, temporal, and heterogeneous tasks. This manual intervention involves modifying the instructions incorporate different objects and settings, resulting in a curated set 1,000 human-annotated and simulation-verificl tasks. This step is essential for maintaining quality and diversity but is impractical for scaling up due its labor-intensive nature.
3. Large-scale generation.
To scale the generation of diverse tasks without extensive manual effort, we leverage the 1,000 annotaed instructions as examples in the LLM prompts. By providing the LLM with both house description and an example task, instruct it to adapt the task
18

---

Published as a conference paper at ICLR 2025
to fit the new setting, such as changing "Clear all dishes from the living room" to "Clear all toys from the bedroom." This approach helps preserve the diversity of the tasks while enhancing the likelihood of successful instantiation in the simulator.
4. Human-in-the-lond filtering.
The scaled tasks undergo a final human-in-thee instructions are eliminated, ensuring the practicality and realism of the tasks. This ensures that tasks such "Move cups to the dining table", where there only cups in the scene are removed. Or "First move the cup from the kitchen to the living room, then place jug in the kitchen, next the cup", which consists of physically infeasible instruction.
filtering where physically infeasible
A.3.1 HABITAT 3.0 AND HSSD EXTENSION
We generate PARTNR using modified HSSD scenes Khanna et al. (2024) and the Habitat 3.0 Puig ct al. (2024) simulator due to its humanoid simulation capabilities and availability features which support modeling of collaborative tasks discussed in Appendix A.I.
Our extensions to the Habitat platform include a set features targeting: object state manipulation (e.g., clean/dirty, powered on/off and filled/un-filles evaluation of object relative relationships (e.g.. next-to, above, within, on-top, on- on-loorr in-region, etc), and procedural clutter generation utilities enabling generation of valid initial scene contents pre-conditioned on the output of J.M--genertted requirements from open-language prompts (see Section 3). For example, in order to evaluate the task of rearranging tea set from furniture A to furniture B, must first generate scene with both types of furniture in accessible locations and a tea set already sitting on or inside of furniture A.
A.4 THE PARTNR EVALUATION SYSTEM
In this section, we formalize the components of the evaluation system, define the resulting metries, and present the details and prompts used for LLMM-based evaluation generation.
A.4.1 EVALCATION PREDICATES
We use logical predicates to query the state of objects, furniture, and rooms at the current timestep in the simulator. The evaluation system operates on the resulting binary state values. The details of all logical predicates are in Table 6.
A.4.2 PROPOSITIONS
The primary component of a task evaluation function is a list of propositions. We define a proposition evaluation predicate instantiated with argument values. Propositions additionally enable the evaluation of instructions with ambiguous references ("on a table" - which table?) and subset counts ("two spoons" any two of the total spoons). Ambiguity is enabled by extending the predicate arguments to lists. Subset counts are enabled by optional arguments number, which defines the subset size, and arg_mat ch, which is boolean indicating whether all entities in the subset must be satisfied with the same second argument. Suppose we want to evaluate the task "Bring spoon to the table.". If we have a single spoon and a single table, the proposition is straightforward:
on is_on_ ___o__top [spoon_ [table_l]]
If we have multiple spoons and need just one (an ambiguous instruction), the following proposition is used:
is is_on_top [spoon_l, spoor_2, spoon_3), [table_1]).
In the above case, a list is treated as a OR of entities. The same holds for multiple tables:
i is_on_topp [sspon__,, spoon_2, spoon_31, [table_1, table_2]),
in which any spoon may be placed on any table. If the instruction specifies bringing two spoons to the table, the number=2 argument is added:
is is_on_top [[spoon_1, spoon_2, apoon_3],
[table_1, table 2], number=2).
19

---

Published as a conference paper at ICLR 2025
Predicate Name is_on_top (01, 02)
Category
Description
Rearrange
is considered on top of o2 if a downward raycast from any of the 01 bounding box cor- ners intersects with 02.
is inside (01.02) 02)
Rearrange
01 is considered inside og if threshold num- ber of opposing raycasts hit the same object. Ray casts 1.00m and the threshold number is 2.
is_ _s_in_room (01,
Rearrange
is considered to be in room if at least 25% of keypoints (bounding box corners and center) are contained in the 3D region.
is_on_floor (0))
Rearrange
01 must be within 0.04m vertically of the igation mesh.
is js_ (01,02)
Spatial
The bounding box of must overlap verti- cally with the bounding box O2 and the hor- izontal L2 distance between bounding boxes is less than or equal to 0.50m.
is_clustered (o1....,0,
Spatial
Each o; satisfies is _e_next_to (0;,0)) for some +
[s_clean(o))
State
01 has affordance cleanable and the state machine indicates that oj clean.
is_dirty (o1)
State
Oy has affordance c cleanable e and the state machine indicates that 01 is not clean.
is filled (01)
State
has affordance fi able and the state machine indicates that 01 filled.
is is_empty (01)
State
has affordance fi aabl and the state machine indicates that 0₁ not filled.
is js_powered_on (01)
State
has affordance powerable and the state machine indicates that o1 powered on.
is jf_powered_off (01)
State
on has affordance powerable and the state machine indicates that oj is powered off.
Table 6: Logical predicates evaluate the relations objects, furniture, and rooms in the PARTNR. Predicates exist for measuring rearrangement of objects (category=Rearrange) spatial placements relative objects or furniture (category=Spatial) and states of objects or furniture (Category=State). For all predicates, o, {object, furniture] and r; is room.
20

---

Published as a conference paper at ICLR 2025
Finally, the instruction may specify that two spoons should be brought the same table. arq_mar mat ch-Tuee enables this:
is is_on_top [spoon_1, spoon_2 spocn_3], [table_l, table 2], number=2, a arg_matcheTrue)
Propositions are represented as JSON within dataset files and as Python function calls (as shown above) during human annotation for interpretability.
A.4.3 DEPENDENCIES
Evaluation functions operate over sequence of simulation states. By default, all propositions in the list of evaluation propositions will be evaluated at every time step. However, many tasks in the PARTNR benchmark involve a required temporal order during execution. Take for example the instruction "Move the cup from the table to the sink. Then, return the cup to the table." This consists of multi-step rearrangement where the proposition checking that the cup is on the table dependent on different proposition first being satisfied. We define proposition dependency the following:
P Proposit t ionDependency
proposition_inaiicem
depends_ood on, relation_tyype
where the argument proposition_indices is list of integers indicating the dependent propo- sitions, the argument depends_on is a list of integers indicating the head propositions, and the argument relation_ttteeeeeeeee type indicates the condition that the depends on propositions must take in order for proposition. indices to be evaluated for satisfaction. The following relation types are supported:
after satisfied: evaluate propositions in proposi proposition_inee ind ces after the propositions in depends_or have been satisfied.
after titer_unsatisfidd:: evaluate propositions in proposi i t on on_iinn i ces after propositions in depends_or have been satisfied some point in the past but no longer satisfied.
while satisfied: evaluate propositions in proposition iindiiiiiniieiieneeeeeeeeeieeeoe indices when propositions in depends_on are currently satisfied.
As concrete example, suppose the instruction is "Move the ball and bat the kitchen and set them next to each other. Then, move them to the closet." The propositions for this task would be
0 is _ec_ioo_ooo______e
room [ball], [kitchen]]
is_ is_in___________ ___in_rromm__ [bat], [kitchen])
2 is_next_to [ball], [bat])
is room [ball], [closet])
4 is room room [bat], [closet]).
and the dependencies would be
0 PropositionDeeendenncy ([2), [0, 11, "while_satisfieed ( [3, 4], [0, 11, "after_satisfiee"
1 PropositionDependeer
The first ensures that the is_next_to predicate is only queried during kitchen placements and the second ensures that placing the ball and bat in the closet not checked at the start of the task execution, which would be inadvertently satisfied upon scene initialization. Each evaluation function has a (possibly empty) list such proposition dependencies.
A.4.4 CONSTRAINTS
PARTNR tasks often require constraining how tasks are completed. Examples of this include complet- ing evaluation propositions in temporal order ("Do [x] before doing [y]" and enforcing links between ambiguities ('Fill one of the cups, then put that cup on the table "). We define set of constraint types that enables evaluating such task complexities:
21

---

Published as a conference paper at ICLR 2025
CoamporalConsrainge A directed acyelic graph (DAG) over the indices of propositions that defines the temporal requirement of when propositions should be satisfied. propositions are satisfied out of order, then the task is marked unsuccessful. The temporal constraint is defined as
TemporalConstraint (graph_edges),
where
set of pairwise temporal constraints. For example, an edge indicates that the proposition at index O must be satisfied at an earlier time step than
"O + 1 the proposition index
SanstrrConsttraint Requires the argument used satisfy proposition to be the same within pre-determined set of propositions. The same argument constraint is defined as
SameArgConstraiit aproposition_indiiii
arg_names),
where proposition_indeeeeeeppppppppppp indices ces is a list of proposition indices to link and arg_names is list specifying which component of each proposition to link.
Requires the argument used to satisfy proposition to be unique within pre-determined set of propositions. The different argument constraint defined as
Di Di.fferrentArgConsrraiii (proposi t ion_ionnnnnoooi ces, a arg_names);
where proposition ces is list of proposition indices to link and arg_names is a list specifying which component of each proposition to link.
Some propositions, once satisfied, should remain satisfied the end of the episode. Others are expected to become unsatisfied, such in multi-step rearrangements or object state changes of a single object. The terminal satisfaction constraint is defined
TerminalSattsfaciinoonntttaiii
Te Terminallattisffactooooootttt oarprp/ccococttccoaooooooooooooo (proposition_ ces)
where proposition_inceeeec
is list of proposition indices that should be satisfied at time = T' for an episode rollout of duration T.
A.4.5 EVALUATION METRICS
task evaluation function is constituted of propositions, dependencies, and constraints as defined above. From this data, the task evaluation function serves to determine what percentage of the task complete for a given human-robot collaboration rollout. From sequence simulation states, evaluate the truth values of each proposition with respect to both the dependencies (when the propositions must be evaluated) and the constraints (how the propositions must be satisfied). The Percent Completion (PC) is defined propositions. Success is defined as S == (PC = 100). Both Percent Completion and Success metrics solely evaluate task completion and thus are agent-agnostic. Given that PARTNR is designed to evaluate multi-agent collaboration, we note that these metrics can be combined with duration- metrics (e.g. simulation steps or time) to measure efficiency; multi-agent aspects like task division and partnered exploration serve to optimize task efficiency. The benefit of an agent-agnostic metric is flexibility to evaluate any number of agents performing the task with respect the high-level goal.
the ratio of satisfied propositions the total number of
A.4.6 EVALUATION FUNCTION GENERATION
This section provides detail on evaluation function generation beyond the overview provided in Section 3. In particular, Figure 8 shows the first two steps of Figure (sim-in-the-loog genera- tion and filtering annotation) in greater detail as they pertain to evaluation generation. Notably, we take three step process to generating evaluation functions; first, an LLM generates a list of evaluation propositions which are parsed into a usable format. Then, an LLM infers the temporal constraint over these propositions by predicting topological generations of temporal graph. For example, the prediction [[0, 1], 32 implies that propositions at indices 0, 1 can be completed in either order, and both must be completed before propositions stored and evaluated more generally as edges of a directed acyclic graph (DAG). We found that the assumption of topological generations sufficiently expresses the tasks in PARTNR while being simpler
indices 2, 3. This prediction is
22

---

Published as a conference paper at ICLR 2025
Tppa
Gameraia Eva Latiot
Task Inttrctionn
1i Propestiona
PARTNR Evaluation Generation
Far canale cadi Brigg
Insnuction
Piad
Modifibbe
Us Wordeel and ao
Suceer
Scane Content
Pradicasss
Some Contant
onutrpt coodle D1 task, mat_1
poned parcol (eeeber next.t ponss plant Di DI)
Predican Vorie
Instruction
Set cancle on een table the herg
atwllase pa
I1 Temparal Constraint
Instuctiin
Promppe
LLM
Contrratet
Vardly
enLjes sandle,
1
potted barnt O [ahehoo
Je- 21, [D- 31 (1-221
Verlyy Verly olgect flumature rooms evet
mot out potted, plant i p paesang 0
Siffrr Aegurent Constraints
Dependendee
Vertty oojee cano offardanioss
"petuel
Instuctiinnn
29. Anp Inice
Predicate Vocabulary
Proposito
Gonaralin
depar
B- 10
(1
dias
day
uuiton iffrrncci
Figure 8: Evaluation function generation pipeline overview. A three-step LLM LLM- generation process produces propositions and constraints during Generate Evaluations. The evaluation function then saved to file for optional human annotation during Manual Annotation, Finally, the evaluation function is packed and verified in simulation during Pack & Verify in Sim to ensure that all entities and affordances exist.
Correctness (%)
Task
Evaluation
Combined
Constraint-Fres
89
97
86
Spatial
87
85
Temporal
91
93
85
Heterogencous
93
91
85
Average
90
92
83
Table 7: Manually-annotated generation accuracy of 100k-scale PARTNR tasks and evaluation functions. Altogether, we find that 83% of episodes are generated without any task or evaluation function errors. Analysis performed on 100 sampled episodes of each task type via Predi
for an LLM to generate than a DAG. The third step is predicting argument constraints, in which the LLM is provided the instruction and propositions list and must predict list of constraints, ei- ther www.Arrrooossrtiinnnnn or wffferentArgConstrrett against CodeLlama (Roziere et al., 2023). All prompts used for evaluation function generation are included in Appendix A.14.
All LLM queries are performed
A.5
HUMAN ANNOTATION AND ACCURACY ASSESSMENT FOR PARTNR DATA
A.5.1 GENERATION ACCURACY FOR TASKS AND EVALUATIONS
It is important for tasks in PARTNR to be solvable by collaboration agents and for the associated evaluation functions to accurately reflect the task being performed. In this section, we analyze the accuracy of scale PARTNR generation with respect to both of these criteria using the PrediViz tool (Appendix A.5.2). In Table 7, we demonstrate that the accuracy of task generation ranges from 87-93% depending on task type and averages 90%. The accuracy of evaluation generation ranges from 85-93% depending on task type and averages 92%. Combining these numbers yields an overall joint accuracy of 83% for our 100k-scale dataset. In Table 8, we annotate the failure modes that lead to unsolvable tasks and incorrect evaluation functions. Common task-related failure modes include:
The produced instruction references objects, furniture, or rooms that do not exist in the scene the instruction was generated for. Example: "Move the clothes the washing machine." produced for an environment that does not contain a washing machine.
Hallucination.
Unresolvable Ambiguity. The produced instruction contains ambiguous directives that cannot be reasonably resolved without further communication detailed understanding of
23

---

Published as a conference paper at ICLR 2025
Task Failures
Evaluation Failures
Mode
%
Mode
%
Hallucination
Incorrect Ambiguity 2
Constraint-Free
Already Satisfied
2
Incorrect Furniture 1
Contradiction
2
Hallucination
6
Incorrect Temporal Grouping
7
Spatial
Unresolvable Ambiguity
5
Incorrect Predicate (Other)
5
Already Satisfied
2
Incorrect Predicate (Room vs Furniture)
3
Hallucination
4
Incorrect Temporal Grouping
3
Temporal
Contradiction
3
Incorrect Predicate (Other)
3
Already Satisfied
2
Incorrect Predicate (Room vs Furniture)
Hallucination
Incorrect Predicate (Room vs Furniture)
5
Heterogeneous
Unresolvable Ambiguity
2
Incorrect Object/FurnitureRRonn
2
Contradiction 1
Incorrect Predicate (Other)
2
Table 8: Top three failure modes of 100k-scale task and evaluation generation reported for each task type. Failures of task generation are led by the hallucination of non-existent entities, while failures of evaluation generation are led by incorrect temporal predictions and incorrect predicate functions. Analysis performed on 100 sampled episodes of each task type via PrediViz.
the task-issuer's 's preferences. Example: "Set the table for dinner": how many place settings are necessary? Should we set the formal dining table nook table? What cutlery is needed?
Contradiction. The produced instruction involves two taneously be satisfied. Example: "Set the scissors on the coffee table. Set the bowl on the counter next to those scissors."
sub-tasks that cannot simul-
Already Satisfied. The produced instruction dictates sub-tasks that are all already satisfied at the start of the episode. Example: "Move the laptop to the living room and turn on". when the laptop is already powered on and in the living room.
According to Table 8, hallucinations are the most common fallure mode for task generation. While in-the-loop filtering avoids this issue for evaluation generation, such filtering is inconsistent for tasks; language has a looser grounding to scene entities than statements of propositional logic. For example, a home might not have a formal dining room, but table in the living room may serve the purpose dining table. Moving on to evaluation generation, common failures are as follows:
Incorrect Temporal Grouping. The instruction implies temporal order among sub-tasks (either explicitly via sequencing words, or implicitly via multi-step manipulations) and the predicted temporal constraint fails to reflect this order over the propositions. Example: allowing propositions to be satisfied in any order for the task "First, return the plates to the kitchen. Then, tidy up the living room."
Incorrect Predicate (Other). The evaluation function uses the wrong predicate function evaluate the task. Example: using is _s__owered__o instead of is fillie when the instruction asks to fill the kettle.
Incorrect Predicate (Room vs Furniture). A task specifies that an object should be rearranged to another room, but the propositions overly-constrain the rearrangement to a target furniture for placement. This failure is separate from the one above because the most common, and it indicates the tendency for the LLM to produce single solution instances rather than reflect the full space of ambiguity. Example: producing a proposition like is_oo__tttt_oo to ( deelectrone canttrrrniccllle cable to the bedroom." "
[bed] for the task "Move the electronic
Incorrect Object/Furnins urniture/Room. incorrect entities are selected for satisfying propo- sition. Example: the instruction calls for rearranging the cushion to living room table, but table e__ the proposition is is_on_top onnsshioon]],, [table_4] exist in bedrooms.
Incorrect Ambiguity, For an instruction that can be satisfied n different ways, the evaluation affords options for solution, where * Example: "Move a toy to the kid's room".
24

---

Published as a conference paper at ICLR 2025
where two more toys exist in the toy bin but the evaluation function does not list out all possible toys for rearrangement.
According to Table 8, the primary failures modes of evaluation generation are incorrect temporal grouping and incorrect predicates. Regarding temporal grouping. observe that many sub-task require multiple propositions to evaluate. Grouping these propositions consistently within the temporal constraint is source of error. Take for example the instruction "Set the shirt and pants next each other on the counter. Then, move them to the dresser." In this case, five propositions will exist; four for placements and one for the spatial relation. The temporal prediction may erroneously link the spatial relation with the dresser placements rather than the counter placements.
On average, this process takes 30 seconds per task. To further ensure task feasibility, use human- in-the-loop to solve the tasks. This ensures that criteria such as object reachability and the absence of conflicting task constraints are met. This process is easily scalable on the web, as it does not require expert annotation, and takes 2 minutes 6 seconds per task on average. Our analysis indicates that automatic generation significantly reduces annotation time by factor of 7x.
A.5.2 VISUALIZATION OF PARTNR TASKS AND EVALUATION FUNCTIONS
-eM-eeeeedtedd PARTNR episodes may contain errors so we evaluate their correctness with human annotators. This is verification problem in which the generated instruction, evaluation function, and contents of the scene must be compared against each other. See Figure for an example of this in code. To make this process faster, easier, and more accurate, we designed a visualization and annotation system (PrediViz) that illustrates the state of the world and the evaluation function relative to task instruction (Figure 9). We chose a 2D illustration style to capture the relevant structure in the data:
Rooms. Each room is drawn as box with name provided underneath. Names repeat if there are multiple instances of that room category in the scene. We model the rooms as a fully-connected graph of accessibility. We treat rooms as spatially independent and reorder or wrap them as needed.
Objects. We visualize objects as boxes with category names underneath. Each proposition of the evaluation function is assigned a unique color to separate them from other propositions. For consistency, objects are colored by the color of the first proposition they appear in.
Receptacles. Receptacles are the furniture in each room. We designed bespoke set of icons for the 25 categories of furniture such that they can be easily subitized you can tell a chair is chair and table is a table just by glancing at it.
States. Both objects and receptacles have states that change over time (empty/filled, dirty/- clean, powered on/off). These states are displayed using motifs for objects and textual labels for receptacles, when necessary.
Affordances. Different furniture affords different object placements. For example, a cup can be placed on top of the fridge or inside it, but only on top of table. These relationships are annotated by hand at the category-level. We visualize the target affordance as a dark box indicating where an object is supposed to go. The initial affordances are provided with the metadata to initialize the objects on corresponding receptacles or rooms.
To represent the requested movement of an object, we draw an arrow from the object's 's initial position to the intended position. A simple placement is represented by a single arrow. Multiple allowable placement targets (c.g. multiple tables in the room) are represented by multiple arrows. We use solid arrows for AND placements (c.g. "place the doll and the toy truck on the couch") and dotted arrows for OR placements (c.g. "place the doll or the toy truck on the couch"), "). We also have support for choosing out of objects for
Placements.
placement (e.g. "place two out of three dolls on the couch"), "), for which we use a numerical label on the dotted lines.
Temporal Constraints. To visualize temporal constraints, we split the instruction into multiple frames. For example, if the instruction requires to "place the doll the couch and the toy truck in the chair, then put the stuffed toy inside a chest of drawers, " we create one frame represent the first half of the statement (i.e. "place the doll on the couch and
25

---

Published as a conference paper at ICLR 2025
Bring the laptop and phone stand from the bedroom to the office. Turn on the laptop. Leave the
phone stand next to the laptop.
(a) Example task #1
First, take the file sorter and folder from the office table and put them on the same living room shelves.
After that, the backpack the office chair put in the kitchen cabinet.
1/2
(b) Example task #2: first temporal frame
First, take the sorter and folder from the office table and put them on the same living room shelves.
After that, take the backpack from the office chair and in the kitchen cabinet.
AR
F
T F
HeWe
(c) Example task #2: second temporal frame
Figure 9: PARTNR tasks visualized in PrediViz. The design distills the task and scene to only the components necessary for verification. In example task #2, the split frames signify that the agents must first rearrange the file sorter and folder (Figure 9b), then rearrange the backpack (Figure 9c).
26

---

Published as a conference paper at ICLR 2025
the toy truck in the chair") and another for the second half (i.e. "put the stuffed toy inside chest of drawers").
Special Relations. We also illustrate special relations like text_to, same_as, and di faifreen_froo in the style of an informational legend shown on the side.
Resulting visualizations wrapped in web-based annotation tool that affords binary verification and failure model labeling. See Figure for results derived from this tool. We ran a small-scaae experimental study (n=22) comparing human annotation using PrediViz text-based representation. Using Predi Viz, verification was 2.6 times faster, 8% more accurate, and perceived as 24% easier.
A.5.3 HUMAN-ASSISTED CORRECTION ANNOTATION
Below is an example task and evaluation function saved in plain text for human annotation. The instruction, propositions, and constraints can be modified necessary ensure the task is feasible and that the evaluation function reflects it. Annotators also have access to a file containing the objects, furniture, rooms, and relations thereof reference during this process.
Dataset Correction Annotation Trial
types 1gnoze
INSTRUCTION
modify necessazy, but keep in mind the scene is fixed.
instruction
Help me organize the entryway, Fixet, place the phone, watch, and keychala on the table to each other.
PRCPOSTTIONS
1s_on_cop (objecta, receptacles, number=1, 1a_ Le is_inside (objects, zeceptscles, number-1, is_ssme_eereiaaaceeeealll numberel, at__sme_rromvvaall is_in_room to__jerssm artities_d, numbers1, is_aamo__-aaae 12 12_threshold=0.5) is_next_te to (entities_a, is_on_flocrr (objects,, aumber=1)
Argo:
aumber: sejjects/eniiiee.. must satisty
is_same: the same entity must satisły all a objects
propositions
is_on_top [ceelphhoe__"" ['cable_4")),
is_on_top| [[watcch_"") (")))_((aabb ("table_4']1,
is_on_topl (on_topp[lxeypeeeeee ['"eechhan_DD" D'], ['table_4']]
is_next_to (('eellpooee
is je_ (otptttc(o"stc001 O'', ("keychein_d"))
TEMPORAT. GROUPS Place pzopositions in groups e.t. one group must be satiefied befoze the next. Example: [0, 1), [2, 3] meens propa 0 must be satisfied before propa 2 3,
Lemporal
Lamporal_groups groups -[ 1, 2, 3, di,
TIE CONSTRAINTS optiona: SameArgconstra'nt, Differrentrrgonntraaint
Args:
proposision
[int]
ing_indiess: [int]int]
Example:
SameArgtonstsssint ([0, 21, [1, 11). Propositions march values on the argument at argument index and 1, respectivaly
consttaatitts
27

---

Published as a conference paper at ICLR 2025
TERMINAL SATISFACTION CONSTRAINT:
Ne sasume all progositions must remain satiatled to the end of the eplaode. if proposition AshouldA become ussatisfied, add its index here. exclude Axcludd_final_coneeeeee
mork True if the taak has fatal isave
skip_episede False reasen
A.6 WORLD GRAPH: THE PERCEPTION FRAMEWORK FOR LLM AGENTS
Scene-graph style hierarchical graphs have been shown to be effective for planning problems Gu et al. (2024); Agia al. (2022); Rana et al. (2023). Inspired by such prior work, as illustrated in Figure 10, our world graph is a hierarchical multi-edge directed graph with K = 3 levels for representing the entities in the world. The nodes at first level correspond rooms in the environment, followed by furniture at second, and objects and agents at third level. The root of this graph is an abstract house- node denoting the environment where tasks are taking place. Apart from the semantic information, each node also stores the 3D location of the entity and its affordance states e.g., clean/dirty, on/off, open/close, etc. The graph can then be serialized or accessed via specialized tools by ReAct policies. Prompts in Section 15 provide example of how the world graph is used by the LLMs in our baselines.
A.6.1 BUILDING AND UPDATING THE WORLD GRAPH
House
Rocms
House
Parmdure
Orjects
The initial graph is built by reading the the ground-truth room region annotations and furniture placements asso- ciated with a particular scene. The location of furniture and the region boundaries are used to associate each fur- niture to specific room. This creates the initial two tiers, Room and Furn ture, of the privileged world-graph. In partial-observabiiiiyy setting this is all the planner gets to start planning. However, in the full-observability set- ting, boject-to-furnitur assignments are read from each episode's 's initialization information and a third Ob Object tier added the world- wrrddgrrphh.
Apents
Kitohon
Living Room
Bathroom
Isb e.0
table_0
counter._2
chair.1
chair.
their_2
seap_0
apple_
remoti_2
cup_3
Human
Plobol
Under purtial-bbservabili setting during execution, we use the panoptic sensors attached both agents detect all visible objects in the current-frame. Then ground-truth simulation information is used to extract the housing fur- niture agent for each of these objects, as well the location, and this new information is added to the main- tained graph. For full-observability setting, each frame latest graph is mined from simulator using current information for objects and overwrites the previous world-graph.
Figure 10: World graph. The high-level policies of the human and robot agents leverage hicrarchical world graph con- taining information about rooms, fur- niture, and objects in the environment. This graph is updated based on the ob- servations and actions of both agents.
is not guaranteed in our setup that the images will pick up an object that was placed, filled powered on/off in the previous step. Therefore, in partial-observabiiity setting, we also add deterministic updates to the graph based on previous action and its result, e.g. successful placement triggers deletion of edge between agent who was holding the object and an addition of edge between placed object and the furniture was placed on per action arguments.
A.6.2 PON-PRIVILEGED PERCEPTION USING CONCEPTGRAPHS
In order to study the dependence of the planner on the underlying world-graphe also follow modified version of ConceptGraphs (CG) pipeline Gu et al. (2024) to create the initial layout that is used initialize world-graph representation.
28

---

Published as a conference paper at ICLR 2025
Modified Pipeline for Creating ConceptGraphs. We adapt the original CG pipeline to only use Meta-CLIP models for getting object name and category; using YOLO and SAM for object segmentation; and LLaMA3.1 for annotating room-label and inter-object spatiosemantic relations. Instead of using LLaVA and GPT for getting open-vocab descriptive object names, we use multi- perspective averaged CLIP embeddings of each object to classify its category given our closed object-vocabulary. For the required moo-tooffuriieee relations, we extend CG pipeline by adding another prompt and query Llama3.1 model assign room-labels to each entity given the categories of 10 closest entities to it.
Updating ConceptGraphs. Like the privileged graddd there are two modes of updating a ConcepGraphs--iittaliice non-privileged world-graph
1. Observations. Using the same panoptic sensors we extract all the visible objects in current frame for both the agents. We use depth-sensors extract the point-cloud associated with this object. We use this 3D location to first check if this is redetection of a previously detected object. We use location, category and whether assess if this is a degenerate detection. If this is a new object detection then we use its location along-with bounding boxes of existing furniture to check if this object is contained- within or on-top-of any of the recognized furniture-pieces If it is then the world-graph is updated with this node and the edge.
features to
2. Actions. Just like privileged world-graph, we not guarantee sensors will pick up changed state of an object that is placed, powered, etc. Thus we add similar action based updates this version as well. A special case is when privviiiee graph upated by human agent's 's action arguments which are grounded in privieleged world-graph and may refer to same physical entity with different given names, e.g. backpack is backpack _13 in non- con-privileged graph. We use a simple proximity and category matchig heuristic to match human's arguments to known entities in non-privileged graph, falling back on proximity based matching when no entity of same category are found.
Using ConceptGraphs with Simulated Skills Simulated skills require sim-handles of the place- ment furniture snap objects onto them. The furniture in non-privileged graph do not have these sim-handles by design. Thus we come up with a simple proximity and category-name based match- ing to match a ground-truth furniture entity detected furniture-entity. If we can not find any ground-trui entity of the same category close to the detected furniture, we fallback on matching to the closest entity.
Prompts and Models used in ConceptGraphs Creation. CLIP model and pretrained back- bone checkpoints: ViT-H----- H-11 1114-uiiigeeeee mew.ciip_aaaeeeeeeceeeeeeleeeeee fcc. Object detector: YOLO checkpoint Aloov8xwworldwe LLM: Llama3. 1 10000--000 Instruct
Room Annotation Prompt for Modified CG
< <lbeyin_of_ x aatatt ala:t_hesder__ttee _ilir_idd___e payaLems head_reeder__a _dl are an expert on house layouts. will given an input which will describe (V2RY_OBJBCT) This object be described by ita name and the 10 pieces of furniture closest to it. You will assign
Figure 11: ConceptGraph pipeline. Left to right: Point-cloud of the scene built from trajectory; All objects detected by ConceptGraphs pipeline after assigning a category- based on CLIP: Semantic visualization of the apartment with color-coded class categories per furniture
29

---

Published as a conference paper at ICLR 2025
Jsoa fields "QUERY_ "QUERY_OBUECT DBUECT "CIOSEST "CIGSEST_ORJECTS" ORJECTS*. Your output should also be a 45ON consisting of key "ROGM TABEL". You should only output the JSON and nothing else.
RAME objest. input
You should only assign one of the following lebels: bedroam
2. Living roon
4. dining roon
5. hallway
bathrocm
7. unknown: only when none of the above strings describe the object<lect_jdl>>
A.7 LEARNED LOW-LEVEL ROBOT SKILLS
We provide the implementation details for training the learned low-level robot skills: [Explore, Navigate, OpenFurniture Closefurniture, FickObject, PlaceObject) These skills based two types of basic skills: Navigate skill, which outputs robot base control velocity command given robot depth and sensor observations, and Sanipulatio on skill, which outputs joint angles and base control velocity commands reach the target location given robot depth and sensor observations. We then use these two skills to get the above robot skills.
The LLM receives a binary signal of successful or unsuccessful execution, as well a simple explanation of the failure. For example, if navigation fails because the robot was not able to reach the target the maximum allowed number of steps, then the LLM receives an 'observation' "Unsuccessful execution skill timed out on the way target". There are some mistakes such the robot picking up the wrong object, the observation says "Successful execution!" (since the robot holding object). The LLM needs to take into account the new scene graph to recover from these mistakes, however, is unable to so.
Most salient failure cases per skill are as below:
Navigation Not able to reach the target in allowed steps (timing out)
Pick Picking the wrong object, being too far away from target object, unable to pick because of learned skill failure (object out of reach of arm, requiring complex motion planning to reach)
Place: Target place position too far, skill timing out, learned skill failure (target place location out of reach).
A.7.1 NAVIGATE SKILL
We follow the learned low-level robot skill from Puig et al. (2024) to get the skill. We briefly describe the details for completeness. The goal of Navigate skill is to navigate to the target object given the target object location. The observation space includes (1) an arm depth camera (224 x 171 with hFOV of 55), (2) the relative pose of the target object location in 2-dim polar coordinate system. The action space includes (1) the linear and (2) angular robot base velocities with the range of - 10 and +10 m/s. The reward function location with a correct orientation. An additional navigation success reward is given if the robot can navigate close enough the target object, and collision penalty is given if the robot collides with obstacles in the scene. Moreover, a slack reward is given to let the robot navigate to the target in as few steps as possible. Finally, the skill trained with DD-PPO distributing training.
encourage the robot to move forward while facing the target
Below is the list of skills based on navigate skill.
Navigate: As described above.
Explcre: It is a composed skill that involves calling Navigate skills sequentially given sequence of navigation waypoints.
A.7.2 MANTPULATTON SKILL
We follow the work from Puig et al. (2024) to train the skill. We briefly describe the details for completeness. The goal of Mani
Manipulate at in skill is drive the robot's and base reach the
30

---

Published as a conference paper at ICLR 2025
target object's location. The observation space includes (1) an arm depth camera (224 x 171 with hFOV of 55). (2) the relative pose of the target object in a 3-dim Cartesian coordinate system, (3) the 7-dim arm joint angles, (4) a binary holding detector, and (5) the relative pose of arm end-effector to the target resting location in a 3-dim Cartesian coordinate system. The action space includes (1) the linear and (2) angular base velocities with the range of 10 and +-10 m/s, (3) the delta arm joint angles applied to the arm with the range of -5 x 10-2 and -5 x 10 (7-dim), and (4) binary command to grasp or release the object. The reward function is encourage the arm to move toward the object. In addition, a success reward is given if the robot interacts with the right target object. Moreover, a slack reward is given let the robot complete the task in few steps as possible. Finally, the skill is trained with DD-PPO distributing training.
Below the list of skills based on man pu ate skill.
OpenFuree Furni ture: Given the drawer handle location, man punippula pww.atteeeeee te skill drives the arm and the base to the target. The articulated furniture is opened if the gripper location is close enough to the handle location.
CloseFurni ture: Given the drawer handle location, manipulate ate skill drives the arm and the base to the target. The articulated furniture is closed if the gripper location is close enough to the handle location.
P Pic PickOb ject:: Given the target object location, manipulate skill drives the arm and the base to the target. The object is snapped to the gripper if the gripper location is close enough to the target object location.
P PlaceObjects Given the target place location, manipulate skill drives the arm and the base to the target. The object desnapped from the gripper to the target place location the gripper location close enough to the target place location.
A.8 IMPLEMENTATION DETAILS FOR REACT AGENTS
For all experiments, LLM inferrence is performed on two Nvidia A100 GPUs using the gpt- fast inference engine Py Porch (2023). Inference on LLama-3.1-700 (using tensor parallelism over two A100s), resulted in an average generation speed of 11.43 tokens/s. Each planning step required an average of 52 tokens resulting in a latency of 4.55 seconds per planning step. The average wall time to complete and entire episode (planning steps for both agents and simulation time) was 36.0 minutes. The finetuned model based on Llama-3.1 3.B required an average of 0.53s per planning step. For those experiments, simulation time and human agent inference time remained unchanged, giving final wall time of 25.3 minutes per episode.
All decentralized baselines had maximum timeout of 50 replanning calls, while centralized baselines had a maximum timeout of 100 replanning calls (to account for the fact that one planner would need to plan for both agents). Additionally all baselines had maximum timeout of 20000 simulation steps.
A.8.1 SKILL API LIBRARY
Below is a list of the skills available across all baselines. Agents acting in the robot role do not have access to state-modifying actions (Clean, Fill, Pour, PowerOff, PowerOn). The ReAct agents considered in the main paper do not have access to perception tools (FindAgentAAction Tool, FindOb- jectTool, FindReceptacle Tool, findRoomTooll We additionally study ReAct agents that query the environment via those tools, which we name ReAct-Tools.
Clean Used for cleaning an object.
Close Used for closing articulated entity.
Explore Search specific room by visiting various receptacles furnitures in that room.
Fill: Used for filling an object.
Navigate: Used for navigating to an entity.
Open: Used for opening an articulated entity.
Pick: Used for picking up object. The agent cannot hold more than one object at time.
31

---

Published as a conference paper at ICLR 2025
Place Used for placing object target location. This requires the name of the object to be placed, the name of the furmiture where it should be placed, spatial rela- tion ("on" "within") describing the relation between the object and furniture. The ob- ject to be placed must already be held by the agent (i.e. picked previously). Addition- ally, you can request to place the object near another object. For that you optionally provide a spatial constraints ("next_to") and the name of the reference object. To place next an object, the reference object must already be on the target furniture. API tem- <furniture spa-
pate: Place Koojject_ settttee__teeeeeeeee _e__bee__dddd,,_ <spatial_relation>> ceeference_objecte tial cial_contraiin and reference_ _bjjcct should be set to "None" when necessary.
to be placed on>, xspatial_constralle www.traintst,tooom
Pour Used for pouring from one container to another. This skill will pour into the specified
container from whichever container is currently held by the agent.
PowerOff Used for turning off a powered object
PowerOn Used for turning on a powered object
object from its current location the target location. This requires the name of the object to be rearranged, the name of the furniture where it should be placed, spatial relation ("on" or "within") describing the relation between the object and furniture. This skill will automatically pick the specified object and move to the target furniture and attempt to place it. Additionally, you can request to place the object near another object. For that you can optionally provide spatial constraints ("next_to" and the name of the reference object. To place next to an object, the reference object must already be on the target furniture. API tempate: <furniture
Rearrange Used for moving
Rearrange [<ob]]] jobtetc__t be placed>, <spatial_relationt treffrrence_object...
to be placed on>, <spati i i lts___ooostttttttt Opatial_constraing and ffferncceoojjed should be set to "None" when necessary.
Wait Used to make agent stay idle for some time
FindParttnerAgeetAActoon
This tool will return a summary of the other agent's actions.
FindObjectTool Used find the exact name/names of the object/objects interest. An LLM will be used to distill relevant objects from the user query. Example (FindObjectTool(toss on the floor] or TonoOheeeTTolllappleee
FeceRecep Tool Used to know the exact name of receptacle. An LLM will be used to distill relevant receptacles from the user query. Example (FindRecepttaceeTool] kitchen counter) FindRoomTool Used to know the exact name of room in the house. LLM will be used to distill relevant rooms from the user query. Example (FindRoomTooll) room which might have something to eat))
Done Used indicate that the agent has finished the task.
A.8.2 CONSTRAINED GENERATION
We follow the procedure described in Geng et al. (2023), constraining token sampling only select tokens that consistent with least one accepting string the specified grammar. For each call the LLM we build grammar which will only accept valid tool calls observed entities. Below is the base grammar used tool calls for all experiments. For experiments utilizing a summary of the world representation (i.e. ReAct, Finetuned see Section 4.1) the perception tools (F (FindObject Tool, rindReceptacleTood etc.) are omitted. The rules for ob object, furniture, and room are set dynamically based the agent's current world graph. This ensures skills are called only for entities that the agent knows exist.
Tool Call Grammar
root ::- Navigate Pick Place Open Close Resrzange PowerOa PowerCff Cleen F111 Pour Explore Wait FindReceptacleTool I FindDojec:Tool Piadkgeat&eciooTTo.. FindRoomTool Done Navigate ::- "Ravigate(" nav_target "]* Pick ::= "Pick[" object "]"
Place ::- "Place[" object "," ws apattal_reeationn relation * NS furniture *," MS Mespatial_consteee "None"!)) "j"
"," wa d'bioorffunituue (*"none" "None") NB "," ("nose"
Open ::- "Open[" tucniture
32

---

Published as a conference paper at ICLR 2025
::8 "close[" furniture "]"
Rearrange 1:= "Rearrange [* object "," Ks spatial__rot rrolocion "," Firniture "," N8
conatiaal_conttraile "," RS "None"!)) "]"
)1 (!"none" "Rone") RB "," ("none"
PowerOn ::- "Pcwezün(" obi_or_ _oj_or_furniurre *]"
Poweroff "Foveroff[" @bj_oor_frnituu *)"
Clean ::- "Clean[" coj_orrrurritn "J"
::= "Fill[" object "]"
Pour ::- "Pour[" object "J" Explere ::- "Explore(" room "]" Wait "Wait[" *]"
Vindksceptacleiioal ::= "rindkeceptacleiooll free_tex: *]"
Tol "FindOhjectToal free_taxt, "]"
LindggentAciiontin ::- AentdgenttctionToool "J"
FindRoomtcol :1= "FindRaonToal free_text *]*
Done ::- "Donel]"
nav nav_target 1:- (Eurniture room object)
object "object_]" "object_2" ...
obj_er_furnitune :s= (furniture object)
furniture "fu"niture_1" "furriture_2"
coom ::- "soor_1" "zoom_2" I... spatisi_cenatraing :1- "next_to"
spalial_relaation *on" | "within" free_text ::- "l 1e:,la-zh- zh- 2--9-1- [+
A.8.3 -AUHEVENT- GENERATION FOR REACT AGENTS
Retrieval- Setrieval-Augmennedd Generation (RAG) is the method of optimizing LLM text generation by querying an external database. However, there are several challenges to applying RAG in our setup. First, when implementing RAG, necessary to provide an external database from which LLMs can retrieve information. However, it is unclear how to effectively generate such dataset and determine which content is most beneficial for solving the task. Second, once LLMs retrieve the information, is unknown where ingest this information into the generation process. To solve the above challenges, we develop an approach inspired by the recent literature that uses an LLM to generate its training dataset iteratively optimize performance (Pang al., 2024; Madaan al., 2024), As shown in Fig. 12, we first construct the dataset by collecting the successful traces generated by LLMs for solving the training tasks. Then, during time, we select the most relevant trace by comparing the sentence similarity between a test instruction and the ones in the dataset. The selected trace is passed back the LLM's promppt as a successful example trace. This represents a refinement process that uses its own past success experience to increase the chance of solving downstream tasks. Figure 12 illustrates the high-level idea.
Specifically, we use the PARTNR train dataset to generate ReAct traces (where the human agent is ReAct-Tools and the robot agent is ReAct). In total, we generate 925 traces that successfully solve the tasks to form the RAG dataset. During evaluation time, base-v2 Reimers & Gurevych (2019) to select the most similar instruction to the instruction at hand in the dataset, followed by adding the trace into the ReAct prompt.
sentence similarity computed by
A.9 FINETUNING LLM AGENTS
In this section, we describe how we finetune an LLM to build our Finetuned baseline. We detail the process to generate the data for finetuning our model and the training details below.
A.9.1 DATA GENERATION FOR FINETUNING
We train the model using successful traces from the ReAct baseline, which obtains the best decentral- ized results. In particular, we run this baseline together with ReAct-Toool human on the training set, and keep the episodes with 1.0 success rate. If an episode reaches 100% success rate some time during the task, but fails at the end, we keep the actions up until the success step and replace the last action with a Done[] action to finish the episode. This process results 1,226 episodes. We then split each episode into the sequence of robot actions, and filter out those that resulted in failure. Our training set is constructed by building a prompt for each of the successful actions, as shown in Sec. A.15. This process results in 15,889 training samples. Note that each action prompt contains the
33

---

Published as a conference paper at ICLR 2025
generation step
Task instruction, reasoning trace, success
Taeks
High-Level Policy (LLM)
Task 2: Instraction, reasoning trace, fallure
success episodes dateset
Task instruction, reasoring trace, failure
Runtime Inference step
Tasks
instruction embedding
Dot product, select the nighest score the shot prompt
Put
High-Leve Policy (LLM)
instruction embedding
success
embedding
episades
dataset
instruction embedding
Figure 12: Overview of ReAct-RAG. RAG consists of two steps: (1) We first use a training dataset to obtain traces from LLMs and log traces with task instructions. (2) During runtime, retrieve the traces by doing dot product on instructions and obtaining the trace with the highest score, and finally put the trace in the prompt. We ensure that the dataset used for the first step different from the one in the second step.
current state of the environment and the previous robot and human actions, but filters out the thought produced by the human robot.
We also explored using the Couristic-Expert baseline as data source for finetuning, but we did not observe improvements in the resulting model. Given that this bascline plans using the ground truth evaluation function, we hypothesized would help distill the natural language task into the right predicates. We followed the same process described above, obtaining 1250 episodes and 13939 training samples. We trained a model with ReAct, Heuristic-Exper and both data sources. We show the evaluation results for each model in Table 9, with the model trained with only React data performing the best.
Data source
Sim Steps |
Success Rate f
Percent Complete 1 Planning Cycles 4
Couristic-Expert
3477.82 82 78.19
0.63 0.02
0.79 = 0.1
15.50 = 024
ReAct
3228.96 75.14
0.70 001
0.84 0.00
12.85 024
ReAct+ Heuristic-Expert
3552.96 61.95
0.69 001
0.83
14.47
Table Performance Finetuned model when using different data sources for finetuning. We measure performance using simulation steps required to finish the episode, success rate and percent complete on the tasks, and the average number of planning cycles performed by the planner. Mean and standard error are reported over the validation set. Heuristic centralized expert has access the task evaluation function.
A.9.2 IMPLEMENTATION DETAILS
We train the model to predict, for every example, the action taken by the agent, which corresponds to the text after the rresserve_speccaa_ttoooo
00 token.
use a low rank adapter Hu et al. (2021) with r = 132, a 128, dropout=0.01, on top of the value and query projection layers W W We train all models 4 A100 GPUs, with batch size of per GPU, The models are trained for 40,000 steps, which takes around 24 hours.
A. ADDITIONAL. RESULTS
To supplement the results in Table 2, we have included Table 10, containing results from additional baselines the validation set, and Table 11 containing results on the PARTNR test set. ReAct
34

---

Published as a conference paper at ICLR 2025
Sim
Success Rate
Completion Rate 1
Planning Cycles +
Method
Controllability
Skills
Observability
Steps
Heuristic-Exper
Centralized
Oracle
Full
1260.88 88 26997
0.84 om
0.94 no1
N/A 17.49 0.4
ReAct
Centralized
Oracle
Full
1347.43
33.80
0.74 a
0.88 a 0o1
ReAct
Decentralized
Oracle
Full
1915.63
0.74 +
0.86
14.20 0.44
ReAct
Centrulized
Oracle
Partial
2298.
61.39
0.74 0.0
0.85 001
20.73 8:5
ReAct
Decentralized
Oracle
Partial
3295.20
76.20
0.73 1:000
0.86
15.24
ReAct-Tools
Decentralived
Oracle
Partial
3622.52
79.00
0.71 o:0m
0.85 001
21.41 0.0
ReAct RAG
Decentralized
Oracle
Partial
3467.47
=: 82.39
0.71 +
0.84 = 0.00
14.75 0:331
Finetuned
Decentralized
Oracle
Partial
3228.96
75.34
0.70
0.84
12.85
ReAct
Decentralized
Leamed
Partial
6494.88
181.52
0.57 +
0.76
22.72
ReAct
Decentralized
Learndd
ConceptCiraph
12274.27
212.68
0.25 o:00
0.53 2:01
26.74
ReAet-Single
Single Agent
Oracle
Partial
2519.02
57,48
0.73 0.00
0.85 0.01
18.68
ReAct-Single
Single Agent
Oracle
Full
1590.20
=: 40.33
0.73 0.01
0.85 0.11
18.60 0.
ReAct-8B
Decentralized
Oracle
Partial
3699.45 67.40
0.64 0.02
0.80 0.01 23.15 t.0.7
Table 10: Baseline results PARTNR validation set. We measure performance using simulation steps required to finish the episode, success rate and completion rate on the tasks, and the average number of planning cycles performed by the planner. Mean and standard error reported over the validation set. Heuristic centralized expert has access to the task evaluation function. Collaboration enables higher task completion and success compared to single agent task execution (shown in gray), at the expense of more simulation steps highlighting the coordination "burden".
Sim
Success
Completion Rate ?
Planning Cycles
Method
Controllability
Skills
Observability
Steps
Rate
Couristic-Expert
Centralized Centralized
Oracle
Full
1184.74 74 22.88
0.69 2000
0.89
N/A
ReAct
Orucle
Full
1348.71 a 53.34
0.67 oxe
0.86 a 001
21.14
0.9
ReAct
Centralized
Oracle
Partial
2590.82 1: 90.71
0.56 0.0
4 0.1
25.57 0.64
ReAct
Decentralized
Oracle
Partial
3353.33 # muss
0.63 0.00
0.84 + 0.1
17.38 0.33
ReAct-Tools
Decentralized
Oracle
Partial
3810.15
0.61
0.83 =
25.79 0.4
ReAct RAG
Decentralized
Oracle
Partial
3489. 18
0.62 0.00
0.83
17.55
Finetuned
Deventralized
Oracle
Purtial
3460.60
1: 18.33
0.51 oxg
0.78
14.73 0:55
ReAct
Decentralized
Learned
Partial
5905.88
162.35
0.50
0.76
24.30 0.00
ReAet-Single
Single Agent
Oracle
Partial
2632.30
60.04
0.68 0.00
0.85
21.28 0.3
ReAct-Single
Single Agent
Oracle
Full
1559973
16,02
0.73 = 0.01
0.88 no1
21.06 0.38
ReAct-8B
Decentralized
Oracle
Purtial
4100.21 98.7
0.51 ox0
0.77 1.
27.65 1.
Table 11: Baseline results on PARTNR test set. We measure performance using simulation steps required to finish the episode, success rate and completion rate on the tasks, and the average number of planning cycles performed by the planner. Mean and standard error are reported over the set. Heuristic centralized expert has access to the task evaluation function.
35

---

Published as a conference paper at ICLR 2025
use the same summary prompting format as the baselines in 2. ReAct-Tools requires the agents to use perception tools to observe the environment instead. ReAct - BB uses Llama-3.1 1..11-8--Insttue as the robot model for equal capacity comparison with Finetuned.
We also present two additional baselines that use VLMs to extract visual information. We design two new baselines that combine the ReAct approach with VLM to leverage vision during planning, as detailed below:
In the first baseline, we measure whether visual information can be combined with graph representation to improve planning. For this, we still provide a privileged graph representation, but also add a caption of the agent's current egocentric observation, generated using Llama-3.2-1 1B- Iision-Instruct, This provides strictly more information to the planner, as it uses both the visual information and accurate world graph. On the validation set, this baseline achieves a success rate of 0.72 (within the error margin of the baseline that does not use VLM, Table 2e), showing that adding the VLM output did not improve performance.
In the second baseline, we measure whether visual information alone can help perform better planning. when no graph is provided. For this, we use Llama-3.2 De-Vision- observations seen by the agent so far. The planner uses these descriptions to generate an action. On the validation set, this baseline obtains a success rate of 0.69, lower than the baseline that uses a graph representation, Table 2 row (c). Studying how to best leverage visual inputs improve planning in PARTNR is an important direction that we leave for future work.
to describe the visual
A.11 ANALYSIS OF COLLABORATIVE BEHAVIOR AND EFFICIENCY OF LLM AGENTS
Since task state success and percent complete metries look at overall team performance for the human and robot agents in our tasks, we also evaluate metrics that allow us to look at different aspects of collaborative behavior of the agents. We measure percentage of sub-tasks done by the robot (task offloading), ratio of unnecessary rearrangements over total successful rearrangements done by both agents (extraneous effort), and number of exploration steps needed before first object picked (exploration efficiency) to analyze agent behaviors (Table 12). Our main findings are below:
Method
Sim Steps
Task Offloading
Extraneous Effort]
Exploration Efficiency.
Decentralized
3295.20
#76277
0.596 =001
0.21 #001
994.88 #24,800
Centralized
2298.13 151.39
0.49
0.04 10.004
684.06 121.71
Single agent
2519.02 157.48
0.047 1001
1121.65 131.256
Table 12: Analysis of collaboration characteristics for LLM agents. Average and standard errors for task offloading, extraneous effort, and exploration efficiency are reported over the successful episodes from the validation set for LLM agents using ReAct approach in partially observable setting.
Agents are able to find objects faster when collaborating as compared to solo, but only when they successfully co-ordinate. The exploration efficiency increases i.e., agents are able to find task-relevant objects in fewer steps, in centralized and decentralized settings. By computing the average number of exploration steps taken before the first object is picked up for a task, we find that single agents require on average 127 steps more locate objects compared However, in centralized setting, where the co-ordination between agents is better owning to single LLM co-ordinating the actions of both agents, shows higher gains in such efficiency as compared to decentralized settings. The challenge LLMs face in coordinating exploration in multi-agent settings also negatively impacts human-I.I.M team performance in our HITL experiments when paired with humans (Table 3).
multi-agent.
Poor co-ordination also leads to wasted effort and more steps to complete the tasks than solo execution. Despite multiple agents working together, agents take longer to complete the tasks in decentralized settings as compared to solo execution owing to poor co-ordination. Poor co-ordination is further highlighted by extraneous effort, which increases by 300% in decentralized settings as compared to solo execution. The agents often repeat parts of the task unsure of whether the other agent really executed that part, and sometimes even undo successfully completed tasks.
36

---

Published as a conference paper at ICLR 2025
Method
Success Rate per Task-type
Constraint-fres
Spatial
Temporal
Object states
Decentralized
0.82 #0.02
0.82
0.60=0.03
0.66+0.03
Centralized
0.84 20.02
0.85+0.02
0.59
0.6611.08
Single agent
0.85+0.02
0.81
0.58
0.68:+0.03
Table 13: Task performance per task type. Average and standard errors task success rate for episodes from the validation set categorized by task type. Performance is shown for LLM agents that use ReAct approach in partially observable setting.
The robot is able to offload more than half of the tasks from the human partner. The human-robot team takes longer to complete the task in decentralized setting, however, the robot offloads 60% tasks from the human partner, reducing their load of task execution. This highlights the potential of robots assisting humans more effectively LLMs continue advance in reasoning, coordination, and planning capabilities.
LLMs struggle to reason about temporal constraints and agent capabilities while planning PARTNR tasks. Constraint-frre and spatial tasks in PARTNR require the LLMs to reason about only the final configuration and states of objects. Instead, the temporal tasks in PARTNR require tracking states of one or more objects over the entire episode, making them challenging (Table 13). Likewise, heterogeneous tasks necessitate reasoning about task distribution conditioned on each agent's capabilities, which make them challenging.
A.12 HUMAN-IN-THE-LOON EVALUATION FOR PARTNR TASKS
A.12.1 HITL INTERFACE AND WEB HOSTING
We adapt the existing human-in teree-innttt-oooo loop (HITL)
infrastructure from Habitat 3.0 (Puig et al., 2024)
to support a more robust server-client architec-
ture, with the server component hosted on AWS. Habitat3. 0 HITL includes the ability to extend functionality to resource-constrainet environ- ments such as web browsers and VR devices, making the platform versatile for different user needs and experimental setups. Figure 13 shows our HITL system running on a web browser. De- tailed interface is shown in Figure 14. Our adap- tation to AWS hosting crucial for handling multiple clients simultaneously, especially non- experts without access to powerful machines or large Habitat datasets. The server-client archi- tecture not only enhances scalability but also ensures flexibility, allowing the system ac- commodate variety of operating systems and hardware platforms. Furthermore, the system in- cludes matchmaking service that enables pair- ing participants for multi-user sessions. When a participant requests a task, they are redirected to "lobby screen" where they are instructed to wait until the next participant arrives.
12P0099 tastinden tere hamur
Mixi the seleonesandd and the electionic cablle from
The couch the chalves the Withen and the kettl with water. Turn on the tamp, and dean Th oouch.
Figure 13: HITL Web-browser. Our HITL sys- tem can be deployed on web browsers enabling large-scale collection.
A.12.2 PARTICIPANT RECRUITMENT AND QUALITY CONTROL
The study was performed through 3rd party company specializing in large-scale annotations. The participants were recruited and compensated for their time by this company. The participants were English speakers, 18 years older from the United States. For training them, created a project and task overview video and guidelines. The participants were instructed to complete the tasks correctly and efficiently by themselves with partner. The participants also went through tutorial where
37

---

Published as a conference paper at ICLR 2025
Instructions
Controls
Help organize my workout room. place dumbbells the Then place the basketball and helmet next to each other on the floor.
H
Hide Help
WASD
Move
R/Middle Click
Look Around
Look Up/Down
Left-Click
Select
Double Click
Pick-up
Double- Click
Open/Close
Right- Click
Drop
Finish Task
Dumbbell
Release place.
WorkounGGym/Exerciee
Dirty
Clean
Figure 14: HITL Interface. Participants control human and robot agents using keyboard/mouse controls to complete the PARTNR tasks. Each participant has access to their partner's viewpoint and thereby current actions via small viewport on the bottom right.
they performed some tasks to get acquainted with the interface before performing the main tasks. Each task took average 3-5 minutes complete. We recruited 129 non-expert participants in total.
Filtering data:
Each task completed up times, in all settings, until deemed successful through task evaluation (conducted online as the participants complete the tasks). With the Failure explanation output from the evaluation function, users describing what went wrong in an episode, if the task was not successful. For example, if actions were completed in the wrong order, or a spatial constraint like next-to was not respected appropriately. Users can use this information to update their actions in the next episode, improving their overall performance over time. By giving tutorial episodes before the start of each experiment, and also tutorial episode in the same house, we ensure that the users are deeply familiar with the tool and tasks before starting the actual experiment.
also given a natural language feedback the end of an episode,
Once each task was completed successfully at least once retried 3 times, we assimilate data by selecting one of the tries per episode: we select the successful try for successfully completed episodes, and the highest percent complete try for unsuccessful episodes. This collection of 1000 episodes for test and val is used for collecting performance statistics described in the next section.
A. 12.3 HITL EXPERIMENTAL DETAILS
Task Evaluation and Data Collection:
Using the enhanced server-client setup. we conduct comprehensive evaluations of 1000 tasks from the validation set, and 1000 tasks from the test set. These evaluations are designed capture data in both single-user and multi-user scenarios. In the single-user setting, participants individually control human agent within the simulator using traditional keyboard and mouse inputs, completing tasks without any external assistance. Conversely, the multi-user setting involves collaborative efforts where two participants work together, each controlling either human robot agent. This collaborative approach specifically designed to study the dynamics between multiple users and to evaluate whether such collaborations lead to more efficient task completion compared to single-user efforts in the PARTNR tasks.
Human-Al Collaboration Experiment:
38

---

Published as a conference paper at ICLR 2025
Success Rate f
Percent Complete 1
Sim Steps +
Task Offloading f
Exploration Efficiency +
Extraneous Effort +
Method
Validation 1,000 episodes
Single-user Multi-user
0,93 a: 0.01
0.96 0.00
3046.99 80.79
N/A
2459.22 4: 26.75
0.09 0.01
0,93 = 001
0.96 + 0.00
2369.55
49.33
0.59 + 0.11
1762.47 + 13.99
0.15 = 0.01
Human-ReAct
0.91 = 0.01
0.96
+ 0.02
4267.71 = 83.40
0.16 + 0.1
2624.39
+ 26,08
0.12 0.00
Human- -umann-inettnnd
0.92 = 0.01
0.96 + 0.00
3443.33 = 61.46
0.26 + 0.01
2164.94 +: 21.31
0.13 = 0.01
Test 1,000 episodes
Single-user
0.89 + 0.01
0.95 = 0.00
3937,87 = 110.53
N/A
2737 .4 + 25.27
0.13 = 0.00
Multi-user
0.85 + 0.1
0.95
+ 0.00
2667.86 = 58.07
0.60 + oat
1889.56 + 14.9-60
0.20 + 0.00
Human-ReAct ReAct
0.87 +: 0.01
0.95 4: 0.00
4080.10 = 72.24
0.12 4: 0.00
2449.50 +: 19.98
0.18 #: 0.01
Finetn--
0.87 = 0.01
0.00
3403.03 62.08
0.26 0.01
2162.20 + 19.40
0.20 = 0.01
Table 14: Human-in-the-Loop evaluation. We evaluate the performance of 2-person human teams and human-LLM teams, comparing them to solo human performance on PARTNR tasks using metrics described in Section 4.1. The human-LLM teams with SOTA LLMs is slower than solo human.
In addition human-only interactions, conduct experiments where human collaborates with robot controlled by Language Model (LLM), specifically using the ReAct and Finetuned models as described in Section 4. The primary goal of these experiments is evaluate the effectiveness of LLM-controlled agents in real-time collaboration with non-expert humans who have not previously interacted with these Al models. We track and compare the success rates (SR) and the percentage of tasks completed (PC) across various settings including single-user, multi-user, and human-Al AI collaborations. The results of these experiments are systematically documented and analyzed in Table 14, providing insights into the collaborative capabilities of human-Al pairs.
To enable this setting, we host LLMS on AWS nodes, and query them intermittently based on robot observations and actions. The HITL server queries two clients human and LLM. The human client sends commands to control the human agent, and the LLM client uses the hosted LLM to control the robot. Different baselines need different numbers of GPUs to keep the inference time reasonable. For hosting and 70B models, as used by the ReAct baselines, we use 4 A100 GPUs per model. For hosting smaller 8B model used by the Finetuned baselines, use 1 A100 GPU. This makes deploying smaller Finetuned models much scalable than larger 70B models.
Efficiency and Task Offloading Metrics:
To further understand the efficiency of task completion across different experimental setups, we measure several key performance metrics beyond success rate and percent complete. These include the number of steps taken to complete tasks and the exploration efficiency, which is assessed by the number of steps participants take to pick the first object. Additionally, we evaluate the extraneous effort by noting actions that do not contribute directly to task completion. Another critical metric we analyze is the ratio of work completed by the robot, referred to task offloading. Ideally, in well-coordinated AAman-Al team, the task offloading ratio should approach 0.5, indicating an efficient division of labor between the human and the robot.
A.12.4 HITL ANALYSIS
Humans are significantly better than LLMs at PARTNR tasks. Both single and multi-human settings achieve success rate of 0.93 on PARTNR validation tasks, while ReAct without any privileged information only achieves 0.30 (Table 2, row (i)). This indicates a huge gap in LLM planning performance. We observe slightly lower human performance on the test set (0.89), also in line with lower LLM performance in Table 11 on this dataset. This indicates that the tasks in our test set are more challenging than the validation set for both humans and LLMS alike, potentially due to human annotations aimed at making them more complex and diverse.
Finetuned LLMS perform better than ReAct when coordinating with real humans. When deployed with real humans-in-the-loog on the validation set, the finetuned model is faster than ReAct at task completion (3275 steps with finetumed versus 4484 with ReAct on the validation set). It is also able to offload more tasks from humans than ReAct (26% with finetuned as compared to 16% with ReAct). This reflects that smaller models with faster inference can improve human experience in real-world deployment. This result is also reflected in the test where finetuned model outperforms
39

---

Published as a conference paper at ICLR 2025
ReAct. Interestingly, the automated eval performance of finetuned is worse than ReAct the test set, but the HITL performance is better, indicating that faster inference more critical than task success when working with real humans.
LLMs struggle at coordination, hampering human performance. Despite the Finetuned being faster than ReAct when collaborating with humans, both approaches slower than the human doing the task alone. In contrast, two humans working together complete the task faster than a single human (2369 steps 3046 with multi- and single-user respectively). This result is in line with the automated evaluation we observed in Table I, where multi-agent LLMs are also slower than a single-agent LLM. This result further reinforces that LLMs suffer coordination; while humans are able to coordinate and divide tasks between each other, decentralized LLMs are unable to do so. We observe the same effect the test set, further reinforeing this finding.
LLMs are able to offload tasks from humans. Despite the aforementioned increase in the number of steps for task completion, robots guided by the finetuned model successfully offload 26% of tasks from humans. This indicates that LLMs can still offer assistance when collaborating with real human partners, Nonetheless, there remains significant potential for improvement.
LLM's inefficiency to explore reduces the team performance when paired with humans. In multi-user condition, the two humans start different parts house, and explore efficiently to locate task-relevant objects more quickly than single user evidenced by the reduced number of steps before first pick (1762 steps with multi-user vs. 2459 steps for single user). However, this efficiency is reduced when humans are paired with LLMs (2120 steps with finetuned and 2791 steps with ReAct), indicating that LLMs struggle to coordinate at both task completion and exploration.
A.13 PROMPTS FOR BENCHMARK TASK AND EVALUATION
A.13.1 TASK GENERATION PROMPTS
Prompts are similar for the different task types, with the primary difference being the in-context examples.
Constraint-free task generation prompt
You system that generstes tacks for robots pezform with himans. Do not be verbose. Anawer the queation with no added qualifications or caveats. Just directly provide the answer JSON.
You
given a description to provide instruct ions for tasks that a robot and human could be doing together in that house, using the objects and furniture.
a house objects furniture your
For each tack, provide the initial state of objacts in the house, the instruction that should performed,
objects after instruction performed. The initial and final state will contain a list of dicticnaries, each with an object furniture floor, and
final
type, the number objects that type, their the region of the house where they are in e.g., bedroom.
Follew the principles:
human doing task perform part of it. In some cases the task will be done togother, in other cases, the human and robot will be coing different tasks.
instruction should given
2. The initial and finai state should contain objects of different Lypes, and sometime multiple objects of a type.
3. Seme of the instructions should semantically rich, partieular they should rafer
to claases or groups of objects.
The instructions shouldn't detailed and explain all steps, but high-level.
5. Tha robot can only rearrange objecta and open containers, the human can do more taaks
e.g., turn on lam9, clean plates, fill up pitcher,
6. The instraction should contein clear
and at leass two associated with
the goal.
instructions should contain spacial apecifiers such aa "next to", "left", "right", "beside", "near", contain temporal ordar, be specified using worda such as *after", *chen", *before" etc. For instance: "Fill the kettle and then tuzn it on. Aflez that, bring two cups to the dianing Lable."
Ensure that instructions
diverse from other.
"front", "side",
some
instractions
40

---

Published as a conference paper at ICLR 2025
instructions should contain diverse actions as "tura on/off", *Eill*, "clear", "set" etc. and object states auch as "aleon", "cirty", "open", *close" etc. while reterring to jejtts
You will
all the piaces of furniture the housa.
You will also be ven all the dirretent types of objects that you can vse, You can specify multiple instances en abfest type. Make sure instruction inc includea the object types and furn'ture presert in the Hst below.
The house has the following rooms, each with the following furniture: [house_ [house_"urning (urnitura)
You can use the following objects: [objests_list)
llece 1s an example with Lwo Lnstructions:
JSON_OUTPUT;
[1
"initial stase";
(1
jobjeet_tyee";; "Lamp", "hew_many": 1, "furniture _ame=" "table_10",
"region": "oom__ Fh
peojjct_ttpp"" "book", "hew_many";
mamiitture__mme.. "table_
"region": "Living_roon_
(1 "abject_typnn "toy_vehiclo",
"how_many": 2, "furaiture_namee "floor",
"region": "Living_roon_
1),
vabject_type"" "toy_cactus",
"how_many""
"fuzniture_nanee "table_1",
"region": room_
1)
"final state": (1
jobject_typet" "Lamp", "hew_many": 1, "Eurniture_nanee "table_10",
"ragion": "Living_room_ H1,
"abject_typae "book", "how_many":
"furniture_nance name": "shelves_2",
"region": "Living_room__"
1),
(1
"object_type": "toy_vehicle", "how_many"" 2, "furniture_nanee "bed_2",
"region": "bedroom_1"
(1
"hox_many": "Loy_cactus",
"ob
"furniture_name"" "bed_2", "region": "bedroom_
1),
iontt "M need to clean the living room. Move all toys and books to the shelf in the living room.",
41

---

Published as a conference paper at ICLR 2025
"reascn";
invoives moving multiple objects shelé living
room.
F)r
"initial stare":
(1
"abject_Etpe"; "plate",
"how_many"
Mturitturennamed "cabinet_2",
"region": "kitchen_
Fl,
"glass",
"how_mony"s
"furniture_iie _amerree___m_____________ "counter_1",
"region": "kitchen__"
(1
"abject_tyyee type": "fork", "how_many"; 5,
"furniture_nane _ame= "cabinet_F", "region": "kitcher__"
1h
"finol state":
"ob "object_type". "plate",
"hew_many":
"furniture_name", "Lable_8",
"region": "Living_
(1
jejjeect_ype"" "glass", "hex_many": 2,
"furniture name": "cable_8",
"region": roon_"
"ob jooject__ype"" "fork", "hew_many"; "furniture_nanc" "table_8", "region": "Living_room_ 1),
Minstructiont "Felp Place 2 plates and 2 glasees or the table. ", "reascn"; "The includes sementically rich descriptions up table)
table dinner the Hivingroom for people.
1)
with (k) instructions like examples Your output should only be: JSOILOUIPUT: result_list where remuls_lies should be JSoN list with the instructions. bet think through thía carefully, step by step.
Generate
Spatial task generation prompt
You are system that generates tasks robots perform humang. Do not be verbose. Anawer the queation with added qualifications or caveats, Just directly provide the answer in JSON.
You will be given a description of a house with objects and furnitere ard your task is could doing together
provide Instructions that house, using the objects and furniture.
For each Lask, provide the initisl state of objects in the house, the instruction that should be performed, and final state of the objects after the instruction is performed. The inital and final type, the number of objects of that type, their location on a furniture or floor, and e.g., bedroom.
will contain list of dicrianaries, each with object
the region the house
they
Follow the next princigles:
42

---

Published as a conference paper at ICLR 2025
instruction should given perforn part of it. In some cases the task will be done together, in other cases, the human and robot will be coing different tasks.
human doing
2. The initial and final state should contain objects of different Lypes, and sometimes multiple objects of a type.
3. Some of the instructions should semantically rich, particular they should rafer
to claases or groups of objects.
sceps, but high-level. 5. The robot can only "earrange objecta and open containers, the h:man can do more taaks
The instructions shouldn't
detailed explain
e.ger turn on lang, clean plates, fill up pitcher,
6. The instraction should contain clear and least two associated with
the goal.
Ensure that instruct
from other.
8, All the instructions should contain at least one of the spatial specifiers from this
list:
te", "left", "right", "beside", "near", "front", "sids". 9. The instructions should contain diverse actions such as "turn en/otr", *fill", "clear", "set" ezc. and object states such as "clean", "cirty", "opea", "close" etc.
You will given the piaces of furriture You will also be given all the different Lypes of objects that you can use. You can specify mul:iple instances an objest type. Make sure instruction includea the object below.
and furniture present In the 1ist
The house has the fowinw rooms, each with tha following furniture: [house_"urniture)
You can use the following objects: [objecte_liat)
Here is an example with two instructions:
JSON_OUTPUT:
[1
"initial state":
(1
"object_Ltye";; "vase", "hex_many": 1, "furniture_ name": "table_10",
"region": "Loving_room_1" F,,
"objjec_typee "stufTed_toy",
"hew_many":
"furniture " "Elcor",
"region": "bedroam_2"
(1
"abject__type "candle", "hew_many": 1,
"furniture_nati "chest_of_drawers.2"
"ragion": "bedroom_1"
11
"final state":
"object_ttpe": "vase",
"hew_many"s
Fturnituro_namee "shelves_2",
"region": "_iving_room_1"
(1
jeajjetttee""""" "stuffed_toy", "hex_many": 2,
"furniture name": "shelven_2",
"region": iving_room_1"
H,,
(1
jobjeect_yyee "candle",
"hew_many":
"furniture_nane name": "shelves_2",
"region": "'iving_roon_""
F
43

---

Published as a conference paper at ICLR 2025
"instructicn": decorate! stušfed_toy on each side of the vage.", "reason"; "The taak includes apatial constraint specified by "side"." Fhr
vase Ther, candle
"initial stace";
(1
rabject_type"" "plate",
"how_many"; 3, "furniture_name"" "csbinet_2",
"ragion": "kitcher__"
(1
tabject_type"" "glass",
"how_many"
"Eurniture_name": "counte:_1",
"region": "kitchen_
F
"object_Lyyee" "Lork", "hew_many":
"turniture_name"" "cabinet_5",
"region": "kitchen__" 1),
"Final state";
"abject_type"; "plate",
"hox_many": "Eurniture_namee' "table_8",
"ragion": "" H,,
"ubject_typet "glass", "hew nnew_many::
"furniture_name"s "table_8",
"region"; "Living_room__"
(1 "object_type"; "Lork", "hex_many" 2,
"furniture_ name": "table_8", "region": "Loving_room__" 1),
"instruction": "Eelp me set up a table for dinner in the livingroom for people. Place plates glasses on table. should plate",
each
"reason"; task includes semantical spatial constraints specified the word 'next!." 1)
descriptions laet up a table)
Conerate JScK list with [k] instructions like the examples above. Your output should only
JSON_OUTPUT: result_list
where reaul reaut=_list ahould be 530N Hsst with the instructions Let's Uhrough this cazefully, step by step.
Temporal task generation prompt
You are system that generates tacks for robots to pezform with humans. the question with added qualifications
not be verbese. directly provida the anawer in JSoN.
Just
You will be given a description of a house with objects and furaiture and your task is provide inscruct ions for tasks that a robot and human could be doing togecher in that house, using the objects end furniture.
each task, provide the initial should be perforned, and final state of the objecta after the instruction la performed. of dicsionarees, each with object type, the number of objects of that type, their location on fuzniture or floor, and the region of the house wheze they are in e.g., bedroom.
of objacts the hause, the instruction that
inital and
44

---

Published as a conference paper at ICLR 2025
Follow the next principles:
1. The instraction should be given if the human doing the task wanted the rebot perferm par: of it. in some cases the task will be done together, in other cases, the human and robot will be coing difforent tacks.
2. The initial and fina: state should contain objects of different types, and somet ines multiple objects
type.
3. Some of the instruct' ons should be semantical rich, in partfoular they should refer classes oz groups of objects.
4. The instructions shouldn't be detailed and explain all the but the high-level.
5. the robot can only zearrange objects and open containers, the human can do more tasks
e.g.,
lamp, clean plates, fill up pitchar. 6, The instruction should contain clear goal and at least two steps associated with
the goal.
Ensure that Instructiona are diverse from each other.
8. All the instructions should contain tempozel order, specified using one of the words from this list: "after", "then", "before", "finally", "first", for instance: *sill
the kattle and then tusn it ar. After that, bring two cups to the dinning table." The instructions should cortain diverse actions
"turn on/ofr", "fill", "cleaz", "set" ecc. and object states such as "clean", *dirty", "opea", *close" etc.
You will be given all the pieces of furniture in the house.
You will also be all the different types of objects that you specify fy mulziple instances of on abjeat type.
inatruction includes object types furniture present the List
below.
The house the following each with the following furniture: [house_Surniture]
You can use the following dbjects: [objeste_list)
ia example with tratructions:
JSON_OUTPUT:
[1
"initial stose":
ttopett_typp": "Lamp",
"how_man thow_many":
peanniture_namm""" "Lable_10",
"region": rLiving_room_i" H),
(1
"object_type": "book", "how_many": 3,
"furniture_nante "table_1l",
"region": "Living_room_1"
(1
project_typp"" "toy_vehicle",
"how thew_many":
"furniture_nnaee name"; *floor",
"region": r_iving_room_1"
H,
(1
"abject_ttppn type": "toy_cactus", "how_many": 1, "furniture_name": "table_1", "region": "_iving_room_1"
final state":
(1
"ob "ubject_Lyge"; "Lamp",
"hox_many":
"furniture_mame". "t",
"region": roon_"
H
(1
"ob "object_ type"" "book", "how_many": 3, "Eurniture_namee: "shelves_2",
45

---

Published as a conference paper at ICLR 2025
"ragion"; "Living_room__
11,
(1
"ebject_type";; "toy_vehicle", "how_many": 2,
"furniture_niii "bad_2",
"region"; "bedroom_1"
H1,
(1
"ob "Loy_cactus", "hox_many";
"furniture_namer "bad_2", "region": "bedroam_
11,
"instruction": "We need to clean the living room. Move all toys to the bedroon and the books to the shelf. After that, tuzn on the lamp in the living room.", ", "reasen"; "The task includes temporal constraints specifled by 'after'."
1h
"initisl state":
"abject_type": "plate",
"hex_many":
"furniturronaaee'. "cabinet_2",
"region"; "kitcher_1" H
"ob tybpeet "glass", "hew_many":
"furniture_nee "counter_1",
"region": "kitcher_1"
C1
cabject_type""
"hox_many";
"furniture_nannt name": "cabinet_5",
"ragion": "kitcher__"" H1,
"final state": (1
"object_type"; "plate",
"heN_many": 2, "furniture_naaee "table_8*,
"region"; rooning
(I
"abject_ type": "glass",
"hex_many";
"furniturronaaoe" "table_8",
"region": r_oving_room_1"
(1
"abject_ type": "fork", mow__many*:
"furniture_narrt "table_8", "region": iving_room_1"
"instruction"; "selp me set up a First place 2 plates on the table. Then, place glasses and forks next to each plate.",
livingream people.
"reasen"; "The task includes semantically rich descripsions Iset table) and temporal conatraints specified by the words 'fires! and "then".*
Generate a JSON 1ist with (k: instructions like the examples above. Your output should only be: 1SON_OUTPUT: result_list resuls_iis: should Let's think through this carefully, step by step.
1ist instructions.
46

---

Published as a conference paper at ICLR 2025
Large-scale task generation prompt
You are system that generates tasks robots perform humana. Do not be verbose, Anawer the queation with added qualifications or caveats, Directly provide the enswez in J8ON.
You will be givan a description a house with objects and furnitere. You will also be Your job to modify this sample
gäven sample this house.
applicable
Hlee is example:
Task: [ (I
"instruction": "Move kettle jug living zoom
fill the kettle with water, then turn on the kettle. Finally clean the living room table.",
"initisl_ttse""
{1
"number";
"ob "abject_cclasee""
"kottle*
[turniture_names
"table_0"
1
"allowed peged_rreeosst "Loving_room__"
(1
"number": 1, "abject__laasse"""
"Jug"
1, Cturiiture_namess
"Lable_i"
1, "allowed_ r viiinn r
13
JSON JSON_OUTPUT:
"instructicn"; "Move the jug, kettle, tespet, and cup from the dining table the kitchen fill water. Turn on the lamp, dining table.", "initial_stace"; (1
"number": "ob "objject_claaeeee
"jug"
1, "furniture_namee : "table_i"
"allowed regionst "Living_room_
11.
(1)
"number": 1, "abject_clasne classes": "kettle"
1,
"Eurniture "furniture_names"" names"; "table_""
b,
"allowed : room___
H),
"number": 1,
47

---

Published as a conference paper at ICLR 2025
Seeject_classestt "teapot"
1, "Eurniture_names" "table_i"
"allowed regionss" [living_room_
Hh (4
"number"; 1,
"cup"
Trurniture_namess
"Lable_0"
"allowed regions": room_
11,
"number": 1,
"abject_classes": *lamp"
"furniture_names" namess; "table_i"
"alloxeé_regionn
"ooving_room_
1
1)
The house the foliiwing rooms, each with the following furniture: (oouse_"urniture)
You can use the following objects: [abjesta_list!
Hare is the task: [task]
Modify this task generate this house.
tasks, using the rooma furniture
change oojects furniture.
initial final locations objects different.
Include act ions auch as turn on/orf, fill and clean.
Make tasks mil multi-step, constating of more than one object and act on.
Your output should enly be: JSOIUOUIPUT: result_list
where resul m_aul=_list ahould be 350N list with the tasks.
A.14 EVALUATION GENERATION PROMPTS
Here we share the LLM prompts used for proposition generation, temporal constraint prediction, and argument constraint prediction. In each, the task accomplish is described in the system prompt and between 6-13 few-shot examples follow.
Evaluation Generation: Propositions
Sourea: system
48

---

Published as a conference paper at ICLR 2025
You will given instrustion deseribing household description initial staze of the house. You willl defina a list of python functions that must be satiafied for the task to marked complete,
You can call the following funetions:
-is_on__cp (objact__names: top of furniture
any object abjectt_nomas is iis_inside (objact_names: str, furmiture_nane stt) insida of a "urniture
furriture_name:
any object abject_namas
is_in_icom(db [en__iooommoeeedd_nnleeee name_n str, zoom_name: atz)
any object object_namee 18 in
[os_o__flooricbjetttoo
str)
any object in abject_names is on
the floor
is_next_to (ob joctteaaar stx, objects_ár stxi
any object in objecta_a 1a nex: object in object_námea 1s
any object objecta_h
1s_oleen (objees__eess strl
:lean
is__iityllejeec__nnee
any object object_names is
dirty
any object object_rames
filled, like with a Liquid
is_empty (objpttyobjees__amee stz)
any object object_nemes any object object_nemes is any object in object_ramms is
emply
is_powered_ (objecttnmee st:l
powered
is_powered _f (objact_namess str)
off
expressed appears objects list Objects object_namer OR of object ("stuffed_toy__ or stuffed_toy_2"), ("etuffed_toy_1") Surniture_name can be expressed as it appears in the furniture list le.g. "table") or appears the furniture-rsem relation ("table in living_rocm").
Only
Listed
Each function should test single instruction anbiguous such multiple objects could to satisfy function (an CR relationchipl, then include all possible objects. Jefine as many functions as necessary.
relation.
Write each function
line.
It ls essencial to wrap each function in delimiters [FM) and 1/39,
End your
[[ND].
Let ' sea some examples. Suppose
stase is:
objects:
pants__
shirt_]
shirt_2
shirt_
Furniture:
washer_dryen
table
Laundzyrcom
jbjeet-rurnituze-Booo Relations:
pants_l on table in laundryroam
ahf on table laurdryroom
shirt_2 floor laundryroom
Furniture-Roo Rolationa:
washer_dzye in laundzyroom
* table in laundryzocm
Instruction "Put the pants on the tabla" maans
[PN] is_on_top ("pant___"" "table in laundryroon") [/FN]
Instruction
pants in [FN] is_inside ("pants_1", "washer_dryer laundryzocn") [/EN]
Instruction "Put shist in the washes" means
[FN] is Liliiniidi""iitttt or shirt_2 or shirt_3", "vasher_deye: in laundryzoon") [/eX)
instructicn "Put all the shirts in the washer" means
[FK] L__innide""siiir "waaher_drye in laundrysoon") [/58.
[PN] is [neide""shit__"" seshee_ddyee in laundrysoca") [/23]
[rn] in_ins ("shirt_3", "washer_dryes in laundrysocn") [/23
49

---

Published as a conference paper at ICLR 2025
Instruction "Put ghirt on the floor next to the pants" moans
shirt_2 shitt_3") [/FN] [FN] ttt.eettttotshirttt.c or shirt_2 or shirt_3", "pants_i") [/EN]
[PN] ('or
Instruction "Bring shirt to the laundryroon' means
[FR] (oonnooon("ah" rt_1 shirt_2 shirt_3", "laund"yroom") [/FN]
<step> Source:
Objecta:
casserole_0
aqueezer_0
Furniture
cabinet
table
Rooms:
bathroom
hallway
kitchen
living_room
Object Purntturrnteteeeeeeeeeeeet Room Reletions:
casserole_0 cabinet kitchen
aqueezer_ 0 on casinet in kitchen
Furnituze- Relations:
table in bathroom
hallway
table living_:oom
Instruction: *cat the casserale and squeezar from the kitchen and put them table in the Hiving room."
<step> Source: assistant
[PN] c__on_sop("caseerccc
"table in living_room") [/EN]
[HN] is_on_sop it_on_ ("quueeee__", "table in living_room") [/H6]] (RND)
<step>
Source: user
The initial
objects:
toy toy_truck_1 atuffed_toy.
atuffed_Loy_@
Furniture:
table
cabinet
Rooms:
bedroom
other_cocm
living_rcom
object-Furnittn paaect-runnitur--ooo Relations:
toy_truck_]
in bedroom
stuffed_tay_4 floor bedroom
feu_feddtoy__ on floor in other_room
Furnituze Roon. Relations:
x table in living_roea
Instruction: *Bring toy from the hedroom and put it in the living room."
<step> Source: sssistant
[PN] is_in_ (Loy bty, "living_room" [BND]
<atep> Source: user
The initial state is:
Objects:
cushion_1
fork_1
plate_]
Furniture:
bed
couch
50

---

Published as a conference paper at ICLR 2025
Rooms:
bedroom
living_reom
Object Seteet--uunnttene -urniture-Rocn Reletions:
cushion_l couch in _iving_room
fork_1 on couch in rivingrrom
plate_1
living_roon
Furniture-Rose Room Re Relations:
bed in bedroom
couch living_roem
counter in kitchen
Instruction: *Bring the diahes back to the kitchen counter. After that, put the cushion tha
<step> Source: assietant
[F8] ta_on_=op("plata_] "ssunter in kitcher*) [2F8]
o__on_top("forkc *courter kitchen") [/7N] [PN] 1s_on__op [END]
"bed in bedroca") [/BN]
<step> Source: user
The initial state 1s:
jbjtaa::
candle_0
fork_0
[oo__o
Furniture:
table
bedroom
living_rcom
- wwweettttnnituue-eeeooooo kelstiens:
candle_0 table in living_roos
tork_0 on counter in kitchen
foy__ood_ counter
jurniture-Roon Relations:
bedroom
counter in kitchen
table in living_rocm
Instruction:
*Pus the toy food on the table where the candle is, and place the Tozk on the toy
<atep>
Source: asst stant
[FN] ia_on_sop (*coy_ food_0", 0*, "table in living_room") [/FN] la_on_sop ("tork___ *table Living_room") (/EN)
[FN] is_nex=_to (fffokkk "Loy_Lood_0"; [/FN] [END]
<atep> Source: vser
Objecta:
toy_airplan__0
toy_aippaane_
Furniture:
table
Rooms:
hallway
living_room
ject Purniture Relations:
=oy_airplane_0 on table in hallway
toy_ toy_airplane_1
hallway
Purniture-Roon Relationa:
table bathcoom
table hellway
table in living_:oom
instructicn: *gring toy airplane to the living rocm and put it in on the table."
<atep> Source: aasistant
[VN] is_on_cop sisplaeee toy_aixplane_l", "table in living_rocm") I/EN]
51

---

Published as a conference paper at ICLR 2025
[END]
<atep> Source: user
objects:
kettle_0 (empty, dirtyl
Furniture:
Rooms:
living_room
Object-Purnitun
Relations:
kettle_0 on table in living_rocm
toy_airplane__ table in hallway
toy_airplane_1 on table in hallway
Furniture-Room Relotiona:
table in Living_room
instructien: "First, make the kettle is clean. Then, fill the kettle with water and turn it on."
<aLep> Source: assistant
[/PN] [PN] is_filled("etttee.000 [/FN]
[PN]
[vN] is_powered__nn (" ("kettle_t"** [/rN]
[FN] 1a.00__og("toy_aarrlllee or toy_atrplane_ "table in Hiving_romm") [/PN]
<step> Source: user
The initial state 1:: (INIT_STATE,
Instructicn: (INSTRUCTTON)"
<atep> Source: assistant Destination: [PN]
Evaluation Generation: Temporal
Source:
You will givan instruction describing task perforn in house and propesitiens that define whether the task was done successfully. The task instruction may say thas certain propositions should completed before "rinally"); Your job is to write python code that groups the propositfors in the ordar which they muat be completed.
("chen", *afzer",
The proposizions are we'l-defined pyther functions that return a boolear value.
You will be given a list of propasitions where index i corresponds to the ith propesition. To solve the taak, define the
proposition_ardet which groupa propositiona together that can be completed in order, Bach proposition group appearing in proposition must be sstisfied before the group that comes efter it. For example,
proposition [0, 1]
means that propositions 0 and can be completed any order. This example
propesition_ordar =
[0],
[1]
means that the proposition must be completed before preposition This exemple
proposition_orden =
[2]
52

---

Published as a conference paper at ICLR 2025
means that propositions C and 1 can be completed in either order, but proposition must be completed after,
Start by assuning that propositions can be completed in any order. Order matters if the instruction includes time ordering words such "than", "Einally", "after", this case, propositions should a be in multiple groups.
Double check that the Incex for each proposition is included in proposition_order,
<step> Source:
Instruction: *Bring apple the kitchen table, then bring orange kitchan counter,"
propesitions
is top [""pple_1"), ["able_4"]] is_on_top ["orange_i"], ("counter_0"])
<alep> Source: aasistant
proposition_order [0],
[1]
<step> Source: user
Instruction: *Pus the toy vehicle and the water bottle in the living Next, zeturn the dish to the kitchen."
propositions
1s inn (("ooy_o((__"oolll ttuck___c rriiving_room_"")) is is_in_room(( (eroom(""cup_000" 0",, ["Living_reon_""1. reom_]")i, is is_in_room (o (""boxl_2",, ["kitchen"]]
<step> Source: assistant
propesition_order
10, 11, [2]
<atep> Source:
Instructicn: *Puz apple beach entryway. move closet."
propesiticns
is_on_topl (allon_top("appplla. ("apple__"" "spple_1"], ["bench_2")] is_inedde ("broo___ (*closet_d"))
<atep> Source: assistant
propesition_ordar (D, 1]
<step> Source: user
Instruction: "Bring me the toy truck from the bedroom and put 1t in the Living room. Then two apples the kitchen table."
propesitions
is is_in_room( "ttt =roc__=_ck__"), ["Living_rrom_]"]]
is_on_top ("apple_1*), ("table_i"]] is_on_topt ("apple_2*), [*table_1"]]
<atep> Source: aasistant
propesiticn_order =
[0], 11, 2J
53

---

Published as a conference paper at ICLR 2025
<atep> Source:
user
Instruction: "Bring the clahes back to the kitchen counter, After that, put the cuahlons the bed.
propesitiona
["plate__]__] (*counter_]* 1", "counter 2", "counter_3" 333nnter_3".1 is_on_top ["fork_1" ["counter_]" "counter_ "counter_3"]]
is_on_topi
is_on_top| ["apoon_ [*] (*counter_]] 1", "counter 2", "oounter_3".b
is_on_topi ["cushion_[") ["bed_1, "bed_2"]),
is_on_top ["cushion_1") ["bed_i", "bed_2")),
<step> Bource: aasistant
proposition_oree
[0, 1, 21, 13, 4]
<alep> Source: user
"Bring the diahes back to the kitchen counter, Put the cuatiion on the bed, Then, move the cushions to the kitchen."
Instruction:
propesitions
["counter_1", "counter_2" "counter_3".),
is_on_top ["__on_top(""oor__""" ["counter_l", *counter_2", 2", #counter_3"1], is is_on_top ["cushion_0"), ["bed_i", "bed_2"1), is "s__o_top[[" (Pcushiooooh ("bed_1", "bed_2"]), ["bedrrom_ in_roonrrmoo ("oushion_i"), ("androom_0"]],
is (f__rooom""ussiiou
<step> Source: assistant
orer
10. 1, 2, 31. [4, 5]
<step> Source: user
instruction: "Nove the clothes from the bedroom the washer, After that, Put the aushion on the bed. Finally, put the book in the living room."
propesitions
is_on_top
[___on_tto("hhit______ 1" mrasher_dzyer_111
is_on_top [ ["shirt_2"]]
@rasher_dryer__1
15 ts_on_top1 [[pants_]]] "washer_dryer___ 1]]],
is_on_top [""ushion_i*), ["bed_i", "b)
15 is_in_room ("book_1"), (")iving_room_""))
<step> Source:
proposit order [ [0, 1,
[3),
<atep> Source: user
Instruction: "First, move the spoer and kettle from the kitchen to the living and place them next to each other, Ther, place the toy food in the xitchen cabinet."
propesiticns
is_on_top ["apoon_a"), ("table_1', 'table_2', "table_3', 'table_4', "table_5"]
1s_on_topt [ept[[letlledd "taale_1', 'table_",, "taale_3', "table_ 'table_8")), to_ (o''sLoo spooa_01], ""kettle_0']), is insidee
("sabinet_0"]]
<step> Source: assistant
proposition_ores order 10, 1, 21, [3]
54

---

Published as a conference paper at ICLR 2025
<atep> Source: user
instructien: "iirst, move the phone stard from the bedrocm to the living room and place the table
-amp. Then, the file sorter from the living to the bedrcom and place it on the table rext to the phone stano."
propositions
is_on_top t''phone_staie
[table_l', "Lable_2', 'Labie_3")),
is_next_to ([
['lamp_01)),
is_on_top [[ ['file File__orter__11111 ['table_6"),, 15 is_next_ton (1ft le sorteer___ ("phore_stand_0")
<atep> Source: aasistant
propesiticn_ordet
[0, 1), 3]
<atep> Source: user
instructicn: "Halp next to each other. Then, place
the candle and hand towel to the kitcher Place them
the spatula o-clang
bedside table
other."
propesiticns
is_on_top!! (acooppp"cnnne01111111 :coouneerO')],
is is_on_top ['hand__owel__"11 ['counter_0"]]
is_next_to (("aatt(("addle_"" ['band_rowel_O")]
_s_on_top ['apatula_6"] ("table_6")],
is_on_top ['e clamp_01), ('Lable66']]
is mext_tooo ((l ('spatula__ ('e-clamp_01]
<step> Source: assistant
proposition_orons - [0, 1, 21, [3, 4, 51
<atep> Source: user
Instruction "First, move the dog bowl, then the placemat, and finally plush Loy from living room so
hallway. Place them each other."
propositions
is_on_top dog_bewi__0', ['bench_0']]
is_on_top [ "pppccmat_00)
('aarch_0') 0']],
is_next_tod (('placemat__
['dog_oowl_]
is is_on_topt I plyehhtooy0"" ['bench_ [)),
is_next_to (""pashhtty__fff ('plazemet_0"))
<step> Source: assistant
proposition_order
[0],
[3, 4)
<atep> Source: user
"Nove the toaster and the bread from the pantry to the kitchen and zurn the teaster on, then
Instructien:
kettle."
propesitions
is_in_room_
(al__room((ttsster__000 ('toaster_O"), ('kitchen_O'11, is ro- ('bread_0'], "'Kitchen_''),, is _s__ooered_on (ttt((ttttttrr")
<step> Source: assietant
55

---

on Published as a conference paper at ICLR 2025
propesiticn_com order
[0, 1, 2],
[3]
<atep>
Instruction: "INSTRUCTTOO)"
[PROPOSITIONS;
<atep> Source: assistant Destinations
propesition_order
Evaluation Generation: Argument Constraints
system
perform in house, whether the task was done successfelly. The task is performed by humar: and robot. The instruction may imply constraints that certain groupa propositions satisfied by the same argument or unique arguments. Your job 13 to write pythor code that defines these constrainte.
will given
set propusitions
define
The pcopositiios are well dell-de pythor Lunctions that return boolear. value.
You will be given a list of propasitions where index i corresponds to the ith proposition. To
task, define the varisble tie_constraints, constraints which be empty. The constraints
use are:
Samekegconssraintt
pxoposi-ion_indices List (iat), indices of propositions that this comstraint applies irg_index: (int), indices of arguments that should be natched on
#ifferentArggonssaaintt proposision_indices:: List [int), indices of propositions that this constroint
applies to ang_index: [int', indices arguments that should
llee examples:
SamsArgconssroone ((0, 1', [0,
# that propasitions index and must have matching value in the first argumeat.
safferonttrroonnecoiii (.0, 1;, [0, 0]) means propositions index have di *ferent values in the first argument.
CameAsgConszzaimg ([C, 1,, 11, 1]) means that propositions at index and muat have matching value in the second argument.
constrainta apply
given instruction, just write empty 1sst
<atep> Source: user
Instruction: "Bring an apple and an ozange to a sable the kitchen."
propositions
is_on_topl totpttopppppppeee ("apple_1"), ("table_3*, "table_4"]), Lsp ["ozange_i"), ""Lable_3,, "table_d"])
<step> Source: assietant
tie_constrainta SameArgConetraint (10, 1b, 11)
<step> Source: user
Instruction:
*Bur the tay vehiale in the living and return the dish the kitchen.
propesilions
is is_in_room( (ttooo(:ttco_______ ("Living_coom_1"))
is iso bbowl_"",, ("kitchen")]
56

---

Published as a conference paper at ICLR 2025
<atep> Source: aasistant
tie_constraints
<atep>
Source: user
Instruction: *Place the book on the shelr in the bedroon. Place the picture Trame nex: it."
propositions
1s_on_topi Ltppll ["book_1".,, ["shelves_0r "sheIves_1"" ["]___
is_on_topl (ltpicppr__rraaaaaaaae frame___r____ 0"], ("shelvaa_0) "shelves_]")) is_next_to(( (ttppppeerrreerrree___o ["book_1"])
<atep> Source: assistant
Leecomstzaials -
SamekegConstreint (IO, 11, [1, 11)
<step> Source:
Instruction: *Place candle
table living
propesiticns
is_on_top [["candle_"" "Lable_0", "Lable_2", "Lable_8")) is_on_top ["canéle_""] ""eable_0", "table_2", "table_6")), is_on_top ("cancle_2"), "table_0", "tablo_2", "table_6"))
<step> Source: assistant
tie_constraiats itffereentrrgtonntrrctttt ([O, 1, 21, 11, 11)
<step> Source: user
hedroom the washer, that, cushion the bed. Finally, put the book in the living room." "
Instruction: "Move clothas
propositions
is_on_Lopt con__oopp""ahieeee ["shirt__"] [eeseer_d:yer_11]]
is_on_top (""hirrt_"",
Twasher_dzyes__1
is_on_top ("ppatt_""s" I*],
bissherddyye_111
15 is_on_top ["oushion_1*) ["bad_1", "bod_2")),
is is_in_room ("*sook ("Living_room_1")
<step> Source: assistant
tic_constraints =
<atep> Source: user
Instructien: "IINSTRUCTION"
[PRGPOSITIONS]
<step> Source: assistant Destination: user
tie_constraints
A.15 PROMPTS FOR PLANNING BASELINES
Following prompts were used various planning baselines.
Decentralized Single/Multi Agent ReAct
57

---

Published as a conference paper at ICLR 2025
multi-agent planning problems. The task assigned to you will be situated in a house and will generally involve navigating objecta, picking and placing them different receptacles schieve resrzangement. You strictly follow pay attention to the previous actions taken in order to avoid repeating mistakes, Rooms do need
heirr_hheee_____aayeeee lend header, _d an agent
fornat speci fications and
explored
than
There
same that you You find that that agent has picked relevant objecta is in the process of completing ing parts of the Leak, If that is the case you may want move on to different of the task.
another agent trying
same time.
explored than This means 1: you have explored the living room and have not found the object, then you should explore kitchen, relevant abjact the Iway etc...
need
found, you should explore
lagent__oll__essrippiie
Many calla should Liy different celion.<leot_Aae <latart_header__ite idlaaadeerddddeueeeeeeeeee leaeseeeeeedddeeeeeeeeeeeee header stask: (task_description)
sign something has gore wrong and action In wepplioal_rag_eeee
[wor1d_desaription)
Possible Actions: [tool_descriptions)
is action progress towards completing Return your zesponse in the following format
Thought: <reascning for you taking the next action> <next action 1a Assigned!
an example: Since there objects found should explore room I have explored Explere[<rooe (<room name>] Ansignad!
Thought:
Centralized ReAct
idl beyatems hendee _dd planning tasks. The task assigned to you will be situated in a house and will generally involve navigating objects, picking and placing them different receptacles achieva rearrangement, "hare wf be robot agent (Agent () and humar agert (Agent 1) available solving
system that solves multi agent
Your goal and solve the task, You strictly follow any format fieatiioooo and attention to the previous actions taken in order to avold repeating mistakes.
assign actions
agents
You should try and divide the task between the two agents for efficient task completion. that the human agent wash, clean, fill, pour and on/off devines along with doing objeas rearrangement. However, the robot can only co object rearrangement 1,8., navigating so objects, picking and placing tham.
In the beginsing, you will be provided with the :ask descrigtion and infornation about the rooms plus furniture in each be availsble. Kooms only need to be explored if there is no information available about task-ralevant chjects. to to go to. Also, rooms do not need to be explored more than once. This means if one of your agents has explored the living explore the kitchen, if reaevant object la still noz round, should explore the hallway etc.
for the house. Object information may may
need
exp explored for idant fying which furriture
object, than you
Many calls to the same action in a row are a sign thas something has gane wrong and you should different
You should try complete task if there two jects to be moved should have agent navigate to each objec and then move them to the target location the seme sine.
arount of actions posaible. This means
a previous navigation octioon is still in progreas for an agent, you should reasatgn that action agent observaticns.
succesaful execution is observed the agent's
58

---

Published as a conference paper at ICLR 2025
You continue agents. Once hoth agenta are done, you con outpuz *Cone to ind that the agenta have rinished the task, Output your reaponse about taak completion in the following format.
progress decide
for
Thought: <reasening about why both agenta have complated the entire suceesafullys Donel]
n0 NOT output "Done un completed. I: one of the agent is done with its part of the task, while the other agent is still executing, you assign "Kait action to the agent who doesnt need to act anymere. Flease re-state the tesk description and verify it's completiar before outputting "Dona() [eot_tag] (aaerrtagTaake (input))
are conft dent that the whole task is sucoesafully
(Norld_desaristion)
Possible actions for each agent: (agent_descriptionss]
la the actioon Relurn your response in the following format
agent
progreas completing task?
Thought: <reasoning Ior you taking the next actions
Agent_ 0_Action: <next action call for robet agenz>
Agent_A_Action <next action call for human Aasigned!
llere is an example: Thought:
multiple task-relevent objects to rearreaged,
each agent Nevigate [sob] namels] agent___Acttoon
one of them
Agent_1_Action: Navigate (Koo] name2>] Aasigned! [ss_idddslstatt__eeeeeeee
The agent role description would be one of the following, depending on if the agent played the role of the human or robot.
Agent Role Descriptions
Human Description
You are playing the role of the task giver. This means if the instruction says semething like "You should the object and will wash it", then the other should be movingg the object, and you should washing the it.
Robot Description
You are playing the role of the task receiver. This moans if the instructian an says something Like "You should object and the other agent should wash it
ooject will it", then you should
Below are the tool descriptions. Perceptions tools are only included for the React-Tools agents.
Tool Descriptions
Clese: Used for closing an articulated entity. You munt provide the name of the Example (Close
furniture you to
Exploze: Search a specific zoom by vieiting various zeceptacles or furnitures in that EXACT name
room. The input so the aki provided in the house description. This tool exhaustivly explorea the specified zoom. Example
*oom to be visited. Usse room
rindigentActtomiicad Should be used find current and past agent.
histery other
object/objecte interest. If you weat to find names of objects on specific receptacles or furnitures, please include thas in the query. Example jFineobjecttool [toys on the floor] or Findob Tool [apples]]
Findob:eo:Tool: Used find
name/names
#indRecepsaciefodl: Used to know the exact name of receptacle. A receptacle is furniture entity (like chair, table, bad etc.) where Txample [FindReceptacle"o0 FindRoomTool:
plane object.
kitchen countar])
zoom house. in the house where furniture is placed. Example (FindRoumlooi room which might have something to eat.)
exact
region
59

---

Published as a conference paper at ICLR 2025
entity. must provide name entity you want to novigate to. Example (Navigate (counter__]
Nevigate: Used navigating
Open: Used for opening an articulated entity. You must provide the name the to open. Exemple pick: Used for picking up ar: cbject. You must provide the name of the object to be time. Exarple (Fick [iick[eu_]]] Place: Used for placing an object on tarçet location. You need to provide the nane the object
picked. The agent cannot hold more than of
furniture you
should be placed, spatial relation (*on* or "with'n") describing the relation between the object and furnicore. The ob ject :o be placed muat already be held by the agent (1.e, picked previously). In addition these,
placed,
nome
furniture
zequest place the object can optionally provide a spatial constraints ("next_tor and the name of the reference object. To place next furniture. API tempate (abbe[<abject_to placed onz, capatial_constraiing reference_ob/fe _ojjctt should be set to "None" when necessary.
another object. Pos that
abjest, refarence object <spatial_re.ation <furniture to citerrnno__bjecc22 spatial__onstran
already be the target
and
Rearrange: Used for moving an object from its current lscation to the target location. You need previde the name the object is should be noved, spatial relation (*on" or "within") daseribing the relation between furniture,
moved, the of the furniture where automatioally
speci the Larget furnicure and atlempt to place it, In addicion to these, you can request to place the object near constraints ("next_to") and the name of the zeference jejtc To place object, the reference object must already be on the tazget furniture. API tempate Rearrange hobjeess_tt__b__oooeeet coattaal_constraittt. <referarco_.bjjnt)>) apotial "None" when recessary.
the
object.
you can optionally provide spatisl to
<apetial_relations <furniture
placed on?, and veferenca_ohjact
Rait: Used te make agent stay idle for some time. Exemple (Wait:l) Done:
indicate
agent has finished the task. example (Done
For LLM agents simulating human, additionally actions which modify the state of objects available to be called. For centralized baselines two lists of available actions are provided in the agent description. One for the robot (without state-modifying actions) and one for the human (with state-modifying actions).
Human Only Tool Descriptions
Clean: Used for clean'me ng an object. You need to provide the name of the object to zlean.
Cless: Used for closing articulated entity. You moat previde the name of the furniture you want to close. Example (Close (Close]ches= srrss_oofddwwerrrr
Fill: Used for filling object. You need to provide the name of the object to fill.
-Four: Used for pouring from one container to another. This skill will pour into the
specified container from whichever containar is currantly hold the agent.
PowerOff: Used for turning off powered objeas. You need to provide the name of the object to
Foweron: Used for turning or powered object. You need prov' de the name of the ab object to be turned on.
The world description contains text description all rooms and their contained furniture, along with currently observed objects. Below is example for scene:
World Description Example
Furniture:
badreom_1: floor flacr_bacreon_ bedroon_i, chair_41, 41, chair bed_49, 43, table_54, ofurt__r__dowwrr___ "aaat_oo__rawer___ chest_of_ ppest_of_drawera_P1 chest_o"_orawers closet_l: coorr_coost__,, wardrobe_9r
living_roca_ls floor floor_ lerp__llvvnn___ooe_
chair_13, chair_14, chair_15, chair_16 chair_17, chair_18, chair_19, chair_20, 21, 22, 2ouch_26, couch_28, couch_ cheir_30, s:ool_31, stool_32, table_38, table_39, table_18, table_50, stead_ counter_78
toilet_l: floor lloor_coflet_', to! let_et
bedroon_2: floor_aecroor_,, bed_23, chair_46, chair_4",, table_53,
drawers_______ ohest_of_dravers chest_of drawers_61 beddrcon_3: flocr_ flocr_beirron_3. bed_37, chair_40, ofest_of_drawere,
chest_oi_drawerd
table_77, chest_of_ drawers__s,, 88, cheat of df__werr__ wardrobs_92
theat_of_drasen
bedreon_4: floor_ becro_bed chair_27, bed 51, wardrobe_56, wardcobe_57, chest_of_drowerr__e chest_of chest_of_drseer___e
table_84, table_a6
kitchen_1a "loor_kitchen_] ahelves_2d 24, shelves_25, chair_33, oha'r_34, chair_35, chair_36, cabine__62, cabinet_63, cabinet_64, cabinet_65, cabinet_ countes_67,
counter_68, councer_69, 7o cabinet_?1, cabinet_76, B1, 38,
fridga_90
60

---

Published as a conference paper at ICLR 2025
other_roon__: Floor_other__oom
other_room__2 Floor_ocher_romm
bathroom_ls "loor batthooom_,,, to1let_44 porch_l: Floor_porch_]
entryway_d; #Lcor_entryway_b, table_85
The rollowing turniturea have faccet: cabinet_70 objects:
No objects found yet
For baselines utilizing summary of the scene, after each action, we insert text description of objects in the scene (as perceived by the agent so far).
Scene Summary Example
Thought: Since there objects tound, I should explore yet. will start with the living room, as it is one of the rooms meationed in the task, and it migh: censain the
have not explored
other relevant objects.
[living__oorrt Assi [ her 1
Result: Suscessful execution:
jbjett:: toy_fruits_l; cheat_of_dravera_d1
bedrooa_4
laptep_cover cabinet_59 559 in laundrysocm_1
cup_3: cabinet_63 in kitchen_l http:///oo__lllaattttttoooo__o hat
vase_0: cabinet_55 in
Thought: Now that we have found the locations of the objects, should navigate the kitchen to pick up the vase, it one or the objects mentioned in the task, and 1: lecated in the kitchen.
Navigste(cabinnt_.G. (oot:niddd//ottidd//]=)
hoader >user< lend (ed__headar_id]>
Result: Successiul execution!
Objects: toy fruiftsits_l:
in bedroca_d
laptop_cover_2s cabinet_5t in laundry:oon_1
cup_3: cabinet_65 in kitchen_] vase_0: cabinet_65 in cott
For finetuning, we build the prompt by combining the task description, world description and the previous actions for both agents.
Finetuning Example
Task: I Linished seading my book. Flease move the book to the bedroom table. You move the vase theze too?
Current Environment: [4orld_descriptionn
Previous acstons:
Agent_Action: Eevigate[Lable..
Action Result: Successful execution!
Agent_Acttio: Novigate [vase_0]
Action Result: Succesaful exacutior!
Agent_Actiins Pick[vaae_[]
Actien Result: Successful execution! Agent_Actions kavigate[table
Actien Result: Succesaful execution!
"cher_Agent_Acction Pick [book_1]
agent_Action!: [aonltable_35]
Action Reault: Sucoesaful ion!
Agent_Actionn Navigate [Lable_35]
Actien Result: Successful execution!
Agent_Actions [vase_0,, ar, table_35, none, none]
Actien Result: Successful execution! Agent_Action: Kait(]
Action Result: Succesaful Executior!
Other Beter_Agee___ttte Open[Labie_35)
Other_Agent Oghet__geat_Action (book_, on, table_35, none, none]
agent_Actionn resoo///rrrrrdppeeeeeeee toeesal_token. apatane [] <et>_act>>
61

---

Published as a conference paper at ICLR 2025
A.16 FURTHER ANALYSIS OF FAILURES
We describe here in more detail some of the common failures of our proposed LLM-based planner. To better illustrate the failures, also provide example traces for these.
Failing to maps objects to their corresponding room leads to unnecessary exploration: Some- times objects are detected but assigned to 'unknown' room, when the perception system fails to assign a room. Note that the object and its location is still added to the graph using RGBD obser- vations, but doesn't have the right parent receptacle reasoning problems with the LLM which does not realize that the object is still navigable, In the following example, the LLM keeps exploring rooms looking for the kettle, even though is already in the graph, but in an 'unknown' room.
room in the graph. Such situations lead to
Room Association Failure
jbcts:: 122_clock: 94_bed in bedroom.
126 =2s_phone_stand: 94_bed in bedreom. 126_plont_saueer 65_table 126 =65_ketlle unknown in unkrown
2oom, foom,</eot__ <leot_id ses_iiddessssratt
Thought: I succeasfully explozed the hallway, bus I didn't rind the kettle. Let me try explore the kitchen, it's likely place find [eetcre[[ktchen,
Result: Successful execution!
objects: 122_clock: 94_bed in bedroom.
126 126_phone_ssand, be_ in bedroom. 288_plant_saucet 65_table 126 126_kettle: unknown in unknow room,<jeat_lo bx///tt_dd>>sstrtt....oo harprr/hheoerrddaaasssttttttttte
Thought: I successfully explored the kitchen, bus s:ill didn's find the kettle. let me try explore the bathroom, it's another pessible location.
Explere [bathrocm]
Assigned! ( Susteerrdddddeeeiiiicce l
Result: Unexpected failure! Skill took too long to finish. Objects: 122_elock: 94_bed in bedroom.
128_phone_stand: 94_bed in bedroom.
126_plont_soueerz 65_table in bodroam.
126 =e5_kettle: unknown in an unkrown room, 122_toy_pivplan 67_mirror hatpoo/cl/ottddixxaaattttteaaa
Thought: The exploration of the bathroom timed out I still haven't found the kettle. Since 1 I've already explored the hallway, kitchen, and bathrocm, my next step would be to explora the living to explore the bedroom aqain, as it's a known location and might have missed something. Explore [hedroom] Assigned1 

don't the living room option.
try
Failures in communication between the high-level and low-level perception systems result in planner failures: Our system relies on multiple layers of perception, e.g. high-level perception that tracks the sfate-of-the-wolt and low-level perception for skills to act and inform the planner of successful or unsuccessful execution. We see lower-level perception sometimes provides the LLM the wrong feedback about skill execution. Even if the high-level perception gives the correct information, the LLM is not able to reconcile these two sources of information, and tends to get confused. In the attached trace, for example, the LLM planner believes the agent has both ramekin and potato in hand and tries to place them together on the same receptacle. In our simulation, the agent is only allowed grasp one object time, but the low-level perception does not send this error message back to the planner correctly.
High-Level and Low-Level Perception Mismatch
Thought: : T have auccessful ly ravigated to the 1iving room and car see the ramekin and golato carpet. will now Lay pick up the Pick re_ekamm Ass hanignedt id]><] atart heedeeeeeddedddeeeeee hend_hhader__d]
62

---

Published as a conference paper at ICLR 2025
Result: Successful execution:
88_ramekin: held the agent. zamekin agent's hand (correct high level) 89_pctato: c6_careet in living_rocm.
90_ramakin: w___ceattoo__raveeee 91 cupp "__chest_of_draae b2_bo 24_stcol 93_potato: 56_postec_plant in bedreom,
bedroam.
in. bedroom,
Thought: 1 have successfully picked up the ramekin. Now, 1 will try to pick up the potato that
carpet the living
Pick|89 [89_potato]
Result: Successful execution: agente can not pick up multiple objects; wzong low-level perception and feedback
objects: _i_cup: 32_she in unkrown.
s8_ramekin: held by the 89 89_potato; 76_carpet in living_room. < ast held by the agent bedroom.
held
agent
90_ramekin: 37_chesttoff_revee in bedroom.
91_cupt 13_chest_of_drake 92_book: 21_stcol in kitchen.
93_petato: $6_posted_plant in
Thought: I have successfully picked up the potato. Now, will navigate to the table in the living zoom so place the ramekin and patato.
Navigate (10_chairr
Assigned! sttt_idd//stttt.ebbeee id pusees hnd_eeeder__dd)
Result: Succeasful execution:
<auиmary?
living room, naed place the zamekin and potato the table. Since the cask mentions placing them next each other on the chair in the dining room, will navigate to the dining reom instead. havigate[47_ntt [47_table]
Thought:
successfully nevigated to
Ass Aaalgned!
Result: Successful execution!
<aummary>
Thought: I have successfully revigated to the table in the dining rocm. New, I will place the ramekin
tan
Place (iB__ramekin,,, on, 47_table, sext_to, 89 89_pocato] headerreer_
Result: Dnexpected failure! No valid placementa found for entity 47_tabla, <-- placements found since placement asked for objecs be plazed next-to 69_pctato, while a9_por is actually or 76_carpet. Plannar keeps repeating this placement on different receptacles until
out.
Failing to associate objects over time reduces the ability to do task progress: PARTNR specifically requires agents to reason about task progression and the state the world as objects are moved around by both agents. Even though the robot agent receives observations from both its sensors and human''s 's sensor, object temporal reassociation is hard problem. This leads to complex failures even if the objects only has two bees and toy-truck; however the graph filled with duplicate copies since the agent fails to associate different toy bee detections to the same instance. This leads to the LLM forgetting the context and the goal, resulting in erroneous episode termination as well as increased inefficiency due trying rearrangements over objects that do exist.
correctly identified. For example, in the trace below, the environment actually
Task Progress Failure
Thought: Now that navigated
in Living room, should up
the toy bee.
Pick (113__tty_bbeeeeeee Assigned! /do_ statt_ pulpeer hend__hadde___d)
Result: Successful execution:
objects: 122 122_picture, plorure_ftaame 67, 67_mirror in bedroom.
124_toy_bees unknown
63

---

Published as a conference paper at ICLR 2025
125_coy_bes: O_carpet in living_room.
126_toy_hee:
0_carpet in living_room.
127_toy_beer
unknown in an unknown room.
12e_toy_bee:
53_carpet
room.
129_toy_
53_carpet in living_room.
138_toy_bee:
hold the agent.
13t_toy_beer
78 r8_aheives in l-ving_room,
132_toy_bes:
32_chest_of_drawer
in living_room.
13s_toy_beet
@R_chest_of_drat ofwwwrrdrrerr__.... ir Hovingrrom.
134_toy_beet
19_ty in L_ving_room.
bee_toy_bee: 18_shelves in living_roem.
136 136_toy_bees
unknown in an unknown room.
132 122_toy_bea: 18_ahelves l-ving_room.
133 139_toy_beer 78 18_ahelves in living_room.
cup_ 190_chest_o__draww
14t_toy_bee: W__chest_or_draae drawerr it Hiving_room.
142_toy_bes: 32 3f_cheet_of_drawez in living_room.
143_picture_iram::
__chest_of_draver
bedrocm.
144_pictura_$trme:
4_chest__o__draee _r_werr in badrocm. in
14b_plcture_"ram:
145_picluze_.tamet unknown in aa uzkaowa room.
147_toy_bes: 19_sv liviag_rocm.
148_toy_bee:
32_chest_of_drawee
in living_room.
149_toy_kes: scchest_of_drwee. in living_room.
150_toy_bes:
0_carpet in living_roon.
151 151_toy_bee: 3__chst__o__rraer
ir living_room room.
192_toy_bee: 0_oarpet in living_room.
bse_toy_bes: 0_carpet in living_:com.
154_toy_bes: 32_chest_of_dreeee
in living_room,
1st_toy__ire__tuekk 78_shelves in living_room.
150_toy_ 19_tv in living_rocm.
156_toy_firo_trruk held by the other
[gont.<lot_tddd>([sttt hildddd/startteeeeaeeaee agent.<leot
Thought: successlully picked up toy the the living room. By task is now complete. will end the simulation here. [one
64


Reviews:
[{"review_id": "0YBHDFrGJ4", "reviewer": "Reviewer_63mf", "summary": "This paper presents PARTNR, a benchmark for human-robot collaboration. It focuses on evaluating LLM high-level planning. \n\nThe paper positions itself against language-based embodied AI benchmarks that don't have interaction, embodied multi-agent benchmarks that have simple, short-horizon environments and closed task sets, and generated-task benchmarks that lack its scale, breadth, and simulator-in-the-loop. It also reviews LLMs for decision making and highlights methods it then uses as baselines.\n\nIt then describes the design elements of PARTNR:\n- Four task categories with different types of constraints\n- 1k human-verified instructions and corresponding evaluation functions, and a prompting + manual annotation pipeline that then leads to 100k total task-evaluation function pairs\n- Methods for generating all of these \n- Dataset train, validation, and test splits \n\nFinally, the paper goes through significant experimentation investigating the nature of PARTNR tasks. It tests various LLM planning methods in multiple settings, then discusses various conclusions about the nature of PARTNR tasks - difficult for current methods, legitimately encouraging collaboration and coordination, and doable by humans.", "questions": "- How is \"collaboration data [from] the web\" used? I may have missed this \n- Does performance improve with more sophisticated collaboration strategies? With a more nuanced understanding of the partner's progress? If not, what's the need for learning these skills? \n- How repetitive is the benchmark, fundamentally? Definitely within the 100k, but even within the seed 10k - are there many tasks that require the same exact action trajectories, just with parameters like object ID switched out?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "### Quality\n- Well-designed benchmark. The task types, generation pipeline (especially sim-in-the-loop verification), and comparative experiments are clever and useful \n- Centralized/decentralized baseline is helpful - even though the centralized method wouldn't make sense as a method benchmarked on PARTNR, using it in comparison to the decentralized method does make an important point about the nature of the tasks. In fact it seems more like an ablation than a benchmark, but point being, it's a good experiment showing that collaboration itself (to the degree you can isolate it when considering generative AI) matters for PARTNR tasks.\n\n### Clarity\n- Very well-written paper! \n\n### Originality \nAs far as I am aware, this is the first multi-agent benchmark of this scale, and with attention paid to these particular aspects. The use of LLM planning modules simplifies the work in many ways - since the methods being used aren't complicated robotic systems, the failures are more constrained to being in semantic space and likely being related to multi-agent coordination, i.e. the benchmark seems especially likely to do what it claims to do. \n\n## Significance \nThis is a considerable work that will likely enable useful research.", "weaknesses": "### Clarity\n- More details would help in the main text. I realize benchmark papers require a lot of material, but it would help to have some examples of \n  - Human annotation\n  - Action selection trajectories - I want to know what it looks like when there's a perception failure, for example \n- At times, the writing feels like a laundry list. E.g. section 3.3 - starts to feel like a laundry list of details about the benchmark, without clear direction on why we should care. Dependent rearrangements and multi-step rearrangements of the same object are both interesting, but they aren't brought up again in detail\n- Some technical details could use reminder/explanation in main text. E.g.: \n  - \"LLMs are unable to recover from skill failures\" - what exactly is communicated to the LLM here? Is it the new scene graph that doesn't have the intended outcome? What does the failure look like? \n  - LLMs struggle to correct errors such as misclassification, making them sensitive to errors in perception - this seems obvious. I'd want to see a trace to know how that plays out, and what this says about *PARTNR* as opposed to the fact that LLMs can't do visual reasoning \n\n### Quality\nOverall a convincing paper. Some examples in the main text would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FqqJO2SwO4", "reviewer": "Reviewer_Htsk", "summary": "The work is devoted to the creation of a new benchmark for multiagent planning and execution of textual instructions in a home simulator. The authors proposed a pipeline of task generation and success evaluation scripts using LLM. In the first stage, 1K instructions synchronized with the Habitat 3.0 simulator were obtained mainly due to manual markup, and then 100K instructions were obtained by prompting LLama3-70B-Instruct using 1K as examples. A PDDL-like language with predicates and predicate constraints was proposed for evaluating scripts. 1K scripts were manually marked up, which were then used to generate a complete set with an RAG-like mechanism for the same LLama3-70B-Instruct. The well-known ReAct model and the heuristic expert planner were tested on the resulting dataset. It is shown that centralization, full observability, and accurate execution of skills gives the best result, but there is still a significant gap in the coordination of actions and in the grounding of information.", "questions": "1. How serious were ConceptGraph's errors? Can we give an additional metric of its errors in object detection and description to assess whether LLM agents suffer from even small perceptual errors?\n2. What criteria were used by the annotation operators and how was their work checked?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The article is written in clear language, the text is easy to read, and the idea and experimental setting are well described. An important gap in existing datasets for collaborative interaction with a large number of scenes, objects, and a variety of instructions is filled. The resulting dataset will clearly be in demand in the community. The authors use the latest available methods, models, and data: LLama3, Habitat 3, HSSD, OVMM, ReACT, and ConceptGraphs.", "weaknesses": "It is not quite clear why the authors specified “applications to robotics, autonomy, planning” as the primary area. There are no new planning methods or robotics applications in the paper, all the baselines used are known. Clearly, the primary topical area should have been “benchmarks and datasets”, within which the work should be evaluated. It is from this point of view that the shortcomings of the work are listed below.\n\n1. In general, the data generation pipelines proposed by the authors cannot be called new. The use of LLM is also found in the works mentioned by the authors both in terms of the generation of the instruction set itself and in terms of the generation of validation scripts. At the same time, the authors do not overcome the main limitation of these approaches - a large amount of manual work of annotation operators. The contribution of LLM is still only related to data scaling, which still requires time-consuming manual validation. In this sense, the work lacks novelty in the sense of data generation pipelines and stands out only for its volume, which is largely obtained by using previously created diverse scenes and collections of objects: HSSD and OVMM.\n2. Using a simulator to ground concepts from instructions is reasonable, but this idea is not extended to automatic validation of successful execution of instructions in this environment, for which a heuristic planner could be used. Correctness of instructions and evaluation scripts still relies on manual spot-checking.\n3. The authors consider only one single learnable baseline - ReAct, so it is hard to call it a benchmark. There are a huge number of planners for embodied environments, including multi-agent ones [1], with special grounding per scene [2] and with error analysis from lower-level actions [3]. Therefore, the conclusions that the authors draw, based on only one baseline, that this dataset is challenging for state-of-the-art models are not well-founded.\n4. It is not clear whether there were instructions that explicitly stated which agent, and which actions should be performed. Such a set of instructions could have been useful for assessing the model's ability to assign roles to a task.\n5. The active involvement of human experts requires additional evaluation of the quality of the tasks they perform, both in relation to the generation of instructions and in relation to the manual checking of the correctness of scripts. What criteria were used by the annotation operators and how was their work checked?\n\n[1] https://ieeexplore.ieee.org/abstract/document/10504634/ \n[2] https://arxiv.org/abs/2310.13255 \n[3] https://arxiv.org/abs/2307.00329", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q3mdMyUPpv", "reviewer": "Reviewer_akQq", "summary": "This paper presents a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR), designed to measure collaboration between humans and robots in a household activity setting. The benchmark, semi-automatically generated with the help of LLMs, covers four categories of tasks: (1) constraint-free, (2) spatial tasks, (3) temporal tasks, and (4) heterogeneous tasks. Each task in the benchmark includes instructions and evaluation functions. Across the generated 100,000 tasks, the instructions achieved a 90% accuracy rate, evaluation functions reached 92%, and the overall task validity rate was 83% (90% × 92%). In the final benchmark tests on current mainstream models, it was found that fine-tuning a small model can achieve performance comparable to models nine times larger, while being 8.6 times faster at inference. Additionally, the findings reveal that LLM-guided partners decrease human efficiency compared to working solo.", "questions": "1） The method ultimately generated 100,000 tasks, with a reported success rate of 83%, but it seem unclear whether the unsuccessful tasks are labeled?\n2） Additionally, is there any tendency in the types of tasks generated by this method? Robots may excel in certain task categories, so if the types of tasks generated are those that robots are particularly skilled at—or, conversely, struggle with—would this affect the realism of the test results? In other words, do these tasks' distributions realistically reflect the types of tasks humans encounter in daily life?\n3） Can the framework generate task patterns that are related to the robot characteristics?\n4)   The author mentioned that the fine tuned model performs better than the large model. But the author trained the model on a specific dataset and tested it on the same dataset. Is this possible to cause overfitting？", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "The semi-automatic benchmark generation method proposed in this paper is highly insightful and offers a potential solution to the limited data issue in the field of embodied intelligence. The process for generating test tasks is described in detail, with a comprehensive explanation provided in the appendix, showcasing an ingenious approach to task generation. Additionally, the experimental results validate mainstream models, demonstrating the practical applicability of this benchmark and highlighting the considerable room for improvement in current models.", "weaknesses": "The success rate of the tasks generated by this method remains relatively low. Although human-guided data refinement was used to clean the tasks, the effectiveness rate is still only 83%, which could lead to complications in practical applications. Furthermore, the paper does not seem to address how to identify the unsuccessful tasks within the dataset. This issue may interfere with measuring embodied intelligence's success rate in tasks. Since the primary contribution of this method lies in generating large volumes of data, if manual filtering is still required to address task inaccuracies, the overall value of this method may be called into question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f8YJ3rqcwn", "reviewer": "Reviewer_LbNu", "summary": "This work proposes a benchmark for planning and reasoning tasks in human-robot collaboration, designed to study human-robot coordination in household activities. Numerous experiments have been conducted, which indicate some weaknesses of current LLMs.", "questions": "1. Do you need to use a human-in-the-loop tool to filter all tasks? How much time does it take?\n2. Is the human in section 4.2.1 using ReAct to simulate? Does every setting in this section use this model to represent a human? Why did you choose this model for simulating human behavior? Why do you think that LLMs should plan for humans?\n3. Is there any analysis of failure cases for task completion?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation is reasonable and meaningful.\n2. I think this is a large-scale benchmark that makes significant contributions to the community due to its open-source nature.\n3. The experiments are sufficient and clear.\n4. The paper is well-rounded, demonstrating a huge amount of work, and is practical, potentially being very helpful for research in this direction.", "weaknesses": "Please refer to the \"Questions\" section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)