PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks

## Abstract

Abstract We present a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration ( PARTNR ) designed to study human-robot coordination in household activities. PARTNR tasks exhibit characteristics of everyday tasks, such as spatial, temporal, and heterogeneous agent capability constraints. We employ a semi-automated task generation pipeline using Large Language Models (LLMs), incorporating simulation in the loop for grounding and verification. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks, spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs on PARTNR tasks, across the axes of planning, perception and skill execution. The analysis reveals significant limitations in SoTA models, such as poor coordination and failures in task tracking and recovery from errors. When LLMs are paired with real humans, they require 1.5 x as many steps as two humans collaborating and 1.1 x more steps than a single human, underscoring the potential for improvement in these models. We further show that fine-tuning smaller LLMs with planning data can achieve performance on par with models 9 times larger, while being 8.6 x faster at inference. Overall, PARTNR highlights significant challenges facing collaborative embodied agents and aims to drive research in this direction.

## 1 Introduction

Imagine a domestic robot that collaborates with humans in daily activities using natural language, akin to human-to-human interactions. This scenario requires two key features: the dynamic collaboration between the robot and the human, and the use of natural language for interaction. Current benchmarks in embodied AI typically satisfy one or the other condition; either robots operate in isolation (Shridhar et al., ; Zhu et al., ; Krantz et al., ; Majumdar et al., ), or tasks are not specified in natural language (Yenamandra et al., ; Puig et al., ; Szot et al., ; Jain et al., ). Despite significant progress in the field of embodied AI, there remains a gap in realistic benchmarks that evaluate robots in collaborative settings. To bridge this gap, we introduce Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR), a novel benchmark that evaluates the ability of embodied AI agents to collaborate with humans across a range of household activities in simulated indoor environments (Figure 1).

PARTNR consists of 100,000 natural language instructions paired with tailored evaluation functions, focusing on four task types: (1) constraint-free, where sub-tasks can be completed in any manner by either agent, (2) spatial tasks that contain spatial constraints, (3) temporal tasks that require ordered execution, and (4) heterogeneous tasks that include actions that cannot be completed by one of the agents. Beyond the conventional challenges of long-horizon planning, novel partially observed environments, and large state and action spaces, PARTNR emphasizes the need for effective collaboration dynamics, such as task division and tracking partner’s progress.

Figure: Figure 1: We present PARTNR, a benchmark for planning and reasoning in embodied multi-agent tasks, featuring 100,000 everyday tasks and evaluation functions generated semi-automatically, spanning 60 houses and 5,819 unique objects. We analyze LLM-based planning agents and also provide a human-in-the-loop tool to evaluate how agents collaborate with real humans.
Refer to caption: x1.png

Curating such a benchmark of large-scale, natural language tasks with tailored evaluation functions presents significant challenges. Current benchmarks typically rely on either templated tasks (Shridhar et al., ; Zhang et al., ) or tasks and evaluations crafted by humans (Mandi et al., ; Li et al., ), which can restrict the diversity or the scale of the datasets. To overcome this, we propose a semi-automated generation method using Large Language Models (LLMs) with simulation-in-the-loop grounding. First, a Large Language Model (LLM) generates task and evaluation functions, which are grounded in the objects and furniture of a simulated house. Next, we employ simulation-in-the-loop to filter out hallucinations and infeasible instructions, complemented by human annotation to enhance diversity and accuracy. Subsequently, a set of 1,000 verified instructions and evaluation functions, along with diverse simulation houses, are utilized to guide an LLM through in-context prompting to create 100,000 tasks.

As PARTNR consists of natural language tasks and LLMs have shown strong results in planning (Yao et al., ; Ahn et al., ; Huang et al., ), we explore prompting and fine-tuning LLMs, to assess their effectiveness in collaborative scenarios. We study the effect of observability of the environment (i.e., full or partial), centralized vs. decentralized multi-agent control, learned or privileged oracle robot skills, and different ways of grounding the 3D world information for LLM-based planning. Beyond these automated evaluations with *synthetic* human partners, we also perform evaluations with *real* humans-in-the-loop, with people performing the task alone, with a human partner, or with an LLM-guided robot partner.
Overall, we find that LLMs struggle at coordination, task tracking and dealing with perception and skill errors. While humans are able to solve 93% of PARTNR tasks, SoTA LLMs can only successfully complete 30% under non-privileged conditions. Moreover, in decentralized multi-agent settings, task completion takes 1.3x more steps than single-agent, due to poor tracking of partner actions, resulting in extraneous actions. In contrast, human pairs outperform single humans, in our human-in-the-loop experiments, highlighting potential for improving LLM collaboration strategies. LLMs also struggle to recover from skill failures and perception grounding errors, resulting in lower performance when privileged skills and privileged perception are removed. When comparing model sizes, we observe that a smaller fine-tuned Llama3.1-8B achieves a similar performance to a Llama3.1-70B without finetuning, while being 8.6x faster. This faster inference plays an important role when evaluated with real humans-in-the-loop, where the finetuned model takes fewer steps and offloads more tasks from the human.

In summary, PARTNR enables reproducible, large-scale, and systematic evaluations of embodied agents in a wide variety of collaborative scenarios.
Through systematic evaluation, we reveal critical insights into the current limitations of LLM-based planners, opening interesting future research directions.

## 2 Related Work

Language-based benchmarks in Embodied AI.
A large body of work on language benchmarks in Embodied AI has focused on navigation (Anderson et al., ; Krantz et al., ; Chen et al., ) or Embodied Question Answering (Das et al., ; Majumdar et al., ) which involve navigation and information gathering but do not require agents to modify their environments. Closer to our work are instruction-following benchmarks (Shridhar et al., , ; Puig et al., ; Wang et al., ; James et al., ; Gong et al., ), where agents interact with environments to complete tasks described via language, though the diversity of tasks is limited. In contrast, we leverage LLMs to generate diverse task definitions and scene initializations, and extend them to *multi-agent* settings. The idea of scaling up task generation using LLMs has been explored in a few recent works (Katara et al., ; Wang et al., ; Xian et al., ; Nasiriany et al., ). However, these works tend to focus on single-agent tasks that span relatively short horizons, while we consider long-horizon, multi-agent problems. Table 1 compares relevant benchmarks with PARTNR.

Table: Table 1: Comparison to similar embodied benchmarks. We compare PARTNR to embodied AI benchmarks, focusing on natural language and multi-agent collaboration tasks. Comparison axes are – Environment: Household single room (S), multi-room (M). Action Space: High-Level Actions (HL), Low-level Actions (LL). Task Types: Constraint-free (C), Spatial (S), Temporal (T), Heterogeneous (H) Num tasks: We measure tasks as the number of unique scene-goal pairs.

Embodied multi-agent benchmarks.
Multiple works have proposed embodied multi-agent benchmarks (Puig et al., ; Agashe et al., ; Zhang et al., ; Jain et al., ; Suarez et al., ). Many of these benchmarks focus on coordination in simple 2D environments, limiting their applicability to real world settings (Agashe et al., ; Carroll et al., ). Recent works have developed benchmarks studying collaboration in more realistic environments and activities (Puig et al., ; Zhang et al., ; Jain et al., ; Puig et al., ; Szot et al., ), focusing on rearranging objects or furniture in large, partially observable 3D environments (Puig et al., , ; Jain et al., ; Szot et al., ), or manipulating objects in a counter-top space (Mandi et al., ). However, these benchmarks are typically limited to a predefined and reduced set of tasks, often not described in natural language and primarily involving object rearrangement. In contrast, PARTNR covers an open set of tasks, each described in natural language, requiring agents to rearrange objects with spatial and temporal constraints, as well as requiring heterogeneous actions that can only be done by the human agent, (e.g., washing dishes or turning on the oven).

LLMs for decision making.
Several works use LLMs as interactive policies, demonstrating their strong capabilities but also highlighting challenges in grounding them with observations and actions (Huang et al., ; Yao et al., ; Ahn et al., ; Huang et al., ; Zeng et al., ). Some approaches improve grounding by prompting LLMs with demonstrations and task-specific constraints (Huang et al., ; Yao et al., ), or by integrating LLMs with external modules for multi-modal reasoning (Ahn et al., ; Huang et al., ; Zeng et al., ).
Toolformer (Schick et al., ) allows LLMs to call APIs for information retrieval or environmental interaction. For instance, APIs can be used to call low-level policies (Driess et al., ), to leverage VLMs for obtaining the state of the world (Huang et al., ; Zhang et al., ), or to another LLM serving as a world model (Zhao et al., ). SayPlan (Rana et al., ) maintains a persistent graph of the current world-state (Gu et al., ; Werby et al., ), enabling detailed semantic and geometric queries. Our work synthesizes these ideas by encoding the environment into a graph, using tools to extract relevant information, and executing tasks through motor skills. Another line of work fine-tunes LLMs with data from the target environments, by learning input and output adaptors (Li et al., ; Szot et al., ; Xiang et al., ). We explore low-rank adaptation of LLMs with multi-agent data to enhance coordination and efficiency. While fewer studies focus on LLMs in multi-agent collaboration (Zhang et al., ; Park et al., ; Li et al., ; Zhou et al., ), one notable example is CoELA (Zhang et al., ) that design collaborative agents, though limited in task diversity. Our work addresses a broader range of tasks and considers agents with varying capabilities, pushing the boundaries of multi-agent system collaboration in more complex scenarios.

## 3 Benchmark Generation

Figure: Figure 2: The PARTNR generation pipeline. Task and evaluation generators produce episodes, which are filtered and annotated for correctness. These episodes are then treated as seeds to achieve 100k-scale. Finally, episodes are vetted during human-in-the-loop collection.
Refer to caption: x2.png

Figure: Figure 3: Task and evaluation example. Language tasks have inherent complexity and ambiguity; both of which are supported by the structures of our evaluation functions.
Refer to caption: x3.png

We introduce PARTNR, a benchmark aimed at training and evaluating robots at solving natural language tasks in collaboration with humans. PARTNR covers four types of tasks: (1) Constraint-free tasks, where sub-tasks can be completed in any manner by either agent. For example, "Let’s move all dirty plates to the sink." (2) Spatial tasks that require reasoning about the spatial positioning of objects. For instance, "Let’s place the books on the shelf next to each other." (3) Temporal tasks, where the sequence in which sub-tasks are executed is important. For example, "Let’s remove the candles from the dining table before bringing the plates to the table." (4) Heterogeneous tasks, involving actions that are beyond the robot’s capabilities. For example, "Let’s wash the dishes before putting them in shelves." In scenarios where the robot’s skills do not support washing, completing this task requires reasoning about agent capabilities. Our benchmark consists of natural language instructions and corresponding evaluation functions, both of which are generated at-scale using LLMs. Specifically, we generate 1,000 human-verified instructions and corresponding evaluation functions and use them as in-prompt examples to scale to 100,000 tasks in other scenes with different layouts and objects.
A unique aspect of our automatic generation is the integration of an embodied simulator within the generation loop, which significantly reduces LLM errors such as hallucinations and infeasible actions.

### 3.1 Simulation-in-the-loop task instruction generation

Figure: Figure 4: Distribution of task types in PARTNR. The left plot displays the percentage of tasks with each characteristic. Constraint-free tasks by definition exclude the other types. The top right bars correspond to the dot combination below.
Refer to caption: x4.png

While LLM-based task generation has been studied in literature before (Katara et al., ; Wang et al., ; Xian et al., ; Nasiriany et al., ), these generations are not grounded beyond user-created in-context prompts. In PARTNR, we use a simulation-in-the-loop generation technique to ground the LLM in the environment, agents and available actions. Specifically, we instantiate a simulation environment in the Habitat 3.0 simulator (Puig et al., ), populated with the HSSD dataset (Khanna et al., ), consisting of 60 unique houses and 5,819 OVMM objects (Yenamandra et al., ). The simulated house is parsed into a list of rooms and available furniture, and passed to an LLM, along with all available objects. Using this information, the LLM is asked to generate free-form, viable tasks in the scene, along with an initial scene state description. For example, if the generated task is "Clear dishes from the living room", the LLM should generate an initial scene with multiple dishes in the living room. At this stage, additional objects are also added to the scene to create clutter in the environment. Once generated, the tasks, initial states, and clutter are instantiated in the simulator, and infeasible instructions are filtered. For example, if the house does not have a living room, "Clear dishes from the living room" is invalid. Similarly, if the generated task requires actions not supported by the simulator, such as folding, the task is filtered. Generally, the rate of hallucinations is high, leading to a significant number of episodes being discarded. We observe that after filtering for infeasible instructions, the diversity in generated instructions is typically limited. For example, most of the instructions use the same objects (e.g., dishes) or similar rooms (e.g., kitchen or dining room). To increase diversity of the generated tasks, we manually annotate them to ensure task and object diversity, such as maintaining a balanced distribution of constraint-free, spatial, temporal, and heterogeneous tasks by modifying the instructions to elicit specific characteristics. This process results in 1,000 human annotated and simulation-verified tasks.

Such manual annotation is not practical for large-scale generation. Instead, we leverage the human-annotated 1,000 instructions to scale generation by using them as in-prompt examples. We prompt the LLM with both a house description and an example task, and instruct it to modify the task to fit the new house. For example a task like "Clear all dishes from the living room" is modified to "Clear all toys from the bedroom." This allows us to maintain the diversity of the original annotated instruction set, while ensuring a high likelihood of successful instantiation in the simulator. Qualitatively, we filtered or edited $\sim\!90\%$ of free-form generated instructions and only $\sim\!10\%$ of scaled instructions. We use *LLama3-70B-Instruct* (Dubey et al., ) for all instruction generation. Finally, all tasks go through a human-in-the-loop filtering. In this step, humans attempt to solve the tasks using our human-in-the-loop tool (Appendix A.13) and eliminate physically infeasible instructions that are difficult to detect, such as requiring an object to be at two locations. Figure 2 provides an overview of our pipeline. Details on the generation process can be found in Appendix A.4 and prompts in Appendix A.14.

### 3.2 Evaluation function generation

To determine if an agent successfully completed the instruction "Clear all dishes from the living room", we need an evaluation function that can validate the removal of all spoons, forks, and other dishes from any of the living rooms. However, manually annotating all necessary rearrangements and state changes of a task is time intensive and since all tasks are unique, impractical at scale. Similar to instruction generation, we employ an LLM to create an evaluation function that assesses task completion without requiring any manual annotations. Specifically, we leverage the ability of LLMs to generate predicate-based Python programs using three types of APIs: a list of propositions indicating what relations between entities must be satisfied, a set of dependencies indicating when propositions should be queried, and a set of constraints indicating how propositions must be satisfied. We define an expressive vocabulary of each of these components to afford evaluation of all tasks in the benchmark (e.g., Figure 3). Closely related evaluation systems include defining tasks in PDDL (Ghallab et al., ) or BDDL (Srivastava et al., ). We choose to build a new Python-based evaluation system since neither have the expressivity to evaluate PARTNR tasks while maintaining human and LLM interpretability; for instance, BDDL does not support time-varying evaluation. Since PARTNR tasks have temporal dependencies (e.g. multi-step rearrangement), the input to the evaluation function is the complete sequence of simulator states during task execution.
The evaluation function returns 3 metrics: (1) Percent Complete ($\texttt{PC}\in[0,1]$), the % of propositions that are satisfied w.r.t. constraints, (2) Success ($\texttt{S}\in\{\texttt{True},\texttt{False}\}$), measuring if a task was successfully completed, defined as $\texttt{S}:=(\texttt{PC}=1)$, and (3) Failure Explanation (FE), a human and LLM interpretable language description of why the agents failed to accomplish the task. See Appendix A.5 for details.

We use *CodeLLama-70B-instruct* (Roziere et al., ) for evaluation function generation. Exemplified in Figure 3, producing perfect evaluation functions is non-trivial. The LLM must correctly classify the entire space of possible actions against natural language instructions and the specific simulation environment, which can be quite complex. For example, in Figure 3, the instruction "set the plants on the shelf" refers to "the shelf", but two shelves exist in the room. The evaluation function must allow either shelf while requiring placement of all plants, and finally account for a next-to relation. Any error or missing value in either a proposition or constraint invalidates the evaluation function. Consequently, we observe a large error rate in LLM generation, particularly pertaining to incorrect propositions and temporal sequencing constraints.

To alleviate these inaccuracies, we follow a similar semi-automated procedure to instruction generation. We first generate evaluation functions for the 1,000 human-annotated instructions and perform manual annotation to correct them (Appendix A.6.3). This results in a dataset of 1,000 human-verified instruction and evaluation pairs. Next, we generate evaluations for the scaled 100,000 instruction set. Recall that the scaled instructions are generated by prompting the LLM with an example instruction from the annotated set. We retrieve the corresponding annotated evaluation function and prompt the LLM with it. This is similar to approaches such as retrieval-augmented generation (Lewis et al., ) and improves the accuracy of evaluation function generation from 50% to 92% as measured through manual inspection (Appendix A.6.2). As a final step, we ask human users to solve all PARTNR tasks using our human-in-the-loop evaluation tool (Appendix A.13). All tasks that cannot be solved by humans over 6 retries (3 single-user, 3 multi-user tries) are deemed infeasible, and removed from the dataset. We find that about $90\%$ of instructions, and $92\%$ of evaluation functions from automated generation are accurate, resulting in a combined generation accuracy of $90\times 92=83\%$.

### 3.3 The PARTNR Dataset

The PARTNR dataset comprises of 100,000 episodes in 37 train scenes, 1,000 episodes in 13 validation scenes, and 1,000 episodes in 10 test scenes from the HSSD dataset (Khanna et al., ). After scaled generation, all validation and test set episodes are human annotated for correctness, as well as a 2,000-episode subset of train. See Appendix A.6.1 for correctness analysis of scale-generated episodes. Below, we analyze the characteristics and diversity of this dataset.

Characteristics: As described earlier, PARTNR focuses on four task types: constraint-free, spatial, temporal, and heterogeneous. We show the distribution of these task types in the test split in Figure 4; validation split is similar. PARTNR evaluates collaboration along these axes both independently and jointly. Secondary characteristics of interest include dependent rearranges (e.g., "Place them on the same table") and multi-step rearrangement of the same object (e.g. "Move the cup to the sink, wash it, then place it in the cabinet"). 7% of tasks include dependent rearranges and 6% include multi-step rearrangement. Tasks average 4.7 propositions to be satisfied (indicative of number of steps required to complete tasks). For analysis of linguistic phenomena and more characteristics, see Appendix A.3.

Diversity:
The diversity of tasks in PARTNR is largely enabled by simulation-in-the-loop generation, which utilizes rich HSSD scenes, and the OVMM object set. Consequently, PARTNR tasks reference and require reasoning about 155 unique object types, 20 furniture classes and 13 room types. Note that each instruction, instantiated in each house, brings its own diversity. For example, "move the laptop to the office table", grounds office and table uniquely in each house, as well as different instances of laptop in different instructions. Further discussion can be found in Appendix A.3.

## 4 Experiments and Analysis

Figure: Figure 5: Decentralized architecture. The human and robot agents use a 2-layer hierarchical architecture, with high-level LLM planners that call low-level skills. Both agents build a world graph, updated using observations and actions.
Refer to caption: x5.png

We investigate how state-of-the-art planning and perception methods handle natural language tasks in new environments and coordinate with unseen partners using PARTNR. Since PARTNR consists of diverse spatio-temporal tasks specified in language, we primarily use LLMs in our baselines for planning, and study variants in (1) zero-shot prompting, retrieval-augmented generation or fine-tuning, (2) centralized versus decentralized planning, (3) partially versus fully observed environment, (4) learned versus oracle low-level robot skills, and (5) privileged versus non-privileged perception.

Our experiments are conducted in the Habitat 3.0 simulator (Puig et al., ) with a simulated Spot robot (; Yokoyama et al., ). We adopt a two-layer hierarchical control architecture, similar to (Puig et al., ; Szot et al., ), as illustrated in Figure 5, for the robot and simulated human. At the high level, a planner selects skills from a predefined skill library (e.g., navigate, pick, place, open, close). We also use a textual world graph with a three-layer hierarchy representing rooms, furniture, and movable objects. Each node in the graph stores a semantic category (e.g., kitchen, table or cup), 3D information (e.g., position or bounding box), and states (e.g., clean, powered on). See Appendix A.7 and Figure 10 for details.

### 4.1 Baselines

We evaluate baselines along the following axes:

- 1.
Variations of high-level planner:
•
Heuristic expert: This approach utilizes expert-designed heuristics and privileged information about the task, environment and evaluation function to pre-plan all steps for human and robot based on their capabilities. For instance, both agents might rearrange objects, but only humans perform cleaning, filling, and toggling on/off tasks.
•
Zero-shot ReAct (ReAct): We use ReAct (Yao et al., ) with an API library of functions or *tools* that enable the LLM to take actions. As observation, we provide the LLM with a concise, current world graph description plus a history of actions. The LLM uses this information to choose an action from [ExploreRoom, Navigate, OpenFurniture, CloseFurniture, PickObject, PlaceObject, Wait, Done] for the robot. See Appendix A.16 for prompts and Appendix A.9 for API details (human and robot).
•
ReAct with Retrieval-Augmented Generation (ReAct-RAG): We also evaluate ReAct with RAG (Lewis et al., ) to investigate whether examples of planning on similar tasks improves the performance of ReAct.
We construct a database of planning examples by collecting the successful traces from ReAct from a the 2,000 task training subset (see 3.3). During test time, the most relevant planning trace from the train dataset is selected based on sentence similarity and added to the LLM’s prompt (Pang et al., ; Madaan et al., ).
•
Finetuned LLMs (Finetuned): We also investigate finetuning a smaller LLM (Llama3.1-8B) as our high-level planner, using successful traces from the ReAct baselines (Hsieh et al., ) that use Llama3.1-70B. Using the React-RAG dataset, we split every episode into a sequence of high-level planning actions, filtering for only actions that were executed successfully. For every action, we build an input containing the world-graph and history of actions, similar to ReAct (see Appendix A.10 for more details). We then finetune an LLM to predict the action from the ReAct episode given this input, using a low-rank adapter (Hu et al., ). This model has reduced latency and computational demands, suitable for real world deployment.
All model generations are constrained to only output valid actions on observed objects using constrained generation (Geng et al., ). The constrained generation greatly reduces the hallucinations and ‘grammatical’ errors typical of LLMs. An episode is finished when both agents call Done or reach maximum simulation steps or LLM calls. Refer to Appendix A.9 for details.
- 2.
Centralized versus decentralized planning:
To study the overhead of coordination in multi-agent PARTNR tasks, we compare centralized and decentralized planners. In centralized, a single LLM decides actions for both agents, with complete information about both agent’s states, effectively removing any need for coordination between the agents. In decentralized, each agent is controlled by a different LLM, and each LLM needs to reason about the other agent’s actions.
- 3.
Partial versus full observability: To evaluate if SoTA language models can explore new environments and identify task-relevant objects, we consider a partially observed setting where the planner knows the house’s layout but not the object locations, requiring exploration. This is in contrast to a fully observed setting, where all object locations are known in advance.
- 4.
Learned versus oracle low-level robot skills: We examine the impact of learned neural-network skills versus oracle skills (with privileged simulation information) on overall performance in PARTNR tasks. We create a library of learned skills for pick, place, navigate, open and close actions (Appendix A.8 provides more details) and compare performance with oracle skills.
- 5.
Privileged versus non-privileged perception:
To study perception challenges such as inaccurate detection and approximate localization, we used a non-privileged world graph with modified ConceptGraphs (Gu et al., ), built from agents’ RGBD observations only.
As agents explore and take actions, this world graph is updated using onboard sensing (details in Appendix A.7). In contrast, with privileged perception, this information is available from the simulation.

### 4.2 Results and Analysis

Metrics. We evaluate performance across different settings using four key metrics. First, the *simulation steps* metric measures the number of steps required for agents to complete the task within the simulation environment, serving as an indicator of efficiency. Second, the *success rate* reflects the completion of the task i.e. whether ‘all’ task constraints are satisfied. Given the complexity and long-horizon nature of PARTNR tasks, agents often partially complete the task. To account for this, we also report *percent complete*, which quantifies the ratio of completed task ‘propositions’ (percent complete $=1$ for successful tasks). Lastly, we assess the reasoning efficiency of the planners through the *planning cycles* metric, which counts the number of high-level LLM calls each planner makes in the course of an episode. We cap the maximum planner calls at 50 in all experiments.

#### 4.2.1 Task Performance Analysis

Table 2 presents a comprehensive evaluation of the planning approaches defined in Section 4.1 using the *Llama3.1-70B-Instruct* model (Dubey et al., ) for ReAct baselines, and a finetuned *Llama3.1-8B* base model for the Finetuned baseline. Since PARTNR tasks are multi-agent, we also need a simulated human partner, which we control with a ReAct approach, using *Llama3.1-70B-Instruct*. Our main findings are detailed below.

PARTNR tasks are challenging for LLM-based planners. LLM-based baselines across all observability and controllability conditions perform worse than the privileged heuristic expert, due to errors in task tracking (not completing all steps, performing them in the wrong order, or undoing completed steps), and semantic errors (placing objects on the wrong furniture, or moving the wrong object), indicating a gap in LLM task planning.

LLMs struggle with coordination in decentralized settings.
Decentralized ReAct baselines which do not have privileged access to partner’s intent are significantly slower at task completion than centralized ReAct ($3295$ steps with decentralized-partial in row(e) versus $2298$ with centralized-partial in row(d)). This shows that reasoning about the partner e.g., knowing or inferring partner’s intent can improve task efficiency in PARTNR tasks, and current SoTA LLMs perform poorly at this. Moreover, decentralized ReAct with two agents is even slower than ReAct with a single-agent ($3295$ steps with multi-agent in row(e) versus $2519$ with single-agent in row(a)), indicating that LLMs suffer from a significant coordination “burden". This co-ordination burden is further highlighted in our analysis on *extraneous effort* in Section 4.2.2, where we find that agents end up repeating parts of the task or performing irrelevant actions with much higher frequency in decentralized settings.

LLMs are unable to recover from learned skill failures. When replacing oracle skills with learned skills, the success rate decreases from 0.73 to 0.57 (row(e) vs. row (h)). This decline can be attributed to the higher failure rate and increased number of simulation steps required by learned skills compared to privileged oracle skills. The LLMs struggle to recover from skill errors like failing to pick up an object or performing incomplete exploration, resulting in a lower success rate. Future research could investigate training large models with low-level skills in the loop, enabling them to learn recovery and replanning strategies in the face of such failures.

LLMs exhibit a high degree of sensitivity to errors in perception. When we replace privileged perception with off-the-shelf perception modules, success rate significantly declines (from 0.57 with a privileged, partial world graph in row(h) to 0.30 with Concept-Graphs (Gu et al., ) in row(i)). LLMs rely heavily on accurate world descriptions provided by the world graph and struggle to correct errors such as misclassification (e.g., shelves misidentified as tables) or incorrect room assignments (e.g., a table in the living room mislabeled as being in the bedroom). Multi-modal models like VLMs might be stronger at recovering from such failures, which we leave for future work.

Finetuned 8B model performs on par with a ReAct with a 70B model, while being 8.6x faster.
We find that the finetuned planner with a small 8B model performs on par with ReAct, which uses a much larger 70B model (a 0.73 success rate with the 70B model in row(e), versus 0.70 with the finetuned 8B model in row(g)). At the same time, we find that the finetuned model is 8.6 times faster at inference. This indicates that the finetuning effectively distills task-relevant information from the training set and generalizes to new test tasks. When deployed with humans-in-the-loop, the finetuned model takes fewer steps and offloads more sub-tasks than the 70B model (see Table 3).

**Table 2: Analysis of planner baselines in various settings. We compare performance using simulation steps, success rate and percent complete on the tasks, and the average planning cycles used by the baselines (described in Section 3).**
| Method | Controllability | Skills | Observability | Sim<br>Steps $\downarrow$ | Success<br>Rate $\uparrow$ | Percent<br>Complete $\uparrow$ | Planning<br>Cycles $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| (a) ReAct-Single | Single Agent | Oracle | Partial | 2519.02 $\pm$ 57.48 | 0.73 $\pm$ 0.01 | 0.85 $\pm$ 0.01 | 18.68 $\pm$ 0.33 |
| (b) Heuristic-Expert | Centralized | Oracle | Full | 1260.88 $\pm$ 26.97 | 0.84 $\pm$ 0.01 | 0.94 $\pm$ 0.01 | N/A |
| (c) ReAct | Centralized | Oracle | Full | 1347.43 $\pm$ 33.80 | 0.74 $\pm$ 0.01 | 0.88 $\pm$ 0.01 | 17.49 $\pm$ 0.34 |
| (d) ReAct | Centralized | Oracle | Partial | 2298.13 $\pm$ 61.39 | 0.74 $\pm$ 0.01 | 0.85 $\pm$ 0.01 | 20.73 $\pm$ 0.51 |
| (e) ReAct | Decentralized | Oracle | Partial | 3295.20 $\pm$ 76.27 | 0.73 $\pm$ 0.01 | 0.86 $\pm$ 0.01 | 15.24 $\pm$ 0.31 |
| (f) ReAct + RAG | Decentralized | Oracle | Partial | 3467.47 $\pm$ 82.39 | 0.71 $\pm$ 0.01 | 0.84 $\pm$ 0.01 | 14.75 $\pm$ 0.31 |
| (g) Finetuned | Decentralized | Oracle | Partial | 3228.96 $\pm$ 75.14 | 0.70 $\pm$ 0.01 | 0.84 $\pm$ 0.01 | 12.85 $\pm$ 0.24 |
| (h) ReAct | Decentralized | Learned | Partial | 6494.88 $\pm$ 181.52 | 0.57 $\pm$ 0.02 | 0.76 $\pm$ 0.01 | 22.72 $\pm$ 0.58 |
| (i) ReAct | Decentralized | Learned | ConceptGraph | 12490.80 $\pm$ 208.90 | 0.30 $\pm$ 0.01 | 0.56 $\pm$ 0.01 | 23.84 $\pm$ 0.45 |

#### 4.2.2 Analysis of Collaborative Behavior and Efficiency

Our analysis in Table 2 revealed challenges in LLM collaboration, prompting a deeper investigation into specific collaborative behaviors, explained below and detailed in Appendix A.12 and Table 12.

Robots offload up to 60% of tasks. We evaluate the robot’s ability to offload tasks from the human, measuring the ratio of sub-tasks performed by the robot to the total sub-tasks in successful PARTNR tasks. Despite similar success rates between single- and multi-agent (0.73 vs. 0.74), the robot offloads about 60% of sub-tasks in decentralized multi-agent, reducing human effort (Table 12).

Decentralized agents are prone to performing extraneous tasks.
The agents sometimes end up performing sub-tasks that are not useful for the task such as rearranging an object that is not required by the task or repeating a sub-task already performed by the other agent. To capture such extraneous agent effort, we measure the portion of agent actions that did not increase the percent complete metric i.e., did not contribute to task progress, over the total number of successful agent actions in an episode. We find a 300% increase in extraneous effort in decentralized multi-agent settings compared to single-agent (Table 12), indicating a significant coordination burden.

Temporal and heterogeneous tasks are challenging for LLMs. LLMs struggle in temporal and heterogeneous tasks. Task success drops by $27\%$ for temporal tasks and $20\%$ for heterogeneous tasks compared to constraint-free tasks for ReAct (Table 13). This highlights the limitations of LLMs in reasoning about agent capabilities and following strict ordering constraints.

### 4.3 Human-in-the-loop Evaluation

We build on the human-in-the-loop infrastructure from Habitat 3.0 (Puig et al., ) and adapt it to a server-client architecture, with the server hosted on AWS capable of supporting multiple clients (see Appendix A.13). This allows us to run at-scale evaluation of our tasks with 129 non-expert human participants. We collect single-user and multi-user data on 1000 tasks from the validation and test set using this tool. In the single-user setting, a single participant completes the whole task, by driving the human agent in the simulator via keyboard/mouse controls Figure 14 in appendix shows our HITL interface. In multi-user, two participants complete the task together by controlling a human and a robot agent. The goal of these experiments is to study multi-user dynamics at PARTNR tasks, and see if multiple humans collaborating are more efficient than single human. Finally, we run a human-AI experiment where a human participant collaborates with a robot controlled by an LLM (using the ReAct and Finetuned models from Section 4.1). This experiment aims to evaluate LLM-controlled agents at collaborating with unseen, real humans. Table 3 shows the success rate (SR) and percent complete (PC) of tasks from the validation set in a single-user, multi-user, human-ReAct and human-Finetuned setting. Additionally, we measure the number of steps taken by each approach to complete the task, and the ratio of work completed by the robot i.e., task offloading. We also measure exploration efficiency in human-in-the-loop, by measuring the steps taken to pick the first object, and extraneous effort, indicating actions that were not useful for task completion. These results are summarized in Table 3. Some key findings are below (more results and analysis in A.13.4):

Humans are significantly better than LLMs at PARTNR tasks. In both single and multiple human environments, the success rate achieved is 0.93 on the PARTNR benchmark. In contrast, the ReAct model without any privileged information, achieves a significantly lower success rate of 0.30 (row (i) of Table 2). This highlights a significant gap in the performance of LLMs in planning tasks. Note that the LLM baselines like ReAct and Finetuned achieve a success rate of 0.92 and 0.91 when evaluated with real humans (Table 3), because humans are able to adapt to LLM mistakes. On the other hand, the simulated human in Table 2 is an LLM, which is unable to recover from partner mistakes.

Finetuned LLMs perform better than ReAct when coordinating with real humans. When deployed with real humans-in-the-loop, the finetuned model is faster than ReAct at task completion ($3443$ steps with finetuned versus $4267$ with ReAct). It is also able to offload more tasks from humans than ReAct (26% with finetuned as compared to 16% with ReAct). This reflects that smaller models with faster inference can improve human experience in real-world deployment.

LLMs struggle at coordination, hampering human performance. Despite the Finetuned being faster than ReAct when collaborating with humans, both approaches are slower than the human doing the task alone. In contrast, two humans working together complete the task faster than a single human ($2369$ steps vs. $3046$ with multi- and single-user respectively). This result is in line with the automated evaluation we observed in Table 1, where multi-agent LLMs are also slower than a single-agent LLM. This result further reinforces that LLMs suffer at coordination; while humans are able to coordinate and divide tasks between each other, decentralized LLMs are unable to do so.

LLMs are able to offload tasks from humans. Despite the aforementioned increase in the number of steps for task completion, robots guided by the finetuned model successfully offload 26% of tasks from humans. This indicates that LLMs can still offer assistance when collaborating with real human partners. Nonetheless, there remains significant potential for improvement.

**Table 3: Human-in-the-Loop Evaluation. We evaluate the performance of a 2-person human team and human-LLM teams, comparing them to solo human performance on PARTNR tasks using metrics described in Section 4.1. Additional results and analysis in Appendix A.13.**
| Method | Success<br>Rate $\uparrow$ | Percent<br>Complete $\uparrow$ | Sim<br>Steps $\downarrow$ | Task<br>Offloading $\uparrow$ | Exploration<br>Efficiency $\downarrow$ | Extraneous<br>Effort $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Single-user | 0.93 $\pm$ 0.01 | 0.96 $\pm$ 0.00 | 3046.99 $\pm$ 80.79 | N/A | 2459.22 $\pm$ 26.75 | 0.09 $\pm$ 0.01 |
| Multi-user | 0.93 $\pm$ 0.01 | 0.96 $\pm$ 0.00 | 2369.55 $\pm$ 49.33 | 0.59 $\pm$ 0.01 | 1762.47 $\pm$ 13.99 | 0.15 $\pm$ 0.01 |
| Human-ReAct | 0.91 $\pm$ 0.01 | 0.96 $\pm$ 0.02 | 4267.71 $\pm$ 83.40 | 0.16 $\pm$ 0.01 | 2624.39 $\pm$ 26.05 | 0.12 $\pm$ 0.01 |
| Human-Finetuned | 0.92 $\pm$ 0.01 | 0.96 $\pm$ 0.00 | 3443.33 $\pm$ 61.46 | 0.26 $\pm$ 0.01 | 2164.94 $\pm$ 21.31 | 0.13 $\pm$ 0.01 |

## 5 Conclusion

We present PARTNR, a benchmark for reasoning and planning in multi-agent embodied tasks, featuring 100,000 natural language tasks instantiated in 60 simulated, multi-room houses with 5,819 unique objects. We use a semi-automated LLM-powered pipeline for large-scale instruction and evaluation function generation that uses simulation-in-the-loop grounding. PARTNR exhibits characteristics of everyday tasks, such as temporal and spatial constraints, and allows systematic evaluation of planning approaches. We find a significant gap between SoTA LLMs and human-level performance at PARTNR tasks. While our best LLM baseline only succeeds at 30% of tasks with no privileged information, humans are able to solve 93% of the tasks. Moreover, LLMs face challenges in coordinating with both LLM-based agents and real human partners. Human-in-the-loop evaluations, involving real humans collaborating with LLM-guided robots, reveal that LLM-guided partners decrease human efficiency compared to working solo. This suggests that LLM-based agents require significant improvements to become effective collaborative partners in embodied tasks. PARTNR serves as a challenging benchmark that highlights the substantial limitations of current models.

## Appendix A Appendix

We present PARTNR, a benchmark for reasoning and planning in multi-agent embodied tasks,
featuring 100,000 natural language, everyday tasks. We show at-scale generation of these tasks using LLMs with simulation in the loop for grounding and human in the loop for filtering. We also evaluate several LLM-based planning models on these tasks and highlight avenues for future work. This appendix provides additional details on these contributions and is organized as follows:

A.1 Code and PARTNR benchmark open-sourcing
A.2 HSSD Scene Annotations
A.3 Details and additional analysis on the PARTNR dataset
A.4 Simulation features and prompts for the PARTNR task generation
A.5 The PARTNR evaluation system
A.6 Human annotation and quality assessment for the PARTNR tasks and evaluation functions
A.7 World graph for perception in LLM agents
A.8 Learned low-level robot skills
A.9 Implementation details for ReAct agents
A.10 Details on finetuning LLM agents for the PARTNR tasks
A.11 Additional results
A.12 Analysis of collaborative behavior and efficiency of LLM agents
A.13 Human-in-the-loop (HITL) system and evaluation for the PARTNR tasks and LLM agents
A.14 Prompts for task and evaluation generation
A.16 Prompts for planner baselines

### A.1 Open-sourcing PARTNR Dataset and Codebase

Accompanying this paper, we have released the code and data necessary to reproduce our experiments. Released code includes our PARTNR benchmark tasks, metrics, baseline oracle skills, large planning model framework, and dataset generation utilities. Released data includes extensions of the Habitat Synthetic Scenes Dataset (HSSD)  (Khanna et al., ), generated benchmark task episodes, and model weights for our trained neural network skills and fine-tuned large planning model.

The publicly released codebase accompanying PARTNR depends on the most recent version of the AI Habitat platform (habitat-lab and habitat-sim (v0.3.2)) (Puig et al., ) which it extends to define collaboration tasks and skills.

### A.2 HSSD Scene Annotations

In order to model the space of rich indoor collaboration tasks we propose with PARTNR, we extended HSSD with additional asset authoring and annotation. To enable more realistic indoor object manipulation, we added articulated 3D furniture models such as drawers, cabinets, and appliances. These models were converted from rigid source assets in HSSD and swapped into the scenes. We prepared 60 scenes divided into train, val, and test splits to support our experiments. Each scene is manually adjusted by a human to ensure simulation robustness and minimize potential issues. Furniture is annotated with a set of Receptacles (surfaces which support small object placement such as shelves and drawers) and can be opened and closed by the agents. Receptacles are further filtered contextually in each scene to ensure that the active set is accessible to the agents. Additional annotations include point or marker sets for each furniture, region annotations, and semantic classification of objects. The marker sets indicate either a spread of surface points (for distance/occlusion checking) or the location of key points of interest such as faucets (for cleaning/filling) and handles (for opening/closing) necessary for low-level skill training and oracle skill execution. Region annotations included per-scene region volumes (e.g., kitchen, living-room, bedroom, etc.) for checking and specifying the location of objects and furniture. Semantic annotations indicate the object category or class (e.g. table, chair, cup, toy) to support open language prompt grounding and semantically guided task generation.

### A.3 Dataset Details and Additional Analysis

We expand on the details and analysis of the PARTNR dataset of Section 3.3, including analysis of linguistic phenomena, secondary task characteristics, and the distribution of sampled entities in generated tasks. See Figure 6 for additional task and evaluation examples.

Linguistic Phenomena. In Table 4, we present an analysis of linguistic phenomena manually annotated over 50 random episodes (similar to Ku et al. () and Chen et al. ()). We analyze the following phenomena:

- •
Class Reference: refers to a semantic class of objects, furniture, or rooms in the scene. These references typically, but not necessarily, follow from the classes defined in OVMM.
- •
Instance Reference: refers to a unique object, furniture, or room; this disambiguation between entities of a class is typically achieved by visual description.
- •
Co-Reference: an expression that refers to an entity defined elsewhere in the instruction.
- •
Passive Voice: the instruction is phrased such that the object is receiving the rearrangement or state change; the request is typically asked instead of commanded.
- •
Active Voice: the instruction is phrased such that the rearrangement or state change is to be completed with an object; the request is typically commanded instead of asked.
- •
High-Level Goal Spec: this sets the operating context for the task before the particulars of rearrangement or state change are specified.
- •
Agentic Reference: a reference to one of the agents performing the task. Typically used to incite a suggested task division between the human and robot.

We observe that PARTNR tasks have a high rate of entity class references, such as the table, (6.38/episode), and common occurrences of instance references and co-references. This signals a need for capable natural language understanding, scene grounding, and co-reference resolution. Tasks and sub-tasks are predominantly issued using active voice (92%), but 14% of tasks include at least one occurrence of passive voice. Half of tasks involve a high-level goal specification, which commonly serves to reduce the search space. For example, a task starting with Let’s clean up all the toys in the playroom constrains object search to that room and softly constrains the placement of those objects to locations in which toys would commonly be stored. Finally, agentic references are present in 14% of episodes.

Figure: Figure 6: Additional task and evaluation examples. Expands on Figure 3 to show a task and evaluation containing subset counts and dependent rearranges (right).
Refer to caption: x6.png

**Table 4: Analysis of linguistic phenomena in the PARTNR dataset. $p$ is the % of instructions that contain the phenomenon while $\mu$ is the average number of times the phenomenon occurs within each instruction. A random sample of 50 Test episodes were included in manual annotation.**
| Linguistic Phenomenon | $p$ | $\mu$ | Example |
| --- | --- | --- | --- |
| Class Reference | 100 | 6.38 | The table |
| Instance Reference | 12 | 0.14 | The coffee table |
| Co-Reference | 50 | 0.64 | That, Those, it, … |
| Passive Voice | 14 | 0.18 | Can you bring me…? |
| Active Voice | 92 | 2.40 | Set it on the stool. |
| High-Level Goal Spec | 50 | 0.50 | Let’s tidy up. Move… |
| Agentic Reference | 14 | 0.16 | Do… While I… |

**Table 5: Analysis of secondary task characteristics in the PARTNR dataset. $p$ is the % of instructions that contain the phenomenon while $\mu$ is the average number of times the phenomenon occurs within each instruction. All episodes in Test were included using automatic annotation.**
| Secondary Characteristics | $p$ | $\mu$ | Example |
| --- | --- | --- | --- |
| Subset Count | 1 | 0.01 | Bring two cups… |
| Resolvable Ambiguity | 68 | 1.68 | Move the pants to any chair. |
| Dependent Rearrange | 7 | 0.09 | Place them on the same table. |
| Multi-Step Rearrange | 6 | 0.11 | Cup to sink. Cup to cabinet. |

Secondary Task Characteristics. In Table 5, We present an analysis of secondary task characteristics present in PARTNR as derived automatically from evaluation functions of all episodes in our dataset. We find rare but present occurrences of subset counts, where agents must reason about manipulating a subset of a set of objects (e.g. bring two cups… when more than two cups exist in the scene). We find that every episode contains at least one occurrence of resolvable ambiguity, where an instruction makes an object/furniture/room reference that may be resolved by more than one entity instance. Dependent rearrangements build on this task characteristic; when multiple objects/furniture/rooms can satisfy a sub-task, a task may require that same entity to solve another sub-task. An example of this is the same table being used to solve the placement of both a spoon and a bowl ("bring the spoon and bowl to the same table in the living room."). This occurs in 31% of episodes and the resulting dynamic dependency is a challenging aspect of collaboration. Finally, 6% of tasks include multi-step rearrangement of the same entity. For example, moving a cup to the sink and then to the cabinet after it is washed. Tasks with multi-step rearrangement have two such rearrangements on average.

Distribution of Entities in PARTNR Tasks. In Figure 7, we examine the distribution of task-relevant objects, furniture, and rooms. We define task-relevant objects to be objects requiring rearrangement or a state change, task-relevant furniture to be target furniture for rearrangement, and task-relevant rooms to be target rooms for rearrangement. Object categories are shown in Figure 7(a), furniture categories are shown in Figure 7(b), and room categories are shown in Figure 7(c). The PARTNR dataset contains a long tail of semantic object categories, and within those categories, a wide variety of objects. Thus, for agents to perform well in PARTNR tasks, they must display visual reasoning and collaboration behaviors that generalize across the semantic particulars of a task. The skew of distributions in Figure 7(b) and Figure 7(c) can be understood by the relative occurrences of rooms and objects in HSSD scenes, e.g. there are more tables than couches on average.

Figure: (a) Objects
Refer to caption: x7.png

### A.4 Simulation-in-the-loop Large-Scale Task Generation

In this section, we describe in detail the simulation-in-the-loop task generation pipeline. We follow a 4-step generation pipeline:

- 1.
Simulation-in-the-loop small-scale free-form LLM generation:
In PARTNR, we initiate the process by setting up a simulation environment using the Habitat 3.0 simulator, populated with the HSSD dataset which includes 60 unique houses and 5,819 OVMM objects. This simulated environment is parsed to identify a list of rooms and available furniture. This information, along with a list of all available objects, is then passed to a Language Model (LLM). The LLM utilizes this data to generate free-form, viable tasks within the scene, accompanied by an initial scene state description. For instance, if the task is "Clear dishes from the living room," the LLM would generate an initial scene depicting multiple dishes in the living room. To add complexity, additional objects are introduced to create clutter.
- 2.
Simulation-in-the-loop filtering and annotation.
The tasks and initial states generated by the LLM are instantiated within the simulator. At this stage, tasks that are infeasible due to the layout of the house or the capabilities of the simulator are filtered out. For example, a task like "Clear dishes from the living room" would be discarded if the simulated house lacks a living room. Similarly, tasks requiring unsupported actions, such as "folding," are also filtered out. This filtering process is crucial as it significantly reduces the number of unrealistic or impossible tasks, although it also tends to limit the diversity of the generated instructions. The full list of allowed actions in the simulator are: [open, close, power on, power off, move, clean, fill, pour, pick, place] and related actions. Next, we use manual annotation to diversity of the resulting tasks. To counter the limited diversity resulting from automated filtering, we manually annotate the tasks to ensure a balanced distribution of various types of tasks, such as constraint-free, spatial, temporal, and heterogeneous tasks. This manual intervention involves modifying the instructions to incorporate different objects and settings, resulting in a curated set of 1,000 human-annotated and simulation-verified tasks. This step is essential for maintaining quality and diversity but is impractical for scaling up due to its labor-intensive nature.
- 3.
Large-scale generation.
To scale the generation of diverse tasks without extensive manual effort, we leverage the 1,000 human-annotated instructions as examples in the LLM prompts. By providing the LLM with both a house description and an example task, we instruct it to adapt the task to fit the new setting, such as changing "Clear all dishes from the living room" to "Clear all toys from the bedroom." This approach helps preserve the diversity of the tasks while enhancing the likelihood of successful instantiation in the simulator.
- 4.
Human-in-the-loop filtering.
The scaled tasks undergo a final human-in-the-loop filtering where physically infeasible instructions are eliminated, ensuring the practicality and realism of the tasks. This ensures that tasks such as "Move 4 cups to the dining table", where there are only 2 cups in the scene are removed. Or "First move the cup from the kitchen to the living room, then place a jug in the kitchen, next to the cup", which consists of a physically infeasible instruction.

#### A.4.1 Habitat 3.0 and HSSD Extension

We generate PARTNR using modified HSSD scenes (Khanna et al., ) and the Habitat 3.0 (Puig et al., ) simulator due to its humanoid simulation capabilities and availability of features which support modeling of collaborative tasks as discussed in Appendix A.1.

Our extensions to the Habitat platform include a set of features targeting: object state manipulation (e.g., clean/dirty, powered on/off and filled/un-filled), evaluation of object relative relationships (e.g., next-to, above, within, on-top, on-floor, in-region, etc), and procedural clutter generation utilities enabling generation of valid initial scene contents pre-conditioned on the output of LLM-generated requirements from open-language prompts (see Section 3). For example, in order to evaluate the task of rearranging a tea set from furniture A to furniture B, we must first generate a scene with both types of furniture in accessible locations and a tea set already sitting on or inside of furniture A.

### A.5 The PARTNR Evaluation System

In this section, we formalize the components of the evaluation system, define the resulting metrics, and present the details and prompts used for LLM-based evaluation generation.

#### A.5.1 Evaluation Predicates

We use logical predicates to query the state of objects, furniture, and rooms at the current timestep in the simulator. The evaluation system operates on the resulting binary state values. The details of all logical predicates are in Table 6.

**Table 6: Logical predicates to evaluate the relations of objects, furniture, and rooms in the PARTNR. Predicates exist for measuring rearrangement of objects (category=Rearrange), spatial placements relative to objects or furniture (category=Spatial), and states of objects or furniture (Category=State). For all predicates, $o_{i}\in\{\text{object, furniture}\}$ and $r_{i}$ is a room.**
| Predicate Name | Category | Description |
| --- | --- | --- |
| is_on_top($o_{1},o_{2}$) | Rearrange | $o_{1}$ is considered on top of $o_{2}$ if a downward raycast from any of the $o_{1}$ bounding box corners intersects with $o_{2}$. |
| is_inside($o_{1},o_{2}$) | Rearrange | $o_{1}$ is considered inside $o_{2}$ if a threshold number of opposing raycasts hit the same object. Ray casts are 1.00m and the threshold number is 2. |
| is_in_room($o_{1},r_{1}$) | Rearrange | $o_{1}$ is considered to be in room $r_{1}$ if at least 25% of keypoints (bounding box corners and center) are contained in the 3D region. |
| is_on_floor($o_{1}$) | Rearrange | $o_{1}$ must be within 0.04m vertically of the navigation mesh. |
| is_next_to($o_{1},o_{2}$) | Spatial | The bounding box of $o_{1}$ must overlap vertically with the bounding box of $o_{2}$ and the horizontal L2 distance between bounding boxes is less than or equal to 0.50m. |
| is_clustered($o_{1},...,o_{n}$) | Spatial | Each $o_{i}$ satisfies is_next_to($o_{i},o_{j}$) for some $j\neq i$. |
| is_clean($o_{1}$) | State | $o_{1}$ has affordance cleanable and the state machine indicates that $o_{1}$ is clean. |
| is_dirty($o_{1}$) | State | $o_{1}$ has affordance cleanable and the state machine indicates that $o_{1}$ is not clean. |
| is_filled($o_{1}$) | State | $o_{1}$ has affordance fillable and the state machine indicates that $o_{1}$ is filled. |
| is_empty($o_{1}$) | State | $o_{1}$ has affordance fillable and the state machine indicates that $o_{1}$ is not filled. |
| is_powered_on($o_{1}$) | State | $o_{1}$ has affordance powerable and the state machine indicates that $o_{1}$ is powered on. |
| is_powered_off($o_{1}$) | State | $o_{1}$ has affordance powerable and the state machine indicates that $o_{1}$ is powered off. |

#### A.5.2 Propositions

The primary component of a task evaluation function is a list of propositions. We define a proposition as an evaluation predicate instantiated with argument values. Propositions additionally enable the evaluation of instructions with ambiguous references (“on a table” — which table?) and subset counts (“two spoons” — any two of the n total spoons). Ambiguity is enabled by extending the predicate arguments to lists. Subset counts are enabled by optional arguments number, which defines the subset size, and arg_match, which is a boolean indicating whether all entities in the subset must be satisfied with the same second argument. Suppose we want to evaluate the task “Bring a spoon to the table.”. If we have a single spoon and a single table, the proposition is straightforward:

$$ $\displaystyle\texttt{is_on_top([spoon_1], [table_1])}.$ $$

If we have multiple spoons and need just one (an ambiguous instruction), the following proposition is used:

$$ $\displaystyle\texttt{is_on_top([spoon_1, spoon_2, spoon_3], [table_1])}.$ $$

In the above case, a list is treated as a OR of entities. The same holds for multiple tables:

$$ $\displaystyle\texttt{is_on_top([spoon_1, spoon_2, spoon_3], [table_1, table_2])},$ $$

in which any spoon may be placed on any table. If the instruction specifies bringing two spoons to the table, the number=2 argument is added:

$$ is_on_top( $\displaystyle\texttt{[spoon_1, spoon_2, spoon_3], [table_1, table_2], number=2)}.$ $$

Finally, the instruction may specify that two spoons should be brought to the same table. arg_match=True enables this:

$$ is_on_top( $\displaystyle\texttt{[spoon_1, spoon_2, spoon_3], [table_1, table_2], number=2, arg_match=True)}.$ $$

Propositions are represented as JSON within dataset files and as Python function calls (as shown above) during human annotation for interpretability.

#### A.5.3 Dependencies

Evaluation functions operate over a sequence of simulation states. By default, all propositions in the list of evaluation propositions will be evaluated at every time step. However, many tasks in the PARTNR benchmark involve a required temporal order during execution. Take for example the instruction “Move the cup from the table to the sink. Then, return the cup to the table.” This consists of a multi-step rearrangement where the proposition checking that the cup is on the table is dependent on a different proposition first being satisfied. We define a proposition dependency as the following:

$$ $\displaystyle\texttt{PropositionDependency(proposition_indices, depends_on, relation_type)},$ $$

where the argument proposition_indices is a list of integers indicating the dependent propositions, the argument depends_on is a list of integers indicating the head propositions, and the argument relation_type indicates the condition that the depends_on propositions must take in order for proposition_indices to be evaluated for satisfaction. The following relation types are supported:

- •
after_satisfied: evaluate propositions in proposition_indices after the propositions in depends_on have been satisfied.
- •
after_unsatisfied: evaluate propositions in proposition_indices after propositions in depends_on have been satisfied at some point in the past but are no longer satisfied.
- •
while_satisfied: evaluate propositions in proposition_indices when propositions in depends_on are currently satisfied.

As a concrete example, suppose the instruction is “Move the ball and bat to the kitchen and set them next to each other. Then, move them to the closet.” The propositions for this task would be

$$ $\displaystyle 0$ is_in_room([ball], [kitchen]) $\displaystyle 1$ is_in_room([bat], [kitchen]) $\displaystyle 2$ is_next_to([ball], [bat]) $\displaystyle 3$ is_in_room([ball], [closet]) $\displaystyle 4$ $\displaystyle\texttt{ is_in_room([bat], [closet])}.$ $$

and the dependencies would be

$$ $\displaystyle 0$ PropositionDependency([2], [0, 1], ‘‘while_satisfied’’) $\displaystyle 1$ $\displaystyle\texttt{ PropositionDependency([3, 4], [0, 1], ``after_satisfied '')}.$ $$

The first ensures that the is_next_to predicate is only queried during kitchen placements and the second ensures that placing the ball and bat in the closet is not checked at the start of the task execution, which would be inadvertently satisfied upon scene initialization. Each evaluation function has a (possibly empty) list of such proposition dependencies.

#### A.5.4 Constraints

PARTNR tasks often require constraining how tasks are completed. Examples of this include completing evaluation propositions in temporal order (“Do [x] before doing [y]” and enforcing links between ambiguities (“Fill one of the cups, then put that cup on the table”). We define a set of constraint types that enables evaluating such task complexities:

- •
TemporalConstraint: A directed acyclic graph (DAG) over the indices of propositions that defines the temporal requirement of when propositions should be satisfied. If propositions are satisfied out of order, then the task is marked unsuccessful. The temporal constraint is defined as
$\displaystyle\texttt{TemporalConstraint(graph_edges)},$
where graph_edges is a set of pairwise temporal constraints. For example, an edge “$0\rightarrow 1$” indicates that the proposition at index 0 0 must be satisfied at an earlier time step than the proposition at index $1$.
- •
SameArgConstraint: Requires the argument used to satisfy a proposition to be the same within a pre-determined set of propositions. The same argument constraint is defined as
$\displaystyle\texttt{SameArgConstraint(proposition_indices, arg_names)},$
where proposition_indices is a list of proposition indices to link and arg_names is a list specifying which component of each proposition to link.
- •
DifferentArgConstraint: Requires the argument used to satisfy a proposition to be unique within a pre-determined set of propositions. The different argument constraint is defined as
$\displaystyle\texttt{DifferentArgConstraint(proposition_indices, arg_names)},$
where proposition_indices is a list of proposition indices to link and arg_names is a list specifying which component of each proposition to link.
- •
TerminalSatisfactionConstraint: Some propositions, once satisfied, should remain satisfied at the end of the episode. Others are expected to become unsatisfied, such as in multi-step rearrangements or object state changes of a single object. The terminal satisfaction constraint is defined as
TerminalSatisfactionConstraint(proposition_indices)
where proposition_indices is a list of proposition indices that should be satisfied at time $t=T$ for an episode rollout of duration $T$.

#### A.5.5 Evaluation Metrics

A task evaluation function is constituted of propositions, dependencies, and constraints as defined above. From this data, the task evaluation function serves to determine what percentage of the task is complete for a given human-robot collaboration rollout. From a sequence of simulation states, we evaluate the truth values of each proposition with respect to both the dependencies (when the propositions must be evaluated) and the constraints (how the propositions must be satisfied). The Percent Completion ($PC$) is defined as the ratio of satisfied propositions to the total number of propositions. Success is defined as $S:=(PC=100)$. Both Percent Completion and Success metrics solely evaluate task completion and thus are agent-agnostic. Given that PARTNR is designed to evaluate multi-agent collaboration, we note that these metrics can be combined with duration-based metrics (e.g. simulation steps or time) to measure efficiency; multi-agent aspects like task division and partnered exploration serve to optimize task efficiency. The benefit of an agent-agnostic metric is flexibility to evaluate any number of agents performing the task with respect to the high-level goal.

#### A.5.6 Evaluation Function Generation

Figure: Figure 8: Evaluation function generation pipeline overview. A three-step LLM-generation process produces propositions and constraints during Generate Evaluations. The evaluation function then is saved to a file for optional human annotation during Manual Annotation. Finally, the evaluation function is packed and verified in simulation during Pack & Verify in Sim to ensure that all entities and affordances exist.
Refer to caption: x10.png

This section provides detail on evaluation function generation beyond the overview provided in Section 3. In particular, Figure 8 shows the first two steps of Figure 2 (sim-in-the-loop generation and filtering + annotation) in greater detail as they pertain to evaluation generation. Notably, we take a three step process to generating evaluation functions; first, an LLM generates a list of evaluation propositions which are parsed into a usable format. Then, an LLM infers the temporal constraint over these propositions by predicting topological generations of a temporal graph. For example, the prediction $[[0,1],[2,3]]$ implies that propositions at indices $0,1$ can be completed in either order, and both must be completed before propositions at indices $2,3$. This prediction is stored and evaluated more generally as edges of a directed acyclic graph (DAG). We found that the assumption of topological generations sufficiently expresses the tasks in PARTNR while being simpler for an LLM to generate than a DAG. The third step is predicting argument constraints, in which the LLM is provided the instruction and propositions list and must predict a list of constraints, either SameArgConstraint or DifferentArgConstraint. All LLM queries are performed against CodeLlama70b-Instruct (Roziere et al., ). All prompts used for evaluation function generation are included in Appendix A.15.

### A.6 Human annotation and accuracy assessment for PARTNR data

#### A.6.1 Generation Accuracy for Tasks and Evaluations

**Table 7: Manually-annotated generation accuracy of 100k-scale PARTNR tasks and evaluation functions. Altogether, we find that 83% of episodes are generated without any task or evaluation function errors. Analysis performed on 100 sampled episodes of each task type via PrediViz.**
|  | Correctness (%) |  |  |
| --- | --- | --- | --- |
| Task | Evaluation | Combined |  |
| Constraint-Free | 89 | 97 | 86 |
| Spatial | 87 | 85 | 74 |
| Temporal | 91 | 93 | 85 |
| Heterogeneous | 93 | 91 | 85 |
| Average | 90 | 92 | 83 |

**Table 8: Top three failure modes of 100k-scale task and evaluation generation reported for each task type. Failures of task generation are led by the hallucination of non-existent entities, while failures of evaluation generation are led by incorrect temporal predictions and incorrect predicate functions. Analysis performed on 100 sampled episodes of each task type via PrediViz.**
|  | Task Failures | Evaluation Failures |  |  |
| --- | --- | --- | --- | --- |
|  | Mode | % | Mode | % |
| Constraint-Free | Hallucination | 7 | Incorrect Ambiguity | 2 |
| Already Satisfied | 2 | Incorrect Furniture | 1 |  |
| Contradiction | 2 | - | - |  |
| Spatial | Hallucination | 6 | Incorrect Temporal Grouping | 7 |
| Unresolvable Ambiguity | 5 | Incorrect Predicate (Other) | 5 |  |
| Already Satisfied | 2 | Incorrect Predicate (Room vs Furniture) | 3 |  |
| Temporal | Hallucination | 4 | Incorrect Temporal Grouping | 3 |
| Contradiction | 3 | Incorrect Predicate (Other) | 3 |  |
| Already Satisfied | 2 | Incorrect Predicate (Room vs Furniture) | 1 |  |
| Heterogeneous | Hallucination | 3 | Incorrect Predicate (Room vs Furniture) | 5 |
| Unresolvable Ambiguity | 2 | Incorrect Object/Furniture/Room | 2 |  |
| Contradiction | 1 | Incorrect Predicate (Other) | 2 |  |

It is important for tasks in PARTNR to be solvable by collaboration agents and for the associated evaluation functions to accurately reflect the task being performed. In this section, we analyze the accuracy of 100k-scale PARTNR generation with respect to both of these criteria using the PrediViz tool (Appendix A.6.2). In Table 7, we demonstrate that the accuracy of task generation ranges from 87-93% depending on task type and averages 90%. The accuracy of evaluation generation ranges from 85-93% depending on task type and averages 92%. Combining these numbers yields an overall joint accuracy of 83% for our 100k-scale dataset.
In Table 8, we annotate the failure modes that lead to unsolvable tasks and incorrect evaluation functions. Common task-related failure modes include:

- •
Hallucination. The produced instruction references objects, furniture, or rooms that do not exist in the scene the instruction was generated for. Example: "Move the clothes to the washing machine." produced for an environment that does not contain a washing machine.
- •
Unresolvable Ambiguity. The produced instruction contains ambiguous directives that cannot be reasonably resolved without further communication or a detailed understanding of the task-issuer’s preferences. Example: "Set the table for dinner"; how many place settings are necessary? Should we set the formal dining table or nook table? What cutlery is needed?
- •
Contradiction. The produced instruction involves two or more sub-tasks that cannot simultaneously be satisfied. Example: "Set the scissors on the coffee table. Set the bowl on the counter next to them."
- •
Already Satisfied. The produced instruction dictates sub-tasks that are all already satisfied at the start of the episode. Example: "Move the laptop to the living room and turn it on", when the laptop is already powered on and in the living room.

According to Table 8, hallucinations are the most common failure mode for task generation. While simulation-in-the-loop filtering avoids this issue for evaluation generation, such filtering is inconsistent for tasks; language has a looser grounding to scene entities than statements of propositional logic. For example, a home might not have a formal dining room, but a table in the living room may serve the purpose of a dining table. Moving on to evaluation generation, common failures are as follows:

- •
Incorrect Temporal Grouping. The instruction implies a temporal order among sub-tasks (either explicitly via sequencing words, or implicitly via multi-step manipulations) and the predicted temporal constraint fails to reflect this order over the propositions. Example: allowing propositions to be satisfied in any order for the task "First, return the plates to the kitchen. Then, tidy up the living room."
- •
Incorrect Predicate (Other). The evaluation function uses the wrong predicate function to evaluate the task. Example: using is_powered_on instead of is_filled when the instruction asks to fill the kettle.
- •
Incorrect Predicate (Room vs Furniture). A task specifies that an object should be rearranged to another room, but the propositions overly-constrain the rearrangement to a target furniture for placement. This failure is separate from the one above because it the most common, and it indicates the tendency for the LLM to produce single solution instances rather than reflect the full space of ambiguity. Example: producing a proposition like is_on_top([electronic_cable], [bed]) for the task "Move the electronic cable to the bedroom."
- •
Incorrect Object/Furniture/Room. incorrect entities are selected for satisfying proposition. Example: the instruction calls for rearranging the cushion to a living room table, but table_4 in the proposition is_on_top([cushion_0], [table_4]) exist in bedrooms.
- •
Incorrect Ambiguity. For an instruction that can be satisfied n different ways, the evaluation affords m options for solution, where $\texttt{m}\neq\texttt{n}$. Example: "Move a toy to the kid’s room", where two or more toys exist in the toy bin but the evaluation function does not list out all possible toys for rearrangement.

According to Table 8, the primary failures modes of evaluation generation are incorrect temporal grouping and incorrect predicates. Regarding temporal grouping, we observe that many sub-task require multiple propositions to evaluate. Grouping these propositions consistently within the temporal constraint is a source of error. Take for example the instruction "Set the shirt and pants next to each other on the counter. Then, move them to the dresser." In this case, five propositions will exist; four for placements and one for the spatial relation. The temporal prediction may erroneously link the spatial relation with the dresser placements rather than the counter placements.

#### A.6.2 Visualization of PARTNR Tasks and Evaluation Functions

Figure: (a) Example task #1
Refer to caption: x11.png

LLM-generated PARTNR episodes may contain errors so we evaluate their correctness with human annotators. This is a verification problem in which the generated instruction, evaluation function, and contents of the scene must be compared against each other. See Figure 6 for an example of this in code. To make this process faster, easier, and more accurate, we designed a visualization and annotation system (PrediViz) that illustrates the state of the world and the evaluation function relative to a task instruction (Figure 9). We chose a 2D illustration style to capture the relevant structure in the data:

- •
Rooms. Each room is drawn as a box with a name provided underneath. Names repeat if there are multiple instances of that room category in the scene. We model the rooms as a fully-connected graph of accessibility. We treat rooms as spatially independent and reorder or wrap them as needed.
- •
Objects. We visualize objects as boxes with category names underneath. Each proposition of the evaluation function is assigned a unique color to separate them from other propositions. For consistency, objects are colored by the color of the first proposition they appear in.
- •
Receptacles. Receptacles are furniture. We designed a bespoke set of icons for 25 categories of furniture that can be easily subitized – you can tell a chair is a chair and a table is a table just by glancing at it.
- •
States. Both objects and receptacles have time-varying state (empty/filled, dirty/clean, powered on/off). These states are displayed using motifs for objects and textual labels for receptacles, when necessary.
- •
Affordances. Different furniture afford different object placements. For example, a cup can be placed on top of the fridge or inside it, but only on top of a table. These relationships are annotated by hand at the category-level. Objects are visualized at their initial receptacle/room placements. An associated dark box indicates the target placement of a particular object.
- •
Placements. To represent rearrangement of an object, we draw an arrow from the initial position to the target position. A simple placement is represented by a single arrow. Multiple allowable placement targets (e.g. multiple tables in the room) are represented by multiple arrows. We use solid arrows for AND placements (e.g. "place the doll and the toy truck on the couch") and dotted arrows for OR (e.g. "place the doll or the toy truck on the couch"). We also support for choosing $k$ out of $n$ objects for placement (e.g. "place two out of three dolls on the couch") using a numerical label on the dotted lines.
- •
Temporal Constraints. To visualize temporal constraints, we split the instruction into multiple frames. For example, if the instruction requires us to "place the doll on the couch and the toy truck in the chair, then put the stuffed toy inside a chest of drawers," we create one frame to represent the first half of the statement (i.e. "place the doll on the couch and the toy truck in the chair") and another for the second half (i.e. "put the stuffed toy inside a chest of drawers").
- •
Special Relations. We also illustrate special relations like next_to, same_as, and different_from in the style of an informational legend shown on the side.

Resulting visualizations are wrapped in a web-based annotation tool that affords binary verification and failure model labeling. See Figure 8 for results derived from this tool. We ran a small-scale experimental study (n=22) comparing human annotation using PrediViz to a text-based representation. Using PrediViz, verification was 2.6 times faster, 8% more accurate, and perceived as 24% easier.

#### A.6.3 Human-assisted Correction Annotation

Below is an example task and evaluation function saved in plain text for human annotation. The instruction, propositions, and constraints can be modified as necessary to ensure the task is feasible and that the evaluation function reflects it. Annotators also have access to a file containing the objects, furniture, rooms, and relations thereof to reference during this process.

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Dataset Correction Annotation Trial,
frametitlealignment=]

### A.7 World graph: the perception framework for LLM agents

Scene-graph style hierarchical graphs have been shown to be effective for planning problems Gu et al. (); Agia et al. (); Rana et al. (). Inspired by such prior work, as illustrated in Figure 10, our world graph is a hierarchical multi-edge directed graph with $K=3$ levels for representing the entities in the world. The nodes at first level correspond to rooms in the environment, followed by furniture at second, and objects and agents at third level. The root of this graph is an abstract house-node denoting the environment where tasks are taking place. Apart from the semantic information, each node also stores the 3D location of the entity and its affordance states e.g., clean/dirty, on/off, open/close, etc. The graph can then be serialized or accessed via specialized tools by ReAct policies. Prompts in Section A.16 provide an example of how the world graph is used by the LLMs in our baselines.

Figure: Figure 10: World graph. The high-level policies of the human and robot agents leverage a hierarchical world graph containing information about rooms, furniture, and objects in the environment. This graph is updated based on the observations and actions of both agents.
Refer to caption: x14.png

#### A.7.1 Building and Updating the World Graph in Simulation

The initial graph is built by reading the the ground-truth room region annotations and furniture placements associated with a particular scene. The location of furniture and the region boundaries are used to associate each furniture to a specific room. This creates the initial two tiers, Room and Furniture, of the privileged world-graph. In partial-observability setting this is all the planner gets to start planning. However, in the full-observability setting, object-to-furniture assignments are read from each episode’s initialization information and a third Object tier is added to the world-graph.

Under partial-observability setting during execution, we use the panoptic sensors attached to both agents to detect all visible objects in the current-frame. Then ground-truth simulation information is used to extract the housing furniture or agent for each of these objects, as well as the location, and this new information is added to the maintained graph. For full-observability setting, each frame latest graph is mined from simulator using current information for objects and overwrites the previous world-graph.

It is not guaranteed in our setup that the images will pick up an object that was placed, filled or powered on/off in the previous step. Therefore, in partial-observability setting, we also add deterministic updates to the graph based on previous action and its result, e.g. successful placement triggers deletion of edge between agent who was holding the object and an addition of edge between placed object and the furniture it was placed on per action arguments.

#### A.7.2 Non-privileged Perception using ConceptGraphs

In order to study the dependence of the planner on the underlying world-graph, we also follow a modified version of ConceptGraphs (CG) pipeline Gu et al. () to create the initial layout that is used to initialize our world-graph representation.

Figure: Figure 11: ConceptGraph pipeline. Left to right: Point-cloud of the scene built from a trajectory; All objects detected by ConceptGraphs pipeline after assigning a category-name based on CLIP; Semantic visualization of the apartment with color-coded class categories per furniture
Refer to caption: x15.png

##### Modified Pipeline for Creating ConceptGraphs.

We adapt the original CG pipeline to only use Meta-CLIP models for getting object name and category; using YOLO and SAM for object segmentation; and LLaMA3.1 for annotating room-label and inter-object spatiosemantic relations. Instead of using LLaVA and GPT for getting open-vocab descriptive object names, we use multi-perspective averaged CLIP embeddings of each object to classify its category given our closed object-vocabulary. For the required room-to-furniture relations, we extend CG pipeline by adding another prompt and query Llama3.1 model to assign room-labels to each entity given the categories of 10 closest entities to it.

##### Updating ConceptGraphs.

Like the privileged world-graphs there are two modes of updating a ConcepGraphs-initialized non-privileged world-graph:

- 1.
Observations. Using the same panoptic sensors we extract all the visible objects in current frame for both the agents. We use depth-sensors to extract the point-cloud associated with this object. We use this 3D location to first check if this is a redetection of a previously detected object. We use location, category and whether object-is-being-held-by-agent features to assess if this is a degenerate detection. If this is a new object detection then we use its location along-with bounding-boxes of existing furniture to check if this object is contained-within or on-top-of any of the recognized furniture-pieces. If it is then the world-graph is updated with this node and the edge.
- 2.
Actions. Just like privileged world-graph, we can not guarantee sensors will pick up changed state of an object that is placed, powered, etc. Thus we add similar action based updates to this version as well. A special case is when non-privileged graph is upated by human agent’s action arguments which are grounded in privieleged world-graph and may refer to same physical entity with different given names, e.g. backpack_0 is backpack_153 in non-privileged graph. We use a simple proximity and category matchig heuristic to match human’s arguments to known entities in non-privileged graph, falling back on proximity based matching when no entity of same category are found.

##### Using ConceptGraphs with Simulated Skills

Simulated skills require sim-handles of the placement furniture to snap objects onto them. The furniture in non-privileged graph do not have these sim-handles by design. Thus we come up with a simple proximity and category-name based matching to match a ground-truth furniture entity to a detected furniture-entity. If we can not find any ground-truth entity of the same category close to the detected furniture, we fallback on matching to the closest entity.

##### Prompts and Models used in ConceptGraphs Creation.

CLIP model and pretrained backbone checkpoints: ViT-H-14-quickgelu, metaclip_fcc. Object detector: YOLO checkpoint yolov8x-worldv2. LLM: Llama3.1-70B-Instruct

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Room Annotation Prompt for Modified CG,
frametitlealignment=]

### A.8 Learned low-level robot skills

We provide the implementation details for training the learned low-level robot skills: [Explore, Navigate, OpenFurniture, CloseFurniture, PickObject, PlaceObject]. These skills are based on two types of basic skills: Navigate skill, which outputs robot base control velocity command given robot depth and sensor observations, and Manipulation skill, which outputs joint angles and base control velocity commands to reach the target location given robot depth and sensor observations. We then use these two skills to get the above robot skills.

#### A.8.1 Navigate Skill

We follow the learned low-level robot skill from Puig et al. () to get the skill. We briefly describe the details for completeness. The goal of Navigate skill is to navigate to the target object given the target object location. The observation space includes (1) an arm depth camera ($224\times 171$ with hFOV of $55$), (2) the relative pose of the target object location in 2-dim polar coordinate system. The action space includes (1) the linear and (2) angular robot base velocities with the range of $-10$ and $+10$ m/s.
The reward function is to encourage the robot to move forward while facing the target location with a correct orientation. An additional navigation success reward is given if the robot can navigate close enough to the target object, and a collision penalty is given if the robot collides with obstacles in the scene. Moreover, a slack reward is given to let the robot navigate to the target in as few steps as possible. Finally, the skill is trained with DD-PPO distributing training.

Below is the list of skills based on navigate skill.

- •
Navigate: As described above.
- •
Explore: It is a composed skill that involves calling Navigate skills sequentially given a sequence of navigation waypoints.

#### A.8.2 Manipulation Skill

We follow the work from Puig et al. () to train the skill. We briefly describe the details for completeness. The goal of Manipulation skill is to drive the robot’s arm and base to reach the target object’s location. The observation space includes (1) an arm depth camera ($224\times 171$ with hFOV of $55$), (2) the relative pose of the target object in a 3-dim Cartesian coordinate system, (3) the 7-dim arm joint angles, (4) a binary holding detector, and (5) the relative pose of arm end-effector to the target resting location in a 3-dim Cartesian coordinate system. The action space includes (1) the linear and (2) angular base velocities with the range of $-10$ and $+10$ m/s, (3) the delta arm joint angles applied to the arm with the range of $-5\times 10^{-2}$ and $+5\times 10^{-2}$ (7-dim), and (4) a binary command to grasp or release the object. The reward function is to encourage the arm to move toward the object. In addition, a success reward is given if the robot interacts with the right target object. Moreover, a slack reward is given to let the robot complete the task in as few steps as possible. Finally, the skill is trained with DD-PPO distributing training.

Below is the list of skills based on manipulate skill.

- •
OpenFurniture: Given the drawer handle location, manipulate skill drives the arm and the base to the target. The articulated furniture is opened if the gripper location is close enough to the handle location.
- •
CloseFurniture: Given the drawer handle location, manipulate skill drives the arm and the base to the target. The articulated furniture is closed if the gripper location is close enough to the handle location.
- •
PickObject: Given the target object location, manipulate skill drives the arm and the base to the target. The object is snapped to the gripper if the gripper location is close enough to the target object location.
- •
PlaceObject: Given the target place location, manipulate skill drives the arm and the base to the target. The object is desnapped from the gripper to the target place location if the gripper location is close enough to the target place location.

### A.9 Implementation Details for ReAct Agents

For all experiments, LLM inferrence is performed on two Nvidia A100 GPUs using the gpt-fast inference engine PyTorch (). Inference on LLama-3.1-70B (using tensor parallelism over two A100s), resulted in an average generation speed of 11.43 tokens/s. Each planning step required an average of 52 tokens resulting in a latency of 4.55 seconds per planning step. The average wall time to complete and entire episode (planning steps for both agents and simulation time) was 36.0 minutes. The finetuned model based on Llama-3.1-8B required an average of 0.53s per planning step. For those experiments, simulation time and human agent inference time remained unchanged, giving a final wall time of 25.3 minutes per episode.

All decentralized baselines had a maximum timeout of 50 replanning calls, while centralized baselines had a maximum timeout of 100 replanning calls (to account for the fact that one planner would need to plan for both agents). Additionally all baselines had a maximum timeout of 20000 simulation steps.

#### A.9.1 Skill API Library

Below is a list of the skills available across all baselines.
Agents acting in the robot role do not have access to state-modifying actions (Clean, Fill, Pour, PowerOff, PowerOn). The ReAct agents considered in the main paper do not have access to perception tools (FindAgentActionTool, FindObjectTool, FindReceptacleTool, FindRoomTool). We additionally study ReAct agents that query the environment via those tools, which we name ReAct-Tools.

- •
Clean : Used for cleaning an object.
- •
Close : Used for closing an articulated entity.
- •
Explore : Search a specific room by visiting various receptacles or furnitures in that room.
- •
Fill : Used for filling an object.
- •
Navigate : Used for navigating to an entity.
- •
Open : Used for opening an articulated entity.
- •
Pick : Used for picking up an object. The agent cannot hold more than one object at a time.
- •
Place : Used for placing an object on a target location. This requires the name of the object to be placed, the name of the furniture where it should be placed, spatial relation (“on” or “within”) describing the relation between the object and furniture. The object to be placed must already be held by the agent (i.e. picked previously). Additionally, you can request to place the object near another object. For that you can optionally provide a spatial constraints (“next_to”) and the name of the reference object. To place next to an object, the reference object must already be on the target furniture. API tempate: Place[$<$object_to_be_placed$>$, $<$spatial_relation$>$, $<$furniture to be placed on$>$, $<$spatial_constraint$>$, $<$reference_object$>$]. spatial_constraint and reference_object should be set to “None” when necessary.
- •
Pour : Used for pouring from one container to another. This skill will pour into the specified container from whichever container is currently held by the agent.
- •
PowerOff : Used for turning off a powered object
- •
PowerOn : Used for turning on a powered object
- •
Rearrange : Used for moving an object from its current location to the target location. This requires the name of the object to be rearranged, the name of the furniture where it should be placed, spatial relation (“on” or “within”) describing the relation between the object and furniture. This skill will automatically pick the specified object and move to
the target furniture and attempt to place it. Additionally, you can request to place the object near another object. For that you can optionally provide a spatial constraints (“next_to”) and the name of the reference object. To place next to an object, the reference object must already be on the target furniture. API tempate: Rearrange[$<$object_to_be_placed$>$, $<$spatial_relation$>$, $<$furniture to be placed on$>$, $<$spatial_constraint$>$, $<$reference_object$>$]. spatial_constraint and reference_object should be set to “None” when necessary.
- •
Wait : Used to make agent stay idle for some time
- •
FindPartnerAgentActionTool : This tool will return a summary of the other agent’s actions.
- •
FindObjectTool : Used to find the exact name/names of the object/objects of interest. An LLM will be used to distill relevant objects from the user query. Example (FindObjectTool[toys on the floor] or FindObjectTool[apples])
- •
FindReceptacleTool : Used to know the exact name of a receptacle. An LLM will be used to distill relevant receptacles from the user query. Example (FindReceptacleTool[a kitchen counter])
- •
FindRoomTool : Used to know the exact name of a room in the house. An LLM will be used to distill relevant rooms from the user query. Example (FindRoomTool[a room which might have something to eat])
- •
Done : Used to indicate that the agent has finished the task.

#### A.9.2 Constrained Generation

We follow the procedure described in Geng et al. (), constraining token sampling to only select tokens that consistent with at least one accepting string in the specified grammar. For each call to the LLM we build a grammar which will only accept valid tool calls on observed entities. Below is the base grammar used tool calls for all experiments. For experiments utilizing a summary of the world representation (i.e. ReAct, Finetuned see Section 4.1) the perception tools (FindObjectTool, FindReceptacleTool, etc.) are omitted. The rules for object, furniture, and room are set dynamically based on the agent’s current world graph. This ensures skills are called only for entities that the agent knows exist.
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Tool Call Grammar,
frametitlealignment=]

#### A.9.3 Retrieval-Augmented Generation for ReAct agents

Retrieval-Augmented Generation (RAG) is the method of optimizing LLM text generation by querying an external database. However, there are several challenges to applying RAG in our setup. First, when implementing RAG, it is necessary to provide an external database from which LLMs can retrieve information. However, it is unclear how to effectively generate such a dataset and determine which content is most beneficial for solving the task. Second, once LLMs retrieve the information, it is unknown where to ingest this information into the generation process. To solve the above challenges, we develop an approach inspired by the recent literature that uses an LLM to generate its training dataset to iteratively optimize performance (Pang et al., ; Madaan et al., ). As shown in Fig. 12, we first construct the dataset by collecting the successful traces generated by LLMs for solving the training tasks. Then, during test time, we select the most relevant trace by comparing the sentence similarity between a test instruction and the ones in the dataset. The selected trace is passed back to the LLM’s promppt as a successful example trace. This represents a refinement process that uses its own past success experience to increase the chance of solving downstream tasks. Figure 12 illustrates the high-level idea.

Specifically, we use the PARTNR train dataset to generate ReAct traces (where the human agent is ReAct-Tools and the robot agent is ReAct). In total, we generate 925 traces that successfully solve the tasks to form the RAG dataset. During evaluation time, we use sentence similarity computed by all-mpnet-base-v2 Reimers and Gurevych () to select the most similar instruction to the instruction at hand in the dataset, followed by adding the trace into the ReAct prompt.

Figure: Figure 12: Overview of ReAct-RAG. RAG consists of two steps: (1) We first use a training dataset to obtain traces from LLMs and log traces with task instructions. (2) During runtime, we retrieve the traces by doing dot product on the instructions and obtaining the trace with the highest score, and finally put the trace in the prompt. We ensure that the dataset used for the first step is different from the one in the second step.
Refer to caption: extracted/5961334/figs/rag_3.png

### A.10 Finetuning LLM agents

In this section, we describe how we finetune an LLM to build our Finetuned baseline. We detail the process to generate the data for finetuning our model and the training details below.

#### A.10.1 Data Generation for Finetuning

We train the model using successful traces from the ReAct baseline, which obtains the best decentralized results. In particular, we run this baseline together with a ReAct-Tools human on the training set, and keep the episodes with 1.0 success rate. If an episode reaches a 100% success rate some time during the task, but fails at the end, we keep the actions up until the success step and replace the last action with a Done[] action to finish the episode. This process results in 1,226 episodes. We then split each episode into the sequence of robot actions, and filter out those that resulted in failure. Our training set is constructed by building a prompt for each of the successful actions, as shown in Sec. A.16. This process results in 15,889 training samples. Note that each action prompt contains the current state of the environment and the previous robot and human actions, but filters out the thought produced by the human or robot.

We also explored using the Heuristic-Expert baseline as a data source for finetuning, but we did not observe improvements in the resulting model. Given that this baseline plans using the ground truth evaluation function, we hypothesized it would help distill the natural language task into the right predicates. We followed the same process described above, obtaining 1250 episodes and 13939 training samples. We trained a model with ReAct, Heuristic-Expert and both data sources. We show the evaluation results for each model in Table 9, with the model trained with only React data performing the best.

**Table 9: Performance of Finetuned model when using different data sources for finetuning. We measure performance using simulation steps required to finish the episode, success rate and percent complete on the tasks, and the average number of planning cycles performed by the planner. Mean and standard error are reported over the validation set. Heuristic centralized expert has access to the task evaluation function.**
| Data source | Sim Steps $\downarrow$ | Success Rate $\uparrow$ | Percent Complete $\uparrow$ | Planning Cycles $\downarrow$ |
| --- | --- | --- | --- | --- |
| Heuristic-Expert | 3477.82 $\pm$ 78.19 | 0.63 $\pm$ 0.02 | 0.79 $\pm$ 0.01 | 15.50 $\pm$ 0.24 |
| ReAct | 3228.96 $\pm$ 75.14 | 0.70 $\pm$ 0.01 | 0.84 $\pm$ 0.01 | 12.85 $\pm$ 0.24 |
| ReAct + Heuristic-Expert | 3552.96 $\pm$ 61.95 | 0.69 $\pm$ 0.01 | 0.83 $\pm$ 0.01 | 14.47 $\pm$ 0.30 |

#### A.10.2 Implementation Details

We train the model to predict, for every example, the action taken by the agent, which corresponds to the text after the <|reserved_special_token_0>| token.

We use a low rank adapter Hu et al. () with $r=132,\alpha=128,\textrm{dropout=}0.01$, on top of the value and query projection layers $W^{V},W^{Q}$. We train all models on 4 A100 GPUs, with a batch size of 2 per GPU. The models are trained for 40,000 steps, which takes around 24 hours.

### A.11 Additional Results

**Table 10: Baseline results on PARTNR validation set. We measure performance using simulation steps required to finish the episode, success rate and completion rate on the tasks, and the average number of planning cycles performed by the planner. Mean and standard error are reported over the validation set. Heuristic centralized expert has access to the task evaluation function. Collaboration enables higher task completion and success as compared to single agent task execution (shown in gray), at the expense of more simulation steps highlighting the coordination “burden".**
| Method | Controllability | Skills | Observability | Sim<br>Steps $\downarrow$ | Success<br>Rate $\uparrow$ | Completion<br>Rate $\uparrow$ | Planning<br>Cycles $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Heuristic-Expert | Centralized | Oracle | Full | 1260.88 $\pm$ 26.97 | 0.84 $\pm$ 0.01 | 0.94 $\pm$ 0.01 | N/A |
| ReAct | Centralized | Oracle | Full | 1347.43 $\pm$ 33.80 | 0.74 $\pm$ 0.01 | 0.88 $\pm$ 0.01 | 17.49 $\pm$ 0.34 |
| ReAct | Decentralized | Oracle | Full | 1915.63 $\pm$ 56.84 | 0.74 $\pm$ 0.01 | 0.86 $\pm$ 0.01 | 14.20 $\pm$ 0.34 |
| ReAct | Centralized | Oracle | Partial | 2298.13 $\pm$ 61.39 | 0.74 $\pm$ 0.01 | 0.85 $\pm$ 0.01 | 20.73 $\pm$ 0.51 |
| ReAct | Decentralized | Oracle | Partial | 3295.20 $\pm$ 76.27 | 0.73 $\pm$ 0.01 | 0.86 $\pm$ 0.01 | 15.24 $\pm$ 0.31 |
| ReAct-Tools | Decentralized | Oracle | Partial | 3622.52 $\pm$ 79.09 | 0.71 $\pm$ 0.01 | 0.85 $\pm$ 0.01 | 21.41 $\pm$ 0.34 |
| ReAct + RAG | Decentralized | Oracle | Partial | 3467.47 $\pm$ 82.39 | 0.71 $\pm$ 0.01 | 0.84 $\pm$ 0.01 | 14.75 $\pm$ 0.31 |
| Finetuned | Decentralized | Oracle | Partial | 3228.96 $\pm$ 75.14 | 0.70 $\pm$ 0.01 | 0.84 $\pm$ 0.01 | 12.85 $\pm$ 0.24 |
| ReAct | Decentralized | Learned | Partial | 6494.88 $\pm$ 181.52 | 0.57 $\pm$ 0.02 | 0.76 $\pm$ 0.01 | 22.72 $\pm$ 0.58 |
| ReAct | Decentralized | Learned | ConceptGraph | 12274.27 $\pm$ 212.65 | 0.25 $\pm$ 0.01 | 0.53 $\pm$ 0.01 | 26.74 $\pm$ 0.45 |
| ReAct-Single | Single Agent | Oracle | Partial | 2519.02 $\pm$ 57.48 | 0.73 $\pm$ 0.01 | 0.85 $\pm$ 0.01 | 18.68 $\pm$ 0.33 |
| ReAct-Single | Single Agent | Oracle | Full | 1590.20 $\pm$ 42.73 | 0.73 $\pm$ 0.01 | 0.85 $\pm$ 0.01 | 18.60 $\pm$ 0.38 |
| ReAct-8B | Decentralized | Oracle | Partial | 3699.45 $\pm$ 87.40 | 0.64 $\pm$ 0.02 | 0.80 $\pm$ 0.01 | 23.15 $\pm$ 0.47 |

**Table 11: Baseline results on PARTNR test set. We measure performance using simulation steps required to finish the episode, success rate and completion rate on the tasks, and the average number of planning cycles performed by the planner. Mean and standard error are reported over the test set. Heuristic centralized expert has access to the task evaluation function.**
| Method | Controllability | Skills | Observability | Sim<br>Steps $\downarrow$ | Success<br>Rate $\uparrow$ | Completion<br>Rate $\uparrow$ | Planning<br>Cycles $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Heuristic-Expert | Centralized | Oracle | Full | 1184.74 $\pm$ 22.88 | 0.69 $\pm$ 0.02 | 0.89 $\pm$ 0.01 | N/A |
| ReAct | Centralized | Oracle | Full | 1348.71 $\pm$ 53.34 | 0.67 $\pm$ 0.02 | 0.86 $\pm$ 0.01 | 21.14 $\pm$ 0.59 |
| ReAct | Centralized | Oracle | Partial | 2590.82 $\pm$ 90.71 | 0.56 $\pm$ 0.02 | 0.80 $\pm$ 0.01 | 25.57 $\pm$ 0.64 |
| ReAct | Decentralized | Oracle | Partial | 3353.33 $\pm$ 70.03 | 0.63 $\pm$ 0.02 | 0.84 $\pm$ 0.01 | 17.38 $\pm$ 0.33 |
| ReAct-Tools | Decentralized | Oracle | Partial | 3810.15 $\pm$ 86.52 | 0.61 $\pm$ 0.02 | 0.83 $\pm$ 0.01 | 25.79 $\pm$ 0.41 |
| ReAct + RAG | Decentralized | Oracle | Partial | 3489.18 $\pm$ 79.54 | 0.62 $\pm$ 0.02 | 0.83 $\pm$ 0.01 | 17.55 $\pm$ 0.38 |
| Finetuned | Decentralized | Oracle | Partial | 3460.60 $\pm$ 78.33 | 0.51 $\pm$ 0.02 | 0.78 $\pm$ 0.01 | 14.73 $\pm$ 0.25 |
| ReAct | Decentralized | Learned | Partial | 5905.88 $\pm$ 162.35 | 0.50 $\pm$ 0.02 | 0.76 $\pm$ 0.01 | 24.30 $\pm$ 0.60 |
| ReAct-Single | Single Agent | Oracle | Partial | 2632.30 $\pm$ 60.04 | 0.68 $\pm$ 0.01 | 0.85 $\pm$ 0.01 | 21.28 $\pm$ 0.37 |
| ReAct-Single | Single Agent | Oracle | Full | 1559.73 $\pm$ 36.02 | 0.73 $\pm$ 0.01 | 0.88 $\pm$ 0.01 | 21.06 $\pm$ 0.38 |
| ReAct-8B | Decentralized | Oracle | Partial | 4100.21 $\pm$ 98.97 | 0.51 $\pm$ 0.02 | 0.77 $\pm$ 0.01 | 27.65 $\pm$ 0.52 |

To supplement the results in Table 2, we have included Table 10, containing results from additional baselines on the validation set, and Table 11 containing results on the PARTNR test set. ReAct rows use the same summary prompting format as the baselines in 2. ReAct-Tools requires the agents to use perception tools to observe the environment instead. ReAct-8B uses Llama-3.1-8B-Instruct as the robot model for an equal capacity comparison with Finetuned.

### A.12 Analysis of collaborative behavior and efficiency of LLM agents

Since task state success and percent complete metrics look at overall team performance for the human and robot agents in our tasks, we also evaluate metrics that allow us to look at different aspects of collaborative behavior of the agents. We measure percentage of sub-tasks done by the robot (task offloading), ratio of unnecessary rearrangements over total successful rearrangements done by both agents (extraneous effort), and number of exploration steps needed before first object is picked (exploration efficiency) to analyze agent behaviors (Table 12). Our main findings are below:

**Table 12: Analysis of collaboration characteristics for LLM agents. Average and standard errors for task offloading, extraneous effort, and exploration efficiency are reported over the successful episodes from the validation set for LLM agents using ReAct approach in partially observable setting.**
| Method | Sim Steps $\downarrow$ | Task Offloading$\uparrow$ | Extraneous Effort$\downarrow$ | Exploration Efficiency$\downarrow$ |
| --- | --- | --- | --- | --- |
| Decentralized | 3295.20 $\pm$76.27 | 0.596 $\pm$0.01 | 0.21 $\pm$0.01 | 994.88 $\pm$24.890 |
| Centralized | 2298.13 $\pm$61.39 | 0.49$\pm$0.01 | 0.04 $\pm$0.004 | 684.06 $\pm$21.71 |
| Single agent | 2519.02 $\pm$57.48 | - | 0.047 $\pm$0.01 | 1121.65 $\pm$31.256 |

Agents are able to find objects faster when collaborating as compared to solo, but only when they successfully co-ordinate. The exploration efficiency increases i.e., agents are able to find task-relevant objects in fewer steps, in centralized and decentralized settings. By computing the average number of exploration steps taken before the first object is picked up for a task, we find that single agents require on average $127$ steps more to locate objects compared to multi-agent. However, in centralized setting, where the co-ordination between agents is better owning to a single LLM co-ordinating the actions of both agents, shows higher gains in such efficiency as compared to decentralized settings. The challenge LLMs face in coordinating exploration in multi-agent settings also negatively impacts human-LLM team performance in our HITL experiments when paired with humans (Table 3).

Poor co-ordination also leads to wasted effort and more steps to complete the tasks than solo execution. Despite multiple agents working together, agents take longer to complete the tasks in decentralized settings as compared to solo execution owing to poor co-ordination. Poor co-ordination is further highlighted by extraneous effort, which increases by $300\%$ in decentralized settings as compared to solo execution. The agents often repeat parts of the task – unsure of whether the other agent really executed that part, and sometimes even undo successfully completed tasks.

The robot is able to offload more than half of the tasks from the human partner. The human-robot team takes longer to complete the task in decentralized setting, however, the robot offloads  $60\%$ tasks from the human partner, reducing their load of task execution. This highlights the potential of robots assisting humans more effectively as LLMs continue to advance in reasoning, coordination, and planning capabilities.

LLMs struggle to reason about temporal constraints and agent capabilities while planning PARTNR tasks. Constraint-free and spatial tasks in PARTNR require the LLMs to reason about only the final configuration and states of objects. Instead, the temporal tasks in PARTNR require tracking states of one or more objects over the entire episode, making them challenging (Table 13). Likewise, heterogeneous tasks necessitate reasoning about task distribution conditioned on each agent’s capabilities, which make them challenging.

**Table 13: Task performance per task type. Average and standard errors of task success rate for episodes from the validation set categorized by task type. Performance is shown for LLM agents that use ReAct approach in partially observable setting.**
| Method | Success Rate per Task-type |  |  |  |
| --- | --- | --- | --- | --- |
|  | Constraint-free | Spatial | Temporal | Object states |
| Decentralized | 0.82 $\pm 0.02$ | 0.82$\pm 0.02$ | 0.60$\pm 0.03$ | 0.66$\pm 0.03$ |
| Centralized | 0.84 $\pm 0.02$ | 0.85$\pm 0.02$ | 0.59$\pm 0.03$ | 0.66$\pm 0.03$ |
| Single agent | 0.85$\pm 0.02$ | 0.81$\pm 0.02$ | 0.58$\pm 0.03$ | 0.68$\pm 0.03$ |

### A.13 Human-in-the-loop Evaluation for PARTNR tasks

#### A.13.1 HITL Interface and Web Hosting

We adapt the existing human-in-the-loop (HITL) infrastructure from Habitat 3.0 (Puig et al., ) to support a more robust server-client architecture, with the server component hosted on AWS. Habitat3.0 HITL includes the ability to extend functionality to resource-constrained environments such as web browsers and VR devices, making the platform versatile for different user needs and experimental setups. Figure 13 shows our HITL system running on a web browser. Detailed interface is shown in Figure 14. Our adaptation to AWS hosting is crucial for handling multiple clients simultaneously, especially non-experts without access to powerful machines or large Habitat datasets. The server-client architecture not only enhances scalability but also ensures flexibility, allowing the system to accommodate a variety of operating systems and hardware platforms. Furthermore, the system includes a matchmaking service that enables pairing participants for multi-user sessions. When a participant requests a task, they are redirected to a "lobby screen" where they are instructed to wait until the next participant arrives.

Figure: Figure 13: HITL on Web-browser. Our HITL system can be deployed on web browsers enabling large-scale collection.
Refer to caption: extracted/5961334/figs/hitl_web_app.png

#### A.13.2 Participant Recruitment and Quality Control

The study was performed through a 3rd party company specializing in large-scale annotations. The participants were recruited and compensated for their time by this company. The participants were English speakers, 18 years or older from the United States. For training them, we created a project and task overview video and guidelines. The participants were instructed to complete the tasks correctly and efficiently by themselves or with a partner. The participants also went through a tutorial where they performed some tasks to get acquainted with the interface before performing the main tasks. Each task took on average 3-5 minutes to complete. We recruited 129 non-expert participants in total.

Filtering data:

Each task was completed up to 3 times, in all settings, until deemed successful through task evaluation (conducted online as the participants complete the tasks). With the Failure explanation output from the evaluation function, users were also given a natural language feedback at the end of an episode, describing what went wrong in an episode, if the task was not successful. For example, if actions were completed in the wrong order, or a spatial constraint like next-to was not respected appropriately. Users can use this information to update their actions in the next episode, improving their overall performance over time. By giving 4 tutorial episodes before the start of each experiment, and also a tutorial episode in the same house, we ensure that the users are deeply familiar with the tool and tasks before starting the actual experiment.

Figure: Figure 14: HITL Interface. Participants control human and robot agents using keyboard/mouse controls to complete the PARTNR tasks. Each participant has access to their partner’s viewpoint and thereby current actions via a small viewport on the bottom right.
Refer to caption: extracted/5961334/figs/hitl_ui.png

Once each task was completed successfully at least once or retried 3 times, we assimilate data by selecting one of the tries per episode: we select the successful try for successfully completed episodes, and the highest percent complete try for unsuccessful episodes. This collection of 1000 episodes for test and val is used for collecting performance statistics described in the next section.

#### A.13.3 HITL experimental details

Task Evaluation and Data Collection:

Using the enhanced server-client setup, we conduct comprehensive evaluations of 1000 tasks from the validation set, and 1000 tasks from the test set. These evaluations are designed to capture data in both single-user and multi-user scenarios. In the single-user setting, participants individually control a human agent within the simulator using traditional keyboard and mouse inputs, completing tasks without any external assistance. Conversely, the multi-user setting involves collaborative efforts where two participants work together, each controlling either a human or a robot agent. This collaborative approach is specifically designed to study the dynamics between multiple users and to evaluate whether such collaborations lead to more efficient task completion compared to single-user efforts in the PARTNR tasks.

Human-AI Collaboration Experiment:

In addition to human-only interactions, we conduct experiments where a human collaborates with a robot controlled by a Language Model (LLM), specifically using the ReAct and Finetuned models as described in Section 4.1. The primary goal of these experiments is to evaluate the effectiveness of LLM-controlled agents in real-time collaboration with non-expert humans who have not previously interacted with these AI models. We track and compare the success rates (SR) and the percentage of tasks completed (PC) across various settings including single-user, multi-user, and human-AI collaborations. The results of these experiments are systematically documented and analyzed in Table 14, providing insights into the collaborative capabilities of human-AI pairs.

To enable this setting, we host LLMs on AWS nodes, and query them intermittently based on robot observations and actions. The HITL server now queries two clients - a human and a LLM. The human client sends commands to control the human agent, and the LLM client uses the hosted LLM to control the robot. Different baselines need different numbers of GPUs to keep the inference time reasonable. For hosting and 70B models, as used by the ReAct baselines, we use 4 A100 GPUs per model. For hosting a smaller 8B model used by the Finetuned baselines, we use 1 A100 GPU. This makes deploying smaller Finetuned models much scalable than larger 70B models.

Efficiency and Task Offloading Metrics:

To further understand the efficiency of task completion across different experimental setups, we measure several key performance metrics beyond success rate and percent complete. These include the number of steps taken to complete tasks and the exploration efficiency, which is assessed by the number of steps participants take to pick the first object. Additionally, we evaluate the extraneous effort by noting actions that do not contribute directly to task completion. Another critical metric we analyze is the ratio of work completed by the robot, referred to as task offloading. Ideally, in a well-coordinated human-AI team, the task offloading ratio should approach 0.5, indicating an efficient division of labor between the human and the robot.

**Table 14: Human-in-the-Loop evaluation. We evaluate the performance of 2-person human teams and human-LLM teams, comparing them to solo human performance on PARTNR tasks using metrics described in Section 4.1. The human-LLM teams with SoTA LLMs is slower than solo human.**
| Method | Success<br>Rate $\uparrow$ | Percent<br>Complete $\uparrow$ | Sim<br>Steps $\downarrow$ | Task<br>Offloading $\uparrow$ | Exploration<br>Efficiency $\downarrow$ | Extraneous<br>Effort $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Validation 1,000 episodes |  |  |  |  |  |  |
| Single-user | 0.93 $\pm$ 0.01 | 0.96 $\pm$ 0.00 | 3046.99 $\pm$ 80.79 | N/A | 2459.22 $\pm$ 26.75 | 0.09 $\pm$ 0.01 |
| Multi-user | 0.93 $\pm$ 0.01 | 0.96 $\pm$ 0.00 | 2369.55 $\pm$ 49.33 | 0.59 $\pm$ 0.01 | 1762.47 $\pm$ 13.99 | 0.15 $\pm$ 0.01 |
| Human-ReAct | 0.91 $\pm$ 0.01 | 0.96 $\pm$ 0.02 | 4267.71 $\pm$ 83.40 | 0.16 $\pm$ 0.01 | 2624.39 $\pm$ 26.05 | 0.12 $\pm$ 0.01 |
| Human-Finetuned | 0.92 $\pm$ 0.01 | 0.96 $\pm$ 0.00 | 3443.33 $\pm$ 61.46 | 0.26 $\pm$ 0.01 | 2164.94 $\pm$ 21.31 | 0.13 $\pm$ 0.01 |
| Test 1,000 episodes |  |  |  |  |  |  |
| Single-user | 0.89 $\pm$ 0.01 | 0.95 $\pm$ 0.00 | 3937.87 $\pm$ 110.53 | N/A | 2737.44 $\pm$ 25.27 | 0.13 $\pm$ 0.00 |
| Multi-user | 0.85 $\pm$ 0.01 | 0.95 $\pm$ 0.00 | 2667.86 $\pm$ 58.07 | 0.60 $\pm$ 0.01 | 1889.56 $\pm$ 14.9=69 | 0.20 $\pm$ 0.00 |
| Human-ReAct | 0.87 $\pm$ 0.01 | 0.95 $\pm$ 0.00 | 4080.10 $\pm$ 72.24 | 0.12 $\pm$ 0.00 | 2449.50 $\pm$ 19.98 | 0.18 $\pm$ 0.01 |
| Human-Finetuned | 0.87 $\pm$ 0.01 | 0.95 $\pm$ 0.00 | 3403.03 $\pm$ 62.08 | 0.26 $\pm$ 0.01 | 2162.20 $\pm$ 19.40 | 0.20 $\pm$ 0.01 |

#### A.13.4 HITL analysis

Humans are significantly better than LLMs at PARTNR tasks. Both single and multi-human settings achieve a success rate of 0.93 on PARTNR validation tasks, while ReAct without any privileged information only achieves 0.30 (Table 2, row (i)). This indicates a huge gap in LLM planning performance. We observe a slightly lower human performance on the test set (0.89), also in line with a lower LLM performance in Table 11 on this dataset. This indicates that the tasks in our test set are more challenging than the validation set for both humans and LLMs alike, potentially due to human annotations aimed at making them more complex and diverse.

Finetuned LLMs perform better than ReAct when coordinating with real humans. When deployed with real humans-in-the-loop on the validation set, the finetuned model is faster than ReAct at task completion ($3275$ steps with finetuned versus $4484$ with ReAct on the validation set). It is also able to offload more tasks from humans than ReAct (26% with finetuned as compared to 16% with ReAct). This reflects that smaller models with faster inference can improve human experience in real-world deployment. This result is also reflected in the test set where finetuned model outperforms ReAct. Interestingly, the automated eval performance of finetuned is worse than ReAct on the test set, but the HITL performance is better, indicating that faster inference is more critical than task success when working with real humans.

LLMs struggle at coordination, hampering human performance. Despite the Finetuned being faster than ReAct when collaborating with humans, both approaches are slower than the human doing the task alone. In contrast, two humans working together complete the task faster than a single human ($2369$ steps vs. $3046$ with multi- and single-user respectively). This result is in line with the automated evaluation we observed in Table 1, where multi-agent LLMs are also slower than a single-agent LLM. This result further reinforces that LLMs suffer at coordination; while humans are able to coordinate and divide tasks between each other, decentralized LLMs are unable to do so. We observe the same effect in the test set, further reinforcing this finding.

LLMs are able to offload tasks from humans. Despite the aforementioned increase in the number of steps for task completion, robots guided by the finetuned model successfully offload 26% of tasks from humans. This indicates that LLMs can still offer assistance when collaborating with real human partners. Nonetheless, there remains significant potential for improvement.

LLM’s inefficiency to explore reduces the team performance when paired with humans. In multi-user condition, the two humans start in different parts of a house, and explore efficiently to locate task-relevant objects more quickly than a single user – as evidenced by the reduced number of steps before first pick ($1762$ steps with multi-user vs. $2459$ steps for a single user). However, this efficiency is reduced when humans are paired with LLMs ($2120$ steps with finetuned and $2791$ steps with ReAct), indicating that LLMs struggle to coordinate at *both* task completion and exploration.

### A.14 Prompts for benchmark task and evaluation

#### A.14.1 Task generation prompts

Prompts are similar for the different task types, with the primary difference being the in-context examples.
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Constraint-free task generation prompt,
frametitlealignment=]

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Spatial task generation prompt,
frametitlealignment=]

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Temporal task generation prompt,
frametitlealignment=]

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Large-scale task generation prompt,
frametitlealignment=]

### A.15 Evaluation Generation Prompts

Here we share the LLM prompts used for proposition generation, temporal constraint prediction, and argument constraint prediction. In each, the task to accomplish is described in the system prompt and between 6-13 few-shot examples follow.
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Evaluation Generation: Propositions,
frametitlealignment=]

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Evaluation Generation: Temporal,
frametitlealignment=]

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Evaluation Generation: Argument Constraints,
frametitlealignment=]

### A.16 Prompts for planning baselines

Following prompts were used for various planning baselines.
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Decentralized Single/Multi Agent | ReAct,
frametitlealignment=]

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Centralized | ReAct,
frametitlealignment=]

The agent role description would be one of the following, depending on if the agent played the role of the human or robot.
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Agent Role Descriptions,
frametitlealignment=]
Human Description

Robot Description

Below are the tool descriptions.
Perceptions tools are only included for the React-Tools agents.
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Tool Descriptions,
frametitlealignment=]

For LLM agents simulating a human, additionally actions which modify the state of objects are available to be called. For centralized baselines two lists of available actions are provided in the agent description. One for the robot (without state-modifying actions) and one for the human (with state-modifying actions).
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Human Only Tool Descriptions,
frametitlealignment=]

The world description contains a text description all rooms and their contained furniture, along with currently observed objects. Below is an example for one scene:
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=World Description Example,
frametitlealignment=]

For baselines utilizing a summary of the scene, after each action, we insert a text description of objects in the scene (as perceived by the agent so far).
{mdframed}[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Scene Summary Example,
frametitlealignment=]

For finetuning, we build the prompt by combining the task description, world description and the previous actions for both agents.

[
linewidth=1pt, linecolor=black, backgroundcolor=gray!10, roundcorner=5pt, frametitle=Finetuning Example,
frametitlealignment=]

## Contributions

Matthew Chang: Implemented the state-modifying skills and built out the object state management system. Implemented constrained generation for planning baselines. Implemented ReAct based baselines and optimized their prompting and performance. Contributed to finetuning code and LLM inference infrastructure.

Gunjan Chhablani: Implemented PrediViz and helped analyze PARTNR tasks and evaluation functions.

Alexander Clegg:
Designed and implemented foundation simulation features and APIs including: geometric relation utils, object states, kinematic relationship management, articulation logic, procedural sampling features, metadata handling, and some oracle skills. Contributed to design, implementation, and better engineering efforts for evaluation, HITL, simulated world graph, perception, and robot skills subsystems. Led the 3D asset dataset preparation effort extending HSSD with articulations, annotations, and quality iteration to enable large scale experimentation on collaboration tasks.

Mikael Dallaire Cote:
Led the development of a cloud platform capable of collecting collaboration data, at a large scale, by extending the Habitat 3.0 human-in-the-loop (HITL) system to (1) Run in the browser; (2) Support multiplayer sessions; (3) Evaluate LLM agents; (4)
Support the full articulated HSSD dataset. Created a GUI application within that system that allows crowdsource workers to complete collaboration tasks.
Deployed the HITL platform to the cloud and rolled out the data collection campaign. Created paper figures.

Ruta Desai: Defined benchmark characteristics and developed the simulation-in-the-loop LLM-based task instruction generation pipeline. Proposed and developed ReAct baselines and its analysis. Wrote experiments and analysis section in the paper.

Michal Hlaváč:
Designed PrediViz, the visualization and annotation system for the PARTNR task evaluation. Wrote the section on PrediViz in the paper.

Vladimir Karashchuk:
Contributed to 3D asset dataset preparation effort extending HSSD with articulations, annotations, and quality iteration to enable large scale experimentation on collaboration tasks. Authored annotations such as regions, articulations, and marker sets as well as authoring 3D assets including render, collision, and receptacle meshes. Worked on quality assurance for annotations and assets originating from vendor workstreams.

Jacob Krantz:
Designed and implemented the PARTNR evaluation system and large-scale generation of evaluation functions. Led the dataset annotation effort and maintained, analyzed, and balanced all dataset splits. Guided the development of PrediViz.

Roozbeh Mottaghi: Managed the team and coordinated different workstreams including benchmark generation, planning models, analysis, and human-in-the-loop infrastructure.

Priyam Parashar: Contributed to the world-graph representation and logic for integration into the planning stack, specifically owning the concept-graph pipeline to create non-privileged scene descriptions and non-privileged graph updates. Built out the integration of LLM agents and planner into the HITL tool enabling single-learn evaluations.

Siddharth Patki:
Designed and implemented centralized and decentralized task planning frameworks for baseline experiments in PARTNR. Proposed and developed tool based ReAct planner baselines using instruct-tuned Llama3 models, and analyzed planner’s performance using various performance metrics and failure modes. Designed and supported implementation of hierarchical scene graphs to encode information about agent’s environment which is consumed by all baseline planners. Paper writing and figures.

Ishita Prasad: PXFN alignment and approval for data collection/annotation for model training. Secured alignment with other orgs, XFN to enable open-sourcing/release of dataset and code for multi-agent collaboration benchmarks.

Xavi Puig: Contributed to the planning codebase, building infra to scale up planning baselines and LLM inference. Led agent finetuning experiments and oracle-based agents, and debugged simulation and oracle skill failures. Initial exploration of automated benchmark generation.
Built tools for HITL data analysis and visualization. Paper writing and figures.

Akshara Rai: Implemented benchmark generation for guided scaling of rearrange, spatial, temporal and heterogenous tasks. HITL analysis, paper writing. Supported all workstreams of the project.

Ram Ramrakhya:
Implemented and helped design LLM fine-tuning pipeline, auto regressive evaluation and metrics for fine-tuning LLM planners. Ran experiments to train oracle-planner traces based agents in full and partial observation settings. Implemented initial version of HITL data parsing to convert episodes into traces for LLM finetuning. Implemented and deployed HITL campaign/experiment management service end-to-end using PsiTurk to:
(1) Automatically launch and manage large campaigns and metadata. (2) Matchmaking for multi-user tasks, single user task assignment and automatic task recycling. Integrated habitat-llm and initial version of task in HITL to support collaboration tasks.

Daniel Tran: Annotated PARTNR datasets and provided feedback leading to improved generation.

Joanne Truoung: Contributed to the dataset effort by refining scene compositions and ensuring validity of the scenes for HITL data collection.

John Turner:
Contributed implementation of foundation simulation features and APIs including: region queries, articulations, metadata management, marker sets, and panoptic sensor. Contributed to 3D asset dataset preparation effort extending HSSD with articulations, annotations, and quality iteration by designing scenes, building editor interfaces, and implementing automated quality checks.

Eric Undersander: Contributed to HITL architecture. Implemented HITL networking capabilities for single-user evaluation. Contributed to the HITL cloud infrastructure and tooling.

Tsung-Yen Yang: Led the effort of integrating neural network skills and designing a RAG system for running planner evaluation on the benchmark. Performed planner and skill results analysis, and contributed to better engineering efforts for designing skills CIs. Improved the planner’s success rate by identifying edge cases, and better script for finding bugs.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - How is "collaboration data [from] the web" used? I may have missed this 
- Does performance improve with more sophisticated collaboration strategies? With a more nuanced understanding of the partner's progress? If not, what's the need for learning these skills? 
- How repetitive is the benchmark, fundamentally? Definitely within the 100k, but even within the seed 10k - are there many tasks that require the same exact action trajectories, just with parameters like object ID switched out?
soundness: 3
strengths: ### Quality
- Well-designed benchmark. The task types, generation pipeline (especially sim-in-the-loop verification), and comparative experiments are clever and useful 
- Centralized/decentralized baseline is helpful - even though the centralized method wouldn't make sense as a method benchmarked on PARTNR, using it in comparison to the decentralized method does make an important point about the nature of the tasks. In fact it seems more like an ablation than a benchmark, but point being, it's a good experiment showing that collaboration itself (to the degree you can isolate it when considering generative AI) matters for PARTNR tasks.

### Clarity
- Very well-written paper! 

### Originality 
As far as I am aware, this is the first multi-agent benchmark of this scale, and with attention paid to these particular aspects. The use of LLM planning modules simplifies the work in many ways - since the methods being used aren't complicated robotic systems, the failures are more constrained to being in semantic space and likely being related to multi-agent coordination, i.e. the benchmark seems especially likely to do what it claims to do. 

## Significance 
This is a considerable work that will likely enable useful research.
summary: This paper presents PARTNR, a benchmark for human-robot collaboration. It focuses on evaluating LLM high-level planning. 

The paper positions itself against language-based embodied AI benchmarks that don't have interaction, embodied multi-agent benchmarks that have simple, short-horizon environments and closed task sets, and generated-task benchmarks that lack its scale, breadth, and simulator-in-the-loop. It also reviews LLMs for decision making and highlights methods it then uses as baselines.

It then describes the design elements of PARTNR:
- Four task categories with different types of constraints
- 1k human-verified instructions and corresponding evaluation functions, and a prompting + manual annotation pipeline that then leads to 100k total task-evaluation function pairs
- Methods for generating all of these 
- Dataset train, validation, and test splits 

Finally, the paper goes through significant experimentation investigating the nature of PARTNR tasks. It tests various LLM planning methods in multiple settings, then discusses various conclusions about the nature of PARTNR tasks - difficult for current methods, legitimately encouraging collaboration and coordination, and doable by humans.
weaknesses: ### Clarity
- More details would help in the main text. I realize benchmark papers require a lot of material, but it would help to have some examples of 
  - Human annotation
  - Action selection trajectories - I want to know what it looks like when there's a perception failure, for example 
- At times, the writing feels like a laundry list. E.g. section 3.3 - starts to feel like a laundry list of details about the benchmark, without clear direction on why we should care. Dependent rearrangements and multi-step rearrangements of the same object are both interesting, but they aren't brought up again in detail
- Some technical details could use reminder/explanation in main text. E.g.: 
  - "LLMs are unable to recover from skill failures" - what exactly is communicated to the LLM here? Is it the new scene graph that doesn't have the intended outcome? What does the failure look like? 
  - LLMs struggle to correct errors such as misclassification, making them sensitive to errors in perception - this seems obvious. I'd want to see a trace to know how that plays out, and what this says about *PARTNR* as opposed to the fact that LLMs can't do visual reasoning 

### Quality
Overall a convincing paper. Some examples in the main text would help.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. How serious were ConceptGraph's errors? Can we give an additional metric of its errors in object detection and description to assess whether LLM agents suffer from even small perceptual errors?
2. What criteria were used by the annotation operators and how was their work checked?
soundness: 3
strengths: The article is written in clear language, the text is easy to read, and the idea and experimental setting are well described. An important gap in existing datasets for collaborative interaction with a large number of scenes, objects, and a variety of instructions is filled. The resulting dataset will clearly be in demand in the community. The authors use the latest available methods, models, and data: LLama3, Habitat 3, HSSD, OVMM, ReACT, and ConceptGraphs.
summary: The work is devoted to the creation of a new benchmark for multiagent planning and execution of textual instructions in a home simulator. The authors proposed a pipeline of task generation and success evaluation scripts using LLM. In the first stage, 1K instructions synchronized with the Habitat 3.0 simulator were obtained mainly due to manual markup, and then 100K instructions were obtained by prompting LLama3-70B-Instruct using 1K as examples. A PDDL-like language with predicates and predicate constraints was proposed for evaluating scripts. 1K scripts were manually marked up, which were then used to generate a complete set with an RAG-like mechanism for the same LLama3-70B-Instruct. The well-known ReAct model and the heuristic expert planner were tested on the resulting dataset. It is shown that centralization, full observability, and accurate execution of skills gives the best result, but there is still a significant gap in the coordination of actions and in the grounding of information.
weaknesses: It is not quite clear why the authors specified “applications to robotics, autonomy, planning” as the primary area. There are no new planning methods or robotics applications in the paper, all the baselines used are known. Clearly, the primary topical area should have been “benchmarks and datasets”, within which the work should be evaluated. It is from this point of view that the shortcomings of the work are listed below.

1. In general, the data generation pipelines proposed by the authors cannot be called new. The use of LLM is also found in the works mentioned by the authors both in terms of the generation of the instruction set itself and in terms of the generation of validation scripts. At the same time, the authors do not overcome the main limitation of these approaches - a large amount of manual work of annotation operators. The contribution of LLM is still only related to data scaling, which still requires time-consuming manual validation. In this sense, the work lacks novelty in the sense of data generation pipelines and stands out only for its volume, which is largely obtained by using previously created diverse scenes and collections of objects: HSSD and OVMM.
2. Using a simulator to ground concepts from instructions is reasonable, but this idea is not extended to automatic validation of successful execution of instructions in this environment, for which a heuristic planner could be used. Correctness of instructions and evaluation scripts still relies on manual spot-checking.
3. The authors consider only one single learnable baseline - ReAct, so it is hard to call it a benchmark. There are a huge number of planners for embodied environments, including multi-agent ones [1], with special grounding per scene [2] and with error analysis from lower-level actions [3]. Therefore, the conclusions that the authors draw, based on only one baseline, that this dataset is challenging for state-of-the-art models are not well-founded.
4. It is not clear whether there were instructions that explicitly stated which agent, and which actions should be performed. Such a set of instructions could have been useful for assessing the model's ability to assign roles to a task.
5. The active involvement of human experts requires additional evaluation of the quality of the tasks they perform, both in relation to the generation of instructions and in relation to the manual checking of the correctness of scripts. What criteria were used by the annotation operators and how was their work checked?

[1] https://ieeexplore.ieee.org/abstract/document/10504634/ 
[2] https://arxiv.org/abs/2310.13255 
[3] https://arxiv.org/abs/2307.00329

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 4
questions: 1） The method ultimately generated 100,000 tasks, with a reported success rate of 83%, but it seem unclear whether the unsuccessful tasks are labeled?
2） Additionally, is there any tendency in the types of tasks generated by this method? Robots may excel in certain task categories, so if the types of tasks generated are those that robots are particularly skilled at—or, conversely, struggle with—would this affect the realism of the test results? In other words, do these tasks' distributions realistically reflect the types of tasks humans encounter in daily life?
3） Can the framework generate task patterns that are related to the robot characteristics?
4)   The author mentioned that the fine tuned model performs better than the large model. But the author trained the model on a specific dataset and tested it on the same dataset. Is this possible to cause overfitting？
soundness: 2
strengths: The semi-automatic benchmark generation method proposed in this paper is highly insightful and offers a potential solution to the limited data issue in the field of embodied intelligence. The process for generating test tasks is described in detail, with a comprehensive explanation provided in the appendix, showcasing an ingenious approach to task generation. Additionally, the experimental results validate mainstream models, demonstrating the practical applicability of this benchmark and highlighting the considerable room for improvement in current models.
summary: This paper presents a benchmark for Planning And Reasoning Tasks in humaN-Robot collaboration (PARTNR), designed to measure collaboration between humans and robots in a household activity setting. The benchmark, semi-automatically generated with the help of LLMs, covers four categories of tasks: (1) constraint-free, (2) spatial tasks, (3) temporal tasks, and (4) heterogeneous tasks. Each task in the benchmark includes instructions and evaluation functions. Across the generated 100,000 tasks, the instructions achieved a 90% accuracy rate, evaluation functions reached 92%, and the overall task validity rate was 83% (90% × 92%). In the final benchmark tests on current mainstream models, it was found that fine-tuning a small model can achieve performance comparable to models nine times larger, while being 8.6 times faster at inference. Additionally, the findings reveal that LLM-guided partners decrease human efficiency compared to working solo.
weaknesses: The success rate of the tasks generated by this method remains relatively low. Although human-guided data refinement was used to clean the tasks, the effectiveness rate is still only 83%, which could lead to complications in practical applications. Furthermore, the paper does not seem to address how to identify the unsuccessful tasks within the dataset. This issue may interfere with measuring embodied intelligence's success rate in tasks. Since the primary contribution of this method lies in generating large volumes of data, if manual filtering is still required to address task inaccuracies, the overall value of this method may be called into question.

---

confidence: 2
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Do you need to use a human-in-the-loop tool to filter all tasks? How much time does it take?
2. Is the human in section 4.2.1 using ReAct to simulate? Does every setting in this section use this model to represent a human? Why did you choose this model for simulating human behavior? Why do you think that LLMs should plan for humans?
3. Is there any analysis of failure cases for task completion?
soundness: 3
strengths: 1. The motivation is reasonable and meaningful.
2. I think this is a large-scale benchmark that makes significant contributions to the community due to its open-source nature.
3. The experiments are sufficient and clear.
4. The paper is well-rounded, demonstrating a huge amount of work, and is practical, potentially being very helpful for research in this direction.
summary: This work proposes a benchmark for planning and reasoning tasks in human-robot collaboration, designed to study human-robot coordination in household activities. Numerous experiments have been conducted, which indicate some weaknesses of current LLMs.
weaknesses: Please refer to the "Questions" section.

Meta-review: The paper presents PARTNR, a large-scale benchmark for studying human-robot collaboration in household activities through planning and reasoning tasks. The work received generally positive reviews, with scores ranging from 6 to 8, and all reviewers recommending acceptance after thorough discussion. Reviewers consistently praised the well-designed benchmark creation pipeline, comprehensive experimental validation, and potential impact on advancing research in human-robot collaboration.

The key strengths highlighted across reviews were the benchmark's scale (100,000 tasks), the innovative simulation-in-the-loop verification approach, and thorough empirical analysis revealing limitations of current LLM-based methods. Reviewers also appreciated the paper's clear presentation and practical significance for the robotics community.

Initial concerns focused on three main areas: the novelty of the LLM-based data generation pipeline, limited baseline comparisons, and lack of detailed analysis of perception failures and task characteristics. The authors provided comprehensive responses, clarifying their novel contributions in grounding LLM outputs through simulation, introducing additional VLM-based baselines, and providing extensive analysis of perception system errors and their impact on task performance. They also clarified the human annotation process and provided detailed failure case analyses.

While some reviewers maintained reservations about the novelty of using LLMs for data generation, they acknowledged that the overall contribution, particularly the simulation-grounded verification and comprehensive evaluation, merits acceptance. The thorough empirical analysis and potential impact on the field outweigh these limitations.

Decision: Accept (Poster)