Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environments
Authors: Siddharth Nayak
Adelmo Morrison Orozco
Marina Ten Have
Jackson Zhang
Vittal Thirumalai
Darren Chen
Aditya Kapoor
Eric Robinson
Karthik Gopalakrishnan
James Harrison
Anuj Mahajan
brian ichter
Hamsa Balakrishnan
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-agent robotics
large language models
Primary area: robotics
TL;DR: a VLM-based planner for long-horizon multi-agent robotics in partially observable setting that does not rely on privileged information from the simulator/oracle.

Abstract: The ability of Language Models (LMs) to understand natural language makes them a powerful tool for parsing human instructions into task plans for autonomous robots. Unlike traditional planning methods that rely on domain-specific knowledge and handcrafted rules, LMs generalize from diverse data and adapt to various tasks with minimal tuning, acting as a compressed knowledge base. However, LMs in their standard form face challenges with long-horizon tasks, particularly in partially observable multi-agent settings. We propose an LM-based Long-Horizon Planner for Multi-Agent Robotics (LLaMAR), a cognitive architecture for planning that achieves state-of-the-art results in long-horizon tasks within partially observable environments. LLaMAR employs a plan-act-correct-verify framework, allowing self-correction from action execution feedback without relying on oracles or simulators. Additionally, we present MAP-THOR, a comprehensive test suite encompassing household tasks of varying complexity within the AI2-THOR environment. Experiments show that LLaMAR achieves a 30\% higher success rate than other state-of-the-art LM-based multi-agent planners in MAP-THOR and Search \& Rescue tasks. Code can be found at [https://github.com/nsidn98/LLaMAR](https://github.com/nsidn98/LLaMAR)

Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environments
Siddharth Nayak!" !M
Adelmo Morrison Orozco
Marina Ten Have
Vittal Thirumalai'
Jackson Zhang'
Darren Chen'
Aditya Kapoor
Eric Robinson Karthik Gopalakrishnan"
James Harrison Brian Ichter
5 Anuj Mahajan 6 Hamsa Balakrishnan'
MIT "TCS USAF-MIT AI Accelerator
Stanford Google DeepMind 6 Apple
Abstract
The ability of Language Models (LMs) understand natural language makes them a powerful tool for parsing human instructions into task plans for autonomous robots. Unlike traditional planning methods that rely domain-specifie knowl- edge and handcrafted rules, LMs generalize from diverse data and adapt to various tasks with minimal tuning, acting compressed knowledge base. However, LMs in their standard form face challenges with long-horizon tasks, particularly partially observable multi-agent settings. We propose an LM-based Long-Horizon Planner for Multi-Agent Robotics (LLaMAR), a cognitive architecture for plan- ning that achieves state-of-the-arr results long-horizon tasks within partially observable environments. LLaMAR employs plan- aeau--ct--orrret-veriie framework, allowing self-correction from action execution feedback without relying on oracles or simulators. Additionally, we present HAP-THOR, a comprehensive test suite encompassing household tasks of varying complexity within the AI2-THOR envi- ronment. Experiments show that LLaMAR achieves a 30% higher success rate than other state-of-the -ttth-art LM-based multi-agent planners MAP-THOR and Search & Rescue tasks. Code can be found at httpss//githbbcccom//i/////XLLL
Introduction
Creating embodied agents that assist humans in real-life scenarios is a significant challenge when humans communicate their intentions in natural language. Certain tasks like moving furniture [ 2). search-and-rescue (3) environmental monitoring [4. etc.. require coordination among multiple agents to solve the tasks efficiently compared to single-agent scenarios. This challenge of understanding these natural language inputs and effectively coordinating these agents to solve the task is exacerbated in the multi-agent scenario. Recent works (5 6 7 I0
II 12 [3 have shown that Language Models ( effectively use language instructions to develop plans for robots. However, most studies focus single-agent long-horizon task planning. Naïve extensions of single-agent planning algorithms to multi-agent settings often fail due to enviromment non -tn-station,rtty where the policies of other agents-modeled
part of the environment-art are continuously changing [14 15). Such failures lead to suboptimal performance as agents struggle to anticipate and adapt to the actions of others. We therefore formulate a centralized process which decisions are made simultaneously for all agents based on their (partial) observations, similar to the centralized multi-agent system framework (CMAS) proposed in [6]. Leveraging the ability of pre-trained LMs to generalize across diverse tasks, we aim to use LMs for long-horizon embodied multi-agent task planning.
The key insight of our work is that integrating plun-act-corrrett-vrij framework with LMs enables robust and adaptive approach to multi-agent task planning in dynamic, partially observable
Equal Contribution. Now Physical Intelligence. Work done outside Apple.
We denote Large
Language Models LLMs, Vision Language Models as VLMs
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

environments that allows agents to: (1) plan subtasks required to complete the task, (2) select high- level actions for each agent complete the proposed subtasks, (3) identify and correct failures after high-level action execution, and (4) self-verify subtask completion based on high-level action execution. Unlike existing methods, our approach uses real-time execution feedback, observations, and agent histories to iteratively refine action planning and execution. This allows agents to adjust strategies based on reasoned insights on action execution, effectively addressing failures without relying perfect environmental knowledge or oracle feedback. The correction and verification process in our cognitive architecture [[7] is grounded in the environment's reality, which sets apart from LM veff-verificatio methods that lack such grounding [T8]. This framework enhances agents' ability to complete complex, long-houn horizon tasks, yielding substantial improvement over current state-of-tee of-tthe-art methods.
24 16] utilize LMs for multi-agent planning. often adopting hierarchical decision-making structure. The LMs are used for high-level planning to determine subtasks, sometimes in conjunction with planning domain definition language (PDDL) that together with the LM planner, functions feasibility solver. Specific actions are executed using low-level policies pre-trained through reinforcement learning, behavior cloning, or heuristic approaches. While these methods effectively use LMs high-level planners, they assume perfect low-level primitive action policies and simulator or saulle-proviee environmental information. By contrast, LLaMAR does not assume perfect knowledge of the environment, does not rely on oracle feedback, and does not assume perfect execution of low-level primitive policies. This approach moves us closer to enabling real-world robots that operate independently of privileged knowledge.
Similar to our approach, recent works 19
To avoid ambiguity, we use the following conventions. We refer to the objectives within the environ- ments as "tasks" or "goals" and "subtasks" to describe the breakdown of tasks or goals. "High-level actions" are defined as skills the agent can perform, while "low-level actions" or "primitive actions" refer to existing policies-eithe learned or predefined using heuristics- execute sequence of actions to accomplish a high-level action. More details and examples can be found in Appendix A
The main contributions of this paper are:
LLaMAR:
Planner for Multi-Agent Robotics, designed for iterative planning of long-horizon, multi-ohjective tasks in partially observable environments, with the following key features:
An LM-based
It operates without prior knowledge of the environment, allowing agents to explore and make decisions based new observations.
It evaluates outcomes through direct observation of images, rather than relying on oracles for feedback, enabling independent identification and correction of action failures.
MAP-THOR (Multi-Agent Planning in THOR): a benchmark suite of tasks within the A12- THOR simulator under partial observability to standardize methodologies and metrics for evaluat- ing multi-agent planning effectiveness and robustness.
2 Related Work
Reinforcement Learning (RL) for Long-Horizon Planning: While RL algorithms have shown promise many applications, they still struggle with long-horizon tasks. Hierarchical reinforcement learning (HRL) has been used to address these challenges in both single-agent [25] 26, 27 (8] and multi-agent settings 29 30 BT. However, these approaches are typically applied single-task, stationary environments, such as games, where agents solve for one goal in fixed environment. Consequently, these methods do not generalize well across multiple environments or tasks. Multi-task RL has been explored as potential solution, requiring sophisticated task planning to handle diverse objectives B32 33). This often involves decomposing tasks into manageable subtasks, a process well-suited for hierarchical frameworks. However, subtasks are known apriori in multi-task RL formulations. Real-world long-horizon RL necessitates robust task planning, and LMs have emerged as promising approach for this purpose.
LMs for Embodied Single-Agent Planning:
Recent studies have demonstrated the effectiveness of LMs in generating and executing plans in embodied single-agent environments 34
35
36
37
38
39
40
and creating plans single-agent embodied robotic environments
44
45
46
48
49
50
51 52) Works like SayCan 55] and Grounded Decoding
6 use combination of value
functions and L.M predictions for long-horizon tasks. ProgPrompt
& and Zero-Shot Language
Planner [3] generate static plans executed in the environment, which may fail in partially observable

---

and dynamic settings. To mitigate this, LLM-planner [53 updates plans based on new observations, similar our approach.
LMs for Multi-Agent Planning: Xu et al. [54 use LLMs in multi-agent games, while CPN 23
creates global plans for robots in embodied environment. RoCo 24 and CoELA 22
assign separate LMs to each agent for decentralized action prediction, allowing natural language communication between agents. However, RoCo and CoNavGPT require detailed environment information for planning, and CoELA's action space is filtered by an oracle. Relying on privileged information from an oracle is impractical in real-world applications. By contrast, our work focuses on free-form action generation and handles tasks with more ambiguous descriptions. Prior work [1 compare centralized (CMAS) and decentralized (DMAS) planning frameworks, showing that centralized planners perform better, though their experiments are in simple, known environments with limited number of agents. Two-Step [1] decomposes goals for main and helper agents, using PDDL planners for high-level actions. SmartLLM [19] uses multiple LLM modules for subtask decomposition, multi-robot group formation and task allocation but assumes robots have complete knowledge of the environment, making plans prone to errors in unknown settings. S-ATLAS 220 use LLMs with conformal prediction for safe multi-agent planning, but the action choices are limited to small set of objects. Table | presents comparison of the characteristics of different LM-based approaches to multi-agent planning with our work.
LMs can interpret high-level instructions and break them down into feasible subtasks, making them ideal for hong-horizon, multi-task scenarios. Our work leverages LMs enable long-horizon planning across a variety of tasks and environments, building on these advances to address the limitations of traditional RI. and HRL methods. By integrating LMs into our planning framework, we enhance the ability to generalize across diverse tasks and scenarios, making significant strides toward practical, real-world applications of RL in dynamic, multi-agent settings.
Dynamic
Local Information
Failure Correction
Self
Method
Planning
Verification
Two-Sie SrooSttppPpr [2]
X
X
Smart LLM [19
x
X
x
S-ATLAS 20
X
X
CoELA 22
X
LLaMAR (this paper)
Table The proposed model, LLaMAR: 1) performs dynamic planning, avoiding open-looy plan
plan-and-an and execute paradigm: 2) operates without privileged simulator information (e.g., access to all objects the environment); 3) re- plans when low-level actions fail, not assuming perfect execution; and 4) self-verifies subtask completion without relying the simulator.
3 Background
Problem Setting: We consider setting where multiple robots perform series of tasks (a) such as cleaning a room or putting groceries in the fridge, in a home-like environment, and (b) rescuing missing personnel and putting out forest fires in search & rescue environment (SAR). These tasks typically require long-hone horizon planning, involving around 100 low-level actions to reach the goal. Our objective is to compute plans for team of robots execute high-level language instructions. /. We formalize these tasks as partially observable Markov decision processes (POMDP) (55] 56). denoted P, N is the number of agents and I is the high-level language
as I, (O+). instruction set. Here, E S represents the joint state of all agents, and o E O denotes the observation for all agents. Particularly, o, e O is the observation set of agent i, that captures incomplete environment state information. a E A = Ai x Ao -N action space comprises of different categories of high-level actions A ANAV AINT U AAXx. where ANAV is the joint navigation action set, AINT is the joint interaction actions which allow the agents to interact with objects, and AEXP are the joint exploration actions which allow the agents to explore the environment. Examples of the high-level actions include Pi PickUp (object) AINT and NavigateTo (location) E ANAV.. Each high-level action is associated with low- level primitive action (pre-trained RI, behavior cloned, or heuristic-based policy). These actions are executed synchronously by agents every high-level decision step. ((ss|s. a) the joint transition probability function that defines the probability of arriving at s' E S after taking joint
An represents the joint action space. The joint
3

---

1-t+1
I
I
Verifier LM
0,11
Oc-1
Environment
G0.+1
Go.-11
Cc++
Go,11
9r
Me-1
Gcs
Oe
M
M,
Actor LM
Oct
Open Subtasks Goz ["transport tomato to tridge". "pick lettuce place the fridge")
Completed Subtasks 9cri I'lnnd fridge", "look for groceries in the kitchen']
Language Task Instruction I: Fetch the groceries and place them in the fridge
Observation oy:
OLe: I tomato, lettuce, and sink 02ui1 can see tomato, bread, and a fridge
Actions a,: cpickup (Lettuce), Navigate (Tomato)
Memory M, Fridge
location co-ordinates
Low-level primitive feedbback dy List of that
(1, 2) facing south. Tomato kept
Corrected actions G...' Mavigate (Apple),
conter shelf and lettuce is
countertop near the sink.
None]
Figure An overview LLaMAR's modular cognitive architecture. LLaMAR leverages LMs within four key modules: Planner, Actor, Corrector, and Verifier. each with specific roles. The Planner breaks down the high-level language instruction into feasible subtasks to achieve the environment goal. The Actor determines the high-level actions each agent should perform. These actions trigger low-level policies that generate and execute sequence of primitive actions in syne across all agents. Based exccution foedback, the Corrector suggests corrections for high-level actions and the Verifier Module validates complction subtasks.
action E A in E S. [gl.-... defines the subtasks that the agents need to perform to accomplish the language instruction task. is the length of the planning horizon.
Environment: To simulate open-ended, long-horizon tasks that resemble everyday activities in home-like environment, we use the A12Thor simulator [T], which supports diverse set of interactions and photorealistic rendering. Since our approach does not require any parametric training, can potentially translate other similar embodied environments like VirtualHome (57], Habitat
42 58 [59,, and ThrleDWorld [60], possibly extending beyond household domains. Similarly, to
simulate the search & rescue scenario, we create custom Search Rescue environment (SAR). More information about the MAP-THOR and SAR environments can be found in Appendix B and respectively. We instantiate the problem with N agents cooperating to accomplish a long-horizon rearrangement task [6]] in indoor environment. The agents do not know the objects present in the environment a priori and are encouraged to explore the environment to gather more information to complete the task. Unlike previous solutions in similar settings [22 [9 20), do not rely oracle/simulator to verify subtask completion. In prior works, predefined conditional satisfiability check is used subtask completion feedback.
Approach
We describe approach in this section. Figure ILLMMRRS architecture comprising four modules: Planner, Actor, Corrector, and Verifier, each an LM with a distinct role. Prior work [62] shows that splitting roles across different LMs improves performance in sequential decision-makin Our initial experiments confirm that LMs tasked with reasoning about multiple inputs and providing long outputs perform poorly. We iterate through these four modules every high-level decision step. The pseudocode for our approach in Appendix We define some key notation below:
Memory M: A textual description of the joint memory of all agents, summarizing past obser- vations, high-level actions, plausible reasons for action failures, and specific subtasks that each agent is attempting to solve.
Open Subtasks Go C G: Feasible subtasks proposed by the Planner LM to achieve the environ- ment task that are yet to be accomplished by the agents.

---

Completed Subtasks G3 C G: Subtasks completed by the agents. Corrective Actions agi Corrective actions for each agent based on failure information from the previous step.
At the start of each episode, Memory M, Open Subtasks Go, Completed Subtasks Is, Actions a, Corrective Actions ac, and Failure Information F are initialized as empty sets,
Consider an example of a kitchen with groceries, fridge, and counter. Two agents are tasked with "Fetch the groceries and place them in the fridge". This example will help illustrate the utility of each module. All LMs receive language task instruction I, joint observations from all agents, and information about open and completed subtasks and memory unless stated otherwise. We next discuss the various components in architecture in detail:
Planner Module The Planner LM module suggests feasible subtasks to ensure the completion of the environment task. This method, similar to SmartL.MM's [19] zero-shot planning, uses only observations in the agents' field of view. The Planner suggests subtasks related to objects seen in the current observation memory of all the agents. For the example considered, it decomposes the task into subtasks like "transport the tomato to the fridge" and "transport the lettuce to the fridge", which are added to Go-
Actor Module The Actor LM additionally uses corrective actions suggested by the Corrector module in the previous time step to predict high-level actions for the current step. These actions are then executed in the environment to progress subset of subtasks in the open subtask set Go and accordingly updates the joint memory. For instance, the Actor module might suggest actions such as a Pickup (Tomato), NavigateTo (Lettuce)), updating memory with "We saw tomato on the counter-top, Alice is picking up the tomato, and Bob is navigating to the lettuce".
Corrector Module The Corrector LM self-corrects high-level actions suggested by the Actor LM after controller failures in the previous step's execution It suggests corrective high-level actions and provides reasons for failures and chosen corrections. For example, it might suggest "The action of picking up the tomato failed because it is too far away. Alice first needs to navigate closer to the tomato." "; a. [Naviateeoo (Tomato). (Toomat)) None].
Verifier Module After executing high-level actions, the Verifier LM assesses whether these actions have completed any subtasks in the open subtask set. Successful subtasks moved to the completed subtask set. Without the Verifier LM, the method would need to rely on the simulator/oracle for success or failure information. The Verifier LM along with other information uses the successfully executed high-level actions proposed by the Actor LM to predict subtask completion. For example, after transporting the lettuce to the fridge, the Verifier updates the completed subtasks with "transport lettuce to the fridge".
Admissible Action parsing with Semantic Translation When LMs generate action plans, natural language outputs often fail to translate to executable high-level actions. This happens when the output does not match the predefined format refers to unrecognized contextually similar objects. We use a cosine similarity method from 13 fine-tuning a pre-trained sentence-B [63] to transform the free-form text into admissible high-level actions. Hyperparameters and additional details of the sentence transformer fine-tuning are provided in Appendix |
Exploration Strategy In unexplored environments, agents need to search for task-relevant objects. If agents cannot find the required objects, the language model can choose an 'exploration' action heuristic to determine the choice of region to be explored.
Geep- We use semantically The agent rotates to four cardinal directions d E N North, South, East, W est, capturing image observations 0n,d. These images are processed through pre- trained CLIP image encoder [64] to obtain embeddings Ja- The list of open subtasks Go is processed through the corresponding CLIP text encoder to get text embeddings 90,i- The exploration score E in direction d is defined as [ool -o. [| The direction with the highest score d= = arg max, Ea is chosen. Summing the scores helps select the best direction to explore in expectation. The agent rotates towards d' and moves = steps, repeating this process K times in one explore action. This approach ensures that images relevant to identifying potential subtasks are prioritized. For example, if Go includes "locate a computer", it is more likely to find a computer on a table than on a sofa, resulting in a higher
ε, = ==
2 We use the simulator just to provide a boolean value about the success of high-level action execution.
5

---

cosine similarity score between the subtask CLIP text embedding and table CLIP image embedding. Refer to Appendix B.2 for more details about the exploration heuristic.
Motivation for Proposed Framework The specifie order of the modules in LLaMAR is due to natural causal relationships in which environment feedback is received. We use the Planner as the first module because it allows LLaMAR to come up with an initial list of open subtasks that could be completed based on the current observation and past memory to satisfy the task. This list serves as a rough high-level plan. The actor then uses this information to suggest the necessary actions. The Corrector is used after the Actor module to identify reasons for failures in the execution of the actions suggested by the Actor Note that the failure module is inert and only suggests corrective actions. Only the Actor module decides the final actions be executed. This role distinction allows for clear reasoning on failures when they occur and lets the actor module focus on choosing actions. The Verifier used after the action is executed to update the list of closed subtasks so that L.LMMRR can be current with the progress toward the completion of the environment task. This allows the planner to update the list of open subtasks in the next step. In essence, the Planner and the Verifier ensure that the progress of the agents is tracked and the actor and the corrector ensure that the actions are executed successfully to advance towards completion of the task.
Multi-Agent features in LLaMAR While our method can be easily adapted to a single-agent setting, our design choice for the architecture was motivated to include the following aggett features:
Agents share their state information with the centralized LLaMAR modules predict actions, enabling them coordinate and avoid conflicts. This information sharing allows for the agents to cooperate and achieve the collective goal.
Coordination through communication:
Dynamic Role Assignment: Agents are dynamically assigned roles based on the current task requirements and their capabilities. This flexibility allows LLaMAR to adapt to changing environ- ments and task demands.
Hierarchical Task Decomposition: To handle the complexity of multi-agent planning, LLaMAR decomposes the action space by creating specific subgoals/subtasss available for any agent to assign itself (done by the actor module) based on the observation and current context. This decomposition reduces the overall search space and improves planning efficiency.
Experiments
MAP-THOR: To evaluate the performance of LLaMAR and benchmark other baseline methods, we create a benchmark dataset of tasks which we call MAP-THOR (Multi-Agent Planning tasks in AI2-THOR). While Smart-LLM [19] introduces a dataset of 36 tasks within A12-Thor [4] classified by complexity, their tasks are limited to single floor plans. This limitation hinders testing the robustness of planners across different room layouts. Additionally, some tasks in their dataset cannot be performed by multiple agents, regardless of task division, such as Pick up the pillow, Open the laptop to turn it on, and Turn off the lamp.
By contrast, MAP-THOR includes tasks solvable by both single and multiple agents. We classify the tasks into four categories based on the ambiguity of the language instructions. To test the planner robustness, we provide five different floor plans for each task. We also include automatic checker modules to verify subtask completion and evaluate plan quality. Our dataset comprises 45 tasks, each defined for five distinct floor plans, ensuring comprehensive testing and evaluation.
We conduct experiments with tasks of varying difficulty levels, where an increase in difficulty of the tasks corresponds to an increased ambiguity in the language instructions. The complete task list of each category can be found in the Appendix C
Explicit item type, quantity, and target location: Agents are explicitly instructed to transport specific items to specific target locations. For example, put bread, lettuce, and a tomato in the fridge clearly defines the objects (tomato. lettuce, bread) and the target (fridge). Explicit item type and target location but implicit item quantity: The object type is explic- itly described, but its quantity not disclosed. For example, Put all the apples in the fridge. Agents must explore the environment to locate all specified items and also predict when to stop.
. Explicit target location but implicit item types and quantity: The target location is explicitly defined but the item types and their quantities are concealed. For example, Put all groceries in the fridge.

---

Implicit target location, item type, and quantity: Item types and their quantities along with the target location are implicitly defined. For example, Clear the floor by placing the items at their appropriate positions. The agent is expected to place items like pens, books, and laptops on the study table, and litter in the trash can.
Search & Rescue Environment (SAR):
To showcase the effectiveness of LLaMAR with respect to explicit coordination in multi-agent settings, we evaluate LLaMAR in a partially observable search & rescue and fire relief environment in a grid world. Depending on the scene, there is a mix of missing people to be found, and wildfires to be stopped before they spread geographically. More details about the environment can be found in Appendix D
Fire Extinguishing: Fires consist of expansive flammable regions with a fixed set of sources that propagate over time. The rate of fire spread is proportional to its intensity; higher intensities result in faster spread. Fires categorized as either Class A Class B, which extinguished using water or sand, respectively. These extinguishing resources are sourced from reservoirs distributed across the environment.
Each individual is initially located at an unknown position within the environ- ment. The objective to locate, rescue, and transport the individuals to a designated drop-off location, which is known beforehand. Transporting a person requires the coordinated effort of two agents simultaneously, who must carry them to the specified drop-off point.
Human Rescue:
Metrics
We evaluate the algorithms using the following metrics to compare their performances on the tasks:
Success Rate (SR): The fraction of episodes in which all subtasks are completed. Success equals
1 if all subtasks are successfully executed in an episode, otherwise it is O. Transport Rate (TR): The fraction of subtasks completed within an episode, provides a finer granularity of task completion.
Coverage (C): The fraction of successful interactions with target objects. It is useful to verify the LMs can infer the objects to interact with, in scenarios where the tasks have objects that are specified implicitly.
Balance (B): The ratio between the minimum and maximum number of successful high-level actions executed by any agent that contributed towards task completion. We only check for a subset of high-level actions that must be executed to accomplish critical subtasks that lead to the successful completion of the language instruction task. If each agent i out of n agents completes
(s1."".8n)) 8; successful tasks, the balance is defined as: B == max/ snafe" This measures how evenly the work is distributed among agents. A balance of zero indicates at least one agent performed no successful high-level actions, while balance of one indicates all agents performed the same number of successful high-level actions. Here = le 4 is small number to avoid division by
zero.
Average steps (L): The number of high-level actions taken by the team to complete the task, capped at L = 30 in our experiments. If the task is not completed within L steps, the episode is deemed failure. Note that the metric L is presented in the table providing the complete results, located in Appendix
For all the metrics, we report the means along with the 95% confidence interval across all the tasks (refer appendix[F]oor complete results). Since SR is a binomial metric, we report the Clopper- Pearson Interval as the confidence interval.
Baselines
For a fair comparison with our method, we make modifications to the baselines to make them work in partially observable settings with limited reliance the simulator. More details about implementations can be found in Appendix H
Act: We query the LLM with the task and the observations to suggest a high-level action.
Chain-of-Thought 65 We modify the Act prompt with a thain-off-though style addendum to let the LM reason about the possible implications while selecting high-level action.
ReAct [66]: We use a ReAct-style prompting to let the LMs reason after suggesting high-level
actions and possibly suggest ways to correct any failures.
SmartLLM [9]: We modify the official codebase to only include information from the local observations of the agents instead of assuming full observability.
7

---

We modify the list of available high-level actions to include all possible valid combinations of actions with interactable objects in the agent's 's local observation. As the scene becomes more cluttered, this list and the prompt becomes combinatorially longer. In the original implementation, the list of available actions is filtered based on the feasibility of the actions as suggested by conditional checker.
CoELA 12]:
It should be noted that Act, Chain-of- Thooughht, ReAct, and SmartLLM are all CMAS frameworks where CoELA follows the DMAS framework.
6 Results and Discussion
Choice of the underlying LM: To understand the impact of the underlying LM's quality on decision- making, we initially experimented with different LMs on MAP-THOR. Specifically, we utilize both the language-only and rision-language models of GPT-4 [7]. IDEFICS-2 2 [68 69] LLaVA [70] 7 and CoGVLM [72). Among these, GPT-4, when used solely with text inputs, exhibits the poorest performance. This is attributed the agents' inability to reason about visual observations, which is particularly detrimental for the Corrector module. Substituting GPT-4V with other vision-language models results in a decline in performance (refer Table 2 and hence we use GPT-4V as the underlying VLM while comparing to the baselines.
Baseline Comparisons: T compares our method, LLaMAR, with other baselines in 2-agent scenario using GPT-4V as the underlying VLM. Act and ReAct show similar performance, with Act struggling due to its lack of strategic planning or correction, and ReAct performing slightly better by dynamically adjusting actions based on reasoning on immediate feedback. CoT's performance declines with longer planning horizons due to its inability to maintain coherence over extended planning sequences, consistent with findings in (3), showing its effectiveness only with highly specific prompts. SmartLLM, operating in a plan-and-an and-eeetteeeette paradigm, generates impractical plans with issues like infinite loops and failure to handle low-level action failures, leading to lower success rates and poor transport metries. It also tends to hallucinate objects. CoELA, using decentralized multi-agent system (DMAS), performs poorly due to large input prompts and struggles to select the correct action from numerous choices. Its decentralized decision-making is less efficient than the centralized multi-agent system (CMAS) used by LLaMAR. Previous research [66] confirms CMAS frameworks are more effective than DMAS frameworks. Overall, our method, LLaMAR, benefits from its modular cogmitive architecture, which integrates planning. acting, correcting, and verifying through distinct LLM roles, resulting in superior performance across various evaluation metrics, By avoiding reliance on privileged information and incorporating robust exploration strategy that allows to scout for objects that are not initially visible, LLaMAR ensures higher success rates and balanced task execution among agents.
To demonstrate the effectiveness of the various modules in our cognitive architecture, we performed ablation studies by evaluating performance metrics with each module removed individually. The results are summarized Table Using only the Actor module corresponds to the "Act" baseline. which demonstrates its fundamental capabilities in isolation but shows limited effectiveness without planning and correction due to relatively lower success and transport rates. Adding the Planner and Verifier modules improves performance, benefiting from better task planning and validation, increasing the overall SR and TR, and ensuring more effective task completion and even work distribution, as indicated by the increase in balance (B). However, in scenarios where the suggested action fails, the actor suggests the same action in the next decision step since it is not able to reason why the action failed until the end of the planning horizon. Incorporating the Corrector module, with access to privileged information from an environment oracle, significantly boosts performance, enhancing the SR, TR, C, and further improving B, consistent with the findings in [17]. This highlights the Corrector module's importance in adjusting actions based on controller feedback. resulting in higher task success and more efficient task completion, albeit with reliance on oracle knowledge. Finally, the complete LLaMAR system, without privileged information, achieves SR. TR. C. and B values close to those of the oracle setup. This demonstrates the system's 's robustness and effectiveness in a realistic setting. The Corrector module plays a crucial role in enabling agents to learn from past failures and avoid repeating actions, preventing task failures due to timeout. Despite lacking oracle knowledge, LLaMAR performs nearly as well as the enheneee setup. These results highlight the importance of each module in our cognitive architecture. Removing any module diminishes effectiveness.
Roles of different modules in LLaMAR:
Increasing the number of agents Increasing the number of agents in the environment shows distinct
8

---

Algorithm
LM
SRT
TRT
C7
Act
GPT-IV
0.33
0.67
0.91
0.59
Modules Used
ReAct
GPT-4V
0.34
0.72
0.92
0.67
SR t
TR t
C
B
CoT
GPT4V
0.14
0.59
0.87
0.62
Actor
0.33
0.67
0.91
0.59
SmartLLM
GPT-VV
0.11
0.23
0.91
0.45
Planner
CoFLA
GPT-4V
0.25
0.46
0.76
0.73
Actor +
0.45
0.78
0.92
0.69
LLaMAR
GPT-4
0.51
0.85
0.95
0.83
Verifier
LLaMAR
ILaVA
0.54
0.84
0.91
0.75
Planner +
LLaMAR
IDEFISS22
0.57
0.86
0.94
0.78
Actor
0.67
0.91
0.97
0.84
LLaMAR
CogVIM
0.61
0.89
0.95
0.80
Corrector
LLaMAR
GPT-VV
0.62
0.87
0.95
0.82
Planner +
expl)
Actor
LLaMAR
0.66
0.91
0.97
0.82
GPT-4V
0.66
0.91
0.97
0.82
Corrector
(w/cxpl)
Verifier
Table 2: Comparison of evaluation metrics against base- lines averaged across all tasks for the 2-agent MAP- THOR scenarios. The complete table with confidence intervals can found in Appendix(F] More details about peculiar behaviors for the baselines can be found in Ap-
Table 3: Performance in the 2-agent scenarios in MAP-THOR obtained by ablating different mod- ules L.M.MAR with GPT-4V as the underlying
I.M
pendix H
MAP-THOR
SAR
# of agents
SR
TR
Ct
B1
SR1
TR
Cf
B
1
0.37
0.67
0.87
1.00
0.28
0.75
0.86
1.00
2
0.62
0.87
0.95
0.82
0.44
0.86
0.94
0.91
3
0.70
0.91
0.98
0.66
0.68
0.92
0.96
0.80
4
0.68
0.90
0.99
0.62
0.72
0.94
0.98
0.78
5
0.62
0.90
0.99
0.54
0.74
0.96
1.00
0.73
Table 4: LLaMAR with various number of agents in the scenario in both MAP-THOR and SAR environments
trends in method's performance metrics for both MAP-THOR and SAR environments (refer Table In MAP-THOR, with two agents, we establish a solid baseline for success rate (SR) and transport rate (TR), which is similarly reflected in the SAR environment. Adding third agent improves both SR and TR in both environments, indicating enhanced task completion and transportation efficiency. Coverage (C) also increases, suggesting better exploration and interaction with objects across both environments. There is a slight decrease in SR and TR when the number of agents increases from to in the MAP-THOR environment. The decrease these metrics can be attributed the rooms in MAP-THOR becoming crowded with and agents hence blocking the agents from navigating without colliding with other agents. But this phenomenon is not seen in the SAR environment which is comparatively more spacious and navigable. However, balance (B), which measures the even distribution of tasks among agents, decreases with more agents. This drop highlights the challenge of ensuring equal contributions from all agents in a larger multi-agent system. While SR remains high, the balance metric drops significantly from 2 to agents, indicating some agents do more work than others. In summary, adding more agents improves task performance and efficiency but introduces challenges in maintaining balanced contributions. Addressing this imbalance is crucial for refining multi-agent planning algorithms.
Correcting Failures: In numerous instances, the actions proposed by the Actor module, such as pick <object>>, are unsuccessful due to the agent's insufficient proximity to the target object. In such situations, the Corrector module recommends appropriate corrective actions, such as navigate to <object> to facilitate closer proximity. Figure shows examples where the Corrector module interprets low-level action failures and suggests remedies, highlighting its importance.
visual feedback to learn from these failures and
7 Limitations and Future Work
Higher number queries to the LM: Since each high-level decision step requires querying 4 different LM-based modules, the cost and the compute times are higher than other baselines, especially compared to the plan-and-execute baselines like SmartLLM. An interesting future direction to

---

Tack: bow, mug and plate frdge
bow, mug plate the trigge
Task Put tssuebok, end remate control table
Acsor action "Pick up
Corrective action Ne plaae
Aoter action: "Put plas in frigee Vroge coen.
Corrective wction: "Oper ridgo"
Actor action "Naxgua
NNniiada to iat
Corractive
blocked obstaole
"Rotalo
Corrector module reasoning: Aloce led 30 pick up the p earier becsuse aht wis ciose ctject attompting mawigate disee
Corrector module reasoning: "Alie lailed to put plan resige becsuse the frkdow is not open:
Corrector module reasoring: Alase ahocald rotate the right get clear
spen tidge
path
Insido"
Figure few examples the Corrector module mitigate failures in predicted actions by the Actor module. (a) the Corrector suggests getting closer to the agent before attempting to pick it up. (b) the Corrector recommends opening the fridge because the previous action of placing the plate failed, (c) the Corrector advises rotating right that access the table to place the tissue box on it when the low-level navigation policy failed to find path to the table
improve this would be to fine-tune smaller LMs with trajectories collected in the simulator (eg: ALFRED (46) done in [22]. Another potential direction worth exploring is using different sizes of L.M for each module based on their specific utility.
Limited spatial reasoning: Although use both textual descriptions and visual features to guide the language model's actions, it still lacks the ability to reason about the spatial features of the environment. Spatial reasoning is crucial in scenarios such navigating around obstacles to reach an object, or determining the shortest path to collect multiple items scattered across different locations. One way to address this limitation is to inject information about the 3D world into the LM, as done in [74), which is an interesting direction for future work.
Performance limited by the underlying VLM: Although LMs make correct reasoning most of the time, they still occasionally make mistakes, including misunderstanding the environment rules specified in the prompt. For example, the agent assumes that the cleaning task requires putting soap, drying, and putting it in the sink when all it needs is the action "CleanObject", and can't figure out the appropriate level of abstraction. The performance of the algorithm is limited by the instruction following and reasoning capability of the underlying LM 75 76]; this calls for developing LMs that fine- muned to inttucttionimage pairs relevant to the environment (as done in (22).
Conclusion
We address long-horizon planning in dynamic, partially observable multi-agent environments with LLaMAR, an LM-based planner using four specialized modules: Planner, Actor, Corrector, and Verifier. This framework iteratively refines action planning, adapts to failures, and verifies subtask completion using real-time observations and action feedback, without privileged information. We also introduce a theuiisti-bbase exploration strategy to guide agents to semantically relevant regions. Additionally, present MAP-THOR a benchmark dataset for multi-agent tasks in the AI2Thor simulator. Empirical results show LLaMAR outperforms existing LM-based approaches, achieving a 30% higher success rate on MAP-THOR.
Acknowledgements
We would like thank Keerthana Gopalakrishnan, Sydney Dolan, Jasmine Aloor, and Victor Qin for helpful discussions and feedback. OpenAl credits for GPT-4 access was provided through OpenAl's Researcher Access Program. The research was sponsored by the Department of the Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number FA8750-199 1-550019-2--111 1000. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Department of the Air Force the U.S. Government. The U.S. Govermment is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexan- der G. Schwing. A Cordial Syne: Going Beyond Marginal Policies For Multi-Agent Embodied Tasks. In
10

---

ECCV, 2020. first two authors contributed equally. L
[2] Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G. Schwing, and Aniruddha Kembhavi. Two Body Problem: Collaborative Visual Task Completion. In CVPR, 2019. first two authors contributed equally.
[3] Vijay Kumar. D. Rus, and Sanjiv Singh. Robot and sensor networks for first responders. IEEE Pervasive Computing, 3(4):24 33, 2004.
[4] Matthew Dunbabin and Lino Marques, Robots for Envirommental Monitoring: Significant Advancements and Applications. IEEE Robotics & Automation Magazine, 19(1):24-33 2012.
[5] Brian lchter, Authony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz. Alex Irpan, Eric Jang. Ryan Julian, Dmity Kalashnikov, Sergey Levine. Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanct, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan. Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do As Can, Not As I Say: Grounding Language in Robotic Affordances. In Karen Liu, Dana Kulic, and Jeff Ichmowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 287-318. PMLR, 14-18 Dec 2023. 12
[6] Wenlong Huang, Fei Xia, Dhruv Shah, Danny Driess, Andy Zeng, Yao Lu, Pete Florence, Igor Mordatch, Sergey Levine, Karol Hausman, and Brian lchter. Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 59636-59661. Curran Associates, Inc., 2023. 0
[7] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models. In Karen Liu, Dana Kulic, and Jeff Ichnowski. editors, Proceedings of The 6th Conference Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1769-1782. PMLR, 14-18 Dee 2023.
[8] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox. Jesse Thomason, and Animesh Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages
11533-1530,, 2023.
2
[9] Jacky Liang Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control. In arXiv preprint
arXiv:2209.07753. 2022. 1
[10] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Text2Motion: from natural language instructions to feasible plans. Autonomous Robots, 47(8):1345-1366 November 2023.
[11] Dhruv Shah, Blazej Osinski, Brian lchter, and Sergey Levine. LM-Nav: Robotic Navigation with Large rraTrei Models of Language, Vision, and Action. In 6th Annual Conference on Robot Learning. 2022.
[12] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual Language Maps for Robot Navigation. In Proceedings the IEEE International Conference Robotics and Automation (ICRA), London, UK, 2023.
[13] Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models Zero- Shot Planners: Extracting Actionable Knowledge for Embodied Agents. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 9118 9147, PMLR, 17-23 Jul 2022. 2
[14] Ming Tan. Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents. In In Proceedings of the Tenth International Conference Machine Learning, pages 330-337. Morgan Kaufmann, 1993.
[15] Ardi Tampuu, Tambet Matisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent Cooperation and Competition with Deep Reinforcement Learning. CoRR, abs/1511.08779. 2015. I

---

[16] Yongchao Chen, Jacob Arkin, Yang Zhang, Nicholas Roy, and Chuchu Fan. Scalable Robti Collaboration with Large Language Models: Centralized or Decentralized Systems? arXiv preprint arXiv:2309. 15943, 2023.
[17] Daman Arora and Subbarao Kambhampati. Learning and Leveraging Verifiers to Improve Planning Capabilities of Pre-trained Language Models. arXiv preprint arXiv:2305.17077, 2023.
2
8
[18] Karthik Valmeckam, Matthew Marquez, and Subbarao Kambhampati. Can Large Language Models Really Improve by Self-critiquing Their Own Plans? arXiv preprint a/X1::2110008118, 2023. 2
[19] Shyam Sundar Kannan, Vishnunandan LN Venkatesh, and Byung-Cheol Min. SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models. arXiv preprint arXiv:2309. J0062, 2023.
[20] Jun Wang. Guocheng He, and Yiannis Kantaros. Safe Task Planning for Language- Instructed Multi-Robot Systems using Conformal Prediction. arXiv preprint arXiv:2402. 15368,
15366, 2024. 2
24
[21] Ishika Singh, David Traum, and Jesse Thomason. TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models, arXiv preprint 1xXx::2403.12240 2024.
24
[22] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tianmin Shu, and Chuang Gan. Building Cooperative Embodied Agents Modularly with Large Language Models. ICLR, 2024.
8 I0
[23] Bangguo Yu, Hamidreza Kasaei, and Ming Cao. Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation using Large Language Models. ArXiv, abs/2310.079377, 2023. 2
[24] Zhao Mandi, Shreeya Jain, and Shuran Song. RoCo: Dialectic multi-robot collaboration with large language models. arXiv preprint 23l.::3007004335 2023.
3
[25] Andrew G. Barto and Sridhar Mahadevan. Recent Advances Hierarchical Reinforcement I Learning.
Discrete Event Dynamic Systems, 13:41-77, 2:03--77.003.1 2
[26] Ofir Nachum, Shixiang (Shane) Gu, Honglak Lee, and Sergey Levine. Data-Efficient Hierarchical Reinforvement Learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Besa-Bianchi, and R. Gareett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates,
Inc., 2018. 2
[27] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hicrarchical Reinforcement Learning: A Comprehensive Survey, ACM Comput. Surv. 54(5), jun 2021.
[28] Jan Wohlke, Felix Schmitt, and Herke van Hoof. Hierarchies of Planning and Reinforcement Learning for Robot Navigation. In 202/ IEEE International Conference Robotics and Automation (ICRA). IEEE, May 2021.
[29] Hongyao Tang, Jianye Hao, Tangjie Lv. Yingfeng Chen, Zongzhang Zhang. Hangtian Jia, Chunxu Ren, Yan Zheng. Zhaopeng Meng, Changjie Fan, and Li Wang. Hierarchical Deep Multiagent Reinforcement Learning with Temporal Abstraction, 2019. 2
[30] Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical Cooperative Multi- Alutt--Aeett Reinforcement Learning with Skill Discovery. CoRR, abs/1912 2019. 2
[31] Jesse Meyer, Ryan Pracuner, and Luke Vanderbcek. Hicrarchical Multi-Agent Reinforcement Leam- ing.
htttps://cse. www.eee...e.eeeee..e.......eee.e hwtp:///////eeeee////.......// htt.s///ee......a111aaaaaaaaaaae /t/ss//aarr/eeiiarrrrr...
pdf
2020.
[32] Adam Stooke, Kimin Lee, Pieter Abbeel, and Michael Laskin. Decoupling representation learning from reinforcement learning. In International Conference on Machine Learning, pages 9870-9879 PMLR,
2021.
2
[33] Ruihan Yang, Huazhe Xu, Yi Wu, and Xiaolong Wang. Multi-task reinforcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767 4777, 2020. 2
[34] Sherry Yang. Ofir Nachum, Yilun Du Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. arXiv preprint 230.:233330012299 2023,
[35] Lei Wang. Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang. Jingsen Zhang, Zhiyuan Chen, Jiakai Tang. Xu Chen, Yankai Lin. Wayne Xin Zhao, Zhewei Wei, and Ji- ii-Rong Wen. A Survey on Large Language Model based Autonomous Agents, 2023. 2
12

---

[36] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang. Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang. Limao Xiong, Yuhao Zhou, Weiran Wang. Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin. Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang. Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui. The Risc and Potential of Large Language Model Based Agents: A Survey, 2023. 2
[37] Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive Architectures for Language Agents, 2023.
[38] Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill Induction and Planning with Latent Language. CoRR, abs/2110.01517, 2021.
[39] Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with Large Language Models via Corrective Re-prompting. January 2022. 2
[40] Maitrey Gramopadhye and Daniel Szafir. Generating Executable Action Plans with Environmentally-Awaan Language Models, 2023. 2
[41] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AL arXiv, 2017.
2
[42] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AT Research. CoRR, Abss/990.00120 2019. 2
[43] Fei Xia, Amir R. Zamir, Zhi- Yang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson Env: Real- World Perception for Embodied Agents. CoRR, abs/1808. 10654, 2018. 2
[44] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martin-Martin, Chen Wang. Gabrael Levine, Michael Lingelbach, Jiankai Sun, Mona Anvari, Minjune Hwang, Manasi Sharma, Arman Aydin, Dhruva Bansal, Samuel Hunter, Young Kim, Alan Lou, Caleb R Matthews, Ivan Vilte-Renteria, Jerry Huayang Tang. Claire Tang, Fei Xia, Silvio Savarese, Hyowon Gweon, Karen Liu, Jiajun Wu, and Li Fei-Fei. BEHAVIOR--K K: A Benchmark for Embodied Al with 1,000 Everyday Activities and Realistic Simulation. Karen Liu, Dana Kulic, and Jeff Ichnowski, editors, Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 80-93. PMLR, 14-18 Dec 2023. 2
[45] Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Sarayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilck Hakkani-Thu TEACh: Task-driven Embodied Agents that Chat, 2021.
[46] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. CoRR, Abs/1912.01734, 2019. 1
[47] Dipendra Kumar Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson. Max Shatkhin, and Yoav Artzi. Mapping Instructions to Actions 3D Environments with Visual Goal Prediction. CORR, abs/1809 809,00086.
2018. 2
[48] Yuke 7hu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozheh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In Proceedings of the IEFF international conference on computer vision, pages 492, 2017. 2
[49] Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat. Hugo Larochelle, and Aaron C. Courville. HoME: Houschold Multimodal Environment. CORR, abs/1711.11017, 2017.
[50] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A SimulAted bart-based Interactive ENvironment. CoRR, abs/2003.08515, 2020. 2
[51] Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G. Schwing. and Aniruddha Kembhavi. Two Body Problem: Collaborative Visual Task Completion. CoRR, abs/1904.05879. 2019, 2
13

---

[52] Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing. A cordial syac: Going beyond marginal policies for multi-agent embodied tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt V 16. pages 471-490 1490. Springer, 2020. 2
[53] Chan Hee Song. Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LLM- Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models, 2023. 33
[54] Lin Xu, Zhiyuan Hu, Daquan Zhou, Hongyu Ren, Zhen Dong, Kurt Keutzer, See Kiong Ng, and Jiashi Feng. MAgIC: Benchmarking Large Language Model Powered Multi Agent- in Cognition, Adaptability, Rationality and Collaboration. arXiv preprint 1iXii:2311108502. 2023, 3
[55] Martin L. Puterman. Markov Decision Processes. Wiley, 1994.
3
[56] Leslie Pack Kaelbling, Michael I., Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101 134, 1998. g
[57] Xavier Puig. Kevin Ra, Marko Boben, Jiaman Li. Tingwu Wang. Sanja Fidler, and Antonio Torralba. VirtualHome: Simulating Household Activities via Programs, 2018. 4
[58] Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimír Vondruš, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 4
[59] Xavi Puig, Eric Undersander, Andrew Szot, Mikael Dallaire Cote, Ruslan Partsey, Jimmy Yang. Ruta Desai, Alexander William Clegg, Michal Hlavac, Tiffany Min, Theo Gervet, Vladimír Vondrus, Vincent- Pierre Berges, John Turner, Oleksandr Maksymets, Zsolt Kira, Mrinal Kalakrishnan, Jitendra Malik, Devendra Singh Chaplot, Unnat Jain, Dhruv Batra, Akshara Rai, and Roozbeh Mottaghi. Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots, 2023. 4
[60] Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel L. K. Yamins, James DiCarlo, Josh McDermott, Antonio Torralba, and Joshua B. Tenenbaum. The Three- DWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically
Realistic Embodied AI, 2021. 4
[61] Dhruv Batra, Angel Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladien Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. Rearrangement: A Challenge for Embodied AL. CoRR, abs/201 1. 101975, 2020. [
[62] Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. ADaPT: As-Needed Decomposition and Planning with Language Models, 2023.
[63] Nils Reimers and Iryna Gurevych. Sentence- Centence-BERT:: Sentence Embeddings using Siamese BERT-Networks. 2019.
S
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. CoRR, abs/2103.0000 2021. 5
[65] Jason Wei. Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoe Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. CORR, abs/2201. 1 19003 2022.
[66] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models, 2023. 7
[67] OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jcff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-L.uis Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning. Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman,
14

---

Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni. Gabriel Goh, Rapha Contijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu. Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billic Jonn, Hecwoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee. Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin. Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe. Patricia Lue, Anna Makanju. Kim Malfacini. Sam Manning. Todor Markov, Yaniv Markovski. Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine Melcavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishehenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajecv Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambatiist Parascandolo, Joel Parish. Emy Parparita. Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipee de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shihani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherhakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Fric Sigler. Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky. Yang Song. Natalie Staudacher. Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang. Nikolas Tezak, Madeleine Thompson, Phil Tillet. Amin Tootoonchian. Elizabeth Tseng, Preston Tuggle, Nick Turley, Jenry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang. Marvin Zhang. Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 Technical Report, 2023.
g
[68] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents, 2023.
g 32
[69] Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision- language models?, 2024.
32
[70] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. In NeurlPS, 2023.
32
[71] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved Baselines with Visual Instruction Tuning, 2023.
[72] Wenyi Hong, Weihan Wang. Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang. Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. CogAgent: Visual Language
Model for GUI Agents, 2023. g 32
[73] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of Thoughtlessness: An Analysis of CoT in Planning. arXiv preprint arXiv:2405.0477% 2024. g
[74] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3D-LLM: Injecting the 3D World into Large Language Models, 2023. To
[75] Subbarao Kambhampati. Can large language models reason and plan? Annals of the New York Academy of Sciences, 1534(1):15-18, 2024. 10
[76] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36.75993-76005. 2023.
10
15

---

A Terminology
We differentiate between the terms subtasks and high-level actions in this section. In essence, multiple high-level actions are needed be carried need to be satisfied to complete the high-level language instruction.
sequence complete subtask. Multiple subtasks
Subtasks: A task is split up into multiple subtasks. For example, if task is "Fetch all the groceries and put them in the fridge". then the initial subtasks could include: "Locate the groceries", "transport the groceries". "Locate the fridge". These subtasks could get updated with new observations. For example, while locating the groceries, the agents come across tomato and lettuce. Then the subtasks "transport the tomato the fridge" and "transport the lettuce the fridge" gets updated in the subtasks list. This basically splits up the high-level instruction I into multiple mid-level subtasks
These are the set of actions required to complete the subtasks. For example, to complete the "transport the lettuce the fridge", we would require: the following set of actions:
High-level actions:
Navigate to lettuce
Pickup lettuce
Navigate to the fridge Open fridge
Put lettuce in the fridge
Close fridge
Note that different agents have to complete different high-level actions that progress the subtasks efficiently whilst avoiding conflicts.
Conflicts arise in the following ways:
Same actions:
Agents performing the same action at the same time. Example: "Open the
fridge".
Blocking: Agent is blocking Agent and not allowing it to complete its high-level action. Example: Agent 1 is attempting to execute PlaacOOject(Tomalle place the tomato in its hand in the fridge and Agent is attempting to execute "OpenFreezer("" needs interact with the fridge. Would require some form of conflict resolution the state cell domain. Agent should move away to allow fridge access to Agent 2. In LLaMAR. the Corrector module helps figuring out these conflicts and suggest different corrective high-level actions.
front of the fridge to
B MAP-THOR Environment
The environment is based the A12Thor simulator with multi-agent setup. All the experiments were performed in the single-room floor plans. When more than agents are added to some floor plans (especially the kitchen floor plans), the environment gets crowded and does not allow for a lot of free space to navigate to different objects (the number of reachable paths reduces).
(a) Kitchen
(b) Bedroom
(c) LivingRoom
(d) Bathroom
Figure 3: Photorealistic rendering of household scenarios in the A12Thor simulator enables the usage of multiple automomous robots to carry out daily tasks.
B.1 Observation Space
The observations each robot include an image size resolution 1000 x 1000 x 3. The textual observation for each agent in the prompt is the list of objects visible in this image and the agents' current location and rotation. The field of view 90 degrees. The agents can interact with the objects only if it is within its visibility range of 1.5m.
16

---

B.2 Action Space
The actions space A consists of navigation actions ANAV, interaction actions AINT, exploration action AExr.
Navigation actions AN consists of the following actions:
Move (wvd<<rrectoo>))) Moves the robot by 0.25m towards the specified direction where <direction>> can be one of (Ahead, Back, Right, Left)
Rotate (<dite(<direction::
Rotates the robot by 90 degrees towards the specified direction where, <direction> can be one of (Right, Left)
LookUp rotates the pitch of the robot camera upwards by the specified angle.
LookDown<anggle>>>>>> rotates the pitch of the robot camera downwards by the specified angle. NavigateTo (a (<object, id>> makes the rohot navigate the specified object. The path is found using the A shortest path algorithm. Note that the robot is only able to find path to the specified object in the environment only it has encountered that object previously during the episode. Otherwise, the NavigateTo action will be unsuccessful and the agent will have to explore.
Interaction actions AINT consists the following actions:
Pickup Picks up the object
Put (<treeeptaacle_iddd)) Puts the object the rohots hand on the receptacle
Open(<objectt_id>>> Opens the object
Close<<object_idd>) Closes the open object
Slice Slice(<object_ide) Slices the object
Clean (lban(<object_id>>> Cleans the object
Foggle0n<<objeett
Toggles the object on
Toggleoff
Toggles the object off
Explore action Agxp: The explore action is carried out by the heuristic explained in Algorithm model for the CLIP weights which
|
and Figure 4 We use the
download from
The action space consists of several executable actions, including (ooireedtc))))))) Rotate (fdrrettd)))))on))) LookUp (<angl<<a))e>>, LookDoun(<anglees NavigateTo (<object_ide Pickup (cobkup(oooecttbiiie jebpjeettd>>>>>>> Put (<receptacle_idd> and others. These actions can be combined in numerous ways, as the <object_idse <directions>, and <angles> can vary significantly, resulting combinatorially large action space. To address this complexity, we do not impose constraints on the language model (LM) modules to select actions in predefined format from this Instead, allow for free-form language outputs, offering greater expressivity. The SentenceBERT module then maps these free-form natural language outputs to executable actions within the environment.
Exploration Store
S
Ral
Boyh laal [4
Borfs
CLIP Encader
Is
CLIP Text
J/
Rd
Burds To
Enoocer
computer
14
&
Open Subtiakk
$o
Explaration (ussa(e)
Figure 4: Choice direction for the exploration heuristic: The agent (Alice) rotates towards 4 cardinal directions to get observations. The cosine similarity between the CLIP embeddings la for these 4 images are calculated with the CLIP embeddings for each subtask in the open subtasks Go get the exploration score &a for each direction. The direction with the highest is chosen explore and agent moves 2 steps in that direction.
17

---

Algorithm A12Thor Exploration Heuristic
Input: Agent ID n, Environment ent, number of exploration steps K, number of move steps J
1: 90 CLIP (ex))
2: while k < do
3: Exploration Score &
4: for E { (North, South, East, West) do
5: 0n env.step( (eeeggettotteegggh n))
6: La CLIP (On,))
7:
Ea TTalll loel
1a-80
8:
end for
9:
d' arg max, &
while
do
11:
0N www. env.step(Move(d'' j j+1 end while
13:
t- k
15: end while
C MAP-THOR Task Types
The complete list of task for each task type:
Explicit object type, explicit quantity and target:
put bread, lettuce, and a tomato in the fridge
Put the pots and pans the stove burners
Slice the bread and tomato and crack the egg
Put the butter knife, bowl, and mug in the sink
Turn off the faucet and light if either is on
Put the tissue box, keys, and plate in the box
Put the computer, book, and pen on the couch
Put the bowl and tissue box on the table
Put apple in fridge and switch off the light
Put the watch and Keychain inside the drawer
Wash the bowl, mug, pot, and pan
Put the Box on the sofa and the bowl in the box
Explicit object type and explicit target: Here we explicitly describe the object type but keep the quantity of the objects ambiguous. E.g. all the apples in the fridge. For this, the agents have to explore the environment ensure that they find all of them.
Open all the drawers
Open all the cabinets
Turn on all the stove knobs
Put all the vases the table
Put all the potatoes in the bowl
Put all pencils and pens in the box
Move all lamps next to the door
Turn off all light switches
Turn on all light switches
The object types are implicitly defined whereas the target is explicitly defined. E.g. Put all groceries in the fridge. This tests whether the model can identify objects of certain categories.
Explicit target, implicit object types:
Put all groceries in the fridge (should identify the tomato, bread, apple, potato, and lettuce)
Put all shakers the closest drawer (should identify the salt shaker and pepper shaker)
Put all tableware on the countertop (should identify the bowl, plate, mug)
Put food on countertop (should identify the tomato, bread, apple, potato, and lettuce)
Put all school supplies on the couch (should identify the pencil, computer, and book)
Put all kitchenware the cardboard box (should move the bowl and plate)
Put all silverware in the sink
18

---

Move everything on the table to the desk (should move the laptop, pencil, pen, plate, credit card, book, and newspaper)
Slice the lettuce, trash the mug and switch off the light
Put all electronies on the couch
Make dish by microwaving eggs and tomato
Put all readable objects on the sofa
Wash all fruits
Implicit target and object types: Here both the object type and the target are implic- itly defined. E.g. Clear the floor by placing the items at their appropriate positions. Here the model expected to keep items like pens, book, laptop on the study table, litter in the trash can, etc.
Clear the floor by placing items at their appropriate positions (depending on what's on the floor)
Clear the table by placing the items in their appropriate positions (depends on the floorplan, e.g. bread, apple, tomato, knife, bowl, book)
Clear the countertop by placing items in their appropriate positions (should move the lettuce, mug, and paper towel roll)
Clear the desk by placing the items in other appropriate positions (should move the statue,
watch, and remote control)
Clear the table by placing the items in other appropriate positions (should move the book, credit card, laptop, plate, newspaper, pen, and pencil)
Clear the couch by placing the items in other appropriate positions (should move the pillow)
Make the living room dark
Make mug of coffee and toast the bread
Trash all groceries
Slice all sliceable objects
D Search & Rescue Environment (SAR)
The Search & Rescue enviromment consists of multiple agents in an unknown environment that has multiple wildfires and missing personnel in the environment. The agents are tasked to extinguish all the fires before they spread and rescue the missing humans. Here, each fire is composed of a large flammable region with fixed set of sources that spread through time. The higher the intensity, the faster the fire will spread. The fires can be of class A or B, extinguished through the of water and sand respectively. Both these resources can be collected through resource reservoirs spread geographically. Each person is initially stranded in unknown location. The goal is to rescue and transport each person to drop-off location (known apriori). The person must have two agents simultaneously carrying them to be transported to the drop-off location.
Figure 5: The search rescue environment consists of multiple drones an unknown environment with missing people, fires of different types, and water and sand reservoirs.
provide justification to why this task is relevant to long-term planning and multi-agent collabo- ration below.
19

---

D.1 Multi-agent Nature
Scalability Unlike the AI2THOR environment, the search & rescue environment is spa- cious and hence prevents congestion amongst agents. With the addition of multiple parallel emergency disaster scenarios once, the environment scales comfortably to agents. Addi- tionally, the complexity slowly increases as the model has to coordinate between more actors and competing emergencies.
Tasks requires explicit cooperation-
-o move group of humans, at least agents are required. Thus, explicit multi -agett
collaboration is required.
Due the time pressure, order to successfully stop the fire, agents must all effectively collaborate in collecting and strategically using resources in high-intensity regions. Exploration & Task Assignment- There explicit tradeoff between allocating agents towards exploring to find lost people and fighting the current fires.
D.2 Long-term Planning
To resolve a fire it requires that the source is identified, the type of fire is identified, and the appropriate resources to stop it acquired and then used. Uncertainty: Inherent uncertainty as to where the lost people can found, and what point in the timeline they will.
Dependency Chain:
Irreversibility and forced consequences With a fire that spreads, present actions have irre- versible future consequences. A balance needs to be struck between fighting the fire's source (to stop it from continuing) versus a periphery (to prevent geographic spread).
D.3 Description of Scenes
We evaluate LLaMAR in five different scenarios in the search & rescue environment. Here each scene evaluated on 5 different random seeds.
This consists of Type A fire with initial sources, Type B fire with initial source, and lost person at random location in the environment.
Scene 1:
Scene 2: This consists of 'yype A fire with initial source, Type B fire with 2 initial sources,
and lost person at random location in the environment.
Scene 3: This consists of 2 Type A fires each with 1 initial source, Type B fire with 1 initial
source, and lost person at a random location in the environment.
Scene 4: This consists of 1 Type A fire with initial sources, and lost person at random location the environment.
Scene 5: This consists of Type A fire with I initial source, 1 Type B fire with initial source, and lost persons at random locations the environment.
D.4 Observation Space
The agent's observation O = Og UO O is union of the following observations.
Global Observations OG consists of all the objects that are globally visible by the agent:
Fires:
If visible, fire name, type, and average intensity.
Fire Regions: fire visible, fire region name, type, and average intensity.
Reservoir:
If visible, reservoir name, and reservoir resource type.
Deposit:
If visible, deposit name, inventory all the resources (water, sand) and persons in
deposit.
Person: visible, person name, carried or not status, dropped-off or not status.
Agent inventory: List of all the resources (water, sand) and person (being carried) in the agent's inventory.
Local Observations O, consists of all the objects that are visible in grid cells adjacent to the agent:
For all the directions <direction> (Up, Down, Left, Right, and Center), output of the following.
20

---

<direction>: At direction <direction>, either 'Empty' if there is no object, 'Flammable' along with corresponding intensity & fire name if object is part of a fire, or 'Obstacle' for any other object.
Names ON consists of list of the name of all visible and interactable objects.
D.5
Action Space
The action space A consists of navigation actions ANAV, interaction actions AINT, exploration action Agxp.
Navigation actions ANAV consists of the following actions:
Move (wwd<recctio))) Moves the agent to the neighhoring grid cell in the specified direction
where <direction> can be one of (Up, Down, Left, Right, and Center)
NavigateTo ( <targetID>>: Moves the agent next the location of the object <targetID> <targetID> visible.
Interaction actions AINT consists of the following actions:
Carry (<perssop::::>): Makes agent earry a person <person> <person> is visible and inter- actable. The person is successfully 'group' carried, if at least the required number of agents successfully does the 'Carry (<person>) action. If carry action is successful, all other resources in agent's ss inventory are dropped.
DropOff (<person>, <deposit>): Drops off person <person> at location <deposit>. This action is only successful if the person has been 'group' carried, all the agents carrying the person have the deposit be visible and interactable, and after all the agents do this action.
StoreSupply (<deposit>): Stores all the resources from the agent's current inventory in the deposit <deposit>.
UseSupply<ffire>, (<fire>, <supply-type> Uses all the supplies of type hupply-typol on the fire <fire> at the location the agent is at.
(sieplyydddpsstt>.. <supply-type>); Fills the remaining of the agent's inventory space with the available supply of type <supply-type in <deposit>.
GetSupply (<reservoir>): Collects unit of supply from reservoir <reservoir> and stores it in the agent's 's inventory.
Exploration action AEXP consists of the following actions:
Explore(). Takes the agent a direction of exploration as described by the heuristic exploration function described in algorithm 2.
Algorithm 2 SAR Exploration Heuristic
Input: Agent A, Environment eno, Previous Direction D, Current Position P, Max steps M Initialize: New direction ND <-
ND +- randomly choose an angle from [0, 2T) at 7/4 intervals
2: while N D = D or N = D + π mod 2n barrier exists most ]/2 M steps direction N.D from P do
3:
4: ND +- randomly choose an angle from [0, 2T) at T/4 intervals
5: end while
6:
Move agent A, M steps in direction ND in environment env
21

---

E
Pseudocode for LLaMAR
Algorithm 3 LLLMMAR
Input:
agents, Task instruction Z, Environment ene
Initialize: Memory M < 0: Open Subtasks Go +- 0;
Completed Subtasks Ic t 0; Actions <- 0;
Corrective Actions de f- 0
Actions Executed d +
1: o= (01, ... .NN) = enw.resel(
2: while
do
3:
Go t- Planner(I,, o, 9o: M)
4:
a, M M ActorrZZ ac 9o M)
5:
(o,...., ov), d= (di, dN) env.step(a)
6:
a. Corrector(I, o, a, 90,
7:
Gc + Verifier[I, o, a, d, Go. M)
8:
if To then
9:
break
end
11:
t 1. ++1
end while
F Full Results
Success Rate
Transport Rate
Algorithm
LM
Coverage
Balance
Steps
Act
GPT-VV
0.33 (0.19, 0.49)
0.67 (0.59, 0.76)
0.91 (0.86.095)
0.59 (0.52, 0.66)
24.92 (22.12.27.33)
ReAct
GPT-4V
0.34
0.72 (0.6..088)
0.92
0.67 (0.61.0733)
24.08 (21.27,
CoT
GPT-VV
(0066 0.25)
0.59 0.67)
0.87 (0.81, 0.92)
0.62 (0. (0.56.669)
28.40 (26.91,29.9)
LMaRLLM
GPT-4V
0.11 (0005, 0.23)
0,23 (0.13, 0.313
0.91 (0.80, 0.96)
0.45 (0.37, 0.52)
29.87 (26.20, 30,00)
CoELA
GPT-4V
0.25 (0.10,
0.46
0.76 0.85)
0.73 0.80)
28.93 (27.77.20.00)
LLaMAR
GPT-4
0.51 (0.36, 0.66)
0,85 (0.80, 0,91)
0.95 (0.91, 0.98)
0.83 (0.78, 0.86)
25.80 (23.72, 27.88)
LLaMAR
LLaVA
0.54 (0.11,
0.84 (0.71,
0.91
0.75 (0.64,
26.21 (21.56,
LLaMAR
meFIcS-2
0.57 (0.43, 0.67)
0.86 (0.74, 0.91)
0.94 (0.89, 0.98)
0.78 (0.65, 0.84)
25.27 (20.14, 28.57)
LLaMAR
CogVLM
0.61 (0.47, 0.68)
0.89 (0.73, 0.95)
0.95 (0.89, 0.99)
0.80 (0.73, 0.86)
23.21 (20.57, 26.82)
LLaMAR
GPT-VV
0.62 (0.46, 0.76)
0.87 (0.80, 0.93)
0.95 (0.91, 0.98)
0.82 (0.77, (0.77,088)
23.44 (20.88. 26.00)
(wlo exploration)
LLaMAR
GPT-4V
0.66 (0.50, 0.78)
0.91 (0.51,0.96)
0.97 (0.93,0.99)
0.82 (0.75.087)
21.87 (18.76.24.23)
(w/ exploration)
Table 5: Comparison of evaluation metries against baselines averaged across all tasks for the 2-agent MAP.
THOR scenarios.
Modules Used
Success Rate
Transport Rate
Coverage
Balance
Steps
0.33
0.67
0.91
0.59
24.92
Actor
(086,0055)
0.66)
(22.12.27.73)
Planner
0.45
0.78
0.92
0.69
24.87
Actor
(0.29 0.57)
(0.67, 0.84)
(084, 0.95)
(0661, 0.75)
(20.48 27.95)
Verifier
Planner
0.67
0.91
0.97
0.84
22.81
Actor
0.96
(19.95, 25.76)
Corrector
0.66
0.91
0.97
0.82
21.87
LLaMAR
0.96
(0.93.0.99)
0.87)
Table Ablating different modules LLaMAR with GPT-4V as the underlying VLM, 2-agents scenarios.
22

---

MAP-THOR
SAR
# of agents
Success Rate
Transport Rate
Success Rate
Transport Rate
Coverage
Balance
Steps
Coverage
Balance
Steps
0.37
0.67
0.87
1.00
28.44
0.28
0.75
0.86
1.00
28
0.51)
(0.58, 0.774
0.00)
(1,0,100)
(25.3) 50.00)
0.62
0.87
0.95
0.82
23.44
0.44
0.86
0.94
0.91
27.76
0.76)
(0,50, 0.93)
0..91,0.88)
(0.772087)
(20,8,
(0.24.0.65)
(0.79, 0.94)
(0.88,
095)
30)
0.70
0.91
0.98
0.66
21.30
0.68
0.92
0.96
0.80
21.88
00.55, 0.82)
00.85, 0,95)
(0.95, 0,99)
(0.61.
(18.00, 23,99)
(0.45, (0.45.0855) 0.85)
(0.86, 0.98)
(0.91.0))
(0.73,
(17.33 25.92)
0.68
0.90
0.99
0.62
22.83
0.72
0.94
0.98
0.78
22.00
(0.52, 0.79)
0.79
00.54, 6:94)
(0.95, 0.99)
(0.57, 0.68)
(19.6), 25.60)
(0.50, 0.85)
(0.880995)
(0.93, 1.00)
0.83)
(17.96 26,03)
0.62
0.90
0.99
0.54
22,91
0.74
0.96
1.00
0.73
24.52
(0.46, 0.75)
(0,85,0.94)
(0.97.1.00)
0.59)
(20.36; 25.57)
(0.52.0.86)
(0.94, 0,99)
(1.0.100
(0.67.0 0.79)
(20.24,28.9)
Table 7: LLaMAR with more agents
G
Failure Cases
One the main motivations of our paper was to achieve better success rates in planning for a variety of tasks compared to previous approaches. While our method outperforms other baselines, we acknowledge that the success rate is still under the expectation for real-world deployment. We believe that our approach LLaMAR and the MAP-THOR benchmark would for future research to improve the success rates in multi-agent embodied robotics tasks. Here we describe few major types of failures in our experiments:
a starting point
:iialznneaaiiztion: The agent can sometime fail to properly infer the level of abstraction to perform a task at, even if it should be clear from the action space. For example, for the washing tasks, the LM assumes that it must put the objects in the sink and add some, rather than just using the "CleanObject" action. We observe this error in the direction of performing actions more low-level than necessary, and not the other way around (more high-level than is feasible).
Mutual Interference (limited spatial reasoning): The agents sometimes block each other and thus fail to carry out actions such as placing an object on receptacle, opening an object, etc. In particular, see this behavior in the "put objects the sofa" and the "put objects in box" tasks, where the LM does not prevent the agents from blocking each other.
Improper Object Visibility (un-observability bias): The LM often fails prioritize exploration for tasks that require objects not yet seen, simply due to it no being able to see it. This bias causes it to improperly assume that further exploration is not necessary, Bent mismatch: The free-form natural language output from the Actor is incorrectly mapped to the executable action. Based on our experimental results, 96.7% of the time the free-form natural language was correctly mapped to the feasible actions. The 2 main reasons for failures in the few cases failed were:
fails find relevant objects.
Incorrect mapping of objects: When the agents have not explored the environment enough and the Actor module suggests the agents interact with objects that are not yet available in the agents' combined memory. Example: The free-form output "navigate to the table" was mapped to the action
Wrong object numeration: When there are multiple objects of the same type in the memory, the sentenceBERT maps "Cabinet_3" to "Cabinet_I" which sometimes leads to incorrect actions being executed for the completion of the plan.
Limited horizon: While each task solvable by a single-agent human-come conan-conttolled agent within our chosen episode length of L 30, this is under the assumption of failures during control and maximum visibility. For the autonomous agents the visibility distance is restricted 1.5 meters, while human could identify objects from afar (from the rendered images) the agents have to explore the environment under stricter partial observability constraint.
We hypothesize that higher episode length would lead to higher metrics, but the cutoff' was chosen for computational budget considerations. As can be observed in figure the coverage and transportation rate metrics in particular seem to plateau around our chosen episode length L, so L 30 is appropriate.
23

---

Pagents
1.0
agantss
2.gents
rete
04
0.4
02
0.2
622
0.
0.0
0.0
s0ops
steps
steps
(a.
(b)
(c)
Transport
0.
0.6
0.8
0.4
0.4
0.4
0.2
6.2
10
15
20
25
30
5
10
15
75
30
5
10
20
25
35
staps
steps
50ops
(d)
(e)
(f)
Figure Plots of the average transport rate (a).(d); coverage (b),(e): and success (c).(f) metrics maximum horizon steps for GPT-4V, LLaMAR algorithm on MAP-THOR. Shown for both the agent (a),(b)(cc) and 5 agent cases (f), along with an error region of one standard deviation.
G.1 Failure Cases
have observed the following categories of failures for the LM on the SAR environment.
Incorrect Causal Ordering: The LM sometimes fails to execute the steps in the proper causal ordering. For instance, we have observed the agent trying to drop off lost person before picking them up, or properly getting supplies to stop the fire but not navigating the fire location before using them.
Sub-Optimal Action Sequence: Due to the time-constraints the fire spread, it is not sufficient for the LM to do the sequence of tasks in order. They must also be done in non-redundant, timely fashion. Otherwise, we have observed that the LM takes the right action sequence, but does not do it efficiently enough to stop the fire spread. For instance, if the agent prioritizes exploration to find the person, then the fire might spread until beyond a critical point where cannot be extinguished within the horizon.
Catastrophic Failure Misunderstanding: Even after the LM has completed the task, it can misunderstand later failure cause for continuing the task. For instance, have observed that if the LM has successfully dropped off a person, but then fails to pick them up again (which is expected since they're already dropped), it will think the task incomplete.
H Baselines
While there a lot of impressive LLM-based multi-agent planners mentioned in Table they vary in the assumptions about access to information about the environment. We were not able to find the official codebase for the Safe Multi-Agent Planning with Conformal Prediction (20) and TwoStep 21]. We describe the prompts used for our model as well as every baseline. Note that we show the prompt for the 2-agent case, but is easily modified to generalize the n-agent case. The italics and bolding added for emphasis.
H.1 LLaMAR
We describe the prompts used for each of the modules used in LLaMAR:

---

Prompt for Planner Module in LLaMAR
You excellent planner who is tasked with helping embodied robots named Alice and Bob carry out task. Both robots have partially observable view of the environment. Hence they have explore around environment to task.
You will get description of the task robots are supposed to do. You will get an image of the environment from Alice's perspective and Bob's perspective the observation input. To help you with detecting objects
image, you will also get a list objects each agent is able to see environment. Here the objects are named as <<object_name's <ooc>>>__obeeiidieh So, along with the image inputs you will get the following information:
### INPUT FORMAT ###
(Task: description the task the robots are supposed do,
Alice's observation: list of objects the Alice is observing,
Bob's 's observation: list of objects the Bob is observing.
Robots' open subtasks: list of subtasks the robots supposed carry out finish the task. If no plan has been already created, this will None. Robots' completed subtasks: list of subtasks the robots have already completed. If no subtasks have been completed, this will be None.
Robots' combined memory: description of robots' combined memory)
Reason over the robots' task, image inputs, observations, open subtasks, completed subtasks and memory, and then output the following:
Reason:
The reason for why new subtasks need to be added.
* Subtasks: A list of open subtasks the robots supposed take complete the task. Remember, as you get new information about the environment, you can modify this list. You can keep the same plan you think it still valid. Do not include the subtasks that have already been completed. The "Plan" should be in list format where the subtask are listed sequentially. For example:
("locate the apple", "transport the apple the fridge", "transport the book the table"] ("locate cup". "go to cup". "clean cup"]
When possible do not perform additional steps when one sufficient (e.g. CleanObject is sufficient to clean an object, other actions need to be done) Your oulput should be in the form of python dictionary shown below.
Example output:
["reason": "Since the subtask list is empty, the robots need to transport the apple to the fridge and transport the book to the table.". "plan": ["transport the apple to the fridge", "transport the book the table"])
Ensure that the subtasks are not generic statements like "do the task". They should be specific to the task at hand.
Do not assign subtasks to any particular robot. Try not modify the subtasks that already exist in the open subtasks list. Rather add subtasks to the list.
NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN
SPECIFIED Let's work this out in step by step way to be have the right answer.
25

---

Prompt for Verifier Module in L.L.MAR
You excellent task verifier who is tasked with helping embodied robots named Alice and Bob carry out task. Both rohots have partially observable view of the environment. Hence they have explore around the environment to task.
You will get description of the task robots are supposed to do. You will get an image of the environment from Alice's perspective and Bob's perspective the observation input. To help you with detecting objects in image, you will also get a list objects each agent is able to see environment. Here the objects are named as So, along with the image inputs you will get the following information:
sareee__amc____aee id>>"
### INPUT FORMAT ###
(Task: description the task the robots are supposed do,
Alice's observation: list of objects the Alice is observing,
Alice's state: description of Alice's state,
Alice's previous action: the action Alice took in the previous step and if it was successful,
Bob's 's observation: list of objects
Bob observing.
Bob's state: description of Bob's 's state, Bob's 's previous action: the action Bob took the previous step, Robots' open subtasks: list of open subtasks the robots in the previous step. If no plan has been already created, this will be None.
Robots' completed subtasks: list of subtasks the robots have already completed. If no subtasks have been completed, this None.
Robots' combined memory: description of robots' combined memory]
Reason the robots' task, image inputs, observations, previous actions, open subtasks, completed subtasks and memory, and then output the following:
Reason: The reason for why you think a particular subtask should be moved from the open subtasks list to the completed subtasks list.
* Completed Subtasks: The list of suhtasks that have been completed by the rohots. Note that add subtasks this list only they have heen successfully completed and in the open subtask list. subtasks have been completed the current step. return empty The "Completed Subtasks" should be a list format where the completed subtasks are listed. For example: ["locate the upple". "transport the apple to the fridge". "trunsport the book to the table")
Your output should in the form of python dictionary as shown below.
Example output:
"reason": "Alice placed the apple in the fridge the previous step and successful and Bob picked up the the book from the table. Hence Alice has completed subtask transporting the apple to fridge, Bob has picked up the book, but Bob has still not completed the subtask of transporting the book to the table". "completed subtasks": ["picked up book from the table", "transport the apple to the fridge")
NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED When output the completed subtasks, make sure to not forget to include the previous ones in addition to the ones.
Let's work this in step by step way to be sure have the right answer.
26

---

Prompt for the Actor Module in LLaMAR
You excellent planner and robot controller who tasked with helping 2 embodied robots named Alice, and Bob carry task. All robots have partially observable view of the environment. Hence they have to explore around the environment to do the task.
They can perform the following actions:
["navigate object <object id>". "rotate in <rolation>> direction". "pick up object <object id>". "put object on <receptacle id>". "open object <object id>", "close object <object sibeettdd>>... "slice object <object_id>> "toggle object <object_idh on" "toggle object <object id> off", "clean object <object_id>> jid_",dd"" "look up by angle <angle>", "look down by angle <angle>", "move in <translation> direction". "stay idle". "Done"]
Here "Done" used when all the robots have completed the main task. Only use when you think all the subtasks are complete.
"stay idle" is used when want the robot to stay idle for one-time step. This could be used to wait for the other robot complete its subtask. Use only when you think it is necessary. Here <rotation>> can one of ['Right". "Lefi"].
Here <angle> the angle degrees and can only be one of (30, 60, 60,90, 120, 150, 180). Here <translation> can be one of "Ahead", "Back", "Left", "Right").
So. along with the image inputs you will get the following information:
#A# INPUT FORMAT ###
(Task: description of the task the robots are supposed to do.
Alice's observation: list of objects the Alice is observing.
Alice's state: description of Alice's state, Alice's previous action: description of what Alice did in the previous time step and whether it was successful,
Alice's previous failures: if Alice's 's few previous actions failed, description of what failed,
Bob's observation: list of objects Bob observing.
Bob's Bob's state: description of Bob's state, Bob's previous action: description of what Bob did in the previous time step and whether it was successful,
Bob's previous failures:
Bob's few previous actions failed, description of what failed,
Robots' open subtasks: list of subtasks supposed to carry out to finish the task. If no plan has been already created, this will be None.
Robots' completed subtasks: list of subtasks the robots have already completed. If no subtasks have been completed, this will be None.
Robots' subtask: description of subtasks robots were trying to complete the previous step, Robots' combined memory: description of robot's combined memory)
### OUTPUT FORMAT #
First all you are supposed to reason over the image inputs, the rohots' observations, previous actions, previous failures, previous memory. subtasks and the available actions the robots can perform, and think step by step and then output the following things:
Failure reason: any robot's previous action failed, use the previous history, your current knowledge of the room (i.c. what things are where), and your understanding of causality to think and rationalize about why the previous action failed. Output the reason for failure and how to fix this in the next timestep. If the previous action Common failure reasons to lookout way. agent can't 't see an object or its destination and must explore (such as move, rotate, or look in a different direction) to find it, agent doing extraneous actions (such as drying objects when cleaning). the previous action successful, output "None".
successful, output "None".
include: one agent blocking another so must move out of
* Memory: Whatever important information about the scene you think you should remember for the future as memory. Remember that this memory will be used in future steps to carry out the lask. So. you should not include information that is not relevant to the task. You can also include information that is already present in its memory if you think it might be useful in the future.
27

---

"memory": "Alice finished putting the knife in the cabinet when Alice was at co-ordinates (1, .5) and (CONTINUED) Prompt for the Actor Module in LLaMAR
Reason:
The reasoning for what each robot is supposed to do next
*
Subtask: The subtask each robot should currently try to solve, choose this from the list of open subtasks.
* Action:
The actions the robots are supposed to take just in the next step such that they make progress towards completing the task. Make sure that these suggested actions make these robots more efficient in completing the task as compared to only one agent solving the task. Your output should just be in the form of a python dictionary as shown below.
Examples output:
Example 1:
"failure reason": "Bob failed to put the mug in the cabinet earlier because Alice was blocking it when she was putting the knife. To fix this, Alice should close the cabinet and move away Charlie should move away to different open area than Alice to avoid congestion, and Bob should wait until the next timestep until Alice can move aside.". "
was facing north. Bob wanted put the mug in the cabinet when Bob was at co-ordinates (1, 0.25) and was facing north.".
" "reason": "Alice can close the cabinet door and then later back out in order help Bob with completing the task. Bob can be idle until the next timestep when Alice moves aside, by then Bob can navigate to the cabinet.", ",
"subtask"" "Alice is currently closing the cabinet door, Bob is currently waiting to get navigate the cabinet".
"Alice's action": "close the Cabinet_]" "Bob's action": "stay idle" }
Example
"failure reason": "Bob failed to clean the cup earlier because Bob had not navigated to it, Bob assumed the cup to be in the sink which was erroneous. To fix this, Bob should navigate to the cup and in the next step clean cup.",
"memory": "Alice finished navigating to the dish when Alice was at co-ordinates (-.5, .5) and was facing east, Bob was not able to clean the cup in the cabinet when Bob was at co-ordinates (1, .25) and was facing north.".
"reason": "Alice can now clean the dish since Alice has navigated to it. Bob should navigate to the cup order be close enough to clean the cup.", ",
"subtask": "Alice currently trying to clean the dish, Bob is currently trying to navigate to the cup".
"Alice's action": "cleea the dish object".
"Bob's action": "navigate to the cup")
Note that the output should just be dictionary similar the example outputs.
#ill Important Notes #ilff
* The robots can hold only one object at time.
For example: If Alice is holding apple, she cannot pick up another object until she puts the apple down.
" Even if the robot can see objects, it might not be able to interact with them if they are too far away. Hence you will need to make the robot navigate closer to the objects they want to interact with. For example: An action like "pick up <object_iie _ip___dd"""" is feasible only if robot can see the object and is close enough
So you will have to navigate closer to it before you can pick it up.
* some scenarios, the agents might not see the objects that they want to interact with. In such cases,
you will have to make the robot explore the environment to find the object. In such scenarios you can use actions to rotate in place or look up/ / down navigate to explore the environment.
* If you open an object, please ensure that you close before you navigate different place. * Opening object like drawers, cabinets, fridge can black the path of the robot. So open objects only when you think it necessary.
NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED
28

---

H.2 Act
We describe the prompt used for the Act bascline:
Prompt for the Act Baseline
You are an excellent planner and robot controller who tasked with helping 2 embodied robots named Alice and Bob carry out task. Both robots have partially observable view of the environment. Hence they have explore around in the environment to do the task.
They can perform following actions:
["navigate to object <object id>". "rrotate in <rotation>> direction". "pick up object <object id>", "pul object on <receptacle id>", "open object <object id>", ", "close object <object id>", "slice object <object id>", ", "toggle object <object id>> on". "toggle object <object id>> off", "clean object <object_id>" "look up by angle <angle>", "look down by angle <angle>", "move <translation> direction", "stay idle". "Done"]
Here "Done" is used when all the robots have completed the main task. Only use it when you think all the subtasks complete.
"stay idle" " used when you want the robot to stay idle for a one-time step. This could be used to wait for the other robot to complete its subtask. Use only when you think it is necessary. Here <rotation>> can be one of ["Right", "Left"].
Here <angle> the angle in degrees and can only be one of (30, 60,90, 120, 150, 180). Here <translation> can be one of '"Ahead" "Back", "Lefft, "Right").
You need to suggest the action that each robot should take at the current time step.
#illi Important Notes #i##i
* The robots can hold only one object at time. For example: If Alice is holding an apple, she cannot pick up another object until she puts the apple down.
" Even if the robot can objects, it might not be able to interact with them if they
far away.
Hence you will need make the robot navigate closer the objects they want interact with. example:
action like "pick up <object_ids id>" " is feasible only robot can see the object close enough to it. So you will have to navigate closer to it before you can pick it up.
In some scenarios, the agents might not see the objects that they want to interact with. In such cases, you will have to make the robot explore the environment find the object. In such scenarios you use actions to rotate place or look up down navigate to explore the environment.
If you open an object, please ensure that you close it before you navigate to a different place. Opening object like drawers, cabinets, fridge can block the path of the robot. So open objects only when you think it is necessary.
#iH INPUT FORMAT ###
You will get a description of the task robots are supposed to do. You will get image of the environment at the current time step from Alice's perspective and Bob's perspective the observation input. Here the objects named "<object_ jaaeecet__mm______ <object_id>> * You will get trace the steps taken by the robots and the actions they took at each time step and whether it was successful or not.
### OUTPUT FORMAT ###
In your output, do not have any extra content outside of the python dictionary below. Do NOT put any text, spaces, or enter keys (i.e. "/n") outside
Your output should ONLY be in the form of python dictionary, without any reasoning or extra text, as shown below: ("Alice": "action to be taken by Alice".
"Bob": "action to be taken by Bob)
For example: If you think Alice should pick up apple and Bob should navigate to the fridge, you will have give the output as:
("Alice": "pick up apple".
"Bob": "navigate to fridge")
NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED
29

---

H.3 ReAct
We describe the prompt used for the ReAct baseline:
Prompt for ReAct Baseline
You are an excellent planner who is tasked with helping embodied robots named Alice and Bob carry out task. Both robots have a partially observable view of the environment. Hence they have to explore around in the environment to do the task.
They can perform the following actions: ["navigate to object <object_ids id>", "rotate in <rotation>> di- rection". "pick up object <object id'". "pul object on <receptacle id>". "open object <object id>"". "close object <object id>"". "slice object <object id>", "toggle object <object id> on", "toggle object <object id> off". "clean object <object id'>". "look up by angle <angle>", ", "look down by angle <angle>>, "move in <translationn> direction", "stay idle", "Done") Here "Done" is used when all the robots have completed the main task. Only use it when you think all the subtasks are complete.
"stay idle" is used when you want the robot to stay idle for one-time step. This could be used to wait for the other robot complete its subtask. Use only when you think is necessary.
Here <rotation>> can one ["Right". "Left"). Here <angle>> is the angle in degrees and can only be one of [30, 60, 90, 120, 150, 180). Here <translation> can be one of ["Ahead", "Back", "Left", "Right"].
You need to suggest the action that each robot should take the current time step. #### Important Notes ####
* The rohots can hold only one object at a time. For example: If Alice is holding an apple. she cannot pick up another object until she puts the apple down.
* Even if robot can see objects, it might not be able to interact with them if they are too far away. Henee you will need to make the robot navigate closer to the objects they want to interact with. For example: An action like "pick up cobject_id>> is feasible only if robot can see the object and is close enough it So you will have to navigate closer to it before you pick it up. * In some scenarios, the agents might you will have to make the robot explore the environment to find the object. such scenarios you can use actions to rotate in place or look up / down or navigate to explore the environment.
the objects that they want interact with. In such cases,
* If you open an object, please ensure that you close it before you navigate to a different place. * Opening object like drawers, cabinets, fridge can block the path of the robot. So open objects only
when you think it necessary.
#illi INPUT FORMAT ####
You will get a description of the task robots are supposed to do. * You will get image of the environment at the current time step from Alice's perspective and Bob's s perspective the observation input. Here the objects are named as "<ooject__anne sanjjctt_ame>_____ namee>_<bjjct__te ". * You will get trace the steps taken by the robots and the actions they took at each time step and whether it was successful or not.
### OUTPUT FORMAT ###
You are supposed think and suggest the action each robot is supposed to take the current time step. Before suggesting an action you need to think, which requires that you reason over the inputs and logically reflect on the task, observation and course of actions needed to complete the task. Output Requirements: At each time step you must ONLY output PYTHON DICTIONARY of the following elements:
*First Element: Key = "Think" 1 Value:(Type: String): A logical reflection of the best action to be
taken given the inputs: task at hand, observations, and trace.
*Second Element: Key- "Action" Value: Python Dictionary):
The value should be in the form of a python dictionary shown below. {" ("Alice": "action be taken by Alice", "Bob": "action be taken by Bob")
For example: If you think Alice should pick up an apple and Bob should navigate to the fridge, you will have to give the output as: ("Alice": "pick up apple", "Bob": "navigate to fridge") Here is example output:
("Think": "To solve the task, need to find and put the apple. The apple is likely on the countertop or table. Then find the fridge." "Action": {" ""lice": "pick up apple". "Bob": "navigate to fridge")
NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED
30

---

H.4 Chain of Thought
We describe the prompt used for the Chain-of-Thought baseline:
Prompt for Chain of Thought Baseline
You are an excellent planner who is tasked with helping embodied robots named Alice and Bob carry out task. Both robots have a partially observable view of the environment. Hence they have to explore around in the environment to do the task.
They can perform following actions: ["navigate to object <.ohject_id' "votate <rotationn> di- rection". "pick up object <object id'", "put object on <receptacle id> "open object <object id>".. "close object <object id>", "slice object <object id>", " "toggle object <object id'> on", "toggle object <object id> off". "clean object <object id>" "look up by angle <angle>", "look down by angle <angle>", "move <translation"> direction", "stay idle", "Done") Here "Done" used when all the robots have completed the main task. Only use when you think all the subtasks are complete. "stay idle" is used when you want the robot to stay idle for one nim-time step. This could be used to wait for the other robot to complete its subtask. Use only when you think it necessary. Here <rolation>> be one of ["Right", "Left"].
Here <angle> the angle degrees and can only be one of [30, 60, 90, 120, 150, 180). Here <translation> can be one of ("Ahead", "Back", "Left", "Right").
You need to suggest the action that each robot should take the current time step.
#### Important Notes ####
The rohots can hold only one object at time. For example: If Alice is holding an apple, she cannot pick up another object until she puls the apple down.
* Even if the robot can objects, it might not be able to interact with them if they
far away. Hence you will need to make the robot navigate closer to the objects they want to interact with. For example: An action like "pick up <object_ids id>" fcasible only robot can see the object and is close enough to it. So you will have to navigate closer to it before you can pick it up.
" In some scenarios, the agents might not the objects that they want to interact with. In such cases, you will have to make the robot explore the environment find the object. In such scenarios you use actions to rotate place or look up / down or navigate to explore the environment.
If you open an object, please ensure that you close it before you navigate to a different place. Opening object like drawers, cabinets, fridge can block the path of the robot. So open objects only when you think it is necessary.
#ill INPUT FORMAT ####
You will get a description of the task robots are supposed to do. You will get image of the environment at the current time step from Alice's perspective and Bob's perspective the observation input. Here the objects are named as "<object__nane satbocttamme______ mawne_<cojjett,_________aaaa <object_id=---_... "iccttid"".. * You will get a trace the steps taken by the robots and actions they took each time step and whether it was successful or not.
### OUTPUT FORMAT #*#
You are supposed to FIRST reason through the situation logically and step by step, then suggest the action each robot is supposed to take at the current time step.
In your output, do not have any extra text or content outside of the python dictionary as below.
Your output should ONLY he in the form of python dictionary shown helow:
{" ("reason": "Reasoning for action plan....", " " "Alice": "action be taken by Alice", "Bob": "action be taken by Bob")
Put all of your reasoning inside of the "reason" key of the dictionary. Do NOT put any text, spaces, or enter keys (i.e. "/n") outside of it.
For example: you think Alice should pick up an apple and Bob should navigate to the fridge, you will have to give the output as: {" ""reason": "since the subtask list is empty, the robots need to transport the apple to the fridge", "Alice": "pick up apple", "Bob": "navigate to fridge")
Let's think step by step, make sure to put all your reasoning inside of the "reason" " key of the dictionary!
NOTE: DO NOT OUTPUT ANYTHING EXTRA OTHER THAN WHAT HAS BEEN SPECIFIED
31

---

H.S SmartLLM
We adapt the prompt from the official codebase of SmartLLM (master branch; commit PurdueSMARF1LMMbbomarrrrs
given here: https///ttuu..comSSAAAA 1lm.py with slight modification. Instead of letting the agents access all the objects in the environment through the simulator metadata, we just give the list of objects visible from the agents' point-of-view.
H.6 COELA
We adapt the prompt from the official codebase of CoELA (master branch: commit (c35ee68t77/bab04332222e266663
given here: https://gitubb.co//MMMaa mat/LLM We modify some aspects of the prompt as described: Instead of relying on the simulator/pre-ddine defined conditional logic for generating the list of available action options, we give list of all possible actions based on the observation. This includes the option to send the communication message, all navigation actions, and all combinations of valid actions with the interactable objects in the current observation.
Foundation-Modell Model/Co-LL/MM Agents//rre//msstr//iieneo
I Open Source VLMs
We list the source of the weights we used for the open-source VLMMs:
We use the 8B base model fine-tuned on mixture of supervised and instruction datasets (lext-only and multimodal datasets) from HuggingFace. The weights were downloaded from httpss//huggingface wttp://wee.ouuugggggaaeeeeeeeeee
Idefics 2 668 69
with the commit We chose Idefics because is able take multiple images as input similar to GPT-4V and reason on them.
LLaVA [70]: We use the 7B model trained by fine-tuning LLaMA/Vicuna on GPT-generated multimodal instruction following data. The weights were down- 1.5-7b-hf with the commit 785b7.
loaded from https:/hugginngface instruction-following 35a22433cb44033eedbbbcacc
CogVLM 772)
We use the 18B model. The weights were downloaded
from
with the commit
SentenceBERT fine-tuning
We finetuned pre-trained BERT model to function semantic mapper between free-form natural language output and the robot's s admissible actions the environment. The pre-trained weights were obtained from https://hugiingface www.aawwww..seeenc..eee...e http:://hnginngcccc.osmsttneeeee www../enneee---tasffoeeeeeeee Mitoomers/allMMiiLMM-LLL trained on a dataset consisting of 2800 free-form input, valid action output pairs. It ran on one (1) Apple MI core for wall clock time of minutes. Table shows the hyper-paree seper-parameters used for the pre-training of the BERT model.
The model was
Epochs
10
Max gradient norm Learning rate
2 x 10
Batch size
64
Encoding dimension
384
Optimizer
Adam
Scheduler
Warm-up linear
Warm-up up steps
Weight decay
0.01
Loss scale
20
Loss type
Multiple negatives ranking loss
Similarity function
Cosine similarity
Table 8: Hyper-parameteo for the model fine- tine-tuning including the Toss.
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract are reflected in the experimental results shown in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
we mention the limitations of our paper in separate section in the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification: We do not include any theoretical results in the paper and hence we do not report the theoretical assumptions. Although we note the assumptions the Al2Thor environment in which we run all our experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide much detail we can for the reproducibility of the experiments. Although there will be some variations to the underlying LLMs/VLMs we note the specific LLM/VLM model we used for our experiments in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
34

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provide a link to the GitHub repository the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://mips.co
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips. htcp//blcc//uudeeeeeeeeeeeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify as many details of our experiments as we can in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, we report means and the 95% confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
35

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we denote all these details the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///waaaorp.....o/oooo.cccaa
Answer: [Yes]
Justification:
We have read the code ethics on the website.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: In our opinion, our work does have any significant societal impact other than being able to progress towards enabling embodied multi-agent robots for household planning.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
36

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: No high risks in the paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the authors of the works we base our work on especially the A12Thor simulator. We have also provided links to the weights used for the open-source VLMs in the appendix with the commit # number on HuggingFace.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
37

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide documentation for the initialization of the multi-agent setting in the A12Thor simulator in the supplementary material.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
38

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "4wnHXJM2Iz", "reviewer": "Reviewer_nYvP", "summary": "The paper addresses the problem of long-horizon household tasks in a multi-agent setup using LLMs and for this, the authors propose LLaMAR. This framework integrates planning, acting, correcting, and verifying parts. LLaMAR plans a high-level action sequence, assigns them to multiple agents, and checks if actions fail to be executed, followed by a heuristic exploration strategy. It also converts the planned actions to the admissible ones in a retrieval manner based on text similarity. The approach is validated in MAP-THOR, an extended version of AI2-THOR that includes household tasks in the multi-agent setting.", "questions": "See weaknesses above.", "limitations": "The authors do not address the potential negative societal impact.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper tackles an important problem of LLM-based multi-agent scenarios.\n- The proposed pipeline is straightforward and easy to replace due to its modular nature, yet raising concern about its justification (see weaknesses).\n- The paper tries to remove unrealistic aspects of prior work, such as no perfect execution of low-level primitives, which potentially improves the deployability of the proposed approach to real-world scenarios.", "weaknesses": "- The proposed framework takes a plan-act-correct-verify pipeline but this is not well-justified. Why do we need the four components in that order? As each plan, act, correct, and verify part has been actively investigated in the literature, the core contribution may come from how we combine them.\n- L49 says that the proposed approach does not rely on oracle feedback, but L176 says it uses the simulator to check whether an action is executed successfully, which is contradictive. Using such GT action failure checking seems quite unrealistic.\n- One argued contribution is MAP-THOR, but the current description (L207-L219) lacks the details of it. How many (training/evaluation) episodes are used? How different is MAP-THOR compared to previous benchmarks for various aspects? How to collect the dataset? Summarizing them in some tables with qualitative analysis might be helpful.\n- What are the values in the parentheses in the tables? Describing what they are in the table captions might improve readability. Are they min/max values for the corresponding metrics?\n  - If yes, then the variation seems quite high. For example, in Table 3, the min of \"LLaMAR (w/ exploration)\" is lower than the min of \"LLaMAR (w/o exploration),\" which raises a concern about the outperformance argued.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comprehensive review and insightful feedback on our paper. We appreciate your thoughtful comments and suggestions, and we have provided our responses and clarifications below.\n\n### Justification for the Proposed Framework\n\nThe specific order of the modules is due to natural causal relationships in which environment feedback is received, which restricts our freedom in permuting the order significantly.\n\nWe use the planner as the first module because it allows LLaMAR to come up with an initial list of open subtasks that could be completed based on the current observation and past memory to satisfy the task. This list serves as a rough high-level plan. The actor then uses this information to suggest the necessary actions. \n\nThe Corrector is used after the Actor module to identify reasons for failures in the execution of the actions suggested by the Actor. Note that the failure module is inert and only suggests corrective actions. Only the Actor module decides the final actions to be executed. This role distinction allows for clear reasoning on failures when they occur and lets the actor module focus on choosing actions.\n\nThe verifier is used after the action is executed to update the list of closed subtasks so that LLaMAR can be current with the progress toward the completion of the environment task. This allows the planner to update the list of open subtasks in the next step. In essence, the planner and the verifier ensure that the progress of the agents is tracked and the actor and the corrector ensure that the actions are executed successfully to advance towards completion of the task.\n\nOur revised manuscript will include this justification with illustrative examples so as to explain why we converged on this solution.\n\n### Oracle feedback clarification\nThe terminology is confusing and should be cleared up. Thanks a lot for pointing it out. In L49, when we say our approach does not rely on oracle feedback, we mean to say that the *environment* does not provide any feedback. Instead, it uses controller feedback—a boolean variable indicating action success or failure, such as obstacles in planned paths or failed object grabs. \n\nIt is indeed true that, in general, loss of that low-level controller information could make the problem harder, and it is a fruitful direction to explore in future work.\n\n### MAP-THOR details\n\nMAP-THOR includes 45 evaluation scenarios each initialized in 5 different rooms in AI2THOR. All of these 45 tasks are described in Appendix C. We refer to L726-L787 for a detailed list of all the tasks in MAP-THOR. A more detailed description of how to use the dataset to run experiments will be provided in the code’s repository when it is open-sourced. Additionally, a description of how to append new tasks to MAP-THOR will also be provided there. MAP-THOR as a dataset consists of AI2Thor scene pre-initializations, task information (task description, timeout, etc.) pairs, and an oracle performance evaluator (to get the metrics). Thus, there isn’t an explicit collection procedure since we create and simulate the tasks based on each floor plan.\n\nThe MAP-THOR dataset was not used to fine-tune/ train any of the modules in the cognitive architecture. Indeed we should provide a fixed training and evaluation division of the dataset in order to aid future work. \n\nMAP-THOR is designed to include tasks that could be done by multiple agents. Previous benchmarks like AlfWorld, and SmartLLM include many tasks that could be done with just a single agent. Example: “Slice the tomato”, “throw the spatula in the trash”, etc. Link:https://github.com/SMARTlab-Purdue/SMART-LLM/blob/master/data/final_test/FloorPlan6.json. Although many of the tasks in these benchmarks require single agents, there are a few tasks that could be solved with multiple agents. We include those tasks in our benchmark. Other benchmarks like the ones used in CoELA in the ThreeDWorld and CWAH environments include tasks that require multiple agents. But there are only limited tasks, 5-10, in those experiments. Another major difference in MAP-THOR is that we instantiate each task in 5 different rooms with random initialization to test for the robustness of the underlying LM based cognitive architecture to variations in the room layouts. This allows us to perform a systematic study of its performance when compared to previous approaches. Along with this, we have created a script to add custom scenarios in AI2THOR. We will add more details about its usage in our documentation of the code and in our appendix along with pointing out specific details about how it is qualitatively different than previous approaches.\n\n### Clarification on the metrics in tables\n\nThe metrics shown in all tables are means. The values in the parentheses are the 95% confidence intervals. Since success rates are binomial, we use the Clopper-Pearson Interval as the confidence interval. For the metrics: “Success Rates (SR)”, “Transport Rates (TR)”, “Coverage (C)” and “Balance(B)”, higher is better and for “Steps (L)” lower is better\n\nIn Table 3, the means for SR, TR, C, and B are higher for LLaMAR (w/ exploration) compared to LLaMAR (wo/ exploration). Similarly, the lower bound of the confidence intervals for SR, TR, and C are higher for LLaMAR (w/ exploration). Similarly, the upper bounds for SR, TR, C, and B for LLaMAR (w/ exploration) are higher than or equal to the upper bounds for LLaMAR (wo/ exploration). For Balance (B), the means are equal for w/ and w/o exploration and the lower bound for w/ exploration (0.75) is slightly smaller than the lower bound for w/o exploration (0.77)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cbU7UrLpyH", "reviewer": "Reviewer_Eixs", "summary": "This paper explores the use of Language Models (LMs) to generate task plans for autonomous robots, emphasizing adaptability across diverse tasks without the need for extensive customization. The authors introduce LLaMAR, an LM-based agent framework that employs a “plan-act-correct-verify” strategy to manage long-horizon tasks in partially observable environments. The effectiveness of LLaMAR is assessed using MAP-THOR, a new benchmark featuring multi-agent scenarios and various household tasks at different complexity levels within the AI2-THOR environment. LLaMAR demonstrates a higher success rate than competing models in these tests.", "questions": "1. The use of fine-tuned sentence-BERT for action parsing is innovative, but the paper should discuss potential performance issues and the implications of incorrect mappings more quantitatively, for example, by comparing with more general function calling in LLMs. \n2. As indicated in Algorithm 2, the corrector works after the robot performs the actions, which may cause safety problems in real-world experiments. For example, the place action is dangerous when the fridge is not open. Is it possible to pre-check the action before the actual execution? Moreover, solely relying on VLM as a verifier may be unreliable; more traditional perception tools (e.g., segmentation and detection) may help improve the system's reliability.", "limitations": "YES", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and structured, making technical details accessible and easy to follow.\n2. LLaMAR introduces a unique \"plan-act-correct-verify\" framework, integrating Language Models to offer a fresh approach for long-horizon planning in dynamic, partially observable multi-agent environments.\n3. Instead of straightforwardly applying the general framework of an LM-based agent, LLaMAR proposes multiple essential components, particularly for solving planning problems in embodied AI, i.e., verifier, corrector, and action parsing. For example, to mitigate the hallucination of LLMs/VLMs, LLaMAR presents a verification module for feedback and self-reflection, which is important when interacting with practical environments. \n4. The introduction of MAP-THOR as a test suite for evaluating the effectiveness of the planner across various household tasks underscores the robustness and practical applicability of LLaMAR, providing a sense of confidence in its potential use.", "weaknesses": "1. The method’s reliance on multiple LM-based modules for planning and execution might introduce significant computational overhead and complexity.\n2. The detailed, step-by-step reasoning process, while beneficial for accuracy and responsiveness, could incur high costs, especially when utilizing commercial LM APIs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comprehensive review and insightful feedback on our paper. We appreciate your thoughtful comments and suggestions, and we have provided our responses and clarifications below.\n\n### SentenceBERT\n\nYes, we agree with the concern about potential performance issues and incorrect mappings with SentenceBERT. Based on our experimental results, 96.7% of time the free-form natural language was correctly mapped to the feasible actions. The 2 main reasons for failures in the few cases it failed were:\n\n1. Incorrect mapping of objects: when the agents had not yet explored the environment enough and the Actor module suggested the agents interact with objects that were not yet available in the agents’ combined memory. Example: The free-form output “navigate to the table” was mapped to the action “NavigateTo(ArmChair)”. \n2. Wrong object numeration: When there are multiple objects of the same type in the memory, the sentenceBERT maps “Cabinet_3” to “Cabinet_1” which sometimes leads to incorrect actions being executed for the completion of the plan.\n\nWe will add more specific examples of the failure cases of the sentenceBERT module in the appendix of our revised manuscript. \n\nAs for using more general function calling with LLMs, we saw in our initial experiments that the LLMs performed well with function calling only when we gave a few examples of the exact format (eg., JSON, pythonic plans, dictionary, etc.) we wanted the LLM to answer in. This meant that we would have to use a significant amount of prompt tokens to include the examples in the input prompt. A way to get around this is to fine-tune the underlying LLMs on the type of outputs we want which we would like to do in our future work along with having specialized smaller LLMs for each module.\n\n### Checking the safety of actions before execution\n\nThis is a great point! One possible way to tackle this is to let the LMs build a world model of the environment as done in a very recent work [1,2]. This would allow the agents to reason about what would happen if certain actions were executed in the current context of the environment. This could act as a pre-execution action checker and allow for filtering unsafe or infeasible actions. However, a drawback of using another LM for the world model would be the added computational complexity.\n\n[1]: Xiang, J., Liu, G., Gu, Y., Gao, Q., Ning, Y., Zha, Y., ... & Hu, Z. (2024). Pandora: Towards General World Model with Natural Language Actions and Video States. *arXiv preprint arXiv:2406.09455*.\n[2]: Zhang, H., Wang, Z., Lyu, Q., Zhang, Z., Chen, S., Shu, T., ... & Gan, C. (2024). COMBO: Compositional World Models for Embodied Multi-Agent Cooperation. arXiv preprint arXiv:2404.10775.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "edg1OGYVHm", "reviewer": "Reviewer_XzUd", "summary": "This paper focuses on the long-horizon multi-agent planning tasks. They propose a new agent structure  composed of four language models (Planner, Actor, Corrector, Verifier), named Language-Model-based Long-Horizon Planner for Multi-Agent Robotics (LLaMAR). LLaMAR operates without prior knowledge or privileged information from the environment, making it suitable for practical robotic applications. Additionally, the authors develop a multi-agent planning benchmark upon the AI2-THOR simulator, encompassing 45 household tasks. Empirical results demonstrate that their method outperforms existing agent methods.", "questions": "1. Since the agent structure can be easily adapted to a single-agent setting, a comparison between single-agent and multi-agent systems would be valuable to demonstrate the effectiveness of using a multi-agent system for these tasks.\n\n2. I would expect more in-depth analyses of the failure cases, such as examining the behavior of different methods across various categorized tasks and identifying the causes of the significant gap between the baselines and the proposed method. Additionally, it would be helpful to understand the conditions under which the proposed method tends to fail.\n\n3. Some visualized examples would be helpful for understanding how the agents cooperate to complete long-horizon tasks.", "limitations": "The authors have addressed the limitations of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. They propose a novel agent structure for building a centralized multi-agent system using (vision-)language models. The integration of the corrector and verifier modules enables the system to perform effectively in practical scenarios. An ablation study on different modules highlights the significant contributions and effectiveness of these two components.\n\n2. The paper demonstrates clear and well-organized writing and includes extensive experiments that support their statements.", "weaknesses": "While the agent structure design is commendable, its contribution to the multi-agent setting appears limited. Firstly, the proposed benchmark focuses on evaluating the effectiveness and robustness of multi-agent planning, yet these tasks are not necessarily crucial for multi-agent cooperation. As shown in Table 5, increasing the number of agents does not lead to improvement and, in fact, results in a decline in almost all metrics when the number of agents increases from 3 to 5. Secondly, task allocation, which is critical for a centralized multi-agent system, is not thoroughly discussed in this paper.\n\nMoreover, considering the overall performance is relatively poor, with at least one-third of the trials failing, more in-depth analyses of the failure cases would be expected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comprehensive review and insightful feedback on our paper. We appreciate your thoughtful comments and suggestions, and we have provided our responses and clarifications below.\n### Single-agent comparison in MAP-THOR:\nWe perform experiments with just a single agent in the environment for the task in MAP-THOR. We report all the metrics for the single agent setting below. We report the means and the 95% confidence interval in parentheses.\n| # agents | Success Rate | Transport Rate | Coverage | Balance | Steps |\n| --- | --- | --- | --- | --- | --- |\n| 1 | 0.37 (0.21,0.51) | 0.67 (0.58, 0.74) | 0.87 (0.81, 0.90) | 1.00 (1.00,1.00) | 28.44 (25.23,30.00) |\n\nWith just a single agent in the environment, it is not able to complete the task in a majority of the episodes. This is because the number of steps required to complete the episode is higher than the horizon length of $L=30$. Hence the success rate and the transport rate are lower than n>1 agents. Also, the singular agent in the environment is not able to explore the environment enough within the planning horizon and hence the coverage is lower than $n>1$ agents.\n\n### Multi-agentness in experiments\nWe perform experiments in a multi-agent search-and-rescue environment where multiple drones are required to collaborate together. More information can be obtained in the global rebuttal. The key relevant points in this new environment are:\n- **Scalability**: With the addition of multiple parallel emergency disaster scenarios at once, the environment scales comfortably to more agents. Additionally, the complexity slowly increases as the model has to coordinate between more actors and competing emergencies.\n- **Explicit cooperation**:\n    - In order to move a group of humans, at least 2 agents are required. Thus, explicit multi-agent collaboration is required.\n    - Due to the time pressure, in order to successfully stop the fire, drones must all effectively collaborate in collecting and strategically using resources in high-intensity regions.\n- **Exploration task assignment**: There is an explicit tradeoff between allocating drones towards exploring to find lost personnel and fighting the current fires.\n\n### Table 5 Analysis\nIn Table 5, there is a decrease in the metrics when the number of agents increases from 3 to 5. The reason for this is that as the number of agents increases, the rooms in MAP-THOR get crowded. Hence the agents block each other while navigating in the environment leading to a decrease in success rates due to successive failures in navigating to desired locations. We will update the analysis for Table 5 in our revised manuscript to explain the decrease in the metrics as we increase the number of agents. We apologize for the confusion!\n\nWe have performed experiments in a search-and-rescue environment that requires explicit cooperation. Please refer to the global rebuttal for the results in the search-and-rescue environment and more analysis.\n\n### Task Allocation\nIn our proposed architecture, the planner module plays the role of creating subtasks necessary for completing the tasks. The actor module then takes these subtasks and allocates them to different agents based on each agent's state. Centralized planning ensures that the actions suggested are not in conflict.\n\n### Analysis of failures cases\n Multi-agent planning in embodied robotics environments has been a challenging problem. Our initial goal was to achieve better success rates in planning for a variety of tasks compared to previous approaches. While our method outperforms other baselines, we agree that the success rate is still not as good as expected for real-world deployment. We believe that our approach LLaMAR and the MAP-THOR benchmark would serve as a starting point for future research to improve the success rates in multi-agent embodied robotics tasks. \n\nA few major types of failures in our experiments were:\n- **Mis-generalization**: For example, in the wash tasks the LM assumes that it must put the objects in the sink, and add soap; rather than using the “CleanObject” action.\n- **Mutual interference (limited spatial reasoning)**: For example, in the put objects on the sofa task, the LM allows the agents to block each other, and therefore fail to put the objects on the sofa. Similarly, for the put objects in a box task.\n- **Improper receptacle visibility**: For example, in the put objects on table task, the table is not initially visible yet the LM does not prioritize exploring to find it. Thus, it fails to navigate to it to put the necessary objects.\n- **SentenceBERT mismatch**: The free-form natural language output from the Actor is incorrectly mapped to the executable action. Based on our experimental results, 96.7% of the time the free-form natural language was correctly mapped to the feasible actions. The 2 main reasons for failures in the few cases it failed were:\n    1. Incorrect mapping of objects: when the agents had not yet explored the environment enough and the Actor module suggested the agents interact with objects that were not yet available in the agents’ combined memory. Example: The free-form output “navigate to the table” was mapped to the action “NavigateTo(ArmChair)”. \n    2. Wrong object numeration: When there are multiple objects of the same type in the memory, the sentenceBERT maps “Cabinet_3” to “Cabinet_1” which sometimes leads to incorrect actions being executed for the completion of the plan.\n    \n    We will add a comprehensive list of different reasons for failures in each of the task categories in our revised appendix to help the readers better understand the gaps in performance between the baselines and our proposed approach. Along with this, we agree that having more illustrative examples of cooperation among agents in completing tasks would be helpful to showcase the effectiveness of our method. We will add these illustrations in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mB2dxCnR9r", "reviewer": "Reviewer_snqg", "summary": "This paper presents a multi-agent planning and execution engine called LLaMaR that is based on several multi-modal foundation models such as GPT4V, i.e., large models  (LMs) that can reason about text and visual inputs. LLaMaR has 4 LMs: planner, actor, corrector, and verifier. The planner decomposes the task into subgoals. The actor chooses which subgoal to perform and which low-level actions to perform to achieve it. The corrector modifies the actions chosen by the actor in case the chosen action is known to have failed. The verifier verifies which subgoals have been achieved. No additional knowledge is given to LLaMaR. LLaMaR controls all agents in a centralized manner. \nThe paper also introduce briefly a benchmark for evaluating LLaMaR which is based on the AI2Thor simulator, and shows that LLaMaR outperforms similar LM-based planners. The main novelty in LLaMaR is the corrector module, and the fact that it is designed for partially observable situations. The latter is addressed by allowing the agents to invoke an exploration strategy if the object they seek is unavailable.", "questions": "1.\tThe separation of the Actor and the Corrector seems artificial. What does the Actor know that the Corrector does not? Since the Corrector needs to be smart enough to know the subgoal, know past failures, and suggest a course of action, it seems it has all the input that the Actor has and more. \n2.\tWhat elements of LLaMaR are really multi-agent? The agents are sharing information in a seamless way, and there is nothing LLaMaR does to avoid the combinatorial challenge of planning for multiple agents (huge number of actions, interactions between the agents). \n3.\tYou say that you didn’t use the Smart-LLM benchmark since some of the tasks did not support multiple agents. Wouldn’t it be good to at least use the tasks there that do support multiple agents?\n4.\tWith respect to Table 5 and the impact of adding more agents, you wrote that the success rate “remains high” and the number of steps “generally decreases with more agents”. Yet when moving from 3 to 4 agents and from 4 to 5 agents, the success rate drops and the number of steps increases. Is the conclusions drawn from Table 5 indeed incorrect?", "limitations": "Not relevant", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe problem is extremely challenging, and the proposed approach works well on the benchmarks presented in this work. \n2.\tThe topic is highly relevant.\n3.\tThe notion of a corrector as a means of learning from mistakes is interesting. \n4.\tHaving exploration actions to discover unobserved objects is also a nice contribution. \n5.\tThe authors present a new benchmark for evaluating algorithms on this problem. This will help future research.", "weaknesses": "1.\tWhile the title promises some multi-agent aspect, I didn’t see anything in LLaMaR that is specific for multi-agent planning. What about interactions between actions? communication issues? Or decomposition of the action-space for efficiency purposes? \n2.\tThe analysis of Table 5 is, as I see it, incorrect. See questions below. \n3.\tLLaMaR is described in a hand-wavy and shallow manner. \n4.\tThe exploration strategy seems ad-hoc and domain specific.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comprehensive review and insightful feedback on our paper. We appreciate your thoughtful comments and suggestions, and we have provided our responses and clarifications below.\n\n### LLaMAR explanation\n\nThe specific order of the modules is due to natural causal relationships in which environment feedback is received, which restricts our freedom in permuting the order significantly.\n\nWe use the planner as the first module because it allows LLaMAR to come up with an initial list of open subtasks that could be completed based on the current observation to satisfy the task. This list serves as a rough high-level plan. The actor then uses this information to suggest the necessary actions. \n\nThe Corrector is used after the Actor module so as to check for any failures in the execution of the actions suggested by the Actor. Note that the failure module is inert and only suggests corrective actions. Only the Actor module has the final say in the actions that are supposed to be executed. This distinction allows for clear reasoning on failures when they occur and lets the actor module focus on choosing actions.\n\nThe verifier is used after the action is executed to update the list of closed subtasks so that LLaMAR can be current with the progress toward the completion of the task. This allows the planner to update the list of open subtasks in the next step. In essence, the planner and the verifier ensure that the progress of the agents is tracked and the actor and the corrector ensure that the actions are executed successfully to advance towards completion of the task.\n\nOur revised manuscript will include this justification with illustrative examples so as to explain why we converged on this solution.\n\n### Multi-agent features in LLaMAR\n\n- **Coordination Through Communication:** Agents share their state information with the centralized LLaMAR modules to predict actions, enabling them to coordinate and avoid conflicts. This information sharing allows for the agents to cooperate and achieve the collective goal. A limitation of having a centralized module is that as the number of agents increases the observation input size (both textual and visual features) increases leading to a large input query (prompt).\n- **Hierarchical Task Decomposition:** LLaMAR decomposes complex tasks into subgoals that are distributed among agents. This hierarchical approach simplifies the planning process and reduces computational complexity by focusing on manageable sub-tasks.\n- **Dynamic Role Assignment:** Agents are dynamically assigned roles based on the current task requirements and their capabilities. This flexibility allows LLaMAR to adapt to changing environments and task demands.\n- **Decomposition of Action-Space:** To handle the complexity of multi-agent planning, LLaMAR decomposes the action space by creating specific subgoals/subtasks available for any agent to assign itself (done by the actor module) based on the observation & current context. This decomposition reduces the overall search space and improves planning efficiency.\n\n### The corrector and actor module separation\n\nCorrect, the actor and corrector have similar information as inputs, but they are queried with a different task prompt. That is, we task the Actor module to suggest the actions and the Corrector module to explicitly reason why certain actions failed. This explicit designation of tasks in LM agents is more effective [1] as compared to letting a single LM module predict both the actions and reason for failure. Finally, only the ‘actor’ LM has the final say in the action to be performed, and the ‘corrector’ module is inert, only serving as a reasoning tool for the ‘actor’ LM.\n\n[1] Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., ... & Zhang, X. (2024). Large language model based multi-agents: A survey of progress and challenges. *arXiv preprint arXiv:2402.01680*.\n\n### Table 5 Analysis\n\nIn Table 5, there is a decrease in the metrics when the number of agents increases from 3 to 5. The reason for this is that as the number of agents increases, the rooms in MAP-THOR get crowded. Hence the agents block each other while navigating in the environment leading to a decrease in success rates due to successive failures in navigating to desired locations. We will update the analysis for Table 5 in our revised manuscript to explain the decrease in the metrics as we increase the number of agents. We apologize for the confusion! We perform experiments with LLaMAR in a multi-agent drone search-and-rescue environment (SAR) to showcase the effectiveness of LLaMAR in a different environment. Please refer to the global rebuttal for results and analysis.\n\n### Exploration Strategy\n\nWe agree that the exploration strategy used is domain-specific as we wanted the agents to search in more semantically related regions for objects of interest. We hope to use more sophisticated & generic approaches for exploration in our future work such that can be generalized well to other scenarios.\n\n### SmartLLM Benchmark\n\nMAP-THOR is designed to include tasks that could be done by multiple agents. SmartLLM includes many tasks that could be completed with just a single agent. Example: “Slice the tomato”, “throw the spatula in the trash”, etc. [Link](https://github.com/SMARTlab-Purdue/SMART-LLM/blob/master/data/final_test/FloorPlan6.json). Although many of the tasks in these benchmarks require single agents, there are a few tasks that could be solved with multiple agents. We include most of such tasks in our benchmark. We will ensure that we include any other tasks from the SmartLLM benchmark that are not present in MAP-THOR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0mlk6HLTNw", "reviewer": "Reviewer_QApg", "summary": "The paper introduces the Long-Horizon Planner for Multi-Agent Robotics (LLaMAR), a cognitive architecture leveraging Language Models (LMs) for planning tasks in partially observable environments. Unlike traditional methods, LLaMAR uses a plan-act-correct-verify framework, enabling real-time self-correction and task verification without relying on oracles or simulators. The system's effectiveness is demonstrated through the MAP-THOR benchmark, which includes various household tasks in the AI2-THOR environment. LLaMAR achieves a 30% higher success rate than other state-of-the-art LM-based multi-agent planners.", "questions": "1. Do you have any strategies in mind for optimizing or reducing this computational overhead?", "limitations": "1. How to scale to the real world remains a real concern.\n2. Computational complexity is a big concern in more complex environments.\n3. Too many restrictions might limit the capabilities of LMs.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The division of tasks into specialized modules (Planner, Actor, Corrector, Verifier) allows for efficient task management and error correction.\n2. The system can self-correct based on real-time feedback, reducing the reliance on perfect environmental knowledge or external oracles.\n3. Experimental results show that LLaMAR achieves a significantly higher success rate compared to other LM-based planners, indicating its effectiveness.", "weaknesses": "1. As with most LLM-based TAMP work, how to scale to the real world remains a real concern.\n2. The use of multiple LM-based modules for each decision step increases computational and time costs compared to simpler baselines.\n3. Some other LLM-based planning framework also has a correction strategy [1]. This framework utilizes A* with symbol operators to verify the plan. Also, this strategy can ensure that the generated plan is the shortest, which is a limitation in this paper.\n4. In the appendix, the number of actions used by these tasks is at most about 10. In complex real-world environments, LMs may need to select the required actions from hundreds of different actions according to different tasks. This complex framework may lead to problems such as difficulty in finding the optimal solution and high computational complexity.\n5. Too many restrictions might limit the capabilities of LMs.\n\n[1]. Li, Z., Yu, K., Cheng, S., & Xu, D. (2024). LEAGUE++: Empowering continual robot learning through guided skill acquisition with large language models. In ICLR 2024 Workshop on Large Language Model (LLM) Agents. Retrieved from https://openreview.net/forum?id=xXo4JL8FvV", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for your comprehensive review and insightful feedback on our paper. We appreciate your thoughtful comments and suggestions, and we have provided our responses and clarifications below.\n\n### Computational complexity\n\nYes, we agree that having multiple LM-based modules would increase the computational complexity of the planning. Our initial goal was to create a benchmark and a method for getting the multi-agent robot teams to complete the tasks with a higher success rate as compared to previous approaches and hence it necessitated the need for more modules. While our initial experiments (refer to Table 2) compared the performance of different open-source along with GPT-4v, we found that GPT-4v was the best-performing one. Our future work would include using specialized fine-tuned LM modules. This would allow us to use smaller LMs and decrease the computational resources required to utilize all 4 modules.\n\n### Number of actions\n\nWhile our action space contains the following executable actions: Move(<direction>), Rotate(<direction>), LookUp(<angle>), LookDown(<angle>), NavigateTo(<object_id>), Pickup(<object_id>), Put(<receptacle_id>), etc. There are a lot of different combinations available where the <object_ids>, <directions>, and <angles> can change. Hence the action space inherently becomes combinatorially large. For this reason, we do not constrain the LM modules to choose actions in a fixed format from this set; rather, we obtain a free-form language output. This allows for more expressivity in the outputs. The sentenceBERT module maps these free-form natural language outputs to the executable actions in the environment.\n\n### Scaling to real-world settings\n\nWe agree that scaling LLM-based TAMP methods is a challenge. We perform experiments with LLaMAR in a multi-agent drone search-and-rescue environment (SAR). Refer to the global rebuttal. The idea of having these experiments was to test out the capabilities of LLaMAR in other multi-agent environments. While the metrics reported with LLaMAR are in a simulator, we are hoping to run more real-world experiments with these drones to test how they fare when executed in a more realistic setting.\n\n### League++ Reference\n\nThanks a lot for bringing this very recent paper (April 2024) to our attention. While League++ does tackle the TAMP problem in conjunction with reinforcement learning a few key differences to our work are:\n\n1. They focus on single-agent scenarios whereas we focus on multi-agent planning\n2. They use LLMs for reward generation and skill acquisition.\n3. The agent is given a list of all objects as a part of the observation which allows for creating plans and performing an A* search to verify the plan. In our case, we have a partially observable view of the environment which does not allow for performing search over the possible tasks/actions.\n\n### Challenge posed by over-restricted environments \n\nHaving over-constrained systems in various robotics settings is a well-known challenge (eg., freezing-robot problem) that could lead to issues in scaling. We agree with the comment on how our proposed method is also limited in similar settings. We will add this in our limitations section in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We would like to thank you all for your comprehensive reviews and insightful feedback on our paper. We appreciate your thoughtful comments and suggestions. A few of the reviewers pointed out that some of the tasks do not require explicit cooperation within the multi-agent team and the performance metrics degraded when the number of agents was increased from 3 to 5. The decrease in metrics can be attributed to the rooms in MAP-THOR becoming crowded with 4 and 5 agents hence blocking the agents from navigating without colliding with other agents. We wanted to address this with an experiment with our proposed approach LLaMAR in a different environment.

We implemented a partially observable search & rescue and fire relief environment for multi-agent systems in a grid world. Depending on the scene, there is a mix of missing people to be found, and wildfires to be stopped before they spread geographically.

- Each fire is composed of a large flammable region with a fixed set of sources that spread through time. The higher the intensity, the faster the fire will spread. The fires can be of class A or B, extinguished through the use of water and sand respectively. Both these resources can be collected through resource reservoirs spread geographically.
- Each person is initially stranded in an unknown location. The goal is to rescue and transport each person to a drop-off location (known apriori). The person must have two agents simultaneously carrying them to be transported to the drop-off location.

We provide a justification to why this task is relevant to long-term planning and multi-agent collaboration below.

### *Multi-agent nature*

- Scalability - prevents crowdedness, can increase complexity
    - With the addition of multiple parallel emergency disaster scenarios at once, the environment scales comfortably to more agents. Additionally, the complexity slowly increases as the model has to coordinate between more actors and competing emergencies.
- Task require explicit cooperation
    - In order to move a group of humans, at least 2 agents are required. Thus, explicit multi-agent collaboration is required.
    - Due to the time pressure, in order to successfully stop the fire, drones must all effectively collaborate in collecting and strategically using resources in high intensity regions.
- Exploration task assignment
    - There is an explicit tradeoff between allocating drones towards exploring to find the lost people, and fighting the current fires.

### *Long-term planning*

- Dependency chain
    - In order to resolve a fire it requires that the source is identified, the type of fire is identified, and the appropriate resources to stop it are acquired and then used.
- Uncertainty
    - Inherent uncertainty as to where the lost people can be found, and at what point in the timeline they will.
- Irreversibility and forced consequences
    - With a fire that spreads, present actions have irreversible future consequences. A balance needs to be struck between fighting the fire’s source (to stop it from continuing) versus a periphery (to prevent geographic spread).

### Description of all scenes:

Each scene has one reservoir for each fire type.

- *Scene 1* - (one) Type A fire with 2 initial sources, (one) Type B fire with 1 initial source, and (one) lost person
- *Scene 2* - (one) Type A fire with 1 initial source, (one) Type B fire with 2 initial sources, and (one) lost person
- *Scene 3* - (two) Type A fires each with 1 initial source, (one) Type B fires with 1 initial source, and (one) lost person
- *Scene 4* - (one) Type A fire each with 3 initial sources, and (one) lost person
- *Scene 5* - (one) Type A fire with 1 initial source, (one) Type B fire with 1 initial source, and (two) lost people

We evaluate each scene on 5 different random seeds.

## Results

The metrics shown are means along with the 95% confidence intervals shown in parentheses. Since success rates are binomial, we use the Clopper-Pearson Interval as the confidence interval. For the metrics: “Success Rates (SR)”, “Transport Rates (TR)”, “Coverage (C)” and “Balance(B)”, higher is better and for “Steps (L)” lower is better

| # agents | Success Rate | Transport Rate | Coverage | Balance | Steps |
| --- | --- | --- | --- | --- | --- |
| 2 | 0.44 (0.24, 0.65) | 0.86 (0.79, 0.94) | 0.94 (0.88, 0.99) | 0.91 (0.88, 0.95) | 27.76 (24.15, 30) |
| 3 | 0.68 (0.46, 0.85) | 0.92 (0.86, 0.98) | 0.96 (0.91, 1.0) | 0.80 (0.73, 0.86) | 21.88 (17.83, 25.92) |
| 4 | 0.72 (0.50, 0.85) | 0.94 (0.88, 0.98) | 0.98 (0.93,1.0) | 0.78 (0.74, 0.83) | 22.00 (17.96, 26.03) |
| 5 | 0.74 (0.52, 0.86) | 0.96 (0.94, 0.99) | 1.0 (1.0, 1.0) | 0.73 (0.67, 0.79) | 24.52 (20.24, 28.79) |

## Analysis of results

There is an observed increasing trend in success rate, transport rate, and coverage when the number of agents increases from 2 to 5 suggesting that more agents lead to better completion of the tasks on average.

The balance metric, however, follows a monotonically decreasing trend from 2 to 5 agents. This would be expected due to the increasing hardness of equal task allocation between many agents. Similarly, the steps metric shows a decrease from 2 to 3, but monotonic increases from 3 through 5. This fact can similarly be expected due to emergent inefficiencies in the performance of many agents.

We will include these in our revised manuscript.

Meta-review: This paper presents LM-based Long-Horizon Planner for Multi-Agent Robotics (LLaMAR) an approach that combines LLMs in a  plan-act-correct-verify framework to solve long-horizon multi-agent tasks. It also introduces the MAP-THOR test suite for evaluating results. The proposed method performs well in the experiments. 

LLMs are becoming more widely used but current methods are still not very good at complex, partially observable, long-horizon tasks. Developing methods that can solve more challenging tasks is an important area and the proposed method is a promising step in this direction.  

There are some questions about the motivation and details of the particular architecture, the high computational complexity, and whether the current approach is really multi-agent but the authors addressed them in the rebuttal/discussion. The authors should include these points (as well as other feedback from the reviewers) in the final paper.
Decision: Accept (poster)