Building Cooperative Embodied Agents Modularly with Large Language Models
Authors: Hongxin Zhang
Weihua Du
Jiaming Shan
Qinhong Zhou
Yilun Du
Joshua B. Tenenbaum
Tianmin Shu
Chuang Gan
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Embodied Intelligence
Multi-Agent Cooperation
Human-AI Interaction
Communication
Primary area: applications to robotics, autonomy, planning
TL;DR: We present CoELA, a modular framework integrating LLMs to address the challenging multi-agent embodied cooperation problem with decentralized control, costly communication, and long-horizon multi-objective tasks.

Abstract: In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments. While previous research either presupposes a cost-free communication channel or relies on a centralized controller with shared observations, we harness the commonsense knowledge, reasoning ability, language comprehension, and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspired modular framework that integrates with perception, memory, and execution. Thus building a Cooperative Embodied Language Agent CoELA, who can plan, communicate, and cooperate with others to accomplish long-horizon tasks efficiently. Our experiments on C-WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication. Though current Open LMs like LLAMA-2 still underperform, we fine-tune a CoELA with data collected with our agents and show how they can achieve promising performance. We also conducted a user study for human-agent interaction and discovered that CoELA communicating in natural language can earn more trust and cooperate more effectively with humans. Our research underscores the potential of LLMs for future research in multi-agent cooperation. Videos can be found on the project website https://vis-www.cs.umass.edu/Co-LLM-Agents/.

Published as a conference paper at ICLR 2024
BUILDING COOPERATIVE EMBODIED AGENTS MODULARLY WITH LARGE LANGUAGE MODELS
Hongxin Zhang', Weihua D Jiaming Shan?, Qinhong Zhou' Yilun Du', Joshua B. Tenenbaum', Tianmin Shu', Chuang Gan 1,5 Uniiveriity of Massachusetts Amherst, Institutee for Interdisciplinary Information Sciences, Tsinghua University, SShangaai Jiao Tong University, "MIT, MIITIIBM Watson Al Lab
ABSTRACT
In this work, address challenging multi-agent cooperation problems with de- centralized control, raw sensory observations, costly communication, and multi- objective tasks instantiated in various embodied environments. While previous search either presupposes a cost-free communication channel or relies on a central- ized controller with shared observations, reasoning ability, language comprehension. and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspire modular framework that integrates with perception, memory, and execution. Thus building Cooperative Embodied Language Agent CoELA, who can plan, communicate, and cooperate with others to accomplish WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication. Though current Open LMs like LLAMA-2 still underperform, we fine-tune munettnee a CoLLAMA with data collected with our agents and show how they can achieve promising performance. We also conducted a user study for human-agent interaction and discovered that CoFiAA communicating in natural language earn more trust and cooperate more effectively with humans. Our research underscores the potential LLMs for future research in multi-agent cooperation. Videos can be found the project website ht tpsps: / //vis www. CS umass edu/ Coo LLM- Agents//tts
harness the commonsense knowledge,
tasks efficiently. Our experiments on C-
INTRODUCTION
Humans are adept at cooperating and communicating with others when solving complex tasks (Woolley et al., 2010). Building embodied agents that can also engage in and assist humans in everyday life is valuable but challenging task, considering the complexity of perception, partial observation, long- horizhorizon planning, natural language communication, and so on (Deitke et al., 2022).
Large Language Models (LLMs) have exhibited remarkable capabilities across various domains, implying their mastery of natural language understanding, dialogue generation, rich world knowledge, and complex reasoning capability (OpenAI, 2023; Touvron al., 2023; Brown et al., 2020; Bubeck et al., 2023). Recent research has also demonstrated that L.MMs can drive embodied agents for single-agent tasks through zero-shot prompting for instruction following tasks (Huang et al., 2022a) or few-shot prompting for more complex long-horizon tasks (Song et al., 2022). However, building cooperative embodied agents work with other agents or with humans under decentralized settings with costly communication remains challenging and rarely explored, where they also need to have strong abilities for cooperative planning and efficient communication. To date, it still remains unclear whether LLMs have such abilities necessary for distributed embodied multi-agent cooperation.
Therefore, this paper aims investigate how to leverage LLMs build cooperative embodied agents that can collaborate and efficiently communicate with other agents and humans to accomplish long- horizon multi-objective tasks in a challenging decentralized setting with costly communication. To this end, we focus an embodied multi-agent setting shown in Figure 1, where two decentralized
denotes equal contribution.

---

Published as a conference paper at ICLR 2024
Alice (LLM)
"Hi, Bob, found larget objects the hitchen. remeember
Bob (Human/AI)
"OK, thanks for information, I'll go to transport them."
you were holding an empty container, can you come here to pick themm up while go to explore other roorss?"
Destge
Apshc
Apple
Figure 1: A challenging multi- cooperation problem with decentralized control, raw sensory observations, costly communication, and long- oon--horizon multi-objective tasks.
embodied agents have to cooperate to finish a multi-objective household task efficiently with complex partial observation given. Specifically, communication in our setting takes time as in real life, so the agents can't 't simply keep free talking with each other. To succeed in this setting, agents must perceive the observation to extract useful information, ii) maintain their memory about the world, the task, and the others, iii) decide what and when to communicate for the best efficiency and iv) plan collaboratively to reach the common goal.
Inspired by prior work in cognitive architectures (Laird, 2019), we present CoELA, a Cooperative Embodied Language Agent, a cognitive architecture with novel modular framework that utilizes the rich world knowledge, strong reasoning ability and mastery natural language understanding and generation capability of LLMs, who plan and communicate with others to cooperatively solve complex embodied tasks. Our framework consists of five modules, each to address a critical aspect of successful multi-agent agent cooperation, including a Perception Module to perceive the observation and extract useful information, a Memory Module mimicking human's long-term memory to maintain the agent's understanding of both the physical environment and other agents, Communication Module to decide what to communicate utilizing the strong dialogue generation and understanding capability of LLMs, a Planning Module decide high-level plans including when communicate considering all the information available, and an Execution Module to execute the plan by generating primitive actions using procedures stored in the memory module.
We instantiate our challenging setting and evaluate our framework on two embodied environments: ThrleDWorld Multi-Agent Transport (TDW-MAT) and Communicative Watch-And-Helly (C-WAH). Our experimental results indicate that CoELA can perceive complex observations, reason about the world and others' state, communicate efficiently, and make long-horizon plans accordingly, as showcased in Figure 1 where CoELA divide the labor with its partner through natural language communication effectively. In particular, CoELA driven by GPT-4 can outperform strong planning- based baselines by achieving more than 40% efficiency improvements and exhibiting emergent efficient communication. Though Open LMs like LLAMA-2 still underperform, we utilize parameter- efficient fine-tuning techniques LoRA (Hu et al., 2021) train CoI.AMM on few data collected with our agents and gain promising performance. In the user study, we also discover that CoELA communicating with humans in natural language can earn more trust. Our contribution includes:
We formalized a challenging multi-agent embodied cooperation problem with decentralized control, complex partial observation, costly communication, and long-horizon multi-objective tasks, and instantiated it in two embodied environments: C-WAH and TDW-MAT.
We presented a novel Dognitive-inspieee modular framework that utilizes the strong planning and communication capability of the LLMs to build cooperative embodied agents CoELA, surpassing strong planning-base based methods.
We conducted user study to evaluate the possibility of achieving effective and trustworthy human- AI cooperation using LLMs.
RELATED WORK
Multi-Agent Cooperation and Communication The field of multi-agent cooperation and commu- nication has a long-standing history (Stone & Veloso, 2000). Many platforms have been proposed for various multi-agent tasks (Lowe et al., 2017; Resnick et al., 2018; Shu & Tian, 2018 008;; Jaderberg
2

---

Published as a conference paper at ICLR 2024
et al., 2019; Samvelyan et al., 2019; Suarez et al., 2019; Baker et al., 2019; Bard et al., 2020). Other works focused on methods that improves communication efficiency (Jiang & Lu, 2018; Das et al., 2019; Wang et al., 2021; Wan et al., 2022), cooperation in visually rich domains (Jain et al., 2020), or grounding communications in environments (Patel et al., 2021; Mandi et al., 2023; Narayan- et al., 2019). For embodied intelligence, Puig et al. (2021; 2023) explored the social perception of the agents during their cooperation. However, these platforms either neglects communication (Jaderberg et al., 2019; Samvelyan et al., 2019; Carroll al., 2019; Puig et al., 2021; 2023), or use uninterpretable continuous vectors (Jiang & Lu, 2018; Das et al., 2019) or limited discrete sym- bols (Lowe et al., 2017; Jaques et al., 2019; Jain et al., 2020; Patel al., 2021; Resnick al., 2018) for communication. In contrast, we propose a more challenging setting where no presupposed free communication channel exists, and distributed agents need to use natural language to communicate efficiently with others, especially humans.
Language Agents Recently, numerous studies have explored language agents which use LLMs for sequential decision-making (Yang et al., 2023; Wang et al., 2023b; Xi et al., 2023; Sumers et al., 2023). Although LLMs still face challenges when solving complex reasoning problems (Bubeck et al., 2023), substantial body of work demonstrates their capacity to make plans (Sharma et al., 2021; Raman et al., 2022; Pallagani et al., 2022; Gramopadhye & Szafir, 2022; Yuan et al., 2023; Li et al., 2022; Wang et al., 2023d), especially in embodied environments (Li al., 2023a; Padmakumar et al.,
2022: Kolve et al., 2017; Shridhar et al., 2020; Misra et al., 2018; Zhu et al., 2017; Brodeur et al.,
2017; Xia al., 2018: Savva al., 2019: Xiang al., 2020; Jain et al., 2020; 2019). Specifically, Liang et al. (2022); Song et al. (2022) used codes or few-shot prompting to directly generate plans, Huang et al. (2022b) built an inner monologue with environment feedback to improve planning, Ahn et al. (2022) combined robotic affordances and LLMs for grounded instruction following. There has also been a line of work utilizing multiple LLMs to cooperate or debate with each other "in mind" to strengthen the single agent's s capability to solve complex tasks (Li et al., 2023b; Du et al., 2023; Wang et al., 2023c), different from their "free self-talk" setting, our decentralized language agents must plan about when and what to communicate carefully since it's costly in real-life. More recently, Park et al. (2023) built an agent socicty using LLMs augmented with memories to simulate human behavior. In contrast the above, our work addresses a more challenging multi-agent cooperation problem, characterized by decentralized control, complex observations, costly communication, and long-horizon multi-objective tasks. We also study the capability of Open LMs like LLAMA-2 and tine-tune CoLLAMA using LoRA with data collected by our agents in embodied environments to demonstrate their promising performance for building better cooperative embodied agents.
3 COOPERATIVE PLANNING UNDER DEC-POMDP-COM
Our setting can be defined as an extension of the decentralized partially observable Markov decision process (DEC-POMDP) (Bernstein al., 2002; Spaan al., 2006; Goldman & Zilberstein, 2003), which can be formalized by (n, S, (Ex), {A:), {Ot), T, G. R, 7, h). where n denotes the number of agents; S is a finite set of states; A, = AWW AÇ is the action set for agent i, including finite set of world actions AWW and a communication action AÇ to send a message d; E E;; O O. x of is the observation set for agent i, including world observations O.W the agent receives through its sensors, and O = E, X ... x Σ,, the set of possible messages the agent can receive from any of its teammates; T(s,a, 8') = p(s'|s [ss a) is the joint transition model which defines the probability that after taking joint action E A₁ X ..... x A, in E S, the new state s' E S is achieved; G = {91, ....... .9k] defines the task with several sub-goals for the agents to finish; R(s,a, = -c(a) + Et_ f= 1(s' = 9:) 1 (s 9:) is the reward function to the team, where c(a) is the cost for action a, and 1(-) checks if the sub-goal gi is satisfied in the world state 8; is the discount rate and / is the planning horizon. In the remainder of this paper, we focus on noise-free broadeast communication and limit our discussion to two agents, though our methods and experiments are generalizable to more than two agents.
We instantiate the problem with two decentralized intelligent embodied agents (including humans) cooperating to accomplish a long-horizon rearrangement task (Batra et al., 2020) in an indoor multi- room environment. The agents are capable of executing one of the actions from the action space A ANAV U AINT U AcOM, where ANAV includes navigation actions, ANNT includes interaction actions and ACOm includes a communication action with which the agent can send a message in natural language broadcast to others. The rearrangement task is defined with several predicates
3

---

Published as a conference paper at ICLR 2024
Our Agent
(b)
Memory Module
Environment
(a)
Update
Semantic Mem
Simulator
Perception Module
(b1)
Obs.
Semantic map Task progress
(c)
Retrieve
Agent state
Communication
Module
Episodic Mem
Message
(b2)
Obs.
Act.
Act.
(d)
Retrieve
Action history
Planning Module
Dialogue history
Other Agent/Human
Update
Plan
Procedural Mem
Retrieve
(b3)
Execution Module
Agent code
(e)
LLM param
An overview of CoELA. There are five key modules in our framework: (c) The Communi- cation Module and (d) the Planning Module leverage LLMs generate messages and make plans, (b) The Memory Module stores the agent's 's knowledge and experience about the world and others in semantic, episodic and procedural memory respectively, (a) The Perception Module and (e) the Execution Module interact directly with the external environment by perceiving raw observations and generating primitive actions. More design details can be found in Appendix A.
Figure 2:
Ii with counts to be satisfied, such as ON (p ate, dinnertable) :2 representing a sub-task of putting two plates onto the dinner table.
BUILDING COOPERATIVE EMBODIED AGENTS MODULARLY WITH LLMS
4.1 FRAMEWORK OVERVIEW
Inspired by the cognitive architectures (Langley et al., 2009; Laird, 2019; 2022), we build CoELA, Cooperative Embodied Language Agent with novel modular framework integrating the strong reasoning ability and language generation capability of LLMs. As shown in Figure 2, CoELA consists of five key modules: (a) Perception, (b) Memory, (c) Communication, (d) Planning, and (e) Execution. At each interaction step. CoELA first uses (a) Perception Module to perceive the raw sensory observation received from the environment, then updates the (b) Memory Module with extracted new information, which stores its knowledge and experience of the world and others. CoELA tackles the challenge of efficient communication with two-step method: first decide on what to send, then decide whether to send this message or choose another plan by deliberately using (c) The Communication Module to retrieve related information from (b) and utilize an LLM to generate the best message to send "in mind" beforehand, then leverages (d) the Planning Module driven by LLM with strong reasoning ability to make the decision on which plan to take given the related information retrieved from (b) and available actions proposed regarding the current state. The generated plan is then used to update (b2) the Episodic Memory. Finally. (e) the Execution Module retrieves procedural knowledge stored in (b3) turn the high-level plan into primitive actions executable in the environment.
4.2 PERCEPTION MODULE
For embodied agents to be helpful in the real world, they have to perceive raw observations gained through sensors and extract useful information for downstream higher-order reasoning. We incor- porate the Perception Module to deal directly with the complex visual observation received from the environment by training Mask-RCNN (He et al., 2017) to predict the segmentation masks from the RGB image, then build 3D point clouds using the RGB-D image, extract useful high-level information such the states of the key objects and build local semantic map.
4.3 MEMORY MODULE
It's of vital importance for an agent to maintain a memory of the knowledge and experience it has of the world and others, we mimic human's long-term memory (Atkinson & Shiffrin, 1968; Wang &
4

---

Published as a conference paper at ICLR 2024
Laird, 2006; Nuxoll & Laird, 2012) and design Semantic memory, Episodic Memory, and Procedural Memory for CoELA.
Semantic Memory stores CoELA's 's knowledge about the world including semantic map, the task progress, the state of self, and the state of others. Each time a new observation is received and perceived by the Perception Model, the Semantic Memory is updated accordingly. To be noticed, COELA'S knowledge about the world may not be accurate since other agents may interact with the objects and change their states without its awareness. Dealing with imparities between the memory and the description of the world from others adds even more challenges.
Episodic Memory stores CoELA's experience about the past including the action history and dialogue history. Each time CoELA executes a new action including sending out a message or receiving a new message, the related information is added to the Episodic Memory.
Procedural Memory contains knowledge including how to carry out specific high-level plans in a specific enviromment implemented in code and the neural models' parameters.
4.4 COMMUNICATION MODULE
To deal with the what to send problem, we deliberately design a Communication Module utilizing the strong free-form language generation capability of the LLMs to act as a message generator. To better condition the LLMs on the cooperative task and avoid inefficient casual chatting, the Communication Module first retrieves the related information from the Memory Module including the semantic map, task progress, agent state, others state, and the action and dialogue history, then convert these into text descriptions using templates, finally prompt the LLMs with the concatenation of Instruction Head, Goal Description, State Description, Action History, and Dialogue History to generate the message to send. To better constrain L.LMs' generated messages, a note at the end of the prompt added and two seed messages are appended at the beginning of the Dialogue History to elicit deserved effective communication behavior. Detailed prompt design in Appendix. A.3.
4.5 PLANNING MODULE
CoELA needs strong Planning Module to make decisions which action to take utilizing all avail- able information gathered and stored so far to maximize cooperation efficiency. While designing such module from seratch consumes large human expert efforts and is nearly impossible generalize, we utilize powerful LLMs directly as the Planning Module by first retrieving the related information from the Memory Module and converting them into text descriptions in the Communication Module, then compile an Action List of all available high-level plans proposed according to the current state and the procedural knowledge stored for the LLMs to make the choice, which formalization makes easier for the LLMs to concentrate on the reasoning and make an executable plan without any few-shot demonstrations easily, finally prompting the LLMS with current information and the proposed Action List to generate a high-level plan. We also use the zero-shot chain-of- oftthhuhhttttt prompting technique introduced by Kojima et al. (2022) to encourage the LLMs to carry out more reasoning before giving the final answer. More details can be found in Appendeix. A.4.
4.6 EXECUTION MODULE
As shown in (Deitke et al., 2022). solving challenging embodied tasks requires modular methods to tackle the complexity of tasks. We found that while LLMs were effective making high-level plans, they were poor at making low-level controls, as also discussed in (Wu et al., 2023). Thus, to enable effective and generalized cooperation an Execution Module to generate primitive actions to execute a given high-level plan robustly in a specific environment, allowing the Planning Module to be generalizable and focus more on solving the overall task with LLMs" rich world knowledge and strong reasoning ability. Practically, this design can also reduce the LLM inference time and is time-saving and economical. CoELA retrieves the procedures in its Memory Module regarding the plan generated by the Planning Module and then carries out the procedure with primitive actions suitable for the environment.
prcision-making in different environments,
design
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
ThreeDWorld Multi-Agent Transport (TDW-MAT) is a multi-agent embodied task extended from the DWrld Transport Challenge (Gan et al., 2022) with more types of objects and containers, more realistic object placements, and communication between agents supported, built top of the TDW platform (Gan et al., 2021), which is a general-pune purpose virtual world simulation platform. The agents are tasked to transport as many target objects as possible to the goal position with the help
5

---

Published as a conference paper at ICLR 2024
of containers tools. The agents receive ego-centric 512x512 RGB-D images observation and have an action space of low-level navigation control, interaction, and communication. We selected 6 scenes from the TDW-House dataset and sampled 2 out of the two types of tasks food and stuff in each of the scenes, making a test set of 24 episodes, and instantiate the horizon h with 3000 frames.
Communicative Watch-And-Help (C-WAH) is extended from the Watch-And-Help Challenge (Puig et al., 2021) built on a realistic multi-agent simulation platform, VirtuaHHome-Socie (Puig et al., 2018; 2021), where we focus more on cooperation ability and support communication between agents. We conduct experiments under both symbolic and visual observation settings. The task is defined as five types of common household activities and represented as various predicates with counts to be satisfied. We sampled 2 tasks from each of the five types of activities to construct test set of 10 episodes and instantiate the horizon h with 250 steps. More details can be found at Appendix. B.
Metrics We use the Transport Rate (TR), the fraction of the sub-goals satisfied on TDW-MAT, and the Average Steps taken to finish the task on C-WAH as main efficiency metrics respectively and calculate Efficiency Improvement (EI) of cooperating with other agents as AM/Mo. where AM denotes the main efficiency metric difference, and Mo denotes the larger one of the main efficiency metric for numerical stability.
5.2 BASELINES
based Hierarchical Planner(MHPP is adopted from the strongest baseline in the original Watch-And And-Help Challenge, which is a Hierarchical Planner with a high-level planner based on MCTS and a low-level planner based on regression planning (Korf, 1987).
Rule-based Hierarchical Planner(RHP) is adopted from the strong performing baseline in the original ThreeDWorld Transport Challenge, which is a Hierarchical Planner with a high-level planner based on heuristics rules and low-level A-start-based planner to navigate with semantic map, using Frontier Exploration strategy which randomly samples a way-point from an unexplored area as a sub-goal for exploration.
Multi-Agent Transformer(MAT) is MARL baseline that applies centralized decision transformer to generate actions from shared observations (Wen et al., 2022). To apply MAT in our setting, we make the compromise to feed the oracle semantic map and the agent states observation and stack up to 50 frames as an RL step since TDW-MAT is too hard for it with long-horizon and sparse reward signals. We train MAT on the training set with more details in Appendix. C.I.
Implementation Details. We train a RCCN- on the training set for the Perception Module and instantiate CoELA with the most powerful LLM GPT-4 from the OpenAI API' with the default parameter of temperature 0.7, top-p 1, and max tokens 256 unless other stated. We also conduct experiments with Open L.M LLAMA-2-1 1-AMAA----3-hhaa 13b-chat (Touvron et al., 2023) and fine-tune a Col.LAMA with LoRA (Hu et al., 2021) on a small set of human-filtered high-quality trajectory data collected with our agents. More details are deferred the Appendix. C.3.
5.3 RESULTS
5.3.1
COLLABORATING WITH AI AGENTS
CoELA cooperates better with baseline agent As shown in Table 1, compared with RHP doing the task alone, cooperating with CoELA leads to a higher TR and EI than cooperating with another RHP (0.69(36%) v.s. 0.61(29%)). even without any knowledge of the inner working mechanism of others, showing CoELA can rea- son about the other agent's state well without hand-designed heuristics. From Table 2, we can observe the same performance boost of cooper- ating with CoELA on C- C-W of 45% compared to 33% of cooperating with the same MHP.
Symbolic Obs
Visual Obs
MHP
111
141
MHP + MHP
75(133%)
103(126%)
MHP + COELA
59(145%)
94(134%) %
CoELA CoELA
57(149 %
92(134%)
Quantitative results on C-WAH. We report the average (EepscEff Improvement) here over runs for MHP and run for CoELA due to cost constraints. The best performance is achieved when cooperating with CoELA.
Table 2:
CoLLAMA is in competence with GPT-4 to drive CoELA Two CoELA cooperate together can further boost the TR to 0.71 and 0.85 on TDW-
'Ou main experiments are done between 1.2.9.11202339225 and 2023.5.1-2023.5.0 1-222.55.166

---

Published as a conference paper at ICLR 2024
CoELA COELA
MAT
RIIP
RIIP RIP
RIIP + CoELA
GPT-4
LLAMA-2 CoLLAMA-2
TDW-MAT
Food
0.49
0.67(125%)
0.79()39%)
0.82(138%)
0.57(19%)
0.73(133%)
Stuff
0.36
0.54(134%)
0.59("34%)
0.61(141%)
0,48(1119)
0.66()44%)
Total
0.43
0.61(129%)
0.69()36%)
0.71(139%)
0.53(110%)
0.70(138%)
TDW-MAT w/ Oracle Perception
Food 0.52
0.76(133%)
0.85(740%)
0.87(141%) 0.83(141 %)
0.60(13%)
0.78(134%)
0.13(J)
Stuff 0.49
0.74c134%)
0.77("35%)
0.81(138%) 0.63(119%)
0.17())
Total
0.50
0.75(134%)
0.81()37%)
0.85(141%)
0.62(18%)
0.80(136%)
0.15(4)
Table I: Quantitative results on TDW-MAT. We report the average Transport Rate Efficiennyy Improvement) here over runs for RHP and I run for CoELA due to cost constraints. *MAT uses central observation and oracle perception. The best results are in bold. The best performance is achieved when cooperating with CoELA.
adapt plans
respond to requests
Hey Afher Jennde santen bett tha the deeeeoeee heatere be NeTeeee Eeeende butae contue cnliciotto agaui abamncher
Bebl exponne balor desidon
Al andher plan cac dult aloudy in trchen
Bob
aquur be piln ccopaiie with Alie
Pannedhel boan cum war catttaaa
vigge mnd
Mone
015 2ide yolferrceee 2266 Rodioo
c. not to communication
know to request
share information
Alle Lat's wpderccthhdbrrttt..ooo ser
dovegroue Syhnerr/rrij -dhirgraeen (Tvnet
un nb Mlre
MMc gro tame
Fina deaid prirde -iraapenn 1jes trgat dijac oullcu (hW/jj
Aeto www. 221 100aaiaa Eragne
Alle, De 20
Figure 3: Example cooperative behaviors demonstrating CoELA can communicate effectively and are good cooperators.
MAT without and with Oracle Perception. While replacing GPT-4 with open Model LLAMA-2 leads to significant performance drop. our fine-tuned CoLLAMA can gain a competitive performance of 0.70 TR and even surpass GPT-4 on the subtask of Stuff where GPT-4 performs not so well, showing the promising future of fine-tuning open L.MSs with our proposed framework on embodied environments for even better cooperative embodied agents.
CoELA exhibit efficient communication and effective cooperation behavior To better understand the essential factors for effective cooperation, we conduct qualitative analysis of the agents' behaviors exhibited in our experiments and identified several cooperative behaviors: CoELA share progress and information with others, know when to request help and can respond to others' requests, adapt plans considering others and knows when not to communicate, shown in Figure 3. We discuss some here and the remaining in the Appendix. C.4.
5.3.2 COLGABORATING WITH HUMANS
It's our ultimate goal build agents that can cooperate with humans, user study is important. We conducted human experiments on the C-WAH where the agent Alice is controlled by real humans.
We recruited 8 human subjects to perform the experiments under four scenarios: cooperating with the MHP CoELA, CoELA w/o communication, and doing the task alone. Subjects have access to the same observation and action space as the agents, they can click on visible objects and select actions
AA template communication is used here study humans' communication preference, details in Appendix F
7

---

Published as a conference paper at ICLR 2024
Steple CoELA
MHP
CollA
NHP
CaBlA w/o camm
CoflA w/o
GoELA
120
Average Stepe
100
w/o eocurn
60
80
w/o memoryy
GPT-35 difves
20
Human Driiing Rlcee
Agent Driving Mice
Comm Effectivenis
Helpheliness
Trust
Average Stage
(a)
(c)
Figure 4: Human experiments results (a) The Average steps when collaborating with Humans and
agents. (b) Subjective Rating Humans give when cooperating with different agents. Humans trust CoELA communicating in natural language more and cooperate more efficiently with them. Ablation results (c) The light-colored portions represent the number of steps used for communication. The Memory Module and strong LLM for the Planning Module are important, while the Communication Module matters more when cooperating with humans.
to interact with them, including navigation to each room and communication through a chat box. We gave each subject a tutorial and they had the chance to get familiar with the interface in few pilot trials. We evaluate the same 10 tasks as in previous experiments and each task was performed by at least 2 subjects, making 80 trials in total. We made sure each subject do 10 trials with least two trials under each scenario. After each trial including a baseline to cooperate with, we asked subjects to rate the agent they just cooperated with on 7-point Likert Scale based on three criteria adapted from Puig et al. (2021): (i) How effective do you think of your communication with the other agent Bob? Did understand your message andlor share useful information with you? (ii) How helpful do you find the other agent Bob? Did it help you achieve the goal faster? (iii) How much do you trust the other agent Bob? Would you feel safe doing the task with it, or you rather do the task alone?
As see in Figure 4a, when cooperating with humans, CoELA still performs better than MHP, and when communication unable, CoELA w/o communication encounters a performance drop. As reported in Figure 4b, also observe that humans would trust the agents more if they can communicate with humans (trust score of 6.3 v.s. 4.7 for CoELA v.s CoELA w/o communication. p=0.0003 over the t-test), and therefore achieves better cooperation. Compared with MHP using template language to communicate, humans prefer to collaborate with CoELA who communicates in natural language and can understand and respond to Human dialogues. We show effective communication example in Figure 10, where the human first shares his progress with CoELA and suggests a labor division, CoELA understands and responds with its future plan as well, resulting in perfect division of the exploration trajectory. These results imply promising futures for leveraging LLMs to build cooperative embodied agents that can successfully work with humans.
5.4 ANALYSIS
Do we need a strong LLM for the Planning and Communication Module? As shown in Figure 4c, when replace GPT-4 with GPT-3355 to drive CoELA, the agents would need more steps to finish the task. GPT-3.5 makes more reasoning errors about the state and therefore generates more implausible plans, which leads CoELA spend more time finishing the task. GPT-3.5 also tends to generate unuseful messages more often than GPT-4. The performance gap can be attributed to more advanced reasoning and Theory of Mind abilities of GPT-4, which is also observed by Bubeck et al. (2023).
Is the communication effective? Though communication still fails in some cases, as shown in Figure 3, our agent exhibits effective communication behaviors, such as sharing information, requesting help, responding to requests, and knowing when not to communicate. More importantly, natural language communication provides us with lens to understand the decision- decision-making of the agents and could lead to better cooperation between humans and AI (as shown in section 5.3.2). We did not observe a significant performance drop when disabling communication among Al agents (as shown in Figure 4c), because carrying out efficient communication in our setting extremely challenging as communication costs time, requiring agents to model others accurately and understand the ambiguity of the natural language itself, which current LLMs still can not master robustly.
Is the Memory Module and Execution Module effective? As shown in Figure 4c, the steps needed to finish the task for the agent with no Memory Module nearly double, showing the importance of the Memory Module to store and update the knowledge and experience of the scene and the others. We also tried remove the Execution Module and let the Planning Module make low-level control
8

---

Published as a conference paper at ICLR 2024
directly at every step. However, this slows down the inference process largely and all trials perform poorly and struggle to finish any task.
Bob las started to
Prompt:
the burger intr the lea tray now
Aliee: Bob,
Goal Description: Transport 3 peas, 1 lighter, ipods, 2 purses, key bed.
plocae pau
But Alice canmot
your burger (258) into your tea tray (634) and
State Description I've takson 1818/3000 steps, We've already transported ipod (183 I), purse (4143). pen (2912), lighter (5824), purse (763 ipod (2088), ipod (9981), pen (3714) the bed.
infer this.
Let's think step by step.
explore
Livingroom
Reasoning Puth:
Firsi, you need find the remaining turget objects (2 pens),
Failure cases 6t1 TDW-MAT. (a) The Agent fails to teasor other already putting the burger into the container. (b) The LLM counts the number of the remaining target objects wrong.
Figure 5:
5.5 FAILURE CASES AND LIMITATIONS OF LLM
Though CoELA built with sota L.LSs is effective and has achieved impressive results, we find that the agent still falls short in several essential capabilities. We provide an in-depth analysis of its limitations and share some insights on designing better cooperative embodied agents for future work.
Limited usage of 3D spatial information. CoELA did not incorporate the spatial information of objects and rooms into consideration due to the challenge of effectively introducing the spatial information to pure text language models. This may cause the agents come up with a semantic sound exploration plan which is actually time-consuming. Work on multi-modal large models capable of both processing visual modalities effectively and generating natural language fluently (Huang et al., 2023; Driess et al., 2023; Lu et al., 2022) would help overcome this limitation and build better grounded embodied agents.
Lack of effective reasoning on low-level actions. To help LLMs better focus on solving the overall task, abstract high-level plans for LLMs directly reason on, reducing the potential decision space significantly, but also making it unaware of the execution of low-level actions, and impossible reason over them. which may lead to plausible but ineffective decisions. For example in Figure 5a, Alice saw Bob holding a container and a target object in both hands and figured he may not know how to utilize the containers, so sent a message instruct him put the object into the container, though Bob was actually putting in the objects at the same time, which is impossible for Alice to reason over now. Developing agents that can directly make low-level controls is essential for building better cooperative agents.
Unstable performance on complex reasoning. Although LLMs make correct reasoning most of the time, they still occasionally make mistakes, including misunderstanding the environment rules specified in the prompt, and incorrect reasoning over the number of unsatisfied goals (Figure 5b). These mistakes cause failures in planning. This calls for developing LLMs with stronger instruction following and reasoning capability.
6 CONCLUSION
In this work, we propose a novel modular framework integrating the Large Language Models to build cooperative embodied agents CoELA, who can plan, communicate, and collaborate efficiently with other agents and humans in a challenging multi-agent setting with decentralized control, complex partial observation, costly communication, and multi-objective hong-horizon tasks. Our experiments on two extended embodied multi- environments show the effectiveness of our proposed frame- work and exhibit several cooperative behaviors. We fine-tune a CoLLAMA from LLAMA-2 using data collected with our agents in embodied environments and showcase its promising performance to build better cooperative embodied agents. We also discover that CoELA communicating in natural language can cooperate better with humans and earn more trust from them. We believe that our work indicates promising future avenues to design even stronger embodied agents with LLMs for multi-agent cooperation. We further perform in-depth analysis of the limitations of the current LLMs and highlight several potential solutions for building better embodied cooperative agents for the future.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank Zishuo Zheng and Zhiqing Sun for their insightful discussions and help with the experiments, Jeremy Schwartz and Esther Alter for setting up the ThreeDWorld environments. We thank the anonymous reviewers for their helpful suggestions. This work is funded in part by grants from ONR Science of AI Program, Google, Amazon, Cisco, Toyota Motor North America, and Mitsubishi Electric Research Laboratories.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.00 01691,, 2022.
Richard C Atkinson and Richard M Shiffrin. Human memory: A proposed system and its control processes. In Psychology of learning and motivation, volume 2, pp. 89-195. Elsevier, 1968.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv: 1909.07528, 2019.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence. 280:103216 2020.
Dhruv Batra, Angel Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, et al. Rearrangement: A challenge for embodied ai. arXiv preprint arXiv:2011.01975 2020.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819-840, 2002.
Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, and Aaron Courville. Home: A household multimodal environment. arXiv preprint arXiv: 1017, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pp. 3338-1546, PMLR, 2019.
Matt Deitke, Dhruv Batra, Yonatan Bisk, Tommaso Campari, Angel X Chang, Devendra Singh Chaplot, Changan Chen, Claudia Pérez D' Arpino, Kiana Ehsani, Ali Farhadi, et al. Retrospectives the embodied ai workshop. arXiv preprint 2:Xxv:2210006844 2022.
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff. Andy Zeng. Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.088 033788 2023.
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factual- ity and reasoning language models through multiagent debate. arXiv preprint 13355 2023.
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowea, Martin Schrimpf. James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Tyler Feigelis, Daniel Bear, Dan Gutfreund,

---

Published as a conference paper at ICLR 2024
David Daniel Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh Mcdermott, and Daniel LK Yamins. ThreeDWorld: A platform for interactive multi-modal physical simulation. In Thirty-fitth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: wwwpnreeviviieeeeeeee ew net/ forum? ( d=db1 InWAWW2T.
Chuang Gan, Siyuan Zhou, Jeremy Schwartz, Seth Alter, Abhishek Bhandwaldar, Dan Gutfreund, Daniel LK Yamins, James DiCarlo, Josh McDermott, Antonio Torralba, et al. The threedworld transport challenge: A visually guided task-and-motion planning benchmark towards physically realistic embodied ai. In 2022 International Conference on Robotics and Automation (ICRA), pp- 8847-8854. IEEE, 2022.
Claudia V Goldman and Shlomo Zilberstein. Optimizing information exchange in cooperative multi-agent systems. In Proceedings of the second international joint conference on Autonomous agents and multiagent systems, pp. 137-144, 2003.
Maitrey Gramopadhye and Daniel Szafir. Generating executable action plans with environmentally- aware language models. arXiv preprint #rXiv:2210.04964 2022.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint SEIi:22006096883 2021.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma. Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 14045, 2023.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. International Conference on Machine Learning, pp. 91 11-811777 PMLR, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05560 2022b.
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, al. Human- level performance in 3d multiplayer games with population-based reinforcement learning. Science, 364(6443):855 865, 2019.
Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexan- der G Schwing, and Aniruddha Kembhavi. Two body problem: Collaborative visual task comple- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing. cordial sync: Going beyond marginal policies for multi-agent embodied tasks. In Computer 2020, Proceedings, Part V 16. pp. 47 471-490. Springer, 2020.
2020: 16th European Conference, Glasgow, UK, August 23-28,
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social inffuence as intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pp. 3040-3049, PMLR, 2019.
Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in neural information processing systems, 31, 2018
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.1 1916, 2022.
12

---

Published as a conference paper at ICLR 2024
Eric Kolve, Roozbeh Mottagii, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXlv:1712.05474, 2017.
Richard E Korf. Planning search: A quantitative approach. Artificial intelligence, 33(1))65-88, 1987.
John E Laird. The Soar cognitive architecture. MIT press, 2019.
John E Laird. Introduction to soar. arXiv preprint
2022.
Pat Langley, John E Laird, and Seth Rogers. Cognitive architectures: Research issues and challenges. Cognitive Systems Research, 10(2):14-1660 2009.
Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín- Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-Ik: A benchmark for embodied ai with 1, 1,000 everyday activities and realistic simulation. In Conference on Robot Learning, pp. 80-93, PMLR, 2023a.
Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Communicative agents for" mind" exploration of large scale language model society. arXiv preprint 1tt.:23333.17660 2023b.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang. Ekin Akyürek, Anima Anandkumar, et al. Pre-trained language models for interactive decision-maker makion.. Advances in Neural Information Processing Systems, 11331199-11222 2022.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.
Ryan Lowe, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor- critic for mixed cooperative-compeitive environments. Advances in neural information processing systems, 30, 2017.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206.0891 2022.
Zhao Mandi, Shreeya Jain, and Shuran Song. Roco: Dialectic multi-robot collaboration with large language models. arXiv preprint arXiv:2307.04736 2023.
Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi Mapping instructions to actions in 3d environments with visual goal prediction. arXiv preprint arXiv:1809.00886, 2018
Anjali Narayan-Chen. Prashant Jayannavar, and Julia Hockenmaier. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19. 1537. URL https: : aalaaaaaannnoooooooooo wttppoogg...... 1137.
Andrew M Nuxoll and John E Laird. Enhancing intelligent agents with episodic memory. Cognitive Systems Research, 17:34-48, 2012.
OpenAl. Gpt-4 technical report, 2023.
Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 2017-2025, 2022.
Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia. Plansformer: Generating symbolic plans using transformers. arXiv preprint 2Xiv:22/2.0868// 2022.
13

---

Published as a conference paper at ICLR 2024
Joon Sung Park, Joseph C O' O'Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra human behavior. arXiv preprint arXiv:2304.03442, 2023.
Shivansh Patel, Saim Wani, Unnat Jain, Alexander G Schwing, Svetlana Lazebnik, Manolis Savva, and Angel X Chang. Interpretation of emergent communication heterogeneous collaborative embodied agents. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15953-15963, 2021.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang. Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration. In International Conference on Learning Representations, 2021.
Xavier Puig, Tianmin Shu, Joshua B Tenenbaum, and Antonio Torralba. Nopa: Neurally-guided
online probabilistic assistance for building socially intelligent home assistants. arXiv preprint arXiv:2301.05223, 2023.
Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re- re-prompting. arXiv preprint drXiv:2211.09935. 2022.
Cinjon Resnick, Wes Eldridge, David Ha, Denny Britz, Jakob Foerster, Julian Togelius, Kyunghyun Cho. and Joan Bruna. Pommerman: multi-agent playground. arXiv preprint 1809.01809.0712 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung. Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Proceedings the the 18th International Conference on Autonomous
Agents and MultiAgent Systems, pp. 21 1186-1188, 2019.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. Habitat: A platform for embodied ai research. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9339-9347, 2019.
Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint drXiv:2110.01517, 2021.
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020.
Tianmin Shu and Yuandong Tian. rr Mind-aware multi-agent management reinforcement learning. arXiv preprint 1010.0011100014: 2018.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. arXiv preprint ar 1rXiv:22 04088, 2022.
Matthijs TJ Spaan, Geoffrey J Gordon, and Nikos Vlassis. Decentralized planning under uncertainty for teams of communicating agents. In Proceedings of the fifth international joint conference Autonomous agents and multiagent systems, pp. 249-256, 256, 2006.
Peter Stone and Manuela Veloso. Multiagent systems: A survey from a machine learning perspective. Autonomous Robots, 8:345-383, 2000.
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents. arXiv preprint arXiv:1903.0072 2019.
14

---

Published as a conference paper at ICLR 2024
Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09280 2023.
Yanming Wan, Jiayuan Mao, and Josh Tenenbaum. Handmethat:
Human-robot communication in physical and social environments. Advances in Neural Information Processing Systems, 35: 12014-12026, 2022.
Guanzhi Wang, Yugi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291 2023a.
Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308. 11432, 2023b.
Yongjia Wang and John E Laird. Integrating semantic memory into a cognitive architecture. Ann Arbor, MI: University of Michigan Center for Cognitive Architecture, 2006.
Yuanfei Wang, Jing Xu, Yizhou Wang, et al. Tom2c: Target-oriented multi-agen communication and cooperation with theory of mind. In International Conference on Learning Representations, 2021.
Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self- collaboration. arXiv preprint arXiv:2307.05300 2023c.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents, 2023d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903, 2022.
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi- agent reinforcement learning a sequence modeling problem. Advances in Neural Information Processing Systems, 15519969116521, 2022.
Anita Williams Woolley, Christopher F Chabris, Alex Pentland, Nada Hashmi, and Thomas W Malone. Evidence for a collective intelligence factor in the performance of human groups science, 33066004):686 688, 2010.
Yue Wu, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Yuanzhi Li, Tom Mitchell, and Shrimai Prabhumoye. Plan, eliminate, and track language models are good teachers for embodied agents. arXiv preprint arXiv:2305.02412. 2023.
Zhiheng Xi. Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. The rise and potential of large language model based agents: 2023.
A survey. arXiv preprint 20X9v:2309.07888
Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world perception for embodied agents. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9068-9079. 2018.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al. Sapien: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11097-11107, 2020.
Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foun- dation models for decision making: Problems, methods, and opportunities. arXiv preprint arXiv:2303.044 04199. 2023.
15

---

Published as a conference paper at ICLR 2024
Siyu Yuan, Jiangjie Chen, Ziquan Fu, Xuyang Ge, Soham Shah, Charles Robert Jankowski, Deqing Yang, and Yanghua Xiao. Distilling script knowledge from large language models for constrained language planning. arXiv preprint arX1v:2305.05252,2000 2023.
Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations. In Proceedings of the IEEE international conference on computer vision, pp. 483-492. 2017.
A ADDITIONAL. DETAILS ON THE FRAMEWORK
A.1 PERCEPTION MODULE
To deal with raw sensory observations, well-constructed Perception Module is needed for embodied agents to extract useful information for downstream higher-order reasoning.
In TDW-MAT, the environment provides an observation of 512 x 512 first-person view RGB image and Depth image. The agent first utilizes a pre- pre-trained Mask-RCNN (He et al., 2017) to obtain the instance segmentation mask, then combines it with the depth image and the agent's 's position to project each pixel into the 3D world coordinate to obtain a 3D voxel semantic map, and finally accumulates along the height dimension to build a top-down 2D semantic map of size L W x 3, where the first channel represents semantic classes including target objects, containers, destinations, and agents, and the last two channels represent the occupied and explored area respectively. Each element in the map denotes grid of size 0. 125m x 0. 0.125m in the scene. The agent also extracts the relationship of the objects with the help of instance segmentation masks and updates its Semantic Memory with the new information extracted from the observation.
To obtain more suitable model for instance segmentation in a TDW simulation environment, we fine-tune the MASK-RCNM model pre-trained on the MS COCO dataset in training scenes. By random sampling in the training environments, we collected 53K 512 512 RGB images and obtained the ground truth instance segmentation mask from the environment as the training set. The fine-tuned model achieves 81.4% MAP@ 50 in the test set.
Figure 6: A visualization of the semantie map stored in the Semantic Memory and updated with new observations at every time in the TDW-MAT environment. The destination is shown in red, target objects in blue, containers are in green, the agent is denoted with cyan, and the other agent's position in memory is denoted in yellow.
16

---

Published as a conference paper at ICLR 2024
A.2 MEMORY MODULE
We mimic human's long-term memory and design Semantic memory, Episodic Memory, and Proce- dural Memory for CoELA store the knowledge and experience has of the world, other agents, and itself.
Semantic Memory stores CoELA's knowledge about the world including semantic map as shown in Figure built and updated with local map perceived from the Perception Module, the task progress which is initialized with all zeros and updated whenever the agent is in the range of the goal position, the state of self including positions, holding objects status, and the state of others in memory which updated whenever the others is perceived in the observation. To be noticed, CoELA's knowledge about the world may not be accurate since other agents may interact with the objects and change their states without its awareness. Dealing with imparities between the memory and the description the world from others adds even more challenges.
Episodic Memory stores CoELA's experience about the past including the action history and dialogue history. Each time CoELA executes a new action including sending out message receiving new message, the related information is added to the Episodic Memory. Empirically, we only keep the last K actions and D dialogues for storage efficiency.
Procedural Memory contains knowledge including how to carry out specific high-level plans in a specific environment implemented in code and the neural models' parameters including LLMS and Mask-RCNN. In our current implementation, the Procedural Memory is never updated except for fine-tuning the model parameters, while it's interesting to design learning mechanism for it in (Wang et al., 2023a) as well.
A.3 COMMUNICATION MODULE
It's important for cooperative embodied agents to be able to communicate effectively with others. Effective communication needs to solve two problems: what send and when send.
We deal with the what to send problem this module by directly using the LLMs as a Message Generator with designed prompts, constructed from the components of Instruction Head, Goal Description, States Description, Action History, and Dialogue History. To better constrain LLMs' generated messages, also add note at the end of the prompt and append two seed messages at the beginning of the Dialogue History to elicit deserved effective communication behavior. The detailed prompt design is shown below:
Instruction Head This part of the prompts is fixed for an environment, mainly consisting of the task instructions and environmental constraints.
Goal Description For each task, the goal description is converted from G {991, 92. ... gk using a formal template.
State Description
For each step, the state description is converted from task progress, state of self, state of others, and semantic map retrieved from the Memory Module through a template.
Action History The concatenation of the last actions (high-level plans) the agent took.
Dialogue History The Concatenation of the last D dialogues between agents including the messages the agent itself has sent.
To constrain the message generation of the LLMs, add note the end of the prompt:
Note: The generated message should be accurate, helpful, and brief. Do not generate repetitive messages.
And append two seed messages at the beginning of the Dialogue History to elicit deserved effective communication behavior:
Alice:
"Hi, I'll let you know if find any goal objects, finish any subgoals, and ask for your help when necessary."
17

---

Published as a conference paper at ICLR 2024
Bob: "Thanks! I'll let you know if I find any goal objects, finish any subgoals, and ask for your help when necessary."
A.4 PLANNING MODULE
CoELA needs strong Planning Module make decisions on which action to take utilizing all available information gathered and stored so far to maximize cooperation efficiency.
While designing such a module from scratch consumes large human expert efforts and is nearly impossible to generalize, we utilize powerful I. L.MSs directly as the Planning Module by first retrieving the related information from the Memory Module and converting them into text descriptions as in the Communication Module, then compile Action List of all available high-level plans proposed according to the current state and the procedural knowledge stored for the LLMs to make the choice, which formalization makes it easier for the LLMs to concentrate on the reasoning and make an executable plan without any few-shot demonstrations easily, finally prompting the LLMs with current information and the proposed Action List to generate high-level plan. We also use the zero-shot chain-of-thouglle prompting technique introduced by Kojima et al. (2022) to encourage the LLMS to earry out more reasoning before giving the final answer.
Action List We compile all available actions regarding the current state into an Action List for the LLMs to select from. The multi-choice formalization makes it easier for the LLM to make an executable plan without any few-shot demonstrations. All available high-level plans on the TDW-MAT include
go to room *
explore current room
go grasp target object/container
put holding objects into the holding container
transport holding objects to the bed
send message: "*"
Answer Extraction As shown in (Wei et al., 2022), chain-of-thougle prompting can unleash the strong reasoning ability of the LLMs, use the zero-shot chain-of-tie chain-of-thought prompting technique introduced by (Kojima et al., 2022) to encourage the LLM to carry out more reasoning before giving the final answer.
A.5 EXECUTION MODULE
To enable effective and generalized cooperation decision- decision-making in different environments, we design an Execution Module generate primitive actions execute a given high-level plan robustly in specific environment, allowing the Planning Module to be generalizable and focus more solving overall task with LLMs" rich world knowledge and strong reasoning ability. Practically, this design can also reduce the LLM inference time and is time-saving and economical. When facing new environment with a different action space, only the procedural knowledge needs to be rewritten A-star-basse planner to find the
for CoELA to work. For rearrangement tasks, we mainly shortest path for navigation and robustly interact with the objects according to rules.
A.6 A WORKING EXAMPLE ON TDW-MAT
To better understand our method, we present A working example of CoELA on one step in the TDW-MAT in Figure 7. CoELA receives observation of 512x512 first-person view RGB image and Depth image from the environment, first uses the Perception Module implemented with Mask- RCNN to predict an instance segmentation mask. then builds 3D point clouds and extracts the states (positions, names, IDs, objects holding if agents) of the key objects including target objects, containers, and the agents, and builds local occupancy map. The Memory Module uses the extracted states of the key objects and the local occupancy map to construct and update the semantic map, which is stored in Semantic Memory. The Memory Module also stores the task progress, the states
18

---

Published as a conference paper at ICLR 2024
Extracted Information
Environment
capple>(136), capple>(63B), -18] Bob, 1, [-3.6, 0.0, 4.11 holding <bread=(533), corange- Alice,
TOW Simulator
Perception Module
-1.5), holding
Observation
Semantic Map
Memory Module
Move forward 0.5m
Low-Level Action
the Livingroom bread and I've also spotted two apples here. Can you search the Office and Kitchen for the remaining objects?"
Execution Module
explore room
Planning Module
Communication Module
<iivingroom> (4000)
High-Level lan
Message
Planning Module
Communication Module
Instruction Head
Instruction Head
Goal Description
State Description
Goal Description
Dialogue History
Action History
Description
Prograss: two taken 133/3000 staps, I'm holding two target objects <534od-((345043) and sor (orange-1388760) I'm
Action List
cattproommaaoos where 've esplored par of fourd target oljects cappleo 2 -oo--) eapples/ess83/2 (0155/2) 522). Last time haw Ajice was in the colfiael (S000), she was helding nothing, I've esplorad nons of the seingroono-/200 none of the checicoms 2000) and fourd the gosl positios bed there I've exploeed none of the coffinnee 10n0). I've explored none of the -kicchnn-(5000) I've esslored none the süvingroome (6000).
A transport objezts I'm hoiding to the hed
C. to (ledrooom//2000))
& to cilingnoomx(109)
I've soplared
=00fcc=33000
(00tchees(t000)
ceplore curonnd room <LXingroom--4000)
Sond message "el Allic, Lifngroomm...
Action History
Answer Extraction
Dialogue History
Figure 7: A working example on the TDW-MAT. The environment provides an observation of 512 512 first-person view RGB image and Depth image. The Perception Module takes these in, builds 3D point clouds, then extracts the states (positions, names, IDs, objects holding if agents) of the key objects including target objects, containers, and the agents, and builds a local occupancy map. The Memory Module uses the extracted states of the key objects and the local occupancy map to construct and update the semantic map. which stored in Semantic Memory. The Memory Module also stores the task progress, the states of the agents in the Semantic memory, and the agent's 's action and dialogue history in the Episodic Memory, which are also updated when a message received. The Communication Module converts the semantic map, task progress, and agents' states into textual State Description and concatenates it with the Instruction Head, Goal Description, Action History, and Dialogue History the prompt condition the L.M current states and generate the message to be sent beforehand. The Planning Module similarly takes these inputs and converts them into a prompt with the addition of an Action List compiled with all available high-level plans including sending the message just generated, then taking advantage of the chain-of-thought prompting to decide the high-level plan. The Execution Module first an A-Star-based planner find the shortest path from the current location to the target location with the help of the semantic map if needed, then carry out the interaction required to finish the high-level plan.
19

---

Published as a conference paper at ICLR 2024
of the agents in the Semantic memory, and the agent's action and dialogue history in the Episodic Memory, which are also updated when a message is received. The Communication Module converts the semantic map, task progress, and agents' states into textual State Description and concatenates it with the Instruction Head, Goal Description, Action History, and Dialogue History as the prompt to condition the LLM current states and generate the message to be sent beforehand. The Planning Module similarly takes these inputs and converts them into a prompt with the addition of an Action List compiled with all available high-level plans including sending the message just generated, then taking advantage of the bhain-of-thoogee prompting to decide on the high-level plan "explore current room <Livingroom> (4000).. The Execution Module then uses an A-Star-based planner find the shortest path from the current location to the target location with the help of the semantic map and gives the low-level primitive action of "Move forward 0.5m", which carried out in the environment and the new observation will be sent to the agents again.
B
ADDITIONAL DETAILS ON ENVIRONMENTS
B.1 THREEDWORLD MULTI-AGENT TRANSPORT
Scenes
Task A: food
Target Objects
Containers
Training Scenes
Task B: stuff
Target Ohjects
Conlainers
Test Scenes
Figure TDW-MAT scenes, target objects, and containers.
As extension of the ThrleDWorld Transport Challenge(Gan et al., 2021), ThreeDWorld Multi- Agent Transport (TDW-MAT) supports multi-agent cooperation with natural language communication and includes more types of objects with more realistic placements. In the challenge, the latest replicant humanoid provided by the TDW platform as an embodiment.
Tasks Two tasks are available in TDW-MAT: w.w.-MAT:ffoooooooo food-transporiing task and stuff-transporting task. The tasks have different types of target objects and containers. Figure shows an overview of the two tasks: We create 4 floorplans and each of them has 3 layouts, where two floorplans are for the training set and another
are for the test set. The food-transporting task has 6 types of targets (apple, banana, orange, bread, loaf bread, and burger) and 3 containers (bowl, plate, and tea tray). In contrast, the stuff- cuaffftransporting task has different types of targets(calculator, mouse, pen, lighter, purse, and iPhone) and 3 containers (plastic basket, wood basket, and wicker basket). In each task, there are target objects and 2 to 5 containers in total. Additionally, there are types of rooms: living room, office, kitchen, and bedroom, and objects placed in these rooms consistent with common sense. For example, food is more likely to be found in kitchens, while stuff is often in offices.
The agents are tasked to transport as many target objects as possible to the goal position with the help of containers tools. One container carry most three objects, and without containers, the agent can transport only two objects at a time. Agents need to transport target objects as much as possible within 3000 frames.
20

---

Published as a conference paper at ICLR 2024
RGB
Depth
Onccle Perception
Figure 9: The RGB, depth, and oracle perception generated from the TDW-MAT environment.
The embodied agent receives the egocentric RGB image and depth image as the main observation, well as some auxiliary observations. Figure 9 is example of an image generated from the TDW-MAT environment, and the detailed observation space is listed here:
Observation Space
RGB image: the egocentric image comes from the camera facing forward, with screen size 512 2 x 512 and field of view 90;
Depth image: the depth image has the same camera intrinsic parameters as the RGB image;
Oracle Perception (optional): image where each object id is mapped color and the camera intrinsic parameters are the same as the RGB image;
Agent position and rotation: the agent's position and rotation in the simulation world;
Messages: the messages sent by all the agents;
Action Space TDW-MAT, there 7 types of actions for agents interact with the environment
or communicate with each other. Each action takes several frames and the detailed action space is listed here:
Move forward: move forward 0.5m;
Turn left: turn left by 15 degrees;
Turn right: turn right by 15 degrees;
Grasp: grasp an object, only the agent is close to the object can he perform the action successfully. The object can be either a target or container;
Put In: put the target into the container, only the agent is holding target in hand and container in another hand can he perform the action.
Drop: drop the objects held in hand;
Send message: Send message to other agents. In each frame, no more than 500 characters can be sent.
B.2 COMMUNICATIVE WATCH-AND-RR HELP
Communicative Watch-And-Hae Hnd- (C-WAH) is extension of the Watch-And-HH Mhallenge(Puig et al., 2021), which enables agents to send messages to each other. Sending messages, alongside other actions, takes timestep and has upper limit message length.
Tasks Five types tasks are available in C-WAH, named Prepare afternoon tea, Wash dishes, Prepare meal, Put groceries, and Set up dinner table. These tasks include range of housework, and each task contains a few subgoals, which are described by predicates. predicate is in "ON/N(x; y)" format, that is, "Put x ON/N y". The detailed descriptions of tasks are listed in Table 3.
The task goal is to satisfy all the given subgoals within 250 time steps, and the number of subgoals in each task ranges from 3 to 5.
21

---

Published as a conference paper at ICLR 2024
Task Name
Predicate Set
Prepare afternoon tea
ONgpudding.coffeetabll)))
ON(apple.coffeetabll ONfuuic,.cfffetabbl)) ON(wine.coffeetable)
Wash dishes
[Nfplate,disswwahhee)), (N(fork.dishwasher)
Prepare meal
ON(pancake dinneetaall,, ONppoundcake,dinneen
cripudding.iinnertabbee (N(apple,dimnertabbl
DNjuuce.,iinerttae o(wwine,dimnertabll
Put groceries
[N(uupcaee.fride
srreancake,frrddge
NN(poundcake, (frpunddcak,,riidge) (rfuudding.ridgeee) fridge) frjuuec,,riddge, IN(wiee.fridge)
Set up dinner table
ON(plate.dinnnetaable) ON(fork,dinnertable))
Table 3: Task description in C-WAH. There are 5 types of tasks and each of them contains a few predicates.
Observation Space C-WAH has two observation modes, named Symbolic Observation and Visual Observation. For Symbolic Observation, we followed the setting of the original Watch-And- Help challenge, one agent can receive all the object information in the same room as the agent, and the information includes location, status, name, relationship. etc.
For Visual Observation, agents can receive the egocentric RGB image and depth image, as well as some auxiliary observations. The detailed observation space is listed here:
RGB image: the egocentric image comes from the camera facing forward, with screen size 256 x 512 and field of view 60;
Depth image: the depth image has the same camera intrinsic parameters as the RGB image;
Oracle Perception: it is image where each object id is mapped to color and the camera intrinsic parameters are the same as the RGB image;
Agent position: the agent's position in the simulation world;
Messages: the messages sent by all the agents.
Action Space The action space similar to that in the original Watch-And-He Help Challenge, with a new action sending message added. The detailed action space is listed here:
Walk towards:
object in the same with the agents room;
Turn left: turn left by 30 degrees;
Turn right: turn right by 30 degrees;
Grasp: grasp object, only the agent is close the object he perform the action successfully;
Open:
Open closed container, only the agent is close the container can he perform the action successfully;
Close: Close an open container, only the agent is close to the container can he perform the action successfully:
Put: Put the held objects into an open container or onto surface, only the agent is close to the target position can he perform the action successfully;
Send message: Send message to other agents. no more than 500 characters can be sent at time.
22

---

Published as a conference paper at ICLR 2024
C ADDITIONAL DETAILS ON EXPERIMENTS
C.I TRAINING DETAILS ON THE AGLNI-AGENT TRANSFORMERS
Multi-Agent Transformer(MAT) We adopt Multi-Agent- (ransformmeMM(MAAT (Wen et al., 2022), which regards MARL as sequence modeling problem and applies centralized decision transformer to generate actions.
The input of MAT contains two parts, the first part is top-down semantic map with size (12, 24) from the oracle perception. The map has 9 channels, implying whether the place is a free space/obsta- cle/wall/unexplored space/target object location/contaiin container location/goal location/my location/another agent's 's location, and the second part is the agent information( whether holds a container, holding object counts, etc.). The output of MAT is one of the following actions: explore, navigate to the nearest target object, navigate to the nearest container, and navigate to the goal place. Each action will last for up to 50 frames or the action is finished.
We train our RL agents for 2c5 frames with the hidden layer dim 64, learning rate 7e 4, ppo epoch 10 on training sets. After training, we test the RL agent on the test sets.
C.2 ADDITIONAL DETAILS ON OTHER BASELINES
Rule-based Hierarchical Planner (RHP) We adopt the strong performing baseline from the original challenge, which is Rule-based Hierarchical Planner with Frontier Exploration strategy, consisting of rule-based hige-level planner that selects one of the high-level plans from Exploration, Pick up an object, Pick up container, and Place according to some human-defined rules and an A-star based planner to navigate with occupancy map and semantic map obtain and updated from the visual observation. The Frontier exploration strategy randomly samples a way-point from an unexplored area as a sub-goal for exploration.
MCTS-based Hierarchical Planner (MHP) We adopt the strongest baseline from the original Batch-And-Heip Challenge, which is a Hierarchical Planner with a high-level planner based on MCTS and low-level planner based on regression planning (RP). MHP infers the other's intention and adapts subgoal accordingly based on the observation of the other agent.
C.3 ADDITIONAL DETAILS ON CoLLAMA
We collected 2k trajectories from 10 episodes in the training set of TDW-MAT with GPT-4 driven CoELA and manually filtered 572 high-quality data with effective communication behavior and good reasoning trace towards collaborative decision-making. We use LoRA to fine-tune the LLAMA-22 13b-chat with a batch size of 384, a maximal sequence length of 2048, and a max learning rate of - for 30 epochs (approximately 60 steps).
C.4 ADDITIONAL. QUALITATIVE ANALYSIS OF THE AGENT BEHAVIORS
CoELA exhibit efficient communication and effective cooperation behavior To better understand the essential factors for effective cooperation, we conduct qualitative analysis of the agents' behaviors exhibited in our experiments and identified several cooperative behaviors, as shown in Figure 3.
CoELA shares progress and information with others. As shown in Figure 3abde, CoELA commu- nicate with each other share progress and intents, demonstrating the Communication Module can handle the challenge of what to send, harnessing the free dialogue generation ability from the LLMs.
CoELA knows when to request help and can respond to others' requests. In Figure 3d, Bob finds a target object in the living room but his container is already full, so he shares this information and requests Alice to come here to help. Alice responds by going there and grabbing the objects. Similarly in Figure 3b, Alice responds to Bob's requests and questions. These examples show CoELA know when request help and can understand others' requests and responses.
23

---

Published as a conference paper at ICLR 2024
cupcake and juice bedroo, you can search other romm noor
Great, in the kitches the cobizets and applionces here for the wine. spple, and pountkakze. Let me know when you've the expcake and jiice the kitchen table.
have put them the kitchen table. And search livingsson bottroom.
Figure 10: A qualitative example in Human + CoFLA experiments, showcasing CoELA can commu- nicate with Humans well and end up with perfect division of the exploration trajectory.
CoELA adapt plans considering others. In Figure 3a, Bob suggests labor division of himself going to the kitchen while Alice checks the other rooms, but Alice suggests a better plan given her circumstances that she's already in the kitchen which Bob aware of before, and finally, Bob adapts his plan to cooperate with her.
CoELA know when not to communicate. In Figure 3c, though Bob receives Alice's suggestion of sharing any progress and has just found a plate, it's more efficient for him to grab the objects by himself and get the job done since this is the last goal object. He successfully reasons about this and chooses not to communicate to achieve higher efficiency. We also observed this behavior from humans when conducting the same task.
C.5 ADDITIONAL DETAILS ON THE HUMAN EXPERIMENTS
We show an effective communication example in Figure 10, where the human first shares his progress with CoELA and suggests a labor division, CoELA understands and responds with its future plan as well, resulting in perfect division of the exploration trajectory. These results imply promising futures for leveraging LLMs to build cooperative embodied agents that can successfully work with humans.
D ADDITIONAL DISCUSSIONS
CoELA is prone to cooperation Communication doesn't 't ensure consensus, and arguing back and forth can consume significant time, resulting reduced efficiency. Interestingly though understand- able, we did not observe such phenomenon during our experiments. CoELA is prone cooperation and coordinate plans without arguing back and forth which may be credited to LLMs trained follow instructions and trust their cooperators. This behavior is beneficial for cooperation, though it may lead to less efficiency when the cooperator is malicious.
Language Agents for Embodied Planning With the recent advance of Large Language Models, there has been work emerging leverage LLMs to build powerful Embodied Agents. Huang et al. (2022a) used GPT-3 to generate high-level plans directly in non ion-inteeeettve way and used another smaller Language Model translate the plan to available actions on virtualhome. Liang al. (2022); Song et al. (2022) used codes or few-shot prompting to directly generate plans, Huang et al. (2022b) built an inner monologue with environment feedback to improve planning, Ahn et al. (2022) combined robotic affordances and LLMs for grounded instruction following. More recently, Park et al. (2023) built an agent society using LLMs augmented with memories in sandbox environment to simulate human behavior. In contrast to the above, our work addresses more challenging multi- agent cooperation problem, characterized by decentralized control, complex observations, costly communication, and long-horizon multi-objective tasks.
24

---

Published as a conference paper at ICLR 2024
E EXAMPLE PROMPTS
We show an example prompt for the Planning Module on C-WAH in Table 4, and an example prompt for the Planning Module TDW-MAT in Table 6.
Table 4: Example prompt for the Reasoning Module on C-WAH
C-WAH Prompts
I'm Alice. I'm hurry to finish the housework with my friend Bob together. Given our shared goal, dialogue history, and my progress and previcus actions, please help me choose the best available acticn to achieve the geal that I can held two objects
possible. Note time and there are no costs for holding objects. All objects are denoted as <name> (id), auch as <table> (712).
Goal: Find and put wine, pancake, poundcake, 1 juice, apple onto the <kitchentable> (130).
Progress: I'm helding nothing. I'm in the <kitchen>, where I found <kitchentable> (130), <apple> (396), and unchecked containers <kitchencabinet> (137), <kitchencabinet> (138), <kitchencabinet> (139), <kitchencabinet> (140), <steve> (157), <dishwasher> (159), <fridge> (165), <fridge> (166), <microwave> (167). I also see Bob here in the <kitchen>, he holding <apple> (387). found an unchecked container <bathroomcabinet> (25) in the bathroom. The livingroom is unexplored. The bedroer. is unexplored.
Dialogue history:
find any goal objects and finish any subgcals, and ask for your help when necessazy."
Alice: "Hi, I'll 1et you know
Bob: "Thanks! 1'11 1et you know if I find any goal objects and finish any subgeals, and ask for your help when necessary."
Alice: "Bob, I'm in the kitchen with the <kitchentable> (130) and found <apple> (386). Please check the living room and bedroom for the other items. I will search the kitchen cabinets and appliances."
Bob: "Alice, I found <apple> (387) and I'm in the <livingroom>, Can you search the kitchen for the other items? I'll check the bathroom and bedroom next."
Previous actions: [gocxplore] <bathroom> (11), [goexplore] <kitchen> (56), [send_ (send__essage), [gecheck] <kitchencabinet> (133), [gocheck] <kitchencabinet> (134), [gacheck] <ki <kitchencabinet> (135), [gocheck] <kitchencabinet: (136) Available actions: A. Isend messagel <"Bob, I see you have <apple> (397). Please put it the <kitchentable> (130) and check the bedroom while search the remaining kitchen cabinets and appliances for the other items.
B. [goexplore] <livingroom> (198)
C. [goexplore] <bedroom> (294)
D. [gocheck] <kitchencabinet> (137)
E. [gocheck] <kitchencabinett (138)
F. [gocheck] <kitchencabinet> (139)
G. [gocheck] <kitchencabinets (140)
Н. [gocheck] <stove> (157)
[gocheck] <dishwasher> (159)
J. [gocheck] <fridge> (165)
K. [gocheck] <fridge> (166)
L. [gocheck] <nicrowave> (167)
M. [gocheck] <tathroomeabinet> (25)
N. [gograb] <apple> (386)
Answer: Let's think step by step.
25

---

Published as a conference paper at ICLR 2024
LLM outputs
First, we know that Bob is already holding an <apple> (387) and in the <livingroom>, We also know that we need to find and put 1 wine, 1 pancake, 1 poundcake, <kitchentable> (130) Since we are already in the <kitchen> and there several unchecked containers, the best course of action is to continue searching the kitchen cabinets and appliances for the remaining itens.
juice, and 1 apple onto the
So, the pest available action for Alice is:
D. [gocheck] <kitcheneabinet> (137)
26

---

Published as a conference paper at ICLR 2024
Table Example prompt for the Reasoning Module on TDW-MAT
TDW-MAT Prompts
I'm Alice. My friend Bob and want transport many target objects as possible to the bed with the help containers within 3000 steps. I can hold two things at a time, and they can be objects or containers. I can grasp containers and put objects into them to hold more objects at time. Given our shared goal, dialogue history, my progress, and previous actions, please help me choose the best available action to achieve the goal as soon as possible. Note that a container can contain three objects, and will be Lost once transported to the bed. I can only put objects into the container
hold after grasping it. Al1 objects are denoted as <name> (id), such <table> (712) Actions take several steps to finish. It may be costly to go to another room or transport to the bed, use these actions sparingly.
Goal: Transpert 3 pens, 1 lighter, 3 ipods, 2 purses, 1 key to the bed.
Progress: I've taken 1313/3000 steps. we've already transported <koy> (3207585), <purse> (15433283), <ipod> (6544816), <purse> (11543537), <pen> (12835254) to the bed. I'm holding nothing. I'm in the <Bedreem> (2000), where I've explored all of it and found Bob
the goal positien bed. Last time (3000), he was holding nothing. I've explored part of the <ivinngoomm> (1000), and I found a container wicker_basket> (13929244) there. I' explored none of the <Oftice> (3000), and I found target objects <ipod> (2031829), <ipod> (16421571) there. I've explored part the <livingroom>> (4000) I've explored none of the <Kitchen> (5000). I've explored all of the <Livingroom> (6000), and I found a target object <pen> (1440084) there. Dialogue history:
in the <Office>
Alice: "Hi, I'll let you know if find any target objects and containers, finish any subgoals, and ask for your help when necessary."
Bob:
"Thanks! I'll let you know if I find any target objects and containers, finish any subgoals, and ask for your help when necessary."
Alice: "Bob, I suggest you transport the basket with key and purse to the bed in the Hedroom (2000). Meanwhile, 1'11 go to the Livingroom (6000)
grab the pen and the containes from collect
Livingroom (1000) afterward plan further."
itens. Let's meet the bed
Bob: "Alice, I'm heading to the Bedroon. to transport the basket with key and purse. After you grab the pen and container, let's meet at the bed to plan any other target objects."
steps. Let know if you find
Bob: "Alice, I just transported the key and purse to the bed. Please grab the pen and container you found and head to the bed. Let me know if you find any other target objects or containers your way."
Previous actions: go to <Bedroom> (2000) step 396, send a message at step 584, go grasp container <plastic_pasket> basket>
(15946891) at step 585, go grasp target object <ipod> (6544816) at step 880, put <iped> (6544816) into the container splastic_basker (15946891) at step 889, go grasp target object <purse> (11543537) at step 929, put <purse> (11543537) into the container splkstic_pasket>
basket> (15946891) at step 1033, go grasp target object <pen> (12835254) at step 1057, put <pen> (12835254) into the container pplastic_basket> (15946891) at step 1098, transport objects I'm helding to the bed at step 1131
27

---

Published as a conference paper at ICLR 2024
TDW-MAT Prompts
Available acticns: (You can only choose the action in the list) A. send a message; "Bob, I transported an ipod, purse, and pen to the bed. I found a container <wicker_basked <livingroom> (1000) and target objects <ipod> (2031829) and <ipod> (16421571) in the <Office> (3000) Please collect the ipods and explore the <0ffice> for more objects. I'11 collect the container and pen from the <Livingroom> and meet you at the bed."
(13929244) in the
B. grasp target object <pen> (1440084)
C. go grasp target object <ipod> (2031829)
D. go grasp target object <ipod> (16421571)
E, go grasp container twicker_basken (13929244)
F. to <livingrcom> (1000)
G. to <0ffice> (3000)
Н. go
<Livingreem>> (4000)
I. go to <Kitchen> (5000)
J. go to <Livingreom> (6000)
Answer: Let's think step by step.
LLM outputs
First, we need to collect the remaining target objects. You should go the <Livingroom> (6000) grasp the <pen> (1440084) since it's a target object. So, the best available action is: B, go grasp target object <pen> (1440084)
F TEMPLATE LANGUAGE COMMUNICATION FOR MCTS-BASED HP AGENT
To enable the MCTS- based Hierarchical Planner to also leverage the new communication capability, design three types of template language communication for it.
Progress Sharing An important motivation for communication to share progress with others to update each other about the total progress of the task. So we design the first template communication to share progress.
When the agent carries out the action of put, which implies new subgoal has been achieved by the agent, it will send a message such as:
P successfully put poundcake <383> kitchentable <130>, and they are in kitchen <56>.
When the agent receives such a message, it will process it and extract the sub-goal satisfied, and use it update its inner tracking of the task progress, so avoiding taking an already satisfied sub-goal as a sub-goal again to better cooperate.
Intent Sharing I Another important motivation for communication is share intent with each other so that all the agents can plan coordinately together. So we design template communication to share intent.
When the agent changes its sub-goal (practically, the Monte Carlo Tree Search High-Level Planner gives a new plan), will tell the other agents current sub-goal by sending a message such as:
'T:: 'Now I want to put cutleryfork <369> in dishwasher </04>, and have not found it yet.
When the agent receives such message, will process it and extract the other agents' new sub- sub-goal and update its belief about the others' intents, so it will not choose the same sub-goal -obl with the others to avoid duplicate and improve efficiency.
28

---

Published as a conference paper at ICLR 2024
Belief Sharing B Sharing the scenes the agent just the other agents can help them update their belief of the location of the object as well, and more importantly, this can help agents to build common ground the belief of the objects to better cooperate together. So we also design a template communication to share beliefs.
When entering room, the agent will send all goal objects found or containers newly checked with no findings or target objects in it to others, such as:
"B:: found nothing is inside kitchencabinet <75>. nothing is inside kitchencabinet <76>. nothing is inside dishwasher <104>. nothing is inside cabinet <216>. culleryfork <369>, cutleryfork <370> and plate <373> inside kitchen <11>.'
When the agent receives such a message, it will process and extract the information maintained in the message to update its belief of the location distributions of the objects just as it has been seen by itself.
Also be noticed, the agents may combine these three types of template communication send one combined message at one time instead of multiple messages over several steps to improve efficiency.
29


Reviews:
[{"review_id": "N66UF33vWj", "reviewer": "Reviewer_tquX", "summary": "The manuscript presents a comprehensive study on constructing cooperative embodied agents using Large Language Models (LLMs), aiming to address multi-agent collaboration in decentralized settings with challenges like raw sensory observations, costly communication, and multi-objective tasks. The authors introduce CoELA, a Cooperative Embodied Language Agent, which integrates the LLMs' capabilities with a modular framework encompassing perception, memory, execution, communication, and planning. The system's performance is evaluated in two embodied environments: C-WAH and TDW-MAT, demonstrating its ability to outperform planning-based methods, particularly when driven by GPT-4. The use of natural language for agent communication is highlighted as a significant advantage, fostering trust and effectiveness in human-agent interactions.", "questions": "1. How to evaluate the communication cost? Is there any tradeoff study on communication cost and effectiveness?\n2. How does the system perform in scenarios with an increased number of objects and more complex interactions, and what measures are in place to maintain scalability?\n3. For turning left/right, what will happen if the object in interest is on the back of the agent? How is the visual-related textual input organized? Did the work try to use the oracle information of the whole environment, including objects and relations?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "**Robust Motivation:** The paper is grounded in a strong and compelling motivation to enhance agent collaboration in complex environments. By addressing the need for agents to effectively communicate and plan their actions in a coordinated manner, the authors establish a solid foundation for their research, showcasing a clear understanding of the challenges and opportunities in the field.\n\n**Carefully Designed Pipeline:** The system architecture demonstrates a thoughtful and meticulous design, integrating multiple modules and dual language models to manage both communication and planning. This comprehensive approach ensures that each aspect of the agent's interaction is given due consideration, resulting in a pipeline that is both balanced and well-reasoned. The deliberate inclusion of separate models for different functions reflects the authors' dedication to creating a system that is tailored to meet the specific demands of agent collaboration.\n\n**Thorough Analysis and Discussion:** The paper excels in providing an in-depth analysis and discussion of the results, helping readers to fully grasp the implications and nuances of the study. The authors do not shy away from addressing the limitations of their work, offering a balanced view that adds credibility to their findings. This level of detail ensures that the paper serves not only as a presentation of the proposed framework but also as a valuable resource for future research, encouraging further investigation and innovation in the field of agent collaboration.", "weaknesses": "**Complex Model and System Design:** The architecture of the system is intricate, requiring each agent to manage five different modules and two distinct LLMs for handling communication and planning. This complexity can lead to instability in the LLM's performance, especially when processing lengthy textual inputs describing complicated scenarios. Furthermore, the challenge to maintain scalability becomes apparent as the addition of objects and details can potentially overwhelm the system, limiting its extendability. The intricate design also indirectly contributes to the limited utilization of spatial information and the difficulty in effective reasoning over low-level actions, as these scenarios would require even longer prompts.\n\n**Effectiveness of Communication:** The effectiveness of communication within the system seems to be suboptimal according to the ablation study. Therefore, a natural question is whether there is a possible improvement for the communication module.\nPersonally, I am interested in the issues in Figure 5a. I am curious if Alice misinterprets Bob's actions due to an incorrect perception or what. Will bi-directed communication, such as Alice asking Bob if he has placed the object in the container when she needs to know, plus Bob telling Alice what he is doing at the beginning and the end of one mission, might serve as a straightforward solution to the problems, highlighting the need for a more responsive and interactive communication system.\n\nBy addressing these weaknesses and optimizing the system accordingly, there is potential to enhance the performance and scalability of the framework, leading to more effective agent collaboration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ZldPMFVG9", "reviewer": "Reviewer_2bPF", "summary": "This paper presents an innovative approach to address multi-agent cooperation challenges in decentralized settings with costly communication and raw sensory observations. The authors introduce a modular framework integrating Large Language Models (LLMs), resulting in the Cooperative Embodied Language Agent (CoELA), capable of efficient planning, communication, and cooperation.", "questions": "See in weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a unique integration of Large Language Models within a modular framework for decentralized multi-agent cooperation, addressing practical challenges in varied embodied environments.\n2. Robust empirical support is provided through comprehensive experiments and a user study, showcasing the effectiveness of the approach and its positive impact on human-agent cooperation.\n3. The paper is well-articulated and structured, offering clear insights and setting a strong foundation for future research in multi-agent cooperation with embodied agents.", "weaknesses": "1. This method assumes a skill library that is manually defined for a specific domain, i.e. execution module. However, this limits its applicability in other domains where predefined skill libraries are not available.\n2. The pipeline appears to be quite complex and relies on several hand-defined modules, including perception modules, three memory modules, planning, and execution. Are all of these modules necessary? Conducting an ablation study would provide better understanding.\n3. The experimental design lacks breadth as it only considers one scenario. Including evaluations across multiple scenarios would strengthen the findings.\n4. There is some ambiguity regarding the execution module, memory module, and perception module details. Were they all designed by humans using language-based approaches?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYZhNYWreh", "reviewer": "Reviewer_iNRX", "summary": "This paper proposes a modular framework that integrates the Large Language Models to build Cooperative Embodied Language Agents CoELA, which focuses on the multi-agent setting with decentralized control, complex partial observation, costly communication and multi-objective tasks. Empirical experiments on C-WAH and TDW-MAT show that CoELA can achieve promising cooperative performance. Additional experiments with real humans demonstrate that CoELA can earn more trust and cooperate more effectively with humans.", "questions": "1. How does CoELA determine the valid high-level plans at each state? Is it determined manually?\n2. Have there been previous works using LLM for implementing embodied agents? Can more discussion or even experimental comparisons be made with these works? What are the core contributions and innovations of CoELA compared to them?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This work proposes one feasible approach to building embodied agents with Large Language Models which seems to be sound. This demonstrates the prosiming potential of LLMs to build cooperative embodied agents and inspires future research well.\n2. The experimental setup is relatively comprehensive, including cooperative evaluation  with AI agents and real humans. Besides, the discussion of the experimental results is also very interesting and thorough.\n3. The discussions about failure cases and limitations are appreciated.", "weaknesses": "1. Despite the fact that the structure of the manuscript is organized well, the description of the method is relatively brief, with some details not sufficiently elaborated. For example, the manuscript states that, for the execution module, CoELA will utilize the procedure stored in its Memory Module to execute the high-level plan. However, it is unclear what form these procedures take and how they are obtained for a specific environment.\n2. If I'm not mistaken, given a specific environment, CoELA needs to manually design/list the possible high-level plans, which may be a relatively tedious workload. Similar issues may arise when determining the possible high-level plans on each state and defining the procedure for each plan. This significantly influences the generality of the method and it may be challenging to implement these manual work in complex problems. Besides, listing the possible plans may significantly increase the prompt length, which might influence the performance.\n3. The baselines in the experiments are relatively simple and heuristic. More baselines, especially methods designed for embodied agents, are recommended.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "57vwseDxjZ", "reviewer": "Reviewer_xiUJ", "summary": "This work proposes a framework called Cooperative Embodied Language Agent (CoELA), which explores the potential of Large Language Models (LLMs) for multi-agent communication. The framework is composed of five modules, among which the communication module helps the agent cooperate more effectively than traditional methods. Experiments conducted on TDW-MAT and C-WAH have demonstrated the effectiveness of CoELA when compared to strong planning-based methods, showcasing effective communication. Furthermore, the authors also explore the potential of using open LMs as LLMs, which is an impressive aspect of their work.", "questions": "1. How does the traditional MHP cooperate with MHP or CoELA?\n2. How does CoELA handle the scenario where no consensus is reached? For example, Alice wants Bob to goto A and Bob wants Alice to goto B.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The comprehensive experiments demonstrate that the proposed method can efficiently cooperate with other agents, which is very impressive.\n2. The presentation is clear.", "weaknesses": "1. Details regarding how CoELA cooperates with other agents are lacking. Sections 4 and 5 do not mention the mechanisms for cooperation between agents, such as how one MHP cooperates with another MHP or with CoELA.\n2. Section 5.3.1 is missing the results of CoELA when driven by CoLLAMA.\n3. The authors appear to be focused on exploring the potential of using Large Language Models (LLMs) in the Discrete Execution setting through communication. However, it's important to consider that communication may lead to challenges in certain scenarios, such as agents failing to reach a consensus. A discussion of such scenarios seems to be missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)