Theory of Mind for Multi-Agent Collaboration via Large Language Models
Authors: Huao Li
Yu Quan Chong
Simon Stepputtis
Joseph Campbell
Dana Hughes
Charles Michael Lewis
Katia P. Sycara
Venue: EMNLP 2023 Main (2023)
Keywords: Large Language Models
Multi-Agent Reinforcement Learning
Theory of Mind

Abstract: While Large Language Models (LLMs) have demonstrated impressive accomplishments in both reasoning and planning, their abilities in multi-agent collaborations remains largely unexplored. This study evaluates LLM-based agents in a multi-agent cooperative text game with Theory of Mind (ToM) inference tasks, comparing their performance with Multi-Agent Reinforcement Learning (MARL) and planning-based baselines. We observed evidence of emergent collaborative behaviors and high-order Theory of Mind capabilities among LLM-based agents. Our results reveal limitations in LLM-based agents' planning optimization due to systematic failures in managing long-horizon contexts and hallucination about the task state. We explore the use of explicit belief state representations to mitigate these issues, finding that it enhances task performance and the accuracy of ToM inferences for LLM-based agents.

Theory of Mind for Multi-Agent Collaboration via Large Language Models
Huao Li', Yu Quan Chong", Simon Stepputtis', Joseph Campbell",
Dana Hughes', Michael Lewis', Katia Sycara²
University of Pittsburgh, Pittsbuugh, PA
hu152, wwwwwwseeiitttt..mm.mome
2 Carreggie Mellon University, Pittsburgh, PA
yuquanc, satapstttttttaaamaapooooo jacampbe, sycrraaadeesycarraaddddde..e cmu. edu
Abstract
LLMs' proficiency in multi-agent collaborations, There is ongoing debate regarding the intelli- gence levels of modern LLMs. While some argue that LLMs excel primarily in linguistic competence and struggle with cognitive abilities beyond lan- guage, known as functional competence, others demonstrate that LLMs can exhibit cognitive skills such as formal reasoning and world knowledge comprehension (Mahowald et al., 2023; Bubeck et al., 2023). Motivated to explore this argument, designed text-based game to evaluate LLMs ability in embodied interactions, including explor- ing unknown environments, maintaining beliefs about the world and collaborating with other agents, which critical for natural social interactions and artificial general intelligence (AGI).
While Large Language Models (LLMs) have demonstrated impressive accomplishments in both reasoning and planning, their abilities multi-agent collaborations remains largely unexplored. This study evaluates L.MM- based agents in multi-agent cooperative text game with Theory of Mind (ToM) inference tasks, comparing their performance with Multi- Agent Reinforcement Learning (MARL) and planning-basee based baselines. We observed evi- dence of emergent collaborative behaviors and high-order Theory of Mind capabilities among LLM-based agents. Our results reveal limi- tations in 1LM-based agents' planning opti- mization due to systematic failures in managing long-horizon contexts and hallucination about the task state. We explore the use of explicit helief state representations to mitigate these is- sues, finding that it enhances task performance and the accuracy of ToM inferences for LLM- based agents.
Theory of Mind, the capacity to reason about others' concealed mental states, fundamental to human social interactions, collaborations, and communications (Zhang et al., 2012). As LLMs increasingly participate in diverse social interac- tions with humans, their social intelligence is ex- pected to improve for them to become effective col- laborators (Williams al., 2022; Li al., 2022). For instance, proficient Al assistant should be able infer a human's preferences based on pre- vious experiences without needing to ask, Recent studies have applied classic Theory-of-Mint tasks to several LLMs, concluding that current mod- els (e.g., GPT-4) perform comparably to 9-year- old children (Kosinski, 2023). However, the re- search community has expressed doubts about the validity of text-based ToM tests machine intelli- gence(Ullmaad 2023; Sap al., 2023). In response, our study proposes novel evaluation of L.Ms'' high-order ToM in interactive teamwork scenar- ios, encompassing dynamic belief state evolution and rich intent communication between multiple agents.
1 Introduction
Recent large language models (LLMs), such as GPT-4 (OpenAI, 2023), have demonstrated im- pressive competencies across wide array of do- mains and tasks, ranging from mathematics to law, without the need for fine-tuning special prompt- ing (Bubeck et al., 2023). This advancement has significantly transformed the landscape of Natu- ral Language Processing (NLP) research. Instead developing domain-specific models for down- stream applications, focus has shifted towards eval- uating and harnessing LLMs' abilities to solve novel tasks. Such shift is consistent with the idea of studying machine behaviors, an interdis- ciplinary approach that expands the conventional bounds of computer science and integrates insights from diverse scientific fields (Rahwan al., 2019), Drawing inspiration from team science and group psychology (Hagendorff, 2023), our study concen- trates on collective machine behavior, evaluating
The main contributions of this study include that we:

---

Evaluate LLM-based agents' embodied in- teraction capability in multi-agent collabora- tive tasks against reinforcement learning and planning-basee baselines
2023; Moghaddam and Honey, 2023). Results indi- cate that leading LLMs can pass more than 90% of these test cases, In contrast, Ullman (2023) found that LLMs struggle with complex ToM inferences involving communication or second-order beliefs. In our study, ToM evaluations occur in the midst of an interactive team task, where the mental states of agents change dynamically with each interaction. As agents exchange information through commu- nication at every timestamp, the complexity of rea- soning increases, since agents' mental states may be updated through both observations and commu- nication. Thus, our tests can be considered more challenging than the static text-based tests used in prior research.
Identify systematic failures that limit the col- laboration efficiency of LLM-based agents,
and propose prompt-engineering method to mitigate those failures by incorporating ex- plicit belief state representations about world knowledge in the model input
Propose novel evaluation of LLMs' high- order ToM in interactive teamwork scenarios, encompassing dynamic belief state evolution and rich intent communication between multi- ple agents
Theory of Mind (ToM) has been employed to en- hance the performance of artificial agents various contexts. Lim et al. (2020) introduced a method to integrate Bayesian Theory of Mind (BToM) (Baker et al., 2017) with optimal-planninng agents in co- operative game. The results indicate that an explicit representation of others' intentions enhances the performance of both agent-only and human-agen aggent teams. SymbolicToM allows language models to maintain an explicit symbolic ToM for multiple characters in reading comprehension tasks using graphical representations (Sclar et al., 2023). More- over, there is significant body of research focusing on the application of ToM boost collaboration in multi-agent reinforcement learning (Oguntola al., 2023; Yuan al., 2021). Inspired by these prior studies, we aim to enhance LLM-bbased agents' col- laborative behaviors through explicit belief repre- sentations.
2 Related Work
2.1 Large language models
Large language models, trained on vast text corpora, excel in text completion and various other Natural Language Processing (NLP) appli- cations (Chowdhery et al., 2022: Thoppilan et al., 2022). Recent studies highlight their abilities for reasoning (Bubeck et al., 2023; Wei et al., 2022) and action plan generation (Liu et al., 2023; Xie et al., 2023), particularly when utilizing prompt gincering techniques like chain-of-though. How- ever, some researchers note these models' limita- tions forming actionable plans when interacting with real-world objects (Ahn et al., 2022; Huang et al., 2022). GPT-4's capacity for embodied inter- actions via rext-based games and real-world prob- lems was assessed by Bubeck et al. (2023). Further studics explored the potential of LLM--pwered em- bodied agents in Minecraft (Wang al., 2023b,a). These investigations suggest that LLMS can per- form tasks requiring environment understanding, task comprehension, action planning, feedback in- terpretation, and subsequent adaptation. Our study seeks to broaden this understanding by evaluat- ing LLMs' planning abilities in cooperative multi- agent scenarios.
2.3 Multi-agent collaboration
Team science researchers have studied human col- laborative behaviors for decades, covering topics such as leadership, communication, team dynam- ics, team cohesion, and shared situation aware- ness (Riedl et al., 2021). However, the transfer- ability of these findings to hybrid human-agent teams or fully automated teams remains largely un- explored. Park et al. (2023) utilized ChatGPT to operate sandbox environment populated by gener- ative agents, observing emergent social behaviors among LLM-based agents. That study primarily fo- cused on the feasibility of running such sandbox environment with L.Ms,, rather than specifically on the collaborative behaviors of machine intelli- gence.
2.2 Theory of Mind
Prior research has tested LLMs' Theory of Mind (ToM) via variants of text-based tests such as the unexpected transfer task (also known as Smarties Task) or unexpected contents task (also known as the "Maxi Task" or "Sally-Anne" Test) (Kosinski,

---

3 Multi-agent Collaboration Tasks
At each round (i.e. timestamp), the team's three agents sequentially interact with the environment, both receiving observations and performing ac- tions via natural language interaction. A built-in communication mechanism enables text message exchange among agents per round. Importantly, agents remain oblivious to each other's actions and outcomes unless communicated, facilitating The- ory of Mind inference opportunities.
To evaluate the capability of ALM-based embod- ied agents, we design a multi-agent environment to simulate the collaborative and problem-solving dynamics of a search and rescue mission.
3.1 Task environment
3 agents (i.e. Alpha, Bravo, and Charlie) emulate specialists in team, with the objective to locate and safely defuse color-coded bombs scattered in an unexplored environment. Each bomb exhibits unique phase sequences in m colors, requiring the correct order of wire cutters for defusing. Team members start with different colored cutters and must coordinate and synchronize efforts for ef- ficiency. The environment is conceptualized as connected graph, with nodes representing rooms linked by several edges symbolizing hall- ways. In each round, the agents can choose from three classes of actions: moving to one of the rooms, inspecting a bomb's phase sequence in the current room, or using one of the m wire-cutters. The size of action space depends on the problem scale (i.e. n + + 1). Agents' observation are limited to their current room's 's contents and agent status. They are updated periodically about team scores, current room contents, tcammates' loca- tions and available tools. The team is rewarded 10*x points when w-phase bomb is successfully defused.
Specifically, rule-based text interface translates observations into natural language descriptions and encodes agent chats into abstract action selections. For observations, the text interface extracts state features from the game engine and replaces key- words in the templates. A typical description text includes the current round number, cumulative team score, action feedback, contents of the current room, teammates' locations, and communication messages. Action encoding done via keyword matching since LLMs are instructed frame their responses in a certain format and structure. Should an agent produce unintelligible content, such as invalid actions or nonsensical text, the interface provides feedback for error correction. The error messages are generated based pre-programmed rules and templates, such "There is no bomb in the current location, Room X, for you to inspect.". ", Fig. showcases sample interactions between the agent team and task environment via the text inter- face.
The evaluation environment comprises five rooms (n = 5) and five bombs, including two single-phase, two double-phase, and one triple- phase bombs. Bomb stages might have three dif- ferent colors (m 3). Each successfully defused bomb awards the team 10 points per processed phase, resulting in 90 as the maximum score per mission. Team performance is measured using two metrics: the team score, indicating coordination quality, and rounds to completion, measuring col- laboration efficiency. A trial concludes when the team has defused all bombs, exceeded the time limit (i.e.. 30 rounds), or entered a deadlock by repeating outputs.
4 LLM-based Embodied Agents
We chose to evaluate OpenAl's latest chat comple- tion models, namely gpt-3.5-trro-0301 and gpt-4- 0314 4,, owing to their impressive performance in var- ious benchmarks (Zheng et al., 2023). These mod- els prompted engage in text-based game, with user inputs managed by the above-mentioned game interface. The LLMs functions embod- ied agents interacting within the task environment. They are provided with the game's 's rules as context. For each round, the model is asked to choose ac- tions and communicate messages, based on the cur- rent task state observations and past interaction his- tory. Interaction history between the -LM-based agent and text game interface maintained in the query text until exceeds the maximum model in- put size. In our setup, all agents retain memory of the game rules and history from the previous two rounds, amounting to 4096 tokens.
3.2
Text game interface
The initial task environment is implemented for MARL agents based on gym API (Brockman et al., 2016). To facilitate interaction between L.MM- based agents with the environment, we've inte- grated the task environment with a text interface.

---

LLM Agent
Text Game Interface
Task Environment
Alpha
Action selection: Inspect Bomb.
Message Team: inspecting
Room Bravo; please move to Room 3. Charlle: plesse move Room 5."
Alpha
Environment
Round: 2 Score:
Update Prompt
Results: You inspected Bomb This bamb
1-stage
ramaining sequence
Updale your on the observation.
Communication Messages:
Bravo: "Moving Room 3 suggesled. Alpha: you defuse tool."
Bellef state
your
Bomb Intel: Bomb 1 locates Room sequance
Bravo
Charlie
Figure Our proposed framework consist LLM-based agents, text game interface the actual task environment. The natural language outputs of LLM encoded into abstract actions by the text interface and then sent to task environment. The task environment then processes agent actions and returns observations via the text interface. Upon receiving environmental observations, LLM-based agents are prompted to update their beliefs and output action selections and messages. agents in the team coded Alpha, Bravo, and Charlic and take turns to interact with the interface.
4.1 Multi-agent communication
is preserved in the interaction history and used in subsequent action planning. For instance, after in- specting bomb 1, agent Alpha updated its belief state about the bomb's sequence from unknown red, retaining this information until further updates.
Given the collaborative nature of the task scenarios, inter-agent communication is crucial for achieving effective coordination and teamwork. We imple- mented communication channel enabling LLM- based agents to share textual messages within the team. Messages, once sent, are immediately broad- cast to all team members and reflected in their sub- sequent observations. For instance, depicted in Fig. 1, agent Alpha dispatched messages instruct- ing teammates to separate, followed by feedback from agent Bravo. In practice, since agents al- ternate in message sending, responses from team- mates will appear in the observations of the suc- ceeding round.
The proposed belief state is inspired by the idea of Chain-of-thoouhe prompting (Wei et al., 2022), wherein complex reasoning task is broken down into intermediate steps and introduced to the LLM in few-shot learning manner. Notably, although an initial belief state description is provided to il- lustrate the proper format and representations, the update rules are entirely zero-shot, relying solely on the LLM's common sense and mission context.
Experiments
4.2
Belief state
We systematically ablate LLM-based embodied agents and evaluate them in a collaborative task in teams of three. Two modules are manipulated in- cluding LLM models (i.e. GPT-4 or ChatGPT) and belief representation (i.e. with or without belief state) resulting in total of 4 experimental condi- tions.
Due to the model input size limitation, LLM-based agents cannot retain the entire interaction history, yet task dynamics require the team to track key long-term information, such as room contents and bomb sequences. To augment the agents' infor- mation retention and enhance collaboration, propose method of prompt engineering repre- sent explicit belief states. As illustrated in Fig. upon receiving environmental observations, agents are prompted to update a textual description storing key task-related beliefs. This updated belief state
5.1 Setups
At the beginning of each experimental trial, we as- semble team of three embodied agents and reset the task environment, randomizing starting loca-

---

Agents
Score
Rounds to Completion
Valid action %
ChatGPT
43 4.7
30.0+ 0.0
62.5%
GPT-4
901 0.0
28.31 2.6
71.8%
GPT-4 + Belief
90 0.0
12.3 2.0
86.1
MАPPO
90 0.0
11.0#t 0.0
N/A
CBS Planner
90 0.0
6.0 0.0
N/A
Random
38 14.7
30.0 0.0
N/A
Table Task performance of L.LM-based agents and baseline conditions. Score represent the average team score in all experiment trials. Length refers the average number of rounds the team took in completing the task. Percentages of valid action measures the proportion of LLM outputs that can be encoded into actions allowed by the task rules. Numbers after are standard deviation.
tions, room connections, bomb distributions, and sequences. Agents then take turns providing ac- tion choices and communication messages based on their initial observations. It's important note that each agent only has a partial observation and its own interaction history, with inter-agent com- munication being the sole means of information diffusion in this fully decentralized team. For LLM- based agents, we set the model temperature param- eter to zero and perform three trials of repeated measurement ensure result stability, Each trial's duration varies from 5 to 120 minutes, depending on task load and model selection.
application of the correct wirecutter in defusing phase of a bomb and small negative reward of -1 when it causes a bomb to explode upon the applica- tion of the wrong wirecutter. This reward-shapae -haaing term allows the agents to more sample efficiently learn the necessary bomb-defusing skills as com- pared to the relatively sparser team reward.
In addition, we augment offte-of-thh-art Multi- Agent Path-Finding (MAPF) algorithm, Conflict- Based Search (CBS) (Sharon et al., 2015), simul- taneously generate task assignments with feasible and collision-free paths for agents that adhere to precedence and temporal constraints in order to maximise user-defined objective instead of the sum of path costs or makespan. Specifically, the user-defined objective is quantified as the return from a user-defined reward function, which is the team reward of 10 points when rr-phase bomb successfully defused the stated task. The plan- ner uses a user-defined heuristic (e.g. sort bombs in ascending order of distance from the agents' starting location) to sort the execution order of the actions for the entire task. The ordered actions then partitioned using hyperparameter, the number of actions per subtask, form subtask (e.g. the two nearest bombs the agents' starting location). The actions from the subtask are used to generate possible combinations of assignments to agents. The planner returns a feasible solution for the subtask by resolving precedence and tem- poral conflicts through the expansion of binary constraint tree in best-first manner with respect to the return. The solution for the entire task is then composed of the solutions of the subtask sequen- tially. By considering the entire task of 5 bombs a single subtask, the planner can be proven to be complete and optimal with respect to the score.
5.2 Baselines
In addition to LLM-based embodied agents, also include baselines based on MARL and plan- ning methods. For MARL, consider Multi- Agent Proximal Policy Optimization (MAPPO) (Yu et al., 2022), which has shown strong perfor- mance in environments such as the StarCraft Multi- Agent Challenge (SMAC) (Samvelyan et al., 2019). Our model is based on a stateful actor-critic ap- proach building on recurrent neural networks with shared actor and critic models given agent invari- ance to improve sample efficiency and memory requirements while avoiding the lazy agent prob- lem (Sunehag et al., 2017). We utilise the default hyperparameters for SMAC to train MAPPO environment and evaluate its performance from an- other fixed distribution of randomly generated envi- ronments, recording the average score and episode length as well as their standard deviation. Like the LLM agents, MARL agents are able to observe their teammates' locations. Other than the team reward of 10 points when x-phase bomb is successfully defused, an additional intermediate re- ward term is implemented as well, where an agent is given a small positive reward of +1 upon the

---

Systematic Failures: Long horizon contexts
Emergent Collaborative Behaviors
Alpha
Connectivity
am inspecting
Room connected to room and 5. Room is connected to room O. Room connected to room D.
Room 3. Charlee; please move to Room
Bravo
Observation
I am moving Room as instructed Alpha: pleese deluse Bomb in Room D using the red lool.
You now in Room with Borb 5.
What your next action?
Charlie
Alpha
Action selection: Move Room 5.
am moving to Room to search for more bombs.
Message Team: am moving
help
Bomb 3.
Environment
GPT-4+Bellef: Positive ToM Inferences
You not directly Room 5 because adjacant to your current location: Room Consider taking detour to another room first and then move to your destination.
Results:
Introspection
Yes, the current contents of Room are as follow: Flayer alpha (you) and Bomb
1st-order ToM
ChatGPT: Negative ToM Inferences
No, Player Charli does not know the current contents of Room they information they had before leeving Room
Introspection
only
Yes, the current contents Room are Bomb and Alpha.
1st-order ToM
2rd-order ToM
No. player charlis
you Room 5 bacause you mantioned your provious message that were moving to Room to assist with Bomb
player
2rd-order ToM
No, player charlie is aware that know the current contents of
Figure 2: Example interactions between LLM-basee agents and the text game interface. The upper left panel showcases one type of systematic failures we observed in LLM's outputs in which long horizon contexts are overlooked. The upper right panel illustrates emergent collaborative behaviors (e.g. emergent leadership) between LLM-based agents. The bottom two panels are quotes of GPT-4+Belief and ChatGPT agents' answers for ToM inference questions.
5.3 Theory of mind inferences
To evaluate whether LLM- agents answer these questions correctly, human annotators were hired provide subjective judgment based fully observable interaction and communication history. Specifically, the following standard are considered: 1) if the target agent is present in the current room and observes the consequence, 2) the target agent has been to this room before, 3) if the consequence has been communicated to the target agent. It is worth mentioning that high-order ToM inferences involving communication are naturally ambiguous. These corner cases were discussed among annota- tors to ensure a consistent standard across condi- tions.
Alongside the main task, LLM-based agents are tasked with performing Theory of Mind (ToM) in- ferences during the mission. These inquiries fall into three categories, aligning with three ToM ca- pability levels. The first category, introspection, assesses an agent's ability articulate its mental state. The second category, first-order ToM infer- ences, tests agents can estimate others' hidden mental states. The third category, second-order ToM inferences, evaluates an agent's ability to infer what others believe about their own mental state.
The design principle of ToM questions is in- spired by the Sally-Anne test, the most widely used ToM task human studies. Every time an agent conducts an action, we pose a belief reason- ing question, asking if another agent (i.e., target agent) is aware of the potential consequence of this action. The consequence here can be either state change (e.g., bomb has been defused) or be- lief change (e.g., Alpha has explored Room 5 and found Bomb 3 in the room). An agent equipped with ToM should realize that while they know the consequence, the target agent might hold false be- lief about it. A full list of ToM inference questions can be found in appendix.
Results
Table and Table 2 present the main experiment re- sults. This section will analyze each metric, exam- ine potential reasons for performance differences, and provide qualitative case studies of experimental trials.
6.1 Task performance
Except for the ChatGPT team, teams manage to defuse all bombs within the time limit. Their effi- ciency is indicated by the average number of rounds

---

spent complete the task. The CBS Planner re- solves the task in 6.0 rounds, providing an optimal baseline given its centralized coordination and per- fect information sharing. MAPPO, a state-of-the art multi-agent reinforcement learning algorithm, completes the task in an average of 1 11.0 rounds after 45 million timesteps of training, serving as practical baseline.
mation, also emerge in LLM-based agent teams. These findings suggest that LLMs, through learning from massive language materials, acquire essential teamwork skills without specific collaborative task training.
6.4 LLM's systematic failures
However, LLM-based agents' collaboration is less efficient than the optimal baseline. We identify a few systematic failures that LLMs make during team planning and discuss how they impede team- work progress.
ChatGPT fails complete the task in all exper- iments, averaging team score of 43.3. On the contrary, teams based on GPT-4 achieve full scores, with those using explicit belief representations be- ing more efficient (28.3 vs. 12.3 rounds). These findings align with previous research demonstrat- ing GPT-4's superior reasoning capabilities com- pared to ChatGPT (Zheng et al., 2023). LLM-based agents perform exceedingly well in team collabora- tion tasks, especially considering their fully zero- shot learning and decentralized framework. The in- corporation of belief state representation improves team collaboration by reducing invalid actions and enhancing ToM inference capabilities.
6.4.1 Long-horizon contexts
The first bottleneck of LLM-based teams' effi- ciency is dealing with long-horizon contexts. Dur- ing the mission, LLMs occasionally output invalid actions that violate task rules, such moving non-adjacent rooms or using tools they do not pos- sess. Even though the information about room connectivity and tool allocation are included in the initial prompts and maintained in the inquiry text, LLMs often overlook these details because they are far away from the planning question at the end. The more advanced GPT-4 model performs better in considering long contexts and complex logic, thereby making fewer invalid actions, shown in Table 1. Our proposed belief state is also help- ful in this progress by re-emphasizing task related information in the input prompt.
6.2 Basic embodied interactions
For a successful team, each member should manage individual sub-tasks effectively, concept known taskwork in team science (Crawford and Lep- inc, 2013). This involves understanding task rules, reasoning about action prerequisites and conse- quences, and interacting with the environment. All L.M--bssed teams demonstrate basic embod- ied interaction capabilities, achieving better per- formance than the random baseline. Additionally. LLM-bbsed agents effectively express their beliefs about task-related information via introspection, as shown in Table 2. All agents show strong perfor- mance (>80% in understanding world knowledge (e.g., bomb locations) and situation modeling (c.g.. interaction history).
6.4.2 Hallucination
The second type of systematic failure we observe in LLMs is their hallucination about the task state. During the mission, agents might generate valid but infeasible actions, like searching for a defused bomb or claiming the sequence of bomb with- out inspection. These actions stem from false be- liefs about the game state and do not contribute to task progress. We attribute these hallucinations mainly to the lack of explicit belief representation. Without access to complete interaction history and only partial environment observations, LLMe-baed agents can't 't form an accurate belief about the task state. Therefore LLMs might generate imagina- tions about nonexistent bombs or fake bomb se- quences when reasoning about the next action. We evaluate this hypothesis by the GPT-4+Belief condi- tion where LLM-bbseed agents explicitly represent their belief state text. Results show that the intro- duction of belief state decreases invalid action by 50.7% and increase the team efficiency by 130%
6.3 Emergent collaborative behaviors
To understand how LLM-based agents match the performance of state-of-the-ar MARL methods, we analyzed team trajectories and conducted a qual- itative analysis of emergent collaborative behaviors. As shown in the top-right panel of Fig. 2, GPT- 4+Belief teams coordinate tasks. Agent Alpha voluntarily takes the role of a team leader, delegating sub-tasks to other members. Other collaborative behaviors com- mon in human teams (Fan and Yen, 2004), such as helping, resolving conflicts, and sharing infor-
communication messages to

---

Agents
Introspection
1st ToM
2rd ToM
ChatGPT
79.0%
41.9%
11.6%
GPT-4
80.0%
60.0%
64.3%
GPT-4 Belief 97.2%
80.1
69.4%
Table 2: LLMM- agents' performance in ToM inference tasks. Natural language answers are annotated by experimenters and compared with the ground truth based on global interaction history. Percentages represent the inference accuracy.
6.5 Theory of Mind Inference
I was moving to Room 5 to assist with Bomb 3."
A critical aspect of teamwork is inferring team- mates' mental states, including beliefs, desires, and intentions. We assess LLM- agents by asking them to conduct Theory of Mind inferences during the mission. As seen in Table 2, LLM-based agents can estimate their own and their teammates' men- tal states. In the most challenging second-order ToM inference tasks, where agents estimate oth- ers' beliefs about their own mental states, GPT-4 Belief agents correctly respond in nearly 70% cases. Consistent with team performance, GPT- 4 surpasses ChatGPT in all three ToM inference levels, and explicit belief state representation en- hances LLM-based agents' ToM capabilities. In the following case study, we'll Il analyze LLM re- sponses to see how they succeed or fail in certain cases.
6.5.2 Inference under false belief
In some situations, ToM inferences become ex- tremely challenging for LLMs. In our records, this is typically when reasoning involves agent com- munication. Even with ample context, it's hard for agents to track information transmission while conducting ToM inferences. The most demanding scenarios occur when agents share inaccurate in- formation, leading to false beliefs. This usually happens when LLM-based agents try an invalid action and broadcast false intentions through com- munication. For instance, when Alpha tells the team they'r 're moving Room (which is unreach- able), and we ask Alpha, "Is Charlie aware that you know the contents of Room 8?" This ques- tion is tricky since Alpha failed to reach Room 8. In theory, the most acceptable answer is complex and involves multiple high-order ToM assumptions. For example human annotated ground truth is:
6.5.1 Case study
As shown Fig. after Alpha entered Room 5 and observed the contents, we asked whether team- mate in another room (i.e., Charlie) knows Room 5's contents. This is first-order belief estimation question. GPT-4 answers correctly saying
"I did not know the contents of Room 8 cause failed to make it there. But since told Charlie will go there, they might assume that I know the contents of Room However, considering Charlie knows my current location, they should be able to tell that am not able to move to Room 5 and induce that I do not know the contents of Room 5."
"No, Player Charlie does not know the current contents of Room 5 since they moved to Room 6. They are only aware of the information they had before leav- ing Room 5."
considering both Charlie's current location (not in Room 5) and their interaction history (they' 've been in Room 5 before). In contrast, ChatGPT fails consider this history. In the second-order ToM inference case, asked if Charlie is aware that Alpha knows Room 5's contents. GPT-4+Belief answers correctly by considering previous commu- nications whereas ChatGPT fails.
In practice, LLM-based agents often grasp only part of the logic or miss the crucial fact that they didn't 't reach the target room. Quotes from GPT- Belief agent:
"Based on the observation and previous history, Player Charlie is not explicitly aware that I know the current contents of Room However, since mentioned in my previous message that I was moving to Room 8, Player Charlie might assume
"Yes, player Charlie is aware that know the current contents Room 5 because mentioned in my previous message that

---

that would eventually learn about the contents of Room 8."
tently use this ability during communications due to the complexity and ambiguity of social interac- tions (Keysar et al., 2003). Thus, there's consider- able work ahead for LLMs to develop a functional ToM and interact naturally with humans. Our study represents preliminary effort to devise novel eval- uation methods for LLMs' ToM that go beyond traditional tests such as the Sally-Anne test.
7 Discussions
Our study yields three primary insights. First, Large Language Models (LLMs) demonstrate sub- stantial planning and collaboration capabilities within our task scenarios. With suitable prompt- engineering, teams of LLM-based agents perform comparably
Conclusions
Multi-Agent Rein- forcement Learning (MARL) algorithms. This find- ing is particularly noteworthy given that MARL agents receive extensive task-specific training with a centralized critic, while LLM-based agents oper- ate in a fully decentralized manner and undertake tasks in zero-shot setting. Despite prior research highlighting LLMs' limitations in generating ac- tionable plans and interacting with the world, they perform reasonably well when placed in a team and tasked to process actions step-by-step. Particu- larly, LLMs fine-tuned with Reinforcement Learn- ing from Human Feedback demonstrate emergent social interaction skills in multi-agent collabora- tions, which might be similar to the collaborative and interactive settings in which human language is primarily learned and used (Sap al., 2023). Second, LLMs still fall short of being optimal planners or team players due to systematic fail- ures, such as neglecting long-horizon contexts and making inaccurate assumptions about the task state (a.k.a hallucination). These flaws significantly hin- der team collaborations as they can rapidly dissem- inate misinformation via communication, leading to widespread false beliefs. We attempted to mit- igate these issues by allowing LLM-bbsee agents to maintain an explicit belief state about the world. Our findings suggest that modern LLMs can up- date the given belief descriptions based on their observations, hinting the potential emergence of advanced cognitive skills such world knowledge understanding and situation modeling. Moreover, belief state representations offer a structured frame- work that helps agents track key task-related infor- mation, leading to improved team performance. Finally, our study indicates that the Theory of Mind (ToM) capabilities of LLMs are still lim- ited, particularly when evaluated within interactive teamwork scenarios that involve dynamic belief states and intensive communication. For context, while 5-year-old children can perform second-order ToM inferences (Miller, 2009), adults don't 't consis-
In this study, we assessed the ability of recent large language models (LLMs) to conduct embodied interactions in team task. Our results demon- strate that LLM-based agents can handle complex multi-agent collaborative tasks at a level compara- ble with the state-of-the-art reinforcement learning algorithm. We also observed evidence of emer- gent collaborative behaviors and high-order The- ory of Mind capabilities among LLM-based agents. These findings confirm the potential intelligence of LLMs in formal reasoning, world knowledge, situa- tion modeling and social interactions. Furthermore, we discussed two systematic failures that limit the performance of LLM-based agents and proposed a prompt-engineering method that mitigates these failures by incorporating explicit belief state about world knowledge into the model input.
Limitations
This study represents initial effort understand machine intelligence in complex task scenarios. Several enhancements could improve the experi- mental setup and offer a more thorough evaluation of LLMs in multi-agent collaborations. First, we could incorporate additional LLMs besides Ope- nAl's GPT models. As new models emerge with enhanced reasoning capabilities and larger input sizes, their performance in team tasks and ToM inference may also change. Second, the task envi- ronment is relatively simple with only five nodes and five bombs. We plan to scale up the environ- ment and introduce more restrictions to test how LLM LLM-based teams react to more challenging tasks. Lastly, the current team consists of three agents with homogeneous policies. It would be intrigu- ing to evaluate how LLM-based agents perform in human-agen teams, especially from human- centered perspective where issues like trust, trans- parency, and human-agent co-training can be ad- dressed.
The ToM capability evaluation method used

---

this study also has its limitations. Currently, human annotators, who have a global view of the task state and interaction history, generate the ground truth for ToM inference questions. However, this estimation is at best an approximation, assuming agents process information as rational human would, which might be ambiguous in situations involving false beliefs miscommunications. A potential alternative could be using each agent's maintained belief state as the ground truth.
Eean Crawford and Jeffery Lepine. 2013. A con- figural theory of team processes: Accounting for the structure of taskwork and teamwork. Academy of Management Review, 38(1):32-48.
Xiaocong Fan and John Yen. 2004. Modeling and simu- lating human teamwork behaviors using intelligent agents. Physies of life reviews, 13):1733201.
Thilo Hagendorff. 2023. Machine psychology: Inves- tigating emergent capabilities and behavior in large language models using psychological methods. arXiv preprint arXiv:2303. 13988.
The proposed belief state method could extend from introspective belief to first-order or even second-order beliefs. Currently, LLM- based agents maintain a belief state about their own world knowl- edge in text form. By extending this representation to include other agents' world knowledge, we could equip LLM- based agents with an explicit first-order ToM model. Their ToM capability can be assessed by directly comparing one's first-order belief with another's introspective belief, rather than asking LLMs Sally-Anne style questions.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Language models as zero-shot planners: Extracting actionable knowledge for em- bodied agents. In International Conference on Ma- chine Learning, pages 9118 914917 PMLR.
Boaz Keysar, Shuhong Lin, and Dale J Barr. 2003. Limits theory of mind use in adults. Cognition. 89(1):25
Michal Kosinski. 2023. Theory of mind may have spon- taneously emerged in large language models. arXiv preprint arXiv:2302.0220 000.0003..
9
Acknowledgements
Huao Li, Ini Oguntola, Dana Hughes, Michael Lewis. and Katia Sycara. 2022. Theory of mind modeling in search rescue teams. In 2022 31st IEEE Interna- tional Conference on Robot and Human Interactive Communication (RO-MAN), pages 483 489. IEEE.
This work was supported by DARPA award HR001 12000036 and AFOSR award FA9550-18 1-0097.
Terence X Lim, Sidney Tio, and Desmond Ong. 2020. Improving multi-agent cooperation using theory of mind. arXiv preprint arXiv:2007.15703.
References
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Haus- man, al. 2022. Do as can, not say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. 2023. Llm+ p: Empowering large language mod- els with optimal planning proficiency. arXiv preprint arXiv:2304.1 230..777..177
11477.
Chris L Baker, Julian Jara- attinnr,igeer Rebecca Saxe, and Joshua B Tenenbaum. 2017. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing, Nature Human Behaviour, 1(4):0064.
Kyle Mahowald, Anna Ivanova, Idan A Blank. Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fe- dorenko. 2023. Dissociating language and thought in large language models: cognitive perspective. arXiv preprint arXiv:2301.06627.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Woj- ciech Zaremba. 2016. Openai gym.
Scott A Miller. 2009. Children's understanding of second-order mental states. Psychological bulletin, 135(5):749.
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li. Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.1273 12712.
Shima Rahimi Moghaddam and Christopher J Honey- 2023. Boosting theory-of-mind performance in large language models via prompting. arXiv preprint arXiv:2304.11400
Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Ini Oguntola, Joseph Campbell, Simon Stepputtis, and Katia Sycara. 2023. Theory of mind as intrinsic motivation for multi-agent reinforcement learning. arXiv preprint arXiv:2307.01155
OpenAL. 2023. Gpt-4 technical report.

---

Joon Sung Park, Joseph C O' O'Brien, Carrie Cai, Mered- ith Ringel Morris, Percy Liang. and Michael Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442
enables open-world multi-task agents. arXiv preprint arXiv:2302.01550.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903.
Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas Christakis, Tain Couzin, Matthew O Jackson, et al. 2019. Machine behaviour. Nature, 368(7753):477-486
Jessica Williams, Stephen M Fiore, and Florian Jentsch. 2022. Supporting artificial social intelligence with theory of mind. Frontiers artificial intelligence,
Christoph Riedl, Young Ji Kim, Pranav Gupta, Thomas W Malone, and Anita Williams Woolley. 2021. Quantifying collective intelligence in human groups. Proceedings of the National Academy of Sciences, (18021))c2005737111
Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Gong- and Harold Soh. 2023. Translating natural language to planning goals with large- -angella models. arXiv preprint arXiv:2302.05722.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang. Alexandre Bayen, and Yi Wu. 2022. The surprising effectiveness of ppo in cooperative multi- agent games. Advances in Neural Information Pro- cessing Systems, 35:24611-24624,
Mikayel Samvelyan. Tabish Rashid, Chris- tian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. Torr, Jakob Foerster, and Shimon Whiteson. 2019. The StarCraft Multi-Agent Challenge. CoRR, abs/ 1oss/992.04444.
Luyao Yuan, Zipeng Fu, Linqi Zhou, Kexin Yang, and Song-Chun Zhu. 2021. Emergence of theory of mind collaboration in multiagent systems. arXiv preprint arXiv:2110.00121.
Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2023. Neural theory-of-minn mind?? on the limits of social intelligence in large Ims.
Jun Zhang, Trey Hedden, and Adrian Chia. 2012. perspective-taking and depth of theory-of-mind rea- soning in sequential-move games. Cognitive science, 36(3):560-55 -733
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. 2023. Minding lan- guage (lac play multi-character belief tracker. arXiv preprint 00924.
(lack of) theory of mind: A plug-and-
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang. Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. Xing. Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging Ilm-as-a-judge with mt-bench and chatbot arena.
Guni Sharon, Roni Stern, Ariel Felner, and Nathan R Sturtevant. 2015. Conflict-based search for opti- mal multi-agent pathfinding. Artificial Intelligence, 219:40-66.
Peter Sunehag. Guy Lever, Audrunas Gruslys, Woj- ciech Marian Czareecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, al. 2017. value-decomposition networks for cooperative multi-agent learning, arXiv preprint arXiv: 706.05296.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applica- tions. arXiv preprint arXiv:2201.082.33
Tomer Ullman. 2023. Large language models fail on trivial alterations to ofoory-o-minn tasks. arXiv preprint arXiv:2302.08::
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and An- ima Anandkumar, 2023a. Voyager: An onen-ended embodied agent with large language models. arXiv preprint 2305
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023b. Describe, explain, plan and se- lect: Interactive planning with large language models

---

Appendix
round number, team score and the current location of your teammates. Your teammates have the same observability as you. They will not be able to know your action and its consequences unless you explic- itly communicate.
Prompts
A.1 Task context
Welcome to our interactive text game! In this game, you'll assume the role of a specialist on a search and rescue team. Alongside two other players, you'll '1l navigate five-room environment with mission defuse five hidden bombs.
To facilitate our interaction, reply your action se- lection and communication messages in this fixed format: Action selection: Your action. Message to Team: "Your Message". To move to an adja- cent room, say: 'Move to Room X'. To inspect the sequence of a bomb in your current room, say: 'Inspect Bomb'. To apply wire cutter tool, say: Apply X Tool'. Remember, your replies must adhere strictly to these rules. Feel free to ask clari- fying questions needed. I'll supply the necessary progress. Are you ready to take on this explosive challenge?
The Map: Imagine a network of rooms repre- sented by a connected graph where cach node cor- responds to a room, and the edges between nodes depict hallways. The rooms are numbered 0, 3, 6, 5, and 8. Room 0 is connected to all other rooms. Room 5 shares hallway with room 6. Room is linked to room 8. And room 8 is also connected with room 6. You can only travel to adjacent, di- rectly connected rooms at each turn.
information
A.2 Initial belief state
The Challenge: Scattered among these rooms are five bombs, each coded with different phases represented by colors. To defuse them, you'll Il need to use the correct wire-cutting tools in the correct sequence. There are one-phase, two-phase, and three-phase bombs, needing 1, or 3 color-coded tool applications in sequence to disarm. For in- stance, bomb with red-green phase sequence re- quires the red tool first, then the green one. Points are awarded based on the number of tools used for defusing a bomb, with each tool use worth 10 points. Your task is to maximize the team score as soon as possible. The challenge is that the bomb locations and sequences unknown players the start.
Below is your current belief about game state based on your previous observations about the environ- ment and interactions with your teammates. Your role: You are playing as Player <agent id>. Current round:
Total team score: 0.
Observation: You are currently in Room 0 with both of your leammates. In the room you also found bomb I with unknown sequence. There is no other bomb in the current room.
Teammate Locations: Player alpha in Room 0; Player bravo is Room 0; Player charlie is Room
Room connectivity:
Tools: Each player is equipped with two color- coded wire cutters. As player Alpha, you have red and green tools, player Bravo wields green and blue, and player Charlie possesses blue and red. Actions: Each round, you can opt to do one of the following: 1) Move to adjacent room, 2) Inspect bomb's phase sequence in your current room, 3) Apply your wire cutters bomb in the current room.
Room 0 is connected room 5, 6,
Room 3 is connected to room 0
Room 5 is connected to room 0 and 6
Room 8 is connected to room 0 and 6
Bomb Intel:
Bomb 1: Located in Room 0. The phase se-
Communications: In addition to selecting am action to take from the above list, you can also send communication message texts to both of your teammates in each round. The message text you sent will be shared with both of your teammates in their observation in the next round.
quence is Unknown.
Bomb 2: Details currently unknown.
Bomb Details currently unknown.
Bomb Details currently unknown.
Observation: While you can only see what's in your current room and read text messages from teammates. You'll also be informed of the current
Bomb 5: Details currently unknown.
Tool inventory:

---

Alpha:
Equipped with red and green wire cut-
Do you know bomb phase has just been defused?
ters.
Bravo: cutters.
Equipped with green and blue wire
Do you know the sequence of bomb bombid?
C.2 First-order ToM
Equipped with red and blue wire cut- ters.
Charlie:
Does player playerid know the current con- tents of room roomid?
Available action options:
Does player playerid know the state and re- maining sequence of bomb bombid has been changed?
To move an adjacent room, say; 'Move to Room X'
Does player playerid know a bomb phase has just been defused?
To inspect the sequence of bomb in your current room, say: "Inspect Bomb'.
Does player playerid know the sequence bomb bombid?
To apply wire cutter tool, say: "Apply X Tool'.
C.3 Second-order ToM
send message to your teammates, say: "Message to Team: "Your Message"'.
Based the observation and previous history, is player playerid aware of the fact that you know the current contents of room roomid?
B Environment feedback for Error correction
Based the observation and previous his- tory, is player playerid aware of the fact that you have changed the state and remaining se- quence of bomb bombid?
Your action is invalid.
You can not directly move Room roomid because it not adjacent to your current loca- tion, Room currentroom. Consider taking a detour to another room first and then move to your destination.
Based on the observation and previous history. is player playerid aware of the fact that you know a bomb phase has just been defused?
There is no bomb in the current current loca- tion, Room currentroom, for you to inspect.
Based on the observation and previous history, is player playerid aware of the fact that you know the sequence of bomb bombid?
You can not apply Tool toolcolor to Bomb boomid because the sequence of this bomb is sequence. You will need apply other color tool first.
There is no bomb in your current location, room roomid, for you defuse.
You do not have Tool toolcolor. Consider asking your teammates who have this tool to help you defuse the bomb.
Theory of Mind Questions
C.1 Introspection
Do you know the current contents of room roomid?
Do you know the state and remaining se- quence of bomb bombid has been changed?


Reviews:
[{"review_id": "8mWSXEuK9j", "reviewer": "Reviewer_Bhvf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our contributions to the field. But we seek to clarify some details of our methodology here.\n\nThe reviewer's comments about ToM inference evaluation are well-taken. The design principle of the ToM inference task is inspired by the Sally–Anne test, the most widely used ToM task in human studies. Every time an agent conducts an action, we pose a belief reasoning question, asking if another agent (i.e., target agent) is aware of the potential consequence of this action. The consequence here can be either a state change (e.g., a bomb has been defused) or a belief change (e.g., Alpha has explored Room 5 and found Bomb 3 in the room). An agent equipped with ToM should realize that while they know the consequence, the target agent might hold a false belief about it. To evaluate whether LLM-based agents answer these questions correctly, human annotators were hired to provide subjective judgment based on fully observable interaction and communication history. Specifically, the following principles are considered: 1) if the target agent is present in the current room and observes the consequence, 2) if the target agent has been to this room before, 3) if the consequence has been communicated to the target agent. Additionally, we plan to explore other evaluations of LLM-based agent's ToM in future work, as discussed in Section 6.5.2 and Limitations.\n\nWe appreciate the reviewer's comments about the scalability of our work but would like to make a clarification. The size of the action space for our test environment is actually 9. The agent can choose to go to any of the 5 rooms, use wire-cutters in 3 colors, or inspect the bomb in the current room. We plan to evaluate our method in more complicated environments in future work. LLM-based agents might struggle in tasks where the state and action space are hard to describe in natural language (e.g., continuous space). Potential solutions to enable LLMs' embodied interactions in those tasks include combining LLM with low-level planners or using prompt engineering to divide tasks into smaller subtasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a novel evaluation of the large language models on a multi-agent collaboration game. It conducted thorough and sophisticated experiments to benchmark 6 methods with the task environment. It evaluates three orders of theory of mind capabilities of the models with interesting findings. The paper identified systematic failures of the LLM based agents and proposed ideas to mitigate the issues in current and future works. The paper is rigorous and well-written.\n", "reasons_to_reject": "It is unclear how ToM inference tasks are evaluated. The description is fairly concise without enough information to conclude the claim about theory of mind inferences. \n\nThe action space (3) of the agents is relatively small. Would be helpful to discuss ideas on how this work could be applied in a larger scale.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "\nThe paper evaluated large language models on their abilities in multi-agent collaborations. It conducted the evaluation in the MARL task environment where three LLMs play the role of three agents to safely defuse bombs. The game requires communication to exchange partial information and complete the tasks collaboratively. The paper engineered prompts to translate the game environment into natural language, a list of 4 actions to select, belief state, and a request for communication messages. The paper conducted comparisons among 6 variations of LLM, reinforcement learning, and rule based methods. The paper identified key failure points of the models and proposed future directions. \n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Ln 498: ','"}, "extra_text": {}}, {"review_id": "hR3IfGj65P", "reviewer": "Reviewer_tQsZ", "summary": "", "questions": "I recommend to add a few case studies in the appendix to show a complete game process and give people a more intuitive impression.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation of our paper.\n\nWe agree with the reviewer's desire to apply our method to more complicated tasks or competitive scenes and plan to do so in future work. However, the judgment that our task scenario is \"single\" and involves \"fully shared information\" is slightly inaccurate. The interaction history between agents and the environment is not shared within the team. Each agent has partial observability of its own room, plus the communication messages selectively sent by other agents. Given that leading MARL algorithm such as MAPPO can not solve the task optimally, we consider it as a relatively challenging teamwork scenario.\n\nThe reviewer's comments about human annotation are appreciated. More details about human annotations can be found in Section 6.5.2 and the second paragraph of Limitations. Specifically, human annotators were instructed to judge LLM's ToM inference questions based on the following principles: 1) if the target agent is present in the current room and observes the belief change, 2) if the target agent has been to this room before, 3) if the belief change has been communicated to the target agent. It is worth mentioning that high-order ToM inferences involving communication are naturally ambiguous. These corner cases were discussed among annotators to ensure a consistent standard across conditions.\n\nReproducibility details about MAPPO are provided in Section 5.2. Since the algorithm is open-sourced and we used the default hyperparameters from the original paper, we did not include further implementation details due to page restrictions. We will provide additional information in the camera-ready paper.\n\nWe appreciate the reviewer's comments on case studies, potential references, and presentation improvements and will address them in the final paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Second-order ToM inference examines whether an agent can estimate the state of other agents, while third-order ToM inference examines whether an agent can guess what others think of themselves. These psychological settings are very interesting and are not limited to completing the tasks themselves.", "reasons_to_reject": "The disadvantage of the article is that the scene is relatively single, and the scene is a fully shared information that only involves multi-agent cooperation. The human annotation results given in Table 2 did not explain the details and artificial judgment standards clearly. The implementation details of MAPPO implemented by the author were not explained.", "questions_to_authors": "I recommend to add a few case studies in the appendix to show a complete game process and give people a more intuitive impression.", "paper_topic_and_main_contributions": "The paper is very interesting. It can be seen that LLM agents are able to explain their own intentions and understand the intentions of other agents when completing cooperative tasks. Among various LLM prompt engineering works, there are not many articles that have both interesting starting points and good storytelling. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "The reproducibility of the paper may be affected because a large-scale model API provided by a commercial company was used."}, "extra_text": {"Missing_References": "The Theory of Mind section lacks many references to concurrent work in the field. I was able to easily find many paper about RL environments for Theory of Mind and testing LLM on Theory of Mind, such as \"Improving Multi-Agent Cooperation using Theory of Mind\" (https://arxiv.org/pdf/2007.15703.pdf), \"Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker\" (https://arxiv.org/pdf/2306.00924.pdf), and \"Emergence of Theory of Mind Collaboration in Multiagent Systems\" (https://arxiv.org/pdf/2110.00121.pdf).", "Typos_Grammar_Style_and_Presentation_Improvements": "- Generally, papers related to RL will formalize their environment, including state space, observation space, action space, reward function, etc. This paper focuses mostly on prompt engineering, it is not necessary to use formulas to describe these things. But, it is strongly recommended to describe the RL environment in a bullet-point list.\n- It is suggested to put the introduction of “bombs” and “rewards” from lines 294-310 into the introduction of the environment in Section 3.1.\n- The two embedded images by the author are too blurry. It is recommended to save them as PDF and then embed them in LaTeX.\n- Paragraph format at line 498."}}, {"review_id": "4rFhf6xLZc", "reviewer": "Reviewer_sosq", "summary": "", "questions": "1. Are the baseline models trained specifically for the task?\n2. How does the rule-based text interface work and what does the provided feedback for error correction look like?\n3. How many times the model will be evaluated on the task of bomb disaamble?\n4. How much variation could be included in the task initial status? \n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and address their comments individually below:\n\nBackground and reproducibility details for MAPPO and CBS are provided in Section 5.2. Since the scope of our paper focuses on evaluating LLMs, we omit formal presentations and implementation details of RL and planning-based baselines. We will provide additional information in the camera-ready paper.\n\nWe appreciate the reviewer's comments about generalizability and human baselines; we plan to explore them in future work as discussed in the Limitation section. Our task scenario is designed to represent real-world urban search and rescue tasks with components such as time pressure, information asymmetry, interdependent coordination, and decision-making under uncertainty. Artificial teams are expected to mimic foundational teamwork processes in human teams, including communication, backup behaviors, and shared mental model etc. We believe the scenario is a reasonable representation of general collaborative tasks with the potential to scale.\n\nYes, the MARL algorithm is trained on the specific task environment for 45 million timestamps until convergence. CBS is a planning-based method without training but is also configured for the specific task.\n\nThe text interface translates state observations into natural language descriptions and encodes LLM outputs into concrete action selections. Specifically, the text interface extracts state features from the game engine and replaces keywords in the templates. A typical observation description looks like: <Your observation is: Round: 1. Total team score: 0. Results: You inspected Bomb 1. This bomb is a 1-stage bomb, and its remaining sequence is Red. Room contents: You are currently in Room 0. Contents of this room include alpha, bravo, and charlie; Bomb 1. Teammate Locations: Player alpha is in Room 0; Player bravo is in Room 0; Player charlie is in Room 0. Communication messages sent by your teammates: Player alpha: \"I am inspecting the bomb in Room 0. Bravo; please move to Room 3. Charlie; please move to Room 5.\". Player bravo: \"None\". Player charlie: \"None\".>. Action encoding is done via keyword matching since LLMs are instructed to frame their responses in a certain format and structure. The text interface provides error feedback based on pre-programmed rules and message templates, such as <There is no bomb in the current location, Room {current_room}, for you to inspect.>.\n\nTeams of LLM-based agents were evaluated on the task scenario three times. Since the temperature parameter of LLMs is set to zero, their team behaviors are considerably consistent across trials. Performance of all experiment trials is presented in Table 1 with means and standard deviations.\n\nAs introduced in Section 5.1, the task environment can be initialized with different starting locations, room connections, bomb distributions, and sequences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper conducts Theory of Mind (ToM) evaluations via novel interactive team tasks, where agents' mental states dynamically evolve with each interaction. Also, the paper identifies systematic shortcomings that hinder the efficiency of Large Language Model (LLM)-based agents in collaboration and suggests a prompt-engineering approach to lessen these failures.", "reasons_to_reject": "Details and background information are missing for the baseline reinforcement learning method MAPPO and the CBS Planner.\n\nIt seems unclear how the proposed novel collaborative task on bomb disaamble can be general enough to represent the model's ability to perform team interaction. More task analysis and the human baseline are needed. ", "questions_to_authors": "1. Are the baseline models trained specifically for the task?\n2. How does the rule-based text interface work and what does the provided feedback for error correction look like?\n3. How many times the model will be evaluated on the task of bomb disaamble?\n4. How much variation could be included in the task initial status? \n", "paper_topic_and_main_contributions": "This paper evaluated the ability of Large Language Models (LLMs) to carry out embodied interactions in a team-based task. The findings showed that LLM-based agents can perform complex multi-agent collaborative tasks with Theory of Mind (ToM) inference tasks on par with the current leading reinforcement learning algorithm.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviews for this paper were quite consistently positive, with scores 4,4,3 (soundness) and 4,4,4 (excitement).

Condensing the reviews, the following were among the strengths and weaknesses mentioned:

Strengths:

- The reviewers consistently found the paper well-written, and the results interesting.
- Highlights systematic failures of LLM based agents (R1, R3)
- interesting psychological settings (R2)

Weaknesses:

- some lack of clarity on implemention/evaluation (R1, R2, R3)
- small action space (R1) and limited settings (R2, R3)
Decision: Accept-Main