Introduction
Recent large language models (LLMs), such as GPT-4 OpenAI ( 2023 ) , have demonstrated impressive competencies across a wide array of domains and tasks, ranging from mathematics to law, without the need for fine-tuning or special prompting Bubeck et al. ( 2023 ) . This advancement has significantly transformed the landscape of Natural Language Processing (NLP) research. Instead of developing domain-specific models for downstream applications, focus has shifted towards evaluating and harnessing LLMs’ abilities to solve novel tasks. Such a shift is consistent with the idea of studying machine behaviors, an interdisciplinary approach that expands the conventional bounds of computer science and integrates insights from diverse scientific fields Rahwan et al. ( 2019 ) . Drawing inspiration from team science and group psychology Hagendorff ( 2023 ) , our study concentrates on collective machine behavior, evaluating LLMs’ proficiency in multi-agent collaborations.

There is ongoing debate regarding the intelligence levels of modern LLMs. While some argue that LLMs excel primarily in linguistic competence and struggle with cognitive abilities beyond language, known as functional competence, others demonstrate that LLMs can exhibit cognitive skills such as formal reasoning and world knowledge comprehension Mahowald et al. ( 2023 ); Bubeck et al. ( 2023 ) . Motivated to explore this argument, we designed a text-based game to evaluate LLMs’ ability in embodied interactions, including exploring unknown environments, maintaining beliefs about the world and collaborating with other agents, which is critical for natural social interactions and artificial general intelligence (AGI).

Theory of Mind, the capacity to reason about others’ concealed mental states, is fundamental to human social interactions, collaborations, and communications Zhang et al. ( 2012 ) . As LLMs increasingly participate in diverse social interactions with humans, their social intelligence is expected to improve for them to become effective collaborators Williams et al. ( 2022 ); Li et al. ( 2022 ) . For instance, a proficient AI assistant should be able to infer a human’s preferences based on previous experiences without needing to ask. Recent studies have applied classic Theory-of-Mind tasks to several LLMs, concluding that current models (e.g., GPT-4) perform comparably to 9-year-old children Kosinski ( 2023 ) . However, the research community has expressed doubts about the validity of text-based ToM tests on machine intelligence Ullman ( 2023 ); Sap et al. ( 2023 ) . In response, our study proposes a novel evaluation of LLMs’ high-order ToM in interactive teamwork scenarios, encompassing dynamic belief state evolution and rich intent communication between multiple agents.

The main contributions of this study include that we:

Evaluate LLM-based agents’ embodied interaction capability in multi-agent collaborative tasks against reinforcement learning and planning-based baselines

Identify systematic failures that limit the collaboration efficiency of LLM-based agents, and propose a prompt-engineering method to mitigate those failures by incorporating explicit belief state representations about world knowledge in the model input

Propose a novel evaluation of LLMs’ high-order ToM in interactive teamwork scenarios, encompassing dynamic belief state evolution and rich intent communication between multiple agents