RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Inference under false belief
In some situations, ToM inferences become extremely challenging for LLMs. In our records, this is typically when reasoning involves agent communication. Even with ample context, it’s hard for agents to track information transmission while conducting ToM inferences. The most demanding scenarios occur when agents share inaccurate information, leading to false beliefs. This usually happens when LLM-based agents try an invalid action and broadcast false intentions through communication. For instance, when Alpha tells the team they’re moving to Room 8 (which is unreachable), and we ask Alpha, "Is Charlie aware that you know the contents of Room 8?" This question is tricky since Alpha failed to reach Room 8. In theory, the most acceptable answer is complex and involves multiple high-order ToM assumptions. For example a human annotated ground truth is:

"I did not know the contents of Room 8 cause I failed to make it there. But since I told Charlie I will go there, they might assume that I know the contents of Room 8. However, considering Charlie knows my current location, they should be able to tell that I am not able to move to Room 5 and induce that I do not know the contents of Room 5."

In practice, LLM-based agents often grasp only part of the logic or miss the crucial fact that they didn’t reach the target room. Quotes from GPT-4+Belief agent:

"Based on the observation and previous history, Player Charlie is not explicitly aware that I know the current contents of Room 8. However, since I mentioned in my previous message that I was moving to Room 8, Player Charlie might assume that I would eventually learn about the contents of Room 8."
Discussions
Our study yields three primary insights. First, Large Language Models (LLMs) demonstrate substantial planning and collaboration capabilities within our task scenarios. With suitable prompt-engineering, teams of LLM-based agents perform comparably to state-of-the-art Multi-Agent Reinforcement Learning (MARL) algorithms. This finding is particularly noteworthy given that MARL agents receive extensive task-specific training with a centralized critic, while LLM-based agents operate in a fully decentralized manner and undertake tasks in a zero-shot setting. Despite prior research highlighting LLMs’ limitations in generating actionable plans and interacting with the world, they perform reasonably well when placed in a team and tasked to process actions step-by-step. Particularly, LLMs fine-tuned with Reinforcement Learning from Human Feedback demonstrate emergent social interaction skills in multi-agent collaborations, which might be similar to the collaborative and interactive settings in which human language is primarily learned and used Sap et al. ( 2023 ) .

Second, LLMs still fall short of being optimal planners or team players due to systematic failures, such as neglecting long-horizon contexts and making inaccurate assumptions about the task state (a.k.a hallucination). These flaws significantly hinder team collaborations as they can rapidly disseminate misinformation via communication, leading to widespread false beliefs. We attempted to mitigate these issues by allowing LLM-based agents to maintain an explicit belief state about the world. Our findings suggest that modern LLMs can update the given belief descriptions based on their observations, hinting at the potential emergence of advanced cognitive skills such as world knowledge understanding and situation modeling. Moreover, belief state representations offer a structured framework that helps agents track key task-related information, leading to improved team performance.

Finally, our study indicates that the Theory of Mind (ToM) capabilities of LLMs are still limited, particularly when evaluated within interactive teamwork scenarios that involve dynamic belief states and intensive communication. For context, while 5-year-old children can perform second-order ToM inferences Miller ( 2009 ) , adults don’t consistently use this ability during communications due to the complexity and ambiguity of social interactions Keysar et al. ( 2003 ) . Thus, there’s considerable work ahead for LLMs to develop a functional ToM and interact naturally with humans. Our study represents a preliminary effort to devise novel evaluation methods for LLMs’ ToM that go beyond traditional tests such as the Sally-Anne test.
Conclusions
In this study, we assessed the ability of recent large language models (LLMs) to conduct embodied interactions in a team task. Our results demonstrate that LLM-based agents can handle complex multi-agent collaborative tasks at a level comparable with the state-of-the-art reinforcement learning algorithm. We also observed evidence of emergent collaborative behaviors and high-order Theory of Mind capabilities among LLM-based agents. These findings confirm the potential intelligence of LLMs in formal reasoning, world knowledge, situation modeling and social interactions. Furthermore, we discussed two systematic failures that limit the performance of LLM-based agents and proposed a prompt-engineering method that mitigates these failures by incorporating an explicit belief state about world knowledge into the model input.
Limitations
This study represents an initial effort to understand machine intelligence in complex task scenarios. Several enhancements could improve the experimental setup and offer a more thorough evaluation of LLMs in multi-agent collaborations. First, we could incorporate additional LLMs besides OpenAI’s GPT models. As new models emerge with enhanced reasoning capabilities and larger input sizes, their performance in team tasks and ToM inference may also change. Second, the task environment is relatively simple with only five nodes and five bombs. We plan to scale up the environment and introduce more restrictions to test how LLM-based teams react to more challenging tasks. Lastly, the current team consists of three agents with homogeneous policies. It would be intriguing to evaluate how LLM-based agents perform in human-agent teams, especially from a human-centered perspective where issues like trust, transparency, and human-agent co-training can be addressed.

The ToM capability evaluation method used in this study also has its limitations. Currently, human annotators, who have a global view of the task state and interaction history, generate the ground truth for ToM inference questions. However, this estimation is at best an approximation, assuming agents process information as a rational human would, which might be ambiguous in situations involving false beliefs or miscommunications. A potential alternative could be using each agent’s maintained belief state as the ground truth.

The proposed belief state method could extend from introspective belief to first-order or even second-order beliefs. Currently, LLM-based agents maintain a belief state about their own world knowledge in text form. By extending this representation to include other agents’ world knowledge, we could equip LLM-based agents with an explicit first-order ToM model. Their ToM capability can be assessed by directly comparing one’s first-order belief with another’s introspective belief, rather than asking LLMs Sally-Anne style questions.
Acknowledgements
This work was supported by DARPA award HR001120C0036 and AFOSR award FA9550-18-1-0097.
A.1 Task context
Welcome to our interactive text game! In this game, you’ll assume the role of a specialist on a search and rescue team. Alongside two other players, you’ll navigate a five-room environment with a mission to defuse five hidden bombs.

The Map : Imagine a network of rooms represented by a connected graph where each node corresponds to a room, and the edges between nodes depict hallways. The rooms are numbered 0, 3, 6, 5, and 8. Room 0 is connected to all other rooms. Room 5 shares a hallway with room 6. Room 3 is linked to room 8. And room 8 is also connected with room 6. You can only travel to adjacent, directly connected rooms at each turn.

The Challenge : Scattered among these rooms are five bombs, each coded with different phases represented by colors. To defuse them, you’ll need to use the correct wire-cutting tools in the correct sequence. There are one-phase, two-phase, and three-phase bombs, needing 1, 2, or 3 color-coded tool applications in sequence to disarm. For instance, a bomb with a red-green phase sequence requires the red tool first, then the green one. Points are awarded based on the number of tools used for defusing a bomb, with each tool use worth 10 points. Your task is to maximize the team score as soon as possible. The challenge is that the bomb locations and sequences are unknown to players at the start.

Tools : Each player is equipped with two color-coded wire cutters. As player Alpha, you have red and green tools, player Bravo wields green and blue, and player Charlie possesses blue and red.

Actions : Each round, you can opt to do one of the following: 1) Move to an adjacent room, 2) Inspect a bomb’s phase sequence in your current room, or 3) Apply your wire cutters to a bomb in the current room.

Communications : In addition to selecting an action to take from the above list, you can also send communication message texts to both of your teammates in each round. The message text you sent will be shared with both of your teammates in their observation in the next round.

Observation : While you can only see what’s in your current room and read text messages from teammates. You’ll also be informed of the current round number, team score and the current location of your teammates. Your teammates have the same observability as you. They will not be able to know your action and its consequences unless you explicitly communicate.

To facilitate our interaction, reply your action selection and communication messages in this fixed format: Action selection: Your action. Message to Team: “Your Message”. To move to an adjacent room, say: ’Move to Room X’. To inspect the sequence of a bomb in your current room, say: ’Inspect Bomb’. To apply a wire cutter tool, say: ’Apply X Tool’. Remember, your replies must adhere strictly to these rules. Feel free to ask clarifying questions if needed. I’ll supply the necessary information as we progress. Are you ready to take on this explosive challenge?
A.2 Initial belief state
Below is your current belief about game state based on your previous observations about the environment and interactions with your teammates. Your role : You are playing as Player <agent id>. Current round : 1 Total team score : 0. Observation: You are currently in Room 0 with both of your teammates. In the room you also found bomb 1 with unknown sequence. There is no other bomb in the current room. Teammate Locations: Player alpha is in Room 0; Player bravo is in Room 0; Player charlie is in Room 0. Room connectivity :

Room 0 is connected to room 3, 5, 6, 8

Room 3 is connected to room 0

Room 5 is connected to room 0 and 6

Room 8 is connected to room 0 and 6

Bomb Intel :

Bomb 1: Located in Room 0. The phase sequence is Unknown.

Bomb 2: Details currently unknown.

Bomb 3: Details currently unknown.

Bomb 4: Details currently unknown.

Bomb 5: Details currently unknown.

Tool inventory:

Alpha: Equipped with red and green wire cutters.

Bravo: Equipped with green and blue wire cutters.

Charlie: Equipped with red and blue wire cutters.

Available action options:

To move to an adjacent room, say: ’Move to Room X’.

To inspect the sequence of a bomb in your current room, say: ’Inspect Bomb’.

To apply a wire cutter tool, say: ’Apply X Tool’.

To send a message to your teammates, say: ’Message to Team: "Your Message"’.
C.1 Introspection
Do you know the current contents of room {roomid} ?

Do you know the state and remaining sequence of bomb {bombid} has been changed?

Do you know a bomb phase has just been defused?

Do you know the sequence of bomb {bombid} ?
C.2 First-order ToM
Does player {playerid} know the current contents of room {roomid} ?

Does player {playerid} know the state and remaining sequence of bomb {bombid} has been changed?

Does player {playerid} know a bomb phase has just been defused?

Does player {playerid} know the sequence of bomb {bombid} ?
C.3 Second-order ToM
Based on the observation and previous history, is player {playerid} aware of the fact that you know the current contents of room {roomid} ?

Based on the observation and previous history, is player {playerid} aware of the fact that you have changed the state and remaining sequence of bomb {bombid} ?

Based on the observation and previous history, is player {playerid} aware of the fact that you know a bomb phase has just been defused?

Based on the observation and previous history, is player {playerid} aware of the fact that you know the sequence of bomb {bombid} ?
Figure 1: Our proposed framework consist of 3 LLM-based agents, a text game interface and the actual task environment. The natural language outputs of LLM are encoded into abstract actions by the text interface and then sent to task environment. The task environment then processes agent actions and returns observations via the text interface. Upon receiving environmental observations, LLM-based agents are prompted to update their beliefs and output action selections and messages. 3 agents in the team are coded as Alpha, Bravo, and Charlie and take turns to interact with the interface.; Our proposed framework consist of 3 LLM-based agents, a text game interface and the actual task environment. The natural language outputs of LLM are encoded into abstract actions by the text interface and then sent to task environment. The task environment then processes agent actions and returns observations via the text interface. Upon receiving environmental observations, LLM-based agents are prompted to update their beliefs and output action selections and messages. 3 agents in the team are coded as Alpha, Bravo, and Charlie and take turns to interact with the interface.
Figure 2: Example interactions between LLM-based agents and the text game interface. The upper left panel showcases one type of systematic failures we observed in LLM’s outputs in which long horizon contexts are overlooked. The upper right panel illustrates emergent collaborative behaviors (e.g. emergent leadership) between LLM-based agents. The bottom two panels are quotes of GPT-4+Belief and ChatGPT agents’ answers for ToM inference questions.; Example interactions between LLM-based agents and the text game interface. The upper left panel showcases one type of systematic failures we observed in LLM’s outputs in which long horizon contexts are overlooked. The upper right panel illustrates emergent collaborative behaviors (e.g. emergent leadership) between LLM-based agents. The bottom two panels are quotes of GPT-4+Belief and ChatGPT agents’ answers for ToM inference questions.
Task performance of LLM-based agents and baseline conditions. Score represent the average team score in all experiment trials. Length refers the average number of rounds the team took in completing the task. Percentages of valid action measures the proportion of LLM outputs that can be encoded into actions allowed by the task rules. Numbers after \pm are 1 standard deviation.
LLM-based agents’ performance in ToM inference tasks. Natural language answers are annotated by experimenters and compared with the ground truth based on global interaction history. Percentages represent the inference accuracy.