T MBENCH: Benchmarking Theory of Mind in Large Language Models (2024)

Abstract: Theory of Mind (ToM) is the cognitive capability to perceive and ascribe mental states to oneself and others. Recent research has sparked a debate over whether large language models (LLMs) exhibit a form of ToM. However, existing ToM evaluations are hindered by challenges such as constrained scope, subjective judgment, and unintended contamination, yielding inadequate assessments. To address this gap, we introduce T MBENCH with three key characteristics: a systematic evaluation framework encompassing 8 tasks and 31 abilities in social cognition, a multiple-choice question format to support automated and unbiased evaluation, and a build-from-scratch bilingual inventory to strictly avoid data leakage. Based on T MBENCH, we conduct extensive experiments to evaluate the ToM performance of 10 popular LLMs across tasks and abilities. We find that even the most advanced LLMs like GPT-4 lag behind human performance by over 10% points, indicating that LLMs have not achieved a human-level theory of mind yet. Our aim with T MBENCH is to enable an efficient and effective evaluation of LLMs’ ToM capabilities, thereby facilitating the development of LLMs with inherent social intelligence.

heading: 1 Introduction
text: Theory of Mind (ToM) is a fundamental cognitive process, involving the capability to attribute mental states – beliefs, intents, desires, emotions, knowledge, etc. – to oneself and others (Premack and Woodruff, 1978). ToM is essential for human social cognition (Baron-Cohen et al., 1985) and plays an important role in social activities like empathetic communication (Decety and Jackson, 2004), relationship maintenance (Slaughter et al., 2002), decision making (Carlson and Moses, 2001), and childhood education (Caputi et al., 2012).
∗ Equal contribution. † Corresponding author.
With the advent of the era of large language models (LLMs), powerful LLMs like GPT-4 (Achiam et al., 2023) and LLaMA (Touvron et al., 2023) have demonstrated comparable performance to humans in solving tasks. Consequently, researchers have grown increasingly curious that “Do large language models have a theory of mind?”, while the subsequent assessments have yielded inconsistent and even contradictory results. Kosinski (2023) and Bubeck et al. (2023) find that GPT-3.5/4 achieve remarkably high performance on the false belief task (Wimmer and Perner, 1983), suggesting that ToM may have spontaneously emerged in LLMs. However, later studies reveal that LLMs’ ToM performance dramatically decreases when faced with trivial alterations (Ullman, 2023; Shapira et al., 2023). This suggests a phenomenon called “Clever Hans” (Lapuschkin et al., 2019) denoting that LLMs rely on unexpected spurious correlations (Simon and Simon, 1977) rather than truly possessing a “Mind Reading” ability. This ongoing debate underscores the pressing need for a holistic ToM benchmark.
Despite its critical importance, the development of such a benchmark still faces three major challenges. 1) Constrained scope. Existing studies mainly evaluate LLMs on specific ToM dimensions like emotions (Sap et al., 2019) and beliefs (Wu et al., 2023). While inspiring, they fall short of providing a complete ToM assessment. 2) Subjective judgment. Most existing inventories for ToM evaluation involve open-ended questions where responses require manual scoring by experts (Beaudoin et al., 2020). While this approach works on small-scale and focused ToM tests for children, it is impractical for benchmarking due to the high annotation cost required for comprehensively evaluating LLMs. Additionally, human evaluations towards diverse answers may introduce potential inconsistency and bias (Klie et al., 2023). 3) Unintended contamination. Although merely combining different inventories can create a corpus involving
15959
multiple ToM aspects, it is still not qualified for being a benchmark. Since most ToM inventories in psychology research are released far before the launch of LLMs, they are likely to have appeared in the training process of LLMs, whether during pre-training, SFT, or RLHF (Golchin and Surdeanu, 2023; Li and Flanigan, 2023). We present a simple verification in Appendix A. The potential contamination indicates a risk that LLMs might complete ToM evaluations in an “open-book” status, leading to inflated performance (Shapira et al., 2023).
To address the above challenges, we propose T MBENCH, a ToM benchmark for LLMs featuring three key characteristics. 1) Systematic framework. We first review broad psychological literature and identify 8 well-defined theoryof-mind tasks in T MBENCH. Then, since tasks are the means and abilities are the core, we further ground T MBENCH in a well-structured psychological framework, ATOMS (Beaudoin et al., 2020), expanding our evaluation to encompass 31 core theory-of-mind abilities. We hereby design both task- and ability-oriented evaluations to comprehensively assess the ToM capabilities of LLMs. 2) Multi-choice question format. We define every test sample in T MBENCH as a story, followed by a question, and several plausible options where only one answer is correct and the others are high-quality but misleading wrong answers. This multiple-choice question (MCQ) format enables automated and efficient evaluation, avoiding the high labor costs and potential inconsistencies in manual scoring. 3) Build-from-scratch inventory. We do not use any existing inventories. Instead, after being trained by professional psychologists who are proficient in ToM, we authors draw inspiration from daily social scenarios and manually create
2,860 original test samples, each available in both Chinese and English. We also implement strict annotation and validation procedures to ensure the data quality. The resulting inventory includes the above-mentioned 8 tasks and 31 abilities for ToM evaluation, covering a wide range of real-world human social scenarios across diverse topics.
Using T MBENCH, we conduct both task- and ability-oriented evaluations of 10 popular LLMs with vanilla and CoT prompting. We also recruit 20 graduate students to provide an expected human baseline ToM performance. The experimental results show that, the GPT-4 series outperforms other LLMs across ToM tasks and abilities, yet their average performance is significantly lower than human levels by over 10% points. Further analysis shows that when facing a more rigorous and comprehensive understanding of social scenarios, the gap in ToM performance between LLMs and humans widens. Moreover, inspecting the internal attention of LLMs reveals their decision-making processes notably differ from those of humans.
To the best of our knowledge, T MBENCH is the first systematic ToM benchmark for automated ToM evaluation of LLMs. We hope that T MBENCH will drive construction through evaluation, facilitating the development of LLMs in the realm of social intelligence, thereby complementing their task-solving capabilities. A broader objective is to enable LLMs to transcend the superficial layer of natural language and delve deeper into understanding the mental states and cognitive processes behind human words, hereby foster more efficient and seamless human-AI interaction. We release T MBENCH at https://github.com/ zhchen18/ToMBench.

heading: 2 T MBENCH Framework

heading: 2.1 8 Theory-of-Mind Tasks
text: The essence of the theory of mind is social understanding, which refers to the ability of humans to perceive and ascribe mental states to themselves and others in social scenarios. In T MBENCH, we first identify 8 social cognitive tasks that have been widely proven in psychology to effectively assess ToM capabilities: Unexpected Outcome Test (Dyck et al., 2001), Scalar Implicature Task (Goodman and Stuhlmüller, 2013), Persuasion Story Task (Kołodziejczyk and Bosacki, 2016), False Belief Task (Wimmer and Perner, 1983), Ambiguous Story Task (Bosacki and Wilde Astington, 1999), Hinting Test (Corcoran et al., 1995), Strange Story Task (Happé, 1994), Faux-pas Recognition Test (Baron-Cohen et al., 1999). We then construct test samples for these tasks based on definitions, descriptions, and examples in the original psychology literature. Due to space constraints, we present the details of all tasks in Appendix B.

heading: 2.2 31 Theory-of-Mind Abilities
text: Drawing from expertise in psychology, we gain insight into the principle that tasks are means, while abilities are the core (Quesque and Rossetti, 2020). Thus, we further resort to a well-defined psychological framework, “Abilities in the Theory-ofMind Space (ATOMS)” (Beaudoin et al., 2020). Originally designed for an extensive analysis of psychological ToM measures towards young children, ATOMS outlines 7 distinct ability dimensions: Emotion, Desire, Intention, Percept, Knowl-
edge, Belief, and Non-literal Communication, embracing 39 specific theory-of-mind abilities. After removing the Percept dimension requiring visual cues and some mixed abilities, we retain a total of 6 dimensions and 31 abilities in T MBENCH.
However, as shown in Figure 2, the abovementioned 8 tasks only encompass 19/31 abilities in ATOMS. Therefore, to address the remaining 12 abilities, we again refer back to the original literature to supplement extra ability-specific test samples, thereby extending the evaluation scope of T MBENCH to a complete set of 31 abilities. Due to space limitation, we here briefly introduce 6 ability dimensions, while the complete details of 31 specific abilities can be found in Appendix C.
1) Emotion involves 7 abilities to understand that situational factors influence people’s emotional states, that people can experience complex emotions, and that people can regulate emotional expressions. 2) Desire involves 4 abilities to understand that people have subjective desires, preferences, and wants that influence their emotions and actions. 3) Intention involves 4 abilities to understand that people undertake actions in pursuit of goals and intentions. 4) Knowledge involves 4 abilities to understand that others have access to different knowledge based on their perceptions, information they have received, or familiarity with things. 5) Belief involves 6 abilities to understand that people can hold beliefs about the world that are different from reality or different from one’s own beliefs. 6) Non-literal Communication involves 6 abilities to understand that communication can convey meaning beyond the literal words spoken.
Exploring Family Attic
Siblings Belongings
Sibling Relationships
Parenting Challenges
Cultural Traditions
Office Organization
Office Gossip
New Employee Orientation Unexpected Meeting
Achievement Recognition
Classroom Exploration
Imitation Behavior in a Robot Artificial Intelligence
Social Media Addiction Interest in Science
Exhibition Visit
Wo rkp
lac e
Sc ho
ol /C
ol le ge
Neighborhood
Recr eatio
nF rie
nd sh
ip
R om
an tic Technology Civic
Family
Legal Support Theft and Response Community Service Volunteer Selection
Volunteer Activity
Balancing Homework and Leisure Library Mischief Academic Pressure Group Project Dynamics
Community Theft and Response
Unexpected Discovery in the Basement
Intriguing Discovery
Conflict Resolution
Interaction Between Neighbors
Amusement Park Experience
Sports
Unfortunate Picnic Experience Birthday Party
Escape Room Game
Privacy and Bully Supporting a Friend in Need Supporting a Friend's Success Support and Encouragement Conflict and Reconciliation
Trust and Loyalty Planning a Romantic Vacation Gift Giving or Receiving Relationships after Breakup Long Distance Relationship
Figure 3: Topics of social scenarios in T MBENCH. Under 9 primary topics, we highlight the top-5 subtopics with the highest frequency.

heading: 3 T MBENCH Construction

heading: 3.1 Overview
text: Principles We authors act as workers and build T MBENCH from scratch. We do not use any existing inventories from psychological literature due to the potential risk of data contamination and the limited size of test samples. To ensure the effectiveness of inventory, all workers have undergone training by psychology experts, gaining a solid understanding of ToM. Furthermore, all workers have thoroughly reviewed the related psychology literature, adhering strictly to the specific definitions, descriptions, and examples of tasks and abilities.
Procedure We first craft samples for 8 tasks, with each worker responsible for at least one, covering 19 abilities in total. Owing to the natural difficulty of tasks and abilities, we ensure that each task has ≥100 samples, and each ability within a task has ≥20 samples. This step results in the creation of 2,470 samples. Subsequently, we add extra samples for the remaining 12 abilities not previously covered, with ≥20 samples for each ability, bringing the total number of samples to 2,860.
Statistics In Table 1, we present the data statistics of T MBENCH. Due to space constraints, the detailed statistics of the 31 abilities are present in Appendix D. As shown in Figure 3, T MBENCH includes diverse daily topics, such as school, workplace, family, neighborhood, etc. These real-life social scenarios are crucial for an effective ToM evaluation, just as we mentioned earlier, the essence of the theory of mind is social understanding.
#S #Q ASL(En) ASL (Zh) Agr.
Task View 934 2,470 61.22 97.69 99.4% Unexpected Outcome Test 100 300 38.46 62.01 100.0% Scalar Implicature Task 100 200 47.17 76.89 100.0% Persuasion Story Task 100 100 36.58 51.35 95.0% False Belief Task 100 600 49.15 77.54 100.0% Ambiguous Story Task 100 200 102.57 164.07 100.0% Hinting Test 93 103 49.63 79.92 100.0% Strange Story Task 201 407 70.42 112.97 100.0% Faux-pas Recognition Test 140 560 95.77 156.79 98.2%
Ability View 1,584 2,860 66.57 107.21 99.4%
Emotion 300 420 52.34 83.50 99.8% Desire 160 180 50.19 74.91 97.2% Intention 273 340 82.56 131.20 100.0% Knowledge 170 290 56.38 94.26 100.0% Belief 440 882 55.70 88.99 100.0% Non-Literal Communication 241 748 88.02 143.91 99.4%
Table 1: Data statistics. #S: Number of stories, #Q: Number of questions, ASL(En/Zh): Average story length (English/Chinese). Agr.: Final agreement.

heading: 3.2 Data Collection
text: We here detail the process of data collection. We construct T MBENCH in the form of multiplechoice questions to avoid the high costs of manual scoring, and ensure an unbiased and consistent evaluation. Each sample is defined as a combination of a story, a question, and several options.
Story describes a scenario from everyday life, including characters’ actions and interactions, to set the context for evaluation. The inspiration for stories primarily comes from posts on social platforms like Reddit, Twitter, Zhihu, and Weibo, such as “What kind of hints have you understood from others?” The design of stories refers to the setting in psychology literature while ensuring variety in social scenarios.
Question asks subjects to understand specific aspects of the story, which strictly adheres to psychological definitions of ToM tasks and abilities. Every question is designed to be answerable by humans. A story can correspond to multiple questions, each exploring different aspects of a social scenario to assess a comprehensive understanding, where each question represents a unique test sample.
Options include one correct answer and several misleading incorrect answers. The incorrect options are designed to be plausible, avoiding outliers that are easily dismissed. Options are generally of two types: for true/false questions, such as “Is what PersonA said true?”, the options are simply yes/no. For explanatory questions, such as “Why did PersonB say this?”, there are four options provided.

heading: 3.3 Data Validation & Translation
text: After the data collection, we conduct two rounds of validation to ensure the data quality. In 1st round, worker A would first complete all samples created by worker B. For stories, questions, and options where there are disagreements, workers A and B would discuss and modify them to reach a consensus as much as possible. In 2nd round, for samples where consensus is still not reached, another worker C would discuss with workers A and B to determine the final answer. After two rounds of discussion, the final average agreement reaches 99.4%. The inventory is initially crafted in Chinese. We then carefully use GPT-4-0613 to translate it into English, and manually check all translated samples to support bilingual ToM evaluation. Note that we do not provide correct answers in translation, thus there is no data leakage. The translation prompts can be found in Appendix E. Since we authors act as workers, we do not pay any other individuals for data collection.

heading: 3.4 Evaluation Method
text: In T MBENCH, we organize the test samples in two perspectives: the task view groups them into 8 theory-of-mind tasks, and the ability view categorizes them into 31 specific theory-of-mind abilities. The former approach is more general and commonly used in psychology research; the latter is more comprehensive, allowing us to inspect the performance of each specific ability. Accordingly, we report both task-oriented and ability-oriented results by averaging the samples related to a specific task or ability. For evaluation, we present LLMs with a story, a question, and several options, then ask them to pick the correct answer.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: We evaluate a total of 10 popular LLMs, including GPT-4-1106 (OpenAI, 2023b), GPT-40613 (Achiam et al., 2023), GPT-3.5-Turbo1106 (OpenAI, 2023b), GPT-3.5-Turbo-0613 (OpenAI, 2023a), ChatGLM3-6B (THUDM, 2023), LLaMA2-13B-Chat (Touvron et al., 2023), Baichuan2-13B-Chat (Baichuan-Inc, 2023), Qwen14B-Chat (Bai et al., 2023), Mistral-7B-v0.2 (Jiang et al., 2023), and Mixtral-8x7B-Instruct-v0.1 (Mistral AI, 2023). For GPT* and other open LLMs, we strictly abide by their terms and get access through official APIs and model weights, respectively.
We employ two prompting methods: the vanilla prompting directly asks LLMs to give a choice, while the CoT prompting elicits step-by-step reasoning. The prompts are detailed in Appendix F. For all models (except GPT-4-*), to avoid the bias from option IDs (Zheng et al., 2023), we shuffle the option orders five times and choose the most frequently selected option as the final answer. For the GPT-4-* models, our pilot experiments show very consistent answers across different option orders, so we only use the result from one round of answering. Accuracy is used as the metric.
To establish a human baseline, we recruit 20 native graduate students (each paid with $15) to complete the Chinese T MBENCH together. No extra tutorials or examples are provided to ensure a fair comparison. We directly use this result as human performance since studies have found that cultural and language differences do not significantly affect ToM task performance across native-English and native-Chinese speakers (Bradford et al., 2018).

heading: 4.2 Main Results
text: We show the ToM performance of LLMs across 8 tasks (Table 2) and 6 ability categories (Table 3), with detailed results on the 31 specific abilities presented in Appendix G. We now dissect the results and highlight several critical observations.
Human vs. LLMs The average ToM performances of all LLMs are significantly lower than that of humans, with the smallest gap being 10.1% in task view (Human 85.4% vs. GPT-4-1106 75.3%) and 10.8% (Human 86.1% vs. GPT-4-0613 + CoT 75.3%) in ability view, respectively. Interestingly, in the false belief task (FBT), LLMs like GPT-4-1106 even outperform humans, which we believe is explainable. In existing ToM inventories, there are relatively abundant samples for FBT, such as the Sally-Anne (Baron-Cohen et al., 1985) and Smarties (Wimmer and Perner, 1983) tests. Moreover, the stories used in FBT are strictly templated, which further benefits LLMs’ generalization after being trained on existing samples. For example, they often involve PersonA moving something to a new location without PersonB knowing, and then asking about PersonB’s belief. It’s normal for LLMs to surpass humans in specific tasks and abilities, but this shouldn’t lead us to conclude that LLMs possess a human-level ToM. Instead, we should consider a general and robust ToM across tasks and abilities as the criterion for assessment.
UOT: Unexpected Outcome Test SIT: Scalar Implicature Task PST: Persuasion Story Task FBT: False Belief Task AST: Ambiguous Story Task HT: Hinting Test SST: Strange Story Task FRT: Faux-pas Recognition Test
LLMs’ ToM Performance GPT-4 series stands out with superior results in LLMs, and surpasses the second-tier GPT-3.5-Turbo series by over 10% points. Among the open LLMs, Qwen-14B-Chat impressively outperforms other LLMs and even closely matches GPT-3.5-Turbo. Mistral-7B is also notable, not only surpassing ChatGLM3-6B but even outperforming LLaMA2-13B-Chat and Baichuan2-13B-Chat. The MoE-type Mixtral8x7B does not outperform Mistral-7B, which contrasts to its task-solving performance and also underscores the unique nature of ToM evaluation.
Vanilla vs. CoT Prompting Both task- and ability-oriented results indicate that CoT prompting almost never improves ToM performance and can even lead to declines. We believe CoT reasoning primarily serves to provide intrinsic context, breaking down complex tasks into simpler ones. However, if an LLM inherently lacks a robust ToM, its CoT decomposition process may struggle to align with genuine human cognitive reasoning, thereby failing to boost performance. We present a failure case of CoT prompting in Appendix I.
Differences Across ToM Tasks In Table 2, we calculate the “Grand Mean” performance of all LLMs on specific tasks. Notably, the faux-pas recognition test (FRT) shows relatively high accuracy, partly because FRT includes a notable number
of True/False questions, reducing its difficulty. The unexpected outcome test (UOT) follows, mainly because a substantial portion of UOT involves direct emotion understanding tasks, a domain where NLP has abundant datasets, thus LLMs perform well. The poorest performance is observed in the scalar implicature task (SIT), which requires understanding a character’s grasp of information based on quantifiers, involving mathematical reasoning where LLMs are typically weak. The persuasion story task (PST) also proves challenging, focusing on the complex psychological activities of characters persuading others and requiring a comprehensive understanding of social contexts.
Differences Across ToM Abilities Further exploring Table 3, the best-performing ability dimensions are Non-literal Communication (overlaps with FRT) and Emotion, consistent with the taskoriented results. The weakest category is Knowledge (overlaps with SIT)—not referring to the LLMs’ own knowledge but testing whether LLMs understand the information propagation between characters in stories, which LLMs often struggle with. In Appendix G, we delve deeper into LLMs’ performance across the 31 abilities.
The Necessity of Bilingual Inventory The averaged results show LLMs performing similarly in Chinese and English, but this does not imply that
LLMs perform identically in both languages. To assess the significance of any differences, we first conduct a Kolmogorov-Smirnov Test (AN, 1933) on Chinese and English results, finding that neither task-oriented nor ability-oriented results follow a normal distribution (p<0.01). Thus, we employ the Wilcoxon Signed-Rank Test (Wilcoxon, 1947) to examine the differences caused by languages, yielding p=0.019 (task) and p=0.016 (ability), both below the significance threshold of 0.05. Obviously, although the bilingual data are generated through translation with minimal semantic differences, LLMs exhibit significant performance discrepancies between Chinese and English contexts.

heading: 4.3 In-Depth Analysis
text: A Harder Coherent Test for LLMs When evaluating task-oriented ToM performance, we use the simple average accuracy across all questions at that time. In fact, for all ToM tasks except PST, most stories are associated with multiple coherent questions probing different aspects of the social scenario within the story. A common and intuitive perspective is that for a story, subjects need to answer all coherent questions correctly to demonstrate a complete understanding (Kim et al., 2023), rather than just making educated guesses. Therefore, we further design a more challenging story-level coherent test for evaluation, where an LLM is considered to have failed a story if it incorrectly answers any question associated with that story.
We present the average performance variation of all LLMs in Figure 4. Under the coherent test, the ToM performance of all LLMs drops dramatically by at least 20% points, while humans only experience a decrease of 13.6% points. This widens the performance gap between the top-performing LLMs and humans from the original 10.1% to 16.2%, highlighting LLMs’ limitations in fully comprehending social scenarios like humans.
LLMs Fail on Trivial ToM Questions We now delve into an ability where LLMs perform particularly poorly: Knowledge/Knowledge-Pretend Links. It’s important to note that this deliberate selection does not introduce bias against LLMs because, methodologically, failing even one ToM task implies the absence of a general ToM ability
(Shapira et al., 2023). Specifically, this ability tests whether subjects can understand that a person or object cannot imitate something they are unaware of. This task is intuitive and straightforward for humans (achieving 93.3% accuracy) but proves difficult for LLMs. GPT-4-1106 and GPT-4-0613 achieve only 26.7% and 3.3% accuracy on English samples, respectively, which is nearly equivalent to or even worse than random guessing (full results can be found in Table 21).
To understand the reasons behind this poor performance, in Figure 5, we visualize the decisionmaking process of an open LLM, ChatGLM3-6B, of which the weights are accessible. Specifically, when ChatGLM3-6B generates an incorrect option A, we average the attention scores across the attention heads in 20 layers, identifying the top-10 keywords that ChatGLM3-6B focuses on when answering the question. Clearly, LLMs still rely on semantic associations to answer ToM questions, aligning with previous research on LLMs’ illusory theory of mind (Kim et al., 2023). For comparison, we also ask a student to provide the top-10 keywords. The results show that the student could correctly answer the question after reading only half of the story, further emphasizing the distinction between human and LLMs in ToM capabilities.

heading: 5 Related Work
text: As LLMs have reached and sometimes even surpassed human performance in task solving, researchers have started to explore whether LLMs possess a ToM and exhibit human-like social intelligence. Sap et al. (2022) tested GPT-3 with SocialIQA (Sap et al., 2019) and ToMi (Le et al., 2019), finding its performance significantly lower than humans. However, when it comes
to instruction-tuned davince-002/003, Kosinski (2023) reported their performance as comparable to children aged 7 and 10, respectively, according to the experiments on 40 samples testing false beliefs. Bubeck et al. (2023) discovered that GPT-4 achieved impressive results on the Sally-Anne test with 10 samples. van Duijn et al. (2023) found that GPT-3.5/4 outperformed students aged 7 to 10 on various early inventories. This sparked some discussions about the “theory of mind may have spontaneously emerged in large language models.”
However, opposing views emerged quickly. Ullman (2023) and Shapira et al. (2023) showed that trivial alterations to test samples could drastically decrease LLM’s ToM performance, highlighting their reliance on spurious correlations. Jones et al. (2023) found poor performance on tasks like scalar implicature, while Ma et al. (2023a), Wu et al. (2023), and Kim et al. (2023) observed limited abilities in reconstructed/high-order/conversational false belief tests, respectively. The debate on whether LLMs possess ToM underscores the need for a holistic benchmark. Ma et al. (2023b) made an inspiring attempt by testing about 9 ToM abilities of simulated geometric bodies in virtual 2D grids, but this setting cannot reflect human activities in real-world social scenarios.
Different from existing ToM assessments for LLMs, our T MBENCH includes 8 tasks and 31 abilities, comprising 2,860 multi-choice questions built from scratch and covering diverse real-world social scenarios. T MBENCH establishes a systematic, automated, and original ToM benchmark that can continuously contribute to future research.

heading: 6 Conclusions
text: We introduce T MBENCH, the first systematic ToM benchmark for LLMs with an MCQ-type inventory entirely built from scratch. We reveal that even the most advanced LLMs significantly lag behind human performance in ToM. Further indepth analysis indicates that LLMs still struggle to comprehensively understand social scenarios, and tend to rely on semantic associations rather than human-like cognitive processes when addressing ToM questions. By proposing T MBENCH, we aim to provide an effective and efficient platform for evaluating the ToM capabilities of LLMs and spur further research into LLMs that innately understand ToM, thereby improving how artificial intelligence can serve us in a human-friendly way.
Limitations
We discuss the limitations of our work as follows.
Evaluation Protocol In T MBENCH, we aim to cover a broad range of well-defined ToM tasks frequently used in psychology. It is important to recognize that additional ToM tasks exist, such as the imposing memory task (Kinderman et al., 1998), which introduces unique evaluation protocols. Additionally, T MBENCH includes 31 abilities from the ATOMS framework that can be evaluated through textual modality. However, visual ToM tests are also classic and common in psychology, such as visual perspective taking - acknowledging that others have different visual percepts and adopting the visual perspective of another person. As multimodal LLMs like GPT-4V rapidly develop, ToM evaluations based on images and videos will also be applicable for LLMs. We advocate for future research to broaden the range of ToM assessments for LLMs, covering more tasks and abilities for a thorough evaluation.
Inventory Size Due to the difficulty and cost of manual construction, although T MBENCH contains a total of 2,860 samples, the minimum number for some abilities is only 20. This may lead to inadequate testing of specific abilities. There is an opportunity for future work to expand the sample size, allowing for a more detailed and accurate analysis of each ability.
Inventory Languages T MBENCH currently supports Chinese and English evaluation, where the Chinese data is originally created and the English data is obtained via translation. Although we have carefully designed the prompt for GPT-4 to complete the translation and manually reviewed all data, some cultural differences may still exist, e.g. fictitious person names, fictitious location names, etc. Future efforts could focus on localizing evaluation content and expanding language coverage to better reflect diverse cultural contexts.
LLMs and Prompting Methods In the experiments, we evaluate 10 popular representative LLMs, though additional LLMs could be included in future analyses. We use vanilla and CoT prompting methods for evaluation, while the recently emerging prompting methods such as “perspective taking” (Wilf et al., 2023) and “foresee and reflect” (Zhou et al., 2023) could also be explored to enhance the LLMs’ ToM performance.
ToM Understanding vs. Application Our examination of LLMs’ ToM capabilities in T MBENCH is conducted by posing ToM-related questions and evaluating the responses. It’s crucial to note that an LLM’s intrinsic awareness of ToM principles does not necessarily translate to effective extrinsic ToM performance in human-AI interactions. These aspects represent progressive steps—from understanding to application. While T MBENCH aims to address the foundational aspect, future research could explore more complex interactions, e.g., considering LLMs as autonomous social agents and assessing their dynamic ToM abilities in customized social scenarios.
Ethical Considerations
We here elaborate on the potential ethical issues.
ToM and Anthropomorphism The theory of mind is a unique social cognitive ability inherent to humans. Evaluating LLMs’ ToM capabilities through T MBENCH might lead to anthropomorphic interpretations, where LLMs are ascribed to human-like mental states. However, it’s crucial to clarify that our intention is not to humanize LLMs. Instead, we aim to enhance LLMs’ ability to understand and interpret human mental states and cognitive processes, thereby improving human-machine interaction. This endeavor is about bridging the communication gap, not erasing the fundamental differences between humans and machines.
Content Safety We authors have undergone professional training in psychology to understand ToM and reviewed broad psychological literature related to the tasks and abilities. We ensure that all test samples in T MBENCH are free from unsafe, toxic, biased, and offensive content. All social scenarios in test stories are fictitious and do not involve any specific countries, locations, events, characters, or other identifiable information.
Human Baseline To establish the human baseline for T MBENCH, we recruit graduate students who are native speakers of Chinese. This decision is made based on objective conditions available to us and does not reflect any intention of bias or unfairness towards individuals of any race, nationality, or other personal characteristics. Participants are selected based on the order of registration and get paid timely after the experiment. Participants complete experiments in an online document without providing any sensitive or personal information.

heading: Acknowledgments
text: This work was supported by the National Science Foundation for Distinguished Young Scholars (No. 62125604) and the NSFC Key Project (No. 61936010). This work was also supported by Tsinghua Precision Medicine Foundation, Tsinghua University - Beijing Tsingshang Architectural Decoration Engineering Co., Ltd. Joint Institute for Smart Scene Innovation Design, and China Postdoctoral Science Foundation (No. 2023M741944).

heading: B Details of Theory-of-Mind Tasks
text: In T MBENCH, we refer to the “Abilities in Theory of Mind Space (ATOMS)” framework (Beaudoin et al., 2020) and define ToM as “a person’s cognitive representation of self and others’ mental
states”. Under this definition, both representing mental states (such as inferring a person’s emotional reactions based on situations) and distinguishing one’s own and others’ mental states (such as understanding someone’s false beliefs) are considered ToM tasks. Thereby, we identify 8 social cognitive tasks from a broad view of psychology literature for ToM Evaluation.
Unexpected Outcome Test (Dyck et al., 2001) evaluates participants’ ability to infer characters’ mental states in scenarios with apparent discrepancies between emotion-eliciting contexts and the emotions actually elicited. Example data is present in Table 5.
Scalar Implicature Task (Goodman and Stuhlmüller, 2013) involves scenarios where a speaker uses terms like “some” to imply “not all” in situations of incomplete information. It tests participants’ ability to infer meanings beyond literal expressions in conversational contexts. Example data is present in Table 6.
Persuasion Story Task (Kołodziejczyk and Bosacki, 2016) assesses participants’ ability to understand and choose effective persuasion strategies, reflecting their understanding of how to influence others’ mental states and attitudes. Example data is present in Table 7.
False Belief Task (Wimmer and Perner, 1983) examines whether participants can distinguish between their own beliefs (true beliefs) and others’ beliefs (false beliefs) when they differ. Example data is present in Table 8.
Ambiguous Story Task (Bosacki and Wilde Astington, 1999) presents ambiguous social vignettes,
followed by questions that gauge participants’ understanding of others’ mental states, like emotions and beliefs, in uncertain situations. Example data is present in Table 9.
Hinting Test (Corcoran et al., 1995) assesses participants’ ability to infer mental states from indirect hints in social interactions, reflecting their understanding of implied meanings beyond literal statements. Example data is present in Table 10.
Strange Story Task (Happé, 1994) requires participants to infer characters’ mental states in stories that include complex social communications such as lies, white lies, misunderstandings, persuasion, irony, contrary emotions, jokes, double bluffs, pretense, figures of speech, forgetting, and appearance versus reality. Example data is present in Table 11 (part1) and Table 12 (part2).
Faux-pas Recognition Test (Baron-Cohen et al., 1999) tests participants’ ability to recognize when characters in social stories commit a faux pas, reflecting their understanding of social norms and others’ perspectives. Example data is present in Table 13.

heading: C Details of Theory-of-Mind Abilities
text: In T MBENCH, we further identify 31 social cognitive abilities in 6 dimensions according to the ATOMS (Beaudoin et al., 2020) framework in psychology for ToM Evaluation. Among them, 19 abilities are included in previous tasks, and example data can be found from the corresponding tasks. Additionally, we have constructed extra test data for the rest 12 abilities, and example data can be found in Table 14 (Emotion), Table 15 (Desire), Table 16 (Intention), and Table 17 (Knowledge).
Emotion involves the abilities to understand that situational factors influence people’s emotional states, that people can experience complex emotions, and that people can regulate emotional expressions. This dimension encompasses 7 abilities.
(I) Typical emotional reactions (Knafo et al., 2009): Inferring a person’s emotional reactions based on situations that typically elicit certain emotions/inferring a preceding event based on a person’s emotional reaction.
(II) Atypical emotional reactions (Denham, 1986): Inferring or explaining a person’s emotional reactions based on situations eliciting emotions that are atypical compared to what is usually expected.
(III) Discrepant emotions (Borke, 1971): Understanding that people may have discrepant feelings about an event.
(IV) Mixed emotions (Gordis et al., 1989): Understanding that people may feel mixed emotions or different emotions successively.
(V) Hidden emotions (Harris et al., 1986): Understanding that other people may hide their emotions.
(VI) Moral emotions (Pons and Harris, 2000): Understanding that negative feelings might arise following a reprehensible action.
(VII) Emotion regulation (Pons and Harris, 2000): Understanding that others might use strategies to regulate their emotions.
Desire involves the abilities to understand that people have subjective desires, preferences, and wants that influence their emotions and actions. This dimension encompasses 4 abilities.
(I) Discrepant desires (Repacholi and Gopnik, 1997): Understanding that different people may have discrepant desires.
(II) Multiple desires (Bennett and Galpert, 1993): Understanding the co-existence of multiple desires simultaneously or successively in one person.
(III) Desires influence on emotions and actions (Wellman and Bartsch, 1988): Understanding that people’s emotions and actions are influenced by their desires/preferences.
(IV) Desire-action contradiction (Colonnesi et al., 2008): Producing plausible explanations when actions contradict stated desires/preferences.
Intention involves the abilities to understand that people undertake actions in pursuit of goals and intentions. This dimension encompasses 4 abilities.
(I) Completion of failed actions (Meltzoff, 1995):
Understanding another person’s intent, as demonstrated by completing their failed action.
(II) Discrepant intentions (Killen et al., 2011): Understanding that identical actions/results can be achieved with different intentions.
(III) Prediction of actions (Phillips et al., 2002): Predicting people’s actions based on their intentions.
(IV) Intentions explanations (Smiley, 2001): Producing plausible intention explanations for different types of observed social events.
Knowledge involves the abilities to understand that others have access to different knowledge based on their perceptions, information they have received, or familiarity with things. This dimension encompasses 4 abilities.
(I) Knowledge-pretend play links (Aronson and Golomb, 1999): Understanding that someone who does not know something exists cannot engage in “pretend play” that incorporates that knowledge.
(II) Percepts-knowledge links (Pillow, 1989): Understanding that someone who does not have access to perceptual information (i.e., by looking, hearing, etc.) may not have access to knowledge.
(III) Information-knowledge links (Peskin et al., 2014): Understanding that someone who was not informed or is not familiar with something may not know.
(IV) Knowledge-attention links (Moll et al., 2006): Understanding that something new is more interesting to someone than something already known.
Belief involves the abilities to understand that people can hold beliefs about the world that are different from reality or different from one’s own beliefs. This dimension encompasses 6 abilities
(I) Content false beliefs (Hogrefe et al., 1986): Familiar container with an unexpected content: Understanding the false belief held by someone who never opened the container.
(II) Location false beliefs (Wimmer and Perner, 1983): Understanding the false belief held by someone who did not witness or was not informed of a displacement or change of action.
(III) Identity false beliefs (Flavell et al., 1986): Understanding that when something looks/sounds/smells like something else, a person may hold a false belief about its identity.
(IV) Second-order beliefs (Perner and Wimmer, 1985): Understanding the second-order belief or false belief held by someone who doe not know
somebody else was informed (e.g., of a misleading identity, a misleading location, etc.).
(V) Beliefs based action/emotions (Swettenham, 1996): Predicting another emotions or actions based on their stated beliefs/Inferring another person’s belief based on their stated action or emotion.
(VI) Sequence false beliefs (Brambring and Asbrock, 2010): Understanding the false belief created when a predictable sequence of stimuli is broken with the intrusion of an unexpected stimulus.
Non-literal Communication involves the abilities to understand that communication can convey meaning beyond the literal words spoken. This dimension encompasses 6 abilities.
(I) Irony/sarcasm (Sullivan et al., 1995): Understanding that other people may lie in order to be ironic/sarcastic.
(II) Egocentric lies (Happé, 1994): Understanding that someone may consciously lie in order to avoid a problem or to get its way.
(III) White lies (Happé, 1994): Understanding that someone may lie in order to spare another’s feelings.
(IV) Involuntary lies (Happé, 1994): Understanding that someone may tell a “lie” without knowing.
(V) Humor (Happé, 1994): Understanding that someone may tell a “lie” in order to make a joke.
(VI) Faux pas (Baron-Cohen et al., 1999): Ability to recognize faux-pas (social gaffe) situations.

heading: D Detailed Data Statistics
text: We present the detailed data statistics of 31 theoryof-mind abilities in Table 18. For each task, we collect ≥100 samples, covering 19/31 abilities. Additionally, for the rest 12 abilities not covered in the 8 tasks, we further collect ≥20 extra samples per ability.

heading: E Prompts for Data Translation
text: We present the prompts for Chinese to English translation of T MBENCH in Table 19.

heading: F Prompts for ToM Evaluation
text: We present the prompts for T MBENCH evaluation in Table 20.

heading: G Detailed Results of 31 Abilities
text: We present the detailed experimental results of 31 theory-of-mind abilities in Table 21. Overall, LLMs have not yet achieved a general and
robust human-level ToM performance. The GPT4 series shows the best performance in most specific abilities and even surpasses human performance in 9 out of 31 abilities. We attribute this to two main reasons: First, certain abilities, such as Emotion/Typical emotional reactions, relate to samples that are quite direct and simple. LLMs have encountered many similar training datasets, thus achieving high accuracy. Second, limited by human resources, some abilities have a relatively small number of samples, insufficient to showcase the general advantage of humans over LLMs.

heading: H Detailed Results of Coherent Tests
text: We present the detailed results of coherent tests on all LLMs with vanilla/CoT prompting in Table 22.

heading: I Case Study for CoT Prompting
text: In Figure 6, we show a failed CoT reasoning process of GPT-4-0613. It’s evident that while LLMs can break down questions into detailed parts, they still struggle to provide the correct answer.

heading: J Use of AI Assistants
text: We use ChatGPT to polish some of the content.
UOT: Unexpected Outcome Test SIT: Scalar Implicature Task PST: Persuasion Story Task FBT: False Belief Task AST: Ambiguous Story Task HT: Hinting Test SST: Strange Story Task FRT: Faux-pas Recognition Test

heading: Future Work
text: We advocate for future research to broaden the range of ToM assessments for LLMs, covering more tasks and abilities for a thorough evaluation. Inventory Size Due to the difficulty and cost of manual construction, although T MBENCH contains a total of 2,860 samples, the minimum number for some abilities is only 20. This may lead to inadequate testing of specific abilities. There is an opportunity for future work to expand the sample size, allowing for a more detailed and accurate analysis of each ability. Inventory Languages T MBENCH currently supports Chinese and English evaluation, where the Chinese data is originally created and the English data is obtained via translation. Although we have carefully designed the prompt for GPT-4 to complete the translation and manually reviewed all data, some cultural differences may still exist, e. g. fictitious person names, fictitious location names, etc. Future efforts could focus on localizing evaluation content and expanding language coverage to better reflect diverse cultural contexts. LLMs and Prompting Methods In the experiments, we evaluate 10 popular representative LLMs, though additional LLMs could be included in future analyses. We use vanilla and CoT prompting methods for evaluation, while the recently emerging prompting methods such as “perspective taking” (Wilf et al. , 2023) and “foresee and reflect” (Zhou et al. , 2023) could also be explored to enhance the LLMs’ ToM performance. ToM Understanding vs. Application Our examination of LLMs’ ToM capabilities in T MBENCH is conducted by posing ToM-related questions and evaluating the responses. It’s crucial to note that an LLM’s intrinsic awareness of ToM principles does not necessarily translate to effective extrinsic ToM performance in human-AI interactions. These aspects represent progressive steps—from understanding to application. While T MBENCH aims to address the foundational aspect, future research could explore more complex interactions, e. g. , considering LLMs as autonomous social agents and assessing their dynamic ToM abilities in customized social scenarios. Ethical Considerations
We here elaborate on the potential ethical issues. ToM and Anthropomorphism The theory of mind is a unique social cognitive ability inherent to humans. Evaluating LLMs’ ToM capabilities through T MBENCH might lead to anthropomorphic interpretations, where LLMs are ascribed to human-like mental states. However, it’s crucial to clarify that our intention is not to humanize LLMs. Instead, we aim to enhance LLMs’ ability to understand and interpret human mental states and cognitive processes, thereby improving human-machine interaction. This endeavor is about bridging the communication gap, not erasing the fundamental differences between humans and machines. Content Safety We authors have undergone professional training in psychology to understand ToM and reviewed broad psychological literature related to the tasks and abilities. We ensure that all test samples in T MBENCH are free from unsafe, toxic, biased, and offensive content. All social scenarios in test stories are fictitious and do not involve any specific countries, locations, events, characters, or other identifiable information. Human Baseline To establish the human baseline for T MBENCH, we recruit graduate students who are native speakers of Chinese. This decision is made based on objective conditions available to us and does not reflect any intention of bias or unfairness towards individuals of any race, nationality, or other personal characteristics. Participants are selected based on the order of registration and get paid timely after the experiment. Participants complete experiments in an online document without providing any sensitive or personal information