SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs

Authors: Yuling Gu, Oyvind Tafjord, Hyunwoo Kim, Jared Moore, Ronan Le Bras, Peter Clark, Yejin Choi
Primary area: datasets and benchmarks
Keywords: Theory of Mind, social reasoning, LLM benchmark, mental state, behavior, judgment, false belief

Abstract:
While prior work has explored whether large language models (LLMs) possess a "theory of mind" (ToM) - the ability to attribute mental states to oneself and others - there has been little work testing whether LLMs can implicitly apply such knowledge to predict behavior, or to judge whether an observed behavior is rational. Such skills are critical for appropriate interaction in social environments. Our approach to study such capabilities is to create a new dataset, called SimpleToM, containing concise, diverse stories (e.g., "The can of Pringles has moldy chips in it. Mary picks up the can in the supermarket and walks to the cashier."), each with three questions that test different degrees of ToM reasoning, asking models to predict (a) mental state ("Is Mary aware of the mold?"), (b) behavior ("Will Mary pay for the chips or report the mold?"), and (c) judgment ("Mary paid for the chips. Was that reasonable?"). To our knowledge, SimpleToM is the first dataset to systematically explore downstream reasoning requiring knowledge of mental states in realistic scenarios. Our experimental results are intriguing: While most models can reliably predict mental state on our dataset (a), they often fail to correctly predict the behavior (b), and fare even worse at judging whether given behaviors are reasonable, despite being correctly aware of the protagonist's mental state should make such secondary predictions obvious. We further show that we can help models do better at (b) and (c) via interventions such as reminding the model of its earlier mental state answer and mental-state-specific chain-of-thought prompting, raising the action prediction accuracies (e.g., from 49.5% to 93.5% for GPT-4o) and judgment accuracies (e.g., from 15.3% to 94.7% in GPT-4o). However, while this shows that models can be coaxed to perform well, it requires task-specific interventions, and the natural model performances remain low, a cautionary tale for LLM deployment. SimpleToM thus breaks new ground in probing real-world ToM reasoning, and reveals surprising, new insights about current model capabilities. We hope the dataset enables further exploration by the community into this critical area of model behavior.

Decision: reject
Reviewer scores: 8, 5, 3

Reviews and discussion:
<thread>
creation date:
1730199382135

thread_type:
official_review

<thread_object>
creator: reviewer_j5pr

<content>
summary:
The paper introduces SimpleToM, a dataset designed to test large language models' (LLMs) ability to apply theory of mind (ToM) reasoning in practical scenarios. While previous research has explored whether LLMs possess ToM capabilities, this work specifically examines how well models can use ToM knowledge to predict and evaluate behavior in realistic situations. They created SimpleToM dataset featuring concise stories with Mental state prediction, Behavior prediction, and Rationality judgment questions. They demonstrated that current LLMs show uneven ToM capabilities and developed interventions to improve model performance in applied ToM.

strengths:
* The paper introduces SimpleToM, a  well-designed dataset for evaluating both explicit and applied Theory of Mind capabilities in LLMs.
* The three-level assessment approach (mental state, behavior, judgment) is innovative and provides a more comprehensive evaluation framework than previous work.
* These results highlight an important gap in current LLM capabilities: understanding mental states in isolation versus applying that understanding to reason about behavior and make appropriate judgments.

weaknesses:
1. The biggest weakness of this work is that **the motivation for evaluating the Theory of Mind (ToM) capabilities of large language models (LLMs) is not clearly articulated.** The statement in lines 49-50, "Given the increasing use of LLMs in human interactions... it is crucial to assess their ToM capabilities," does not sufficiently explain why these evaluations are necessary. Specifically, it remains unclear how improvements in "prediction" and "judgment" in certain human behaviors or narrative scenarios would benefit the practical application of LLM systems or AI agents. 
2. Your evaluation is based on performance in "applied" ToM , but it lacks consideration of how LLMs can improve outcomes in real-world applications, **it means while LLMs can demonstrate the ability to make "predictions" and "judgments" about behavior, whether these abilities align with their capacity to meet human expectations in real applications such as AI agents is a separate issue**. There seems to be a gap between these two aspects, and I hope the authors can provide more discussion on this matter.
3. The "RESULTS AND ANALYSIS" section lacks sufficient analysis of your results. For example, in lines 316-317, "Only the latest o1-preview model ... on this question type (84.1%)," you should provide an analysis of why built-in inference-time reasoning tokens improve the model's performance.
4. A human baseline is needed for your evaluation. As you mentioned in lines 42-43, "People infer what others know, anticipate their actions, and expect them to choose cost-minimizing behaviors," I think you should include an evaluation of human performance on your test for comparison.

questions:
As mentioned in Weaknesses 1, I hope the authors can provide more details on the benefits of achieving a higher score in SimpleToM. Specifically, how does this relate to the performance of large language models or their applications in real-world scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730600135511

thread_type:
official_review

<thread_object>
creator: reviewer_jf3j

<content>
summary:
This paper exams whether LLMs can infer and more importantly, apply the mental state of character in stories. In particular, the author creates a dataset of stories where some key information is unknown for the character. Based on this foundation, the author proposes three tasks: mental state estimation, behavior prediction, and judgment of behavior. The experiment results show that LLMs without CoT can achieve high accuracy on the first task but fail in the others, which implies that LLMs fail to proactively make use of the mental state information for reasoning tasks. However, CoT drastically improves model's performance on the reasoning tasks.

strengths:
1. The author proposes a new task with a new dataset. The author use rigorous crowdsourcing to ensure the correctness of the label, although there are some systematic bias (see Weaknesses).
2. The experimental design is thorough, including a wide range of SOTA LLMs and exploring various prompting strategies.

weaknesses:
1. It’s unclear how this paper’s findings differ fundamentally from the large body of existing work that LLMs can acquire various types of knowledge (e.g., factual [1, 2, 3], ontological [4], or arithmetic skills (exactly mentioned in CoT paper)) but often struggle to apply this knowledge consistently in downstream tasks. Is reasoning about mental states inherently different in how it’s processed by LLMs compared to other types of knowledge?
2. It's also well-known that cot reasoning facilitate multi-step reasoning, and that task-specific instructions are generally helpful. Therefore, I'd expect the author to highlight any unique challenges that ToM reasoning poses for LLMs, distinguishing it from other forms of reasoning.
3.  The curated dataset appears biased. Every story assumes that characters are unaware of key information, and thus the ground-truth answer to the judgement question is always "reasonable". An ideal dataset would include a balanced mix of cases, with characters both aware and unaware.
4. While the author hints throughout the paper that low performance on SimpleToM with naive decoding may signal a risk for LLM deployment, this connection feels weak to me. I'd expect more concrete examples of applications where failing SimpleToM might lead to problematic outcomes.


[1] Allen-Zhu, Zeyuan, and Yuanzhi Li. "Physics of language models: Part 3.2, knowledge manipulation." arXiv preprint arXiv:2309.14402 (2023).

[2] Kazemnejad, Amirhossein, et al. "Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models." The 2023 Conference on Empirical Methods in Natural Language Processing.

[3] Berglund, Lukas, et al. "The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”." The Twelfth International Conference on Learning Representations.

[4] Wu, Weiqi, et al. "Do PLMs Know and Understand Ontological Knowledge?." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

questions:
1. See Weakness 1 & 4
2. Why do models generally achieve higher accuracy in behavior prediction than in judgment tasks? Both tasks should require the same level of reasoning
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644782440

thread_type:
official_review

<thread_object>
creator: reviewer_wyts

<content>
summary:
This paper proposes a new dataset called SimpleToM, designed to assess existing large language models on their Theory of Mind reasoning abilities. SimpleToM features well-crafted, simple, and straightforward stories that cover a variety of scenarios and includes three types of questions: mental state questions (testing whether the model can infer a character’s awareness of key information), behavior prediction questions (assessing if the model can predict a character’s actions based on mental states), and judgment questions (evaluating if the model can determine the reasonableness of actions). The paper experiments with various inference-time interventions to improve the performance of LLMs and concludes that these interventions only temporarily enhance model performance, indicating that there is still significant potential for LLMs to improve their Theory of Mind capabilities.

strengths:
- SimpleToM is a straightforward and comprehensive dataset for evaluating models' Theory of Mind abilities.
- The asymmetric information ensures diversity in the dataset, allowing for a more thorough assessment of models' capabilities, aligning better with the diversity characteristic of humans.
- This paper evaluates the Theory of Mind capabilities of both closed-source and open-source LLMs.
- The dataset is publicly available, and the experiments are highly reproducible.

weaknesses:
- The results and analysis in section5 are quite interesting, but each paragraph contains too many bolded words, making it difficult to follow the main points. Clarifying this section would improve readability.

- In Table 2, Llama-3.1-8B stands out as an outlier, compared with Llama-3.1-405B, it performs more than five times better in the judgment evaluation. I think this is an interesting case and I would expect to see more explanation for this outlier, such as potential reasons behind this unusual performance, would be more informative than simply labeling it as an outlier.

questions:
I'm curious about whether the humans personality or demographic characteristics in the mental state phase might influence the model's mental state evaluation. Some ablation experiments would be insightful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732315131615

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to reviews

comment:
Thank you for your review of our work! We’re grateful for the positive feedback on the following aspects of SimpleToM:
- Well-designed, well-crafted, simple, and straightforward stories for evaluating both explicit and applied Theory of Mind capabilities in LLMs.
- Comprehensive dataset covering a variety of realistic scenarios and information asymmetry, allowing for a thorough assessment of models’ capabilities.
- Rigorous crowdsourcing to ensure the correctness of labels, resulting in a high-quality dataset. 
- Comprehensive evaluation framework covering three levels of assessment - mental state, behavior, and judgment.
- Thorough experimental design, including a range of SOTA LLMs, both close-source and open-source LLMs, and exploring various prompting strategies.
- Results that highlight an important gap in the current LLM capabilities (understanding mental states in isolation vs applying that understanding to reason about behavior and make appropriate judgments)
- Dataset is publicly available and experiments are highly reproducible.

A concern raised by some reviewers is the general motivation behind evaluating the ToM capabilities of LLMs. To further emphasize the importance of the capabilities tested in SimpleToM, we provide concrete examples of applications where failing on applied ToM would be problematic:

**The case of a bad personal AI assistant** - failing to implicitly reason over other’s mental states to predict behavior:

Matt is a professional athlete. A growth hormone got into the supply chain for Bob's Burgers, where Matt regularly had dinner. Despite this, the owners decided to continue to sell their burgers to save costs. Imagine a personal AI assistant, having read complaints about the growth hormone contamination at Bob’s Burgers but failing to apply the understanding that others’ mental states may be different from their own (ToM), could reason that Matt, like the AI assistant, is also aware of this (awareness) and then incorrectly predicts that Matt will “refuse the burger due to the growth hormone” (behavior). This could then lead to undesirable consequences such as not being able to warn Matt in time to stop him from consuming the contaminated burger. In the untimely case of an unannounced blood test, Matt could even show up with positive traces of this very illegal growth hormone and be accused of doping. In this case, such an AI assistant, lacking a nuanced understanding of human awareness and motivations, might also falsely assume Matt's intentional wrongdoing. This highlights a critical limitation: without robust ToM, the AI fails to grasp that Matt’s actions could stem from unawareness rather than culpability, leading to flawed judgments that could unjustly tarnish his reputation or career.

**The case of a bad AI judge** - failing to make appropriate judgments of behavior:

Alice visited the supermarket to purchase some carrots to pack lunch for her husband Bob. After consuming the lunch Alice packed, Bob succumbed to a severe E. coli infection. It turned out the supermarket's carrots were contaminated with E. coli and were subsequently recalled. Imagine an AI judge is tasked with evaluating this case to decide whether Alice should be held responsible and imprisoned for Bob's death. Imagine a bad AI judge, failing to apply the understanding that others’ mental states may be different from their own (ToM), incorrectly assuming that Alice was aware of the E. coli and judging that Alice’s act of packing the contaminated carrots for her husband was wrong (judgment). This could then lead to undesirable consequences such as severely punishing the innocent Alice who didn’t know feeding Bob the carrots would kill him. In common law jurisdictions, whether a defendant is found guilty is often decided taking into account both mens rea (“guilty mind”) and actus reus (“guilty act”). Therefore, the ability to apply ToM is important for potential AI judges to appropriately assess whether an individual has a “guilty mind” when making key judgments, such as determining whether to convict someone.

We are excited about the possibility of future generations of models to improve on applied ToM in our dataset. This could pave the way for models to effectively interact with humans – for instance, serving as reliable personal AI assistants as well as trustworthy AI judges. We hope SimpleToM, as the first resource of its kind to measure LLMs’ capability on such diverse applied ToM scenarios, will help facilitate the community in pursuing exciting directions that bring us there.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732778653934

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Updated PDF

comment:
We thank all the reviewers for their helpful comments and suggestions! We have now updated the PDF with the following changes:

- Further describe motivational applications of applied ToM (details in Appendix A - Importance of Applied ToM)
- More explicit description of the difference between behavior prediction and judgment (Section 2.3: "As the inference graph in Figure 1 illustrates, the judgment question goes beyond behavior prediction as it requires two levels of implicit reasoning, first implicitly predicting the behavior of Person X, which itself relies on implicitly understanding their mental state.")
- More in-depth result analysis, such as Llama-3.1-8B outlier performance on judgment questions and the o1-preview's reasoning tokens (details in Appendix K - Further analysis on interesting behavior)
- Results on performance across personas (details in Appendix M  - Performance across personas)
- Fewer bolded words (Section 5)

We hope we have addressed the questions raised by the reviewers through our authors’ responses, as well as updates to the paper. And we’re happy to follow up if there are any remaining concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
SimpleToM is a new dataset designed to assess Theory of Mind (ToM) reasoning abilities in large language models (LLMs). The dataset features simple and concise stories covering a variety of scenarios where some key information is unknown to a character, creating a basis for ToM reasoning tasks. The dataset includes three distinct question types to test ToM reasoning: (i) Mental State Questions that test whether the model can infer a character’s awareness of key information, (ii) Behavior Prediction Questions that assess if the model can predict a character’s actions based on their mental state, and (iii) Judgment Questions that evaluate whether the model can determine the reasonableness or rationality of a character’s actions. Experimental findings show LLMs achieve relatively high accuracy in mental state prediction but struggle with behavior prediction and judgment tasks. It also shows that using CoT reasoning drastically improves model performance in behavior prediction and reasoning tasks. The authors experimented with interventions to improve performance but found that these improvements were only temporary, indicating room for further development in ToM reasoning. 

**Strengths identified**:
1. SimpleToM is a well-designed, straightforward, and comprehensive dataset for evaluating Theory of Mind (ToM) reasoning abilities in LLMs.

2. The experiments are rigorous and reproducible, involving a wide range of state-of-the-art (SOTA) LLMs and exploring various prompting strategies.

3. The three-level assessment framework (mental state prediction, behavior prediction, judgment) is innovative and provides a more comprehensive evaluation compared to previous work in ToM.

**Weaknesses to be addressed**:
1. The motivation for evaluating Theory of Mind (ToM) capabilities in LLMs is not clearly articulated. Why is assessing ToM abilities in LLMs important for real-world applications or AI agent interactions? The connection between low performance on SimpleToM and potential risks in real-world LLM deployment is weak. The paper could benefit from more concrete examples of practical applications where failing ToM tasks might lead to problematic outcomes. 

2. The findings do not seem to differ fundamentally from previous research that demonstrates LLMs' struggles in applying acquired knowledge (e.g., factual, ontological, or arithmetic knowledge) in downstream tasks. More specifically, The paper does not highlight any unique challenges specific to ToM reasoning that distinguish it from other forms of multi-step reasoning or knowledge application. 

3. Every story assumes that characters are unaware of key information, making the ground-truth answer to the judgment question always "reasonable". A more balanced dataset, with scenarios where characters are both aware and unaware of key information, would be ideal.

4. The paper does not sufficiently analyze why specific models or interventions (e.g., inference-time reasoning tokens) improve performance. Also, the paper does not include a human performance baseline for comparison, which is important given the claim that humans infer mental states, predict behavior, and make judgments effectively.