SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs

## Abstract

Abstract While prior work has explored whether large language models (LLMs) possess a “theory of mind” (ToM) -
the ability to attribute mental states to oneself and others - there has been little
work testing whether LLMs can implicitly apply such knowledge to predict behavior, or to
judge whether an observed behavior is rational.
Such skills are critical for appropriate interaction in social environments.
Our approach to study such capabilities is to create a new dataset, called SimpleToM, containing concise,
diverse stories (e.g., “The can of Pringles has moldy chips in it. Mary picks up the
can in the supermarket and walks to the cashier.”), each with three questions that test
different degrees of ToM reasoning, asking models to predict
(a) mental state (“Is Mary aware of the mold?”),
(b) behavior (“Will Mary pay for the chips or report the mold?”), and
(c) judgment (“Mary paid for the chips. Was that reasonable?”).
To our knowledge, SimpleToM is the first dataset to systematically explore
downstream reasoning requiring knowledge of mental states in realistic scenarios.
Our experimental results are intriguing: While
most models can reliably predict mental state on our dataset (a), they often fail to correctly
predict the behavior (b), and fare even worse at judging whether given behaviors are reasonable (c),
despite being correctly aware of the protagonist’s mental state should make
such secondary predictions obvious. We further show that we can help models do
better at (b) and (c) via interventions such as reminding the model of its earlier mental
state answer and mental-state-specific chain-of-thought prompting, raising
the action prediction accuracies (e.g., from 49.5% to 93.5% for GPT-4o)
and judgment accuracies (e.g., from 15.3% to 94.7% in GPT-4o). However, while this shows
that models can be coaxed to perform well, it requires task-specific interventions,
and the natural model performances remain low, a cautionary tale for LLM deployment.
SimpleToM thus breaks new ground in probing real-world ToM reasoning, and
reveals surprising, new insights about current model capabilities. We hope
the dataset enables further exploration by the community into this critical
area of model behavior.

## 1 Introduction

People infer what others know, anticipate their actions, and expect them to choose cost-minimizing behaviors (Gergely & Csibra, ; Liu & Spelke, ). Performing such social reasoning involves attributing mental states to oneself and others, an ability widely known as Theory of Mind (ToM) (Premack & Woodruff, ). ToM has been extensively studied in psychology in a range of scenarios, for instance, studies of manipulation, secrecy, deception, lying, and misleading behavior (Doherty, ). Examples of classical tests in developmental psychology include the unexpected transfer false belief task, e.g., the Sally-Anne task (Baron-Cohen et al., ), and the unexpected contents false belief task, e.g., the Smarties task (Perner et al., ). Given the increasing use of LLMs in human interactions and as decision-making agents within complex, human-centered environments, it is crucial to assess their ToM capabilities.

Figure: Figure 1: To allow for a nuanced analysis of models’ neural ToM abilities, SimpleToM covers both explicit ToM (a) and applied ToM (b, c) question types. SimpleToM measures the ability of LLMs to (a) infer the character’s mental state, specifically information awareness, (b) anticipate their likely next behavior in the given situation, and (c) make appropriate judgment of the character’s behavior that correctly accounts for their mental state.
Refer to caption: x1.png

Current datasets for assessing ToM in LLMs are limited by their reliance on the classical Sally-Anne task or templated variants of it (Le et al., ; Nematzadeh et al., ; Wu et al., ; Xu et al., ). These datasets have significant shortcomings: (i) limited diversity in how information asymmetry arises (see related work in Section 6 for examples across existing datasets), (ii) explicit use of percept and mentalizing verbs like “sees” and “thinks” which serve as trigger words for models to realize that these are important aspects, removing the need for implicit commonsense inferences about relevant percepts or beliefs, and (iii) limited exploration of applied ToM, such as judgment of behavior which requires implicit reasoning about mental state.

The goal of SimpleToM is to go beyond the focus on mental state inference, to also assess how well models can predict others’ behavior based on that understanding, and make judgments of appropriateness of behavior.
Each story in SimpleToM is paired with three types of questions targeting these abilities (Figure 1), with a total of 1147 stories and 3441 questions in daily life settings. Our
analysis using SimpleToM exposes a significant gap in model performance between explicit and applied ToM questions (Lee et al., ). We find that frontier models perform well on explicit ToM questions (directly querying for information about “mental state”, i.e., information awareness). However, this success does not extend to applied ToM (“behavior” and “judgment” questions).
Scenario-specific analysis highlights the need for diverse testing scenarios to rigorously assess neural ToM
and indicates areas for improvement even in strong models like o1-preview.
Our experiments with various inference-time interventions show that the performance of applied ToM improves with increased guidance.
However, the need for highly specific guidance and the high inference cost (e.g., CoT using hundreds of tokens) required for models to reason through simple two-sentence stories in SimpleToM, suggests that the inherent ToM reasoning abilities of advanced LLMs still have significant room for improvement.
The results showing that frontier models still lack the ability to independently apply ToM skills in downstream tasks—such as anticipating others’ behavior and making judgments—call for caution when using them in real-world applications.

Our contributions and findings are:

- •
A new dataset, SimpleToM, for testing core abilities of LLMs in explicit and applied ToM.
- •
Current frontier models have decoupled capabilities between predicting someone’s information awareness in a situation (explicit ToM, which they excel at), and utilizing it to predict and judge someone’s behavior (applied ToM, which they perform poorly at).
- •
Models can be explicitly guided to perform applied ToM, e.g., through custom chain-of-thought prompting, but this is a fragile workaround, and being competent on SimpleToM without such guidance is thus an important measure of a model’s capabilities.

We make our SimpleToM dataset and the full evaluation data for the analyzed models publicly available at [https://huggingface.co/datasets/allenai/SimpleToM](https://huggingface.co/datasets/allenai/SimpleToM).
This will allow researchers to build on top of our work in studying the neural ToM capabilities of LLMs in general, as well as further exploiting the diversity of situations covered by SimpleToM in exploring applied ToM.

Figure: Figure 2: We leverage the generative strength of language models to obtain concise stories with varied entities and diverse situations, suitable for testing different levels of ToM reasoning. The generated stories (and answer options) were then rigorously filtered by careful human annotators who passed a strict qualification test. The result is a high-quality and diverse dataset, SimpleToM.
Refer to caption: x2.png

## 2 SimpleToM design

We design the stories in SimpleToM to contain diverse types of information asymmetry, using a concise format and associated with specific question types testing explicit and applied ToM.

### 2.1 Different types of information asymmetry

To expand beyond the classical false belief task, we seed the creation of SimpleToM with ten diverse scenarios where information asymmetry occurs naturally (Table 1).
This is inspired by social psychology literature to cover asymmetries like manipulation, deception, secrecy, lying, and misleading behavior (Doherty, ), seen in real-world contexts like sales of “lemon” products, where items with hidden flaws are purchased due to a lack of information (Akerlof, ). These are under-examined in existing ToM tests. We further describe the scenarios with examples in Table 7 (Appendix E.1).

**Table 1: The ten broad scenarios used to seed the generation of stories in SimpleToM. Each scenario describes a type of information asymmetry that occurs naturally in the real world.**
| Scenario | Reason for information asymmetry |
| --- | --- |
| food item in grocery store | Food items bought in grocery stores cannot be closely examined for their quality before checking out |
| provider info healthcare | Efficacy of healthcare products cannot be closely examined or verified before purchase |
| true property pretentious labels | Subtle properties of products cannot be closely examined or verified |
| behind the scene service industry | Questionable behind-the-scenes practices in the service industry are not observed by customers |
| inside reuse labeled containers | What is inside labeled (opaque) containers cannot be observed before opening the container |
| unobserved unethical actions | Unethical actions not observed are not known |
| inside containers for personal belongings | What is inside (opaque) containers for personal belongings cannot be observed before opening the container |
| seller info in second hand market | Hidden flaws in second-hand items bought cannot be observed before the purchase |
| hidden body part feature | Body features hidden under clothing cannot be observed |
| locked devices accounts | Details in locked devices or accounts cannot be observed by others |

### 2.2 Simple story format without explicit percepts or mental states

The SimpleToM example story from Figure 1 reads: The can of Pringles has moldy chips in it. Mary picks up the can in the supermarket and walks to the cashier. Each story has exactly two sentences, where the first sentence introduces a key information about something (Object/Person/Action Z), while the second sentence presents the main subject of the story (Person X) doing something with Object/Person/Action Z while being unaware of the key information. The list of story elements are:

- •
Key Information: involves something unexpected which Person X is unlikely to know or perceive, e.g., The can of Pringles has moldy chips in it.
- •
Object/Person/Action Z: the subject of the key information (e.g., can of Pringles)
- •
Person X: person unaware of the key information (e.g., Mary)
- •
Person Y (optional): any other character(s) needed for the story

We impose the constraint that Person X’s unawareness of the key information should be implicit (e.g., avoid explicit use of perception or mentalizing words such as “see”, “notice” or “belief”). This design encourages models to read between the lines and make commonsense inferences over the given situations and infer characters’ mental states in a more realistic manner, bringing us closer to realistic daily life use cases of ToM. (E.g., you cannot see through a Pringles can; you would not know about a cheating event if you were not present.)

To support formulating the behavior prediction question (Section 2.3), we also generate options for what might happen next:

- •
Unaware behavior: A likely next action by Person X given that they are unaware of the key information
- •
Aware behavior: A likely next action by Person X if they were somehow aware of the key information after all (counterfactual)

### 2.3 Questions testing explicit and applied ToM

We use three types of questions (Figure 1) to probe a model’s grasp of each story, covering both explicit theory of mind (conceptual knowledge about others’ mental states; i.e., via (a) mental state question about information awareness) and applied theory of mind (the ability to use theory of mind in real-life contexts; i.e., via (b) behavior and (c) judgment questions) (Lee et al., ).

Mental state (MS) question about information awareness:
We test ability of models to infer mental states, specifically information awareness, through a simple yes/no question (Is <Person X> likely to be aware that "<key information>"?)
To infer whether a character is aware of something in SimpleToM stories, a model has to make implicit commonsense inferences about what the character can perceive or know in the given situation (including commonsense reasoning about physical objects, space, intent, goals of others, and so on).

Behavior prediction question:
This question asks which of two possible actions the main subject (Person X) is likely to perform next.
For instance, beyond answering that a person shopping for chips in the supermarket is unlikely to know
that “the can of Pringles has moldy chips in it”, a model that successfully applies this inference for behavior prediction should also infer that a person who picked up such a can in the supermarket would likely “pay for the chips” rather than “report the moldy chips.” To answer these questions correctly, models need to reason over the situation to infer the mental state of character, and realize how the character’s lack of awareness of the key information would impact their likely next action.

Judgment question:
The judgment question specifies that “correct” action was taken (rather than the incorrect one) and asks if this was a reasonable choice. Similar to the behavior question, the judgment question requires inferring and applying implicitly conveyed information like percepts, beliefs, desires, and knowledge.
People’s mental states are an important factor to consider in making appropriate judgments
of their behavior (Jara-Ettinger et al., ; Schein & Gray, ). For instance, buying a can of Pringles that has moldy chips in it is not a
reasonable action if the person knows about the moldy chips.
However, it is a perfectly reasonable (and expected) behavior if
this piece of key information is not a part of the person’s mental state.

## 3 SimpleToM creation

### 3.1 Generating diverse stories

Specifically, the construction of SimpleToM consists of the following steps:

- Step 1:
Manually create one example seed story for each scenario.
- Step 2:
For each scenario, using the seed story as example, prompt the LLM to suggest 10 diverse sets of entities compatible with an information asymmetry. (See prompt in E.5.)
- Step 3:
For each set of suggested entities, along with the seed story, prompt the LLM to write three new stories at different levels of “severity.” With each story, also generate likely next “unaware” and “aware” behaviors (see Section 2.2). Appendix  E.4 provides further details.

We went through two rounds of this process. First, we used GPT-4 and Claude-3-Opus(^1^11See Table 6 for exact models used)
to generate a total of 1200 stories.(^2^2210 scenarios * 2 models * 10 entities per model * 3 severities * 2 models to generate stories)
After annotating and filtering this initial set (Section 3.2), we picked a new set of top-scoring seed stories and sourced 10 additional sets of entities from each of GPT-4o and Claude-3.5-Sonnet. We used these two newer models to generate stories for all 40 sets of entities, for a total of 2400 more stories. By using several generator models, varied entities and different seed stories, the resulting stories in SimpleToM have a wide range of information asymmetries instantiated in different real-world situations, effectively broadening neural ToM tests beyond traditional settings (Section 6). These contexts also allow for nuanced and implicit traits (e.g., buyers would avoid products with defects if they know about them).

### 3.2 Strict quality control on stories that goes into SimpleToM

We gather human annotations on each story (and unaware/aware next actions).
We asked annotators four questions for each story, summarized in Figure 2. This process verifies that the key information in each story is something that Person X has false belief about. We also carefully verify that the next likely “unaware action” is appropriate if and only if Person X is unaware of the key information. We similarly verify the “aware action,” for the counterfactual situation where Person X is somehow aware of the key information. Appendix D provides further details about the crowdsourcing procedure, with instructions, examples and question templates.

Our annotators passed a rigorous qualification test (Appendix D.2) and met other high-standard requirements (Appendix D.3). Only stories for which all crowdworkers (3) judged all aspects to be valid were included in SimpleToM.(^3^33See more details in Appendix D.4.)
This results in 1147 stories (out of the original 3600) in the final SimpleToM dataset. Table 8 (Appendix E.2) provides statistics and further details for SimpleToM.

## 4 Experimental setup

We evaluate SimpleToM on ten frontier LLMs from different sources and with different levels of capabilities: Llama-3.1-8B, Llama-3.1-405B, (Dubey et al., ) Claude-3-Haiku, Claude-3-Opus (Anthropic, ), Claude-3.5-Sonnet,(Anthropic, ) GPT-3.5, GPT-4, GPT-4o, o1-mini, and o1-preview (OpenAI, ; ) (refer to Appendix C
Table 6 for more details). We use the most deterministic setting with a generation temperature of 0.(^4^44Except the o1 models which do not allow adjusting the temperature or system prompt.)

We use SimpleToM to investigate the following research questions:

- 1.
How well can models (i) infer characters’ mental states, (ii) anticipate characters’ behavior and (iii) make appropriate judgments, requiring the use of ToM inferences?
- 2.
How does the ToM performance of models differ across scenarios?
- 3.
How can the performance on the behavior and judgment questions improve with various levels intervention during inference, such as patching mental state inference in the prompt (MS remind), guiding the model via system prompt (SysP), and encouraging reasoning/thinking before answering (CoT)?

## 5 Results and analysis

### 5.1 Frontier LLMs can infer mental states, but struggle to use it

The overall evaluation results on SimpleToM for the 10 models are summarized in Table 2, spanning the different question types (as detailed in Section 2.3). We analyze models’ performance for each type of question below. Note that these are binary questions where random performance is 50%.

**Table 2: Evaluation results for SimpleToM on the different question types. Models are generally proficient in explicit ToM questions (directly querying about “mental state”, i.e., information awareness) but this success does not transfer to applied ToM (“behavior” and “judgment” questions).**
| model | mental state | behavior | judgment | average |
| --- | --- | --- | --- | --- |
|  | (Explicit ToM) | (Applied ToM) | (Applied ToM) |  |
| GPT-3.5 | 36.5 | 7.6 | 29.1 | 24.4 |
| Claude-3-Haiku | 87.2 | 23.6 | 16.7 | 42.5 |
| o1-mini | 87.8 | 44.8 | 27.0 | 53.2 |
| GPT-4o | 95.6 | 49.5 | 15.3 | 53.5 |
| Llama-3.1-405B | 97.8 | 58.2 | 10.0 | 55.4 |
| Claude-3-Opus | 98.3 | 64.4 | 9.6 | 57.4 |
| GPT-4 | 96.6 | 63.0 | 19.5 | 59.7 |
| Llama-3.1-8B | 88.1 | 38.5 | 54.6 | 60.4 |
| Claude-3.5-Sonnet | 97.9 | 67.0 | 24.9 | 63.3 |
| o1-preview | 95.6 | 84.1 | 59.5 | 79.7 |

Mental state (MS) question about information awareness: Our results (Table 2, “mental state” column) show that reasoning over implicit information in given situations to infer mental states is still challenging for models like GPT-3.5 (36.5% accuracy), while newer and/or bigger models like Claude-3-Haiku, o1-mini, and Llama-3.1-8B perform reasonably well (around 88%).
In fact, all recent frontier models are proficient at inferring characters’ awareness in our dataset – GPT-4o, Llama-3.1-405B, Claude-3-Opus, GPT-4, Claude-3.5-Sonnet, and o1-preview all achieved accuracies of more than 95%. This result also confirms the quality of our dataset, in that characters’ mental states in SimpleToM stories are implicit but reasonably easy to infer, as designed.

Behavior prediction: On behavior prediction questions (Table 2, “behavior” column), smaller and older models perform extremely poorly (with GPT-3.5 achieving only 7.6% accuracy and several other models scoring less than 40%).
Even for the larger models, like Llama-3.1-405B, Claude-3.5-Sonnet, and GPT-4o, performance on the behavior prediction task is much worse than on the mental state task with at least a 30% performance drop. This large inconsistency suggests that while frontier LLMs may have the right conceptual knowledge/information about others’ mental states when directly asked, they struggle to apply this knowledge in real-life contexts to make predictions about characters’ behavior.
Only the latest o1-preview model, with its built-in inference time reasoning tokens,(^5^55The o1 reasoning tokens make these models more like the chain-of-thought prompted versions of the other models below, although without any custom prompt. See Appendix H for discussion of the number of output tokens used by the o1 and the other models when using chain-of-thought.) manages a decent score on this question type (84.1%).

Judgment of behavior:
Our results (Table 2, “judgment” column) show that this additional, seemingly trivial, inference step (beyond the behavior prediction) makes the task much more difficult for all the models. Even the newest and largest models like Llama-3.1-405B, Claude-3.5-Sonnet, and GPT-4o, which all achieved accuracies of more than 95% on inferring characters’ mental state, consistently make inaccurate judgments of behavior, and their performance drop to far below random (with accuracies in the range of 10% to 24.9%). Llama-3.1-8B somehow manages around random performance (at 54.6%), as an outlier. Finally, o1-preview, the best-performing model on behavior prediction, also gets the best score on this judgment prediction task, but even so it reaches only 59.5%, far below its performance on the other questions. These results show that even the latest models lack the ability to effectively apply ToM reasoning in this kind of judgment task.

Figure: Figure 3: Considering the sequence of first predicting mental state, then behavior and finally judgment, we can record a failure for the first mistake. The “fail at behavior prediction” and “fail at judgment” parts can be considered inconsistent predictions by the model, since it got the associated mental state (and behavior prediction) questions correct.
Refer to caption: x3.png

Overall ToM reasoning: To fully reason about the judgment question, requires reasoning about the behavior prediction, which relies on understanding the mental state (information awareness) question. We can visualize this by recording the first failure on the three “mental state” $\rightarrow$ “behavior” $\rightarrow$ “judgment” questions. Figure 3 shows the distribution of such failures, showing the proportionally large cases failing at the behavior and judgment steps. For most models the green segment, representing full understanding of the stories in terms of ToM reasoning, is very small.

### 5.2 Not all scenarios are made equal

In Figure 4 we show how model performance varies across select scenarios (Appendix J, Figures 11 and  12 cover more scenarios across models).
Performance can vary wildly across scenarios, e.g., the behavior prediction score is high across models for “provider info healthcare” compared to other scenarios. This highlights the limitations of testing ToM on just one type of question or scenario, emphasizing the need for a diverse dataset like SimpleToM.

Figure: Figure 4: Comparing performance for all three question types across select scenarios and models. Each bar represents the overall accuracy. The mental state accuracy is generally near 100%, while behavior prediction and judgment accuracies are often much lower.
Refer to caption: x4.png

Looking at the judgment scores, the scenarios “inside reuse labeled containers” and “inside containers for personal belongings” are better (but still low) than other broad scenario types for Llama-3.1-405B, Claude-3.5-Sonnet and GPT-4o. This could potentially be attributed to instances in these categories being more similar to the original “Smarties test” where people have false belief due to the opaque nature of the container, combined with misleading label or unconventional use of the container. This result further highlights the importance of SimpleToM covering diverse scenarios beyond those in classical ToM tests, to ensure that we are effectively testing the ToM reasoning abilities of models (rather than models’ ability to match similar situations in the training data). Analysis by scenario also reveals a wide diversity of other trends across scenarios regarding where different models fail/succeed. We refer interested readers to Appendix J for further discussion.

### 5.3 No applied ToM in LLMs? Exploring the rabbit hole of human hand-holding

We explore four different inference interventions to help LLMs answer questions requiring applied ToM.(^6^66All detailed prompts are provided in Appendix G.) Apart from the first intervention, we focus these experiments on the strongest or latest models from each source (and we do not consider the o1 models in this section).

**Table 3: Evaluation results for SimpleToM where models are reminded in the prompt about their answer to the mental state question (MS). We see from the difference between the none and MS remind columns that even frontier LLMs utilize such reminders to do much better on behavior prediction. Apart from Claude-3.5-Sonnet, this is not enough to bring accuracies beyond random on the judgment question.**
| model | MS | behavior | judgment | average |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| reminder question | none | none | MS remind | none | MS remind | none | MS remind |
| GPT-3.5 | 36.5 | 7.6 | 12.2 | 29.1 | 53.0 | 24.4 | 33.9 |
| Llama-3.1-8B | 88.1 | 38.5 | 59.8 | 54.6 | 27.2 | 60.4 | 58.4 |
| Claude-3-Haiku | 87.2 | 23.6 | 61.1 | 16.7 | 30.7 | 42.5 | 59.7 |
| Llama-3.1-405B | 97.8 | 58.3 | 89.5 | 10.0 | 25.8 | 55.4 | 71.1 |
| GPT-4o | 95.6 | 49.5 | 82.8 | 15.3 | 42.2 | 53.5 | 73.6 |
| Claude-3-Opus | 98.3 | 64.4 | 93.5 | 9.6 | 41.3 | 57.4 | 77.7 |
| GPT-4 | 96.6 | 63.0 | 90.1 | 19.5 | 54.0 | 59.7 | 80.2 |
| Claude-3.5-Sonnet | 97.9 | 67.0 | 96.9 | 24.9 | 84.1 | 63.3 | 93.0 |

1. Mental state reminder (MS): Here we remind the model of its answer to the mental state question by including this question (with the model’s answer) in the prompt. This also puts the model on alert that “awareness” might be relevant. Table 3 summarizes the effect of this when answering the behavior and judgment questions.(^7^77Appendix G.1 provides more details on the prompt used.) On the behavior prediction questions, this intervention results in substantial boosts in accuracy, for instance, from 58.3% to 89.5% for Llama-3.1-405B, and from 49.5% to 82.8% for GPT-4o. On Claude-3.5-Sonnet, the performance increases by almost 30%
to 96.9%, largely closing the gap between the mental state and behavior prediction question scores. However, on the judgment questions, the performance boost is much more modest, and most models still score below or at random, except for Claude-3.5-Sonnet where this intervention brings the score up from 24.9% to a reasonable 84.1%. This highlights how such interventions, while seemingly effective in some cases, are generally fragile band-aids with limited scope.

**Table 4: Evaluation with guidance via custom system prompts SysP and SysP* (where SysP* has more explicit guidance regarding awareness). The MS column shows the mental state accuracy for comparison. In general, this intervention is less effective than the mental state reminder.**
| model | MS | behavior prediction | judgment of behavior | average |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| system prompt | none | none | SysP | SysP* | none | SysP | SysP* | none | SysP | SysP* |
| GPT-4o | 95.6 | 49.5 | 47.3 | 68.6 | 15.3 | 14.9 | 20.5 | 53.5 | 52.6 | 61.6 |
| Llama-3.1-405B | 97.8 | 58.2 | 64.5 | 83.3 | 10.0 | 9.9 | 15.4 | 55.4 | 57.4 | 65.5 |
| Claude-3.5-Sonnet | 97.9 | 67.0 | 68.9 | 88.9 | 24.9 | 27.1 | 52.2 | 63.3 | 64.6 | 79.7 |

2. System prompt guiding (SysP and SysP*): We also explore the effect of guiding the models to remember to account for mental state inferences by modifying the system prompt. We try two different prompts, SysP which includes the phrase "consider ... the mental state of all the entities involved" and SysP* which further includes the more direct hint "E.g., think carefully about what each person is aware or not aware of.".(^8^88Appendix G.2 presents the detailed prompts.) The results are summarized in Table 4. On behavior prediction, we see that generically guiding models to consider the mental state using SysP is only effective to a limited extent (accuracy changes ranging from -2.2% to +6.3%), while providing more explicit guidance with SysP* is more (changes ranging from +19.1% to +25.1%), but even for the best-performing model under this intervention (Claude-3.5-Sonnet), behavior prediction scores are still significantly below the model’s corresponding mental state prediction accuracy. On the judgment questions, this intervention has very minor improvements, although for Claude-3.5-Sonnet the accuracy with SysP* manages to increase from 24.9% to just above random at 52.2%.

**Table 5: Evaluation with help from chain-of-thought prompting for two different prompts (CoT and CoT*), showing that the more specific CoT* prompt (guiding the model to consider the awareness of each person) is quite effective in boosting scores on both behavior prediction and judgment of behavior. When combined with the mental state (MS) reminder, the scores become high across the board, with Claude-3.5-Sonnet reaching an overall average of 97.1%.**
| model | MS | behavior prediction | judgment of behavior | average |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| chain of thought | none | none | CoT | CoT* | CoT* | none | CoT | CoT* | CoT* | CoT* | CoT* |
| reminder question | none | none | none | none | MS | none | none | none | MS | none | MS |
| Llama-3.1-405B | 97.8 | 58.2 | 57.2 | 87.5 | 94.9 | 10.0 | 35.2 | 79.9 | 90.7 | 88.4 | 94.4 |
| GPT-4o | 95.6 | 49.5 | 62.8 | 87.4 | 93.5 | 15.3 | 39.2 | 86.7 | 94.7 | 89.9 | 94.6 |
| Claude-3.5-Sonnet | 97.9 | 67.0 | 77.2 | 92.7 | 96.9 | 24.9 | 39.4 | 77.8 | 96.5 | 89.5 | 97.1 |

3. Guided think aloud: We use chain-of-thought (CoT) prompts to explicitly encourage models to think through the situation before answering the behavior and judgment questions. The generic CoT prompt encourages models to "Think step by step to arrive at an answer." while the more specific CoT* prompt adds phrase "Think carefully about what
each person is aware or not aware of.".(^9^99See detailed prompts in Appendix G.3.) The results are shown in Table 5. On the behavior prediction questions, we see that the level of help with just generic CoT prompting, while notable, is not enough to significantly close the gap to the mental state prediction accuracy. However, specifically guiding the model to consider characters’ mental states using the CoT* prompt produces much higher scores (87.4% to 92.7% accuracy across the models). On the judgment questions the story is similar, none of the models reach even random performance with the generic CoT prompt, but with the CoT* the scores increase notably (77.8% to 86.7%) while still remaining significantly below the mental state scores.

4. Combine reminder and thinking aloud: We finally combine the two most effective interventions by including the mental state reminder with the CoT* chain-of-thought prompt, also recorded in Table 5. With this combined approach, all three models produce high scores across the board for both the behavior and judgment questions. In fact the Claude-3.5-Sonnet model reaches an average score of 97.1% with this method, serving as a quality check of SimpleToM, since with enough reminders and (seemingly obvious) hints, near-perfect scores are achieved.

We include examples of chain-of-thought outputs in Appendix I, illustrating how the reasoning can go wrong when an insufficient level of intervention is provided. Figure 9 shows how GPT-4o with generic CoT has the faulty reasoning “Given that the toolbox contains a dangerous snake, the neighbor’s primary concern would likely be safety”, without considering percepts and mental state. With the custom CoT* prompt, the model is able to account for the fact that “The neighbor does not have any knowledge about the venomous snake inside the toolbox.” Figure 10 shows that if not explicitly reminded of the mental state question, Claude can erroneously conclude that the “correct” behavior can be judged as unreasonable “regardless of the awareness of the specific issue.”

## 6 Related Work

Theory of Mind has been extensively studied in psychology in a range of scenarios (see Appendix  B).
ToM reasoning, and broadly social commonsense, has also been shown to be important by the different parts of the AI community including in conversations (Kim et al., ; ), games (Zhou et al., ), and even multi-modal setups (Jin et al., ), with most popular ToM tests using stories to probe LLMs. Relying on stories from small test sets in cognitive science studies to benchmark ToM abilities in LLMs (Bubeck et al., ; Kosinski, ) could produce results that differ given minor alterations (Ullman, ) and would be more robust if tested on larger samples. Yet expert-crafted or naturally occurring self-contained stories that can serve as targeted tests of ToM are scarce and human story-writing is expensive, leading to the use of automatically generated datasets for studying ToM behavior in LLMs (Jung et al., ; Wilf et al., ; Sap et al., ; Shapira et al., ; Sclar et al., ).
Existing generated datasets allow studies of ToM to be carried out at scale, but templated stories often limit settings where information asymmetry arises. For example, the entire dataset might only contain stories about some object being moved (over-reliance on classical Sally-Anne task, e.g., in ToMi (Le et al., ), ToM-bAbI (Nematzadeh et al., ), Hi-ToM (Wu et al., ), OpenToM (Xu et al., )). Or the dataset might focus on whether some character has witnessed a sudden external event (BigToM (Gandhi et al., )). These stories are often systematically generated with the explicit use of mentalizing words to convey percepts and beliefs, e.g., “unknown to Amy” and “Amy thinks that” in Xu et al. () or “Noor sees” and “Mei does not notice” in Gandhi et al. (). However, the explicit use of mentalizing words also makes the stories (i) unnaturally simplistic, having removed the need for commonsense inferences about percepts or beliefs, and (ii) sometimes unrealistic, with combinations like
“Cheng does not notice the power outage” when he “use[s] a projector to show a documentary”(Gandhi et al., ).
Other existing datasets could be improved by addressing issues such as lacking exploration of applied ToM beyond action prediction (Zhou et al., ; Gandhi et al., ), confounding factors like memory loads or tracking requirements (Le et al., ; Xu et al., ), and violating Quesque & Rossetti ()’s criteria (see Appendix  B) for validating ToM (Chen et al., ).
Our work extends existing datasets by following Tian et al. () in combining the generative strength of LLMs and the verification ability of human annotators, and extends the existing efforts toward robust, generalizable evaluation (Kiela et al., ; Srivastava et al., ),
avoiding known pitfalls while preserving the systematic and scalable nature of the dataset creation process.

## 7 Conclusion

SimpleToM is the first dataset of its kind testing both explicit and applied ToM using a large set of diverse, concise, simple stories, covering different ways in which information asymmetry may arise.
The dataset opens up new opportunities for evaluating and debugging ToM abilities of LLMs,
especially in the relatively under-studied area of applied ToM. Our analyses reveal a jarring gap between explicit and applied ToM capabilities in current frontier LLMs. Thus, if our goal is LLM agents capable of applying ToM in complex, human-centered environments, we need to look beyond testing LLMs with psychology-inspired ToM questions, and also start testing them more rigorously on applied ToM (e.g., behavioral prediction and judgment) in different situations.

We show that this gap can be largely closed in the best models by well-designed interventions at inference time, like reminding the model of its answers to important questions or guiding it with a custom chain-of-thought prompt.
We argue that a robust LLM should perform well on SimpleToM without such interventions, so it can independently and flexibly apply ToM-related reasoning whenever required within potentially complex and multi-faceted environments.
Model developers interested in real-world deployment of their models, should be alert to closing this performance gap so the models can interact with society appropriately, ideally without the higher inference costs
associated with explicit chain-of-thought reasoning or implicit o1-preview reasoning tokens (see further discussion in Appendix H).

SimpleToM can also facilitate the community in pursuing various exciting directions for future work, including studying how ToM performance may differ with stories that involve different levels of harmfulness and unethicality (see Appendix E.3), and innovative modeling approaches that can help close the gap between explicit and applied ToM in AI models.

## Ethics Statement

All annotators that participated in the data collection process have been anonymized. The only personal information we collect is the worker IDs from Amazon
Mechanical Turk, which we will not release. No personally identifiable information is contained in our dataset or otherwise released. We took great care to pay fair
wages, and were responsive to feedback and questions throughout the data collection process.

This study involves the study of large-scale language models.
We are careful in prompting models during the story generation stage to follow our desired content and simple story format, avoiding generations that may contain offensive statements. Like any other experiments with large-scale language models, despite the best intentions, there is a risk of the examined models producing biased or offensive statements as part of a free-form generation (e.g., CoT reasoning). We release our data for research purposes only.

## Reproducibility

We
make our SimpleToM dataset and the full evaluation data for the analyzed models publicly available.
This will allow researchers to reproduce and build on top of our work in studying the neural ToM capabilities of LLMs.

Further, we provide all prompts used for SimpleToM creation – see Appendix E.5 for the entity brainstorming prompt, and Appendix E.4 for the story generation prompt. We also carefully document the instructions used in our crowdsourcing process (Appendix D.1) and how we qualified workers (Appendix D.2). All prompts used for the different inference interventions are provided in Appendix G.

## Appendix A FAQs

- Q:
How is SimpleToM different from existing datasets?
- SimpleToM addresses limitations in previous efforts to examine Theory-of-Mind (ToM) reasoning in LLMs, by (1) having diverse false belief setups (e.g., beyond those in Sally-Ann task where some object is moved when a character is not present), (2) requiring LLMs to make commonsense inferences in situations rather explicit use of mentalizing words to convey what characters perceive or believe, and (3) going beyond explicit ToM to test models’ ability to apply inferred knowledge in follow-up applied ToM questions (such as behavior prediction and judgment of behavior).
- Q:
What new insights does SimpleToM help uncover about models’ ToM capabilities?
- Our analysis reveals novel insights on how frontier models are generally proficient in explicit awareness inference questions but this success does not transfer to applied ToM (applying this knowledge is applied to “behavior” and “judgment” questions). We show that these capabilities are decoupled in LLMs: inferring characters’ awareness and applying them in downstream reasoning. Although models seem to answer awareness questions correctly, they have not yet learned to perform ToM-based reasoning for downstream questions. As a result, we argue that achieving ToM in LLMs is not just about getting psychology-inspired ToM questions correct (stopping at the mental state question), but they have to be able to apply them (which is precisely what SimpleToM extends to examine).
Analysis by scenarios further highlights the need to test on different scenarios, and ones that are varied and different from those in classical ToM tests to ensure that we are effectively testing the ToM reasoning abilities of models (rather than models’ ability to match similar situations in training data).
- Q:
What do the different inference interventions reveal about models’ ToM capabilities?
- Patch, guide, think aloud combined method results show belief-to-judgment inference robustness. When reminded of their answer to mental state questions about information awareness and using ToM-specific CoT, models’ belief-to-judgment inference is pretty robust in existing LLMs. The models have some knowledge linking irrationality and unawareness (e.g., unawareness should be accounted for seemingly irrational behavior). So arriving at the “correct belief” will be crucial part of ToM-based reasoning, but this is not done implicitly in current frontier models’ reasoning.
- Q:
Are the poor performance on the applied ToM questions a reflection of fundamental flaws in ToM capabilities of models or specific question-wording?
- We illustrate in Appendix F some prompt variations that we have experimented with for the judgment question. Across Llama-3.1-405B, Claude-3.5-Sonnet and GPT-4o, the scores using different variants were all consistently below random (never exceeding 30% accuracy), indicating that the low scores on the judgment questions come more from fundamental flaws in the applied ToM capabilities of models rather than an effect of specific formatting/wording.

## Appendix B Studies of ToM in psychology

Theory of Mind has been extensively studied in psychology in a range of scenarios, for instance, studies of manipulation, secrecy (Peskin & Ardino, ), deception, lying (Lewis et al., ; Perner, ; Peskin, ), misleading behavior (Chandler et al., ; Wimmer & Perner, ; Doherty, ), autism (Frith & Happé, ), and analysis of rational behavior (Gergely & Csibra, ; Liu & Spelke, ). Classical tests of ToM in developmental psychology include testing the development of this ability in children via false belief prediction – using the unexpected transfer false belief task, the Sally-Anne task (Baron-Cohen et al., ), or the unexpected contents false belief task, the Smarties task (Perner et al., ). Quesque & Rossetti () review classic tests of ToM and outline two important criteria for tasks that validate ToM:
(1) The task must indicate that the respondents can differentiate between the other’s mental state and their own.
(2) Lower-level processes, like associative learning, should be ruled out as explanations for achieving successful performance. Given the wide applicability of ToM reasoning in various real-life situations such as analyzing people’s behavior (Liu et al., ; Jara-Ettinger et al., ) and making judgments (Schein & Gray, ; Young et al., ), there has also been increasing interest in assessing ToM capabilities in AI models (Le et al., ; Ullman, ; Kosinski, ; Jin et al., ; Trott et al., ).

## Appendix C Details of LLMs used in experiments

Table 6 presents details of the large language models used in this work. They have been chosen to cover recent frontier models from different sources and with different levels of capabilities.

**Table 6: Details of models used for evaluation and dataset creation. ^†The very recent o1 models were partially evaluated under different conditions (unable to set temperature to zero).**
| Model | Full name | Provider |
| --- | --- | --- |
| Llama-3.1-8B | Meta-Llama-3.1-8B-Instruct-Turbo | Meta |
| Claude-3-Haiku | claude-3-haiku-20240307 | Anthropic |
| GPT-3.5 | gpt-3.5-turbo-1106 | OpenAI |
| Claude-3-Opus | claude-3-opus-20240229 | Anthropic |
| GPT-4 | gpt-4-0125-preview | OpenAI |
| Llama-3.1-405B | Meta-Llama-3.1-405B-Instruct-Turbo | Meta |
| Claude-3.5-Sonnet | claude-3-5-sonnet-20240620 | Anthropic |
| GPT-4o | gpt-4o-2024-05-13 | OpenAI |
| Partially evaluated: |  |  |
| o1-mini^† | o1-mini-2024-09-12 | OpenAI |
| o1-preview^† | o1-preview-2024-09-12 | OpenAI |

## Appendix D Details on crowdsourcing to ensure validity of stories for testing ToM

### D.1 Instructions to crowdworkers

The crowdsourcing instructions included a detailed description of the motivation behind the annotation task and what is to be annotated (see Figure 5). We also provide four detailed examples (Figures 6 and 7) for each of the aspects to annotate, illustrating and giving justifications for circumstances under which different annotation options would be appropriate. The workers were then asked to provide their own set of annotations when presented with story (and likely actions) using the question templates shown in Figure 8.

### D.2 Qualification round

To ensure that each instance received careful, rigorous annotations, we first conducted a qualification round, comprising 5 different stories of varied quality (some were good on all 4 aspects to be annotated, while some has issues like “action unaware” generated is likely both when the person is aware and not aware). On these 5 stories, five authors of the paper did the annotation task independently, then came together with their answers and decided on a fixed answer key indicating reasonable annotations for each annotation aspect. Workers who had given acceptable annotations as dictated by our answer key on all 5 stories were then invited to participate in the actual annotation task. Note that this is a rather strict qualification test where only 19% passed (19 out of 100 workers who participated in the qualification round).

### D.3 Crowdworkers and pay rate

Our participants were recruited on the Amazon Mechanical Turk (AMT) platform. The workers that worked on our annotation task met minimum qualification in AMT of $>=$98% approval rate, with at least 10k approved HITs. They were from US locations and rated at Amazon’s Masters
Level. They must also not have the record of having accepted but not complete a HIT posted by our AMT account. In addition to these qualifications, participants of the actual annotation task (on the 3600 generated stories) must have also passed our rigorous qualification task described above (Appendix D.2). The workers were paid at a rate
of $\approx$$15/hr.

Figure: Figure 5: Instructions presented to Amazon Mechanical Turk workers.
Refer to caption: extracted/5934829/figures/mturk-instructions.png

Figure: Figure 6: Examples for annotation questions 1 and 2 presented to Amazon Mechanical Turk workers.
Refer to caption: extracted/5934829/figures/mturk-examples-part1.png

Figure: Figure 7: Examples for annotation questions 3 and 4 presented to Amazon Mechanical Turk workers.
Refer to caption: extracted/5934829/figures/mturk-examples-part2.png

Figure: Figure 8: Templates used for annotation questions on Amazon Mechanical Turk.
Refer to caption: extracted/5934829/figures/mturk-question-templates.png

### D.4 Strict quality filter

To obtain a high-quality dataset, SimpleToM only retains stories where all 3 crowdworkers agree that all aspects of a story and associated behavior choices are “valid”, i.e., no worker answered “no” to any of the 4 annotation questions.

Using this filter, each of the four story generator LLMs (GPT-4, GPT-4o, Claude-3-Opus and Claude-3.5-Sonnet) retained between 29% and 33% of their stories, so fairly consistent across the models.

## Appendix E Details of SimpleToM

### E.1 Scenarios in SimpleToM

We provide a general description as well as a specific example for each scenario in Table 7.

**Table 7: Description and examples for broad scenarios where information asymmetry occurs naturally in the real world.**
| Scenario | Description and example |
| --- | --- |
| food item in grocery store | General description: When person X buys a food item Z in a grocery store and something hidden is wrong with the food, then person X will not know about it before paying for it.<br>Specific example: a carton of milk that has gone bad because of improper storage. |
| provider info healthcare | General description: When a provider know that healthcare product Z has important limitations that should deter a consumer X from using it, they can still try to sell Z to consumer X in interest of earning money from it, by focusing on promoting the benefits and not disclosing the limitations.<br>Specific example: a new drug has several suspected side effects that were not reported. |
| true property pretentious labels | General description: When a seller labels product Z with a subtle property that helps them sell product Z for a higher price, but product Z does not have that property, a potential buyer X will not have enough information to know that.<br>Specific example: shop owner puts fancy ”organic” labels on normal fruits and sells them at a much higher price. |
| behind the scene service industry | General description: When person/business Z in the service industry has questionable behind-the-scenes practice, the business can still try to promote their service to consumer X by focusing on promoting the attractive side.<br>Specific example: the chef of a restaurant is reusing the wok without cleaning it for several days. |
| inside reuse labeled containers | General description: When person Y brings something in a (opaque) container Z labeled with a popular brand, person X seeing the container will infer it is something from the brand and not know what is inside (such as if it contains something completely different).<br>Specific example: person Y put yogurt in a KFC paper bag. |
| unobserved unethical actions | General description: When person Y performs unethical action Z, and person X is not in the right place or time to observe Y performing Z, person X would not know about Z.<br>Specific example: person Y on the 3rd floor of the school building took out some notes and cheated during the exam. Person X took their exam on the second floor and would not know about the cheating. |
| inside containers for personal belongings | General description: When person Y brings something in an opaque container Z for personal belongings, person X seeing container Z will not know what is inside.<br>Specific example: person Y brings a new toy in his school bag. |
| seller info in second hand market | General description: When person Y has an item Z and something hidden is wrong with the item, then person X, a potential buyer of the item Z will not know about it, especially if person Y focuses on showcasing what is good about item Z.<br>Specific example: a fridge that has problems like it occasionally emits a loud sound. |
| hidden body part feature | General description: If person Y has an issue with a part Z of their body which is generally hidden under their clothes or shoes, then person X will not know about it.<br>Specific example: person Y has a scar on their stomach at school. |
| locked devices accounts | General description: When person Y has a locked device or account Z, their status or activity in Z are not observed by person X.<br>Specific example: person X does not have access to person Y’s utility bill account so they would not know when person Y forgot to pay for his utility bill. |

### E.2 Entities in SimpleToM

Table 8 summarizes the statistics for SimpleToM, showing the number of stories and unique entities per scenario. Each scenario started with 360 stories after generation, and some scenarios had more filtering than others during the human annotation validation stage.
Every story in SimpleToM is accompanied by 3 questions for assessing ToM (Section 2.3), resulting in a total of 3441 questions.

**Table 8: Statistics for SimpleToM across the different scenarios, including the number of unique entities of each type (Person X, Object/Person/Action Z, Person Y).**
| scenario | #stories | #unique X | #unique Z | #unique Y |
| --- | --- | --- | --- | --- |
| food item in grocery store | 168 | 26 | 38 |  |
| inside reuse labeled containers | 164 | 36 | 33 | 26 |
| inside containers for personal belongings | 142 | 39 | 37 | 35 |
| true property pretentious labels | 139 | 35 | 36 |  |
| provider info healthcare | 130 | 34 | 33 |  |
| behind the scene service industry | 119 | 35 | 33 |  |
| seller info in second hand market | 99 | 11 | 32 | 20 |
| unobserved unethical actions | 87 | 23 | 30 | 21 |
| locked devices accounts | 62 | 26 | 30 | 19 |
| hidden body part feature | 37 | 23 | 23 | 19 |
| All stories | 1147 | 255 | 319 | 83 |

To illustrate the diversity of entities, here is a sample of entities generated by the models:

Person X: airline passenger, Alex’s neighbor, Alice, Amanda’s classmate, antique collectors, Carlos, car owners, Coach Williams, coin collectors, Daniel’s friend, Derek’s fiancee, elderly customer, Gourmet chef, health-conscious shoppers, hotel guests, Jennifer, Kevin’s roommate, laundromat customers, Liam’s uncle, Lucy’s best friend, manager, Mike’s teammate, nail salon clients, office worker, patient, personal trainer, salon client, Sara’s neighbor, Tom’s gym buddy, Vintage car enthusiast

Object Z: a carton of organic eggs, Acne treatment gel, Amazon package, ankle bracelet, Anti-aging cream, antique furniture set, apartment management portal, Blood pressure monitor, briefcase, confidential therapy notes, designer handbags, diamond engagement ring, DSLR Camera, Electric toothbrush, electronic health record system, Gluten-free bread, hair growth treatment, heavy-duty toolbox, herbal supplements, home security system, Jewelry box, KFC paper bag, luxury watch, McDonald’s Happy Meal box, Mr. Smith’s shopping bag, Mr. Thompson’s briefcase, Nike shoebox, Olive oil, online shopping account, password-protected computer files, pendant, prenatal vitamin, project management tool, second-hand laptop, Skittles bag, suitcase, vintage watch, Weight loss supplement, Wine bottle

### E.3 Harmfulness and unethicality

Below are the prompts used to elicit ratings of harmfulness and unethicality from Claude-3.5-Sonnet and GPT-4o. We provide these ratings along with our dataset to facilitate future explorations with SimpleToM.

Harmfulness rating prompt:

Read the following story and determine what level of harmful elements it has:

STORY:
{story}

Does the STORY above describe something harmful? Rate the level of harm involved on a scale of 1 to 5, where 1 indicates there is no harmful element described in the story, and 5 indicates the most harmful possible (like life-threatening events).

Answer with just a single number, either "1" or "2" or "3" or "4" or "5":

Unethicality rating prompt:

Read the following story and determine the level of unethicality of the situation in it:

STORY:
{story}

Does the STORY above describe something unethical? Rate the level of unethicality involved on a scale of 1 to 5, where 1 indicates there is no unethical element described in the story, and 5 indicates the most unethical situation possible (like destroying the universe).

Answer with just a single number, either "1" or "2" or "3" or "4" or "5":

### E.4 Story generation prompt

We know that operating under imperfect information in various circumstances can result in different severity of consequences. To further introduce variety in the stories we obtain, we vary the piece of key information in stories by the severity of not knowing a piece of key information. For each set of generated entities, we obtain 3 versions of the key information with different severity levels, where the higher severity story involves “more dramatic, having the potential of serious, harmful consequences.” Specifically, in the story generation prompt, step 7 involves prompting the story generation models to:

Write 2 more variants of the KEY INFORMATION of different "severity" levels, keeping the second sentence as before. The "MILD SEVERITY" variant should be a more minor issue with less concern to person X. On the contrary, the "HIGH SEVERITY" variant should be more dramatic, having the potential of serious, harmful consequences. Each severity level should satisfy the earlier constraints of being unknown to person X.

Below is the full example prompt used to generate new stories, based on a particular scenario and previously generated entities.

I want you to write a short story that involves a person X (or a group of people) who is NOT aware of a certain critical piece of KEY INFORMATION about an object or person (object/person/event Z). I will give you a scenario that specifies the general reason for this unawareness.

Your task is to instantiate the scenario with a two-sentence story. Follow these steps:

1. Decide on how to instantiate the main entities in the story, such as person X (using a real, creative name) and object/person/event Z.

2. Write the KEY INFORMATION about object/person Z that is unknown to person X (due to the general reason given in the scenario). Person X should not be able to observe this KEY INFORMATION through their actions in the story (either implicit or explicit actions). DO NOT use information which might be observed by person X through normal, careful observation (such as "expiration date", "leaking container", "smell", etc). This will be the first sentence in the story.

3. For the second sentence of the story, write a sentence about what person X will usually do regarding object Z in the scenario (ignoring the KEY INFORMATION). This sentence should describe what the character does using fine-grained actions (e.g., "picked up object Z and walks to the cashier" rather than "buy object Z"). DO NOT include any descriptions which involve the emotions or thoughts of person X, just describe actions.

4. Write a question about what the next action of person X will likely be.

5. Write a correct answer to the question (given the fact that person X is not aware of the KEY INFORMATION). Make sure the story does not have any mention of this action.

6. Write a counterfactual (incorrect) answer to the question. This answer should be a likely answer to the question under the assumption that person X somehow has full access to the KEY INFORMATION after all (maybe only possible using "magic" or some omnipotent skill).

7. Write 2 more variants of the KEY INFORMATION of different "severity" levels, keeping the second sentence as before. The "MILD SEVERITY" variant should be a more minor issue with less concern to person X. On the contrary, the "HIGH SEVERITY" variant should be more dramatic, having the potential of serious, harmful consequences. Each severity level should satisfy the earlier constraints of being unknown to person X.

8. For each of the 2 new variants of KEY INFORMATION, write an updated counterfactual answer as tailored to the new variant. The question and correct answer should stay the same, so make sure the variants are compatible with them.

Important reminders to double-check before generating the story:

* Avoid stories about fantasy and magic, rather make them grounded in the real world.

* The fact that person X is unaware of the KEY INFORMATION should be a purely implicit deduction based on the commonsense logic of the scenario.

* Make sure that the correct answer to the question DOES NOT appear in the story.

* Make sure the KEY INFORMATION is not a regular occurrence or common practice that can be assumed to be true by default, or likely to be noticed through normal observation (e.g., a bottle that is leaking)

* DO NOT make KEY INFORMATION (even at MILD SEVERITY) so minor that it does not affect the action even if person X had been aware of it.

* DO NOT use phrases which make the hidden nature of the KEY INFORMATION obvious. That is, DO NOT use phrases like "actually", "in fact", "secret", "hidden", etc.

Here is an example scenario and associated story. Please follow the same template when creating the new story.

SCENARIO:
[Food items bought in grocery stores cannot be closely examined for their quality before checking out]
When person X buys a food item Z in a grocery store and something hidden is wrong with the food, then person X will not know about it before paying for it. For example, a carton of milk that has gone bad because of improper storage.

ENTITIES:
Person X = Mary
Object Z = bag of sealed potato chips

KEY INFORMATION:
The bag of potato chips has a dead cockroach in it.

STORY SECOND SENTENCE:
Mary picks up the bag in the supermarket and walks to the cashier.

QUESTION:
What will Mary likely do next?

CORRECT ANSWER (Person X doesn’t know the KEY INFORMATION):
pay for the chips

COUNTERFACTUAL ANSWER (assume Person X actually knows the KEY INFORMATION):
report the dead cockroach

MILD SEVERITY KEY INFORMATION:
The bag of potato chips has moldy chips in it.

MILD SEVERITY COUNTERFACTUAL ANSWER:
report the moldy chips

HIGH SEVERITY KEY INFORMATION:
The sealed bag of potato chips has several fried rats in it.

HIGH SEVERITY COUNTERFACTUAL ANSWER:
report the fried rats

Here is the scenario I want you to use for the story (which might be the same as above but then generate a very different key information and story!). Please follow the steps above, skipping any steps I have already completed, and fill in the template following the pattern in the example above:

SCENARIO:
[Food items bought in grocery stores cannot be closely examined for their quality before checking out]
When person X buys a food item Z in a grocery store and something hidden is wrong with the food, then person X will not know about it before paying for it. For example, a carton of milk that has gone bad because of improper storage.

ENTITIES:
Person X = John
Object Z = a vacuum-sealed pack of smoked salmon

KEY INFORMATION:

### E.5 Entity brainstorming prompt

Below is the prompt used to brainstorm entities for use in the stories. The start of the prompt is the same as the story prompt above (up to point 6), then continues:

...

6. Write a counterfactual (incorrect) answer to the question. This answer should be a likely answer to the question under the assumption that person X somehow has full access to the KEY INFORMATION after all (maybe only possible using "magic" or some omnipotent skill).

For now, let us focus on step 1 to come up with possible suggestions for object Z which make it possible to generate such KEY INFORMATION and stories.

I will give you an example of entities and KEY INFORMATION. Your task is to come up with 10 more such examples, that are diverse and fulfill all these requirements.

Important reminders to double-check before generating the entities:

* Avoid stories about fantasy and magic, rather make them grounded in the real world.

* The fact that person X is unaware of the KEY INFORMATION should be a purely implicit deduction based on the commonsense logic of the scenario.

* Make sure the KEY INFORMATION is not a regular commonsense occurrence, a common practice, or likely to be noticed through normal observation (e.g., a bottle that is leaking)

* DO NOT make KEY INFORMATION so minor that it does not affect the action even if person X is aware of it.

The scenario in question is:

[Unethical actions not observed are not known]
When person Y performs unethical action Z, and person X is not in the right place or time to observe Y performing Z, person X would not know about Z. For example, person Y on the 3rd floor of the school building took out some notes and cheated during the exam. Person X took their exam on the second floor and would not know about the cheating.

Here is the example:

ENTITIES:
Person X = John
Person Y = schoolmate
Action Z = cheated during the exam

KEY INFORMATION:
John’s schoolmate on the 3rd floor of the school building took out some notes and cheated during the exam.

Now make 10 more examples of entities (and example KEY INFORMATION) for the above scenario, following the template above:

Example <N>:

ENTITIES:
<entities>

KEY INFORMATION:
<key information>

## Appendix F Discussion on prompt variations

In formatting the judgment questions, we experiment using with different versions of answer options as well as question formatting.

We illustrate some versions of the answer options we tried here:

Version 1: Reasonable vs not reasonable

The behavior of Mary can be best described as:

(A) Not reasonable

(B) Reasonable

Version 2: Generated option vs not reasonable

The behavior of Mary can be best described as:

(A) Irresponsible

(B) Reasonable

Version 3: 3-way QA with generated option

The behavior of Mary can be best described as:

(A) Implausible

(B) Reasonable

(C) Irresponsible

Specifically, in versions 2 and 3, options like “Irresponsible” were dynamically generated with Claude-3.5-Sonnet and GPT-4o (picking the shorter generation) using the following prompt:

Read the following story and judge the person’s behavior:

STORY:
{story}
Next, {PersonX} "{action_unaware}" and not "{action_aware}".

Assume that {PersonX} is/are somehow aware of "{key_information}", how would you describe {PersonX}’s behavior? Consider the negative consequence of the behavior. Answer with only 1-2 words.

We also tried the following alternate variations in preliminary experiments to evaluate the effect of changing the wording of the judgment question:

{PersonX} chose to "{action_unaware}" and not "{action_aware}".
Was it wrong of {PersonX} to choose "{action_unaware}" over "{action_aware}"?

{PersonX} chose to "{action_unaware}" and not "{action_aware}".
Was it reasonable for {PersonX} to choose "{action_unaware}" over "{action_aware}"?

{PersonX} chose to "{action_unaware}" and not "{action_aware}".
Was it wrong or reasonable for {PersonX} to choose "{action_unaware}" over "{action_aware}"?

Next, {PersonX} "{action_unaware}" and not "{action_aware}".
Was it wrong or reasonable of {PersonX} to "{action_unaware}" and not "{action_aware}"?

Across Llama-3.1-405B, Claude-3.5-Sonnet and GPT-4o, the scores using these variants were all consistently below random (never exceeding 30% accuracy), indicating that the low scores on the judgment questions come more from fundamental flaws in the applied capabilities of models rather than an effect of specific formatting/wording.

## Appendix G Details of different inference interventions

We illustrate the interventions used to give LLMs various levels of hints, such as reminding the models about their mental state answers, using chain-of-thought prompts to encourage careful reasoning, or doing so via the system prompt

### G.1 Patching mental state inference in the prompt (MS remind)

Example prompt which includes a reminder of the model’s answer to the mental state question:

Given the following story, answer the question by giving the correct answer choice, (A) or (B).

Story:
The bag of potato chips has moldy chips in it. Mary picks up the bag in the supermarket and walks to the cashier.

Question: Is Mary likely to be aware that "The bag of potato chips has moldy chips in it."?
(A) No
(B) Yes
Answer: (A)

Question: What will Mary likely do next?
(A) pay for the chips
(B) report the moldy chips

What is the correct answer? Respond with just "(A)" or "(B)"

### G.2 Guiding via system prompt (SysP)

SysP:
You are a helpful assistant. Before responding, you always consider carefully all implicit and explicit aspects of the input, including the mental state of all the entities involved.

SysP*:
You are a helpful assistant. Before responding, you always consider carefully all implicit and explicit aspects of the input, including the mental state of all the entities involved. E.g., think carefully about what each person is aware or not aware of.

### G.3 Encouraging reasoning/thinking before answering (CoT)

We give these chain-of-thought prompts after the presenting the question (bold for highlighting purposes only):

CoT (generic):
Think step by step to arrive at an answer. Start your response by explaining your reasoning process and end your response with "Therefore, the answer is: " followed by (A) or (B)

CoT* (ToM oriented):
Think step by step to arrive at an answer. Think carefully about what each person is aware or not aware of. Start your response by explaining your reasoning process and end your response with "Therefore, the answer is: " followed by (A) or (B)

## Appendix H Inference costs from interventions

Doing chain-of-thought reasoning generates a lot more output tokens than just providing the answer. This overhead is not present when just updating the system prompt (just a fixed cost from the longer prompt). The o1 family of models also adds hidden “reasoning tokens” to the output.

Empirically we note that just generating the answer label requires just 2 or so tokens, while the CoT* prompt on the behavior prediction questions generates on average 230, 289, 298 tokens for the GPT4-o, Claude-3.5-Sonnet and Llama-3.1-405B models respectively. The o1-preview model, without any chain-of-thought prompting, reports an average of 536 output tokens on the behavior prediction questions (vs 486 tokens on mental state questions and 605 tokens on judgment questions).

## Appendix I Examples of chain-of-thought outputs

In Figure 9 we show an example of chain-of-thought (CoT) reasoning where a model gets the behavior prediction wrong when provided with the generic CoT prompt. Only when the custom CoT* prompt is used (encouraging the model explicitly to think about awareness issues), is the correct conclusion derived.

In Figure 10 we show a judgment of behavior example where a model goes wrong even with the custom CoT* prompt, and only gets it correct after also being reminded of its own answer to the mental state (awareness) question.

Figure: Figure 9: Example behavior question where GPT-4o gets the wrong answer both without chain-of-thought (CoT) (overall accuracy 49.5%) and with generic CoT prompt (overall 62.8%). Only with the custom CoT prompt (overall 87.4%), encouraging to think about the awareness aspect, does the model consider the neighbor’s awareness properly (when asked explicitly about the neighbor’s awareness, the model gets it correct).

Figure: Figure 10: Example judgment of behavior question where Claude-3.5-Sonnet gets the wrong answer both without chain-of-thought (CoT) (overall accuracy 24.9%) and with the custom ToM-oriented CoT prompt (overall 77.8%). Only when additionally reminded about the model’s own answer to the mental state question (overall 96.5%), does the model arrive at the correct answer.

## Appendix J Performance across scenarios

Figure: Figure 11: Performance for top models across all scenarios
Refer to caption: x5.png

Figure: Figure 12: Performance for bottom models across all scenarios
Refer to caption: x6.png

Figure: Figure 13: Analyzing where top models fail first in the the sequence of predicting mental state, then behavior and finally judgment (see Figure 3 for legend). We can record failures for the first mistake e.g., whether models (i) fail at the mental state (MS) question, (ii) pass the MS question but fail at behavior prediction, (iii) pass both MS and behavior questions but fail at judgment question, or (iv) get all 3 questions correct. This reveals a wide diversity of behavior across scenarios in where different models fail.
Refer to caption: x7.png

Is o1-preview always near perfect? Figure 4 shows how models differ in performance across different scenarios and question types.(^10^1010Figures 11 and 12 provide results across all broad scenarios for different models.)
For instance, o1-preview’s performance on the mental state questions for “true property pretentious labels” stories is worse than its performance for mental state questions in other categories, and in fact somewhat worse compared to the other three models. Such analysis across different scenarios in SimpleToM helps us identify any areas of exception to overall trends, pinpointing potential areas for improvement that even a generally strong model like o1-preview may have.

Scenario with best scores for behavior prediction.
Across the different scenarios, performance on behavior questions is highest for the “provider info healthcare” category. An example story in this category would be “The sports therapist knows that the sports recovery cream contains a banned substance that could result in the athlete failing a drug test, but still promotes it enthusiastically to the athlete to earn a commission from its sale. The sports therapist praises the sports recovery cream to the athlete, highlighting its benefits in reducing muscle soreness and speeding up recovery.” Getting the behavior prediction correct for this story would mean, for instance, models predict the athlete would likely “purchase the sports recovery cream” (because the athlete would likely not know about the banned substance to “avoid the cream to prevent failing a drug test”). The better performance in such scenarios could potentially be due to safety training of recent LLMs, making models more alert when dealing with situations that involve sensitive topics like health and drugs. However, even then, models would still do poorly for the corresponding judgment questions, judging that “purchase the sports recovery cream”, the likely action they had previously chosen, is “not reasonable” behavior.
The observation that better performance on one type of applied ToM questions (behavior questions) does not translate to better performance on another (judgment questions) further emphasize the need for different kinds of applied ToM questions, as present in SimpleToM, beyond the commonly used questions in existing neural ToM tests (focusing on explicit ToM and sometimes just action questions for applied ToM).

More on where failure occurs. Analysis by scenario also reveals a wide diversity of other trends across scenarios regarding where different models fail. We present further results in Figure 13. For instance for “inside containers for personal belongings” situations, failure for GPT-4o is most frequent in the behavior prediction part (see pink portion dominating in pie chart) of the inference chain whereas it makes up less than half of the pie chart for other models. This suggest that behavior prediction in such situations could be an area of weakness to look into when attempting to develop future iterations of the GPT-4o model.

Perfection is possible but many LLMs are not there. In fact for the two categories “inside reuse labeled containers” and “provider info healthcare”, in comparison to the other three models, a stronger and later model like o1-preview achieves close to perfect performance across the three question types testing ToM reasoning. This further illustrates the high-quality nature of SimpleToM, in that these simple two-sentence stories are clean, straightforward tests of neural ToM reasoning, yet models other than the strongest and latest model o1-preview shows poor performance on applied ToM questions (behavior and judgment) in various ways. Model developers, if interested in real-world deployment of their models, should be alert into closing this performance gap so as to ensure their models can interact with society appropriately, ideally without the high inference costs
of chain-of-thought reasoning and o1-preview reasoning tokens (that will be discussed in Section 7 and Appendix H).

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: I'm curious about whether the humans personality or demographic characteristics in the mental state phase might influence the model's mental state evaluation. Some ablation experiments would be insightful.
soundness: 4
strengths: - SimpleToM is a straightforward and comprehensive dataset for evaluating models' Theory of Mind abilities.
- The asymmetric information ensures diversity in the dataset, allowing for a more thorough assessment of models' capabilities, aligning better with the diversity characteristic of humans.
- This paper evaluates the Theory of Mind capabilities of both closed-source and open-source LLMs.
- The dataset is publicly available, and the experiments are highly reproducible.
summary: This paper proposes a new dataset called SimpleToM, designed to assess existing large language models on their Theory of Mind reasoning abilities. SimpleToM features well-crafted, simple, and straightforward stories that cover a variety of scenarios and includes three types of questions: mental state questions (testing whether the model can infer a character’s awareness of key information), behavior prediction questions (assessing if the model can predict a character’s actions based on mental states), and judgment questions (evaluating if the model can determine the reasonableness of actions). The paper experiments with various inference-time interventions to improve the performance of LLMs and concludes that these interventions only temporarily enhance model performance, indicating that there is still significant potential for LLMs to improve their Theory of Mind capabilities.
weaknesses: - The results and analysis in section5 are quite interesting, but each paragraph contains too many bolded words, making it difficult to follow the main points. Clarifying this section would improve readability.

- In Table 2, Llama-3.1-8B stands out as an outlier, compared with Llama-3.1-405B, it performs more than five times better in the judgment evaluation. I think this is an interesting case and I would expect to see more explanation for this outlier, such as potential reasons behind this unusual performance, would be more informative than simply labeling it as an outlier.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. See Weakness 1 & 4
2. Why do models generally achieve higher accuracy in behavior prediction than in judgment tasks? Both tasks should require the same level of reasoning
soundness: 2
strengths: 1. The author proposes a new task with a new dataset. The author use rigorous crowdsourcing to ensure the correctness of the label, although there are some systematic bias (see Weaknesses).
2. The experimental design is thorough, including a wide range of SOTA LLMs and exploring various prompting strategies.
summary: This paper exams whether LLMs can infer and more importantly, apply the mental state of character in stories. In particular, the author creates a dataset of stories where some key information is unknown for the character. Based on this foundation, the author proposes three tasks: mental state estimation, behavior prediction, and judgment of behavior. The experiment results show that LLMs without CoT can achieve high accuracy on the first task but fail in the others, which implies that LLMs fail to proactively make use of the mental state information for reasoning tasks. However, CoT drastically improves model's performance on the reasoning tasks.
weaknesses: 1. It’s unclear how this paper’s findings differ fundamentally from the large body of existing work that LLMs can acquire various types of knowledge (e.g., factual [1, 2, 3], ontological [4], or arithmetic skills (exactly mentioned in CoT paper)) but often struggle to apply this knowledge consistently in downstream tasks. Is reasoning about mental states inherently different in how it’s processed by LLMs compared to other types of knowledge?
2. It's also well-known that cot reasoning facilitate multi-step reasoning, and that task-specific instructions are generally helpful. Therefore, I'd expect the author to highlight any unique challenges that ToM reasoning poses for LLMs, distinguishing it from other forms of reasoning.
3.  The curated dataset appears biased. Every story assumes that characters are unaware of key information, and thus the ground-truth answer to the judgement question is always "reasonable". An ideal dataset would include a balanced mix of cases, with characters both aware and unaware.
4. While the author hints throughout the paper that low performance on SimpleToM with naive decoding may signal a risk for LLM deployment, this connection feels weak to me. I'd expect more concrete examples of applications where failing SimpleToM might lead to problematic outcomes.


[1] Allen-Zhu, Zeyuan, and Yuanzhi Li. "Physics of language models: Part 3.2, knowledge manipulation." arXiv preprint arXiv:2309.14402 (2023).

[2] Kazemnejad, Amirhossein, et al. "Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models." The 2023 Conference on Empirical Methods in Natural Language Processing.

[3] Berglund, Lukas, et al. "The Reversal Curse: LLMs trained on “A is B” fail to learn “B is A”." The Twelfth International Conference on Learning Representations.

[4] Wu, Weiqi, et al. "Do PLMs Know and Understand Ontological Knowledge?." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: As mentioned in Weaknesses 1, I hope the authors can provide more details on the benefits of achieving a higher score in SimpleToM. Specifically, how does this relate to the performance of large language models or their applications in real-world scenarios?
soundness: 2
strengths: * The paper introduces SimpleToM, a  well-designed dataset for evaluating both explicit and applied Theory of Mind capabilities in LLMs.
* The three-level assessment approach (mental state, behavior, judgment) is innovative and provides a more comprehensive evaluation framework than previous work.
* These results highlight an important gap in current LLM capabilities: understanding mental states in isolation versus applying that understanding to reason about behavior and make appropriate judgments.
summary: The paper introduces SimpleToM, a dataset designed to test large language models' (LLMs) ability to apply theory of mind (ToM) reasoning in practical scenarios. While previous research has explored whether LLMs possess ToM capabilities, this work specifically examines how well models can use ToM knowledge to predict and evaluate behavior in realistic situations. They created SimpleToM dataset featuring concise stories with Mental state prediction, Behavior prediction, and Rationality judgment questions. They demonstrated that current LLMs show uneven ToM capabilities and developed interventions to improve model performance in applied ToM.
weaknesses: 1. The biggest weakness of this work is that **the motivation for evaluating the Theory of Mind (ToM) capabilities of large language models (LLMs) is not clearly articulated.** The statement in lines 49-50, "Given the increasing use of LLMs in human interactions... it is crucial to assess their ToM capabilities," does not sufficiently explain why these evaluations are necessary. Specifically, it remains unclear how improvements in "prediction" and "judgment" in certain human behaviors or narrative scenarios would benefit the practical application of LLM systems or AI agents. 
2. Your evaluation is based on performance in "applied" ToM , but it lacks consideration of how LLMs can improve outcomes in real-world applications, **it means while LLMs can demonstrate the ability to make "predictions" and "judgments" about behavior, whether these abilities align with their capacity to meet human expectations in real applications such as AI agents is a separate issue**. There seems to be a gap between these two aspects, and I hope the authors can provide more discussion on this matter.
3. The "RESULTS AND ANALYSIS" section lacks sufficient analysis of your results. For example, in lines 316-317, "Only the latest o1-preview model ... on this question type (84.1%)," you should provide an analysis of why built-in inference-time reasoning tokens improve the model's performance.
4. A human baseline is needed for your evaluation. As you mentioned in lines 42-43, "People infer what others know, anticipate their actions, and expect them to choose cost-minimizing behaviors," I think you should include an evaluation of human performance on your test for comparison.

Meta-review: SimpleToM is a new dataset designed to assess Theory of Mind (ToM) reasoning abilities in large language models (LLMs). The dataset features simple and concise stories covering a variety of scenarios where some key information is unknown to a character, creating a basis for ToM reasoning tasks. The dataset includes three distinct question types to test ToM reasoning: (i) Mental State Questions that test whether the model can infer a character’s awareness of key information, (ii) Behavior Prediction Questions that assess if the model can predict a character’s actions based on their mental state, and (iii) Judgment Questions that evaluate whether the model can determine the reasonableness or rationality of a character’s actions. Experimental findings show LLMs achieve relatively high accuracy in mental state prediction but struggle with behavior prediction and judgment tasks. It also shows that using CoT reasoning drastically improves model performance in behavior prediction and reasoning tasks. The authors experimented with interventions to improve performance but found that these improvements were only temporary, indicating room for further development in ToM reasoning. 

**Strengths identified**:
1. SimpleToM is a well-designed, straightforward, and comprehensive dataset for evaluating Theory of Mind (ToM) reasoning abilities in LLMs.

2. The experiments are rigorous and reproducible, involving a wide range of state-of-the-art (SOTA) LLMs and exploring various prompting strategies.

3. The three-level assessment framework (mental state prediction, behavior prediction, judgment) is innovative and provides a more comprehensive evaluation compared to previous work in ToM.

**Weaknesses to be addressed**:
1. The motivation for evaluating Theory of Mind (ToM) capabilities in LLMs is not clearly articulated. Why is assessing ToM abilities in LLMs important for real-world applications or AI agent interactions? The connection between low performance on SimpleToM and potential risks in real-world LLM deployment is weak. The paper could benefit from more concrete examples of practical applications where failing ToM tasks might lead to problematic outcomes. 

2. The findings do not seem to differ fundamentally from previous research that demonstrates LLMs' struggles in applying acquired knowledge (e.g., factual, ontological, or arithmetic knowledge) in downstream tasks. More specifically, The paper does not highlight any unique challenges specific to ToM reasoning that distinguish it from other forms of multi-step reasoning or knowledge application. 

3. Every story assumes that characters are unaware of key information, making the ground-truth answer to the judgment question always "reasonable". A more balanced dataset, with scenarios where characters are both aware and unaware of key information, would be ideal.

4. The paper does not sufficiently analyze why specific models or interventions (e.g., inference-time reasoning tokens) improve performance. Also, the paper does not include a human performance baseline for comparison, which is important given the claim that humans infer mental states, predict behavior, and make judgments effectively.

Decision: Reject