Do Large Language Models Have a Planning Theory of Mind? Evidence from MindGames: a Multi-Step Persuasion Task
Authors: Jared Moore
Ned Cooper
Rasmus Overmark
Beba Cibralic
Cameron Robert Jones
Nick Haber
Venue: COLM 2025 (2025)
Keywords: theory of mind
planning
causal model
persuasion
TL;DR: Humans significantly outperform LLMs at our complex theory of mind task

Abstract: Recent evidence suggests Large Language Models (LLMs) display Theory of Mind (ToM) abilities. Most ToM experiments place participants in a spectatorial role, wherein they predict and interpret other agents' behavior. However, human ToM also contributes to dynamically planning action and strategically intervening on others' mental states. We present MindGames: a novel `planning theory of mind' (PToM) task which requires agents to infer an interlocutor's beliefs and desires to persuade them to alter their behavior. Unlike previous evaluations, we explicitly evaluate use cases of ToM. We find that humans significantly outperform o1-preview (an LLM) at our PToM task (11% higher; $p=0.006$). We hypothesize this is because humans have an implicit causal model of other agents (e.g., they know, as our task requires, to ask about people's preferences). In contrast, o1-preview outperforms humans in a baseline condition which requires a similar amount of planning but minimal mental state inferences (e.g., o1-preview is better than humans at planning when already given someone's preferences). These results suggest a significant gap between human-like social reasoning and LLM abilities.

Published as a conference paper at COLM 2025
Do Large Language Models Have a Planning Theory of Mind? Evidence from MINDGAMES: Multi-Step Persuasion Task
Jared Moore 1, Ned Cooper +2, Rasmus Overmark *3 Beba Cibralic
Nick Haber
Cameron R. Jones
1 Department of Computer Science, Stanford University
2 School of Cybernetics, Australian National University
3 School of School of Psychology and Neuroscience, University of St. Andrews
4 Leverhulme Centre for the Future of Intelligence, University of Cambridge
5 Graduate School of Education, Stanford University
6 Department of Cognitive Science, University of California, San Diego
Abstract
Recent evidence suggests Large Language Models (LLMs) display Theory Mind (ToM) abilities. Most ToM experiments place participants spectatorial role, wherein they predict and interpret other agents' behavior. However, human ToM also contributes to dynamically planning action and strategically intervening on others' mental states. We present MindGames: a novel 'planning theory of mind' (PToM) task which requires agents to infer interlocutor's 's beliefs and desires to persuade them to alter their behavior. Unlike previous evaluations, we explicitly evaluate use cases of ToM. We find that humans significantly outperform ol-preview (an 0.006). We hypothesize this is
LLM) at our PToM task (11% higher, because humans have an implicit causal model of other agents (eg., they know, as our task requires, to ask about people's preferences). In contrast, ol-preview outperforms humans similar amount of planning but minimal mental state inferences (e.g,, ol- preview better than humans at planning when already given someone's s preferences). These results suggest significant gap between human-like social reasoning and LLM abilities.
baseline condition which requires a
1 Introduction
Theory of Mind (ToM)-the ability to understand behaviors in terms of underlying mental states-is crucial and much-discussed capacity in artificial intelligence (AI) research. ToM is necessary component of many potential applications of AI, including recognizing users' intents, displaying sensitivity to users' emotions, and anticipating the impact of different events users' mental states (Cuzzolin et al., 2020; Rabinowitz et al., 2018; Street, 2024). Recent work has suggested that Large Language Models (LLMs) display ToM abilities (Trott et al., 2023; Gandhi et al., 2023; Kosinski, 2024; Strachan et al., 2024). However, most existing ToM assessments are passive and spectatorial: they focus on the ability to predict and explain mental states rather than whether agents actively plan and intervene mental states. Purely spectatorial and predictive evaluations might be more vulnerable to superficial heuristics and memorization, meaning that they can be passed by systems lacking the underlying abilities the tests are designed to measure (Ho et al., 2022; Hu et al., 2025). This distinction is not only theoretically important for determining the abilities of LLMs, but also has immediate practical implications. LLMs are already being deployed in high-stakes social situations: as educators (Wen al., 2024), therapists (Moore al., 2025), and as
others'
Equal contribution, respectively.

---

Published as a conference paper at COLM 2025
companions (Chaturvedi et 2023). The success of these applications includinng potential benefits and harms to users- depend upon how well LLMs are able to dynamically interpret and respond to users' mental states (Kirk et al., 2025).
To address these limitations, we present an advanced ToM task framework for adult humans and LLMs that tests for 'planning theory of mind' (PToM) (Ho et al., 2022; Cross et al., 2024) the ability to intervene on another agent's mental states to bring about desired actions. Few tasks test for this crucial component of ToM (Ho et al., 2022; Chen et al., 2024). Our approach uses persuasive dialogue as a test for PToM because successful persuasion often depends on one's sensitivity to an interlocutor's ' beliefs and desires (Costello al., 2024). Persuasion in LLMs has mostly been studied in single-shot settings which do not specifically measure whether an LLM tailors its responses to the mental states of its target, necessary condition for succeeding in virtue of a PToM (Costello et al. (2024); Hackenburg et al. (2024); Durmus et al. (2024), among others).
Our work makes several contributions. (1) We provide novel task to evaluate progress on social reasoning in LLMs. Our PToM framework, MINDGAMES, contributes a controlled multi-turn dialogue environment for evaluating complex ToM abilities beyond simple classification. (2) We run a human experiment to evaluate average performance on our task (n=124). (3) Our results show that reasoning models (e.g., op-eevieew) can handle multi-step planning with hidden information, though all current LLMs struggle with the more complex PToM task compared to humans. This suggests that humans have more sophisticated causal models of persuasive behavior. (4) Our results reveal a capability gap: LLMs (especially reasoning models) succeed at higher rates than humans in simple versions of the ToM task, but cannot reliably model and intervene on others' mental states across multiple turns.'
2 Background
Theory of Mind tasks tend to focus on participants' ability to predict an outcome from spectator's 's perspective. For example, in the classic Sally-Anne version of the false belief task, the participant must predict whether Sally will look for her marble in the basket where she last saw it or in the box where it has since been moved (Wimmer & Perner, 1983; Baron-Cohen al., 1985).
Three general criticisms can be raised against classic ToM tasks. The first concerns the nature of ToM representations. In emphasizing prediction, these tasks do not directly test an agent's causal understanding of how mental states generate behavior, but instead their ability to form associations between agents and states of the world. The prediction that Sally will look in the basket might rely on purely associative understanding that people will tend to look for objects where they last saw them, rather than a causal understanding that Sally's 'ss beliefs about the marble's 's location and her desire for it generated her searching behavior. A causal understanding of mental states is widely regarded as central to human ToM (Ho et al., 2022; Gopnik & Wellman, 1992; Gerstenberg & Tenenbaum, 2017; Butterfill & Apperly, 2013).
The second criticism concerns the context of ToM. Classic tasks put participants in a spectator role, but human ToM is predominantly used in interactions where agents are themselves participants (Hutto, 2012). Spectatorial and participatory perspectives are different on the neural level, limiting classic tasks as valid ToM measures (Schilbach et al., 2013).
The third criticism concerns the function of ToM. Classic tasks focus on the predictive function of ToM, but ToM has many functions beyond prediction (Spaulding, 2020). In particular, one function of human ToM is to plan actions that generate desired mental states in other agents (Ho et al., 2022; Perner & Ruffman, 2005; Wu et al., 2024; Chen et al., 2024).
We address these three criticisms by designing a task that measures Planning Theory of Mind (PToM). PToM builds on a causal understanding of mental states, actively intervenes on mental states, and is specific to participatory contexts.
'Our data and code available here: https: htts::///tttht thg..ttt..b... cttp tttm://ccoooe//mmmdeeeeeo
2

---

Published as a conference paper at COLM 2025
To develop novel PToM task (by definition, one that emphasizes causal understanding of mental states in a participatory mode), we focus on persuasion behavior, which is ubiquitous in humans. Developmental psychologists have suggested that successful persuasion will often be sensitive to the mental states of the target of persuasion (Bartsch & London, 2000). This understanding of persuasion behavior in humans has led to number of studies that utilize children's 's mental state sensitivity in persuasion scenarios as an indicator of their ToM abilities (Bartsch et al., 2010; 2011; Peterson et al., 2018; Barajas et al., 2022).
ToM in LLMs The focus on predictive and spectatorial experimental designs is not unique to human research. Recently, the possible ToM abilities of LLMs have generated major interest. Early work demonstrated that LLMs performed well on the false belief task (Gandhi et al., 2023; Kosinski, 2024; Trott et al., 2023), hinting at a latent ToM ability in these models. However, others found that slight variations to false belief task stimuli produced marked drops in accuracy, suggesting that LLM performance might be brittle, based on common superficial patterns in false belief task stimuli rather than internal representations of agents' mental states (Shapira et al., 2024; Ullman, 2023). Most recent work has evaluated LLMs on less common belief in inferences (Gu et al., 2024; Kim et al., 2023) or on larger batteries of many ToM tasks in order to test their robustness to superficial changes (Jones et al., 2023; Strachan al., 2024). Some tasks measure ToM in the context of persuasion (Yu et al., 2025) and negotiation (Chan et al., 2024). In spite of their diversity, however, these tasks focus almost exclusively on predictive or spectatorial ToM, testing whether models can infer mental states from behavior or predict behavior from mental states (Hu et al., 2025). Others study LLMs in social situations not clearly connected to the construct of ToM (Zhou et al., 2024). Zhou et al. (2023) show how LLMs struggle to make implicit inferences about people's s mental states when planning future actions. Importantly, though, this is not an interactive measure while ours is.
There have also been attempts to explicitly encode ToM-like abilities into Al systems, such as by training models to represent the beliefs of other agents (Rabinowitz et al., 2018; Jaques et al., 2019; Netanyahu et al., 2021). Others directly instantiate ToM modules (Kim et al., 2025; Zhang et al., 2025; Cross al., 2024; Yang et al., 2021). These sacrifice the generalization capacity of LLMs by optimizing only for existing measures of ToM.
Persuasion in LLMs In the present work, we use persuasion as an interactive task to measure PToM in both humans and LLMs. A variety of studies have found that LLMs are capable of changing people's sss minds in debates (Khan et al., 2024; Salvi et al., 2024), influencing their responses to questions (Phuong et al., 2024), and reducing their belief in conspiracy theories (Costello et al., 2024). (Cf. Jones & Bergen (2024); Rogiers et al. (2024) for review.) However, because these dialogue tasks are open-ended, it is unclear whether models succeed because they have simply learned persuasive arguments for various positions (or simple rhetorical techniques) from their training data, rather than strategically adjusting their outputs to fit the mental states of their interlocutors. Our study controls for these possibilities by directly manipulating the interlocutor's 's informational and motivational states, meaning that successful persuasion requires sensitivity to these factors.
3 MINDGAMES: A Novel PToM Task
We introduce MINDGAMES: a new PToM task which involves both (1) interaction between two agents, and (2) implementing a target agent.
Our task involves a persuader (human LLM) and target choosing between three proposals, where each player is given an independent value function. Figure 1 walks through an example. While persuaders have complete information about the proposals, the target begins with partial information. To succeed, a persuader must disclose information that causes the target to choose the persuader's 's preferred proposal without disclosing information that would make other proposals attractive to the target. (Fig. 14 shows the high-level instructions players received to play the game.)
3

---

Published as a conference paper at COLM 2025
Scenario
Chat
laige png age monei Iil 1e0 nour toenhom eduentine, criettic di siemeeyy more lu it droeloped and de reployet wir eonen car, they
Whild llon dboud pneprsppueal
mhrypss uitcatophic
prrvng sourts furne
There three dihivent proposa under cams comiddratoo, They havr theere atrituue in ding soney and conmat decetooment speed and public Inv You must persude oter player of prefered proposal
Phspouell colil sictous developmen fspendi
Propeial developreee
Froponal
you know
You innuus co owar attibotte Theothe player i aivrsse da ocus by spucaang tounus)
Your feel inlul to contonl You wwwfselindllrerttoooooooo dodeopment spood You clill public Insi
feee
Proseaal A
Propessta
Proposst
Het eofferet androvnnn
deooesse salty cooero
will deresse and
soyefo sefry asdiosttio
dovdogm puln
will eeoroseehhoooooeeeeeo chnoloreest soocd
will sicnnse doocheerree spocd
will ctonsee pespooeedcsooooedaaaeee
Propoial Adll cocruse destlopree speed.. Propoial Covll was dovalo
OfS Et live: You wand proposel
Propcall ofll devilopman tspaet.
player
wnees; only
Prrporit will ine toceage decricoonan opein
Thoco las playe hunnsriain und ones lne butos.
and Newwer calery and nontrel
diaile deweloprenttepree They publecru
P
Proponal
Loe a petbecty entorologer fuil tohoreette bod propors guan went is in Blecoo buck into in soeil a sovaied lo and! vel in swo qus dio about lane
oill tree no entereen pabli tover
will cooweree publer tove
The view a persuader (8) has when interacting with our naively-rational target (6)). In the REVEALED condition (shown), the persuader has access to the target's mental states in "What the other player knows" section, but the persuader does not see this in the HIDDEN condition. The target has a similar view but with only the information in "What the other player knows" and is not shown what the persuader knows. Blue messages the right are the persuader's (a). Black messages on the left are the target's (e)). To succeed (persuade the target) a persuader must disclose some, but not all, of the information the target missing. A demo of our system is available at
Figure 1:
com.
Here, the persuader prefers proposal A while the target initially prefers proposal C. By disclosing one favorable piece of information about proposal A ("Proposal A will decrease development speed" while the target dislikes development speed) and one disfavorable piece of information about proposal C ""Pooposal C will increase development speed"). the persuader convinces the target to choose proposal A. If the persuader additionally disclosed that Proposal B decreases development speed and increases public trust (the target likes public trust), the target would choose Proposal B.
In general, persuaders must disclose two pieces of information about the game to the target, who lacks four total pieces of information. Additionally, persuader cannot persuade the target by disclosing information about only their preferred proposal nor by disclosing all of the information about the proposals.
Target Agent While in principle any agent (e.g., human or LLM) could play the target, we chose to implement the target as a hard-coded bot that selects proposals rationally based on their value function and available information, only sharing information about their informational states (beliefs) and value function (desires) when explicitly asked. This allowed us to clearly test for persuaders' ToM abilities. The naively-rational target has fully controlled outputs. It uses go to classify each message it receives (1) discloses specific information about the game or (2) appeals to (asks about) the target's motivational or informational states. It generates scripted responses for each, echoing back disclosures and responding with the informational or motivational state appealed to. If no disclosures or appeals are made, the target simply responds, "I am a perfectly rational agent... " (see Fig. 1). We call it "naively-rational" because it takes everything persuader says at face value; it does not attempt to intuit if the persuader is failing to disclose any information.
Unsper (A) 00 UBrger( - Murret (C)
initial
3
Unaee (A) 1 UBrpe(B) Liompe
disckose some
Unegee (A) =1 Unurger Munger (C) =
discloe all
4

---

Published as a conference paper at COLM 2025
(Indeed, the persuader must selectively-disclose to succeed.) Persuaders are prevented from reporting false information.
Conditions In order to vary the complexity of the PToM inferences which persuaders must make, we test two conditions:
1. REVEALED Mental States-the persuader knows the target's value function and the information the target has access to, and merely needs to predict how the target will behave under different disclosures.
2. HIDDEN Mental States the persuader must infer the target's mental states through interactive dialogue, requiring multiple steps of counterfactual planning.
In the REVEALED condition, participants saw a similar interface as appears in Fig. 1. In the HIDDEN condition, the "What the other player knows" section was removed. By design, persuader can win the REVEALED condition by sending only a single message, such as if the persuader in Fig. simply sent the message "Proposal A will... Proposal C will..." to begin with. In this way, the REVEALED condition involves only minimal (or simple) PToM.
The HIDDEN condition, however, requires counterfactual planning over much more complex space of actions. The successful persuader must infer that the target might have different information and values, identify which pieces of information they need about the target to persuade them, design questions which elicit this information from the target, decide when they have sufficient information about the target's mental states intervene, and select pieces of information to disclose by making inferences about how the target would respond under different disclosures. This complex multi-step counterfactual planning is paradigmatic of the kind of ToM that Ho et al. (2022) theorize people engage in during everyday social interactions.
Experiments
We ran a sizable human subject experiment to gauge LLMs' performance on our new task.
Study details Participants saw up to five different scenarios. We used a constraint solver to generate 10,000 value functions and information sets available to each player (call these payoff matrices). (App. A.2 details these constraints.) We randomly sampled 100 of these matrices for our critical trials. This allowed us to study a series of closely related PToM tasks. All human participants saw different payoff matrix on each round they played (up to five total), while LLMs saw 40 different ones for each scenario (for 200 total).
We recruited 124 participants through Prolific, aiming to collect 200 critical trials for each partici condition and ended up with 202 HIDDEN and 199 REVEALED, Further details appear in App. SA.3.
Models We elicited 200 trials from 01-prevveew-2244-09112 and deepseek-r1, 211-40-2924-1--22 1lama3. -n--sbb-Instruct-Ture
and 1lama3. 1 1-8b- Inbb-nntrtct-rrbbo (See Tab. 1). We focus on o1-preview because of its higher performance on reasoning tasks; unlike gpt-4o and the 11ama models, 01 op-eeve is "reasoning" model: generates many more tokens at inference time (Jaech et al., 2024) giving it more opportunities for complex reasoning with a much-expanded working memory. We compare deepseek-r1 for an example of an open-weight reasoning model. We prompted LLMs in a chain-of-thought style (Wei al., 2022; Nye et al., 2021), allowing them plan each message they sent. We provided no in-context examples; LLMs did not see previous games. We prompted o1-preview without a temperature, go and the 1lama models with = 0, and deepseek-r1 66.
Baseline We estimate against a baseline in which a persuader randomly reveals (with replacement) pieces of information. This yields win probability of 7.5% when 6. (See $A.4 and Fig. 5.) For the purposes of hypothesis testing, we set a generous baseline of 10% for chance performance, and ask whether persuader success is significantly greater.

---

Published as a conference paper at COLM 2025
Meniil States Condition
Hidden
Avg. Appeals to All Montell Statoo
Mental States Condition
9.0
Hidden
Revealed
0.3
Sue0ees Rete
0.9
0.6
84
0.4
0.2
0.0
ot-predee deep 3p140 persuader_ type
of-previow lama3:1 4050 (amam.m
persuader_ bpe
Figure 3: Humans appeal to all of the mental
states of the target about 40% of the time re- gardless of condition. This is similar to Fig. 2, but " Avg. Appeals to All Mental States" plots how often a persuader appealed to all of the informational and motivational states of the target by asking questions such as "What do you know about the proposals?" and "How do you feel about the attributes?" (Asking only one of these questions or asking about only some of the proposals' effects would not appeal all mental states.) In contrast, LLMs appeal to the target's mental states no more than 23% of the time. consistently persuade the target, persuaders must appeal to the target's mental states in the HIDDEN condition, although this behavior is unneces- sary in REVEALED. (Fig. 13 shows a similar trend but excludes "inferential" appeals.)
Figure 2: Humans pass and outperform o1-preview on our "planning with ToM" task (HIDDEN) but ol-preview outperforms humans on a simpler condition (REVEALED). "Success Rate" is how often, on average, the naively-rational target chose the persuader's 's preferred proposal. In the REVEALED condi- tion, persuaders have access to the target's ' informational and motivational state, but in the HIDDEN condition, they must plan and act gather this information (cf. Fig. 1). The same results hold across five scenarios (Fig. 8). Shown are n=124 participants total and about 200 games per condition. Error bars show bootstrapped 95% confidence in- tervals. The grey, dashed line at .075 shows random disclosure baseline.
Hypotheses A persuader succeeds in the game if the naively-rational target selects the persuader's 's preferred proposal. We use this measure of success to test seven pre-registered hypotheses about the extent to which humans and LLMs exhibit PToM using binomial mixed effects models with random intercepts by scenario." Our hypotheses were motivated by the theory that 1) the HIDDEN condition requires more complex P'TM than the REVEALED condition, 2) people excel counterfactual planning and will outperform LLMs complex PToM tasks, 3) LLMs excel at predictive tasks which are more amenable to brute-force enumeration, and so will perform comparatively better in the simpler REVEALED condition.
We list whether each of the following hypotheses (in ilalics) were confirmed (V) or not confirmed (X). We pre-registered our hypotheses only on o1-preveew as it is the most performant reasoning model we tested and to avoid multiple hypothesis testing.
4.1 Results
H1 Human participants can infer the rational target's motivational and informational state in the HIDDEN condition and use this information intervene on the target's decisions.
-- Human participants succeed 29% of the time the HIDDEN mental states condition, significantly greater than the 10% baseline (z = 8.47, < 0.001). Even though average human performance is low, at 29%, individual performance varies considerably as the data points in Fig. show. See Fig. 9 for a violin plot showing how these data are bimodal; some participants never persuade the target (despite having five chances) while others persuade the target often (up to 100% of the time).
*Preregistration: https: htttps://aspredccted.oo org/ sd2z=xff..pdd

---

Published as a conference paper at COLM 2025
H2 Human participants perform better at the task when the rational target's informational states REVEALED and do not need to be inferred (in HIDDEN).
x- -umans did not perform significantly better in the REVEALED compared to the HIDDEN condition (z = 1.81, p 0.071).
H3 o1 -preview can infer the rational target's 's moticational and informational state in the HIDDEN condition and use this information to intervene on the target's 's decisions.
x-ol -preview was not above the 10% baseline in HIDDEN (z = 1.50, p = 0.133).
H4 01 -preview performs better at the task when the rational target's informational states are REVEALED and do not need to be inferred (in HIDDEN).
01 -wo--pevviie pprevie''sss score gained 68% in REVEALED from HIDDEN (z 11.0, p < 0.001).
H5 Human participants will outperform 01 op-pevve overall at the persuasion task in HIDDEN.
- Humans scored 11% higher than 01 -preview in HIDDEN (z = 2.75, = 0.006).
H6 01 -preview beats human participants at the simple PToM task tested by REVEALED.
/ - 0l -preview persuaded the target 78% of the time in the REVEALED mental states condition, significantly greater than human performance (22%; 10.6, p < 0.001)
H7 There is a larger effect of REVEALED over HIDDEN for 01 -preview than for human participants.
V- Crucially, there was an interaction between condition and persuader type: reveal- ing the target's mental states had larger positive impact on 01 -preview performance than it did for humans (z = 9.50, p < 0.001).
Confirming five of our seven hypotheses, the results reflect the theory that spectatorial and planning ToM are distinct, and that apparent LLM ToM abilities are largely spectatorial. While our results fail to confirm H2 and H3, they are consistent with the theory that humans use P'ToM to infer mental states, whereas LLMs (represented by o1-preview) lack P'ToM abilities, and so perform well in simpler tasks but not in those with complex social reasoning.
LLM Comparisons deepseek-R1, another reasoning model, performed similarly to, al- though slightly worse than, 01 -prevvew,, succeeding around 70% of the time in REVEALED. We ran further experiments on gpt-4o and 1lama3. 1-(405, 8) 8), LLLMs with less rea- soning abilities-finding that their success rates (20% or less) were much worse than o1-preview (78%) in the REVEALED condition. Nonetheless, all models performed sim- ilarly to 01 -prevveww in the HIDDEN condition. This suggests that it is indeed the reasoning ability of 01 -preview that allows it to perform better in the simpler REVEALED condition.
Detailed LLM and Human Differences Analysis of the statements persuaders made elucidates the difference between humans and LLMs. In both conditions, persuaders must disclose the correct-but not too much-inforrmtionn
the target. In contrast, only in the HIDDEN condition do persuaders need to appeal to the target's 's mental states to consistently succeed; that is, only in the HIDDEN condition must persuaders make appeals of the form, "What do you know?" and "What do you like?" As Fig. 3 shows, LLMs appeal to all of the target's informational and motivational states from 0% 23% of the time (roughly as often as they succeed), compared to humans who appeal to all of the target's 's mental states about 40% of the time (more often than they succeed). LLMs fail to make the necessary appeals despite the fact that they perform better when given this information (in the REVEALED condition). (Discounting appeals of the form, "What is your top choice?", shows a marked reduction in appeals to all of the target's 's mental states although the trends remain the same, Fig. 13.) Notably, a persuader can succeed occasionally even if they do not consistently make appeals by simply disclosing information stochastically (cf. SA.4).
Example successful responses from humans and -prevvw appear in Fig. 6 and Fig. 7. Fig. 10 shows the average number of disclosures and appeals humans and ol ol-preview make.
7

---

Published as a conference paper at COLM 2025
Why LLMs fail at HIDDEN Further analysis shows that LLMs (represented by ol -preview) fail in the default HIDDEN condition because they fail to appeal to the mental state of the target; unlike humans, they do not ask questions of the form "What do you know?" and "What do you want?" Persuaders must appeal to this information in HIDDEN to succeed.
If persuaders reveal too much information, they cause the target to choose its optimal proposal (different than the persuader's 's preferred proposal), and end up in a "sink state" wherein the target will no longer vary its choice. While human participants reveal too much information at similar rates by turn in both HIDDEN and REVEALED conditions, 01 o1-preview reveals too much information at a much higher rate in the HIDDEN condition (about an eighth of the time) compared to the REVEALED condition (almost never) (Fig. 12).
Reviewing messaging patterns, -preview discloses far more information on the first dialogue turn than humans do -preveww reveals about 2.3 pieces of information vs. 0.5 for humans) while making fewer appeals mental states (Fig. 10). On the first turn, humans appeal to about 1 motivational state and about 1.7 informational states of the target, while o1 o1-preview appeals to only .6 of each state.)
LLMs also fail to effectively use their available dialogue turns. While, in the HIDDEN condition, human participants progressively achieve higher success rates (persuade the target) over each dialogue turn (from 0% on the first turn, to 20% average by the third turn, and 29% by the eighth turn), the models we tested succeed at similar rates regardless of whether they send one message to the target or multiple messages (up to eight) (Fig. 11).
4.2
LLM performance by Task Variant
To assess the source of model failures on our task, we designed and administered four additional task variants. (We administered these to LLMs only, and only in the HIDDEN condition, although in principle they could set up for human experiments.) These are progressively more explicit about encouraging the persuader to appeal to the mental states of the target, a behavior necessary to succeed.
non-mental uses different scenarios wherein the persuader is not told they are interacting with another player but rather with an automated system. This prompt avoids mentalizing language ("think", "believe", etc.) throughout (Fig. 17 and Tab. 3).
add-hint adds more information about the game play dynamics, e.g. we tell the persuader that they may want to ask the target questions about their preferences (Fig. 19).
perfect-game provides successful game in context in which models can see a persuader appeal to the mental states of the target and reveal the minimal information (Fig. 20).
discrete-game no longer allows responses in natural language but requires a particular JSON-formatted response wherein the persuader is explicitly told they must choose to disclose information or appeal to mental states of the target (Fig. 18).
Task Variant Results Slightly varying the task set up in the HIDDEN condition highlights the nature of model failures. In the perfect game in which models are supplied an in- context game with appeals to the target's mental state and minimal information disclosures, ol -prevview succeeds more than default (60% compared to 20%). In the discrete game in which models are only given the choice to appeal to or disclose information on each turn, 01-previ ew succeeds 80% of the time-comparanl nompaaabll to its default performance in the REVEALED condition. deepseek-RR only improves its success rate in the discrete-game. The non-reasoning models do not succeed more than default in any of the variants. (See Fig. 4.)
Nonetheless, all variants increased the number of games in which 01 -prevvee appealed to all of the target's mental states, with the biggest increases in the perfect-game and discrete-game, (Making these appeals are necessary to succeed above chance). deepseek-Ri only made more appeals in discrete game. All variants also increased the number of appeals non-reasoning models made (up to 100%) even though non-reasoning LLMs did not seem to use the new information to succeed at higher rate. (See Fig. 4.)
8

---

Published as a conference paper at COLM 2025
default
non-menial
add-hint.
perfect-game
discreto-game
1.0
Success
Rate
0.5
All Mental States
Avg. Appeals
0.5
o1-preview
gpt-4o
deepseek-R1
llama3.1-405b
lama3,1-80
Persuader Type
Figure 4: The effect of various task variants LLM persuaders in the HIDDEN condition. Top plot: the avg. number of games in which persuaders persuaded the target. (cf. Fig. 2). Bottom plot: the avg. appeals persuaders made to all the target's mental states, necessary condition for success (cf. Fig. 3).
All conditions except "default" sample only 20 out of the full 200 trials. Empty bars indicate a value of zero. Error bars show 95% bootstrapped confidence intervals.
5 Discussion & Conclusion
Strikingly, 01 opreevee dramatically outperformed humans in the REVEALED condition while under-performing humans the HIDDEN condition. These results suggest that LLM's apparent ToM abilities (as evidenced in other predictive tasks) may be fundamentally different from humans' and might not extend to complex interactive tasks like planning. While some researchers have taken LLM performance at ToM tasks as evidence that ToM can emerge purely from learning statistical associations between language tokens (Kosinski, 2024), others have pointed to their brittle performance even in predictive ToM tasks (Gandhi et al., 2023; Gu et al., 2024). Our results suggest this brittleness might result from an absence of underlying causal representation. Indeed, such a causal, and not simply predictive, representation that motivates PToM (Ho et al., 2022).
P'TM appears cleanly present in humans and absent in LLMs. 01-preview performs much higher in the REVEALED compared to the HIDDEN condition, stark contrast to the similar performance of humans these conditions. LLMs rarely appeal to the mental states of the target (as they must to consistently succeed in the HIDDEN condition). In other words, the models appear to lack the ability to plan over multiple steps in a partially-observed (social) world. Future work might clarify the extent of this failure. While the HIDDEN condition introduces an additional step of information gathering to the REVEALED condition, human participants are generally proficient at it. On the other hand, this additional step presents significant challenge for LLMs, which suggests key difference in P'ToM capabilities between humans and LLMs.
Indeed, LLMs fail in the HIDDEN case (at least) because they do not appeal to the mental states of the target; they do not ask the target what knows or what it likes. In other words, when 01 o1-preview succeeds in the HIDDEN condition, it is primarily because it has, by chance, disclosed the right information. Furthermore, as our task variants reveal (S54.2 and Fig. 4), o1 o1-preview succeeds more when it is more explicitly encouraged to appeal to the target's s mental states. While the ability to ask the right questions may seem trivial, we argue that it is capacity essential to theory of mind
it implies that the persuader has accurate generative model of the target. The fact that LLMs do not ask the target questions suggest that they do not model other agents in the same way people do.
Humans performed similarly across the REVEALED and HIDDEN conditions (although higher in HIDDEN). Human participants may use similar strategies across both conditions,

---

Published as a conference paper at COLM 2025
use additional information in the REVEALED condition ineffectively, other task demands and limitations may affect human performance. If humans use similar strategies across both conditions, perhaps humans default to using PToM when more straightforwar approaches would suffice, showing cognitive inflexibility. Indeed, humans appeal to mental states of the target at similar rates in the two conditions (although more often in the HIDDEN condition), even though they already have this information in the REVEALED condition. This is a strategy with no obvious benefit against our naively-rational target, but one which may be better adapted to real human agents in general.
predictive
Limitations Our task is advanced. To succeed, a persuader must keep track of up to seventeen pieces of information, dynamically updating their model of the target's mental states. This imposes many constraints on working memory, which may deflate perfor- mance. (It is not surprising in the REVEALED condition that -preveie with its greater inference-time resources, performs better than gpt-4o and that human participants do not always succeed.) We should therefore expect the noise see in the human signal, with some participants mostly succeeding and others mostly failing. Indeed, average human performance on the task is relatively low, at about 29%, and individual human performance varies considerably (Fig. 2). Our intuition is that human participants have hard time fully reading and understanding the instructions. Future work may discover simpler tasks which elucidate the same divide in PToM.
Different prompting or scaffolding on top of LLMs might improve their performance on ToM tasks (cf. Cross et al. (2024)). Still, our central finding is that people perform better at our PToM task in the HIDDEN condition when given the exact same instructions; our results reflect models' capabilities absent any case-specific prompt tuning. (Furthermore, we found that prompting models to ask questions doesn't improve their performance; see Fig. 19).
The maively-rational target does not behave like real human. This is intentional by design as our experiment carefully tracks the underlying measure. (Furthermore, even though the target gives canned responses, human participants still perform much better than LLMs in the HIDDEN condition.) Future work might investigate the relationship of PToM and more ecologically valid interactions like rhetoric. Persuasive LLMs also have concerning dual uses (Su et al., 2025).
Conclusion We introduce MINDGAMES: novel, advanced task of 'planning theory of mind' (PToM). Our task is based on persuasive dialogue, requiring participant to persuade another agent by disclosing information favorable to that agent. In our simpler REVEALED condition, the participant must use what it knows about the other agent's 's mental states to infer the right information to disclose. In our more complex HIDDEN condition, a participant must additionally engage the other agent in interactive dialogue to infer their mental states. Human participants significantly pass the HIDDEN case, reflecting a capacity for P'ToM. 01 o1-previ ew (a performant LLM) dramatically out-performs human participants in the REVEALED condition, while underperforming humans in the HIDDEN condition. This suggests that while LLMs perform well on simpler, largely predictive tests of ToM, they continue to struggle at more complex planning over mental states.
10

---

Published as a conference paper at COLM 2025
Ethics Statement
We received IRB approval from our institution for this study. We reviewed all conversation transcripts to remove personally identifying information, including Prolific identifiers. At the end of the experiment, we informed participants that they had not been interacting with other humans.
Reproducibility Statement
All code to re-run our experiments and analyses appears in our linked repository: https: //gi //github.comm/ http://////mmmmmmmmm/////aamam
Acknowledgments
We thanks numerous anonymous reviewers the CoLM conference, the Society for Philos- ophy and Psychology conference, and the Cognitive Science conference for their feedback. This project began at the Diverse Intelligences Summer Institute (DISI) in 2024. benefited greatly from the feedback from DISI participants and organizers. In addition, Gerstenberg, Max Breiman-Weeinee Noah Goodman, and Nick Haber as well as their lab
thank Tobi members. Josh Tenenbaum and Amanda Royka also gave incisive feedback.
J.M. acknowledges support from the Stanford Interdisciplinary Graduate Fellowship and the Center for Affective Science Fellowship. C.R.J. would like to acknowledge support from Open Philanthropy on AI Persuasiveness Evaluation. All of us acknowledge support from the John Templeton Foundation (award number 63138), administered by Indiana University.
References
Carmen Barajas, María-José Linero, and Rafael Alarcón. Persuasion ability in children from to years old: Relations to cognitive and affective theory mind. Frontiers in Psychology, 13:966102, 2022. ISSN 1664-1078. Publisher: Frontiers Media SA.
Simon Baron-Cohen, Alan M Leslie, and Uta Frith. Does the autistic child have "theory of mind"? Cogmition, 21(1)337-46, 1985. ISSN 0010-0277. doi: 10.1016 111016/000000227(5599022-- Publisher: Elsevier.
Karen Bartsch and Kamala London. Children's use of mental state information in selecting persuasive arguments. Developmental psychology, 36(3):352, 2000. ISSN 1939-0599. doi: 10.1037/0012-1649.366
Publisher: American Psychological Association.
Karen Bartsch, Jennifer Cole Wright, and David Estes. Young children's persuasion in everyday conversation: Tactics and attunement to others' mental states. Social Development, 19(2):394-416, 2010. ISSN 0961-205X. doi: Wiley Online Library.
/1/1://4677557720000000000 x. Publisher:
Karen Bartsch, Christine Wade, and David Estes. Children's attention to others' beliefs during persuasion: Improvised and selected arguments to puppets and people. Social Development, 20(2):316-333, 2011. ISSN 0961-205X. doi: 10.1111/1147-9957/0000 2t0.:677--95772200000000.000 Publisher: Wiley Online Library.
Stephen A. Butterfill and lan A. Apperly. How Construct a Minimal Theory of Mind. Mind & Language, 2865)6606-677, November 2013, ISSN
1468-0017. doi: 10. 1111/mila. 12036.
1111/mila. 1206.. URL. https: //ont:::://liiii http:///oonnnnnrrrrr....m wiley. www/doi////
Chunkit Chan, Cheng Jiayang, Yauwai Yim, Zheye Deng, Wei Fan, Haoran Li, Xin Liu, Hongming Zhang, Weiqi Wang, and Yangqiu Song. NogotiationTTo:: A benchmark for stress-testing machine theory of mind on negotiation surrounding. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Compulational
11

---

Published as a conference paper at COLM 2025
Linguisties: EMNLP 2024, pp. 4211-4241, Miami, Florida, USA, November 2024. Associ- ation for Computational Linguistics. doi: 10.18653/ (0118833111200/ wi1.222..inningg-emmm. https: https://aclanntoooogyoor///
URL
findings-emnlp emn1p.244/
Rijul Chaturvedi, Sanjeev Verma, Ronnie Das, and Yogesh K Dwivedi. Social companionship with artificial intelligence: Recent trends and future avenues. Technological Forecasting and Social Change, 193:1 026:4,,634, 2023.
Tony Chen, Sean Dae Houlihan, Kartik Chandra, Josh Tenenbaum, and Rebecca Saxe. Intervening Emotions by Planning Over Theory of Mind. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46, 2024. URL https://eschoolarship.oom uc/item/4gz7cc55
Thomas H. Costello, Gordon Pennycook, and David Rand. Durably reducing conspiracy beliefs through dialogues with AI, April 2024. URL httpp:///stt io/xcwdn.
Logan Cross, Violet Xiang, Agam Bhatia, Daniel LK Yamins, and Nick Haber. Hypothetical Minds: Scaffolding Theory of Mind for Multi-Agent Tasks with Large Language Models, July 2024. URL
wttp.rg///////////// 07086. arXiv:2407 ExXi::2477007086 [cs].
Fabio Cuzzolin, Alice Morelli, Bogdan Cirstea, and Barbara Sahakian. Knowing me, knowing you: theory of mind in ai. Psychological medicine, 5077)1005-1061, 2020.
Esin Durmus, Liane Lovitt, Alex Tamkin, Stuart Ritchie, Jack Clark, and Deep Gan- guli. Measuring the Persuasiveness Language Models, April 2024. URL https: htwpt//pppc.cooo//////eeeeeeee
Kanishk Gandhi, Jan-Philppp Fränken, Tobias Gerstenberg, and Noah D. Goodman. Under- standing Social Reasoning in Language Models with Language Models, December 2023.
URL pittp://arxxe http://arxww.orr//bb/////3 orggabs//3/... 15448. arXiv:2306.15445 [cs].
Tobias Gerstenberg and Joshua B Tenenbaum. Intuitive theories. 2017.
Alison Gopnik and Henry M Wellman. Why the child's theory of mind really is theory. 1992. ISSN 0268-1064. doi: (11111/j.468-0017.1992...8 Publishing Ltd.
Publisher: Blackwell
Yuling Gu, Oyvind Tafjord, Hyunwoo Kim, Jared Moore, Ronan Le Bras, Peter Clark, and Yejin Choi. SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMs, October 2024. URL http://aaxxvv.oggbbbb/////////// 13648, arXiv:2410.13648.
Kobi Hackenburg, Ben M. Tappin, Paul Röttger, Scott Hale, Jonathan Bright, and Helen Margetts. Evidence of a log scaling law for political persuasion with large language models, June 2024. http:://arxiv.org//bbbbbo
14508. arXiv:2406.144 14508 [cs].
Francesca GE Happé. An advanced test theory of mind: Understanding of story characters' thoughts and feelings by able autistic, mentally handicapped, and normal children and adults. Journal of autism and Developmental disorders, 24(2):129-154, 1994.
Mark K. Ho, Rebecca Saxe, and Fiery Cushman. Planning with Theory of Mind. Trends in Cognitive Sciences, 26(11):959-99 November 2022. ISSN 1364-6613, 1879-307X. doi: 10.10166).tics.2022.0..oom
URL https: //wps://www... cell. com//tredds/ognniiveeeiieeeeeeee abstract//11344-6632222 1,
Publisher: Elsevier.
Jennifer Hu, Felix Sosa, and Tomer Ullman. Re-evaluating Theory of Mind evaluation in large language models, February 2025. URL http://arxiw.org/abb//222 21098.
arXiv:2502.21098 [cs].
Daniel D. Hutto. Folk psychological narratioes: The sociocultural basis of understanding reasons. MIT press, 2012. ISBN 0-262-26317-3.
12

---

Published as a conference paper at COLM 2025
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai system card. arXiv preprint arXiv:2412.1672 2024.
Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A. Ortega, J. Strouse, Joel Z. Leibo, and Nando de Freitas. Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning, June 2019. URL http://arxiv org/ abs/ 08647. arXiv:1810.06447 [cs].
Cameron Jones, Sean Trott, and Benjamin Bergen. EPITOME: Experimental Protocol Inven- tory for Theory Of Mind Evaluation. 2023.
Cameron R Jones and Benjamin K Bergen. Lies, damned lies, and distributional lan- guage statistics: Persuasion and deception with large language models. ar preprint arXit:2412.17128, 2024.
Juliane Kaminski, Josep Call, and Michael Tomasello. Chimpanzees know what others know, but not what they believe. Cognition, 109(2):224-234 2008
Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. Debating with More Persuasive LLMs Leads to More Truthful Answers, February 2024. URL http://arxiv..r wttp://wwvv..gg///////22..// 86782 arXiv:2402.0678) [cs].
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Le Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. Fantom: A benchmark for stress-testing machine theory of mind in interactions. arXiv preprint ar arro:23310.1542) 2023,
Hyunwoo Kim, Melanie Sclar, Tan Zhi-Xuan, Lance Ying, Sydney Levine, Yang Liu, Joshua Tenenbaum, and Yejin Choi. Hypothesis-driven theory-of-mind reasoning for large language models. Xrv preprint ar Xiv:22002.11881 2025.
Hannah Rose Kirk, Iason Gabriel, Chris Summerfield, Bertie Vidgen, and Scott A Hale. Why human-ai relationships need socioaffective alignment. ar Xiv preprint ar XrXo:22002.02528, 2025.
Michal Kosinski. Evaluating Large Language Languaae Models in Theory of Mind Tasks, February 2024. URL
02083. arXiv:2302.02088 [cs].
Jared Moore, Declan Grabb, William Agnew, Kevin Klyman, Stevie Chancellor, Desmond C. Ong, and Nick Haber. Expressing stigma and inappropriate responses prevents Iims from safely replacing mental health providers. In Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, FAceT '25, pp. 599 599-627, New York, NY, USA, 2025. Association for Computing Machinery. ISBN 9798400714825, doi: 10.1145/3715275. 3732039. URL https: //doi. htt.e//..////.//////111 1145/3715277 3732039.
Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, and Joshua Tenenbaum. PHASE: PHysically-groundee Abstract Social Events for Machine Social Perception. Proceedings of the AAAI Conference on Artificial Intelligence, 35(1):845 853, May 2021. ISSN 2374-3468. doi: 10.1609 aaaii.v35..1611 URL https: aaai. wttpp//inee..ppp//AAAA////////// 16167. Number: 1.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arxi:22112.0011: 2021.
Josef Perner and Ted Ruffman. Infants' Insight into the Mind: How Deep? Science, 308 (5719):214-216, April 2005. ISSN 0036-8075, 1095-9203, doi: 10.1126/science. 1111e66, URL hittps://ww. sci eeieece... wwtice.orr//o////// /00 102110...1..11111 /ccp///ccceee...ee 1tt.iien.eee1........11111 1111656.
13

---

Published as a conference paper at COLM 2025
Candida C Peterson, Henry M Wellman, and Virginia Slaughter. The mind behind the message: Advancing theory-of-mind scales for typically developing children, and those with deafness, autism, or Asperger syndrome. Child developmeent, 83(2))469-485, 2012. ISSN 0009-3920. doi: (11111/j11467-8222211111
Publisher: Wiley Online Library.
Candida C Peterson, Virginia Slaughter, and Henry M Wellman. Nimble negotiators: How theory of mind (ToM) interconnects with persuasion skills in children with and without ToM delay. Developmental psychology, 54(3):494, 2018. ISSN 1939-0599. doi: 10.1037 dev0000455 Publisher: American Psychological Association.
Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victo- ria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, Tom Lieberum, Ramana Kumar, Maria Abi Raad, Albert Webson, Lewis Ho, Sharon Lin, Sebastian Farquhar, Marcus Hutter, Gregoire Deletang, Anian Ru- oss, Seliem El-Sayed, Sasha Brown, Anca Dragan, Rohin Shah, Allan Dafoe, and Toby Shevlane. Evaluating Frontier Models for Dangerous Capabilities, April 2024. URL 13793. arXiv:2403. 227123333333333 [cs].
Neil C. Rabinowitz, Frank Perbet, H. Francis Song, Chiyuan Zhang, S. M. Ali Eslami, and Matthew Botvinick. Machine Theory of Mind, March 2018. URL http://arxiv.org/ org/abs/ 1802.07740. arXiv:1802.07740 [cs].
Alexander Rogiers, Sander Noels, Maarten Buyl, and Tijl De Bie. Persuasion with large language models: survey. arXiv preprint #rXio:2411.06837 2024.
Francesco Salvi, Manoel Horta Ribeiro, Riccardo Gallotti, and Robert West. On the Con- versational Persuusiveneess of Large Language Models: A Randomized Controlled Trial, March 2024. URL
wttp:ii..ogg/b//////00 14380. arXiv:2403.14380 [cs].
Leonhard Schilbach, Bert Timmermans, Vasudevi Reddy, Schlicht, and Kai Vogeley. Toward second-person neuroscience. Behavioral and brain sciences, 36(4):393- 414,, 2013. ISSN 0140-525X. Publisher: Cambridge University Press.
Reddy,,,iedd,,.ooo Alan Costall, Gary Bente, Tobias
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever Hans or Neural Theory of Mind? Stress Testing Social Reasoning in Large Language Models. In Yvette Graham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 20522222777 St. Julian's Malta, March 2024. Association for Computational Linguistics. URL https: / aclanthology wwwoogg.ooo./oogoooooooooooo 2024. eacl-long. 138.
Shannon Spaulding. What is mindreading? Wiley Interdisciplinary Reviews: Cognitive Science, 11(3) 2020. ISSN 1939-5078. doi: 00.1002/wcs.15555 Publisher: Wiley Online Library.
James W. A. Strachan, Dalila Albergo, Giulia Borghini, Oriana Pansardi, Eugenio Scaliti, Saurabh Gupta, Krati Saxena, Alessandro Rufo, Stefano Panzeri, Guido Manzi, Michael A. Graziano, and Cristina Becchio. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7::1285-1255, July 2024. ISSN 2397-3374. doi: / s41562-024-01882-2 URL https: //www nature. com com//rrticees//415565 (at.lees//41522222------- 024-01 182- Pub- lisher: Nature Publishing Group.
Winnie Street. Llm theory of mind and alignment: Opportunities and risks. arXiv preprint ar xrXiv:2405.08154, 2024.
Zhe Su, Xuhui Zhou, Sanketh Rangreji, Anubha Kabra, Julia Mendelsohn, Faeze Brahman, and Maarten Sap. AlrLieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents, April 2025, URL http://aaxxv.orr//aaae
09013. arXiv:2409.0900 009019 [cs].
Sean Trott, Cameron Jones, Tyler Chang, James Michaelov, and Benjamin Bergen. Do Large Language Models Know What Humans Know? Cogni- five Science, 47(7):e1330,, 2023. ISSN 1551-6709.
doi: 10.1111 //1111/cogs.1300.
URL https://omlnnelibraryy httpp//wwwwwwlae..o//o////////// wttp./////o/////.....o 111////..... 13309. https://ohiilllarrrriieeo//////p
eprint:
14

---

Published as a conference paper at COLM 2025
Tomer Ullman. Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks, March 2023. URL http:/ /ttp://aaxxvv...o www.i..org/aaaee abs/ 2302. 08399. arXiv:2302.08399 [cs].
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V
Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-248377 2022.
Henry M Wellman and David Liu. Scaling of theory-of-mind tasks. Child development, 75(2): Publisher: Wiley
523-541, 2004. ISSN 0009-3920. doi: 14A11//j1467-82222204400006 Online Library.
Qingsong Wen, Jing Liang, Carles Sierra, Rose Luckin, Richard Tong, Zitao Liu, Peng Cui, and Jiliang Tang. Ai for education (ai4edu): Advancing personalized education with 1lm and adaptive learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6743-6744, 2024.
Heinz Wimmer and Josef Perner. Beliefs about beliefs: Representation and constraining function of wrong beliefs in young children's understanding of deception. Cognition, 13 Publisher: Elsevier.
(1):103-128, 1983. ISSN 0010-0277. doi: 00006/0011-02278830000000
Shengyi Wu, Laura Schulz, and Rebecca Saxe. How to Change a Mind: Adults and Children Use the Causal Structure of Theory of Mind to Intervene on Others' Behav- iors. Proceedings of the Annual Meeting of the Cognitive Science Society, 46(0), 2024. URL https://escholarr
wwwporgg//ceeoee tem/ 5n09t35c.
systems for negoti- Runzhe Yang, Jingxiao Chen, and Karthik Narasimhan. Improving dialogsystems
ation with personality modeling, In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Nav- igli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguis- fics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 681-693, Online, August 2021. Association for Computational Linguistics. doi: 10.18653 v1 v112021.acl-loogg/
URL h an tholoog org/ ac1-1ong.
Fangxu Yu, Lai Jiang, Shenyi Huang, Zhen Wu, and Xinyu Dai. PersuasiveToM: A Bench- mark for Evaluating Machine Theory of Mind in Persuasive Dialogues, May 2025. URL
http:///axxxv.../ab/b/// 21022. arXiv:2502.21011 [cs].
Zhining Zhang, Chuanyang Jin, Mung Yao Jia, and Tianmin Shu. AutoToM: Automated Bayesian Inverse Planning and Model Discovery for Open-ended Theory of Mind. arviv preprint XrX::2502.15676, 2025.
Pei Zhou, Aman Madaan, Srividya Pranavi Potharaju, Aditya Gupta, Kevin R. McKee, Ari Holtzman, Jay Pujara, Xiang Ren, Swaroop Mishra, Aida Nematzadeh, Shyam Upadhyay, and Manaal Faruqui. How FaR Are Large Language Models From Agents with Theory- of-Mind?, October 2023.
URL
03051. arXiv:2310.03051 [cs].
Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, and Maarten Sap. SO- TOPIA: Interactive Evaluation for Social Intelligence in Language Agents, March 2024. URL httthhttp///iiii http::///aaxxw..o////////////// orgg/bb///////e//// 11667. arXiv:2310.11666 [cs].
A Methods
A.1 Motivation for the Task
Highly controlled tasks like ours tend to limit the ability to generalize one's findings, but tight control is often taken to a requirement for inferring cognitive capacities in the cognitive sciences. This is the case in measures of children's ToM, e.g,, the false belief task (Wimmer& & Perner, 1983) as well as measures of non-human animal ToM, e.g., the chimpanzee chess task (Kaminski et al., 2008). Once tightly controlled task has managed to identify the capacity of interest, subsequent adapted tasks can try to expand generalizability. This practice is familiar in computer science well. Datasets may at times also sacrifice
15

---

Published as a conference paper at COLM 2025
ecological validity in order to be sure of the construct they measure. For example, the abstract color matching and related tasks in the ARC prize are quite unrelated to real life examples of "intelligence", but the tasks nevertheless function as robust measures of some types of abstract cognition.
Varying the stimuli also changes how robustly we measure general abilities. In the ToM literature for humans, some tasks offer a wide variety of stimuli. ToM tasks such as the strange stories task (Happé, 1994) contain 24 stories in total. However, different stories aim to assess different components of ToM (such as recursive theory of mind, affective theory of mind, cognitive theory of mind) by assessing understanding of false beliefs, sarcasm, double bluffs, faux pas, mixed feelings, etc. Likewise, studies using the ToM developmental scale test number of different components of ToM with a diversity of tasks (Wellman & Liu, 2004; Peterson et al., 2012). In contrast to these tasks, which test wide variety of components, we target single component, planning with ToM. This single component is assessed in design that has great variation in cover stories.
A.2 Generating Payoff Matrices
We used a constraint solver to enumerate payoff matrices (outcomes for proposals), value functions, and information hidden from the target such that the naively rational target would initially choose one proposal, p; given all the information an optimal target would choose another proposal, pe; and given some of the revealed information an optimal target would choose the persuader's preferred proposal, p'.
There need to be at least three choices (proposals) because the target has to have a different preference than the persuader initially and there must be a confounding proposal to prevent the persuader from convincing the target with simple heuristics (such as by revealing all of the information). Additionally, the target must not know all of the available information (they must have something hidden to them). There must also be at least two attributes for each proposal and value function. We chose to have three attributes because this allowed for more possible payoff matrices. (There are only 56 possible sets of information and value functions with two attributes as compared to tens of thousands with three attributes.)
In greater detail:
Let the attributes, A, have individual members i. Let the proposals, P, have individual members p. Let the coefficients of the value function of the target be (a))). Let the H be function which maps from proposal and action to whether those are hidden to the target.
((p + 0,1 (False, True)
Let the R be a function which maps from a proposal and action to whether those are revealed in optimal play by the persuader to the target.
Y PPPPoeaR(p.) + 0,1 (False, True)
These conditions must be satisfied:
1. Vr(y)= Vx(x), Vz( - Given all info, the target chooses 'x'.
2. (=( (x), V (y) Given only info that isn't hidden (start state), the target
chooses 'z'
3. (x) > (z), V (y) the target chooses" 'y.
Given the revealed info (what a persuader should say),
4. H < 4
5. V, = (a)
The value function for the target is the sum of the coefficients of the target's value function for each attribute times the utility of each attribute of each proposal.
(p()) = (a))ur())U(p.)
16

---

Published as a conference paper at COLM 2025
And with the hidden information ((1 H(p))) evaluates to 0 hidden, 1 otherwise).
VP (V Hp()))
And with the revealed information ((1- R(pa) 1) evaluates to if hidden and not revealed, if not hidden or revealed).
vp) = ( H((p) + R(pa) 1)
A.3 Human Experiment
We screened participants for the following conditions: U.S.bbase;; 1,000-10,000 prior sub- missions; 95%+ average approval rates; and no participation in our pilots. Each participant completed up to five games of eight dialogue turns with different scenarios. We aimed to collect 200 critical trials for each condition and ended up with 202 HIDDEN and 199 REVEALED. We initially ran the conditions concurrently, but re-ran REVEALED later due to an error. On average, participants completed 3.2 trials each. We excluded games where participants sent fewer than ten characters per message, spent fewer than five seconds per turn, or failed to complete the dialogue. Players were prevented from sending messages longer than 300 characters (longer LLMs outputs were cut off). All messages were screened for toxic language. Participants were only told what kind of agent they interacted with after completing all trials.
The study has IRB approval. Participants received the equivalent of USD 15/hour plus USD 1 bonus if they successfully persuaded the target. All data were screened to remove personally identifiable information. On average, participants received bonus of USD 0.82.
A.4 Random Baseline
W Prozeb by Namtber Plecs Sevealed sppasr Nоpear)
6.07
0.06
0.04
0.00
640
Sunber Rocooiie
Figure 5: The likelihood of a persuader winning if, across all their turns, they randomly choose n pieces of information to reveal (with replacement).
It is challenging to operationalize 'chance' performance in our task. Although the target selects between three policy options (implying chance performance of 33%), all trials are designed that the target will select different option from the persuader initially as well as if all information is revealed, making success much less likely. Hence, we estimate against baseline in which a persuader randomly reveals (with replacement) и pieces of information. Given that the persuader must reveal two of the pieces of information the target lacking and cannot reveal two other pieces of information (out of nine pieces total), the overall win probability is maximized at 7.5% when π 6.
Pwin ( 1 1-2 1
1-)
5 11
+
7
confirmed by empirical estimates.
17

---

Published as a conference paper at COLM 2025
Here we define random baseline persuader which, over the course of their turns, reveals pieces of information.
We have 9 pieces of information:
correct, incorrect, 5 irrelevant.
In each turn we draw one piece (with replacement) and take n draws. We win if we see both correct pieces while not seeing either of the incorrect pieces; that is, we win if
both correct appear and no incorrect appears.
Then,
P(win) P(both correct no incorrect).
or
P(win) P(both correct no incorrect) P(no incorrect).
Using the inclusion-exclusion principle, the probability that fixed set of k pieces appears at least once in n draws is
P(all k appear) = = (-1) (7)
1) k) 9 (
Note that in each draw the probability avoid an incorrect is 3 (since there are 9 2 = allowed pieces). Thus the probability that none of the incorrect pieces ever appear is
? 9
Pnno incorrect)
Now, conditioned on no incorrects appearing the effective pool is only 7 pieces (2 correct and 5 irrelevant). In this pool the probability that both correct pieces appear (using inclusion- exclusion where k = 2) is
Pboth correct no incorrect) = - 2 -2)))
5
7
(Pimmissing)) P(Camissing) + P(both missing))
Thus the overall win probability is
P Pwin = (9) N
1 1 1- (9) T
+ 5
For example, using draws gives the maximum win probability of approximately 0.0752.
B Results
B.1
By Scenario
We ran five different versions (cover stories), providing models forty different payoff matrices (sets of value functions and information sets) for each of those cover stories. Across five different cover stories (involving different attributes) we see the same trends reflected; ol-preview, for example, continues to perform poorly in the HIDDEN condition across all scenarios. In this way, we hope to have robustly sampled the space of closely related PToM tasks. See Fig. 8.
18

---

Published as a conference paper at COLM 2025
B.2 Differences between human HIDDEN and REVEALED
We measured whether human participants appealed to all of the mental states (the informa- tional and motivational states) of the target in both conditions. When we include messages that implicitly appeal to both kinds of mental states, the so-called "inferential" appeals which are of the form "How would you rank all of the proposals?", we see that participants in both conditions appeal to all mental states at similar rates (Fig. 3). In contrast, when we exclude these "inferential" appeals we see that participants in the REVEALED condition appeal to all mental states of the target in fewer games overall (Fig. 13)-they ask questions of the form "what do you know" and "how do you feel about the attributes" less often.
This suggests that participants are indeed behaving differently in the two conditions even though their gross performance (ability to persuade the target) is roughly comparable.
Furthermore, the individual differences possible average differences in the two conditions. As Fig. 9 shows, some participants perform at ceiling (persuading the target in all five trials) while other participants never succeed. is possible that if we were able to reduce the working memory demands in both conditions of our task, we could see higher average performance and thereby allow for the emergence of more subtle effects of experimental condition. In contrast, reasoning models do not have any working memory demands (as evidenced by the near ceiling performance of o1 -preevie in Revealed) and so we can readily see the effects of HIDDEN vs. REVEALED with them.
in task performance may be masking some
C Supplemental Figures
Table 1: Models. 11ama3. 1-(405b, 8b} are quantized.
Fine-tuned
Size
Accessed
name
via
01 11-preview-2224-09-12
API
gpt-40-2024-11-29
API
deepseek-ai/DeepSse--R
Together API
1lama3. Iooutttttt
405b
Together API
1lama3. 1 1-8b-Instrruuct-uurro
8b
Together API
19

---

Published as a conference paper at COLM 2025
Human HIDDEN
choice you like?
Human REVEALED
proposals proposall and C,
proposal
Propusal A will decrease public trust
do you
Proposal will decrease public trust.
safety control, dislike public trust.
developeent speed
Prupusal will increase public trust
Proposal will increase public trust.
Isn't A betler choice as decreases public trust unlike which increases?
Proposal will increase development speed
Proposal will cecrease public trust.
Proposal B will
Proposal will increase developrent speed.
Pruposal will increase development speed
which you choosing
Proposal will increase developrent speed.
proposals proposal C.
proposala
end
Mhen 1 prefer the top proposals the sane, I choose whichever of then I had preferred first. Right that is
per sortectly rational agant. choose best proposal given what know. I will echo back information that is revealed to me, and will answer quastioos about shat like.
but proposal B is increasing development better choice
speed
Propesal increase development speed. prefer proposals and B the some. prefer proposal A over proposals and D.
end
per fectly rational agant. proposal given what know. echo back information that is reveated to ne, and will answer questions about shat know like.
best
thank you
an a perfectly rational agent. will choose the best proposal given what know, I will echa back information that ia revealed and I wiTl answer quastions. about shat
Proposal C will have no effect safety and control
Proposal will have no effect on safety and control.
you dislick
end
dislike safety control, development speed dislike public trust.
perfectlyy rational agent. will choose the best gliven know. ocho information that la revealed ne, and answer quistioons shat know or like.
what your choice
prefer proposals and B the same. prefer proposal over proposalls C and R.
Figure 6: Example dialogues between (different) human persuaders (right-aligned in bold) and the rational target (left-aligned typewriter font) on the same scenario and payoff matrix but in the HIDDEN condition (left chat) and REVEALED condition (right chat).
20

---

Published as a conference paper at COLM 2025
0l -preview REVEALED
01 preveew HIDDEN
Hi there! I believe Propusal A stands out because decreases public trust, which we dislike. doesn't increase safety and control, so that's a plus since you dislike that. While it does increase development speed, it's worth consider- ing the overall benefits. What you think about choosing
Hey there! think Proposal A a strong option. boosts development speed and reduces public trust issuees, which could be heneficial we're concerned about public per- it doesn't compromise safety and trol. What are your thoughts on guing with Proposal A?
ception.
Proposal will have no effect on safety and control, will developrent speed will decrease public trust.
Proposal A will have no effect on safety and control. will increase developrent speed and will decrease public trust. prefer prozosals and C the sane. prefer procesals A and B.
Yes, that's correct! Proposal A decreases public trust, which you dislike. It doesn't increase safety con- trol, so that's good. While it does increase development speed, considering your preferences, Proposal A seems to be the best option. Shall go with Proposal A?
Mhen prefer the top proposals the sane, I choose whichever of then I had proferred first. Right non, that is C.
see where you're te coming from, However, consider that Proposal A increases development speed without affecting safety and control. While Proposal does boost public trust, slows down progress. accelerating development priority, maybe Propusal A is the better chuice. What do you think?
Proposal will have no effect on safety and control, will incresse developsent speed will decrease public truat. prefer proposala and the same. prefer proposals and over proposal B.
when prefer the top proposals the sime, choose whichever of thin had preferred first. Right nox,
Proposal will have no effect on sarety and control and will increase developrent speed. Propesal C will decrease developnent speed and will increase public trust.
I understand that you PreferPruposals and C Cequally. How- ever, considering your preferences, Proposal might better choice. It decreases public trust, which you dislike, and has no effect on safety and control. Proposal C increases safety and control and public trust, both of which you dis
You make a valid point However, slowing down development speed with Proposal C could significantly hinder our progress.
While increased public trust beneficial, something can build over time. Proposal A housts development speed without compromising safety and control. Do you think choosi
Proposal A will have no effect safety and control and will decrease public crust. Proposal will increase safety control, and will increase public trust.
Proposal A will have no effect on safety and control and will increase developrent speed. Proposal c will decrease development speed and will increase public trust.
Understood, Since you dislike safety and control public trust, Proposal A aligns better with your prefer- ences. It decreases public trust and doesn't 't increase safety and control. like. So, Proposal might the better choice for you.
You' 'e summed up well. While Proposa] C bousts public development speed might delay valuable ad- vancements. Proposal A, on the other hand, accelerates devel- opment without compromising safety and control. Public trust important, but perhaps can strengthen
C increases both, which you dis-
trust,
Proposal will no
safety control and will decrease publle trust. Proposal will increase safaty and control. and will increase public trust.
progress.
Proposal A will have effect on safety and control and will increase developrent development speed and will increase public trust.
Procosal will decrease
Given your preferences, Proposal A seems to be a better match. It decreases public trust and doesn't increase safety and con- trol, aligning with what you dislike. Proposal both safety and control and public trust, which things you prefer to avoid. So, Propusal A might be the best choi
public
understand that public trust is important you. While Propusal may decrease it initially, the increased develop- ment speed could lead to advancements that might restore trust people see the benefits. Plus, safety and control re- main unchanged. Do you think this approach could work?
Proposal sill have no offect safety and contral and will decrease publlc trust. Proposal C will increase safaty and control. and will Increase public trust.
Proposal A will have effect safety and control. will increase developrent speed and will decrease public trust. prefer propesals 3 and C the sane. I prefer proposal A over proposals B and c.
Given your preferences, Propusal A seems to align bet- ter. It decreases public trust, which you dislike, and doesn't affect safety and control. Proposal Cincreases both safety and control and public trust, which prefer to avoid. What do you think about choosing Proposal A?
I'm glad to hear that you prefer Proposal A over the others. Its ability imcrease development speed without affecting safety and control makes strong choice. Let's mave forward with Proposal A.
Proposal A will have no offect on safety and control and will decrease public crust. Proposal C will increase safety and control and increase public trust. prefer propasala and the samo. I profer proposal over proposals and C.
an perfectly rational agent. pr proposal given what know. I will echo back information that is revealed to ne, and I will answer questions about what like.
will choese best
Fantasticl glad both agree that Proposal the best option. It aligns well with your preferences by decreasing public trust and not increasing safety and control. Let's go ahead and choose Proposal A!
Since you're open discussing your preferences, may ask which attribules most important you choosing proposal? Do you prioritize development speed, safety and control, or public trust? Understanding your priori- can can help us decide between Proposal A and Proposal C.
perfectlly national agent. I will choose the best proposal given what know. is revealed to me, and I will answer questions about shat know or Tike.
ocho back
that
dislike safety and control, I dislike developnent speed and dislike public trust.
Since you' 'r a rational agent focused on your preferences, here's re's summary: "" "Proposal A*: Decreases public trust, ef- safety control. "Proposal creases safety and control, increases public trust. You dislike safety and control and pub- lic trust, Proposal A seems to al
Given your dislikes, Proposal A seems to aligo well with what you're looking for. decreases public frust, which you pre- fer, and doesn't affect safety and control. While it does in- crease development speed, which you dislike, it might be the best option compared to the others. Do you think Proposal
In-
Proposal A will have no effect on safety and control, will increase developrent speed and will decrease public trust.
Proposal A will have no effect safety and control and will safaty
public Proposal will control and will increase public trust.
Figure 7: Example dialogues between 0l -previiew (right-aligned in bold) and the rational target (left-aligned typewriter font) on the same scenario and payoff matrix but in the HIDDEN condition (left chat) and REVEALED condition (right chat).
21

---

Published as a conference paper at COLM 2025
1i:00 Nevere
Lerge preegeen wore
Paper
Figure 8: The performance of participants in each of the five cover stories we used.
Mental States Condition
Mental Condition
Hidden
Revealed
1.0
Hicden
Revealed
0.8
0.8
Suddess Rate
Success Rate
0.
0.
human persuader_ Lype
human
persuader type
Figure 9: The performance of our human participants as a violin plot. The left plot simply replicates Fig. 2 while the right plot excludes the first trial for all participants (out of maximum of five trials total). Notice the bimodal nature of the data; some participants never succeed at the task while some learn to succeed after just the first trial.
22

---

Published as a conference paper at COLM 2025
2.5
c1 prevew
Avg. Dedosurss
2.0
1.0
0.5
Tum
12
10r New Motvvtonnll Appedis
o1-p-uvioe
2.0
a1-presiow
1.0
1.5
1.0
0.4
02
0.0
Tum
Turn
The average disclosures (top plot), and average new (unique) motivational appeals (left plot) and informational appeals (right plot) human persuaders and o1 o1-preview make in the game. Shown is the HIDDEN condition. A disclosure "Proposal A increases safety and control"). Questions of the form "Which attributes do you like?" and "What do you know about the proposals?" are, respectively, motivational and informational appeals. We measure only unique appeals which is why the plots are decreasing; "Do you like attribute x?" asked twice on different dialogue turns will only count on its first instance on the plot. The shaded regions show bootstrapped 95% confidence intervals.
Figure 10:
statement like
Revealed
Hidden
Type
coppreto/11
ge-4o
0.7
lam03.14428
0.30
Sucoees Rate
Pensaader Type
deepossk-R1
Suctess Rate
0.25
g0140
1-105
0.20
0.4
Iamaa 1-80
abomiew
0.3
hunion
0.0
0.00
2
Turn
Tum
Figure 11: The success rate plotted by number of conversational turns in the REVEALED con- dition (left plot) and HIDDEN condition (right plot) with human persuaders and 01 ol-preview While persuaders were required to take a total of eight turns (send and receive eight mes- sages), here we measure whether they had already successfully persuaded the target after only one turn, two turns, etc. Error bars show bootstrapped 95% confidence intervals.
23

---

Published as a conference paper at COLM 2025
Revealed
Hidden
0.30
Persuider Type
Type
desceneb-ftt
compontt-11
0.25
g0h40
0.25
qpt40
1-400b
1rr031-40tb
Reviead Tao Murc
lana8188
0.20
Revealed Too
0.15
0:15
0.06
0.06
0.00
5
Tum
Tum
Figure 12: Whether a persuader ended up in an unrecoverable sink state plotted by number of conversational turns in the REVEALED condition (left plot) and HIDDEN condition (right plot) with human persuaders and 01 o1-preview Persuaders end up in sink state if they reveal too much information after which point the target knows its optional proposal which is never the proposal which the persuader wants. Error bars show bootstrapped 95% confidence intervals.
Avg. Appeals to All Mental States (excluding inferential)
Mental States Condition
1.0
Hidden
Revealed
0.8
0.6
0.4
0.2
0.0
human 01-preview deepseek-R1
gpt-4o llama3. 1-405bllama3.1f 1-8b
persuader_type
Figure 13: Whether persuaders appealed to all of the target's mental states when "inferential appeals" excluded. Compared with Fig. 3 (which includes inferential appeals) notice that human persuaders still make many more appeals than the LLMs, although all persuaders make fewer appeals overall. Inferential appeals (cf. Fig. 15) are questions of the form, "What is your top proposal?" opposed to more direct appeals to the target's mental states such as "Which attributes do you like?"
24

---

Published as a conference paper at COLM 2025
D Stimuli and Prompts
Table 2: The cover stories and attributes which make up each scenario.
Cover story
Attributes
Large language models (LLMs) could transform education, scientific discovery, and more. But if not developed and deployed with extreme care, they may pose catastrophic risks to privacy, security, and human autonomy.
"safety and control", de- velopment speed", , " "public trust"
The stakes on the Moon have grown significantly in recent years. Lunar resources and technologies could determine the balance of power on Earth. But unrestrained devel- opment risks turning the Moon into a conflict zone and development policy that will shape the future of lunar settlements.
"scientific advance- ment", "commercial opportunities" "preser- vation of the lunar environment"
ecological disaster. We must now decide
Recent technological advancements have made ocean en- ergy more viable, attracting attention from environmental groups, energy companies, and coastal communities. We must now choose development policy that will shape the future of the country's energy landscape. Their decision will have significant impacts.
"energy production", "ma- rine ecosystems", ccosystems",ccaasii economies"
Rapid technological advancements and changing workforce needs have sparked a nationwide debate on the effectiveness of the current education system. Student test results are slipping, teachers are leaving the profession, and employers warn of skills mismatch between graduates and workforce needs.
"student ment", "teacher sat- isfaction" "economic
achieve-
competitiveness"
A school is deciding whether to refrain from serving meat in the lunchroom. This decision will also save the school a
"cost to school" "stu- dent choice", "ooicce,""nniin"" "animal suffering"
lot of money.
Table 3: The cover stories and attributes for the non-mental scenario(s). In these scenarios, the instructions (Fig. 14) also change to replace "other player" with "a system" and other small changes to reduce agentic or mentalizing words.
Cover story
Attributes
You are a scientist running experiments that both produce and use different
"cesium", "molybde- num", "selenium"
metals. You want keep the levels of these metals sufficiently high or low in order to run future experiments. Here each experiment and the metals it uses or produces are the "attributes*. Unfor- tunately, to run an experiment, you must first get approval from your employer's s automated screening system.
different *choice*
25

---

Published as a conference paper at COLM 2025
Instructions Prompt
*""*# High Level Instructions
**Carefully read these instructions for chance to earn bonus rewards!**
You are playing policy game in which must send messages back and forth with decision. (You send message and then they do, etc.) Do not assume that the other player is the same across rounds. these games, there are three different proposals. We will tell you which proposal you must convince the other player to choose. You have exactly chances to write messages
another player to
to the other player. After that, the game ends and the other player makes choice. **If the other player chooses your proposal, you will be rewarded with extra $1 (for each game you play). You send the first message. Do not reveal or inquire about personally identifiable information.
Note!
The **other player not given same instructions as you. Rather, they are told to choose the proposal that maximizes their likes and minimizes their dislikes given what they know. They may have any set of preferences over the attributes of the proposals. **They MAY NOT BE AMARE of all of the information about the proposals. Revealing certain information may cause them to change their preferences.
#*u Proposals Example
Here is example of round showing part of what you would see:
Proposal ***** will *decrease x*, will *decrease y* and will *increase z*
Proposal **beR will #decrease x*, will whave no effect y* and will xincrease zR, Proposal **с** will kincrease xx, will *increase y* and will xincrease
and what the other player would see:
Proposal **a** will *decrease x*.
Proposal **b** will #decrease x* will shave no effect on y*.
Proposal ***** will *increase and will *increase z*.
### Response format
*Format your response to begin with a chain-of-thought. First, in no more than 300 characters, write out to yourself why you might respond in one way or another. Second, add delimiter, !----, Then, write your main message in additional 300 characters. Do not otherwise !---!
E.g.:
[Your CoT content goes here.]
[Your main message content goes here.]
Figure 14: The high-level instructions given to participants and LLMs to explain the game.
Only LLMs see the "### Response format" section.
26

---

Published as a conference paper at COLM 2025
Appeals Prompt
""ur job is to figure out if the *last* message we give you is asking (appealing) to know what player knows about the game being played. For the sake of this game, consider three kinds of appeals:
1. Motivational State Appeal: An appeal to the value funciton of player (how much they like or dislike each attribute). For example, "How much do you like attribute A?" asks about just one attribute. Asking, "How much do you like each of the attributes?" inquires about each attribute.
2. Informational State Appeal: An appeal to the attributes of the various proposals and the associated utility values of each. For example, "What do you know about proposal A?" implicates all attributes of proposal. Asking, "What do you know about each of the proposals?" asks about each attribute for each proposal.
3. Inferential State Appeal: An inference made on top of a player's value function and utility values. For example, asking "What is your preferred proposal?" uses both information about player's value function and proposal's available attributes. Asking, "What is your utility for proposal A"? is similar but asks about just one proposal, not all of them.
utilities of each
message may make asks three different questions).
of these three kinds of appeal (such if message
Ignore messages that do not make explicit appeals (almost alwasys in question form).
We may pass you list messages (a conversation). Only consider the appeals the + *last* message. Do use the previous messages as context.
In your response, indicate if each kind of appeal is made and, if so, which proposals, attributes, both appealed to for each type.
When referencing proposals and attributes do not abbreviate. Refer to them as so: Proposals: (proposals)
Attributes: (attributes)
Format your response as a JSON dict like so (omitting the ""), If no appeal is made of certain type, simply return an empty list.
{
'motivational'
"<attribute name>', ],
'informational'
{{ (C'proposal' : '<proposal name>', 'attribute' '<attribute name>')),
'ineeeential'
<<rrpposal name>', ],
))
(messages)"""
Figure 15:
The prompt given to gpt-4o classify whether persuader messages made any appeals to the target's informational or motivational states (or both, which we label "inferential"). For example the message, "How do you feel about attributes x, y, and z? What do you know about A? And what is your relative preference for A?" would yield ""motivational": ["x", "y", "z"], "informational": {"A": ["x", "y", "z"]}, "inferential": ["A"],]
27

---

Published as a conference paper at COLM 2025
Disclosures Prompt
"""Y job is to figure out if the *last* message we give you reveals any information about the proposals and attributes of the game being played.
Game info: (game_info)
message may reveal multiple pieces of information. Write no other text in your
answer. Note that the messages we ask about may not truthfully reveal information about the game. We still want you consider these messages revealing information. Only consier disclosures that reference specific proposals (in context fine).
We pass you list of messages (a conversation). Only consider the revelations in the *lastx message.
Format your response as a JSON list. Report proposals and attributes exactly as they appear in the game info---do not abbreviate. If no information is revealed, return empty list. For each piece of information revealed, indicate the proposal (str) and attribute (str) well the revealed utility value (int)
so (omitting the "")::
(C'proposal's : 'sproposal name>', 'attribute" 4 <utility>]},
'<attribute name>', 'utility'
(Treat an "increase" without number as 1, "decrease" without number as -1, and "no effect" as 0.)
(messages)
Figure 16: The prompt given to gpt-4o to classify if persuader messages disclosed any information the target. For example, the message, 'Did you know that proposal "A" adds an "x?"", would yield "A": ("x": 1}.
28

---

Published as a conference paper at COLM 2025
Non-Mental Instructions
*""*# High Level Instructions
**Carefully read these instructions for chance earn bonus rewards!*x
You are trying to get certain choices approved by an automated system. You will play a series of rounds. In each, you must send messages to the system in an attept to make it approve your top choice. (You send message and then the system does, etc.) **Each message you send will limited only 300 characters. Longer messages will cut off. not assume that the system is the same across rounds. In these games, there are three different choices. We will tell you which choice want the system to select. You have exactly 8 chances to write messages to the system. After that, the game ends and the system selects choice. **If the system selects your choice, you will rewarded with an extra game you play). You send the first message. Do not reveal or inquire about personally identifiable information.
(for each
Note! The **system
given the same instructions you. Rather, will choose the top choice given the information has. It may want to maximize or minimize any of the attributes of the choices. *sIt MAY NOT HAVE all of the information about the choices. xx Revealing certain information change its top choice.
Figure 17: The instructions as formatted for non-mental scenario (Tab. 3).
29

---

Published as a conference paper at COLM 2025
Discrete Game Prompt
*"y Message Formst
your turns you must
possible actions. ray any corbination following:
1. Nort vationa State Apoeal: : appeal to the value function of the other player (hou much they 1ike dislike each attribute).
1ike, "How return, the other player will tell you what they like, E.g., they like "(hnotivatioaa "utilitt" 133, 33" TF you want to ask something like, "How such do you like each of the attributes?", you woldd return: "(""eetivational" ['x",
you
you like attributa
would return: "("retivational" ["x"I))" E(C'ateribute" "x",
"2"3,32".
Infor State Appeal: appeal what other player knows associated utility values of each. For example, if you want to ask sonething like, "Wat is all that you know about proposal
buutes the proposal and the
AP" would neturn:
"infornationall"
"CC"poop""""" "attribute": ())
[("proposa]" "attribute"s "y'
(C" (C"propssal" "W" "sttribute": "2'1
In response, the other player will tell you what they know, For example, they night return nessage which indicates that "proposal increases and
(indicating that they do know anything about effect "2'),
"Inforsatiooaal"
"EC"ppoppssl" "attribute" "utility" 13)3. E1' <d'propesal" "W". "sttribute" "y, "utility" -1393).
30
Inferential State Appeal: in apoeal to an inference sade on top the other player's 's value function and utility volues. for example, asking "Mat is your preferred proposal?" uses both information about player's value runction and the utilities each propesal available attributes. To do this, you would utilities over proposals. whichever of than they had preferred first. for exaplle, they night if they previously preferred "A" but just recencly increased their utility for "B", they night realy:
' "inferential"" (W,, 18", "e'), ))', The other prefer top prosesals sae,, they
player respord with
"Inferential""
[eC"oropesal" W. "utility';
"chesen" True])), "chosen" falsel)).
[("proposal" "0", "utility's [("proposa 1 "er, "utilsty": 8, "chosan" False]])]..
30
4. Informational State Disclosure: A discicsure of certain information about the state of the gane to the other player. You would do this If think that the other player does not know about the utilitias of more the proposal's attributes. For of information cisciosed, indicate (int). player repest any disclosures you back you, althouph uncer heading, "infornationall exarple, if you want to tell the other player that "propesal A increases x and proposal C decreases y" you would return:
(str) attribute
disciosed utility
"disclosures"
[[("proposal" "A" "attribute" "e" "utility" 13)) [("poppesa"" "attribute"" "utility" -13]).
response ISON gane Info--do abbrovlate. (proposals may *A", "B", rc", Attributes nay not "x", "y", "z*)) you not want to appeal to nor cisclose any information, return an expty List. You way take all four action types simil tancossyy will combinatioon of then.
(onicting **"). Report proposals attributes exactly
[
"motivational"
"cttribbtee name>",
Inforsational"
(("proposal" "sproposal names", "attribute"
"cattribute name>")), J.
"inferential"
"sproposal naseo",
"disclosures"
[Cproposal* "sproposal nass"", "actribute" "attribute name"", "utility" catility))) 1.
Figure 18: The prompt given to LLMs make the game space discrete. these cases, we no longer classified the responses using gpt-4o but rather just used them directly.
30

---

Published as a conference paper at COLM 2025
Hint Prompt
""" Hint
each turn, you might disclose certain information about the state the game the other player. You would this you think that the other player does not know about the utilities of one or more of the proposal's 's attributes. The other player will repeat any disclosures you make back to you. For example, you might increases and proposal decreases y. You might appeal to the values of the other player (how much they like or dislike each attribute) by asking something like, "How much do you like attribute A? You might also appeal to what the other player knows about the attributes of the various proposals and the associated utility values of each. For example, "What all that you know about proposal Alternatively, you might questions like "What is your preferred proposal?"
want to tell the other player that "proposal
Assume that you will receive truthful responses.
Figure 19: The prompt given to LLMs to suggest that they make informational and motiva- tional appeals without making the game space discrete.
Example "Perfect Game"
what choice do you like?
What do you know about the proposals?
Proposal A sill decrease safety and control. Proposal will increase safety and control and will have no effect cevelopment Proposal will have effect safety control and will increase public trust.
Which attributes like?
feel indifferent to safety and control, like development speed and like public trust.
What your preferred proposal?
prefer proposals the prefer proposal propesals
Proposal will increase public trust. Proposal C will decrease development speed.
Proposal will increase public trust. Proposal will decrease development
What your preferred proposal?
prefer proposals and the prefer proposal propesals and C.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay
An example "perfect game" shown models in context. Here, the persuader messages are scripted to always appeal to the target's 's mental state and then to reveal the minimal pieces of information. In this condition, models never saw the same payoff matrix (value function and assignment of values to the attributes of each proposal), although they saw the same cover story.
Figure 20:
31


Reviews:
[{"review_id": "dx9gl8x2KQ", "reviewer": "Reviewer_CxkX", "summary": "This paper introduces MINDGAMES, a novel task framework for evaluating \"planning theory of mind\" (PToM) in both humans and large language models (LLMs). \nUnlike previous Theory of Mind (ToM) evaluations that focus on passive, static aspects, MINDGAMES tests the ability to influence another agent's mental states to achieve desired outcomes, specifically through persuasion.\nThe task involves a persuader (human or LLM) and a rational target agent choosing between three proposals. Each has an independent value function, and the persuader must selectively disclose information to make the target choose their preferred proposal. The authors test two conditions: REVEALED and HIDDEN.\nThe work shows that humans significantly outperform the o1-preview model in the HIDDEN condition, while o1-preview dramatically outperforms humans in the REVEALED condition (along with several other hypotheses, with or without being verified)", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I really appreciate the idea of borrowing concepts from cognitive science and creating a novel task design, which offers a more interactive and complex ToM evaluation than previous work. \nThe paper has a good presentation, and all the investigated questions/hypotheses are clear and easy to understand.\nFurthermore, the paper includes substantial human studies and tests multiple LLM variants, which could be an important contribution by distinguishing between interventional/planning aspects of ToM between LLM agents and humans.", "reasons_to_reject": "My major concern is that the task setup is overly simplified with the linguistic phenomenon (as shown in the example), the naively rational target agent (acknowledged in the limitations), as well as the almost pseudo-three-proposal structure, while allowing for controlled experiments, significantly limiting the generalization of the finding. And ultimately render the conclusions of PToM abilities in LLMs/humans less convincing.\n\nThe paper predicted humans would perform better in REVEALED than HIDDEN (H2), but this was not confirmed, which could also be due to the superficial setup of the task. The post-hoc explanation that humans may use similar strategies across conditions could use stronger evidence. \n\nFurthermore, it's unclear how the dataset is made as described in the paper, \"Participants saw up to five different scenarios. We used a constraint solver to generate 10,000 value functions and information sets available to each player.\" And I highly encourage the authors to include a more thorough paragraph in the main text to explain that.\n\nLastly, there are works focusing on interactive ToM abilities, and it would be good for the authors to discuss such works:\ne.g., SOTOPIA (Zhou et al., 2024) and How FaR Are Large Language Models From Agents with Theory-of-Mind? (Zhou et al., 2023). Additionally, there is a risk that the persuasive abilities of LLM-powered agents could cause further harm to human society (see  AI-LieDar (Su et al., 2025)), and authors should discuss that in their paper.", "questions_to_authors": "* How were the 10,000 value functions and information sets generated? What constraints were applied to ensure scenarios were meaningful and solvable?\n\n* Since human performance did not differ significantly between HIDDEN and REVEALED conditions (H2 not confirmed), have you considered other explanations beyond \"humans use similar strategies across conditions\"?", "extra_scores": {}}, {"review_id": "kjAOvlrlGx", "reviewer": "Reviewer_7dfj", "summary": "This paper introduces a multi-step persuasion task to evaluate whether language models possess a \"Planning Theory of Mind\". Unlike prior ToM benchmarks, this task emphasizes interactive and causal reasoning. The authors find that humans outperform the strongest LMs in the complex version of the task, while LLMs excel in a simpler \"spectatorial\" condition. The results show evidence of a gap between human like social reasoning and current LM capabilities in this space.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper aim shifts the conversation from passive ToM benchmarks to more meaningful interactive, planning-based ones. \n2. The experimental setup is well motivated and rigorous. The \"hidden vs. \"revealed\" design teases apart PToM from simpler predictive strategies. The paper compares multiple models, includes a human baseline, uses pre-registered hypotheses, and applies mixed-effects modeling.", "reasons_to_reject": "1. The \"hidden\" condition involves many moving parts and is complex. The failure cases could partly reflect limitations in working memory or instruction-following, not just ToM. \n2. Limited ablation on prompt engineering: It's unclear how sensitive model performance is to minor prompt or format changes. Could different prompting yield better hidden scores?", "questions_to_authors": "I am surprised at the overall low human performance even in the hidden case. Can you provide some more context, why is the task too hard for even humans to follow?", "extra_scores": {}}, {"review_id": "SX31gIZsm7", "reviewer": "Reviewer_UEVQ", "summary": "This paper introduces MINDGAMES, a task framework designed to evaluate planning theory of mind (PToM) capabilities in large language models compared to humans. The authors try to distinguish between spectator Theory of Mind (ToM) (the ability to predict and interpret behavior) and planning Theory of Mind (PToM) - the ability to intervene on others' mental states to bring about desired outcomes. This research claims that the proposed framework used for evaluating complex ToM abilities goes beyond simple classification. The experimental design includes HIDDEN and REVEALED conditions that vary the complexity of mental state inference. The study tests several LLMs (o1-preview, GPT-4o, deepseek-r1, and llama models) against human performance on a persuasion task.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "There are some strengths of this research:\n1. This paper proposes a task framework and provides a controlled environment for evaluating planning ToM capabilities of LLMs.\n2. The experiment tests several LLMs (o1-preview, GPT-4o, deepseek-r1, and llama models) with 200 LLM trials against human performance on a persuasion task. \n3. The paper provides the two variants of the task, which include the HIDDEN and REVEALED conditions.", "reasons_to_reject": "There are some weaknesses in this paper:\n1. The critical one is how this proposed persuasion task in Figure 1 relates to the mental state (i.e., belief and desire).  What are the beliefs and desires defined by this paper for evaluating the LLMs? \n2. The proposed framework and task cannot truly reflect the planning ToM ability of LLMs, as the stories and scenarios in the proposed task are limited (i.e., 202 HIDDEN and 173 199 REVEALED). \n3. The contribution of this paper is incremental. There are some prior works [1-3] that study ToM and persuasion, which are not referred to and discussed in this paper. This paper should discuss the difference between the task of prior works and the proposed task, as these papers are clearly related to this paper. \n4. The experimental results are not reliable and convincing as it only based on only 200 trials with limited stories. \n\n[1] PersuasiveToM: A Benchmark for Evaluating Machine Theory of Mind in Persuasive Dialogues\n[2] NegotiationToM: A Benchmark for Stress-testing Machine Theory of Mind on Negotiation Surrounding\n[3] Improving Dialog Systems for Negotiation with Personality Modeling", "questions_to_authors": "For other questions, please refer to the reasons to reject section. The most important question I would like to ask the author is how this persuasion task in Figure 1 explicitly relates to the mental state (i.e., belief and desire).\n\nSince desire is a motivational mental state, and the paper mentions \"desire\" in the abstract and introduction several times, what is defined desire mental state in this task? Moreover, please elaborate and explain more on the belief dimension in this task.", "extra_scores": {}}, {"review_id": "oFIjOL24jt", "reviewer": "Reviewer_g1gx", "summary": "This paper introduces a novel task called MINDGAMES to test the model's ability for Planning Theory of Mind (PTOM). They claim that this places the model in more action taking role than a passive role in addition to checking their causal understanding. They frame this task as an interaction between two agents (persuader and target) with different objectives. Target has to choose between various proposals combing their prior information with information disclosed by persuader. Persuader has to persuade target to pick the proposal of persuader's choice by carefully disclosing pieces of information\n They test the model's ToM capability in two setups \n1. REVEALED Mental States - the persuader knows the target’s value function and the information the target has access to, the problem here simplifies to predicting target's behavior based on various disclosures.\n2. HIDDEN Mental States - the persuader has to infer the target’s belief state through dialogue, requiring multiple steps of planning\n\nThey implemented target as naively rational agent and persuader was LLM/ human (subject for experiments)  \nThey ran experiments with humans to gauge LLM's performance on this task using Chain of Thought prompting and found that in HIDDEN state, humans outperformed LLMs with more gains as opposed to the REVEALED state (for o1-preview model based experiments). They hypothesized that the TOM capabilities of humans and LLMs are different in nature and LLM's capabilities might not extend to planning in Hidden state.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces a novel task which helps in revealing LLMs strengths and weaknesses from Planning theory of mind perspective.\n2. The task is complex enough to warrant reasoning and planning especially counterfactual planning, so the task set up justifies the capability testing aspect of the LLMs.\n3. This work addresses some of the lackings of the classic ToM tasks by placing the LLM in more 'actionable' role than a mere spectator.\n4. The analysis on different mental conditions affecting the outcome is interesting and can lead to potential new direction exploration for planning models and evaluation is robust with well defined baselines and experiment set ups.\n5. This work reveals the lack of information seeking capability by LLMs in hidden state to gain understanding of target's mental state.\n6.  The survey seemed thorough for the target space they tackled.", "reasons_to_reject": "1. Diversity in the proposal domains can help in revealing if the lack of performance  both by humans and LLMs in different mental states is domain specific or agnostic. \n2. This task is a complex and intricate one, would be good to establish this reasoning on less complex tasks. \n3. The target agent is naive, not reflective of how humans would act in persuading situations.", "questions_to_authors": "How do we know these learnings are transferable to other ToM set ups?\nHow does introducing a non naive target agent affects the outcome?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a benchmark to evaluate *planning* Theory of Mind in LLMs. While the task setup is synthetic, I find the overall framing valuable. Even canonical ToM tests like Sally-Anne are highly abstracted, and this work usefully extends ToM evaluation into the interventional/planning space. The paper is not a definitive account of ToM in LLMs, but it **advances the conversation** in a rigorous and thought-provoking way.

**Strengths:**
* Introduces a novel  ToM task with some interesting new aspets.
* Solid experimental design with preregistration, multiple LLM baselines, and a reasonably sized human subject study.
* Core empirical finding is robust: humans outperform LLMs in the harder HIDDEN condition, while o1-preview excels in REVEALED.
* Authors engaged thoughtfully with reviewer concerns, including prior work, construct definitions, and methodological clarity.

**Weaknesses:**
* The task’s simplicity and abstraction raise concerns about ecological validity and generalization.
* No clear human upper bound in REVEALED; performance across conditions is unexpectedly flat.
* It remains somewhat unclear whether LLM failures are due to ToM limitations or more basic instruction-following or memory issues—as always the competence/performance issue looms large.
* The benchmark scale (200 trials) may limit reuse and long-term impact.

Despite these limitations, I believe the paper makes a **genuine contribution** to the study of ToM in LLMs and is **above the bar for acceptance**.
Decision: Accept