Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Models
Authors: Hyunwoo Kim
Melanie Sclar
Tan Zhi-Xuan
Lance Ying
Sydney Levine
Yang Liu
Joshua B. Tenenbaum
Yejin Choi
Venue: COLM 2025 (2025)
Keywords: theory of mind
reasoning
large language model
inference-time algorithm
TL;DR: We introduce a novel inference-time algorithm, ThoughtTracing, which uses LLMs to probabilistically trace and weight hypotheses about agents’ evolving mental states without relying on questions and ground-truth answers in benchmarks.

Abstract: Existing LLM reasoning methods have shown impressive capabilities across various tasks, such as solving math and coding problems. However, applying these methods to scenarios without ground-truth answers or rule-based verification methods - such as tracking the mental states of an agent - remains challenging. Inspired by the sequential Monte Carlo algorithm, we introduce ThoughtTracing, an inference-time reasoning algorithm designed to trace the mental states of specific agents by generating hypotheses and weighting them based on observations without relying on ground-truth solutions to questions in datasets. Our algorithm is modeled after the Bayesian theory-of-mind framework, using LLMs to approximate probabilistic inference over agents' evolving mental states based on their perceptions and actions. We evaluate ThoughtTracing on diverse theory-of-mind benchmarks, demonstrating significant performance improvements compared to baseline LLMs. Our experiments also reveal interesting behaviors of the recent reasoning models - e.g., o3 and R1 - on theory-of-mind, highlighting the difference of social reasoning compared to other domains.

Published as a conference paper at COLM 2025
Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Models
Hyunwoo Kim!
Melanie Sclar2
Tan Zhi-Xuan
Lance Ying
Sydney L
Yang Liu
Joshua B. Tenenbaum
Yejin Choi
INVIDIA UUiiveriity of Washington
MIT "Stnnford University
Harvard University AAmazon
Abstract
Existing LLM reasoning methods have shown impressive capabilities across various tasks, such as solving math and coding problems. However, applying these methods to scenarios without ground-truth answers or rule-based verification muth as tracking the mental states of an agent-remains challenging. Inspired by the sequential Monte Carlo algorithm, we introduce ThoughtTracing, an inference-tine reasoning algo- rithm designed to trace the mental states of specific agents by generating hypotheses and weighting them based on observations without relying on ground-truth solutions to questions in datasets. Our algorithm is inspired by the Bayesian theory-of-mind framework, using LLMs to approximate probabilistic inference over agents' evolving mental states based on their perceptions and actions. We evaluate ThoughtTracing on diverse theory-of- mind benchmarks, demonstrating significant performance improvements compared to baseline LLMss. Our experiments also reveal interesting be- haviors of the recent reasoning models e.g, o3 and R1 on theory-of-mind, highlighting the difference of social reasoning compared to other domains.
1 Introduction
New reasoning algorithms and training paradigms for large language models (LLMs) have recently achieved remarkable breakthroughs in domains where well-defined ground-truth answers are readily available, enabling partial-solution evaluation and objective verification -e.g., mathematics, coding, and puzzles (Yao et al., 2023; Besta et al., 2024; OpenAI, 2024b; DeepSeek-AI, 2025b). This stands in stark contrast social reasoning, where objective answers are not casily obtainable, making portial-solution evaluation less straightforward. Furthermore, the information-syymmettt
nature of theory-of-mind (ToM) - which requires inferring hidden mental states makes social reasoning more challenging due to its increased uncertainty Zhi-Xuan et al., 2024b; Nafar et al., 2024).
Existing AI research on theory-of-mind has focused on either natural language benchmarks and tailored approaches to solve them (Sclar et al., 2023; Kim et al., 2023; Wilf et al., 2024; Jung et al., 2024; Jin et al., 2024), (Baker et al., 2009; Doshi & Gmytrasiewicz, 2009; Zhi-Xuan et al., 2020; Alon et al., 2023). As L.M-powered AI agents increasingly interact with humans (Collins et al., 2024), the ability to track and infer others' mental states from open-ended textual input will become crucial for broader applications and data synthesis in the social interaction domain.
mental state inference in non-linguistic settings
To this end, we propose ThoughtTracing, an inference-time reasoning algorithm for LLMs that can infer and track the mental states of target agents. Our method is conceptually inspired by the Bayesian theory-of-mind framework (BТoМ; Baker et al., 2017), viewing the input context as a sequence of states and agent actions, and treating ToM reasoning as probabilistic inference about the agent's 's mental states based on their perceptions and actions.
Data and code are available at https:: httpp::/hhynw.... https////huuwnii//tteeeeeeeeeeee wwm/tthoggtttrratttteoo
1

---

Published as a conference paper at COLM 2025
Parse Agent's Trajectory
Perception: lile perceived the warm, inviting interior her bakery. with shelses lired with freshly babed bread and pastr ies, She the oven properly funciioning.
State 1: is baiker in small
Perception
Fronch village proparing her bakery for day
Shee bake her agnatuuee baguettes using her trusyy old oven.
Hypothesis Inference
Initialize Hypotheses
Hypothesis believec
oven needed to be checked.
Action 1: Lile checked the oven's tempenttur eatlier
Hypothesis believec
wos proserly prehcatee.
Updato Weights
based on Action Likellhood
pown is important for her baking.
Original Imput Contout
Hypothesis lila believes the customers will want warm breads.
Resample Hypotheses
Rejuvenate Hypothases
Perception: Lila noticed the suddenn power outage and saw the oven was cooling down, making unsu table for baking.
Perception Inference
State 2: However, a oudden power the oven cool down, making it mursuttale for baking
Hypothesis Inference
Propagate Hypotheses
Hypothesis I: life belleve the
oven shoule have been working.
Action 2: outsge and the oven turning off
the posoer
Update Weights
Hypothesis 2: Llb believec the oven needed to be checked again.
based on Action Likellhood
Hypothesis N: Lil believed the
beers affected
Figure 1: An illustrative overview of our ThoughtTracing algorithm. The original context is parsed into trajectory of the target agent. At each time step, the algorithm generates multiple hypotheses about the agent's 's beliefs and weighs them based on the likelihood of the agent's 's action. These weights are then used to resample the hypotheses, prioritizing more promising ones to be used for the next time step. Further details are in Section
To account for the inherent uncertainty in this task, we follow the high-level structure of sequential Monte Carlo inference (SMC; Del Moral et al., 2006; Lew et al., 2023b), tracking multiple weighted hypotheses about the agent's mental states. Importantly, however, these hypotheses are represented in open-ended natural language, and are both generated and weighted by LLMs. By using LLMs in this way, our method offers a generalizable approach to uncovering the underlying thoughts that drive agent behavior, without requiring the explicit probabilistic models assumed by BToM algorithms, and without relying on question- answer annotations or benchmark-spean speciifc assumptions.
Although Traccig is not specifically designed for solving benchmark questions, we evaluate the algorithm using -tate-of-the-are LLMs on four theory-of-mind benchmarks to assess whether the traced thoughts can enhance downstream question answering task performance. Experiments show that (1) ThoughtTracing consistently improves performance on all tested models across all benchmarks, (2) additional inference- compute (e-g., shain-of-thoughl further aids models to process contexts interleaved with mental states, and (3) models with Thought'raccing outperform reasoning models (e.g., 03-mini and R1) despite using significantly shorter reasoning traces.
Furthermore, our results reveal interesting behavioral patterns in existing reasoning mod- els on theory-of-mind tasks: (1) they do not consistently outperform vanilla LLMs using Shain-of-though reasoning, (2) they fail to generalize to similar scenarios, (3) they produce significantly longer reasoning traces for theory-of-mine questions than for factual ques- tions, and (4) reasoning effort (e.g., output length) does not correlate with performance. These findings highlight that social reasoning differs from mathematical or programming reasoning, areas where reasoning models typically excel.
These results suggest that Trought represents a promising step towards more robust tiference-time ToM reasoning. We aim to spark new discussions on inference-tine reasoning in the social domain, contrasting with the predominant focus on math and coding.
2 ThoughtTracing
We introduce ThoughtTracing, an inference- time algorithm that performs ToM reasoning by inferring and propagating weighted hypotheses about agents' thoughts. These hypotheses
2

---

Published as a conference paper at COLM 2025
represented in natural language, and are generated and weighted using LLMs. Thought- Tracing follows the conceptual structure of Bayesian Theory of Mind, and takes inspiration from sequential Monte Carlo to perform approximate inference over agents' mental states.
2.1 Background
Bayesian Theory of Mind (Baker et al., 2017) frames mental state attribution as probabilistic inference over a generative model of a rational agent. It focuses on several important roles that beliefs and goals play in theory-of-mind: the agent's perceptions and their prior beliefs jointly effect the current belief, and beliefs and goals are the causes of the agent's actions. As such, beliefs and goals can be inferred in various ways: forward simulation of beliefs from the agent's ''s perceptions and prior beliefs, backward inference of from the agent's observed actions, or through joint integration of all available information.
Thought follows this framework's structure without using explicit models of rational belief updating and action selection. Instead, we LLMs to simulate how an agent is likely to update their mental states in response to their perceptions, and to evaluate the likelihood of an agent's actions given their mental states. This enables greater generality, decomposing social inference into simpler tasks: forward simulation and likelihood evaluation by LLMs.
Sequential Monte Carlo (SMC) refers to a family of algorithms designed for incremental inference over sequences of posterior distributions (Del Moral et al., 2006), such as posterior inference over latent dynamics from time series data. SMC uses collection of weighted hypotheses (called particles) to approximate each distribution in the sequence. Given particles for step f in the sequence, SMC generates particles for step t via propagation (extending each previous particle with new latent states that exist at step and reweighting (weighting each particle by the likelihood of the observed data under that particle's latent states). The particles are then resampled according to their weights (focusing samples regions of high posterior probability) and optionally rejuvenated with Markov chain Monte Carlo (MCMC) to increase particle diversity (Chopin, 2002). To improve inference quality, SMC can make use of custom proposals at for propagation or rejuvenation (Lew et al., 2023a), using data-driven cues to generate more plausible hypotheses (Perov et al., 2015).
To infer hidden mental states that change over time, thoughtTracing follows an SMC-like structure, using LLMs as proposals for propagating hypotheses about agents' mental states, and weighting these hypotheses by likelihood scores generated by LLMs. However, for simplicity, ThoughtTracing does not compute full importance weights for each particle, since this would require accessing LLM log probabilities not provided by most APIs.
2.2 Algorithm
Thought/racing leverages sequential Monte Carlo principles to maintain a dynamically updated set of natural lan- guage hypotheses about the target agent's
Algorithm 1 Thought
1:
procedure TRACE(text c, target agent A)
2:
Trajectory T PREPROCESS(C)
mental state, weighting them in response to new observations. These hypotheses are updated based on the agent's perception, prior beliefs, and actions. See Algorithm 1.
3:
for == 1 to T do
4:
(s,, + T(I]
5:
if = 1 then
6:
H, + INITIAL (4,
7:
else
8:
H, PROPAGATTH((,-L-S944,,
Preprocessing Given a text input C, we
9:
if a, Ø then
parse it into trajectory (Lee, sequence of state-action pairs) of the target agent A. Specifically, we use the LLM for action la- beling of each sentence of the input. We
10:
W, < UPDATEWEIGHTS(Hf..e
11:
if effective sample size of H, is low then
12:
H, + RESAMPLE(H))
13:
if text similariiy of H, is high then
classify each sentence whether it includes 14:
H, + REJUVENATE(HH)
actions (e.g., physical movements or utter-
15:
7- HYOTHEESSUUMARIITATOOO H)
ances) from A. If the sentence does not include any actions from A, it is consid-
16:
return 7
> T with traced thoughts
ered a state sentence, which includes both information about the world state (descriptions of
3

---

Published as a conference paper at COLM 2025
other entities or the environment) and the agent state (the agent's features or characteristics). After we obtain the state-action sequence { (st, ())a we use the LLM for perception infer- ence, producing percept descriptions {p:} that describe whether the agent A perceived each state and what they perceived during each action dt. As result, we obtain target agent A's trajectory Tι (st, pt)
Initialization ThoughtTracing begins by sampling collection of weighted hypotheses H1 {h, ) where fn denotes hypothesis and w/") is the associated weight, from an initial distribution that approximates the step-1 posterior philsi,, P1). In practice, the LLM is given (s,, P1) as input, and produces a list of N hypotheses, with each
hypothesis assigned an initial weight of w(t) N We set N for our experiments.
Propagation At every step each hypothesis (1) is propagated forward to produce a new hypothesis nn(0) based on the trajectory so far: 1! ~ pa (h+ (1 (rr) Pr)) Here, 0 denotes the parameters of an LLM and {(s}, a, pj) denotes the trajectory (i.e., state, action, perception) up to time f. In effect, the LLM is used as data-driven proposal that generates plausible mental state h, given the observations.
Weight Update After propagation, each hypothesis h! receives a weight update based on how well it explains the action or utterance a, The LLM is used to output the likelihood
that 4; would be produced under each hypothesis: w!" := pa (a; h {(j,, 4j pj) St, Pt). While it is possible to compute this likelihood from a model's log probabilities (as done in Zhi-Xuan et al. (2024b)), in practice we find that instructing the LLM to choose from six options-from "very likely (around 90%)" to "very unlikely (below --ields better performance and stability. Moreover, most closed-source APIs (e.g,, GPT-4, Gemini) do not provide access to token-level log probabilities. We map these options to numerical scores and normalize so that Σ w
(1) =1.
Resampling & Rejuvenation Over time, some hypotheses may have very low weights, while others have very high weights. This is referred to as the degeneracy problem in SMC algorithms, and a well-known mitigation is resampling the hypotheses based on their weights (Douc & Cappé, 2005). Therefore, after the weight update, we check the effective sample size (ESS) to decide if resampling is needed. the ESS is lower than specific threshold, we sample N new hypotheses with replacement from the existing ones H; according to their normalized weights. This discards low-weight hypotheses and duplicates higher-weight ones to obtain new set of hypotheses. Additionally, we check the text similarity of the hypotheses to prevent them from collapsing into single point. If similarity is over certain threshold, we rejuvenate the hypotheses via paraphrasing.
Hypotheses Summarization After iterating through the entire trajectory, we aggregate the hypotheses at each time step leveraging their weights. In SMC, computing statistics such as means and variances a common practice to summarize the distribution represented by the particles. Since our hypotheses are in natural language, we instead use the LLM to generate a weighted summary of the hypotheses set H, at time step f, given the past trajectory and the hypotheses along with their weights. This process can be viewed as deriving weighted consensus over the hypotheses at each time step.
The final output of ThoughtTracing is trajectory of the target agent A with the summarized hypotheses interleaved. An example is in Table 1. State, action, perception, and beliefs of A at each time step is concatenated as reasoning trace formatted with HTML-style tags.
The full prompts used in Trought can be found in the Appendix C. A case study on how the hypotheses and weights evolve over time is in the Appendix D.
4

---

Published as a conference paper at COLM 2025
<context 2>
He then proceeds to open the third Room C cabinet and closes it as well. <note>David perceived that the third Room C cabinet is empty. </not>> <David's updated thoughts>David believed that the third Room C cabinet might contain an Object E. This belief is inferred from his action of opening the cabinet to check its contents, suggesting that he thought there was possibility of finding Object E there. However, upon opening it, he perceived that it was empty, confirming that his belief was incorrect regarding the presence of an Object E in that cabinet.</Davine updated thoughts>
<lcontext 2>
<context 3>
Following this, he opens and closes the stove.
<note>David perceived the stove in Room C, which contained an Object H and an Object J, as he opened and then closed it </note>
<David's updated thoughts>David David believed there might be Object E in the stove. This belief inferred from his systematic search through the storage spaces Room C, as he had not yet found the Object E he was looking for in the cabinets. His actions of opening and closing the stove suggest that he was checking to see if contained the desired object. However, upon opening it, he penceived that it contained an Object H and piece of Object J, confirming that his belief regarding the presence of an Object E in the stove was cecorrectt<DDaviie updated thoughts> </context
Table 1: An example traced thoughts from our Thought algorithm on MMTOM-QA.
3 Experiments
Our ThoughtTracing algorithm offers a general method for tracing the mental states of target agent, given context. While the algorithm is not explicitly designed to solve benchmark questions, we evaluate its validity by testing whether it can improve existing models' performance on four theory-of-mind (ToM) benchmarks. We apply Thought to off-the-shelf LLMs and feed the generated thought traces along with the original context and compare their performance against other baselines, such as reasoning models.
(1) ParaphrasedToM: (Sclar et al., 2023) is revised version of the classic tfeory-of-mind benchmark ToMi (Le et al., 2019) that addresses the limited linguistic diversity of the original ToMi by rewording all ToMi templates using GPT-3 with additional manual filtering. The resulting dataset is considerably more complex, as actions are expressed in a less straightforward way. We use total of 600 questions and report three metrics: accuracy of (i) all questions, (ii) false belief questions, and (iii) true belief questions.
(2) BigToM (Gandhi et al., 2023) is a benchmark featuring synthetic narratives centered on person's 's desires, actions, and beliefs, along with an event that changes the environment's state. It includes questions about the person's beliefs and actions. We find some of the narra- tives and ground-truth answers in BigToM include incoherent stories and errors. To address this issue, we randomly sampled 600 scenarios from the dataset and re-annotated by crowd- sourcing 510 online participants (see App. A). We only retained samples with an agreement rate higher than 90% (249 samples). We report the average accuracy of the questions.
(3) FANTOM (Kim et al., 2023) is multi-party conversation question-answering dataset designed to test coherent theory-of-mine capabilities. In FANToM, the speakers join and leave the conversation while it continues, making the participants hold both false and true beliefs. The benchmark includes first-order and second-order theory-of-ming about the beliefs of conversation participants in various formats, such multiple-choice and list type questions. We use 64 sampled conversations from the short version of FANTOM, containing total of 1,086 questions. We report three metrics: (i) All Qs, (ii) Answerability All Qs, and (iii) Ince-Accessibillity All Qs, each requiring the model to correctly answer all questions for the corresponding question types for given conversation snippet.
questions
crestion-answering benchmark that includes questions covering the beliefs and goals of person searching for object (e.g., remote controller). We use 200 questions in its text-only subset, which describes a person's activities in household environment. Because some objects associated with unusual
(4) MMToM-QA (Jin et al., 2024) is a multi-modal
5

---

Published as a conference paper at COLM 2025
Pora
BigToM
FANToM
MMToM-QA
Model
Avg.
False Belief
True Belief
Avg.
All
Answer- ability All
Info Access Qs
Avg.
Belief
Goal
Qs
GPT-4o
59.5
38.5
80.3
98.0
11.1
33.3
32.7
56.3
74.5
39.8
GPT-4o CoI
67.0
76.0
58.0
98.0
40.7
53.7
71.2
56.0
82.3
56.1
GPT-4o + TT
76.0
84.5
67.5
98.8
41.5
71.7
76.5
60.0
78.4
37.8
GPT-4o CoT
74.3
65.5
99.2
54.7
74.5
69.0
93.1
42.9
ol-preview
36.0
ol-low-effort
67.8
98.5
37.0
99.2
28.3
41.5
51.0
76.5
96.1
57.1
ol-medium--ffoot
70.8
96.01
45.5
98.8
30.2
41.5
58.8
76.0
94.1
60.2
ol-high-effort
73.0
97.0
49.0
98.8
37.9
44.8
63.0
76.5
95.1
59.2
ol-mini
60.0
60.0
97.6
41.5
23.5
60.0
91.2
27.6
os-mini-tow-eloort
64.0
62.5
65.5
95.2
17.0
11.8
71.6
37.8
64.5
90.5
38.5
97.2
1.9
22.6
19.6
69.0
94.2
42.9
03-min o3-mimihiigh-fffor
64.0
99.5
28.5
97.2
1.9
35.8
33.3
71.5
97.1
44.9
DeepSeck R1
68.3
77.0
59.5
96.8
37.9
62.1
48.1
49.0
73.5
24.5
Llama 3.3 70B
57.3
44.5
70.0
96.4
0.0
7.4
0.0
47.0
50.0
42.9
Llama 3.3 70B CoT
64.5
62.5
92.4
11.3
23.5
47.0
Ilama 3.3 70B TT
71.3
62.0
80.5
99.2
9.4
35.8
21.6
44.0
59.8
27.6
Llama 3.3 70B TT CoT
77.0
87.5
66.5
77.0
17.0
45.3
33.3
58.0
79.4
35.7
Gemini 1,5 Pro
54.8
43.5
66.0
98.0
1.9
7.5
2.0
50.0
70.6
28.6
Gemini 1.5 Pro + CoT
58.3
70.5
46.0
17.0
24.5
27.5
51.0
72.5
28.6
Gemini 1.5 Pro TT
74.3
83.0
65.5
30.2
58.5
52.9
54.0
75.5
21.4
Gemini 1.5 Pro TT + CoT
61.5
73.5
49.5
87.6
45.3
67.9
62.7
64.0
80.4
46.9
Qwen 2.5 72B
57.3
39.0
75.5
94.0
0.0
5.7
2.0
41.5
51.0
32.7
Qwen 2.5 72B + CoT
67.8
63.0
72.5
96.4
1.9
3.8
27.5
41.5
51.0
32.7
Qwen 2.5 72B TT
74.3
21.6
29.6
Qwen 2.5 72B-2--- TT + CoT
76.0
80.0
72.0
96.4
32.1
62.3
68.6
53.0
63.7
40.8
QwQ 32B preview
58.8
27.5
90.0
93.6
0.0
0.0
0.0
51.5
71.7
29.6
Table 2: Results across four theory-of-mind benchmarks. The underlined metric indicates the primary metric for each benchmark. The notation '+TT' denotes that our ThoughtTracing method has been applied, while '+Co'I indicates that chain-of-thought has been used. Full evaluation results can be found in Table in Appendix.
Docationsssuhe as a remote controller and a fridge-contrary to commonsense, we replace room names and object labels with letters (e.g., room A and object A). We report three metrics: average accuracies of (i) all questions, (ii) belief questions, and (iii) goal questions.
Baseline & Setup We apply our method to four state-of-the-art LLMs and compare their performance, and also compare with five recent reasoning models: GPT-4o (OpenAI, 2024a), ol, ol-mini (OpenAl, 2024b), o3-mini (OpenAI, 2025), DeepSeek-R1 (DeepSeek-AI, 2025a), Llama 3.3 70B Instruct (Dubey et al., 2024), Gemini 1.5 Pro (Gemini-Team, 2024), Qwen 2.5 72B (Yang et al., 2024), and QwQ 32B preview (Qwen-Tey Team, 2024). We also compare with zero-shot thain-of-thought (CoT)) reasoning (Kojima al., 2022).
3.1 Overall Results
ThoughtTracing (+TT) improves all baseline models across all benchmarks. Table 2 shows the results on all four benchmarks. Although ThoughtTracing does not take bench- mark questions as input, its output significantly improves the model performance the Bog MMTOM-QA, and the AllQs score on FANTOM. For example, Llama 3.3 70B + TT, Gemini 1.5 Pro TT, and Qwen 2.5 TT significantly outperforms GPT-4o CoT on ParaphrasedToMi These results indicate that the traced mental states from ThoughtTracing provide accurate intermediate reasoning steps that helps models correctly answer mental state related questions.
main metrics - the average accuracy (Avg.) on Paraphrased
ThoughtTracing guides better reasoning trajectories. The fact that ThoughtTracing out- performs CoT indicates the traced thoughts exhibit better reasoning traces. Moreover, when CoT is applied on top of the traced thoughts, models' performance improves further in many cases. For example, Llama 3.3 and Qwen 2.5 achieve the first and second best scores

---

Published as a conference paper at COLM 2025
on ParaphrasedToMMi respectively. All models achieve the best score when Trocigt and CoT is applied sequentially on MMToM-QA. Interestingly, Qwen 2.5 does not show any improvement on FANToM, when ThoughtTracing is applied. However, when CoT is further applied its performance jumps to 32.1, even outperforming o1-medium-effort and o3-mini.
On one hand, we find that, for GPT-4o and Gemini 1.5 Pro, the combination of +T un- derperforms compared to +TT alone on ParaphrasedToMM: This underperformance is linked to the degradation on True Belief scenarios. In these scenarios, there is information asym- metry-ie., all participants share the same belief aligned with the ground-trume truth-making over-reasoning potentially harmful. Notably, vanilla CoT alone sigmificantly reduces their performance on True Belief questions, suggesting that the extra reasoning steps may intro- duce noise or unnecessary complexity. This effect appears to be significantly stronger in dlosed-wiigh models (e.g., GPT-4o, Gemini 1.5 Pro) than open-weight ones (e.g. Llama 3.3, Qwen 2.5). Interestingly, the degradation of TT CoT relative to TT alone is only observed in these weight models, hinting at possible post-training differences.
3.2 Comparison with Reasoning Models
ThoughtTracing applied models show better performance than reasoning models. Ex- cept for ol on MMToM-QA, models with Thought significantly outperform ol, ol-mini, 03-mini, DeepSeek R1, and QwQ 32B preview on all benchmarks. Moreover, even LLMs with CoT perform better than reasoning models, such 03-mini, DeepSeek R1 and QwQ 32B. The ol model also underperforms than CoT on FANTOM. This suggests that reinforcing mathematical or programming reasoning does not generalize social reasoning, necessitating separate training scheme in this domain.
Reasoning models show performance trade-off. A performance trade-off can be observed between false belief scenarios and true belief scenarios in P'rraphrasedooMi, While ol and o3-mini achieve near-perfect scores (i.e., 100) on the false belief scenarios, their performance on the easier true belief scenarios drops sharply to below 50 and even below 30 in some cases. similar pattern observed with DeepSeek R1, whose scores on the easier true belief scenarios are lower than those on the false belief scenarios. In contrast, the QwQ 32B preview follows the pattern seen in other vanilla LLMs, displaying higher scores for true belief scenarios but lower scores for false belief scenarios. Notably, QwQ significantly underperforms on FANToM, primarily due to excessive false positives. For instance, it frequently predicts that characters are aware of certain information when they are, in fact, unaware. This issue arises consistently in both list-type and binary (yes/no) no) questions.
Although these models do not release their training data, interesting future direction would be to investigate how reinforcement learning influences this trade-off. On the other hand, ThoughtTracing applied models show more balanced performance across false belief and true belief scenarios, with improvements observed in both cases, except for GPT-4o.
Output token counts and performance Figure 2 counts for 03-mini, o1, DeepSeek R1, and GPT-4o with Trought on Paraphrased ToMi,
summarizes the average output token
(1) Output token counts in oll and 03-mini do not correlate with performance. OpenAl's ol and o3-mini models include an additional parameter that controls 'reasoning effort' using three settings: low, medium, and high. This parameter guides the model in determining how many reasoning tokens to generate before producing a final response. Although the ol model with high reasoning effort outperforms the medium and low settings on ToMi, this pattern does not consistently appear across other benchmarks. For example, on BigToM, FANTOM, and MMTOM-QA, the ol model with low reasoning effort achieves the highest performance. Similarly, while increased reasoning effort in o3-mini improves performance on MMTOM-QA, this is not the case for other benchmarks. Moreover, performance on true belief scenarios in ParaphrasedToMi gets worse reasoning effort increases.
RRasoning models were provided with extra formatting prompts to align with the evaluation setup of these benchmarks. Without those formatting prompts, the scores are significantly lower.
7

---

Published as a conference paper at COLM 2025
3500
Pasponsesste to Factual Questions [ToHij
Conrect
Quastions
Aug. Dutbut Folth
2500
Incorect Responses to ToM Questions [ToMi]
Conrect Respunses ToM Questions (HMTcM)
1300
Incoreet Reeponses to TeM Questions (MMTaM]
1000
500
DeepSeekR1
o1-medium
o1-iigh
c3-mini-oww
63-mini-medium
63-minitiggh
4T
Figure 2: The average number of tokens in the reasoning trace from the reasoning models and ThoughtTracing for ParaphrasedToMMi and MMToM-QA.
(2) Theory-of-mind (ToM) questions elicit more output tokens than factual questions. In ParaphrasedToMi, all reasoning models use significantly more output tokens when address- ing ToM questions compared factual ones. This suggests processing social information is more computationally demanding for these models.
(3) Token usage for incorrect and correct responses to ToM questions differs across datasets. Although recent analysis reported incorrect responses from DeepSeek R1 and QwQ-32B preview have significantly higher token usage (Wang et al., 2025), this does not hold for ToM benchmarks. In Tora all reasoning models show significantly higher token usage for incorrect responses to ToM questions than correct ones. Moreover, the pattern is different for MMToM-QA, which the token usage is similar between correct and incorrect responses or is higher for correct ones. On the other hand, ThoughtTracing exhibits balanced token usage regardless of whether the responses to the ToM questions are correct or incorrect. We note that the relative difficulty of the questions in ToMi and MMToM-QA is consistent compared to benchmarks in other domains, and all models achieve near-perfect scores on ToMi's factual questions.
(4) Models with ThoughtTracing produce significantly shorter reasoning traces while achiev- ing better performance. As shown in Table 2, GPT-4o+T and GPT-4+TT+Col deliver the best and second-best performance among these models on ToMi, with scores of 76 and 74.5, respectively. Despite better performance, these models generate significantly fewer output tokens than o3-mini, ol (with medium and high reasoning effort), and R1. Furthermore, compared to ol-low and 03-mini-low, GPT-4o+TT and GPT40+TT+Col score almost 10 points higher. Similarly in MMToM-QA, GPT-40+T++Co1 outperforms R1 and shows performance comparable with m3-mini-medium while using less than 50% of the tokens.
Thought switching and performance We also analyze the thought switching patterns for DeepSeek R1, following (Wang et al., 2025), on ParaphraseddoMi and MMToM-QA. The authors initially reported that frequent thought switching leads to underthinking and ultimately lower performance. For Paraphrased ToMi, incorrect responses included an average of 9.69 thoughts, while correct ones included an average of c.2s--cnnistent with the reported pattern. However, for MMTOM-QA, responses for incorrect responses included an average of 6.83 thoughts similar to correct responses, which contained an average of 6.45.
3.3 Analysis on ThoughtTracing
Qualitative Error Analysis We manually evaluated ThoughtTracing' errors on Para- porasedTooMi, FANToM, and MMTOM-QA. Due to the model's 's near-perfect performance on BigToM, we did not conduct error analysis, as the number of errors was too limited to draw meaningful conclusions.
(1) Ti:: We observed recurring pattern of incorrect perception predictions. In true belief scenarios, the sometimes incorrectly infers that the agent did not witness often due to conservative estimation of perception (e.g., assuming the agent was not paying attention).
another agent moving an object object-often
The model tends to overestimate the target agent's prior knowledge, occa- sionally assuming familiarity among characters. However, FANToM is designed such that characters are meeting for the first time, making these inferences incorrect.
(2) FANToM:
8

---

Published as a conference paper at COLM 2025
These cases from ParaphrasedIoMi and FANToM reflect our intentional design choice to avoid benchmar--specfffc assumptions- as presuming agents always observe ev- erything have predefined relationships. We believe this is principled trade-off for generalizability. Incorporating benchmark-specinn cues or more explicit assumptions in the prompts will reduce these errors.
(3) M-ToM-QA: We find an inherent bias across models. In many cases, when an agent searches for something and repeatedly encounters an object-despite not interacting with it-the models generate hypotheses suggesting that the object is what the agent is looking for. However, according to both social common sense and explicit models of ToM (Baker et al., 2017; Ying et al., 2025), if an agent repeatedly sees an object but continues searching elsewhere, it is likely that they are actually searching for something else. Since this error may be a side effect of perception inference in ThoughtTracinng we conduct ablation experiments.
Ablation We conducted ablation exper- iments three models-GPT-4o Llama 3.3 70B, and Gemini 1.5 Pro ing either the perception inference the weight update ($2.2). For the perception inference removal, we directly sample hy- potheses only from the state-action pairs
Paraphrased ToMi (Avg.)
EANIOM (AllQ=)
MMIOM
Model
(Avg.
Pro-by by remov-
GPT-4o IT
Without Perception Without Weight Update With Self-corretion
-16.2
-17.0
-7.0
-4.2
0.0
-4.0
-.2
-3.8
-4.0
Llama 3,3 70B + 'TI
Without Perception Without Weight Update With Self-corretion
-12.8
-9.4
0.0
in the trajectory each time step. For the weight update removal, we apply uni- form weights for all hypotheses across the trajectory at every time step.
-188
-2.5
-2.0
+0.2
-3.7
-1.0
Gemini 1.5 Pro TT
Without Perception Without Weight Update With Self-correction
-19.5
-20.8
-2.0
-4.3
-5.7
-9.0
Table shows the results. Both modifica- tions lead to reduced performance, with the removal of perception inference caus- ing greater drop. This suggests not only perception inference is crucial compo-
Table 3: Score difference comparing variants of ThoughtTracing the original algorithm on ParaphrasedToMi FANToM, and MMToM-QA.
nent in LLM's theory-of-mind reasoning (Jung et al., 2024), but also the fact that LLMs tend not to incorporate perception inference internally when generating mental state hypotheses. Additionally, this indicates the weight update based on action likelihood helps the model prioritize more promising hypotheses.
Self-correction during Propagation We also tested another variant of ThoughtTracing that self-corrects its previous hypothesis and predict a new hypothesis during the propagation step (62.2). Table shows the results. Although self-correction has shown a performance increase in some domains (Madaan et al., 2023; Shinn et al., 2023), it does not show better performance for hypotheses propagation on the three ToM benchmarks. In most cases, it leads to worse performance, indicating self-correction capability is not effective in our iterative updates, potentially due to the accumulation of correction errors. We suggest future works to investigate the effects of LLM's self-correcting behavior in the social domain.
4 Related Work
Model-based Tfeory-of-Mind Reasoning Cognitive science research has shown that humans reason about other agents' minds by assuming their actions are rational and goal- directed (Dennett, 1981; Csibra et al., 1999; Baillargeon et al., 2016). These processes have been formalized as Bayesian Theory of Mind (Baker al., 2017; Jara--ttingee et al., 2019), which models agents as rational actors that plan and act upon their beliefs and desires to achieve their goals. This probabilistic model can then be inverted via inverse planning, inferring agents' mental states from their behavior (Baker et al., 2009; Ying et al., 2023; 2025), including via SMC methods (Zhi-Xuan et al., 2020; 2024a). While ThoughtTracing is patterned on this model-based approach, we do not explicitly model rational agents, but instead use LLMs as implicit models of agents' behavior, and as hypothesis generators for agents' thoughts, enabling our method to be applied to open-endes inputs.

---

Published as a conference paper at COLM 2025
Theory-of-Mind Reasoning in LLMs Debates on whether LLMs are capable of ToM reasoning have sparked extensive controversy (Ullman, 2023; Whang, 2023; Ma et al., 2023; Shapira et al., 2024). As result, many benchmarks for evaluating ToM reasoning have been released (Le et al., 2019; Gandhi et al., 2023; Wu et al., 2023; Shapira et al., 2023; Jin et al., 2024; Chen et al., 2024; Xu et al., 2024, inter alia) along with task complexity assessments (Huang et al., 2024). While LLMs succeed on some tasks, analyses on these benchmarks show that they are not yet at the level of human ToM, with signs of overfitting (Sclar al., 2023) and overall poor capabilities (Sap et al., 2022; Kim et al., 2023). To mitigate this, several inference- time methods have been introduced (Sclar al., 2023; Wilf al., 2024; Jung et al., 2024). However, they rely on specific assumptions or few-shot examples, making them less scalable. Recently, Sclar et al. (2025) showed search for generating ToM training data and Cross al. (2025) introduced modular design for ToM hypothesis generation in multi-agent game scenarios. In this work, we aim to minimize assumptions and introduce a flexible algorithm capable of generating traces of target agent's 's mental states in open-ended text.
Inference-time Reasoning for LLMs Inference-time reasoning has emerged as pivotal area of research for LLMs, particularly in mathematics, coding, and puzzle solving. Early work, such as Chain-of-Thouugh (Wei et al., 2022), demonstrated the benefits of generating intermediate reasoning steps, and was later augmented by approaches capable of exploring multiple reasoning paths (Yao et al., 2023; Besta et al., 2024), aggregation across reasoning chains (Wang et al., 2023), and problem decomposition (Zhou al., 2023). Recently, rea- soning models trained via reinforcement learning ol (OpenAT, 2024b), R1 (DeepSeek-AT, shown remarkable performance. Most of these approaches leverage objectively verifiable answers to enable reinforcement and accurate selection among multiple reasoning paths. However, such verification challenging in social reasoning due to subjectivity and uncertainty. To handle uncertainty, recent methods perform probabilistic inference in LLMs via sequential Monte Carlo (Lew et al., 2023b; Zhao et al., 2024; Loula et al., 2025), but have not been used to infer and track mental states. In this work, we introduce general SMC-like algorithm capable of operating effectively in the social domain while bypassing the need for objective verification.
2025a), and QwQ (Qwen-Team 2024)
Conclusion
In this paper, we introduced Thought'Tracing, new inference-time algorithm that performs approximate inference over agents' mental states. Our approach draws inspiration from the sequential Monte Carlo (SMC) method and follows the conceptual structure of Bayesian Theory of Mind (BToM; Baker et al., 2017). ThoughtTracing leverages an SMC-like structure by using LLMs as proposals for natural language hypotheses, which are then weighted with actions from the given text-all without requiring ground-truth answers to questions in datasets. Our evaluation on four theory-of-mind benchmarks demonstrates its effectiveness, significantly improving upon baseline LLMs and outperforming recent reasoning models such o3-mini and R1. Additionally, our findings reveal that reasoning models show different behaviors in the social domain compared to their remarkable performance in areas like math and coding. Nonetheless, there remains room for improvement due to errors in areas such as how hypotheses about agents' goals are generated (Section 3.3). This suggests that more work needs be done to achieve the reliability that explicit BToM models demonstrate in closed-wordd settings (Zhi-Xuan et al., 2022; 2024b; Ying et al., 2025). We hope these insights pave the way for future research on inference-time reasoning in social settings, where AT agents will play an increasingly important role.
Acknowledgments
This work was supported in part by the Amazon Research Gift Grant. Tan Zhi-Xuan is funded by the Open Philanthropy AT Fellowship.
10

---

Published as a conference paper at COLM 2025
References
Nitay Alon, Lion Schulz, Jeffrey S Rosenschein, and Peter Dayan. A (dis-) information theory of revealed and unrevealed preferences: emerging deception and skepticism via theory of mind. Open Mind, 7:608-624 2023.
Renée Baillargeon, Rose M Scott, and Lin Bian. Psychological reasoning in infancy, Annual review of psychology, 67:159-186, 2016.
Chris L Baker, Rebecca Saxe, and Joshua Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):329-33) -4999-3499400 2009.
Chris L. Baker, Julian Jara-Ettinger Rebecca Saxe, and Joshua B. Tenenbaum. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nature Human Behaviour, 144):1-10, April 2017. doi:
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Infelligence, volume 38, pp. 17682-17690, 2024.
Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, and Minlie Huang. ToMBench: Benchmark ing theory of mind large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15959-15983, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ ///////1222242cllooo/ https://aclanthologg. org/2024 acl- 1ond.loog..80 847/.
URL
Nicolas Chopin. A sequential particle filter method for static models. Biometrika, 89(3):
539-552, 2002.
Katherine M Collins, Ilia Sucholutsky, Umang Bhatt, Kartik Chandra, Lionel Wong, Mina Lee, Cedegao Zhang, Tan Zhi-Xuan, Mark Ho, Vikash Mansinghka, et al. Building machines that learn and think with people. Nature human behaviour, 8(10):851-1863, 2024.
Logan Cross, Violet Xiang, Agam Bhatia, Daniel LK Yamins, and Nick Haber. Hypothetical minds: Scaffolding theory of mind for multi-agent tasks with large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: wwtpeerrvveee.ee/fffoonnlidodooe
Gergely Csibra, Gyórgy Gergely, Szilvia Bíró, Orsolya Koos, and Margaret Brockbank. Goal attribution without agency cues: the perception of 'pure reason's 'in infancy. Cognition, 72 (3):237-267, 1999.
DeepSeek- AI. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning, 2025a. URL
12948.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint ar XrXi:22001.12448 2025b.
Pierre Del Moral, Arnaud Doucet, and Ajay Jasra. Sequential monte carlo samplers. Journal of the Royal Statistical Society Series B: Statistical Methodology, 68(3):411-433) 2006.
Daniel Clement Dennett. The Intentional Stance. MIT Press, 1981.
Prashant Doshi and Piotr J Gmytrasiewicz. Monte carlo sampling methods for approximat- ing interactive pomdps. Journal of Artificial Intelligence Research, 34:297-337, 2009.
Randal Douc and Olivier Cappé. Comparison of resampling schemes for particle filtering. In ISPA 2005. Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis, 2005., pp. 64 69. Teee, 2005.
11

---

/22054////224.cc---// Published as a conference paper at COLM 2025
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arv preprint or arv: 217883 2024.
Kanishk Gandhi, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah Goodman. Under- standing social reasoning in language models with language models. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
URL httpt:: http:///wwevii....eefforrriiiia/ wwwprrrnnmndddiirrrraar00
Gemini-Team. Gemini 1.5:
Unlocking multimodal understanding across millions of tokens of context. Xro preprint ar XrXo:24403.05530, 2024.
X Angelo Huang, Emanuele La Malfa, Samuele Marro, Andrea Asperti, Anthony Cohn, and Michael Wooldridge. notion of complexity for theory of mind via discrete world models. Xrviv preprint ar Xiv:2406. 11911, 2024.
Julian Jara-Ettinger, Laura Schulz, and Josh Tenenbaum. The naive utility calculus as unified, quantitative framework for action understanding. PsyArXiv, 2019.
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua Tenenbaum, and Tianmin Shu. MMToM-QA: Multi- modal theory of mind question answering. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), PP. 16077-16102, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: acl long........ 851/.
URL
Chani Jung, Dongkwan Kim, Jiho Jin, Jiseon Kim, Yeon Seonwoo, Yejin Choi, Alice Oh, and Hyunwoo Kim. Perceptions to beliefs: Exploring precursory inferences for theory of mind in large language models. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 19794-19809, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 wtt.//2...22lppp111 1105. URL https: //tp//ahaaannnnoy....oo. org/ 20244emmlpmmal 1105/.
Hyunwoo Kim, Melanie Sclar, Xuhui Zhou, Ronan Bras, Gunhee Kim, Yejin Choi, and Maarten Sap. FANToM: A benchmark for stress-testing machine theory of mind in interactions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, pp. 14397-14413, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/ 2023.emnlp-main.55 8900. URL https: htttps::/aclaanthologg// www../2...........oo.. 2023. emnlp-main main.890/.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: http///pppeevvieeeeee wwt.wviwwwennotnnnieeeeee met//oorum?id==e2BTByyyy/
Matthew Le, Y-Lan Boureau, and Maximilian Nickel. Revisiting the evaluation of theory of mind through question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP) Pp. 5877, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 00.18633vv1/D1// 1598. URL https: https: / org/D19-1598/.
Alexander K Lew, George Matheos, Tan Zhi-Xuan, Matin Ghavamizadeh, Nishad Gothoskar, Stuart Russell, and Vikash K Mansinghka. SMCP3: Sequential monte carlo with proba- bilistic program proposals. International conference on artificial intelligence and statistics, pp. 7061-7088. PMLR, 2023a.
Alexander K Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K Mansinghka. Sequential monte carlo steering of large language models using probabilistic programs. arXiv preprint arXit:2306.03081, 2023b.
12

---

Published as a conference paper at COLM 2025
João Loula, Benjamin LeBrun, Li Du, Ben Lipkin, Clemente Pasti, Gabriel Grand, Tianyu Liu, Yahya Emara, Marjorie Freedman, Jason Eisner, Ryan Cotterell, Vikash Mansinghka, Alexander K. Lew, Tim Vieira, and Timothy J. O'Donnell. Syntactic and semantic control of large language models via sequential monte carlo. In The Thirteenth International Conference on Learning Representations, 2025. URL https:://ppereee
Ziqiao Ma, Jacob Sansom, Run Peng, and Joyce Chai. Towards a holistic landscape of situated theory of mind in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1011-1031, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/ /v11 2023. 2023. 0023.indings-emnllp72 URL https://aclanthhon
org/2023. f Findings-emnp.... 72/.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty- Conference on Neural Information Processing Systems, 2023. URL https://openreviiee net/ forum?1dd=S37horrQL3
Aliakbar Nafar, Kristen Brent Venable, and Parisa Kordjamshidi. Probabilistic reasoning in generative large language models. arXiv preprint arXio:2402.09611 2024.
OpenAl. Gpt-4o system card. arxiv preprint arXio:2410.21276, 2024a.
OpenAL. Openai ol system card. arXiv preprint 2x11:00122222222.762222000000 2024b.
OpenAL 03-mini system card, 2025. httphttss///cn.oor pdf. Accessed: 2025-02-05.
coom/o3-minn www.:ooiiiisssse---oooeee
Yura N Perov, Tuan Anh Le, and Frank Wood. Data-driven sequential monte carlo in probabilistic programming. Xiv preprint ar arXiv:1512.04387,, 2015.
Qwen-Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. URL https://qwenIm. gi thub. htthbblog/wq--//0 1:bb-preview/.
Maarten Sap, Ronan Le Bras, Daniel Fried, and Yejin Choi. Neural theory-of-mind? on the limits of social intelligence large LMs. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 3762-3780, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 006653//1/22222222 http///2.eemmmmmai.nnn.. URL https://aclantthologyy hwwp://aaannnttooooororee///// org/ emnlp-main. 248/
Melanie Sclar, Sachin Kumar, Peter West, Alane Suhr, Yejin Choi, and Yulia Tsvetkov. Mind- ing language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker. Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13960-13980, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1 202/2223.allloong acl acl-long. 780/.
URL https: htapps:///cchi http:///aaaaatthhhoooo orgoorgg///// 2023.
Melanie Sclar, Jane Dwivedi- Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https::/
Natalie Shapira, Guy Zwirn, and Yoav Goldberg. How well do large language models perform on faux pas tests? In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 10438-10451, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 2023. 1D23.findings-cle acl.nng....l.66... URL https://acllntthoo.ge .rrgey.org//o
ort.org////// 22233.22. findings-acl acl. 663/.
13

---

Published as a conference paper at COLM 2025
Natalie Shapira, Mosh Levy, Seyed Hossein Alavi, Xuhui Zhou, Yejin Choi, Yoav Goldberg, Maarten Sap, and Vered Shwartz. Clever hans or neural theory of mind? stress testing so- cial reasoning in large language models. In Yvette Graham and Matthew Purver (eds.), Pro- ceedings of the 18th Conference of the European Chapter of the Association for Compulational Lin- guistics (Volume 1: Long Papers), pp. 01--2257-225 St. Julian's, Malta, March 2024. Association for Computational Linguistics. URL httpp:://ccciiiiie https://aclanthology www.aannthooooooooo/22224/4 org/2o24. eareeeaaclllggogo 138/.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In Thirty-sementh Conference on Neural Information Processing Systems, 2023. URL
net/
forum?id=vvAEIncccKW.
Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08339 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Repre- http/forummidd11PIIIMMMM
sentations, 2023. URL httpss://operrevieet
Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. Thoughts are all over the place: On the underthinking of ol-like 1lms. Xiv preprint arXiv:2501. 18585, 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrreview. net/forrum?dd__jjl1MBBB
Oliver Whang. Can machine know that we know what it knows? The New York Times, 2023. httpe//aaaa-neiiieeeeeeennnnnnno
URL https://www. http:://ww.nytiims.comm/ html.
Alex Wilf, Sihyun Lee, Paul Pu Liang, and Louis-Philippe Morency: Think twice: Perspective- taking improves large language models' theory-of-mind capabilities. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8292-8308, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024.acl-long.455 451. URL https: httpss///aaaathoogyy.ooooo org 2024. acl-long.
Yufan Wu, Yinghui He, Yilin Jia, Rada Mihalcea, Yulong Chen, and Naihao Deng. Hi-ToM: A benchmark for evaluating higher-order theory of mind reasoning in large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10691-10706, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653 v1 wtt.//2...nninnn...... findinngs-emnll.777. URL h https://aclanthooooyy awthology.oornooeeoooooo 2023. iidings-emnlpp... 717/-
Hainiu Xu, Runcong Zhao, Lixing Zhu, Jinhua Du, and Yulan He. OpenToM: A comprehen- sive benchmark for evaluating theory-of-mind reasoning capabilities of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 8593-8623 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 2tt/3//11112244alloooggg../ 466/
URL https: //acl// acwaacolonntyyyyo org.org///222 ac4-
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXin=2412.15115, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language
14

---

Published as a conference paper at COLM 2025
models. In Thirty-seventi Conference on Neural Information Processing Systems, 2023. URL https://apenreview net nett/forumidd=Sxceeeeeee
Lance Ying, Katherine M Collins, Megan Wei, Cedegao E Zhang, Tan Zhi-Xuan, Adrian Weller, Joshua B Tenenbaum, and Lionel Wong. The neuro-symbolic inverse planning engine (nipe): Modeling probabilistic social inferences from linguistic inputs. arXiv preprint (iX::23366.13355 2023.
Lance Ying, Tan Zhi-Xuan, Lionel Wong, Vikash Mansinghka, and Joshua B Tenenbaum. Understanding epistemic language with language-augmenteer bayesian theory of mind. Transactions of the Association for Compulational Linguistics, 13:613-637, 2025.
Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Baker Grosse. Probabilistic inference in language models via twisted sequential monte carlo. In Forty-first Inter- national Conference on Machine Learning, 2024. URL https://ppenreiiww.nee wttperrewiiieeeeooreeiiiii wtw.se/wwntooormimiiid
frAeNNBS1n.
Tan Zhi-Xuan, Jordyn Mann, Tom Silver, Josh Tenenbaum, and Vikash Mansinghka. Online bayesian goal inference for boundedly rational planning agents. Advances in Neural Information Processing Systems, 33, 2020.
Tan Zhi-Xuan, Nishad Gothoskar, Falk Pollok, Dan Gutfreund, Joshua Tenenbaum, and Vikash K Mansinghka. Solving the baby intuitions benchmark with hierarchically bayesian theory of mind. In RSS 2022 Workshop on Social Intelligence in Humans and Robots, 2022.
Tan Zhi-Xuan, Gloria Kang, Vikash Mansinghka, and Joshua Tenenbaum. Infinite ends from finite samples: Open-ended goal inference as top-down bayesian filtering of bottom- up proposals. Proceedings of the Annual Meeting of the Cognitive Science Society, 46(46), July 2024a.
Tan Zhi-Xuan, Lance Ying, Vikash Mansinghka, and Joshua B Tenenbaum. Pragmatic in- struction following and goal assistance via cooperative aanguage-guided inverse planning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 2094-2103, 2024b.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh Interna- fional Conference on Learning Representations, 2023. URL https: /ttp::////eeereiew..noe net.orerw..t/orrmm id=WZH7899tgfM.
15

---

Published as a conference paper at COLM 2025
A BigToM Re-annotation
The original Big dataset (Gandhi et al., 2023) includes tasks, each with 200 stimuli. In our re-annotation, we took the first 100 stimuli from each task category.
We recruited 510 participants (Average age 41.33, 238 males, 265 females, 7 others) via Prolific. To minimize potential bias, we did not conduct tutorials for this task. The experi- ment took place over customized interface. Each participant was randomly assigned to complete 30 stimuli randomly sampled from the set of stimuli. Each stimulus received on average 13 ratings.
Table 4 shows the average agreement for each question type. We manually went over the re-annotation and found samples with an agreement rate below 90% often contained incoherent stories or debatable answer options. Therefore, we only use samples with an agreement rate above 90% in our experiments.
Backward Belief (FB)
Backward Belief (TB)
Forward Action (FB)
Forward Action (TB)
Forward Belief (FB)
Forward Belief (TB)
Mean Std Dev
0.79
0.68
0.69 0.22
0.90 0.10
0.94 0.11
0.78
0.17
0.21
0.14
Table 4: Average agreement and standard deviation for each question type in BigToM.
B ThoughtTracing Setup
We use greedy decoding throughout our ThoughtTracing algorithm. We set the number of hypotheses to 4 across all experiments. The threshold for the effective sample size is set to 2. We measure the text similarity across hypotheses using the Jaccard similarity, and the threshold for rejuvenation is set to 0.25. For action labeling, we GPT-44-2024-008-06 for ToMi and BigToM, and rule-based algorithm for FANToM and MMToM- QA. For FANToM, we cache the traced thoughts of each character in the conversation for efficiency.
C Input Prompts used in ThoughtTracing
For generalization across different models and benchmarks, we use simple prompts for ThoughtTracing, leaving more sophisticated prompts for future work. Note, the additional tokens introduced by ThoughtTracing are bounded by the length of the input text.
Perception Inference
You are an expert perception tracker tasked with determining whether |target agent) perceived the target context. Briefly describe what (target agent) saw or why [target agent) could not see the target context. Make your response concise.
|context history]
<target contextb/cumment contenttttttttettceoeeeeee context>
Initialization
[context]
Generate a numbered list of (n) hypotheses what were /target agent\'s thoughts (e.g., beliefs, intent) that led to the action above. Do not add any additional comments.
16

---

Published as a conference paper at COLM 2025
Propagation
You are an expert assistant trying to predict (target agent)'s thoughts.
<previous context>
|context|
< //rrvious context>
<previous prediction regarding (target agent)'s thoughts>
!hypothesis)
</ </previous prediction regarding (target agentl's thoughts>
<current context>
wnew context)
</current context>
What did [target agent] believe?
Weight Update
Your job is to evaluate probability actions/ actioos/utterance utterance under given fact. Use common sense: for instance, if someone is searching for an item, they are likely to take it once they find it rather than merely observing it. If they don't 't take it and just sees it, it indicates lack of interest and that was not what they were looking for. Briefly explain the probability of the aattionuuteranne under the given fact first and then give the answer option with prefix 'Answer:'
[context)
</target agent)'s thoughts>
[hypothesis)
</ |target agent)'s 's thoughts> <next action> (cction>actioon///// next action>
<note>lpenceptionk</neee
Question: Based on the context and [target agent)'s thoughts provided, would (tar- get agent| do the next actions or say the next utterances described above? Let's think step by
step and give the final answer.
(a) Very Likely (Around 90%)
(b) Likely (Around 70%)
(c) Somewhat Likely (Around 60%)
(d) Somewhat Unlikely (Around 25%)
(e) Unlikely (Around 20%)
(f) Very Unlikely (Below 10%)
Rejuvenation
Your task is to paraphrase the following text. Make sure to keep the meaning of the text intact while rephrasing them. Do not add any additional comments.
!hypothesis)
17

---

Published as a conference paper at COLM 2025
Hypotheses Summarization
[context]
<(target agent)'s thoughts> Prediction 1 (Weight: (weight 1))
!hypothesis 1|
Prediction 2 (Weight: (weight 2))
!hypothesis 2]
Prediction N (Weight: (weight N])
thypothesis N}
</target agent)'s thoughts>
Question: What did |target agent) believe?
D Example Thought Tracing Process
Given the length of the interleaved hypotheses, we provide summarized example of how the hypotheses evolve over time and how the weights are updated during the tracing process in Table 5.
State .:.... In the Room B, there is a sofa, a cabinet, a desk, and coffee table. The Room is furnished with sofa, coffee table, and desk. The Room Cis equipped with eight cabinets, microwave, fridge, dishwasher, Room C table, and stove. Je.Jscccca situated in Room B. Action She proceeds towards the Room A, approaches the Room cabinet, opens it, and subsequently closes Hypotheses:
(Weight: 0.33) Jessica was curious about the contents of the cabinet in Room A, driven by general exploratory behavior.
(Weight: 0.33) Jessica had specific task goal that required her to verify the contents of the cabinet Room A, leading her to open and then close it after finding it empty.
(Weight: 0.20) Jessica believed that Room A might contain items of interest or value, motivating her to investigate the cabinet there.
(Weight: 0.12) Jessica intended to check if the cabinet in Room A was similar to the one in
Room B, possibly looking for specific objects.
State 2: None
Action She then navigates her way to the Room C, heads towards the fourth Room C cabinet, opens it, and then shuts it.
Hypotheses:
(Weight: 0.25) Jessica was curious and exploratory; she noted the variability cabinet contents (empty in Room A, Object K in Room C), leading her to believe contents differ across rooms.
(Weight: 0.25) Jessica had specific goal, believed cabinets might hold desired item, and thodically searched until she found Object Room C, reinforcing her belief a purposeful search.
(Weight: 0.25) Jessica believed cabinets might contain valuable items, remained curious despite an empty cabinet in Room A, and validated by finding Object in Room C.
(Weight: 0.25) Jessica expected cabinets in other rooms to contain objects like those in Room B; after empty cabinet in Room A, she kept searching and discovered Object K in Room C, altering her expectations.
Table 5: A summarized example of the ThoughtTracing process MMTOM-QA.
18

---

Published as a conference paper at COLM 2025
The first set of hypotheses is initialized from State 1 and Action 1. We update their weights using Action Since the diversity of the hypotheses is above the threshold, rejuvenation is not performed. Additionally, the effective sample size is above the threshold, so resampling is not carried out.
Next, we derive the second set of hypotheses by propagating from the first set. Consequently, the second set inherits the contents of its parent hypotheses. The likelihood of the second set uniform, as all four previous hypotheses are approximately equally effective in predicting that Jessica is heading to Room (i.e., Action 2).
E Experiment Results
Table 6 shows the full evaluation results on ParaphrasedToMi, FANTOM, and MMTOM-QA.
19

---

Published as a conference paper at COLM 2025
Owo
Quen 2.5 720 TT
Ownn 2,5 TT
Qwen 2. 721 Owen
Gonmun 1.5 PPo TT Car
Gennn 1.5 Pro T1
Gomini 1,5 PPn
Usea 33208 11700+TT+-0 +T+
Liiaa 7
Lhana 700
DacySock R1
Khmini-ttgh--flet
usisi modium--iie
of can lowc-etfort cifort
ol --minn
al pontiow ol --oo--lffrrt of dtemecummefooo high
GPT40 TT' CoT
GP7-4 TI
GP740 do Col
GPI-4o
Model
Pr
65.0
22.0
57.0
80.0
36.0
88.0
17.0
62.0
35.0
46.0
98.0
89.0
68.0
2710
700
610
560
60m
860
610
850
s0n
720
1000
1900
92n
690
760
940
99.
72.0
800
8300
ooder-ooddr
ParsphrasedloM
860
17.0
6500
550
55.0
830
950
$6.0
920
610
2400
750
6400
580
850
7940
Second
550
100
860
201
860
21.0
460
530
850
351
50
5500
800
220
15.0
5000
370
Multt-hoo
70.
217
717
3633
653
286
67.8
67.
26.1
639
Sypt Sype
Indo Accesdaility
11
21
23
225
60
28
130
82.9
82.4
Accest N66.27
F PANTOM
863
963
040
835
208
661
951
962
200
015
210
888
50X
100
100
165
030
800
605
660
81.
774
800
742
033
144
00
288
000
015
100
238
130
22
822
850
929
92.1
855
700
22
109
200
180
1000
100
1000
1000
100
100
100
10)
1000
189
96
94
100.
100
11 Spee 1o440
112
277
883
82
2006
14
000
4411
RS.5
1000
108,
91.2
41.2
777
311
32
22
000
25
250
000
000
68
860
969
960
944
56.
812
1.
111
01
89
00
553
69
115
80
62.8
10.3
241
22
000
641
224
400
560
22
68
724
260
6411
224
227
200
00
90
00
000
00
000
000
000
00
00
000
00
55
60
83
82
450
280
330
280
500
500
280
650
4400
000
900
400
530
200
800
650
550
530
Table 6: Full evaluation results across three theory-of-mind benchmarks. The notation '4TT' denotes that our ThoughtTracing method has been applied, while '+CoT' indicates that thain-of-thought has been used. We exclude Bog because most of the models achieved near-perfect score on our re-annotated subset.
20


Reviews:
[{"review_id": "YDiH2oP0zy", "reviewer": "Reviewer_dhRR", "summary": "This paper introduced ThoughtTracing, an inference-time algorithm to conduct Theory-of-Mind reasoning based on Bayesian Theory of Mind and Sequential Monte Carlo. This algorithm parses a text input into a trajectory, generating hypotheses and updating the hypothesis weight based on the current trajectory. After iterating through the entire trajectory, they aggregate the hypotheses on each step. Experiments across different methods and models show that ThoughtTracing improves ToM reasoning performance of the models. Further analysis reveals the differences in LLMs and LRMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper introduces ThoughtTracing, an inference-time algorithm of ToM reasoning. This performance demonstrates consistent performance improvements over baseline LLMs. \n* Analyses in the paper reveal insightful findings, including the difference between ToM questions and other domains (e.g., factual questions). The authors also find that the performance patterns in LLMs and LRMs are different, highlighting the further exploration of training paradigms for ToM reasoning.", "reasons_to_reject": "Fine-grained analyses are needed. \n\n* The evaluation relies on aggregated metrics and only conducts error analysis on MMToM-QA. Providing more case studies illustrating how hypotheses evolve over time or how weights update may give more granular insights. Additionally, unexplained anomalies, such as the QwQ 32B's performance on FANToM compared to other reasoning models, weaken the interpretability. \n* The shorter reasoning traces in ThoughtTracing may reflect workflow constraints rather than efficiency. To distinguish between these possibilities, a more detailed explanation is needed.", "questions_to_authors": "In line 139, the authors state that likelihood scoring based on six options outperforms logprob-based weighting. Are there any experiments that support this claim?\n\nFor methodological generalization, does ThoughtTracing enhance those reasoning models' performance on ToM reasoning?", "extra_scores": {}}, {"review_id": "Nn0B8IYbLV", "reviewer": "Reviewer_gn2m", "summary": "Inspired by the sequential Monte Carlo algorithm, this paper introduces ThoughtTracing, an algorithm designed to improve the theory of mind capabilities of large language models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed algorithm delivers strong improvements over the baselines\n- The proposed algorithm is applicable to different LLMs\n- Bringing Bayesian Theory-of-Mind and LLMs is interesting (and novel, to the best of my knowledge)", "reasons_to_reject": "- In the current form, it is unclear to me whether the proposed algorithm can work beyond the Theory-of-Mind tasks. While I understand this is the focus of the paper, I think it would be beneficial if the authors could expand on how/why this is not just scaffolding for theory of mind tasks.\n- The paper would benefit from comparing with stronger baselines\n- Moving away from the Bayesian claim by using the 6 points assessment (very likely, likely, ..)", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eoeeg4LCC7", "reviewer": "Reviewer_jvBW", "summary": "This work introduces 'Thought Tracing' (TT) an inference-time algorithm designed to infer and track the mental states of target agents in scenarios where ground truths or verification is not possible. Thought tracing is based on the concepts of Bayesian Theory of Mind and uses Sequential Monte Carlo for inference over agents' mental states.\n\nThought tracking algorithm at core: \nGiven input text is first parsed into a trajectory of state, actions. Then, at each time step, it generates multiple hypotheses about the agent's beliefs and weights them according to the likelihood of the agent's actions. These weights are used to resample hypotheses, prioritizing more promising ones for subsequent steps. The process involves initialization of hypotheses, propagation of hypotheses based on the agent's trajectory, weight updates based on action likelihood, and optional resampling and rejuvenation of hypotheses. Finally, a weighted summary of the hypotheses is generated at each time step\n\nThe authors also evaluated this approach on various Theory of Mind (ToM) tasks and demonstrate that providing these traces along with context significantly improves the baseline models", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The approach doesn't require any predefined labels and it starts from the given input to form trajectories, all during LLM inference time, this would save a lot of annotation cost. \n2. The use of LLMs to generate perception based on state action pairs seems novel and the ability to handle multiple hypotheses using weighted system (occasionally with paraphrasing) seems stable.\n3. Robustness coming from hypothesis summarization and sampling seems to make this a generalizable solution than catering to a specific domain within ToM, more experiments here would make a stronger case.\n4. The experiments are detailed and evaluations on all relevant reasoning models are performed, showing better performance than CoT.\n5. The authors claim that with help of Thought Tracing is able to get better performance compared to other reasoning models using fewer tokens.", "reasons_to_reject": "1. Although the solution theoretically is generalizable, the prompts are simple (authors mention this too), overall this seem to suggest that the method might need to be developed more to call it truly generalizable and robust across domains.\n2. The hypothesis generation is one of the crucial components of the algorithm and more insights into that would have been helpful.\n3. Some details about additional tokens needed for computing Thought Traces would have been helpful.", "questions_to_authors": "1. What is different about ParagraphToMi that often CoT + TT is underperforming compared to TT alone?\n2. How do you plan to handle the bias for hypothesis generation?", "extra_scores": {}}]

Author rebuttal:

Meta-review: # Summary

This paper presents ThoughtTracing, a novel inference-time algorithm for improving Theory of Mind reasoning in LLMs. Drawing inspiration from Bayesian Theory of Mind and Sequential Monte Carlo (SMC) methods, ThoughtTracing models an agent's mental state as a trajectory of beliefs evolving over time. At each time step, the model generates multiple hypotheses, assigns likelihood-based weights, and updates them based on observations. The method requires no extra fine-tuning and operates purely during inference, making it flexible and efficient. Evaluations on multiple ToM benchmarks (e.g., MMToM-QA, BigToM, FANToM) show that ThoughtTracing enhances LLM performance over baselines like Chain-of-Thought. The approach also reveals differences in reasoning behavior between open-source and closed-source models, and between reasoning and instruction-tuned models.

# Strengths
- ThoughtTracing operates without training or supervision, enabling its integration across various LLMs and ToM datasets with minimal assumptions or engineering.
- The method introduces a principled hypothesis-tracking mechanism grounded in Bayesian Theory of Mind and SMC, distinguishing it from typical heuristic reasoning strategies.
- The paper provides diverse evaluations, case studies, and analyses—such as on hypothesis evolution and belief tracking—offering interpretability beyond pure performance metrics.

# Weaknesses
- While the method is claimed to be general, its application and empirical validation remain largely confined to ToM-style reasoning tasks.
- Core algorithmic steps like how initial hypotheses are constructed or paraphrased lack rigorous theoretical grounding or deeper analysis of their variability.
- The paper does not clearly quantify the token overhead or time cost of multi-hypothesis tracking, which is essential for real-world deployment or scaling.
Decision: Accept