Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner
Authors: Chunhui Zhang
Zhongyu Ouyang
Kwonjoon Lee
Nakul Agarwal
Sean Dae Houlihan
Soroush Vosoughi
Shao-Yuan Lo
Venue: ICML 2025 spotlightposter (2025)
Keywords: Theory-of-mind
neuro-symbolic concept
long-form planning
Primary area: deep_learning->large_language_models
TL;DR: A scalable Bayesian Theory-of-Mind planner that breaks multimodal reasoning into stepwise belief updates and uses post-trained small LMs to guide larger ones, yielding a performance boost on ToM benchmarks.

Abstract: Theory-of-mind (ToM) enables humans to infer mental states—such as beliefs, desires, and intentions—forming the foundation of social cognition. Existing computational ToM methods rely on structured workflows with ToM-specific priors or deep model fine-tuning but struggle with scalability in multimodal environments. They remain trapped within the gravitational pull of multi-step planning complexity, failing to generalize as task demands increase. To overcome these limitations, we propose a scalable Bayesian ToM planner. It breaks down ToM complexity into stepwise Bayesian updates. Meanwhile, weak-to-strong control specializes smaller LMs to refine ToM-specific likelihood estimation, transferring their ToM reasoning behavior to larger LMs (7B to 405B) for social and world knowledge integration. This synergistic approach enables scalability, aligning large-model inference with human mental states with Bayesian principles. Extensive experiments demonstrate a 4.6% improvement in accuracy over state-of-the-art methods on multimodal ToM benchmarks, including unseen scenarios, establishing a new standard for modeling human mental states in complex environments.

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Planner
Chunhui Zhang'
Zhongyu Ouyang'
Kwonjoon Lee² Nakul Agarwal
Sean Dac Houlihan Soroush Vosoughi" 1 Shao-Yuan Lo
Abstract
CoT Liama-3,1-88
FT Llama-3.1-88
offory-of-Mind (ToM) enables humans infer mental states-such as beliefs, desires, and in- tentions-formingg forming the foundation of social cogni- tion. Existing computational ToM methods rely on structured workflows with ToM-specific pri- ors or deep model fine-tuning but struggle with scalability in multimodal environments. They remain trapped within the gravitational pull of multi-step planning complexity, failing general- ize task demands increase. To overcome these limitations, we propose scalable Bayesian ToM planner. It breaks down ToM complexity into stepwise Bayesian updates. Meanwhile, weak-to- strong control specializes smaller LMs to refine ToM-specific likelihood estimation, transferring their ToM reasoning behavior larger LMs (7B to 405B) for social and world knowledge integra- tion. This synergistic approach enables scalabil- ity, aligning large-model model inference human mental states with Bayesian principles. Extensive experi- ments demonstrate a 4.6% improvement in accu- racy over state-of-the-art methods on multimodal ToM benchmarks, including unseen scenarios, es- tablishing new standard for modeling human mental states in complex environments.
CoT Llama-3.1-708
FT Llama-3.1-708
Llama-3.1-405B
Ours Llama-3.1-405B
CoT o1-mini
0.9
Accuracy
0.8
0.7
0.6
0.5
0.4
0.3
0.2
1
9
11
Plan Steps
Figure I. Comparison of models on planning tasks in VirtualHome (a simulator used in multimodal ToM). As plamning steps increase, smaller models (e.g.. Llama3, 1-8B, 1-BBB, 70B) and inference-time seal- ing (e.g.. ol-mini, CoT) fail to sustain accuracy. Only larger models (c.g., Llama3.1-405B) maintain performance, demonstrat- ing that sustaining accuracy requires model scaling.
tinct from one's own (Dennett, 1988; Gopnik & Wellman, 2012). Similarly, ToM tasks in AI involve perceiving ob- servable cues sucs-such an agent's 's actions and surrounding context- to predict their goals and beliefs. Equipping Al systems with ToM capabilities could unlock their potential for human-like social understanding and interactions (Lake et al., 2017; Wu et al., 2021; Ma et al., 2023).
Existing approaches ToM reasoning follow main strategies: (i) structured planning workflows with ToM- specific priors (Baker et al., 2017; Jara-Ettinger, 2019; Shu et al., 2021), ),, and (ii) integrating these priors into language models (LMs) via specialized training (Rabinowitz et al., 2018: 8; Shu et al., 2021; Sclar et al., 2022; Jin et al., 2024). The challenges extends beyond method designs- envi- ronments demand multimodal reasoning, integrating visual, textual, and contextual inputs into coherent mental state inferences. Understanding and predicting the goal of agent planning behaviors require models to reason across multiple steps while integrating multimodal cues.
1. Introduction
Theory-of-Mind (ToM) is cornerstone of human social cognition, enabling individuals to attribute and infer men- tal states themselves and others. This capacity under- pins the recognition that others may have perspectives dis-
See the contributions of student and senior authors. 'Dartmouth College, Hanover, NHI USA 'Honda Research Institute USA, San Jose, CA. USA. Correspondence to: Soroush Vosoughi <soroush, esoroush.vosoughiee @ dartmouth.edu>. Shao- Yuan <shao- yoan ri.mm.........
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
However, these challenges introduce generalization barri- ers, making it unclear whether standard approaches can

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
scale effectively. To examine these challenges, Fig.1 com- pares the performance of various models in planning tasks with the physical simulator used by multimodal ToM bench- marks. While inference-time scaling methods-Suun such as CoT reasoning, ol systems, and fine-tuning experts-provide in- cremental improvements, they fail to maintain accuracy as task complexity increases. Only larger models, such as 1 405B, sustain performance over increasing planning steps, while smaller LMs exhibit rapid accuracy degradation. These scalability challenges stem from two fundamental factors: First, reasoning boundary exists, limiting the number of effective reasoning steps regardless of the method used. Prior studies (Chen et al., 2024; Zhang et al., 2024; Ye et al., 2025; Gao et al., 2025) show that CoT reasoning and ol/rl-like scaling methods plateau in effec- tiveness as task complexity grows, leading to diminishing returns. Second, multimodal ToM reasoning relies heav- ily on integrating broader social and world knowledge, which correlates strongly with model scale. Unlike closed- form logic tasks, ToM reasoning in complex environments requires grounding in extensive, diverse contexts. As shown in Zhang et al. (2024); Yu et al. (2024); Sun et al. (2024); Gao et al. (2024); Yuan al. (2025); Diao et al. (2025b); Zhang al. (2025b), smaller LMs struggle to generalize ef- fectively due to their limited pretraining capacity to encode and utilize world knowledge. These findings emphasize that simple fine-tuning or inference-time scaling alone is insufficient to improve ToM reasoning at scale. Instead, sustained performance requires both structured frameworks and models with expansive generalization capabilities, as demonstrated by 1, deviating from the existing methods.
while maintaining Bayesian consistency for stable and in- terpretable reasoning. The theoretical foundation of our approach is established by Theorem 1, which formalizes its effectiveness through KL divergence analysis. Empirical results demonstrate that our scalable solution enhances the generalizability of Bayesian inference and achieves a 4.6% accuracy improvement over state-of-the-arr methods on mul- timodal ToM tasks, including unseen scenarios, setting a new standard for modeling ToM in complex environments.
2. Preliminaries
Behaviour modelling: a Markov decision process formu- lation The behaviour of an agent can be formulated as a forward generative model based on a Partially Observable Markov Decision Process (POMDP), defined by the tuple (S, A. T. G, R, Ω2 O,)) (Kaelbling et al., 1998; Jin et al., 2024). Here, s E S and a² E A represent the state and action at time t, respectively. T(s*Is, a) denotes the state transition probabilities. The goal E G determines the reward r' = R(ss, a', a'.9). The agent's 's observation of e Q2 is obtained via the observation function of = O(s*). The dis- count factor is y E (0, 1]. Crucially, the agent's belief, b(s), is probability distribution over the state. This belief is dy- namically updated during belief evolution P(b 60-1,s77 where b(s) is factorized into probabilities over the possible locations of individual objects.
In our practice, 6 is initialized as all possible object lo- cations at the start. At each step т, if an object is ob- served inside container, b" is updated to include the con- tainer; otherwise, unobserved locations are removed from br P(bT)b--1, 5²) can be approximated using likelihoods from an LM.
Motivated by these scalability challenges, we introduce more scalable Bayesian ToM solution that directly addresses inference-time reasoning complexity through Bayesian In- verse Planning (BIP) (Baker et al., 2007; 2009; 2017: Shum et al., 2019; Jin et al., 2024) and overcomes world knowl- edge dependency by scaling LMs up to 405B for ToM- specific generalization. First, BIP mitigates the reasoning boundary by decomposing multimodal ToM reasoning into modular, stepwise Bayesian updates-stt -uch as state transi- tions, belief updates, and action likelihoods. This structured framework refines beliefs and hypotheses iteratively, en- suring tractability even in complex environments. Second, overcoming the reliance on broad social and world knowl- edge requires scaling to larger LMs, as illustrated in 1ig.1. Prior approaches (Jin et al., 2024) relied on smaller LMS for likelihood estimation, but their limited capacity hindered generalization in rich ToM settings. Our weak-to-strong control mechanism enables smaller, post-trained LMs to specialize in ToM-specific tasks and transfer these learned behaviors larger LMs (up to 405B) during inference. This innovation allows the larger LM to serve as the primary policy model, leveraging its extensive world knowledge
Inverse inference: from observed behaviours to men- tal states Bayesian inverse planning (BIP) infers an agent's goals and beliefs by inverting the forward POMDP model (Baker et al., 2017). Given observed states gh# and actions a 1:t-1, the posterior probability of a goal g and belief b' is expressed as:
P(g, P(g,b'|s b'isstt, aa 2
x II (aa'g,bb)) r=1
- P(1)))))))) (1)
where <<a" [g. b²) represents the agent's policy-the the proba- bility of taking action a" given its goal g and belief 67. This process combines the likelihood of actions and belief up- dates with prior probabilities, refining beliefs incrementally as new evidence observed. To compare hypotheses about
2

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
an agent's goals, we evaluate their relative log-likelihoods:
two phases: In instruction tuning phase, a scenario-specific policy æ0o is refined using an action-policy experience pool D {(si, bi, 9i, a.)] where 5, b, g. and represent states, beliefs, goals, and actions. The tuning objective maximizes the likelihood of observed actions:
P(g1, by aalssitt""a al.t)) log
6r)
log
Plaz, bl g1:t, al:t)
ata" 192,
steps comparison
+ log T m(a b1) т Taall92 122)
+ log P( B st)
(2)
LIT(
Σ log n fo
18 bis D:).
(5)
P(b b
Current step comparison
This step builds mapping from multimodal environments to goal-directed actions. Then in preference optimization phase, LM is further aligned by distinguishing between effective (a and ineffective (a") actions. a* corresponds to concise, successful outputs, while a represents long descriptions or failed actions. The preference loss, modified from DPO (Rafailov et al., 2023), is defined as:
The first term compares cumulative likelihoods across prior steps, while the second term evaluates how well each hypoth- esis aligns with the agent's 's latest action and belief update,
In practice, the hypothesis with the highest accumulated likelihood is selected, and π raa lg, by) can be approximated using likelihoods generated by language model.
Lpo -Exxa aaaa-aa-yaa jaD [log (3 A log क r)] T (a s. g)
3. Bayesian ToM Planner at Scale
(6)
+d Ex.anx
log T (a s, b.b.g)
Our scaled Bayesian planner infers an agent's mental state based the unified representations about scene, per- son's actions, and the mental state hypotheses from multi- modal inputs, and then post-train an LM to conduct contex- tual inverse symbolic planning, based on unified symbolic representations (Jin et al., 2024). Then, as shown in Fig.2, the LM used in our scaled Bayesian planner is from 7B up to 405B parameters at test-time compute. This approach harnesses the world knowledge and reasoning capabilities of large LM, avoiding additional post-training.
where A log nf = log (at
s, log
8, b, g) represents the log-odds. Here, 3 controls the sharp- ness of preference learning, while regularizes deviations from the initial policy = ensuring stable training.
Inference stage: large policy model with behavioral guid- ance. During inference, we leverage the behaviour acquired by the ToM preference-optimizee smaller LM to guide the reasoning of larger, more capable LM. This approach dy- namically adjusts the output of the large LM based on the shift observed between a post-trained small LM of and a naive small LM T At each inference step t, the policy distribution for the redirected large LM is given by:
Weak-to-strong controlled large policy model When augmenting likelihood estimation with the guided large LM's broad capabilities, we scale up the LM used only for inference-time computing and avoid the direct post-training on large LM. The true policy π =(a", g. b²) is estimated through a LM esttmmttd probability (aa ss'g,, 6*):
1 T(a²| Is' s'',g,8') = E ('',,,"))) T 7((a' s S.gg.0 8') NNa 8 g, 80)
T (aa 6") = (aa" | b¹) +e,
(3)
where L s.gg 89) represents the policy distribution from the naive large LM. The post-training effect to policy function is approximated through the ratio nf (s''s""..88 offering an on-the-fly redirecting mechanism. The nor- malization factor is calculated by Z = E.. (a I
where E represents the inherent approximation error. When applied to the Bayesian inverse planning (1), the posterior probability expressed as:
P(g. b' stt a
II ((a" 1 .g) +
<=(aals'...bb) ==(a''''....") It ensures the resulting probabilities being aligned, reflecting both the post-training adjustments and the base likelihood from the larger LM. Our overall method facilitates ToM behaviour transfer from the post- trained small LM (nf) ) to the larger LM (Tr scaling the large policy model's 's capabilities in BIP at inference time.
s", g. 8*)
P(b 1 s²) ((b))(()) (4)
It integrates the approximation into Bayesian planner, con- sidering 8;, bi, gi, 0; updates over time.
Post-training stage: ToM optimization To specialize the LM's behaviors to estimate likelihoods and reduce approx- imation error & in Eq 3, the post-training stage consists of
Below Eq 8 shows this behavior control relies on the learned As to approximate the scaled gradient V.EC.C y) with higher-order terms contributing to the residual error.
3

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Bayesian Inverse Inference
K
Weak to Strong
P(a) P(b)
+da
T+4
4g
Latent Behavior Change 4
Pear
in
basket
Pear
the basket
Large Langue Model
Base Small
Past- trained
Likelhood from the base small LM
Likellhood from large
LM
Likelihood post-trained small
Likeliheed adjusted from the small LM
(S9, 20,6,
(S* b". 8)
Hypo a/b
(i) post-training small LM
(ii) extroct lotent behavior change (d)
(iii) apply 4 on the lorge LM
Figure 2. (left) The large LM operates a scaled policy model (e.g.. 405B) to estimate the likelihood of an agent's actions in dynamic environments, based multimodal symbolic imputs (video and description). (right) The latent reasoning the large LM is guided by the ToM behaviours from post-trained small LMs, which acts weak-to-strong scaling control. Overall, Bayesian inverse planning compares hypotheses about the agent's 's goal and belief, using the large LM as policy model to infer ToM.
described by Jin et al. (2024). The dataset comprises 1,000 procedurally synthesized videos within realistic household simulator, each annotated with states, goals, beliefs, and actions. (ii) For evaluation, we use the MMToM- QA (Jin et al., 2024), evaluation benchmark aimed at evaluating ToM reasoning over multimodal situations. The dataset consists of 134 videos, each showing person searching for household objects, with an average of 1,462 frames per video representing approximately 36 human actions. These videos are accompanied by 600 questions (detailed in App. $D.1). evenly divided between the categories of belief in- ference (with 1.1, 1.2, and 1.3 subtasks) and goal inference (with 2.1, 2.2, 2.3, and 2.4 subtasks). The questions assess the ability of models to infer goals and beliefs jointly.
Let x² be the directly post-trained base model. Suppose the output adjustment (x( $xe (Xx(==<t) Saxx (X,( approximates the scaled negative gradient of the CE loss for outputs, i.e., As -IVV,CcE(S,.. where is the learning rate. Then, the model т, defined by 8* (Xx xxe))= = 8x2 (X) + As( (((x((=)) approximates π². The KL divergence be- tween their output distributions is:
Theorem 1.
1/2 (Rr (P Xmax V +
(8)
where Amax is the maximum eigenvalue of the Hessian of the cross-entropy loss for the outputs.
T does not need to strictly approximate the exact loss gradient of x Instead, r can use its intrinsic capacity to adapt to ToM scenarios, based only on the approximated As learned by the small LM. See App for the proof.
We include three types of baselines. text- only evaluation, we compare performance in the text-only subset of MMToM-QA using various LMs, including GPT- 4 (OpenAI, 2023a), GPT-3.5, Llama2-7B (Touvron et al., 2023), OpenAl-o3-mini, DeepSeck-R1-6711 (Guo et al., 2025). Advanced prompting methods, such as Sim'oMM (Wilf et al., 2024) and SymbolicTooN (Sclar et al., 2023), which enhance GPT-4's reasoning capabilities, provide ad- ditional baselines (e.g.. SimToM with GPT-4 and Symbolic- ToM with GPT-4). For multimodal evaluation, we include GPT-4V (OpenAI, 2023a), Video-Llama2 (Zhang al., 2023), and LLaVA (Liu et al., 2023), BIPALM (Jin et al., 2024). For human, 180 participants answer 120 randomly
Baselines
4. Experiments
Fig.1 first shows the scaling benefits in physical simulated scenarios. To dive into our BIP method in multimodal ToM, for the strong component, we scale up the large LMs to 70B and 405B parameters. In contrast, for the weak component, we reduce the size of the small LMs from 8B to 4B parame- ters. First, the results reveal a positive correlation between model size and ToM capabilities, especially when the larger models are guided by the post-trained behaviours of the smaller models. Interestingly, these post-trained behaviours are also effectively captured by smaller LMs. We also illus- trate how the large LMs are progressively redirected to the answer space during the Bayesian process.
sampled questions, covering all question types, as reported by Jin et al. (2024).
Post-training We post-train Llama (Touvron et al., 2023; Dubey et al., 2024) policy model with LoRA (Hu et al., 2022), as outlined in Tab.7. Following the setup recom- mended by Jin et al. (2024), we use learning rate of le-3 over 3 epochs. LoRA is configured with a rank of 16 and an alpha value of 32 for the 7B and 8B LMs. For 70B, we use lower rank of 8 and an alpha of 16.
4.1. Setup
(i) For post-training, we use MMToM sam- pled from an apartment environment simulator, Virtual Home (Puig et al., 2018), using the procedural methods
Datasets

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
4.2. Main results
the 8B model (65.19% and the 70B model (66.62%). No- tably, the performance of the 405B model approaches that of the post-trained Llama2-7B. Furthermore, the improvement from 70B to 405B suggests that the benefits of scaling have not yet reached saturation, indicating potential for further gains with larger models. (ti) Post-training significantly enhances LMs' performance on ToM tasks, even when larger LMs already perform well in zero-shot scenarios. This effect is consistent across model sizes, from smaller models such as 7B/8B to larger models up to 70B, regardless of the specific version (I.ama2,, Llama3, or Llama3.1), For belief inference tasks, which are closely tied to world knowl- edge, post-training helps align the large models' knowledge more precisely with the input questions. For goal inference tasks, which are linked to environmental dynamics, post- training refines the models' atomic-level reasoning (i.e.. predicting based on s, b, g). resulting in greater improve- ments compared to belief inference. This suggests that post- post-training provides a more substantial benefit for tasks that require dynamic reasoning. (iii) Our weak-to-strong control approach approximates the benefits of direct post- training in Bayesian inference. When comparing models such as Llama2, Llama3, and Llama3.1, we find that direct bost-training the 70B model, even with adjusted hyperpa- rameters from the 8B model (e.g., reducing the alpha value from 32 to 16), does not produce results as stable as our method. We attribute this to the difficulty of finding opti- mal hyperparameters for larger models, which require more extensive tuning. In contrast, our weak-to- to-strong control, which uses a well-trained smaller LM to guide the larger LMs, allows for more consistent improvements without the need for extensive hyperparameter trials.
Tab.1 uses human performance as the gold standard, with humans achieving clear lead of 93.0% accuracy on tasks with multimodal input. Among the models, our solution with multimodal input achieves the highest performance, highlighting the critical role of integrating both visual and textual modalities. Comparisons between LMs alone and those augmented with ToM workflows (e.g., SimToM, Sym- bolicToM,, or BIP) further demonstrate the benefits of the ToM workflow. Specifically, the ToM workflow improves performance by decomposing the complexity of multi- modal ToM reasoning into modular steps. In belief infer- ence, which is strongly linked to world knowledge, models like GPT-4 and GPT-3.5 perform exceptionally well, par- ticularly on task 1.1, where GPT-4 achieves an accuracy of 94%. This result underscores the importance of large-scale models in capturing and applying vast amounts of pretrained world knowledge. However, despite their impressive perfor- mance in belief inference, these models do not perform as effectively on goal inference, where adaptation to specific ToM contexts and dynamic environments is crucial. This highlights the need for models to be better aligned with the specific requirements of ToM scenarios. Models with smaller scales, such as those with 6B, 7B, and 13B parame- ters, face inherent capability limitations, which restrict their performance on belief inference tasks, particularly when compared to larger models like GPT-4 on task 1.1. However, these smaller models, such as BIPALM w/ GPT-J-6B and Llama2-7B, benefit from post-training specifically designed for ToM scenarios. This allows them to perform better on goal inference tasks, where understanding and adapting to scenario-specifie environmental dynamics is essential. De- spite their size constraints, these models demonstrate the value of targeted post-training in compensating for the lack of large-scale pretrained knowledge. Our ap- proach goes beyond seesaw effects in prior methods and has both strengths: while its strong component leverages the extensive world knowledge embedded in large pretrained models, also its weak component incorporates post-training to the ToM contexts and environmental dynamics required. This dual advantage allows a balanced performance across both task types (belief & goal inference). with an over- all 81.3% accuracy on multimodal tasks and exhibits a 4.6% improvement over the existing best baseline.
4.4. Scaling-down small LMs are effective controllers
In the Bayesian planner, prior experiments show that post- trained behaviours from small LMs can effectively guide the pretrained capabilities of larger LMs during test time. To further study the role of post-training to weak-to-strong control, Tab.3 investigates whether post-trained behaviours can be learned effectively with reduced computational re- sources, while the pretrained capabilities of larger LMs are still available. Specifically, we examine whether downsized smaller LMS can effectively capture these post-trained be- haviours and guide the pre- trained capabilities of larger LMs without compromising performance. We use 8B LMs as baselines in normal size, and we also downsize them to two 4B variants: Llama3. I -Miintronn- Soreittron----aaiee Width, which reduces the hidden size of each layer; and L --Minttonn--- Depth, which cuts the model depth (Sreenivas et al., 2024). Despite their smaller size, they maintained comparable ac- curacy in weak-to-strong control. While the 4B-Width LM underperformed the 4B-Depth LM in zero-shot scenarios, its
4.3. Stronger Large LMs enhance likelihood estimation
In the Bayesian framework, we explore the role of LMs in likelihood estimation and examine how their scale and post-training affect performance across various ToM tasks. According to Tab.2, (i) our results demonstrate a positive correlation between LM size and ToM task performance. For instance, in the zero-shot setting of 1lama3.1, the 405B model achieves an accuracy of 69.43%, outperforming both
results surpass the 4B- 4B-Depth, especially
5

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Table Comparisons between humans and models across task types from 1.1 2.4 are provided. The hest results for each modal setting are highlighted in bold. The second best results in multimodality are underlined. Rows of ours are highlighted in color.
belief inference
goal inference
method
all
1.2
1.3
avg.
2.2
2.3
2.4
avg.
Human
96.0
95.8
81.3
91.0
85.8
76.7
65.0
68.3
74,0
82.5
GPT-4
97.0
12.0
77.0
62.0
48.0
42.7
2.7
42.7
34.0
48.0
SimToM w/ GPT-4
96.0
15.0
82.0
64.3
61.3
44.0
2.7
54.7
40.7
52.5
text only
SymbolicToM w/ GPT-4
100
61.0
74.0
78.3
73.3
66.7
0.0
50.7
47.7
63.0
BIPALM w/ GPT-J-6B
88.0
69.0
88.0
81.7
77.3
68.0
30.7
70.7
61.7
71.7
DeepSeck-R116611
92.0
50.2
72.4
71.5
68.3
44.0
45.0
48.2
51.4
61.5
OpenAT-o3-mini
83.1
47.6
62.3
64.3
64.0
38.6
38.7
46.0
46.8
55.6
Ours (w/ Jamm3.114055P
90.1
70.5
87.4
82.7
68.8
75.5
75.3
71.8
72.9
77.8
Human
69.1
64.3
86.4
73.3
58.5
60.0
76.7
63.3
64.6
68.9
video only
Video-Llama2 Llama2-13B
24.0
32.0
67.0
41.0
50.7
45.3
56.0
52.0
$1.0
46.0
LLaVA-7B
33.0
15.0
69.0
39.0
44.0
24.0
56.0
57.3
45.3
42.2
GPT-4V
64.0
34.0
39.0
45.7
54.7
26.7
48.0
56.0
46.3
46.0
BIPALM w/ GPT-J-6B
63.0
57.0
72.0
64.0
45.3
62.7
50.7
62.7
55.3
59.7
BIPALM w/ Llama2-7B
69.0
63.0
60.0
64.0
62.7
54.7
53.3
62.7
58.3
61.2
Human
95.8
96.7
100
97.5
90.0
91.7
83.3
88,9
88.5
93.0
36.0
38.0
52.0
42.0
36.0
41.3
30.7
45.3
38.3
40.2
multimodal
LLaVA-7B
46.0
14.0
69.0
43.0
65.3
22.7
40.0
48.0
44.0
43.5
GPT-4V
94.0
13.0
59.0
55.3
56.0
26.7
4.0
52.0
34.7
44.0
BIPALM w/ GPT-J-6B
90.0
69.0
86.0
81.7
68.0
78.7
56.0
73.3
69.0
75.3
BIPALM w/ Llama2-7B
88.0
68.0
85.0
80.3
62.7
77.3
72.0
80.0
73.3
76.7
Ours (w/ Llama3.1-405B)
92.1
76.0
93.0
87.1
73.4
80.0
75.5
78.7
76.9
81.3
when controlling the 70B large LM, demonstrating its supe- rior transferability. These results highlight two key points: (i) downsizing the weak component can still effectively guide larger LMs without signiticant loss in accuracy, and (ii) reducing model width, rather than depth, tends to be more generalizable, as deeper models demonstrate better transferability-aßie aligning with learning principles of the width-depth trade-off's in small-scale studies (Telgarsky, 2016; Lu et al., 2017; Raghu et al., 2017).
achieved in the original apartment scenario and also remains close to the performance of the 8B LMs. This consistency suggests that downsizing the weak component does not sig- nificantly affect performance, even in new and diverse test environments. These results indicate that our approach has strong potential for continually downsizing smaller LMs as controllers since they also are capable of captur- ing the post-trained behaviours. It allows saved resources to be potentially allocated to stronger controlled LMs, while still keeping stable to scenarios unseen previously.
4.5. Transferability of scaled Bayesian planner
4.6. Weak-to-stroe control redirects large LM
Although the small LMs are nost-trained on the apartment, overall framework is expected to be stable and generaliz- able across various unseen scenarios. To evaluate the trans- ferability, Tab.4 compares our method with baseline models in five previously unseen scenarios: Andersen fairy tales, an- cient Egyptian, outer space, wild west, and medieval castle. These diverse settings assess the generalisability of our ap- proach beyond the post-training scenario. When scaling the strong component (i.e., the large controlled LMs) from 70B to 405B across these new scenarios, there are continuous im- provements in ToM understanding. This demonstrates that the increased capacity of our scaled solution enhances the transferability of ToM reasoning across multiple dynamic and unseen environments. Furthermore, when the weak controller component is reduced from 8B to 4B, performance remains stable, ranging between 78.0% and 79.15%. This result is comparable to the 79.05% accuracy
To quantify the influence of the weak controller in Bayesian planner, we analyze the estimated likelihood changes before and after applying weak-to-strong control at each step. Fig.3 samples ten test cases from five datasets and averages the results. It illustrates the progressively increasing magnitude of likelihood changes as Bayesian inference progresses:
(i) At the beginning, when the large LM is close to general initial state, the likelihood changes are minimal. This is be- cause the general state aligns closely with pretrained world knowledge, requiring little correction from ToM-specific behaviours; (ii) As the model approaches a more specialized final hypothesis, the likelihood estimates are increasingly redirected. This occurs because the specialized scenarios demand ToM-specific behaviours, which the post-trained small LMs are fine-tuned to capture. The post-trained small
6

---

Overcoming Multi-step Complexity in Multimodal -heord Reasoning
Table 2. Scaling-up performance on strong component (large LMs) in weak-to-strong control.
LM
config
belicf inference
goal inference
all
1.1
1.2
1.3
avg.
2.1
2.2
2.3
avg.
78-zero-shot
44.00
37.00
84.00
55.00
64.00
65.33
62.67
64.00
64.00
60.14
Llama2
7B- "B--pos-ttrained
80.00
60.00
89.00
76.33
74.67
60.00
78.67
66.67
70.00
72.71
70B-zes zero-zhoot
64.00
47.00
93.00
68.00
56.00
72.00
25.33
70.67
56.00
61.14
pos--oost-raiied
90.00
70.00
87.00
82.33
78.67
76.00
61.33
72.00
72.00
76.43
70B-ours
89.00
70.00
90.00
83.00
73.33
74.67
76.00
73.33
74.33
78.05
8B-zero-shot
88.00
72.00
91.00
83.67
65.33
57.33
13.33
53.33
47.33
62.90
Llama3
SB-post-trained
92.00
72.00
83.00
82.33
77.33
73.33
72.00
70.67
73.33
77.19
70B-zcro-shot
69.00
67.00
95.00
77.00
42.67
70.67
16.00
52.00
45.33
58.90
poB-post-trained
91.00
70.00
89.00
83.33
73.33
74.67
44.00
69.33
65.33
73.05
70B-ours
91.00
75.00
92.00)
86.00
68.00
72.00
74.67
78.67
73.33
78.76
8B-zero-shot
88.00
72.00
91.00
83.67
65.33
62.67
22.67
54.67
51.33
65. 19
8B SB-post-traing
90.00
71.00
93,00
84.67
69.33
72.00
62.67
72.00
69.00
75.71
Llama3.1
70B-zero-shot
85.00
63.00
93.00
80.33
72.00
76.00
16.00
61.33
56.33
66.62
70B- 7OB-post-rrained
91.00
69.00
95.00
85.00
69.33
80.00
29.33
69.33
62.00
71.86
405B-zcro-shot
86.00
70.00
90.00
82.00
73.33
78.67
21.33
66.67
60.00
69.43
70B-ours
90.00
74.00
93.00
85.67
74.67
77.33
70.67
76.00
74.67
79.38
405B-ours
92.10
76.00
93.00
87.10
73.40
80.00
76.50
78.67
77.14
81.29
WOB-post-traings
91.20
71.21
94.10
85,50
74.50
79.43
66.50
79.00
74.86
80.18
70B-ours
92.33
72.00
94.00
86.11
75.20
81.10
75.00
77.85
77.79
81.95
Table Scaling-dows effect on weak part (small LMs) scaled Bayesian planning. All models based on Llama3.
LM
config
belief inference
goal inference
all
1.1
1.2
1.3
avg.
2.1
2.2
2.3
2.4
avg.
zero-shot
88.00
72.00
91.00
83.67
65.33
62.67
22.67
54.67
51.33
65.19
8B
post-trained
90.00
71.00
93.00
84.67
69.33
72.00
62.67
72.00
69.00
75.71
8B 4- 70B
90.00
74.00
93.00
85.67
74.67
77.33
70.67
76.00
74.67
79.38
zero-shot
79.00
69.00
89.00
79.00
60.00
69.33
24.00
52.00
51.33
63.19
wid.
41
post-trained
90.00
72.00
87.00
83.00
70.67
72.00
68.00
78.67
72.33
76.90
4B-width 9- 70B
90.00
71.00
90.00
83.67
74.67
74.67
76.00
73.33
74.67
78.52
zero-shot
91.00
74.00
88.00
84.33
69.33
77.33
20.00
66.67
58.33
69.48
dep.
4B
post-trained
91.00
71.00
90.00
84.00
65.33
65.33
76.00
69.33
69.00
75.43
4B-depth 7 70B
91.00
72.00
91.00
84.67
72.00
74.67
84.00
64.00
73.67
78.38
LMs are specifically fine-tuned to the ToM context, enabling them to model human actions, goals, beliefs, and environ- mental states across unseen scenarios. Overall, this anal- ysis finds that the weak component progressively redi- rects the output of larger models, guiding them toward more accurate ToM predictions among unseen scenarios throughout the Bayesian inference.
likelihood estimation at the concept level. Fig.4 shows the execution of ten inference trials with temperature of 0.7. The scenario involves the agent James interacting with objects in an apartment, aiming retrieve a bottle of wine. The initial state s; is pear in the basket, no wine, the belief b; is wine in the cabinet, the goal gi is obtain bottle of wine, and the action o; is open basket, walk to cabinet. The baseline small LM assigns lower likelihoods to fine- fineggrained item-level concepts (e.g., wine, wine glass). After post- training, the small LM significantly shifts its focus toward item-level concepts, aligning its predictions more closely with the action space. This adjustment increases the like- lihood assigned critical items like wine and wine glass, which necessary for accurately predicting the agent's
4.7. Post-training aligns large LM's likelihood estimation at the concept level
Previous experiments demonstrated that post-training small LMS can progressively guide the behaviour of large LMs throughout Bayesian inference. Now, we further fo- cus on how post-trained small LMs influence large LMs'

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Table Transfer performance of the Bayesian method with different scaling settings (zero-shot, direct post-training, and our weak-to-
strong control) from the apartment scenario to various unseen environments. All models are based on Llama3.1 1. Results are average accuracy of belief inferenceigoal inferenceloveraal for each scenario. Detailed unseen scenarios and results are in $E.6&E.7.
solution
apartment (seen)
Andersen tales
ancient Egyptian
outer space
wild west
medieval castle
Raw
70B-zero-shol
80.3/56.3/6.66
83.6/60.6/700
33.6/60.6/69.
$4.0/58.0/69.1
27.6/77.6/68.3
82.6/57.6/68.1
70B-post (0B-post-trained
85.0/62.0/71.. 1.8
84.6/66.3/74.1
84.6/66.3/75.3
83.0/66.0/73.2
81.0/65.0/71.8
$1.0/65.0/71.
4B-wide 9- 70B
83.6/74.6/78.5
84.0/75.3/79.0 0
83.0/75.3/79.1
82.6/75.3/78.4
84.0/74.6/78.6
84.6/73.0/78.0
Ours
4B-depth 9- 70B
84.6/73.6/78.3
85.0/71.3/77.1
85.3/71.3/77.9
81.6/71.0/555
83.3/71.3/76.4
83.3/64.0/72.2
8B 9+ 70B
85. 85.6/74.6/79.3
02.6/76.0/78.
83.6/76.0/77.7
84.0/75.0/78.
83.3/74.0/78.
83.6/75.0/78.77
8B 9- 405B
87.0/77.0/81.3
85.8/76.0/80.2
86.0/76.3/80.4
87.2/75.5/80.5
85.3/76.0/79.9
85.6/75.2/79.7
6
Table S. Directly adding small LM (8B) to large LM by naive logit combination, without
control, is suboptimal.
LM
Method
Belief Avg.
Goal Avg.
All Avg.
Llama3.1 70B
Naive Logit Add
82,50
66.40 65.67
74.45
Llama3. I 405B
Naive Logit Add
83.67
74.67
likelihood change 3
andersen fairy tales
ancient egyptian
goal. Consequently, post-training enables the small LM to better capture fine-grained details of the agent's ' behaviour, improving ToM predictions. In contrast, the large LM dis- tributes its likelihood more evenly across all levels, from rooms to items, reflecting broad understanding of the en- vironment. While this approach captures general spatial wwareness-iddenifyingg key areas like the kitchen and furni- ture like the cabinet-it lacks the sharp focus on fine- fine-grained details, such as wine and wine glass, which are crucial for this task. As result, the large LM may struggle with tasks that require precise, item-level predictions.
outer space
wild west
medieval castle
Bayesian state
Figure 3. Likelihood change during Bayesian inference under weak-to-strong control. Results are averaged over sampled cases across five different unseen scenarios.
Rooms
Furiitrres
lange
Overall, post-training helps small LMs focus on iteml concepts, making it more effective for this task. While the large LM captures a broader understanding of the physical environments, it benefits from post-trained behaviours that redirect its likelihood estimation toward fine- tineggrined, item- level predictions. This finding reflects the role of post- trained small LMs in guiding large LMs' concepts in ToM reasoning.
arga Lt
1.25
1.55
1.00
6.75
0.75
0:50
8:00
Ldee
Entreen:
Rizten
Items
base small LM
post-trained small LM base large LM
1.25
1.00
0.75
Is Simply Adding Small LM Effective? We also ex- amine whether naïvely combining small LM with large LM-by directly adding their logits, without structured W2S adjustment-can close the gap. As shown in Tab.5, this approach yields lower performance than our method, The results demonstrate that naively incorporating a small LM suboptimal; explicit weak-to-strong control is essen- tial to effectively abstract the specialized ToM behaviors of the small LM and fuse them with the large LM's pretrained world knowledge. These ablation studies conclusively val- idate that our weak-to-strone control mechanism is both critical and independently responsible for the strong gener- alization and ToM grounding observed in our method.
0.50
0.25
0.00
Apple
Bosk
Chips
Condinent
Cuptalie
Dich Bont
Plate
Rint. Carl
Saimon
Water Glacs
Wiw
Win Giass
Figure 4. Likelihood estimation across different levels of concept granularity (rooms, furniture, and items) for base small LM, post- trained small LM, and base large LM. The Bayesian framework uses an LM the policy model to infer actions conditioned on states, beliefs, and goals, where actions often refer to fine-grained item-level concepts (e.g.. wine, wine glass). It highlights the trend of how each model allocates likelihood across these concept levels. The ToM scenario of this case is detailed at $E.5.
8

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
5. Related Work
6. Discussion and Conclusion
5.1. Modelling human mental states
This study investigates scalable Bayesian inference in com- plex and dynamic ToM environments. Existing methods based on normal-sized LMs often fail to provide sufficient reasoning capabilities and world knowledge, particularly when used as likelihood estimators in diverse challenging ToM scenarios. Therefore, to overcome these limitations, our solution abstracts and transfers the post-trained behav- ioral patterns of smaller LMs. This approach allows the extensive world knowledge of large LMS to be progressively redirected towards ToM reasoning at inference time. Con- sequently, we avoid additional post-training resources for large models, yet allow effective inference-time scaling of Bayesian ToM reasoning even in dynamic and complex physical scenarios.
There are many studies on understanding human behaviour by classifying and predicting physical motion patterns (Ag- garwal & Ryoo, 2011; Caba Heilbron et al., 2015; Choi & Savarese, 2013; Shu et al., 2015). Beyond physical be- haviour, some studies focus specifically on modeling hu- man mental states, i.e. ToM. ToM models have followed two broad approaches: Bayesian methods and end-to-end deep learning. Bayesian ToM models (Baker et al., 2017: Jara-Ettinge,, 2019; Shu et al., 2021) rely on structured probabilistic frameworks infer mental states from sparse observations of human behaviour. On the other hand, end- to-end models such as ToMnet (Rabinowitz et al., 2018: Shu et al., 2021; Sclar et al., 2022) have been trained di- rectly on ToM tasks, learning relationships between data patterns without explicit causal models of mental states (Sap et al., 2022; Zhi-Xuan et al., 2022; Ullman, 2023). More recently, neurosymbolic reasoning systems use the neural models for feature extraction, while also incorporating prob- abilistic models for structured reasoning (Wong al., 2023; Ying et al., 2024; 2023). They face challenges in dynamic and multimodal environments, where both physical and mental state reasoning are required. Different from prior studies, our work operates in more complex and dynamic multimodal ToM environments, where physical actions and mental state reasoning are intertwined.
Impact Statement
This study advances Bayesian ToM inference at scale and contributes new datasets representing diverse cultural con- texts. These datasets are based on ancient or fictional cul- tures, mitigating potential sensitivities related to contempo- rary societal issues. To ensure ethical integrity, the datasets have been carefully reviewed to minimize concerns regard- ing discrimination, bias, and faimess. They do not contain real individuals, eliminating risks to privacy or security.
We are committed to responsible research practices and encourage ongoing scrutiny of potential biases or unin- tended consequences. Our methods are designed to be fully reproducible, with detailed descriptions of datasets, experimental settings, and methodologies provided in this paper and our repository: https: //ghthhhtt..///// com/ chunhuizng scale bayesian cesssiinnplannne
5.2. Post-training LMs for downstream tasks
Post-training can project the LMs' pre-trained capabilities into downstream tasks such as dialogue generation (Ouyang et al., 2022), multilingual understanding (Yang et al., 2025b;a), prosocial alignment (Bai et al., 2022; Liu et al., 2024b), calibration (Fu et al., 2025), and multimodal tasks (OpenAI, 2023b; Liu et al., 2023; Jian et al., 2023; Liu et al., 2025a; Jian et al., 2024; Diao et al., 2024; Zhang et al., 2025a; Diao et al., 2025b;a; Liu et al., 2025b). Previous ap- proaches also use activation engineering/vector steering to adjust the output predictions of fine-tuned LMs, interpolat- ing the effects of fine-tuning with pre-trained knowledge for diverse downstream tasks (Liu et al., 2021; Mitchell et al., 2024; Liu et al., 2024a; Tan et al., 2024; Cao et al., 2024). Recently, LLMs are in embodied agents, allowing them to in- teract with and explore environments (Kim et al., 2024: Szot et al., 2024; Li et al., 2024). Our study differs by framing LMs as policy models in the context of Bayesian inverse inference, specifically to model human mental states. We address the limitations of existing ToM methods by scaling large policy models at test time using a likelihood redirec- tion strategy, reasoning more accurately in complex ToM scenarios. See A.p.A.1 for additional discussions.
Author Contributions
Student Authors Chunhui Zhang led the conceptual de- velopment, implementation, figure design, manuscript writ- ing, and preparation of the rebuttal. Zhongyu Ouyang con- ducted additional experiments, contributed figure design, refined the manuseript, and assisted with the rebuttal.
as action/policy models for
decision-makee
Senior Authors Shao- Lo supervised Chunhui Zhang during the internship phase of this project. Soroush Vosoughi supervised the overall publication process and contributed to idea generation in his capacity as Chunhui Zhang's PhD advisor. Sean Dae Houlihan, postdoctoral fellow working with Soroush Vosoughi, provided expertise in cognitive science. Kwonjoon Lee and Nakul Agarwal provided valuable discussions and feedback throughout the internship period.

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
References
Dennett, D. C. Précis of The Intentional Stance. Behavioral and Brain Sciences, 1988.
Aggarwal, J. K. and Ryoo, M. S. Human activity analysis: A review. ACM Computing Surveys, 2011.
Diao, X., Zhang, C., Wu, T., Cheng, M., Ouyang, Z., Wu, W., and Gui, J. Learning musical representations for music performance question answering. In Findings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J.. Conerly, T., El-Showk, S., Elhage, N., Suffield-Doddss Z., Heran- dez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Train- ing helpful and harmless assistant with reinforcement learning from human feedback. Anthropic, 2022.
Diao, X., Yang, T., Zhang, C., Wu, W. Cheng, M., and Gui, J. Learning sparsity for effective and efficient music per- formance question answering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025a.
Baker, C. L., Tenenbaum, J. B., and Saxe, R. R. Goal inference as inverse planning. In Proceedings of the annual meeting of the cognitive science society, 2007.
Diao, X., Zhang, C.,, Wu, W., Ouyang, Z., Qing, P., Cheng, M., Vosoughi, S., and Gui, Temporal working memory: Query Query-guided temporal segment refinement for enhanced multimodal understanding. In Findings of the 2025 An- nual Conference of the North American Chapter of the Association for Computational Linguistics, 2025b.
Baker, C. L., Saxe, R., and Tenenbaum, J. B. Action under- standing inverse planning. Cognition, 2009.
Baker, C. L., Jara-Ettinger, J., Saxe, R., and Tenenbaum, J. B. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing, Nature Human Be- haviour, 2017.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint 2777:2240: 2024.
Blukis, V., Paxton, C., Fox, D., Garg, A., and Artzi, Y. A persistent spatial semantic representation for high-level natural language instruction execution. In Conference on Robot Learning, 2022.
Fu, Y., Yu, Z., Li, J., Qian, J., Zhang. Y., Yuan, X., Shi, D., Yakunin, R., and Lin, Y. C. Amoeballm: Constructing any-shape large language models for efficient and instant deployment. Advances in Neural Information Processing Systems, 2025.
Caba Heilbron, F., Escorcia, V., Ghanem, B., and Car- los Niebles. J. Activitynet: A large-scale video bench- mark for human activity understanding, In JEEE Confer- ence on Computer Vision and Pattern Recognition, 2015.
Gao, Z. Niu, B., He, X., Xu, H., Liu, H., Liu, A., Hu, X., and Wen, L. Interpretable contrastive monte carlo tree search reasoning. arXiv preprint arXiv:2410.01707, 2024.
Cao, Y., Zhang, T., Cao, B., Yin, Z., Lin, L., Ma, F., and Chen, J. Personalized steering of large language models: Versatile steering vectors through bi-directional prefer- ence optimization. In The Thirty-eighth Annual Confer- ence on Neural Information Processing Systems, 2024.
Gao, Z., Chen, L., Zhou, J., and Dai, B. One-shot entropy minimization, 2025. URL https::/arxitsi httpss//axxvv.ooog/./
0as/2505.20285
Chen, Q. Qin, L., WANG, J., Zhou, J., and Che, W. Un- locking the capabilities of thought: reasoning boundary framework to quantify and optimize chain-of of-inuof-thought. In The Thirty-eighah Annual Conference on Neural Informa- tion Processing Systems, 2024.
Gopnik, A. and Wellman, H. M. Reconstructing construc- tivism: causal models, bayesian learning mechanisms, and the theory theory. Psychological bulletin, 2012.
Guo, D., Yang, D.,, Zhang. H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.1 250.. 2025.
Choi, W. and Savarese, S. Understanding collective activi- tiesof people from videos. IEEE Transactions Pattern Analysis and Machine Intelligence, 2013.
Han, C., Xu, J., Li, M., Fung, Y., Sun, C., Jiang, N., Ab- delzaher, T., and Ji, H. Word embeddings are steers for language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Chuang, Y.-S., Xie, Y., Luo, H., Kim, Y., Glass, J. R., and He, P. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth Inter- national Conference on Learning Representations, 2024.
10

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. DExperts: Decoding-time controlled text generation with experts and anti-experts. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing, 2021.
Jara-Ettinger, J. Theory of mind as inverse reinforcement learning. Current Opinion in Behavioral Sciences, 2019.
Liu, H.,, Li, C., Wu, Q. and Lee, J. Visual instruction tun- ing. In Thirty-sevetth Conference on Neural Information Processing Systems, 2023.
Jian, Y., Gao, C., and Vosoughi, S. Bootstrapping vision- language learning with decoupled language pre-training. In Proceedings of the 37th Conference on Neural Infor- mation Processing Systems, 2023. Spotlight.
Liu, T., Guo, S., Bianco, L., Calandriello, D., Berthet, Q.. Llinares, F., Hoffmann, J., Dixon, L., Valko, M., and Blondel, M. Decoding time realignment of language models. In Proceedings of the International Conference on Machine Learning. 2024a.
Jian, Y., Liu, T., Tao, Y. Zhang, C., Vosoughi, S., and Yang, H. Expedited training of visual conditioned language generation via redundancy reduction. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics, 2024. Oral Presentation.
Liu, Z., Dou, G., Tan, Z., Tian, Y., and Jiang, M. Towards safer large language models through machine unlearn- ing. In Findings of the Association for Computational Linguistics: ACL 2024, August 2024b.
Jin, C., Wu, Y., Cao, J., Xiang, J., Kuo, Y.-L.,, Hu, Z.,, Ullman, T., Torralba, A., Tenenbaum, J. B., and Shu, T. MMToM-QA: Multimodal theory of mind question answering. In Annual Meeting of the Association for Computational Linguistics, 2024.
Liu, Z., Dou, G., Jia, M., Tan, Z., Zeng, Q., Yuan, Y., and Jiang, M. Protecting privacy in multimodal large language models with mllmu-bench. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies, 2025a.
Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Plan- ning and acting in partially observable stochastic domains. Artificial intelligence, 1998.
Liu, Z., Dou, G., Yuan, X., Zhang, C., Tan, Z.. and Jiang, M. Modality neuron pruning for unlearning in multi- modal large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025b.
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E. P., Sanketi, P. R., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., and Finn, C. OpenVLA An open-source vision-language-aeee Conference on Robot Learning, 2024.
model. In Annual
Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang, L. The expres- sive power of neural networks: A view from the width. In Advances in neural information processing systems, 2017.
Kotha, S., Mitchell Springer, J., and Raghunathan, A. Un- derstanding catastrophic forgetting in language models via implicit inference. In International Conference on Learning Representations, 2024.
Ma, Z., Sansom, J., Peng, R., and Chai, J. Towards a holistic landscape of situated theory of mind in large language models. In Findings of the Association for Computational Linguistics: Conference on Empirical Methods in Natural Language Processing, 2023.
Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Mitchell, E., Rafailov, R., Sharma, A., Finn, C., and Man- ning, C. D. An emulator for fine-tuning large language models using small language models. In The Twelfih International Conference on Learning Representations, 2024.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 2017.
Mitchell, J., Springer, S., Goyal, K., Wen, T. K., Yue, X., Malladi, S., Neubig, G., and Raghunathan, A. Over- trained language models are harder to fine-tune. In Pro- ceedings of the ICLR 2025 Workshop on ICBINB, 2025.
Li, X., Liu, M., Zhang, H., Yu, C., Xu, J., Wu, H., Cheang, C., Jing, Y.. Zhang, W., Liu, H., Li, H., and Kong. T. Vision-language foundation models as effective robot imitators. In The Twelfth International Conference on Learning Representations, 2024.
OpenAI. Gpt-4 technical report. arXiv:2303.08777 2023a.

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
OpenAI. (isooV( system card, 2023b.
Shu, T., Bhandwaldar, A., Gan, Smith, K., Liu, S., Gut- freund, D., Spelke, E., Tenenbaum, J., and Ullman, T. Agent: A benchmark for core psychological reasoning. In International conference on machine learning, 2021.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang. C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.., Simens, M., Askell, A., Welinder, P., Christiano, F., Leike, J., and Lowe, R. Training language models follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Shum, M., Kleiman- Kleiman-Weiner, M., Littman, M. I.., and Tenen- baum, J. B. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of the AAAI conference on artificial intelligence, 2019.
Puig, X., Ra, K., Boben, M., Li, Wang. T., Fidler, S., and Torralba, A. Virtualhome: Simulating household activi- ties via programs. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Lim pruning and distillation in practice: The minitron approach, 2024.
Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. A., and Botvinick, M. Machine theory of mind. In International conference machine learning, 2018.
Su, Y., Lan, T., Wang, Y., Yogatama, D., Kong, L.,, and Collier, N. A contrastive framework for neural text gen- eration. In Advances in Neural Information Processing Systems, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et Language models are unsupervised multitask learners. OpenAl blog, 2019.
Sun, K., Xu, Y., Zha, H., Liu, Y., and Dong, X. L. Head- to-tail: How knowledgeable are large language models (LLMs)? A.K.A. will LLMs replace knowledge graphs? Proceedings of the 2024 Conference of the North Amer- ican Chapter of the Association for Computational Lin- guistics: Human Language Technologies, June 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty- seventh Conference on Neural Information Processing Systems, 2023.
Szot, A., Schwarzer, M., Agrawal, H., Mazoure, B., Metcalf, R., Talbott, W., Mackraz, N., Hjelm, R. D., and Toshev, A. T. Large language models generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2024.
Raghu, M., Poole, B., Kleinberg. J., Ganguli, S., and Sohl- Dickstein, On the expressive power of deep neural net- works. In International Conference on Machine Learning, 2017.
Sap, M., Le Bras, R., Fried, D., and Choi, Y. Neural theory- of-mind? on the limits of social intelligence in large LMs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022.
Tan, D. C. H., Chanin, D., Lynch, A., Paige, B., Kanoulas, D., Garriga-Alonoo A., and Kirk, R. Analysing the generalisation and reliability of steering vectors. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Sclar, M., Neubig, G., and Bisk, Y. Symmetric machine theory of mind. In International Conference on Machine Learning, 2022.
Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.,, Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models a practical size. arXiv preprint arXiv:2408.00 001118 2024.
Selar, M., Kumar, S., West, P., Suhr, A., Choi, Y., and Tsvetkov, Y. Minding language models' (lack of) theory of mind: A sang--nd--la multi culti-character belief tracker. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2023.
Telgarsky, M. Benefits of depth in neural networks. In Conference on learning theory, 2016
Shi, H., Ye, S., Fang, X., Jin, C., Isik, L., Kuo, Y.-L., and Shu, T. Muma-tom: Multi-modal multi-agent theory of mind. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv:2307.09980 2023.
Shu, T., Xie, D., Rothrock, B., Todorovic, S., and Zhu, S.- C. Joint inference of groups, events and human roles in aerial videos. In IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Ullman, Large language models fail on trivial al- terations to tfeory-of-mind tasks. arXiv preprint arXiv:2302.08399 2023.
12

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Wilf, A., Lee, S., Liang, P. P., and Moreney, L.-P. Think twice: Perspective-taking improves large language mod- els' theory-of-mind capabilities. In Proceedings of the Anmual Meeting of the Association for Computational Linguistics, 2024.
Yuan, X., Zhang, C., Liu, Z., Shi, D., Vosoughi, S., and Lee, W. Superficial self-improved reasoners benefit from model merging, arXiv preprint arXiv:2503.021003 2025.
Zhang, C., Jian, Y., Ouyang, Z., and Vosoughi, S. Working memory identifies reasoning limits in language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
Wong, L.., Grand, G., Lew, A. K., Goodman, N. D., Mans- inghka, V. K., Andreas, J., and Tenenbaum, J. B. From word models to world models: Translating from natural language to the probabilistic language of thought. arXiv preprint arXiv:2306. /2672, 2023.
Zhang, C., Jian, Y., Quyang, Z.,, and Vosoughi, S. Pretrained image-text models are secretly video captioners. In Pro- ceedings of the 2025 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2025a. Oral Presentation.
Wu, B., Yu, S., Chen, Z., Tenenbaum, J. B., and Gan, C. STAR: A benchmark for situated reasoning in real-world videos. In Thirty-fitth Conference on Neural Informa- fion Processing Systems Datasets and Benchmarks Track, 2021.
Zhang, C., Wang. S., Ouyang, Z., Yuan, X., and Vosoughi. S. Growing through experience: Scaling episodic grounding in language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025b.
Yang, I,, Ma, W., Zhang, C., and Vosoughi, S. Is it navajo? accurate language detection in endangered athabaskan languages. In Proceedings of the 2025 Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2025a. Oral Presentation.
Zhang, H. Li, X., and Bing, L. Video-LLaMA: An instruction tuned audio visual language model for video understanding. In Proceedings of the Conference on Em- pirical Methods in Natural Language Processing: System Demonstrations, 2023.
Yang, I. Zhang, C., Wang. Y., Ouyang, Z., and Vosoughi, S. Visibility as survival: Generalizing nip for native alaskan language identification. In Findings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025b.
Zheng, J., Cai, X., Qiu, S., and Ma, Q. Spurious forgetting in continual learning of language models. In International Conference on Learning Representations, 2025.
Ye. Z. Xia, K., Fu, Y.. Dong, X., Hong, J., Yuan, X., Diao, S., Kautz, J., Molchanov, P., and Lin, Y. C. Long- mamba: Enhancing mamba's ' long-contes capabilities via training- frre receptive field enlargement. In Interna- tional Conference on Learning Representations, 2025.
Zhi- Zhi-Xuan, T., Gothoskar, N., Pollok, F., Gutfreund, D., Tenenbaum, J. B., and Mansinghka, V. K. Solving the baby intuitions benchmark with hierarchically bayesian theory of mind. In Social Intelligence in Humans and Robots, RSS Workshop, 2022.
Ying, L., Collins, K. M., Wei, M., Zhang, C. E., Zhi-Xuan, T., Weller, A., Tenenbaum, J. B., and Wong, L. The neuro-symholic inverse planning engine (nipe): Modeling probabilistic social inferences from linguistic inputs. In ICML Workshop on Theory of Mind in Communicating Agents, 2023.
Zhou, Z., Liu, J., Dong, 7.,, Liu, J., Yang, C., Ouyang, W., and Qiao, Y. Emulated disalignment: Safety alignment for large language models may backfire! In Proceed- ings of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024.
Ying, L., Zhi-Xuan, T., Wong, L., Mansinghka, V., and Tenenbaum, J. Grounding language about belief in a bayesian theory-of-mind. In Proceedings of the Annual Meeting of the Cognitive Science Society, 2024.
Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H. Yao. Z.. Zhang, X., Li, H., Li, C., Zhang, Z., Bai, Y., Liu, Y., Xin, A., Yun, K., Gong, L., Lin, N., Chen, J., Wu, Z., Qi, Y., Li, W., Guan, Y., Zeng, K., Qi, J., Jin, H., Liu, J. Gu, Y., Yao, Y., Ding, N., Hou, I.., Liu, Z.,, Bin, X., Tang, J., and Li, J. KoLA: Carefully benchmarking world knowledge of large language models. In The Twelfth International Conference on Learning Representations, 2024.
13

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
A. Comparison of Methodologies for ToM Inference
tasks into the domain of social cognitive reasoning. Our framework uses language models approximate policy behaviours for probability estimation in embodied simula- tors, based on the cognitive science-inspire Bayesian ToM framework. Unlike previous work focusing on text-based tasks such as sentiment or factuality control, our method addresses the unique challenges of ToM tasks, which require complex reasoning and the integration of world knowledge. These tasks involve multimodal scenarios that require un- derstanding of agents' beliefs, goals and actions domain distinct from the text generation problems addressed in pre- vious studies.
Table 6. Attributes of each method for ToM task.
methool
waalahility
structured reasoning world knowlsdge multmodility
Bayesian Teld models end to and ToM models
6ab.6 provides comparative analysis of various method- ologies for ToM inference, supplementing the discussion in the introduction 1) Our proposed approach differs signifi- cantly from the underlying philosophies of Bayesian ToM models and end-to-end models. While Bayesian models emphasize structured reasoning guided by principles from cognitive science, they often lack scalability and struggle to handle multimodal inputs. In contrast, end-to-end models in- corporate extensive world knowledge but lack the structured reasoning capabilities essential for accurate ToM inference.
B. Data Flow and Processing in Scalable Bayesian ToM Inference
B.1. Overall Data Flow
For a detailed depiction of the data flow in our method, refer to Fig.5. The symbolic representation tools first con- vert video and textual descriptions into structured symbolic inputs, which are then processed by the Bayesian infer- ence framework. This framework leverages large LM as scaled policy model, dynamically controlled by task-specific priors provided by a post-trained small LM, enabling accu- rate estimation of action likelihoods in dynamic scenarios.
Our method integrates these attributes: scalability (e.g., up to 405B), structured reasoning, world knowledge, and the ability process multimodal inputs. Furthermore, our method demonstrates superior scalability, leveraging the stronger reasoning capabilities of large LMs at test time without the need for extensive post-training on large models. This allows our approach to efficiently handle complex and dynamic ToM scenarios.
B.2. Data Preprocessing: Unified Symbolic Representations
A.1. Our Theoretical Rationales in Scaled ToM Inference and Related Work
To enable Bayesian ToM inference at scale, following es- tablished methods mentioned in MMTOM (Jin et al., 2024; Blukis et al., 2022), multimodal data (video and textual descriptions) are transformed into structured symbolic rep- resentations. This process involves three key components: visual perception, text parsing, and information fusion. Together, these components provide unified representation of states, actions, and hypotheses required for ToM tasks.
Our approach is based on a high-level principle derived from Theorem and its proof, which implies that smaller models can approximate the scaled gradient of the loss func- tion for larger models. This mechanism bypasses direct parameter updates in the larger model, capturing the pri- mary adjustments needed for fine-tuning while exploiting the innate generalisation capacity of the larger model. By re- lying on the approximate knowledge provided by the smaller model, our framework reduces computational overhead and improves scalability.
This principle is related with previous studies that have explored reweighting mechanisms for various applications (where not necessarily the same as our perspective of scal- ing or embodied policy model), including avoiding toxicity in text generation (Liu et al., 2021), 1), mitigating harmful out- puts in aligned models (Zhou et al., 2024), adjusting code generation (Mitchell et al., 2024), controlling sentiment in text (Han et al., 2024), and reducing hallucination or degen- eration in neural text (Chuang et al., 2024; Su et al., 2022). These works demonstrate how reweighting can approxi- mate the behaviour of large language models, mimicking direct fine-tuning in specific contexts. In contrast, our scaled ToM inference extends this principle beyond text generation
Visual Perception. The visual perception module is de- signed to process video frames and extract symbolic rep- resentations of the environment. For each frame. scene graph is generated to capture the spatial and relational prop- erties of objects and agents with the scene graph gener- atoriBlukis et al., 2022). Following established methods in MMToM (Jin al., 2024), voxel maps and 3D bound- ing boxes utilized to infer object positions, contain- ment relationships, and human poses. For instance, objects such as pear and basket are represented by predicates like In (pear, basket). These predicates effectively sum- marize the physical state of the environment, serving as critical inputs for subsequent reasoning steps.
14

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Symbolic Representation
Bayesian Inverse Inference
Latent Behavior Change
Scene
Video
(ST,b",
a
Prob Prob b
Base Small LM
()".b.g)
(Extreme) Large Language Model
control
Hypothesis
Goal:
Belief: In( (
Pear
in
the
basket
Desc.
Hypothesis b
Post-trained Small LM
Goal:
(S ,,,
(ST,b".g) bb....
Hypo a/b
Belief: NotIn )
Figure The data flow in our scalable Bayesian ToM inference framework. Vidco scenes and their corresponding descriptions first processed by multimodal symbolic representation tools (Jin et al., 2024), generating structured symbolic inputs (states, belief's, goals), These symbolic representations are then integrated into the Bayesian inference process, where large language model (LM) operates as scaled policy model to estimate the likelihood of an agent's actions in dynamic environments. The right panel demonstrates the latent hehavioral changes introduced by the post-trained small L,M, which provides task-specific priors to guide larger LM a control mechanism.
Text Parsing. To extract symbolic representations from textual descriptions, one LLM (c.g., GPT-4) processes the text into three distinct components: (i) the initial state of the environment, (ii) human actions, and (iii) the question. Each component is translated into symbolic predicates. For example:
question provides two hypotheses--oal and belief-that guide the reasoning task.
C. Theoretical Rationale
Theorem 1. Let If be pretrained base model, Tf and TV be smaller tunable models where Tf is fine-tuned on the target task, and π be the directly tuned base model. Suppose the output adjustment As(X,] = 8 (Xtl=<t)= - 8gN (x)((e)) approximates the scaled neg- ative gradient of the cross-entropy loss for ouppts,, i.e.. As = -IVV,Ce((s,, v), where is the learning rate. Then, the proxy-tuned model म, defined by s (X,((ee)) S (xe((e) + As(X,] approximates the directly tuned base model π². The KL divergence between their output distributions has this relation:
The state is represented as predicates like In (pear, basket.).
The action is represented commands such as walk
towards kitchen.
The question is decomposed into two hypotheses, each comprising goal (e.g., pear) and belief (e.g., In (pear, basket) or its negation, -In (pear, basket)).
1)2 Amax V /.ecc( 2) O(1²),
D
2
This symbolic parsing ensures compatibility with the struc- tured reasoning framework.
where Amax is the maximum eigenvalue of the Hessian of the cross-entropy loss for the outputs.
Fusion. The fusion module integrates symbolic informa- tion from video and text into unified representation. First, predicates extracted from video inputs (e.g., spatial relation- ships) are aligned with those parsed from text to form the initial state. Next, human actions detected from the video matched with text-based actions, and the video sequence segmented into discrete time steps corresponding these actions. Starting from the initial state, the symbolic repre- sentation of the environment is updated at each time step based on newly detected predicates. This process results in a sequence of symbolic states and actions, which serve as the input for Bayesian inference. Additionally, the parsed
Proof. When the learning rate is small, and the cross- entropy loss Lee is smooth and twice differentiable with respect to the outputs s, then the output adjustment As approximates the scaled negative gradient of the loss as:
As - (CF(
(10)
The outputs of the directly tuned base model x² after fine- tuning are updated using gradient descent:
" 8n" = - (.C.nVVLLen(s... v)+ u) (.
(11)
15

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
where Hg is the Hessian of Lce with respect the outputs. The outputs of the proxy-ture model # are:
This inherent flexibility allows the large model harness its pre-trained potential, activated by the weak-to-strong control mechanism, to effectively adapt to the current ToM (12) task. Consequently, our method achieves stable advanced performance even novel scenarios where the small LM provides only a coarse approximation the gradient. This significantly reduces the reliance on strict fine-tuning and maximizes computational efficiency, ensuring the approach is both scalable and practical for the physical VirtualHome environment.
8x S + As.
When As Rs &CE (8,,, 1),, we have:
Rd 8 V
(13)
The difference in outputs between the directly tuned model and the proxy-tuned model is:
(14)
D. Experimental Details
Then we consider their expressions:
D.I. Belief and Goal Inference Types and Their Characteristics to LMs
2 2 H. H.(VoLCE(SSAA) O(n))
c =
(15)
MMToM apartment scenario questions are split into seven types, assessing ToM reasoning (Jin et al., 2024): Belief Inference includes 50% of questions on True Belief (Type 1.1), False Belief (Type 1.2), and Long-Term Belief Tracking (Type 1.3). Goal Inference covers the remaining 50% on True Belief (Type 2.1), False Belief (Type 2.2), Updated Belief (Type 2.3), and Future Actions (Type 2.4).
The KL divergence between the output distributions of π and ã constrained using the properties of the softmax function and the Lipschitz continuity of the KL divergence:
(P=-1111PP)) <
( 1
(16)
Short-term Belief Inference relies heavily on world knowl- edge, making it more responsive enhancements from large LMs" pretrained capabilities. In contrast, long-term reasoning-boot for Belief and Goal Inference- focruns on the dynamic nature of the environment and benefits from post-training specifically aligned to ToM scenarios.
Using the norm of ex:
||> 2 T 4 CCCCV., v))
(17)
The Hessian's norm is constrained by its maximum eigen- value:
D.2. Post-training configurations
(V.(VlCc S Amax VcCce(s
(18)
Tab. 7 summarizes the LoRA post-training configurations applied to Llama2, Llama3, and Llama3.1 models during policy model training. We carefully adjust a, rank, and other hyperparameters to optimize performance across different (19) model sizes. Notably, following prior engineering studies, higher and rank are used for smaller models (7B and 8B), while reduced values employed for the larger 70B model to ensure efficient adaptation without overfitting.
which gives:
||ea112 S 1 4
Finally, the KL divergence is:
(K |Pa) S 122 Amax V V.LCE(88e, 0 (20)
2
D.2.1. TUNNNGNING PROCESS AND RESOURCES
The fine-tuning process for smaller models (e.g., Llama3. 1- 8B) was conducted using a single NVIDIA H100 GPU, leveraging BF16 mode optimize memory usage and main- tain GPU memory consumption under 60GB. This config- uration enabled efficient training of policy models tailored for Theory of Mind (ToM) tasks. The fine-tuning process was executed with the following parameters:
For theoretical implications for practical applicability, this analysis demonstrates that the weak-to-strong control mech- anism relies on the learned As to approximate the scaled gradient V CVVC(EEss y) with higher-order terms con- tributing the residual error. Importantly, our method does not require the small LM ( to strictly approximate the exact gradient of the cross-entropy loss for the large model. Instead, the large model (x leverages its intrinsic capacity for generalization and adaptation, based only on the approx- imate adjustment As learned by the small L.M.
Batch size: 16 (achieved via a per-device batch size of 4 and gradient accumulation steps of 4),
Learning rate: 5 x 10-5
16

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Number of epochs: 3.
ularly for smaller models and larger datasets. However, for larger models, LoRA provides an effective alternative with near-parity in performance and significantly reduced compu- tational overhead. Furthermore, our weak-to-strong control mechanism demonstrates stability to fine-tuning methods, enabling scalable ToM-specific behavior elicitation with high aceuracy in larger models.
Under this setup, the fine-tuning process required approxi- mately 8 hours to converge.
D.2.2. DATASET SIZE
The training pool size N for post-training was set to 20,000 data points, sourced from the MMToM dataset's training split and our released data sampled from an embodied simu- lator. For tasks involving transfer to new themes, the training dataset size remained consistent at 20,000 data points, en- suring a fair and uniform setup across different experiments.
E.2. Impact of Pre- Pre-Training Quality on MMToM Tasks
The differences in performance between the Llama2, Llama3 and Llama3. 1 models provide insight into the role of pre-training quality, especially at large model scales. Based the experimental results in Tab.2 and Tab.3, the influence of pre-training quality diminishes primarily due to ceiling effect, but this is only observed when comparing models within the same scale, such as the 70B parameter range. However, when comparing smaller models to larger ones, the effect of pre-training is more pronounced. For example, moving from Llama2 7B Llama2 70B after ToM- foM-specific post-training leads 6% improvement in belief inference accuracy (from 76.33% to 82.33%) and a 2% improvement in goal inference accuracy (from 70% to 72%), highlighting the role of scaling in encoding richer representations.
E. Additional Experiments
E.1. Comparison of Fine-Tuning Methods on MMToM Tasks
To evaluate the relative performance of full fine-tuning (FFT) and LoRA fine-tuning, conducted experiments on two smaller models, GPT2-large (Radford et al., 2019) (774M parameters) and Gemma-2B (2B parameters) (Team et al., 2024). Each model was fine-tuned using datasets of 20,000 and 8,000 datapoints, over two epochs, on NVIDIA A100 80GB GPUs. The results are summarised in Tab.8.
When examining why pre-training becomes less effective at larger scales, such as comparing Llama2-70B (pre-trained with 2.2 trillion tokens) to Llama3. 1-7BB (pre-trained with 15 trillion tokens), the results suggest that larger pre-training corpora improve performance primarily for tasks that rely heavily on world knowledge: Tasks involving belief infer- ence, which rely on short-term reasoning and general world knowledge, show significant improvements due to improved representations learned during pre- tr-itrainin.. For example, L achieves a 3.67% improvement in belief inference accuracy over Llama2 (from 83.00% 85.67%) These tasks benefit from richer pre-training datasets that refine the model's understanding of common human behaviours and object interactions.
The results show several important trends. First, when suf- ficient training data is available (e.g.. 20,000 data points), full fine-tuning consistently outperforms LoRA, with accu- racy gains of 0.9-1.2 percentage points. This suggests that full training is better at exploiting richer data, especially for smaller models. Second, the performance gap between FFT and LoRA narrows for larger models. For example, Gemma-2B shows minimal differences between FFT and LoRA (0.3 percentage points on 20,000 data points), sug- gesting that larger models are stable to LoRA's parameter efficiency constraints. Finally, the influence of dataset size is evident: while FFT shows greater improvements over LoRA on smaller datasets, LoRA maintains competitive performance in resource-constrained scenarios, especially for larger models. Tab.9 further demonstrates the stable performance of weak-to-strong control when transferring ToM-specific fine-tuning knowledge from a smaller model (Minitron-4B-Wiine Widtth) to a larger model (7)8)) The difference in accuracy between FFT and LoRA is only 0.15 percentage points when weak-to-strong control is ap- plied, indicating that the mechanism is highly effective at bridging the gap between fine-tuning methods. Importantly, this highlights the ability of the proposed method to scale ToM-specific behaviors efficiently, leveraging both compu- tationally intensive FFT and parameter-efficies LoRA.
In contrast, goal inference tasks that rely on long-term rea- soning, including integrating temporal observations and dynamically updating beliefs, show smaller gains from larger pre-training corpora. For example, Llama3.1 im- proves goal inference accuracy by only 1.67% over Llama2 (from 72.33% to 74.00%). Such tasks are more dependent on the fine-tuning stage and the use of task-specific rea- soning frameworks, such as weak-to-strong control. These results suggest that for complex reasoning tasks, the primary performance bottleneck shifts from pre-training quality to the reasoning strategies employed during fine-tuning.
In summary, pre-training quality has a significant impact on smaller models and tasks that rely heavily on world knowledge, such as belief inference. However, as models scale up to 70B parameters, the influence of pre-training
Overall, these experiments highlight a tra between computational efficiency and performance gains. Full fine- tuning achieves modest but consistent improvements, partic-
17

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
diminishes due ceiling effects, and logical reasoning tasks such as goal inference rely more on task-specific adaptations during fine-tuning.
justment ratio to refine its likelihood estimation dynamically, * The overall policy distribution is computed as π xx, where Tf is the policy from the larger model. This mechanism allows the larger model retain its broad reasoning and world knowledge, ensuring its capacity for generalization while aligning with ToM-specifie task structures.
E.3. How Consistent is Theory of Mind Across Different Phrasings?
As shown in Tab.4, the "Alf" column across different themes (e.g. Apartment, Andersen Fairy Tales, etc.), there is no- ticeable performance variance even within models of the same scale. To quantify this, we measured the range of variance for three configurations: 70B-zero-shot, 70B-post- trained and 8B 9- 70B: (1) For 70B-zero -oB-zero-hhot, performance ranged from 66.62 to 70.52 across themes, yielding a vari- ance range of 3.90; (2) For -os-tsst- the variance range of post- trained LMs is 3.47, with performance ranging from 71. 86% and 75.33%; (3) For our solution 8B 9- 70B, the weak-to-strong control mechanism further stabilised the performance, reaching only the smallest variance range of 1.62, with scores between 77.76% and 79.38%.
To validate this framework, we compared the performance of the 8B + 70B model to the 70B-post-trained model across five unseen themes, including Andersen Fairy Tales, Ancient Egyptian, and Outer Space. As shown in Tab. 10, the weak-to-strong mechanism achieved consistent improve- ments across all ToM tasks, demonstrating its ability to preserve and transfer the larger model's ' general reasoning capabilities while aligning with ToM-specific requirements. These results, combined with theoretical insights from Sec- tion C, demonstrate that the weak-to-strong framework ef- fectively utilizes the smaller model as a task-specifi, lens to guide the larger model's 's predictions. This collaborative dynamic ensures alignment with ToM-specific task require- ments while preserving general reasoning capabilities.
These results suggest that specific topics have different ef- fects on ToM skills, but our solution demonstrates relative stability to distributional changes caused by topic shifts. For example, 70B-zero-shot achieves its highest performance up to 70.52% and its lowest up to 66.62%, highlighting the model's pronounced sensitivity to thematic variations in reasoning trajectories without adaptation. In contrast, our proposed solution, 8B 9- 70B, significantly reduces this gap. demonstrating the effectiveness of the weak-to-strong con- trol mechanism in adjusting the ToM behaviour of the larger model while preserving the framework's s general reasoning capacity across diverse and -agnosicc contexts.
E.5. Theory-of-Mind Case Study: Agent James in Apartment Interaction
Fig.6 provides detailed visual and language- baase descrip- tion of the test case described in experiment $4.7 of the experiment, where the likelihood estimation behaviour of different LMs is discussed across varying concept levels.
E.6. ToM Transfer Effect on unseen scenarios
Tab. 1l supplements the results in experiment $4.5, provid- ing a detailed comparison between the baselines and our scalable solution across belief inference and goal inference subtasks various unseen ToM scenarios. Our experi- mental observations are consistent with those outlined in $4.5: (i) The increased capacity of our scalable solution significantly improves the transferability of ToM reasoning across dynamic and previously unseen environments. (ii) Our approach demonstrates strong potential for downsiz- ing small LMs as controllers, as they successfully capture the post-trained behaviours and exhibit stable performance in guiding larger models. (iii) Notably, our method can approximate-ande in some cases outperform- the results achieved by directly post-training large-scale LMs (such as the 70B model). These findings underscore the flexibility and scalability of our approach for handling practical ToM tasks in diverse, complex environments.
E.4. On the Role of the Weak-to-Strong Framework
The weak-to-strong framework presented in this paper fo- cuses on aligning the larger model's 's distribution with ToM- specific beliefs and task structures while preserving its gen- eral reasoning capabilities, rather than primarily relying on the smaller model's reasoning abilities. This design enables efficient transfer of ToM-specific task structures without compromising the broader capabilities of the larger model.
The smaller model (e.g., 4B or 8B parameters) undergoes ToM-specific post-training to encode task-relevant priors, such as belief states and potential goals, without requiring advanced independent reasoning capabilities. During infer- ence, the smaller model functions as an assistive scaffold, conditioning the larger model's 's likelihood estimation in Bayesian framework. This role is formalized through the adjustment ratio: _ where o is the post-trained smaller model's iss task-specific policy, and yr is the naive pre-trained smaller model's policy.
E.7. Thematic Scenario Data for ToM Task Transfer
As described in $E.6, five new thematic scenarios are used for evaluation: Andersen Fairy Tales, Ancient Egyptian, Wild West, Outer Space, and Medieval Castle. These en-
The larger model (c.g., 70B parameters) integrates this ad-
18

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
vironments are not seen during the post-training phase of our method and are different from the original apartment setting.
to-Strong controlled models and their naively post-trained counterparts. Across all model scales and architectures, the absence of Weak-to-Strong guidance results in consistent and often substantial drops in generalization performance. These results underscore that Weak-to-Strong control is critical for maximizing the large LM's ability to leverage its pretrained world and mental-state knowledge, enabling stronger ToM reasoning.
The transfer to these scenarios demonstrates the general- isability of our solution to dynamically adapt to different domains, with each thematic environment presenting unique challenges and contextual shifts from the apartment scenario. Fig.7 provides a visual summary of these key differences, statistically extracted and mapped to illustrate the transfor- mation of concept and environment across these themes. These distinctions are used to evaluate ToM task transfer across different dynamic environments.
F. Practicality under Real-Time and Resource Constraints
A key concern in deploying advanced ToM reasoning sys- tems is their feasibility under stringent computational and latency requirements. Our approach addresses this by lever- aging a small, post-trained LM (e.g.. 4B or 8B) to provide dynamic, parallel guidance to a large, pretrained LM (e.g.. 70B or 405B) during inference.
E.8. Generalization to Diverse Real-world Social Interactions
In Tab. we expand our evaluation using MuMA-ToM (Shi et al., 2025), a benchmark explicitly designed for nuanced social interaction, including: (1) Belief inference: Under- standing environmental dynamics. (2) Social Goal infer- ence: Interpreting subtle social objectives. (3) Belief-of- Goal inference: Attributing complex mental states. Results in Table A show that our method performs competitively with the state-of-the-art GPT-4o-based LIMP (Shi et al., 2025) and outperforms all the other baselines. Note that this achieved by using open-source models, avoiding the expensive GPT-4o API cost required by LIMP (Shi et al., 2025). Our weak-to-strong control leverages large pretrained LMs, effectively adapting to real-worl world social reasoning without compromising generalization.
Both LMs can be deployed simultaneously on standard NVIDIA HI00 GPUs (80GB, BF16 precision), with the small LM introducing negligible computational overhead. For instance, in the 7B+70B configuration, inference over 600 tasks requires approximately 14-15.5 minutes (1.4-1.55 seconds per question), matching the runtime of an unguided 70B model. This efficiency is attributed to the lightweight nature of the small LM's 's computations relative to the dom- inant cost of the large LM, and to efficient parallelization with minimal synchronization (passing only compact likeli- hood tensors).
Moreover, both the small and large LMs only perform like- lihood estimation (i.e., prefilling, typically up to 1024 to- kens), which is well supported by contemporary accelera- tion frameworks such as NVIDIA Dynamo, vLLM (Kwon et al., 2023). These optimizations make our Bayesian ToM planner well-suited real-time and resource-constrained environments.
E.9. Comparison with Parameter-Efficiee Fine- Fine-Tuning
The proposed weak-to-strone tt-sonon control is fully orthogonal and complementary to PEFT techniques, i.e., we can com- bine our method with any PEFT technique. In fact, our small LMs are trained by LoRA, as described in Li9i-tight. Tab. 13 further confirms the consistent effectiveness of our method, regardless of the PEFT choice for small LM.
Finally. our method circumvents the prohibitive costs of fine-tuning very large LMs. Directly fine-tuning a 405B model would typically require upwards of 50-64 H100 GPUs, which beyond the reach of most institutions. In contrast, our approach enables effective adaptation by fine- tuning only small LM (e.g., 8B), reducing hardware re- quirements to a single H100 GPU, while still achieving superior performance through guided inference. This de- sign makes our approach both scalable and accessible for practical deployment.
Directly applying PEFT to large pretrained LMs performs worse than our method. As discussed above, our method avoids fine-tuning large LMs and thus preserves their pretrained mental/world knowledge (Kotha et al., 2024; Mitchell et al., 2025; Zheng et al., 2025), essential for gen- eralization in multimodal ToM tasks.
E. The Necessity of Weak-to-Strong Control
To elucidate the contribution of our Weak-to- toak--ngSttong mecha- nism, we conduct two ablation studies: (1) replacing W2$ with naive post-training, and (2) simply adding a small LM (8B) a large LM without structured control.
Tab. 14 presents a comparative ablation between our Weak-
19

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
Table 7. LoRA configuration settings Llama2, Llama3, and Llama3.1 during post- post-training for policy models.
configs
7B
70B
bias
none
none
fan-in fan-out
false
false
false
inference made
true
LoRA initialization
true
32
dropout
0.05
0.05
rank
target modules
[q (q-proj, v-proj]
task type
causal 1m
Table 8. Comparison of full fine-tuning (FFT) and LoRA fine-tuning for GPT2-large and Gemma-2B across different MMToM data sizes.
Fine-tuning Method
Data Size
Model Size
Accuracy (%)
GPT2-large
FFT
20,000
774M
63.4
GPT2-large
LoRA
20,000
774M
62.4
GPT2 large
FFT
8,000
774M
62.8
GPT2-large
LoRA
8,000
774M
62.1
Gemma-2B
FFT
20,000
2B
68.8
Gemma-2B
LoRA
20,000
2B
68.5
Gemma-2B
FFT
8,000
2B
67.5
Gemma-2B
LoRA
8,000
2B
67.3
Table Comparison weak-to-strong control for Llama3.1-Minitro, 1-MiniMoon--BBB Width and Llama3.1-705 using different fine-tuning methods
on the smaller model.
Fine-tuning Method
Data Size
Model Size
Accuracy (%)
Llama3. I- I-Minitron-1B--i -Batttonn---Wittt Width
FFT LoRA
20,000
4B
77.00
Llama3.1 I Miiiitoon--B-V
20,000
4B
76.90
Reak-to-sttrogg control results:
Width 9- Llama3. 1-70B
FFT-trained 4B
20,000
70B
78.67
4B Width 9- Llama3. 1-70B
LoRA-trained 4B
20,000
70B
78.52
Table 10. Performance of the 8B 70B LMs unseen themes compared 70B--ost--rriined/-eer--- LMs across all ToM tasks.
Unseen Theme
Scale
1.2
1.3
Avg.
2.1
2.2
2.4
Avg.
Andersen Fairy Tales
70B-2ero-shot
88.00
73.00
90.00
83.67
70.67
80.00
25.33
66.67
60.67
70.52
70B-post-train
90.00
71.00
93.00
84.67
73.33
61.33
61.33
69.33
66.33
74.19
8B 9- 70B
92.00
71.00
85.00
82.67
82.67
76.00
68.00
77.33
76.00
78.86
Ancient Egyptian
70B-2ero-shot
89.00
71.00
91.00
83.67
74.67
74.67
25.33
60.00
58.67
69.38
70B-post-train
89.00
69.00
96.00
84.67
72.00
76.00
61.33
64.00
68.33
75.33
8B 9- 70B
90.00
73.00
88.00
83.67
69.33
76.00
73.33
74.67
73.33
77.76
Outer Space
70B-zero-shot
88.00
72.00
92.00
84.00
72.00
64.00
25.33
70.67
58.00
69,38
70B post- train
91.00
68.00
90.00
83.00
69.33
65.33
61.33
68.00
66.00
75.33
8B 9- 70B
90.00
70.00
92.00
84.00
73.33
81.33
66.67
78.67
75.00
77.76
20

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning
ToM exemplar
What's inside the apartment:
bedroom, there is coffee table with plate on The bathroom houses cabinet, which currently empty. The living is furnished with cabinet, coffee table, sofa, and desk. The cabinet is filled with two apples, condiment bottle, three wine glasses, two water glasses, a cupcake, two bags of chips, a remote control, and bottle of wine. Both water glass and a wine glass are placed on the coffee table. The kitchen is equipped with a fridge, an oven, kitchen table, and microwave. Inside the oven contains salmon, Meanwhile, the microwave houses salmon and two cupcakes.
The apartment consists a bedroom, bathroom, living room, and kitchen,
fridge, there
two apples. Actions taken by James:
James is in the kitchen. He strides towards the stove, opens it, and then shuts it. He then opens the fridge, closes it, opens the microwave, and closes It as well. Finally, he walks towards the living room and approaches the cabinet.
State Modelling:
(a) James has been trying to get a bottle of wine.
(b) James has been trying to get an apple. X
St: apples in fridge, wine
by: wine in the cabinet
St: obtain bottle of wine
aj: open fridge, walk cabinet
Figure 6. Theory-of Mind scenario used in the main experiments 84.7, involving an agent (James) interacting with objects in an apartment.
Table 11. Detailed transfer performance of the Bayesian method with different scaling strategics (zcro-shot, direct post-traie bost-training. and our
weak-to-strong control) from the original apartment scenario various unseen environments. All models are based Llama3.
Belief Inference
Goal Inference
Theme
Scale
1.1
1.2
1.3
Avg.
2.1
2.2
2.3
2.4
Avg.
All
70B-zeroshot
88.00
73.00
90.00
83.67
70.67
80.00
25.33
66.67
60.67
70.52
70B toB-post-train
90.00
71.00
93.00
84.67
73.33
61.33
61.33
69.33
66.33
74.19
Andersen fairy tales
8B 9+ 70B
92.00
71.00
85.00
82.67
82.67
76.00
68.00
77.33
76.00
78.86
4B-width 9- 70B
90.00
73.00
89.00
84.00
80.00
81.33
76.00
64.00
75.33
79.05
4B-depth 9- 70B
91.00
74.00
90.00
85.00
74.67
73.33
64.00
73.33
71.33
77.19
70B-zeroshot
89.00
71.00
91.00
83.67
74.67
74.67
25.33
60.00
58.67
69.38
70B-post-train
89.00
69.00
96.00
84.67
72.00
76.00
61.33
64.00
68.33
75.33
ancient Egyptian
8B + 70B
90.00
73.00
88.00
83.67
69.33
76.00
73.33
74.67
73.33
77.76
4B-width 70B
90.00
69.00
90.00
83.00
70.67
80.00
85.33
69.33
76.33
79.19
4B-depth 9- 70B
91.00
69.00
96.00
85.33
76.00
68.00
69.33
76.00
72.33
77.90
70B-zeroshir 70B-post 70B-post-train
88.00
72.00
92.00
84.00
72.00
64.00
25.33
70.67
58.00
69.38
91.00
68.00
90.00
83.00
69.33
65.33
61.33
68,00
66.00
75.33
outer space
8B 9- 70B
90.00
70.00
92.00
84.00
73.33
81.33
66.67
78.67
75.00
77.76
4B-width 9- 70B
90.00
70.00
88.00
82.67
73.33
76,00
80.00
72.00
75.33
79.19
4B-depth 70B
90.00
69.00
86.00
81.67
70.67
73.33
68.00
72.00
71.00
77.90
70B-zeroshot
88.00
72.00
92.00
84.00
72.00
64.00
25.33
70.67
58.00
69.14
70B-post-train
91.00
68.00
90.00
83.00
69.33
65.33
61.33
68.00
66.00
73.29
wild west
8B 4+ 70B
90.00
70.00
92.00
84.00
73.33
81.33
66.67
78.67
75.00
78.86
4B-width 70B
90.00
70.00
88,00
82.67
73.33
76.00
80.00
72.00
75.33
78.48
4B-depth 9- 70B
90.00
69.00
86.00
81.67
70.67
73.33
68.00
72.00
71.00
75.57
70B -eB-zeroshot
88,00
71.00
89.00
82.67
62.67
74.67
20.00
73.33
57.67
68.38
70B-post-tran train
85.00
69.00
89.00
81.00
65.33
69.33
57.33
68.00
65.00
71.86
medieval castle
8B 9. 70B
90.00
72.00
89.00
83.67
72.00
76.00
68.00
84,00
75.00
78.71
4B-width + 70B
92.00
71.00
91.00
84.67
77.33
77.33
69.33
68.00
73.00
78.00
4B-depth 70B
90.00
70.00
90.00
83.33
58.67
72.00
53.33
72.00
64.00
72.29
21

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Ming Reasoning
Andersen Seirys fairy_tttrtytte tetpspiigeesnnssingsss. "apartment": "cottage",
ancient uncient_Eeyytian Seyyotiannaaginggg "apartnent": "palace",
"bedroon": "chamber",
"bedroom": "sleeping chanber", "bathroan": "bathing reon", "living roon": "sudience hall", "kitchen": "kitchen", "cotabbta "stone table", "desk"; "writing table", "kitchentable". "dining table", "sofa": "'ushioned bench", "kit chencebinet": "storage chest", "cabinet":: "treasure chest", "bat hhw.hccabnnnnt "washstand", "dis hwasher": "servant", "fridge": "cool roon", "microwave": "hesting pot", "stove": "fire pit", "spple": "fruit", "papyrus scroll", "chips": "flatbread", "condimentbottle" "spice jar", "cupcake"": "honey pastry"", "dishbow!""; "clay bowl", "plate": "ceranic plate", "renotecontrol"" "scepter", "salmon": "dried fish", "waterglass "chalice", "wine": "wine", "vineglass": "goblet")
"bathroon"; "washroor""
"Living room": "great hall", "kitchen": "hearth", "cof feetabebll "wooden table", "desk": "uriting desk", "kitchentable": "feasting table", "sofa": "wooden bench", "kit chhencainee"" "pantry", "cabinet": "cuphoard", "bathrooncaainet""" "washstand", "dishwasher"; "washing basin", "fr "fridge": "cooling box", "microwave": "heating stone", "stove": "fireplace", "apple": "asple",
"book": "tone",
"chips": "dried berries", "condinentbattle": "spice jar", "cupcake": "honey cake". "dishbowl": "clay bowl", "plate": "wooden plate", "remotecantral" "nagic wand", "sa Inon": "snoked fish", "waterglass"; "goblet", "wine": "nead" waineglass": "goblet", "kitchencabinett: "pantry shelf")
west west_nappings
outer saarr_spac____ee papaccenappings--
nedieval_cast caapslllasslleetaeeeeeeeeeeeeeee sagieppp__atttle__ppinngg "apartment"; "saloon", "bedroon": "bunk roon", "bathroon": "outhouse", "Living roon": "bar area", "kitchen":: "cooking area", "coffeetable": "woaden table", "desk": "uriting desk", "kitchentable"; "dining table", "sofa": "wooden bench", "kitchencabinett" "starage shelf", "cabinet": "supply cabinet", "bathroomcabinet"" "washstand", "dishwasher": "wash basin", "fridge": "icebox", "nicrowave": "stove", "stove": "woad stove" "apple": "fresh "fpee" "book": "ledger", "chips": "corn chips", "candinentbottle": "sauce bottle", "cupcake": "pastry", "dishbowl": "ceranic bowl", "plate": "ceranic plate", "renotecontrol"" "telegráph key", "sa Lnonn" "salted fish", "waterglass". "glass", "wine": "whiskey", "wineglass" "shot glass")
"apartment"; "saloon",
aapartnen"" "quarters",
"bedroan": "bunk roon", "bathroon": "outhouse", "living roon": "bar area", "kitchen": "cooking area", "cof feetable"; "wooden table", "desk"; "writing desk", "kitchentable": "dining table", "sofa": "wooden bench", "kitchen chiiccaaanaiee" "storage shelf", "cabinet" inatn "supply cabinet", bbathrooncabinet:: "washstand", "dishwasher": "wash basin", "fridge": "icebox", "nicrowave": "stove", "stave": "wood stove" "apple": "fresh apple", "book": "ledger", "chips": "corn chips", "condinentbottlee: "sauce bottle", "cupcake": "pastry", "dishbowt"; "ceranic bowl", "plate": "ceranic plate", "renotecontrol"" "telegraph key", "satnon" "salted fish", "waterglas wwtterglass:: "glass", "wine": "whissey"" "wineglass": "shot glass")
"bedroon": "steeping quarters", "bathroon": "sanitation roon". "living roon": "recreation area", "kitchen": "replicator station", "caffeetable" "control consote", "desk": "comand station", "kitchentable": "ness table", "safa": "lounger",
"kitchencabinett" "storage unit", "cabinet": "storage unit", "bathrooncaaiiett: "hygiene compartnent", "dishwasher": "sterilizer unit", "fridge": "cold storage", "nicrowave"; "food synthesizer",
"stove": "heating unit", "apple": "synthesized apple", "book": "data pad", "chips": "nutrition chips", "condinentbottlee: "flaver vial". "cupcake": "synthesized pastry", "dishbowl": "serving bowl", "plate": "serving plate". "eeooecoonrra""" "control pad", "sa inon": "replicated fish", naterglass". "hydration vessel", "wine": "synthesized wine", "wineg lass": "drinking vessel", "kitchencabinet": "storage unit">
Figure 7. Primary changes from VirtualHome simulator between the original apartment scenario and the five transferred thematic environments used in our ToM experiments.
22

---

Overcoming Multi-step Complexity in Multimodal Theory-of-Ming Reasoning
Table 12. Performance social interaction MuMA-ToM bench-
mark.
Method
Belief
Social Goal
Belief of Goal
All
Human
98.9
94.4
93.5
Gemini 1,5 Flash
53.9
33.0
41.4
42.7
Gemini 1,5 Pro
78.9
43.9
46.9
56.4
Llava 1.6 13B
70.2
43.2
17.9
43.7
Llava 1.6 34B GPT-4o
93.6
37.2
27.5
52.8
67.9
39.6
44.4
50.6
InternVL 2 26B
59.3
44.9
35.5
46,6
VadeoL 2 7B
70.1
45.6
37.7
51.1
BIPALM Bamaa2-B
41.2
34.1
30.6
33.9
LIMP (GPT-40)
93.4
67.7
76.6
Ours (8B+405B)
94.0
64.5
67.5
75.3
Table 13. Performance comparison: Our wcak-to-strong method
vs. fully fine-tuned and PEFT baselines.
Fine-tuning
Data
LM
Acc.
Method
FFT
20,000
4B
77.00
Llama3.1 11 --Minitronn-B-Wiiae Width
LoRA Abit-QLoRA
20,000
4B
76.90
20,000
Weak-to-Strong
FFT-tained 4B
20k
70B
78.67
4B-Width - Llama3,11700 1-778
LoRA-trained 4B
20k
70B
78.52
Siit-QloRA-ttan
PEFT Large LM:
FFT
20,000
70B
71.45
Llama3.1-708
LoRA
20,000
70B
71.86
4bit-QL.RA
Table Ablation study on the contribution of Weak-to-Strong (W2S) Control (with 8B as weak small LM).
Comlig
Belief Avg.
Goal Avg.
Avg.
Llama2 70B
Post-trained (No W2S)
72.00
76.43
Llama2 70B
Weak-to-Strong
78.05
Llama3 70B
Post-trained (No W2S)
83.33
65.33
73.05
Llama3 70B
Neak-to-Strone
86.00
73.33
78.76
Llama3. 1 70B
Post-trained (No W2S)
85.00
62.00
71.86
Llama3.1
Weak-to-Strong
79.38
Llama3.1 405B
Weak-to-Strong
87.10
77.14
81.29
23


Reviews:
[{"review_id": "22M5UsnbT5", "reviewer": "Reviewer_dLAb", "summary": "The paper presents a Bayesian ToM method using stepwise belief updates and weak-to-strong LM transfer, unifying social and world knowledge to achieve 4.6% higher accuracy on multimodal tasks (including unseen settings) than prior approaches, resolving scalability/generalization trade-offs.\n\n## update after rebuttal\nThe authors have addressed most of my concerns. Although minor issues remain, the paper is satisfactory overall, and I lean toward an accept recommendation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank Reviewer dLAb for their insightful comments and support.**\n\n---\n\n**Q1:** Practicality under strict real-time/resource constraints?\n\n**A1:**  \nOur method uses a small post-trained LM (4B/8B) to dynamically guide the large pretrained LM (70B/405B) at inference. Practically, both models comfortably fit on individual NVIDIA H100 GPUs (80GB, BF16 precision) and run in parallel with minimal synchronization (small likelihood tensors). For example, for the 8B+70B model, its inference for 600 tasks takes ~14–15.5 min (1.4–1.55s per question), nearly identical to a single unguided 70B model, as the extra computational overhead of the 8B model is almost negligible compared to the 70B model.\n\nFurthermore, the large LM only performs likelihood estimation (**prefilling** ~1024 tokens), typically completed in ≤1 second per GPU [NVIDIA refs 1, 2]; correspondingly, the small LM requires only ~0.5 seconds per question as an individual. Such prefilling tasks are highly amenable to acceleration tools (e.g., NVIDIA Dynamo, vLLM), making our Bayesian ToM planner practically suitable even under resource constraints.\n\nAdditionally, our method avoids the costly fine-tuning of large LMs. For example, directly fine-tuning a 405B model is practically infeasible for most institutions due to extreme GPU requirements (~50–64 Nvidia H100 GPUs). In contrast, our method can fine-tune an 8B model to guide a 405B model, requiring only a single H100 GPU and achieving superior performance. We will include this discussion in our revised paper.\n\n---\n\n**Q2:** Generalization to diverse, nuanced real-world social interactions?\n\n**A2:**  \n\n**Table A: https://anonymous.4open.science/r/response_tom-BD87/tableAB.md**\n\nWe evaluated generalization to diverse, complex unseen scenarios (Tables 4, 9 &10), covering Andersen fairy tales, Ancient Egypt, Outer Space, Wild West, and Medieval Castle. Results consistently show stable, robust generalization.\n\nAdditionally, inspired by your feedback, we expanded our evaluation using MuMA-ToM [3], a benchmark explicitly designed for nuanced social interaction, including:\n\n- **Belief inference:** Understanding environmental dynamics.\n- **Social Goal inference:** Interpreting subtle social objectives.\n- **Belief-of-Goal inference:** Attributing complex mental states.\n\nResults in Table A show that our method performs competitively to the state-of-the-art GPT-4o-based LIMP [3] and outperforms all the other baselines. Note that this is achieved by using open-source models, avoiding the expensive GPT-4o API cost required by LIMP [3]. Our weak-to-strong control robustly leverages large pretrained LMs, effectively adapting to real-world social reasoning without compromising generalization. We will include these new results in our revised paper.\n\n---\n\n**Q3:** Why were prominent CoT models (Deepseek, GPT O3) not included?\n\n**A3:**  \n\n**Table B: https://anonymous.4open.science/r/response_tom-BD87/tableAB.md**\n\nThank you for highlighting CoT models. We initially did not include Deepseek R1 (671B) or GPT O3-mini (released in Jan 2025), due to timing close to the submission due (also in Jan 2025). Additionally, our multimodal ToM tasks prioritize implicit world knowledge and nuanced mental-state reasoning, which differ fundamentally from CoT models' strength in explicit logical reasoning.\n\nFollowing your valuable suggestion, we conducted new evaluations in Table B, which clearly show the performance ranking:  \n**Our method > Deepseek R1 > GPT O3-mini**.\n\nThis demonstrates that the core challenge in multimodal ToM tasks is the depth and breadth of implicit mental-state and world knowledge—areas where large pretrained representations excel over logic-specialized CoT models. Further, our cognitive-inspired BIP framework effectively mitigates the overthinking/hallucination pitfalls observed in specialized logical reasoning models. We will include these new results in our revised paper.\n\n---\n\n**Q4:** (minor) The setup is sound, but more details on data partitioning and confounding factors would be beneficial.\n\n**A4:**  \nWe provide some detailed dataset construction, partitioning strategies, and confound mitigation in Sec. 4.1 and App. D. Specifically, following MMToM-QA's setting, the training set is derived from 1,000 procedurally-generated videos annotated with structured sequences (states, goals, beliefs, actions). The test set uses 600 questions derived from other 134 videos, which are entirely disjointed environments and narratives. To enhance clarity, we will add more details to App. D, as well as explicitly reference App. D and the accompanying repository readme file in the revised main paper.\n\n---\n\n**References:**\n\n- [1] NVIDIA MLPerf AI Benchmarks. “Llama 2 70B: MLPerf Benchmark.”\n- [2] NVIDIA Technical Blog. “Boost Llama 3.3 70B Throughput 3x with TensorRT.”\n- [3] MuMA-ToM: Multi-modal Multi-Agent Theory of Mind, AAAI-25 (Oral).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I verified the proof on decomposing ToM reasoning into stepwise Bayesian updates and the one on transferring ToM reasoning from smaller to larger language models. Overall, the proofs are sound, though some high-dimensional scalability assumptions need further clarification.", "experimental_designs_or_analyses": "We reviewed the experimental design on multimodal benchmarks and the analysis reporting a 4.6% improvement. Overall, the setup is sound, but more details on data partitioning and confounding factors would be beneficial.", "relation_to_broader_scientific_literature": "This work bridges Bayesian cognitive modeling (via modular belief updates) and large language models (LLMs) to overcome scalability and generalization issues in traditional Theory-of-Mind methods. Its novel weak-to-strong knowledge transfer integrates social reasoning with real-world knowledge in LLMs, achieving state-of-the-art accuracy in complex, unseen multimodal scenarios.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n+ The Bayesian ToM planner decomposes complex theory-of-mind reasoning into stepwise Bayesian updates. This design enables effective scaling across different model sizes (from 7B to 405B parameters), overcoming the typical scalability issues found in previous methods.\n+ The approach leverages a “weak-to-strong” control strategy by using smaller language models to refine ToM-specific likelihood estimates. These estimates are then integrated into larger models, which combines specialized reasoning with broader social and world knowledge.\n+ The method demonstrates a 4.6% accuracy gain over state-of-the-art techniques on multimodal ToM benchmarks, including in unseen scenarios, suggesting tangible benefits in real-world applications.\n\nWeaknesses:\n- Although the method is scalable, combining multiple models (smaller ones for likelihood estimation and larger ones for integration) may lead to increased computational complexity. This might limit its practicality for applications with strict real-time or resource constraints.\n- Although experimental results are promising, it remains uncertain how well the approach will perform across the diverse and nuanced landscape of real-world social interactions.\n- Why Not Use Chain-of-Thought Models Like Deepseek or GPT O3.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you elaborate on why chain-of-thought models like Deepseek or GPT O3 were not experimented with in your approach, given their recent prominence?\n2. What are the practical computational limitations of your Bayesian ToM planner in real-world, resource-constrained scenarios, and how might these affect its scalability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "760q4H31BH", "reviewer": "Reviewer_DnoB", "summary": "This paper proposes a scalable Bayesian Planner that employs small models for stepwise Bayesian updates, refining the likelihood estimation of larger models. Experimental results demonstrate that this approach outperforms existing methods on multimodal Theory of Mind (ToM) benchmarks and generalizes well to unseen scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely thank Reviewer DnoB for the insightful comments and valuable suggestions.**\n\n---\n\n**Q1:** *Does direct post-training limit the generalization of the large LM, or does guidance from smaller post-trained LMs enhance generalization?*\n\n**A1:**  \nThank you for raising this insightful question. Our experiments (Tables 2 & 3) suggest that directly post-training the large LM (70B) achieves suboptimal generalization. Specifically, the directly post-trained 70B LM performs worse than the proposed method (8B+70B, 4B-depth+70B, and 4B-width+70B).\n\nThis indicates that direct post-training on large LMs may inadvertently diminish their inherent generalization capabilities, likely due to partial overwriting of implicit world knowledge and mental-state reasoning abilities acquired during extensive pretraining. This interpretation aligns closely with recent literature on generalization degradation and catastrophic forgetting in extensively fine-tuned large LMs [1,2,3].\n\nIn contrast, our proposed weak-to-strong control addresses this by post-training only small LMs (4B or 8B), which subsequently guide the larger LM exclusively at inference time without modifying its pretrained weights. Thus, these smaller models function as specialized lightweight controllers that effectively enhance ToM reasoning without compromising the broader pretrained capabilities. This strategy enables our framework to effectively balance task-specific adaptation with robust generalization, yielding better performance on previously unseen scenarios. We will include this discussion in our revised paper.\n\n**References:**  \n- [1] “Overtrained Language Models Are Harder to Fine-Tune,” arXiv:2503.19206.  \n- [2] “Understanding Catastrophic Forgetting in Language Models via Implicit Inference,” ICLR 2024.  \n- [3] “Spurious Forgetting in Continual Learning of Language Models,” ICLR 2025.\n\n---\n\n**Q2:** *$g_1/g_2$ in Eq (2)*\n\n**A2:**  \nIn Equation (2), $g_1$ and $g_2$ represent two candidate **goal hypotheses** within our Bayesian inverse planning framework. Each hypothesis $H_i = \\langle g_i, b_i^t \\rangle$ comprises a goal $g_i$ and a corresponding belief $b_i^t$, jointly providing potential explanations for the observed agent behavior. In our experimental setup (Sections 3 and 4), these hypotheses directly correspond to the provided multiple-choice answer options (e.g., options (a) and (b)). Equation (2) thus computes and compares the posterior likelihoods of these candidate hypotheses given observed states and actions. We will explicitly clarify this linkage near Equation (2) in the revised manuscript.\n\n---\n\n**Q3:** *Results on text-only benchmarks to strengthen the claim of generalizability?*\n\n**A3:**  \n\n**Table C: https://anonymous.4open.science/r/response_tom-BD87/tableC.md**\n\nFollowing your valuable suggestion, in Table C, we conducted additional experiments specifically evaluating our method on text-only Theory-of-Mind tasks (the same MMToM-QA benchmark but without visual or multimodal inputs). We are pleased to provide these new results and will include them in our revised paper.\n\nIn these text-only evaluations, despite lacking the fine-grained temporal state transitions provided by multimodal observations, our model maintains robust performance. As our smaller LM was post-trained on activity data derived from the multimodal ToM simulator, it effectively leverages familiarity with underlying scenarios to accurately infer high-level ToM states from text alone, thereby strongly supporting our claim of generalization.\n\n---\n\n**Q4:** *Performance on scenarios in Figure 1*\n\n**A4:**  \n\nFollowing your recommendation, we conducted additional evaluations of our weak-to-strong control framework specifically on scenarios depicted in the updated Figure 1 ([link](https://anonymous.4open.science/r/response_tom-BD87/accuracy_vs_steps_update.png)). \n\nOur model (8B+405B) consistently outperforms the 405B with CoT on tasks ranging from 1-step to 7-step planning. For tasks exceeding 8 planning steps, performance between methods converges. This convergence occurs because these highly complex multi-step tasks fall beyond the distribution of the post-training dataset. As a result, successful reasoning at higher complexities increasingly depends upon the intrinsic pretrained grounding capabilities of the large LM. We will include this discussion and the updated Figure 1 in our revised paper.", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "The paper introduces a novel hierarchical modeling approach where specialized smaller models assist larger models, effectively balancing robustness and generalization while reducing the complexity of multistep training. The control mechanism in Equation (7) is particularly well-designed, allowing the language model to focus on differences between the base model and the post-trained model.\n\n\nThe experimental design is strong, including evaluations of models' performance over extended planning steps, tests on both text-only and multimodal benchmarks, and assessments in unseen scenarios.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I have some clarification questions:\n- How does post-training affect generalization in unseen scenarios? In Table 4, the 4B-depth and 8B+70B models outperform the 70B post-trained model. Does this suggest that post-training limits the generalization of the 70B model, or that the smaller post-trained model enhances the generalization of larger models?\n\n- What is the notation g1/g2 in Equation (2)?\n\n- What are the results on text-only benchmarks? Given the framework’s design, it should theoretically generalize well to text-based ToM tasks. Providing these results would strengthen the claim of generalizability.\n\n- What is your model’s performance on the scenarios depicted in Figure 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OrlGNOGMRo", "reviewer": "Reviewer_RMFT", "summary": "This paper addresses the scalability limitation of Theory-of-mind (ToM) models in multi-modal environments. Predicting agents' goals and beliefs in complex mutli-modal environments involving vision and language requires visual understanding, multiple steps planning and reasoning, as well as extensive world knowledge. While LLMs in the order of hundreds of billions of parameters posses such capabilities, fine-tuning them to ToM tasks is expensive. To address this, the paper presents a weak-to-strong guidance framework where small LMs (in the order of 4-8B parameters) are fine-tuned on ToM tasks and guide a large LM to perform such tasks without further fine-tuning. This is achieved by modifying the LLM's predictions by the difference in predictions between the base small model and the same model after it was fine-tuned on ToM tasks. The authors conduct several experiments that show the efficacy of their approach on a plethora of models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer RMFT for the positive and encouraging assessment, and for clearly recognizing the unique complexity in multimodal ToM we aim to address. The core philosophy of our weak-to-strong guidance BIP framework is precisely to leverage specialized small language models to efficiently guide large pretrained models, thus preserving extensive implicit world knowledge essential for multimodal ToM. \n\nYour confirmation of our theoretical rigor, experimental soundness, and alignment with the broader literature greatly encourages us!", "claims_and_evidence": "The paper is very well written and the claims are supported by evidence.", "methods_and_evaluation_criteria": "The evaluation methods make sense.", "theoretical_claims": "The proof of Theorem 1 seems correct to me.", "experimental_designs_or_analyses": "The experimental design is sound.", "relation_to_broader_scientific_literature": "The paper addresses an important issue in the area of ToM modeling and provides a sound solution.", "essential_references_not_discussed": "Prior work is properly discussed,", "other_strengths_and_weaknesses": "Nothin to add.", "other_comments_or_suggestions": "Nothing to add.", "questions_for_authors": "Nothing to add.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VF7REdRuV4", "reviewer": "Reviewer_Dtd9", "summary": "This paper primarily aims to tackle multimodal ToM, though in practice, their main focus is still on complex multi-step reasoning tasks, with the multimodal aspect being somewhat secondary in their method.\n\nThey propose the \"Weak-to-Strong Control\" strategy, which modifies the probability distribution at the output layer of a small model to directly adjust the corresponding probability distribution of a large model. This allows them to enhance ToM reasoning capabilities without fine-tuning the large model, thereby reducing computational costs.\n\nThe experimental results support their claim.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**We sincerely appreciate Reviewer Dtd9’s valuable comments and suggestions.**\n\n---\n\n**Q1:** *Comparison with fine-tuned large LMs (e.g., 70B/405B)*\n\n**A1:**\n\n**Table D: https://anonymous.4open.science/r/response_tom-BD87/tableDEFGH.md**\n\nDirectly fine-tuning a 405B model is practically infeasible for most institutions due to extreme GPU requirements (~50–64 Nvidia H100 GPUs). Therefore, we approximated this scenario using the more manageable Llama-3.1 70B model. In our paper, Tables 2 & 3 have demonstrated that the fully post-trained 70B LM performs worse than the proposed method (8B+70B, 4B-depth+70B, and 4B-width+70B). This suggests that our method not only reduces resource usage but also achieves superior generalization, aligning with recent literature showing generalization degradation after fine-tuning large LMs [1-3]. Results (Table D) on Llama-3.3 70B (widely considered equivalent to 3.1-405B by the community) further support this conclusion. (See also our detailed discussion in Q1-A1 to Reviewer DnoB).\n\n---\n\n**Q2:** *Comparison with parameter-efficient fine-tuning (PEFT) such as adapters or LoRA.*\n\n**A2:**\n\n**Table E: https://anonymous.4open.science/r/response_tom-BD87/tableDEFGH.md**  \n\nThe proposed weak-to-strong control is fully orthogonal and complementary to PEFT techniques, i.e., we can combine our method with any PEFT technique. In fact, our small LMs are trained by LoRA, as described in L191-right. Table E further confirms our method's consistent effectiveness regardless of the PEFT choice for small LM.\n\nDirectly applying PEFT to large pretrained LMs performs worse than our method (Table E). As discussed above, our method avoids fine-tuning large LMs and thus preserves their pretrained mental/world knowledge [1-3], essential for generalization in multimodal ToM tasks.\n\n---\n\n**Q3:** *If other scenarios such as social interactions or psychological reasoning can be added*\n\n**A3:**  \n\n**Table A (same as Table F): https://anonymous.4open.science/r/response_tom-BD87/tableAB.md**\n\nWe sincerely appreciate your recognition of the difficulty of testing nuanced social generalization. Inspired by your feedback, we expanded our evaluation using MuMA-ToM [4], a benchmark explicitly designed for multi-agent social interaction tasks involving belief inference, social goal inference, and belief-of-goal inference (Table A). Our model achieves competitive results to LIMP [4], the expensive GPT-4o-based SoTA method, demonstrating clear generalization across social relationship tasks. (See also our detailed discussion in Q2-A2 to Reviewer dLAb).\n\n---\n\n**Q4:** *Connection / how the method addresses multimodal ToM challenges.*\n\n**A4:**  \nMultimodal ToM tasks uniquely require integrating implicit world knowledge and dynamic temporal mental-state reasoning, which significantly differ from standard STEM reasoning tasks. Our method addresses these challenges explicitly through two key innovations:\n\n1. Leveraging the vast implicit knowledge encoded in scaled-up pretrained LMs (e.g., 405B parameters), crucial for understanding complex social interactions and environmental contexts.\n2. Structuring dynamic belief inference explicitly using Bayesian Inverse Planning (BIP), a cognitive-science-based framework designed specifically for ToM.\n\nThis combination strategically bridges the gap between abstract multimodal contexts and nuanced mental-state reasoning, effectively resolving the multimodal complexity identified in our Figure 1.\n\n---\n\n**Q5:** *Ablations clearly isolating Weak-to-Strong Control's contribution.*\n\n**A5:**  \n**Tables G & H: https://anonymous.4open.science/r/response_tom-BD87/tableDEFGH.md**\n\nTo clarify its impact, we demonstrate two ablation studies:\n\n- We compare with naïvely post-trained models. Results summarized in Table G (i.e., Tables 2, 3 & D) show consistent performance drops without weak-to-strong control. This demonstrates clearly that Weak-to-Strong guidance enhances the large LM’s ability to generalize by effectively leveraging its pretrained world and mental-state knowledge.\n\n- In Table H, when directly combining a small LM and a large LM (directly adding their logits) without our structured weak-to-strong adjustment, the performance is lower than our method. Thus, naïvely adding a small LM is suboptimal. Explicit weak-to-strong control is essential for abstracting the smaller LM’s specialized ToM behavior to leverage the large LM’s pretrained knowledge.\n\nOverall, these ablations conclusively validate that our mechanism is both critical and independently responsible for our method’s demonstrated role in ToM grounding.\n\n---\n\n**References**  \n- [1] Overtrained Language Models Are Harder to Fine-Tune, arXiv:2503.19206\n\n- [2] Understanding Catastrophic Forgetting in Language Models via Implicit Inference, ICLR 2024\n\n- [3] Spurious Forgetting in Continual Learning of Language Models, ICLR 2025\n\n- [4] MuMA-ToM: Multi-modal Multi-Agent Theory of Mind, AAAI-25 (Oral)\n\n---", "claims_and_evidence": "1. The authors claim that their \"Weak-to-Strong Control\" strategy enhances ToM reasoning while reducing computational costs, but their experiments only compare 405B + 8B control to an untrained 405B model, not to a task-trained 405B. This means they only prove performance improves under cost constraints, but not that their method is the best way to enhance ToM reasoning if cost were not a concern.\n\n2. The authors demonstrate that their method achieves better generalization by outperforming direct inference in unseen environments such as outer space, ancient Egypt, fairy tales, the Wild West, and medieval Europe. However, these tasks remain structurally similar, primarily involving object-location-based reasoning. It remains unclear whether their approach would generalize equally well to other types of ToM tasks, such as inferring social relationships or tracking psychological shifts in negotiations, where reasoning dynamics differ significantly.", "methods_and_evaluation_criteria": "The proposed method is actually more similar to adapter tuning and LoRA-style approaches, which aim to achieve better performance by fine-tuning fewer parameters rather than performing full model fine-tuning. A more reasonable baseline for comparison should be these parameter-efficient fine-tuning methods. The paper itself feels somewhat inconclusive, as I don’t see a strong connection between the proposed method and multimodality ToM.", "theoretical_claims": "Theorem 1 is mathematically valid and justifies that Weak-to-Strong Control allows the large model to approximate a post-trained model without fine-tuning, ensuring minimal divergence through controlled adjustments.", "experimental_designs_or_analyses": "The experiments show that 405B + 8B control improves performance compared to an untrained 405B model, but without comparing to a fine-tuned 405B or LoRA/adapter tuning baselines, it's unclear if this is the best approach for improving ToM reasoning. Also, an ablation study is needed to confirm whether the improvement comes from Weak-to-Strong Control itself or just adding a small model.", "relation_to_broader_scientific_literature": "The paper builds on ToM reasoning and efficient fine-tuning methods by having a small model adjust a large model’s outputs instead of fine-tuning, but without comparing to LoRA, adapter tuning, or fine-tuned large models, it’s hard to tell if this is actually the best approach, and the multimodal angle feels more like an add-on than a core focus.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is well-structured.", "other_comments_or_suggestions": "The generalization tests focus only on object-location-based tasks, it would be more convincing if some other scenario such as social interactions or psychological reasoning can be added. But this might be hard, so if not able to, it is fine.", "questions_for_authors": "As I discussed above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously found the work to be well-written, the claims supported by strong evidence, and the experimental design sound. The reviewer found the core method (particularly the weak-to-strong guidance which avoids costly fine-tuning of large LLMs while preserving their world knowledge) to be a strong contribution.

Reviewers raised several questions regarding computational cost, generalization beyond tested scenarios, comparison to alternative methods (CoT models, PEFT, full fine-tuning), and the precise contribution of the proposed control mechanism. The authors provided an exceptionally thorough and convincing rebuttal, and the reviewers generally found majority of concerns being resolved.

Given the paper's novelty, technical strength, compelling results (demonstrating state-of-the-art performance and generalization), strong reviewer consensus I strongly recommend acceptance.
Decision: Accept (spotlight poster)