Unveiling the Cognitive Compass: Theory-of-Mind–Guided Multimodal Emotion Reasoning
Authors: Meng Luo
Bobo Li
Shanqing Xu
Shize Zhang
Qiuchan Chen
Menglu Han
Wenhao Chen
Yanxiang Huang
Hao Fei
Mong-Li Lee
Wynne Hsu
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Affective Computing
Multimodal Understanding and Reasoning
Reinforcement Learning
Primary area: datasets and benchmarks
TL;DR: We present HitEmotion, a ToM-grounded benchmark, and TMPO, a reinforcement learning approach with ToM-guided reasoning chains, to enhance emotional reasoning in multimodal LLMs.

Abstract: Despite rapid progress in multimodal large language models (MLLMs), their capability for deep emotional understanding remains limited. We argue that genuine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise. To this end, we introduce HitEmotion, a ToM-grounded hierarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth. Second, we propose a ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning. We further introduce TMPO, a reinforcement learning method that uses intermediate mental states as process-level supervision to guide and strengthen model reasoning. Extensive experiments show that HitEmotion exposes deep emotional reasoning deficits in state-of-the-art models, especially on cognitively demanding tasks. In evaluation, the ToM-guided reasoning chain and TMPO improve end-task accuracy and yield more faithful, more coherent rationales. In conclusion, our work provides the research community with a practical toolkit for evaluating and enhancing the cognition-based emotional understanding capabilities of MLLMs.

Published as a conference paper at ICLR 2026
UNVEILING THE COGNITIVE COMPASS: THEORY-OF- MIND-GUIDED MULTIMODAL EMOTION REASONING
Meng Luo't Bobo , Shanqing Xu², Shize Zhang', Qiuchan Chen', Menglu Han', Wenhao Chen', Yanxiang Huang", Hao Feil, Mong-I. Lee', Wynne Hsu
National University of Singapore
"Huazhong University of Science and Technology
'hh Hong Kong Polytechnic University
ABSTRACT
Despite rapid progress in multimodal large language models (MLLMs), their ca- pability for deep emotional understanding remains limited. We argue that gen- uine affective intelligence requires explicit modeling of Theory of Mind (ToM), the cognitive substrate from which emotions arise. To this end, we introduce HitEmotion, a ToM-grounded hicrarchical benchmark that diagnoses capability breakpoints across increasing levels of cognitive depth. Second, we propose ToM-guided reasoning chain that tracks mental states and calibrates cross-modal evidence to achieve faithful emotional reasoning. We further introduce TMPO, a reinforcement learning method that uses intermediate mental states as process- level supervision guide and strengthen model reasoning. Extensive experiments show that HitEmotion exposes deep emotional reasoning deficits in state-of-the- art models, especially on cognitively demanding tasks. In evaluation, the ToM- guided reasoning chain and TMPO improve end-task accuracy and yield more faithful, more coherent rationales. conclusion, our work provides the research community with practical toolkit for evaluating and enhancing the cognition- based emotional understanding capabilities of MLLMs. Our dataset and code are available at: httplliiiEimmitoioiggiuiiie.
INTRODUCTION
Emotional intelligence (Picard 2000) lies at the heart of machine intelligence and plays pivotal role in the development of human-centric AI systems. Despite the remarkable progress of Multimodal Large Language Models (MLLMs, (OpenAl 2025 Gemini et al.) 2023 Li et al. 2024c)) across various tasks, their capability in deep emotional understanding remain suboptimal (Huang et al.) 2023 Sabour et al., 2024). Existing studies primarily focus surface-leved emotion recognition, often neglecting the dynamic, context-dependent nature of emotions and their intricate relationships with other mental states such as beliefs and intentions (Khare al. 2024).. Such oversimplification overlooks the complexity of human affect, limiting the interpretabilirg lity and performance of MLLMs
in emotional understanding (Zhang et al. 2025c Luo al. 2024b Kang et 2025).
Recent benchmarks such as EmoBench (Sabour et al., 2024) and EmotionHallucer (Xing et al., 2025) have empirically validated this bottleneck: even state-of-the-e (SOTA) MLLMs struggle with emotionally complex tasks that require nuanced perspective-takingg or reasoning over conflict- ing multimodal cues (Yang et al.] (204)). These failures often manifest as emotional hallucinations and other distortions. However, while these evaluations (Hu et al., [025 Huang et al. 2023) suc- cessfully expose the symptoms, their own fragmented task design limits deeper diagnosis of the underlying causes. We argue that the core limitation of current evaluation paradigms lies in the absence of a unified cognitive framework, a veritable Cognitive Compass, to guide the evaluation of mental state reasoning (Chen et al. 2025). Specifically, they fail to organize emotional reason- ing tasks according to developmental levels of Theory Mind (ToM, (Lake et al. 2017) -...)...g..
"Emiil: mluo@ u.nus.edu
1 "Correponning author: Hao Fei (haofei37 @ (ud.ccu.ss.)

---

Published as a conference paper at ICLR 2026
Theery of Wind (terl)
EPR
EUA
ECR
CO
Large sonte 2044 sumples
Stadees cxpacity to andenstard stares sach as cortion, setort lan, to peoduct pituriees baadd ie this anderstureiin
Eastios Percsption Recognteten
Exotiee Inderstanding and Anelysts
Cogritian and Reasominy
level 3
Ressenable classsfication
level 2
level
Levels
18 tasks
Lasks
tasks
FESDI ESA NESA
NER ISAA
Mattlendet mad. bxt
28
163
F8
Fust-deaer attrouucian
1.
Tmage Sentinnnt Jealysts
Decode Als> ha Sigut
Eyttrddn x Veral
ABl bul fan we ser's Fangedive
EUA
R
ECR
Dara
4
Contectint
Desile M. in wol Dalagic
Esaloge
Atlbulo the Spadlers
Secogritdan
Colltc dloo with Rule
Conducte for Label
Rancemale Cowet tounl wrattigeece
Hkdttindaa Exetion Caase Pade: Ectrectionn
Secoen chattr
1
Decoca Taget Tun
Iuid Cankent
Coust Lisingg Prly Utioa de
HE Emtotion
Figure 1: Overview of our HitEmotion benchmark.
first-order belief inference (Wimmer & Perner 1983 vs. second-order recursive reasoning (Perner & Wimmer 1985) Without such a compass, benchmarks provide only a coarse overall score and cannot pinpoint the exact ceiling or breaking point of model's reasoning capacity.
This lack of precision in evaluation, in turn, hides fundamental flaws in the models' reasoning pro- cess. Even with generic Chain-of-Thought (CoT, (Wei et al. 2022 approach, the reasoning abilities of MLLMs (Fei et al.] 2025: Luo et al. 2025 tend to emerge from general properties rather than from oegnition-specifie coherent but are ultimately unfaithful. Specific problems include substituting causal attribution with simple template matching, being highly sensitive to small changes in wording and prompts, lacking robust ways to update in response to counterfactuals, and failing to explicitly track maintain sistency among intermediate mental states like beliefs and intentions. Ultimatlly, these systems are measuring shallow emotional fact retriever, skilled only mapping superficial cues, rather than deep mental state simulator capable of inferring the complex interplay between mind and emotion.
supervised training. This leads reasoning chains that often look
To tackle these dual shortcomings in evaluation and reasoning, and to help shift the paradigm in emotion understanding from fact retrieval to mental simulation, this study makes two core con- tributions. ① A hierarchical, Theory-of-Mind-- Mind-based benchmark for multimodal emotion under- standing moitEmotion)) As presented in Figure [ HitEmotiion systematically arranges evaluation tasks into three cognitive levels of increasing depth: Emotion Perception and Recognition, Emotion Understanding and Analysis, and Emotion Cognition and Reasoning. This hierarchical structure designed to precisely pinpoint and measure a model's capability breakpoints at different cogni- tive depths. @ A novel framework for Theory-of-Mind reasoning chain preference optimization (TMPO). This approach begins by designing structured reasoning templates for specific tasks based on ToM principles. It then pioneers the use of intermediate mental states from these reasoning chains as both supervisory signals and reward sources for reinforcement learning. The method aims to shift model's reasoning from "general emergent" ability to "domain-acquire" acquired" skill, significantly boosting its performance, robustness, and auditability in complex situations.
To validate the proposed framework, the first step was to construct a new evaluation resource. We curated 24 diverse datasets spanning sentiment, humor, sarcasm, and causal reasoning, and sys- tematically cleaned and re-purposed them. Following the cognitive hierarchy of the proposed ToM framework, these datasets were rigorously restructured and aligned to create a benchmark capable of fine-grained assessment of model capabilities. Extensive experiments on our benchmark yielded three key findings. ① The performance of baseline models decisively substantiated our critique. Even SOTA MLLMs performed inconsistently across tasks and exhibited profound deficiencies at the highest tier of our framework. @ ToM reasoning chain prompting by itself shows considerable potential. When used simply as prompting strategy, it significantly improves the performance of powerful closed-source models, offering initial proof of ToM's effectiveness fold." " O The TMPO optimization delivers significant and consistent improvements across all evalu- ation tasks. It not only scores higher than most baseline models but also generates reasoning chains with demonstrably greater faithfulness and logical consistency, highlighting the advantages of the "domain acquisition" approach. In conclusion, the HitEmotion benchmark and TMPO method offer the research community a powerful toolkit for evaluating and advancing the deep emotional intelli- gence of MILMs,, facilitating the development of genuinely empathetic AI systems.
reasoning "scaf-
2

---

Published as a conference paper at ICLR 2026
Eowa on Senkrant Deaszon
10 Task
Enstet
Percpiion aNE RecopeNoc
Vere Sertnny Anaylle
Naaroce Eritis ThEps
Intel Hiturty feadya
Beaediee commpee Do af
Gong Emetton on Racogetin
Speeth Erriion Recogiaion
Commentt Emrtton Araly
08
Deleclon of Penuasoon Technig in Mernes
Erolon-eva Beaee IntrrA yous
Hamer Lnckrntancing
Task
Rinockal
Sundmorr
multtmodel Gusmtuple Extraction
Hu dincccl Stance Derection
Mull party Osakggu Ereo Jon Rooogg Rice
3000
Errckin Elictalion Reasoring
Einotion intepestatont
Lougfe Rlasoring
06
Waader aigh??
Nalirocal Eradiinn Cleas Par Cernction
Denection
Task
Figure 2: Task taxonomy and examples in our HitEmotion benchmark.
Table 1: Comparison with other benchmarks related to emotional intelligence. Psych-based
indicates grounding in psychological theory; Rea-chain indicates whether reasoning traces are pro- vided; Rationale indicates whether model rationales are included.
Benchmark
Task
Modality
# Instances
Type
Psy-based
Rea-chain
Rationale
EQ EQ-bench (Paech
20236
Text
60
Open-ended Open-ended
Emetionnergg Ho03gg Clal. 2023
Text
428
EmoBench (Sahoir drain 2014
Text
400
MCQ MCQ
MOSABendi [Song et
2024bb
Image
1,047
MM-Instrootte
vatprrooettvvttllnaaa (1.ng al 2025
Image
34,602
MCQ
X
FimoBench-M eHu
Viden
5,646
MCQ, Open-ended
MER-Unnleent Unillench [Lian et al.
2025
Video
12,799
Open-endeed
EmetionHalluuer Xing et 2015
Video, Image
2.742
Binary QA
MME-Emotion Ziiing ar
20238
Video
6,500
Open-caded
HitEmottin (Onrs)
24
Video, Image
20,114
MCQ. Open-ended
RELATED WORK
Multimodal Affective Computing. Multimodal Affective Computing aims to understand human emotions by learning cross cross-modal representations from heterogencous signals like language, vision, and acoustics (Ramirez et al. 201T 20rT.Diang et al. 2021 Zhu et al., 2024b a). Evolving from unimodal studies (li et al. [020:] Donnelly & Prestwich 2022), the field now emphasizes unified alignment- and-fusion frameworks, a shift accelerated by large-scale pretraining (Gemini et al., 2023) and en- abled by parameter--fffiient adaptation (Houlsby et al. 2019). Fusion techniques have similarly advanced from early/late strategies (Tsai et al. 2019:] Chen et al., 2024a) to more sophisticated in- termediate schemes that deepen cross-modal interaction and yield more discriminative features (Luo) ct al. 2021: Zou et al. 2023). Recent work further refines how affect is modeled, for instance by treating emotion inherently ordinal to better infer intensity (Mai et al. 2025) or by leveraging human actions as a sparse but highly credible signal for emotional understanding (Yu et al. (005)).
Evaluation of Emotional Intelligence. The evaluation of emotional intelligence has progressed from text-based queries to comprehensive multimodal benchmarks. Initial text-only assessments established reproducible formats for testing emotional inference and reasoning (BQ-Bench, (Paech) 2023a); EmotionBenchh, (Huang et al.] 2023); EmoBench, (Sabour et al. 2024)). The focus has since expanded to multimodal settings that probe for more contextual understanding. Current benchmarks assess a wide range of capabilities, including multi-object sentiment analysis (MOSABench: (Song al. at, 2024b)), carructiion-oolooinng
Instucttt;;;; (Yang et al. 2025)). ). hierarchical skills from recognition to social awareness (EmoBench-M: (Hu et al., (025)) ), and unified evaluation of clas- sic and free-form responses (MER- (MER-UniBench; (Lian et 2025)). Complementary work explicitly audits emotion-related hallucinations (EmotionFHallucer:: (Xing et al.) 2025)) provides holistic, multi-agent scoring across diverse scenarios MMMEE-Emtioo; (Zhang et al. (0255)). As detailed in Table [| while extensive, prior benchmarks offer fragmented evaluation. We are the first to con- nect psychological theory with the model's 'ss reasoning process and its ability to generate rationales, thereby providing a unified evaluation framework.
3

---

Published as a conference paper at ICLR 2026
Table 2: Evaluation tasks of our HitEmotion benchmark. "N-CLS" denotes an n-class classi- fication task, and "GEN" represents a generation task. "ACC", "MF", "WAF" and "EMF" denote accuracy, micro F1 score, weighted average Fl score and exact match FI, respectively.
Task
Data Source
Type
iInstances
Metric
Level I: Emotion Perception and Recognition
Face Expression Sentiment Detection (FESD)
CH-SIMS
3-CLS
450
ACC, WAF
Image Sentiment Analysis (ISA)
EmoSet
8-CLS
2,000
ACC, WAF
Meme Sentiment Analysis (MESA)
Memotion
5-CL.
2,000
ACC, WAF
Multimodal Emotion Recognition (MER)
MER2023
6-CLS
400
ACC, WAF
Multimodal Sentiment Analysis (MSA)
CH-SIMSv2
3-CLS
500
ACC, WAF
Opinion Sentiment Analysis (OSA)
CMU-MOSI
3-C1.
ACC, WAF
Sentiment Intensity Analysis (SIA)
CMU-MOSEI
7-CLS
1,000
ACC, WAF
Song Emotion Recognition (SOER)
RAVDESS
6-CLS
500
ACC, WAF
Speech Emotion Recognition (SPER)
RAVDESS
8-CLS
500
ACC, WAF
Stock Comment Emotion Analysis (SCEA)
FMSA-SC
5-CLS
250
ACC, WAR
Level 2: Emotion Understanding and Analysis
Detection Persuasion Techniques Memes (DPTM)
SemEvall-2221 Task
Multi-lubel
550
MF
Emotion-Raaee Based Intent Analysis (EBIA)
MC-BIU
7-&8-CI.
ACC
Humor Understanding (HU)
UR-FUNNY
2-CLS
400
ACC, WAF
Implicit Attribute Value Extraction (IAVE)
ImplicitAVE
N-CLS
2,000
ACC, WAF
Multimodal A Aspeci-Based Semtiment Analysis (MABSA)
Twitte2015/2017
3-CLS
2,000
MI
Multimodal Quintuple Extraction (MQE)
PanoSent
GEN
500
MF
Multimodal Stance Detection (MSD)
MMWTWT
4-CLS
2,000
ACC
Multiparty Dialogue Emotion Recognition (MDER)
MELD
7-CLS
500
ACC. WAF
Level 3: Emotion Cognition and Reasoning
Emotion Elicitation Reasoning (EER)
FilmStim
7-CLS
64
ACC, WAF
Emotion Interpretation (EI)
ElBench
GEN
1,500
LLM
Laughter Reasoning (LR)
SMILE
GEN
500
LLM
Multimodal Emotion Cause Pair Extraction (MECPE)
ECF
GEN
500
MF
Sarcasm Detection (SD)
MUStARD
2-CLS
500
ACC, WAF
Sentiment Flip Analysis (SFA)
PunoSent
GEN
500
EMF
Theory-of-Mind Reasoning. ToM is the capacity to represent and infer others' mental states such beliefs, intentions, and emotions Premack & Woodruff 1978; Baron-Cohen al.) 1985 Decety & Jackson| 2004 Lake et al., 2017). providing a cognitive foundation for affective computing. Psy-
chology formalizes emotion
core ToM dimension (Beaudoin et al. 2020 Chen et al. 2024b)
and shows that tracking mental states crucial for its attribution (Lillard) 1993] Qu et al.
2015).
These insights have inspired inference-time strategies for LLMs that decompose ToM queries into tractable subproblems, such as simulating perspectives and checking knowledge access, thereby im- proving reasoning without extra training (Wei al., 2022, Sarangi et al. 2025 Rahwan al. 2019)
ToM reasoning also extends to temporal. counterfactual and non-literal communication, which are vital for affect interpretation (Byre (017). However, recent benchmarks reveal that even state-of- the-art MILI .MM still lack robust ToM capabilities (ToMBench, (Chen et al.) (024bb); MMToM--A, (Jin et al. 2024)), highlighting key challenge. Consequently, the absence of targeted optimization in current models hinders the acquisition of more robust and complex ToM reasoning.
HITEMOTION BENCHMARK
3.1 TASK TAXONOMY
As shown in Figure we organize our benchmark into three hierarchical levels of emotional in- telligence, each targeting progressively advanced capabilities (see Appendix C for more details). O Emotion Perception and Recognition (EPR) establishes the foundation by evaluating models' ability to perceive and classify explicit emotional states across modalities, mapping multimodal in- puts to predefined categories. @ Emotion Understanding and Analysis (EUA) requires contextual awareness and relational reasoning, emphasizing the interpretation of emotions' functions and in- tents in situational settings. @ Emotion Cognition and Reasoning (ECR) advances to causal and second-orde reasoning, requiring models to explain emotion causes, track temporal dynamics, and interpret nuanced expressions, thereby engaging with the cognitive processes underlying emotions.
4

---

Published as a conference paper at ICLR 2026
Step 1: Reasoning Chain Generation
Enotion- Sased Intent Analysis Audio dlues:
multimodal affective theory mind. Model the utterance as...
grounded
Ite speabor canays anger eyceg)
lawa and FEAM
Highorkceed lnne
video
Speahr's Brenkn
"ufal the clanse to ba ore the bou
Communouine trimt
Conduce and Map Jleb
2: Label-Aligned Filtering Correction
(1) Decsda Mullnnde Caagpi Care
Gerinn
(3) Abribb 2e Spuake's Ersdice
Labes
Inter Comes Wrknnt lovent the Acdewaoe is cusd siredd riect quartorin...
Initiel chote
Geld Angil Guesforirg
Gold
Labdlc
(eection: Sangen
InlkGal
Corrected cheta
Anger
(wnstoer Shasagg
'reeet Inuestoring
cheic
Coraatin:: Ruse Sraeet Vooent Aroeet-liaaannnnn inceatioingg
Gueatoring
Teseet Santtaaptonn
Self-Refiection Normalization
Step Verification
Complety
Reficedd chate
It Standded pnd Presiin
Cumpattiyy
Figure 3: Our reasoning chain curation pipeline.
3.2 BENCIMARR CONSTRUCTION
To curate data and construct our benchmark while reducing annotation costs, we leverage publicly available datasets from the field of multimodal affective computing with task-specific annotations. Building on this foundation, we aggregate and curate 24 publicly available datasets spanning diverse affective domains, including emotion recognition, sentiment analysis, humor understanding, and causal reasoning. As shown in Table these datasets are systematically organized into a three-tiered hierarchy reflecting increasing cognitive complexity: Emotion Perception and Recognition, Emotion Understanding and Analysis, and Emotion Cognition and Reasoning. While preserving the original task names, data structures, and evaluation metrics, we unify all datasets into a standardized closed- label QA format. To ensure the benchmark's 's integrity and reliability, enhancements without altering the source semantics. First, we institute a rigorous quality assurance protocol, wherein stratified sample representing one-third of each dataset undergo dual-annotator cross-review and arbitration process to validate the consistency of "prompt-answer-cont" Second, to prevent data leakage and ensure fair evaluation, we exclusively incorporate the official test splits from each source dataset. This meticulous curation process yields a benchmark with high internal consistency and more uniform label distribution, providing robust and systematic environment for assessing the affective capabilities of MLLMs.
implement two critical
triplets.
4 METHODOL.OGY
This section introduces TMPO, our framework for enhancing emotional understanding in MLLMs. We organize this section into four core components: task definition, ToM based prompting, a super- vised fine-tuning stage, and ToM preference optimization.
4.1 TASK DEFINITION
Our objective to leverage a Multimodal Large Language Model (MLLM) to infer an emotion- related output (o) and the underlying cognitive reasoning chain (r) from multimodal inputs (Text T, Audio A, and Video V). This task can be formally represented as mapping: (T, A, - (r, o). Since ground-truth reasoning chains (r) are unavailable in existing datasets, we construct a gold- standard version to guide model generation. As illustrated Figure 3 this achieved through strict four-step pipeline involving LLM-driven generation, filtering, enhancement, and correction (see Appendix D.4 for for details).
4.2 TOM-STYLE PROMPTING MECHANISM
To elicit the desired reasoning chain т, we utilize a ToM-style prompting mechanism, denoted as task-specific prompt P, to structure the expected output format. Our prompt P is structured across three levels of cognitive complexity to elicit increasingly sophisticated reasoning chains. For concrete examples of these prompts, please refer to Figures 24 47 in Appendix F
5

---

Published as a conference paper at ICLR 2026
Level 1: First-Order Mental State Attribution. Prompts this level guide the model map multimodal cues to an immediate emotional state. This involves synthesizing observable signals into first-order attribution of what the subject feels, while remaining flexible to task-specific modalities like text-image incongruities in memes.
Level 2: Relational & Contextual Mind Modeling. This level requires reasoning about the rela- tionship between an emotional state and its context, such as a specific entity or communicative goal. builds upon Level attributions by contextualizing them, for example, by linking an emotion to specific target in aspect-based sentiment analysis.
Causal Attribution & Second-Ordes Reasoning. The highest level elicits reasoning about the causes of emotions and their social interpretation, involving causal inference and second- order ToM, which involves inferring what others believe about a subject's state. Prompts guide the model to explain why emotion arises detect incongruity between literal and intended meaning, as in sareasm, moving beyond what is felt why felt and how it is meant be interpreted.
Level 3:
4.3 STAGE 1: TOM-ALIGNER SUPERVISED MINE-TUNING
To establish foundational capability for structured reasoning, we first perform SFT on mul- timodal backbone model. The core objective of this stage to teach the model to generate re- sponses that not only produce the correct task-specific output but also articulate the underlying cognitive process a clear, step-by-step manner. We explicitly wrap the intermediate reasoning steps (T) with a <think></thinn
tag and encapsulate the final task-specific output (o) within an fanswer></answer> answer>> tag. This structural disentanglemen forces the model to learn the dis- tinet functions of cognitive deliberation and final conclusion generation. The model is trained on the native multimodal inputs (T', A, V) along with task-specific prompt P. The target for the model is generate the complete structured string y = <think>r</think> <ankk>aanswerrr
ansser>> The fine-tuning objective is to minimize the standard negative log-likelihood loss over our dataset:
LSFT (0) -..((PPTT.AA)))..
To (TT
(1)
where Te is the policy of the MLLM with parameters 0. After this stage, the model acquires preliminary ability to mimic structured, multi-step reasoning patterns from our curated data.
4.4 STAGE 2: TOM-BASED PREFERENCE OPTIMIZATION WITH GRPO
While SFT imparts the basic structure of Too--aligned reasoning, the generated chains may still lack factual grounding, exhibit logical inconsistencies, or fail to generalize robustly across diverse scenarios. To overcome these limitations, we further refine the model using Group-wise Reward Policy Optimization (GRPO) (DeepSeek) (025). which enhances the model's ability to generate reasoning chains that are structurally correct as well cognitively plausible and factually accurate.
The GRPO process begins by sampling N candidate outputs {y1, yN from our current policy for a given prompt, where each y: = <think>>,< wtte/thiniiaett <aw.er>>oowwwooo/o/o.... /eeero////nswer answer>. Each candidate is then evaluated using a custom-designed, cuult--dimensionll reward function R(y). The resulting scores guide the policy update via the GRPO objective:
No (36r)
max E
A
3D (Toll
(2)
Tola (TA)
where A; are the computed normalized advantage scores and the KL-divergence term penalizes deviation from a reference policy aee (typically the initial SFT model) stabilize the optimization process. The advantage scores A; are derived from the relative ranking value of the rewards R(y.) within the sampled group, guiding the model to prefer higher-scoring responses.
4.4.1 REWARD ASSIGNMENT
The comerstone of our GRPO strategy is comprehensive reward function R(y) that decomposes the quality of a response into four distinct, complementary components. This function is formulated weighted sum:
R(v) M1 Ratructure + P2 Roonteee M&g Rprocess pa Reonsistency

---

Published as a conference paper at ICLR 2026
Table 3: Performance on Emotion Perception and Recognition, with ACC as the evaluation metric.
Bold and underlined indicate the best and the worst results among all models, respectively.
Categury Modd
FESD
ISA
MESA
MER
MSA
OSA
SIA
SOER
SPER
SCTA
31:60
18
66.62
42.00
LLaV
HaVA-NeXTA ViT---
41.31
56.11
65.80
S-VL-7
11.25
56.75
61.21
64.20
47.35
InterNN15-000 801
21.00
53,00
65300
03.10)
31,20
Source
Open
Qwea2.5 V-L-225-VL-322
InnreVV..3300 3-30
63.78
57:51
61.21
53.70
57.58
05.19
34.00
25.25
15.15
57.16
57.14
50.15
62.65
65.80
08.80
65,50
08.90
52.45
34,20
12.42
15.7%
46.90
43,40
$3.46
41.30
57.59
48.00)
47.60
$0.80
R1-Ommi-- 5B
42.25
$1.55
23.73
$0.88
41.24
$2.20
19.50
20.12
24.18
43.60
HamanUroii 7g
04.64
53.77
23.82
50.75
18.20
23
33.50
50.31
40.30
47.00
Owee2 5-Oumi-78
64.67
$1.56
$6.00
64.00
65.00
Euanico_LLaMA-T
53.63
34.00
43.75
44.40
47.37
A1tratiP1:11
GPT 40
54,48
30.12
69.20
69.53
49.00
49.96
IoM prompee
74.00 (+3.78)
(-196)
(5.18()3.00)
63:32 (15.8)
78.60
(360015:000
(17.44
41.36
54.31 (+4.1)
(+2.58)
GPT-41
71,45
$6,50
31.43
64.00
72.46
69.50
40.81
65.19
$5.20
$3.20
Closed
TeM pronpe
14.74155 (+3.79)
58.00
(-1.76)
16.55 (+3.17)
66:00
(00:000000
76.08 (+7
74 24.80048877 70)
43.17
(+3.355
69.19
(+300)
57.30 (-3.00)
56.01 (+7.8))
Soony
Cemini-) S-Pash
67.13
55.41
22.12
56.93
68:40
70.91
18.44
$7.47
56.51
$0.81
TeM peooppe
76.44 (19 (19.35)
25.01 (18.00)
29.20 (12.00)
(15.4)
74.84 (15/44
76.00 (5:122)
43.30 (1492)
63:33
(1380
62.22 (15.71)
53.04 (12.21)
Gemini 2.5 Pro
78.39
61.12
28.96
72.11
74.20
75.71
46.55
67.56
05.00
55.00
T&M prooage
79.11 (40.72)
63.33
(-2.81)
31.02 (=2000)
72:92 (4101)
77.97 (+1.27)
79:19
(45 480
(65:21)
69.00
(+1.6)
69.31
(-5:11)
62.25 (47.29)
Owr
TMPO
GRPO
75.41
66333
79.12
72.49
77.03
21.39
$0.20
These components evaluate the reasoning process from different perspectives: the Structure Re- ward (Rateeture) enforces the correct sequence of reasoning steps; the Content Reward (Reontens)
evaluates the final answer's correctness; the Process Reward ( (Rgrocess) encourages domain-specific language; and the Consistency Reward (Reonsistency) penalizes logical and factual inconsistencies. The weights ((+) are calibrated to prioritize correctness and logical grounding, A full description of each component and the rationale for weight assignments are provided in the Appendix D.I
Table 4: Performance Emotion Understanding and Analysis. By default, ACC is used as the evaluation metric, while DPTM, MABSA, and MQE use the MF metric.
Category Model
DPTM
EBIA
HE
TAVE
MARSA
MQR
MSD
MIKR
31.17
14.42
44.39
62.50
61.95
23,67
51.15
42.61
LLaVA-Cho--Vee Vision-7
31.54
11.33
42.25
60.37
63.89
14,072
59.75
33.00
LLaWA-NeXT-Vides
$1.28
12.57
40.50
13.45
24:75
25.25
Swen2.5-VLL-78
11.02
32.50
$2.55
45.60
36.77
14.79
60.13
33.13
Source
Open
2wwm2.5-VL-320
MinCCPM-VV..666- 6B
40.22
30.80
14.62
14.51
19:25
57.50
62.67
55.03
63.29
61,82
27.77
32.38
$2.08
39.85
48.20
43.51
40.55
16.49
59.25
64.74
64,80
32,64
$3.85
46.00
RI 21-Ommi-0.. 0.5B
37.54
13:45
46.25
50.03
58.40
29.58
47.85
29.81
HumanDeell-72
12.55
47.90
36.20
Owee2.5
11.42
55.79
31,09
Favotion LMA-78
39.54
1546
$7.92
$2.08
34.18
AffectGPP1-78
34,17
12:27
$6.50
40.02
60.48
30.95
42.40
37.92
GPT-lo
42.33
17.45
60.00
66.13
61.76
35.32
55.76
49.68
ToM prompe
45,90
(48255)
66.03 (45.63)
71.19 (45:06)
68.30 (63.54)
36.45 (+1.13)
61 (45.25)
53.72 (44.04)
GPT-4.1
47.50
18.62
67.68
37.98
65.76
Clored
ToM prompe
27.65 (3)
72.91
39)
(+2.93)
Source
feeminl-2...Plash
47.18
16.34
64.14
66.71
36,55
+ ToM prompt
56.35 (-9.17)
24.87 (+8.53)
63.74 (+1.08)
72.63 (+8.49)
73.21 (66.50)
38.12(+-1.7)
61.83 (+4.18)
55.56 (H1.15)
Gemini-2 Prr
49.23
19.25
60 30
70.67
67.61
39.23
64 64.95
$2.65
ToM prompt
(-991))
(19.43)
71.83 (12.84)
77.20
(-6.53)
75.43 (62)
(+5.24)
73.79 (18.84)
$8,90(H6.22) (+6.25)
Owrs
(SFT)
GRPO
46,42
32,82
23.11
68.40
64.18
78.16
69.47
45.68
37.24
60.45
61.08
$1.81
EXPERIMENTS
5.1 SETTINGS
We use Qwen2.5-0mni-7B as our base model, trained on 8 x NVIDIA A800 80 GB GPUs. For our reward function, the weights μl, P2, His 144 are set to 0.4, 1.0, 0.1, and 1.0, respectively. During training. videos are sampled into 16 frames. The model first undergoes SFT for two epochs with a learning rate of le-5, followed by our GRPO strategy with a learning rate of 1e-6. For evaluation, we select the checkpoint with the best validation performance and conduct comprehensive assessment on both open-source models (0.5B to 38B parameters) and closed-source models (GPT and Gemini series). Further implementation details are provided in the Appendix D.3
5.2 RESULTS AND ANALYSIS
The experimental results reveal significant limitations in the multimodal emotion analysis capabili- ties of current ML.MMs. As shown in Tables 3 model performance is evaluated across the three
7

---

Published as a conference paper at ICLR 2026
Table 5: Performance on Emotion Cognition and Reasoning. By default, ACC is used as the evaluation metric, while MECPE uses the MF metric and SFA uses the EMF metric.
Category Model
EER
LR
MECPE
SD
SEA
Vide33L
45.31
31.29
39.56
13.09
37.56
13.16
LLaVA-Ome- Vision-7B
46.88
47.40
44.60
10.83
45.20
16.22
LLaVA-Nehh NeX. Video 7B
35.94
46.80
43.10
13.05
46.36
18.42
Qwea2.5-VL-22
40.62
50,53
48.20
15.07
49.00
14.64
InternVL33
50.00
MiniCPM-V.2.6-88
39.68
33,93
50.40
16.44
51.40
21.83
Open
Qwen2.5-VL-32B
54,69
53.40
19.60
55.60
23.79
Source
ItteraLL3338B
50.31
50.67
51.40
19.28
35.80
25.73
R1-Ommi-0.5B
39.67
43.73
43.00
16.13
53.00
19.93
HumanOmmii77B
38.85
47.93
49.40
16.43
Qwen2.5 Samnnii--mmiii--e 7B
51.25
48.67
49.20
13.83
53.40
17.76
Famotion- Paaiion-LLaMA-.a.
42.81
19.02
AffenGPT-7B
43.75
46.27
50,40
10.81
52.73
15.12
GPT-4o
57.81
54.13
55.83
20.93
56.60
25.77
ToM prompt
(+2.01))
64.33 (0.20))
(+10.17)
(+1.55)
64.80 (+8.20)
42.29 (=16.52)
GPT-4.1
60.31
57.67
61.04
26.86
66.20
36.73
Closed
+ ToM prompt
65.86 (+5.55)
69.00 (-11.3))
71.79 (+10.75)
28.11 (+1.25)
68.67 (42.47)
47.75 (+11.02)
Souore
Gemini-2.5.FBe
58.33
54.47
58.20
27.11
61.49
28.02
+ ToM prompt
(45.80)
63.93 (49.46)
66.60 (+8.40)
31.43 (+4.32)
64.10 (+2.61)
45.22 (-17.20)
Gemini-2.5-Pro
66.13
65.13
59.23
33.33
66.61
41.22
+ ToM prompt
71.94 (+5.81)
70.27 (45,14)
68.20 (+8.97)
37,70 (+4.37)
69.00 (+2.39)
52.78 (-11.66)
Ours
TMPO (SFT)
60.10
62.36
59.75
26.33
59.92
40.50
GRPO
73.13
72.27
72.45
39.34
70.13
54.16
Level
Level
Level
Average Score
50
30
VilooL-MAR-1
20
LtaVi- MaWi-Oreellae Vlaion-
Owatt 548-22
Intami
Inlen 2.300
Orend 5-0aak-77
Emvaline LLaMt-77
apeddo
GPE41
Grmael Frash
Gended 3.pea
Minicist4-
Figure 4: Average performance across our HitEmotion benchmark levels. Comparison of 17 multimodal models our HitEmotion benchmark, showing average scores for each level per model.
hierarchical task categories. At the foundational level of EPR, only three of the ten tasks FESD, MSA, and OSA-yield average scores above 60. Even the best-performing model, Gemini-2.5- Pro, achieves 78.39 on FESD, 74.20 on MSA, and 75.71 on OSA, while most other models remain around the 50-point range, reflecting limited robustness. As task complexity increases, performance declines markedly. In EUA level, only two tasks surpass the 60-point threshold. Most critically, within the cognitively demanding ECR level, task achieves performance hierarchy underscores our benchmark's 's ability to differentiate models across distinct levels of reasoning. Taken together, the findings show that current MLLMs possess only rudimentary emotional intelligence and continue to struggle with higher-order emotional reasoning, highlighting an urgent need for advances in both model architectures and training methodologies.
average score above 60. This clear
Closed-Source, Tuned, and Scaled Models Lead in Emotional Intelligence. As shown in Fig- ure 4 proprictary models such as the GPT and Gemini series consistently outperform open-source counterparts, owing to their large parameter scales and extensive pretraining on diverse datasets. Even in the zero-shot setting, Gemini-2.5-Pro achieves 78.39 on the FESD task, while GPT-4.1 reaches 71. 46, both substantially ahead of most open-source models. The Gemini series further surpass GPT in multimodal emotion recognition due to its native capacity for processing video and audio inputs. Nevertheless, open-source models, though constrained by scale, can achieve com- petitive results through task-specific fine-tuning. AmootoonLLLMA-- task, outperforming most untuned baselines. While Emotion-LLaie LLLMA benefits from domain fine-tuning, still lags behind zero-shot proprictary models, indicating that a significant capabil- ity gap persists between existing open-source solutions and top-tier proprietary systems. Likewise,
attains 34.18 on the MQE
8

---

Published as a conference paper at ICLR 2026
Table 6: Ablation study on reward components.
Vedix
CXT
To00bet
T0007
Reward Components
Average Score
Aurring Stare
Rancare
Bessver
Roustuesyy
Rprces
L1
L.2
L3
54,52
$2.
d
62.63
61.07
59.31
65.12
64.03
62.82
56.20
55.10
55.05
65.82
64.70
63.58
GP74
Gronse
Figure 5: Ablation study on ToM-style prompting.
Qwen2.5- V-L33BB achieves 53.40 on the LR task, closely approaching GPT-4o's 55.83. These find- ings show that large-scale pretraining provides the foundation for advanced emotion intelligence, but targeted fine-tuning offers open-source models a practical pathway to close the gap with proprietary systems and achieve broader accessibility.
Effects of ToM Prompting on Emotional Intelligence. Closed-soure source models such as GPT-4.1 and Gemini-2.5-Pro gain clear advantages when ToM prompting is applied, especially on the most challenging tasks. High-capacity open-source models, including Dwenn.55VVL-321 and Intern VL.3- 38B, also benefit, with improvements observed across most tasks and effects becoming more pro- nounced at higher levels. This suggests that models with stronger baseline reasoning are better able to leverage intermediate reasoning chains provided by ToM. By contrast, weaker models don't ' ex- hibit consistent gains and in some cases deteriorate. For instance, VideoLLaMMA3-7 drops from 61.78 to 54.18 on FESD under ToM prompting, and LLLVA-NeXT--VE Video-7B shows minimal im- provement IAVE, increasing only from 40.50 40.81. These outcomes imply that models with insufficient representational and reasoning capacity cannot stably exploit ToM, and are more prone hallucinations that compromise the reasoning chain.
TMPO Unlocks Advanced Reasoning Capabilities. The experimental results consistently demon- strate the remarkable efffcctiveness of our TMPO framework, which provides a substantial perfor- mance uplift to the backbone model across all task categories. Both the SFT stage and the GRPO stage contribute to this success, with GRPO delivering a particularly significant boost in perfor- mance. Crucially, on more complex tasks requiring nuanced reasoning, our fully-optimized model not only closes the gap but often surpasses the performance of top-tier proprietary models, emerging as the top-performing model on 16 of the 24 tasks. This highlights TMPO's exceptional capability in teaching models how to reason. Conversely, for some direct, perception-drins model still lags behind some leading sys-
tasks, such as
inferring emotions mainly from facial expressions, tems. This likely due to the inherent limitations in the base model's s raw multimodal perception capabilities, which the reasoning- focused optimization cannot fully overcome.
5.3 ABLATION STUDIES
ToM-style Prompting. To validate prompt engineering, conduct an ablation study its key design choices. We compare three strategies: (1) CoT, which instructs the model to "please think step-by-step"; (2) Toi-Init, which establishes cognitive reasoning path without specific terminological guidance; and (3) ToM-Full, which enhances ToM-Init by explicitly integrating task- relevant ToM keywords. The results in Figure 5 show that ToM-Init consistently outperforms the generic CoT, confirming the inherent benefit of a ToM-aligned framework over unguided reasoning. In addition, ToM-Full yields a further substantial performance gain over ToM-Int Init, validating that the explicit integration of key ToM concepts is crucial for unlocking the model's full reasoning potential.
Reward Components. To validate our reward function, we conduct complementary ablation study on its individual components. We progressively add each reward to the GRPO objective, with results summarized in Table Using only Rarneture establishes baseline by enforeing coherent format. The most substantial performance gain is observed with the introduction of Reontent underscoring the necessity of directly optimizing for the correct final answer. Furthermore, integrating Reonsistency yields another significant boost, validating its crucial role in eliminating logical fallacies and ground- ing the reasoning. Finally, Rpocess provides a complementary refinement by encouraging the use of Toe--pcccfic terminology. This progressive enhancement demonstrates that all four components work in synergy to produce high-quality, reliable, and cognitively aligned reasoning. Additionally, investigate the necessity of Ratagurre by removing it from the full configuration. This exclusion

---

Published as a conference paper at ICLR 2026
leads to significant performance drop. Without the explicit structural penalty, the model exhibits Format Collapse, failing to adhere to the XML schema required for extracting answers. This con- firms that Attruture acts as the foundational prerequisite that enables the effective optimization of other reward components.
6 CONCLUSION
In this work, we introduce HitEmotion, hierarchical benchmark that systematically diagnoses MLLM's capability breakpoints across increasing cognitive depths. To improve reasoning, we develop TMPO, novel preference optimization method that uses intermediate mental states as process-level supervision. Our experiments confirm that HitEmotion exposes deep reasoning deficits in top-tier models, while TMPO substantially boosts the backbone model's performance. The optimized model surpasses leading proprictary systems on many cognitively demanding tasks by im- proving end-task accuracy, faithfulness, and the coherence of its reasoning. Together, HitEmotion and TMPO form a robust toolkit for evaluating and enhancing cognitive- based affective intelligence. This approach pushes MLLMs beyond superficial recognition toward deeper, more human-like mental state simulation, facilitating the development of more empathetic Al.
7 ETHICS STATEMENT
This work relies exclusively publicly available datasets released by prior publications. We did not collect new suban-subject data, and no personally identifiable information was used. All datasets were used in accordance with their original licenses. No institutional ethics review required. We adhere to the ICLR Code of Ethics.
8 REPRODUCIBILITY STATEMENT
To facilitate reproducibility, we have released all data preprocessing scripts, model training and inference code through an anonymous repository, with the link provided in the Abstract. In addition, we have uploaded the dataset samples used in our experiments, together with detailed configuration files. We further provide comprehensive description our experimental setup, including model architecture, training methodology, and hyperparameter settings in Section 5. and Appendix D.3 These resources ensure that the experimental results this paper can be faithfully reproduced.
ACKNOWLEDGMENTS
This work is supported by the Ministry of Education, Singapore, under its MOE AcRF TIER Grant (MOF-MOET33002-0000)
REFERENCES
Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo, Jiajun Cao, Hao Liang, Ying Chen, Qi She, al. Mc-llava: Multi-concept personalized vision-language model. arXiv 2024.
preprint arXiv:2411.1172
Ruichuan An, Sihan Yang, Renrui Zhang, Zijun Shen, Ming Lu, Gaole Dai, Hao Liang, Ziyu Guo, Shilin Yan, Yulin Luo, et al. Unictokens: Boosting personalized understanding and generation via unified concept tokens. arXiv preprint arXiv:2505. 14671, 2025.
Ruichuan An, Sihan Yang, Ziyu Guo, Wei Dai, Zijun Shen, Haodong Li, Renrui Zhang, Xinyu Wei, Guopeng Li, Wenshan Wu, al. Genius: Generative fluid intelligence evaluation suite. arXiv preprint arXiv:2602.1 144, 2026.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
10

---

Published as a conference paper at ICLR 2026
Simon Alan M Leslie, and Uta Frith. Does the autistic child have "theory of mind"? Cognition, 21(1):37-46, 1985.
Cindy Beaudoin, Élizabel Leblanc, Charlotte Gagner, and Miriam H Beauchamp. Systematic review and inventory of theory of mind measures for young children. Frontiers in psychology, 10:2905, 2020.
Ruth MJ Byrne. Counterfactual thinking: From logic to morality. Current Directions in Psycholog- ical Science, 26(4):314-322) 2017.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. Towards multimodal sarcasm detection (an obviously perfect paper). arXiv 2019.
preprint
Qikai Chang, Zhenrong Zhang, Pengfei Hu, Jun Du, Jiefeng Ma, Yicheng Pan, Jianshu Zhang, Quan Liu, and Jianqing Gao. Thor: Tool-integrated hierarchical optimization via rl for mathematical reasoning. arXiv preprint arXiv:2509.1376. 2025.
Ruirui Chen, Weifeng Jiang, Chengwei Qin, and Cheston Tan. Theory of mind in large language models: Assessment and enhancement. In Proceedings of ACL, pp. 31539-31558, 2025.
Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. Pad: Personalized align- ment at decoding- arXiv e-prints, pp. WXXv-22410, 2024a.
Sheng- Yeh Chen, Chao-Chun Hsu, Chuan-Chun Kuo, Ting-Hao (Kenneth) Huang, and Lun- Wei Ku. Emotionl Line: An emotion corpus of multi-party conversations. arXiv preprint arXiv:1802.08379, 2018. Presented at LREC 2018.
Zhuang Chen, Jincenzi Wu, Jinfeng Zhou, Bosi Wen, Guanqun Bi, Gongyao Jiang, Yaru Cao, Mengting Hu, Yunghwei Lai, Zexuan Xiong, et al. Tombench: Benchmarking theory of mind in large language models. arXiv preprint arXiv:2402. 15052, 2024b.
Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Jingdong Sun, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Emotion-llama: Multimodal emotion recognition and reasoning with instruction tuning, arXiv preprint arXiv:2406.11 2024.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Tox- vaerd, and Nigel Collier. Will-they-wool t-they: A very large dataset for stance detection twitter. arXiv preprint S00i:::0005003888 2020. Accepted at ACL 2020.
Jean Decety and Philip L Jackson. The functional architecture of human empathy. Behavioral and cognitive neuroscience reviews, 300, 2004.
DeepSeek. Deepseek-rl incentivizes reasoning in Ilms through reinforcement learning. Nature, 645 (8081):633- 66866333333666 2025.
Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. Semeval-2021 task 6: Detection of persuasion techniques in texts and images. arXiv preprint arXiv:2105.09288. 2021.
Patrick Donnelly and Alex Prestwich. Identifying sentiment from crowd audio. In 2022 7th International Conference Frontiers of Signal Processing (ICFSP), pp. 64-69. IEEE, 2022.
Hao Fei, Bobo Li, Meng Luo, Qian Liu, Lizi Liao, Fei Li, Min Zhang, Bjöm W Schuller, Mong- Li Lee, and Erik Cambria. Cogmaec' 25: The 1st workshop on affective and empathetic computing. In Proceedings of the 33rd ACM International Conference Multimedia, pp. 14323-14325 2025.
multimodal
11

---

Published as a conference paper at ICLR 2026
Gemini, Rohan Anil, Sebastian Borgeaud, Yean-Baptiss Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Md Kamrul Hasan, Wasifur Rahman, Amir Zadeh, Jianyuan Zhong, Md Iftekhar Tanveer, Louis- Philippe Morency, et al. Ur-funny: A multimodal language dataset for understanding humor. arXiv preprint arviv: 1904.06618, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficinnt transfer learning for nlp. In International conference machine learning, pp. 2790-2799 PMLR, 2019.
He Hu. Yucheng Zhou, Lianzhong You, Hongbo Xu, Qianning Wang, Zheng Lian, Fei Richard Yu, Fei Ma, and Laizhong Cui. Emobench-m. Benchmarking emotional intelligence for multimodal large language models. arXiv preprint 0rXiv:2502.04424. 2025.
Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. Emotionally numb or empathetic? evaluating how Ilms feel using emo- tionbench. arXiv preprint arXiv:2308.03666 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Lee Hyun, Kim Sung-Bin, Seungju Han, Youngjae Yu, and Tac-Hyun Oh. modal dataset for understanding laughter in video with language models. arXiv preprint arXiv:2312.000 12.09/8 2023.
Smile: Multi-
Yunjic Ji, Hao Liu, Bolci He, Xinyan Xiao, Hua Wu, and Yanhua Yu. Diversified multiple instance learning for document-leved multi-aspect sentiment classification. In Proceedings of the 2020 con- ference on empirical methods in natural language processing (EMNLP), pp. 7012-7023, 2020.
Dazhi Jiang, Runguo Wei, Hao Liu, Jintao Wen, Geng Tu, Lin Zheng, and Erik Cambria. A multitask learning framework for multimodal sentiment analysis. In 2021 International conference on data mining workshops (ICDMW), pp. 151-157. IEEE, 2021.
Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B Tenenbaum, and Tianmin Shu. Mmtom-qa: Multimodal theory of mind question answering. arXiv preprint S0xiv:2401.084444 2024.
Zhaolu Kang, Junhao Gong, Jiaxu Yan, Wanke Xia, Yian Wang, Ziwen Wang, Huaxuan Ding, Zhuo Cheng, Wenhao Cao, Zhiyuan Feng, et al. Hssbench: Benchmarking humanities and social sciences ability for multimodal large language models. arXiv preprint vrXiv:2506.03922. 2025.
Smith K. Khare, Victoria Blanes- Vidal, Esmaeil Nadimi, and U. Rajendra Acharya. Emotion recognition and artificial intelligence: A systematic review (2014-2023) and research recommen- dations. Inf. Fusion, 102: 102019, 2024.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and bruin sciences, 40:e253, 2017.
Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park. and Kwanghoon Sohn. Context-aware emotion recognition networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2019.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 2024a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. next-ieeeiit Tackling maulti-maaee, video, and 3d large multimodal models. arXiv preprint arXiv:2407.07895, 2024b.
12

---

Published as a conference paper at ICLR 2026
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye Gan, Ding Qi, Chaoyou Fu, et al. A survey on benchmarks of multimodal large language models. arXiv preprint 088.022200..8663, 2024c.
Ya Li, Jianhua Tao, Linlin Chao, Wei Bao, and Yazhu Liu, CHEAVD: chinese natural emotional visiol database. Journal of Ambient Intelligence and Humanized Computing, 8:913-924, 2017. doi: 011777111262201--406666
Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning. In Proceedings of the 31st ACM international conference on multimedia, pp. 9610- 9614, 2023.
Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. ICML, 2025.
Bin Liang, Ang Li, Jingqian Zhao, Lin Gui, Min Yang. Yue Yu, Kam-Fai Wong, and Ruifeng Xu. Multi-modal stance detection: New datasets and model. arXiv preprint arXiv:2402. 14298, 2024.
Junfeng Liao, Qizhou Wang. Shanshan Ye, Xin Yu, Ling Chen, and Zhen Fang. Explainable Ilm unlearning through reasoning. In The Fourteenth International Conference on Learning Repre- sentations, 2026.
Angeline S Lillard. Pretend play skills and the child's theory of mind. Child development, 64(2): 348-371, 1993.
Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wen- tao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain, caption, and segment anything in images and videos, 2025a. URL 05302
Yuxiang Lin, Jingdong Sun, Zhi-Qi Cheng, Jue Wang, Haomin Liang, Zebang Cheng, Yifei Dong, Jun- Yan He, Xiaojiang Peng, and Xian-Sheng Hua. Why we feel: Breaking boundaries in emo- tional reasoning with multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1196-52001 2025b.
Rui Liu, Haolin Zuo Zheng Lian, Xiaofen Xing, Björn W Schuller, and Haizhou Li. Emotion and intent joint understanding in multimodal conversation: A benchmarking dataset. arXiv preprint arXiv:2407.0275. 2024.
Steven R Livingstone and Frank A Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamicc multimodal set of facial and vocal expressions in north american english. PloS one, 13(5)ee016339,, 2018.
Huaishao Luo, Lei Ji, Yanyong Huang, Bin Wang. Shenggong Ji, and Tianrui Li. Scalevlad: Improv- ing multimodal sentiment analysis via multi-scale fusion of locally descriptors. arXiv preprint arXiv:21 2021.
Meng Luo, Hao Fei, Bobo Li, Shengqiong Wu, Qian Liu, Soujanya Poria, Erik Cambria, Mong- Li Lee, and Wynne Hsu. Panosent: A panoptic sextuple extraction benchmark for multimodal conversational aspect-based sentiment analysis. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 7667-7676, 2024a.
Meng Luo, Han Zhang, Shengqiong Wu, Bobo Li, Hong Han, and Hao Fei. Nus-emo at semeval- 2024 task 3: Instruction tuning Ilm for multimodal emotion-cause analysis in conversations. arXiv preprint arXiv:2501.1726) 2024b.
Meng Luo, Shengqiong Wu, Liqiang Jing, Tianjie Ju, Li Zheng, Jinxiang Lai, Tianlong Wu, Xinya Du, Jian Li, Siyuan Yan, et al. Dr. A hierarchical pnaceptionteemporal-cooo frame- work to diagnose video hallucination by fine-grained spatial-temporal grounding. arXiv preprint arXiv:2509. / 1866, 2025.
13

---

Published as a conference paper at ICLR 2026
Sijie Mai, Ying Zeng, and Haifeng Hu. Learning by comparing: Boosting multimodal affective computing through ordinal learning. In Proceedings of the ACM on Web Conference 2025, pp. 2120-2134, 2025.
Shreyash Mishra, Suryavardan, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, Asif Ekbal, and Srijan Kumar. Memotion Dataset on sentiment and emotion analysis of codemixed hindi-english memes, 2023. URL
OpenAl. Gpt-4.1 system card. httt:s://opeeaa.com.
4-1 2025.
Samuel Paech. Eq-bench: An emotional intelligence benchmark for large language models. arXiv preprint arXiv:2312.06281/ 2023a.
Samuel Paech. Eq-bench: An emotional intelligence benchmark for large language models,
2023b. URL httph:t//axxiii
htt////s/////// 2312. 06281
Josef Perner and Heinz Wimmer. "john thinks that mary thinks that... attribution of second-order beliefs by 5- to 10-year-old children. Journal of Experimental Child Psychology, 39(3):337-477 1985.
Rosalind W Picard. Affective computing. MIT press, 2000.
Soujamya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv: 1810.02508, 2018.
David Premack and Guy Woodruff. Does the chimpanzee have theory of mind? Behavioral and brain sciences, 144):515-526, 1978.
Li Qu, Pinxiu Shen, Yu Yan Chee, and Luxi Chen. Teachers' theory-of-minee coaching and children's executive function predict the training effect of sociodramatic play on children's theory of mind. Social Development, (6(4):716-733 2015.
Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean- Can-Fraaççis Bonnefon, Cynthia Breazeal, Jacob W Crandall, Nicholas A Christakis, lain Couzin, Matthew Jackson, et al. Machine behaviour. Nature, 568(7753))477 2019.
Geovany A Ramirez, Tadas Baltrušaitis, and Louis-Philippe Morency. Modeling latent discrimi- native dynamic of multi-dimensional affective signals. In International Conference Affective Computing and Intelligent Interuction, pp. 396-406. Springer, 2011.
Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, Alvionna S. Sunaryo, Juanzi Li, Tatia M. C. Lee, Rada Mihalcea, and Minlie Huang. Emobench: Evaluating the emo- tional intelligence of large language models, 2024.
12071
Sncheel Sarangi, Maha Elgarf, and Hanan Salam. Decompose-ton tom:: Enhancing theory of mind reasoning in large language models through simulation and task decomposition. arXiv preprint arXiv:2501.09055 2025.
Alexandre Schaefer, Frédéric Nils, Xavier Sanchez, and Pierre Philippot. Assessing the effectiveness of a large database of emotion -eleitiineiiiiingg films: A new tool for emotion researchers. Cognition and emotion, 24(7): 152-1172, 2010.
Lingyun Song, Siyu Chen, Ziyang Meng, Mingxuan Sun, and Xuequn Shang. Fmsa-sc: A fine- grained multimodal sentiment analysis dataset based on stock comment videos. IEEE Transac- tions on Multimedia, 26:7294-7306 2024a.
Shezheng Song, Chengxiang He, Shasha Li, Shan Zhao, Chengyu Wang, Tianwei Yan, Xiaopeng Li, Qian Wan, Jun Ma, Jie Yu, and Xiaoguang Mao. Mosabench: Multi-object sentiment analysis benchmark for evaluating multimodal large language models understanding of complex image, 2024b. URL URLhttps:: www.vv.vvvvvv/.////vv//aa///.vvv www.oorgaaaooo abs// 2412. 00060
14

---

Published as a conference paper at ICLR 2026
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for computational linguistics. Meeting, volume 2019, pp. 6558, 2019.
Fanfan Wang, Zixiang Ding, Rui Xia, Zhaoyu Li, and Jianfei Yu. Multimodal emotion-cause pair extraction in conversations. IEEE Transactions on Affective Computing, 14(3):1832-1844, 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-2444 2022.
H. Wimmer and J. Perner. Beliefs about beliefs: representation and constraining function of wrong beliefs in young children's understanding of deception. Cognition, 13(1):103-28,, 1983.
Bohao Xing, Xin Liu, Guoying Zhao, Chengyu Liu, Xiaolan Fu, and Heikki Kälviäinen. Emotion- hallucer: Evaluating emotion hallucinations in multimodal large language models. arXiv preprint arXiv:2505. / 1405, 2025.
Jin Xu, Zhifang Guo. Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang. Bin Zhang, Xiong Wang. Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215. 2025. URL
/ /ttp:p//////vi..oo//a//////////
20215
Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Emoset: A large-scale visual emotion dataset with rich attributes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 20383-20394 2023.
Qu Yang. Mang Ye, and Bo Du. Emollm: Multimodal emotional understanding meets large language models. CoRR, abs/2406.16442. 2024. URL
wro.:.oorg/11 48550/
arXiv.2406. 16442
Xiaocui Yang. Wenfang Wu, Shi Feng, Ming Wang, Daling Wang, Yang Li, Qi Sun, Yifei Zhang, Xiaoming Fu, and Soujanya Poria. Mmmiisttructeall: Zero-shot evaluation of (multimodal) large language models on multimodal reasoning tasks, 2025. URL htttps://arxiss org/ abs/
2405. 27229
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.01800 2024.
Jianfei Yu and Jing Jiang. Adapting BERT for target-oriented multimodal sentiment classification. In Sarit Kraus (ed.), Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 5408-5414. ijcai.org, 2019. doi: 1149663/jcai.20199777. URL http://o...oo /tt.///o.oo/oo////oo//ooo/ooo 10. 249 /ttp:///...aa1111/...... 2019/75
Tan Yu, Jingjing Wang, Jiamin Luo, Jiawen Wang, and Guodong Zhou. Tacl: A trusted action- enhanced curriculum learning approach to multimodal affective computing. Neurocomputing, 120992 2025.
Wenmeng Yu, Hua Xu, Fanyang Meng, Yilin Zhu, Yixiao Ma, Jiele Wu, Jiyun Zou, and Kaicheng Yang- Ch-sims: A chinese multimodal sentiment analysis dataset with fine-grained annotation of modality. In Proceedings of the 58th annual meeting of the association for computational linguisticx, pp. 3718-3727, 2020.
Zhenlong Yuan, Xiangyan Qu, Chengxuan Qian, Rui Chen, Jing Tang, Lei Sun, Xiangxiang Chu, Dapeng Zhang, Yiwei Wang, Yujun Cai, et al. Video-star: Reinforcing open-vocabulary action recognition with tools. arXiv preprint arXiv:2510.084. 2025a.
Zhenlong Yuan, Jing Tang, Jinguo Luo, Rui Chen, Chengxuan Qian, Lei Sun, Xiangxiang Chu, Yujum Cai, Dapeng Zhang, and Shuo Li. Autodrive-r2: Incentivizing reasoning and self-reflection capacity for vla model in autonomous driving. arXiv preprint arX1v:2509.01944, 2025b.
15

---

Published as a conference paper at ICLR 2026
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal cor- pus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint (7Xx:::606600259, 2016.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computaaionall Linguis- 2018.
tics (Volume 1: Long Papers), pp.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Vidcollama 3: Frontier multimodal foundation models for image and video under- standing. arXiv preprint arXiv:2501. 13106 2025a.
Fan Zhang, Zebang Cheng, Chong Deng. Haoxuan Li, Zheng Lian, Qian Chen, Huadai Liu, Wen Wang, Yi-Fan Zhang, Renrui Zhang. Ziyu Guo, Zhihong Zhu, Hao Wu, Haixin Wang, Yefeng Zheng, Xiaojiang Peng, Xian Wu, Kun Wang, Xiangang Li, Jieping Ye, and Pheng-Ann Heng. Mme-emotion: A holistic evaluation benchmark for emotional intelligence in multimodal large language models, 2025b. Utt.ttsss::/ //arx v. org/ abs 2508. 09210
Fan Zhang, Zebang Cheng, Chong Deng, Haoxuan Li, Zheng Lian, Qian Chen, Huadai Liu, Wen Wang, Yi-Fan Zhang, Renrui Zhang, et al. Mme-emotion: holistic evaluation benchmark for emotional intelligence in multimodal large language models. arXiv preprint arXiv:2508.0m 092100 2025c.
Han Zhang, Zixiang Meng, Meng Luo, Hong Han, Lizi Liao, Erik Cambria, and Hao Fei. Towards multimodal empathetic response generation: A rich tet--peech--iiooooe avatar-based benchmark. In Proceedings of the ACM on Web Conference 2025, pp. 2872-2881, 2025d.
Jaxxng Zhao, Xihan Wei, and Liefeng Bo. Rl-omni: Explainable omni-multinoal emotion recog- nition with reinforcement learning. arXiv preprini arXiv:2503.05379, 2025a. URL https:
7/ //rxxiv.org//bb//2333333 05379
Jaxing Zhao, Qize Yang. Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Weixuan Chen, Xihan Wei, and Liefeng Bo. Humanomni: large vision-speech language model for human-centrie video understanding. arXiv preprini arXiv:2501.15111 2025b.
Pengcheng Zheng, Chaoning Zhang, Jiarong Mo, GuoHui Li, Jiaquan Zhang, Jiahao Zhang, Sihan Cao, Sheng Zheng, Caiyan Qin, Guoqing Wang, et al. Llava-fa: Learning fourier approximation for compressing large multimodal models. arXiv preprint arXiv:2602. 00.000033 35, 2026.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10477 2025.
Zhihong Zhu, Xuxin Cheng, Guimin Hu, Yaowei Li, Zhiqi Huang, and Yuexian Zou. Towards multi- modal sarcasm detection via disentangled multi- grained multi-modal distilling. In Proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (LREC- COLLCC 2024), pp. 16581-16591 2024a.
Zhihong Zhu, Xianwei Zhuang, Yunyan Zhang, Derong Xu, Guimin Hu, Xian Wu, and Yefeng Zheng. Tfed: Towards multi-modal sarcasm detection via training-free counterfactual debiasing. In Proc. of IJCAI, 2024b.
Henry Peng Zou, Vinay Samuel, Yue Zhou, Weizhi Zhang, Liancheng Fang, Zihe Song, Philip S Yu, and Cornelaa Caragea. Implicitave: An open-source dataset and multimodal lims benchmark for implicit attribute value extraction. arXiv preprint arXiv:2404.15592, 2024.
Shihao Zou, Xianying Huang, and Xudong Shen. Multimodal prompt transformer with hybrid contrastive learning for emotion recognition in conversation. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 5994-6003, 2023.
16

---

Published as a conference paper at ICLR 2026
APPENDIX OVERVIEW
Appendix outlines how LLMs are utilized in the paper.
Appendix B
summarizes the limitations and future work of this work.
Appendix presents the taxonomy of the tasks and dataset details.
Appendix provides additional implementation details.
Appendix presents the extended experimental results.
Appendix describes the design of ToM-style prompts.
Appendix presents representative samples from each dataset used.
Appendix presents representative case studies from multiple perspectives.
A THE USE OF LLMS
In this work, LLMs are employed as an auxiliary tool for language editing. Their use is limited grammar correction, refinement of sentence structure, and improvements textual fluency and readability. LLMs are not used for any substantive aspects of the research, including the design of methodology, execution of experiments, data analysis, or interpretation of results.
B LIMITATIONS AND FUTURE WORK
While our TMPO framework demonstrates significant advancements in multimodal emotion reason- ing, we acknowledge certain limitations that outline directions for future research.
Scope of Applicability. Our framework is grounded Theory of Mind (ToM), which refers to the ability to attribute mental states-suc as emotions, intentions, and beliefs-tt to oneself and others. In essence, ToM involves "putting yourself someone else's shoes" to infer hidden mental states. This stands in stark contrast to domains like mathematics, coding, and logical puzzles, where well- defined ground-trute answers are readily available, enabling objective verification. Social reasoning, however, is characterized by its information-asyminn aasmmmmtiee nature and increased uncertainty, where ob-
jective answers are not easily obtainable (Chang et al. 2025) Yuan et al. 20225b a Liao et al. 2026 Zheng et al. 2026 An et al. 2026 Lin al. 2025a An et al. 2025 2024). Consequently, TMPO is tailored specifically for these social complexity challenges. holds strong promise for broader Social Intelligence domains (e.g., negotiation, intent analysis) that rely the same underlying cog- nitive mechanisms.
Base Model and Modality Constraints. Our choice of the 7B parameter backbone was driven by the necessity for native omni-modal processing Mideo+AAdio+Texxl as audio is critical for emotion perception. Currently, few open-source models larger than 7B support native audio-visual integra- tion. While TMPO significantly boosts cognitive reasoning (Level 2 & 3 tasks), the performance on direct perception tasks (Level 1) remains bounded by the inherent sensory quality of the base encoders. As larger cmnn-modaa models become available, we anticipate that scaling up TMPO will yield further gains, leveraging the stronger reasoning priors of large-scale backbones.
Computational Efficiency. Despite using compact 7B model, our approach achieves performance competitive with proprietary systems (e.g., Gemini-2.5-pro). This highlights the efficiency of our method: by optimizing the reasoning process via RL, we extract maximal cognitive intelligence from a lightweight architecture, offering a practical solution for resource-constrann deployment.
C TASK TAXONOMY
Level 1: Emotion Perception and Recognition. This level forms the foundation emotion intelli- gence; its core function is to directly identify and classify explicit emotional states across modalities. This layer evaluates MILMs' ability to accurately extract and integrate emotional information from multimodal inputs and map it predefined emotion categories. This capability is a prerequisite for higeer-level competencies. Specific evaluation tasks draw on several specialized datasets. For Facial Expression Sentiment Detection, CH-SIMS Yu al. 2020) provides Chinese video clips
17

---

Published as a conference paper at ICLR 2026
with fine-grained multimodal annotations to assess models' capacity for integrated perception of visual and linguistic emotions in realistic scenarios. EmoSet (Ynng et al. (023) is a large-seale image-sentiment dataset that focuses on recognizing emotions from static visual content. To capture internet-specific expressions, Meme Sentiment Analysis employs the Memotion dataset (Mishral et al.] 2023), which contains large collection of memes annotated for sentiment and humor and challenges models' ability to comprehend text-image interplay. The Multimodal Emotion Recogni- tion task uses MER2023 (Lian et al., 2023) to evaluate models' generalization in broader multimodal contexts. Opinion Sentiment Analysis uses CMU-MOSI (Zadeh et al. 2016), a corpus of monologue videos that targets sentiment polarity from speech and facial expressions. Emotion Intensity Analy- sis extends this by using the larger CMU-MOSEI dataset (Zadeh et al. 2018), which requires models not only to identify emotion categories but also to quantify their intensity. Song and Speech Emotion Recognition employ RAVDESS (Livingstone & Russo 2018), comprising speech and song clips by professional actors with matched lexical content and emotions presented at varying intensities. Fi- nally, to assess domain-spectie
performance, Stock Comment Emotion Analysis uses FMSA-SC Song et al. 2024a) to analyze emotions in financial-domain comments.
Level 2: Emotion Understanding and Analysis. This level constitutes advanced layer of emo- tion intelligence, which goes beyond basic classification, requiring models to analyze emotions within complex contexts. This capability entails not only identifying emotions but also modeling their complexity and interpreting their function and intent in specific situations. Accordingly, mod- els must exhibit robust contextual awareness and relational reasoning. To evaluate these abilities, this layer incorporates several challenging tasks. For internet culture, Detection of Persuasion Tech- niques in Memes employs the SemEval-2021 Task dataset (Dimitrov et al.) 202T) to identify per- suasive intent in memes. Emotion-Bas Based Intent Analysis uses the MC-EIU dataset (Liu et al. 2024) to examine links between emotional expression and users' underlying intent. Humor Understanding employs UR-FUNNY (Hasan et al., 2019), corpus of TED-talk clips that requires integrating lin- guistic, visual, and acoustic cues to determine whether content is humorous. Implicit Attribute Value Extraction uses ImplicitAVE (Zou et al., 2024), in which attribute values are not stated explicitly. Multimodal Stance Detection leverages the MWTWT datasets (Liang et al., (024).. The Multimodal Quintuple Extraction task, based the PanoSent dataset (Luo al.) 2024a), aims to parse five core elements of -oodiment-holder, target, aspect, opinion and sentiment. Multimodal Aspect- Based Sentiment Analysis uses Twiitte2015/2017 (Yu & Jiang] 2019) to evaluate models' ability to identify fine-grained sentiment toward specific entities or aspects in text and images. Lastly, to approximate real-world social interaction, Multiparty Dialogue Emotion Recognition uses MELD (Poria et al., 2018). a corpus of multiparty conversational clips from Friends, and requires tracking the emotional dynamics of each character in pelti- interactions.
Level 3: Emotion Cognition and Reasoning, This level constitutes the highest tier of emotion intelligence, which requires models not only to perceive and understand emotions but also to reason about their causal relationships, temporal dynamics, and underlying cognitive processes. This level approximates a computational account of human emotional cognition, encompassing tasks such as explaining emotion causes, predicting consequent behaviors, and interpreting complex expressions. Evaluation at this layer focuses on models' cognitive and reasoning abilities. Emotion Elicitation Reasoning uses FilmStim (Schaefer et al. 2010) to assess whether models can infer emotions likely to be elicited in audiences from film-clip content. Emotion Interpretation leverages ElBench (Lin al. 2025b), requiring models to explain the deeper meaning and motivation behind emotional pressions. Laughter Reasoning uses the SMILE dataset (Hyun et al.) 2023). which requires models to explain the specific reason for person's laughter in video, demanding nuanced understanding of social context. Multimodal Emotion Cause Pair Extraction employs the ECF dataset (Wang et al., 2022), focusing on precisely identifying the event or cause that leads to a particular emotion from multimodal signals. Sarcasm detection uses the MUSTARD dataset (Castro et al.) 2019). which contains sarcastic dialogue clips from TV shows. Models must integrate contextual, prosodic, and facial cues to identify the incongruity between an utterance's 's literal meaning and its intended mean- ing. Finally, Sentiment Flip Analysis also uses the PanoSent (Luo et al. (024a), requiring models to detect shifts in emotional state during conversation and to identify the key causes of such flips.
18

---

Published as a conference paper at ICLR 2026
C.I DATASET DETAILS
We benchmark a total of 22 publicly available multimodal affective computing datasets, which to- gether constitute 24 distinct tasks. The following section details each dataset included in our bench- mark, and representative samples are provided in Appendix G
CH-SIMS: CH-SIMS is constructed from 60 Chinese movies, TVV series, and variety shows involving 474 unique speakers. The dataset comprises 2,281 curated video clips with an average duration of 3.67 seconds. Its key characteristic is the provision of both unimodal and multimodal annotations across text, audio, and vision under in-the-wild conditions. Labels are assigned to five sentiment categories: negative, weakly negative, neutral, weakly positive, and positive.
CH-SIMSV2: CH-SIMS v2.0 extends CH-SIMS with Mandarin video data from films, TV, talk shows, interviews, vlogs, and other sources. It offers 4,402 supervised seg- ments (2,281 relabeled and 2,121 new) alongside 10.161 unsupervised clips. Designed for text-acoustic-visual analysis, it employs 720p+ sources, active speaker detection, and strict modality separation. Labels comprise unimodal and multimodal scores mapped to five sentiment categories ranging from negative to positive.
keywords from social media and artistic image platforms. It comprises 3.3 million images, of which 118.102 are carefully annotated by humans. Distinguished by attribute diversity, the dataset records brightness, colorfulness, scene type, object class, facial expression, and human action. Labels follow Mikels' model, encompassing eight categories: amusement, awe, contentment, excitement, anger, disgust, fear, and sadness.
EmoSet:
EmoSet is curated using 810 emotion-relatee
Memotion is compiled from Reddit and Google Images through automated crawling and enriched with OCR text via Google Vision. The dataset consists of 10,000 Hinglish memes, divided into 8,500 for training, 1,500 for validation, and 1,500 for testing, with annotations verified by bilingual raters. distinctively integrates multimodal content with code-mixed language, providing sentiment labels (positive, neutral, negative), four emotion types (humorous, sarcastic, offensive, motivational), and graded intensity levels.
Memotion:
MER2023: MER 2023 extends CHEAVD (Li et al. 2017) by automatically collecting expression-centri video clips and releasing rigorously curated splits for community bench- marking. The corpus comprises 3,373 Train & Val samples and three test Aartitins-MERR MULTI, NOISE and MER-SEM SEMIIamountin to about 68 hours of audiovisual data. Emphasizing robustness, it provides three tracks (multi-label learning, modality- ro- bustness, and supi-supe learning) and supplies annotations for six discrete emotions (neutral, anger, happiness, sadness, worry, surprise) together with a continuous valence dimension.
CMU-MOSI: MOSI is multimodal opinion-level corpus for sentiment intensity and sub- jectivity in online vlogs. It comprises 93 videos (89 speakers), 3,702 segments, and 2,199 opinion clips labeled on a -3...+3 scale by five AMT raters. Releases include word- and mhoneme-aligned transcripts, millisecond acoustic features, frame-level visual cues, and gesture tags, with fine-grained subjectivity boundaries and high inter-annotator agreement. Baselines demonstrate that multimodal fusion especilyy a word-gesture "multimodal dictionary" ootperrfrrrts text-only models.
CMU-MOSEL: CMU-MOSEI is one of the largest multimodal sentiment analysis corpora, derived from 3,228 YouTube videos featuring about 1,000 speakers across 250 topics. offers 23,453 sentence-level segments with synchronized text, audio, and visual modalities. Distinguished by its scale and fine-grained alignment, the dataset facilitates cross-modal learning. Labels cover 7-point sentiment scale from -3 (strongly negative) to +3 (strongly positive), supporting both polarity and intensity prediction.
RAVDESS: RAVDESS a validated multimodal corpus of speech and song by 24 profes- sional actors in North American English. Speech covers neutral/calm, happy, sad, angry, fearful, surprise, and disgust, while song includes neutral/calm, happy, sad, angry, and fearful, each at two intensity levels. The 7,356 recordings available in audio-visual, audio-only, and video-only formats. Each clip is rated 10 times by 247 raters, demonstrat- ing strong validity and test-retest reliability.
19

---

Published as a conference paper at ICLR 2026
FMSA-SC: FMSA-SC consists of 1,247 stock comment videos. Its novelty lies in offering ine-grained sentiment anmotations, aligning textual phrases with corresponding visual and acoustic cues. Labels span five sentiment levels from strong negative to strong positive, establishing the first multimodal benchmark for financial sentiment analysis.
SemEval-2021 Task 6: SemFval-2021 Task 6 introduces a multimodal benchmark for detecting persuasion techniques in memes collected from 26 public Facebook groups. The dataset comprises 950 memes, divided into 687 training, 63 development, and 200 testing samples. Its novelty lies in addressing propaganda in multimodal context, offering three subtasks on text, spans, and complete memes, Annotations cover 22 persuasion techniques, encompassing both textual and visual strategies.
MC-EIU: The MC-EIU dataset offers large-scale open-source resource for multimodal emotion and intent understanding in conversation. It includes 4,970 clips with 56,012 utter- ances from English and Mandarin TV series, totaling 53 hours of dialogue. Distinguished by its bilingual coverage and tri-modal design (text, audio, and video), it provides seven emotion and nine intent categories, establishing the first comprehensive benchmark for joint affective analysis.
UR-FUNNY: UR-FUNNY is derived from TED talks, using transcripts and laughter mark- ers to label punchlines and contexts. The corpus covers 1,866 videos with 1,741 speakers and 417 topics, containing 8,257 humorous and 8,257 non-humorous instances. It features tri-modal alignment (text, audio, vision), ppeaker-independee chronization to support robust multimodal modeling, with balanced negatives sampled from the same videos. Tags: humor detection, multimodal language, TED, punchline-context modeling, laughter cues, speech-vision- fusion.
splits, and word-level syn-
ImplicitAVE: ImplicitAVE is the first publicly available multimodal dataset for implicit attribute value extraction in e-commerce. It includes 68,604 training and 1,610 human- verified test instances across five domains and 25 attributes. Its novelty lies in curating implicit values absent from text but inferable from images or context, supplemented with product photos and rigorous human re-annotation. Labels span 158 attribute values across clothing, footwear, jewelry, food, and home products.
Twitter2015/2017: The Twitter2015 and Twitter2017 datasets are benchmark corpora for target-oriented multimodal sentiment classification. Together they comprise over 5,000 tweets paired with images, annotated for sentiment polarity toward specific opinion targets. Their main contribution enabling fine-grained alignment between textual and visual con- tent to model sentiment at the target level. Labels span three categories-positive, negative, and neutral supttrll--pppiiinng multimodal sentiment research.
PanoSent: The PanoSent dataset establishes large-scale benchmark for Multimodal Con- versational Aspect- based Sentiment Analysis. It comprises 10,000 dialogues and over 47,000 sextuples across English, Chinese, and Spanish, integrating text, image, audio, and video. Its novelty lies in panoptic sentiment sextuple extraction and dynamic sentiment flipping analysis, capturing holders, targets, aspects, opinions, sentiments, and rationales. Annotated through both human experts and GPT-4 synthesis, it supports multi-scenario, and implicit sentiment reasoning, with labels covering fine-grained and causal dynamics.
The MWTWT dataset originates from the Multi-modal Stance Detection project. It extends the textual Hill-They-Wont-TTe a multimodal form by incorporating both tweets and images. The dataset comprises 1,747 annotated examples focused on corporate merger debates, where each instance is labeled as Support, Refute, Comment, or Unrelated. Its uniqueness lies in capturing stance expres- sion across text-image pairs, enabling research on multimodal opinion dynamics. Labels highlight nuanced stance categories relevant to corporate decision-making
MWTWT:
dataset (Conforti al.) 2020) into
MELD: The MELD dataset extends the Emotionl corpus (Chen et al., 2018) into a multimodal benchmark for emotion recognition in conversations. contains over 13,000 utterances from 1, 1,433 dialogues in the TV series Friends, each annotated with emotion and sentiment labels across audio, visual, and textual modalities. By emphasizing multi-party interactions, MELD captures complex phenomena such as emotion shifts and inter- speakes dependencies, providing a challenging resource for multimodal conversational emotion recognition. Tags: multimodal, emotion, conversation, multi-party.
20

---

Published as a conference paper at ICLR 2026
FilmStim:
The FilmStim dataset is developed to provide validated collection of emotion- eliciting film excerpts for experimental research. It comprises 70 clips selected through expert surveys and validated on 364 participants, offering rich tool for controlled emotion induction. Its distinctive feature lies in covering both basic emotions and mixed feelings, with validated criteria including arousal, valence, and emotional discreteness. Labels span anger, fear, sadness, disgust, amusement, tenderness, and neutral states, making it a com- prehensive benchmark for affective studies.
The ElBench dataset is constructed from CAER-S (Lee et al. 2019) and EmoSet to advance the task of Emotion Interpretation, which asks why an emotion arises rather than merely identifying which emotion is present. It contains 1,615 basic samples and 50 com- plex cases, requiring models to generate causal explanations across explicit and implicit triggers. Its key contribution is the Coarse-to-Fiis Self-Ask (CFSA) annotation pipeline, which combines Vision-Language Models with human refinement capture nuanced, context-dependent emotional reasoning. Labels span four primary emotions-ange anngr., sad, happy, and excited-with complex subsets featuring overlapping emotional states.
ElBench:
SMILE: The SMILE dataset curated from TED talks and sitcoms to explore the task of Video Laugh Reasoning. It comprises 887 clips with 4,434 annotated segments, each paired with textual explanations of why laughter occurs. Its unique focus lies in audience laughter, reducing subjectivity and highlighting multimodal cues across visual, acoustic, and semantic channels. Labels provide free-form explanations rather than fixed classes, enabling deeper analysis of social intelligence.
ECF: The ECF dataset is constructed from the sitcom Friends to support the task of Mul- timodal Emotion-Caus Pair Extraction in Conversations. It contains 1,344 conversations with 13,509 utterances and 9,272 annotated emotion-cause pairs. Its distinguishing fea- ture lies in integrating text, audio, and video modalities to capture diverse causal triggers, categorized as events, opinions, emotional influence, and greetings. Emotion labels follow Ekman's six basic categories:
anger, disgust, fear, joy, sadness, and surprise.
The MUStARD dataset is constructed from TV shows such Friends, The Big Bang Theory, The Golden Girls, and Sarcasmaholics Anonymous to advance multi- modal sarcasm detection. It comprises 690 balanced video clips evenly divided between sarcastic and non-sarcastic utterances, each paired with transcripts and conversational con- text. Its distinctive contribution lies in integrating text, audio, and visual cues with dialogue history, enabling nuanced analysis of incongruity across modalities. Labels are binary: sar- castic versus non-sarcastic.
MUSIARD:
D IMPLEMENTATION DETAILS
D.1 REWARD COMPONENT DETAILSS
The weights for the reward function are set based on a principled hierarchy reflecting each component's importance. We /42 1.0 (content) and '4 1.0 (con- sistency) to assign the highest priority to the foundational requirements of correctness and logical- factual grounding. A moderate weight of µ = 0.4 (structure) ensures compliance with the reason- ing format without overriding correctness. Lastly, a minimal weight of μuر 0.1 (process) serves as a gentle stylistic nudge, guiding the model towards Nomain-specife language while mitigating the risk of superficial keyword stuffing. The four components of our comprehensive reward function are detailed as follows:
Weight Assignment Rationale.
Structure Reward (Astrruturee:: This reward fosters the generalization of structured rea- soning by enforcing the unique cognitive framework required for each training task. The reward system is task- it first identifies the task from the input prompt to select the corresponding reasoning template. The reward Rauuctrre is then calculated as the proportion of required step headers that are correctly present and sequenced within the reasoning chain T.
Content Reward (Reonten):: This reward evaluates the correctness of the final output o by comparing against the ground-truth label using the standard evaluation metric appropriate
21

---

Published as a conference paper at ICLR 2026
for each task's specific format. factually accurate conclusion.
This ensures the model's reasoning ultimately leads to
Process Reward (prrcesss: This reward promotes the articulation reasoning using ToM- specific language. We curate a lexicon of ToM-related keywords (e.g., "belief," "intention," "desire"). Rprcess calculated the normalized count of unique keywords from this lexicon found within This encourages the model not just to follow a structural template, but to fill with rich language reflecting genuine cognitive reasoning.
Consistency Reward ( (Reonsistency): ): To penalize logical fallacies, this reward assesses the consistency of the reasoning chain т. We employ a large language model to detect two types of inconsistencies: (1) Internal Contradictions, where the chain contradicts itself, and (2) External Contradictions, where the chain describes a fact inconsistent with the input multimodal context. Reonsistency is a penalty-basee reward, yielding high value (1.0) for consistent chains and significantly lower value (0.1) if any contradictions found.
The computational formulas for the four reward components are defined follows:
Structure Reward (Rstructure). This reward calculates the proportion of required structural ele- ments that are correctly present and sequenced. Let Seg be the ordered sequence comprising the mandatory XML delimiters and the task-specific step headers {1a} derived from the prompt:
Sne = [<think>,
lx, </think>, <answer>, </answer>)
(4)
Let idx(s, y) denote the index of token s y. We define validity indicator is E (0,1) for the i-th token in Sreq:
(xx; v) xo A idx(s;, v) > max((idx(sj)) v) 1 1, 1)u(-1}] (5)
This recursive condition strictly enforces the topological order. Let N |Stegl be the total number of required elements. The reward is the proportion:
N
Rstructure (y)
Σ
(6)
N
1=1
Content Reward (Reontent). This evaluates the correctness using the standard metric Mussk (e.g.. Accuracy, FI) comparing the extracted answer to the ground truth o":
Reontent (y) = MMuskk (o, o²)
Process Reward (Rproces). Consistent with the description of a normalized count, let VIeM be the ToM lexicon and S, be the set of unique tokens in r. We normalization factor yy:
Rymieess (v) = min 1.0, SSt VroM
(8)
"
Consistency Reward (Reonsistency) ) To rigorously enforce logical soundness, we employ an LLM Judge to detect inconsistencies, the Judge evaluates two logical predicates:
Jin(T): Returns True if the reasoning chain is free of internal contradictions.
Jox(T, Input): Returns True if the reasoning chain is consistent with the inputs (T, A, V).
The final reward applies a penalty if either condition fails (i.e., if any contradiction is found);
1.0
if (T) ^ (eu Input)
Reunsistency (v)
(9)
0.1
otherwise
22

---

Published as a conference paper at ICLR 2026
D.2 HYPERPARAMETER SENSITIVITY ANALYSIS
To empirically validate the optimality of our reward weight configuration (f) = 0.4, μ22 - 1.0, p43 0.1, /14 1.0), conducted fine-grained grid search. We report the Average Score (mean of L1, 1.2, and L3 tasks) to quantify the optimization objective.
Sensitivity to Process Reward (Aproces). are shown in Table
We fixed Mstrict 0.4 and varied Pprocess
The results
Table 7: Ablation study on Process Reward Weight.
Pprocess
0.0
0.1
0.3
0.5
Score
63.99
64.70
63.65
62.31
The configuration μ 0.0 serves as the baseline without stylistic constraints. Introducing minimal weight (μ 0.1) yields the optimal performance. However, increasing the weight beyond this point (u 2 0.3) leads to degradation in reasoning quality, as the model prioritizes keyword frequency over logical correctness.
Sensitivity to Structure Reward (Harnee).. We fixed Mprocess = 0.1 and varied Pittrct- illustrates the impact structural constraints.
Table
Table 8: Ablation on Structure Reward Weight.
Mstruct 0.1
0.4
0.7
1.0
Score 61.25
64.70
64.10
63,80
At µ 0.1, the penalty is insufficient to enforce the XML schema, leading Format Collapse and parsing failures. Conversely, high weights (μx 2 0.7) cause Structural Rigidity, where the model strictly adheres to templates the cost of the reasoning flexibility required for complex multimodal inputs, resulting in diminished accuracy.
D.3 EXPERIMENT SETTINGS
During inference, we allow 16-64 frames with a resolution up to 256 28 x 28 pixels per frame. Training hyperparameters for both SFT and GRPO stages, including learning rate, scheduler, batch size, and rollout settings, are summarized in Table 9 The model uses a context window of 8,192 tokens and maximum generation length of 4,096. The closed-source models included in our eval- uation are accessed through their official APIs.
We evaluate total of 17 MILMS,, comprising 4 closed-source and 13 open-source models. A brief introduction to each model is provided below:
VideoLLLaMA3 (Zhang et al. 2025a) is the third- hiiddggeneration VideoLLaMA series designed for both image and video understanding. It introduces flexible resolution tokenization and efficient frame pruning to reduce redundancy while preserving temporal context. With progressive training pipeline, VideoLLaMA3 achieves strong performance diverse video reasoning and description benchmarks, particularly in long-horizon and fine-grained temporal comprehension.
-nn---One-Viion (Li et al. 2024a) is a unified vision-language model built to handle images, documents, and charts under one interface. Its transfer framework distills knowl- edge from multiple pretrained encoders into single instruction- -uuned backbone. LLaVA- One-Vision enables broad task coverage and practical deployment real-world multimodal applications, ranging from general VQA to structured document analysis.
LLaVA-NeXT-Vidde. (Li et al, 2024b) extends the LLaVA-NeXT series to video under- standing. It employs interleaved frame encoding with preference-optimizzd alignment to
23

---

Published as a conference paper at ICLR 2026
Table 9: Key hyperparameters for the SFT and GRPO training stages.
Hyperparameter
SFT Stage
GRPO Stage
Base Model
Twen2.5-Omni-e
SFT-tuned Model
Learning Rate
1.0 10-5
1.0 x 10-6 Constant
LR Scheduler
Cosine
Warmup Ratio
N/A
Epochs
2
1
Batch Size
16
8
Precision
b
b
Rollout Samples (N)
N/A
8
KI. Coefficient
N/A
0.001
enhance temporal reasoning and dialogue quality. LLaVA-NcXT: Video proves effective for video QA and conversational analysis in long-horizon scenarios, delivering more coherent and context-aware responses.
Qwen2.5-VL. (Bai et al. 2025) is lasion-language model family developed by Alibaba. combines dynamic-resolution processing, fine-grained localization, and progressive align- ment to support documents, diagrams, and long videos. Qwen2.5-VL. delivers reliable perception and reasoning across diverse multimodal benchmarks, excelling tasks that require detailed and structured visual understanding.
InternVL3 (Zhu et al. 2025) is the third-generation InternVL. family, integrating stronger vision encoders with Qwen2.5 backbones. introduces efficient token reduction and im- proved preference optimization for reasoning- tasks. InternVL3 achieves notable improvements OCR, document analysis, and complex visual understanding, offering a more balanced trade-off between efficiency and accuracy.
MiniCPM-V Yao et al. 2024) a lightweight multimodal LLM optimized for on-device use, including phones and edge platforms. With efficient visual encoding, multilingual tuning, and system-level optimizations, it supports privacy-presevvigg and eeffccecffiiennt interaction. MiniCPM-V enables practical multimodal deployment resource-constrained environments, making advanced perception and reasoning accessible on everyday devices.
R1-Omni (Zhao et al. 2025a) is an omni-modal model focused on emotion reasoning. Building on HumanOmni, integrates reinforcement learning with verifiable rewards to enhance interpretability. R1-Omni generates step-by-step explanations that clarify how vi- sual and acoustic cues shape predictions, leading improved generalization in challenging emotional tasks.
HumanOmni (Zhao et al., 2025b) is human-centric mmni-multimodal model trained for emotion and interaction understanding. It employs dedicated perception branches for faces, bodies, and interactions, fused with audio signals. HumanOmni excels human-related applications such emotion recognition and social behavior analysis, enabling more fine- grained comprehension of real-world scenarios.
Qwen2.5-Omni (Xu et al., 2025) is a flagship omnimodal model that unifies text, im- ages, audio, and video while generating both text and speech. Its Thinker-Talker architec- ture and efficient streaming design support real-time interaction. Qwen2.5-Omni enables ppeech-in/speechhon multimodal assistants for continuous audiovisual tasks, combining responsiveness with versatile cross-modal reasoning.
Emotion-LLaMA (Cheng et al. (024)) is a multimodal model tailored for affective com- puting. It fuses audio, visual, and text encoders through two-stage training pipeline for recognition and explanation. Emotion-LLaMA advances emotion-awand understanding across diverse modalities, supporting both accurate recognition and interpretable rationale generation.
AffectGPT (Lian et al., 2025) is a multimodal emotion model that introduces MER- Caption, the largest fine-graine grained emotion dataset gathered via novel model- crowd- sourcing strategy. It also embeds pre-fusion operations for enhanced cross-modal align- ment and proposes MER-UniBench, unified evaluation benchmark tailored to natural
24

---

Published as a conference paper at ICLR 2026
language emotion understanding. AffectGPT optimizes emotion-aware reasoning and de- scriptive understanding in multimodal LLMs.
GPT-4o (Hurst et al., 2024) is one of OpenAT's latest multimodal large language models, offering APIs that can seamlessly handle text, vision, and audio. It shows strong perfor- mance across numerous benchmarks, with notable progress in perception, comprehension, and multimodal reasoning. Built on a unified design that supports smooth cross- moss- in- tegration, GPT-44 is efficient and versatile, making it well-suited for practical multimodal applications.
GPT-4.1 (OpenAl, 2025) a recently released multimodal model that emphasizes both cost-effectiveness and reliability. It enhances programming capabilities and instruction following, while also introducing an extended context window of up to one million tokens. With this improvement, GPT-4. is able to deliver more robust long-context reasoning and significantly improve task efficiency.
Gemini-2.5-Flase (Comanici et al. 2025) is a multimodal reasoning model designed to balance speed, performance, and resource usage. introduces selective reasoning modes, enabling users to trade off accuracy and efficiency depending on their needs. With its fine- grained control of reasoning steps, Gemini-2.5-Fii achieves competitive results across broad range of multimodal understanding benchmarks.
Gemini-2.5-Pro (Comanici et al.) 2025 is the flagship model in the Gemini family, advanc- ing multimodal understanding with stronger perception and reasoning. It supports longer contexts and delivers improved mrosl- alignment, while excelling in domains such as programming, mathematics, and scientific analysis. Equipped with more capable reasoning abilities, Gemini-2.5-Pro is optimized for demanding. knowledge-intensive tasks.
D.4 TRAINING DATA GENERATION
Our training data generation methodology follows multi-stage pipeline designed to produce high- fidelity reasoning chains. The pipeline integrates the generative capacity of advanced MLLMs with automated filtering and human-in- -o-t-loo verification, ensuring both efficiency and reliability. comprises four key stages, as outlined below.
Step 1: Reasoning Chain Generation. For each sample, we generate initial reasoning pathways by providing GPT-5 with the video input and our tailored prompt, augmented by an auxiliary report from Qwen2-Audio that identifies and supplies additional information present in the soundtrack. This module extracts salient audio cues such crying, laughter, changes in tone, speech rate, pauses, emphasis and stress, or voice trembling -which are integrated with the visual and textual context according to the task. GPT-5 then produces three independent candidate reasoning chains from these multimodal inputs.
Step 2: Label-Aligne Filtering & Correction. The generated reasoning chains are automatically evaluated by comparing their predicted labels against the ground-truth answers. Based on this com- parison, each sample is categorized into of three groups. If least one of the three candidate chains produces a final output that exactly matches the ground truth, it is preserved as correct and advanced the next stage. If all three chains fail, the sample is classified as completely incorrect and flagged for intensive correction. In multi-label tasks, chains that correctly predict part of the labels but miss others are deemed partially incorrect; for these cases, the full original reasoning path together with the gold-standard labels are provided to the correction model, which is instructed to revise only the erroneous parts while keeping the valid portions intact.
Step 3: Self-Refiection Normalization. The correction process is carried out using Gemini-2.5. pro, which generates either fully new reasoning chain for completely incorrect samples or targeted revisions for partially incorrect ones. Once all corrected and initially correct chains consolidated, they undergo a final self-reflection step with GPT-4o. this phase, the model standardizes format- ting, ensures logical clarity, and refines the articulation of ToM concepts, resulting in coherent and high-quality reasoning outputs.
Step 4: Human Verification. As the final stage, the complete set of refined reasoning chains undergoes numan-in-the-loog verification. Two computer science PhD students manually review each chain with the primary goal of cross-validating the reasoning against the source multimodal
25

---

Published as a conference paper at ICLR 2026
information. If any factual inaccuracies, logical inconsistencies, misinterpretations of the visual context are detected, the annotators intervene to edit and finalize the chain, thereby ensuring its correctness and reliability.
Our data generation pipeline yields a two-stage training corpus tailored for reasoning alignment. First, in the SFT stage, we collect approximately 10,000 high-quality prompt-response pairs to bootstrap the model's output format, reasoning style, and baseline behavior. Then, in the GRPO stage, we select another 10,000 prompt instances emphasizing tasks with complex reasoning struc- ture and high diversity; these prompts serve as seeds for multiple rollouts and pairwise preference comparisons to train a policy aligned to human judgments.
D.5 EVALUATION SPECIFICATIONS
Our evaluation framework is designed to rigorously assess model performance across spectrum of emotion-related tasks. We employ a set four primary metrics: ACC (Aceuracy). WAF (Weighted Average Fl-score), MF (Micro Fl score), and EMF (Exact Match F1). The evaluation is stratified into three hierarchical levels, with the complexity of tasks and the sophistication of metrics increas- ing each level.
Level 1: Emotion Perception and Recognition. This foundational level focuses on the direct perception and classification of emotional and sentimental states from various data modalities. encompasses 10 tasks: FESD, ISA, MESA, MER, MSA, OSA, SIA, SOER, SPER, and SCEA. These tasks are measured using ACC and WAF.
ACC offers direct measure of overall correctness by calculating the ratio of correct pre- dictions to the total number of samples.
Number of Correct Predictions
ACC
(10)
Total Number of Samples
WAF addresses class imbalance by computing the F1 score for each class and averaging them, weighted by the number of true instances per class ((S.)). This yields more bal- anced assessment, especially when certain emotion labels are underrepresented. For a set of classes C' and total sample size of [S), is defined as:
|ccl
WAF =
Σ
X
FL.
(11)
S
rEC
where the F1 score for an individual class, Fla, is the harmonic mean of its precision and recall:
Precision x Recall,
Flc = 2 x
(12)
Precision + Recall,
Level 2: Emotion Understanding and Analysis. This intermediate level requires deeper ana- lytical capability, moving from simple recognition to understanding context and implicit attributes. It includes eight tasks: DPTM, EBIA, HU, IAVE, MABSA, MQE, MSD, MDER. For the classifi- cation tasks at this level, we continue to utilize ACC and WAF. Additionally, to provide a holistic performance view on more granular tasks, also the MF score.
MF assesses performance by aggregating the counts of true positives (TP), false positives (FP), and false negatives (FN) across all classes before computing the final score. This makes it equivalent to overall accuracy in single-label classification but provides robust metric for more complex scenarios.
Eeec TPc
Precision,
(13)
(Te FP.) FP..)
Eneo TP. Recall,, EcEc (TP. FN.)
(14)
Precision, Recall,
MF = 2 x Precision, + Recall,,
(15)
26

---

Published as a conference paper at ICLR 2026
Level 3: Emotion Cognition and Reasoning. This highest level probes the model's ability to perform complex reasoning and generate human-like explanations. comprises six tasks: EER, EI. LR, MECPE, SD, and SFA. The evaluation methodology this level is diversified to match the task requirements. For classification-orieneed tasks like SD, we continue to employ ACC and WAF. For tasks that demand the generation of free-form text, we also two specialized evaluation strategies: EMF for answers with a high degree of expected lexical overlap, and sophisticated LLM-based evaluation for open-ended, creative responses.
EMF designed for generative tasks where the desired output is specific, factual ex- planation, such as in LR. It quantifies the word-level overlap between the predicted and ground-truth texts after normalization. The texts are treated as bag of words, and the score is computed based on the common words. Let Wordsprod and Words be the set of words in the prediction and ground truth, respectively.
Worddpped N Wordsgr
Precision
(16)
WordSpred
WordSpred O Wordsgl
Recallword
(17)
Wordsgr
EMF Precisionword Recallwor
(18)
Precisionwad
Recallwor
LLM-based Semantic Evaluation is employed for open-ended tasks like EI, where multi- ple, distinct answers can be valid and word-level overlap metrics like EMF are inadequate. In this paradigm, we leverage GPT-4.1 as a semantic judge. The LLM prompted to com- pare the meaning of the generated response against the ground-truth answer(s), assessing its semantic relevance, plausibility, and correctness. This approach transcends word-level matching to capture the true quality of nuanced and diverse generative outputs.
In practice, some models fail to strictly follow the required output format due to differences in ciiructtionfoolowinn ability. In such cases, we also employ GPT-4.1 to normalize and extract the intended answers, ensuring consistent and fair evaluation across all models.
27

---

Published as a conference paper at ICLR 2026
E EXTENDED EXPERIMENTS RESULTS
In this section, we extend our evaluation across all three hierarchical levels and introduce metrics beyond accuracy for more in-depth analysis. We report results in two parts: Tables 10 12 tablish baseline performance of vanilla models, while Tables [3] show the effects of integrating provides per-task radar visualizations that directly contrast each representative model with vs. without ToM prompting across all benchmark tasks, revealing heterogeneous impacts--uubsaantal improvements on some tasks and occasional regressions on others. In addition, we provide task-level comparisons on labeled tasks, where fine- grained F1 scores across n emotion categories are reported for all MILLMs; visual summaries appear We also include confusion matrices for Gemini-2.5-Pa (Figures 22 and 23)
ToM prompting. Complementing these tables, Figure [6)
in Figures N 21
LAAAS Ved 76
MEEP
NOTP
6A
AAA
S7C
103
100
36E
MEC
BOLL
MBC
MAM
SIPA
351
Figure 6: Per-task radar performance for representative models. Each polar chart groups tasks by benchmark level and juxtaposes vanilla model vs. ToM-prompted results.
E. TASK-LEVEL PERFORMANCE CHARACTERISTICS OF CURRENT MLLMS.
100
Level
Level 2
Level 3
60
20
FESD
MESA
M12
914
sozR
NCEd
nens
EnIa
no
142
MAnSA
Msp
MDrR
E2R
1B
MRCPE
SPA
Figure 7: Score distributions by task and level. For each task, the left box corresponds to the vanilla model and the right box to the ToM Prompting.
Figure 7 summarizes the performance of MLLMs across tasks spanning three hierarchical levels. Current models perform relatively well on explicit emotion recognition: for example, Gemini-2.5- Pro scores 78.39 on FESD, and GPT-4.11 reaches 71.46, demonstrating that large-scale MILMSs can classify direct emotional cues with reasonable accuracy. However, performance declines sharply on tasks requiring implicit inference or complex contextual reasoning. In EBIA, even Gemini-2.5-PP attains only 19.25, underscoring the difficulty of intent recognition. Likewise, in structured extrac- tion tasks such MQE and MECPE, most models achieved MF scores below 30. These results reveal persistent weaknesses in handling implicit cues, causal reasoning, structured extraction, and higher-level pragmatic understanding such dialogue dynamics, sarcasm, and humor.
28

---

Published as a conference paper at ICLR 2026
E.2 VERIFICATION OF LLM-BASED EVALUATION RELIABILITY
To validate the reliability of GPT-4.1 I in our evaluation pipeline, we consider two representative sce- narios. First, for free-form generation tasks in Level 3, we randomly sample one-third of the dataset and compare GPT-4. I's judgments with human annotations. Agreement is measured using accuracy and Cohen's Kappa. GPT-4.1 achieves 98.5% agreement with Cohen's Kappa of 0.98, demonstrat- ing near-human reliability. Second, for classification-styy tasks where models often fail to follow the required output format, we employ GPT-4.1 normalize predictions and extract the intended labels. On a stratified sample of 2,000 such cases, GPT-4.1's extracted labels match human inter- pretations with 96.5% agreement and a Cohen's Kappa 0.96. These results confirm that GPT-4.1 provides a consistent and trustworthy mechanism both for semantic judgment and for standardizing model outputs, ensuring fair and reliable evaluation across diverse task types.
Table 10: Performance on Emotion Perception and Recognition using vanilla model. Bold and underlined indicate the best and the worst results among all models, respectively.
FRSD
ISA
MRSA
MER
MSA
OSA
SIA
SOFR
SPER
SCEA
ACC:
WAT
ACC
WAT
ACC
WAI
AFC
WAT
ADC
WAI
ACC:
WAI
AEX:
WAF
AFT:
WAF
AET:
AFT:
WAF
Opens:
Scatie Midi!!
ValeoLLMM33-1 7B
61.51
52.18
$1.91
41.03
43.50
36.02
42.00
LLaN Que Visia--8 TB
63.64
64.84
49.19
48.3
12.05
16.35
11.50
36.55
63:40
66.57
62. (..00
66.74
27.00
25.30
$3.40
46.04
44.50
34.04
34.80
37.07
N-W---NTT 78
54.44
53.53
41 23
37.4
3,63
41.31
33
56.11
61.65
65.90
T0.E3
25.08
30.25
43.00
46.00
43.40
40.54
11.20
32:12
63.50
07.73
40.00
Mirst 6-2.-311
57.53
01.12
09.39
47.08
18.85
30.00
62.00
52.45
62.15
57.42
14.11
46.90
15.39
30.84
45.21
38.93
Qweals-vv..e 5-1--22
63:77
07:18
33.00
32.13
24-28
22.85
57.16
57.00
63.00
16.00
54.20
27.38
45.40
39.66
41.80
11.13
41.00
4123
65.37
71.82
31.92
53.45
48.18
50.9
04.94
00.05
53.77
33.81
23.82
24.97
68.31
09.67
46.54
35.90
14.35
30.31
40074
46.20
31.31
41.69
31.33
Queat 3-Ommm-111
64.07
05.13
31
32.85
22.71
23.35
56.10
56.87
14.00
18.
68.00
16.18
$2.30
28.05
56.72
40.10
44.50
34,00
48:00
49.35
45:44
Shure-S Soart MAde!
CFF-1
71.00
23:25
10.1
66.64
5.30
55.35
56.34
Comini Faaad
67:11
2007
41.11
51.00
25.2
55
00.6
68
07 18
T191
76
18.44
76.55
17.47
16.11
17.37
10.11
17.47
Centni-....Po
78.09
61.12
61.29
72.11
72.78
7631
7966
665
43.9
67.06
65.00
63,8
5548
11.40
Table 11: Performance on Emotion Understanding and Analysis using vanilla model.
DPTM MF
EBIA
HU
LAVE
MABSA
MQE MF
MSD
MDER
Method
ACC
ACC
WAF
ACC
WAF
MF
ACC
ACC
WAF
Open-Source Model
31.17
14.42
44.89
34,80
62.50
61.46
61.96
23.67
51.15
42.61
40.71
LLaVA-One Misn-Viiion-8
31.54
11.33
42.25
29.96
60.37
58.70
63.89
14.02
39.75
33.00
26.52
JLaVA-NeXi VeXTe-e---d
31.28
12.37
43.50
39.54
40.50
35.38
59.40
13.45
44.75
25.25
19.08
Qwen2.5-VL-7B
31.41
54.25
54.19
64.49
63.16
64.65
32.59
52.55
45.60
44.18
IntemVL33-8B
36.77
14.79
53.50
53.25
60.13
57.03
63.11
33.13
50.90
39.48
34,80
MiniCPM-V-2.6-8B
30.80
14.51
49.25
47.32
55.03
54.04
61.82
27.77
39.85
43.51
41.38
Qwen2.5-VL--22B
57.50
57.43
62.67
62.20
63.29
32.38
52.08
48.64
In33mVL3-38B
40.55
16.49
59.25
59.19
64.74
63.82
64.80
32.64
53.85
46.00
44.88
R R1-Ommi-0.5B
37.54
13.45
46.25
45.41
50.03
50.92
58.40
29.58
47.85
29.81
28.24
HumanOmni-7B
35.59
12.55
49.50
49.50
53.50
53.67
59.89
32.98
47.90
36.20
33.30
Qwen2.5.0mni
$3.00
$5.79
53.85
61.93
44.65
35.09
LLotion-LLaMA-TB
39.54
15.46
57.92
57.92
52.08
52.04
60.13
34.18
44.15
47.59
48.04
AffeaGPT-7B
34.17
12.27
56.50
56.47
40.07
37.99
60.48
30.95
42.40
37.92
37.28
Closed-Sowree Model
GPT-4o
42.33
17.45
60.00
59.92
66.13
65.87
64.76
35.32
55.76
49.68
50.64
GPT-4.1
70.19
70.17
67.68
67.78
70.81
37.98
65.76
53.82
Gemini-2.5-Flassh
47.18
16.34
64.66
64.38
64.14
64.45
66.71
36.55
57.65
51.41
53.07
Gemini-2.5.PM.
49.23
19.25
69.39
66.98
70.67
70.99
67.61
39.23
64.95
$2.65
53.59
29

---

Published as a conference paper at ICLR 2026
Table 12: Performance on Emotion Cognition and Reasoning using vanilla model.
EER
EI
LR
MECPE
SFA
Method
ACC
WAF
LLM
LLM
MF
ACC
WAF
EMF
Open- Source Model
Videol.LaMA3-7B
45.31
37.85
31.29
39.56
13.09
37.56
36.90
13.16
LLaVA-One- -ision-7B
46.88
42.43
47.40
44.60
10.83
45.20
40.37
16.22
LLaVA-NeXTI-vido-777
35.94
28.33
46.80
43.10
13.05
46.36
36.37
18.42
Qwen2.5-VL-7B
40.62
29.14
50.53
48.20
15.07
49.00
45.43
14.64
Intern
50.00
44.72
47.00
46.40
16.41
51.40
51.37
17.61
MiniCPM-V-2.6-8B
39.68
34.54
33.93
50.40
16.44
51.40
39.60
21.83
Qwen2.5-VL-32B
54.69
51.41
53.40
53.40
19.60
55.60
45.73
23.79
ViterWVL3-3BB
50.31
46.63
50.67
51.40
19.28
55.80
55.58
25.73
R1-Omni-0.55
39.67
38.65
43.73
43.00
16.13
53.00
52.91
19.93
HumanOmni-78
38.85
30.76
47.93
28.40
13.19
49.40
49.15
16.43
Qwen2.5 Omni-mnniii-
51.25
51.98
48.67
49.20
13,83
53.40
53.25
17.76
AinotionLLLaMA-1
42.81
40.53
49.53
53.00
19.28
52.60
52.45
19.02
AffectGPT-7B
43.75
45.75
46.27
50.40
10.81
52.73
52.64
15.12
Closed-Suurce Model
GPTT4o
57.81
59.48
54.13
55.83
20.93
56.60
53.61
25.77
GPT-4.1
60.31
65.49
57.67
61.04
26.86
66.20
65.09
36.73
Ge5ini-2.5-Flash
58.33
60.29
54.47
58.20
27.11
61.49
59.76
28.02
Gemini-2.5-Pro
66.13
65.41
65.13
59.23
33.33
66.61
66.62
41.22
Table 13: Performance on Emotion Perception and Recognition with proposed ToM prompting.
FESD
ISA
MESA
MER
MIS
OSA
SIA
SOER
SPER
SCEA
(lpee-Suare Suarr MAGB!
LLaNA-One Viaaa-185
$2.55
57.03
26.01
46.41
22.85
22.33
62.66
42.00
56.56
66.71
69.00
15.89
32.15
29.15
15.25
2003
40-40
38.00
$3.16
51.83
55
63.84
33
70 12
90
15
50.10
59 00
53.71
56.08
39.00
76
37.11
35.15
80.39
78.45
31.30
11.11
Quend V-17-7
04.71
33.90
65.6
12.30
42.19
13.06
76.79
19:70
MiwIE3PA--V--AA 6-14-111
48.00
51.33
25.88
63.08
45.5
68.5
70/74
12.00
39.51
30:17
20.32
14.00
41.03
60.07
77.33
91.17
70.90
27.79
09:00
72.33
75.23
18.63
16.31
47.18
47.11
48.30
41.15
18.15
48.41
29.88
19.08
16:31
17.76
11.80
Murosttenii-1
07.00
61.21
07:97
55.85
61:40
85
11.04
10.11
1045
19.00
10.12
15-11
Qwe11 6-mmmi-11
69.27
01:11
29.61
07
24.21
22
16.55
16
68.31
71
61.00
05
13.10
12.41
15.97
11.09
13.35
31.79
13:40
41.76
1.MA-111
00:31
31.21
18.22
06.00
11.50
18.30
11.08
76:35
Chunel Scurc Mali)
GFT-11
43.17
69.19
57.20
5601
0.5ini Fash
76.44
74.75
59
37.83
21.3
25.86
69.19
65.0
74564
14.97
76.05
77.54
43.56
43.33
63.35
63.00
62.33
64.06
$3.04
$3.72
29.11
79.42
03.13
66.80
31.02
24
72.2
7306
77.97
7X.11
79.19
MUS5
51.74
51.15
69.00
69.99
69.31
6943
63.15
61.45
30

---

Published as a conference paper at ICLR 2026
Table 14: Performance on Emotion Understanding and Analysis with proposed ToM prompting.
Method
DPTM
EBIA
II
LAVE
MABSA
MQE
MSD
MDER
MF
ACC
ACC
WAF
ACC
WAF
MF
MF
ACC
ACC
WAF
Open- Seen-Souree Model
VidooLLMMA3-78
26.75
12.42
56.00
53.07
66.41
65.15
61.37
22.86
43.60
34.49
31.83
24.72
10.21
51.01
47.85
64.44
62.80
60.66
20.18
39.15
34.90
31.62
LLaVA-NeXT-Video-7
25.89
14.53
54.50
53.84
40.81
38.32
61.32
24.18
43.12
32.58
31.66
Qwen2.5-VL-77H
20.12
12.27
54.25
46.41
44.17
41.84
64.30
22.79
46.90
41,30
40.16
IntemVLL-888
29.76
12.88
56.75
53.00
62.08
59.09
66.67
22.42
44.00
38.00
34.01
MiniCPM-V-2..-8BB
29.36
56.75
70.26
61.42
22.63
38.15
38.28
Qwen2.5-V1.-22h
41.20
16.12
65.25
64,04
75.46
74.33
68.37
33.56
59.95
49.64
49.17
IntemVLL3-38B
40.28
19.08
64.75
61.07
67.45
66.05
66.50
34.21
55.65
48.83
43,30
R1-Ommi-0.5B
14.57
46.43
55.74
56.76
56.29
22.24
42.78
31.95
28.42
HumanOmmi-7B
26.14
12.89
55.64
54.24
57.21
57.75
66.39
22.61
42.49
36.13
Qwen2.5-0mmi-71
25.67
11.20
49.75
34.08
57.23
55.08
64.54
25.38
46.60
35.55
32,56
Emotion-LLaMA-7B
25.25
10,91
50.00
33.33
32.51
25.84
58.52
22.37
39.51
39.06
32.11
AffectPP1:7B
52.31
63.29
34.94
Closed
Model
GPT-4o
45.90
25.70
66.63
66.51
71.19
70.92
68.30
36.45
61.04
53.72
55.24
GPT-411
49.47
27.65
78.00
77.68
72.91
72.75
77.70
40.91
66.18
57.85
59.83
Gemini-2.5-Flade
56.35
24.87
65.74
65.57
72.63
72.01
73.21
38.
61.83
55.56
57.36
59.21
28.68
71.83
71.42
77.20
77.78
75.43
44.47
73.79
58.90
60.13
Table 15: Performance on Emotion Cognition and Reasoning with proposed ToM prompting.
EER
EI
LR
MECPE
SD
SFA
Method
ACC
WAF
LLM
LLM
MF
ACC
WAF
EMF
Open-Sun Source Model
Vid33LL
50.00
43.30
55.00
51.80
15.18
48.88
32.09
18.29
LLaVA-One- -isionn--
40.62
37.90
52.00
50.60
12.42
50.10
33.67
19.22
LLaVA-NeXT-Video-75
28.81
25.97
36.67
51.28
10.37
53.60
53.36
20.33
Qwen2.5-VI--7B
48.44
43.90
55.53
59.59
19.30
50.00
44.91
18.33
Intern VL.3-8B
54.69
51.70
62.73
57.56
15.32
51.70
38.03
30.54
MiniCPM-V-2.6-8BB
39.68
34.41
60.13
56.80
17.46
48.35
40.16
19.99
Qwen2.5-VL--22
58.42
52.10
57.16
64.21
22.14
61.53
53.24
34.91
Intern VL3-38B
55.50
51.33
55.32
57.13
23.33
64.72
57.54
33.61
R1-Omni-0.5B
41.67
43.23
30.59
60.72
15.02
56.68
56.21
21.22
HumanOmmi-77
37.50
36.90
49.07
43.09
17.62
51.60
38.61
22.94
Qwen2.5 Omni-7B
40.62
35.36
50.07
63.60
13.65
46.20
42.36
30.65
Amotion-LLaMA-..
38.71
44.02
38.73
59.80
12.38
49.60
35.47
31.07
AffectGPT-7B
32.26
37.83
46.73
60.08
12.83
53.51
52.50
40.49
Mloxed-Sourel Model
GPT-4o
60.00
63.41
64.33
66.00
22.48
64.80
62.49
42,29
GPT-4.1
65.86
77.98
69.00
71.79
28.11
68.67
68.63
47.75
Gemini-2, 2.5mii2..-Flash
64.13
75.71
63.93
66.60
31.43
64.10
64.08
45.22
Gemini-2.5-Pros
71.94
75.73
70.27
68.20
37.70
69.00
68.63
52.78
31

---

Published as a conference paper at ICLR 2026
Regative RA4
181
13.4
181
61
180
133
22.4
763
-3
VideotLaMM
LLav LLava-one-iatlon
LLWA-BorTMMoTT
Gwend P44-188
Infomiis
Minic -M
Gwanz B. JDM
Interii
R1-dmas-nn ...0
Homardmend 78
Owren Owrenz S-mmar77
Emuaion www.e.LaMaa LLM47
Affectom1100
arp4e
0PE4.1
Geminh Gemini-2.5-iia 2.5a
Geminin- S
Figure 8: Task-level Performance Comparison on FESD task.
10
467
882
38.2
Ame
184
27.8
884
362
45.4
48.
17A
66.4
4TA
413
44.
14.
11.4
11.
Contentment
4/4
924
489
414
36.
46.1
47A
92.
46.2
486
42.1
583
90.
24.3
36.3
Dogut
665
664
964
618
37A
60.8
64.2
623
64.0
66.
61A
68.4
62.6
723
75
6677
16.1
Eedtement
14.7
174
26A
113
414
05.1
Pour
47.
1LB
4.3
362
212
142
95.
2.1
Sadeass
723
21.3
72.3
72
VilooLLMMA3
LLati- oeie-nn---tttttl
LLaVA Hea 7M
Denna 44-778
InterniL 30
Mal 2.2.6.
oweet M.-2
Inde V-deslaa... A8
Homerdion--i 7M
Omeoz Somm--7
Emanion LLaMA-M
0P74
Gumiel S
Gomoni 2-P-P
Figure Task-level Performance Comparison on ISA task.
Very, Pesitive
147
34.4
3.
20
Pesitive
26A1
15.
17.6
27.8
21.
26.
22.
22.7
22.
9.
167
26.7
25.3
22.1
24A
Nogative
18.3
18.4
124
22.4
244
33.8
137
25.8
267
35.4
21.2
16.
244
26
142
169
VibolLLMM3-D -0
LLaV LAVEE WoT-oeo7-
Itternit 2-8
Ma 220
InterniL3 3008
RJ Pl--ma--aa 5g
Homandhenl17 78
Ownstt 5-d
Emeation LLaMA-78
ArfuasPe- y8
SPR41
Guminii d-Fh
Genstal 5-Pm
Oieend
Figure 10: Tesk-level Performance Comparison on MESA task.
32

---

Published as a conference paper at ICLR 2026
Happy
11.
414
264
382
69.4
114
64.2
748
544
284
618
41.4
P677
122
0.1
414
811
763
893
124
PA.1
85.
758
884
MMA
40.4
11A
A1A
92.L
00
Suprre
VidenLLaMA3-7
LLaV LLavi-one Vision-
LLaVe 1o
Owen2-8-8--7
Intenti 2-88
Misit 2.688
Owend W-322 3288
Intev J--tem---555
Homendmen-1
Oweat R-dont:77
Emetton- Lanke
Affectop
GP140
DP142
Gemoni S-Fiash
Gamoni 2,5 4-Pro
Figure 11: Task-level Performance Comparison on MER task.
Negetive
413
6..3
392
684
113
67.
18.3
484
46.9
66.3
41.3
643
16.3
723
612
72.7
Pouitive
VidenuLaMMA-78
LLaVA-onee Vision-8
LL (La7A--e71104077
Owea 28.78
Intera
MalcHe 6888
Owm22----1228
Intera 8.308
81-0mai-0.33 38
Pernatt S.ani-78
Emoriion LaMe-78
6PX4.3
Gemind 2,5 S-yiishh
Gamini 2.5m
Figure 12: Task-level Performance Comparison on MSA task.
Strong Negativa
derate Negative-
35.8
123
31.5
44.
418
362
46.3
461
284
46.7
47A
372
MA
551
994
532
415
Slight Negative
24.5
22.
214
16A
2M
14-2
MA
165
10.7
M41
264
244
463
404
40A
514
Nestral
Slight Positive
21.8
21A
264
12.4
28.
20.4
13.2
10.1
11.
267
21,8
312
31.
odarate Posltive
14.
113
12.0
41.8
41.
67.8
89.
9M
68.
28.1
162
91.
14
12.
Strong
One- One-Wiiin-888 78
dwenz V1-78
Internti
Ownad -----
Internit. 188
#1 1--mmidd.3
Homsedeent
Oenand 3-0mni-78
Emettove Latt 78
@PM4 BP540
Gemini-2 A-Hlasii
Gemiss-2 S-Pro
Figure 13: Task-level Performance Comparison on SIA fask.
Culm 11.6
2.1
22.2
3.7
32.
223
22
2.
20,0
2.6
53
279
281
343
15.6
465
54.0
Happy
Sad
11.2
15.8
0.
15.2
15.4
613
643
123
96.
7LA
383
183
624
666
15.1
Angry
Tearful 21.
11.3
313
28.
12.
81
LLaVik-One- -taim-78
LLaV LLokk.aaE-vdeee..
Oueett. Mu-78
Internst..a
Owinz 3-4 228
Internv Je8
Homasdmanl-1
Oureal d-Deel-78
Emeeti (LaaM--77
Affect G788
dpEde
GP441
Demino Riri
Genuall 2. S-Pre
Figure 14: Task-level Performance Comparison on SOER task.
33

---

Published as a conference paper at ICLR 2026
Nevtral
48.
666
53.
213
25.4
37a
26.2
51.2
32.8
293
90.6
28.1
Calre
24.1
23.8
24
15.5
2.1
32.8
363
26.0
31.4
15.0
Happy
143
Sad
Angry
Surprsed
VidallaM63-78 77
LLV.W-ne--tttte 7B
LL La
Intern Aa
Owne V--333
Intarn P-30
Oroolo.... SB
HomanOmni-18
Qwenz 5-Demi-7
Emation LL
aftecorrs78
GP144 BPV44
Gemini 2.5 Fei
Gental 2.5--a
Melcc
Figure 15: Tase-level Performance Comparison on SPER task.
Weak negative
Strong nngative
451
174
408
16.
41A
462
144
246
1631
26.
46A
422
142
944
AL7
46
Strong positire
Weak positire
482
10
12.
20,1
LLat Cne---i---illll
LLaVA Rar
Phend 14-778
MosCCM 2-266-98
Owent AVL-ba
Intern Jay
Owenz Comai-77
Emotion-Lae Lafta
AfecCOPP-75
Opfee
GPP-4
Gommint 2 Seleshh
Domini Pro
Figure 16: Task-level Performance Comparison on SCEA task.
354 Slueve
66.
37.8
26
626
53.9
38.4
61.4
925
485
267
318
33.6
26.1
65.
66.9
622
Sleeve
403
892
06.
812
17.
963
843
612
35.
143
903
563
861
033
845
718
Short Sleeve
66.0
58.7
263
153
612
66.1
66
363
223
6.1
615
Strapp/
VidentLanAS-78
LL Mitloo-88
LL wo
cress 341-78
Owen Sat 228
InterveL J88
A..ooiii 8,58
Mona 7B
Smurticn Smoriio-LLattt LaMt-77
AffertPP78
GFF4
Gemboi 2.5 Fiaii
Gamint- Pro
Figure 17: Task-level Performance Comparison on LAVE task.
Neutral
964
582
0.1
989
512
592
644
Positive
44)
163
VidenlloNAA-78
LLaKA-One- Visi--iiiion788
LLavk-teurtiideeee
Owand 14.-2
IntemnVL
Owanz MoccIMM-.... 2.2.
S--.2.-----22 228
R1-0ma--a Inte
0.3
HumanDmall7
Owenz24 S.mml778
Emetioe Emetoo-LLaMA-77 LLMMA-7
or54a
GPP411
demini 6-Flaohh
Denine 5-rro
Figure 18; Task-level Performance Comparison on MABSA task.
34

---

Published as a conference paper at ICLR 2026
Support
162
M4
414
143
93
M.7
531
152
463
461
46
412
436
14.
662
964
663
Rafate
13.4
664
412
114
114
6641
367
184
61
483
68.1
642
42
814
963
Comment
11.4
422
163
M4
114
44
412
418
416
462
943
Unrelated
Lavi Holt--
Owen2 542-78
Internl a8
Owenz Gweaz25-n-20 290-278
Internt sa0
Rt cane 0.38
Oweet 3-0mai-78
Emeaiinn LaMt--8
AffendPP778
OFF46
6PT41
Dewini-- S-Piatt
Gemini 2.5
Figure 19: Task-level Performance Comparison MSD task.
Surprise
166
34
424
240
2.1
41.
22.0
475
Fear
260
260
15.4
166
240
210
X5.
31.0
166
20.4
248
346
26.
36.
424
416
43.
Jay
Diegust
21,
146
13.
26
188
21.8
21.
186
276
18.8
Uie
45
26
20
VidentlaHM
LLaVl-Onee-e vision-
LLaV LLaVA-tocerideee
Droez 4--778
InterVL
MiNiCPEAA- 6:40
Gwent 2-2---2--2
Inswrrl L3-30
Orenld... 5.
Oweezz 2-8 A-Dani-78
Emoiinn LLMM--7
Afhetopi-18
GPE4.1 6PS40
Gemiial Pias
Gentin
Figure 20: Task-level Performance Comparison on MDER fask.
Amucament
19.1
11.
722
65.4
033
214
K1.
46.1
114
142
782
M61
IA4
23
544
164
86.
66.
011
125
144
Diaguet
Fear
Nestral
78.8
283
22.1
21.
27A
Sadnesa
03.3
383
148
341
014
214
21
01
722
810
LLate--ne Vister-70
Intemvikhd0
Owend 2L-228
Internvikaa
Hamaachent-7
Ornetal Desil-7 78
Ametiinn LLaMA-78
Demini Flach
Gomsel Pr
Figure 21: Task-level Performance Comparison EER task.
35

---

Published as a conference paper at ICLR 2026
seatnal
8.5
poottbe
0L55
0.27
0.66
diegua
0.3
Angothe
6:05
621
0.35
0.34
6.66
polllire
9J2
0.13
835
way www..eegeteeeee
-81
dispest
wet wwr.eeeeeerr
OSA
Baggy
87
aestral
-0.7
aestral
6.74
0.10
0.51
Wortied
0,00
9:02
0.32
645
0.77
0.45
-02
poodirv
6114
pootive
(g) STA
DO ROER
STER
0.36 nat
0.00
Nugattre
9.6
0..7
cales
Stggh Nogative 6.9
dbagest 0.82
Nentral 0:22 8.7 0.s 0.50 0.33 0.55 0.62
94
hapgy
0.01
0.80
0.81
0.19
0.87
8.00
0.00
0.02
eat
Madecuie Pelttre
0.2
Streeg Poiitive eat est
suppeiee cas
Stoogp Howdtee
Whght Poetter
Minsted
Motrrale Fratten
Frattes
Anppy
Medecade
Figure 22: Confusion matrices for Gemini-2.5-Pro on each task (Part 1).
36

---

Published as a conference paper at ICLR 2026
(10 12EA-11
quentionieg TINM
huppy
9:47
0.03
0.5
0.26
0.43
6.00
0.80
ovatral
diggast
comeling ALC 0.N..111 a.1..00..000
0.29
angee 0.00 0.13
0.00
auppsttigg 022
9:2
0:31
wisiing ALE Gue 00w.6602.... 6822-12
wesk porttre
6.90
0.29
oestral
wess tongestee
prette
Oni SAVE
MAIIA
Steere
0.7
6:3
641
0.06
0.5
Sloryo 9.0
0.31
0.10
0.81
TXXI
0.5
0.5
Skott Sloreo
0.50
0.53
+0.4
022
Strappy 8:00
6/0
9.05
6.1
0.1
-L.
0.
Si Sttrre
Elerre
dlsert Meeee
Amppost
10 MDEER
(g) ESR
AAE 0.1 0.33 631 BUN
Amaanmanit 8:00 bub Dub 1.11 0.51
0.5
Diagast
Audeom 0.2 0.9 034 9:66 636 6:04 6.05
Peor 0.10 9.00 0.10
0.19
922
0.18
Nestral 9.00 0.0
enDer 014
"innermos 9.00 680
Figure 23: Confusion matrices for Gemini-2.55 2.5-Pro on each task (Part 2).
37

---

Published as a conference paper at ICLR 2026
F DESIGN OF TOM-STYLE PROMPTS
We adopt a unified Toyes prompting scaffold across three levels, aligned with the progres- sion of our framework. Level operationalizes first-order affect attribution through four- stage chain--errceptull Simulation, Cognitive Empathy, Prsspective-Takinng and Conclude-and- Map-illustrated Figure(24]to Figure Level extends this scaffold to relational and contextual mind modeling, where perceived states are linked to entities, aspects, and communicative goals (state - about(entity, context)). Representative templates are shown in Figure 34 to Figure [41] Level advances causal attribution and second-order reasoning. focusing why emotions arise, how they shift, and how they are socially interpreted (cause attribution and recursive mind modeling). Illustrative templates are provided in Figure to Figure 47
F.1 DETAILED PROMPT DESIGN RATIONALE
Face Expression Sentiment Detection. This task instantiates basic ToM attribution by inferring a subject's immediate affect from facial expro-exprssssons, gaze, posture, and prosody. The model integrates convergent and divergent cues into a coherent here-and- now hypothesis, explicitly excluding observer bias or trait-based assumptions. Attribution remains grounded in the subject's perspective, ensuring that emotion labels reflect their mental state rather than external interpretation.
Image Sentiment Analysis. This task extends ToM reasoning to full-scene interpretation, requiring attribution of an affective stance from either visible human subjects or environ- mental affordances such as threat, celebration, or serenity. The model infers what an ex- periencer-depicted or implied-would feel, grounding sentiment in context rather than in the observer's reaction. This design highlights scene-level ToM attribution by linking visual evidence to an imagined experiencer's 's mental state.
Meme Sentiment Analysis. This task reframes sentiment detection as communicative in- tent attribution. The model integrates text and image cues, treating convergence straight- forward reinforcement and divergence as deliberate rhetorical strategy (c.g., sarcasm, irony, humor). Sentiment is attributed from the creator's perspective toward the intended audi- ence, embedding ToM reasoning in the recognition of communicative goals.
Multimodal Emotion Recognition. This task generalizes first-order ToM attribution across multiple input channels- behavior, prosody, and lexical content. The model integrates these cues into coherent hypothesis of the speaker's immediate emotional state, resolving convergence and divergence strictly on observable evidence. Attribution reflects the speaker's perspective, ensuring recognition captures their current mental state.
Multimodal Sentiment Analysis. This task shifts from emotion recognition to polarity attribution, asking whether the speaker expresses a positive, negative, or neutral stance. Multimodal cues are synthesized into a stance hypothesis, with attribution grounded in the speaker's evaluative perspective rather than external judgments. The design distinguishes evaluative positioning from emotion states while preserving ToM-based reasoning.
Opinion Sentiment Analysis. This task emphasizes propositional attitudes, attributing po- larity toward a stated proposition. The model decodes evaluative lexical markers alongside multimodal cues and synthesizes them into a hypothesis about the speaker's s stance. Attri- bution is explicitly tied to the speaker's 's point of view, ensuring that polarity judgments context-sensitive rather than generic affect labels.
Sentiment Intensity Analysis. This task advances polarity attribution by incorporating graded strength. The model decodes lexical intensifiers, prosodic emphasis, and visual force/tension to distinguish slight, moderate, or strong polarity expressions. Attribution remains anchored in the speaker's immediate evaluative stance, enabling finer-grained dis- tinctions within ToM-based sentiment reasoning.
Song Emotion Recognition. This task applies ToM reasoning to performance contexts, at- tributing enacted emotional states conveyed by singers or performers. The model integrates cues, with lyries considered when present. Attribution
facial, bodily, and acoustic-mustt is framed from the performer's 's expressive perspective, capturing intended affective enact- ment rather than audience response.
38

---

Published as a conference paper at ICLR 2026
Speech Emotion Recognition. This task applies ToM attribution to spoken interaction, decoding croustic-prssodi features, articulatory-vin cues, and lexical content as evi- dence of inner state. The model integrates these signals into hypothesis of the speaker's 's immediate emotion, ensuring attribution reflects the speaker's mental world rather than the listener's impression.
Stock Comment Emotion Analysis. This task adapts ToM reasoning to financial dis- course, attributing a commenter's ' evaluative stance toward a financial target with graded polarity strength. Lexical cues such hedging, certainty, or numeric framing cen- tral, with prosodic/visual markers incorporated when available. Attribution reflects the commenter's current evaluative orientation, distinguishing weak versus strong polarity in context.
Emotion- Intent Analysis. This task extends ToM reasoning from first-order emo- tion attribution to relational modeling of communicative intent in dialogue. The model de- codes lexical, prosodic, visual, and dialogic cues as traces of the speaker's s state, integrates them into an emotion hypothesis, and then contextualizes this stance as intent toward the addressee (e.g., questioning, consoling, encouraging). Attribution reflects the transition from state to state aaboutadddeeeseeeeeeee.. context), binding emotions to pragmatic communica- tive goals.
Humor Understanding. This task applies second-order ToM reasoning, requiring the model to capture how speaker amuses audience by violating expectations. The model decodes setup, punchline, and delivery cues, constructs an audience expectation baseline, and checks for mismatches such as reversals or double meanings. Attribution made from the saudier-taudience perspective, framing humor as communicative intent based on in- congruity resolution.
Implicit Attribute Value Extraction. This task adapts ToM reasoning to product interpre- tation, treating product presentation as a communicative act between designer and observer. The model decodes visual design cues together with metadata, interprets them intentional signals of hidden properties, and maps them onto valid attribute values. Attribution thus reframes classification as relational reasoning about design intent and observer inference.
Multimodal Aspect-Based Sentiment Analysis. This task extends sentiment attribution to multiple targets and aspects, requiring structured stance separation. The model de- codes multimodal and referential cues, integrates them into stance hypotheses for each target/aspect, and interprets divergences as possible rhetorical devices while grounding strictly in evidence. Attribution reflects the author's perspective toward each entity, pro- ducing distinct and contextually bound polarity labels.
Multimodal Quintuple Extraction. This task formalizes relational stance mapping by ex- tracting structured units of evaluation. The model decodes evaluative cues, resolves holder identity and coreference, and infers quintuples (holder, target, aspect, opinion, sentiment), ensuring attitudes are contextual- ized, evidence-basee and relationally organized beyond raw polarity classification.
tatgeta-sppeeect relations. Attribution is expressed
Multimodal Stance Detection. This task links an author's evaluative state to a specific claim target, distinguishing stance from generic sentiment. The model decodes multi- modal stance cues, attributes the author's 's immediate attitude toward the target, and maps into support, refute, comment, unrelated. Attribution explicitly conditions inference target-specific positioning, aligning with ToM reasoning about communicative orientation.
Multiparty Dialogue Emotion Recognition. This task situates emotion attribution within multi-party -artt exchanges. The model decodes lexical, prosodic, and visual cues for the focal speaker, integrates them into an emotion hypothesis, and refines the attribution using roles, turn-taking, and interactional context. Attribution reflects ToM reasoning about how speaker's ' state is shaped and signaled within dialogue structure, ensuring role- and context- sensitive recognition.
Emotion Elicitation Reasoning, This task shifts ToM reasoning to second-order attri- bution, modeling how a generic viewer, rather than the characters, appraises events. The model decodes narrative and cinematic appraisals along dimensions such as goal congruence, threat, or attachment, and maps them to single elicited emotion. Attribution is grounded in the causal link between specific
potential affect triggers, constructs viewer
39

---

Published as a conference paper at ICLR 2026
events/devices and the audience's 's reaction, highlighting what the makes viewers feel and why.
Emotion Interpretation. This task explains why a subject experiences a given emotion by reconstructing appraisal pathways. The model decodes observable cues (expressions, posture, events, objects), builds a subject-centered appraisal hypothesis (e-g., goal obstruc- tion, threat, social evaluation), and attributes the emotion proximate, visible causes. ToM reasoning is operationalized as event + appraisal + emotion mapping, producing concise, seidence-based explanations.
Laughter Reasoning, This task applies second-order ToM reasoning to explain why laugh- ter occurs. The model decodes setup, punchline, and delivery cues, models audience expec- tations, and identifies the humor trigger (e.g., incongruity, reversal, irony, norm violation). Attribution explains how the mismatch causes reinterpretation into amusement, framing laughter the outcome of expectation management and speaker intent.
Multimodal Emotion Cause Pair Extraction. This task extends emotion recognition to explicit cause-effect mapping in dialogue. The model decodes cues in a target utterance, builds a subject-centered appraisal hypothesis, and links it to the most proximate prior utterance that explains the emotion, enforcing temporal precedence. Attribution results in explicit emotion-catte cause pairs, embedding ToM reasoning about interpersonal dynamics and conversational elicitation.
Sarcasm Detection. This task frames sareasm as nonliteral intent attribution requiring second-order reasoning. The model decodes the literal proposition and surface polarity, models speaker-audience dynamics, and tests for incongruity- reversal where context sig- nals the opposite of what is said. Attribution distinguishes sarcasm from humor or exag- geration by grounding meaning in the speaker's intent for the audience to infer a reversed stance.
Sentiment Flip Analysis. This task tracks how sentiments evolve across dialogue, at- tributing changes to conversational causes. The model decodes polarity and discourse cues, builds sentiment timelines for each speaker, and detects flips from one stance to another. At- tribution assigns trigger types (c.g.. new information, argument, feedback, relf-reflcction) by enforcing temporal and causal reasoning. This highlights ToM's role in modeling dy- namic shifts in evaluation rather than static judgments.
40

---

Published as a conference paper at ICLR 2026
Face Expression Sentiment Detection
muliimodal affective computing expert grounded in theury of mind. Your task is infer the subject's current emotional state by reasaning about their mental world from observable evidence. Complete the face expression emotion detection task as follows:
Decode Affective Siguals (Perceptual Simulation):
Systematically decode observable affect displays facial features, gaze, body posture, prosedy (if available), and verbal conten/tramseripl (i provided) outward evidence of an inner state. Brielly describe each present eue its likely valenee.
Syathesize Mental State Hypothesis (Coguitive Empathy)::
Inlegrate the decoded siguus to forn cohereni firsi-under hypothesis about what the subject is feeling now. the signals converge, state that they support the same valence. some signals diverge, acknowledge discrepancy but ground your hypothesis strietly
evidence without specnlating heyond
(3) Attribute from the Subject's Perspective (Perspective Taking))
Refine the hypothesis to ultribute the most plausible here-amd-now affective valence from the subject's perspective, Exclude interpretations
on viewer's reaction subject's personality
traits.
(4) Conclude and Map to Label:
Provide your reasoning following the steps above, then choose exactly one label from: neutral, negative, positive, and output exnctly f'omotion': 'labol").
Figure 24: ToM-style prompting for Face Expression Sentiment Detection.
2) Immage Sentiment Analysis
You are multimodal affective computing expert grounded theory of mind. Your task is to infer the single dominant emotion conveyed by the image by reasoning about an (explicit or implied) human experiencer's mental from observuble visual evidence. Complete the image sentiment analysis follows:
Decode Affective Siguals (Perceptual Simulation):: Systematically decode visible affective visible, note facial expression, body posture, and interpersonal actions. If no clear subject visible, note the scene's affective affordances (egg, threat/safety cues, contamination /decay vs cleanliness, celebration'- goal attaioment, damage, majjssy, serenitylooziness) together with color/lightng and compo- sition. Briefly desetibe the present and their likely affective ocicutation.
oulward evideniee of slate. If human subjects
(2) Synthesize a Mental State Hl puthesis (Cognitive Empathyy):
Integrate the decoded cues into coherent first-order hypothesis about what a person in or observing this scene would now. diverge, leknowledge the discrepancy ground the hypothesis strietly what visible (avoid specula- tion beyond the image).
cues converpe on the emotion partially diverge; they
(3) Attribute from the Experiencer's Perspective TrrspeediveTaakinng the plausible here-and affect from experiencer's perspective depicted person, present; otherwise implied observer embedded the seene). Exchude interpretations hased your own idiosyneratic preferences or on stable personality traits.
(4) Conclude and Map to Label:
Provide your reasening following the sleps above, choose exacily contentment, disgust, excitement, fear, sadness, and output exactly ('emation': "label').
amusementi anger,
Figure 25: ToM-style prompting for Image Sentiment Analysis.
41

---

Published as a conference paper at ICLR 2026
3) Meme Sentiment Analysis
You multmmddl affective computing expert grounded in theory of mind. Your rask is infer the meme's ' overall offective attiruee by reasoning about the communicative intent behind ir. Complete the meme sentiment analysis task as follows:
Decade Affective Signals (Perceptual Simulation):
Systematically decode observable signals from both the image (faces, netions, symbols) and the text (wording, amphasis, emojis). Briefly note eash eue and its literal, surface-level polarity. polarrityy
(2) Analyze Multimodal Interaction Infer
(Cognitive Empathy):
Analyze the relationship between the imuge and other express strightforward attitude If cues diverge: Treat this conflict critieal clue. Consider if the discrepancy is intentioaal to create pon- literl menning like surcasm, ireay, or hunzor. The mome'' true attitude uften arises from interpeetnng this very conflict. Formulate a hypothesis about the memme's intended commumicative goal.
Ifuees converge: State that they rein force ench
(3) Attribute from the Meme's Communicatiie Perspective PPrspecttre-aatteg))
Based on your analysis of the multimodal interaction and inferred intent, attribute the single most plausible affective attitude heing expressed by mame as communicative act. This is the attitude the meme's creator
generic viewer understand.
your own reaction or judgments about people
depicted.
(4) Conclude and Map to Label:
Provide your reaseming following the steps above, them choose exactly one label from: very pasitive, positive, nentral, negative, very negative, output exactly ('emotion': 'label").
Figure 26: ToM-style prompting for Meme Sentiment Analysis.
4) Multimodal Emotion Recognition
are a multimodal affective computing expert grounded theory of mind. Your infer speuker's emotional state by reasoning about their mental world from multimodal evidence. Complete the multimodal emotion recegnition lusk follows:
(1) Affective Signals (Perceptuul Simulation):
Systematically decode observable in each avnilable channel visual behavior (lace, gaze, posture, gesture), speech presody (tonc, pitch, intensity, tempo) available, and lexical tranent/transcript provid- ed as outward evidence of an inner state. Briefly note the present cues and their tikely cmotion implicas tions.
Synthesize Mental State Hlypothesis (Cogaltive Empathy):
Integrate the deceded enes into colsereut first-order hypothesis about what the speaker is feeling now. If the cues converge, state that they remfforce the same valence. If some caes diverge, scknow ledge the discreuancy bui ground the hypothesis strictly in the observable evidence without specullting beyond
Attribute from the Speaker's Perspective (Perspective Takknng
Refine the hypothesis attribute the single most plausible here-and-now castional state from the speaker's perspective. Exclude interpretations based on the viewer's 's reaction or on the speaker's 's stable traits.
Conclude and Map Label:
Provide your reasoning following the above, then choose exactly one label from: happy, sad, neutral, wurried, surptise, and oulput exuctly l'emotion': 'label").
Figure 27: ToM-style prompting for Multimodal Emotion Recognition.
42

---

Published as a conference paper at ICLR 2026
5) Multimodal Sentiment Analysis
You are a muttrimodal offective computing expert grounded theory of mind. Your task is infer the speaker's overall evaluative polarity by reasoning about their current mental state from multimodal evidence. Complete the multimodal sentiment analysis task follows:
(I) Decode Affective Signals (Perceptual Simulation):
Systematically decode observable eues each available shannel-visual behavior (face, gaze, pesture, gesture). speech prosody (tone, pitch, intensity, tempo) available, and lexical content/ranseript provid- ed outward evidence of iamer evaluative stance. Brielly note the present and their likely polarity implications.
(2) Synthesize a Mental Hypothesis (Cognitive Empathy): Integrate the decoded expressing the converge. state that they reinforce the same polarity. If some eues diverge. acknowledge the discrepancy but ground the hypothesis siriclly in the observable evidence without speculating heyond it.
coherent first-order hypothesis about what overall polarity the speaker
Attribute from the Speaker's Perspective Perspectiie-Takiine
Refine the hypothesis to attribute the single most plausible here-and-now evaluative polarity from the speaker's 's perspective (not the viewer's ' reaction and not slable trail).
Conclude and Map to Label:
Provide your reasoning following steps above, then choose exactly label from: neutrol, negative, pusitive, and output exactly ('ennotion" 'label').
Figure 28: ToM-style prompting for Multimodal Sentiment Analysis.
h
6) Opinion Sentiment Analysis
You are multimodal affective computing expert grounded in theory of mind. Your task is infer the spouker's is current first-order evaluative antiude (overall polarity) by reasoning about their mental world from observable evidence. Complete the opinion sentiment analysis as follows:
(I) Decode Propositional & Affective Signals (Perceptual Simulation):
Systematically decode the spoken' textual content to identify opinionuted predicates, evaluntive terms, intensifeers/negations modulity markers (lexical comten//tunseripp if provided), Note supportive cues speech prosody (tone, pitch, emphasis) and visual behovior (facial expression, posture (gestur)) if available. Briefly describe each present and its likely polarity.
(2) Synthesize a Prupasitional Attitude Hypothesis (Cognitive Empathy):
Integrate decoded cues into coherent here-and-now hypothesis about the speaker's evaluative stance toward what being talked about. If diverge, acknowledge the discrepancy but ground the hypothesis strictly in the observable evidence w without speculating beyond it.
converge, state they reinforce the same polarity. If some
Speaker's Perspective (Perspective- Taking): Refine the hypetbesis attribute the single plausible immediate overall polarity from speaker's perspective (not the viewer's seaction and stable trait).
(3) Attribute
(4) Conclude and Map to Label:
Provide your rensoning Collowing steps above, then exactly one from: neutral, negative, positive, and output exactly l'emotion': "aabel)..
Figure 29: ToM-style prompting for Opinion Sentiment Analysis.
43

---

Published as a conference paper at ICLR 2026
7) Sentiment Intensity Analysis
You multimodal affective computing expert grounded theory of mind. Your nask is infer the speaker's current first-order evaluative state with graded intensity by reasoning about their mental world from multimodal evidence. Complete the sentiment intensity analysis task as follows:
Decude Pularity & Intensity Signals (Perceplual Simulation):
Systematically decode observable cues as outward evidence an inner state lexical content/tramseripe (if provided: evaluative temms, intensfierdddminnsheerr, negation, capitalization, repetitionssllngation emojis), speech prosody available: pitch, londness/cnergy, tempo, emphasis), visual behavior (facial muscle activation, gesture force, posture tension). Briefly each present with polarity direction and intensity strength.
(2) Synthesize a Graded Mental State Hypothesis (Cognitive Empathy):
Integrate decoded caes into a cobereat here-and-now hypothesis consisting a base polarity (positive' weagttveeeegaavvvvuttttae and intensily level (sighar/mwueenteesttunng) Indicate whether converge, if some diverge, acknowledge the discrepancy but ground the hypothesis strictly observable evidence without inventing unobserved camses.
(3) Attribute from the Speaker's Perspective (Perspective- Taking); Reinee hypothesis attribute single most plausible immediate evaluntive with intensity from the speaker's 's perspective (nol the viewer's reaction and not slable trail).
(4) Conclude and Map to Label:
Provide your reasoning following the steps above, then choose exactly one from: Strong Negative, Modcrate Negative, Slight Nogative, Neutral, Slight Positive, Moderate Positive, Strong Positive, and oulput exactly ('emotion': "label").
Figure 30: ToM-style prompting for Sentiment Intensity Analysis.
8) Song Emotion Recognition
multimodal offectivv computing expert grounded theory of mind. infer performance's first-order emotional (as enacted by the singer/performer) by reasoning about their mental world from multimodal evidence. Complete the song emotion recognition task follows:
(1) Decode Affective Signals (Perceptual Simulation):
Systematically decode observable performance cues as outward evidence of an inner state- state-facial expres- sion and micro-cxpressions, gaze and body/brcath posture movement quality: musical-acoustic prosody audible (tempo, intensity/enerry. pitch rangolcentour, timbre, articulation); and lyrics/enscreen text provided (evaluative stance). Briefly note the present and their likely emotion implica- tions.
(2) Synthesize a Performance-Stir Hyppothesss (Cognitive Empathy):
Integrate the decuded inlo a coherent lure-and--uow hypothesis about what emotiio performer expressing. Indicate whether the converge; if some diverge, acknowledge the discrepancy while gronding your hypothesis strictly in observahle evidence without speculating heyond it.
(3) Attribute from the Performer's Perspective (Perspective- Taking):
Refiee the hypothesis attribute single most plausible immediate emotion from the performer's enacted perspective (not the viewer's reaction and stable truit the performer).
(4) Conclude and Map to Label:
Provide your reasoning following the steps above, then choose exactly one label from: neutral, calm, happy, sad, angry, fearful, and output exaclly ('emotion!: "abel!).
Figure 31: ToM-style prompting for Song Emotion Recognition.
44

---

Published as a conference paper at ICLR 2026
9) Speech Emotion Recognition
You multimadal affective computing expert grounded in theory of mind. Your task is infer the speaker's is curvant first-ordd first-ordar emotional state during speech by reasoning ahout their mental world from observable evidence. Complete the speech emotion recognition task as follows:
Decode Affective Signals (Perceptual Simulation):
Sysiematically decode sperch-related eues as outward evidence of an inner state acoustic acoussic-pruoodi features (pitch levelinngs/contour loudaess/ecrrgy tsmpeo/tydmm emphasis, voice quality such as kyeathyy/ease/oouhh) articulatory-visuall caes (mouth shaping. facial expression. gaze, head upper-body movement, posture). and transcript provvdde.. Briefly note the present and their hikely emotion implications.
(2) Synthesize a Mental State Hypothests (Coguitive Empathy):
Integrate the decoded cues into cobereat here-and-now emotion hypothhsis.. eues converge, state that they reinforce hypothesis strietly in observable evidence without speculuting beyond it.
emotion;
diverge, acknowledge discrepancy while grounding your
(9) Attribute from the Speaker's Perspective Perspectiie-Takinng Refine hypothesis to attribure the single most plausible immediate emotion from the speaker's is perspoc- tive listemer's reaction and not stable traif).
(4) Conclude and Map to Label:
Provide your rensoning following the steps above, then choose exnetly one label from: neutral, calm, happy, sad, angry, fearful, supprised, disgust, and cutput exactly l'emotion": 'label").
Figure 32: ToM-style prompting for Speech Emotion Recognition.
10) Stock Comment Emotion Analysis
multimodal offective computing expert grounded theory of mind. commenter'ss current first-order evaluative attitude toward the financial target with graded strength reasoning about their mental world from multimodal evidence. Complete the stock comment emotion analysis task as follows:
infer
(1) Decode Evaluative Signals (Perceptual Simulation):
Systematically decode observable cues as outward evidence of an inner stnnee lexical content (fioanee-laden evaluations, polarity terms, certainty/hedging comparativeisuupellii wording), speech prosody if available modality markers, mumeric framing, confidence doubt), and visual behavior if available (facial expression, nods/shuees, pasture/gesturr tension). Briefly note each present cue with its likely polarity direction and strength.
indicating
(2) Synthesize a Graded Attitude Hypothesis (Cognitive Empathy): Integrate decoded cues and strength level (weak loward the specific financial largel implied the comment. If cues converge, state that they reinforee the same gruded attitude; if some diverge, acknowl- edge the discrepancy but ground the hypothesis strictly in observable evidence without inventing unobserved causes.
cohereat here-and-now hypothesis consisting
polanit
gosstiveengaativeeeeeee
Attribute from the Commenter's Perspective (Perspective Tukinn)):
Refine the hypothesis to attribute the single most plmusible immediate evnluative attitude from the commenter's 's perspective toward the financial target. Exclude interpretatiens based on the viewer's reaction. actual market outcomes, or stable personality truits.
(4) Conclude and Map to Lahel:
Provide reasoning following the steps above, then choose exactly one label from: weak negative, strong uegative, neutral, weak positive, strong positive, and output exactly (emottoon: "labet).
Figure 33: ToM-style prompting for Stock Comment Emotion Analysis.
45

---

Published as a conference paper at ICLR 2026
1) Detection of Persuasion Techniques in Memes
You are multimudal offective computing expert grounded theory mind. Your lask to detect persuasion wccniiqes the by modeling communicative act from This requires relational and contextual reasoning chain: first infer mental from observahle cues (cues state), then reason about how this state directed toward entities and serves communica- tive purpose (state -- aboutfentity, context)).
audience.
Decode Propositional & Allective Content (Perceplual Simulation):
Systematically decode observable signals as oalward evidence of menial atritude. For the image (lices, actions, symbols, flaps, crowds, threat/victim imagery) and text (wording, slogans, emphasis such as CAPS, repetition, emojis), identify key tugets, in-group/ouut-ron framing if present, and affoctive value-laden Note whether channels converge deliberately
(2) Infer Communicative Intent toward the Audience (Cognitive E'puthy Perspective Cenpective-Takingg:
Adopt the crentor-as-peaker perspective and model the generic viewer as the mudience. Infer what belief, attitude, emotion meme designed to slicit. text and image diverge, treat potential thetori- device (e.g. sarcasm, irony, exaggeration) rather than noise.
(3) Attribute Persuasion Techniques (Relational Mapping):
Map the inferred intent and its supporting cucs to zero or more persuasion tochniques from the given txonomy. Link each chosen technique relevant cue-intent reiation. but keep reasoning concise and evidence-based. Select only techniques directly supported by multimodal content.
(4) Conclude and Output: Provide your rensoning follewing the steps above, then outpur exactly: (techniques: ['lahell' Jahel2 ...).
Figure 34: ToM-style prompting for Detection of Persuasion Techniques in Memes.
2) Emotion-Based Intent Analysis
You are a multimodal affective computing expert grounded theory of mind. Model the utterance communicative act from speaker to addressee. Follow relational contextual reasoning chain: first infer mental state from observable the addressee and serves communicative purpose within the dialogue (state aboutladdressee contexli).
state), then how that is directed toward
Decode Multimodal Diulogic Cues (Perceptual Simulatioo):
Systematically decode observable signals as outward evidence of an inner state state-lexical content, prosody (if available), visual behavior (if available), and immedinte dialogue context. Note present cues and their emotionn'nttet implications.
Attribute the Speaker's Emotion (Cognitive Empathy, First Order):
Iotegrate the deceded cues infer the speaker's here-and-now emotional state from the speaker's 's perspec- tive (not the listener's 's reaction and not stable trait). efenss converge, state they support the same emotion; if some diverge, scknowledge discrepancy but ground attribution strictly observable evidence.
Infer Communicative Intent toward the Addressee (Perspective Takenn Relational Mapping): Using dinlogue context and speech-act indientors, infer the single most plausible intent the speaker directs at the addressce. Treat emotion as modulating tone, but do not override clear linguistic or pragmatic markers, Use "neutral" intent is indicated.
Conclude and Output:
Provide rensoning tollowing the steps above, then choose exactly emotion from: happy, supprise, sad, disgust, anger, fear, ncutral, and exactly one inteut from: questioning, agrecing, scknowledging. encouraging, consoling, suggesting. wishing, neutral. Output exactly: f'emotion': 'label', 'intent' "lahe!").
Figure 35: ToM-style prompting for Emotion-Based Intent Analysis.
46

---

Published as a conference paper at ICLR 2026
3) Humor Understanding
mullimodal affective computing expert grounded theory of mind. Model the communicutive from speaker audience. Follow relational and contextual reasoning chuin: first infer mental state from observable cues and contect (cues state), then reuson about how that stute used to amuse an audience within the diologue (state - about(audience,, context)).
(I) Decode Context & Delivery Cues (Percepiual Simulation):
Systematically decode observable signals as outward evidence of an ini ended stance context/setup content, the punchline sentence, any available delivery cues (prosody, timing, factal/gestural markers). Briefly the salieut propositions and the baseline expectation established the sctup.
(2) Construct an Audience Expectation Model (Cognitive Empathy):
Frrom the setup, infer what reasonable audience would expect nexi. Specify the assumed beliel/interpreet audienoe
hearing punchline (the contextual baseline).
(3) Detect Incongruity and Seek Coherent Reinterpretation (Perspective-Takee & Intent Inference): Compare the punchline to the baseline. Determine whether there is an intentional mismatch (e.g.. reversal, double meaning, absurd shill) that invites coherent reinterprelation resolving the surprise benign/ac- the inference stricily in the provided content;
ceplable way.
import external facts.
Attribute Commonicative Intent (Relational Mapping):
Decide whether the speaker is using that irssoggtiiytttsoottion to amuse the audience bere-and--ow (i.s, humor as the communicative purpose). no plausible humorous reinterpretation emerges from the given not
content,
(S) Conclude and Output:
Provide your reasoning following the steps above. then choose exactly one label from: true, false, and oulpul chosen label only.
Figure 36: ToM-style prompting for Humor Understanding.
4) Implicit Attribute Value Extraction
You multimadal affective computing expert grounded in theory of mind. Your task to infer unstated product attribute value by reasoning about the designer's intent and the ohserver's interpreia- tion. Treat the product image and metadata as communicative evidence: what is visually presented meunt imply hidden attribute values. perceptual contexti).
this, follow relational reusoning chain; first infer state), then map them cutegory-specife attribute (state about(atribute,
Cues (Percopiual Simulation):
Systematically analyze observable details product image such shape, cut, seams, length. noekline, sleeve, strap. collar, shoulder, shaft height, heel, toe style, or other relevant features and combine them with the provided mctadata (category and attribute typc). Treat these as outward signals of the product's design properties.
Attribute Implied Design Property (Cognitive Empathy):
Infer the most plausible bere-and-now design property implied by the decoded cues, simulating the intent behind how the preduct is visually presented. Ground the inforence strictly in visual and metadata evidence, avoiding speculation beyond the product.
Relational Mapping Attribute Taxonomy (Perspective-Tiie Takiin))--Tang:::
Map the inferred design property exnetly one valid value within the provided taxonomy for that entegory and attribute. Ensure the choice is cousisteat with the evidence and fits the requested attribute class.
Conclude Output:
Provide rensoning following the steps then choose exactly attribute value from the given list of opptions for the tarpet attribute. Output only the chosen lahel.
Figure 37: ToM-style prompting for Implicit Attribute Value Extraction.
47

---

Published as a conference paper at ICLR 2026
5) Multimodal Aspect-Based Sentiment Analysis
multimodal offective computing expert grounded theory mind. Model the post communicative from author oudience. Follme relational and contextual reasaning chain: first infer an evaluative state from observable cues (cues state), then relute that state to each specified turget/spect (state -a about(entity, contect)).
(1) Decode Multimodal & Refervntial Cues (Percepfual Simulation):
Systenatically decode observable signals as oulward evidence an immer evaluative slamce textual content (opinion predicates, polarity words, intensifiewinegations, stance verhs), image cues (faces, actions, symbols, scenes), and prosody if available. Identify mentions, aliases, and pronouns referring each listed target'sspeet. Note whether text and image coaverge deliberately contrast.
(2) Build Target-L.eee Attitude Hypotheses (Cognitive Empathy)::
For speci fied lurgetinspect, inlegrate only cues that pertain (direct mentions, co-references, clearly implied links) form here-and-now mismatch possible thetorical device (e.g., sareasm irony) but ground interpretation strictly in the peovided content. If evidence for tarect/aspect insufficient or non-commmttall, assign neutral.
hypethesis cues modalities diverge, trent the
Attribute Attribute the single plausible immediate polarity the author holds toward each turgetiaspect (oot the viewer's 's reaction, and net stable trait). Keep judgments targef-separable; de not let sentiment toward one larget bloed into snother.
Perspective (Perspective Taking &
Mapping):
Conclude Output:
Provide your reasoning following the steps above, then for every tarpet output exactly lahel from: positive, neutral, negative. Retim exactly one ISON object mapping each target to its label, ogg. (TTaggetA:: "positive', 'TargetB: 'ncutral").
Figure 38: ToM-style prompting for Multimodal Aspect-Rased Sentiment Analysis.
6) Multimodal Quintuple Extraction
muliimodal offective computing expert grounded theory of mind. Model each utterance/post communicative from specific holder loward turgets in context Follow relational and contex- tnal reasoning chain: first infer evaluative states from textuul cues (cues - state), then bind each stute concrate target and espect within the discourse or some (state - about(entity, contexti) and express structured quintuples.
(1) Decade Propositional & Referencing Cues (Perceptunl Simulation):
Systematically parse content affectiveicvaluutiv markers opinion prodicates, polarity words, pneessffeessinegation hedges/modality-ane resolve speaker identity (holder). Perform basic corefer- enceialias resolution link prenouns aliases to explicit targets. Keep strictly the given text: not import external facts.
Attribute Target-I
(Cnguitive Empathy, First-Order):
each holder, infer the here-and-now evaluative toward specific target and aspect indiented implied by the text. If cues diverge (e.g., positive term with negative inteusifier), acknowledge and ground the inference in the most coherent observable reading. If stance is wcak/uncertain, treat sentiment as neutral.
Relational Mapping Structured Quintuples (Perspective-Takiee Context Binding):
Compose quintuples the form (holder, target, aspect, opinion, sentiment), holder: the speaking author of the utterance: target: the catity being evaluated brrandplacccppsseeitttmtt the target (explicit noun phrase: allow implicit facet only clearly signaled); opinion: the minimal surface phrase expressing the evaluation; sentiment:
aspect: the facet/property
(positive, neutral, negalive), consistent Create separate quintuples different turgets/aspects.
(4) Couclude and Output:
Provide your reasoming following the steps above, oulgut a single Py thou-style de list quintuples using single quotes
commentary, e eg-[(hooler "target' "aspett sspinion", sentiment), ...].
Figure 39: ToM-style prompting for Multimodal Quintuple Extraction.
48

---

Published as a conference paper at ICLR 2026
7) Multimodal Stance Detection
are muitimodal affective compuring expert grounded theory of mind. Model the postutterance communicative act from author audience. Follow relational and contextual reasoning chain: first infer evalaative state from observable cues (cues state), then relate that state to the specified lurget/caam to derive stunce in context (stale - aboul(turge,, contexi)).
Decode Multimodal Referential Cues (Perceptual Simulation):
Systemutically decode textual content (upinionsstance markers, polarity, negation/mmddaltt. image cues (faces, actions, symbols, scenes), and prosody available, as outward evidence of an immer attinude: Identily explicit mentions, aliases, promouns the given trrget/claim, image converge deliberately contrast. Do import external facts.
whether
(2) Attribute the Author's Attitude toward the larget Claim (Cognitive Empathy, First Order): Inlegrate only cues that pertain to the specified lurgeticlaim to forn here-und-nowe now uttitude hypothesis from author's perspective. If cues diverge, ucknowledge discrepuncy but ground attribution strietly observable evidence. If evidence non-committal, keep the hypothesis neutral.
(3) Map Attitude to Stance (Perspective Taking & Relational Mapping): Derive exactly one stance lahel conditioned on the target/claim:
support expresses aedorseme//allgmmeenn
lurget/claim.
refute expresses opposition conttiion//tttiiitiiinnn
targotcllaim.
comment discusses the lopie/context without clear support rofutation. uunelat eaaeeeeedd no meaning ful relation to the specifted lunge//clum.
Keep judgments lunel-condiiioned do not let general semtiment loward other entities bleed into this mapping.
(4) Conclude and Output:
Provide your reasoning following the steps above, then choose exactly one label from: support, refute, comment, unrelated. Outpul exaclly: ('stance': "label").
Figure 40: ToM-style prompting for Multimodal Stance Detection.
8) Multiparty Dialogue Emotion Recognition
multimodul affective computing expert grounded theory of mind. Model
communicative produced a specific speaker within multi- parti--prtyyaee exchange. Follow relational and contextual reasoning chain: first infer mental state from observable cues (cues state), then relate that state to dialogue roles and nearby turns to finalize the attribution (stale - ahout(role, context)).
(1) Decode Multimodal Dialogic Cues (Perceptual Simulation):
Systematically decode observahle signals as outward evidence of an inner state lexical content/transeript provided), speech prosody available), Resolve the larget speaker for this and note turn-taking markers, address terms, and response type.
pasee//gv//ppsuureee gesture available).
(2) Altribule the Speaker's 's Emotiin (Cognitive Empathy, First-Onder)::
Integrate the decoded cues infer speaker's here-and-now emotional state from the speaker's perspec- (not listener's reaction and not stable trait). diverge, acknowledge the discrepancy and ground the attribution strictly observable evidence; if e'evidence is weak, prefer neutra).
(3) Contextualize with Roles & Surrounding Turns (Perspective Sesspectiv--akking & Relational Mapping): Use inmediate context (preceding/follow following turns and participant roles such as uddressee/tiin party) disambiguate ultribution egg, whether the tum reply, challenge, lease, agreement, correction. Let context modulate, but override, cleur in-tum affective signals. Do import exteral
facts.
(4) Conclude and Map to Label:
Provide your reasoning following steps above, then choose exactly one from: neutral, surprise, saduess, joy, disgust, anger, Output exeetly; ('amotion': "labe!))
Figure 41: ToM-style prompting for Multiparty Dialogue Emotion Recognition.
49

---

Published as a conference paper at ICLR 2026
1) Emotion Elicitation Reasoning
mullimmdda offective computing expert grounded theory of mind. Your is infer emation elicited in typical viewer this clip, using causal and second-order order reasoning (modeling the viewer's 's appraisals rather than the characters' foelingss. Proceed as follows:
Decode Narrative Affective (Perecptual Simulation)::
Systemutically extract salient orents/oulcommes churacter expressions and behaviuue spoken lone/prosody (if audible), and cinematic signals (music, pacing, camera focus, lighting) as outward evidence of inteended affect.
Viewer Appraisal Model (Second- Order Perspective- TTkinng):
From the perspective of generic adult viewer. appraise the depicted events along dimensions: congruemceoobsiratiion threat/safety, agency/blame, nomm purity violation, atlachment/care or loss. Keep this distinet from the churacters' internal states.
Infer Elicited Emotion (Causal Attribution):
Map the dominent viewer appraisals one bere-and-now elicited ennotion for the viewer. Resolve any conflicts by priortizing the most consequential event- sent-appraissl pattern for the viewer, not character mood.
Identify Proximate Causes Modulators:
Name the specific scene events/dynamis and any einematic devices that directly produce intensify the elicited emotion chose.
Figure 42: ToM-style prompting for Emotion Elicitation Reasoning.
2) Emotion Interpretation
You are multimodal offective computing expert grounded in theory of mind. Given a stated turget emotion to explain, attribute plausible causes for the focal person(s) using causal and second-order reusoning. Proceed as follows:
(1) Decode Scene Affective Cues (Perveptual Simulation):
Systematically parse ohservahle evidence faccial expression, gave, body postureemmvemeet interpersomal spacing touch, salient objscts/events, setting eues, and any evidense of internal appraisals.
text. Treat these outward
(2) Build a Subject Centered Appraisal Hlpothhesi (Cognitive Empathy Perspective Taking); Frro subject's perspective (not viewer's 3), infer appraisals that the given emotion along core dimensions such goal tenemmenee/obsmmuttioon loss, threat/safety, ageney-blame, social evaluation, norm/purity violation, control/umcrtaingg Keep this distinct from other characters foelings.
(3) Causal Aftribution (Event - Appraisal Emotion):
Identily conerete, proximate causes would produce those appriisals and thus the larget emotion. Prefer immedialely visible allow only minimol inferences directly suggested by the evidence. Do not import external facts.
events, agents, objects, or conditions-
(4) Comelude and Outpul:
Provide your reasoning following steps plausible camses, phrnsed short clanses prounde in the scene, with extra commentary.
output
numbered (1., 2., ...)
Figure 43: ToM-style prompting for Emotion Interpretation.
50

---

Published as a conference paper at ICLR 2026
3) Laughter Reasoning
You are multimodal affective computing expert grounded in theory of mind Your task is explain why the andience laughed in the clip. using causal and second-ore order reasoning that models the audience's expectations and speaker's communicative
Proceed as follows:
(1) Decode Setup & Delivery Cues (Perceptual Simulation):
Systematically parse the setup content the punchline region logether with delivery cues cues-lexical lexical propositions, prosodytiming (pauses, pilchlenergy shifls), facial'gestural markers, und any on-screen audience reactions. Summatize the baseline interpretation established by the setup.
(2) Model Audience Expectations (Second Order Perspective Taking);
From generic audiemce perspective, state what they are expect immediately before the trigger (the heliet/interprctation they momentarity hold based
setup and delivery).
(3) Identify the Humor Trigger & Mechanism (Causal Attribution):
Locate the specific element that violates or reinterprets the baseline (eg incengruity. reversal, irony. hyperbole, double meaning. frame/rol shift, berign nomn violation). Ground this identification strictly the provided content; do import extemaa
(4) Explain the Causal Link & Intent (Relational Mapping):
Explain how this trigger causes a quick reinterpretation that resolves surpeise into amusement and signals playful intent from the speaker to the audience. Note any delivery features (timing' emphasis) that amplify effect.
(S) Conclude and Output:
Provide your reasoning following the steps above, then oulpul single sentencce statting with The audietuee laughed because
Figure 44: ToM-style prompting for Laughter Reasoning.
Multimodal Emotion Cause Pair Extraction
You are a multimodal offective computing expert grounded in theory of mind. For EACH turget utter- ance infer the speaker's emotion identify the single most plausible prior utterance that causes using causal and second-order reasoning. Proceed as follows:
(I) Decode the Target Turn (Perceptuul Simulation):
Systematically parse target utterance's ohservable lexical content, prosody a'avillllle (pitch/en- pegyttempo/mmhhais, and visual behavior available fhcc//gzz/posstrre/ gestturree ourward evidence of an inner state. Note the speaker, addressee, and tarn finction.
(2) Build a Subject- Centered Appruisal Hypothesis (Cognitive Empathy & Perspective -Takiing: From targel speaker's perspective, the appraisals that can expluin the larget emotion (eg-, goul obstruction itsttuetioonattainmnennn threat/safety. agency/blume, social evaluation, violation, Keep this distinct from other participants' feelings. Ground all in the given dinlogue/video; do not import externa facts.
Linking Scon earlier find the single prior utterance that direstly elicits the inferred appruisals for this target turn. Enforee temporal precedence (cause preeede effect). Prefer the most proximal utterance that minimally sufficient to trigger the target emotion und is semamtically aligned with the appraisals. If multiple candidates exist, choose the nearest one that explains the largel emotion, evidence is weak mixed, keep
(Causal
Recond-Order Reasoning):
still
most plausible prior cause supported by context.
(4) Conclude and Outpat:
Provide your reasoning following the steps above, then output exactly one ISON object mapping cach target utterance to f'emotion': 'labol', 'cause utterance id: 'ID") using single quotes. Choose emotion from: joy, sadness, anger, disgusi, fear, supprise, neuiral. Output nothing
Figure 45: ToM-style prompting for Multimodal Emotion Cause Pair Extraction.
51

---

Published as a conference paper at ICLR 2026
5) Sarcasm Detection
multimodal affective computing expert grounded theory of mind. Your task is decide whether the statement is sarcastic by modeling the speaker's nonliteral intent toward an audience using causal and second-order reasoning. Proceed as follows:
Decode Literal Proposition & Affective Cues (Perceptual Simulation):
Systematically purse lexical contemt logether with delivery cues prosody if available (piich/energe ppiich/energyllem- polemphasis) and visual bebavior if available aice/gaze/gesturgg State the literal proposition and its surface polarity as outward evidence ao initial stance.
Model Speaker Intent & Audience Uptake (Second Order Perspective Taking): From the speaker-audience perspective, infer whether the speaker intends the andience recover meaning that contrasts with the literal stance, based only on contextual evidence in the clip (preceding events, shared situational facts shown, discousse expectations). not import external fiets.
Incongruity Reversal Test (Causal Attribution):
Check for deliberate, coherent mismatch between (a) the literal words and (b) the situational appraisal signaled by context or by prosodic cues. Determine whether this mismatch supports stable opposite interpretation (ic., the spenker means reverse attitude) rather mere ambiguity or general humor.
Decision Rule:
the vvidenice indicates an intended opposite attitude that a reasoable audience would recognize (nonlit- eral reversal with supportive cmntextual deliverr cues), label as sarcastic true, otherwise lahel false
Conclude and Output:
Provide your rensoning following the above, then output exaetly one tokon: true or false.
Figure 46: ToM-style prompting for Sarcasm Detection.
Sentiment Flip Analysis
multimodal affective computing expert grounded theory of mind. Your task detect ALL sentiment flips in the dialogue and anribute a trigger type for each flip, using causal and second-order reasoning. Proceed as follows:
Decode Turns Textual Cues (Perceptual Simulation):
Systematically purse each utterance's textual signals Texical polarity words, negations, intensifiers, hodges, modality markers, discourse markers ourward evidenee of an inner evaluative stanee. Record the holder (speaker) for every tura and note its relation to the ongoing dialoguc.
Track Hokler Sentiment Timeline (Cognitive Empathy, First-Order):
For each holder, infer the here-and-now sentiment for each turn one of (pusitive, negative, neutra).. Establish baseline (the earliest clearly expressed stance) and update the timeline across turns. cuues are ambiguous, assign noutral.
Defect Plips & Attribute Triggers (Causal Attribution & Second-Order Reasoning): Scan each holder's 's timelime to detect changes of sentinent category. For each flip, atiribute the most plausible trigger type based ONLY on dialogue context, enforcing temporal precedence: Introduction Information, Logical Argumentation, Participant and Self-refleztion.
Interaction, Personal Experience
(4) Conclude and Output:
Provide your reasening following the steps above, then outgut ONLY single JSON-like list where each deseribes one Ilip: [[holder": !.,, 'initial sentiment': bosiiivenneativeeeurre 'Dipped sentiment': "posttieenegaatvve neutrall, 'triggr type': Introduction New Information| Argunentation Partici- pant Feedback and Interaction/eersoaal Experience and Self-reflection)). Output nothing clsc.
Figure 47: ToM-style prompting for Sentiment Flip Analysis.
52

---

Published as a conference paper at ICLR 2026
G
DATASET CASES
CH-SIMS
QUESTION
ANSWER
The person in video says: really thought couldn't beat you. Determine the emotion conveyed. Do not provide any additional explanations or extra content. Choose of the following labels as your final answer.
NEUTRAL
NEGATIVE
POSITIVE
Figure 48: Representative sample CH-SIMS dataset.
CH-SIMSv2
QUESTION
ANSWER
The person in vidco says: When you die, I'll burn paper to tell you. Goodbye, bastard. Determine the emotion conveyed. Do not provide any additional explana- tions or extra content, Choose one of the following labels as your final answer.
NEUTRAL
NEGATIVE
POSITIVE
Figure 49: Representative sample of CH-SIMSv2 dataset.
EmoSet
QUESTION
Recugnize the emotion conveyed by the visual content. Do not provide any explanations or extra text. Choose of the following labels your final
ANSWER
AMUSEMENT
ANGER
SADNESS
DISGUST
CONTENTMENT
EXCITEMENT
FEAR
AWE
Figure 50: Representative sample of EmoSet dataset.
53

---

Published as a conference paper at ICLR 2026
+
Memotion
QUESTION
WHY
Recognize the overall sentiment of meme. Do not provide any explanations or extra text. Choose one of the following labels your final answer.
ANSWER
VERY POSITIVE
POSITIVE
NEUTRAL
NEGATIVE
HELLO
VERY NEGATIVE
Figure 51: Representative sample of Memotion dataset.
MER2023
CERL TO
QUESTION
ANSWER
Okay, anyway, you can just bring it back. As a little under- cover agent, can be considered as helping you climb the Blade Mountain. Next. Determine the emotion conveyed. Do not provide any additional explanations extra content. Choose one of the following labels as your final answer.
SAD
NEUTRAL
ANGRY
HAPPY
WORRIED
SURPRISE
Figure 52: Representative sample of MER2023 dataset.
A
CMU-MOSI
66
QUESTION
ANSWER
NEUTRAL
The person in video says: IM JUST KINDA LIKE NO THIS IS GOING TO FAR. Determine emotion conveyed not provide any additional explanations content. Choose one of the following labels your final answer.
NEGATIVE
POSITIVE
Figure 53: Representative sample of CMU-MOSI dataset.
54

---

Published as a conference paper at ICLR 2026
CMU-MOSEI
QUESTION
The transcript of the video is: (uhh) The action sequences thought didn't even hold up. Determine the sentiment intemsity category based on the video content and its transcript. Do not provide any explana- tions or extra text. Choose one the following labels as your final answer.
ANSWER
NEUTRAL
MODERATE
STRONG NEGA-
SLIGHT NEGA-
SLIGHT POSITIVE
STRONG POSI-
Figure 54: Representative sample of CMU-MOSEI dataset.
RAVDESS Song
QUESTION
ANSWER
Please watch the provided video and determine the emotion it conveys. Do not provide any additional explanations or extra content. Choose one of the following labels as your final answer
SAD
CALM
ANGRY
NEUTRAL
FEARFUL
HAPPY
Figure 55: Representative sample RAVDESS (song) dataset.
55

---

Published as a conference paper at ICLR 2026
RAVDESS speech
QUESTION
The transcript of the video (uhh) The action sequences thought were didn't even hold up. Determine the sentiment intensity category based the video content and its transcript. Do not provide any explana- tions or extra text. Choose one of the following labels as your final answer.
ANSWER
CALM
SAI>
FEARFUL.
DISGUST
SURPRISED
NEUTRAL
Figure 56: Representative sample of RAVDESS (speech) dataset.
FMSA-SC
QUESTION
ANSWER
Person video says: He has major advantage that fell first compared all other sectors. fell the beginning of the year, but the cycle has been rising for 10 months since the beginning of the year. Deternine the emotion conveyed. Do not provide any additional explanations extra content. Choose one of the following labels as your final answer.
WEAK NEGATIVE
STRONG NEGATIVE
NEUTRAL
STRONG POSITIVE
WEAK POSITIVE
Figure 57: Representative sample FMSA-SC.
56

---

Published as a conference paper at ICLR 2026
SemEva12021 Taskó
QUESTION
*President* Biden?
Detect which persuasion techniques are used this meme. Choose zero more labels from the following list. Appeal to authority, Appeal to fear/prejudice, Black-amd-white FallacyyDictuatrsship. Causal Oversimpli- fication, Doubt, Exaggeration/ Minimisation, Flag-waving, Glittering generalities (Virtue), Loaded Language, Mistepresentation of Someone's Position (Straw Man), Name calling/Labeling, Obfuscation, Intentional vaguemess,, Confusion, Presenting Irrelevant Data (Red Herring), Reductio and hitlerum, Repetition, Slogans, Smears, noought-eemminating cliché, Whataboutism, Bandwagon, Transfer, Appeal (Strong) Emotions.
Please, no.
ANSWER
Smears, Doubt
Figure 58: Representative sample of SemEval2021. Taskó dataset.
MC-EIU
QUESTION
The person video says: Oh! Okay, you come with me and you tell that the house is haunted. Analyze the emotion and intent. Choose one emotion: happy, surprise, sad, disgust, anger, fear, and neutral. Choose one intent: questioning, agrecing, acknowledging, encouraging, consoling, suggest- ing, wishing, and neutral.
ANSWER
emmotion: 'neutral", 'intent': 'suggesting"
Figure 59: Representative sample of MC-EIU dataset.
57

---

Published as a conference paper at ICLR 2026
UR-FUNNY
QUESTION
ANSWER
The context sentences in the video are: was so innocent and so sweet and her voice was pure, and i even loved seeing behind her could see the little teddy bear sitting the piano behind her in her room, such an intimate video, and had this idea if could get people to do this same thing sing their parts soprano alto tenor bass wherever they were in the world post their videos we could it all together and choir, wrote my. The punchline sentence in the video is: actually wrote hopeful- ly for the last time in public ever. Does this statement express humor? Choose one of the following labels as your answer: true, false.
FALSE
TURE
virtual
Figure 60: Representative sample of UR-FUNNY dataset.
ImplicitAVE
QUESTION
[Category] Shirts or Tops Columbia Women's Everyday Kenzic Tec. What is Sleeve Style of this product based on the given information and the given image? Answer with the option from the given choices directly.
ANSWER
Long Sleeve
Short Sleeve
Sleeveless
Strappy
3/4 Sleeve
Figure 61: Representative sample of ImplicitAVE dataset.
Twitter2015 2017
QUESTION
The tweet of the image reads: Barcelona Crowned 2016 Liga Champions, Suarez Sink. Determine sentiment toward each target. Choose of the following labels for every target.
ANSWER
Barceloma: positive, La Liga: neutral, Suarez: negative
Figure 62: Representative sample Twitter2001//201 dataset.
58

---

of minor side Published as a conference paper at ICLR 2026
PanoSent
QUESTION
Below is conversation (utterance list). Each line shows utterance ID speaker: text.
I. M.Mkk: I've been rescarching acupuncture and its effectiveness alternative medicine, and think the pain relief aspect is impressive because many patients report decreased symptoms after their sessions.
2.Samir:: Actually, I find the safety aspect somewhat concerning; there effects, such as bruising.
3.Elena: The cost of acupuncture is relatively affordable compared to other treatments, which believe makes accessible.
4. DeDenis: But find the scientific evidence aspect lacking; lot of studies have inconclusive results, which makes me hesitant.
Task: Extract ALL quintuples in this conversation. Each quintuple must be (holder, target, aspect, opinion, sentiment). Respond ONLY with single list of quintuples using single quotes.
ANSWER
('Mrrk,, 'acupuncture', 'pain relief,, 'impressive', ppositive')
('Samira', 'acupuncture,, 'safety', 'concerning', 'negative") ('Elena', accupuncture,, 'cost', 'affordable', 'positive') ('Dennis', 'acupuncture', 'scientific evidence', 'lacking', 'negative')
Figure 63: Representative sample of PanoSent dataset.
MMWTWT
QUESTION
TRIAGE
The tweet of the image reads; What Aetnas $37B acquisition of Humana could mean for digital health (via Juice.li..Targer Merger and acquisition between Aetna and Humana. Determine the stance toward target. Choose one of the following labels your final answer.
Prestuss
ANSWER
Support
Refute
Comment
Unrelated
Figure 64: Representative sample of MMWTWT dataset.
59

---

Published as a conference paper at ICLR 2026
MELD
Sust staiing
nennn Packs
QUESTION
ANSWER
The person in video says: Ohh, yeah she really would've 've wanted to be there. Please watch the provided video and determine the emotion it conveys. Do not provide any additional explana- tions or extra content. Choose one of the follow- ing labels as your final answer.
NEUTRAL
SURPRISE
FEAR
JOY
DISGUST
ANGER
SADNESS
Figure 65: Representative sample of MELD dataset.
FilmStim
AA
QUESTION
ANSWER
Recognize the viewer's elicited emotion after watching the clip. not provide any expla- nations extra text. Choose of the following labels as your final answer.
AMUSEMENT
FEAR
SADNESS
TENDERNESS
NEUTRAL
DISGUST
ANGER
Figure 66: Representative sample of FilmStim dataset.
60

---

Published as a conference paper at ICLR 2026
ElBench
QUESTION
Based only the visual content (facial expressions, body posture, social context, setting), list plausible causes for why the person(s) in the image would feel this emotion. Write concise num- bered list.
ANSWER
TYPE OF REVERAGE IN THE MUG
EVENTS
NATURAL DEMEANOR
WEATHER
TIME OF DAY
RECENT EVENTS IN HIS LIFE
PHYSICAL DISCOMPORT
Figure 67: Representative sample of ElBench dataset.
SMILE
What
What
What
TED
TEL Sus
Sus
Sus
QUESTION
You answer why the audience laughed given the video clip. The corresponding subtitles provided below: O: PENNY: You mean when he was little.
1: PERSON: Not as little as you'd want him to be.
PENNY: You know, this stuff horrible. You want to if can find the cafeteria and get real
coffee? Sure.
3: PERSON: We're going to the cafeteria to get some coffee. You want anything?
4: LEONARD: I'm fine.
RAJ: That's nice that they'r 're getting along
6: RAJ: Your girlfriend and your ex-girlfriend are hanging out together? Oh, ycah, that only be
good for you. Explain why the audience laughed.
ANSWER
The audience laughed because David Birch revealed that he works for banks and hates them more than the protesters do, which was unexpected and ironic, This was underscored by the acoustic FO and energy, and increase in jitter and shimmer, indicating features showing decrease change in Birch's delivery style.
Figure 68: Representative sample of SMILLE dataset.
61

---

Published as a conference paper at ICLR 2026
ECF
QUESTION
Below is a conversation (utterance list). Each line shows [utterance_]]
ID] speaker: text.
[1] Monica: All right, definitely taste nutmeg.
[2] Phoebe: You
[3] Monica: You do not Well, that the difference between professional and layman.
[4] Phoebe: That and arrogance...
[S] Joey: Hey. [6] Monica: Hey How was sailing For ALL target utterances in this conversation,
determine emotion expressed and identify its cause utterance from the conversation. Choose
emotion from: joy, sadness, anger, disgust, fear, surprise, neutral.
ANSWER
2:: l'emotion:: 'surprise', 'cause utterance id': '1', '3:: ('emotion': 'surprise', 'cause. utterance id':
'I') 44: f'emotion:: 'disgust', 'cause utterance '3":
Figure 69: Representative sample of ECF dataset.
MUSIARD
QUESTION
ANSWER
The person in the video says: There girlfriend in there? 'Cause you might need one. Does this statement express sarcasm? Do not provide any additional explanations or extra content. Choose one of the following labels as your final answer.
FALSE
TRUE
Figure 70: Representative sample of MUStARD dataset.
62

---

Published as a conference paper at ICLR 2026
PanoSent
QUESTION
Below a conversation (utterance list). Each line shows [utterancel TD] speaker; text.
[O] Chen: I think the color palette 'Sunrise Valley' is vibrant; every scene feels alive because of the bold saturations.
[1] Amina: The voice acting in the show, however, sounds monotone; it misses emotional highs and lows for me.
[2] Sophia: Actually, I'd argue that the animation style is sophisticated; the subtle character move-
ments reflect attention detail you rarely see in other series.
[3] Marcus: That's fair point about the animation style, Sophia At first, felt the animation style was plain, but given how natural the gestures appear in that dinner scene, have to say it's actually refined.
Identify ALL sentiment flips in this conversation. For each flip, output V/holder': 'People's names', 'initial sentiment': 'positive, negative, neutral" 'flipped's sentiment': 'positive, negative, neutral', 'trigger' type': 'one of the predefined types")3.
ANSWER
('holder': 'Marcus', 'initial sentiment': 'neutral', 'flipped sentiment': 'positive', 'trigger type': 'Logi- cal Argumentation')
Figure 71: Representative sample of PanoSent dataset.
63

---

Published as a conference paper at ICLR 2026
H CASE STUDY
We present representative case studies to complement the quantitative analyses. In each case, the QA specification is fixed so that comparisons are controlled along three axes. First, Figures 72-74 74 compare different models under identical prompts for the same QA, revealing substantial variability final predictions. Second, Figures 75 77 fix the model but switch the response mode between direct answer and ToM prompting, showing how explicit reasoning reshapes intermediate justifica- tions and can alter predicted emotions intents. Third, Figures [78 80 return to the across model setting, applying standardized ToM prompting for the same QA and examining step-by-step traces; despite explicit reasoning, divergences remain and some systems still err. Together, these qualitative results highlight both the strengths and the limitations of current reasoning procedures.
Level 1: Multimodal Emotion Recognition
Question
The person in video says: Also, what do you think Zhao Chong in our family? Determine the emotion conveyed. not provide any additional explanations or extra content. Choose one of the following labels as your final answer: Happy, Sad, Neutral, Angry, Worried, Surprise.
2
video
Answer
Gold label
Happy
Internvi38h
Gp14o
Videollama3
Llavaonevision
Qwcn32b
Internvi8b
QwenviBb
Gemini-pro
Happy
Minicpm
Noutral
Figure 72: Model answers the same QA. A side-by-side comparison of different models' pre- dictions for the same QA, illustrating variability responses across models.
64

---

Published as a conference paper at ICLR 2026
Level 2: Multiparty Dialogue Emotion Recognition
Question
The person video says: You 're not? See, that's why you'le 're great! Please watch the provided video and determine the emotion conveys. not provide any additional explanations or extra content. Choose one of the follawing lahels as your final answer: Neutral, Surprise, Fear, Sadness, Joy, Disgust, Anger.
video
Answer
Gold label
Happy
1
Gpt4o
4
Videoilaama
Llavaonevision
2
Qwen32b
InternviBb
8
QwenviBb
Gemini-pro
Surprise
Minicpm
Internvi38b
Joy
Figure 73: Model answers the same QA.
Level 3: Emotion Elicitation Reasoning
Question
Recognize the viewer's elicited emotion after watching the clip. provide any explanations text. Choose one of the following labels as your final answer: Amusement, Anger, Disgust, Fear, Neutral, , Sadness, Tenderness.
video
Answer
Gold label
Happy
Gpt4o
Videcilamaa
Llavannevision
Qwen32b
Internvißb
QwenviBb
Gemini-pro
Sadness
Minicpm
Infernv/38b
Fear
Figure 74: Model answers the same QA.
65

---

Published as a conference paper at ICLR 2026
Level 1: Sentiment Intensity Analysis
Question
The transcript of this video is (uhh) The action sequences thought didn't even hold up. Determine sentiment intensity category based on video content transcript. not provide any explanations or extra text. Choose one of the following labels as your final answer: Strong Negative. Moderate Negative, Slight Negutive, Neutral, Slight Positive, Moderate Pasitive, Strong Positive.
video
Answer
without Mtyle prompting
Gold label
Gemini2.5-pro
Moderate Negative
Strongly Negative
Question
You multimodal uffective computing expert grounded in theory of mind. Model the...
Answer
with ToM-style prompting
Gemini 5-prr
Decode Polarity Intensity Signals (Perceptual Simulation):
The phrase "this movie fails all levels" direct expression of negative sentiment. The word "fails" curries strong negalive connolations, and qualifier further intensifies the negative judgment, indicating a comprchensive failure. Speech Prosody: The prosody is characterized by stable pitch and moderate loudness, with a deliberate tempo, especially when articulating "fails on all levels." This steady delivery suggests a well-considered, firm opinion rather than spontameous emotional reaction. Visual Rehav ior: The speaker exhibits neutral facial expression, with minimal facial muscle activation, such relaxed brows and lips. The absence of dynamic gesturing exaggerated facial expressions suggests calm demeanor, reinforcing the sense of reasoned critique rather than emotional outburst.
Lexical Content:
levels"
Synthesize Graded Mental State Hypothesis (Cognitive Empathy):
The lexical analysis reveals strong negative sentiment due the comprehensive criticism of movie. The prosodic visual cues, however, indicate compesed and controlled delivery. While emotional intensity appears subdued, the strength of the lexical content implies deep-seated negative judgment. Thus, the speaker's mental state holds a strong negative evaluntive stance, conveyed through calm and measured presentation.
(3) Attribute from the Speakerlu2019, Perspective [Perspective-Iakinne
From the speaker's perspective, the evaluative state is one of decisive disapproval. The speaker is likely experiencing strong of disappointment dissatisfaction with the movie, which is reflected in the assertive language used. The calm delivery undersoores confident assertion of this negative judgment, suggesting clarity their viewpoint rather than emotional detachment.
(4) Conclude and Map Labels:
The combination of strongly negative lexical content and the composed delivery reflects confident and definitive negutive assessment of the movie, Despite the lack of emotional arousal in prosody and visual cues, the speaker's evaluative state is clearly Strongly Negative based on the comprehen- sive nature of their critique.
Figure 75: Same model, direct answer vs. ToM prompting. For the same QA, we compare one model's output when answering directly and when using our ToM prompting, illustrating how ex- plicit reasoning changes the predicted label.
66

---

Published as a conference paper at ICLR 2026
Level 2: Emotion-based Intent Analysis
Question
The person in video says: WOW, THAT SANDWICH REALLY DOES SMELL GOOD. Analyze the emotion and intent. Choose one emotion: Happy, Surprise, Sad, Disgust, Anger, Pear, and Neutral.
intent.Choose Choose intent: Questioning, Agrecing, Acknowledging, Encouraging. Consoling, Suggesting, Wishing, and neutral. Respond in the format: ('emation' 'lahel', 'intent': 'lahel"..
video
without -ayle prompting
Gold label
Geminiz. 2.5-po
Surprise
Acknowledging
Surprise, Neutral
Question
You are a multimodal uffective computing expert grounded in theory of mind. Model the...
Answer
ToM-style prompting
Gemini 2.5 5-ppo
(1) Decode Multimodal & Dialogic Cues (Perceptual Simulation):
Lexical content: The interjection "WOW" is strong indicator of surprise or wonder, often associated with sudden, intense reuction. The phruse "THAT SANDWICH REALLY DOES SMELL GOOD" is emphatic positive appraisal of sensory experienee, with "REALLY DOES" intensifying the aflirmation and implying the speaker is getuinely impressed that the smell exceeded expectations. Prosody: (Not provided text form, but implied by capitalization and "WOW"): The "WOW" and all caps strongly suggests exelamatory and enthusiastic tone, reinforcing the surprise and positive valence. Visual behavior: (Not available). Immediate dialogue context: The utterance spontancous, immediate reaction to perceiving an aroma. functions as a declarative statement sharing a personal, positive sensory experience. While makes an observation that addressee might relate to, it primarily conveys the speuker's internal state rather than initiating a turn for specific response or action from the addressce.
(2) Attribute the Speaker 2019s Emotion (Cognitive Empathy, First-Order)::
The convergence of the strong interjection "WOW" and the highly positive, emphatic evaluation ("REALLY DOES SMELL GOOD") strongly suggests the speaker experieneing surprise. This pleasunt surprise, urising from unexpeciedly delightful or particularly strong sensory encounter with the sandwich's aroma. The "WOW" captures the suddenness the realization, while the descrip- tive phrase grounds emotion a positive experience.
(3) Infer Communicative Intent toward the Addressee (Perspectie-Takiinn & Relational Mapping): The statement primarily functions as an expression of the speaker's immediate, internal sensory experience. It shares this experience with the addressce but does not explicitly roquest a response, suggest an action, question premise, or agree with previous statemend. The speaker is simply voicing observation about their present perception. Therefore, the communicative intent toward the addressce neutral.
Conclude and Map Labels:
'emotion'"Supriie 'intent:: 'Neutral"
Figure 76: Same model, direct answer vs. ToM prompting.
67

---

Published as a conference paper at ICLR 2026
Level 3: Emotion Eliciation Reasoning
Question
Recognize the viewer's elicited emotion after watching the clip. Do not provide any explanations extra text. Choose one of the following labels your final answer: Amnsement, Anger, Disgust, Fear, Neutral, Sadness, Tenderness.
video
Answer
without -tyle prompting
Gold label
InternVL
38b
Neutral
Sadness
Question
You multimodal affective computing expert grounded in theory of mind. Model the...
Answer
with ToM-xtyle prompling
InternVL
38b
Decode Narrative Affective Cues:
The video opens with woman sitting desk classroom, suggesting teaching administra- tive role. expression is neutral, but setting implies professional environment. scene transitions a hallway where two women are conversing. lighting is warm, the interaction seems serious. A close-up shows woman looking at herself mirror, with a contemplative or conserned expression. The mirror scene suggests introspection or self-reflection. Another woman, wearing a beret, is shown in classroom sctting, looking thoughtful or concerned. The final scone shows woman with a distressed expression, possibly indicating sadness worry. Cinematic signals include warm lighting, close-up shots, and reflective pauses, which contribute to somber and introspective mood.
Construct Viewer Appraisal Model:
suggest narrative personal professional challenges, possibly involving conflict or difficull decisions Threat/Safely: expressions sellings imply emotional or relational threats rather than physicul ones. Agency/Blams: characters uppear to grappling with personal or professional responsibilities, possibly questioning their actions or decisions. Norm or Purity Violation: The reflective and somber tone may hint at moral or ethical dilemmas. Attachment or Loss: The introspective moments suggest personal loss or emotional strain.
Goal Congruence/Obstruuciin
(3) Infer the Elieited Emotion:
The dominant viewer appraisal likely be sadness, driven by the characters' expressions of concern, introspection, and distress.
(4) Identify Proximate
Modulators:
The proximate causes are the characters' somber expressions reflective actions, such as looking in the mirror. Cinematic devices like close-up shots and warm, subdued lighting intensify the emotional weight of the scenes.
(5) Conclude and Map to Label:
The combination of introspective actions, somber expressions, and reflective settings leads to the elicitation of Sadness the viewer.
Figure 77: Same model, direct answer vs. ToM prompting.
68

---

Published as a conference paper at ICLR 2026
Level 1:
Song Emotion Recognition
Question
Please watch the provided vidco and determine the emotion conveys. provide any additional explanations or extra content. Chouse one of following labels as your final answer: Neutral, Calm Happy, Sud, Angry, Fearful.
video
Answer with ToM-style prompting
Gold label
Calm
Qwenvl-32b
(1) Decode Affective Signals (Perceptual Simulation) Facial Expression and Micro- Micro-Expreess Expressions
Facial Expression: The performer displays softening of the facial muscles, with a slightly relaxed mouth and eyes that seem thoughtfull or introspective. 'hhe eyes are not sharply focused, often glancing downwards, which adds to calm Serne there is no audio provided... Oneserenn Text: There is onsereen text
(2) Synthesize Performance-State Hypothesis (Cognitive Empathy) Integration of Cues: The facial expression and gaze consistently point toward negative emotional state indicate negative emotional state, specifically that aligns with sadness reflective, somber mood.
(3) Attribute from the Performer's Perspective (Perspective Taking) Refining the Hypothesis: From performer's 's enacted perspoctive... here-and-now emotional state being expressed through their performance.
(4) Conclude and Map to Label Reasoning:
The facial expression, gaze, and body posture all align with negative emotional state. the hypothe- sis that the performer is expressing
Sadness
Answer with MoM-style prompting
Gemini 5-pro
(1) Decode Affective Signals (Perceptual Simulation):: Facial Expression:
Facial Expression: The performer displays softening of the facial muscles, with slightly relaxed mouth and eyes that seem thoughtful or introspective... Iyric: The lyries convey simple imagery, such "leaves rustling in the breeze." which descriptively neutral but delivered with tone suggesting peace and introspection.
(2) Synthesize a Performance-State Hlypothesis (Cognitive Empathy): The convergence of cues points towards state of calmness. The relaxed facial expression, soft gaze, steady posture, and serene vocal qualities all suggest a tranquil emotional state...
(3) Attribute from the Performer 2019s Perspective (Perspective- SaapecttieTTakknggh From the perspective of the performer, they are embodying moment of culm reflection. The delivery suggests peaceful contentment mindfulness,, where the emotion is one of balance and quictude.
(4) Conclude and Map to Label:
The combination the relaxed facial cues, steady gaze, the serene musical delivery strongly points emotional expression of Calm bothe performer.
Figure 78: ToM prompting answers from different models on the same QA. We compare models' reasoning and decisions, illustrating cross-model differences in analytical paths and outcomes.
69

---

Published as a conference paper at ICLR 2026
Level 3: Sarcasm Detection
Question
The person in video says: no, you don't 't have fly back, really. Does this statement express sarcasm? Choose one of the following lahels as your final answer: True, False.
video
Answer with ToM-styye prompting
Gold label
False
Qwen2.5.
Let's analyze statement "No no, you don't have to fly back, really" stop-by-step:
(1) Decode Literal Proposition Affective Cues (Perceptual Simulation):
The literal proposition is "you don't have to fly back." The surface polarity suggests positive stance, implying that the speaker is offering concession or rcassurance.
(2) Model Speaker Intent & Audience Uptake (Second-Order Perspective-Takimg
The speaker seems to be addressing someone... imply that the speaker believes the other person doesn't need to take the action being suggested.
(3) Incongruity Reverall Test (Causal Attribation): There mismaich The "really" at end adds emphasis, suggesting degree disbelief sarcasm.
(4) Decision Rule:
Given the casual tone and the lack of urgency, appears the speaker is not genuinely offering a concession hut rather dismissing the idea with hint of sarcasm.
(5) Conclude and Output: Based the amalysis, the statetent likely contains sarcastic undertone Oulput
Answer with ToM-style prompting
Gemini 2. 5-pro
Tet's break down the statement "No no, you don't have to fly back, really" to see it's sarcastic.
(I) Literal Meaning and Delivery:
The speaker's literal words express that the recipient does not need to return by flying... Therefore, we interpret this direct, unculomed stalement.
(2) Speaker's Intention:
Based solely words and without any additional context, the speaker appears genuinely communicate suggettt the speaker intends audienoe to understand meaning opposite to the literal one.
(3) Checking for Mismatch:
There is no apparent incongruity or contradiction between the literal statement and what might be implied by hidden cues context. The statement seems straightforward, without signs of iromy a deliberate reversal of meaning.
(4) Sareasm Decision:
Given the lack of any evidence for intended opposite mcaning, and no supporting contextual delivery cues that would signal sarcasm, statement is not considered sarcastic. Falss
Figure 79: ToM prompting answers from different models on the same QA.
70

---

Published as a conference paper at ICLR 2026
Level 3: Sarcasm Detection
Question
The person in video says: no, you don't 't have fly back, really. Does this statement express sarcasm? Choose one of the following lahels as your final answer: True, False.
video
Answer with ToM-styye prompting
Gold label
False
Qwen2.5.
Let's analyze statement "No no, you don't have to fly back, really" stop-by-step:
(1) Decode Literal Proposition Affective Cues (Perceptual Simulation):
The literal proposition is "you don't have to fly back." The surface polarity suggests positive stance, implying that the speaker is offering concession or rcassurance.
(2) Model Speaker Intent & Audience Uptake (Second-Order Perspective-Takimg
The speaker seems to be addressing someone... imply that the speaker believes the other person doesn't need to take the action being suggested.
(3) Incongruity Reverall Test (Causal Attribation): There mismaich The "really" at end adds emphasis, suggesting degree disbelief sarcasm.
(4) Decision Rule:
Given the casual tone and the lack of urgency, appears the speaker is not genuinely offering a concession hut rather dismissing the idea with hint of sarcasm.
(5) Conclude and Output: Based the amalysis, the statetent likely contains sarcastic undertone Oulput
Answer with ToM-style prompting
Gemini 2. 5-pro
Tet's break down the statement "No no, you don't have to fly back, really" to see it's sarcastic.
(I) Literal Meaning and Delivery:
The speaker's literal words express that the recipient does not need to return by flying... Therefore, we interpret this direct, unculomed stalement.
(2) Speaker's Intention:
Based solely words and without any additional context, the speaker appears genuinely communicate suggettt the speaker intends audienoe to understand meaning opposite to the literal one.
(3) Checking for Mismatch:
There is no apparent incongruity or contradiction between the literal statement and what might be implied by hidden cues context. The statement seems straightforward, without signs of iromy a deliberate reversal of meaning.
(4) Sareasm Decision:
Given the lack of any evidence for intended opposite mcaning, and no supporting contextual delivery cues that would signal sarcasm, statement is not considered sarcastic. Falss
Figure 80: ToM prompting answers from different models on the same QA.
71


Reviews:
[{"review_id": "6qev9WfFGK", "reviewer": "Reviewer_orYG", "summary": "This paper enhances emotional understanding in multimodal language models by incorporating Theory of Mind (ToM) into reasoning. The authors introduce HitEmotion, a benchmark of emotion-related tasks organized by cognitive depth, and propose ToM-guided reasoning chains that trace beliefs, intents, and feelings before answering. They develop TMPO, a reinforcement learning framework that aligns model reasoning with intermediate mental state sequences using custom rewards. Experiments show that ToM-tuned models outperform both open and proprietary baselines on high-level emotional reasoning tasks, yielding more coherent and human-like rationales. This work contributes a new diagnostic toolkit and a training strategy for cognitively aligned affective reasoning in AI.", "questions": "- How generalizable is the ToM-guided reasoning approach to domains outside of emotion cognition?\n- Do the authors anticipate even better performance if the approach is applied to a larger backbone (e.g., 13B or 70B model)? Or are there diminishing returns because larger models might already internalize some ToM-like patterns?\n- How efficient is the TMPO approach?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- HitEmotion Benchmark - The benchmark’s hierarchical ToM-based structure is unique, filling a gap in existing evaluations by mapping tasks to cognitive reasoning levels (first-order, second-order ToM, etc.). This will be valuable for the community.\n- Ground Truth Reasoning - The reasoning chain curation pipeline shown in Figure 3 is also valuable, where the authors generate intermediate reasoning chains with an LLM and then refine them with human review. This yields high-quality supervision for the model’s thought process, which is a robust approach that enhances the credibility of the results.\n - TMPO Framework - The reward function in TMPO is particularly well thought out – it combines four complementary objectives (Structure Reward, Content Reward, Process Reward and the Consistency Reward) which are shown via ablation to work in synergy.\n- Comprehensive Quantitative Results - The experiments are extensive, covering 24 diverse tasks and comparing a wide range of models. Such evaluation adds confidence that the improvements are real and not cherry-picked.", "weaknesses": "- Scope - The work's focus is confined only to the domain of emotion understanding. It’s unclear how well the proposed ToM-guided reasoning and TMPO training would generalize to other domains (e.g., logical reasoning puzzles, mathematical problem solving, or non-emotional tasks). The paper would be stronger if it discussed or demonstrated applicability beyond emotion-centric scenarios.\n- Scalability to Larger Models - The experiments only use a 7B parameter model (Qwen2.5-Omni-7B). What is the rationale behind choosing Qwen2.5-Omni-7B as the base model? Does the approach scale to larger models as well, or are there diminishing returns because larger models might already internalize some ToM-like patterns, as already shown in Tables 3-5?\n- Base Model Dependency - As acknowledged by the authors, for direct perception driven tasks (e.g. recognizing facial expressions from images), the model lags, primarily due to inherent limitation in the base model. Extending on the previous point about scalability, the reasoning would be more effective/convincing if other better base models are also used. \n- Computational Cost - How efficient is TMPO, both in training and during inference? How does it compare to other baselines? Since the authors choose only a small baseline model (7B parameters) and compare against larger open and closed source models, it would make sense to compare efficiency as well, and even claim it as an advantage if the numbers reflect that.    \n- Reward Component Ablation Study - Although Table 6 indicates that the different reward components are effective and synergistic, it is not exhaustive, e.g. how does the model perform if R_structure is removed? Does it help or degrade performance for other combinations?\n- Comparison with open source models - The authors state that \"Emotion-LLaMA-7B attains 34.18 on the MQE task, outperforming most untuned baselines\" (L420-421). Are there any open-sourced models in Tables 3-5 which are untuned? Emotion-LLaMA-7B only outperforms some open source models, and none of the zero-shot (untuned) closed-sourced models. Comparing against untuned open-source baselines would not be fair.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZJyGbdwXuL", "reviewer": "Reviewer_3ihr", "summary": "This work addresses the limited emotional understanding of multimodal large language models (MLLMs) by emphasizing the need for explicit Theory of Mind (ToM) modeling. The authors introduce HitEmotion, a hierarchical ToM-grounded benchmark, and TMPO, a reinforcement learning method that leverages intermediate mental states for process-level supervision. Experiments demonstrate that HitEmotion reveals deep emotional reasoning deficits in current models, while the proposed ToM-guided reasoning and TMPO significantly enhance emotional reasoning accuracy and coherence.", "questions": "1. Could you analyze the relation among the three levels: EPR, EUA and ECR on model performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. It provides a large-scale datasets on emotion reasoning and a method for constructing ToM reasoning chains.\n2. The experimental results show that the benchmark is useful and valuable, and the ToM reasoning chains can improve the model performance.", "weaknesses": "1. The paper models cognitive abilities into three levels. But it didn't analyze the relation among the three levels on model performance.\n2.  The paper lack some analysis and discussions on the benchmark, e.g., data distribution, data quantity for each task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3Mjf7tcVp", "reviewer": "Reviewer_uj2G", "summary": "This work introduces HitEmotion, a three-tiered evaluation benchmark, and TMPO, a novel preference optimization method. The benchmark is constructed from 24 tasks across 22 datasets with standardized prompts to assess MLLMs' affective capabilities at progressive cognitive depths. The proposed TMPO method is designed to significantly enhance model performance on this benchmark.", "questions": "1. Does the HitEmotion benchmark evaluate the model's capability in processing inputs that may require multi-stage affective analysis? For instance, when provided with Input A (assuming A necessitates the simultaneous application of all three stages for a complete analysis), can the benchmark effectively assess the model's holistic competence across all three stages?\n\n2. If a specific data instance from the original 22 datasets qualifies for multiple tasks among the 24 defined tasks, how is this data instance allocated? What is the principle or methodology for partitioning such multi-qualifying data to avoid data leakage or ensure unambiguous evaluation?\n\n3. Regarding the reward function R, what are the precise computational formulas for each component on the right-hand side of the equation (e.g., R_structure, R_content, R_process, R_consistency)? A detailed description of their calculation is requested.\n\n4. Concerning the parameters (µ1, µ2, µ3, µ4) in the reward function, does the current parameter selection demonstrably represent the optimal configuration? What evidence or ablation studies justify that this particular set of weights is the most effective for the optimization objective?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The study features clear figures and a comprehensive set of experiments. The HitEmotion benchmark establishes itself as a systematic framework for the objective evaluation of MLLMs across the three stages of affective understanding.", "weaknesses": "The methodologies employed in TMPO are relatively common, indicating a potential lack of methodological novelty. The HitEmotion benchmark primarily focuses on evaluating model capabilities at isolated stages—for instance, assessing Stage 1 competence through Input A and Stage 2 through Input B. However, it overlooks the evaluation of multi-stage reasoning capabilities based on a single integrated input. A comprehensive emotional understanding of an event often requires holistic multi-stage reasoning, which the current benchmark design fails to adequately capture. Additionally, there is an issue with duplicated labeling in Figures 24 and 25.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)