MME-Emotion: A Holistic Evaluation Benchmark for Emotional Intelligence in Multimodal Large Language Models
Authors: Fan Zhang
Zebang Cheng
Chong Deng
Haoxuan Li
Zheng Lian
Qian Chen
Huadai Liu
Wen Wang
YiFan Zhang
Renrui Zhang
Ziyu Guo
Zhihong Zhu
Hao Wu
Haixin Wang
Yefeng Zheng
Xiaojiang Peng
Xian Wu
Kun Wang
Xiangang Li
Jieping Ye
Pheng-Ann Heng
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models
Affective Computing
Primary area: datasets and benchmarks
TL;DR: We propose MME-Emotion, the first-ever comprehensive benchmark for emotional intelligence in multimodal large language models (MLLMs).

Abstract: Recent advances in multimodal large language models (MLLMs) have catalyzed transformative progress in affective computing, enabling models to exhibit emergent emotional intelligence. Despite substantial methodological progress, current emotional benchmarks remain limited, as it is still unknown: (a) the generalization abilities of MLLMs across distinct scenarios, and (b) their reasoning capabilities to identify the triggering factors behind emotional states. To bridge these gaps, we present MME-Emotion, a systematic benchmark that assesses both emotional understanding and reasoning capabilities of MLLMs, enjoying scalable capacity, diverse settings, and unified protocols. As the largest emotional intelligence benchmark for MLLMs, MME-Emotion contains over 6,000 curated video clips with task-specific questioning-answering (QA) pairs, spanning broad scenarios to formulate eight emotional tasks. It further incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning, analyzed through a multi-agent system framework.
Through a rigorous evaluation of 20 advanced MLLMs, we uncover both their strengths and limitations, yielding several key insights: (1) Current MLLMs exhibit unsatisfactory emotional intelligence, with the best-performing model achieving only $39.3\%$ recognition score and $56.0\%$ Chain-of-Thought (CoT) score on our benchmark. (2) Generalist models (\emph{e.g.}, Gemini-2.5-Pro) derive emotional intelligence from generalized multimodal understanding capabilities, while specialist models (\emph{e.g.}, R1-Omni) can achieve comparable performance through domain-specific post-training adaptation. By introducing MME-Emotion, we hope that it can serve as a foundation for advancing MLLMs' emotional intelligence in the future.

we hope that can serve Published as a conference paper at ICLR 2026
WWE-Umetion
MME-EMOTION: A HOLISTIC EVALUATION
BENCHMARK FOR EMOTIONAL INTELLIGENCE IN MUL- TIMODAL LARGE LANGUAGE MODELS
Fan Zhang 11.2, Zebang Cheng* Chong Deng" 22, Haoxuan Li Zheng Lian", Qian Chen", Huadai Liu", Wen Wang", Yi-Fan Zhang,, Renrui Zhang', Ziyu Guo Zhihong Zhu", Hao Wu?, Haixin Wang", Yefeng Zheng", Xiaojiang Pengl.',
Xian Wu', Kun Wangi.10, Xiangang Li Jieping Ye", Pheng-Am Heng
'The Chinese University of Hong Kong Tongyi Lab 3SZTU *Peking University TTnggi University "CASIA Tencent SULA 9 Westlake University INNTU fzhangê 1 ink. cuhk. edu.hk, hk, phengêcse cse.
ABSTRACT
Recent advances in multimodal large language models (MLLMs) have catalyzed transformative progress in affective computing, enabling models to exhibit emer- gent emotional intelligence. Despite substantial methodological progress, current emotional benchmarks remain limited, as it is still unknown: (a) the generalization abilities of MLLMs across distinct scenarios, and (b) their reasoming capabilities to identify the triggering factors behind emotional states. To bridge these gaps, present MME-Emotiion understanding and reasoning capabilities of MLLMs, enjoying scalable capac- it,, diverse settings, and unified protocols. As the largest emotional intelligence benchmark for MILMs, MME-Emotion contains over 6,000 curated video clips with task-specific questioning annwerring (QA) pairs, spanning broad scenarios to formulate eight emotional tasks. It further incorporates a holistic evaluation suite with hybrid metrics for emotion recognition and reasoning. analyzed through a multi-agent system framework. Through rigorous evaluation of 20 advanced MLLMs, we uncover both their strengths and limitations, yielding several key insights: ① Current MLLMs exhibit unsatisfactory emotional intelligence, with the pest-performig model achieving only 39.3% recognition score and 0% Chain-of-Thoughee (CoT) score on our benchmark. @ Generalist models (e.g., Gemini-2.5-Pa and GPT-4o) derive emotional intelligence from generalized mul- timodal understanding capabilities, while specialist models (e.g., R1-Omni and AffectGPT) can achieve comparable performance through domain-specific post- training adaptation. By introducing a foundation for advancing the emotional intelligence of MLLMs in the future. Project Page: https: /mne- emot i on., gi thub. io/.
systematic benchmark that assesses both emotional
INTRODUCTION
Affective computing (Picard, 2000; Tao & Tan, 2005; Zhang et al., 2024c; 2025a; Mao et al., 2025) is fascinating interdiseiplinary field seeking bridge the gap between human emotional intelligence and machine capabilities, with significant applications in education (Wu et al., 2016; Yadegaridehkordi et al., 2019), healthcare (Yannakakis, 2018; Liu al., 2024b; Zhang et al., 2025b; Zhu et al., 2025), and human-robot interaction (Filippini et al., 2020; Gervasi et al., 2023). Catalyzed by emergent multimodal large language models (MLLMs) (Wu et al., 2023; Zhang et al., 2024b), research focus has increasingly shifted from exploring emotion recognition capability (Kosti et al., 2017) to investigating the interpretability of clues behind emotional states (Lian et al., 2024c).
Equaal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2026
Distribution Somples Across Different Tasks
Gumedyy
Russicc
TV Droro
Servar
Number of Quastiars
1500
100
1600
1600
Fumily
vilg
900 5000
Lifratyle
Addiiee
Enage Bur Nodie
Welcchunm
MME Emotion
Horse EP
ER-14b BX-Wid Nobe-ER 76-ER
ML-ER
5A
PS-54
Longenge
Intonirs
Distribution of Videos Across Different Tasks
Modero
Camedy
Length of Videos (3)
5.07
4.74
Love
3.81
Film
Sappesse
ER-14b BR-Wid Nobe-E 76-ER
ML-ER
5A
P3-54
Figure 1: Overview of MME-Emotion Statistics. Left: Task Types. MME-Emotion encompasses eight emotional tasks across 27 distinct scenario types, enabling fine-grained analysis of diverse video contexts. Right: Data Distributions. MME-Emotion features balanced distributions of question volume and video duration, facilitating comprehensive evaluation of temporal understanding.
Recognition Score
Reasoning Score
CoT T Score
Ei-Lob
EB-140
Ei-Lub
IR-WI
IR
Wid
IR
ER-Wid
F6-54
Nase-ER FS-S4
Make Ek F6-54
No
96-BR
F6-ER
16-8R
ML-ER
ML-ER
ML-ER
Audio VL-
GPT-46
GPT-4.1
Genini-2
Figure 2: Performance Comparison of Leading MLLMs on MME-Emotion. Our evaluation suite assesses MLLMs using three unified metrics (Ree-S, Rea-S, and CoT-S) across eight emotional tasks.
While a growing body of methodological research (Cheng et al., 2024a; Zhao et al., 2025b; Lian al., 2024c) explores the use of multimodal (audio, visual, and textual) data to boost emotional intelligence in MI.LMs, existing evaluation benchmarks (Liu et al., 2024a; Hu et al., 2025) suffer from notable limitations, including (a) inadequate scenario coverage and (b) inconsistent evaluation protocols. Consequently, the extent to which current advanced MLLMS demonstrate emotional intelligence remains unclear under open and fair settings, as are still unknown: ① how effectively they generalize across different real-world scenarios, @ how accurately they can recognize emotional states, and ① how well they are capable of identifying the triggering factors behind those emotions.
Towards this end, we present benchmark for MI.LMs, featuring scalable capacity, diverse settings, and unified protocols. As shown in Figure 1, MME-Emotion consists of 6,500 video clips associated with task-specife question- answering (QA) pairs across 27 distinct scenario types to formulate eight emotional tasks, including emotion recognition in the lab (ER-Lab), emotion recognition in the wild (ER-Wild), emotion recognition under noise (Noise-ER)) fine- tine-grained emotion recognition (FG-ER), multi-label emotion recognition (ML-ER), sentiment analysis (SA), fin-grained sentiment analysis (FG-SA), and intent recognition (IR). The distributions of question volume and video duration are balanced across all tasks, with each task containing minimum of 500 QA pairs and video clips averaging >3. seconds.
the first-ever comprehensive emotional intelligence
2

---

Published as a conference paper at ICLR 2026
Going beyond this, provide holistic evaluation suite for assessing the capabilities of MLLMs in emotion recognition and reasoning using unified protocols across all sub-tasks within AME-Emotion. For each question, employ multi- agent system framework to enable automated evaluation of MLLMs' responses with an MLL-Maas-judge strategy. The visual clues, extracted audio clues, ground- truth emotion labels, and partitioned answer steps of specific MILMM are fed into GPT-based judge agent to evaluate the performance using three metrics: recognition score, reasoning score, and Chain-of-Though (CoT) score. To further validate our evaluation approach, we also ask five human experts to cros--evaluate the performance of MLLMs on sampled data and manually annotated scores at each answer step. The comparison between GPT and expert scores demonstrates high consistency across multiple statistical metrics, confirming the effectiveness of our automated evaluation strategy.
Applying our evaluation suite to 20 state-of-the-an MLLMs, we uncover both their strengths and limitations, yielding the following key insights: ! The overall emotional intelligence of current MLLMS remains far from satisfactory (Figure 2). Even the top-performing model Cremini-2.5-PPrr achieves merely 39.3% recognition score and 56.0% CoT score our benchmark, respectively. The average performance across all evaluated MLLMs (29.4% recognition score, 49.5% reasoning score, and 39.5% CoT score) indicates there is still substantial room for improvement. @ While generalist models (e.g., Gemini-2.5-PP
(Google, 2025c) and GPT-4o (OpenAI, 2024)) derive emotional intel- ligence from generalized multimodal understanding capabilities, specialist models (e.g., R1 R1-Omni (Zhao et al., 2025a) and Audio-Reasone (Xie et al., 2025)) can achieve comparable performance through emotion-specific post-training adaptation techniques, such as supervised fine-tuning (SFT) (Zhang et al., 2023b; Liu et al., 2023b) and human preference alignment (Guo et al., 2025; Liu et al., 2025). 2 Generally, response step count positively correlates with model performance, underscoring the necessity for equipping MLLMs with emotion reasoning capabilities in future development.
The main contributions of this paper can be summarized as follows:
Comprehensive Benchmark: We introduce MME-Emotion the largest benchmark for emotional intelligence in MLLMs that encompasses eight emotional tasks and 27 distinct scenarios, enabling a comprehensive evaluation of MILMs' generalization capabilities across diverse settings.
Holistic Evaluation Suite: We provide a multi-agent system framework that can automatically assess the emotion recognition and reasoning abilities of MILMs using three unified metrics across all tasks. The validity of our evaluation strategy is also fully verified by five human experts.
Empirical Analysis: Through rigorous evaluation of 20 advanced open-source and closed-source MLLMs EMMtioooion, along with in-depth analysis, we reveal their strengths and limitations, paving the way for future research efforts to advance emotional intelligence in MLLMs.
2 RELATED WORK
2.1 EMOTIONAL INTELLIGENCE IN MULTIMODAL LARGE LANGUAGE MODELS
The success of MILMSs (Liu et al., 2023a; Zhu et al., 2023; Luo et al., 2025b; Wang et al., 2025) has advanced affective computing, prompting growing interest in equipping these models with emotional intelligence (Cheng et al., 2024a; Zhao et al., 2025b; Lian et al., 2025a; Zhu al.; Lian et al., 2025b). On the one hand, general MILMS (such as the Gemini (Team et al., 2023), GPT (Achiam al., 2023), and Qwen-VL (Bai et al., 2023) series) have demonstrated strong multimodal understanding capabilities. Thanks to their powerful LLM backbones and extensive pretraining on diverse datasets, emotional intelligence emerges other hand, MILMs with relatively small parameter sizes (ranging from 0.5B to 7B) have not yet shown strong general multimodal understanding capabilities, primarily due to limitations in data scale and model capacity. However, with post- post-training adaptation strategies, it is still possible to elicit strong performance on emotion-related tasks. For example, Emotion-L.LaM. (Cheng et al., 2024a) employs amotion-specifie encoders to project video, audio, and text modalities into a unified LLM space, followed by instruction tuning to activate emotional intelligence. RI-Omni (Zhao et al., 2025a) enhances emotional reasoning capabilities in MLLMs through reinforcement learning with verified feedback (RLVR). AffectGPT (Lian et al., 2024c), by training on large-scale emotional dataset and introducing pre-fusinn projector to integrate multimodal emotional signals, builds powerful emotion -seciiln- model. However, these models are often evaluated only on specific tasks and limited scenarios, leaving it unclear how they perform under open and fair evaluation settings.
byproduct of these general models. On the
3

---

Published as a conference paper at ICLR 2026
Table I: Comparison of MME-Emotion with other Benchmarks related to Emotional Intelli-
gence. Rec-A and Qea-Q are short for recognition accuracy and reasoning quality, respectively.
Benchmark
Task
Modality
Rec-A
Rea-Q
QA
LLM Eval
Human Assist
EmotionBench (Huang et al., 2023)
Text
K
/
EmoBench (Sabour et al., 2024)
Text
K
/
MOSABench (Song et al., 2024)
Image
X
MM-InstrucEVal (Yang et al,, 2025)
Image
X
J
IEMOCAP (Busso ct al., 2008)
Video
X
MC-EIU (Liiu et al., 2024a)
Video
OV-MER (Lian al., 2024a)
Video
EmoBench-M (Hu al., 2025)
Video
MER-Uni BeRRhhhienchhh (Lian et al., 2024c)
Video
MME--mmttion (Ours)
Video
2.2 DATASETS AND BENCHMARKS FOR EMOTIONAL INTELLIGENCE
Numerous efforts (Busso et al., 2008; et al., 2017) have been made to establish an open and fair environment for evaluating emotional intelligence in artificial intelligence (Al) models. However, most existing datasets and benchmarks were developed before the emergence of LLMs and MLLMs, primarily aimed at assessing the performance of traditional machine learning and deep learning models. Although some recent studies (Lian et al., 2024c; Hu et al., 2025) have attempted to adapt these datasets for MILMs using prompt templates, common limitation still remains: they focus solely on evaluating emotion recognition capabilities, while overlooking the models' abilities to reason about emotional cues. To address this gap, propose automated evaluation strategy, along with newly introduced metric termed reasoning score, to quantify the reasoning abilities of MLLMs.
3 MME-EMOTION
In this section, first introduce the data curation process, followed by an elaboration evaluation suite, including a multi-agent system framework and three unified metrics. As shown in Table compared with existing emotion-related benchmarks, MME-Emotion stands out as the only one that simultancously accounts for both recognition accuracy and reasoning quality.
3.1 BENCHMARK CONSTRUCTION
To curate data and construct our benchmark while reducing annotation costs, we collect videos and their corresponding emotion labels from publicly available resources. For longer videos that exhibit emotional shifts over time, we segment them into shorter video clips based on timestamps and the consistent emotion labels within specific intervals. We then convert various tasks into a QA format using prompt templates. Given that current MLLMs still lack sufficient emotional intelligence to handle open-ended tasks, we include all candidate emotion labels as a predefined label set within the prompt, enabling the model to make predictions in a closed-set setting. By aggregating and resampling data from multiple public datasets (Busso et al., 2008; Poria et al., 2018; Jiang et al., 2020; Liu et al., 2022b; Lee et al., 2019; Liu al., 2024a; Feng al., 2024; Luo al., 2020; Lian et al., 2024b; Zadeh et al., 2018; 2016; Liu et al., 2022a), we compile total of 6,500 QA pairs along with their corresponding video clips. To prevent potential data leakage. all involved samples are exclusively drawn from the test sets. These samples span eight emotion-related tasks across 27 distinct scenarios, including emotion recognition in the lab (ER-Lab), emotion recognition in the wild (ER-Wild), emotion recognition under noise (Noise-ER), fine- tne-grained emotion recognition (FG-ER), multi-label emotion recognition (ML-ER), sentiment analysis (SA), fine-grained sentiment analysis (FG-SA), and intent recognition (IR). By constructing the MME-Emotion benchmark, we provide comprehensive and systematic environment for assessing emotional intelligence MLLMs.
3.2 EVALUATION SUITE
Evaluation Strategy. Evaluating the reasoning capabilities of MLLMs' emotional intelligence is challenging task. To address this, we propose annotation-free strategy based multi-agent
4

---

Published as a conference paper at ICLR 2026
Question: As on emotion recognition expert, identify the most prominent emotion displayed the video.
Candidate labels: Fear, Disgust, Neutral, Anger, Happiness..... Please think step by step.
MAK LOAERT
Enclose your enswer within <answer>/answer> tags,
Extract Audio Clues
Extract Key Steps
Gemini
2.5-
Ploas onchaa whiic festuna the i neflect that speaher is feaking eEiente Lebell Pay specea otteonion to arstiaa fechures eeplosis and stress, viice treetiing and aecitheer enartion diensceeetiic
Sieen the follzwing (fledd Respradd about the enotion corveved tha viden plaase satnect sowr key stepe frue asouar Exch the let step, cheecore the predected anotion.
Gemini
(Wodd laspond) icernity mast vouil
anation
(Brettion Labell aager
vidoo
Audio
Agents
I Answer to Step
WaeeLLerA
AfRICEGIT
through their auprassive fere. laat usicing and the of foroffid langage wellcting they one mediingg
The spesker ech
urtep:Step Anayyae foozl express and eyetione peshar, stertify poosible epatiant. SSay andyas
neath operners
Evaluated MLLMs
spect
Human Verification
Step 1: Stop 2:
0.O
Step 5:
Spearmon:
Strong Foulbive Perfect Excellent
Visual Clues
Audio Clues
Emotion Labels
Answer Steps
Judge Agent
Koppo: 2.8626
ICC: 0,9704
Overkipping Hittor and beaity Gures
Step 1: 1/1, Step 2: 1/1, Step 3: 0/1, Step 4: 1/1, Step 5: 0/1
Exert
Reasoning Score: 75
Recognition
Evalucting
Evaluation Suite
COT Score: 0.375
Sampling
Figure 3: Illustration of Our Evaluation Strategy. We leverage a multi-agent system framework to assess the recognition and reasoning capabilities of MLLMs across different tasks with three unified metrics. To validate the effectiveness our MLLM-as -uujuee strategy, we further compare the results of the judge agent on sampled data against results cross-evaluated by five human experts.
system (Figure 3). This approach allows us to assess the reasoning performance of MLLMs without the need for manually annotated ground-truth reasoning steps. Specifically, we first obtain the MI.MMss response to a given question, and then employ a unimodal step agent to automatically extract key reasoming steps from the model's answer. This process can be formulated as:
= MLLM V), 5 = Step-LLMMP...A),
(1)
where Q. V. and A are short for question, video, and answer, respectively. P. and S denote the prompt for the step agent and the corresponding answer steps. Here, we opt for GPT-4.1 (OpenAI, 2025) our step agent. Note that refrain from incorporating additional dlements-Suce as the original question or the video content-durinn the step partition process, as we aim to avoid introducing external biases that could influence the identification of reasoning steps. Afterward, we employ another powerful multimodal agent as the judge to rate the performance of the extracted answer steps. To ensure an accurate and comprehensive assessment, is essential to provide the judge agent with complete multimodal information, thereby minimizing the risk of misjudgment due to missing context. However, mainstream multimodal agents such as GPT-4o (OpenAI, 2024) are currently unable to process all modalities simultaneously. To address this limitation, we propose and-c--unn perspective. Specifically, for visual clues, we can directly convert the original video into frames and feed them into the judge agent. For audio clues, utilize separate, powerful audio-lange audio-language model as an audio agent to extract relevant audio information. Once we obtain the visual and audio clues, combine them with the answer steps and the ground-truth emotion labels and input them into the judge agent for final evaluation. This process can be formulated as:
C. CV) C. Audio-LLM((P V), Rec-S, Rea-S = Judge-MLLLM(?. C., Ca, Y,S),
(2)
(3)
where Co, C.. and Y denote visual clues, audio clues, and emotion labels, respectively. P. and P, are the prompts for extracting audio clues and evaluating the performance. Here, we employ Qwen2-Audio (Chu et al., 2024) the audio agent and GPT-4o (OpenAI, 2024) the judge agent. We adopt three unified evaluation metrics across all tasks: recognition score (Ree-S), reasoning score (Rea-S), and Chain-of-Thhinee -fhouhh score (CoT-S), each of which will be detailed in the following section.
5

---

Published as a conference paper at ICLR 2026
Table 2: Overall Performance Comparison (%) on MME-Emotion. The top three performing
results are highlighted in red (1*), blue (2"h), and yellow (3t) backgrounds, respectively.
Model
LLM Open-source MLLMs
T
Avg Step
Avg Token
Ree-S
Rea-S
CoT-SS
Qwen2-Audio (Chu et 2024)
7B
3.0
40.3
34,1
50.4
42.3
Audio-Reasoner (Xie et al., 2025)
7B
5.0
356.8
38.1
71.6
54,8
Qwen2-VL-7B (Wang et al. 2024)
7B
4
V
2.9
68.0
29.2
38.1
33.7
Qwen2-VL-722 (Wang al., 2024)
72B
1.5
31.1
10.5
Qwen2.5-VLL-7B (Bai al., 2025)
7B
V
4.8
28.4
64.8
Qwen2.5-VI.-228 (Bai et al., 2025)
72B
V
V
4.8
266.3
31.3
75.7
53.5
QVQ (Team, 2024)
72B
V
5.5
Video-LLaVA (Lin al,, 2023)
7B
V
2.3
19.1
25.8
32.8
29.3
Video-LLaMA (Zhang al., 2023c)
7B
V
Y
4.5
122.5
26.1
48.5
37.3
Vidco-LLaMA2 (Cheng ct al., 2024b)
7B
V
V
2.6
37.7
29.2
27.7
28.4
Qwen2.5-Omni (Xu et al., 2025)
7B
V
3.7
78.6
17.4
59.3
38.4
Emotion-LLaaMA (Cheng
2024a)
1.0
25.1
HumanOmmi (Zhan al., 2025b)
7B
1.0
1.3
36.0
0.3
18.1
R R1-Omni (Zhao ct al., 2025a)
0,5B
V
5.0
156.2
26,3
58.6
42.4
AffecGPPT (Lian et al., 2024c)
7B
4.9
122.8
11.9
50.6
31.2
Closed-source MLLMS
GPT-4o (OpenAI, 2024)
4.4
169.4
79.8
GPT-4, (OpenAT, 2025)
5.2
141.2
28.8
65.2
47.0
Gemini-2.0-Flam (Google, 2025a)
4.1
64.7
36.3
60.0
48.1
Gemini-2.5-Flan (Google, 2025b)
v
4.3
261.8
52.7
43.7
Gemini-2.5-Por (Google, 2025c)
5.1
538.6
39.3
72.7
56.0
Evaluation Metric. Next, provide detailed introduction to the evaluation metries employed in this paper. During the earlier step extraction process from MLLM answers, we consistently place the task-specific prediction in the final step, while treating all other steps the reasoning process. To compute the recognition score, we compare the final prediction step against the ground-truth emotion labels. For tasks involving single emotion label, adopt standard accuracy as the recognition score, following conventional evaluation practices (Li & Deng, 2020; Tzirakis et al., 2017). For tasks involving multiple emotion labels, the recognition score is defined as the ratio between the number of correctly predicted emotions and the total number of ground-truth emotions. We then derive the reasoning score from the extracted reasoning steps. Each step treated as binary classification problem, where the judge agent determines whether the reasoning is correct not. To eliminate potential bias caused by varying step lengths, we compute the reasoning score as the average correctness across all reasoning steps. Finally, we obtain the CoT score by taking weighted combination of the recognition score and the reasoning score, providing holistic measure both recognition accuracy and reasoning quality. The CoT score is defined as:
CoT-S Rec-S (1) a.) Rea-S,
where a E [0, 1] is hyperparameter that controls the relative contribution of the recognition score and reasoning score in the overall evaluation process. By default, we a 0.5.
Human Verification. To evaluate the reliability of our MLLM-as-judge evaluation strategy, we recruited five experts to perform manual validation. Specifically, the human experts annotated 373 reasoning steps drawn from 100 randomly sampled questions and their corresponding answers from evaluated ML.MMs. Then we assessed the consistency between the GPT scores and the expert scores. As shown in the bottom-right panel of Figure 3, the Spearman's rank correlation coefficient (Spearman, 1961) between the two sets of scores is 0.9530, indicating a strong positive correlation. terms of inter-rater reliability, the two sets of scores achieve Cohen's 's Kappa coefficient (McHugh, 2012) of 0.8626, which falls into the category almost perfect agreement. Additionally, the intra-class correlation coefficient (ICC) (Koch, 2004) of 0.9704 further confirms excellent reliability between the two scoring methods. These results collectively demonstrate that our MLLLM-as-jude strategy aligns closely with human judgment and serves highly reliable evaluation approach.

---

Published as a conference paper at ICLR 2026
Task- level Recognition Score
Task-level Reasoning Score
Task-level CoT Score
Adde- Aeasinur
22.6
26:0
474
61
AL4
502
305
214
T24
604
472
275
B25
749
Addl Kaabonar
$2.
644
2622
593
4012
Spead RL-7DD
340
300
807
Qpent RU--
15.4
Ownll 5-11-70
20.9
274
135
234
36.
122
Genel 5-7
420
953
02.9
022
152
Cwril 5-11-700
434
533
344
953
4092
11:2
314
294
05.0
434
430
118
452
624
61b
727
674
152
0212
410
44.7
50.4
40.2
537
303
356
54-8
LuaRA
20.7
6PA
31.
Wilee MAAMA
Video LLMM42
25.4
45.5
213
481
375
150
442
13.0
Video HoMAT?
110
150
155
325
372
155
Certov-LaRA
210
14.5
$0.4
1037
57
80L
54
213
Bruinee LLAMA
OZ
60
0.0
85
16.3
204
75
32
131
2.5
120
22.1
560
52.8
67A
412
N2
Ki-denk
302
442
#fferdry
309
176
64.
419
505
402
#tferder
412
250
953
501
20.3
6m--4.1-
21:2
15.5
32.4
22.4
33C
333
74
267
PP--LL--
466
548
613
673
N22
743
GIT-4.1
3133
140
472
40.9
554
543
561
512
54.5
722
274
10.5
502
402
M4
Cenri
S73
912
613
50.0
35.
420
26-140
Figure 4: Task-level Performance Comparison (%) on MME-Emotion. We showcase fine-grained comparison results of 20 MILMs using Rec-S, Rea-S, and CoT-S across 8 emotional tasks.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate the performance on a total of 20 cutting-edge MLLMs, including Qwen2-Audio (Chu al., 2024),
(Xie et al., 2025), Qwen2-VL--7B/72B (Wang et al., 2024), Qwen2.5- VL-7B/72B (Bai et al., 2025), QVQ (Team, 2024), Video-LLaVA (Lin et al., 2023), Video-LLaMM (Zhang et al., 2023a), Video-LLaMA2 (Cheng et al., 2024b), Qwen2.5-Omni (Xu et al., 2025), ELLMiin-LLaMA (Cheng et al., 2024a), HumanOmni (Zhao et al., 2025b), R1-Omni (Zhao et al., 2025a), AffectGPT (Lian et al., 2024c), GPT-4o (OpenAI, 2024), GPT-4.1 (OpenAl, 2025), Gemini- 2.0-Flash (Google, 2025a), Gemini- 2.5- (Google, 2025b), and Gemini-2.5-PPr (Google, 2025c). For most of the open-sourced MLLMs, we use the vLLM (Kwon et al., 2023) framework to perform inference on our benchmark. For models that are not compatible with the vLLMM framework, we conduct experiments using the official open-source code provided by the authors. For glosed-sourced MLLMs, inference is conducted directly via their official APIs. All the experiments are conducted under zero-shot settings. To ensure a fair comparison across models, we maintain consistent prompting wherever possible, with only minor modifications when necessary. For example, for audio-language models, the word "video" in prompts is replaced with "audio"; for vision- language models that do not support video input but accept multiple images, "video" is replaced with "video frames".
4.2 MAIN RESULTS
We showcase the overall performance comparison on MME MME-Emotion, covering three performance metrics: recognition score (Rec-S), reasoning score (Rea-S), and Chain-of-Though score (CoT-S) as well as two response length metrics: average step count (Avg Step) and average token count (Avg Token), as shown in Table 2. Notably, even the top-performing models, Gemini-2.5-PP (Google, 2025c), Andio-Reasoner (Xie et al., 2025), and GPT-4o (OpenAl, 2024), achieve CoT scores of only 56.0%, 54.8%, and 53.8%. respectively. All evaluated MLLMs report recognition scores below 40%, and most closed-source MLLMs also score below 40% in terms of CoT quality. This performance landscape highlights the challenging nature of the MME-Emotion benchmark. It also suggests that even Mfte-of-thhe-ar MILMss are still in the early stages of developing emotional intelligence.
7

---

Published as a conference paper at ICLR 2026
Average Staps vs CoT Score
Reasoning Score vs Recogrition Score
Convelation Matrix
Agg Tole
5:43
1.40
GTT Segr
0.55
Mec-
4006
6U8
1.00
000
Fex-5
6:50
0:00
0.96
1.00
Kag Dry Auy
Bark
Figure 5: Relationships among Model Evaluation Metrics. Left Panel: Relationship between average steps and CoT scores. Center Panel: Relationship between reasoning and recognition scores. Right Panel: Pearson correlation quantifying inter-dependencies among five evaluation metrics.
We also observe that current MLLMs have yet fully and effectively leverage multimodal information. As shown in Table 2, Audio-Reasoned which processes only audio and textual modalities, achieves a strong CoT score of 54.8%. Similarly, many closed-source vision-language models, utilizing only visual and textual information, also demonstrate competitive performance, In contrast, omnimodal models, which are designed process audio, visual, and textual information simultaneously, often exhibit noticeable performance drops. This phenomenon points to two potential issues: Multimodal data often contains redundant or inconsistent emotional clues across different modalities. @ Existing omnimodal models still lack effective strategies for robust multimodal emotional clues fusion.
In addition, Figure presents fine-grained comparison of 20 MLLMs across eight distinct emotional tasks, evaluated using three performance metrics. The results reveal notable differences in task difficulty. For more challenging tasks such as FG-SA and IR, even the top- pop-pefoormin MLLMs achieve recognition scores of only around 30%. When comparing the results of two related tasks, ERId and ER-Lab, we observe clear performance gap across different MILMMs This discrepancy suggests that most existing MLLMs are primarily trained in-the-wild data and therefore struggle to generalize effectively to controlled, in-the-lab settings, leading to noticeable performance degradation. These findings further underscore the discriminative capability of the AME-Emotion benchmark.
4.3 OBSERVATIONS AND INSIGHTS
Delving into the experimental results, derive the following key observations and insights.
Obs. 1: Emotional intelligence in MLLMs can be bootstrapped through diverse approaches. On the one hand, generalist MLLMs, such as the Gemini and GPT series, are equipped with large model capacities and trained on massive datasets. As result, they exhibit strong general understanding capabilities and perform well across various tasks (Li et al., 2024; Fu et al., 2023; Zhang al., 2024d; Fu et al., 2025; Jiang et al., 2025). Although these models have not been explicitly adapted to emotion-specific domains (Fu et al., 2024), our experimental results show that they can generalize from general intelligence emotional intelligence, illustrating one viable path forward. On the other hand, specialist models, such as AffectGPT and RI R1-Omni, may not match generalist models in broad understanding domains due the limitation of scaling laws (Kaplan al., 2020; Zhang et al., 2024a). However, by constructing large-scale emotional data and applying post-training adaptation, they can achieve comparable performance against generalist models. These findings demonstrate that there is no one-size-fite fitt-alll solution: emotional intelligence can be bootstrapped through a variety of strategies, opening the door to a broad range of future research directions.
Obs. 2: Encouraging deeper reasoning can enhance emotional intelligence in MLLMs. In Figure 5, there is general positive correlation between the average number of answer steps and overall performance. While certain non non-reasoning MLLMs, such as Emotion-LLaMA, HumanOmni, and Qwen2-VL-72B, can generate direct answers without explicit reasoning through instruction tuning, most caaasoning-capabll MLLMs demonstrate a strong synergy between reasoning and recognition: improvements in one metric often accompanied by gains in the other. These findings suggest a promising research direction for advancing the emotional intelligence of MLLMs: encouraging deeper reasoning by incentivizing models to reflect and identify the underlying triggers of emotions.
8

---

Published as a conference paper at ICLR 2026
Question: As amotion recagnition expert, identify the most prominent emotion disployed the video. Please think step step.
Condidate lobels: Happy, Neutral, Angry, Sod, Surprise, Fear, Disgust.
Audio Clues: The speaker exhibits several key emational indicators of fear: 1. High pitch and fast speech nate: This indicates that the speaker be nervous ponicked and trying to express his emations quickly. 2, Trembling voice: Trembling voice monifestation of which usually occurs
the speoker under much pressure emationally excited, Nasal and inkaled sounds: These vocal features may indicate that speaker trying control his breathing
thus relieve emotional pressure. 4. Internittent pouses and repeated words: This sentence structure may indicate that the specker is trying to ongarize the longuage, while showing that they ore uneasy afraid of what they are about to say. 5, Loud and crying Tone: This further emphasizes the mension and panic of the speaker's situation, Combining the analysis, speoker this audio clip shows abvicus fear. Emotion Lobels: Fear.
LLoMA:
Roting:
Step Observe the young man't behavior facial expressions
situations within video. Step 2: Note, that despite being in potentially streseful scerarios (e.g. a spider crawling on his face in dork cave). he does
0/1
not displcy any strong emotional reactions.
Step 3: Conclude that the young mon consistently shows no visible emotional expression throughout the video.
0/1
Step 4: The prodicted emotion is neutral.
0/1
Qwen2. 5-
Rating:
Step I: Anolyze the person's eyes for emotional clues, noting that wide open eyes of indicate surprise or shock.
0/1
Step 2: Examinz the mouth, observing that a slightly open mouth can be sign of surprise or fear.
1/1
Step Based
visual clues, determine that the predicted
surprise.
Gemini-2,5 2.mini-2,5-Prr::
Roting:
Step 1: Examine facial clues in each frame, focusing on eyes, eyebrows, and mouth positions.
1/1
Shep 2: Identify key features such as wide open eyes, roised eyebrows, and dropped
clearer fromes.
1/1
Step 3: Compare observed features with typical expressions of condidate emotions (neutrol, happy, angry, sad, fear,
1/1
surprise, disgust).
Step 4: Determine that the combination wide eyes, raised eyebrows, on open mouth best matches expression
surprise.
Step 5: The predicted emotion is surprise.
0/1
Figure 6: Cases of Error Responses. We showcase some typical error types of cutting-edge MLLMs.
Obs. 3: Limited visual perception capabilities constrain the emotional intelligence in MLLMs. As illustrated in Figure 6, the failure cases of Video-LLaMAN and Qwen2.5-Omn reveal that both models struggle primarily due to insufficient visual perception capabilities. In this example, Video- LLaMA2 fails accurately capture subtle facial expression changes required and thus incorrectly classifies the emotion as neutral. Qwen2.5-0mmi demonstrates notable progress by narrowing down the candidate emotions to surprise and fear, eliminating many irrelevant options. However, when it comes to making a fine- anneggrined distinction between these two emotions, it still fails to recog- nize the subject's fearful expression, leading to an incorrect prediction. These findings highlight the importance of improving the fine-grained visual perception capabilities of MLLMs, with par- ticular emphasis on the accurate interpretation of facial expressions and body movements. Such advancements are essential for the continued development of emotional intelligence in MLLMs.
Obs. 4: Incomplete multimodal information limits the emotional intelligence in MLLMs. The failure case of Gemini-2.5-Prr in Figure 6 reveals that the error primarily stems from the absence of audio information. In this example, the character's fearful emotion is clearly conveyed through audio clues. However, the model fails recognize it due to its inability jointly process both audio and visual modalities from the video clip. Most current generalist MILMS are still limited in their capacity to integrate audio and visual information simultaneously, leading to misinterpretations when critical emotional signals are missing. These findings suggest that although current leading models are primarily vision- lingon-language models, the future of emotional intelligence lies in advancing more powerful omnimodal models capable of jointly processing audio, visual, and textual information.
5 CONCLUSION
In this paper, we introduced MME-Emotion, a comprehensive multi-task benchmark for evaluating emotional intelligence in MILMs,, accompanied by holistic evaluation suite. The assessment
9

---

Published as a conference paper at ICLR 2026
process was fully automated within multi-agent system framework and thoroughly validated by human experts. Through rigorous evaluation using three unified metrics and in-depth empirical analysis, uncovered both the strengths and limitations of cutting-edge MLLMs, laying solid foundation for future research aimed at advancing emotional intelligence in MILMMs. In future work, intend build upon the observations and insights derived from this paper and systematically address the existing shortcomings of MLLMs, developing powerful emotion-specialls model.
ACKNOWLFDGMENT
The work described in this paper was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Project T45-401/22-N and Project CUHK 14202125.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A versatile vision-language model for understanding, localization. Text Reading, and Beyond, 2, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jean- nette N Chang, Sungbok Lee, and Shrikanth S Narayanan. lemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42:335-359, 2008
Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander Hauptmann. Eimotion-llama Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805-11083 2024a.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi modeling and
Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing audio understanding in video-Ilms. arXiv preprint arxi::2406.0 074767,77 2024b.
Yunfei Chu. Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng Hc, Junyang Lin, et al. Qwen2-audhe audio technical report. arXiv preprint arXiv:2407.10759 2024.
Yueying Feng, WenKang Han, Tao Jin, Zhou Zhao, Fei Wu, Chang Yao. Jingyuan Chen, et al. E3: Exploring embodied emotion through large-scale egocentric video dataset. Advances in Neural Information Processing Systems, 37:118182-118199 2024.
Chiara Filippini, David Perpetuini, Daniela Cardone, Antonio Maria Chiarelli, and Arcangelo Merla. based affective computing and its application to facilitate human robot review. Applied Sciences, 10(8):2924, 2020.
Thermal infrared imaging- maseng interaction:
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: a comprehensive evaluation benchmark for multimodal large language models. corr abs/2306. 13394 (2023), 2023.
Chaoyou Fu, Yi-Fan Zhang, Shukang Yin, Bo Li, Xinyu Fang. Sirui Zhao, Haodong Duan, Xing Sun, Ziwei Liu, Liang Wang, et al. Mme-survey: comprehensive survey on evaluation of multimodal llss. arXiv preprint a251 2024.
10

---

Published as a conference paper at ICLR 2026
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 24108-244 24118, 2025.
Riccardo Gervasi, Federico Barravecchia, Luca Mastrogiacomo, and Fiorenzo Franceschini. Ap- plications of affective computing in human-robot interaction: State-of-art and challenges for manufacturing. Proceedings of the Institution of Mechanical Engineers, Part Journal of Engineering Manufacture, 237(6-7):815-8322 2023.
Google. Gemini 2.0 flash model card, 2025a. URL http:pss//stoongg..oollaaaaa...p com/ model-cards, www.rdd/ddocumene doc.mentsssnocmmmccmmmmmm httg/emmiiiiiii -itmmiii----ii 2 pdf.
Google. Gemini 2.5 flash preview model card, 2025b. URL
com/mode cacards/////aa documents 2rt.mm/ennngggg..aa22222----aaar
Google. Gemini 2.5 pro preview model card, 2025c. URL com/mode -acdd hooidi/dooooneeooooeee 2ttp:///moooeee//aaaaaaa.aaaa///
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874 2021.
He Hu, Yucheng Zhou, Lianzhong You, Hongbo Xu, Qianning Wang, Zheng Lian, Fei Richard Yu, Fei Ma, and Laizhong Cui. Emobench-m: Benchmarking emotional intelligence for multimodal large language models. arXiv preprint arXiv:2502.04424, 2025.
Jen-tse Huang, Man Ho Lam, Eric John Li, Shujie Ren, Wenxuan Wang, Wenxiang Jiao, Zhaopeng Tu, and Michael R Lyu. Emotionally numb empathetic? evaluating how Ilms feel using emotionbench. arXiv preprint urXiv:2308.03655 2023.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprini arXiv:2502.09621, 2025.
Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions the wild. In Proceedings of the 28th ACM international conference multimedia, pp. 2881-28828 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Gary G Koch. Intraclass correlation coefficient. Encyclopedia of statistical sciences, 2004.
Ronak Kosti, Jose M Alvarez, Adria Recasens, and Agata Lapedriza. Emotion recognition in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1667-1675, 2017.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jiyoung Lee, Seungryong Kim, Sunok Kim, Jungin Park, and Kwanghoon Sohn. Context-aware emotion recognition networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10143-10152, 2019.
11

---

Published as a conference paper at ICLR 2026
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan.
Seed- benchb Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 55299-13308 2024.
Shan Li and Weihong Deng. Deep facial expression recognition: A survey. IEEE transactions on affective computing, 13(3):1195-122) 1215, 2020.
Shan Li, Weihong Deng, and JunPing Du. Reliable crowdsourcing and deep learning for expression recogmition in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1862-2861, 2017.
Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, multimodal emotion recognition: Dataset,
Siyuan Zhang, Hailiang Yao, et al. metric, and benchmark. arXiv preprint arXiv:2410.01444 2024a.
Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, et al. Mer 2024: Semi-superviset learning, noise robustness, and open-vocabulary multimodal emotion recognition. In Proceedings of the 2nd International Workshop Multimodal and Responsible Affective Computing, pp. 41-48, 2024b.
Zheng Lian, Haiyang Sun, Licai Sun, Jiangyan Yi, Bin Liu, and Jianhua Tao. Affectgpt: Dataset and framework for explainable multimodal emotion recognition. arXiv preprint 207752340..06633,
2024c.
Zheng Lian, Licai Sun, Lan Chen, Haoyu Chen, Zebang Cheng, Fan Zhang, Ziyu Jia, Ziyang Ma, Fei Ma, Xiaojiang Peng, et al. Emoprefer: Can large language models understand human emotion preferences? arXiv preprint arXiv:2507.04277 2025a.
Zheng Lian, Fan Zhang, Yazhou Zhang, Jianhua Tao, Rui Liu, Haoyu Chen, and Xiaobai Li. Affectgpt- r1: Leveraging reinforcement learning for open-vocabulary multimodal emotion recognition. arXiv preprint 2rXiv:2508.0/3/8 2025b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by aligmment before projection. arXiv preprint arXiv:2311.10122, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024. URL https://arxiv org/abs/2310.03344, 3(4):5, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34998 2023b.
Rui Liu, Haolin Zuo, Zheng Lian, Xiaofen Xing, Björ W Schuller, and Haizhou Li. Emotion and intent joint understanding in multimodal conversation: A benchmarking dataset. arXiv preprint arXiv:2407.02751. 2024a.
Yihe Liu, Ziqi Yuan, Huisheng Mao, Zhiyun Liang, Wanqiuyue Yang, Yuanzhe Qiu, Tie Cheng, Xiaoteng Li, Hua Xu, and Kai Gao. Make acoustic and visual cues matter: Ch-sims v2. dataset and av-mixup consistent module. In Proceedings of the 2022 international conference multimodal interaction, pp. 247-258, 2022a.
Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. Mafw: A large-scale, mudti-modal, compound affective database for dynamic facial ex- pression recognition in the wild. In Proceedings of the 30th ACM international conference on multimedia, pp. 24-32, 2022b.
Yuanyuan Liu, Ke Wang, Lin Wei, Jingying Chen, Yibing Zhan, Dapeng Tao, and Zhe Chen. Affective computing for healthcare: Recent trends, applications, challenges, and beyond. arXiv preprint arXiv:2402. 13589, 2024b.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0/78. 2025.
12

---

Published as a conference paper at ICLR 2026
Junyu Luo, Zhizhuo Kou, Liming Yang. Xiao Luo, Jinsheng Huang, Zhiping Xiao, Jingshu Peng. Chengzhong Liu, Jiaming Ji, Xuanzhe Liu, et al. Finmme: Benchmark dataset for financial multi-modal reasoning evaluation. arXiv preprint wrxiv:250.22774. 2025a.
Junyu Luo, Weizhi Zhang, Ye Yuan, Yusheng Zhao, Junwei Yang, Yiyang Gu, Bohan Wu, Binqi Chen, Ziyue Qiao, Qingqing Long, et al. Large language model agent: A survey on methodology. applications and challenges. arXiv preprint arXiv:2503.2/ 24600 2025b.
Luo, Jianbo Ye, Reginald B Adams, Jia Li, Michelle G Newman, and James Z Wang. Arbee: Towards automated recognition of bodily expression of emotion in the wild. International journal of computer vision, 12:11225, 2020.
Shuyi Mao, Xinpeng Li, Fan Zhang, Xiaojiang Peng, and Yang Yang. Facial action units as a joint dataset training bridge for facial expression recognition. IEEE Transactions on Multimedia, 2025.
Mary L. McHugh. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276 282, 2012.
OpenAl. Gpt-4o system card, 2024.
URL https: https://openai. com/ index/
gpt - 40- pppt-4--yyse --------stte--araaaaaae
OpenAl. Gpt-4.1 system card, 2025. URL wwwphtttp//ccc https::/openai....tt./a/// http:/ss//pppaaa...coco///////ga
Rosalind W Picard. Affective computing. MIT press, 2000.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-pare partttaattyy dataset for emotion recognition in conversations. arXiv preprint wrXiv:1810.025000 2018.
Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M Liu, Jinfeng Zhou, Alvionna S Sunaryo, Juanzi Li. Tatia Lee, Rada Mihalcea, and Minlie Huang. Emobench: Evaluating the emotional intelligence of large language models. arXiv preprint 2rXiv:2402.12071, 2024.
Shezheng Song, Chengxiang He, Shasha Li, Shan Zhao, Chengyu Wang, Tianwei Yan, Xiaopeng Li, Qian Wan, Jun Ma, Jie Yu, et al. Mosabench: Multi-object sentiment analysis benchmark for evaluating multimodal large language models understanding of complex image. arXiv preprint arXiv:2412.00060, 2024.
Charles Spearman. The proof and measurement of association between two things. 1961.
Jianhua Tao and Tieniu Tan. Affective computing: A review. In International Conference on Affective computing and intelligent interaction, pp. 981-995. Springer, 2005.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean Cann-Bapistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312./11805 2023.
Qwen Team. Qvq: To see the world with wisdom, December 2024. URL httttt:///weennn github. ic/blog////// /qvq-/vvvvvv-/ 72b preview/.
Panagiotis Tzirakis, George Trigeorgis, Mihalis Nicolaou, Björn W Schuller, and Stefanos Zafeiriou. End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of selected topics in signal processing. 11(8): 1301-1199,, 2017.
Kun Wang, Guibin Zhang, Zhenhong 7hou, Jiahao Wu, Miao Yu, Shiqian Zhao, Chenlong Yin, Jinhu Fu, Yibo Yan, Hanjun Luo, et al. A comprehensive survey in Imm (-agent) full stack safety: Data, training and deployment. arXiv preprint arXiv:2504. 2025.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et Qwen2-vl: Enhancing vinion-language model's 's perception of the world at any resolution. arXiv preprint arXiv:24099 /2/91 2024.
Chih- Wu. Yueh-Min Huang, and Jan-Pan Hwang. Review of affective computing in educa- tion/learning: Trends and challenges. British Journal of Educational Technology, 17(6):1304-133 2016.
13

---

emotion Published as a conference paper at ICLR 2026
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pp. 2247- 2256. IEEE, 2023.
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chumyan Miao. Audio- reasoner: Improving reasoning capability in large audio language models. arXiv preprint arXiv:2503.02318, 2025.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, al. Qwen2. 5-omni technical report. arXiv preprini arXiv:2503. 2025.
Elaheh Yadegaridehkordi, Nurul Fazmidar Binti Mohd Noor, Mohamad Nizam Bin Ayub, Hannyz- zura Binti Affal, and Nornazlita Binti Hussin. Affective computing in education: A systematic review and future research. Computers & education, 142:103649, 2019.
Xiaocui Yang, Wenfang Wu, Shi Feng, Ming Wang, Daling Wang, Yang Li, Qi Sun, Yifei Zhang, Xiaoming Fu, and Soujanya Poria. Mm-instructeval: Zero-shot evaluation of (multimodal) large language models on multimodal reasoning tasks. Information Fusion, pp. 103204, 2025.
Georgios N Yannakakis. Enhancing health care via affective computing. 2018.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. Mosi: multimodal cor- pus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259, 2016.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philiges Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2236 2246, 2018.
Biao Zhang, Zhongtao Liu, Colin Cherry, and Orhan Firat. When scaling meets IIm finetuning: The effect of data, model and finetuning method. arXiv preprint 24X1v:2402.171993 2024a.
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li, Dan Su, Chenhui Chu, and Dong Yu. Mm-Ilms: Recent advances in multimodal large language models. arXiv preprint (iti:22001.13000 2024b.
Fan Zhang, Zhi-Qi Cheng, Jian Zhao, Xiaojiang Peng, and Xuelong Li. Leaf: unveiling sides of the same coin in semi-supervised facial expression recognition. arXiv preprint arXiv:2404. 15041, 2024c.
Fan Zhang, Haoxuan Li, Shengju Qian, Xin Wang, Zheng Lian, Hao Wu, Zhihong Zhu, Yuan Gao, Qiankun Li, Yefeng Zheng, et al. Rethinking facial expression recognition in the era of multimodal large language models: Benchmark, datasets, and beyond. arXiv preprint arX1v:2511.00389, 2025a.
Fan Zhang, Tianyu Liu, Zhihong Zhu, Hao Wu, Haixin Wang, Donghao Zhou, Yefeng Zheng, Kun Wang, Xian Wu, and Pheng-Ann Heng. Cellverse: Do large language models really understand cell biology? arXiv preprint arXiv:2505.07865 2025b.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0220 02858, 2023a.
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: a survey. arxiv. arXiv preprint arXiv:2308. 10792, 2023b.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal Ilm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408. / 13257, 2024d.
Jiaxing Zhao, Xihan Wei, and Liefeng Bo. RI-omni: Explainable recognition with reinforcement learning. arXiv preprint (rXiv:2503.05379 2025a.
14

---

Published as a conference paper at ICLR 2026
Jixxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Xihan Wei, Liefeng Bo, et al. Humanomni: A large vision-speech language model for ceman-centric video understanding. arXiv preprint 11X1v:2501.1511,, 2025b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 2023.
Zhihong Zhu, Fan Zhang, Yunyan Zhang, Jinghan Sun, Zhiqi Huang, Qingqing Long, Bowen Xing. and Xian Wu. A survey multi-modal intent recognition: Recent advances and new frontiers.
Zhihong Zhu, Yunyan Zhang, Xianwei Zhuang. Fan Zhang, Zhongwei Wan, Yuyan Chen, Qingqing- Long QingqingLong Yefeng Zheng. and Xian Wu. Can we trust ai doctors? a survey of medical hallucination in large language and large vision-language models. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 6748-6769, 2025.
15

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Related Work
2.1
Emotional Intelligence in Multimodal Large Language Models
2.2
Datasets and Benchmarks for Emotional Intelligence
3
MME-Emotion
3.1
Benchmark Construction
3.2
Evaluation Suite
4
Experiments
4.1
Experimental Setup
4.2
Main Results
4.3
Observations and Insights
5
Conclusion
The Use Large Language Models (LLMs)
17
Limitations and Social Impact
17
C More Related Works
17
C.1 Datasets and Benchmarks for Multimodal Large Language Models
17
D Implementation Details
17
D.I Evaluated MILMS
17
D.2 Prompt for Evaluated MLLMs
19
D.3 Prompt for Extracting Audio Clues
20
D.4 Prompt for Extracting Key Steps
20
D.5 Prompt for Rating Performance
E
Additional Experimental Results
22
E.1 Additional Experimental Results across Tasks
22
E.2
Additional Experimental Results across Categories
22
E,3 Additional Experimental Results across Metrics
22
E,4
Additional Case Study
22
16

---

Published as a conference paper at ICLR 2026
A
THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs were not involved in the research ideation or the writing of this paper.
B
LIMITATIONS AND SOCIAL IMPACT
While MME-Emotion represents a significant step forward the field of affective computing and multimodal large language models (MLLMs), several limitations remain worth acknowledging:
First, although we provide multi-task and multi-scenario benchmark for evaluating emo- tional intelligence in MLLMs, it would be beneficial to incorporate task difficulty classifica- tion. Similar to the categorization of mathematical problems (Hendrycks et al., 2021) into primary school, high school, college, and Olympiad levels, assigning a difficulty rating or level to each sample in our benchmark could offer more nuanced understanding of MLLMs' capabilities in handling affective computing tasks across varying levels complexity.
Second, since our benchmark is constructed from publicly available video datasets, the included data covers a wide range of multilingual scenarios. However, we do not currently distinguish between different languages in the benchmark, nor do we analyze model perfor- mance across different linguistic contexts. future work, we aim to extend our evaluation from a multilingual perspective, enabling more comprehensive assessment of emotional intelligence in MILMs.
Overall, we present a systematic benchmark for evaluating both the recognition and reasoning capabilities of MLLMs in emotion-related tasks, which we believe will serve as valuable resource for both the MLLM and affective computing communities. We hope this work will inspire growing research efforts aimed at bootstrapping emotional intelligence in MILMs, ultimately advancing downstream applications in education, healthcare, and human-robo interaction.
C MORE RELATED WORKS
C.I DATASETS AND BENCHMARKS FOR MULTIMODAL LARGE LANGUAGE MODELS
With the rise of large language models (LLMs) and the growing popularity of multimodal large language models (MLLMs), an increasing number of studies have focused on evaluating MLLM performance across a wide range of domains. By collecting data and employing human annotations, MME (Fu et al., 2023) evaluates the general understanding capabilities of MILMS across 14 subtasks, ranging from perception tasks to cognition tasks. Video-MME (Fu al., 2025) assesses the video analysis capabilities of MLLMs using a diverse and high-que nigh-quality collection of vidco data. MME- RealWorld (Zhang et al., 2024d) aggregates large set of high-resolution images, combined with manual annotations, to construct a suite of practical and challenging tasks, including optical character recognition (OCR) in the wild, remote sensing, and autonomous driving. FinMME (Luo al., 2025a) leverages annotations from 20 domain experts to comprehensively evaluate MLLMs' understanding capabilities in the financial domain. These benchmarks have significantly accelerated progress in MLLM development, enabling improved performance in both general- -prppose and domain- specific scenarios, ultimately contributing wide array of real-world applications.
D IMPLEMENTATION DETAILS
D.1 EVALUATED MLLMS
We evaluated a total of 20 MLLMs, comprising 15 closed-source and 5 open-source models. A brief introduction each model is provided below:
Qwen2-Audio (Chu et al., 2024) is the latest open-source large audio-language model developed by Alibaba Group. It is designed to handle a wide range of audio inputs and can perform detailed audio analysis or generate direct textual responses based on spoken
17

---

Published as a conference paper at ICLR 2026
instructions. Qwen2-Audio achieves state-of-the-ait performance across multiple audio- related tasks, including automatic speech recognition (ASR), apeech-to-text translation (S2TT), speech emotion recognition (SER), and vocal sound classification (VSC).
Audio-Reasone (Xie et al., 2025) is a large audio-language model built upon Qwen2-Aude enhanced with advanced reasoning capabilities. By curating structured thain-of-thougle (CoT) data and applying supervised fine-tuning, it achieves superior performance and sets new state-of-ther -he-ttrtttrttttt in audio reasoning tasks.
Qwen2-V VL (Wang al., 2024) is a newly developed series of large vision-language models from Alibaba Group. A key innovation in Qwen2-VL. is the incorporation of naive dynamic resolution mechanism, which allows the model to flexibly adapt to images of different resolutions by converting them into variable numbers of visual tokens. This design enhances both the efficiency and precision of visual representations, in manner that closely mirrors human visual perception. Consequently, Qwen2-VL series achieve cutting-edge results visual understanding tasks and exhibits strong capabilities in complex reasoning and Decision-making
Qwen2.5- VL (Bai et al., 2025) series, developed by Alibaba Group, represents one of the most advanced large vision- languuag model families date. Leveraging streamlined and efficient vision encoder alongside dynamic resolution and frame rate training strategies, the Qwen2.5-VI. models set new state-of-the-art performance a wide range of visual tasks. In particular, they demonstrate exceptional capabilities in fine-grained video understanding.
QVQ (Team, 2024) a large vision-language model developed by Alibaba Group based on the Qwen2-VI. architetuure, with particular focus on strengthening visual reasoning capabilities. represents a significant advancement Al's ability to perform visual under- standing and solve complex problems. QVQ achieves remarkable performance, especially in reasoning-intensive. tasks such as mathematical problem solving, marking a notable breakthrough in the field.
Vidco-LLaVA (Lin et al., 2023) is large vision- languag model capable of processing both images and videos. Its core innovation lies in projecting image and video features into a unified representation space, enabling the language model to learn cross-modal interactions from a consistent visual embedding. Video-LLaVA demonstrates strong performance across both image understanding and video analysis tasks.
Video-LLaMA (Zhang et al., 2023a) is large omnimodal model designed to process both visual and audio information through dedicated visual and audio branches. Unlike many existing models that are limited to a single modality, Video-LLaMA enables simultancous understanding of both visual and acoustic content within videos. By integrating multimodal data, it addresses critical limitations in current video understanding approaches.
Video-LLaMA2 (Cheng et al., 2024b) builds upon its predecessor by integrating a cus- tomized spatio-temporal convolution connector (STC), which effectively captures the in- tricate spatial and temporal dynamics inherent in video data. Furthermore, the model incorporates audio branch through joint training. allowing it to seamlessly fuse audio clues and significantly enhance its multimodal understanding capabilities.
Qwen2.5-Ommi (Xu al., 2025) is a flagship large omnimodal model developed by Alibaba Group. It adopts the Thinker-Talker architecture, specifically designed for comprehensive multimodal perception. The model can seamlessly process a wide range of input modalities, including text, images, audio, and video, while also supporting streaming text generation and natural speech synthesis outputs. @wen2.5-0mmi achieves strong performance across both unimodal and multimodal understanding tasks.
Emotion-LLaMA (Cheng et al., 2024a) is an omnimodal emotion-speciilist model designed for multimodal emotion recognition. It leverages audio, visual, and textual encoders to capture comprehensive multimodal representations and employs a two-stage training strategy to enhance learning effectiveness. As a result, Emotion -LaMoA achieves strong performance across various emotion recognition tasks.
HumanOmni (Zhao et al., 2025b) is an omnimodal specialist model tailored for human- centric tasks, including emotion recognition. It utilizes three distinct visual projectors facc-- related, body-related, and interaction-relatee related-comminee with learnable gating mech- anism to capture fine-grained, task-relevant visual representations. This design enables
18

---

Published as a conference paper at ICLR 2026
HumanOmni to achieve strong performance across wide range of human-centric down- stream tasks.
R1-Omni (Zhao et al., 2025a) is an omnimodal emotion-specialiss model built upon the foundation of HumanOmni. By combining reinforcement learning with verifiable reward (RLVR) and rule-based rewards, R1-Omni effectively enhances the model's reasoning capabilities on emotional tasks.
AffectGPT (Lian et al., 2024c) is an omnimodal emotion-specialis model. Leveraging model-based crowdsoureing strategy, the authors curated a large-scale emotion under- standing dataset rich in emotional cues. Leveraging this dataset for training, alongside the incorporation of pre-fusion operations to enhance multimodal integration, AffectGPT demonstrates robust and advanced capabilities in emotion understanding.
GPT-4o (OpenAI, 2024) is one of the latest MLLMs developed by OpenAl, providing convenient APIs for processing textual, visual, and audio modalities. demonstrates state- of-the-art performance across wide range of benchmarks, with significant advancements in visual perception, understanding, and reasoning. Equipped with a unified architecture that facilitates seamless cross-modal interaction, GPT-4o is both highly efficient and adaptable, making well-suited for real-world multimodal applications.
GPT-4.1 (OpenAI, 2025) is powerful and cost-effective MLLM released by OpenAL. demonstrates significant improvements in programming capabilities and instruction fol- lowing. Additionally, GPT-4.1 features an expanded context window supporting up to one million tokens, enabling enhanced long-context understanding that better leverages information to improve both efficiency and performance.
Gemini-2.0-Flaash (Google, 2025a) supports input and output across multiple modalities, including images, videos, and audio, while maintaining efficient inference speed. By leveraging advanced reasoning capabilities along with extended context understanding, Ge-ini-2.0-Flash achieves strong performance on a variety multimodal reasoning tasks.
Gemini-2.5-Flash (Google, 2025b) is a multimodal hybrid reasoning model that strikes balance between performance and efficiency. By allowing users selectively enable reasoning, it offers a flexible trade-off among performance, speed, and cost. Through fine- grained control over reasoning tokens, Gemini-2.5-Flas achieves outstanding performance across variety of multimodal understanding tasks.
Gemini-2.5-PP (Google, 2025c), recent release from Google DeepMind, advances mul- timodal large language modeling with enhanced capabilities in visual understanding. supports longer context windows and improves the efficiency of cross-modal alignment. Equipped with robust reasoning skills, Gemini-2.5 Pro excels across a wide spectrum of tasks, such as programming, mathematies, and scientific problem solving.
D.2 PROMPT FOR EVALUATED MLLMS
adopt unified prompts across all evaluated MLLMs to ensure fair comparison, with slight modifications made for accommodate their specific input modalities. Examples of the prompts are provided below.
models, lasion-language models, and omnimodal models to
Prompt for ER-Lab, ER- Wild, and Noise-ER
As emotion recognition expert, identify the most prominent emotion displayed in the video. Candidate labels:
Please think step by step. Enclose your final answer within <answer><<answer> tags.
Prompt for FG-ER
As an emotion recognition expert, identify one or several emotions displayed in the video.
Candidate labels:
Please think step by step. Enclose your final answer within <answer-</anseer> tags.
19

---

Published as a conference paper at ICLR 2026
Prompt for ML-ER
As emotion recognition expert, identify multiple emotions displayed in the video. Candidate labels:
Please think step by step, Enclose your final answer within <answer></nswer>> tags.
Prompt for SA
As emotion recognition expert, identify the most prominent sentiment displayed in the video.
Candidate labels:
Please think step by step. Enclose your final answer within <answer></answer>> tags.
Prompt for FG-SA
As an emotion recognition expert, identify the most prominent fine- grained sentiment displayed in the video.
Candidate labels:
Please think step by step. Enelose your final within <answer><<answer>> tags.
Prompt for IR
As emotion recognition expert, identify the most prominent intent displayed in the video.
Candidate labels:
Please think step by step. Enclose your final answer within <answer></answer>> tags.
D.3 PROMPT FOR EXTRACTING AUDIO CLUES
An example prompt for extracting audio clues is shown below. Please note that minor adjustments are needed depending on the specific task requirements.
Prompt for Extracting Audio Clues
You speech emotion analysis expert, specializing in analyzing the tone and intonation of input audio. Please analyze which features the audio reflect that the speaker is fceling (Emotion labels). Pay special attention to emotional features such as crying sounds, laughter, changes in tone, speech rate, pauses, emphasis and stress, voice trembling, and other emotional characteristics. Emotion labels:
D.4 PROMPT FOR EXTRACTING KEY STEPS
An example prompt for extracting key answer steps is shown below. Please note that minor adjust- ments needed depending the specific task requirements.
20

---

Published as a conference paper at ICLR 2026
Prompt for Extracting Kye Steps
You are expert in affective computing and very good at handling tasks related emotion recognition. Given the following answer about the emotion conveyed video, please help extract several key steps from the answer. Each step should as concise as possible and prefixed Step X. the last step, showcase the predicted emotion. If there are no reasoning steps, please showcase the predicted emotion at Step I. Enclose your result within <step><<step> tags.
Answer:
Example
Answer:
In the video, elderly man wearing green shirt is in an outdoor nighttime setting. He appears focused and serious, with slightly furrowed brows and serious expression. His eyes scanning the other person, suggesting wants to convey something important or ask about certain thing. As time passes, his gaze shifts from scanning to direct engagement, eventually relaxing and showing hint of happiness, indicating fluctuations in his emotions. Overall, the elderly man experiences a moderate intensity of neutral emotion, mixed with slight positive fluctuations. Result:
<step>Step Observe setting characters the video, noting any relevant details such as location and participants' appearance. Step 2: Analyze the facial expressions and movements, such as furrowed brows and downturned eyes, looking for indicators of specific emotions. Step 3: Consider verbal and non-verbal and emotional responses. Step Synthesize observations understand the overall emotional state conveyed by the participant. Step 5: Based on the analysis, determine that the predicted emotion neutral, </step>
including mouth movements and gaze direction, that signify communication
Example
Answer:
happy
Result:
<step>Step 1: The predicted emotion is happy.</step>
D.5 PROMPT FOR RATING PERFORMANCE
An example prompt for evaluating the performance shown below. Please note that minor adjust- ments are needed depending the specific task requirements.
Prompt for Evaluation the Performance
You are an expert in affective computing and very good at handling tasks related to emotion recognition. I will first give you some ground truth information about the emotion in video: visual clue, audio clue, and emotion label. I will also give you model prediction. Please help rate the performance of the prediction.
Rating Requirements:
1. Rate each step using 0 or 1. (0=wrong, !=correct)
2. For the last step. your rating should if the predicted emotion matches the ground truth emotion label, otherwise Don't consider visual/audio clues at this step.
3. For each other step, only consider predictions clearly contradicting visual/audio clues as incorrect.
4. Ensure the number of steps in your rating is equal to that in the model prediction.
5. Output format: <score>Step 1: 0/1, Step 2: 1/l..../ssooe>
Input Data:
Visual cluc: The video frames (images)
Audio clue:
Emotion label:
Model prediction:
Example Output:
<score>Step 1/1. Step 0/1, Step 3: 0/1</score>
21

---

Published as a conference paper at ICLR 2026
E ADDITIONAL EXPERIMENTAL RESULTS
E ADDITIONAL EXPERIMENTAL RESULTS ACROSS TASKS
As shown in Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, and Table 10, we present
fine-grained performance comparison of cutting-edge MLLMs across various emotional tasks.
E.2 ADDITIONAL EXPERIMENTAL. RESULTS ACROSS CATEGORIES
As shown Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, and Figure 12, we present a fine- grained performance comparison of cutting-ed MLLMs across different categories of emotions, sentiments, and intents. We include only single-label tasks and exclude the results of multi-label comparisons.
E.3 ADDITIONAL EXPERIMENTAL RESULTS ACROSS METRICS
As shown in Figure 13, we provide additional results to showcase the relationships among model evaluation metries. The results are consistent with our previous observations.
E.4 ADDITIONAL CASE STUDY
As shown in Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, and Figure 21, we present additional examples responses from three representative MLLMs across a range of emotional tasks.
Table Overall Performance Comparison (%) on ER-Lab. The top three performing results are highlighted in red (1²), blue (2nd), and yellow (3td) backgrounds, respectively.
Model
LLM Size
A
V T
Avg Step
Avg Token
Rec-S
Rea-S
CoTTS
Open-soures MLLMS
Qwen2-Audio (Chu et al., 2024)
7B
1.7
13.4
20.4
32.3
26.3
Audio-Rcasoner (Xie 2025)
7B
5.2
358.5
32.6
72.6
52.6
Qwen2-VL-7B (Wang et 2024)
7B
4
V
1.2
13.1
15.2
3.0
9.1
Qwcn2-VL-72B (Wang ct al., 2024)
72B
V
V
2.1
58.5
20.6
21.7
21.2
Qwen2.5-VI-7B (Bai et al., 2025)
7B
4.6
180.1
22.0
62.0
42.0
Qwen2.5-VI.-728 (Bai al., 2025)
72B
V
4.5
60.2
QVQ 2024)
72B
V
5,4
928.4
21.2
62.4
41.8
Video-LLaVA (Lin et al., 2023)
7B
V
Y
1.6
14.7
11.2
8.7
9,9
Vidco-LLaMA (Zhang ct al., 2023a)
7B
V
V
4.4
94.8
15.0
34.7
24.8
Vidco-LLaMA2 (Cheng et al., 2024b)
7B
1.0
23.4
0.3
11.9
Qwen2.5-Omni (Xu al,, 2025)
7B
4.2
109.6
50.7
Emotion-LLaMA (Cheng al., 2024a)
7B
V
1.0
1.0
21.0
0.0
10.5
HumanOmmi (Zhao
2025b)
R1-Omni (Zhau al., 2025a)
0.5B
4,8
151.4
20.6
58.0
39.3
AffecGPP (Lian al., 2024c)
4.6
64.4
Clased source MIL.Ms
GPT-4o (OpenAI, 2024)
4
4.8
242.4
21.8
79.4
50.6
GPT-4,1 (OpenAI, 2025)
4
5.6
150.7
21.2
46.6
33,9
Semini--2.--Faas (Google, 2025a)
V
v
4.1
24.0
55.6
39.8
Gemini-2.5 Flass (Google, 2025h)
V
4.5
363.4
19.6
48.3
33.9
Gemini-2.5.PP (Google, 2025c)
5.2
62.7
22

---

Published as a conference paper at ICLR 2026
Table Overall Performance Comparison (%) on ER-Wild. The top three performing results are highlighted in red (1*)), blue (2"H), and yellow (3ml) backgrounds, respectively.
Model
LLM
T
Avg Step
Avg Token
Ree-S
Rea-S
Open source MILMS
Qwen2-Audio (Chu ul., 2024)
7B
2.8
32.
22.3
38.4
30.3
Audio-Reasoner
2025)
7B
5.1
28.8
60.4
44.6
Qwen2-VL-7B (Wang al., 2024)
7B
2.7
53.8
31.5
32.3
31.9
Qwen2-VL-728 VL-22 (Wang al., 2024)
72B
V
V
1.9
30.7
33.3
15.4
24.3
Qwen2,5-VL-7B (Bai ct al., 2025)
7B
V
V
4.8
162.7
28.9
55,3
42.1
Qwen2.5-VI--22B (Bai et al., 2025)
72B
V
V
4.9
293.7
32.4
67.2
49.8
QVQ (Team, 2024)
72B
V
V
5.4
826.8
31.8
61.6
46.7
Video-LLaaA (Lin et al,, 2023)
7B
V
V
1.0
5.4
21.2
2.2
11.7
Video-LLaMA (Zhang al., 2023a)
7B
4.3
25.5
44.7
Vidco-LLaMA2 (Cheng al., 2024b)
7B
4.3
94.2
25.4
44.7
35.0
Qwen2.5-Omni (Xu ct al., 2025)
7B
V
V
3.7
78.1
19.0
54.3
36.7
Emotion-LLaMA (Cheng ot al., 2024a)
7B
v
4
1.0
1.1
32.3
0.0
16.1
HumanOmni (Zhao al., 2025b)
7B
1.0
1.5
52.8
0.0
26.4
RI 11-0mii (Zhao et al., 2025a)
0.58
V
5.1
156.3
34,3
52.8
43.6
AffecGGPT (Liun el ul., 2024c)
7B
4.8
115.3
8.3
41.9
25.1
Closed-source MLLMs
GPT-4o (OpenAI, 2024)
4.6
GPT-4,1 (OpenAl, 2025)
/
5.3
148.3
29.3
54.8
42.1
Gemini-2.0-Flas (Google, 2025a)
v
4.2
62.1
41.1
56.9
Gemini-2.5 2.5ni---.--Fash (Google, 2025h)
4.6
279.9
35.8
Comini-2.5-PPa (Google, 2025c)
v
5.1
527.5
36.7
67.3
52.0
Table 5: Overall Performance Comparison (%) Noise-ER. The top three performing results highlighted in red (I**), blue (2"d), and yellow (3rl) backgrounds, respectively.
Model
LLM Size
V
T
Avg Step
Avg Token
Rec-S
Rea-S
CoT-S
Open-source MLLMs
Qwen2-Audio (Chu al., 2024)
3.1
38.8
$7.6
Audio-Reasoner (Xie al., 2025)
7B
4.9
353.7
47.6
80. 4
64.0
Qwen2-VLL Owen2-VL-7B (Wang et al, 2024)
/
3.4
Owen2-VL-72B (Wang
2024)
72B
Y
1.9
38.0
47.4
21.6
34.5
Owen2.5-VL-7B (Bai a 2025)
7B
4
V
4.7
147.4
37.8
62.9
50.3
Bwcn2.5-VLL-22B (Bai et al., 2025)
72B
V
V
4.9
284.7
41.2
74.6
57.9
QVQ (Team, 2024)
72B
V
5.4
718.3
39.6
72.7
Video-LLaaV (I.in al,, 2023)
V
1.7
22.3
Video-LLaMA (Zhang
2023a)
7B
4
Y
4.2
70.4
28.0
38.1
33.0
Video-LLaMA2 (Cheng et al., 2024b)
7B
4
V
1.5
8.5
45.0
13.0
29.0
Qwen2.5-Omni (Xu et al., 2025)
7B
V
3.7
82.8
24.2
60.1
42.1
Emotion-LLaMA (Cheng al., 2024a)
7B
V
V
V
1.0
1.1
58.4
0.0
29.2
HumanOmmi (Zhao al., 2025b)
R R1-Omni (Zhao al., 2025a)
0.5B
5,3
185.7
52.0
67.4
59.7
AffectGPT (Lian et al., 2024c)
7B
V
V
4.9
121.3
16.8
59.5
38.1
Closed-sounee MLLMS
GPT-4o (OpenAl, 2024)
4.4
144.1
27.8
78.2
53.0
GPT-4.1 (OpenAl, 2025)
5.2
138.3
32.8
(Google, 2025a)
V
v
4.2
60.2
52.4
67.5
60.0
Scmini-2.5-Flash F-5sshsh (Google, 2025b)
v
5.0
301.6
54.4
73.2
Gemini-2.5 Pro (Google, 2025c)
v
v
5.3
503.6
57.6
81.2
69.4
23

---

Published as a conference paper at ICLR 2026
Table Overall Performance Comparison (%) on FG-ER. The top three performing results are highlighted in red (1*)), blue (2"H), and yellow (3t)) backgrounds, respectively.
Model
LLM
T
Avg Step
Avg Token
Ree-S
Rea-S
Open source MILMS
Qwen2-Audio (Chu al., 2024)
7B
2.7
$1.2
21.7
22.0
21.8
Audio-Reasoner (Xie 2025)
7B
5.0
9.1
47.3
Qwen2-VL-7B (Wang al., 2024)
7B
1.1
10.9
26.4
4.5
15.4
Qwen2-VL-728 VL-22B
(Wang al., 2024)
72B
V
V
1.0
9.2
23.9
2.9
13.4
Qwen2.5-VL-7PB (Bai ct al., 2025)
7B
V
V
4.8
197.4
19.5
62.2
40.9
Qwen2.5-VI-722 (Bai et al., 2025)
72B
V
V
4,9
279.7
27.3
69.8
48.6
QVQ (Team, 2024)
72B
5.6
26.6
67.8
47.2
Vidco-LLaVA (Lin al., 2023)
7B
V
2.7
26.0
26.6
18.7
Video-LLaMA (Zhang
2023a)
7B
5.1
236.7
33.7
34.6
34.1
Vidco-LLaMA2 (Cheng
2024b)
7B
1.2
5.3
Qwen2.5-Omni (Xu et al., 2025)
7B
V
3.3
72.9
10.4
42.3
26.3
Emotion-I.I (Cheng et al., 2024a)
7B
4
1.5
15.4
10.7
4.3
7.5
HumanOmmi (Zhao et al,, 2025b)
7B
v
1.0
1.5
16.5
0.8
8.6
R1-Omni (Zhao et al., 2025a)
0.5B
4.7
135.1
49.5
27.3
AffectGPT (Lian al., 2024c)
7B
4.8
115.5
5.2
40.6
22.9
Closed MLLMS
GPT-4o (OpenAl, 2024)
4
4
4.4
198.0
31.8
68.6
50.2
GPT-4.1 (OpenAI, 2025)
v
/
5.2
145.3
28.4
67.3
47.8
Gemini-2.0-Flas (Google, 2025a)
4.4
104.2
58.6
44.7
Gemini-2.5 1a5ini-2.5-Flash (Google, 2025h)
V
3.5
354.1
20.5
37.8
Gemini-2.55 P.ooPo (Google, 2025c)
v
5.1
473.7
38.6
61.0
49.8
Table 7: Overall Performance Comparison (%) on ML-ER. The top three performing results highlighted in red (I**), blue (2nd), and yellow (3tl) backgrounds, respectively.
Model
LLM Size
V
T
Avg Step
Avg Token
Rec-S
Rea-S
CoT-S
Open-source MLLMs
Qwen2-Audio (Chu al. 2024)
7B
3.9
101.0
53.5
72.8
63.2
Audio-Reasoo Radio-Reesoneer (Xie et al., 2025)
7B
5.2
447.6
41.1
77.5
59.3
Owen2-VL-77 (Wang et al, 2024)
Owen2-VL-72B (Wang
2024)
72B
V
V
1.0
33.5
0.8
17.1
Qwen2.5-VL-77 (Bai et al 2025)
7B
V
4.8
193.4
23.6
75.2
49.4
Qwen2.5-VIL-22B (Bai al., 2025)
V
88.6
QVQ (Team, 2024)
72B
V
5.7
892.3
25.2
82.3
53.7
Video-LLaVA (Lin ct al. 2023)
7B
V
V
3.7
58.5
52.7
$7.1
54.9
Video-LLaMA (Zhang ul. 2023a)
7B
5.0
231.7
29.6
68.7
49.2
Video-LLaMA2 (Cheng et al. 2024b)
7B
2.2
40.1
31.0
35,5
Qwen2.5-Omai (Xu al., 2025)
7B
3.4
64.9
38.0
Emotion-LLaMM (Cheng al., 2024a)
7B
1.0
1.2
5.7
0.5
3.1
HumanOmmi (Zhan et al., 2025b)
7B
v
1.0
1.5
34,3
2.7
18.5
R1-Ommi (Zhao et al., 2025a)
0.5B
5.1
149.8
23.1
76.2
49.6
AffecGPPT (Lian al., 2024c)
4.9
65.9
Closed Closed-source source MLLMs
GPT-4u (OpenAI, 2024)
4.2
162.9
29.9
82.5
56.2
GPT-4,1 (OpenAl, 2025)
5.5
163.5
35.0
76.2
55.6
Seminii-2.0-Fie (Google, 2025a)
y
v
4.3
81.8
38.3
71.7
55.0
Gemini-2.5 2-5-lah (Google, 2025b)
2.5
38.7
30.1
Gemini-2.5 Pro (Google, 2025c)
v
v
5.1
650.2
50.6
80.5
65.5
24

---

Published as a conference paper at ICLR 2026
Table 8: Overall Performance Comparison (%) on SA. The top three performing results are highlighted in red (1*), blue (2"h), and yellow (3td) backgrounds, respectively.
Model
LLM Size
T
Avg Step
Avg Token
Ree-S
Rea-$
CoT-$
Open-soures MLLMS
Qwen2-Audio (Chu al., 2024)
7B
3.2
35.3
53.6
65.4
Madio-Reasoner (Xie al., 2025)
7B
4.9
230.2
62.2
81.9
72.0
Qwen2-VL-7B -L-B (Wang et al,, 2024)
/
4.0
34.1
60.6
Ww-n2-VL-72B (Wang al., 2024)
72B
4
1.4
22.2
34.8
9.3
22.1
Qwen2.5-VL-7B (Bai al. 2025)
7B
V
V
4.9
161.0
36.5
70.7
53.6
Qwen2.5-VL-72B (Bai al., 2025)
72B
4.8
227.7
37.5
81.9
59.7
QVQ (Team, 2024)
72B
V
V
5.7
823.1
43.0
76.2
59.6
Vidco-LLaaA (Lin ct al., 2023)
7B
V
V
3.0
18.4
35.8
55.8
45.8
Video-I.LaMA (Zhang al., 2023a)
7B
V
4.5
96.2
31.3
50.7
41.0
Video-LLaMA2 (Cheng el al., 2024b)
7B
7
211
Qwen2.5-Omni (Xu al., 2025)
7B
V
3.7
73.8
24.6
68.1
46.3
Emotion-LLaMA (Cheng al.. 2024a)
7B
V
1.0
1.0
26.1
0.1
13.1
HumanOmmi (Zhao et al., 2025b)
7B
V
1.0
1.1
32.1
0.0
16.0
RI-Omni (Zhao et al., 2025a)
0.5B
V
5.1
155.7
27.8
60.6
AffectGPT al., 2024c)
4.9
118.4
52.3
Closed- Closed-source source MLLMs
GPT-4o (OpenAl, 2024)
4
v
4.2
144.3
34,6
85.7
60.2
GPT-4, (OpenAT, 2025)
v
4.9
119.5
33.8
74.3
$4.0
Gemini-2.00 2.0mmii-..--Fashhh (Google, 2025a)
V
v
4.0
52.5
42.1
61.1
51.6
Gemini-2.5 5-Fassh (Google, 2025b)
4.2
44.2
55.9
Seminni22.5PPon (Google, 2025c)
v
v
5.0
477.5
48.2
78.7
63.4
Table Overall Performance Comparison (%) on FG-SA. The top three performing results are highlighted in red (1*),, blue (2"d), and yellow (3nd) backgrounds, respectively.
Model
LLM Size
T
Avg Step
Avg Token
Ree-S
Rea-S
CoT-S
Open-source MLLMs
Qwen2-Audio (Chu al., 2024)
7B
3.1
62.5
41.1
Audio-Rcasoner (Xic al., 2025)
7B
4.8
310.9
30.5
82.6
Owen2-VI.-7B (Wang al. 2024)
7B
V
3.1
37.6
Qwen2-VL-72h VL-72B (Wang et al., 2024)
72B
V
1.3
17.1
10.0
3.9
7.0
Qwen2.5- (Bai
2025)
7B
V
V
4.8
165.0
12.2
56.6
34.4
Qwen2.5-VL-72B (Bai al., 2025)
72B
V
5.0
75.8
QVQ (Team, 2024)
72B
V
Y
5.4
907.6
13.0
60.2
36.6
Vidco-LLaVA (Lin ct al. 2023)
7B
V
V
3.0
13.8
5.8
73.6
39.7
Vidco-LLaMA (Zhang
2023a)
7B
V
4.6
1348
54.6
Video-IIaMA22 (Cheng et 2024h)
V
2.9
Qwen2.5-Omni (Xu et al,, 2025)
7B
Y
3.6
79.0
6.2
45.2
25.7
Emotion-LLaMM (Cheng al., 2024a)
7B
1.0
1.3
5,8
0.0
2.9
HumanOmmi (Zhao et al., 2025b)
7B
4
1.0
1.0
7.8
0.0
3.9
R1-Omni (Zhao et al., 2025a)
0.5B
V
4.9
142.7
7.6
51.9
29.8
AffectGPT al., 2024c)
7B
4.8
120.7
51.3
32.1
Closed source MLLMs
GPT-4o (OpenAl, 2024)
4
v
4.4
144.0
7.6
79.7
43.6
GPT-4. I (OpenAI, 2025)
v
5.0
116.7
7.4
70.9
39.1
Gemini-2 2las-lassh (Google, 2025a)
v
3.7
50.8
44.0
27.5
Gemini-2 Sommini-2.5-Flasff (Google, 2025b)
4.5
47.4
Gemini-2.5 2.5mii-2.-Pmmr Pro (Google, 2025c)
v
5.0
571.6
16.6
64.3
40.5
25

---

Published as a conference paper at ICLR 2026
Table 10: Overall Performance Comparison (%) on IR. The top three performing results are highlighted in red (1*), blue (2"h), and yellow (3tl) backgrounds, respectively.
Model
LLM Size
T
Avg Step
Avg Token
Ree-S
Rea-$
CoT-S
Open-source MLLMs
Qwen2-Audio (Chu ct 20244)
7B
3.0
41.6
27.3
45,9
36.6
Audio-Reasoner (Xic al., 2025)
7B
4.7
361.2
23.4
74.9
49.2
Qwen2-VL- -e-72 (Wang et al,, 2024)
3.8
61.0
Qwen2-VL--221 (Wang et al., 2024)
72B
V
1.0
7.5
30.7
0.0
15.3
Qwen2.5-VL-7B (Bai al., 2025)
5.0
183.7
28.1
Swen2.5-VLL-72B (Bai ct al., 2025)
72B
V
V
4.8
319.4
27.9
93.4
60.6
QVQ (Team, 2024)
72B
5.5
1417,8
25.7
84.0
Vidco-LLaVA (Lin al. 2023)
7B
3.1
27.6
Vidco-LLaMA (Zhang al., 2023s)
7B
V
V
4.7
130.8
20.2
62.5
41.4
Video-I. (Cheng et al., 2024h)
7B
V
V
1.5
10.5
16.8
14.2
15.5
Qwen2.5-Omni (Xu el al,, 2025)
7B
4
3.4
59.4
5.4
81.8
43.6
Emotion-LLaMA (Cheng al., 2024a)
7B
1.0
23.8
HumanOmmi (Zhao
2025b)
7B
1.0
15.8
0.2
R1-Omni (Zhao et al., 2025a)
0.5B
V
5.2
175.6
16.6
58.7
37.7
AffectGPT (Lian et al., 2024c)
7B
V
4.9
117.5
1.8
44.0
22.9
Closed source MLLMs
GPT-4o (OpenAl, 2024)
4.5
195.5
29.1
88.9
GPT-4.1 I (OpenAI, 2025)
v
5.3
172.6
28.7
75.8
52.2
Gemini-2.6 2.0-Flash (Google, 2025a)
v
4.4
75.6
24.7
67.9
46.3
Gemini-2.5-Flan (Google, 2025h)
4.7
277.7
65.5
Gemini-2.5-Por (Google, 2025c)
v
5.2
660.5
28.1
84.7
56.4
26

---

Published as a conference paper at ICLR 2026
lategory-level Recognition Score (ER-Lab)
Angee
59
34.0
0.0
0.0
0.
39.6
13.2
45,
17.0
17.0
472
3211
19
0.0
0.0
1.9
Excited
12
36
0.0
0.0
36
12
133
24
00
0.9
36
00
0.0
0.0
25.3
49
12
24
10.8
16.9
90
00
08
00
00
0.
136
0.
0.0
0.0
Frustratien
36.7
33
0.0
19.7
Hoppiness
8L3
28.2
89.7 94.6
62.1
76.9 76.9
8233
74A
6155
$2.1
94.9
94.9
61.5
17.9
0.0
0.0
87.2
66.7
71.8
Noutrol
4.5
03.9
41.6
78.7
75.3
77.5
69.5
0.0
44.3
60.2 67.4
65.2
75.7
5L7
9.0
9L.
69.9
10.3
9.0
20.2
Other
00
00
0.0
0.0
0.0
0.0
1311
00
6.9
14
5.6
15.3
37.5
Surprise-
10
42.1
0.0
0.0
10.5
0.0
0.0
0.0
0.0
3G6
0.0
0.0
3L6
5.3
0.0
0.0
0.3
211
10.5
Quend Aodio
Audio- Remannrr
Quest: VL V1-78
Quent- MaL------ -28
Quent 8 B--VL-8
Qwent 3-VL, -28
Video- LLaVA
QvQ
Video LLoMA
Video- LLeM44
Qwest Onei
Emation- Laat- LiMH
Ri Cren
AffeciePT
GPT 40
Ganisi D.Pluth
GFt 4.7.4.1
Genini S.S. Flash
Ganini 2.. 2-Pee
Category- level Reasoning Score Lab-
100
Anger
34.0
00
40.3
32
15
1.9
36
36
13.
0.0
0.0
25.
16.9
Fear
13.6 00 00 0.0 0.0
0.0
00
0.0
00
0.0
0.0
000
4.
0.0
13.6
000
0.0
0.0
0.0
0.0
Frestratien-
0. 367 00
0.0 08
33
0.0
0.0
0.0
0.0
0.0
08
25.0
13.2
16.7
242
30,8
19.2
60
Hoppiness
2822
846 821
769
615
94.9
17.9
0.0
718
53.9
78.7 75.3
685
66.2
65.2
76.7
9.0
89.9
20.2
Other
0.0 00
0.0 0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.
0.0
0.0
0.0
0.0
0.
0.0
0.0
0.0
0.0
200
Sodness 48.6 431 56 0.0
69 42 00 14 0.0 6.9 28 14 56 14 42 14 56 15.3
19.4
37.5
Surprise- 10.5
0.0
80.5
00
316
0.0
53
0.0
10.5
Qwend- Asdio
Audio-Rotonn Recsoner
Qrest VL VL-7B
Cheand VL- V2-
Queet 55-V.-11 VL-7B
Quennt & V8 -728
Video- Video-LLaVA LLaVA
QvQ
Video LLMMA
Video- Vides-LLate LLatt 42
Quest 5- Onni
Emerion LatkA
HomanChesi
Cresi
AffecnGit
GPr Ao
Gentnl 2.D-Puu D-Plssh
GPT-441
Gentos 2 5-F-aath
Gemini 2
Audio
Category level CoT Score (ER-Lab)
Anger-
49.
55.8
08
4.1
22.4
20.9
224
237
220
22.6
17.4
85
85
601
52.0
37.1
22.3
252
22.7
23.8
Escited
2477
25.7
0.4
98
291
30.4
374
69
189
0.4
29.1
00
00
272 45.4 379
125 278 34.4 44.7
80
265
20.8
18.5
0.0
45.6
8.8
14.4
20.3
Frestration
58.5
25.9
30.4
16.5
0.3
0.0
29.3
46.5
29.1
37.9
38.1
Hoppiness
53.2 52.3
47.4
614
87A
81A
83.
52.6
72.1
90.8
87.0
47.4
47.4
62.
42.9 $1.2
19.9
878
69.6
80.1
Nautral
11.3 64.5
23.0
16
82.8 82.9
77
22
46.4
52.6
75.0
52.6
99.3
56.7 52.5
94
98.5
57.8
19.9
43.7
40
Other
233
Sodnege
40.9
51.2
10.
33.5 32.4 5.
6.8
3.5
19.0
0.7
2.8 27.8 29.5 42.1 24.0 332 34.7
52.5
Surpetce
21.1
84.9
2,2
11.6
31.4
10.2
25
12.7
7.0
15.8
18,5
0.0
7.9
40.0 40.0
37.6
8.0
17.1
32.4
24,9
Qwenzz Audio
Audio Audio-Rentai Reatoner
Chest-yp VL WL-799
YL-78
Quent VL-18-8
Quent 5-WL 720
Video- L
QvQ
Video- Laak LLMMA
Guant O-Onn
Emerion Smuroa-LLLMMM HomenDonii
LookA
Rt O-Onii
Affectgit
GPT -4
Ganial 2 D-Paush
GPT 4.4
Genins 2 F-Plish
Gemini- 2 S-Pro
Owest-vV
Vidao
Figure 7: Category-level Performance Comparison (%) on ER-Lab. We showcase fine-grained
comparison results of MI.LMs using Rec-S, Rea-S, and CoT-S across emotion categories.
27

---

Published as a conference paper at ICLR 2026
Category- -level Recognition Score (ER-Wild)
100
Aager
23.
104
179
19
19
134
18D
R7
25.4
32.8
34.3
Aealitty
00
2.4
103
172
35.0
152
0C
50
00
6.
00
343
17.2
241
27.
44.0
242
Cartungt
4.
4.
13.8
00
DO
00
00
02
36.4
18.2
273
40.3
Disppikement
15.0
1433
23.0
95
23.8
66.7
997
43
00
95
91
4.
21.8
28
381
Digat
15.2
0.4
8.5
6.2
6.
6.2
31
00
33.1
53.
0.5
1.5
83
14C
24.3
12.6
52
156
52
52
16.5
172
Hoppinos
14.6
25.4
60.3
051
5629
55.5
65.5
16.9
66.7
60.5
2511
91.4
85.7
50.9
23.2
5.0
1000
600
06.2
Doy
50.5
50.
30.6
2933
300
41.5
12.
26.0
20.8
12.5
17.1
36.5
42.
122
2.5
12.2
415
55.7
46-3
Sedwar
25.6
23.2
29.3
14
182
28.6
257
12.
6-4
12.3
40.2
47.3
257
15.
172
21.
5LC
46.8
47.
Acored
157
16.7
6.8
12.5
125
16.
0.
500
00
12.
0.0
8.8
0.5
70.8
167
20.9
Shy
180
00
300
47.2
Suprttx
29.3
28.9
12
374
46.2
15.
28.0
22.6
K2
924
2.8
PB
22.8
Queed Audio
Audio Heueetoe
Oroud 18, 1p
Geant & MM-
Qenat B- -788
Vidco L
Vidro LaaM
Vilte cloRAe
Gemit Oved
Eootien LZaaA
Affertery
40%
Gontes 2.0 F--iishh Flahh
APT-4
Category- level Reasoning Score (ER- Wild)
42.4
63.2
25.8
127
35.2
320
545
25.3
235
443
00
0.0
53
26.5
731
44.2
47.
53.0
44.3
Cwrteegt
9.6
53.6
07.4
13.
506
22.
57.3
0.0
25.4
25.
356
0.
00
46.
299
16.
61.8
47.
415
68.5
Dhupperiment
317
66.9
59.5
58.5
5511
553
19.0
00
539
95.9
825
Digaat
2019
48.5
211
11.0
2548
50.3
47.5
27.2
272
35.6
00
403
26.0
600
32.4
472
40.4
08.4
Fear-
4009
18.3
30.9
57.8
58.6
26.7
54.7
00
232
441
72.1
347
552
Heppiees
27.3
55.2
37.8
27.9
731
30.2
75.
75.0
72.7
62.0
0.0
533
576
70.2
90.
75.0
79.0
Halkerwas
01.7
13.5
25.4
67.9
61.
662
69.4
60.2
74.6
0.0
82.4
42.5
6377
70.2
64.7
72.2
00.4
72.0
1.6
6669
561
53
66.2
0.0
392
720
92.0
Neutral
15.7
65.5
$2.0
933
45.7
22.5
53.5
15
60.6
652
64.3
0.0
00
46.1
262
512
84.
60.2
26.3
54.
6688
50.5
Sarcartc
14.0
75.2
22.3
343
503
77.2
53.5
0.0
21.6
22A
54.2
00
00
238
233
045
784
51.5
50.7
70.7
Gly
212
40.2
5.
13.4
32.5
382
46.
00
301
131
402
00
00
27.3
142
61.
223
243
315
52.4
Supprice
Greent- Resceeer
Cancal -7
Qweet-- 77M
Qrard 14--77
Vidce Lane
Vitu HaMMA
Afhecto
4PT-44 Au
Greal 8-
1.5
Category-leve level CoT Score (ER- Wild)
100
35.0
2611
51.5
Apcicty
3.4
26.2
35.2
15.4
$50
72.5
38.0
0.0
15.5
15.7
393
09
172
30.0
43.2
50.7
93.
56.1
40/4
45.9
Deappthtmert
25.4
44.0
423
25.8
30.2
43.
312
23.5
77.4
70.8
36.9
02
48
32.2
28.6
400
332
548
521
635
Miguet
Fear
25.2
31.5
11.7
13.3
28.0
417
37.
11
16.2
16.0
25.0
03
16.3
2966
24.
14.
355
33.4
418
47.3
Happiess
30.6
6.7
29.2
50.7
42.9
$7.9
0.6
04.7
34.5
37.
0.
19.0
75.9
21
58.5
916
28.5
04.5
400
35.4
Noutral
25.8
99.8
30.8
40.7
52.5
33.2
48.6
12.5
09.1
94.
48.5
15.9
34.3
44.0
20.3
756
75.4
57.6
31.5
Sodrcas
32.9
17.3
31.
45.4
28.7
22.2
230
33.4
24.3
23.4
3411
43.3
22.3
50.0
Sarsaricc
14-1
43.4
11.2
72
34.3
54.7
14.
92
10.7
32.4
02
12.7
66.5
72.4
35.2
Scored
25,5
192
31.
134
115
253
03
16.
375
32.8
12.6
162
181
252
91
26.6
00
7.4
30.
47.7
Supriee
20.7
43.1
IM
65.7
19.4
12.3
27.
52
262
40,1
173
370
205
14.
677
Geonkt Asdla
Rusreer
Quoot v8.78
736
DVQ
Kide- cloRA
Indor Laata
Smutien
Aftecton
CPt
Genin Pias
Genor- Plad
Figure& 8: Category-level Performance Comparison (%) on ER- Wild. We showcase fine-grained
comparison results of 20 MLLMS using Rec-S, Rea-S, and CoT-S across 15 emotion categories.
28

---

Published as a conference paper at ICLR 2026
Category-level Recognition Score (Noise-ER)
100
Aegry
69.3
15.8
26.7
30.7
Happy
27.4
20.0
71.6
811
73.7
77.9
75.8
80.0
99.8
91.6
44.2
77.9
94.7
71.6
31.6
6.3
5.3
91.6
87.4
84.2
Neutral-
15.2
46.5
70.7
57.6
35.4
25.3
17.2
35.4
12.1
29.3
10.1
66.7
46.5
28.3
1.0
59.6
71.7
28.3
4,0
60
95.0
17.6
Surpeise
40.6
43.8
0.0
12.5
65.6
46.9
40.6
18.8
0.0
40.6
18.8
6.2
53.1
53.1
3.1
3.1
944
25.0
59.4
78.1
20
Worried
51.9
29.3
72.2
81.5
35.2
5.6
6L1
85.2
Quent- Audio
Audlo Recconer
Qwent VL-78
Qwenz VL-77 ML-728
Qwent VL-78
Quant B-VL VLL-7BB
Video Video-LLaVA LLaVA
QvQ
Video- Video-LaMM LiaMA
Video-LLMM LLoMA2
Quend 5- O-Onni
Emation-Ue AllHA
Humon
AffecraPY RecToPT
GFT
Genini O-Flash
4.1
Gemini-2 B.B-Flesh
Gemini 2. 8-Pro
Flesh
Category- level Reasoning Score (Noise ER)
100
Asgry
39.1
58.8
Happy
46,9 72.0
62.4
23.3
84.6
86.7
82.3
39.5
91.4
112
71.
00
0.0
725
716
75.0
40.5 87.0 86.2
87.8
Neutral-
43.0
60.3
56.8
89.
60
Sod
61.1 78.4
38.4
24.8
59.1
73.5
77.2
12.5
38.0
25.4
68.3
0.0
0.0
76.3 62.9 72.5 57.0
84.2
80.3
95.2
Surprise-
68.5
74,2
27.6 14.6 79.4 77.1 69.9 35.4 14.3 14,6 45.8 0.0 0.0 59. 37.7 63.8 382 39.6 79.7 86.8
20
78.1
0.0
Quenz- Audio
Audio Recsoner
Quenz VL-78
Owen VL-778
Qwen? 8-VL. V-VL-78
Qunet 8 M2-728
Vides Vides-LLaVA LLaVA
QvQ
Video- Video-LLMM LaMA
Video- LLMMAR
Guen? 5-Omni
Emotion LOMA
HumonOmei
RI --nn
Affactapy
GFT
Genini - O-Flash
GPT-4,
Gemini-2 Plesh
Gemini 2 8-Pro
-ategory-level CoT Score (Noise ERR)
Angry
68.1
86.5
36.1
28.5
20.8
34.7
42.6
24.5
332
23.5
16.8
42.6
78.2
43.6
49.9
44.7
38.1
51.0
55.3
80
Happy
37.1
46.0
67.0
52.2
791
82.3
79.1
59.7
93.6
51.4
57.8
38,9
47.4
72.1
31.6
40.7
22.9
89.3
86.8
86.0
50.3
80.4
60
3ed
48.2
58.9
42.3
27.9
42.2
52.3
51.2
26.0
33.7
28.2
49.3
47.5
42.9
68.4
38.2
42.1
37.5
69.4
78.6
68.8
40
Surprise
84.6
62.0 55.3
32.3
33.5
23.8
32.3
20
Wornied
61.7
70.0
34.6
64.3
82.2
87.9
0.8
56.6
2.8
0.9
35.5
42.2
75.3
67.9
87.9
06.3
87.8
QwenZ- Audio
Audio- Reasoner
Qwent- -L--7
Qwen2-1 72-
Owent 5-VL-7B 47-78
Owent & VL-728
Videa- Videa-LLaVA ALLVA
QvQ
Video-Le LLoNA
Video LLaMA2
Quean 5 Oreii
Emation-Uu LLOMA
HumanOmo
R1 Omni
AffectaPy
GPT-4o
Gemini -2 O- Flesh
GPT
Gemini 2 Flash
Gemin 2 5
Figure 9: Category-Ievel Performance Comparison (%) on Noise-ER. We showcase fine- ane-grained
comparison results of MI.LMs using Rec-S, Rea-S, and CoT-S across emotion categories.
29

---

Published as a conference paper at ICLR 2026
Category-level Recognition Score (SA)
100
Negative
634
69.3
11.
11.6
18.0
22.0
28.7
212
0.0
15.
72
25
6.4
7.3
19.6
18.6
18.3
26.3 40.6 49.2
Neutral
40.9
57.6
76.5
84.3
71.
56.3
45.5
18.9
67.4
36.4
41.9
90.7
86.1
68.4
3.8
77.8
83.3
65.7 50.8
36.9
Positive
57.6
7.5
Qwenz Audia
Audio Reasoner
Quenz- QwenZ-V2-1 VL-78
Qwven VL VL-728
Quen2. D-VL-78
Qwent 5 VL VL-722
Video- LLoVA
QVQ
Video- LLaMA
Video Video-LLaMA LLaM42
Owenz 5. Omusi
Emotion- LLoMA
HumonOmasi
OOmii
AffectOPT
GPT-40
Genini-2 2.0-a D--Flssh
GPT -4
Gomini 2 5-Flesh
Gemini: - Pro
Category-level Reasoning Score (SA)
100
Negative 69.9
83,9
48,9
12.7
559
73.3
66.5
49.2
23.1
23.6
50.9
0.0
0.0
63.3
56.9
789
599
45.8
52.6
76.6
Neutral
525 77.4
78.0
8.6
83.8
8866 79.5 45.0 76.2 34.3 80.9
03
0.0 541 42.6 95. 92.8 75.6 52.6 76.5
Positive 69.3
82.8
61.3
6.1
77.9
86.7 84.5 70.4 63,3 26.5 77.9 0.0
0.0 62.2 53,9 86.4 77.2 67.8 6L7 82.6
Queen2 Audio
Audio Reasoner
Quenz RvenZ-VL-1 VL-78
Cavan -L VL-728
Quee2 B-VL-70 VL-78
Qwent & B-VL-i VL-722
Vidao- LLaV
QvQ
Video LLGMA
Video LLaM42
Gwenz 5. O-Onni
Emation- LLMMA
HumanOmni
RI OOOnn
AffectGRT
GPT ..
Genini-z 2 D-Flach
GPT -4,
Gemini 2 F-Flssh
Gentni Pro
Category- CoT Score (SA)
100
Negative
66.7
76.6
36.9
35.2
11.6
29.0
13
38.3
36.0
46.6
Neutral
46.7
67.5
77.3
46.5
77.5
72.5
62.5
32.0
71.
35.3
61.4
45.5
431
61.
23.2
86.6
88.1 70.6 517
56.7
50
Positive
60.5
70.2
45.6
16.1
55.4
64.1
70.8
66.9
52.0
44.7
54.9
3.7
11.5
42,2
39.5
54.4
46.8
35.5
52.7
68.7
Qweesz-u Audio
Audio Reaconer
Quant VL. KB
Qwan2 VL-728
Owen2 5-VL-70
Qwent 5 VL--727
Video- LLaVA
QvQ
Video- LLoHA
Video Video-LLaMAt -LaM44
Quest B- Omei
Emohion-Ue LLaMA
HrmonDrnii
.OOml
AffectSPT
GPT-A GPT-4o
Genini - D-Fahsh
GPT-4.1
Gomini-1 2.5mi-22.... P-Pass
Genini -rrr
Figure 10: -ategory-level Performance Comparison (%) on SA. We showcase fine-grained comparison results of 20 MILMs using Rec-S, Rea-S, and CoT-S across sentiment categories.
30

---

Published as a conference paper at ICLR 2026
Category-level Recognition Score (FG-5A)
100
Negottee
7.6
23.5
16.8
Nautral
8300
800
0.
4.0
56.0
96.0
76.0
12.0
1000
52.0
26.0
24.0
Pesiiine
2.3
395
66
53
6.6
211 105 0C
26
316
92
00
79
1LB
118
25
2.6
224
18
211
Streeg Negottve- 13.0
272
22
98
54
22
8.7
26.1
66.5
22
00
11
0.
43
6.5
11
4.3
11
87
36.1
Strong Positive 3
13.5
388
77
115
96
19.2
38
58
13.5
19
00
125
1.
5.6
3.
19
38
192
15.4
Streeg Megetive-
3.0
40.0
0.0
0.0
0.0
0.0
3.0
40
Very Strong
33.9
33.9
Negotive
17
0.
33.3
1.7
17.2
15.5
Weak Pasitive.
2.9
7.2
20.3
0.
18
0.0
8.0
s0.
18
Gwasd Audia
Ressoner
Quwez- -B-78
Quest Qeest W. 728
Quest 5 -2.733
2----37 -8
Video LalfA
QvQ
Video LinkA
Vrido Ltaksz
Guent &. Orail
Emotion LLMMA
Humon
Omes
Affecrten
GFT- 6FT-A0
Gemino O-Plesh
GPT .4.3
Genany 2. Plach
Genind S-Pra
Category leeell Reasoning Score (FG-SA)
100
Nagotive- 79,3
31.9
39
54.4
55
92. 78.9 40.5
0.0
531
56.9 79.2 71. 359 48. 58.2
Neutral 25.3 7L7
717
8.0
824
93.
29.
65 27.0 68.0 89.3
00
0.0 58.5 48.0 96.0 99.0 297 49.7 66.6
Posttine 47.9 769
43.0
3.6
700 80. 70.7 52. 27.3 40.5 67.5 0. 0.0 57.9 39.5 50.7 77. 51. 60.9 03.5
Strong Negetive 50.0 896
26.0
3.4
99.7 53.0 50. 92. 75.1 35.6 246
0.0 0.0 46.7 613 713 57.0 257 34.5 48.4
60
Stroeg Pesitive
58.1 25.8 42.0
48.0
423
462 418
60.5
Very Strong Nagottve
3.0
58.3 36.7
63.3
Very Strong Pasitive
100,0
3.0
66.7
50.0 111
75.0 66.7
556 77.8
Wesk Negative 76,4
839
29.4 4.0 88.8 7288 60.1 77.3 76.0 44,4 35,8 0.0 0.0 47.0 62.1 83.2 616 40.8 45.5 71.0
Week Pacttiee-
43.0
73M
47u
4.8
62.5 90.3 F2.1 b39 34,9 46.0 56.5 0D u.p. b0 28.4 87.9 2322 603 b5 735
Qwest Asdio
Audia Rassoner
Queed V -78
Grest 728
Queed 2-V
Quentt -28
Video- LlaVa
QvQ
Vides- LoNA
Vidao Li Llaaaz
Quent Orne
Bmotten- LaMe
#1 Omee
Affecisnt
GFT -o
Gamand Plach
Gremia- 2.B. Flash
Gemanl 8-Pra
Category level CoT Score (F6-SA)
100
Negotive
5322
62b
10.8
57
333
40.0
38
46. 411 219
205
2.5 0.4 28.2 402
413
30.8
209 32.5 33.7
Nautral 14.7
515
798
44.0
712
645
355
327
155 620 76.7 440 48.0
67. 30.0 98.0 99.5 558 38.6 45,3
80
Pesitive
38.3
264 15.0 363
34.8
42.0
39.7 50.8
Strong Nagoties-
Strong Poxitive
21.7
3.8
34.5
29 15.8 27.7
25.0
23.9
250
30.4
37.9
Very Streng Negetive $5.0
90.0
10.0
3.0
10.0 40,0 10.0 60.0 29,2 18.3
3.3 0.0 0.0 31.7 35.0 25.0 20.3
3.3
15.0 43.7
Vary Stroeg Positive- $3.3
66.7
16.7 16.7 4:7 53.3 98.3 25.0 56 82 194 0.0 35.3 37.5 35.5 50.0 53.3 27.8 55.6 66.7
20
Wesk Negative
54
60.9
15.5 2.9 32.9 4599 430 386 37. 38.9 19.5 0. 0.0 24.4 40.5 43.3 32.5 29.0 30.5 48.4
Week Posttive
257
3.9
34.9
26.8 17.5 32.4 29.7 0.0 25.2 22.3
41.6 33.0 32.0 45,7
Guent Audio
Audic-ale Restoher
Qweed -78
Qreat- -L VL-228
Queed P--V--7
Quant & 720
Video- LLoVA
cyQ
Video-LL LLMMA
Viduo Llokht
Emalion- L
Ote
Homan
#1 Grani
Afrecight
EPT- -o
Gammni O. Plash
GPT
Gemuai 2.8, Plash
Gemint 2. S-Rre
Quest&
Figure 11: Category-level Performance Comparison (%) on FG-SA. We showcase fine tne-geained comparison results of 20 MILMS using Rec-S, Rea-S, and CoT-S across sentiment categories.
31

---

Published as a conference paper at ICLR 2026
Category-leve Recognition Score (IR)
100
Acknowledging
16.7
50.3
16.7
0.0
16.7
33.3
6.7
0.0
0.0
3.0
0.0
0.0
0.0
0.0
0.0
3.4
8.3
5.9
25.0
92
Agrezing
5.8
15.4
15.4
26.9
23.1
0.0
0.0
3.0
19
0.0
0.0
0.
19 135 212
19.2
25.0
15.4
800
Consoling
18.2
7.3
00
00
0.
0.
0.0
Encouroging
3.4
17.2
24.1
3.4
37.9
24.1 20.7
3.4
0.0
55.2
0.0
3.4
0.0
0.0
6.9
24.1
27.5
17.2
6.9
27.6
Neutral
3.8
9.8
36.1
60.2
20.3
18.8
12.0
20.3
8.3
47.4
10.5
672
59.
60.9
0.9
23.3
36.1
15,8
3.8
8.3
40
Questioning
44.4
46.3
58.3
46.3
25.9
76.9
0.9
9.3
1.9
45.4
37.0
48.6
Suggesting
48.3
0.0
1.7
0.0
0.0
0.0
60.0
38.3
Wishing
14.0
10.0
0.0
0.0
2.0
4.0
4.0
0.0
0.0
9.0
2.0
0.0
0.0
0.0
0.0
6.0
6.0
10.0
6.0
16.0
Qorent Audio
Audio Retconer
Quent VL-778
Qwent VVL-78
Qwen? 5. VL-77
Quan? 5. S-VL.1 --VB-728
Video- Videa-LLaVe LLaVA
QVQ
Video -Loo--LoAAA
Video- LLaHAZ
Queet 8 O-Omeii
Emation- LLOMA
HumanDnn
Al -OOmsi
AfferGGT
GPT
Gemini- 2. O-Flach
GPT-4.1
Gemini 2. 5---lach
Senini- 2.oni---... 5-5
Category-level Reasoning Score (IR)
100
Acknowledging-
56.6
86.1
46.5
0.0
57.6
91.7
81.2
15.3
20.8
4.2
64.6
0.0
0.0
54.9
28.6
79.2
64.5 62.5 54.2
76.6
Agreeing
34,9 66.1
64.4
0.0
802
94.5
78.4
30.4
43,9
4.2
86.9
144
0.0
49.4
32.9
96.0
74.7
69.9
68.8
60.8
Consoling
57.0
65.3
43,8
0.0
73.5
86.4
65.4
412
54,8
22.3
78.3
0.0
0.0
56.5
40.2
85.2
72.3
57.0
5599
79.0
-0
Encouroging
95.4
58.1
83.0
Neutral-
46.1 70.9
65.7
0.0
86.5
96.5
80.9
44.4
77.1
26.7
89.2
0.0
0.8
60.7
50.7
92.0
68.1
65.8
56
83.0
Questioning
37.7 83.8
89.1
0.0
82.4
91.9 85.4
39.7
74,2
7.6
72.8
0.0
0.0
62.0
41.9
86.3
70.8 72.3
75.4
67.3
Suggesting 58.4
06.2
73,8
0.0
87.5
94.2 91.9 39,9 55.1
5.7
82.5
0.0
0.0
70.8 52.9 93.5
69.9 78.2 80.5
92.2
20
Wishing
11.7
82.0
0.0
0.0
45.5
41.4
Quenz. Audia
Audio Reazoner
Owent VL-78
Qwenz VL-728
Qwan? 5. VL.78
Quan2 5 VL. -28
Qva Video- LLaVA Video- LLMHA
Vidao- Video-LLaMA LLMM42
Quent b-Omni
Emation LioMA
HumerDDnni
Rt -mmm
AffectGPT
GPT-46
Gemini- Flash
GPT -4.1
Gemini- 2.5. Flash
Genini- 2 B..
leve CoT Score (IR)
100
Acknowledging-
21
32.3
27.4
14.3
Agreeing
20.4
40,8
37.0
77
47.
60.7
50.8
152
22.0
21
44.4
0.7
0.0
24.7 17.4 47.9 47.9
44.6
45.9
48.1
Consoling
60.3
36.1
24.7
4.5
43.1
47.7
51.8
39.7
33.8
14.8
39.2
0.0
0.0
28.3 20.1
44.3
39.1
33.0
33.4
46.6
Encouroging
28,2
45.3
45.1
1.7
63.0
59.8
53.9
21.3
30.7
32.5
42.2
2.6
0.0
29.1
23,9
55.3
30.7
45.4
40
59.2
30.0
Questioning
50,8
64.1
52.7
26.4
70.3
71.4
65.8
32.8
75.5
4.3
41.0
0.9
0.0
31.9
22.8
65.9
53.9
60.4
68
70.0
Suggesting
414
64.4
50.2
1.7
61.2 69.6 70.1 19.9 27.6
2.8
42.1
0.0 0.0
35.4 27.3
76.7 54.1
58.3 61.1
77.8
Wishing
26.7
39.5
26.5
0.0
41.6
48.9
43.4
21.0 22.7
5.8
42.0
0.0
0.0
22.7
20.7
46.5
35.0
35.0 31.0
50.7
0
Quent-un Audia
Audio Retconar
Qhent VL-78
Qwent- Monnt-VL-77 -VL-728
Quant 5 VL-78
Quan2 VL- VL-728
Video- Video-LLaVA LLaVA
Qva
Video- LLAMA
Video- Vido--LaMAt LLaM42
Qwend 8- 8-Oenn
Emation-La LIMMA
HumanDrnn
Rt -Oeri
AffectarT
GPT 4o
Gemini- 2.0 3.0- Flash
GPT 4
Gamini- 5.5--lesh
Gemini- B.Prr
Figure 12: Category-level Performance Comparison (%) on IR. We showcase fine-grained com- parison results of 20 MLLMs using Rec-S, Rea-S, and CoT-S across 9 intent categories.
32

---

Published as a conference paper at ICLR 2026
Matrix Bivariate Relationships and Univariate Distributions
Average Average Steps
Recognition Score 20
Reasoning Score
100
CoT Score
100
50
1000
20 40
0
100
0
50 CoT Score
Average Steps
Average Tokens
Recognition Score
Reasoning Score
Figure 13: Multivariate Analysis of Evaluation Metrics. We showcase pairwise scatter plots with distribution diagnostics across five different evaluation metrics on 20 state-of-the-art MLLMss.
33

---

Published as a conference paper at ICLR 2026
Question: As emotion recognition expert, identify the prominent emotion displayed the video.
Please think step by step.
Candidate labels: Anger, Sadness, Surprise, Hoppiness, Excited, Fear, Frustration, Neutral, Other. Audio Clues: The speaker exhibits frustration through their struggle with financial constraints, as indicated by the expression 'I can't afford to back to school." This statement conveys sense of helplessness or distress due to inability to financially support educational pursuits. Emotion Labels: Frustration.
AffectGPT:
Rating:
Step 1: Analyze the audio clues for indications of emotional states, such as excitement and anxiety.
1/1
Step Examine facial expressions and body language for signs of strong emotional responses.
1/1
Step Infer possible emotions (anger, frustration, fear) based on combined audio and video analysis.
Step 4: Note the lack of specific emotional cues, making it difficult to determine the exact emotion.
0/1
Step 5: The predicted emotion is unclear due to insufficient evidence.
0/1
Reasone::
Rating:
Step Assess the speaker's and initial statements for emotional
1/1
Step Analyze specific phrases for indicators of uncertainty, helplessmess, and financial burden.
Step 3: Consider and rule out alternative emotions such anger, surprise, happiness, excitement, fear, and neutrality.
1/1
Step 4: Focus on the presence of sadness and frustration based on the combination of cues. 1/1
Step The predicted emotion is frustration.
Gemini-2.5- 2.5-Po-
Rating:
Step Identify the primary subject for emotion recognition (the person on the left, who 1/1
more expressive and actively speaking).
Step Analyze the subject's facial expressions and gestures across key frames, focusing on changes animation, hand movements, and facial cues.
Step 3: Compare observed expressions to candidate emotion labels, ruling out anger, sadness, surprise, happiness, excitement, fear, and neutral based on specific cues.
1/1
Step 4: Conclude that the combination of raised eyebrows, open-mouthhed speech, and open 1/1 hand gestures most closely aligns with frustration.
Step The predicted emotion is frustration.
Figure 14: Case Study on ER-Lab, We showcase the answers and ratings of three MLLMs.
34

---

Published as a conference paper at ICLR 2026
Question: As emotion recognition expert, identify the prominent emotion displayed the video. Please think step by step.
Candidate labels: Neutral, Surprise, Happy, Anger, Sad, Fear, Disgust.
Audio Clues: The speaker exhibits combination of vocal and non-verbal clues that indicate surprise, The following are key indicators: 1. High-pitched and rapid speech: This usually indicates anxiety shock, which are common emotions during surprise situations. 2. Changes in pitch and volume: There might be abrupt shift in the speaker's pitch and volume, suggesting sudden increase in emotional intensity. 3. Sighs, groans, or exclamation aanamaiio--lkee sounds can convey feelings of surprise or disbelief. 4. Paced speech: The speaker may speak quickly or hesitantly, reflecting their state of surprise or confusion. Eye contact: Lack eye contact or prolonged gaze in surprised situation may suggest uncertainty or shock. summary, the speaker's rapid, high-pitched speech, unusual vocalizations changes in pitch and volume, and paced speech, along with possible lack of eye contact, all contribute the perception of surprise in this audio. Emotion Labels: Surprise.
Unusual vocalizations:
AffectGPT:
Rating:
Step Analyze the subtitle content V'Sister, sister\ for context.
1/1
Step Observe the woman's 's crying expression and body language in the video for emotional clues.
Step 3: Note the presence of laughter in the audio additional clue.
0/1
Step Infer that sentence may be a greeting or expression concern based on the woman's negative emotional state.
Step 5: The predicted emotion is negative.
0/1
Aadio-Reasonert
Rating:
Step Assess the audio clip, focusing the spoken word and its intonation.
1/1
Step Analyze the rising pitch and emphasis the speaker's voice identify emotional clues.
Step Compare the observed vocal features with typical characteristics various emotions.
Step 4: Weigh the evidence, considering context and vocal clues to narrow down the
1/1
emotion. Step 5: The predicted emotion surprise.
Gemini-2.5-Pre:
Rating:
Step Analyze the facial features in the provided frames, focusing eyes, eyebrows, mouth, nose, cheeks, overall tension.
1/1
Step 2: Evaluate the observed features against candidate emotion labels, ruling out those that do not match the indicators.
1/1
Step 3: Determine the most prominent emotion by synthesizing the subtle clues and
1/1
comparing them to typical emotional expressions.
Step The predicted emotion is neutral.
Figure 15: Case Study on ER-Wild. We showcase the answers and ratings three MLLMs.
35

---

Published as a conference paper at ICLR 2026
Question: As an emotion recognition expert, identify the most prominent emotion displayed in the video. Please think step by step.
Candidate labels: Sad, Angry. Neutral, Happy, Worried, Surprise.
Audio Clues: The speaker's neutral emotion can inferred from their steady pace and normal speaking rate, lacking any noticeable changes pitch or volume, There emotional displays. The pauses between words suggest calm and composed delivery. Additionally, there's evidence of stress, trembling voice, or other vocal indicators of strong emotions. Overall, the neutral mood is conveyed through the consistent and calm manner of speaking. Emotion Labels: Neutral.
signs of crying, laughter, other
Affect6PT:
Rating:
Step Exomine the subtitle for self-evaluative or reflective language.
1/1
Step 2: Analyze audio clues such as trembling voice and rapid breathing for emotional indicators.
0/1
Step 3: Observe video clues like closed eyes and contemplative posture for signs of internal conflict.
0/1
Step Integrate verbal, audio, and visual information to infer the emotional state.
Step 5: The predicted emotion is regret or guilt.
0/1
ReaioneR:
Rating:
Step Note initial impression of the audio, focusing on its abrupt sharp nature.
Step 2: Analyze specific vocal features such as pitch, duration, intonation, and vocal quality.
1/1
Step 3: Compare these features with typical characteristics of candidate emotions (happy,
1/1
neutral, angry, surprise, sod, worried).
Step 4: Conclude that the predicted emotion is anger.
0/1
Gemini-2.5-Par
Rating:
Step Analyze the facial features in the provided frames, focusing on eyebrows, eyes,
mouth, and forehead tension.
Step 2: Compare observed features against candidate emotion labels, eliminating those that 1/1 do not match.
Step 3: Differentiate between the most plausible emotions (angry, sad, worried) considering the intensity and nature the expression any contextual clues.
0/1
Step 4: Determine the most prominent emotion based on the synthesis of facial clues and context.
1/1
Step The predicted emotion is sad.
Figure 16: Case Study on Noise-ER, We showcase the answers and ratings of three MLLMs.
36

---

Published as a conference paper at ICLR 2026
Question: As emotion recognition expert, identify one or several emotions displayed the video. Please think step by step.
Candidate labels: Affection, Excitement, Fatiguee, Anticipation, Happiness, Yearning .....
Audio Clues: The speaker exhibits range of emotional clues that indicate sensitivity, sadness, and suffering. The sniffle indicates sense of sadness or distress, while the sigh provides vocalization of weariness emotional exhaustion. Additionally, the soft and possibly subdued manner of speaking suggests level of sensitivity or vulnerability. The use of filler words like 'umm' and hesitation ('uh) further emphasizes feelings of uncertainty or distress. Emotion Labels: Sensitivity, Sadness, Suffering.
AffectGPT:
Rating: 1/1
Step Observe the woman's ss facial expressions and body language in the office environment.
Step 2: Note the presence of crying and other negative visual clues,
1/1
Step 3: Anolyze audio clues such trembling voice, rapid and shallow breathing, and
1/1
tense tone.
Step 4: Consider the caption content \"I m afraid of this as expression of her
1/1
emational state.
Step 5: Synthesize visual, auditory. and textual information to determine the predicted emotions.
1/1
Step The predicted emotions are fear, anxiety, stress, sadness, disappointment.
Aadio-Reasoners
Rating:
Step 1: Listen to the audio and note the characteristics of the sounds (short, sharp, percussive, high-pitched, obrupt, lacking sustained vocolization background noise).
1/1
Step 2: Analyze the qualities of the sounds for emotional clues (abruptness, isolation, lack of context).
0/1
Step Consider possible emotional associations based sound features (discomfort, surprise, disconnection, aversion, fear, disappointment, sadness).
Step 4: Synthesize the analysis and select the mast fitting emotion(s) given the ambiguity sound characteristics.
0/1
Step 5: The predicted emotions are disconnection or aversion.
0/3
Gemini-2.55 2.mini-2.5-Pro:
Rating:
Step Identify main characters in the video and focus on their visible expressions and gestures.
1/1
Step 2: Anolyze the woman the left by observing her posture, gaze, mouth, and hand position across fromes to infer emotions as sadness disquietment.
1/1
Step 3: Analyze the woman on the right by noting her attentive gaze, comforting gesture, and facial expressions infer emotions such sympathy and affection.
1/1
Step Synthesize the observations to determine the dominant emotions for each woman.
Step 5: The predicted emotions are sadness and disquietment for the woman on the left,
1/3
and sympathy and affection for the woman the right.
Figure 17: Case Study on FG-ER. We showcase the answers and ratings of three MLLMs.
37

---

Published as a conference paper at ICLR 2026
Question: As emotion recognition expert, identify or several emotions displayed the video. Please think step by step.
Candidate labels: Helplessness, Fear, Sadness, Anger, Happiness, Disgust, Anxiety ....
Audio Clues: The speaker exhibits several key emotional indicators of happiness and surprise: 1. High- pitched and upbeat tone: The speaker's voice is likely raised, indicating excitement
surprise, 2. Speed variation: There might be rapid changes in speaking rate, reflecting the intensity of emotions. 3. Smiling while speaking: Although not explicitly mentioned, the assumption can be made based on the context of the situation. 4. Enthusiastic delivery: The way the speaker delivers the message conveys sense of cheerfulness and excitement. 5. Use of positive words: Selecting words that convey positivity, such as okay or sure, reinforces the happy and surprised mood. These elements combined suggest that the speaker is experiencing happiness and surprise. Happiness, Surprise.
Emotion Labels:
Affect6PT:
Rating:
Step Analyze subtitle V"I wont be with you'" emotional intent.
Step 2: Observe the woman's ss smiling and laughing expressions and body movements the video.
1/1
Step the trembling and shaking of the individual's voice audio.
Step 4: Infer that the woman is expressing affection or desire, but also feels anxious or nervous.
0/1
Step 5: The predicted emotions are iifectiiondesire and anxiety/nerroossess.
0/2
Audio :eaioneR::::r:
Rating:
Step Assess the audio clip, noting brevity and the importance of tone for emotion recognition.
Step 2: List possible emotional interpretations based vocal characteristics of the
0/1
\"Eek!\" utterance.
Step 3: Acknowledge the lock of context, which makes definitive emotion prediction difficult.
0/1
Step 4: Conclude that the predicted emotions are ambiguous and could include disgust, fear, 0/2 contempt, disappointment, sadness, onger, anxiety, helplessness.
Gemini-2.5-Pres
Rating:
Step Analyze facial expressions in each video frame, focusing on features such as mouth shape, eye crinkling, and eyebrow position.
1/1
Step 2: Note verbal clues and context, such spoken phrases that may indicate recognition or surprise.
1/1
Step 3: Match observed facial and verbal clues to candidate emation labels.
1/1
Step Exclude emotion labels do not fit observed clues.
Step 5: The predicted emotions are happiness and surprise.
2/2
Figure 18: Case Study on ML-ER. We showcase the answers and ratings of three MLLMs.
38

---

Published as a conference paper at ICLR 2026
Question: As emotion recognition expert, identify the most prominent sentiment displayed the video. Please think step by step.
Candidate labels: Negative, Neutral, Positive.
Audio Clues: The speaker expresses negative sentiment by stating they 'don't really go for the animated type films. This indicates lack of interest preference for such movies, which be perceived negative stance, Additionally, there might be hint of disapproval or dissatisfaction within the statement, although it's not explicitly stated. stated. Emotion Labels: Negative.
AffectGPT:
Rating: 1/1
Step 1: Exomine the caption in the text for emotional clues, such as expressions of disappointment anger.
Step Analyze the woman's 's facial expressions and body language in the video for signs of anger frustration.
Step 3: Consider audio clues that indicate of frustration and annoyance.
1/1
Step 4: Infer the overall emotional state based on the combination of text, visual, and audio clues.
1/1
Step The predicted sentiment negative.
Reasonn:
Rating:
Step Assess speaker's initial statement sentiment, noting the lack of strong
positive or negative emotion.
Step 2: Anolyze the longuage used for indications of bias defensiveness, interpreting the
0/1
stance as neutral.
Step 3: Consider alternative interpretations, such as subtle negativity, but weigh them
0/1
against the overall context.
Step Weigh all evidence determine the most prominent sentiment.
Step 5: The predicted sentiment is neutral.
0/1
Gemini-2.55 2.5iiri-
Rating:
Step Analyze each frame for facial expressions and emotional clues.
1/1
Step 2: Identify recurring patterns or dominant expressions across the frames.
1/1
Step 3: Evaluate the intensity and clarity of the emotional clues.
1/1
Step Determine the most prominent sentiment based analysis.
Step 5: The predicted sentiment is positive.
0/1
Figure 19: Case Study on SA. We showcase the answers and ratings of three MLLMs.
39

---

Published as a conference paper at ICLR 2026
Question: As an emotion recognition expert, identify the most prominent fine-grained sentiment sentimeent displayed in the video. Please think step by step.
Candidate labels: Very Strong Negative, Strong Negative, Weak Negative, Negative, Neutral
Audio Clues: The audio contains numerous indicators of strong posítive emation. The speaker's 's voice trembling, which usually indicates they are feeling excited overwhelmed. There are loud laughter, suggesting amusement joy. Furthermore, the rapid pace and upbeat intonation the speech indicate excitement happiness. Additionally, the of little Wolverine' might suggest fondness admiration, further enhancing the positive emotion conveyed, Lastly, the fact that the speaker continues speaking despite the presence of background noise implies eagerness to communicate their feelings. Emotion Labels: Very Strong Positive.
AffectGPT:
Rating:
Step 1: Analyze the acoustic information, focusing on voice characteristics and tone.
1/1
Step 2: Observe visual clues, such the woman's 's smile and gaze into the distance.
1/1
Step 3: Infer the possible intent, suggesting the woman giving a compliment or evaluation.
0/1
Step 4: Note the lock of specific contextual information, making the exact sentiment unclear.
0/1
Step 5: The predicted sentiment indeterminate due to insufficient context.
0/1
Mudio-Reasone
Rating:
Step Assess the audio for key emotional indicators, such as speech and laughter.
1/1
Step 2: Analyze the nature of the laughter and descriptive phrases (e.g.. \'so cute,
1/1
\oso funny\") infer sentiment.
Step 3: Evaluate the speech content for emotional clues, noting if is neutral or emationally charged.
1/1
Step Integrate findings from laughter speech determine the overall sentiment.
Step 5: The predicted sentiment is positive.
0/1
Gemini-2.5-Pres
Rating:
Step Analyze the facial expressions and body language each frame to track emotional
progression.
Step 2: Identify the overall emotional trend and peak emotion the sequence.
1/1
Step 3: Determine the most prominent sentiment based on the most expressive frames.
Step 4: Match the observed sentiment to the candidate labels.
1/1
Step 5: The predicted sentiment very strong positive.
1/1
Figure 20: Case Study on FG-SA. We showcase the answers and ratings of three MLLMs.
40

---

Published as a conference paper at ICLR 2026
Question: As emotion recognition expert, identify the most prominent intent displayed the video. Please think step by step.
Candidate labels: Neutral, Questioning. Suggesting, Consoling, Agreeing, Wishing, Encouraging .... Audio Clues: The speaker exhibits questioning intent through their voice tone, which rises at the beginning of the sentence 'What is this? indicating curiosity confusion. Additionally. there slight hesitation in the voice, possibly suggesting that they are unsure seeking clarification about the subject being referned to. Emotion Labels: Questioning.
AffectGPT:
Rating:
Step Identify the context of the interaction, noting that the woman responding to the man's inquiry or providing an explanation.
0/1
Step 2: Analyze audio clues such as trembling voice and quicker speech pace.
0/1
Step 3: Observe video clues, including the woman's focused and absorbed state.
0/1
Step 4: Infer possible emotions, such as anxiety, tension, or excitement, based on the
0/1
combined clues.
Step The predicted intent is responding question or providing an explanation while feeling anxious and excited.
ReasonRR
Rating:
Step Assess the audio content and note the spoken word and its intonation.
1/1
Step 2: List possible interpretations based on the word and context, such as questioning, surprise, disagreement, or acknowledgement.
Step 3: Analyze the intonation to down the likely intent, focusing the significance of rising intonation.
1/1
Step 4: Consider the limited context and weigh the most probable interpretation. Step The predicted intent is questioning.
1/1
Gemini-2.5-Proo
Rating:
Step Analyze the facial expression for emotional clues, noting features such relaxed eyebrows, downward gaze, and a slightly open mouth indicating calmness and thoughtfulness. Step 2: Evaluate possible intent labels by comparing the observed expression with common emotional clues for each label.
0/1
1/1
Step 3: Narrow down to the most plausible candidates by comparing the nuances of \ssuggesting, \" \'acknowledging.' and "nnettral" based the context and expression. Step Select most prominent intent by considering the proactive and thoughtful nature of the expression while speaking.
0/1
Step 5: The predicted intent is suggesting.
0/1
Figure 21: Case Study on IR. We showcase the answers and ratings of three MLLMs.
41


Reviews:
[{"review_id": "YHBUGkkULH", "reviewer": "Reviewer_XCho", "summary": "This work introduces an emotional reasoning dataset and tests a suite of models on it.", "questions": ".", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a comprehensive testing suite and tests on a wide variety of models. Tests reasoning vs. non-reasoning.\n\nIt is interesting how the authors adapted video data for non-video models.", "weaknesses": "It is hard to know if the MLLMs don’t perform well on emotion tasks because they have trouble with modality fusion/their perception system is weaker or if it is truly a problem with emotion recognition. To test this the authors could have used one of the unimodal datasets cited in Table 1 and tested each model on a text-only version of the task or an image-only version of the task to test the model’s ability. That way the reader would be better informed if the issue was modality/modality-fusion or if the model truly is emotionally unaware. \n\nVideo is an inherently difficult modality to work in, so using it to benchmark models on a non-video primitive seems to be a misguided approach. We can get to how well these models recognize and understand emotion from other modalities. If the issue is in the video processing, that may not be an emotion recognition problem at all! It would be interesting to know the disparity as this can show what the lag from not processing videos well might be.\n\nIt would have been best to have a “human-level” benchmark for this benchmark to compare to.\n\nObservation section is interesting but also a bit obvious. e.g. Obs 3, bad visual perception constrains emotional intelligence.\n\nTable 2 “Colsed-source MLLMs”  “Closed-source MLLMs”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rzK4WZgp2V", "reviewer": "Reviewer_12fu", "summary": "This paper presents MME-Emotion, a holistic evaluation benchmark for assessing the emotional intelligence of MLLMs, addressing the limitations of existing benchmarks (inadequate scenario coverage and inconsistent protocols that overlook reasoning capabilities). Its core contributions include: 1) constructing the largest benchmark to date, with 6,500 curated video clips paired with task-specific QA pairs, covering 8 emotional tasks across 27 scenarios to test generalization; 2) designing a holistic automated evaluation suite via a multi-agent system, using three unified metrics and validating reliability with 5 human experts; 3) conducting rigorous evaluation of 20 MLLMs, revealing key insights. MME-Emotion serves as a foundation for advancing MLLMs’ emotional intelligence, with source code and data in the Supplementary Material.", "questions": "1. Is the use of a multi-agent architecture for model evaluation sufficiently stable? For instance, can consistent results be obtained across multiple runs?\n\n2. Will updates to the version of the core judge model affect the final evaluation results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. MME-Emotion has the leading benchmark scale and scenario coverage, which includes 6,500 curated video clips with task-specific QA pairs and 8 emotional tasks. This benchmark could enable fine-grained evaluation of model generalization and address the insufficient scenario coverage of existing benchmarks.\n\n2. An automated evaluation suite is proposed. A multi-agent system-based evaluation framework is designed, which could evaluate the performance of MLLMs without manual annotation of reasoning steps. The evaluation suite shows extremely high consistency with human experts.\n\n3. This paper has carried out a comprehensive empirical analysis of 20 state-of-the-art multimodal large language models (MLLMs) based on the self-constructed MME-Emotion benchmark, covering both open-source and closed-source models. It not only reveals the insufficient emotional intelligence of current models but also clarifies the emotional intelligence construction paths of generalist models and specialist models, providing guidance for future research.", "weaknesses": "1. Compared with existing emotional intelligence benchmarks, the main contributions of this paper lie in two aspects: first, it incorporates relevant evaluations for emotional reasoning capabilities; second, it designs a large model-based automated evaluation algorithm.\n\n2. The paper only considers various emotion recognition tasks and does not include emotion generation tasks. Can recognition-only tasks sufficiently and comprehensively assess the emotional intelligence of models?\n\n3. The paper evaluates the model's emotional reasoning ability by identifying the triggering factors behind emotional states, which represents a relatively singular perspective.\n\n4. The sensitivity of different MLLMs to prompts may affect the evaluation results.\n\n5. The large model-based automated evaluation method incurs certain costs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jMZ3WSIovn", "reviewer": "Reviewer_73cU", "summary": "The paper introduces MME-Emotion, a comprehensive benchmark designed to evaluate the emotional intelligence of large language models (LLMs) across multiple modalities. Comprising over 6,500 curated video clips spanning 27 real-world scenarios and eight distinct emotional tasks, it covers areas such as emotion recognition in laboratory and natural settings, fine-grained and multi-label recognition, sentiment analysis, and intent recognition. The authors also propose a multi-agent evaluation framework that automatically assesses multimodal LLMs (MLLMs) using three unified metrics: Recognition Score (Rec-S), Reasoning Score (Rea-S) and Chain-of-Thought Score (CoT-S). The framework uses GPT-based 'judge' and 'step' agents to evaluate model responses without the need for human-annotated reasoning traces and validates this method through high inter-rater agreement with five human experts. Using this benchmark, the authors evaluate 20 state-of-the-art MLLMs and reveal that even the top models only achieve 39.3% Rec-S and 56.0% CoT-S, highlighting significant room for improvement. The paper also explores the trade-offs between generalist and specialist models, emphasizing the importance of multimodal fusion and reasoning depth.", "questions": "1) Were ablation experiments conducted with the same MLLM in audio-only, video-only, and audio+video modes?\n2) Using a set list of emotions in the prompts makes the task easier than using an open format. Are you aware of this limitation? Do you think the current wording of the task reduces the validity of the results?\n3) Do you plan to analyze how well the model performs across different cultural groups (e.g. East Asian versus Western)? Have you considered that ignoring cultural differences in emotional expression could lead to inaccurate conclusions about 'universal' emotional competence?\n4) Have you conducted a sensitivity analysis to assess the models' ability to reason temporally? If not, how do you evaluate the models' capacity for temporal reasoning using the available data?\n5) Do you plan to categorize common mistakes? For example, can the current evaluation framework distinguish between logically sound reasoning based on incorrect premises?\n6) You claim that you only use test splits of the original corpus. However, the wording of the questions and the candidate labels may indirectly reveal information about the data distribution, particularly if the prompts are based on the original annotations. How did you verify that there was no such leakage?\n7) Did you carry out further checks on complex or borderline cases? Do you expect the level of agreement to decrease (to much lower than the current 0.953) when the sample is expanded?\n8) What are your plans for offering adaptive $\\alpha$ adjustment on a task-by-task basis?\n9) Do you have any data on the effectiveness of long arguments? Could an average step size that is too high sometimes be a sign of redundancy rather than depth?\n10) In terms of emotions, scenarios and complexity, how exactly was balance achieved? How are the characters in the video distributed by age, gender and ethnicity?\n11) Have you analyzed how noise affects reasoning quality (Rea-S)? For instance, can models retain a logically sound argument but draw incorrect conclusions?\n12) Have you conducted any more extensive experiments on agent replacement?\n13) The benchmark does not include chains of reasoning from humans that are verified as correct. Do you plan to collect such data? Wouldn't you agree that comparing with human reasoning (and not just the final label) would provide a deeper understanding of the quality of CoT in models?\n14) Do you plan to introduce asymmetric penalties or weighted metrics that consider the context of use?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1) MME-Emotion is the first holistic benchmark to evaluate both the presence of an emotion and the reason for it, offering a novel method that goes beyond mere classification. The multi-agent automated evaluation is a creative solution to the absence of annotated reasoning chains.\n2) The large-scale benchmark comprises 6,500 clips, 27 scenarios and eight tasks, and is balanced in terms of duration and question distribution. Human validation of the automated scoring adds credibility.\n3) The paper is clear and precise, with transparent reporting of model performance and precise definitions of metrics.\n4) By exposing the limitations of current MLLMs, even the top models achieving a score of less than 40% in recognition, the work sets out a clear research agenda. It also shows that, with targeted post-training, specialist models can rival generalists, offering practical guidance for future development.", "weaknesses": "1) Although the article acknowledges that specialized models (e.g. Audio-Reasoner) outperform their multimodal counterparts, it does not provide a systematic analysis of the contribution of individual modalities. Ablation experiments (e.g. running the same MLLM in audio-only, video-only and audio+video modes) could reveal whether the problem lies in the modalities' integration being inefficient or in noise/conflict between the modalities. I would like to see more detail on this, either in the form of detailed experiments or a detailed discussion supported by evidence.\n2) Using a set list of emotions in prompts simplifies the task, but may inflate metrics. In real-world scenarios, users do not provide such a list, so the model must generate an open-ended response. Am I correct in understanding that this reduces the applicability of the results to practical tasks, and is therefore a limitation? Or is this an unavoidable fact, and is there no other way to achieve the obtained metrics?\n3) Although the authors acknowledge that the data is multilingual, they do not stratify by language or culture. Emotional expressions and norms can vary greatly between cultures (e.g. East Asian and Western), so ignoring this factor could lead to biased conclusions about the 'universal' emotional competence of models. This is worth mentioning in more detail to avoid any misunderstandings.\n4) Although the average clip length is specified as being greater than 3.3 seconds, there has been no investigation into how performance scales with clip length or temporal complexity (e.g. a change in emotion occurring midway through a clip). This is important for understanding the extent to which models are capable of temporal reasoning, which is one of the key aspects of real emotional perception. More detailed explanations of this should be added.\n5) Am I correct in understanding that the Reasoning Score metric assesses the accuracy of each step, but does not analyze common error patterns? What happens, for example, if the audio context is ignored, microexpressions are misinterpreted, or emotions are mixed up? Can this metric evaluate all these errors and others at once?\n6) From the paper, it seems that the authors use test splits of the original corpus. This may mean that the wording of the questions and candidate labels indirectly reveals information about the data distribution, particularly if the prompts are based on the original annotations. How confident are the authors that this is not happening, and how can this be assessed?\n7) Although there is a high level of agreement between the GPT assessment and the experts (0.953), the experts only evaluated 373 reasoning steps. If a more extensive verification is performed, especially on complex or borderline cases, how can we be sure that the assessment will not be lower and that it is not currently being adjusted for greater consistency?\n8) The CoT-S formula uses a fixed value and does not involve sensitivity analysis. Different tasks may require different balances between recognition and reasoning. For instance, accuracy is more important in clinical diagnostics, whereas explainability is more important in education. So why is the value fixed rather than offering adaptive adjustment?\n9) Table 2 shows 'Avg Token' and 'Avg Step', but how are these metrics analyzed in relation to quality? For example, how effective is a token per point?\n10) Although 'balancing' is mentioned, it is unclear exactly how the even distribution of emotions, scenarios and complexity was achieved. What was the age and gender distribution? How does this affect the final distribution? More details on this matter are needed; otherwise, there will still be many doubts about the samples and how they are used.\n11) There are few details on exactly how noise affects reasoning quality, rather than just final predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X0Llus1d4y", "reviewer": "Reviewer_2ztj", "summary": "This paper introduces MME-Emotion, a new benchmark designed to holistically evaluate the emotional intelligence of MLLMs. The authors argue that existing benchmarks are limited in scope and fail to assess the reasoning capabilities behind emotional state identification. To address this, MME-Emotion provides a large-scale dataset of over 6,000 curated video clips, organized into eight distinct emotional tasks across 27 different scenarios. A key contribution is the proposed holistic evaluation suite, which employs a multi-agent framework to automatically assess model performance. This framework uses a GPT-based judge to score models on three unified metrics: Recognition Score (Rec-S) for accuracy, Reasoning Score (Rea-S) for the quality of step-by-step logic, and a combined Chain-of-Thought Score (CoT-S). The authors conduct a rigorous evaluation of 20 advanced MLLMs, revealing that current models still possess unsatisfactory emotional intelligence. The paper concludes with several key insights, such as the comparable performance of generalist and specialist models and the correlation between deeper reasoning and better performance, paving the way for future research in affective computing.", "questions": "1. Could you elaborate on the conceptual distinctions that make your eight task categories fundamentally different \"tasks\" rather than different \"scenarios\" of a single emotion classification task?\n2. The performance of models like AffectGPT and Qwen2.5-Omni is unexpectedly low. To clarify these results, could you provide details on your model replication and the benchmark's label quality (e.g., inter-annotator agreement)?\n3. Have you performed an ablation study comparing model performance with and without the CoT instruction to justify its necessity? Additionally, how do you ensure fair evaluation for models designed for direct answers that are penalized by the CoT-S metric?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-motivated by identifying an existing gap in the evaluation of emotional intelligence for MLLMs. To address this, it introduces the MME-Emotion benchmark, which includes over 6,000 videos and its structured organization across 8 tasks and 27 scenarios, expanding the scope of previously available datasets in this area.\n\n2. The paper proposes a holistic, automated evaluation suite. A key feature of this framework is its use of a multi-agent, MLLM-as-judge system to assess performance. This approach extends beyond simple recognition accuracy by incorporating metrics designed to measure reasoning quality (Rea-S and CoT-S), offering a multi-faceted view of model capabilities.\n\n3. The paper conducts a large-scale empirical study evaluating the performance of 20 different MLLMs on the proposed benchmark. This comparison provides a broad overview of current model performance on these emotional intelligence tasks. The results highlight the challenging nature of the benchmark for existing models and document performance differences between generalist and specialist approaches.", "weaknesses": "1. The benchmark is framed as \"multi-task,\" but the eight defined tasks (e.g., ER-Lab, ER-Wild, Noise-ER, SA, FG-SA) appear to be variants of a single core task: emotion classification. They primarily differ by scenario (e.g., in-the-lab vs. in-the-wild) or label granularity rather than representing fundamentally distinct tasks. Consequently, the analysis in Section 4, which shows performance differences, highlights varying difficulty levels but fails to provide a deeper analysis of why models perform differently across these closely related settings.\n2. A significant concern arises from the reported results in Table 2. Highly capable specialist models like AffectGPT (11.9% Rec-S) and advanced omni-modal models like Qwen2.5-Omni (17.4% Rec-S) achieve scores that are barely above random guessing. This performance is inconsistent with the capabilities demonstrated in their original papers. This discrepancy raises critical questions about the experimental methodology, suggesting potential issues with either the model replication or, more fundamentally, the label quality and accuracy of the MME-Emotion dataset itself. Such poor performance from top models risks undermining the benchmark's validity.\n3. The evaluation framework is heavily centered on CoT reasoning. However, the paper does not provide an ablation study to justify that CoT is necessary or even beneficial for emotion recognition, a task that often relies more on perception than complex reasoning. This approach also introduces a clear bias against models not optimized for step-by-step output. As seen in Table 2, models like HumanOmni and Emotion-LLaMA produce very short outputs (Avg Token = 1.3 and 2.3) and have near-zero Reasoning Scores (Rea-S = 0.3 and 0.4). Evaluating them with the CoT-S metric, which is heavily weighted by Rea-S, is unfair and does not accurately measure their ability to perform direct emotion recognition.\n4. The paper contains some inaccuracies regarding the capabilities of the models evaluated. For instance, Table 2 indicates that the Gemini-2.5-Pro cannot process audio (the 'A' column is not checked). This is factually incorrect, as it is natively capable of processing audio from video inputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)