EmotionHallucer: Evaluating Emotion Hallucinations in Multimodal Large Language Models
Authors: Bohao Xing
Xin Liu
Guoying Zhao
Chengyu Liu
Xiaolan Fu
Heikki Kälviäinen
Venue: ICLR 2026 Poster (2026)
Keywords: Emotion Hallucination
Emotion Understanding
Affective Computing
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Emotion understanding is a critical yet challenging task. 
Recent advances in Multimodal Large Language Models (MLLMs) have significantly enhanced their capabilities in this area. However, MLLMs often suffer from ``hallucinations'', generating irrelevant or nonsensical content.
To the best of our knowledge, and despite the importance of this issue, there has been no dedicated effort to evaluate emotion-related hallucinations in MLLMs.
In this work, we introduce \textbf{EmotionHallucer}, the first benchmark for detecting and analyzing emotion hallucinations in MLLMs. 
Unlike humans, whose emotion understanding stems from the interplay of biology and social learning, MLLMs rely solely on data-driven learning and lack innate emotional instincts. 
Fortunately, emotion psychology provides a solid foundation of knowledge about human emotions.
Building on this knowledge, we assess emotion hallucinations from two perspectives: emotion psychology knowledge and realworld multimodal perception. 
To support robust evaluation, we utilize an adversarial binary question–answer (QA) framework, which employs carefully crafted basic and hallucinated pairs to assess the emotion hallucination tendencies of MLLMs.
By evaluating 41 LLMs and MLLMs on EmotionHallucer, we find that:
(1) most current models exhibit substantial issues with emotion hallucinations;
(2) closed-source models outperform open-source models in detecting emotion hallucinations, and reasoning capability provides additional advantages;
and (3) existing models perform better in emotion psychology knowledge than in multimodal emotion perception.
As a byproduct, these findings inspire us to propose the \textbf{PEP-MEK} framework, which yields an average improvement of 9.90\% in emotion hallucination detection across selected models.
Resources will be available on GitHub.

Published as a conference paper at ICLR 2026
Emotion EVALUATING EMOTION HALLUCI- NATIONS IN MULTIMODAL LARGE LANGUAGE MOD-
ELS
Bohao Xing
Xin Liu
Lappeenranta-L.at University of Technology LUT Finland
Lappeenranta- University of Technology LUT Finland
bohao. xing@lut.......
linuxsino aamol@mmaal..
Guoying Zhao
Chengyu Liu
Xiaolan Fu
ELLIS Institute Finland University of Oulu, Finland guoying. zhao@oulu.
Southeast University China chengyuβseu. edu. cn
Shanghai Jiao Tong University China
fuxiaol ands jtu. edu. cn.
Heikki Kälviäinen
Lappeenranta-Lae Lahti University of Technology LUT Finland
Brno University of Technology
Czech Republic
heikki. waw.kkaavviiiine fi
ABSTRACT
Emotion understanding is critical yet challenging task. Recent advances in Mul- timodal Large Language Models (MLLMs) have significantly enhanced their ca- pabilities in this area. However, MILMS often suffer from "hallucinations" gen- erating irrelevant or nonsensical content. To the best our knowledge, and de- spite the importance of this issue, there has been dedicated effort to evaluate emotion-related hallucinations in MLLMs. In this work, we introduce Emotion- Hallucer, the first benchmark for detecting and analyzing emotion hallucinations in MLLMs. Unlike humans, whose emotion understanding stems from the inter- play of biology and social learning, MLLMs rely solely on data-driven learning and lack innate emotional instincts. Fortunately, emotion psychology provides a solid foundation of knowledge about human emotions. Building on this knowl- edge, we assess emotion hallucinations from two perspectives: emotion psychol- ogy knowledge and real world multimodal perception. To support robust evalu- ation, we utilize adversarial binary question-answer (QA) framework, which employs carefully crafted basic and hallucinated pairs to assess the emotion hal- lucination tendencies of MLLMs. By evaluating 41 LLMs and MILMs Emo- HionHallucer, we find that: (1) most current models exhibit substantial issues with emotion hallucinations: (2) closed-source models outperform open-source mod- els in detecting emotion ballucinations, and reasoning capability provides addi- tional advantages; and (3) existing models perform better emotion psychology knowledge than in multimodal emotion perception. As a byproduct, these find- ings inspire us to propose the PEP-MEK framework, which yields an average improvement of 9 9.90% in emotion hallucination detection across selected models.
Inevitably, emotions inseparable from the idea of good and evil.
Antonio Damasio, The Feeling of What Happens.
"Corresponding author.
Code and benchmark: https://gi com wtten//xxars///noooo http:///fthhhoocccc/////rrmmm/or lucer

---

Published as a conference paper at ICLR 2026
Culture Rule
cognitive appraisals
feelings
Audio
Image
Racsering Cun
fcleodty
Company
Rauoor by Pood
Multmodaiity
Perception
physiological
behaviors
changes
Emotion
Knowledge
challenge
Thurry
Video
Dutittionn
Tod
Input
Output
Input
Verbal
Nonverbal Cues
(a)
(b)
Figure I: Emotion understanding differences and the EmotionHallucer. (a) The difference between how humans and MLLMs understand emotions. Based the component process model (Scherer) 2009) and a dynamic systems approach (Lewis) 2005). human emotion understanding involves dy- namic interactions among cognitive appraisals, physiological changes, feelings, and behaviors. In contrast, MLLMs rely on data-driven learning from external behavioral cues, which limits their abil- ity to accurately infer the underlying emotional states. (b) EmotionHalluucer is organized along two main dimensions, Emotion Knowledge and Multimodality Perception, and includes seven subcate- gories across four modalities.
INTRODUCTION
Emotion understanding is of the most fundamental yet challenging tasks in Al (Koelstral et al. 2011] Hakak et al. 2017). and has attracted significant attention from the research com- munity (Nandwani and Verma 202 Li and Deng 2020 El Ayadi al. 2011 Ezzameli and Ma- hersia 2023 Rahdari et al. 2019 Xing et al.) 2024). Much of the existing work has focused on independent sub-tasks across different modalities: the text modality. tasks such sentiment anal- ysis (Wankhade et al. 2022) and emotion cause detection (Lee et al. 2010): in the image modality, facial expression recognition (Li and Deng
2020 Yuan al. 2024 Liu et al. 2024c Hu al. 2025) and affective scene analysis (Zhao et al.
2021 in the speech modality, speech emotion recogni-
tion (Wani et al. 2021) and in the video modality, multimodal emotion recognition (Abdullah et al.,
2021). dynamic facial expression recognition (Zhao and Liu 202I), and body gesture-based emotion recognition (Liu et al. 2021). among others.
Recently, MLLMs have demonstrated remarkable capabilities in textual and visual understand- ing (Alayrac et al. [022 AAchiam et al. (202)). and have begun to play an increasingly important role |022|Achiam
in emotion understanding (Zhang et 2023h Lian et al.) 2023b 2025 Yuan al. 2025 Cheng jet al., [204]] Xing et al. 2024). Nevertheless, despite their advanced capabilities, MLLMS often
generate incorrect ungrounded responses when operating based on textual or visual inputs (Li et al. 2023b Tong et al. 2024; Petryk et al. (2024). This issue of providing misleading information commonly termed "hallucination" (Rohrbach al. 2018). Hallucination is generally categorized into two types (Bai et al., 2024): (1) factuality hallucination. where outputs conflict with real-world facts; and (2) faithfulness hallucination, where outputs diverge from input instructions, or provided context, or exhibit internal inconsistencies. In response to these challenges, increasing emphasis MILLMs. However, existing halluci-
has been paid to analyzing and mitigating hallucinations
nation benchmarks are designed primarily for general-purposs tasks (Wang et al. 2024). leaving hallucinations emotion understanding tasks largely unexplored.
Human emotion arises from combination of innate biological mechanisms and lifelong social learning (Zeidner et al., 2003). which makes emotions challenging to model, as shown in Figure la Unlike humans, MLLMs Ms rely data-driven learning and they lack the embodied and experiential grounding that humans use to interpret emotions naturally and intuitively. Fortunately, learning how emotion develops help understand more about emotion itself (Shiota and Kalat 2017
2

---

Published as a conference paper at ICLR 2026
Moreover, decades of research in psychology have offered rich insights into how humans perceive, process, and reason about emotions (Niedenthal and Ric 2017). offering a valuable source of knowl- edge support more reliable emotion understanding in MLLMs.
Motivated by these observations, we first consider how emotion-related hallucinations should be defined and categorized. Unlike general hallucinations, emotion hallucinations tend to be more complex, as emotion understanding involves not only objective perception but also psychological and sociological reasoning (Zaki and Ochsner 2012). In view of this complexity, we propose Emo- tionHallucens the first benchmark specifically designed to evaluate emotion hallucinations. Emo- tionHallucer targets two key aspects: hallucinations related knowledge of emotion psychology (focusing on factuality hallucination) and hallucinations in real-world multimodal emotion under- standing (focusing on faithfulness hallucination). as illustrated in Figure Tb To ensure reliable evaluation and reduce confounding factors (Li et al. 2023b; Zhhng et al. 2023a), we adopt binary QA based evaluation framework (Li et al. 2023b; Wang et al. 2024). Specifically, we construct adversarial QA pairs (Tong et al. 2024), where each pair consists of basic question and inten- tionally hallucinated question to test the models. To mitigate language bias, we balance "yes" and "no" answers, and provide concise explanations to reduce misinterpretation.
By evaluating 41 LLMs and MLLMs on EmotionHallucerr our analysis produces three main find- ings: first, most current models exhibit substantial issues with emotion hallucinations; second, closed-source models outperform open-source ones in detecting emotion hallucinations, and reason- ing capability provides additional advantages; and third, existing models perform better in emotion psychology knowledge than in multimodal emotion perception. Building on these findings, we pro- pose PEP-MEK, a plug-and-play framework that incorporates both modality- and emotional knowledge to mitigate emotion hallucinations. Experimental results show that applying PEP-MEK leads a significant performance improvement in results on EmotionHallucerr with an average improvement of 9 9.90%. We believe this framework can support future research and development in the detection and mitigation of emotion hallucinations in MILMSs Our main contributions are summarized as follows:
We present, to the best of our knowledge, the first hallucination benchmark that evaluates emotion psychology knowledge and multimodal perception.
We conduct a comprehensive evaluation of 41 LLMs and MLLMs, from which we derive three key findings.
Building these insights, propose PEP-MEK, and demonstrate through experiments its effectiveness and potential in mitigating emotional hallucinations.
RELATED WORK
2.1 HALLUCINATION IN NATURAL LANGUAGE PROCESSING
Generative models in Natural Language Processing (NLP), particularly Large Language Models (LLMs), have demonstrated impressive performance across wide range of language generation tasks. However, a major challenge remains: these models may occasionally produce text that is inaccurate, irrelevant, or illogical, a phenomenon commonly referred to as "hallucination" in NLP. Hallucination typically refers to instances where the generated content is nonsensical or deviates from the intended meaning or source material (Filippoval 2020). In the NLP community, this issue empirically categorized into two types (Bai et al. 2024): (1) factuality hallucination, which high- lights inconsistencies between generated output and verifiable real-world facts, often manifesting as factual errors or fabrication; and (2) faithfulness hallucination, which refers to deviations from user instructions, input context, or internal consistency within the generated content. Within specialized research domains, opinions diverge regarding the value of factuality hallucinations. Some studies suggest that such hallucinations can be beneficial (Maynez et al. 2020 Thomson and Reiter 2020), arguing that the additional information they introduce may enhance the perceived informational value of the output.
3

---

Published as a conference paper at ICLR 2026
2.2 HALLUCINATION IN COMPUTER VISION
Recent advances in lision-language modeling have led to impressive performance across various generative tasks (Alarrac et al. 2022 Li et al. 2023a Achiam et al. 2023). Alongside these de- velopments, increasing attention has been given the issue of hallucination in this domain. The concept of object hallucination in image captioning, along with the CHAIR metric, was first intro- duced by Rohrbach al. 2018). To provide a more robust evaluation framework, POPE (Li al.) 2023b) proposed binary VQA benchmark specifically aimed at detecting object hallucinations, offering greater reliability than CHAIR. Subsequent studies have broadened the scope of hallucina- tion research to include relationships, attributes, counting, OCR, and other visual phenomena (Sun) et al. 2023 Wang et al. 2023a; Guan et al.] 2024 Cui et al. 2023 Chen et al., 2024 Liu et al, 2024b). More recently, research has extended to video-based hal hallucinations, reflecting the growing complexity of multimodal understanding (Zhang et al. 2024] Wang et al., 2024). However, to the best of our knowledge, and despite the emergence of general hallucination benchmarks, no dedicated benchmark has yet been developed to evaluate hallucinations related to emotion understanding. To fill this gap, we introduce the first benchmark specifically designed to assess emotion hallucinations.
2.3 EMOTION MLLMS
With the rapid advancement of L.LMs, growing body of research has begun to explore their po- tential for emotion understanding. These models facilitate the integration of multimodal informa- tion, making complex emotional reasoning increasingly feasible. Representative research in this direction includes work presenting the AffectGPT (Lian et al. 2025). EMER (Lian et al. b023b ( (Yang et al.
EmAM (Cheng et al. 2024), and
2025), which investigate
how MLLMs can be adapted for emotion recognition and reasoning. In parallel. other studies have focused on more Nomain--pecifet emotion understanding tasks (Li et al. 2024a Xing et al.) 2024] ei 2025), contributing to increasingly flexible and specialized frameworks. However, despite emotion understanding remains largely
these advancements, the critical issue of hallucination
underexplored. One key reason for this gap is the lack of dedicated benchmark assess halluci- nations in emotion understanding. In this work, we address this limitation by introducing the first benchmark specifically designed evaluate hallucination in emotion understanding.
THE Emotion BENCHMARK
3.1 BENCHMARK CONSTRUCTION
To evaluate hallucination in emotion understanding, divide our benchmark into two primary categories: emotion psychology knowledge and real-world multimodal emotion perception. This design yields seven specific evaluation settings spanning four modalities, illustrated in Figure 2 For the emotion knowledge dimension, we collected and curated factual statements from an author- itative textbook in emotion psychology (Shiota and Kalat 2017). For the multimodal perception dimension, we leveraged several widely used datasets across different modalities: SOUL (Deng al. (2023) for text, Twitter15 and Twitter17 (Zhang et al. 2018) for images. RAVDESS (Living- stone and Russo 2018) for speech, and MER 2023 (Lian et al. (023a) and Social-IQ 2.0 (Zadeh) jet al.. 2019 Wilf and collaborators) (2023) for video. These diverse sources allowed us to construct hallucination instances that reflect both knowledge-based and perceptual challenges in emotion un- derstanding. The detailed annotation procedure is described in Appendix A
3.1.1 EMOTION PSYCHOLOGY KNOWLEDGE HALLUCINATION
To the best of our knowledge, EmotionHallucer is the first hallucination benchmark that evalu- ates LLMs and MLLMs on their understanding of emotion psychology knowledge, with a focus on core theories from affective science (Scherer 2009 Lewis 2005). We began by selecting set of well-established and unambiguous statements from an authoritative textbook on emotion psy- chology (Shiota and Kalat 2017). which serve ground truth. Based these statements, constructed hallucinated counterparts that intentionally contradict, distort, or misrepresent the origi- nal content. This setting enables examination of the susceptibility of models hallucination within
4

---

Published as a conference paper at ICLR 2026
Emotion Knowledge Hallucination
Muitimodality Perception Hallucination
Theory Hallucination
Category Hallucination
Intensity Hallucination
Besic:
agri
emotional tookings best describd torms of continuaaa cinnenserrs, nather timn discrete catagariies Hallucinated: Verious dinrerational mocheds agrax lhad emotonnal ngs are best descrbed in tomss of uss calugooti nather than
Basic: You can hear noraal Intenally of sadress the person's speech.
woman the picture appears
Hallucinated: You can hear a strona Intonsitty of sadness the person's speech.
posilvee
Definition Hallucination
Hallucinated: waman
be
Basic: several more recent, widely recognized
Reasoning Cue Hallucination
cogriive appraisals, feelings,
Reasoning Result Hallucination
chysiologiaal gas, behaviors g amalions.
Review: The show had great episodes,
not one them not fer rill opiends it's just hardd follow up "The man that was dealh:" " "All through the house".
Hallucinated: several recent, recogrized deftnilions, coonttive feelings, physio ogical changes, and behaviors
and "Dig that cal, he's real gone "<br I--tor This
oplaeda is about couple that has just boon married
emotions
Peggr (Ammands Puummer and Char les (Stephen Shellen) in the first five minutess you find out that
Basic: The man amirica when
Finding Hallucination
"menttlist" bacause nostaigically raminds
Charles only mamried Paggy for heir money The two go on their moneymoon and their car bradks down on dirl
"olory days megician, eyes momentanity
Basic: People in Westem cultures tend to be righ
widen, quickly eyetrow
on individuallem, which emphasizee indicduud
road and they have seak rallage in an oid abandan
knowing linting memery slight suggest racollaction
uniquanss, pensonal rights, heing thae one's
mansion. Charles soon finds out a secret of Poggy's
and indegendence from others. Most South East cultures emphasize collectivimm Hallucinated: People couli Last
family... car jo-tbe join my cpinion you should watch this
oplsodo, but just don't expect the same foeling as the
Hallucinated:
smirks when referred nostalgically
rest of the opisodes in the first season.
him his "prory magiclann His momontarily
"mantalls""
high individual
Basic: This review a negative sentiment because it
empphssiros insividuual uniguensss personal rights,
hightights ti clear shortcomings in the episode compared ed
knowlngg smilie rinting The he lilts his head slightly and exheles lahily suggests amuusement fond recollaction.
salf, indepancienss being Most Wossaan collectiviam.
to the rest the season...
Hallucinated: This nevew is posttessettieen
because
Figure 2: Example tasks in EmotionHalluuer. Each pairs consists of basic question, used test the basic ability of MLLMs, and hallucinated question, containing hallucinated content to evaluate the model's ability detect hallucination. Emotion Knowledge Hallucination targets emotion psy- chology knowledge (Scherer [2009; Lewis] [0055] [Shiota and Kalat] (017). whereas Multimodality
Perception Hallucination centers on real-world emotion understanding (Deng et al. 2023, ZZhang
et al. 2018: Livingstone and Russo 2018 Lian et al. 2023a Zadeh et al. Wilf and collabo-
rators 2023).
More details can be found in Appendix D
the domain of emotion knowledge, providing a framework to assess both knowledge grounding and hallucination behaviors in the context of human emotion theory.
Theory. Our work introduces an Emotion Psychology Theory Hallucination setting in the context of emotion psychology theory (Cannon 1927| Scherer] 2009; Lewis, 2005), as illustrated in Fig-
ure To support this setting. collected set of core statements derived from well-established theoretical frameworks in emotion psychology, covering a range of foundational perspectives (Can- non 1927: Scherer 2009; Lewis 2005; Shiota and Kalat 2017). Based on these statements, and through an annotation process, we constructed 81 question-answer pairs that serve as test instances for evaluating hallucination in emotion theory knowledge.
We also constructed an Emotion Psychology Definition Hallucination setting, which tar- gets the definitions of widely accepted concepts in emotion psychology (Berkowitz) 1999), as shown in Figure 2 To support this setting, we collected a set of definitions for key emotion-related terms from authoritative academic sources, ensuring that each definition reflects the consensus within the field. This process resulted in 133 question- guestion-answer answer pairs, which serve as evaluation instances for assessing hallucinations related to definitional knowledge in affective science.
Definition.
Finding. We further constructed an Emotion Psychology Finding Hallucination setting, which fo- cuses on empirical findings in emotion research that describe observed phenomena but have not yet been formalized into complete theoretical frameworks (Kagitcibasi] 1997). as shown in Figure 2 This category includes well-documented results such as cross-cultural differences in emotional ex- pression (Hareli et al. 2015), developmental variations between infants and adults (Best et al., 2013), and other empirical observations reported in the literature. This setting yields 178 ineestion-nsswe pairs, which enable the evaluation of hallucination behaviors related to non-theoretical but empiri- cally grounded knowledge in emotion psychology.
3.1.2 MULTIMODALITY PERCEPTION HALLUCINATION
Beyond emotion psychology knowledge, we also constructed a Real-World Multimodal Emotion Perception Hallucination setting, which focuses on assessing hallucination risks in emotion un-
5

---

Published as a conference paper at ICLR 2026
derstanding tasks involving multimodal inputs (e.g., text, audio, and visual signals), as shown in
Figure 2
To build this setting, we collected samples from widely used emotion understanding
datasets (Deng et al. 2023 Zhang et al. 2018 Livingstone and Russo 2018 Lian et al.) 2023a
Zadeh et 2019 Wilf and collaborators, 2023p. We then filtered and refined these samples to con-
struct instances in which hallucinated descriptions contradict misinterpret the emotional content conveyed by the inputs. This setting enables the evaluation of hallucinations in realistic, multimodal environments, where emotion understanding requires integrating ambiguous and context-dependent cues from multiple sources (Zadeh et al. 2017).
Category. Following prior work on object hallucination (Rohrbach et al. 2018), and as shown in Figure 2 we introduced an Emotion Category Hallucination setting, which refers to the incorrect generation or identification of emotion categories (Dredzickis et al. 2020). Specifically, this type of hallucination refers to cases where models generate a nonexistent or inappropriate emotion cate- gory. Additionally, the emotion categories in this setting are not limited to binary sentiment labels commonly used in text-based tasks (e.g., positive/negative), but also cover discrete basic emotion categories frequently adopted in visual and multimodal emotion recognition tasks (e.g., happiness, anger, sadness).
Intensity. We further introduced an Emotion Intensity Hallucination setting, which concerns the misrepresentation of the strength of emotional expressions, shown in Figure 2 Unlike traditional approaches in affective computing that represent emotions using continuous valence-arousal dimen- sions (Russell 1980),, that are more aligned with the representational capabilities of LLMs. In this setting, hallucination arises when the model exaggerates, downplays, or otherwise inaccurately describes the intensity of the emotional state conveyed by the input.
adopted discrete intensity descriptors (e.g., mild/slightly, normal, strong)
Reasoning Result. Following previous works (Lian et al. 2023a Cheng et al., 2024j. we intro- duced a Reasoning Result Hallucination setting. This type of hallucination arises when the model accurately extracts emotional cues from multimodal inputs (e.g.. facial expressions, vocal tone, or text), but still produces an incorrect emotional interpretation, as shown in Figure 2 In this setting, the hallucination does not stem from the misidentification of the input signals themselves, but from incorrect emotional reasoning. This design reflects the critical distinction between perception and reasoning in emotion understanding, highlighting that accurate signal recognition does not neces- sarily guarantee appropriate emotional conclusions.
Reasoning Cue. Following previous works (Lian et al. 2023a; Cheng et al., 2024), we introduced a Reasoning Cue Hallucination setting. as shown in Figure In this setting, hallucination arises when the model overlooks, misinterprets, or fabricates key multimodal signals (e.g., missing the angry tone in speech, misreading a facial expression, text), leading to unreliable reasoning processes. Crucially, Reasoning Cue Hallucination captures cases where the reasoning pathway itself is flawed due to incorrect cue selection or interpretation, independent of the correctness of the final emotion judgment.
inferring unsupported emotional context from
3.2 BENCHMARK STATISTICS
Quantitative Analysis. Table I provides an overview of EmotionHallucer. In total, the benchmark
comprises 2,742 questions, with an average length of 31.6 words. EmotionHallucer covers four modalities: text, image, audio, and video. We further categorized the data collected from MER 2023IIian et al. (023a) and Social-IQ 2.0Zadeh et al. 2211 Wilf and collaborators 2023) into short video and long video subsets. More description can be found in Appendix(A.3
Qualitative Analysis. To provide a more intuitive understanding of EmotionHallucer, key terms are presented using word cloud, as shown in Figure 3 The visualization highlights frequently occurring concepts such as emotion, voice, facial expression, and tone, which correspond closely to the core modalities and reasoning components in our benchmark. These results suggest that EmotionHallucce effectively focuses on the multimodal cues essential for emotion hallucination.
3.3 EVALUATION METRIC
Hallucination Evaluation. Following previous work (Tong et al, 2024 Wang et al. 2024), we adopted Qa-bd benchmark for the following reasons: (1) Susceptibility to external factors.

---

Published as a conference paper at ICLR 2026
make experience
exhibie
furrowed brown Woman
Table 1: Dataset statis- tics of EmotionHallin Hallucer.
evleww
ee
SONEMER
behavior
person
expression
expression ssevvice
Statistic
Count
emot i
rather
suggest
speak
dituation
despite
film
C
Charactee
Ligtly
ma
rev
acterizz
jewer
negat
ellt
Live
zed
negative
inter
emor 1on
positiiv gentiment time
you
sontact outtre
senselue way
Sent immet
posture
Avg Question lea Koowlolge lon
Videos
Quesrioes
2,742
31.6
230
volce
headinges
feel
stall
Ragce
tone
darra
naking
tanird
1ood
Aug
wamar by Jee look
fee
Se woman exhibits
mattall
one
peop
le
indicate
Flach
Leat
Aug Image resolution
Audia lon
$79.5
4.2
466.5
mormooiie iie
Aug Vidzo
resolution
$70.0 476.9
Aug I long Video lon
60.0
Figure 3: Word cloud of EmotionHallucer.
Avg Long Viden resolation
637.9 360.0
Caption-based evaluations, like BLEU (Papineni et al. 2002) and ROUGE (Lin) 2004), are sensitive external variables such as prompt design and caption length (Li al. 2023b), which can distort evaluation outcomes. (2) Evaluation complexity. Existing approaches such as CHAIR (Robrbach) al,, 2018) rely on intricate, manually designed parsing rules, which complicate the evaluation process and hinder scalability. (3) LLM Hallucination Bias: Given the propensity of L.MMs to produce hallucinated outputs, using their own generations for self-evaluation may compromise the reliability and objectivity of the results (Wang al. 2023a).
To reduce these evaluation biases, EmotionHallucer utilizes an adversarial evaluation framework inspired by (Tong et al. 2024 Wang at. 2024). For each evaluation instance, we constructed a pair of complementary questions: a basic question, which tests the model's s core perception and reasoning abilities, and a hallucinated question, which introduces intentionally fabricated content to test the model's 's robustness against hallucination. A response is considered correct only if the model answers both questions accurately of whether a model can detect and resist hallucinations without compromising its performance on fundamental tasks.
pair. This dual-question design enables more rigorous assessment
In addition to the accuracy, we calculated the Yes Percentage Difference (Pet. Diff) and False Positive Ratio (FP Ratio) (Guan et al. 2024) Wang et al. 2024) reveal the bias in MLLMs. Specifically, the Yes Percentage Difference is calculated as
Bias Evaluation.
dy [Pred(m, 4) "yes"""es""peeee (mgg)v
(GT q) = "yes") (m.g)eV (m.s))
(1)
where V is the set of question pairs, represents additional modality information such image, audio, or video (for text-only questions, this component is absent), q refers the question itself, and
GT q) is the ground truth. A smaller dy indicates that the number of "yes" responses from mod- els is closer to the ground truth, revealing less language bias. The False Positive Ratio calculated
[Pred[mm, 9) "yes""mmmneen (m.
(2)
where W is the set of wrongly answered question pairs. Tfp gives the percentage of "yes" in all wrongly predicted answers. A value closer to 50% indicates less bias from the models.
4 EXPERIMENTS
In this section, we evaluate a range of widely used LLMs and MLLMs on EmotionHallucer. The models are grouped according to the input modalities they support. Implementation details and additional analyses are provided in Appendix C
4.1 MAIN BENCHMARK RESULTS
Multimodality. The EmotionHallucer benchmark encompasses four modalities: text, image, audio, and video. However, most existing MLLMs lack the ability to process all four modalities simul- tancously, with many limited to two modalities (e.g., text-image or text-video). To ensure a fair and consistent evaluation, report both aggregate results for models capable of handling all four
7

---

Published as a conference paper at ICLR 2026
Table Performance comparison on EmotionHallucer with additional "Yes/No bias" analysis.
Model
Yes/No Bias
Accuracy Emotion
Methods
Size
Pet. Diff (~0) FP Ratio (~0.55) Basic
Hallucinated f Overall f
Open-source
Qwen2.5-Omni Xn 2025
7B
-0.05
0.44 0.71
52.81
63,46 33,45
18.65
SonotionLLaaMee
2024
7B
0.20
72.88
15.43
Closed-source
Gemini-2.5-Flash
Goople DeapMind 2025
0.01
0.51 0.52
69.41 70.30
68.15 15 67.56
45.06
Gemini 2.5-Pro
Google DeepMind
2025
0.01
44.17
Table 3: Performance comparison on Emotion Noouuluu with additional "Yes/No bias" analysis. More results can be in Table 17 of the Appendix.
Model
Yes/No Bias
Accuracy EmotionFalltuer-Noo
Methods
Size
Pct. Diff (~00) FP Ratio (~0.55) Basic Hallucinated 1 Overall
Open-source
LLaVA Lin
2023
34B
+0.05
0.45
50.25
59.52
10.27
Llama3. 2-vision
Gratiafioii
11B
0.21
0.78
83.05
41.28
29,91
Video-ChatGPP
Maiz
7B
0.09
0.59
61.91
44.67
18.44
Amnotion-LLaMa.
(Chesg
2024
0.12
0.63
66.55
42.43
18.86
Qwen2.55 LL Bri ct
2025
72B
0.08
0.63
78.08
62.15
43.02
Qwen2.5 Omni Xu
2025
7B
0.11
0.65
72.39
49.74
25.44
Closed-source
QvQ-Max
Owea Team 2025
0.07
0,63
78.18
63.39
47.98
GPT-4o at
-0.01
0.48
67.10
69.49
40.98
GPT-5 OpenAl 20236
-0.06
0.40
67.10
78.17
49.35
Gemini-2.5. Jash
Google DeepMind
2025
0.06
0.61
78.55
66.80
50.56
Gemini-24 2.5-i Pro
Googl DespMini
X25
0.07
0.64
81.31
67.01
51.58
modalities and modality-speciffe results for models restricted certain subsets. Additional analyses on model performance and modality-specife analysis are provided in Appendix C.3
As shown in Table 2 we compare models across all modalities available on EmotionHallucer. While the strongest closed-source Gemini models outperform their open-source counterparts, their overall performance remains suboptimal, reflecting the inherent challenges of emotional understanding and hallucination. Notably, all open-source models fail to exceed the 25% accuracy expected from ran- dom guessing, also underscoring the current limitations of MILMS in handling emotional reasoning. In the "Yes/No Bias" evaluation, general- purposs MLLMs demonstrate better neutrality, showing less tendency toward overconfident affirmations. In contrast, the emotion-specific model Emotion- LLaMA performs significantly worse in this regard. This may be attributed its fine-tuning focus on emotional content. It may also be because the model is relatively outdated compared to recent MLLMs, potentially lacking the latest advances.
Given that most popular MLLMs are optimized for vision modalities and lack support for audio- only inputs, developed EmottionHallueer-NoAAde modality (see Table 3). Due to space constraints, we report results for 11 representative models. Consistent with earlier findings, closed-source models outperform their open-source counterparts in both overall accuracy and the "Yes/No Bias" evaluation. Gemini-2.5-PPy achieves the best perfor- mance, followed closely by Gemini-2.5-F Among open-source models, Qwen2.5 VI. performs the best and notably exceeds the random-guese guessing baseline.
benchmark subset that excludes the audio
Unimodality. To enable a more fine-grained understanding MLLM behavior across tasks and modalities, report performance each individual modality. As shown in Figure models per- form best on emotion knowledge, with accuracy decreasing steadily across perception- based tasks, from text to image, and further to audio and video. This trend may be attributed to two main fac- tors: (1) current training is predominantly focused on text data, enhancing models' performance on Knowledge-oriented tasks; and (2) lack of high-quality emotional annotations in modalities,
8

---

Published as a conference paper at ICLR 2026
Model
LLaVA-348
Overall Accuracy (%)
Llama32-visiio-1188
60
ChatGPP7
Emotion-LLaMA-78
Qwen2.5-VL-726
Qwen2.5-0mni
QvQ-max
30
GPT-4o
GPT-5
Gemini-2.5-Flash
Gemini-2.5-Pra
Knowledge Perception-T
Perception-l
Perception-A Perception-V/5 Perception-VIL
Task and Modality
Figure 4: Unimodal performance of partial selected models. T. A, v/s, and V/L stand for Text, Image, Audio, Short Video, and Long Video, respectively. Additional models and implementation details are provided in Appendix C
TE
Directly Predict
Extrestoo
Knoowklle
RMM
(a)
(b)
Figure 5: Emotion hallucination analysis and our proposed framework. (a) Comparison of halluci- nation detection in emotion knowledge and multimodal perception. (b) The PEP-MEK Framework.
which limits the models' ability to learn fine-grained emotion understanding. These results under- score a critical future direction: improving cross-modal emotion understanding through better data quality and balanced modality training. Additional detailed results and analysis are provided in Appendix
5 PREDICT-EEXPLANN-PREDDD WITH MODALITY AND EMOTION KNOWLEDGE
We observed that most models perform significantly worse under the multimodal perception hallu- cination setting compared to the emotion knowledge hallucination setting in EmotionHallucer. Mo- tivated by this observation, we propose simple yet effective explanation- method to mitigate hallucination issues arising from multimodal emotion perception errors.
5.1 EMOTION KNOWLEDGE VS. MULTIMODAL PERCEPTION HALLUCINATION DETECTION
Model performance on emotion knowledge and multimodal perception is shown Figure 5a A key observation is that most models achieve significantly lower accuracy on multimodal perception, highlighting a substantial gap compared to their performance on structured emotion knowledge. This result suggests that while current models handle structured knowledge reasonably well, they struggle with world emotion understanding. Accordingly, this section focuses on enhancing model robustness against multimodal perception hallucinations. Among the evaluated models, QvQ-

---

Published as a conference paper at ICLR 2026
Table Results the PEP-MEK Framework.
Model
Yes/No Bias
Accuracy on Emotion Hallucinated f Overall f
Methods
Size
Pet. Diff (~0) FP Ratio (~0.55)
Basic
Open-source
Qwen2.5-Omni
2025
7B
-0.18
0.30
35.51
71.96
10.49
-0.19
0.29
37.49
74.87
20.15
Emotion LLaMA eu 2025
0.28
0.77
77.34
22.02
9.65
+PEP-MEK
-0.08
0.41
46.35
62.86
26.03
flosed-soure
Gemini-2.5 2.5-Flash
Google DeepMind 2025
0.00
0.50
61.54
62.05
33.44
MEK
-0.07
0.40
56.66
71.17
37.84
Max outperforms Qwen2.5-Vi which attribute differences in its reasoning paradigms. Given that L.Mss are generally more adept at fact detection than hallucination detection (Ji et al. (023). we hypothesize that incorporating more structured and targeted reasoning framework, enriched with emotion-specifie knowledge, may further reduce emotion hallucination in MLLMs.
5.2 PEP-MEK FRAMEWORK
Building on these findings, we propose a novel framework, the Predict-Explain PreedictPrreictt with Modal- ity and Emotion Knowledge (PEP-MEK) framework, designed to enhance MLLMs' performance in detecting multimodal emotion hallucinations. Figure 55 illustrates the framework of the PEP-MEK. Rather than relying direct predictions, PEP-MEK introduces to improve model transparency and decision reliability in emotion understanding. First, PEP-MEK uses prompts to guide the model in autonomously extracting -oselity- and emotional knowl- edge from the input (text, image, audio, video). This extracted information then combined with the original input for an initial prediction. Next, the model generates an explanation for its prediction using both the input and the extracted knowledge. Then, the explanation is incorporated to refine the prediction. This process enables the model reason more effectively and reduce hallucinations.
intermediate reasoning stage
The effectiveness of PEP-MEK evaluated on three representative MLLMs with support for all modalities: general-purpont model (Qwen2.5-Omni) an emotion-specife model (Emotion- LLaMA), and a closed-source model (Gemini-2.55FFlaal is shown in Table 4 The results demon- strate that PEP-MEK consistently improves performance, yielding an average accuracy gain 9.9.9% across models. Notably, EmottonnLLaMM benefits the most from PEP-MEK, achieving 16.38% increase in overall accuracy and a substantial reduction in bias-relate metrics. Full implementation details, ablation study, and qualitative examples provided in Appendix
CONCLUSION
In this work, we introduced EmotionHallucer, the first benchmark designed to detect emotion hal- lucinations MLLMs. Our adversarial evaluation strategy provides a rigorous assessment both emotion understanding and hallucination susceptibility. By evaluating hallucinations from two com- plementary perspectives, emotion psychology knowledge and multimodal emotion perception, Emo- tionHallucer enables fine-grained analysis of large models. Testing of 41 LLMs and MLLMs vealed the pervasiveness of emotion hallucinations, especially in tasks requiring perception across modalities. To address these challenges, we propose PEP-MEK, framework that showed good po- tential for reducing emotion hallucinations and improving model robustness. These findings high- light critical limitations current emotion MILMSs and point promising directions for advancing emotion-awand aware, multimodal reasoning in future research.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
G. Zhao was supported by Research Council of Finland (former Academy of Finland) Academy Professor project EmotionAl (grants 336116, 345122, 359894), HPC project FaceCanvas (grant number 364905), the University of Oulu & Research Council of Finland Profi (grant 352788), and EU HORIZON- SERIOON--MCA-S-E--- project ACMod (grant 1011 130271).
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 techni- cal report. arXiv preprint arXiv:2412.08905 2024.
Sharmeen M Saleem Abdullah Abdullah, Siddeeq Y Ameen Ameen, Mohammed AM Sadeeq, and Subhi Zeebaree. Multimodal emotion recognition using deep learning. Journal of Applied Science and Technology Trends, 2(01))733-75 2021.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Man-BBapistt Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 1332335 2025.
Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. arXiv preprint 2rXiv:2404.1899 2024.
Leonard Berkowitz. Anger. 1999.
Catherine A Best, Hyungwook Yim, and Vladimir M Sloutsky. The cost of selective attention cat- egory learning: Developmental differences between adults and infants. Journal of Experimental Child Psychology, I 116(2):005-119, 2013.
Walter B Cannon. The james-lange theory of emotions: A critical examination and an alternative theory. The American Journal of Psychology, 1061/4) 1927.
Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, and Huajun Chen. Unified hallucination detection for multimodal large language models. arXiv preprint arXiv:2402.03190 2024.
Zebang Cheng. Zhi-Qi Cheng. Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng. and Alexander Hauptmann. Emotion-llamee Multimodal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Systems, 37:110805-110852 2024.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo. Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, al. Qwen2-audee technical report. arXiv preprint arXiv:2407.1075, 2024.
Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:2311.. 03288, 1..02887 2023.
11

---

Published as a conference paper at ICLR 2026
Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. Soul: Towards sentiment and opinion understanding of language. arXiv preprint SKKi:2311.119244 2023.
Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song,
Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:250411823, 2025.
Andrius Dzedzickis, Artüras Kaklauskas, and Vytautas Bucinskas. Human emotion recognition:
Review of sensors and methods. Sensors, 20(3).592, 2020.
Moataz Ayadi, Mohamed S Kamel, and Fakhri Karray. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition, 44(3):572-587, 2011.
K Ezzameli and H Mahersia. Emotion recognition from unimodal to multimodal analysis: A review. Information Fusion, page 101847, 2023.
Katja Filippova. Controlled hallucinations: Learning generate faithfully from noisy data. arXiv preprint arXiv:2010.05873, 2020.
Google
DeepMind. Gemini:
Our most capable model, getting better with think-
wtw.googglleeehhollllgggoooo gemini mooe thinkiin -hinkiigg-uptttesss marchm March 2025.
ing and coding.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The Ilama 3 herd of models. arXiv preprini arXi::2407.21788: 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: advanced diagnostic suite for tangled language hallucination and visual illusion large vision-language models. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375- 14385, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint urXiv:2501.12945 2025.
Nida Manzoor Hakak, Mohsin Mohd, Mahira Kirmani, and Mudasir Mohd. Emotion analy- sis: A survey. In International Conference Computer, Communications and Electronics (COMPTELIX), pages 397-402. IEEE, 2017.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26584-26595, 2024.
Shlomo Hareli, Konstantinos Kafetsios, and Ursula Hess. A cross-cultura study emotion expres- sion and the learning of social norms. Frontiers in Psychology, 6:1501 2015.
Zhuozhao Hu, Kaishen Yuan, Xin Liu, Zitong Yu, Yuan Zong Jingang Shi, Huanjing Yue, and Jingyu Yang. Feallm: Advancing facial emotion analysis in multimodal large language models with emotional synergy and reasoning. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 5677-5686, 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey hallucination in natural language generation. ACM Computing Surveys, 55(12):1-35 2023.
12

---

Published as a conference paper at ICLR 2026
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud. Marie-Amn Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. 2023. URL https:
//arxtt..rrg/ab//2210..0 0682
Albert Q QJiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.04088 2024.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao. and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700-1313 13710, 2024.
Cigdem Kagitcibasi. Individualism and collectivism. Handbook of Cross-cultral Psychology, 3: 1-49, 1997.
Mark L Knapp, Judith Hall, and Terrence Horgan. Nonverbal communication human inter- action. Thomson Wadsworth, 1972.
Sander Koelstra, Christian Muhl, Mohammad Soleymani, Jong-Seok Lee, Ashkan Yazdani, Touradj Ebrahimi, Thierry Pun, Anton Nijholt, and Ioannis Patras. Deap: A database for emotion analysis; using physiological signals. IEEE Transactions on Affectivv Computing. 3(1):18-31, 2011.
Michael Krumdick, Charles Lovering, Varshini Reddy, Seth Ebner, and Chris Tanner. No free labels: Limitations of llm-as-a-judge without human grounding. arXiv preprint arXiv:2503.05060 2025.
Sophia Yat Mei Lee, Ying Chen, and Chu- Chu-Ren uee Huang. text-driven rule-based system for emotion cause detection. In Proceedings of the NAACL HLT Workshop on Computational Approaches to Analysis and Generation of Emotion in Text, pages 45-53, 2010.
Marc Lewis. Bridging emotion theory and neurobiology through dynamic systems modeling. Behavioral and Brain Sciences, 28(2):169-194, 2005.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmente gener- ation for knowledger snwwledge-intesiiee nlp tasks. Advances in neural information processing systems, 33: 1459-9474 2020.
Deng Li, Xin Liu, Bohao Xing, Baiqiang Xia, Yuan Zong, Bihan Wen, and Heikki Kälviäinen. Eald-mlim: Emotion analysis in long-sequential and de- de-identity videos with multi-modal large language model. arXiv preprint arXiv:2405.005T 2024a.
Deng Li, Bohao Xing, Xin Liu, Baiqiang Xia, Bihan Wen, and Heikki Kälviäinen. Deemo: De- identity multimodal emotion recognition and reasoning. arXiv preprint wrXiv:254411944 2025.
Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. Lims-as-judges: comprehensive survey on lameb evaluation methods. arXiv preprint arXiv:2412.05572 2024b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2:
Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730-19742. PMLR, 2023a.
Shan Li and Weihong Deng. Deep facial expression recognition: A survey. IEEE Transactions Affective Computing, (3(3):1195-1221 2020.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024c.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vasion-language models. arXiv preprint arXiv:2305. 10355, 2023b.
13

---

Published as a conference paper at ICLR 2026
Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi semi-supervised learning. In Proceedings of the ACM International Conference Multimedia, pages 9610-9614 2023a.
Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, Shun Chen, Mingyu Xu, Ke Xu, Kang Chen, et al. Explainable multimodal emotion recognition. arXiv preprint 0rX1v:2306.15401, 2023b.
Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang. Hailiang Yao, et al. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark. arXiv preprini urXiv:2410.011
2024.
Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. arXiv preprint arXiv:2501. 2025.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXi:2331.10122, 2023.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches out, pages 74-81, 2004.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.194377 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892-3496,, 2023.
Jiazhen Liu, Yuhan Fu, Ruobing Xie, Runquan Xie, Xingwu Sun, Fengzong Lian, Zhanhui Kang, and Xirong Li. Phd: prompted visual hallucination evaluation dataset. arXiv e-prints, pages arXiv-2403, 2024b.
Xin Liu, Henglin Shi, Haoyu Chen, Zitong Yu, Xiaobai Li, and Guoying Zhao. imigue: An identity- free video dataset for micro-gesture understanding and emotion analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10631-10642, 2021.
Xin Liu, Kaishen Yuan, Xuesong Niu, Jingang Shi, Zitong Yu, Huanjing Yue, and Jingyu Yang- Multi-scale promoted self-adjusting correlation learning for facial action unit detection. IEEE Transactions on Affective Computing, 2024c.
Steven R Livingstone and Frank Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS One, 13(9)cc016339, 2018.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Vidco-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00666 2020.
Batja Mesquita and Nico H Frijda. Cultural variations in emotions: review. Psychological bulletin, 112(2):179, 1992.
Mistral AL. Introducing mistral-3.1: our new small model. mis iist ral smal-mmalll 1--3-1 2024.
Pansy Nandwani and Rupali Verma. A review on sentiment analysis and emotion detection from text. Social Network Analysis and Mining, 11(1):81, 2021.
Paula M Niedenthal and François Ric. Psychology of Emotion. Psychology Press, 2017.
14

---

Published as a conference paper at ICLR 2026
OpenAl. Gpt-4 turbo and the next frontier of intelligence.
httoS://gpenal..omm/nee.
gpt 4-1- November 2024. Accessed: 2025-04-30.
OpenAl. Introducing gpt-oss. 2025a.
OpenAI. 2025b.
Introducing gpt-5. atteer//peeaa..ccm//mee////ttttt
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311-318, 2002.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. arXiv preprint 220::2200..03886 2022.
Suzanne Petryk, David M Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph Gonzalez, and Trevor Darrell. Aloha: A new measure for ballucination in captioning models. arXiv preprint arXiv:2404.02904, 2024.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Ko- rhonen. Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv preprint arXiv:2005.00333, 2020.
Qwen. Latest advancement: Qwen blog post. https://qwen.ai/// froo772errc Latestt-ddvanncmentss--111- 2025. Accessed: 2025-09-22.
Qwen Team. Qwen A family of scalable language models. nttps://qwen.n.gihhiie /log/qwen3/1 2024.
Qwen Team. Preview of qwen-vl-72b: Advancing multimodal understanding. https:// gwenlm. gwenlm.github.r httpammmgg..ab...ooooooogggggoro
2025. Accessed: 2025-04-30.
Farhad Rahdari, Esmat Rashedi, and Mahdi Eftekhari. A multimodal emotion recognition system using facial landmark analysis. Iranian Journal of Science and Technology. Transactions of Elec- trical Engineering, 43:171-189, 2019.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination image captioning. arXiv preprint 1iXi::8809.021560 2018
James Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39 (6):1161, 1980.
Klaus R Scherer. Vocal communication of emotion: A review of research paradigms. Speech communication, 40(1-2):221 (0(1-2):227-256, 2003.
Klaus R Scherer. The dynamic architecture of emotion: Evidence for the component process model.
Cognition and Emotion, 2377):1307-135) 2009.
Michelle N. Shiota and James Kalat. Emotion. Oxford University Press, New York, NY, edition, 2017. ISBN 978-0190635510.
Craig A Smith. Dimensions of appraisal and physiological response in emotion. Journal of person- ality and social psychology, 56(3):339, 1989.
Zhiqing Sun, Sheng Shen. Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:23099 /4525, 2023.
Gemma Team, Aishwarya Kamath. Johan Ferret. Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
15

---

Published as a conference paper at ICLR 2026
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025, URL htttp:://wen.mm. github. ht/bloq//oe ht/blog/qwg--3333333 -wo
Craig Thomson and Ehud Reiter. A gold standard methodology for evaluating accuracy in data-to- text systems, arXiv preprint 1rXiv:2011.03922. 2020.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. Pruceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 9568-9578, 2024.
Junyang Wang. Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, et al. Amber: An llm-free multi multi-dimensional benchmark for milms hallucination evaluation. arXiv preprint arXiv:231 1.07397, 2023a.
Peiyi Wang, Lei Li. Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17920 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, coniisenssy improves chain of thought reasoning in language models. arXiv preprint arXhv:2203.11171 2022.
Yuxuan Wang, Yueqian Wang, Dongyan Zhao, Cihang Xie, and Zilong Zheng. Videohallucer: Evaluating intrinsic and extrinsic hallucinations in large video-language models. arXiv preprint 1rXiv:2406.16338, 2024.
Taiba Majid Wani, Teddy Surya Gunawan, Syed Asif Ahmad Qadri, Mira Kartiwi, and Eliathamby Ambikairajah. A comprehensive review of speech emotion recognition systems. IEEE Access, 9: 47795-47814, 2021.
Mayur Wankhade, Annavarapu Chandra Sekhara Rao, and Chaitanya Kulkarni. A survey on senti- ment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7):5731- 5780, 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in 2022.
neural information processing systems,
Adam B. Wilf and collaborators. Social-iq 2.0 challenge.
www.tttb.cooooocceetoooommootttt abwallwwii
Social-I0-- 0- Challhhallgeee 2023.
Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huanjing Yue, Jingyu Yang, and Heikki Kälviäinen. Emo-llama: Enhancing facial emotion understanding with instruc- tion tuning. arXiv preprint arXiv:2408. 11424, 2024.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.2000 202.1 2025.
An Yang, Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. 2024a. URL hittts://aaxiin or wtgg/bb..... / 2407 10671
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1555 2024b.
16

---

Published as a conference paper at ICLR 2026
Qize Yang, Detao Bai, Yi-Xing Peng. and Xihan Wei. Omni-emotion: Extending video mllm with detailed face and audio modeling for multimodal emotion analysis. arXiv preprint arXiv:2501.09502, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. Science China Information Sciences, 67(11))220105, 2024.
Kaishen Yuan, Zitong Yu, Xin Liu, Weicheng Xie, Huanjing Yue, and Jingyu Yang. Auformer: Vision transformers are parameter--fficient facial action unit detectors. In European Conference on Computer Vision, pages 427-445. Springer, 2024.
Kaishen Yuan, Yuting Zhang, Shang Gao, Yijie Zhu, Wenshuo Chen, and Yutao Yue. Coemogen: Towards semantically-cofee cohherent and scalable emotional image content generation. arXiv preprint arXiv:2508.03533 2025.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250 2017.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Moreney. Social- iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8807-8817, 2019.
Jamil Zaki and Kevin N Ochsner. The neuroscience of empathy: progress, pitfalls and promise.
Nature Neuroscience, 15(5):675-680, 2012.
Moshe Zeidner, Gerald Matthews, Richard D Roberts, and Carolyn MacCann. Development of emotional intelligence: Towards a multi-level investment model. Human Development, 46(2-3); 69-96, 2003.
Jiacheng Zhang, Yang Jiao, Shaoxiang Chen, Na Zhao, Zhiyu Tan, Hao Li, and Jingjing Chen. Eventhallusion: Diagmosing event hallucinations in video llms. arXiv preprint (rXiv:2409.16397 2024.
Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A Smith. How language model hallucinations can snowball. arXiv preprint arXiv:2305. 13534, 2023a.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Yazhou Zhang, Mengyao Wang, Youxi Wu, Prayag Tiwari, Qiuchi Li, Benyou Wang, and Jing Qin. Dialoguellm: Context and emotion knowledge-tunee sunde large language models for emotion recognition in conversations. arXiv preprint arXiv:2310.11 1374, 2023b.
Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Bjoern W Schuller, and Kurt Keutzer. Affective image content analysis: Two decades review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):6729- 6751, 2021.
Zengqun Zhao and Qingshan Liu. Fofr Dynamic facial expression recognition transformer. In Proceedings of the ACM International Conference Multimedia, pages 1553-1561, 2021.
17

---

Published as a conference paper at ICLR 2026
APPENDIX
To facilitate deeper understanding and reproducibility, we organize the appendix as follows:
Appendix A presents additional information on benchmark construction and annotation
procedures, and statistics.
Appendix details the implementation and ablation study of PEP-MEK.
Appendix includes experiment setup and additional experimental results along with in- depth mailysi..
Appendix provides more examples.
Appendix discusses the limitations our work.
Appendix discusses the directions for future improvement.
Appendix is the Ethics Statement.
Appendix is the Reproducibility Statement.
Appendix the L.M Usage Statement.
A BENCHMARK COLLECTION, ANNOTATION, AND STATISTICS
Here, we provide annotation details to better illustrate our data sources as well each modality and category. The annotation process begins with the collection of knowledge from emotion psychology and existing emotion understanding datasets. This foundational step ensures that our annotation grounded in well-established emotional concepts and tasks. Next, we engage group of trained annotators to carefully filter and review the initial QA. Their task to ensure that each QA pair accurate, coherent, and well-grouu cell-grounded in the corresponding source content. Following this, for each basic question, annotators are instructed to generate a corresponding hallucinated question. To ensure high annotation quality, adopt cross-review verification mechanism. Each question (ba- sic and hallucinated) is independently reviewed by second annotator. Only when both annotators reach consensus is the item included in the final benchmark. The overall annotation workflow is shown in Figure
Data Source Selection
Basic QA
Cross Review
Hallucinated QA Construction
Cross Review
Sample Filtering
Construction
Figure 6: Overview of the annotation workflow. The process consists of four key stages: selecting relevant data sources, filtering for appropriate samples, constructing basic QA pairs, and generating hallucinated QA variants by applying diverse hallucination strategies. Each QA construction step is followed by cross-review phase ensure quality and consistency across annotators. This unified pipeline is applied across all modalities, including text, image, audio, and video.
A.
COLLECTION AND ANNOTATION ACROSS MODALITIES
We first present our data collection and annotation procedures from a modality perspective to provide clearer understanding of our data sources and the original objectives of each dataset.
A.1.1
TEXT
To ground our annotation process in established affective science, we selected leading textbook in the field emotion psychology: Emotion (3rd Edition) by Michelle N. Shiota and James W.
Kalat (Shiota and Kalat 2017). This book offers comprehensive overview of key emotional the-
ories (Cannon 1927
Scherer 2009; Lewis (005), physiological and cognitive underpinnings of
emotions (Berkowitz
1999,, well cross-cultural perspectives (Kagiteibasi 1997; Hareli al.
18

---

Published as a conference paper at ICLR 2026
2013 Best et al. 2013). It is widely used in academic settings and is recognized for its balanced integration of classical theories (c.g., James-Lange (Cannon 1927). Schachter-Singen and con- temporary research findings. The text also emphasizes real-world applications and experimental paradigms (Scherer) 2009, Lewis 2005), making it a suitable foundation for constructing psycho- logically grounded emotion-related tasks. We collected subset of statements from the book that conveyed clear emotion psychology viewpoints and insights. These statements were carefully se- lected through manual review to ensure their clarity and relevance. For each selected statement, the basic question was taken directly from the original text, preserving the emotional concept or psycho- logical mechanism as presented in the source. After the basic QA pair was established, annotators were instructed to generate a hallucinated version. Common strategies included addition, deletion, or distortion of key concepts or the intension and extension of a given inference, while ensuring that the question remained grammatically fluent and superficially plausible. As a result, 784 basic and hallucination QA pairs were generated.
In the real-world text modality emotion understanding, we incorporate existing datasets to better examine emotion perception hallucinations in real-world textual contexts. One such dataset is SOUL (Deng al., 2023), which is specifically designed to evaluate fine-grained understanding of sentiment, stance, and opinion in natural language. In its original task format, SOUL presents review text alongside multiple statements that reflect potential interpretations of the opinion ex- pressed. Each statement is annotated as either correct or incorrect, depending on its emotional alignment with respect to the original review. This setup goes beyond traditional sentiment analysis by emphasizing deeper dimensions of subjective understanding, such as emotional appropriateness, implicit attitude, and contextual nuance. In our annotation process, for each review text, we col- lected all associated statements and manually refined them into a single, well-grounded summary statement. This statement was used as the basic question in our QA format. This approach allowed deeper dimensions of subjective understanding, thus
our QA pairs to inherit SOUL's emphasis enabling more fine-grained evaluation of MLLMs' capabilities as regards emotion understanding and hallucination detection. As result, 200 basic and hallucination QA pairs were generated.
A.1.2 IMAGE
We utilized the 15itter15 and 1 datasets (Zhang al. 2018), which were originally devel- oped for named entity recognition and sentiment analysis in social media contexts. These datasets consist of tweets accompanied by images and structured annotations, including entity labels and overall tweet-level sentiment polarity (positive, negative, neutral). The data reflects real-world, noisy, and multimodal content, making a valuable resource for emotion perception tasks.
In our annotation process, we focused on images. Notably, we observed that many images con- tain multiple people, often displaying divergent emotional expressions within the same instance, as shown in Figure TO Thii introduces unique challenge for MI.LMs, as it requires fine grained emotion understanding and the ability to disambiguate multi-actor sentiment based on visual cues. To reflect this, reformulated each image into QA format, selecting or synthesizing emotionally consistent description as basic questions, and constructing hallucinated variants by introducing emo- tionally incongruent elements, such misattributing emotions to the wrong person exaggerating emotional intensity. As a result, we generated 300 basic and hallucination QA pairs and 150 images.
A.1.3 AUDIO
We incorporated the RAVDESS dataset (Livingstone and Russo 2018), which provides multi- modal collection of emotional expressions through both facial and vocal modalities. The dataset consists of high-quality recordings in North American English, featuring actors performing scripted speech and song with controlled emotional expressions. Emotions span wide range (including calm, happy, sad, angry, fearful and more), and are further labeled with varying levels of intensity, making RAVDESS a rich resource for studying graded emotional cues.
In our annotation process, used only the audio modality as input, focusing how emotion is conveyed through vocal tone and prosody rather than linguistic content. For each selected audio clip, annotators generated basic question that captured the emotional tone perceived from the speech alone, based on the original emotion labels provided in the dataset. Hallucinated variants were then constructed by subtly misrepresenting the emotional valence intensity, such as exaggerating.
19

---

Published as a conference paper at ICLR 2026
downplaying. inverting the expressed emotion, allowing us assess the sensitivity of MLLMs to audio signals without relying on semantic content. As a result, we generated 736 basic and hallucination QA pairs and 368 audio clips.
A.1.4 VIDEO
Videos contain both visual and auditory information, and they represent the most natural and com- prehensive modality for capturing human emotional experiences. Motivated by this, we selected two data sources, MER 202nLian al. (2023a) and Social-IQ 2.CZadeh et al. (2019); Wilf and col-j laborators 2023), to evaluate emotion understanding hallucination in realistic, multimodal contexts. Both datasets involve video-based social situations, where emotion understanding relies not only on speech and text, but also on facial expressions, gestures, and interpersonal dynamics.
MER 2023 focuses on multimodal emotion recognition on sentence-length inputs. Each instance consists of a short video clip accompanied by aligned textual transcripts, from which the model is expected to identify the expressed emotion based on the combined audiovisual and textual cues. emphasizes how emotions are conveyed through brief, but richly multimodal expressions.
Social-IQ 2.0, in contrast, designed to assess social intelligence in multimodal contexts, requiring models to understand and reason about complex human interactions. Each video clip in the dataset portrays real-world social scenario sourced from platforms like YouTube, often involving interper- sonal communication, emotional exchanges, or coal-directed behavior. Importantly, each video is paired with multiple questions, each targeting a different aspect of the scene, such as the emotional state of person, their underlying intentions, or the appropriateness of their response. It transcends traditional discrete emotion concepts and aligns more closely with the diverse emotional experiences encountered in real-world contexts. Each question is accompanied by several candidate answers, only one of which is correct, thereby framing the task as a multiple-choice question-answering problem.
In our annotation process, for MER, we first expanded the original emotion label into a basic reason- ing process that incorporates both verbal and non-verbal cues. This reasoning was then formulated into a basic question that captures the emotion and its underlying justifications. Each annotator was subsequently tasked with generating hallucinated version of the question by subtly altering key aspects of the emotional reasoning, for example, by misattributing the cause of the emotion, ex- aggerating its intensity, or neglecting relevant multimodal cues. As a result, got 360 basic and hallucination QA pairs and 180 videos.
For Iocial-IQ 2.0, the annotation process was more complex. We preserved the original multiple- question-per- per-video structure to maintain the richness of the reasoning context. First, we filtered out ambiguous or poorly defined questions. For the remaining high-quality questions, we refined and expanded each one, along with its correct answer, into a more detailed version, which served as the basic question in our QA format. Annotators then created hallucinated variants by introducing subtle distortions in social or emotion understanding, such as misinterpreting intentions. assigning incorrect emotional reactions. This enabled us to evaluate MLLMs' ability to handle hallucinations in multimodal contexts. As a result, we generated 402 basic and hallucination QA pairs and 50 videos.
A.2 ANNOTATION ACROSS CATEGORIES
A.2.1 THFORY
Throughout the development of emotion psychology. various emotional theories have emerged, each reflecting different perspectives on the nature, origin, and structure of emotions (Cannon) [927] Scherer 2009; Lewis. 2005 Shiota and Kalat] 2017). In this setting, we focus evaluating the capability of MLLMs to detect hallucinations in these thcoretical frameworks, rather than judging the correctness or scientific validity of the theories themselves.
To clearly signal the basis of each item, we typically prefaced the question with phrases such as "Ac- cording to [TTeoryy Name]... or "[Theory Name] argues that ... followed by concise statement derived from the theory. When constructing hallucinated questions, we deliberately manipulated key aspects of the original statement, such as reversing causal relationships, disrupting conceptual
20

---

Published as a conference paper at ICLR 2026
quences, modifying the intension extension of key emotional constructs. This strategy allows us to assess whether the model can factually understand the theoretical content.
A.2.2 DEFINITION
In this setting, we focus on evaluating MILMs' capability to detecting hallucinations commonly accepted definitions from emotion psychology (Berkowitz 1999), which can help models better align with human emotion understanding. To probe their sensitivity, we typically preface the ques- tion with phrases such as "[Concept] ... followed by a definition in psychology.
When constructing hallucinated questions, we deliberately manipulate critical aspects of the original definitions, introducing subtle distortions in the intension (i.e., the core conceptual meaning) or the extension (i.e., the range of phenomena the concept applies to) of emotional constructs.
For example, widely accepted definition of anxiety might state: "Anxiety is a general expecta- tion that something bad might happen, without identifying any particular danger." To construct hallucinated version, we might alter it to: "Anxiety might happen, without identifying any particular danger." This version subtly but significantly dis- torts the core meaning of anxiety by replacing negative anticipation with positive anticipation, which undermines the emotional construct's 's essential nature.
general expectation that something good
Such hallucinated definitions are designed to appear structurally and linguistically plausible, yet semantically incorrect. They serve as diagnostic tool to assess whether MLLMs can distinguish between valid and flawed emotional concept definitions.
A.2.3 FINDING
In this setting, we focus on evaluating MLLMss capability to detect hallucinations in a set of empir- ical findings about emotion. Unlike formalized theories, these findings refer observed regularities or patterns in emotional expression, perception, and regulation, often cross-cultural or physiological in nature, without necessarily forming a comprehensive thcoretical framework (Kagitcibasi 1997
Hareli et al., 2015: Best et al., 2013 Smith 1989).
When constructing hallucinated questions, we deliberately altered the scope or the core semanties of the original statement. These manipulations may involve reversing cultural norms, misatrrbbuting physiological phenomena, or distorting the contextual boundaries of emotion.
For example, consider the empirical statement: "Japanese consider it inappropriate to show negative emotion to high-status person." A hallucinated version would state: "Japanese consider it appropri- ate to show negative emotion to a high-status person." While structurally similar, this altered version directly contradicts established cross-cultural observations, thereby testing whether the model can detect such culturally incongruent assertions.
This type of adversarial probing helps reveal how well MILMs understand fine-grained emotion, particularly in contexts involving cultural norms, physiological responses, or socio-emotional regu- lation. Such questions may be challenging even for humans unfamiliar with the specific context, yet the patterns they rely on often stem from deeply ingrained biological or sociocultural tendencies. Therefore, the ability of a model to distinguish between accurate and distorted emotional knowl- edge can serve strong indicator of its deeper emotional competence and alignment with human psychological realities.
A.2.4 CATEGORY
In this setting, we focused on real-world emotion understanding tasks, among which emotion recog- nition serves as a fundamental component (Dzedzickis et al.) 2020). current standard setups, this typically includes text-based sentiment analysis tasks, which often rely on coarse sentiment la- bels such as positive or negative. However, we also considered discrete basic emotion categories, such happiness, anger, and sadness, which are widely adopted in visual and multimodal emotion recognition tasks. Beyond these categories, we incorporated insights from recent work such as OV- MER (Lian et al., 2024), which proposes more fine-gra men-grineed and naturalistic taxonomy of emotions. This framework extends beyond basic emotions by incorporating more nuanced, everyday emotional expressions (e.g., relieved, anxious, content) better capture the richness of human affective states.
21

---

Published as a conference paper at ICLR 2026
In this context, constructed hallucinated examples by intentionally distorting the original emotion label annotations. Specifically, we substituted the original emotion with a semantically related or contrasting category. These substitutions can be adjacent in meaning (e.g., replacing angry with serious) or diametrically opposed (e.g., replacing angry with happy), depending on the intended degree of challenge.
Such manipulations allowed us to probe whether MLLMS can detect inconsistencies in emotional labeling, and whether they possess sufficiently fine-grained understanding of affective semantics distinguish between closely related or contrasting emotions. This process resulted in 334 question- answer pairs.
A.2.5 INTENSITY
In our setting, intensity refers to the strength magnitude of an emotional state expressed in natural language. Rather than relying on continuous valence-arousal scales commonly used in tra- ditional affective computing [Russell 1980), adopted discrete verbal descriptors (e.g., mild, normal, strong), which align more closely with everyday language and the reasoning capabilities of LLMs.
In this context, hallucinations occur when the model misjudges emotional intensity, either by ex- aggerating, underestimating, or otherwise misrepresenting the degree of emotion expressed in the input. For instance, changing "He seemed slightly annoyed by the interruption" to "He seemed very annoyed by the interruption." This form of distortion challenges the model's ability cap- ture nuanced emotional gradients and detect subtle affective cues, which is essential for applications requiring fine- peeiggraidd emotional understanding, such empathetic communication, psychological assessment, and emotion-aware content generation. This process resulted in 325 question-answer pairs.
A.2.6 REASONING RESULT
In the framework, reasoning result refers the final emotional interpretation that a model generates after perceiving and integrating multimodal emotional cues such as facial expressions, vocal tone, and textual context. Unlike perception-level errors, which involve failures in detecting or extract- ing relevant signals, reasoning result hallucinations arise when the model correctly identifies input cues but still draws an incorrect or implausible emotional conclusion based on faulty reasoning or inference.
For example, given a scenario where person has furrowed brow, a tense voice, and says "I can't believe this happened again," correct interpretation might be frustration. A hallucinated reasoning result would be labeling this as excitement or anger, despite all emotional cues being correctly perceived. In this case, the failure lies not in perception but in misinterpreting the emotional meaning of those cues.
This distinction emphasizes that accurate recognition of emotional signals is necessary but insuffi- cient condition for successful emotion understanding. Proper emotional inference requires coherent reasoning over the perceived inputs, making reasoning result hallucinations key diagnostic for evaluating models' deeper emotional competence. This process resulted in 181 question-answer pairs.
A.2.7 REASONING CUE
In addition, we defined reasoning cue hallucination, which targets failures in identifying, attending to, or correctly interpreting the emotional cues necessary for sound emotional regannnnge-rgressssssseeee of whether the final emotion classification is ultimately correct not. In this setting, hallucinations occur when the model overlooks, misinterprets, or fabricates critical multimodal signals, such as failing to detect angry tone in speech, misreading sad facial expression neutral, drawing unsupported emotional implications from text.
We considered two primary forms of cue hallucination: (1) cases where specific cue is modi- fied while the final emotional result remains unchanged, thereby testing the model's robustness to misleading or missing signals; and (2) cases where multiple key altered simultaneously,
22

---

Published as a conference paper at ICLR 2026
often accompanied by shift in the final emotion prediction, which challenges the model's entire reasoning chain, from perception to conclusion.
This setting highlights the importance of the reasoning pathway itself, rather than just the final output. A model may occasionally reach the correct emotion label by chance or superficial pattern matching, but reasoning justified and grounded in valid emotional evidence. As such, this setting is crucial for assessing models' interpretability and trustworthiness in real-world emotion understanding scenarios. This process resulted in 159 question-answer pairs.
hallucination reveals whether the underlying process is semantically
A.3 MORE BENCHMARK STATISTICS
As shown in Table 1 for the emotion knowledge text, lengths range from 4 to 56 words, with an average of 19.9 words. In the case of real-world review text&Deng et al. (2023) (see task details in Appendix A). the length ranges from 12 to 199 words, averaging 108.7 words. For real-world images Zhang al. 2 resolutions range from 320 x 194 to 600 1024 pixels, with an average resolution of 579.5 x 466.5 pixels. The real-world audio clips ([ivingstonee and Russo 2018) vary from 3.0 to 6.3 seconds length, with an average duration of 4.2 seconds. For real-world videos, short videos (Lian et al. 2023a) have average resolution of 870.0 x 476.9 pixels, ranging from 576 x 256 to 1280 720 pixels. Their durations range from 0,52 to 38.29 seconds, with an average of 4.29 seconds. In contrast, long videos (Zadeh et al. 2019 Wilf and collaborators 2023) have fixed duration of 60 seconds. Their resolution averages 637.9 x 360.0 pixels, ranging from 534 x 360 640 360 pixels.
B IMPLEMENTATION DETAIL AND ABLATION STUDY OF PEP-MEK
B. IMPLEMENTATION DETAIL OF PEP-MEK
As illustrated in Figure 5b| there are two main components of the PEP-MEK: modality and emotion knowledge extraction and setfict-exxlain-rrediiii
For the modality and emotion knowledge extraction, asked the model extract the modality (Yin et al., (024) and emotion knowledge. [ the prompt we designed to guide modality and emotion knowledge extraction. considers both modality-specciff knowledge (e.g., visual cues such as what can be seen in an image, or auditory cues such as what can be heard in speech) and emotion-sye -mptiin-specifiic knowledge (e-g., facial expressions, body posture, vocal prosody, emotion types and definitions, overall emotional atmosphere, and inferred causes). By structuring the prompt, we aimed to ensure that the model captures contextually grounded and psychologically meaningful emotional signals across modalities.
Next, we prompted the model to generate an initial answer by combining the extracted modality and emotion knowledge with the target question. As shown in Figure 8 the prompt instructs the model to base its judgment on the input content (c.g.. video, image. audio, or text). and, if needed, to incorporate the accompanying structured knowledge refine its answer. The model is explicitly asked to respond with only one word, YES or NO, to enforce clarity in evaluation, without generating additional explanations.
We then prompted the MLLM to provide an explanation for its initial answer by referencing the extracted modality and emotion knowledge, the target question, and the initial answer. As shown in Figure D the model was asked to first generate a detailed explanation of its reasoning, then assess the factual accuracy and logical soundness of its own explanation, and finally re-affirm its conclusion with a concise binary answer (YES or NO). This structured response format allows us to evaluate not only the model's 's decision, but also the justification process and self-consistency behind it.
B.2 ABLATION STUDY ON PEP-MEK
B.2.1 DIFFERENT COMPONENTS PEP-MEK
As shown in Table 5 we evaluate the contributions of different components in PEP-MEK. MEK de- notes incorporating modality and emotion knowledge perform prediction, while MEK+Fxplain
23

---

Published as a conference paper at ICLR 2026
Please extract useful modality and emotional knowledge from the given text/image/au- dio/video. Your goal is to gather interpretable features that can support accurate emotion prediction, explanation, and refinement. Structure your response according to the following components:
Overall Scene Mood and Context
Describe the general emotional atmosphere j...u. tense, melancholic, peace- ful).
Identify contextual elements in the environment (e.g., indoor/outdoor, weather, time of day, symbolic objects like candles, rain, broken glass).
What kind of situation or event might the scene represent (e.g.. birthday party, farewell, conflict)?
For audio/text: consider background sounds (c.g., music, ambient noise) narra- tive tone (e.g., hopeful, sarcastic, ominous).
Human Presence and Character Analysis
How many people are in the scene (visually or described in text/audio)?
For each individual:
Physical characteristics (e.g., age, gender, clothing style, if visible de- scribed)
Facial expression (e.g., happy, sad, fearful, angry, neutral, from video/image)
Head pose and gaze direction (e.g.. direct gaze, looking away, tilted head)
Body posture and hand gesture (e.g.. open arms, clenched fists, self-touch, leaning in or away)
Voice tone and prosody (e.g., trembling, rising pitch, flat tone, fast or slow pace)
Verbal emotional cues (e.g., exclamations, emotional word choice, metaphor- ical language)
Speech content (e.g., direct expression of emotion: "I'm scared," or indirect hints: "I don't know what to do anymore.")
3. Social and Emotional Interactions
Describe the relationships or interactions between people:
Are they physically close or distant?
Is there eye contact, mirroring expressions, or body synchronization?
Is any touch present (c.g., hugging, pushing, hand-holding)?
Do they seem emotionally aligned or in conflict?
In audio/text: Are their voices overlapping? Do they interrupt, agree, show empathy?
Emotion Type, Intensity, and Diversity
For each person, specify the likely dominant emotion and estimate its intensity (e.g., mild sadness vs. intense grief).
Are there multiple emotions present in the video/audio/text, possibly conflicting ones?
5. Emotion Knowledge and Reasoning Support
For each identified emotion:
Provide short definition and its typical visual, auditory. or linguistic cues.
Example: "Fear is response to perceived threat, often expressed by widened eyes, raised eyebrows, tense voice, and avoidance language."
If possible, suggest potential causes triggers of these emotions based on the multimodal context.
Figure 7: Prompt for modality and emotion knowledge extraction.
24

---

Published as a conference paper at ICLR 2026
Please provide clear response to the question below by watching the video (reading the text, watching the image, listening the audio). If necessary, you can also use the accompanying modality and emotion knowledge help refine your answer.
Modality and emotion knowledge: (knowledge)
Question: {question]
Please answer with ONLY ONE WORD: YES or NO. Do not provide any explanation or additional output.
Figure 8: Prompt for modality and emotion knowledge extraction.
First, please provide detailed explanation for your initial answer to the question. Then, ver- ity both the factual accuracy of your explanation and the logic behind your answer. Finally, give concise response to the question by answering with 'YES' or 'NO'. Modality and emotion knowledge: (knowledge)
Question: {question)
Initial Answer: [initial.answer]
Answer Format:
1. [Explanation]
[Verification]
3. [Final Answer]
Figure 9: Prompt for modality and emotion knowledge extraction.
(i.e., PEP-MEK) represents making second prediction based explanations generated from all inputs. Incorporating MEK consistently improves performance across all models, indicating that grounding predictions with structured emotional knowledge helps suppress spurious responses. Adding the explanation component yields further gains, highlighting the role of explicit reasoning in enhancing prediction faithfulness. Together, these results validate the effectiveness of both com- ponents and show that combining MEK with explanation- based reasoning forms complementary strategy mitigate emotion hallucinations.
Table 5: Ablation study on EmotionHallucer- showing the effect of different components PEP- MEK.
Yes/No Bias
Accuracy on Emotion
Methods
Input
Pct. Diff (~~0) FP Ratio (~-0.5) Basic Hallucinated & Overall 1
Original input
-0.18
0.30
35.51
71.96
10.49
Qwen2.5-Omni
MEK
-0.26
0.20
29.39
81.93
15.58
2005
MEK Explain
-0.19
0.29
37.49
74.87
20.15
Emotion-LL.e LLaMA Original input
0.28
0.77
77.34
22.02
9,65
MEK
-0.14
0.36
38.64
65.87
19.12
Chenge et
MEK + Explain
-0.08
0.41
46.35
62.86
26.03
Original input
0.00
0.50
61.54
62.05
33.44
Gemini-2.5-Flash
MEK
-0.08
0.39
54.95
71.37
35.84
Google DespMind 2025
MEK + Explain
-0.07
0.40
56.66
71.17
37.84
B.2.2 EMOTION KNOWLEDGE IN PEP-MEK
We further conduct ablation study to evaluate the role of prompt design in extracting Modality and Emotion Knowledge (MEK). In particular, we replace the MEK prompt with a generic modality knowledge (MK) prompt that removes emotion-specifie guidance. As shown in Table 6. this sub-
25

---

Published as a conference paper at ICLR 2026
stitution consistently reduces performance across all models. These results indicate that emotion- grounded prompts are essential for capturing emotion mnotion-relevant information, which in turn critical for detecting emotion-related hallucinations.
Table 6: Ablation study EmotionHaallucer- evaluating the role of emotion knowledge in PEP- MEK.
Yes/No Bias
Accuracy on Emotion
Methods
Input
Pet. Diff (~0)
FP Ratio (~0..5)
Basic 1 Hallueinated ? Overall
Original input
-0.18
0.30
35.51
71.96
10.49
Qwen2.5-Ome
PEP-MK
-0.23
0.24
31.93
78.38
13.21
ale 2025
+ PEP-MEK
-0.19
0.29
37.49
74.87
20.15
Original input
0.28
0.77
77.34
22.02
9.65
LaMM
+ PEP-MK
-0.07
0.43
42.66
56.95
16.80
Cheng 2024
+ PPP--EE
-0.08
0,41
46.35
62.86
26.03
Original input
0.00
0,50
61.54
62.05
33.44
Gemini-2..-Fllash
+ PEP-MK
-0.09
0.38
51.26
69.75
30.55
Google Deep 2025
+ PEP-MEK
-0.07
0.40
56.66
71.17
37.84
B.2.3 WALL-CLOCK LATENCY AND TOKEN-COST
To assess the computational overhead introduced by the PEP-MEK framework, we measure both token cost and wall-clock latency for each component (Baseline, Description, PEP-MEK-frst, and TEP-MEK-explain&scon
across all five modalities, these metrics are fundamentally modality- dependent. The results are summarized in Table 7 These measurements were obtained using the Qwen2.5-0mmi-77B model via the Aliyun Bailian API. Token cost collected by enabling True], while wall-clock time was measured
st stream_opt ons= minncludded.. unccludeussage as the elapsed time from sending the request receiving the final streamed chunk. We note that wall-clock latency also reflects network delay and API scheduling overhead, and thus should be interpreted approximation of pser-peeeeived latency rather than pure computational time.
The PEP-MEK components introduce moderate computational overhead but yield consistent per- formance improvements across modalities. Both the first stage and the second stage show stable accuracy gains relative to the baseline. The benefit is particularly pronounced in long-video scenar- ios, where explanation-uuider reasoning (PEP-MEK-expllee -EeE--ME--eplaii&&eeeeeee explain&secoond) outperforms the Description stage, indicating that structured multi-stage reasoning becomes increasingly important as emotional cues unfold over extended temporal sequences. Furthermore, comparisons in Figure show that PEP-MEK achieves stronger performance than alternative strategies operating under similar compu- tational budgets, demonstrating the effectiveness of investing computation toward coottion-specffee reasoning rather than naive prompting. We emphasize that for safety-sensitive applications involving emotional interpretation, robustness often more critical than inference speed. The modest latency increase of PEP-MEK therefore represents meaningful and practical trade-off.
B.2.4 PERFORMANCE ACROSS SUBCATEGORIES
Beyond modality-level efficiency, we further analyze the impact of PEP-MEK on the four halluci- nation subcategories, as shown in Table 8, We observe stable and consistent improvements for both +MEK and +MEK + Explain across all evaluated models. These gains appear consistently across subcategories, indicating that PEP-MEK enhances not only basic perceptual grounding, but also higeer-level emotional reasoning. Importantly, the improvements type but generalize across diverse hallucination categories, demonstrating that PEP-MEK mitigates hallucinations broadly applicable manner rather than overfitting to specific cases.
not confined single failure
B.2.5 COMPARATION TO GENERAL METHODS
To further validate the effectiveness of PEP-MEK, compare Qwen2.5-0mmi (Xu al.) 2025) with several widely used hallucination mitigation strategies. For CoT (Wei et al. 2022), we apply the prompt "Let's think step by step" to guide reasoning. In majority voting (Wang et al. 2022
26

---

Published as a conference paper at ICLR 2026
Table 7: Wall-clock latency and token-cost comparison for each PEP-MEK component across
modalities. Each cell reports Overall Accuracy / Token Cost / Latency (s).
Modality
Baseline
Description
PEP-MEK-fig PEP-MEEK-expaainSsccond
Text
19.00 / 00.6
-/ 13557 10.09
31.00/ / 1357 10.09
36.00 / 1
Image
32.67 / 462 2.47
-/ /1499 / 11.33
37.33 / 1193 / 8.69
40.00 / 1403 / 8,80
Audio
0.82 / 252 / 57.48
-/1103 / 74.54
2.99 / 769 49.68
5,43 / 995 / 20.27
Video-S
13.19 2 /55.09
-13584 23.65
27.08 3307 2.67
29.17 3533 3,91
Video-L
5.47 / 18389 191.43
- / 157.47
6.47 19591 45..66
17.91 19812 / 22.71
Table Overall accuracy comparison of PEP-MEK components across the four hallucination sub- categories: Category, Intensity, Reasoning-Rews Result, and Reasoning-Cue.
Model
Original Input
+MEK
+MEK Explain
Qwen2.5 Onen2.5-Omni
14.37/2.67/188.0 02 11.08/4.58/12.8111 41
18.86/6.87//55.9/1115 18.26/15.27730005/14555
21.56//.92/30.27721..
AmnotionLLLaMa
25.45/22.90/33 50/23.54
Gemini-2.5-Flash
42.51//2200//40322226..
43.71133.002/40.82222
we sample three outputs (temperature = 1.2, p 1.0) to match PEP-MEK's inference budget. Combining CoT with majority voting yields the self-consistency variant (Wang al., 2022). For RAG (Lewis et al. 2020), because our questions are not directly suited for conventional retrieval, variant: the model generates query emotion psychology textbook, retrieves a relevant answer, and then synthesizes its prediction.
design psychology-groundde
As shown in Table PEP-MEK consistently outperforms all alternatives. While these general strate- gies mitigate hallucinations to some extent, their lack of emotion-speciffe grounding limits effective- ness in our benchmark. We also observe that psychology-base -bsee RAG often biases retrieval toward facial wet-eeeexeooo-eelttieeee related cues, overlooking other emotional signals. Importantly, these strategies are not mutually exclusive with PEP-MEK; integrating structured emotion reasoning with retrieval, voting, or self-consistency offers a promising direction for future research.
Table 9: Performance comparison of Qwen2.5-Omni with different reasoning and augmentation strategies on EmotionHallucer-P.
Yes/No Bias
Accuracy on Emotion
Methods
Pet. Diff (~-0)
FP Ratio (~~0.5)
Basic f Hallucinated t Overall
Qwen2.5-Ommi Xuet
2025
-0.18
0.30
35.51
71.96
10.49
+CoT (Wsi al. 2022
-0,23
0.24
31.58
77.94
14.79
Maaoorty Voting Wing
2222
-0.11
0.38
42.73
64.29
10.78
Self-consistency
-0.29
0.17
27.35
85.60
17.31
RAG (Lowis
-0.13
0.36
41.60
66.92
13.40
PEP-MEK
-0.19
0.29
37.49
74.87
20.15
B.3 QUALITATIVE ANALYSIS OF PEP-MEK
As shown in Figure 10 we present an example that benefits from the PEP-MEK approach, where the generated content originates from the Qwen2.5-Omni model. is evident that the initial response, the model exhibited hallucination issue, failing to accurately interpret the facial expressions of in- dividuals in the image. However, with the integration of PEP-MEK, the model was guided to attend to more discriminative visual details, for instance, recognizing that most individuals are smiling, ex- cept for the person wearing a red tie, who has a neutral or serious expression. This targeted guidance significantly improved the model's 's emotion understanding of the image. Ultimately, the model iden- tified the key feature and produced the correct answer, demonstratigg the effectiveness of PEP-MEK in mitigating hallucinations and enhancing vision and emotion reasoning accuracy.
27

---

Published as a conference paper at ICLR 2026
Hallucinated: The person with red lie has happy expression. Initial Answer: YES
PEP-MEK:
Explanation: description states
person happy expression. In the provided image, most individuals are smiligg which aligns with the concept of happiness. However, the description specifically mentions only one individual (the person with the red lie). and his expression natural. Therefore,
a1
ssOnl
not fully capture collective emotion of the entire group but focuses single person.
description
Verification:
Image: Most individuals
image smiling,
indicating happiness. 2. Focus of Description: The description focuses on specific individual (the
with the red rather than the entire group. 3. Accuracy. The description only mentions one person out of severel. rather than describing the expressions of the majorily of the group.
Qualitative analysis of PEP-MEK. In the observed image, everyone except the person wearing the red tie is smiling, displaying a happy expression. PEP-MEK noticed this detail during explanation and ultimately provided the correct answer.
Figure 10:
C MORE EXPERIMENT DETAILS
C.1 SETUPS FOR BASELINES
In our experiment, we selected several well-known LLMs and MILMs for comparison, shown in Table I0 We provide a list of model names, model sizes, supported input and output modalities, all organized by release date enable more detailed analysis. Additionally,
annotate whether each model a reasoning model like DeepSeek-RI (Guo et al.) 2025). To ensure a fair comparison, we adopted the default sypeer-parametess of these models. All models with fewer than 235B parameters were run locally on either single NVIDIA A100 or four NVIDIA A100 GPUs. The remaining models were accessed via APIs provided by their developers.
C.2 RESULT OF UNIMODALITY
We present more fine- fine-grained evaluation and analysis of the model's performance for each indi- vidual modality.
C.2.1 EMOTION KNOWLEDGE
As shown in Table 11 report the performance of various models the EmotionHallucer-EH which focuses on emotion knowledge hallucination. These results are further visualized in Fig- II From the visualization, several key trends emerge: (1) model performance has steadily improved over time; (2) closed-source models generally outperform open-source models; (3) the incorporation of reasoning capabilities open-source models significantly narrows the gap with closed-source models; (4) MI.LMs generally perform worse than pure L.MSs on this task; and (5) within the same model family, larger parameter sizes tend to correlate with better performance. Overall, we observe a clear upward trend in model performance on emotion knowledge halluci- nation tasks, with many models achieving relatively high accuracy. This highlights the growing potential of leveraging LLMs for applications in emotion psychology, and points to future directions for further enhancing open models, particularly multimodal emotion reasoning.
C.2.2 MULTIMODALITY PERCEPTION: TEXT
As shown in Table 12 we report the performance of various models on the EmotionHallucer which focuses on hallucinations in real-world emotion understanding from text. The results are further visualized in Figure 12 From the figure, several key observations emerge: (1) model perfor- mance has gradually improved over time; (2) unlike the knowledge setting, the performance gap between closed- and open-source models is relatively small; (3) the incorporation of reasoning capabilities into open-source models does not lead to better performance; (4) within the same model family, larger parameter sizes do not necessarily correlate with higher accuracy; (5) although both PT and EK benchmarks are based on text, models perform significantly worse on PT, suggesting that LLMs adept at retrieving structured emotional knowledge than interpreting nuanced
28

---

Published as a conference paper at ICLR 2026
Table 10: Comparison of representative MLLMs. For closed-source models, we report the API version date. I, V, A, and T represent Image, Video, Audio, and Text, respectively.
Model
Date
Model Size (B)
Input
Output Reasoning
open-source
LLaVA
04.2023
7& 13834
I+T
T
No
Video- Caleo-ChatGP Maaz et
2025
06.2023
7
V+T
No
Mistral Jinn
2023
08.2023
7
T
No
Qwen (Bai etal 2023
09. 09.2023
7& 14
T
T
No
Chat-UniVi
11.2023
7
++V+T
T
No
LLaMA-VID
2024c
11.2023
7
I+V+T
No
Video-LLa (Lin al.
2023
11.2023
7
I+V+T
No
Onellm et 3024
12.2023
7
++A+V+T
T
No
Mixtral Bang 2014
01.2024
8x7
T
T
No
Llama3
04.2024
8
T
T
No
Emotion-
LnaMi LLMMA (Cheng st
2024
06.2024
7
V+I
No
Llama3. Gratiabori et
2024
07.2024
8&70
T
T
No
Qwen2 Ying el 1024
07.2024
7
Text
T
No
Llama3.2
Gomtaffrri
2024
09.2024
3
T
T
No
Llama3.2
2-ssinn
09.2024
11
I+T
No
Llama3.3 (Grattadion et al
2024
12.2024
70
No
Phi4 Abdin al 2024
12.2024
14
T
T
No
Qwen2.3 Yang
2124
12.2024
3&7 14&
T
T
No
DeepSeck- V3 Liunet
2024a
12.2024
T
T
No
DeepSeck-B
et
2025
2,2025
7&88114832&7086777
Yes
Qwen2.5-VV VL
Bai et
2025
02. 02.2025
32&72
I+V+T
T
Yes
QwQ Team 2025
03.2025
32
T
T
Yes
Gemma3 Team
2025
03.2025
121222
I+T
T
No
Mistral-small3.1
03.2025
24
I+T
T
No
Qwen2.5 5-Omni
Xu et 2023
03,2025
7
A +T
A+T
No
Qwen3 Owen Team 2024
04,2025
4& 342303338235
T
Yes
Kimi-Aus AudioAAdio
2225
04.2025
7
A+T
A+T
No
closed-source
GPT-40 Huns
2024
08,2024
T
No
Qwen- Oieen-Auioo-a Turbo
Clu
2024
12.2024
A
T
No
Qwen-Plus
2024h
01.2025
T
T
No
Qwen-Max
Yangetal
01.2025
T
T
No
Qwen-Vi VL-
Bai et
2025
01,2025
T+++V
No
QwQ-Plus
Teum 2025
03.2025
T
Yes
QvQ-Max Ownn Team 2025
03.2025
I+T
T
Yes
Gemini-22 2.5-
DespMind
2025
03.2025
4VA++
T
Yes
Qwen-VL-Man VL-Max
04,2025
T+4+V
T
No
2.5mni-2.5-Flashl
Google DespMind
2025
04,2025
V++++
Yes
GPT-4.11 OpenAll
2024
04,2025
I+T
T
No
GPT-OSS ApennA) 20250
08.2025
T
T
No
GPT-5 OpmAAl 3029
08. 20.2025
I+T
T
Yes
Qwen3- Qwen
2025
09.2025
T
Yes
emotions in real-world language; and (6) PEP-MEK demonstrates its effectiveness, often achieving performance that matches or exceeds the best-performing models of the same period.
We hypothesize that this performance gap stems from two primary factors: (1) current LLMs strug- gle to detect subtle emotional shifts and cues in naturalistic language, making them prone to halluci- nation text emotion understanding tasks; and (2) existing models may lack sufficient exposure pretraining tasks that require fine-graine emotion reasoning in everyday contexts. These findings highlight a critical challenge for future work: enhancing MLLMs' capacity to understand subtle emotional shifts in textual content.
C.2.3 MULTIMODALITY PERCEPTION: IMAGE
As shown in Table 13 we report the performance of various models on the EmotionHallucer-PP which focuses hallucinations in real-world emotion understanding from images. The results
29

---

Published as conference paper at ICLR 2026
Table 11: Performance comparison on EmotionHHllluce--Kk (Emotion Knowledge).
Model
Yes/No Bias
Accuracy EmotionHalluuer--E
Methods
Size
Pet. Diff (~0) FP Ratio (~0.5) Basic Hallucinated Overall
Open-source
Mistral
Bame et
2025
7B
0.26
0.89
92.47
39.52
34.68
Owen
Bai 05 at.
2023
0.21
0.81
87.63
45.70
37.37
Qwen
Bai
2022
14B
0.11
0.69
82.26
60.48
46.51
Mixtral
Jane
2024
8x7B
0.05
0,42
62.10
72.04
41.13
Llama3
Gralbdioni
8B
0,40
0.98
98.39
18.55
17.47
Llama3.
Gritiatioi al
0.26
0.90
93.28
41.14
37.37
Llama3.
Grattatiori et
2024
70B
0.22
0.94
96.77
51.88
49.73
Qwen2
Yang of al. X034
7B
0.18
0.86
93.01
56.18
50.54
Llama3.2
(iramatiori
3B
0.19
0.78
84.95
46.24
34.68
Llama3.3
Grattatooi
70B
0.20
0.92
95.97
56.18
53.76
Phi4 Abdin es
2024
0.79
91.40
66.67
60.22
Qwen2.5
Yang ct
2024b
3B
-0.13
0.27
58.87
84.95
45.70
Qwen2.5
2024b
7B
0.17
0.82
90.32
57.26
50.54
Qwen2.5
Yang
14B
-0.01
0.46
80.91
83.60
66.40
Qwen2.5
Ying et
32B
0.09
0.75
91.13
73,92
67.47
Qwen2.5
Ying et
2024b
72B
0.11
0.83
94.35
72.31
68.82
DeepSeek-V V3
Lau ctal
2024a
671B
0.10
0.76
90.59
69.89
62.10
DeepSeek-RI
Gso
2025
7B
0.19
0.80
87.90
50.27
43,55
DeepSeek-R
Guo
8B
0.15
0.78
87.63
56.99
48,92
DeepSeek R1
Goo et al.
2025
0.08
0.71
88.17
71.24
62.63
DeepSeck R1
Goo et al.
2025
32B
0.08
0.75
91.67
75.00
68.82
DeepSeek R1
Gso
2025
70B
0.09
0.76
91,40
72.58
65.86
DeepSeek-R
2025
671B
0.08
0.78
93.82
77.96
73.12
QwQ
Team
32B
0.07
0.75
92.74
73.66
73.39
Owen3
Quen Team
2024
0.05
0.63
86.02
76.61
65.32
Qwen3
Q Owen Team
2024
8B
0.04
0.63
86.56
78.23
67.20
Qwen3
Owen Team
2024
14B
0.04
0.61
86.83
70.30
68.82
Qwen3
Owen Team
30B
0.03
0.60
88.17
81.99
72.58
Qwen3
Qwen Team
2024
32B
0.06
0.73
93.01
80.91
76.08
Qwen3
Quen Team
2024
235B
0.05
0.67
91.13
81.72
74.18
GPT-OSS
OpenAT
2025a
20B
0.04
0.62
87.63
79.84
70.97
GPT-OSS
OpesAA
20250
120B
0.04
0.66
91.40
83.06
76.34
Twee3-Next-Imme
Owen
80B
0.02
0.55
85.22
81.99
68.55
Qwen3 Downn--nex-Thinkiing
Thinking (Qwen
2025
-0.04
0.36
82.26
90.05
73.66
LLaVA Lin
7B
0.41
0.97
97.85
16.40
15.59
LI VA Lin
2021
0.85
90.32
43.82
36.29
LLaVA Li
2013
34B
0.43
1.00
99.73
13.71
13.71
Iddo-ChatGPP
Maaz
7B
0,26
0.83
86.56
34.95
25.27
Chat-UniVi
7B
0.50
1.00
100.00
0.00
0.00
LLaMA-VID Liet
2024
0.50
1.00
100.00
0.00
Vidco-LLaVA
1in 2023
0.50
1.00
100.00
0.00
0.00
Onellm ei
2024
7B
-0.01
0.48
66.40
68.82
40.05
Emotion-LLaMiX
Chenget
7B
-0.04
0.45
59.68
67.20
32.53
Mistral-small3.1
Mistral
24B
0.12
0.77
90.05
66.94
59.14
Llama3. Llama3.2-vision 2-visinn
Grottfoooi es
2024
0.88
92.20
45.16
40.32
Qwen2.5- VL. Bai Ptil
2025
32B
0.25
0.97
98.39
48.39
47.85
Qwen2.5 VL Bai
2025
72B
0.09
0.79
93.55
75.54
70.43
Gemma3
4B
0.28
0.90
93.01
36.56
32.26
Gemma3
Team al.
2025
0.19
0.85
91.94
54.30
49.19
Gemma3
Team al
2025
0.92
96.24
55.38
52.69
Qwen2.5 5-0mmi Xa of
7025
7B
0.28
0.96
97.58
41.40
39.78
Closed-source
GPT-4o Hurst al
2074
0.10
0.78
92.74
73.66
68.82
Qwen-Plun Yang
2024b
0.17
0.89
94.89
60.48
58.06
Qwen-Max
Yung dl
2024b
0.14
0.84
93.28
66.59
61.02
OwQ-Plus
Remm 2025
0.07
0.76
93.55
79.30
75.81
QvQ-Max Qwen Team 2025
0.05
0.67
90.59
81.18
74.19
Gemini-2.5-1 5-
Kioople DeepMind
2025
0.02
0.58
88.98
84,68
75.54
Gemini 2.5 Flash Google DeepMind
2025
0.03
0.61
90.05
84.14
75.54
GPT-4.1 OpenAl 2021
0.08
0.81
94.62
77.69
74.19
GPT-5 OpenAA 20256
0.02
0.59
91.94
88.17
81.45
30

---

Published as a conference paper at ICLR 2026
DetoScck R1 N. Pash
GPT GFT-0551200 Qwen Thecinnnnng
Ce
Owen Nesh-thetrurt
cel 5R1 706
Pi
MHI 3.1-244
Liamaz.a 738Gemmb3 278
Overall Overall Accuracy
50
USi 78B
Quen 3B
DecpSeak
40
Nneei 9x77
Ulaaã 2epision1188
Owen2 SeDmni-71
118
Emmtiinnél..o a8M
Gemma3 4B
30
Vides-CherGPT-78
L'ama3
LLa 545
VCitebil AMSSFO771
May
May
Sep
Release Date
Figure 11: Performance comparison on EmotionHallluce--KK Blue denotes LLMs, red denotes MLLMs. Circles indicate known parameter sizes, while stars represent unknown parameter sizes. Red dashed line denotes the top-performing model of the current month.
Gemin S'isen
Quen3-Ne New.n---eiiiiinn Tarkinn
50
Liama
tiven2.5
Miliig 3.1-248
200
Overall Accuracy
40
LLavg-328
Emotion LLaMA 78/GGMMMB
Ulama8 8B
Cle Oaaoth 88
Cween3 2359
304en3- 308
CPT--86-206
(SP-NNK
348
32B
Qwea2:77 7B
Mistual-78 78
Oriiness Sa
Emotiotdaaaalee
Upm
20
Llama3.2.33
10
Owenn 7B
Chat-laeivi-79
Onelar-7B
Release Date
Figure 12: Performance comparison on EmottonnHllluce-.P1. Blue denotes LLMs, red denotes MLLMs. Circles indicate known parameter sizes, while stars represent unknown parameter sizes. Red dashed line denotes the top-performing model of the current month.
31

---

Published as a conference paper at ICLR 2026
Table 12: Performance comparison EmotionHHlluuer- (Perception Text).
Model
Yes/No Bias
Accuracy on EmotionHallucer
Methods
Size
Diff (~0) FP Ratio (~0.5) Basic
Hallucinated Overall
Open-source
Mixtral Jiang et
2024
0.37
0.99
99.00
25.00
25.00
Qwen
Boi al.
2022
7B
0.12
0.60
53.00
29.00
10.00
Qwen
Bai
2023
14B
0.07
0,41
53.00
67.00
34.00
Mixtral
Tanget
2024
8x7B
0.14
0.33
46.00
73,00
36,00
Llama3
Grilblioti etall
2024
0.04
0.58
75.00
66.00
50.00
Llama3.
Krratatiori al.
2024
0.11
0.63
75.00
53.00
42.00
Llama3.
Grataatioii ot
2024
70B
0.40
1.00
100.00
21.00
21.00
Qwen2
M0540
7B
0.34
0.95
96.00
28.00
28.00
Llama3.2 2 Gramatiooi
3B
0.39
0.06
17.00
95.00
16.00
LIama3.3
Grattationi et
2024
1.00
100.00
21.00
21.00
Phi4 Abdin al. 2024
14B
0.36
0.99
99.00
26.00
26.00
Qwen2.5
Yang ml.
2034h
3B
0.19
0.19
50.00
88.00
46.00
Qwen2.5
Yangetall
2024h
7B 0.39
0.98
98.00
20.00
20.00
Qwen2.5
Ying
14B
0.38
1.00
100.00
25.00
25.00
Qwen2.5
Yang ct
2024b
0.29
0.94
96.00
38.00
37.00
Qwen2.5
Yang al.
2024b
72B
0.30
1.00
100.00
40.00
40.00
DeepSeek- V3
-B
0.28
0.93
95.00
38.00
36,00
DeepSeek-RI
7B
0.30
0.93
95.00
35.00
34.00
DeepScek-N R1
Goo al.
2025
0.34
0.97
98.00
29.00
28.00
DeepSeck R1
Goo et al
2025
14B
0.32
0.98
99.00
35.00
35.00
DeepSeck R1
Goo al
2025
32B
0.23
0.93
96.00
49,00
48,00
DeepSeek-R
Glso
2025
70B
0.37
0.99
99.00
25,00
25,00
DeepSeck-R
671B
0.28
0.94
96.00
40.00
40.00
Qw
Trom 2025
0.31
1.00
100.00
38.00
38.00
Qwen
O Qwen Team
2024
4B
0.35
0.99
99.00
28.00
28.00
Owen3
Owen Team
2024
8B
0.29
0.94
99.00
38,00
38.00
Qwen3
Owen Team
14B
0.33
0.95
96.00
30.00
28.00
Qwen3
Qwen Team
2024
30B
0.26
0.89
93.00
41.00
40.00
Qwen3
Qwen Team
2024
32B
0.33
0.97
98.00
32.00
32.00
Qwen3 Quen Team
2024
235B
0.23
0.89
93.00
45.00
42.00
GPT-OSS UpenAA
20B
0.26
0.89
93.00
41,00
40.00
GPT-OSS OpesAi
120B
0.23
0.86
91.00
46.00
45,00
Qwen3-Nexx -ee-----ntstucc
Instruct
Owen
2025
0.92
94.00
35.00
33.00
Qwen3- -eettt---tt-t-- Next-Thinking
(Qenn
2025
80B
0.10
0.68
82.00
62.00
54.00
LLaVA Liner 2023
0.47
1.00
100.00
5.00
LLaVA Lin 2023
13B
0.21
0.18
46.00
88.00
41.00
LL Liu 2003
34B
0.32
0.96
97.00
33,00
32.00
Video-Chat
7B
0.32
0.17
19.00
83.00
18.00
Chat-UniVi Jin 2024)
0.47
0.01
6.00
99.00
6.00
LI ei 2014G
0.20
0.68
64.00
25.00
19.00
Vidco-LLaVA
iinn
2023
7B
0.47
0.98
98.00
4.00
4.00
Onellm Han
7B
0.50
1.00
100.00
0.00
0.00
Emotion-LLaMMA LLaMA Cheng
2024
7B
0.28
0.22
21.00
78.00
21.00
PEP-MEK
0.14
0.32
46.00
75.00
39.00
Mistral-small3..
Mistral A1
2024
24B
0.20
0.84
91.00
51.00
47.00
Llama3 2-vision
GGilltoioo
2024
11B
0.02
0.53
69.00
65.00
47.00
Qwen2.5- VL
32B
0.39
1.00
100.00
23,00
23,00
Qwen2.5
Bai ct
2025
72B
0.32
0.98
99.00
36.00
36.00
Gemma3
Team ot
2025
0.29
0.91
94.00
36.00
35.00
Gemma3
Team 2025
12B
0.35
0.97
98.00
28.00
28.00
Gemma3
Team
27B
0.39
1.00
100.00
23.00
23.00
Qwen2.5 5-0mi
7B
0.41
1.00
100.00
19.00
19.00
+PEP MEK
0.28
0.91
94.00
38.00
36.00
Closed-source
GPT-4o Hurst et al.
2024
0.28
0.94
96.00
39.00
39.00
Qwen- Plus Yaug
2024b
0.31
0.96
97.00
35.00
35.00
Owen-Max
20140
0.36
0.99
99.00
27.00
27.00
QwQ-Plus
0.33
0.99
99.00
33.00
33.00
QvQ-Max Qwen Team 2025
0.24
0.93
96.00
47.00
45,00
Gemini- 2.5 Pro Google DeepMind
2025
0.23
0.92
96.00
51.00
49.00
Semmini-255-Flas
Google DoepMind
3025
0.21
0.89
94.00
51.00
50.00
MPE
0.12
0.74
88.00
54.00
60.00
GPT-4.1 OyenAl
0.30
0.92
94.00
34.00
33,00
GPT-5
OpenAl 2025
0.18
0.80
88.00
51.00
46.00
32

---

Published as a conference paper at ICLR 2026
Table 13: Performance comparison on EmotioonHallucer-P) (Perception Image).
Model
Yes/No Bias
Accuracy on Perception-1
Methods
Size Pet. Diff (~0) FP Ratio (~0.55 Basic
Hallucinated f Overall f
Open-source
LLaVA
Lio
7B
0.50
1.00
100.00
0.00
0.00
LLaVA Liu
13B
0.06
0.44
47.33
58.67
22.00
LLaVA
34B
0.46
0.01
6.67
99.33
6.00
Video ChatGPT
Maase et
2023
7B
0.04
0.54
53.33
45.33
14.00
Chat-UniVi Bin
2024
7B
-0.31
0.18
19.33
82.00
14.00
LLaMA-VID>
X034
7B
-0.21
0.28
32.67
74,00
25.33
Video-LLaaVe
7B
0.46
0.99
99.33
6.67
6.67
Onellm Han 2024
0.48
0.99
98.67
3.33
3.33
Emotion LLaMA Cheng et
2024
7B
0.32
0.81
80.00
15.33
14.00
PER-MEK
0.09
0.60
63.33
44.67
28.00
Llama3. 2-vision Grmmafiooi
2024
11B
0.25
0.79
82.67
33.33
22.67
Qwen2.5-Ve
Bai
32B
-0.02
0.47
60.67
65.33
41.33
Qwen2.5
Bai ctal
2025
72B
-0.07
0.39
64.00
77.33
50.00
Gemma3
Team al
2024
4B
-0.14
0.35
40.00
68.00
22.67
Gemma3
Team 2025
12B
-0.17
0.22
46.67
80.00
39.33
Gemma3
Team
27B
0.13
0.66
74.00
48.67
40.67
Mistral small.
Mistral
2024
-0.44
0.02
10.67
98.00
9.33
Qwen2.5 O-mni
Xu et 2025
7B
-0.11
0.36
48.00
70.67
32.67
+EEP-MEK
7B
-0.11
0.34
54.00
76.00
40.00
Closed-source
GPT-4o Huuss
M24
0.04
0.56
72.00
64.00
46.67
QvQ-Max Owen
0.04
0.57
72.57
64.00
46.00
Gemini-2. 1amnii-2.5-P7
Glople DrepMind 2025
0.06
0.62
82.00
70.00
56.67
Qwen- VL-Plus
Bai 2025
-0.24
0.22
34.67
82.00
26.00
Qwen-V VL-Max
2025
-0.07
0.38
63.33
78.00
50.00
2emini-2.5--lashl
DeepMind
2025
0.05
0.58
74.67
64.67
46.67
0.02
0.53
73.33
69.33
50.67
GPT-4.1 OpenA 2024
0.07
0.61
77.33
64.00
$4.00
GPT-5
OpenA 20256
0.01
0.52
79.33
77.33
64.00
are further visualized in Figure 13 From the figure, several key observations emerge: (1) model performance has steadily improved over time; (2) closed-source models generally outperform open- source models, though the performance gap is relatively small; (3) the incorporation of reasoning capabilities into open-source models does not lead clear performance gains; (4) in most cases, larger parameter sizes tend to yield better results; and (5) PEP-MEK demonstrates its effectiveness, same period.
often achieving performance that matches or exceeds the pest-performing models
Furthermore, the lack of significant improvements from that reasoning paradigms the visual domain are still their early stages and require further de- velopment and refinement. These findings highlight critical challenge for future work: enhancing MLLMs' capacity to understand emotional cues embedded visual content, and to reason more effectively about complex human affect in real-world image contexts, particularly through the tinued advancement and adaptation of reasoning paradigms.
models may indicate
C.2.4 MULTIMODALITY PERCEPTION: AUDIO
As shown in Table 14 we report the performance of various models on the EmotionHallucer-PA benchmark, which focuses on hallucinations in real-world, perception-based emotion understanding from audio. The results are further visualized in Figure] From the figure, several key observations emerge: (1) model performance has gradually improved over time, though remains in the early stages, with most models only slightly exceeding the random guess baseline of 25%; (2) closed- source models significantly outperform open- sourcs models; and (3) PEP-MEK demonstrates strong effectiveness, often achieving performance that surpasses the best models available at the time.
33

---

Published as a conference paper at ICLR 2026
70
60
50
Out P MEEEEE MEKI
Gee Mike
Overell Accuracy
40
QIIITZ 20 TEE RMEE
Ieee 4-0mmi--78
30
CNicn VL Plus
lama's Soationn111
20
Video-C
V-78
10
Mistralsmali 1-248
LLava 34B
Onelin
LLana 78
Mar 2023
May
Jul
Sep
Nov
Jan
Nar
May
Sep
Nov
Jan
Mar
May
Jul
Sep
Nov
2024
2025
Figure 13: Performance comparison on EmotionHallucer-PL Blue denotes MLLMs that accept only the current modality (e.g., image), red denotes models capable of handling additional modalities (e.g., video, audio), and cyan denotes the results obtained using PEP-MEK. Circles indicate known parameter sizes, while stars represent unknown sizes. The red dashed line marks the top pop-prformmng model of the current month.
Table 14: Performance comparison on EmotionHalluuer--A (Perception Audio).
Model
Yes/No Bias
Accuracy on
Methods
Size
Pet. Diff (~0) FP Ratio (~0.5)
Basic
Hallucinated Overall
Open-source
Onellm vtal
2024
0.50
1.00
100.00
0.00
0.00
Emotion LLaMA
Cheng et
2024
7B
0.43
0.92
91.85
6.52
5.16
PPE-
-0.14
0.36
33.68
64.67
22.28
Qwen2.5-Omni
al.
2025
7B
-0.49
0.01
1.36
99.46
0.82
-0.45
0.04
6.25
95.92
5.43
Kimi-Audio Ding ctal.
2025
-0.23
0.22
36.41
81.79
19.29
Closed-source
Owen-Audio- Turbo Cha et al.
2024
-0.49
0.01
2.17
99.46
1.63
Gemini-2.5-1 Gioople DeepMind
2025
-0.14
0.34
41.03
69.02
24.46
Gemini-2.55Flaan Flash
Google DeepMind
2025
-0.13
0.34
45.11
71.74
30.43
-0.12
0.34
48.10
72.83
34.51
We hypothesize that the relatively low performance in the audio modality is due the current focus of audio-based MLLM research, which primarily targets tasks such Automatic Speech Recognition (ASR) and semantic audio understanding. For instance, when refusing to answer, GPT- 4o-Audio responds with: "I'm sorry, but can't 't help with that request." These tasks emphasize transcribing or interpreting spoken content, but often neglect paralinguistic features, such as tone, intensity, and prosody, which are essential for emotion understanding. These findings highlight critical challenge for future work: enhancing MLLMs' ability to perceive and interpret emotional signals embedded in speech, beyond lexical content. This calls for the development of audio audio-specifis specific reasoning paradigms and training objectives that better capture the richness of human emotional expression in voice.
34

---

Published as a conference paper at ICLR 2026
40
30
Semini Flash Owen? 5- Soimenttssss
Accural
20
Kimi-tus
Emoooon MMA
Owenz
Nov
Jan
Mar
May
Jul
Scp
Nov
Mar
May
2023
2024
2025
Figure 14: Performance comparison on EmotionHallucer- Blue denotes MLLMs that accept only the current modality (c.g., audio), red denotes models capable of handling additional modalities (e.g., iamge, video), and cyan denotes the results obtained using PEP-MEK. Circles indicate known parameter sizes, while stars represent unknown sizes. The red dashed line marks the top oop-perfommnn model of the current month.
C.2.5 MULTIMODALITY PERCEPTION: SHORT VIDEO
As shown in Table 15 report the performance of various models the EmotionHallucer-PVV which focuses on hallucinations in real-world, perception- based emotion understanding from short videos. The results further visualized in Figure 15 For models that only support image input, we adopt a key-frame sampling strategy to enable evaluation on video data.
Several key observations emerge from the figure: (1) model performance has steadily improved over time; (2) closed-source models generally outperform open-source models, though the perfor- mance gap remains modest; (3) for MILMS that only support image input, increasing sampled frames typically does not lead to performance improvements: (4) some models, such as LLaMA- VID, Video-LLaVA, and OneLLM, perform particularly poorly on this task; (5) Emotion LLaMA, despite being fine-tuned on emotion recognition tasks, does not outperform general-purposs. models; (6) PEP-MEK continues demonstrate strong performance, often matching or exceeding that of the best models during the same period.
Furthermore, video-based emotion understanding and hallucination detection remains an open and underexplored challenge, with most models still in the early developmental stage. Our results also indicate that supervised finetuning on emotion tasks does not necessarily alleviate hallucination is- sues. Notably, although Emotion-LLaMA achieves excellent performance MER, it performs poorly on EmotionHallucer--Pa constructed from the same data source but reframed for halluci- nation detection.
These findings highlight critical challenge for future work: to develop sideo-language models with stronger emotion recognition and reasoning capabilities, and to identify effective strategies for mitigating hallucinations introduced by supervised finetuning (Perez et al. 2022 Achiam et al,
2023
35

---

Published as a conference paper at ICLR 2026
Table 15: Performance comparison EmotiontHallucer-PPP/ (Perception Short Video).
Model
Yes/No Bias
Accuracy on Emotion//alltuer-PP/S
Methods
Size
Pet. Diff' (~-0) FP Ratio (r~O.5) Hallucinated 1 Overall t
Open-source
LLava et
2003
IF
7B
0.50
1.00
100.00
0.00
0.00
I.Lava einet
VFF
13B
-0.33
0.14
19.44
87.22
11.11
I.vavv
/FF
34B
-0.44
0.04
7.22
96.11
3.89
2-visin-
Granafiorietal al 2004 /1F
11B
0.18
0.72
76.67
40.00
24.44
Llama3.2 Llama3.2-vision
2-viion
Gratation
2024
2024
2F
11B
-0.46
0.02
5.00
97.78
3.33
Llama3 2-vision
Gratafiori
/
11B
-0.49
0.01
1.67
99.44
1.11
Gemma3
V1F
4B
0.21
0.74
76.11
33.33
17.78
Gemma3
Term et
2025
/2F
0.24
0.74
75.00
27.22
14.44
Gemma3
Trom at
2005
/4F
4B
0.22
0.73
75.00
31.67
15.00
Gemma
Team
2005
/1F
12B
0.16
0.69
73.68
42.78
25.56
Gemma3
2025
/2F
12B
0.15
0.66
69.44
39.44
20.56
Gemma3
2025
/4F
12B
0.03
0.53
58.33
52.22
20.56
Gemma3
Tram ct
2025
/FF
27B
0.28
0.82
83.89
27.78
17.22
Gemma3
Team at
2025
12F
27B
0,22
0.73
75.00
31.11
17.22
Gemma3
Team 2005
/4F
27B
0.18
0.70
72.22
36.67
20,00
Mistral-sman small.1
2024
/1F
24B
-0.13
0.35
41.11
67.78
17.78
Mistral-smal13.
Misrrl
/2F
24B
-0.23
0.24
32.78
78.89
17.22
Mistral-smalI3.1
Misrill
1034
/4F
24B
-0.38
0.11
12.22
88.89
6.11
Mazz et
2013
-0.09
0.41
39.44
$8.33
13.89
Chat-UniVi Jim al 2024
7B
-0.16
0.32
39.44
71.67
26.11
LLaMA-VID Lietal
2024c
7B
0.49
0.99
99.44
0.56
0.56
Video-LLaVA
2023
7B
0.50
1.00
100.00
0.00
0.00
Onellm
2024
7B
0.50
1.00
100.00
0.00
0.00
Emotion -naMiion-LLaMAA LLaMA
(Cheng et
2024
0.20
0.70
70.71
31.07
10.71
<PE--MER
-0.12
0.35
45,00
70.00
26.11
Qwen2.5 2ren2..5VV VL
Baä
2025
32B
0.32
0.95
86.11
22.92
15,97
Qwen2.5- VL
72B
0.38
0.92
93.06
18.06
13.89
Qwen2.5 5-mmn
Xu es
2025
0.37
0.91
91.67
17.36
13.19
PPP-MEKK
-0.09
0.38
51.39
70.14
29.17
Closed-source
GPT-40
Hurst ctal.
2024
1FF
0.00
0.50
57.78
$7.22
20,56
GPT-4o Hors al.
2004
/2F
0.05
0.56
62.22
52.78
24.44
QvQ-
3025
0.23
0.76
79.86
34.73
27.08
Gemini-2..m Pro Gauple DrepMind
2025
0.24
0.81
85,90
36.46
30.77
Qwen-V VL-Plus
Hai et
2025
-0.14
0.33
43.75
72.22
20.83
Owen-VL-Man VL-Max
Bai et
XI2A
0.38
0.93
93.75
17.36
14.58
Gemini-2.55 2.5-
Giogle DeepMind
M25
0.19
0.77
83.33
45.51
36.54
-0.04
0.45
62.78
70.00
40.00
GPT-4,1 OpenA 2026 VIF
0.20
0.72
75.56
35,56
19.44
GPT-5
OpenAl 2025h 1F
0.01
0.51
58,33
57.22
23.89
36

---

Published as a conference paper at ICLR 2026
40
Ge EiiGht
Owenz m.0m
G Mari
EverrllAccuuren
vision
Ges 9-12
GPT-
20
Qwee-Vi Sim
Sea
Video ChS
Emocien MA-78
LLa6a 78
Weel Oxffff
Mar 2023
May
Jul
Sep
Nov
Jan
Mar
May
Sep
Nov
Jan
Mar
May
Jul
Sep
Nov
2024
2025
Figure 15: Performance comparison on EmotionHallucerr Blue denotes MLLMs that accept only the current modality (c.g., video), red denotes models capable of handling additional modalities (e.g., image, audio), and cyan denotes the results obtained using PEP-MEK. Circles indicate known parameter sizes, while stars represent unknown sizes. The red dashed line marks the top cop-performing model of the current month.
C.2.6 MULTIMODALITY PERCEPTION: LONG VIDEO
As shown in Table 16 we report the performance of various models on the EmotionHallucer--P PVVLL benchmark, which locuses on hallucinations in real-world, perception-basee emotion understanding from long-form videas. The results are visualized in Figure T6] For models that only support image input, we adopt a key-frame sampling strategy to approximate video-level evaluation.
From the figure, several key observations emerge: (1) model performance has gradually improved over time, but overall results remain low, indicating that this task still in its early stages; (2) closed-source models generally outperform open-source ones, though the performance gap is not as large as in other benchmarks; (3) nearly all models perform at or below random guess levels, underscoring the difficulty of emotion understanding in long video contexts; (4) Emotion LLaMA, despite being fine-tuned on emotion recognition tasks, fails to outperform ceneral-purpose models, suggesting limited transferability to hallucination detection; (5) PEP-MEK demonstrates strong and consistent effecttivensss, in many cases outperforming all models released in the same timeframe.
that many MLLMs perform surprisingly poorly on this task, despite their strength on other modalities and tasks. This discrepancy suggests that current MILMs are not yet equipped reason about fine-grained emotion states and changes over long temporal spans, and may lack both temporal integration and emotion-specific understanding capabilities.
Additionally,
These findings highlight critical challenge for future work: develop temporal-aware. emotion- grounded MLLMs capable of robust reasoning over long-form emotional content. It will also be essential explore new reasoning paradigms and training strategies that directly target ballucination resilience in dynamic emotional contexts.
C.3 RESULT OF MULTIMODALITY
We further present the hallucination performance of models EmotionHallucer-NoA in Ta- ble [7 and Figure [7 providing an overall view of multimodal capability. From the figure, several key trends can be observed: (1) model performance has consistently improved time, reflect-
37

---

Published as a conference paper at ICLR 2026
Table 16: Performance comparison EmottionHalluuer-PP// (Perrccctttoteeieieiiti Long Video).
Model
Yes/No Bias
Accuracy on Emotion//alltcer-PVV/
Methods
Size
Pct. Diff' (~-0) FP Ratio (r~0.5) Hallucinated 1 Overall t
Open-source
LLava et
2003
IF
7B
0.50
1.00
100.00
0.00
0,00
I.Lava einet
VFF
13B
-0.13
0.37
38.31
63.68
5.97
I.vavv
/FF
34B
-0.44
0.05
6.47
95.02
1.99
2-visin-
Gratafforietal al 2004 /1F
11B
0.25
0.77
79.10
29.35
12.44
Llama3.2 Llama3.2-vision
2-viion
Gratation
2024
2024
2F
11B
-0.38
0.11
12.44
89.05
3.48
Llama3 2-vision
Gratafiori
/
11B
-0.49
0.01
0.50
98.51
0.00
Gemma3
V1
4B
-0.03
0.47
49.25
55.22
12.94
Gemma3
Term et
2025
/2F
-0.12
0.37
41.79
65.67
10,95
Gemma3
Trom at
2005
/4F
4B
-0.06
0.43
49.25
61.69
14.93
Gemma
Team
2005
/1F
12B
-0.07
0.42
48.26
62.69
16.42
Gemma3
2025
/2F
12B
-0.10
0.39
44.28
64.18
13.43
Gemma3
2025
/4F
12B
-0.15
0.33
40.30
70.65
13.93
Gemma3
Tram ct
2025
/FF
-0.01
0.49
$2.74
54.23
9.95
Gemma3
Team at
2025
12F
27B
0.00
0.50
54.23
54.73
11.44
Gemma3
Team 2005
/4F
27B
-0.09
0.41
44.78
62.19
11.94
Mistral-sman small.1
2024
/1F
24B
-0.40
0.08
12.44
92.04
4.98
Mistral-smalI3.
Misrrl
/2F
24B
-0.45
0.05
5.47
95.52
2.49
Mistral-smalI3.1
Misrill
1034
/4F
24B
-0.48
0.02
2.99
98.51
1.49
Mazz et
2013
0.17
0.66
64.18
30.85
13.43
Chat-UniVi Jim al 2024
7B
0.13
0.63
62.69
36.82
18.41
LLaMA-VID Lietal
2024c
7B
0.49
0.98
97.51
0.00
0.00
Video-LLaVA
2023
7B
0.50
1.00
100.00
0.50
0.50
Onellm
2024
7B
0.50
1.00
100.00
0.00
0.00
Emotion -naMiion-LLaMAA LLaMA
(Cheng et
2024
0.36
0.86
86.57
14.93
7.96
<PE--MER
-0.04
0.45
52.74
60.70
24.88
Qwen2.5 2ree2.55VV VL
Baä
2025
32B
-0.17
0.31
36.32
71.14
9.95
Qwen2.5- VL
72B
-0.16
0.32
38,81
70.65
11.44
Qwen2.5 3-Omnl Xu es
2025
-0.36
0.12
16.42
88.06
5.47
PPP-MFKK
-0.06
0.43
44.28
57.21
17.91
Closed-source
GPT-40 Hurst ctal.
2024
11F
-0.41
0.08
9.95
92.04
4.48
GPT-4o Hors al.
2004
/2F
-0.37
0.12
15.42
88.56
5.97
QvQ-Mat
3025
-0.05
0.45
49.25
58.71
17.41
Gemini-2..m Pro Gauple DrepMind
2025
-0.03
0.46
55.72
62.19
20.90
Qwen-V VL.
Mai et
2025
-0.49
0.01
0.50
99.00
0.00
Owen-VL-Man VL-Max
Bai et
XI2A
-0.05
0.45
49.25
$9.20
11.94
Gemini-2.55 2.5-
Giogle DeepMind
ME5
-0.06
0.43
48.76
60.70
18.41
-0.19
0.29
37.81
75.12
21.39
GPT-4,1 OpmnA 2026 VIF
-0.29
0.19
24.88
82.59
10.45
GPT-5
OpenAl 2025h VF
-0.42
0.08
9.45
92.54
3.48
38

---

Published as a conference paper at ICLR 2026
25
Emations BIPEE-MER)
Chat M
Qwerd
Overrll Accuracy
15
Video-Gro GPT-78
Liama3 vision 11B
Emotior
LLaVe 338
CPI-4o
Owend mmi--77 7B
GITES
LLava 34B
Yoa Cal Wain
Mar
May
Jul
Sep
Nov
Jan
Mar
May
Sep
Nov
Jan
Mar
May
Jul
Sep
Nov
2023
2024
2025
Figure 16: Performance comparison on EmotionFallluer-PV/ Blue denotes MLLMs that accept only the current modality (c.g., video), red denotes models capable of handling additional modalities (e.g., image, audio), and cyan denotes the results obtained using PEP-MEK. Circles indicate known parameter sizes, while stars represent unknown sizes. The red dashed line marks the top nop-performing model of the current month.
ing the rapid progress in multimodal learning; (2) recent image-based MILMs outperform earlier models with video- rideo-prccessing capabilities, suggesting advances in visual understanding even with- out temporal cues; (3) open-source models still lag behind closed-source counterparts, though the performance gap is gradually narrowing; (4) overall, the Gemini series stands out for both its ability to handle all input modalities and its strong overall performance across tasks.
D MORE EXAMPLES OF EMOTIONHALLUCER
In this section, we provide more cases from EmotionHallucer, as shown Figure 18 Figure 19
Figure 20 Figure 21 Figure [2] and Figure 23 The resources are available at https: anonymous 4open. science/ r Emot on lucer
E LIMITATIONS AND DISCUSSIONS
While our benchmark provides comprehensive evaluation of emotion hallucinations across modal- ities, several limitations remain. (1) Although we take a lot of strategies to make sure the quality of EmotionHHallucer. there is noise introduced by human annotations. (2) The benchmark currently uses English as the interaction language with LLMs. Although the underlying multimodal data come from diverse cultural contexts, the current version does not yet systematically evaluate how cross- lingual or cross-cus cross-cultural variation affects emotion understanding and hallucination. (3) While we ob- serve emotion hallucination phenomena, the underlying causes, such as pretraining biases, modality misalignment, or lack of emotion-specine specific supervision, remain underexplored. Understanding these root causes is essential for designing more robust and interpretable models. (4) Although we treat emotion understanding and hallucination detection separately, real-world applications often require joint emotion understanding and hallucination awareness, which calls for unified modeling strategies that integrate both capabilities. (5) Open-ended evaluation. Our current benchmark primarily relies on binary QA, which ensures clear ground ground-truth supervision but remains insufficient to capture the open-ended hallucinations in real-world settings.
39

---

Published as a conference paper at ICLR 2026
Table 17: Performance comparison on EmooiionHallnueer--oAAuuuu with additional "Yes/No bias"
analysis.
Model
Yes/No Bias
Accuracy on EmotionHHalluerr-No
Methods
Size
Pet. Diff (~00) FP Ratio (~0.5) Basic Hallucinated 1 Overall
Open-source
LLaVA Liu
2025
34B
-0.55
0.45
50.25
59.52
10.27
Video-ChatGiP ChatGPT Maxt al.
2023
7B
0.09
0,59
61.91
44.67
18.44
Chat-UniVe UniVi Jin
7B
0.09
0.59
60.22
42.37
11.07
I.L.MA-VID)
7B
0.36
0.86
85.74
13.66
5.78
Video-LLaVA in 2023
0.49
1.00
99.70
1.50
1.50
Onellm Han er at
2074
7B
0.31
0.85
87.34
26.02
15.35
LLMMA
Cheng
2024
7B
0.12
0,63
66.55
42.43
18.86
Llama3 1lama3.2-visil 2-visoon
11B
0.21
0.78
83.05
41.28
29.91
Gemma3 Teamet al. 2025
0.15
0.70
78.66
49.15
33,90
Qwen2.5 VL Bai et
0025
72B
0.08
0.63
78.08
62.15
43.02
Mistral-small3 Mistral
AT
2024
24B
-0.11
0.35
53.94
75.17
32.20
Qwen2.5 mmiii
Xu
7B
0.11
0.65
72.39
49.74
25.44
Closed-source
QvQ-Max
Owen
0.07
0.63
78.18
63.39
47.98
GPT-4o Hurst etall
-0.01
0,48
67.10
69.49
40.98
GPT-4.1 OpeнA 2024
0.05
0.58
74.58
64.71
44.47
Gemini-2.51 2.. Goople Desd
212
0.06
0.61
78.55
66.80
50.56
(eemini22.544 Pro
Google DeopMind 2025
0.07
0.64
81.31
67.01
51.58
GPT-5
OpenAl
-0.06
0.40
67.10
78.17
49.35
60
GROgEE IND DEWERR
GPTS
Overall Accuracy
1-248
30
Cama3
mmni-77
20
Video-Cha
Einatton MMA-78
Oneelm m
Chat VRAV 7B
LLaM
Video
NM-78
Mar 2023
May
Jul
Sep
Nov
Jan
Mar
May
Sep
Nov
Mar
May
Sep
Nov
2024
2025
Figure 17: Performance comparison on EmotionFHlllcer-NNoAA accept only the image modality, red denotes models capable of handling additional modalities (e.g., video, audio). Cireles indicate known parameter sizes, while stars represent unknown sizes. The red dashed line marks the top-performing model of the current month.
Blue denotes MLLMs that
40

---

Published as a conference paper at ICLR 2026
Basic: suggested by the affect infusion model, emotion humans feel any moment influences how they react other events. even they are unrelated whatever evoked their emotion.
Facial feedback hypothesis is the hypothesis that posed facial expression of emotion can help generate an emotional feeling.
Basic:
Hallucinated:
Facial feedback hypothesis hypothesis that posed facial expression emation has effect emotional feelings unrelated to emotional experience.
Hallucinated: As suggested by the affect infusion
model, the emotion humans feel at any moment influence how they react to other events, but only those events are related to whatever evoked their emotion.
Figure 18: Two examples of basic-hallucinated pair for Theory Hallucination.
Basic: Appraisal is cognitive evaluation what a stimulus or situation means one's goals, concern,, and well-being.
Basic: Anxiety a general expectation that something bad might happen, without identifying any particular danger.
Anxiety is general expectation that something good might happen, without identifying particular danger.
Hallucinated:
Hallucinated:
Appraisal cognitive evaluation of what stimulus or situation means for one's emotional state.
Figure 19: Two examples basic-lallucinated pair Definition Hallucination.
"Happiness" means different things to people different cultures, even setting aside issues language. People in many Asian cultures prefer a sense of contentment and knowledge that one done duty toward family and community, whereas North Americans place greater value on excitement and achievement.
Basic:
Basic: Simply believing that you have some control situation effectively reduce levels.
Hallucinated: "Happiness" means different things to people in different cultures, even setting aside issues of language. People in many North American cultures prefer a sense of contentment and knowledge that one has done one's duty toward family and community, whereas Asian culture place greater value excitement and achievement.
Hallucinated: Simply believing that you have control over situation can effectively reduce stress levels.
Figure 20: Two examples basic-hallucinated pair for Definition Hallucination.
Category Hallucination
Basic:
The person worried
Hallucinated:
The person exhibits angry emotion.
Figure 21: An example of halcchhaluuinntee pair for Category Hallucination.
41

---

Published as a conference paper at ICLR 2026
Intensity Hallucination
Review Text: As poker enthusiast looking forward to seeing this it. cbr /sebbr bsaaicaaly,, Scotty Nyugens short spots in this film all has going for it,cbr /bwcbr /he characters are unlikeable and annoying, the soundtrack is awful and the plot, well, there isn't one.<b <br /><br /sl honestly got headache and found myself reading the barcode number on the DVD box after twenty minutes was THAT bored. Its actually ashame that Nyugen was in this movie as otherwise wouldn't have wasted $16 buying off Ebay.<br fo-brr /> Take it from me- AVOID offsuit!!! Diro.
Especially
Scotty Nyugen
Basic: This review very negative sentiment, because
Hallucinated: review slighlly negative sentiment, because
Figure 22: An example of aasic-hallucinnted pair for Intensity Hallucination.
Reasoning Hallucination
The man does not display obvious pride or shame about his retail instead adopting defensive stance. From his firm posture and slight nod, it is evident that, despite considering a step down, remains composed. Even tone carries hint defensiveness, his brief pause before speaking suggests he acknowledges the perceived step down.
Basic:
retail job. From his somewhat anxious posture and slight nod, is evident that he considers a step down and struggles to remain composed.
Hallucinated:
displays
His tone carries hint of emberrassment, and brief pause before speaking suggests he
how respond.
The red-haired woman looks to the side at 0:45 because she is calling man's girlfriend mean name avoids speaks, gaze shifts away, body
Basic:
looking at him. angling slightly to avoid direct confrontation.
The red-haired woman looks to the side at 0:45 bocause she calling the man's girlfriend a mean name and avoids looking at him. As she speaks, her shifts away, her breathing quickens, and her body angles slightly to avoid direct confrontation.
Hallucinated:
Figure 23: An example masic-halluccnated pair for Reasoning Result and Cue Hallucination.
42

---

Published as a conference paper at ICLR 2026
F FUTURE WORK
In this section, we outline several directions for extending EmotionHalluuer based on the limitations identified in our current design and empirical findings. Rather than treating these limitations as iso- lated issues, we view them as stepping stones toward a progressively richer and more comprehensive framework for evaluating emotion understanding and emotion hallucination in MLLMs. Each pro- posed direction -ranging from open-ended evaluation, multilingual and cross-cultural extensions, and deeper mechanistic probing to fully integrated joint reasoning builds on the diagnostic foun- dation established in this work. These future developments will enable Emotion to evolve from an initial fine-grained benchmark into a broader evaluation suite capable of supporting ad- vanced research on emotionally reliable and trustworthy MLLMs.
Open-ended evaluation. Open-ended setting represents important next step for extending Emo- tionHallucer beyond binary QA. Although our current binary QA format provides a controlled and reliable diagnostic setting, it cannot fully capture the expressive richness of free-form emotional de- scriptions. To examine its extensibility, we further conducted a pilot open-ended evaluation where MLLMS (Gemini-2.55Fllash in our case) were asked generate free-form emotion descriptions, and a judge model (GPT-40 in our case) was used to evaluate whether the generated descriptions aligned with the basic hallucinated references. We present the independent confusion matrices in Fig- ure 25 and in Figure 26 we report pair-level consistency (a pair is counted as consistent only if the description for the basic item is judged aligned with the basic reference and the description for its hallucinated counterpart is judged not aligned). The overall consistency between binary QA and open-ended evaluation is reasonably aligned, suggesting that our binary paradigm may serve as a reliable proxy for open-ended settings.
Beyond these preliminary observations, open-ended evaluation itself introduces additional chal- lenges. Judge models often struggle to evaluate long and diverse free-form responses, especially when emotion cues are subtle, indirect, or embedded in implicit contextual reasoning. Their judg- ments can also be sensitive to prompt wording and instruction framing, echoing findings from prior work on evaluator instability (Wang et al. 2023a bj Li et al. 2024b Krumdick et al. 2025). These issues collectively limit the reliability of fully automated open-ended assessment. One nat- ural solution is to incorporate human-rated open-ended evaluations, where annotators directly as- sess the correctness, grounding, and hallucination tendency of free-form emotional descriptions. However, high-quality human rating for multimodal emotional reasoning is costly and difficult to scalc-particularly when responses require fine- fnn-grained interpretation of emotion cues or long-form multimodal contexts. Given these constraints, we believe that structured open-ended annotation pro- vides a more sustainable and reliable path forward. Instead of rating entire responses holistically, annotators label key interpretable components -ucch as entities, activated versus non-activated emo- tional cues, emotion category and intensity, and the reasoning pathway, as shown in Figure 24 This structured protocol is grounded in long-standing frameworks from psychology research, which em- phasize that emotional understanding emerges from identifiable verbal and nonverbal cues as well as their integration (Scherer 2003 Knapp et al. 1972). Leveraging these well-established theo- retical foundations ensures that the annotation components are not arbitrary, but follow empirically supported principles of human emotional processing. This approach enables fine-grained identifica- tion of hallucinations at both the span level and the holistic reasoning level. Moreover, structured annotations naturally align with the cognitive stages used in EmotionHallucer, offering a principled extension of the current binary QA framework toward richer open-ended evaluation. We emphasize that this structured scheme is a preliminary design intended to illustrate a feasible direction; in fu- ture iterations of Emotion we plan to refine, expand, and further validate this annotation protocol to support more comprehensive open-ended assessments.
Cross Cross-lingual or cross-cultural evaluation. Building on our current English- English-based interaction pro- tocol, a natural next step is to extend EmotionHallucer into multilingual and cross-cultral settings. A more comprehensive gross-lingual evaluation requires systematically examining how MLLMS' emotion understanding and hallucination patterns change when the interaction language itself varies, inspired by Ponti et al. (2020). Additionally, languages differ in structural properties, affective lexi- calization, and pragmatic conventions -fattors that directly influence how to interpret emotion cues, resolve ambiguity, and map linguistic expressions to affective states. Understanding these language- driven shifts is essential for assessing whether emotional reasoning remains stable across different linguistic interfaces.
43

---

Published as a conference paper at ICLR 2026
Beyond interaction language, emotion communication itself is known to vary across cultures, including differences in display rules, emotion appraisal patterns, and nonverbal expressive norms (Mesquita and Frijda, [992). Future versions of EmotionHallueer will explore culturally grounded extensions-sucl as parallel multimodal samples across cultural groups, culturally spe- cific emotional frameworks, and cross-cultural reasoning tasks-to evaluate whether MILMS can adapt to culturally shaped emotional cues.
Deeper mechanistic understanding. While EmotionHallucer reveals clear patterns of emotion hal- lucination, the deeper causal mechanisms behind these failures-such as pretraining corpus biases, modality misalignment, and insufficient emotion-spectie supervision- remain largely unexplored. Progress toward understanding these root causes requires evaluation foundations that make it pos- sible to probe models from multiple complementary angles. In this sense, the current binary QA forms the core foundation for controlled evaluation, while the planned structured open-ended eval- uation and mosss-iiggal//roos--cul mechanistic analysis rather than endpoints themselves. Together, these foundational tools will sup- port future efforts to move beyond surface error characterization toward principled insights into how emotional information is encoded, transformed, and sometimes distorted within modern MLLMs.
extensions will serve as additional pillars that enable deeper
Joint emotion understanding and hallucination. A coherent integration of emotional percep- tion, inference, and intlluiination-waarenn is an important long-term goal for MLLMs. However, achieving robust joint reasoning first requires a phase-by-phar understanding of where each type of error originates. EmotionHalluuce adopts this early-stage decomposition establish such a foun- dation. Building on this basis, future extensions of EmotionHallucer can move toward joint for- mulations where emotion understanding and hallucination mitigation are assessed within unified reasoning process. Our planned structured open-ended annotation scheme is particularly promising in this regard: by explicitly labeling entities, activated and non- con-activated cues, emotional categories, intensities. and reasoning pathways, it provides the representational scaffolding needed to connect perception, inference, and hallucination detection within single coherent framework. This opens the door to future benchmarks and models that more closely mirror real-world emotion reasoning, where understanding and self-moni operate hand in hand.
G ETHICS STATEMENT
This work complies with ethical standards for AI research. All datasets used in this study are pub- licly available and were originally released under appropriate research or academic licenses. No private or sensitive personal data were involved. Our benchmark focuses on evaluating emotional hallucinations in language and multimodal models; however, acknowledge the subjective and culturally nuanced nature of emotion, and we caution against overinterpreting model outputs in high-stakes sensitive applications. Furthermore, hallucinations in emotional reasoning may lead to miscommunication or emotional misjudgment, particularly in domains such as mental health, education, or comanecr interaction. We encourage future work to incorporate robust safety checks, human oversight, and culturally inclusive evaluations when deploying such models in real- world scenarios.
H REPRODUCIBILITY STATEMENT
We have taken several steps ensure the reproducibility of our work. Our benchmark is con- structed from publicly available datasets, and we provide detailed description of the annotation and processing pipeline in the appendix. The prompts used for MLLMs are fully documented the appendix.
I LLM USAGE STATEMENT
In this paper, we used LLMss to assist with writing (language polishing). In addition, we employed several L.MSs and MI.LMss for evaluation purposes. The paper provides a complete list of the models and prompts used in these evaluations.
44

---

Published as a conference paper at ICLR 2026
rentisy"; "..."
"perceptia Lomeet:;
vverbel.cuess"
Nioiicl..onotion in
"eviderce": ("Gevastated", "beartbroken")
"enotion.reletee www.:reeleeddottteeee.
"emmary"; speaker
losing inportant relacfonship."
Pronverbal .cues":
"visual";
paciallexpreset [
"descript peserritiinn ion"; "teary eyes, downceet gane"
"eend_position";
"sascription": "hoad tilted dowroord*
der_posture";
"coscription": b torso_porture";
"coscription": "lcaning forward slightly"
1, "acn.assture":
*coscription":
hard-cesture";
"eascription":
Pacoustio": !
woice.prosody":
"coscription": "low pitch, slow Leng
b
weice.quality";
"cscrippiion": "trembling, weas voice"
"omotion. inference":
"oategusy"; "sadhwas",
"Inzonsity"; "bigh"
"ngasoring";
"ifferred anotion":
Mescegory"; : "sadhwas",
"incenaity": "high"
'appport hupporting.co togs:"
"verbal"; [
"verbal lexicel emotion words",
Rowcbal inn
"riscal": [
Mooneeclal-cwee Lowes.visal facial expowssson",
Maonverbal, www...veua...................... headd pos Lon", "nonverbal cues.visual storso posture"
"acosssio": [
Mgonverbal
iolvoice prosody", Raonverbat wtw.bblll....seooo io. voice qualiietyy
flicingg srbaeeentt _alaett
Mgonverbal wow.eaallsuueu.e.. cues.visual shoulder, des uret
Manneecbal couss visuall asn. arst
gesture"
"acosatio" []
applanaionn "The Inference aupporsed
Figure 24: Envisioned structured annotation format for open-ended evaluation.
45

---

Published as a conference paper at ICLR 2026
Out Butos Mall
Satus Mall
Carkan Midia
(a) Perception-
(b) Perception-1
(c) Perception- V/S
(d) Perception-Vee V/L
Figure 25: Confusion matrices for ballucination detection in the open- ended setting.
Binary Open-ended Consistency
30.0
Binary
46.57
Open-ended
45
41.33
Consistency (%)
36.54
35
24.13
23
18.41
15
Perception--
Perception
Perception-V/S
Perception-VI..
Figure 26: Binary vs open-ended consistency of hallucination detection across different modalities.
46


Reviews:
[{"review_id": "kUr1OMMSB0", "reviewer": "Reviewer_Fm9U", "summary": "The paper introduces EmotionHallucer, the first benchmark designed to evaluate emotional hallucinations in multimodal large language models (MLLMs). The findings indicate that most current models exhibit significant issues with emotional understanding, particularly concerning hallucinations. By introducing the PEP-MEK framework, the authors demonstrate an average performance improvement of 9.90% in emotion hallucination detection. The study draws from emotion psychology knowledge and real-world multimodal perception, providing a comprehensive evaluation perspective. Overall, the paper contributes valuable tools and directions for future research in the field of emotional understanding.", "questions": "In defining emotional hallucinations, how can researchers effectively balance scientific rigor with interpretability?\n\nIs the current benchmark adaptable to different types of multimodal models to ensure consistency in evaluation?\n\nCan the PEP-MEK framework be further optimized for additional emotional understanding tasks beyond those currently evaluated?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This study fills a critical gap in the evaluation of emotional hallucinations, offering the first benchmark tailored for MLLM emotional understanding. The introduction of the PEP-MEK framework shows significant effectiveness, enhancing model performance in hallucination detection. The authors provide robust experimental data and statistical evidence to support their conclusions, increasing the paper's credibility. The research methodology integrates insights from emotion psychology, ensuring the scientific validity of the assessments. Additionally, the use of diverse multimodal data sources adds practical relevance to the findings.", "weaknesses": "Language Limitation: The study is restricted to English, failing to account for cross-linguistic and cross-cultural variations in emotional expression.\nComplex Definitions: The definitions and classifications of emotional hallucinations may be overly intricate, potentially leading to ambiguity in the evaluation process.\nSuboptimal Performance: Model performance in processing multimodal data, particularly in audio and video emotional understanding, remains inadequate.\nResult Stability: The stability and reliability of certain experimental results need further validation to ensure consistency.\nReal-World Reflection: The assessment of existing models may not accurately reflect their performance variations in practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jVgumul8pt", "reviewer": "Reviewer_aDze", "summary": "The paper introduces EmotionHallucer, a benchmark to detect and analyze emotion-related hallucinations in multimodal LLMs. It evaluates two axes: \n- Emotion psychology knowledge (theory, definitions, empirical findings) \n- Real-world multimodal perception (category, intensity, reasoning cues/results) across text, image, audio, and video. \n\nThe benchmark uses adversarial binary QA pairs—a “basic” item and a matched “hallucinated” item—and counts a prediction as correct only if the model answers both correctly, reducing prompt/length confounds seen in captioning metrics and self-evaluation bias.", "questions": "1. Again, I would like to know a wall-clock and token-cost-wise comparison, along with ablations for each PEP-MEK component and per-subcategory gains.\n\n2. Bias balancing details. The authors stated that the yes/no is balanced. Can you share the exact balance per subcategory and modality, and how you prevented position or wording biases between paired items? \n\n3. Open-ended evaluation. Beyond the pilot LLM-judge setup, do you plan a human-rated open-ended benchmark slice to validate the binary proxy and reduce judge-model bias?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This is the first benchmark that is dedicated to emotion hallucinations, spanning both psychology knowledge and multimodal perception; prior hallucination suites are general-purpose. The seven subcategories (theory/definition/finding; category/intensity/reasoning cue/reasoning result) make the construct very concrete. And, the adversarial paired QA design (basic vs hallucinated) is what I call a neat, low-variance way to test detection of hallucination, beyond typical caption/LLM-judge setups. \n\nThe paper is well structured, with clear logic, a well-defined task taxonomy, examples, and an easy-to-follow pipeline; the appendices document the collection/annotation and PEP-MEK details; ethics and reproducibility statements are included. \n\nNon-trivial scale and coverage, with broad evaluation and clear metrics, which is very nice.", "weaknesses": "1. Adversarial pair construction & QA artifacts. The process risks introducing superficial cues between the basic and “hallucinated” versions. Report inter-annotator agreement, pair-level quality controls, and checks against annotation artifacts (e.g., spurious lexical markers).\n\n2. Latency/compute overhead and failure cases are not quantified. A wall-clock and token-cost-wise comparison is needed here, along with ablations for each PEP-MEK component and per-subcategory gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n2YpkBjYXG", "reviewer": "Reviewer_abcp", "summary": "EmotionHallucer introduces the first systematic benchmark for evaluating emotion-related hallucinations in multimodal large language models (MLLMs). The benchmark is grounded in emotion psychology and real-world multimodal perception and uses adversarial binary QA pairs (basic vs. hallucinated versions) across modalities (text, image, audio, video). The authors evaluate 41 MLLMs (open- and closed-source) on multiple subtests (e.g., perception-level, psychology/knowledge-level, reasoning-result), report quantitative metrics (including bias/FP measures and separate accuracy for basic vs. hallucinated items), and analyze model behaviors. They find that many models remain vulnerable to emotion hallucinations, that closed-source models often outperform open-source ones, and that models are typically better at explicit emotion-knowledge tasks than at grounded multimodal perception and inference. To improve detection, the paper proposes PEP-MEK, a multimodal+emotion-knowledge augmentation framework that boosts hallucination detection and is evaluated via ablations. The paper provides dataset construction, annotation procedures, limitations, and plans to release resources on GitHub.", "questions": "SEE WEAKNESS", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "### A novel benchmark (EmotionHallucer) specifically targeting emotion hallucinations in MLLMs:\n- Covers multiple modalities and multiple diagnostic levels (perception, emotion knowledge, reasoning results).\n- Uses adversarially constructed basic vs. hallucinated QA pairs to probe hallucination propensity.\n### Large-scale empirical evaluation and analysis:\n- Systematic evaluation of 41 MLLMs (both open- and closed-source) with detailed metrics (Pct. Diff, FP Ratio, separate Basic vs. Hallucinated accuracy, overall scores).\n- Insights showing systematic weaknesses (e.g., multimodal perception and reasoning produce more hallucinations; closed-source models typically fare better).\n### Proposed mitigation/analysis method (PEP-MEK) and ablation studies:\n- PEP-MEK integrates psychology-grounded emotion knowledge and perceptual cues to improve hallucination detection.\n- Demonstrated consistent improvements across models and includes ablation studies showing the contribution of emotion knowledge and other components.", "weaknesses": "### Annotation noise and scope limited to English and certain datasets:\n- The benchmark relies on human annotation (e.g., creating hallucinated variants), admitting annotation noise.\n- The dataset is English-only and does not address cross-lingual or cultural variability in emotional expression.\n### Partial exploration of root causes:\n- While the paper documents hallucination phenomena and correlates them with modality and model class, it does not deeply investigate underlying causes (e.g., pretraining biases, modality misalignment, lack of emotion-specific supervision) or provide mechanistic explanations.\n### Separation of evaluation axes and incomplete real-world integration:\n- Emotion understanding and hallucination detection are treated separately, whereas practical systems need integrated capabilities (joint perception, inference, and hallucination-awareness).\n- Temporal and long-form audio/video integration remain challenging and less explored; the benchmark and methods may not fully capture these complex real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e9YhBIH2nP", "reviewer": "Reviewer_QF5h", "summary": "This paper introduces EmotionHallucer, a novel benchmark designed to evaluate emotion-related hallucinations in multimodal large language models (MLLMs). The benchmark spans four modalities (text, image, audio, video) and is organized around two main dimensions: emotion psychology knowledge and multimodal emotion perception. The authors also propose PEP-MEK, a reasoning-enhanced framework aimed at mitigating emotion hallucinations. Extensive experiments on 41 LLMs and MLLMs reveal widespread emotion hallucination issues, particularly in multimodal perception tasks, and demonstrate the effectiveness of PEP-MEK in improving model robustness.", "questions": "- Have the authors considered extending EmotionHallucer to include non-English or cross-cultural emotional expressions? If so, what challenges do they anticipate?\n- Could the authors provide more detailed analysis or examples of cases where PEP-MEK fails? Understanding its limitations could help guide future improvements.\n- How might the benchmark be adapted to support more open-ended emotion generation or reasoning tasks, beyond binary QA?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Novel Benchmark: EmotionHallucer is the first comprehensive benchmark targeting emotion hallucinations, with a well-designed adversarial evaluation protocol.\n- Multimodal Coverage: The benchmark spans text, image, audio, and video, enabling a holistic assessment of MLLMs’ emotion understanding capabilities.\n- Large-Scale Evaluation: Experiments on 41 models offer broad insights into current limitations and trends.\n- Practical Mitigation Framework: PEP-MEK demonstrates consistent improvements across models and modalities, offering a simple yet effective approach to reducing emotion hallucinations.", "weaknesses": "- Limited Cross-Lingual and Cultural Analysis: The benchmark is limited to English, and there is no discussion of how cultural or linguistic differences might affect emotion hallucination patterns. This limits the generalizability of the findings.\n- Superficial Error Analysis: While the paper reports performance drops in open-set and multimodal settings, it does not deeply investigate the root causes of failures (e.g., which types of cues are most often misinterpreted).\n- Benchmark Design Limitations: The binary QA format, while useful for controlled evaluation, may not fully capture the complexity of open-ended emotion understanding in real-world scenarios.\n- Lack of Human Baseline: The absence of human performance comparison makes it difficult to gauge the practical significance of the model results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SIcrT5EbZe", "reviewer": "Reviewer_jLHy", "summary": "This paper introduces EmotionHallucer, the first benchmark designed to evaluate emotion hallucinations in MLLMs. The benchmark assesses two complementary aspects, emotion psychology knowledge and multimodal emotion perception. Evaluating 41 models using an adversarial binary QA framework, the paper reports three main findings and propose PEP-MEK, a framework that integrates modality-specific and emotional reasoning to mitigate emotion hallucination. PEP-MEK achieves an average 9.9% improvement across selected models. Therefore, EmotionHallucer provides a benchmark and valuable insights for advancing emotionally reliable MLLMs.", "questions": "See Weakness.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe paper introduces the first study of emotion hallucinations in MLLMs, addressing a crucial yet previously unexplored aspect of emotion understanding.\n\n2.\tEmotionHallucer is designed based on emotion psychology and real-world emotion perception, spanning four modalities and multiple evaluation settings. The adversarial QA framework provides a controlled protocolto assess emotional reasoning errors.\n\n3.\tThe benchmark evaluates a large number of models, providing comprehensive insights into the current state of MLLMs. \n\n4.\tPEP-MEK is compatible with both open- and closed-source models through standard APIs, offering a plug-and-play approach for emotion hallucination mitigation.\n\n5.\tThe paper is well-structured, easy to follow.", "weaknesses": "1.\tWhile the adversarial binary QA framework provides strong objectivity and control, and the authors have additionally performed consistency checks between binary and open-ended results, this setting may still not fully capture open-ended emotion hallucinations in real-world generative scenarios. It would be valuable to further discuss how this limitation could be addressed, and what directions the authors plan to explore in future work.\n\n2.\tThe benchmark primarily focuses on English-language. Although some data sources contain diverse cultural content, explicitly incorporating cross-cultural emotion understanding tasks would further enhance its generalizability.\n\n3.\tWhile this trade-off is reasonable given the improvement in reliability, a more detailed discussion on the efficiency–accuracy balance and potential optimization strategies would further strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)