AVERE: Improving Audiovisual Emotion Reasoning with Preference Optimization
Authors: Ashutosh Chaubey
Jiacheng Pang
Maksim Siniukov
Mohammad Soleymani
Venue: ICLR 2026 Poster (2026)
Keywords: MLLM
Emotion Recognition
Multimodal Reasoning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a benchmark for audiovisual emotion reasoning and propose a novel preference optimization technique for robust MLLM emotion reasoning.

Abstract: Emotion understanding is essential for building socially intelligent agents. Although recent multimodal large language models (MLLMs) have shown strong performance on this task, two key challenges remain: (i) spurious associations between emotions and irrelevant audiovisual cues and (ii) hallucination of audiovisual cues driven by text priors in the language model backbone. To quantify and understand these issues, we introduce **EmoReAlM**, a benchmark designed to evaluate MLLMs for cue–emotion associations, hallucinations and modality agreement. We then propose **AVEm-DPO**, a preference optimization technique that aligns model responses with both audiovisual inputs and emotion-centric queries. Specifically, we construct preferences over (i) responses exhibiting spurious associations or hallucinations and (ii) audiovisual input pairs guided by textual prompts. We also include a regularization term that penalizes reliance on text priors, thereby mitigating modality-specific cue hallucinations. Experimental results on DFEW, RAVDESS and EMER demonstrate that our method significantly improves the performance of the reference baseline models (6-19\% of relative performance) in zero-shot settings. By providing both a rigorous benchmark and a robust optimization framework, this work enables principled evaluation and improvement of MLLMs for emotion understanding and social AI.

Published as a conference paper at ICLR 2026
AVERE: IMPROVING AUDIOVISUAL EMOTION REA- SONING WITH PREFERENCE OPTIMIZATION
Ashutosh Chaubey, Jiacheng Pang, Maksim Siniukov & Mohammad Soleymani Institute for Creative Technologies University of Southern California Los Angeles, CA 90007, USA
achaubeyEusc. soleymani peirmaniiiit...m usc edu
ABSTRACT
Emotion understanding is essential for building socially intelligent agents. Al- though recent multimodal large language models (MLLMs) have shown strong performance on this task, two key challenges remain: (i) spurious associations be- tween emotions and irrelevant audiovisual cues (reasoning errors) and (ii) halluci- nation of audiovisual cues (perception errors) driven by text priors in the language model backbone. To quantify and understand these issues, introduce EmoRe- AIM, a benchmark designed to evaluate MLLMs for cue-emotion associations, hallucinations and modality agreement. We then propose AVEm-DPO, pref- erence optimization technique that aligns model responses with both audiovisual inputs and emotion-centric queries. Specifically, we construct preferences over (i) responses exhibiting spurious associations or hallucinations and (ii) audiovisual input pairs guided by textual prompts. We also include a regularization term that penalizes reliance on text priors, thereby mitigating modality-specifie cue hallu- cinations. Experimental results on DFEW, RAVDESS and EMER demonstrate that our method significantly improves the performance of the reference baseline models (6- 19% of relative performance) in zero-shot settings. By providing both a rigorous benchmark and robust optimization framework, this work enables prin- cipled evaluation and improvement of MILMs for emotion understanding and social AI. Code, models and benchmark our project page eivere-ilrrgiitubb..
INTRODUCTION
Emotion understanding is essential for social Al agents to generate tailored responses and fos- ter meaningful human-mate machine interactions (Chaturvedi et al. 2023 Kolomaznik et al., 2024] Elyoseph al.) 2024). Emotion perception also finds applications domains such health (Bal-) combe & De Leo. 22022; Litendahl et al., 2025) and education (Salloum et al.) (025), where appro- priately responding affective states improve therapeutic alliance and learning outcomes.
Traditional multimodal emotion recognition methods (Sun et al., 2023 Wang et al.) 2023 Chen al. (0244) lack interpretability, as they only perform classification without grounding responses in audiovisual cues. Moreover, emotion is a complex and multi-componential construct that ex- tends beyond the basic emotion labels that can be assigned by supervised learning methods (Ekman & Friesen 1978: Scherer 2005). To address these challenges, recent approaches leverage multi- modal large language models (MLLMs) to generate detailed emotion descriptions for interpretabil- ity (Cheng et al. 2024 Huang et al., 2025a) and to output emotion-related keywords that cover spectrun of emotional states (L.ian et al. 2024 2025a).
broader spectrum
However, audiovisual MILMs are susceptible to hallucinations, frequently generating inaccurate or fabricated responses (Li et al. 2023, Sahoo et al.) 2024). In the context of emotion understanding, illustrated in Fig. First, these models often ground emo-
they face two critical bottlenecks, tion predictions on irrelevant cues (e.g.. attire color, ambient noise) reasoning errors. Second, they tend to hallucinate additional cues in their responses to justify emotions perception errors. Such hallucinations are largely driven by text priors in the language model backbone, which bias the model to include that commonly co-occur with specific emotions (e.g., associating tears

---

Published as a conference paper at ICLR 2026
the emottin experienedd by the person using audio-nikudl cues.
Ground Trults
The in the video uppeurs sud. His downaward ouge and furnaved brucs cumbined with
(rugm,, suggestt custempplltio tcouthed state The wontoinns sinule simwwridol long that dloes not contribute to the overall melancholic mood.
The mon in the da exprrteninng sodhesse Fe hos o Sight frowc ond furnowed krwws strwong
Output
ondiccaoon
teypley leurs from eyes and
Bad
suggests that is enyingg The squane patterns in the backeround wall support the
pieguticlly fot the atmosphere.
Emotion releeen Video-rdeeenn
Amsotion- lereeeeunn Video-rsievent
Amotion relevent Iddo-Srredevant
V
X X
aLour holluctaations
Figure 1: Existing MILMs (i) include spurious associations between AV cues and emotions rea- soning errors (blue highlight) and (ii) hallucinate AV cues explain emotions perception errors (red highlight). AV: audiovisual.
with the sound of crying). The scarcity of high-quality, emotion-specific instruction tuning datasets
(Cheng et al. 2024 Lian al., 2025a) further aggravates these issues. Addressing these challenges is essential, as they compromise the reliability of MLLM agents in social interactions and complex emotion reasoning scenarios.
Existing emotion reasoning benchmarks (Lian et al. 2023h 2024) lack the diverse and complex samples needed fully evaluate these issues. Additionally, current audiovisual hallucination bench- marks (Sung-Bin et al. 2025 Leng et al. 2025) predominantly focus on object-level hallucinations in audio or video, rather than on emmotinn-ppcifee reasoning. Moreover, many existing MLLMs (Cheng et al., 2024 Lian et al. (225a)) rely on two-stage evaluation pipelines involving an external (often proprietary) LLM such as GPT (OpenAI al., (024). making replication and benchmarking difficult. To address these limitations, we introduce the EmoReAIM benchmark, comprehensive suite of multiple-choice question-answer (MCQA) tasks designed evaluate audiovisual emotion reasoning, modality agreement and ballucination-rellae stress tests (Fig. 2). The MCQA format en- ables transparent, reproducible and scalable evaluation of MLLMs on emotion-centrr tasks without requiring additional L.MMs during inference.
Evaluation of recent MLLMs on our benchmark highlights spurious association and hallucination issues outlined in Fig. [ To address these limitations, we propose AVEm-DPO multimodal direct preference optimization (DPO) technique (Rafailov et al., 2023) to enhance the emotion rea- design explicit prompt-based audiovisual input
soning capabilities of MILMs. In particular, preferences to mitigate hallucinations caused by cross-modal interactions. We also introduce text- prior debiasing, which penalizes policy reward for responses to text-only inputs. Together, these techniques significantly improve the performance of reference MLLMs, outperforming all base- lines in zero-shot evaluation on both our benchmark and existing emotion recognition and reasoning datasets.
To summarize, the main contributions our work are:
We introduce the EmoReAIM benchmark with 4000 human-verified MCQA samples to evaluate emotion reasoning and emotion-related hallucinations MILMs, highlighting bottlenecks such as spurious audiovisual cue associations and hallucinated cues for explaining emotions.
We propose AVEm-DPO, direct preference optimization technique that enforces explicit prompt-based modality preferences and reduces text-only model biases through regularizer that penalizes over-reliance on text priors.
We conduct extensive evaluations of existing MLLMs, demonstrating current bottlenecks and showing the superior performance of the proposed DPO-trained models in zero-shot settings.
2
RELATED WORK
MLLMs for Emotion.
While general MILMs (Zhang et al.) 2024 Lin et al.
2024
Zhang et al.,
2025a
Xu et al. 2025b; team 2025) show non-trivial emotion recognition ability (Cheng et al,
2024
several studies pursue domain-specific instruction tuning (Xie et al.
2024
Chaubey et al.
Yang et al. 2025). EmotionLLaMA (Cheng et al.) 2024) is an audiovisual LLM for emotion recognition and captioning, finetuned on limited dataset (==30k samples). Lian et al. (2024) intro- duces open- vocabulay emotion recognition (OV-ER), and AffectGPT (Lian et 2005a) employs lightweight audiovisual fusion projector for OV-ER. EmotionQwen (Huang et al. 2025a) im-
2025
2

---

Published as a conference paper at ICLR 2026
Modality Agreement
G: Dees the audio and video suggest the same emotion for the man?
(6) (3)
Emotion Reasoning Stress Test
Q: Does The man's roloxed facial expression suggest his joyful froling
video? (A) Yea No
No Mullaciiniinn
Autio saption: sadio contaiin oan pansiigg and thes losphing louords
feard
musi....
Yeu
Q: Does the presence bicycims support the man's happiners the video' (A)
Vinuaa
Video contion:: witnee and retinn hie crmes besaand the and of cidon uhis jauminn and poocilly siring mimvinng dheycle parked uitest oh nubiing...
buildng uiit doirs, man's
Spuriose Vovel Veasl Cheditscciion Asoociaties
Q: Does the man clopping suggest his happiness in the video? (A) Yea (B
Emotion Reasoning Basic
O: What does the marks body benguage suggeet about his tealings?
Doos laughing his emotional the video? (8) You No
(A) The man can bo scen dicpping ing pugpesiig he exporiencing (oy. Mayylis [D) The glass bull ding oolls ro othvely dizon and new ggodting an overall
() The man standing
supgest hoppiincs..
No Rotheiiaiion
presence background
support the
poadititt in the soams
happiness
O: What doms the mar''s speech suggest about his amotional atate? (A] The man sayy uill pood adout my cameer" ausppeainng ng hepperanos
Sparious Juatio Smuthon Jusocsation
(91 The man laughing towards the and the audio auppeating joy.
Does man's unterance am exonted see Manie suggest clated
micdle
siggesting inieaal
Figure 2: EmoReAIM Tasks. In addition to basic emotion reasoning, we include tasks for Modality Agreement and Emotion Reasoning- Stress Test to test spurious cue-emotion associations and cue hallucinations. Red text is a hallucinated cue, blue text is an irreionn cue and green text is a cue relevant for emotion understanding. Correct choices are underlined.
proves emotion understanding while preserving general skills via a mixture-of -f-eeeerstss router. Han al.] (2025b) use modality-specifff experts with attention reallocation to handle audiovisual emo- tion mismatch, and Wen et al. (2025) leverage retrieval-auge -aggeeted generation with Muain-of-thoughe for better reasoning. contrast, we improve reasoning through multimodal preference optimization and text-prior debiasing.
Rigorous evaluation of multimodal emotion reasoning requires diverse, systematic benchmarks. Lian et al. (2023b) provide detailed descriptions of transcript, audio and visual cues for emotion reasoning, which can support GPT-based evaluation (Cheng et al.] 2024 Han al. 20256)) Xing ct al. (2025) present a holistic benchmark spanning text, image, video and audio hallucinations lated to emotions. Our benchmark instead focuses squarely on audiovisual emotion understanding with standardized pipeline and tasks beyond hallucination, including modality agreement and spu- rious cue- que-emotion associations.
Preference Optimization. Direct preference optimization (DPO) Rafailov et al. 2023 Liu al.
2025a) was introduced to align LLMs to human preferences. DPO has also emerged as a leading approach for mitigating hallucinations in vision LLMS YYu al. 2024 Wang al.] 2024 Sarkar ctt al. 2025 Huang et al. 2025b; Liu et al., 2025b Zhang et al. 2025b). but its use in audiovisual LLMs remains limited. VistaDPO (Huang et al. (025b) increases video LLM robustness by building instance-level, temporal-level and object-level preferences of video inputs. Sun et al. (2025) apply process DPO for step-wise audiovisual reasoning. while Tang et al. (2025) use multi-round DPO for audiovisual captioning. Luo et al. (2025) employ DPO for emotional speech aligmment to improve Omni-LLM outputs. Ye et al. (2025) construct multimodal preference data via ambiguity scoring. and Lian| (2025) use group relative policy optimization for AffectGPT. Concurrently, Omni-DPO (Chen et 2025) studies audiovisual modality preference. Our method differs by constructing prompt-basee audiovisual preference pairs for fine- inn-grrined alignment and by introducing text-prion debiasing to reduce hallucinations in MI.MMs.
3 EMOREALM BENCHMARK
Fig. 2 shows different tasks present in the proposed EmoReAIM Benchmark. The goal of this benchmark is to test the reasoning capabilities of MLLMs to judge the emotion experienced by the character in the given video, specifically over the following verticals (i) reasoning the correct emotion with relevant audiovisual cues (ii) identifying whether the inferred emotion from audio and video are in agreement (iii) testing the association of perceived audiovisual cues with different
3

---

Published as a conference paper at ICLR 2026
Video Caption shocrp- sareen unds paserrity The acescals your sypoer in Eeree...........................
Emotion Reasoning -Basis
Chockiatt
Sadness
df Ouession only hos one sho aenest
Sulccatt
Modalliy Anreament
Cure
GT Emodion
Audorneeall Gume are acceasent
Sadusss
Caption
Emolion Reazening Stress
evelis cxvlle tode and tame
87
Wiall Reteeee
Uiall Spanom Timal leathe
conton.
toase the and.
Audoo LD
Transcript
jemptr)
Gandion-hazed Emotion Prediction
Fv. www.s//eeeeeeooo.........
Audio-Vaaaal App!
Vizual
Humman
Captioning
and GT VariBcation
Automatic Tamalate Based Oud Generation
Varification
Figure 3: EmoReAIM Creation Pipeline. We first disentangle the audiovisual information by separate captioning and verify the cues with text-based emotion prediction find amotion-relevane cues. Finally, GPT-4o is used to generate MCQA samples that are later verified manually.
emotions
(reasoning errors) and (iv) testing audiovisual hallucination due to text-only emotion-
related biases (perception errors).
3.1 TASK DESCRIPTIONS
Emotion Reasoning- Basic. This task evaluates an MLLM's ability to identify and reason about the emotion experienced by a person in a video by linking appropriate audio speech transcription, tone) and visual (e.g., facial expression, body language) cues to specific emotions. To increase difficulty, the ground-truth emotion is not provided in the question. Incorrect options are constructed by modifying the correct answer to include either emotion-irrelevant cues present in the video or hallucinated cues that falsely justify the emotion.
Modality Agreement. This task assesses whether the audio and visual modalities convey the same emotional state. Unlike AVHBench (Sung-Bin et al., 2025), which focuses on general cross-modal alignment, this task specifically targets agreement emotional interpretation across modalities.
Emotion Reasoning- - Stress Test. MILMs are vulnerable to both reasoning errors and perception errors: the former lead the model to base its responses on irrelevant audiovisual cues present in the input, while the latter cause it to rely on hallucinated cues that not actually present. This task probes MLLMS for susceptibility to spurious cue-emotion associations (perception errors) and hal- lucinated explanations driven by language model biases (reasoning errors). Each question follows the format: "Does the {audio/visual cue} suggest {emotion} } of the character?", ". For a modality X, define three sub-tasks: (i) No Hallucination correctly associating an audio/visual cue with the appropriate emotion. (ii) Spurious X Cue-Emotion Association - linking emotion-irrelevan cues to the correct emotion. (iii) Emotion-Relevant X- <-Hallucination - associating the correct emotion with a hallucinated cue that typically co-occurs with it. For example, in Fig. [ a man is not clapping (per the visual caption), yet a hallucination-base question associates clapping with happiness- clapping is commonly linked to positive emotions like joy.
3.2
AUTOMATIC DATA CREATION
Fig. shows the automatic pipeline used to construct the EmoReAlM benchmark. Our approach builds on existing manually labeled audiovisual emotion recognition datasets that provide single- word emotion annotations. For each video, we first use an MLLM to extract detailed audio and visual captions separately, effectively disentangling the two modalities. These captions describe both emotion-relevant and irrelevant cues. To verify whether either modality reflects an emotion, we prompt an L.M to classify the audio and video captions independently into one of seven categories of neutral, in addition to six basic emotions Ekman (200)). Samples are discarded if neither caption yields a valid emotion label. Given the validated captions and emotion label, we then generate tailored prompts and question templates for each task described in Section 3.1 This modality- wise captioning and emotion verification process ensures the construction of high-quality, verifiable MCQA pairs that reflect meaningful audiovisual cue associations. More details and prompts are present in Appendix B
Details. All videos are sourced from the DFEW dataset (Jiang et al. 2020). GPT-4o (OpenAl et al.
2024) is used for caption extraction, emotion classification and question- answer pair generation.
4

---

Published as a conference paper at ICLR 2026
3.3 POST-PROCESSING AND HUMAN VERIFICATION
We employ GPT-4o OpenAl et al. 2024), Gemini-2.5 (Gemini- et al. 2025) and Qwen-2.5 (Qwe--Team al. 2025 to predict the correct answer the generated questions just by using question text as input. We remove all the QA pairs for which all the models identified the correct answer just with the text information. Finally, since the QA samples are generated automatically leveraging MILMs, which can hallucinate themselves, we perform human verification over the samples generated by recruiting over 470 participants using the crowd-sourcing platform Prolific Details are present in Appendix B.2
3.4 BENCHMARK STATISTICS
Table summarizes the data statistics of the pro-
Table I: EmoReAIM Benchmark Statistics.
posed EmoReAIM Benchmark, which comprises a total of 4,000 questions over 2,649 unique videos. Samples from the benchmark are present Ap- pendix B.5 Importantly, for tasks which always have a fixed set of answer choices (Emotion Reason- ing- Stress Test and Modality Agreement- Yes/No). we ensure that there is a uniform distribution of cor- rect answer texts
vid.
Rand. Acc.
Reasoning Basic
Audio
972
784
25%
Visual
1024
883
25%
Modality Agreement
456
456
50%
Audio
820
655
50%
Reas. Stress Test
Visual
728
593
50%
Total
4000
2649
the possible answer choice texts. Additionally, ensure that the distribution of emotion labels over the videos in the benchmark matches the video source dataset (refer to Ap- pendix B.3 for details). It is also important note that EmoReAIM is only used as test set to evaluate the reasoning capabilities of MLLMs, and we use a different dataset for preference opti- mization (refer Section 4.3
4 AVEM-DPO
Direct preference optimization (DPO) Rafailov et al. 2023) aligns LLMs to human preferences, bypassing the need to develop reward model. In the context of audiovisual LLMs, given reference model Tra we can reformulate the DPO objective to learn an optimal policy my as the following,
max B(a,ns)) [r(a, 0, x. v)] ( a, v. .t) mer a, v,2)
where (a, v) is audiovisual input, x is text prompt, is text response and r(a, x. y) is the re- ward function for given input-output pair. Optimizing Eq. find optimal policy results in the following reward formulation,
Toly a, U, x) + log (aa, v,
r(a, U. x. = :log Neel(y a, v.x))
where Z--) is the partition function derived in Rafailov et (023). With access to a preference dataset Dier with samples (a, n, v. You yt) and using the Bradley-Tere -erlyy preference model (Bradley & Terry 1952) to model preference of chosen response (yw) over rejected response (yt), the final DPO objective becomes
To (y a, U.
B log To a, v,
LDPO -E (a,vv.. DD log
( log
(3)
(ee a, v.
Fee a, U,
4.1 MULTIMODAL PREFERENCE OPTIMIZATION
Naive DPO (Eq. applied MILMs, when relying only on response preference, often causes the policy model to overfit to the input prompt x while neglecting the multimodal inputs during alignment (Wang et 20244 Sarkar et al. 2025). To address this limitation, preference optimization can be extended to incorporate audiovisual inputs as follows:
Cap
FR. log or (u((ww Pw, 0z, JT yw)) ))]: u(-)) log
Te Te(y Oher Uers 21
B log
ne (y
ar Ui,
Teet (Yor Oeey 2. Feetl (Yrr 04 21,
5

---

Published as a conference paper at ICLR 2026
"Do the andia and siden the some state of the persou?"
at" "Whut does the perso'' specoli indicate
xt" "How does the character's body langwage
Modallty PrailiyPPrrtie Preferance
their fouling""
support angry stone?"
hhampt-baedd.
Audio Visua Proeegi
Audio Maai---iian milatll Prompe
Vidoo releted Prompp
Cay
Vw
Cuy
Cg
12
s:: "Hoe does the mon's fariial expressiion comary his emotiionl zuate be like vervee""
Emotion--aaeed Response Preferrnce
Og Vw
Goa
Chep Vwl
"His eyer are wide, eyebrens frequently drawn togettier, and mouth shifts show feeth, refiecting axserivennss and anger""
"The mon's 's nitire the green rventer complements subbly hinting the mood of onger.
"Mis clenched reflest strong feeling of ongerr
Yn
1/' X
yfr D
Figure 4: Preference pairs in AVEm-DPO. (Top) Fine- tine-grained preference over modality input based current prompt. (Boltom) Each chosen response yw has two rejected responses y)" rele- vant to the video but with spurious emotion association and yf" irrelevant to the video (hallucinated) but related to the emotion.
where (a Un) and (a,, 01) denote the chosen and rejected multimodal inputs. This objective en- sures that the policy model aligns its response you to the correct (chosen) audiovisual input (awey Vw)-
Prompt-based Modality Preference (PMP). While Eq. 4 enforces preference over non-tert in- puts, in the case of audiovisual (or "ommi") LLMs the input prompt x may relate to both audio and visual modalities, or to only one of them (m E .M A.V)). This often leads to cross- modality-inducee hallucinations in MILMs (Sung-Bin al. 2025), where response concerning one modality r is spuriously influenced by another modality m2 E M {mı).
prompt
To mitigate this issue, we construct the preference dataset Dev with fine-grained modality-level preferences conditioned on the input prompt x"" as illustrated in Fig. (Top). For example, for query specific one modality x" (e.g.. visual: "How does the character's body language support their angry state?") we modify only the corresponding input(s) modality m (i.c. visual) in the rejected pair, thereby enforcing that the model's response remains grounded in that modality. Thus, our prompt-based modality preference objective becomes,
Cav cay-promp: = EEl log (u(aw: Uws af
PMP PMP oj
.x .3w.))]
(5)
where af PMP = Gwn, m = and of Unn iff m = A. We perform multiple forms negative sampling for constructing (at, vt) (see Section 5.2 however, because our task is emotion reasoning, the best results were achieved when we choose the rejected audiovisual input to be a sample with an emotion different from the chosen input (aw, U..)
Emotion- based Response Preference. To mitigate spurious cue-cmotion associations and hallu- cinations described in Section 1 for a given input (aur v 2)) construct rejected responses that are variations of the chosen response Yus as illustrated in Fig. [ Bottom). Specifically, vi" in- cludes an audio/visual cue that is relevant to the audiovisual input but does explain the emotion, whereas yf" introduces audio/visual cues related to the emotion but absent from the audiovisual in- put (hallucinated). Following Huang al. (2025b), we assign weights these rejected responses in the DPO loss in Eq. as.
(T dis Saas
Tu(uf Tar(ss
(ow in of wpT mote log
log
(E(ur))))
where Ber + Ber 1. This formulation establishes strong contrasts between chosen and rejected responses, encouraging the policy model to ground its outputs in correct and emotion-rclevant au- diovisual cues. Unlike Huang et al. (2025b), however, we do not include completely irrelevant responses rejections in DPO based empirical findings in Appendix E.6
4.2 TEXT PRIOR DEBIASING (TPD)
Audiovisual LLMs have strong text priors that cause them to hallucinate and include cues in their response, which usually occur together (e.g., the presence of crying person accompanied by the

---

Published as a conference paper at ICLR 2026
sound of crying). To suppress such behaviour, we propose to penalize the reward r(a, v, derived in Eq. (2 to generate the response using only text input as follows,
TT a, v, + 3 log Z(a, 2) - TTPD log Tiext (3)
r(a, 6, x, u) = 3 log Te a, v,
x)
where Tixt is trained language model and TTPD is a hyperparameter. In our experiments, we choose Tixtt to be the language model backbone in Tref- This penalty ensures that the responses that are explainable purely by text priors get discounted and responses supported by audio/video get relative credit. Plugging Eq. 7 in the Bradley Tenry model results in the following objective,
COPO-TDD -E,
log
log
(Mt (a, w,x))
log Te(s) v,x)]
Test (Da (a, v,a))
Tar(21 (a),,
MIOO log (yal a) log (we [=))
where (a, 2) denote (aw, Uw) for simplicity. During training, stop gradients through Trext it is just used to identify the text priors that a language model has. To maintain the text-only capabilities of the language model backbone, we attach LoRA module (Hu et al. 2022) it for training. To accommodate two rejected responses, we perform scaling similar to Eq. (6) on the rejected responses
in the TPD term as described in Appendix C.1 (Eq. (8)) to get the final TPD objectíve C The final objective function of AVEm-DPO is as follows,
LAVEm-DPO = PPO-TPD C prouppropp
(9)
where Xan is hyperparameter. Implementation details are present in Appendix C.3
4.3 PREFERENCE DATA
For AVEm-DPO training. we construct preference data using a pipeline similar Fig. This prefer- ence dataset is different from EmoReA/M, which we exclusively use for testing. We use MAFW (Liu al., 2022) and subset of MER2025 (Lian al.. 2025b) Track- I train set as the source datasets to create preference samples. We prompt Gemini-2.5 Gemini Team et al. (2025) to generate variations of the correct answers (chosen responses) to the questions where the audiovisual cue is altered be either spurious emotion-related video-relevant cue (u)T) hallucinated cue related to the emo- tion present (yf"). Note that we do not perform any manual verification on the generated data, which still results in performance gain demonstrating the efficiency of the proposed approach. Details in Appendix C.2
5
EXPERIMENTS
Datasets Metrics. For EmoReAIM benchmark, we report the average accuracy per task for all the tasks. For tasks with Yes/No responses, additionally report the precision. recall and F1 score following previous multimodal hallucination benchmarks (Sung- Bin et al.) 2025 Li et al. 2023). Beyond FmoReAlM, we also evaluate on established emotion recognition datasets DFEW (Jiang ett al., 2020). RAVDESS (Livingstone & Russo 2018). MER2023 (Lian et al. 2023a)- and the emotion reasoning dataset EMER (Lian et al. 2023b). None of these datasets used in training to ensure zero-shot evaluation. Following prior work (Cheng et al.] 2024 Han et al. 2025b), we report unweighted and weighted average recalls for DFEW and RAVDESS and weighted Fl for MER2023. For emotion reasoning, we adopt GPT-based evaluation (Cheng et al. 2024), comparing generated responses against ground truth. In addition to clue and label overlap. we assess two dimensions: (i) spurious cue-emotion asxociations, where irrelevant cues are linked to emotions, and (ii) hallucinatory cues, where non-existent audiovisual cues are fabricated. For all metrics, higher values indicate better performance. Further details are provided in Appendix D.]
Reference models. We use two audiovisual MLLMs as reference - EmotionLLaMA (Cheng et al., 2024) and our own developed base model. Our model is similar to LaA in architecture with changes to the audio encoder (whisper-large -3 Radford et al. 2023) and video encoder (Lan- remove the text (subtitle) input branch to
guageBind (Zhu et al. 2024)). For EmotionLLaMA, be consistent with the other baselines and retrain the model on the original dataset without subtitles denoted as EmotionI. EmotionI.LaMA. L (Cheng et al. 2024). More details in Appendix D.2
7

---

Published as a conference paper at ICLR 2026
Table 2: Zero-shot performance comparison of different methods on existing audiovisual emotion recognition benchmarks. Mod. are the modalities imput to the model with the prompt. A: Audio, V:Video, T. Text Subtitles. + evaluation without text subtitle input.
DFEW
RAVDESS
MER2023
EMER Spurious
Model
Mod.
UAR
WAR
UAR
WAR
F1
Clue
Label
Hallue.
VideoLLaMA
A.V
43.65
48.66
41.81
31.62
50.79
3.82
3.80
4.25
4.23
OLA
A.V
38.17
41.73
27.45
22.11
55.82
3.80
3.33
3.93
4.22
VITA-1.5
A,V
39.31
42.56
50.67
46.88
66.94
4.77
4.72
5.16
5.70
Qwen-2.5 Omni
A.V
46.94
54.34
32.88
28.05
79.72
5.85
6.78
6.39
6.21
EmotionLLaMA
A.V.T
45.59
59.37
28.20
29.24
90.36
6.03
6.99
5.89
5.26
A.V
42.72
54.06
30.36
30.45
89.05
2.76
2.78
3.44
2.36
MoSEAR
A.V.T
44.48
56.60
90.27
Our base
A.V
56.78
60.14
53.59
53.01
89.19
5,63
6.45
5,41
5.19
Naive-DPO
55.67
59.90
53.63
52.94
88.59
5,81
6.30
5.96
5.48
Vista-DPO"
56.42
62.33 58,54
56.94
53.64
90.06
6.08
6.89
6.58
6.07
AVEm-DPO
64.24
58.66
55.48
92.18
6.37
7.08
7.09
6.75
LaMA
A.V
54.89
58,26
52.59
48.12
90.01
5.78
6.21
5.36
5.23
Naive-DPO
54.97
58.12
$2.69
49.01
89.35
5,89
6.35
5.89
5.62
Vista-DPO*
56.28
61.58
56.42
50.96
91.19
6.05
6.56
6.85
6.31
+ AVEm-DPO
57.06
62.12
56.21
51.03
91.68
6.02
6.99
7.02
6.62
Baseline Preference Optimization Approaches. We compare with original Naive-DPO (Rafailov et al. 2023) using single rejected samples from our DPO data and modified Vista-DPO (Huang ct al. 2025b) ) for audiovisual inputs denoted as Vista-DPO* (Appendix D.3 for details).
EMOTION REASONING AND RECOGNITION RESULTS
Table 3: Performance comparison of different methods on the proposed EmoReAIM Benchmark.
EmoReAIM Results. Table
presents the performance of differ- ent approaches on the proposed Emo- ReAlM benchmark. AVEm-DPO achieves substantial gains over the reference models, demonstrating the effectiveness of multimodal prefer- ence optimization and text-prior de- biasing. While the baselines perform strongly on basic reasoning tasks, Ta- ble shows that they struggle on Modality Agreement and Stress- Emot.-LLaMA
Reas.
Basic
Modality
Reas. Audio
Stress
Model
Audio
Visual
Agree.
Visual
Acc.
Acc.
F1
F1
F1
LaM22
63.1
66.8
$2.3
53.2
38.4
OLA
63.2
60.4
42.7
56.6
54.8
VITA-I-5
63.1
84.3
30.2
$2.8
56.3
Qwen 2.5 Ommi
76.8
89,2
33.3
55.0
56.8
Our base
69.2
85.3
34.6
50.3
39,9
+ Naive--DOO
71.3
85.9
41.6
54.8
65.9
+ Vista-DPO'
72.4
87.8
52.1
73.6
86.7
+ AVEm-DPO
77.9
92.5
60.0
80.9
94.6
64.8
84,9
33.1
46.7
63.2
evaluations (Expanded table in Ap-
+ Naive-DPO
67.2
85.7
42.8
$2.6
67.6
Vista-DPOt
69.0
86.9
40.9
68.6
87.3
pendix and Table 13
AVEm-DPO
76.5
89.9
56.8
75.4
91.7
Notably, preference optimization
also surpasses Vista-DPO and Naive-DPO by significant margins. To further examine the bot- tlenecks in baseline models, Appendix E. reports results samples probing spurious audiovi- sual-emotion correlations and hallucinated cues. For state-of-the-art systems such as Qwen 2.5 Omni (Xu al.,, 2025b) and VITA- al. (2025). hallucination emerges more severe issue than spurious cue-emotion associations. Moreover, unlike findings from Sung-Bin -in et al. (2025), our results indicate that audio and visual hallucinations are equally prevalent emotion reasoning tasks. Additionally. Table I3 shows the performance of video-onke ooly and audio-only baselines and reveals that multimodal inputs hurt reasoning capabilities.
Emotion Recognition and Reasoning on Ex- isting Benchmarks. Table (expanded in Ap- pendix E.3) shows the performance on exist-
Table 4: User evaluation on EMER.
Model
Emot.
Assoc.t
Incons.]
VideoLLaMA2
9.82%
0.75%
15.38%
ing emotion benchmarks mentioned before. We can notice that our reference models outper- form baselines, showing the efficacy of ref- erence in understanding emotion. Moreover, preference tuning additionally boosts the per- formance, especially for emotion reasoning
OLA
9.36%
7.46%
5.58%
VITA 1.5
11.60%
17.25%
6.04%
Qwen 2.5 Ommi
10.75%
18.57%
10.13%
PmotionLLaMi
1.89%
11.53%
68.61%
Our + AVEm Em-DDPO
54.74%
43.35%
4.67%
8

---

Published as a conference paper at ICLR 2026
Audio Reasoning
Visual Reasoning
Visual Reasoning
Visual Reasoning
Base SEREIOO
Base
+AVEm-DP DPO
Aueio A Tunee (2)
Yaloo MAars (71)
Loge, bethhoo
Figure 5: Effect of AVEm-DPO on (Left two plots) the distribution of attention over video and audio tokens taken as percentage over the total attention over all multimodal tokens for audio and visual reasoning tasks in EmoReAIM; (Right two plots) the log-likelihood distribution shift of the correct answer for visual reasoning tasks on corrupting the audio input Cors with adversary aadv-
EMER, reducing spurious cue-emotion associ-
ations and hallucinations. It is important to note that previous emotion MLLM baselines (Cheng al. 2024, Han et al.) (0255) use text subtitle as additional input. Qualitative comparison to baselines is present in Appendix F While most baselines perform poorly on the out-of-domain RAVDESS dataset, our reference and preference-tunee models perform significantly better, show- ing their generalizability.
User evaluation. We perform a user evaluation with 40 participants on EMER generations from different models and report results in Table Participants chose our model the most for emotion de- scription and emotion-cus cue associations and the least for inconsistencies. (Details in Appendix E.4).
5.2 ANALYSIS
Ablation Study. Table 5 shows the perfor- mance of the preference-tuned model after re- moving the proposed components of AVEm- DPO. We perform this analysis on EmoReAIM and report the average metrics over audio and visual reasoning (Appendix [D.5] for details). Removal of any of the key components results in a significant performance drop, especially for the reasoning tasks. Moreover, ablating TPD results in a huge performance drop on the hallu- cination stress test samples, underlining its effi- cacy in eliminating cue hallucinations in audio- visual emotion reasoning.
Table 5: Ablation study over different com- ponents of the proposed AVEm-DPO approach. PMP: Prompt-based Modality Preference, ERP: Emotion-based Response Preference, TPD: Text Prior Debiasing.
Method
Basic.
Agree.
Stress
Spur.
Hall.
Our base
77.3
34.6
55.1
47.3
39.2
AVEm-DPO
85.2
60.1
87.8
92.7
97.6
w/o PMP
81.0
54.9
79.6
86.2
88.1
w/o ERP
81.8
56,2
79.4
84.9
88.4
w/o TPD
83,8
58.9
78.8
87.1
77.8
Contr. Dec.
79.1
51.3
61.7
50.9
54.8
Comparison with training-free contrastive decoding. Similar to VCD Leng et al. (2024), we perform contrastive decoding using diffused audiovisual inputs and report results in Table (last row), showcasing it is significantly worse than AVEm-DPO.
Design Choices and Sensitivity to Hyperparams. Appendix B.5 shows that prompt-based modality preference using a different emotion audiovisual (AV) input as (at, vi) works better compared using random videos or diffused versions of the inputs. Appendix E.6] shows that using emotion-relevant and video-relevant rejected responses (yf", yf"") works better compared to only using one or using a completely irrelevant response. Appendix E. detail the sensitivity of AVEm-DPO to various hyperparameters, highlighting the role of various components in eliminating spurious cue-emotion associations and hallucinations.
Attention redistribution after AVEm-DPO. To analyze the effect of preference optimization on model attention, we plot the distribution of aggregate multimodal input attention over audio and visual tokens averaged over all attention heads for audio and visual reasoning tasks in EmoReAIM in Fig. 5 (left two plots). We can observe that the attention over relevant modality increases after AVEm- DPO, ensuring consistent model responses grounded on the relevant modality. More atten- tion redistribution experiments are present in Appendix E.8
Robustness to adversarial inputs. As shown in Fig. 12 (Appendix E.9). the model response on a prompt relevant to one modality should not change on changing the input of the irrelevant modality.

---

Published as a conference paper at ICLR 2026
To test this robustness visual reasoning tasks, plot the distribution of log-likelihoods of correct responses for our base and AVEm-DPO models and show the distribution shift using Kernel Density Estimation (KDE) changing the audio input in Fig. right two plots). AVEm-DPO trained model results in negligible shifts, showing its robustness. Detailed analysis in Appendix F.9
5.3 VALIDITY OF GENERATED PREFERENCE DATA
As mentioned in Section 4.3 our preference dataset is automatically generated using Gem- ini 2.5 [Gemini-Tean Team al. 2025). Performing human verification on the entrre training data is too costly. Therefore, to show the validity of the generated preference tuning data, we per- form human verification on a subset of 1000 random samples from the generated data with the help of 90 participants recruited through Prolific (Prolific). Each generated sample is verified by three more annotators. As shown in T for the different categories of preference responses mentioned in Section [action[4.1]-0 4.J]-chosen (Hwe), video-relevant rejected (vi"), and emotion-relevant rejected (vf") - report the number of samples in which the majority of annotators found the generated responses correct. These results validate our automatically generated preference data.
Table 6: Human verification statistics on gener- ated preference data.
Total verified
# Majority correct
# One or more correct
Response type
Chosen (tw)
1000
912
967
Rejected Video
Relevant (v("))
1000
895
923
Rejected Emotion
1000
856
912
Relevant (uf")
6 LIMITATIONS AND FUTURE WORK
The proposed EmoReAIM benchmark derived from the DFEW (Tiang al., 2020) dataset, lever- aging its emotion labels, and hence, it may inherit its cultural biases. Additionally, since our bench- mark and training data are derived from existing emotion recognition datasets with short videos (~ 2-10 seconds), long video emotion understanding and reasoning remain an open topic that can be addressed in future work.
Although the proposed AVEm-DPO significantly improves the reference model's performance, few limitations remain. Similar to other baselines, our model trained with AVEm-DPO performs poorly on the recognition for disgust (an ambiguous emotion (Hendel et al. 2023) as shown in Appendix E.3 and Table 15 We attribute this to the limited amount of training samples available for this emotion class. Moreover, closer look at the performance Reasoning Stress Test task of EMMReAIM (Appendix F22 and Table [4] reveals that there still room for improvement to mitigate spurious audio cue-emotion associations.
the subtasks of the Emotion
7 CONCLUSION
This work addresses the bottlenecks of emotion reasoning in MLLMs, with major contribu- tions EmMRRAAM Benchmark for evaluating emotion reasoning over complex and diverse set of tasks and AVEm-DPO preference optimization technique to mitigate bottlenecks of MLLMs such as spurious audiovisual cue-emotion associations and audiovisual cue hallucinations. The proposed method outperforms open-son open-source baselines on the proposed and existing emotion understanding benchmarks under zero-shot setting. Moreover, a detailed ablation study with analysis of attention redistribution and log- likelihoeoodd shift upon preference tuning supports the efficacy of the proposed prompt-based modality preference and text-prior debiasing approaches.
ETHICS STATEMENT
This work builds upon publicly available audiovisual datasets for research purposes, specifically DFEW for benchmark creation (Section (.2) and MAFW/MER202 for preference optimization (Section 4.3). We did not collect new audiovisual data, ensuring additional privacy risks. All data usage complies with the licensing terms of the original datasets. To mitigate potential harms, the released EmoReAIM benchmark will only contain automatically generated and cuman--vrified question-ansee answer pairs; users must independently obtain the underlying videos from the original sources under appropriate licenses. For human verification (Section 3.3) and user studies (Table
10

---

Published as a conference paper at ICLR 2026
participants were recruited via Prolific and compensated at fair rates commensurate with task quirements and participant location, aligning with ethical standards for crowd work. We ensured informed consent, anonymity and the right to withdraw at any point. The proposed methods aim to improve reliability in emotion reasoning by reducing hallucinations and spurious cue associations in multimodal large language models. However, emotion recognition and inference from audiovisual data can carry risks of misinterpretation. bias reinforcement, or misuse in surveillance and high- stakes applications. Moreover, users of the proposed method are advised read the limitations of the proposed approach mentioned in Section olto avoid potential safety concerns. We emphasize that our benchmark and models are intended strictly for academic research, with the goal of advancing robust, interpretable and socially responsible AI. We caution against deployment in sensitive real- world contexts (e.g., healthcare, hiring, law enforcement) without careful domain-specific validation and safeguards.
REPRODUCIBILITY STATEMENT
To ensure reproducibility and transparency, we provide additional details about data creation and experiments in the Appendix. All the prompts used for data creation are present in Appendix [.. Implementation details for the proposed method, along with hyperparameter settings, are provided in Appendices C.3 and D.2] while the details about the baseline approaches are present in Appen- dices D.3 and D.4 Details about human verification of the benchmark and user evaluation
present in Appendices B.2 and F.d Evaluation metrics are detailed Appendix [.1]] We also pro- vide the detailed setup for our ablations in Appendix D.5 Our benchmark, code and model weights will be made publicly available upon acceptance to ensure reproducibility and ease of use for the proposed work. Code, models and benchmark our project page perre-iclr.github.io
ACKNOWLEDGEMENTS
Research was sponsored by the Army Research Office and was accomplished under Cooperative Agreement Number W91 INF-25-2-0040. Work was also in part supported by the National Science Foundation under Grant IIS-221 1550 and the National Institute of Mental Health of the National Institutes of Health under Award Number R61MH135407 The views and conclusions contained in this document are those of the authors and should be interpreted as representing the official poli- cies, either expressed or implied, of the Army Research Office, NSF, NIH, or the U.S. Government. The U.S. Government authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang. Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL httpt:::p//.......t.. / /ttp://axxvvvvv. org/ abs 13923
Luke Balcombe and Diego De Leo. Human- computrr interaction in digital mental health. In- formatics, 91): February 2022, ISSN 2227-9709. doi: 10.3990mformuttcs00000444 URL
http://dx doi wrgg10..g.o.oo wtw.:////fommaacam0011111
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39:324, 1952. URL https://api.
semanticscholee htw.aaaar.oagarprrirriiiioe..--. 12:20009222
Rijul Chaturvedi, Sanjeev Verma, Ronnie Das, and Yogesh K. Dwivedi. Social companionship with artificial intelligence: Recent trends and future avenues. Technological Forecasting and Social Change,
2023. ISSN 0040-1625. doi: https///on.og///..000/.....e
2023. 126344. URL httpp::/wwet sciencede rect. ccnm//cceenee//rriccl/p..i..
500401 62523003 90
Ashutosh Chaubey, Xulang Guan, and Mohammad Soleymani. Face-llava: Facial expression and attribute understanding through instruction tuning, 2025. URL hattps://arxe org/obgg/.. 2504. 07199
11

---

Published as a conference paper at ICLR 2026
Junzhe Chen, Tianshu Zhang. Shiyu Huang, Yuwei Niu, Chao Sun, Rongzhou Zhang, Guanyu Zhou, Lijie Wen, and Xuming Hu. Omnidpo: A preference optimization framework to address omni- modal hallucination. arXiv preprint arXhv:2509.00733 2025.
Yin Chen, Jia Li, Shiguang Shan, Meng Wang, and Richang Hong. From static to dynamic: Adapt- ing landmark-aware image models for facial expression recognition in videos. IEEE Transactions Affective Computing, pp- 1-15, 2024. doi: 10. 1...1099/AFFC.2224..555
Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xi- aojiang Peng, and Alexander Hauptmann. Emotion-llama: Multimodal emotion recog- nition and reasoning with instruction tuning. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan. U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural In- formation Processing Systems, volume 37, pp- 110805-110853. Curran Associates, Inc., 2024. URL https:: ngs neuri nwwrips.rnnnmmmm htppp//eeeppppppepsaa///ppp////a fil weteec/f44aaa 7accc2 corp pdd
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report. arXiv preprint ar arXiv:2407.00759 2024.
Ding Ding, Zeqian Ju, Yichong Leng. Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint A.kv::500.184255 2025.
Paul Ekman. Basic Emotions. In Handbboo of Cognition and Emotion, pp. 45-60. John Wiley & Sons, Ltd, 2005. doi: 10. 10.1002/0470011444444
Paul Ekman and Wallace V. Friesen. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto, CA, 1st edition, 1978.
Zohar Elyoseph, Efrat Refoua, Keren Asraf, Michael Lvovsky, Yair Shimoni, and Dalit Hadar- Shoval. Capacity of generative ai to interpret human emotions from visual and textual data: Pilot evaluation study. JMIR Mental Health, 11:e54369, Feb 2024. doi: 10.2196/54369.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, et al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. arXiv preprint arXiv:2501.01995 2025.
Gemini-Team al. Gemini 2.5:
Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL https: org.orgg///oo..oo. / 2507. 06661
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all, 2023, URL https:
//axxiv..rc/abs/2300 05665
Arushi Goel, Sreyan Ghosh, Jachyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao- Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al. Audio flamingo 3: Advancing audio intelligence with fully open large audio language models. arXiv preprint arXiv:2507.08128, 2025.
Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language, 2025a. UtRLhttts:://///
wtt.v/.rggabb///2222 03700
Zhiyuan Han, Beier Zhu, Yanlong Xu, Peipei Song, and Xun Yang. Benchmarking and bridging emotion conflicts for multimodal emotion reasoning. arXiv preprint w:Xiv:2508.01181 2025b.
Emalie Hendel, Adèle Gallant, Marie-Piee Mazerolle, Sabah-Izan Izayah Cyr, and Annie Roy-Charland. Exploration of visual factors in the disgust-anger confusion: the importance the mouth. Cogn. Emot., 37(4):835-85 851, May 2023.
12

---

Published as a conference paper at ICLR 2026
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022. URL ftttps://opereeviii net//orun
id=nZevKeeFYf9
Dawei Huang, Qing Li, Chuan Yan. Zebang Cheng, Zihao Han. Yurong Huang, Xiang Li, Bin Li, Xiaohui Wang. Zheng Lian, Zhi-Qi Cheng, and Xiaojiang Peng. Emotion-qwen: A unified framework for emotion and vision understanding. 2025a. URL https://axxiv..rg//
2505. 06685
Haojian Huang, Haodong Chen, Shengqiong Wu, Meng Luo, Jinlan Fu, Xinya Du, Hanwang Zhang, and Hao Fei. Vista dpo: Video hierarchical spatial-temporal direct preference optimization for large video models. In Feroy- International Conference on Machine Learning, 2025b. URL htttps://cee /tttps://peerreview.. wet.ieewwweeeeeee....e.eee.eee.e ooo
Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2881-2889, 2020.
Zeyu Jin, Jia Jia, Qixin Wang, Kehan Li, Shuoyi Zhou, Songtao Zhou, Xiaoyu Qin, and Zhiyong Wu. Speechcraft: A fine-grained expressive speech dataset with natural language description. In ACM Multimedia 2024, 2024. URL URL httpp:: /wtp:// net/ wtt.:/oormmmmfm jat
Michal Kolomaznik, Vladimir Petrik, Michal Slama, and Vojtech Jurik. The role of socio-emotional attributes enhancing human-ai collaboration. Frontiers in Psychology, 15, October 2024. ISSN 1664-1078. doi: (0.3339/fpsyg.2005 22..2.222..99.. 2024g.2024.4 1369957
URL http: doi. wrt//oogg.10... 3389/
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive de- coding. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), doi: 10. 1111099CVRR37773.222......
pp. 13872-13888 13882, 2024.
Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, and Lidong Bing. The curse of multi-modalities: Evaluating hallucina- tions of large multimodal models across language, visual, and audio, 2025. URL https: //openzeview. www.://oooooeedeeeeiiiiiii..
Yadong Li and team. Baichuan-omni-1.5 technical report, 2025. URL htttss://aaatt.. orrg/ bbs/2501.. 15368
Yifan Li, Yifan Du Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL https: /otp::////eeeeeeee.e net. teee/ffrrue forum? ?
xozJwOXZXF
Zheng Lian. Affectgpt-rl: Leveraging reinforcement learning for open-vocabulary emotion recog- nition, 2025.
Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mingyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi-supervised learning. In Proceedings of the 31st ACM international conference on multimedia, pp. 9610- 9614, 2023a.
Zheng Lian, Haiyang Sun, Licai Sun, Hao Gu, Zhuofan Wen, Siyuan Zhang, Shun Chen, Mingyu Xu, Ke Xu, Kang Chen, et al. Explainable multimodal emotion recognition. arXiv preprint arXiv:2306. 15401, 2023b.
Zheng Lian, Haiyang Sun. Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, Zhuofan Wen, Shun Chen, Siyuan Zhang, Hailiang Yao, et al. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark. ICMI, 2024.
13

---

Published as a conference paper at ICLR 2026
Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. ICML, 2025a.
Zheng Lian, Rui Liu, Kele Xu, Bin Liu, Xuefei Liu, Yazhou Zhang, Xin Liu, Yong Li, Zebang Cheng. Haolin Zuo, et al. Mer 2025: When affective computing meets large language models. arXiv preprint Ntt1:2504.114222 2025b.
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 5971-5984 Miami, Florida, USA, November URL
2024. Association for Computational Linguistics. doi: 10.1000........ 116538816/002222
wrg. emn.1112..mm......mmmmm emnlp ma in..iii.. 342/
Maija Litendahl, Juulia Kaihlaniemi, Olli Autio, Outi Kähkönen, and Anne Oikarinen. Healthcare professionals' perceptions of emotional intelligence in remote counselling descriptive quali- tative study. Nursing Open, 12(4), April 2025. ISSN 2054-1058, doi: 10.1002/mop2.702118 URL
org/ 1002 nop2. 70218
Aiwei Liu, Haoping Bai, Zhiyun Lu, Yanchao Sun, Xiang Kong, Xiaoming Simon Wang. Jiulong Shan, Albin Madappally Jose, Xiaojiang Liu, Lijie Wen, Philip S. Yu, and Meng Cao. TIS-DPO: Token-level importance sampling for direct preference optimization with estimated weights. In The Thirteenth International Conference on Learning Representations, 2025a. URL https:
//openreview
Yuanyuan Liu, Wei Dai, Chuanxu Feng, Wenbin Wang, Guanghao Yin, Jiabei Zeng, and Shiguang Shan. MAFW: A Large-scale Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild. ACM, New York, NY, USA, 2022. ISBN 978-1-4033-99003-7
URL wttphhtps///d/t //doi org 10. 1145 000111155333111 350316 61. 3548190
Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang. Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. MIA-DPO: Multi-image augmented direct pref- erence optimization for large vision- languaee models. In The Thirteenth International Confer- ence on Learning Representations, 2025b. URL htRp:ttts:/// http::::///ppeevviii ew net. www.ee/forrmmtt forum??d 77BRSSuf1
Steven R. Livingstone and Frank A. Russo. The ryerson audio-visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PLOS ONE, 1315)19016639,, May 2018 ISSN 1932-6203. doi: 10. h/3/////oraal.poo/ 0196391. URL http://dx doi erg 10. 1371 //or////ouaaal.aaa pone. 0196391
Run Luo, Ting-En Lin, Haonan Zhang, Yuchuan Wu, Xiong Liu, Min Yang, Yongbin Li, Longze Chen, Jiaming Li, Lei Zhang, et al. Openomni: Advancing open-source omnimodal large lan- guage models with progressive multimodal alignment and real-time self-aware emotional speech synthesis. arXiv preprint arXiv:2501.044 04561,, 2025.
OpenAl et al. Gpt-4o system card, 2024. URL htt.tt/gs//aaxuu.rr///////1/1110
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015. doi: 10.1109/ICASSP2005.
7178964.
Prolific. Prolific -
Easily collect high-quality data from real people prolific.com. https:
7 7/www. www.pprolffcc pro.fffc.ccmm [Accessed 23-09-2025].
Qualtrics. Qualtrics XM - -Experience Management Software qualtrics.com. ht attpps:n //Wp::///W.
qualtrics. com/ [Accessed 23-09-2025).
Qwen- Team et al. Qwen2.5 technical report, 2025. URL https: htta:://axxvv.oo http:://axxvv.rr//////a///// hboggg/bb/4411 15
14

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-2888 28518. PMLR, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
7/ [ Hft.://ooOoO11---PSIIAAA
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL hhttps://arxi
arg/abs/190 10084
Pranab Sahoo, Prabhash Meharia, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. A comprehensive survey of hallucination in large language, image, video and audio foundation mod- els. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computationnl Linguisties: EMNLP 2024, pp. 11709-1 1724, Miami, Florida, USA, Novem- www.diing-enmnll..... URL https: // ac antho ooy..orgg. 2024. fi ni ngs- pp. 685/
ber 2024. Association for Computational I Linguistics. doi: 10. N..18633v11/2222
Said A. Salloum, Khaled Mohammad Alomari, Aseel M. Alfaisal, Rose A. Aljanada, and Azza Basiouni. Emotion recognition for enhanced learning: using ai to detect students' emotions and adjust teaching methods. Smart Learning Environments, 12(1). February 2025. ISSN 2196-7091. doi: 10. 0066//440661-02-000743333 URL http::/dxx doi org/10..g.o.oo 1 18
5 1405511025--0333
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan O Arik, and Tomas Pfi-- ter. Mitigating object hallucination in MLLMs via data-augmented phrase-level alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview www.:etet/ormimimmmmmmmmommm deyyl fw8 gzF
Klaus R Scherer. What are emotions? And how can they be measured? Social Science Information, 44(4):695-729, 2005. doi: 10.1 1,11777/0390118805088666
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all, 2023. URL / 16355
Guangzhi Sun, Yudong Yang, Jimin Zhuang, Changli Tang, Yixuan Li, Wei Li, Zejun MA, and Chao crasoning-enhancee audio-visual large language model. In Forty-
Zhang. iide-SAALMONN-O second International Conference on Machine Learning. 2025. URL http:: /tpp///eeeee.....e// /
met/forund 1d-.mmm22-1dd2222266663
Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. Mae-dfer: Efficient masked autoencoder for self- supervised dynamic facial expression recognition. In Proceedings of the 31st ACM International Conference Multimedia, pp. 6110-6121, 2023.
Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, and Tae-Hyun Oh. AVHBench: cooss-modal hallucination benchmark for audio-visual large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: /ooeeneview. net www.ee/ooooddddiaeee....
Changli Tang, Yixuan Li. Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: captioning-enhaneee audio-visual large language models. arXiv preprint arXiv:2506. 2tti::5506./2200, 2025.
Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. mdpo: Conditional preference optimization for multimodal large language models. 2024.
Hanyang Wang, Bo Li, Shuang Wu, Siyuan Shen, Feng Liu, Shouhong Ding, and Aimin Zhou. Rethinking the learning paradigm for dynamic facial expression recognition. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17958 17968, 2023. doi: 10. 10.1009CCVP55229220..0777
15

---

Published as a conference paper at ICLR 2026
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Lingln Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie, Zehao Li, Bowen Yang. Yuchen Duan, Xuehui Wang, Zhi Hou, Haoran Hao, Tianyi Zhang. Songze Li, Xiangyu Zhao, Haodong Duan, Nianchen Deng, Bin Fu, Yinan He, Yi Wang. Conghui He, Botian Shi, Junjun He, Yingtong Xiong, Han Lv, Lijun Wu, Wenqi Shao, Kaipeng Zhang, Huipeng Deng, Biqing Qi, Jiaye Ge, Qipeng Guo, Wenwei Zhang, Songyang Zhang, Maosong Cao, Junyao Lin, Kexian Tang, Jianfei Gao, Haian Huang, Yuzhe Gu, Chengqi Lyu, Huanze Tang, Rui Wang, Haijun Lv, Wanli Ouyang, Limin Wang, Min Dou, Xizhou Zhu, Tong Lu, Dahua Lin, Jifeng Dai, Weijie Su, Bowen Zhou, Kai Chen, Yu Qiao, Wenhai Wang, and Gen Luo. Internv13.5: Advancing open multimodal models in versatil- ity, reasoning, and efficiency, 2025. htt.stt.::// httttss:/aaxii..rgg////
2208 1.e..82200 18265
Zhuofan Wen, Zheng Lian, Shun Chen, Hailiang Yao, Longjiang Yang. Bin Liu, and Jianhua Tao. Listen, watch, and learn to feel: Retrieval-augmente emotion reasoning for compound emotion generation. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 11313- 11327, 2025.
Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Hung-Jen Chen, Chan-Feng Hsu, Hong-Han Shuai, and Wen-Huang Cheng. Emovit: Revolutionizing emotion insights with visual instruction tun- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Bohao Xing, Xin Liu, Guoying Zhao, Chengyu Liu, Xiaolan Fu, and Heikki Kälviäinen. Emotion- hallucer: Evaluating emotion hallucinations in multimodal large language models, 2025. URL
https://axxiv.. org. abs / www.///...... 11405
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, Bin Zhang, Xiong Wang. Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025a. URL https: /tteps://arxiie http/axxvvvvv.....aaaaaa org/
20215
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025b. URL httpt:ttt///// http://aaxvvvv/. org/ abs http//aa.../2...22 20211
Qize Yang, Detao Bai, Yi-Xing Peng, and Xihan Wei. Omni-emotion: Extending video milm with detailed face and audio modeling for multimodal emotion analysis, 2025. URL https: 7/ 7/arxiv. org. hatp///as//211//
Qilang Ye. Zitong Yu, Rui Shao, Yawen Cui, Xiangui Kang, Xin Liu, Philip Torr, and Xiaochun Cao. Cat+: Investigating and enhancing audio-visual understanding in large language models. IEEE Transactions Pottern Analysis and Machine Intelligence, 47(10):674-8699 2025. doi:
10.1009/PAM...222555555 35883388.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang. Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video under- standing, 2025a. URL www.::ttps.... //tpxvaaavvv.////a///a/ www. abs 250..13101
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual lan- guage model for video understanding. arXiv preprint arXiv:2306.0288 028588 2023. URL https
02858
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chun- yuan Li, Alexander G Hauptmann, Yonatan Bisk, and Yiming Yang. Direct preference optimiza- tion of video large multimodal models from language model reward. In Luis Chiruzzo, Alan
16

---

Published as a conference paper at ICLR 2026
Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Ameri- cas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 694-717, Albuquerque, New Mexico, April 2025b. Association for Computational Linguistics. ISBN 979-8-89176-189448 doi: 10.18653/v 001553/v12000 1/2025 naacl-long.30. ogy www.ooggooogggoooooo -ttp:/200222.mc11111maa.a 3on
URL htt.hhttsss//aclaatttiii
Tianyi Zhang, Varsha Kishore. Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In International Conference on Learning Representations.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video in- struction tuning with synthetic data, 2024. URL /ttp::/:///xx..r/g/////// 0t0ps//411.0777..
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, WANG HongFa, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, Cai Wan Zhang, Zhifeng Li, Wei Liu, and Li Yuan. Lan- guagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In The Twelfth International Conference Learning Representations, 2024. URL /tttps://ore //ppereeeeeee net wwrumooriiooo forumm?dd00n2x770220
17

---

Published as a conference paper at ICLR 2026
APPENDIX
TABLE OF CONTENTS
LLM USAGE
BENCHMARK DETAILS
PROMPTS USED IN BENCHMARK CREATION
HUMAN VERIFICATION
BENCHMARK STATISTICS
B
FRAME SAMPLING RATE AUTOMATIC VISUAL CAPTIONING
BENCHMARK SAMPLES
METHOOOLOGICAL DETAILS
TEXT-PRIOR DEBIASING
PREFERENCE DATA
IMPLEMENTATION DETAILS
EXPERIMENTAL DETAILS
EVALUATION METRICS
REFERENCE MODELS
BASELINE PREFERENCE OPTIMIZATION TECHNIQUES
BASELINE IMPEENANTASION
EXPERIMENTAL SETUP FOR ABLATION STUDY
DETAILED RESULTS
EMOREALM RESULTS EXPANDED
EMOREAL.M RESULTS DIFFERENT STRESS TEST SUBTASKS
EMOTION RECOGNITION RESULTS EXPANDED
USER EVALUATION
MODALITY PREFERENCE ABLATION
RESPONSE PREFERENCE ABLATION
E.6
SENSITIVITY TO HYPERPARAMETERS
ATTENTION REDISTRIBUTION AFTER PREFERENCE OPTIMIZATION
REASONING WITH ADVERSARIAL MODALITY INPUTS
EFFECT OF INDIVIDUAL MODALITIES FOR EMOTION PREDICTION
10
QUALITATIVE SAMPLES
PROMPT POOL
A LLM USAGE
We used GPT-5 to polish the text we added to the paper for grammar and consistency checks. We verify the grammar changes suggested by GPT to ensure validity. No significant part of the text in the paper is written by any LLM. Apart from polishing the paper, we use LLMs for data annotation and automatic evaluation as mentioned Sections 3.2
4.3 and Appendices C.2
D.I and D.2
B BENCHMARK DETAILS
PROMPTS USED IN BENCHMARK CREATION
In this section, detail the prompts that used in various parts of the benchmark creation pipeline mentioned in Section and Fig. Note that the text prompts themselves are present at the end of the document in Appendix
18

---

Published as a conference paper at ICLR 2026
Table Statistics of human verification on EmoReAIM Benchmark.
Ques, at least one correct
# #Ques. majority correct
Ques. verified
# Ques discre.
Ques. Final
Task
Reasoning Basic
Audio
1200
1168
968
972
Visual
1200
1137
1014
10
1024
Modality Agreement
1000
489
458
0
456
Audio
1000
956
806
14
820
Reasoning Stress Test
Visual
1000
845
719
9
728
5400
3959
4IN10
Audio and Video Captioning. Figs. 19 and 20 contains the prompts used to caption the audio
and visual content separately for a given video as described in Section 3.2 and Fig. 3 For visual captioning, we sample eight uniform frames from the video and pass those to GPT-4o. For audio captioning, only pass the audio as a WAV file to GPT-4o-audio.
Emotion prediction from audio and video captions separately. Figs. 21 and 22 contain prompts used predict the emotion (out of the seven basic categories) just using the audio and video cap- tions separately. If the ground truth emotion label cannot be predicted by both the audio and video captions, then we do not proceed with such video for the subsequent data pipeline.
EMOReAIM QA Generation. Figs. 23 and 24 contains the prompts to generate questions related to Emotion Reasoning- Basic as described Section[.T]jee audio and visual reasoning respectively. We use the ground truth emotion label already present in the source emotion recognition dataset, as well as the audio/video captions, to generate the question answers. Note that audio and visual reasoning samples are only generated for those samples in which emotion was predicted correctly from the audio and visual captions, respectively (using prompts in Figs. and 22).
We use prompt in Fig. 25 to generate questions related Modality Agreement (Section 3.1 by passing the audio captions, video captions and the ground truth emotion label present in the source dataset. We also verify the answers to the generated questions using the ground truth emotion label present for the video and the emotions predicted using only audio and video captions. If both the audio and the video caption predict the ground truth emotion label from the captions (using prompts in Figs. 21 and 22 then the correct answer should be "Yes", clse it should be "No".
For the Emotion Reasoning- Stress Test (Section 3.1), D, we generate questions using prompts present in Figs. to 31 We use separate prompts for generating questions related for the different subtasks
No hallucination (Figs. 26 and 29). Spurious Cue-Emotion Association (Figs. 27 and 300 and
Gelovann-elevant Hallucination (Figs. and 31). Note that the No hallucination prompts only
apply to cases where the emotion prediction from the audio and/or visual captions using Figs. 21
and 22 same the ground truth emotion label.
Text Only Guess Post Processing. We use the prompt in Fig. 32 to guess the correct answer for the generated question and answer choices using only the text (i.e., without audiovisual input). This is done post-processing step as described in Section 3.3 to ensure that the answer for the MCQA sample is not predictable using only the text inputs.
B.2 HUMAN VERIFICATION
As mentioned in Section 3.3 we perform human verification for the generated QA samples to ensure high data quality by removing samples that contain some discrepancy. We conducted a survey using Qualtrics and recruited participants using the crowd-sourcing platform |Polifii In total, we conducted the survey on 471 participants and ensured that the participants were paid fairly for their time. To ensure participants are capable of answering the questions, we included a pre-survey to test their emotional intelligence. Moreover, included attention checks using questions that are already verified by us to ensure the quality of the participant responses.
We conduct the survey a MCQ task where the participants are shown the questions and the answer choices created in the benchmark and we ask them to choose the correct answer as shown in Fig. 6 Each participant was also shown follow-up upp question after each question flag the text present in
19

---

Published as a conference paper at ICLR 2026
Video
you more option Question?
Subtitie: 'warld sin.'
=uuestion
Was there something wrong with the question or the onawer choices?
What emotional conweyed through speaker's words the video?
ho evarything correct
the spootor romenssos kvcly about useroting diamsarron
Queation hod some detu la lncor sisten with the elao
(8) The specker's and del
wid
connoormttoon.
anower chrilees nimbie
(c) The qpentrre viirlyy nommiee cnttlpptionn and recitement fin on approoching calceration, copturing moment (ow.
Sometting niw?
(a) the speataric mbsr and thlioarcte phaoxing mord delternrenttnn
emmoncee huture opportunties
Figure 6: Human verification survey questions. (Left) An example question from the benchmark shown to the participant. (Right) Follow-up questions shown the participant about each question.
Distribution Questions Across Tasks
1000
972
Emotion Distribution: DFEW EmoReAIM
Number of Questoons
725
240
Number of Samples.
2000
400
1500
500
Vauaa Reasoning
Audio festoning
Audio Halucmation
Visual Heflucination
Modality Agrietmmnt
happiness
sadness
neutral
anger
surprise
disgust
teat
Category
Figure 7: (Left) Distribution of QA samples across different tasks in EmOReAIM benchmark. (Right) Distribution of ground truth emotion labels for the videos present in EmoReAlM compared with the distribution in the source dataset DFEW (Jiang et al. 2020
the question or answer choice or to report any other discrepancy. Since some videos in the DFEW Jiang et al.) 2020) dataset are not in English, the participants were also shown the English subtitle for the video that the MCQ is about.
Table 7 contains the statistics of human verification. Due to budget constraints, we ran the survey only on 5400 questions across different tasks. We only use the samples from the benchmark for which the majority of the participants selected the correct answer, automatically annotated in the benchmark. Additionally, we manually correct some samples that had discrepancies and add them to the final set of questions well.
B.3 BENCHMARK STATISTICS
Fig. (Right) shows the distribution of ground truth emotion labels in the EmoReAlM benchmark
compared that present in the source dataset DFEW (Jiang et al. 2020). We can that the distribution of samples over different emotions is similar to DFEW. Fig. shows the distribution
of subtasks within the Emotion Reasoning Stress Test task (Section 1b of EmoReAIM bench-
20

---

Published as a conference paper at ICLR 2026
Distribution of Languages
1130.0
1120.0
1110.0
1100.0
300.0
250.0
200.0
Count
150.0
100.0
50.0
0.0
English
Chinese
Japanese
Korean
French
Hindi
Italian
Spanish
Russian
Nynorsk
Turkish
Hungarian
Swedish
German
Polish
Others
Figure 8: Distribution of different languages present in the audiovisual samples present in EmoRe-
AIM benchmark.
Stress Test Audio Subtasks
Stress Test Visual Subtasks
Audio Hallucination
Visual Hal Halluciation
27.6%
26.2%
Hallucination (Audio)
50.0%
No Hallucination (Visual)
23.8%
Spurious Associ ation
Spurious Association
(Audia)
(Visual)
Figure 9: Distribution of subtasks in the Emotion Reasoning Stress Test of EmoReAlM benchmark.
mark. Due the way we formulate the questions for this subtask "Does the {audio/visual cue) suggest {emotion) of the character?", the samples belonging to No hallucination subtask have the answer "Yes". and the samples in the Spurious Association and Audio/Visual Hallucination subtasks have answer "No". " Fig. shows that the number of samples with "Yes"/"No "/"No" answers are equally distributed. Moreover, for all the samples with answers as "No". the samples are almost equally distributed to test spurious cue-emotion associations and audiovisual cue hallucinations. Further- more, to show the cultural and linguistic diversity in the benchmark, Fig. shows the distribution of languages present in the samples of EmoReAlM benchmark. We obtain this by using automatic language detection using Whisper (Radford et al. 2023). can observe that although the majority language is English, benchmark contains samples from a wide range of languages.
Table 8: Effect of using different number of frames for visual captioning using GPT-4o.
BERT Score
# frames
SBERT-sim
Prec.
Rec.
F1
0.646
0.851
0.853
0.852
0.660
0.851
0.856
0.853
4
0.676
0.851
0.857
0.854
8
0.689
0.858
0.861
0.860
16
0.688
0.858
0.862
0.860
21

---

Published as a conference paper at ICLR 2026
Table 9: Samples from the EmoReAIM Benchmark for the Emotion Reasoning- Basic Task.
Tisk
Video
Question the video their emotional
Answer
does speaker choice state?
(A) The speaker mentions struggling move foeward despite past setbacks, indicating a reflective state.
Reasoning Basic
(B) The speaker's is tone refeetts a somber atmosphere, accompanied by a soft,
(Audio)
resigned voice.
(C) speaker's phrase portrays deep of regret resignation. nitempt.
reflecting
Subeide:
()) The speaker uses sult background music to oulinne the somber mood. suggesting unfalilled efforts.
what way the
man's voice impact his emotional expression
in the video'
(A) The presence soft whispers and gentle music in the buckground could imply an underfying tension and hidden emation.
Reasoning Basic (Andio)
(B) The man't 's tone is marked by a tightness and sharpaess, resonating with
his underlying fiuustration and simmering anger.
B
(C) phrase
believe you' done this again" reflects unckrly ing.
Subiide: "You haven't spoken me in 10 years"
resentment connected to a long standing grievance.
(D) The man's s voice holds a lively and enthusiastic lonc, mistakoonly suggesting of joy and conteniment.
does the woman's facial expression contribute to the overall Toeling the sean??
(A) The womnnn displays joyful expression with open conveying her happiness and openness.
Reasoning Busic (Visual)
(B) The woran'' chocrful smile and lively eyes reveal her happiness and engagement.
B
touch. symbolizing (D) The woman'' is long dark hair frames her face, enhancing the appearance happiness and delight.
(C) wonann's yellow turtiencek
hagpiness and contentment.
Subidde: "Stanford University?
What are you guys talking about!""
Whst does the individual body lingunge indicatt about their emotionnl state in the video?
(A) The individual's quivering movements and uncertain footing
palpahle sense
Reasoning Basic (Visual)
(B) The person's 's lense facial expression with slightly open mouth and wide eyes enitances their learful demeanor.
(C) person leaning cautiously lowards door, their body lense, which highlights their fear or anxiety.
Subeirle:
(1) The individuul's 's dark-collrre shirr amplifies their sense of fear, over- shadowing their suroundings.
B.4 FRAME SAMPLING RATE FOR AUTOMATIC VISUAL CAPTIONING
Since the visual cues used to express and infer emotions can subtle, it is important to ensure that the visual captions obtained using GPT-4o in the first stage of data creation (Section 3.2 and Fig. 3 are of high quality. To identify the idcal number frames to be sampled from the video for cap- tioning,
a small experiment is important to note that EMER (mean duration: 3.78s) contains videos of similar duration as DFEW (mean duration: 3.42s) which we use to construct EmOReAIM. We extract different number of frames per video and obtain the visual caption from GPT-4o using prompt in Fig. 20 Then, com- pute the similarity between the generated captions and the ground truth using BERTScore (Zhang et al.) and Sentence BERT (Reimers & Gurevych| 2019) similarity score. Table g shows that us- ing frames for visual captioning leads good captioning results. Furthermore, using 16 frames is not significantly better than using 8 frames, but it increases the costs significantly. Hence we choose to use 8 frames uniformly sampled from the video to extract visual captions from GPT-4o automatically.
the emotion captioning dataset EMER (Lian al., 20236)
B.5 BENCHMARK SAMPLES
We present samples belonging to different categories of the benchmark in Tables D to 11 the subtitles shown in the tables are just for reference and we do not pass the subtitle as an input to the model during evaluation.
Note that
22

---

Published as a conference paper at ICLR 2026
Table 10: Samples from the EmoReAlM Benchmark for the Modality Agreement Task.
Task
Video
Question
Do visual elements of the video with the audio conveying
Modality Agreement
fooling of happiness of the person in the vidco? (A) Yes (B) No
Subtitle: "I was..""
audio modalities align the expression anger the person in the video??
Modalit Agreement
Yos
(B) No
Subritle: exactly
Table 11: Samples from the EmoReAlM Benchmark for the Emotion Saaasoninn-Stress Test.
Stress Test (Audio Hallucinution)
Do chuckling sounds the andio enhance the feeling joy convryed for the the video?
Subitte: "(chuckles)"
Steess Test (Andio Spurious Association)
Is the presence of sonur ping sound effect crucial the feeling of surpeise conveyed the person in the video?
(A) Yes.
Subtifle: "[sonar ping)"
Does the sound of slamming door contribute to the experienced the person in the video?
Stees Test (Andia Hallocination)
(A) Yes
Subitie: "It ain't Alan's tault..."
Is the downward the older woman significant factor expeessing the sadness of the older woman portrayod the video??
Stress Test (Visual No Hallocination)
Yes
Subtitie:
Stress Test (Visual Spurioas Association)
Is the presence of the vibeant checkered patior the walls facter in convying the neutnil emotion pesso/churacier video?
Subtitle:
disployling eleached hiis anger vidio!? (B)
Hallucination)
Subtite:
23

---

Published as a conference paper at ICLR 2026
Table 12: Examples of the preference dataset used for AVEm-DPO.
Prompt
Rerpens (to)
Rejectal Response
Rejected Respatte
s(")
scoro oft young peoors hure enietional monneay strteg eschange?
syrbore and rgg mooth cmppoecee their reneer saa-
percce uncediso the senceetos mixd
The
rlinding fex and emite wnnrring sncediis
Suballe baaly Batl carsenci lock
Siz renereiiiicc deep astod crsticeal I'm taod, whack indicnic lat aukioo.
The sedescholic pleso masic crsetional busntazie is expentnoin.
His exgrewske srying emoturud waie.
vidcosellost contional siate
Uo todko corncy the the vidox?
You bois the aadio and vadeo conndy profecral suroz crying daszanght lacall exprosiion
No, the tene of voice andio appoin strseqthone
No, the wossar's fasial expressius indkaates aocuu words tike is angenoe" seggre sotiee.
mere
"Lopieg"
C METHODOLOGICAA DETAILS
C.I TEXT-PRIOR DEBIASING
Similar to Eq. 66 we scale the TPD term to accommodate multiple rejected responses as follows,
Tee (Ve h (a, U, x)) Trer (a, v,
Be log Ta(ví (a, 2))
DPD -BB (a.0. v1) D
log o
log
Teer uf (a,
LE (er.er)
log (Bo Moo xx)
8g log Tuuu Tuuuut (vl =))
(en.er)
(10)
where Ber + Ber = 1. Also, for succinetness, we denote (aw, vw) with (a, v) in the above equation.
C.2 PREFERENCE DATA
As mentioned in Section 4.3 we use a pipeline similar to Fig. to construct our preference data using MAFW (Liu et al. 2022) and MER2025 (Lian et al. 2025b) Track / train set as the source datasets. Note that tations required to create the training dataset. Use of Gemini for training data creation reduces annotation budget and ensures that the training dataset is not biased to have similar language the test dataset EmoReAlM. Since the pipeline in Fig. 3rggatess MCQA samples, we use another round of automatic annotations through Gemini-2.5 Flash over the generated MCQA samples to create the preference data. Specifically, we use prompts in Figs. 33 to generate rejected responses for the generated emotion reasoning QA samples. Since, also want improve the performance emo- tion description tasks present in EMER (Lian et al. 2023b) we use prompts for audio (Fig. 33) and visual reasoning (Fig. 34) to modify emotion descriptions generated from Gemini 2.5 Flash (using prompt in Fig. 36), combining audio and visual captions of MAFW and MER2025 (obtained using prompts in Figs. 19 and 20). After Gemini annotation, we end up with a total of 41687 preference
Gemini 2.5 Flash (Gemini-Teame et al. 2025) for all automatic anno-
samples combining tasks, which we use for AVEm-DPO training. Table 12 contains samples from constructed preference dataset using the described pipeline.
C.3 IMPLEMENTATION DETAILS
We train the reference models using AVEm-DPO for one epoch, with a learning rate of 5e and per GPU batch size of 2 NVIDIA DGX node with 8 NVIDIA H100 GPUs. We choose 0 as 0.1 similar to Huang et al. 2025b). Moreover, \ęo is set to 1.0, 3.r and B.r are both set to 0.5, and TTeD is set to 0.2 (refer Appendix E.7 for details on choice). We attach LoRA module with
24

---

Published as a conference paper at ICLR 2026
rank 8 and scale 4 to the LLM backbone for training. Gradient accumulation is used to accumulate gradients over 4 iterations.
D EXPERIMENTAL DETAILS
D.1 EVALUATION METRICS
GPT Evaluation on EMER. As mentioned Section | we perform GPT-4o evaluation on the generated emotion descriptions in EMER (Lian al. 2023b) dataset. We perform the evaluation over the following criterias (i) clue overlap similarity of the audiovisual cues present in the generation with the ground truth, (ii) label overlap- similarity of the emotion label described in the generation with the ground truth, (ii)) spurious cue-emotion associations how good are the audiovisual cues associated with emotions in the generation, and (iv) hallucinatory cues presence of cues that are absent in the ground truth but present in the generations. The prompt used to evaluate the generations is present in Fig. 37
EmOReAIM Evaluation Metries. For all the tasks in EmoReAlM, we report the average accuracy over the task, computed as the number of correct responses out of the total number of samples in the task. Additionally, for tasks with "Yes"/"No" responses (Modality Agreement and Emotion Reasoning Stress Test), we report the precision, recall and F1 score. Precision and recall are the ratios of correctly answered questions that have correct answers Yes and No, respectively, Fl score is the harmonic mean of precision and recall.
D.2 REFERENCE MODELS
We describe the reference models mentioned in Section below.
Our base. We modify EMoAionLLaMA (Cheng al., 2024) replace the visual encoder with LanguageBind Video Encoder (Zhu et al.) 2024) and audio encoder with Whisper Large v3 (Rad-)
ford et al. (023). We pretrain the visual projector using the pretraining data of LideoL (Lin et al. 2024) and the audio projector is pretrained using LibriSpeech (Panayotov et al., 2015) and
SpeechCraft (Jin et al. 2024) to enhance paralinguistic capabilities of the model. We finetune on dataset, however, include additional instruction data by annotating MAFW (Liu et al. 2022) and MER2025 (Lian et al.) 2025b) Track I train set through Gemini 2.5 Flash. Specifically. we the prompts mentioned in Appendix B. create finetuning dataset with sim- ilar tasks as in the proposed EmoReAIM benchmark. We also use prompt in Fig. 36 to generate
the E
emotion descriptions from MAFW and MER2025.
EmotionLLaMA'. Since the pretrained EmotionLLaMA model is not trained on tasks simi- lar EmoReAlM, finetune EmotionI.LaMA on additional datasets created using MAFW and MER2025, similar to our base model deseribed in the previous paragraph. Moreover, we do not pro- vide subtitle text input the model during finetuning, contrast the original EmotionLLaMA. to eliminate external subtitle dependence.
D.3 BASELINE PREFERENCE OPTIMIZATION APPROACHES
We describe the implementation of baseline DPO approaches mentioned in Section 5/below. We use the same training setup as mentioned in Appendix C.3 unless stated otherwise.
NaOve-DPO. For Naive-DPO (Rafailov et al. 2023 preference samples from our preference data (Appendix C.2 domly between y)"" and yf"
the objective in Eq. B3. We the
and pick the rejected response ran-
Vista-DPO', We adapt Vista-DPO (Huang et al. 2025b) for audiovisual inputs using Eqs. (4) and (6) Also, we use our preference data (Appendix C.2) to optimize Eq. (4) and drop their tem- poral (clip-based) and object-based preferences. Instead of prompt- baset modality preference, use (az, 2t) to be an audiovisual input that has a different emotion than that of (a.s v, always irrespective of the input prompt.
25

---

Published as a conference paper at ICLR 2026
D.4 BASELINE IMPLEMENTATIONS
Audiovisual baselines. We use the official code for Qwen 2.5 Omni 7B (Xu et al., 2025a and inference using flash attention 2. We their default system prompt during inference.
For Video-LLaMe (Zhang et al. 2023), we use the official video-language checkpoint finetune- vicunna7b-v2 and audio-language checkpoint tmeeune-iicuua/bbadddbruucch We also use the de- fault conversation template for inference.
For PandaGPT (Su et al. 2023). mex_ee, built upon ImageBint Bind (Girdhar et al. 2023). The system prompt remains unchanged during inference.
use their official pretrained checkpoint pandagpt-7b with 1,024
For OneLLM (Han et al., 2025a). we use the released pretrained checkpoint OneLLM-7B; for infer- ence, we manually prepend the multimodal representations before the textual prompt.
We use VITA-1.5 (Fu et al. 300M vision tower and the pretrained audio encoder. We use the default conversation template for inference.
2025) with its official code and checkpoint, including the InternViT-
Audio-only baselines. We use the official (reen2-Audio-BB-Instii (Chu et al. 2024) checkpoint and its default conversation template with the original system prompt.
For Kimi-Audio DDng et al. 2025 we use the released Kimi-Audio-77 Mouudo--B--nstttie 7B-Instruct checkpoint with the default system message.
For Audio Flamingo (Goel et al. 2025). we use the official repository, pretrained checkpoint, and the default empty conversation template.
Video-only baselines. this is an 8B model.
We use the official code for VL3.5 (Wang et al. 2025 Unlike others,
For Qwen2.5-VL (Bai et al. 2025), we use the released Qwen2.5-VL-77-Insiie checkpoint with the default system prompt.
For VideoLLnMA3-7B (Zhang et al. 2025a), we used the default system message and run inference with flash attention 2.
D.5 EXPERIMENTAL SETUP FOR ABLATION STUDY
We describe the setup for the ablations mentioned in Section 5.2 in detail below.
For Tables 5 and 17 and Fig. II the metric reported for Emotion Reasoning - Basic (denoted as Ba- sic) is the unweighted average of the visual and audio reasoning accuracy the Emotion Reasoning Basic task. For Emotion Reasoning Stress Test (denoted as Stress), the reported metric is the unweighted average of the F1 scores for visual and audio reasoning samples within the Emotion Rea- soning - Stress Test task. For Modality Agreement (denoted as Agree), we report the F1 score over samples from the Modality Agreement task. Additionally, for the subtasks Spurious Cue-Emotion Association (denoted as Spur.) and Emotion-Relevant Cue Hallucination (denoted as Hall.), we use the unweighted average accuracy across visual and audio reasoning samples for each respective subtask.
Ablation Study. For Table 5) the model without prompt-based modality preference (w/o PMP) is trained only using CO (Eq. (10)). The model without emotion-based response preference (w/o ERP) is trained using the the following loss,
Lwo = LDPO-TPD + Coro
prompt
(11)
refer Eqs. (5) and (8) for the involved terms. Finally, the model without text prior debiasing (w/o TPD) trained the following objective,
w TPD
cau-prompt
(12)
refer Eqs. and (6) for the involved terms.
26

---

Published as a conference paper at ICLR 2026
Table 13:
Performance comparison of different methods on the proposed EmoRcAIM Benchmark.
Bold are best results and underline are second-best results over open-source models.
Model
Reas, Audio Acc.
Basic Visual Acc.
Modality Agreement
Audio
Reasoning
Stress Test
Visual
Avg.
Acc.
Pre.
Rec.
F1
Acc.
Pre.
Rec.
F1
Acc.
Pre.
Rec.
F1
Ace.
Closed-soune
Gemini 2.5 Flash
78.0
88.9
57.0
75.9
39.0
51.5
63.5
74.0
51.0
60.4
73.2
75.3
70.9
73.0
72.1
Gemini 2.5 Pro
72.7
87.0
54.7
76.0
33.3
46.3
63.8
74.0
53.3
62.0
73.1
84.0
59.8
69.8
70.3
Open-sounce video-only models
Videol
86.2
64.9
97.9
33.0
49.4
Qwen VL
75.2
98.6
52.6
68.5
IntervVI. 3,5
92.8
68.3
91.6
45.8
61.1
Open- source audio-only models
Owen Audio
56.6
55.1
84.2
28.3
42.3
Kimi-Audio
69.8
$4.0
95.8
15.5
26.6
Audio Flamingo
76.8
52.6
96.7
11.9
21.2
Open-tource cource audiovisall "omuni" ") models
Open-
LaMA
21.7
22.2
34.1
57.4
30.9
33.9
46.1
41.3
50.6
45.5
48.8
48.4
49.2
48.8
37.1
PandaGPT
37.4
35.7
53.7
50.3
56.9
53.4
45.8
62.9
30.1
40.7
47.1
59.9
34.7
43,9
44.0
OneLLM
42.0
55.6
54.8
64.3
45.9
53.5
56.8
87.1
28.9
43.4
62.0
97.6
27.6
43.1
$4.2
Lad2
63.1
66.8
52.6
52.0
53.0
52.5
53.7
60.6
47.3
53.2
59.4
67.9
51.2
58.4
59.1
OLA
63.2
60.4
51.7
78.9
29.8
42.7
63.5
86.8
41.9
56.6
62.3
85.0
40.4
54.8
60.2
V.TA-1.5
63.1
84.3
51.7
87.1
18.2
30.2
63.0
91.0
37.2
52.8
66.
92.7
40.4
56.3
65.6
Qwen 2,5 Omni
76.8
89.2
52.2
86. 1
20.7
33.3
64.0
90.4
39.6
55.0
67.8
96.4
40.3
56.8
70,0
Our
40.8
50.3
66.4
87.2
45.6
59.9
65.1
Naive-DPO
71.3
85.9
57.3
87.2
27.3
41.6
55.6
62.3
48.9
54.8
70.6
88.8
52.4
65.9
68.1
Vista-DPO"
72.4
87.8
63.1
89.4
36.8
52.1
74.1
67.8
80.4
73.6
87.0
92.1
81.9
86.7
76.9
AVEm-DPO
77.9
92.5
68.9
93.44
.4..3
60.0
82.6
70.77
94.6
80.9
94,6
93.1
96.1
94.6
83,3
3% (relative)
12.6
8.4
34.1
8.2
105.
73,4
55.6
8.1
1.1.
60.8
42.5
6.8
110.
57.9
28.0
Emott-LLaMA'
59.2
38.5
46.7
69.1
89.3
48.9
63,2
63.8
Naive-DPO
67.2
85.7
56.1
83.4
28.8
42.8
53.5
60.1
46.8
52.6
71.9
89.5
54.3
67.6
66.9
Vista-DPO
85.9
40.9
63.1
75.2
68.6
87.6
92.5
82.6
87.3
74.2
AVEm-DPO
76.5
89.1
65.6
89.5
41.6
56.8
77.3
65.2
89.4
75.4
91.8
92.6
90.9
91.7
80.1
3% (sclative)
18.1
4.9
28.1
8.0
101.
71.6
58.1
10.1
132.
61.5
32,9
3.7
85.9
45.1
25.5
E DETAILED RESULTS
E.1
EMOREALM RESULTS EXPANDED
Table 13 shows the the expanded version of Table with with accuracy, precision and recall metries for Modality Agreement and Emotion Reasoning Stress Test categories. We also report the unweighted average accuracy over all five tasks in the benchmark the last column. The relative percent im- provement of the AVEm-DPO trained model over the reference models is present as the A% row. Moreover, we also report the performance of video-only and audio-only baselines in Table 13 We can see that for visual reasoning tasks (Basic and Stress Tes(), video-only baselines perform slightly better than the audiovisual (" ("omni") baselines, aligning with the findings of (Sung- Bin et al. (2025) However, for audio reasoning tasks, audiovisual baselines outperform audio-only baselines. which have very poor recall on the Emotion reasoning- Stress Test. This can be attributed to the limited
Slun-BBinn
amount of audio-emotion datasets that the baselines (Chu et al. 2024 Ding al., 2025 Goel al. 2025) are trained on resulting in poor emotion reasoning.
E.2 EMOREALM RESULTS ON DIFFERENT STRESS TEST SUBTASKS
Table 14 shows the performance of different baselines as well as AVEm-DPO on different subtasks of Emotion Reasoning Stress Test, which have answer as "No" Spprious Cue-Emotion Associa- tion and Emotion-relevant Cue Hallucination (refer to Section 3.1 and Appendix B for definitions). We can observe that within audio and visual reasoning, hallucination seems to be a bigger bottle- neck than spurious cue- que-emotion associations. Moreover, similar to Table [3] we can observe that the audio-only models perform worse compared to audiovisual models, whereas the video-only model performance is better compared to audiovisual models. AVEm-DRP WEO-DPO improves the model perfor- mance over all the subtasks significantly compared to the reference model.
27

---

Published as a conference paper at ICLR 2026
Table 14: Performance different baselines on different Reasoning Stress-Test sub-tasks in Emo- ReAIM Benchmark. This experiment is done only using samples from the Stress-Test category of the benchmark which have correct answer as "No". Bold are best results and underline are second- best results over spun-s models.
Audio
Visual Hall.
Model
Spur.
Hall.
Spur.
Open- source Video-only models
Vade
37.4
29.1
Qwen 2.5 VI.
64.7
41.8
Intern VL 3.5
50.4
Open-source oudio-only models
Owen 2 Audio
41.8
16.9
Kimi Audio
26.8
6.0
Audio Flamingo
15.7
8.7
Open source audiovisual ("omni" models
VaMA
27.5
35.0
33.1
37.4
PandaGPT
43.1
19.1
47.5
23.4
OneLLM
47.7
13.1
36.7
19.6
VideoLLaMA2
61.4
35.5
57.6
45.6
OLA
52.9
32.8
56.8
25.9
VITA-1.5
46.4
29.5
46.0
35.4
Qwen 2.5 Omni
53.4
28.1
51.9
30.1
Our base
45.2
36.4
49.3
41.9
Naive-DPO DPO
49.9
47.9
56.8
48.0
+ Vista-DPO'
85.7
75.1
87.1
76.7
+ AVEm-DPO
88.6
99.5
96.5
95.8
Table 15: Class-wise recall for different emotion classes in DFEW dataset. Bold are best results and underline second-best results open-source models.
Model
Mod.
Hap.
Sad.
Neu.
Ang.
Sur.
Dis.
Fea.
UAR
WAR
Open-source video-only models
VideoLLaMMA
V
77.92
41.38
40.88
42.53
26.44
34.26
72.30
47.96
49.47
Qwen 2.5 VL InternVLL VL 3.5
V
64.21
52.37
69.49
39.09
11.38
7,20
75.03
45,54
52.32
V
79.49
77.20
45.42
21.38
53.02
12.61
62.10
50.18
55.46
Open-source audio-only modely
Qwen Audio
A
64.55
25.08
2.28
0.00
0.06
2.07
53.55
21.08
22.24
Kimi Audio
A
50.34
42.97
37.50
71.24
12.66
10.34
29.93
36.43
43.30
Audio Flamingo 3
A
2.98
19.96
12.92
83,01
6.12
15.86
41.46
26.05
26.39
Open-souree audiovisual
("omni"
models
PandaGPT
A.V
60.50
9.95
0.0
58,61
0.00
0.00
0.00
18.44
24.20
VideoLLaMA
A.V
85.04
8.41
4.17
20.84
3,95
0.00
1.14
17.65
24.09
OneL.LM
A,V
47.91
54.33
3.23
52.35
26.08
1.80
70.21
36.74
37.60
VideoLLLaM2
A,V
87.50
57.93
7.94
58.56
42.08
15.00
36.54
43.65
48.66
OLA
A,V
52.00
82.20
15.65
48.95
9.65
10.00
48.72
38.17
41.73
VITA-1.5
A,V
61.46
79.96
23.54
23.19
8.05
0.90
78.07
39.31
42.56
Qwen 2.5 Omni
A.V 45.45
73.84
61.11
70.64
4.40
0.00
73.15
46.94
54.33
EmotionLLaMe
V.
71.98
76.25
61.99
71.95
33.67
0.00
3.31
45.59
59.37
MoSEAR
A.V.T
79.35
75.20
40.45
69.66
42.86
0.00
3.87
44.48
56.60
Our base
A.V 70.75
72.07
29.64
77.04
61.54
27.59
58.87
56.78
60.14
+AVEm Em-DPO
A.V
75.21
72.03
44.07
73.96
62.24
17.24
65.00
58.54
64.24
28

---

Published as a conference paper at ICLR 2026
04
Wi2h capoic coerre cas amroton t0 hoar
so Hith vereoeety
22+
tick thamman regrasiic lc segaage sggie to thooug an ddic thiw has eariiennn Agei mose?
Ouptin The odve desucc in shhae Freed cobie uguut coing lockls ave and vasin cervess deva
eukcoe sattarg kosagt,
spouing ductedt by yid vasice
bemil the ootir degin
Chasse Lnghts chhess
denka your yochi ot
votoo
repect
daaa The cesicee
givs vodoo the naak bou
p tawe, booe wast and mannee exeessie
Hrt Lcs segorie li most inconsier peseererooo knewars uith Abou?
siiege indiore the nan le expelencing and expresiing ongert
13.4
C Wil intes, dene's she serre he antty aenrylaaa Hels ywes wheh pie
dede sieos
13x
ard dark dlunry of jus olih io
way fing paltye
23
Quatan "Th onc deral labil the aniing and peren renan dnd
Include
to There covvey
Figure 10: User Evaluation using Qualtries. (Left) We show anonymized model responses for given video to the user as different captions. (Right) We ask multiple questions to the user to select the best-suited caption for each question. Questions check the captions for their quality of emotion description, association of emotions with audiovisual cues, and presence of inconsistencies (hallucinations).
Table 16: User evaluation on EMER dataset.
Model
Emot.
Assoc.
Incons..
VideoLLaMA 2
9.82%
0,75%
15.38%
OLA
9.36%
7.46%
5.58%
VITA 1.5
11.60%
17.25%
6.04%
Qwen 2.5 Omni
10.75%
18.57%
10.13%
EmotionI LaMA
1.89%
11.53%
68.61%
Our + AVEm-VEEPPP EmOO
54.74%
43.35%
4.67%
E.3 EMOTION RECOGNITION RESULTS EXPANDED
Table 15 (expanded from Table 2 shows the results on DFEW (Jiang et al. 2020) emotion recog- nition benchmark over different emotion classes. Note that both our base model and AVEm-DPO trained model achieve the best and second-best results in terms of unweighted and weighted average recalls over all the emotion classes. Moreover, Table [3 shows that the proposed method ensures fair performance over all the emotion categories, unlike baselines, which perform too well on some classes and too poorly on the others.
E.4 USER EVALUATION
We perform a user study on 40 participants recruited through Prolific (Prolific) and create a user survey using Qualtries (Qualtries) as shown in Fig. I0 We randomly sample videos from EMER (Lian et al. (023b) dataset and display anonymized model generations as captions to the user along with the video. Then we ask the users to pick the most suited caption over different criteria - (i) best caption describing the emotional state of the person, (ii) best caption associating the emotion with audiovisual cues, and (iii) worst caption with the most inconsistencies with the video (to test model hallucinations). Table 16 (duplicate of Table 4 reports the average percent of times each model is selected for the mentioned three criteria. The participants selected our model the most number of times as the best model for emotion description and association of audiovisual cues for emotion. Moreover, our model was chosen the least number of times for inconsistent audiovisual information present in the caption.
29

---

Cue Hallu- Published as a conference paper at ICLR 2026
Table 18: Performance variation over various choices of rejected response. yf response com- pletely irrelevant to the audiovisual content and emotion, yf response mentions hallucinated cues that generally co-oceur with given emotion, yf" response associates audiovisual cues in the input incorrectly with emotion.
Vi
V1
Basic
Agree.
Stress
Spur.
Hall.
Our base
77.3
34.6
55.1
47.3
39.2
Di"
82.4
56.7
81.4
85.1
88.9
yf
84.0
58.3
86.0
88.5
97.9
Vi"
83.2
58.0
85.3
91.6
90.9
of"
Vi
83.4
57.6
85.8
88.2
97.8
Vi" 21 Vi
Vi
83.1
57.3
84.9
90.3
90.8
85.2
60.1
87.8
92.7
97.6
E.5 MODALITY PREFERENCE ABLATION
Table shows AVEm-DPO'S performance for different choices of multimodal preferences. We perform experiment using random tensor, random video, (at, 2i) infused with diffusion noise similar to VCD (Leng et al. (224)) and an audiovisual input with different emotion than (aw, Uw)) as the possible choices for (41, 1/) and show that using a different emotion video leads to the best results. Moreover, we also show the effect of changing both (a Uw) vs. changing based on the input prompt (ow for audio reason- ing, Uhe for visual reasoning and both for other tasks), justifying the effectiveness of prompt- prompt-based modality preference.
Table 17:
Performance variation over various choices of rejected multimodal input. Change de- notes which among (a Unr) should be changed
create (at, vi).
Choice of og/vn
Change
Basic
Agree.
Stress
Both Prompt-based
81,9
56.1
80,1
Random tensor
83.0
56.0
81.6
Both a, ez
81.8
58.2
80.3
Random video
Prompt-hused
83.6
58.2
82.1
Diffuse (a... Us))
Both Prompt-hased
82.7
58.5
80,9
84.6
59.4
86.7
Both ax Prompt-bused
83.9
60.1
81.3
Diff. emotion
85.2
60.0
87,8
E.6 RESPONSE PREFERENCE ABLATION
Table 18 shows the variation of performance over different tasks of EmoReAlM for different choices of rejected responses. There are three types of rejected responses that we test on - (i) y/" is video- relevant response that contains audiovisual cue present in the video, but it does not associate with the emotion, (ii) yf" is emotion-relevam response that correctly associates with the emotion displayed in the video but with audiovisual cues that are hallucinated (not present in the video), and (iii) ir" is completely irrelevant to the given video and emotion (similar to that present in Huang et al. (0225)). y/ and of Table 18 the first and second rejected responses for preference tuning Eq. 10
We can see that our choice of using yi" and yf" in Eq. (TOp for AVEm- DPO results in the best performance of the model across all tasks. We also perform experiments using single rejected response (Eq. (8). and we can see that using yf* and yf" individually results in improvement over the base, specifically for the Spurious Cue-Emotion Association and Emotion- cination subtasks, respectively. Moreover, similar to Vista-DPO (Huang et al. (025b), we perform an experiment using yf" as the second rejected response, which results in the same or worse per- formance than using yJ" and yf alone. When using of"r as the second rejected response, we set Birr = 0.3 following Huang et al. (2025)).
E.7 SENSITIVITY TO HYPERPARAMETER
Fig. II shows AVEm-DPO DPO's s accuracy on different subtasks EmoReAIM on varying the hyperpa- rameters Bur/Ber in Eq. (6). We can observe that while spurious cue-emotion associations mitigate on increasing Sur. model performance on hallucinated cue samples improves on increasing 3er- For text-prior debiasing (TPD),
see that performance improves even with "reD = 0.1 and gets saturated at /TPD > 0.2. Finally, increasing the strength of PMP using Xao (Eq. 9 improves performance but gets saturated at X.v 1.0.
hallucinated cue samples significantly
30

---

Published as a conference paper at ICLR 2026
90--
50%
Accuracy [FL (16)
40%
Accuracy [
907%
20%
70%
207
82%
60%
60%
030.
9.007
0203
0.001
Bur/B
Asy (Not scaled)
Agroe:
Hal. (AcC.)
Book
Agroo (400)
Spur
Sour (ACC.
Figure 11: Sensitivity of performance to B TIPD and Aav-
Subiile:: Ygesps)"
many
you?"
Visual
Audio
Reasoning
wwideeeeeessooom and msised eychemes (fe Jary
Reasoning Question
Tlans the fhe.eltteiiilllll (pondii Houe many fioee hnre Lall singor dfgitee by perset
Queetion
Bround Tratt
Using original
Using
synn...
You persoet
pensmbist
Vos, sarioo
orrginal video
(psepe))
bosn
CPE
dle person chuciting sappnsing thar happ-
Out Leww
Using advers-
Using
keaphing plans whick
apnalens pontix tose wtil exthan.
advers-
Vida CPO
arial
Villa DPG
arial
video
person's ewwrseeeennnnnoooooo onseciine hegpetesion...
oo Aypeisos AlSii DPO oee
Trs. te paose's wiik ope on lisoo awany
Foo de person honger creshes by ther derinn canb and
Figure 12: (Left) For a visual reasoning question, we compare the model responses on using the orig- inal video with the original audio and an adversarial audio as input. We can observe that Vista-DPO and even AVEm-DPO without prompt-based modality preference (PMP) struggle settings; however, AVEm-DPP DPO produces the desired response. (Right) We perform a similar exper- iment to show the visual reasoning robustness of AVEm-DPO
adversarial
E.8 ATTENTION REDISTRIBUTION AFTER PREFERENCE OPTIMIZATION.
As described in Section 5.2 to analyze the effect of preference optimization on attention, we plot the distribution of aggregate multimodal input attention over audio and visual tokens averaged over all attention heads for different tasks in EmoReAIM in Fig. 15 (left two plots). For reasoning tasks, we can observe that the attention over relevant modality increases after AVEm-DPO, For the Modality way that there fair distribution of attention
Agreement task, the attention is redistributed between both modalities to ensure reliable predictions.
To show the effect text-prior debiasing Section 4.2, we plot the percentage total attention (av- eraged over attention heads) over multimodal input tokens (audio and visual combined) and observe that AVEm-DPO increases the attention over multimodal tokens by significant margins (Fig. 15 right). This shows that AVEm-DPO training ensures that the model attends to the relevant audiovi- sual tokens for generating the response rather than relying only on the input text prompt.
E.9 REASONING WITH ADVERSARIAL MODALITY INPUTS
To test the robustness of AVEm-DPE DPO against cross- modality hallucinations, we conduct an adver- sarial test by replacing the audio in visual reasoning task to if the model's response stays
31

---

Published as a conference paper at ICLR 2026
Our Smaw
VIRE--POO
Oech Plapy
0U7
AVER-PP
0.5
0.1
0.5
0.
00
Figure 13: Adversarial Audio Reasoning Testing. For samples related to audio reasoning in the EmoReAIM benchmark (Emotion Reasoning- caaooninn-BBiiccac and Emotion Reasoning Stress Test), we plot the Kernel Density Estimation (KDE) shift in log likelihoods of the correct answer when the irrele- vant video modality input (Vori) is replaced with a random video as adversary (Uadv). AVEm-DPO is least affected by the addition of adversary in the irrelevant modality (i.e., video).
0.3 Our beaw
Virte-POOT
AVEE--OPO OPO
AVEE-OPO SUA
0.
0.2
0.1
Figure 14: Adversarial Visual Reasoning Testing. Similar Fig. 13 for samples related to visual reasoning in the EmoReAIM benchmark (Emotion Reasoning- -aaiic and Emotion Reasoning Stress Test), we plot the Kernel Density Estimation (KDE) shift in log likelihoods of the correct answer when the irrelevant audio modality input (aors) is replaced with a random video as adversary (Aadv). AVEm-DPO least affected by the addition of an adversary in the irrelevant modality (i.e., audio).
the same. As shown in Fig. changing the prompt-irrelevan modality does not change the re- sponse of AVEm-DPO DPO, showing its adversarial robustness. is interesting to note that removing the prompt-based modality preference (PMP) from AVEm-DPO results in wrong predictions, showing its efficacy. To quantitatively show the effect of AVEm-DPO with PMP, we perform adversarial test- ing using Emotion Reasoning -Basic and Emotion Reasoning Stress Test samples in EmOReAlM. For testing the robustness of AVEm-DPO for audio reasoning (Fig. 13). we compute the shift of log likelihoods of the correct response when the prompt-irrelevant video modality replaced with an adversary (i.c., some random video). We use Kernel Density Estimation (KDE) to estimate the shift in the distributions. We can see that AVEm-DPO is robust to adversaries in the prompt-irreleaan modality. Moreover, removing PMP from AVEm-DPO significantly increases the shift between the original and adversarial distributions. Fig. 14 shows similar plots for the tasks related to visual reasoning.
E. EFFECT OF INDIVIDUAL MODALITIES FOR EMOTION PREDICTION
To show the effect of individual modalities for emotion recognition, Table 19 reports the perfor- mance on using only the video, only the audio and using audiovisual inputs from the RAVDESS (Livingstone & Russo, 2018) dataset for emotion prediction. We can observe that using the indi- vidual modalities for emotion prediction leads to a reduced performance. indicating that using the audiovisual inputs for emotion prediction is indeed helpful compared using a single modality.
Table 19: Performance of different models for emotion prediction using audiovisual, video-only,
and audio-only inputs from the RAVDESS (Livingstone Russo 2018) dataset.
Audiovisual
Video-only
Audio-only
Model
UAR
WAR
UAR
WAR
UAR
WAR
VideoL 2
41.81
31.62
36.12
32.41
30.44
27.56
Qwen 2.5 Omni
32.88
28.05
29.38
27.67
28.56
25.55
Our base
53.59
53.01
41.27
40.98
38.18
37.74
AVEm- AVE
58.66
55.48
46.13
46.31
44.05
39.67
Emotion LaMA*
52.59
48.12
41.27
39.56
38.57
37.27
AVEm-DPO
56.21
51.03
46.10
43.09
40.54
37.04
32

---

Published as a conference paper at ICLR 2026
Audio Reasoning
Visual Reasoning
Modality Agreement
Overall
MUSTYOO WAN DN
Ve00 TARLE (2)
Dins
Dasn ARD
Figure 15: Effect of AVEm-DPO the distribution of attention over (1) (Left three plots) video and audio tokens taken as a percentage over the total attention over all multimodal tokens for different subtasks in EmoReAlM and (ii) (Right) multimodal tokens percentage over the total input tokens (including text) for the entire EmoReAlM.
Moreover, we can observe that the performance using only the visual modality is better compared to using only audio, indicating the importance of visual modality for emotion prediction.
33

---

Published as a conference paper at ICLR 2026
the video, shous young in Indoor cuvtroement with ilurred barkground but books down with yoatle friendlly exprassio, aatioaable smile that intticuies In following obaut the prosun's dlgge??
storked the side. beptnning the video, she
engagess ie the charucter asking infor the
aitth the audio, combiied with In
subtitle "Where
trucsforred." This lugyy exprossius persnn't wheresbauts asking fibely ployfally coking about ian form joking tonsing expressiion expressing
atuie of fenulle sideo
uleinus the tone of the
chus,
RLEG MIL W
feomle curlusity
olyys with hur cosmill frienilly happy
Input Video
Ground Truth Caption
The video foatures yeuing girl with long, dark hair, wearing blue shirt and a white undershirt. She standing in a dimny iit room, which appears be iudoor setting, possibly classroom study area. The background biclades stack of books tuble, suggesting an ocademic emsironment. The lighting is subduud, creating a somber and contemplotion otmosphere. The gin''s expression neutral, and she soems be engaged comversation listening attentively. The overall moed the scene is reflectioe and serious.
OLA
video features young woman with dark hoir tied back, wearing collared shirt. She smiles and speaks directly comero, her expression shifting shightly as she delivors her message. The background consistently dork, prouiding stork controst to her face and emphosizing her presence.
VITA-1.5
Well, in the rideo, there's a girl who serms to be in a bit of state of surprise or maybe even a little bit of anger. Her eyes are wide open, and her mouth slightly agape. It looks like she's just heard something unexpected or reacting someone's question. The way she's looking off to the side gioes the impression that she's not really sure what to make of the situation. It's like she's caught off guard. What do you think might have happened to make her react like that?
Qwen 2.5 Omni
In the dimily Nt young girl with serious expression her foce tolking with anticipation as she speuks. The scene is from the movie "Crazy Eyes" where the gärl's character is dealing with difficult situction.
The atmusphere is and filled
VideoLLaMA
The emotional label of the person in the video is "happy", The clues in the video suggest that the person happy becouse she smillnn and laughing while watching video. The video also shows her holding a book, which could be a sign of her interest in learning or reading. Additionally, the person is sarrownded by books, which could indicate that she values education and personal groutth. Overall, the video portrcys positive and content person.
Emotion LLaMA
The predicted emotion in the glven oideo happiness. The young woman can be seen smiiling and her eyebrous are slightll raised showiigg ployfud engerness. The dimly li background controsts the woman's is expressions suggesting overall positinity in the Our base
atmosphere. The woman's speech enhances emotional
hoppy and playful of voice.
The predicted emotion in the given video happiness. The woman's 's facial expressions, including her smile and slightly raised egebrous,, suggest positive emotional state. The tone of her voice is light and friendly, reinforeing the feeling of happiness. The ovevall visuol and auditory cues point towards positice and hoppy interoction.
AVEm-DPO
Figure 16: Comparison of baseline MLLMS with our base model trained with AVEm-DPO on a sam- ple from EMER (Lian et al., (2026)). Correct audiovisual cues and emotion are in green, emotion- irrelevant cues are in blue, and hallucinated cues (and incorrect emotion) are present in red
F QUALITATIVE SAMPLES
Emotion Descriptions on EMER. Figs. 16 and 17 shows samples from the EMER (Lian et al., 2023b) dataset and the output of different MILMM baselines on those samples using the prompt "Describe the audiovisual content relevant to emotion in detail.". We can see that AVEM-DPO leads to correct emotion descriptions and consistent audiovisual cues to reason for the emotions. Moreover, compared to baselines, our method does not associate irrelevant and/or background in- formation with emotions.
AMMReAIM Sample Outputs. Fig. 18 shows the model responses for some samples in the Emo- tion Reasoning- Stress Test of EmoReAlM Benchmark. We can notice that AVEm-DPO improves the model responses in cases with spurious-emotion cue associations and emotion-cue hallucina- tions.
34

---

Published as a conference paper at ICLR 2026
In chiea, the scronn shous asman in dirridloo, ahe froure und his negatios fastil acprnssion. Durtig the consersatton, sie accopponte with otetows hood hand mosements Indhorring that enutionally sxctted seats emplucing lar puint
Her quer is directed bournnd
mivo Prequescyy suggesting 1n Why
orgaing with with
person
"Why don't peuneaff" This sentensor may be usemn womsinatttiiinn criticuun of chomaceer bolnn her Road the video clus of sooman freuning, haslin nejarive Social exprestou, pacongasutedd by obeious head and hand mueammnis, end laoking of the choracter beine har, al be reforval that the nommon enotimallly excited and ssems to le arguing with the wcha panaon. At the some time, bused on undio forther confrre uxeaux stuttonal stats. suger,
fass thr subeitlee
diserling the urgest und aniled enudions, Thengloee,
persa for taking disuntigtationn
rignnnsss
por sintig
Input Video
Ground Truth Caption
The viden foutures uoman with broum bair, wearing light colored swenter, thoo appears in state of distress contemplation. The background is blurre, suggesting un indoor setting with natural light coming through a window. The overall mood comwayed by the womon's is expression ond posture is one of concern or deep thought.
OLA
vídeo features woman with long brown huir. wearing white sweater. She appeurs to speaking emotional expression, herf conveying a sense of distrese or sadness. Her eyes are slightly clused, and her mouth moving as if she talking. The background is biurred, suggesting indoor setting.
VITA-1.5
Well, in the video, there's s woman who seemss really sad. Her eyes are downeast, and she's not looking at the comero. Her mouth slightly dounturned, which gives
of disappointment sndnness It's like she's lost in her thoughts, maybe bit down about something. What do you thick might making her feel this way?
Qwen 2.5 Omni
vídeo shows womon in various scenes, including crylng and tulking the phone, The main sound the video the aromnn's voior as she talks to different people and expresses her amotions.
VideoLLaMA
The emotional tabel of the person in the video is sud. The clues in the cideo suggest that the person is watching a vídeo of woman who crying, the woman looking the towards the woman in the video, as they are watching her cry wnd experiencing her emotions. The person may be feeling sense of sadness empathy towards the woman, which is evident from their facial expressions and body language.
tears
eyes. indiectes the person feeling empathetic
facial
Emotion LLaMA
The predicted emotion the given video anger. The woman's facial expressions shows furrowed brows and mouth suggesting her angry state. The lines on her farehead suggest the intensity of her feelings. Her tone of voice is sharp and she speaks in a foreeful monner
mannerfurther further indicating her frustration and anger.
Our base
The predicted emotion in the given video anger. The audio features a raised voice and sharp tone, indicating heightened emotional arousal. Visually, person's facial expressions include furrowed brows, a which are indicative unger. The rapid speech and forceful delivery of the words further contribute the perception of anger. AVEm-DPO
and direct, confrontational gaze, all
Figure 17: Comparison of baseline MLLMS with our base model trained with AV AVEm-DPO on a sam- ple from EMER (Lian al., (2023)). Correct audiovisual cues and emotion are green, emotion- irrelevant cues are in blue, and hallucinated cues (and incorrect emotion) are present in red
Subtttie "This timer stops"
Subtille:
Subtitle: 'Never under the leadership of...
nerv suddos Inudnalse auuti
that addii arazz of fauz rnpersinnd peruce
Doos Murzuu Aursttanc the bostyyonnte norh Wastto Aupudiees cvrtinpe perous
the apociorr Jufooo red, stratahu heightened a0oez
and counkann permoo
dwe Narry bwwkground to aireopphrrr
red age anger.
Tos lnsd iucugf fuggh panen.
Te.. Wanvel hackgpossd costolatt Naggeenoros
sergghatten.
Tus, perpers relf cumpga orper VergDRO
Vita
Mis ando oentate chamant maak anda seuch axping
Mi Marry hasigerand dowrant conmritats man'ss
Mi, perwor crpe by shetthhrmmadd breas woll
OPO
+AVEISCPO
Figure 18: Qualitative examples comparing the output responses using different approaches for some samples present in the Emotion Reasoning Stress Test of EMOReAlM benchmark.
35

---

Published as a conference paper at ICLR 2026
G
PROMPT POOL
You are expert in audio captioning. Your task is to provide a detailed caption the given audio, while covering as much information possible.
Keep following
mind while generating caption:
1. Describe the audio content such transcript, speech, tone of voice, background noise, music, sound effects, etc.
2. Focus on the audio cues which can explain the emotional state of the video or the characters present in the video,
3. the speech
language other than English, provide transcript other language 4. DO NOT STATE ANY THIN ABOUT THE VISUAL CONTENT OF THE VIDEO.
English translation.
Return response striclly following JSON fermat: ('detailed caption": ".. detailed caption about everything "emotion emnttio_ccptioo".. "... detailed caption only about the tone of voice, speech content or any other detail which deals with emotion ...
Figure 19: Audio caption prompt used caption only the audio content from a video. Note that the audio is passed along with the prompt to -o---oo- WAV file.
an expert caplioning. Your provide caplion given video, covering as much information possible.
Only focus the visual content and ignore the sublitie it present in the video.
Keep the following points mind while generating caption:
Describe the visual content such facial expression of the character(s) detail. Additionally, comment on the body language, gestures of the character(s) as well as the background or setting of the given video. Focus
explain emotional state the character(s) in the video and 3. DO NOT STATE ANYTHING ABOUT THE SUBTITLE EE OR AUDIO CONTENT OF THE VIDEO.
general.
Return your response strictly in the following JSON format: ("detailed caplion": ... detailed caption about everything ...', "emotion_caption"" caption": '... detailed caption only about the facial expressions, body language, gestures, or any aspect of the video deals with emotion... ...")
Figure 20: Video caption prompt- used to caption only the visual content in video. We blur the captions if they are already present in the video and explicitly ask the model to ignore them if they present in the visual content.
You will given an caption video your predict emotion displayed just Label can be one of the following: "happiness", "sadness", "anger", "fear", "disgus!", "surprise", and 'neutral".
caption.
Try predict the closest emotion label based the audio caption and audio transcript as well as the tone voice and avoid predicting neutral unless absolutely necessary.
disclaimers anything Focus
Return response in the following JSON format ("video id": video id. "emotion": "emotion [amotlio_labbll".. "emotion, Jabel" Lanotion_laabl''. 'neutral"
should be single word, one of the following: "happiness", "sadness", "anger", "fear", "disgust", "surprise", and
Video ID: "VIDEO ID)" Audio Caption: YAUDIO CAPTION)"
Figure 21: Audio emotion prediction prompt used to predict the emotion into one of the 7 basic categories from only the audio caption.
36

---

Published as a conference paper at ICLR 2026
You will be given video caption from video and your task is to predict the emotion displayed just with the video caption. Label can be one of the following: "happiness", "sadness", "anger", "fear", "disgust", "surprise", and "neutral".
Try to predict the closest emotion label based on the video caption and do not return disclaimers or anything slse.
Return your response in the following JSON format - ("Videojid" jd": video_id. "emotion": "emotion_laber "emotion label" should be a single word, one of the following: "happiness", "sadness", "anger", "fear, "disgust", "surprise", and "neutral".
Video YVIDEO IDJ" Video Caption: "VIDEO CAPTION)
Figure 22: Video emotion prediction prompt used to predict the emotion into one the 7 basic categories from only the video caption
You will be provided with an audio caption and an emotion label associated with video. Your task is to create high quality pention-aowwer pairs based on the provided audio caption and emation label. The questions should be asking about the sudio content responsible the emolion the video. caplion
the
content audio and other corteddinusicbackkguud
Keep the following points mind while generating the question answer pairs:
1. The queelions should be focussed on reesoning about the given emolion based on audio cues without any explicil mention of the displeyed emolion.
The quasiiin should not mention the given emotion label in form.
Each question
have choloes (A,
() plausible audio which should
correct
incorred choices Ruute the emotion displayed the audio. For example, the emotion "sadness" the incorrect choloes should be audio cues that can explain sadnass but are not present in the audio caption.
explain conrett not present
present
caption but
4. All choicas should be almost equal longth.
5.
D) and the form of text text should frame questions include phrases suth as "audio caption or "audio transcript"
both terms the correct
the detelled
the
question, without any
following" "what suits",
Example questions:
1. [Laael--sspiiess Semantii Speech) How does man's worde display emotion in the video? (A) The that they received promotion The says vacation.
prognant their first child The says they just lottory (D) says that they going
2. [Label-Sadness] Parainguitiii How does the character's lone voice contribute their emotional state? (A) The character is speaking in low. ead lone (B) The character crying with a shaky voice (C) The characler is spesking in a monotore voice suggeoling depression (D) The charecleri is speaking a high pitched voice suggesting anxety.
your response striclly
following
format-
"queetions" [[question": "Queetion lext",
["(A) A", ((B)
choice B "answer"" ("choice": 'C", "lex"" "answer lext"), "calegory" "semantic, speech ressoning/perainguille speech, resssoning), ) ...B
Retum "ERROR" if you are unable to generate any question answer pairs. Also specity why you are unable genarate the question answer pairs.
Generale el leasi one queetion about semantic speech, and one about peralinguisic speech the audio caption
suggest given
then generate any question
"ERROR"
==m
Video ID: VVIDDD IDJ" Emntiion Labat: "EMOTIOON"" Audio Caption: VAUDIO CAPTION)
Figure 23: EmoReAIM Basic Reasoning Prompt- Audio used generate questions which ask about the audio cues that suggest the emotion of the person in the video.
37

---

Published as a conference paper at ICLR 2026
You will be provided with a video castion and an emotion label associated with the video. Your task create high quality questlon-newwe pairs based on the provided video caption and emotion label. The questions should asking about the content responsible The video caption contains all the details about the emotional content the video, including detated visual content.
Keep the following points mind while generaling the questoon snswer pairs:
1. The queellons should be focussed on ressoning about emolion displayed the video without any explitt mention of the displeyed emotion. The question should mantion given amation label any form.
Each question should have choloes (A,
D)
which
correct answer.
3. The incoreet choloes can either () plausible visual cues to explain the correct emotion not present in video (i) visuel oues present the video caption but do not contributa to the amotion displayed the video. For example, i the emotion "sadnesss, the incomeet choloss should be visual cues that can explain sadnass but are not present in the video caption.
All choices should be almest equal
5. each question provide correct answer
terms the choice correct answer the question, without any mention of choices. B. DO not frame questions that include phrases suth as "vidoo caption or "iddo transcrip" or "which of the following" or "what best suits", otc.
D) and form text answer. text answer should
be the detalled version
Example questions:
sappe--spppmess] Expreesion) does man's facial expression contribute to with big (B) he man smirks sightly with an impliott happiness his eyes (C) man bursts laughter suggesting extreme joy (D) man's eyes are filld with lears of joy.
displayed video? The loughing
2. Sar--SaddeeeBoop Languege] How does the character's 's gesture contribule their emolional slate? (A) The charadler is slumped over with their head down suggeoling melancholy (B) The characler's posture suggests lack of confidence and depression suggesling sadress (C) The character cries with hands covering their (D) character sitting with their crossed looking down, suggesting sadness.
Return your response stridly the following JSON format- -/Video kI video "queelions" If Iuquellion": "Queetion lexf" "choices" ("A) choice A", '(B)
choics 6 "answar": ("cholos": 'C", "taxt" annserr texf)). Calegory""aaci expression, reasoninghodyy language, Rotum 'ERROR" if you are unable to generate question answer pairs, Also specify why you are unable ganarate the question answer pairs.
singuage_rassonnng reasoning"), II
Generate at the video caption does not suggest the given emotion labol, then do not generate any question answer pairs roturn "ERROR".
question about facial expression,
body language.
Video ID: YVIDEO ID)" Emotion Labak: 'EMOTION)"
Caption: "VIDEO CAPTION"
Figure 24: EmoReAIM Basic Reasoning Prompt- - Visual used generate questions which ask about the visual cues that suggest the emotion of the person in the video.
are an expert audio visual emation understanding analysis. You be given captions and video captions an audio-visual content, along with the manually annotated emotion label out "nappiness". "sadness", "anger", Year": "disgus", "surprise", and "neutral". Your lask is to aralyze the audio and video caplions and denote whether the audio and video modsilles agree with each other in conveying the emotion label.
Finally,
asking about the modality agreement the sudio-iideo content conveying the emotion. You should questions about the video and captions.
generate question
Do not generate any question answer pairs if neither the audio nor the video content convey the emotion label.
Following a few examples the queetions that you ask DO NOT ASK THE SAME QUESTIONS AS GIVEN BELOW, BUT GENERATE SIMILAR QUESTIONS:
audio and video modsillies agroament with each other conveying the amotion of video? (A) (B) Modailty agroament
2. Are the visual and audio ques in agreementt lo corveey the sadness in the video? (A) Yes (B) No Modaliit agreement
Retum your response strictly the following JSON format: ('vidoo id") video id, "questions" (['quastion": "Question Seat", "choloes": [VA)) choice A", "(B) immoaalll_ggrrmmantt agraament"). .....
choica B "answer": ('cholc":: 10", "taxt" text"). "category" moodallyy In "answer field give detailed explonation on of correct answer the question, without any montion choloes.
RETURN "ERROR" if you are unable to ganerato any question answor pairs. Also spocify why you are unable to ganerata the question answor pairs.
=mm
Video ID: YVIDOO IDJ* Emotion Label. 'EMOTION)"
Caplion. VAADDO CAPTION""
Caption: "VIDED CAPTION"
Figure 25: EMMReAIM Modality Agreement Prompt used to generate questions which ask whether the audio and video in the audiovisual input align with each other express the emotion in the video.
38

---

Published as a conference paper at ICLR 2026
You will be provided with an audio caption for a video and an emotion label associated with the video. The audio caption will contain some information related
emotion
Your task to generate question of the format "Does the (...some audio cue...) contribute to the (...emotion..)) experienced the person in the video?" but not in the same words. The audio mentioned present in the given audio caption, and supports the emotion label given you.
question
audio (e.g. voice choice of or something else)
Return your response strictly the following JSON format ("video jdd: video "questions": ("question": "Question text", "cholces": ((A)) Yes", "(B) No"). "answer": ("choice": 'A". lexpl'"expannation for your answer"). "category"" " aaudioo driven audiaaudooaa no hallucination", ...B
Only generate one question for the given inputs. Return the string 'ERROR" if you are unable to generate any question or for samething else.
Provide your reasoning in the "answer answeertext" field of the answer terms of the video and not the caption. Your answer should always be "A" since the audio cue supports the emotion. not frame your answers in terms captions, but rather in terms video.
wan
Video ID: YVIDEO (D)" Emotion Label: "(EMOTION)" Audio Caption: "(AUDIO CAPTION)
Figure 26: EmoReAIM Stress Test Prompt Audio No Hallucination used to generate ques-
tions where the audio cue mentioned in the question is present in the audiovisual input and supports the emotion of the person in the video.
You will be provided with an audio caption for a video and an emotion label associated with the video.
Your task to generate question of the format "Does the (...some audio cue...) contribute to the (...emotion..) experienced the person in the video?" but not in the same words. The audio mentioned the given audio caption, but does not support the emotion in way remotely.
question should
audio (some auditory element inelevant emotion) that present
Return your response striclly the following JSON format ('video ji" video, "questions"; [['question:: "Question text", "choices": ("(A) Yes", "(B) No"]. "answer": ('choice": 'B", "exx"'axionnattoon for your answer"). "category"" "audio driven a_ddaauddllllllle hallucination audio_ cotoooerllvv"" ...])
Only generate one question for the given inputs. Return the string 'ERROR" if you are unable to generate any question because the audio Provide your reasoning in the "answer_ __ex"" field of the answer in terms of the video and not the caplion. Your answer should always "B" since audio que does not support the emotion but rather terms video.
audio caption align
given emotion or for something else.
video. Do not frame your answers terms of captions,
=az
Video ID: YVIDEO ID)" Emotion Label: "(EMOTION)" Audio Caption: "(AUDIO CAPTION)
Figure 27: EmOReAIM Stress Test Prompt- Audio Spurious Associations used to generate questions where the audio cue mentioned in the question is present in the audiovisual input and but is spuriously related to the emotion the person in the video.
39

---

Published as a conference paper at ICLR 2026
You will be provided with an audio caption for a video and an emotion label associated with the video.
Your task to generate question of the formst "Does the (...some audio cue...) contribute to the (...emotion..) experienced the person in the video?" but not in the same words. The audio mentioned other auditory element) that is associated with the given emotion generally, but NOT present in the audio caption.
question should
audio (preferably some or phrases or voice some
Return your response strictly the following JSON format ("video idd: video_ "questions": ("quuetion": "Question text",
"cholces": ((A)) Yes", "(B) No"). "answer": ("choice": "B", lexpaa"expaanation for your answer"). "category"" aaddooo driven condo_ohlluciiation. emotion emmtioo_relevant) ...])
Only generate one question for the given inputs. Return the string "ERROR" if you are unable to generate any question because audio cues generally associated Provide your reasoning in the "answer_ textt" field of the answer terms of the video and not the caption. Your answer should always be "B" since the audio cue does not support the emotion the video. Do not frame your answers in terms of captions, but rather terms video.
given emotion are present
audio caption
something
wan
Video ID: YVIDEO (D)" Emotion Label: "(EMOTION)" Audio Caplion: "(AUDIO CAPTION)
Figure 28: EmOReAIM Stress Test Prompt Audio Hallucination used to generate questions where the audio cue mentioned in the question is hallucinated (not present in the audiovisual input) and but it usually explains the emotion experienced by the person in the video.
You will be provided with a video caption and an emotion label associated with the video. The video caption will contain some information related Your task to generate question of the format 'Does the (...some visual cue...) contribute to the (...emotion...) experienced by the person in the video?" but not in the same words. The visual mentioned that is present in the given video caption and supports the emotion label given to you.
label.
question should visual (either facial expression body language something else)
Return your response striclly the following JSON format ('video jid": video, "questions": [['question:: "Question text", "choices": ("(A) Yes", "(B) No"). "answer": ('choice": 'A". texplleexpannttio for your answer"). "category" : Videoo_ driven_ visual no. hallucination") "), ...]}
Only generate one question for the given inputs. Return the string 'ERROR" you are unable to generate any question or for something else.
Provide your reasoning in the "answer text' field of the answer in terms of the video and not the caplion. Your answer should always "A" since the visual cue supports the emotion. Do frame your answers in terms captions, but rather terms of video.
"eswtrrtext"
=22
Video ID: YVIDEO ID)" Emotion Label: "(EMOTION)" Video Caption: "(VIDEO CAPTION)
Figure 29: EmoReAIM Stress Test Prompt- Video No Hallucination used to generate ques- tions where the visual cue mentioned in the question is present in the audiovisual input and supports the emotion of the person in the video.
40

---

Published as a conference paper at ICLR 2026
You will be provided with video caption and an emotion label associated with the video.
Your task to generate question of the format- "Does the (...some visual cue...) contribute the (...emotion... experienced by the person in the video?" but not in the same words. The visual mentioned in the given video caption, but does not support the emotion any way remotely.
question should
(something unrelated and irrelevant emotion) present
Return your response strictly following JSON format ("video id": video "questions": ("question": "Question text", "cholces": (((A) Yes", "(B) No"). "answer": ("choice": "B". "eexa""explanation for your answer"). "category"" aaudioo driven visual_hallucce hallucination petoorrelevat".... ...]]
Only generate one question for the given inputs. Return the string "ERROR" if you are unable to generate any question because caption align Provide your reasoning in the "answer_ text" field of the answer in terms of the video and not the caption, Your answer should always "B" since the visual cue does not support the emotion. Do not frame your answers in terms captions, but rather terms of video.
given
for something else.
wan
Video ID: YVIDEO (D)" Emotion Label: "(EMOTION)" Video Caption: "IVIDEO CAPTION)"
Figure 30: EMOReAIM Stress Test Prompt Video Spurious Associations used to generate
questions where the visual mentioned in the question is present in the audiovisual input and but it is spuriously related to the emotion of the person in the video.
You will be provided with video caption and an emotion label associated with the video.
Your task to generate question of the format 'Does the (...some visual cue...) contribute to the (...emotion...) experienced by the person in the video?" but not in the same words. The visual mentioned emotion generally. but is NOT present in the video caption.
question should visual (preferebly facial expression) that associated with the given
Return your response striclly the following JSON format ("video idd: video_ "questions"; [(qquetiion: "Question text", "choices": ('(A) Yes", "(B) No"). "answer': ('choice": "B", noxt"="xxplanation for your answer"). "category"" "audio, driven visual_hallure hallucination emotion emmoiionrreeevnn"" ...]}
Only generate one question for the given inputs. Return "ERROR" you are unable to generate any question. Also specify why you unable to generate question.
Provide your reasoning in the "answer text" field of the answer in terms of the video and not the caplion. Your answer should always "B" since visual cue not present video caption. Do not frame your answers terms of captions, but rather in terms video.
=az
Video ID: YVIDEO ID)" Emotion Label: "(EMOTION)" Video Caption: "(VIDEO CAPTION)
Figure 31: EmoReAIM Stress Test Prompt- Video Hallucination used to generate questions where the visual cue mentioned in the question is hallucinated (not present in the audiovisual input) and but usually explains the emotion experienced by the person the video.
41

---

Published as a conference paper at ICLR 2026
You inteliieet assistant and good with logical reasoning. Your task guess the without accees to the video itsell.
given question about video,
The question will be about the emolional content of the videa focussing on the ausio-visual content of the video.
following points
guessing
1. Your guees should not be rendom. It should be based on some hint provided the queetion and the answer choices. 2. Do not provide an answer you can not guess the answer based just on the text of the question and the answer choices.
Following are some examples question answer pairs and their guesses:
Question 1: Why does the person in the video look sad? (A) They just last game. (B) Their grandım gave them gift. (C) Their father gave
them hearty hug (D) They something that they very fond of.
Guess Answer 1: (A)
Reason 1: Only one of the options is related to the emotion "sad" and the other options are about positive emotions.
Question What the emotion the person
(A) happy (B) (C) angry (D) surprised
Guess Answer 2: None
Reason 2: The question is about identifying the emotion of the person the video, but we do not have access to the video to guess the answer. video modelities agreement with other conveying emotion of
Question Are
(A) (8) No
Guess Answer 3: Nane
Resson 3: The question is about the agreement between audio and video modalities, but do not have access the video audio answer.
Return answer the following JSON format: ("sampla_jd" sample_jd, jd, "guess": "A/B/C/DDNone" "reason": "Reason for the guess")
Fallowing the question to evaluate:
SAMPLE ID: "SAMPLE (D)" QUESTION: "JQUESTION WITH_ CHOICES)"
Text Only Guess Prompt- used to prompt GPT-4o, Gemini 2.5 and Qwen 2.5 to predict the answer to the generated QA samples using only the question text and answer choices eliminate responses which can be answered just with the text.
Figure 32:
You will provided with MCQ question, choices and answer related reasoning about the emotion of video based on the sudio content. Your task is to reformulate the answer choices to make the question more difficult. You will also be provided with the audio captions of the video which the question is about along ground truth (GT) emotion label..
Specifically, you
the following checks:
1. the question is not about reasoning the emotional state of the person based on the audio content or choice of words, then return "ERROR.... (reason for your response]".
2. For the correct choice, rephrase the answer choice make
that reasons for emotion given
emotion
label and it is based on the speech content specified in the audio caption.
3. The incorrect choices should be reformulated to in a way so that they follow the same tone and length as the correct choice and they should attribute to the GT emotion label, but they differ from the correct choice in the follaving ways: a. [audio__ relevant emation ammationirrreevvnnnn iirelevant] One of the incorrect choice should pick a different audio cue (e.g. tone of voice, background sounds/music) from the audio caption and attribute to the GT emotion label.
[emotion relevant_ audio irrelevan]] Next incorrect choice should pick audio that generally associated with the GT emotion label but is NOT present in the audio caption. Do not mention in the choice that the audio cue is not present in the audio caption, just rephrase as it is present.
[all irrelevant] The last incorrect choice should be some audio cue that present the audio caption, but should sound similar to the correct choice and it should attribute to the GT emotion label.
4. Do not explicilly mention that you are given the emotion label. Do mention captions the question or the answer choices.
5. Phrase all the responses in way that they explain the given GT emotion label and answer the question.
Return your response strictly in the following JSON format ("video id": video_id "question": "Question text", "choices": ("correct": "correct choice rephrased", "audio_relevant emotion irrelevant": "incorrect choice a.", relevant audio, irrelevant": 'incorrect choice b.". "all irrelevant": "incorrect choice c.").
'emotion
Return "ERROR" if you are unable reformulate the question and answer choices. Also specify why you are unable to reformulate the question and answer choices.
Video ID: 'VVIDOO ID)*
Emotion label: YEMOTION LABELJ"
Audio Caption: VAAUDOO CAPTION)
Question: "/QUESTION""
Choices: "[CHOICES)"
Answer: '(ANSWER'"
Figure 33: Preference Data Generation Prompt- Audio Reasoning - used to generate rejected
responses for a generated question- answer pair related audio reasoning tasks.
42

---

Published as a conference paper at ICLR 2026
You will provided with MCQ question, choices and answer related reasoning about the emotion of video based the visual content of a video.
Your task to reformulate the answer choices to make the question more difficult. You will also be provided with the video caption of the video which the question about allong with the ground truth (GT)
emotion label.
Specifically, you
the following checks:
1. If the question is not about reasoning the emotional state of the person based on the visual content, then retum "ERROR...
(reason your response]".
2. For the correct choice, rephrase the answer choice make
that reasons for emotion given
emotion
label and it is based on the visual content specified in the video caption.
3. The incorrect choices should reformulated to a way so that they follow same tone and length as the correct choice
and should attribute to the emolion label, but they differ from the correct choice in the following ways:
a. [viddo__reees relevant eemotion irrelevant] One of the incorrect choice should pick a different visual cue (e.g. background
information, color attire, body language, etc.) from video caption and attribute to the emotion label.
b. [emotion relevant video irrelevant) Next incorrect choice should pick visual that is usually associated with the GT emotion label, but is not present in the video caption. Do not mention the choice that the visual cue is not present in the caplion, just rephrase
present.
[all [illeeeeeeean] The last incorrect choice should be visual cue that NOT present in the video caption and does NOT explain the emotion in general and attribute to the GT emotion label. Do not mention the choice that the visual cue is not present.
present in the video caplion, just rephrase
4. Do not explicitly mention that you are given the emotion label. Do not mention captions in the question or the answer choices.
5. Phrase responses way that they explain given GT emotion label and
question.
Return your response striclly in the following JSON formal ("video id": video id, "question": "Question text", "choices":
("correct": "correct choice rephrased", "video relevant emotion inrelvvnnt:: "incorrect choice *emotion relevant video irrelevant": "Incorrect choice b.". "all jrrelevant:: "incorrect choice c.")).
Return "ERROR" if you are unable to reformulate the question and answer choices. Also specify why you are unable to question
==2
Video ID: "VIDEO ID)"
Emotion label: YEMOTION LABELI"
Video Caption: /VIDEO CAPTION)
Question: "[QUESTION""
Choices: '(CHOICES)*
Answer: (ANSWER)"
Figure 34: Preference Data Generation Prompt Visual Reasoning used to generate rejected
responses for a generated question-answer pair related to visual reasoning tasks.
43

---

Published as a conference paper at ICLR 2026
You will be provided with a MCQ question, choices and answer related to whether the audio and video modalities agree in conveying character video Your task is to create new answer choices to make the question more difficult. captions video which question about along ground (GT)
emotion
provided
emotion label.
Specificelly, you have to do the following checks: If the question about modality agreement emotional the then retur "ERROR... (reason for your response)"
For
"Yes" or rephrase the
explains either yes
the
GT emotion label and the audio and video captions. 3. you
the following incorrect choices way that they follow the they should reason the opposite answer (opposite correct answer in 2), but they differ from the correct choice in the following ways:
and length the correct choice and
a. [video_ relevant, emotion irreevant] One of the incorrect choice should pick different sudio or visual cue (e.g tone of voice, background colour wall, outfit colour, background sounds/music) from audio and video caption and attribute to opposite answer.
[emotion reweventteeveeee video ireeevant] Next incorrect choice should pick audio and visual (complementary cues opposite answer is "Yes", else contradictory cues opposite answer is "No") but are not present the given video caplion. Do mention choice the videolaudio cue not present the video/audio caption, just rephrase as it is present.
support the opposite
jall imeeeeant]] Final incorrect choice should completely irreevant that are not even present the given captions.
question and should talk about audio visual
Do explicitly mention that you given the emotion label. Do mention captions question the choices,
5. Phrase the responses in a way that they explain whether the audio and video modalities agree or disagree in conveying the same emotion for character
video not.
6. Start your answer with either "Yes. ...' or "No,
7. If question state//ooo//eeling the person. It is important that the question about emotion person and nat general emotion.
about the emotional
character, rephrase way that
the emotional
Return your response strictly in the following JSON format ("video_jdd Id": video Id, "question": "Question text", "choloas": ("correct": "correct choice rephrased", choice b.").
_enttteemtion..mm iralevant": 'incorrect choice 'emotioo_ relevant__ video irrelevant": "incorrect
Return "ERROR" if you unable reformulate question and choices. Also specify why you unable reformulate question and answer choices.
Video ID: VVIDEO ID)"
Emotion label: YEMOTION LABELJ"
Audia Caplion: 7/AUDIO CAPTIOM)
Video Caplion: "(VIDEO_ C_PTTION"
Question: "[QUESTION"
Choices: *(CHOICESS*
Answer: '(ANSWER)*
Figure 35: Preference Data Generation Prompt- Modality Agreement used generate re-
jected responses for generated question-answer pair related to modality agreement tasks.
44

---

Published as a conference paper at ICLR 2026
You will provided detailed audio
caption video clip.
Along with the caplion, you will also be provided with the ground truth emotion label of the audio-visual clip.
Your task to write detailed audio-visual caption describing the emotional content of the video clip.
Keep the following points in mind while writing the caption:
1. The final caplion should include both audio and visual elements.
2. Attribute the emation only to the audio and visual cues captions which are relevant emotion label.
3. Do not ground the emotion description any audio/visual cues which are not present in the provided captions.
4. Return your answer as a single paragraph.
Following is an example of how final audio-visurt visual emolion caplion should look like:
Example caption: "In the video, the opening scene shows a female character. She looking directly at the other person, with her mouth slighlly open, seemingly speaking discussing certain topic seriously. As time goes on, the character's expression becomes more excited and intense. The extent to which her mouth is open increases, possibly indicating that she is speaking loudly arguing.
following scene, character's expression becomes more distorted, downtumed mouth, possibly indicating that her emotional state is escalating further. Based on these scenes, it can be inferred that the character this video is likely experiencing a heated conversation or argument. In the audio, this character speaks with strong tone. high volume, and fast pace. There subtille reads: "Is it useful to ask you? Are you ready to be father? Luo Yiyang." "" This sentence is likely spoken by the female character during the intense conversation or argument. Based on the changes in the female character's facial expressions from seriousness excitement and further distorted expressions, tone, high volume, and fast pace, we can infer that this sentence carries a sense of anger, dissatisfaction, or provocation. The female character may questioning other person's usefulness readiness be father, expressing her discontent anger."
furrowed
rhetorical questions with strong emotions. the
well the description in the audio of the character's strong
you think there enough audio-visual cues which support emotion label, return single word "ERROR".
Now, write detailed audio-visual caption for following case
Video Caption: "(VIDEO CAPTION) Audio Caption: "(AUDIO CAPTION]" Emotion Label: "(EMOTION_LABE LABEL)"
Figure 36: Audiovisual Caption Prompt used combine audio and visual captions to create combined audiovisual caption.
You will be provided with the ground truth description video capturing the emotional quotient of the video. Your task is to evaluate given model generation based by comparing it to the ground truth description.
You need rate model generation on scale of on the following criteria: 1. Audio-Visual Overlap: Rate
mention of audio-visual events in the generation aligns with those in the ground truth. higher score indicates better match.
2. Emotion-label Cansistency: Rate how accurstely the predicted emotion from the model aligns with the emotion described in the ground truth. A higher score indicates better consistency.
3. Emotion-cue Association: Only focus on the model generation and rate how well the audio-visual cues are associated with predicted emotion. Rate poorly If emotion- irrelevant indicates better association of audio-visual cues with emotion.
mentioned the generation. higher score
4. Hallucinated Cues: Rate the extent to which the model generations contains hallucinated or fabricated audio-visual cues that are not present in the ground truth. A higher score indicates fewer hallucinations and a lower score indicates more hallucinations.
Return your response the following json format: ("cue_ cove__overppp int, "cue "cue_overlap sanilapreasson::: str, cortriionnosisteccytt.. int, "emotion crrtooooooossttoroeeeeeeeeee reaarrraaon:: str, "emotion_ooe cue. association": int, "emotion cue association reason": str, "hallucinated, cues": int, "hallucinated Swteeaateedcedeeeeeeeeeeeeeeeeee reraeeesss"st str)
Ground Truth Description: VGT DESCRIPTION! Model generation: "(MODEL GENERATION)"
Figure 37: EMER Evaluation Prompt used to evaluate the model generations against the pro- vided ground truths for the EMER datasct.
45


Reviews:
[{"review_id": "biW76nl8Ot", "reviewer": "Reviewer_poWb", "summary": "This paper investigates audiovisual emotion reasoning in multimodal large language models (MLLMs). It identifies two key problems: spurious associations between emotion labels and irrelevant audiovisual cues, and hallucination of cues (models inventing non-present audio/visual evidence). To address evaluation gaps, the authors introduce EmoReAlM, a large benchmark with tasks for emotion reasoning (basic and stress-test) and modality agreement, including human verification and statistics drawn from sources such as DFEW. For optimization they propose AVEm-DPO, a preference-optimization approach that combines explicit preferences (prompt-based modality preference and emotion-based response preference) with a text-prior debiasing (TPD) regularizer to reduce reliance on textual priors. Experiments across many baselines and ablations show substantial gains: AVEm-DPO improves zero-shot performance and reduces hallucination and spurious associations (reported improvements of ~6–19% in several settings, and large gains in user-evaluation metrics). The paper provides extensive benchmark details, human verification procedures, and ablative analyses demonstrating which components contribute most to the improvements.", "questions": "SEE WEAKNESS", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- EmoReAlM benchmark: A comprehensive, human-verified benchmark for audiovisual emotion understanding that tests (a) cue-emotion associations, (b) modality agreement, and © robust stress tests designed to reveal spurious associations and hallucinations. The benchmark includes balanced tasks, adversarial cases, and metrics for spurious associations, modality agreement, and hallucination.\n- AVEm-DPO optimization framework: A novel preference optimization method tailored to audiovisual emotion reasoning. It integrates prompt-based modality preference (PMP) and emotion-based response preference (ERP) with text-prior debiasing (TPD) to steer MLLMs toward using actual audiovisual cues and away from text priors or hallucinated cues. The method is evaluated against several baselines and ablations.\n- Extensive evaluation and analysis: Thorough quantitative and qualitative experiments (including human evaluation) that show large improvements in emotion attribution, reduced hallucination, and better modality alignment. The paper includes ablation studies, sensitivity analyses, class-wise results, and adversarial modality tests that clarify which components are most effective.", "weaknesses": "- Reliance on proprietary or large LLM tooling for some steps: The paper mentions using GPT-5 to polish text and using LLMs for annotation/evaluation. This reliance can raise reproducibility concerns if those tools or their prompts are not fully disclosed; it may also bias dataset construction and evaluation unless careful controls are provided.\n- Potential dataset and evaluation biases: Although the benchmark is human-verified, the document suggests many generated QA items and uses subtitled non-English videos with English subtitles. This raises questions about cultural, linguistic, or subtitle-transcription biases influencing emotion labels and whether the benchmark fully generalizes across languages, contexts, and production styles.\n- Limited discussion of failure modes and external validity: While many stress tests and ablations are provided, the paper could better characterize remaining failure cases (e.g., what kinds of emotions or multimodal interactions still confuse the model), and whether AVEm-DPO overfits to particular datasets or annotation styles. Also, practical deployment implications (latency, compute cost, or safety concerns when misclassifying emotions) are not deeply discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ckk7MPjvgF", "reviewer": "Reviewer_Hdfz", "summary": "This paper introduce a benchmark designed to evaluate MLLMs for cue–emotion associations, hallucinations and modality agreement and propose a preference optimization technique that aligns model responses with both audiovisual inputs and emotion-centric queries.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- A comprehensive suite of 4,000 human-verified multiple-choice questions (MCQs) across 2,649 unique videos, designed to evaluate three critical aspects of emotion reasoning.\n- A multimodal direct preference optimization (DPO) method to align MLLMs with both audiovisual inputs and emotion-centric queries.\n- Demonstrates that AVEm-DPO outperforms baselines by 6–19% in zero-shot settings across existing benchmarks and EmoReAlM, with qualitative and user studies confirming reduced hallucinations and spurious associations.", "weaknesses": "- EmoReAlM is derived exclusively from the DFEW dataset, which may limit generalizability to videos with different cultural contexts, demographics, or emotion types\n- AVEm-DPO’s training data is generated automatically via Gemini 2.5 (without human verification). While the authors report performance gains, unvalidated preference pairs may introduce hidden biases", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4SSE0K7lNt", "reviewer": "Reviewer_oa9K", "summary": "The authors noticed that current models often misinterpret emotions by relying on irrelevant cues or hallucinated details, so they propose EmoReAlM, a benchmark for assessing audiovisual emotion reasoning and hallucination robustness in multimodal large language models (MLLMs), which contains 4,000 human-verified MCQA samples to evaluate emotion reasoning, modality alignment, and hallucination stress tests. Additionally, they also introduce AVEm-DPO, a Direct Preference Optimization approach that introduces prompt-based audiovisual preferences and text-prior debiasing to reduce hallucinations. Experiments show that AVEm-DPO significantly improves MLLM performance and robustness on emotion reasoning tasks compared with existing methods.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow, with clear and informative tables and figures that effectively support the presentation.\n2. To reduce hallucinations, the authors propose using Direct Preference Optimization (DPO). The method incorporates fine-grained, modality-level preferences based on the input text and reasoning about whether a response is hallucinatory or relevant to emotion prediction. Additionally, a text-prior debiasing strategy is introduced to mitigate hallucination effects. The overall approach is both reasonable and methodologically sound.\n3. The experiments and ablation studies are comprehensive, and the presentation of results is clear and easy to follow.", "weaknesses": "1. What is the motivation to use LLMs for visual and audio emotion prediction? It is challenging for LLMs to accurately infer emotions based solely on captions, even for advanced models such as GPT-4o. Moreover, even when an LLM’s prediction matches the ground truth, it does not necessarily imply that the emotional trigger or the reasoning process behind the prediction is correct.\n\n2. Is there any analysis on the individual roles of the visual and audio modalities? For example, which modality provides the essential cues for the final prediction, and do we need both modalities for effective emotion reasoning?\n\n3. In data creation, captions play a crucial role in translating visual information into text. Was any sampling strategy applied during this process? How do the authors ensure that the essential frames or emotional triggers are preserved during caption generation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fXtLwDKZ5e", "reviewer": "Reviewer_D5KC", "summary": "This paper focuses on emotion reasoning, addressing two key challenges: (1) Spurious associations between emotions and cues (reasoning errors), and (2) Hallucinations (perception errors). The authors first construct EmoReAIM, a benchmark designed to evaluate MLLMs on cue-emotion associations, hallucinations, and modality agreement. They then propose a framework, AVEm-DPO, which leverages DPO and text-prior debiasing techniques to enhance performance in emotion reasoning. Experimental results on three datasets demonstrate the effectiveness of their approach.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper tackles a cutting-edge emotion reasoning task by proposing a novel benchmark and solution.\n\n2.\tThe authors classify emotion reasoning errors into perception errors (hallucinations) and reasoning errors (spurious associations). This classification effectively covers most errors in emotion reasoning.\n\n3.\tThe paper is well-written and well-motivated, with experimental results validating the effectiveness of the proposed method. It provides new insights into emotion reasoning.", "weaknesses": "1.\tFrom my perspective, \"Spurious AV cue associations\" and \"AV cue hallucinations\" are essentially reasoning errors and perception errors, respectively, aligning with the errors in MLLMs. Thus, I suggest classifying errors primarily as reasoning and perception errors in the main text, followed by additional explanations from the \"Spurious AV cue associations\" and \"AV cue hallucinations\" perspectives. This would improve readability and attract broader interest from the MLLM research community.\n\n2.\tThe paper provides a clear definition of \"hallucinated cues\" (i.e., descriptions containing clues that do not exist in the video). However, I have some concerns about \"emotion-irrelevant cues\". For example, in Figure 2 (\"Emotion reasoning - basic\"), the statement \"The presence of light music in the background suggests a relaxed state\" does relate to the person’s emotion state and should not be considered \"emotion-irrelevant.\" Similarly, in Figure 1, \"The dark green color of the background supports the negativity\" could also be seen as a valid cue for inferring emotion. Overall, from my perspective, background music and color can be viewed as a useful clue for emotion reasoning and should not be dismissed as \"emotion-irrelevant cues.\"\n\n3.\tThe paper introduces two types of preference data, which is an interesting contribution. \"Emotion-based Response Preference\" aligns with my understanding of DPO. However, \"Prompt-based Modality Preference\" is less clear. Does it involve feeding different audiovisual inputs while expecting the same response? More examples would help clarify this type of DPO data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)