EmoPrefer: Can Large Language Models Understand Human Emotion Preferences?
Authors: Zheng Lian
Licai Sun
Lan Chen
Haoyu Chen
Zebang Cheng
Fan Zhang
Ziyu Jia
Ziyang Ma
Fei Ma
Xiaojiang Peng
Jianhua Tao
Venue: ICLR 2026 Poster (2026)
Keywords: multimodal emotion recognition
descriptive emotions
EmoPrefer
EmoPrefer-Data
EmoPrefer-Bench
Primary area: datasets and benchmarks

Abstract: Descriptive Multimodal Emotion Recognition (DMER) has garnered increasing research attention. Unlike traditional discriminative paradigms that rely on predefined emotion taxonomies, DMER aims to describe human emotional state using free-form natural language, enabling finer-grained and more interpretable emotion representations. However, this free-form prediction paradigm introduces new challenges regarding its evaluation. Previous works depend on ground-truth descriptions, but emotions are inherently tied to diverse human behaviors, and generating a comprehensive and accurate description is inherently demanding. Other researchers reformulate this problem into a more tractable human preference learning task, but pairwise preference annotation involves substantial manual effort. This leads to a question: *can we leverage multimodal LLMs (MLLMs) to achieve more cost-efficient preference annotation?* To answer this, we propose **EmoPrefer**, a pioneering work exploring the potential of LLMs in decoding human emotion preferences. Specifically, we construct the first emotion preference dataset, **EmoPrefer-Data**, featuring high-quality preference annotations from experts. Additionally, we introduce **EmoPrefer-Bench**, which evaluates the performance of various MLLMs and prompting techniques in preference prediction, while also revealing new strategies to enhance their performance. To the best of our knowledge, this is the first work exploring the capabilities of LLMs in understanding human emotion preferences. Our work advances the field of DMER and lays the foundation for more intelligent human-computer interaction. Our data and code are released at https://github.com/zeroQiaoba/AffectGPT/tree/master/EmoPrefer.

Published as a conference paper at ICLR 2026
EMOPREFER: CAN LARGE LANGUAGE MODELS UN- DERSTAND HUMAN EMOTION PREFERENCES?
Zheng Lian Licai Sun Lan Chen", Haoyu Chen?, Zebang Cheng Fan Zhang', Ziyu Jia', Ziyang Ma², Fei Ma". Xiaojiang Peng". Jianhua Tao 10-
State Key Laboratory of Autonomous Intelligent Unmanned Systems, Tongji University
Frontiers Science Center for Intelligent Autonomous Systems, Ministry of Education UUniversity of Oulu "MAIS, CASIA "Shenzhen University
"Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ.
Thhe Chinese University of Hong Kong SShanghai Jiaotong University 9 "Shenzhen Technology University Department of Automation, BNRist, Tsinghua University
ABSTRACT
Descriptive Multimodal Emotion Recognition (DMER) has garnered increasing research attention. Unlike traditional discriminative paradigms that rely on prede- fined emotion taxonomies, DMER aims to describe human emotional state using free-form natural language, enabling finer-grained and more interpretable emo- tion representations. However, this free-form prediction paradigm introduces new challenges regarding its evaluation. Previous works depend on ground-trufe -trth de- scriptions, but emotions are inherently tied to diverse human behaviors, and gener- ating a comprehensive and accurate description is inherently demanding. Other re- searchers reformulate this problem into more tractable human preference learn- ing task, but pairwise preference annotation involves substantial manual effort. This leads question: can we leverage multimodal L.LMs (MI.LMs) to achieve more cost-efficient preference annotation? To answer this, we propose EmoPre- fer, pioneering work exploring the potential of LLMSs in decoding human emo- tion preferences. Specifically, we construct the first emotion preference dataset, EmoPrefer-Data, featuring high-quality preference annotations from experts. Additionally, we introduce EmoPrefer-Bench, which evaluates the performance of various MLLMs and prompting techniques in preference prediction, while also revealing new strategies to enhance their performance. To the best of our knowl- edge, this is the first work exploring the capabilities L.MS in understanding human emotion preferences. Our work advances the field of DMER and lays the foundation for more intelligent human-computer interaction. Our data and code are released at
INTRODUCTION
Descriptive Multimodal Emotion Recognition (DMER) (Lian et al.
2023 2025a) aims to free-
form natural language to describe emotional states. Unlike traditional discriminative methods that depend on predetined emotion taxonomies (El Ayadi al. 201T Lian et 2026). DMER adopts generative paradigm, offering greater flexibility in emotion representation. This enables fine- fine-grained and interpretable emotional expression, presenting significant opportunities for advancing emotion- intelligent human-computer interaction technologies (Brave & Nass 2007). Recent advancements in Multimodal Large Language Models (MLLMs) (Zhao et al.) [2033] Yin et al.) (0244), with their rich vocabularies and multimodal understanding capabilities, have made this task feasible. Figure demonstrates the distinctions between descriptive and discriminative MER.
However, evaluating the quality of such open-ended descriptions remains non-trivial task. Figure 2 summarizes the existing evaluation strategies. The first approach leverages ground-truth emotion descriptions and employs LLMs, such OpenAl GPT (OpenAl 2024) Google Gemini TTam et al. 2023), to measure the similarity between predicted and ground-truth descriptions (Lian et al.
* *Corresponding Author

---

Published as a conference paper at ICLR 2026
Discriminative MER:
worried
Descriptive MER: At the beginning, she seems bit nervous and sneusy. In the following scenes, her eyebrows are raised, indicating that emphasizing something.. At end the video, she lowers her head and looks down, not looking at the other person while speaking This behavior from inner pressure, anxiety, uneasiness, frusirution, causing her to tend avoid communicaiio with others.
Temporil Dysami
Emotion fntemiit
Uncertainty
Multinodal Cluss
Figure Task comparison. Discriminative MER assigns a label from a predefined taxonomy, whereas Descriptive MER provides a more detailed description by incorporating emotional temporal dynamies, intensity, uncertainty, and the existence and reasonableness of multimodal clues.
2023). Nevertheless, emotions are inherently tied to diverse human behaviors, including facial ex- pressions, (micros)gsstures. head movements, hand actions, and vocal tones (Chen et al. 2023 & Deng 2020; Yang et al.) (026). As a result, generating a comprehensive and accurate descrip- tion of person's 's emotional state inherently challenging, and unreliable ground-truth descriptions can lead to inaccurate evaluations. To address this, researchers propose abandoning costly and of- ten incomplete ground-truth descriptions (Lian al.] 2025b). Instead, they reformulate the complex problem of designing indicators to measure semantic similarity with ground-truth descriptions into a more tractable problem of learning human preferences. However, this approach requires preference annotations for each model pair across multiple samples, entailing substantial human effort. For ex- ample, given M models and N samples, the total number of required comparisons is C(M, 2) x N.
Ground Truth Description
Ground Truth Description
Ground Truth
Description
Repluce?
L.-M-based
Preference
Human Armottior
Julge MULM
Preference
Metrics
Predicted Description
Predicted Description
Predicted Description
Predicted
Predicted Description 2
Description
(2) Deseription-bused
(b) Preference-driven
(c) EmoPrefer
Evaluation strategies for DMER. (a) Description-based evaluation, which relies on costly and often incomplete human-annotated descriptions; (b) Preference-driven evaluation, where pairwise preference annotations require substantial human effort; (c) EmoPrefer, a pioneering ap- proach that explores whether MLLMs can decode human emotion preferences, aiming to replace expensive human annotations with MLLM-based judges.
Figure 2:
Given the high cost of manual preference annotation, heuristic idea emerges: Can leverage MILMs to achieve automatic emotion preference decoding, thereby providing a more cost-efficient evaluation strategy for DMER? To answer this, we propose EmoPrefer, the first work exploring the potential of MLLMs in emotion preference prediction. (1) We introduce EmoPrefer-Date the first multimodal preference dataset centered on human emotions. In this dataset, we provide pair- wise emotion descriptions for videos and recruit multiple expert annotators to label preferences. Only samples with unanimous agreement among all annotators are retained, ensuring high-quality preference annotations. (2) We establish MeoPrefer-Bench the first benchmark for emotion pref- erence prediction. In this benchmark, we conduct a comprehensive evaluation of different MI ML and prompting techniques, further exploring strategies to enhance their alignment with human pref- erences. This paper presents pioneering work that reveals the capabilities of MLLMs in human emotion preference. Beyond evaluation, the dataset and insights from benchmark will support the training of emotion-aware reward models, thereby enhancing the consistency of MLLMs with human emotional understanding. The main contributions of this paper summarized as follows:
(EmoPrefer) This is a pioneering work that explores the potential of MLLMs in emotion pref- erence. Beyond evaluation, the insights from this paper will contribute to training reward mod- els capable of understanding human emotions, paving the way for emotion-intellignnt models.
2

---

Published as a conference paper at ICLR 2026
(I) EmoPrefer-Datt (Human)
(2) EmoPrefer-Bench (Judge MLLM)
(3) Evaluation Metric
Emotion
Model Pool
prefer
Dataset
Des
Des d2
true
Des
prod
Recognition Performance
prred
prefes
Overlassing
Scieatio
Des d, swap
pred
Dataset 2
Emotion
Preference
MLLM+des
MLLM-rdes
preier
Consistency
Swap
(
LEM-+pred
LLM LLMparedd
Des d1
Des d2
pred
Figure 3: Pipeline of EmoPrefer. (I) EmoPrefer-Data. We select overlapping videos across two descriptive emotion datasets, then conduct human preference amnotations and adopt their majority- voted results as final labels. (2) EmoPrefer-Bente
We systematically evaluate various MLLMs and prompting techniques, revealing their potential in emotion preference decoding. (3) Evaluation metric. We introduce two metrics for performance evaluation: recognition performance measures alignment between MLLM-based judges and human annotations; swap consistency evaluates the MLLM-based judges' robustness to swapping the order of description pairs.
(amoPrefer-Data) This paper constructs the first preference dataset centered on emotions. Our work makes a substantial contribution to current research on LLMs-as-judges by not only expanding the modality to full multimodal scenarios (encompassing audio, video, and text) but also extending preference learning to the domain of human emotions.
(EmoPrefer-Bench) This paper proposes comprehensive evaluation metrics and explores di- verse solutions, demonstrating the feasibility of automatic preference prediction for human emotions. Additionally, we introduce model-based crowdsourcing to enhance performance.
EMOPREFER-DATA
Figure illustrates the overall pipeline of EmoPrefer, which is the first work to explore the po- tential of MILMs for emotion preference decoding. To achieve this, we require a high-quality, numan-annotatee emotion preference dataset. In this section, detail the construction process of our Emmorrefe--eee from four aspects: (1) sample selection, (2) description quality analysis, (3) preference annotation, and (4) inter-annotator agreement analysis.
Sample Selection. For preference annotation, we provide pairwise emotion descriptions for each video and ask human annotators to select the hetter one. However, the quality of two descriptions varies significantly, the preference prediction task will become trivial, potentially diminishing the advantages of superior solutions. To ensure high-quality descriptions, we leverage two benchmark datasets for descriptive emotion: MERR-Fine (Cheng et al. 2024) and MER-Caption+ (Lian et al.) 2025a). The raw videos in these datasets are sourced from the MER2024 dataset (Lian et al. (024)) We only select videos that appear in both datasets, resulting in 1,368 samples, each with two descrip- tions (one from each dataset). Appendix [C.T provides visualizations of the selected video samples. These videos primarily feature single front-facing character with complete audio content, ensuring the full expression of emotions for the target speaker.
Description Quality Analysis. This section analyzes the descriptions provided by MERR-Fine and MER-Caption+. These datasets focus on capturing the character's multimodal cues in the video, including facial expressions, body language, vocal tone, contextual events, and environmental fac- tors, offering comprehensive depiction of the person's emotional state. Figure 4 presents word clouds of emotion words and nouns extracted from these descriptions. The detailed extraction pro- cess is described in Appendix C.2 Our analysis reveals that these descriptions contain rich diver- sity of emotional vocabulary and multimodal cues related to emotions, confirming their high quality.
3

---

Published as a conference paper at ICLR 2026
anger neutral
angerr
Intonation wo words
tone
sentence
happy HOT
Sad
mnxlecy
Voice
emoltion tons SUOT OO
Conaa RO
frustrat iesstrationn!!
a anxious
expression
(a) MERR-Fine (Emotion)
(b) MER-Caption+ (Emotion)
(c) MERR-Fine (Noun)
(d) MER-Caption+ (Noun)
Figure 4: Word cloud visualization. We extract emotion words and nouns from the descriptions and generate corresponding word clouds for both datasets. Appendix C.2 details the extraction process.
Preference Annotation. During the annotation process, we recruit master's students from our lab candidate annotators. Given their research focus affective computing, these individuals are al- ready familiar with emotion definitions. To ensure their understanding aligns with that of the general population, we first conduct a preliminary test. Specifically, we select 12 samples that have reached consensus among eight external annotators (i.e., all eight have assigned identical emotion labels). The candidates are then asked annotate these 12 samples, and only those who achieve at least 75% accuracy are retained, resulting in three qualified annotators. The selected annotators are then tasked with determining which of two descriptions more accurately reflects the character's emotional state. Appendix C.3 illustrates the layout of our annotation platform. For each comparison, annotators can choose one three options: the first description better, the second description is better, or lie. We retain only samples where all three annotators reach consensus. This strict selection strategy ensures the reliability of the preference labels.
Inter-annotator Agreement Analysis. After an- notation, we evaluate the inter-annotator agreement for emotion preference recognition. Figure Sa
presents the agreement scores excluding ties, with an average of avg(69.88, 69.31, 68.73) = 69.31 %. This reflects the upper bound of human agreement and confirms that different annotators can reach a reasonable consensus this task. Figure 5b shows the agreement scores including ties, where the aver- age drops to avg(60. (60.87, 56.39, 60.43) 59.23%, 10% decrease compared to Figure 5a This suggests that annotators struggle more to reach consensus ties, indicating that ties represent higher ambiguity than clear preferences (c.g., when one description is distinctly better than another). Table I compares EmoPrefer- Data with existing datasets. Our dataset not only covers more modalities but also extends the preference task to human emotions. To the best of our knowledge, this is the first multimodal preference dataset centered on human emotions, providing valuable resources for future research.
100.00
$1.72
2
60:07
103.00
60.13
(a) w/o tie
(b) w/ tie
Figure 5: Inter-annotator agreement analy-
sis. We present the inter-annotator agreement under two conditions: w/o ties and w/ ties.
EVALUATION METRICS
In this section, we propose two metries to evaluate the performance of MI.LM-based judges: recog- nition performance and swap consistency. Figure 3 Iillustrates the detailed calculation pipeline.
Recognition Performance. This metric evaluates the aligmment between model predictions and human annotations. In EmoPrefer-Data, the labels fall into three categories: (1) the first description is better, (2) the second description is better, or (3) tie. Since human annotators often struggle to reach consensus on ties (as shown Figure(5). we report both the weighted average Fl-score (WAF) and accuracy (ACC) under two settings: two-class (excluding ties) and three-class (including ties) setups. Given the inherent class imbalance in the dataset, we default to the two-class WAF when reporting a single metric in the following experiments.
Swap Consistency. Ideally, swapping the order description pairs should not affect the model's preferred description. However, in practice, we observe that order swaps sometimes lead different
4

---

Published as a conference paper at ICLR 2026
Table 1: Dataset comparison. The second column specifies the modalities used for preference tasks. Since MLLM-as-aa-udde (Chen et al. (2024) relies on image sequences rather than videos, we use the symbol d indicate this. Our EmoPrefer-DDe not only expands the range of modalities but also extends the preference task to human emotions.
Modality
Dataset
Task Description
I
A
MT-Bench (Zheng al.
2023
X
X
writing. math, general knowledge,
LLMEval (Zhang
A125
X
X
storytelling, summarization, dialogue, etc. writing, role play, elc. law, biomedical data, etc.
FairEval Waan et al. 2024a
X
X
X
P Wang cl al.
2024b
X
X
JudgeBench [Tan el al.
1025
X
x
general knowledge, reasoning, math, coding communsense, math, science, etc.
CALM Ye all 2025
x
MLLM-as-a-Judee (Clien
2024
X
captioning, reasoning, etc.
VT.RRewardBench (Ta al. 2025e
V
X
general visual perception, reasoning, etc. captions, safety, reasoning, elc.
MM-RLIIF (Zhang el al. 20Z5
V
X
V
Multimodal RewardBeenc Yusunaga et al. 2025
X
knowledge, safety, reasoning, etc.
EmoPrefer- (Ours)
human emotion understanding
outcomes. This discrepancy may arise because the model's preferences are determined by the order of inputs rather than the actual content of the descriptions. Therefore, introduce swap consis- tency, a metric that evaluates a model's robustness to order swapping and ensures predictions are based on content rather than ordering. Specifically, given N examples { (x=", df, dz Ji=1 where x" is a video and di and d' are two corresponding descriptions, we use MLLM to predict preferences for both the normal order input (x"' df. d3 and the swapped order input (x" dB dft.. This yields We compute swap consistency as follows:
two outcomes: 012 and o3j.
Swap Consistency = Ei I(0 oz1
(1)
N
where II() is the indicator function, and higher scores indicate greater robustness to input order variations. Appendix [] provides the prompt template used in the calculation process.
EMOPREFER-BENCE
This is the first benchmark designed for emotion preference recognition. We evaluate various MLLMS and explore the effectiveness of model-based crowdsourcing. The paper reports the zero- shot performance of these MLLMs, with all experiments conducted on A100 GPUs. To reduce randomness, each experiment is run twice, and both the mean and standard deviation are reported.
4.1 JUDGE MILM
Given that humans express emotions through multiple modalities, we require MLLMs support least audio or video inputs. To enable preference prediction, we propose four prompting strategies (see 3) Strategy 1 (S1): We input the video along with two descriptions into the MLLM and ask the model to determine the better one based on the character's emotional state. Strategy 2 (S2): We decompose S1 into two steps. First, we instruct the MLLM generate detailed description of the video. Then, using the output from the first step as a reference, we prompt the model to determine which of the two descriptions better aligns with Strategy (S3): The second step in S2 relies solely on text input, meaning it can be implemented using an external LLM. The rationale for this alternative is that current MILMM training typically begins with a pretrained L.M,, followed by additional multimodal fine-tuning While this enables the model to process other modalities, such training may compromise its text understanding capabilities. Thus, we opt to use an external LLM for the second step. Strategy 4 (S4): We further break down the second step in S3 into two sub- steps, requiring the model to perform additional reasoning before making decision. This allows us investigate whether the reasoning process is necessary for accurate preference prediction. We adopt Qwen2.5-7B (Yang et al.] 2024) as the external LLM for S3/S4, and the rationale for this choice is discussed in Section 4.2 Appendix D provides the prompt templates for all strategies.
5

---

Published as a conference paper at ICLR 2026
4.2 MAIN RESULTS
proposes four prompting strategies, and their impacts on different MLLMS summarized in Figure For most models, S3 and S4 outperform S1 and S2. The key distinction between these strategies lies in whether they leverage external LLM. These findings highlight the advantages integrating an external LLM, as MLLMs may have limitations in language understanding. However, some exceptions exist. For instance, Qwen2.5-Vi 5-V (Bai et al.) 2025) and Qwen2.5-Onn (Xu et al., 2025) perform better with S1 and S2 than with S3 and S4. This is because these models retain strong language processing capabilities due to their advanced training methodologies, enabling them to interpret complex prompts effectively. In such cases, the longer inference chains in S3 and S4 may introduce error accumulation and degrade performance. Thus, a trade-off emerges: while using an external LLM can compensate for the limited language processing abilities of some MLLMs, longer inference chains may also increase error accumulation.
Impact of Prompting Strategy. Section 4.1
(a) Videe-LLaVA
(b) Quen2-Audio
(C) LLIMAVVID
Chat-UaiV
(e) mwLUG-Owl
① VideoChull
VideoChat2
(h) LLaVA Next Video
VITA-L5
PLLAVA
Qwen2.5 VL
@ QuQeee2555 Omná
Figure 6:
Impact of prompting strategies. In these figures, we compare the performance of S1 $1-S4 across various MLLMs. All experiments are conducted on the EmoPrefer-Data dataset.
Role of External LLMs. S3 and S4 rely on external LLMs. Figure 7 examines the impact of LLMs emotion preference prediction. We observe that Qwen3- 14B (Yang et al.) (025) performs better under the S3 strategy, while Qwen2.5-7B performs better under the S4 strategy. These results reveal interesting phenomenon: different LLMs exhibit varying preferences for different prompt- ing strategies. To further investigate, we conduct statistical analysis of the two LLMs using their respective best strategies. Experimental results demonstrate that Qwen2.5-7B generally outperforms Qwen3-14B These findings suggest that larger LLM does not necessarily guarantee better align- ment with human preferences. In this paper, we adopt Qwen2.5-7H as the default external LLM.
Perfrrancc under 53
Over 31
53
Results
Caccs
25.30
Hndeshatt
Chestow
Aatos Lasi
Orenz Canoe
Ltaite Ville
Lnmba HD
ALLR
Omnec Audia
Dhlice of Aizay
Figure 7: Role of external LLMs. From top to bottom, we show the performance of two LLMs under three configurations: the 83 strategy, the $4 strategy, and the best of the S3 and S4 strategies.

---

Published as a conference paper at ICLR 2026
Performance on Different MLLMs. Table 2 presents the performance of MLLMs under their best strategies. Full results for four strategies are provided in Appendix G Experimental results demonstrate that open-source models do not show noticeable performance gap compared to closed- source models; some open-source models even surpass closed-source ones. Among all models, Qwen2.5-Omni achieves promising results, attaining 67.21% the two-class WAF and 79.09% on swap consistency, highlighting its strong capability in multimodal information processing and human emotion understanding. Additionally, most models outperform random guessing (i.e.. 50%), yet even the best persforero one falls short of the upper bound. This indicates the challenge of emotion preference decoding and suggests that further exploration needed in the future.
Table 2: Main results. The second column indicates the optimal strategy for each MLLM.
Rec. (2-Class)
Rec. (3-Class)
Swap Cons.
Model
S.
WAF(1)
ACC(1)
WAFC)
ACC(1)
Open-source MILMSs
VidcoChat (Li ct al. 2025b
11.79+1.21
52.31
44.88=1.54
40.77+1.57
40.85=+1.83
Video-ChattiPI MMaz
2024
02.10
52.13+0.98
43.97=0.44
40.42+0.35
1..3
Otter (Li al. 2025a
S4 1..12+1.10
52.49++0.98
43.41+1.67
38.15+1.74
46.08+0.96
VidcoChat2 [Li
2024b
S4 52.38+0.10
52.49=00099
45.88=0.96
43.82++0.44
41.46=#1.74
mPLUG-Owl Ye ( al.
2022
S4
53.76+0.117
53.82:+0.18
46.90=0024
44.34+0044
42.33=+1.39
Video-I.TaVA YT.m
2124
54.53+0.81
54.62+0.80
43.61+0.82
38.94+0.44
42.86+33.31
Chat-UniVi (Jin
20:24
14.97=1.64
15.06+1160
47.03-22.22
44.69:42.00
43.90:1:.52
LLaVA-Next- [L)
2024a
56.41+0.98
56.57=+0.8
1.28
10.78+1.31
53.92+22.00
I.AMA-VIDD [Ti et al. 20/44
S4
57.10+0.63
57.10+0.62
50.42+1.17
47.13+
45.
PLLAVA (Xu èt al. 2024
S4
57.29=1.118
57.55=00118
54.02=0.05
52.61=0.000
53 13.83=11222
VITA-1.5 FD et al. 2025
S4
60.08:0.611
60.12+00.2
57.08=0.116
56.01+00009
59.06= 1.22
Qwen2-Audra Chu hu
2024
S4
63.17:10.19
63,23 18
60.15
59.32.10.78
61 10
Qwen2.5-VL. (Bai ct al.
2025
64.43-1.87
65.2840.80
62.60=0.84
64.022:0078
02:10.555
Qwen2.5-Omni [Xu
67.21
67.32++0.00
65.29=00.00
66.03+:0.00
79.09=00000
Closed-source MLLMs
GPT-4o (OpenAl
2024
59.28+00088
59.41
56.57=0.04
53.75+0009
64.55++0.96
Gemini- Gemini-2U-Flash 2.0-Fashh
Google 2025a
59.80-10.85
60.39110.8
08.1410.65
55.66.10.61
2..126
Gemini-2.54. 2.5-Flahh Google 2025b
60.60:1064
61.1910062
58.77=0.55
57.9310.78
64. 64,9811.33
GPT-4,1 (Achiam al. 2025
S1
60.75+0. 18
60.75 18
59.23
59.67+0026
80.54=0.52
Gemini- 1,5-Pro ( Team al. 2024
S1
60.79110.6 68
61,55100.2
59.06
60.37.10.1
02.37.10.33
Gemini-1.5-Flash [eam ct al.
2024
S1
64.6410.12
65.19100.5
62.35
63.50110.09
72.04:10.44
Metric Correlation. Figure [a calculats the Pearson correlation coefficients (PCC) using scores from all strategies (see Appendix G
Sec:1D
3.00
0:59
0.53
RXc-22
3.00
0.88
whereas Figure 8b uses scores from the op- timal strategies (see Table D In Figure Sal we observe strong correlation between two-
Snup Bec
3:29
2.50
Rec.
0.57
1.00
0.02
class and three-class recognition performance, whereas these metrics show only relatively weak correlation with swap consistency. These metries capture
90d0
Bec
Rac/211
Seawp
Rac/21
SwAp
results suggest that the distinct aspects of model capability: one re- flects preference prediction accuracy, while the other reflects model robustness. A good model should perform well on both metries. Figure 8b the correlation between these metrics is higher than in Figure Sa This indicates that models with poor recognition performance may exhibit more varied swap consistency. But for well-performing models, these metrics remain related.
(4) all strategies
(b) hest strategy
Figure 8: Metric correlation analysis.
Effeet of Normal-Swapped Combination. Table 2 reveals that swapping input orders may lead to distinct preference prediction results. This observation naturally raises the hypothesis: Can enhance the reliability of the results by aggregating the outputs of normal and swapped orders? To verify this, execute each model twice in normal order and twice in swapped order, generating four outcomes. We then apply majority voting to determine the final prediction. Figure demonstrates the impact of this strategy across different MI.LMs. We observe that most MILMs benefit from this
7

---

Published as a conference paper at ICLR 2026
approach. However, exceptions exist, such Chat-UniVi (Jin al. 2024). indicating that the effectiveness of this strategy is model-dependent.
Recoghtb Horfrmand
63-
Swtopes
VitroChd
visto ChRcot
Hiber Lork
Owenz Sog.
Orend Droy
Lssie Nede
3
mitic
itats
VideCcat
RLAN
Ownnt Audlo
Figure 9: Normal-only vs. Normal-swapped combinations.
Model-Based Crowdsourcing. This section investigates the impact of model-based crowdsoure- ing. Specifically, we rank models based on their recognition performance and aggregate the predic- tions of the top-k models. As shown in Figure TO crowdsourcing generally improves performance, the extent of the gain depends on k: large l introduces noise, whereas small A: limits effective- ness. We then conduct an analysis from two perspectives: (/) Necessity of hormal-swapper combi- nations. We examine the effectiveness normal-swapped combinations crowdsourcing. Unlike the findings for single models (see Figure 99, normal-swapped combinations yield only marginal improvements and may even degrade performance in some cases (e.g.. Figures 10a and TOd]. There- fore, we do not normal-swapper combinations by default in crowdsourcing. (2) Restricting the model scope. Figures 10a T0c show performance when restricting the model scope, with the scope ranging from all, open-source, closed-soune models, respectively. Compared to restrictions, restricting the model scope to closed- closed-source source models leads to noticeable performance degradation, whereas limiting to open-source models still yields promising results. This suggests that restricting evaluations to open open-source models could reduce costs without significant performance loss.
NS)
Open-sonne NS)
Closad-sanree
(d) All (wd NS)
(e) Open source (wi ISS)
(6 Closed source (wl NS)
Figure 10: Impact of model-based crowdsourcing on recognition performance. In these figures, "w/o "S" and ww/ " denote whether normal-swapper combinations used, while "all", "open- source", and "closed-source" indicate whether the model scope is restricted.
Besides its impact recognition performance, Figure II further reveals its effect swap consis- tency. We observe that selecting an appropriate & also improves swap consistency, indicating greater robustness to input order changes with crowdsourcing. Interestingly, the optimal k differs for swap
8

---

Published as a conference paper at ICLR 2026
consistency and recognition performance (e.g., Figures 10a and Ia Since good model should perform well on both metrics, we should consider both when selecting k. For example, &o = 3 or good choice when restricting the model scope to open-source models.
Open-source
(c) source
Figure 11: Impact of model-based crowdsourcing swap consistency.
5 PRACTICAL APPLICATION
Figure 12 a)) integrates preference prediction with the Bradley- Terry algorithm (Hunter 2004) for model ranking. Specifically, suppose there are M models and N samples, where each sample is denoted as x" and each model as m,- Taking the comparison hetween models m, and m, as an example, first obtain their emotion descriptions for x", yielding dī and dy. We then use judge MLLMs to predict preference. Inspired by the results of EmoPrefer-Bencch, we use model-based crowdsourcing, restrict the model scope open-source models, and set top-k=3. The predicted preference is denoted as p. By repeating this process across multiple samples, we obtain the num- bers of wins, losses, and between m, and mj. denoted as win,y. lose,y. and tie:y, respectively. These scores are stored in the matrix W, and then the Bradley-Tere algorithm is used to estimate the advantages. Appendix provides the detailed calculation process of the ranking algorithm.
Figure 12 provides concrete example. Specifically, MER2025 (Lian et al. 2025b) is prominent challenge affective computing, featuring track focused on descriptive emotions. With permission from the challenge organizers, visualizes the matrix W, and Figure (2(c) presents the estimated advantage scores.
participants submissions for model ranking. Figure 12
Model Preference
Ranking
Scanee
wis los
P
Tulge
10
Rtancing
(M4,, m2))
(2) Ranking pipeline
(b) Manix advantage scoce
Figure 12: Practical application. (a) Ranking pipeline. We first employ pairwise models and my to generate descriptions for x", obtaining d, and dy. Next, we use MLLM- judges to derive the preference prediction p.j. By repeating this process, we collect the counts of wins, losses, and ties for each model pair comparison. These results are then used to compute the model rankings. (b) We visualize the matrix W and advantage scores derived from concrete example.
6 CONCLUSION
This paper proposes EmoPrefer, pioneering work that explores the capabilities of MLLMs in recognizing human emotion preferences. To this end, we introduce EmoPrefer-Date and establish

---

Published as a conference paper at ICLR 2026
MinoPrefer-Bencie the first dataset and benchmark focused human emotion preference. Exten- sive experiments show that different MLLMs favor different prompting strategies, and model-based crowdsourcing enhance both recognition performance and model robustness. Mcanwhile, observe that current MILMS still struggle with accurate emotion preference prediction. This high- lights both the limitations of current MILMS and the inherent challenges in emotion understanding.
Limitations and Future Work. This paper reports the zero-shot performance of MLLMs, without exploring more effective architectures training paradigms. Such exploration is left for our future work. Beyond evaluation, we will leverage our dataset and the insights drawn from our benchmark train reward models and apply reinforcement learning to maximize these rewards. This approach will enhance MLLMs' understanding of human emotions and, in turn, improve their emotional intelligence. Additionally, this paper focuses on emotion preference in descriptive emotion. Beyond descriptive emotions, other forms of emotion representation exist, including categorical emotions (e.g., using discrete emotion words) and dimensional emotions (e.g., using Roat-point values). In the future, will extend our preference task to other emotion representations.
ETHICS STATEMENT
We do not collect new data but instead use the raw samples from MER2024 with the permission of the owners. To construct EmoPrefer-Data, we hire multiple annotators. They are generously compensated for their work. During the annotation process, we provide a clear task description and utilize a well-designed annotation platform. Furthermore, non-commercial purposes under the BY-NC 4.0 license, which explicitly outlines the appropriate and responsible of our dataset. As result, ethical concerns raised in this paper.
restrict the use of EmoPrefer- Data to
REPRODUCIBILITY STATEMENT
Our data and code are released at For dataset annotation, provide detailed introduction in Appendix C In summary, have made every effort to ensure the reproducibility this paper.
ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China (N0.62201572, No.62088100 101, No.62322120 120, No.61831022, No.62276259, No.U21B2010), the European Union's Horizon Europe Research and Innovation Programme under the Marie Sklodowska-Curt Actions grant agreement No. 101 126602 (Data4Healthcare). the Innovative Team Project of Multimodal Ar- tificial Intelligence and Interaction in Guangdong Province (2255KCTD004), the University of Oulu & The Academy of Finland Profi Hybrid Intelligence (grant 352788), and Research Fellow project (grant 37 17111).. We also wish to acknowledge the CSC IT Center for Science, Finland, for computational resources.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392. 13923, 2025.
Scott Brave and Cliff Nass. Emotion human-computer interaction. Human- Humun-Computer Interac- tion Handbook, pp. 103-118. CRC Press, 2007.
John Cacioppo and Wendi Gardner. Emotion. Annual Review of Psychology, 50(1):191-214, 1999.
10

---

Published as a conference paper at ICLR 2026
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. ju-jueee--- assessing multimodal -lm-as-a-judg with vision vision-language benchmark. In Proceedings of the 41st International Conference Machine Learning, pp. 6562-6595, 2024.
Haoyu Chen, Henglin Shi, Xin Liu, Xiaobai Li, and Guoying Zhao. Smg: A micro-gesture dataset towards spontaneous body gestures for emotional stress state analysis. International Journal of 2023.
Computer Vision,
Zebang Cheng, Zhi-Qi Cheng, Jun-Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng. and Alexander Hauptmann. Emotion-llan -lamon Multimodal emotion recognition and reasoning with instruction tuning. In Proceedings of the Advances Neural Information Processing Systems, pp. 118805-110853 2024.
Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Con- ference Neural Information Processing Systems, pp. 4302-4310, 2017.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint 2ivv:2407.10754) 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.1416. 2021.
Raymond Dolan. Emotion, cognition, and behavior. Science, 298(5596:11111-1994, 2002.
Paul Ekman and Dacher Keltner. Universal facial expressions of emotion. California mental health research digesr, 8(4):151-158, 1970.
Moataz El Ayadi, Mohamed S Kamel, and Fakhri Karray. Survey on speech emotion recognition:
Features, classification schemes, and databases. Pattern Recognition, (4235)5722587,2000 2011.
Chaoyou Fu, Haojia Lin, Xiong Wang, Yi-Fan Zhang, Yunhang Shen, Xiaoyu Liu, Haoyu Cao, Zuwei Long, Heting Gao, Ke Li, al. Vita-1.5: Towards gpt-4o level real-time vision and speech interaction. In Proceedings of the Advances in Neural Information Processing Systems, 2025.
Google. Gemini 2.0 flash model card, 2025a URL UKL/stteee///ttrrre.ggggglppppp
model- http:///dcuummttt//eeeee gemini flash.pdf
Google. Gemini 2.5 flash preview model card, 2025b.
ccom/mooeel--arrdd//ccnmeee http:///c....nceeemmeeaaaaa-aaa-
Jiawei Gu, Xuhui Jiang, Zhichao Shi. Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, al. A survey on llm-as-a-judge. The Innovation.
Hatice Gunes, Bjöm Schuller, Maja Pantic, and Roddy Cowie. Emotion representation, analysis and synthesis in continuous space: A survey. In International Conference Automatic Face & Gesture Recognition (FG), pp. 827-834, 2011.
David R Hunter. Mm algorithms for generalized bradley-terry models. Annals of Statistics, 32(1):
384 406, 2004.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700- 13710, 2024.
Woosuk Kwon, Zhuohan Li. Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems princi- ples, pp. 611-611166 166666 2023.
11

---

Published as a conference paper at ICLR 2026
Jennifer S Lerner, Ye Li, Piercarlo Valdesolo, and Karim S Kassam. Emotion and decision making. Anmual Review of Psychology, 66(1):799 823, 2015.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. alava-next-interleave Tackling muulti-maae,, video, and 3d in large multimodal models. arXiv preprint ar arXiv:2407.07895 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mvbench: A comprehensive multi-mode video understand- ing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024b.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. Science China Information Sciences, 68(10):200102, 2025b.
Lei Li, Yuancheng Wei, Zhihui Xie, Xuqing Yang. Yifan Song, Peiyi Wang, Chenxin An, Tianyu Liu, Sujian Li, Bill Yuchen Lin, et al. Vl-rewardbench: A challenging benchmark for vision- language generative reward models. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 24657-24668, 2025c.
Shan Li and Weihong Deng. Deep facial expression recognition: A survey, IEEE Transactions on Affective Computing, 13(3): 2020.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340, 2024c.
Zheng Lian, Licai Sun, Mingyu Xu, Haiyang Sun, Ke Xu, Zhuofan Wen, Shun Chen, Bin Liu, and Jianhua Tao. Explainable multimodal emotion reasoning. arXiv preprint arXiv:2306.1544 2023.
Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, et al. Mer 2024: Semi-supervised learning, noise robustness, and open- vocabulayy multimodal emotion recognition. In Proceedings of the 2nd International Work- shop on Multimodal and Responsible Affective Computing, pp. 41-48, 2024.
Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: new dataset, model, and benchmark for emotion understanding with multimodal large language models. In Proceedings of the 42nd International Conference on Machine Learning, 2025a.
Zheng Lian, Rui Liu, Kele Xu, Bin Liu, Xuefei Liu, Yazhou Zhang, Xin Liu, Yong Li, Zebang Cheng, Haolin Zuo, et al. Mer 2025: When affective computing meets large language models, In Proceedings of the 33th ACM International Conference on Multimedia, 2025b.
Zheng Lian, Haiyang Sun, Licai Sun, Haoyu Chen, Lan Chen, Hao Gu, Zhuofan Wen, Shun Chen, Zhang Siyuan, Hailiang Yao, al. Ov-mer: Towards open- vocabuaayyy multimodal emotion recog- nition. In Proceedings of the 42nd International Conference on Machine Learning, 2025c.
Zheng Lian, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. Mer- bench: unified evaluation benchmark for multimodal emotion recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2026.
Bin Lin. Yang Ye, Bin Zhu, Jiaxi Cui. Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Confer- ence on Empirical Methods in Natural Language Processing, pp- 5971-5984, 2024.
Kristen A Lindquist and Lisa Feldman Barrett, Emotional complexity. Handbook of Emotions, 4: 513-530, 2008.
12

---

Published as a conference paper at ICLR 2026
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics, pp. 12585- 2024.
Marvin Minsky. Society of mind. Simon and Schuster, 1986.
OpenAl. Gpt-4o system card, 2024.
URL https://cdn. com/
gpt 40 cystem-card..pe
Robert Plutchik. A general psychoevolutionary theory of emotion. Emotion: Theory, research, and experience, 1980.
James A Russell. A circumplex model of affect. Journal of Personality and Social Psychology, 39 (6):1161, 1980.
James A Russell and Albert Mehrabian. Evidence for fhree-factor theory of emotions. Journal of Research Personality, 1133):27-222 1977.
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca A. Popa, and Ion Stoica. Judgebench: A benchmark for evaluating Ilm-based judges. In Proceedings of the I 13th International Conference on Learning Representutions, 2025,
Gemini Team, Rohan Anil, Sebastian Borgeaud, can-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Gemini Team, Petko Georgiev. Ving lan Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv preprint arXiv:2203. 05530, 2024.
Peiyi Wang. Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, et al. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 9440-9450 2024a.
Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, Xing Xie, Wei Ye, Shikun Zhang, and Yue Zhang. Pan- dalm: An automatic evaluation benchmark for L.L. instruction tuning optimization. Proceed- ings of the 12th International Conference Learning Representations, 2024b.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20211 2025.
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter- free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.1699 2024.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 251::2412.15115.. 2024.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338, 2025.
Yueyi Yang. Haotian Liu, Fang Kang, Mengqi Zhang, Zheng Lian, Hao Tang, and Haoyu Chen. Saynext-bench: Why do llms struggle with next-utterance prediction? arXiv preprint 20Xi:26602.00322,, 2026.
13

---

Published as a conference paper at ICLR 2026
Michihiro Yasunaga, Luke Zettlemoyer, and Marjan Ghazvininejad. Multimodal reward- bench: Holistic evaluation of reward models for vision language models. arXiv preprint 14111 2025.
Jiayi Ye. Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V. Chawla, and Xiangliang Zhang. Justice or preju- dice? quantifying biases alu-as-a-judge. In Thirteenth International Conference on Learning Representations, 2025.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et mplug-owl: Modularization empowers large language models with multimodalityy arXiv preprint arXiv:2304.14172 2023.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. National Science Review, 11(12):wwac403, 2024.
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper IIm networks are fairer Ilm evaluators. arXiv preprint 00000:2308.01862, 2023.
YiFan Zhang. Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. Mm-rlhf: The next step forward in multimodal I1m alignment. In Forty-second International Conference Machine Learning. 2025.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303.18223 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Proceedings of the Advances in Neural Information Processing Systems, pp- 46595-46623, 2023.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
LLM Usage
17
B Related Works
17
B.1 Descriptive Emotion Representation
17
B.2 LLM-based Judge
17
Details of Eata
18
Data Visualization
18
C22 Description Statistics
18
Annotation Platform Layout
19
D Details of MLLM-baased Strategies
19
D.I Details of ST
19
D.2 Details of S2 and S3
20
D.3 Details of S4
20
Prompt Template for Swap Consistency
Details of Bradley-Terry Algorithm
21
MLLM Performance across Prompting Strategies
21
H
Relationship Analysis Between Humans and MLLMS
23
Model-pair Similarity
23
H.2 Ranking Similarity
23
Rationale for Choosing Binary WAF as the Default Metric
24
J Inference Cost Comparison
24
Visualization of Internal Outputs for S1~$4
25
Emotion Range Analysis of SmnoPrefer-Data
28
M Impact of Emotion Distribution on Preference
28
Impact of Templates on Swap Consistency
29
Ablation Studies on Recognition Performance
30
0.1
Impact of Input Modality
30
0.2 Impact of Prompt Templates
31
0.3
Impact of Temperature and Seed
34
15

---

Published as a conference paper at ICLR 2026
P SmmPrreee-DDtt--
35
Human-only vs. Human-LLM collaboration
36
16

---

Published as a conference paper at ICLR 2026
A LLM USAGE
In this paper, LLMs are merely used to polish language, such as improving the clarity, coherence, and fluency. We do not leverage LLMss in the generation of research ideas. Our research ideation involves deep domain expertise, creativity, and an understanding of unresolved scientific challenges, all of which rely on our insights into the field of affective computing. Therefore, LLMs serve as supportive tools for language enhancement rather than as drivers of idea breakthroughs.
B RELATED WORKS
B. DESCRIPTIVE EMOTION REPRESENTATION
Emotion is intrinsically linked to human cognition, decision- and behavior (Cacioppo & Lernrr et al.] 2013: Dolan 2002) and plays a pivotal role in artificial intelligence
Gardner 1999
(Minsky 1986 Emotion representation methods serve the foundation, aiming map complex
human emotions into computable values (Gunes et al. 2011). Within the field of emotion represen- tation, two predominant paradigms exist: categorical representation and dimensional representation.
Categorical Representation. Rooted in psychological theory, researchers classify human emo- tions into discrete categories. For instance, instance, EEkman & Keltner (1970) proposed that set of universal emotional states, termed basic emotions, exists across all human cultures, including anger, disgust, fear, happiness, sadness, and surprise. Later, [Plutchik] (1980) introduced eight primary emotions, organized in opposing pairs and arranged by intensity, which can combine to form secondary emo- tions. However, human emotions are far more complex than these six or eight labels (Lindquist & Barrett 2008). Restricting the emotional space to such limited categories inevitably overlooks nu- anced emotional experiences. To address this, Lian et al. (2025c) proposed recognizing emotions in an open-vocabulae manner, enabling the prediction of arbitrary emotion categories. This approach extends beyond traditional categorical representation paradigms.
Dimensional Representation. Rather than discrete categories, dimensional representation theory models human emotions as points in a continuous multi-dimensional space. For example, [ussell & Mchrabian (1977) proposed the Aoaance-rouusll-Dmnnaan the positivity of an emotion, arousal reflects its level of excitement, and dominance reflects the perceived sense of control in situation. Later, Russell (1980) introduced simpler Circumplex Model of Affect, reducing emotions to a -iv-dimensional space -aalnne and arousal. However, dimensional emotions are abstract and less intuitive than discrete labels (e.g., happy), limiting their practical use in downstream tasks where simpler categorical models are preferred.
(VAD) model, where valence reflects
Descriptive Representation. Neither categorical nor dimensional representations fully capture human cognition in emotion perception and understanding, rendering the emotion understanding process a black-box operation. To address this limitation, [ian et al. (2023) 2025a) proposed a novel representation strategy called descriptive emotions. They leverage form-form natural language to vi- sualize the human emotion understanding process by incorporating multimodal clues and additional analysis. Within its internal analysis, this representation further accounts for emotions' temporal dynamics, intensity, and uncertainty, thereby providing more human-like emotion representations. Despite its advantages, evaluating the quality of such open-ended descriptions remains non-trivial challenge. Previous studies rely on costly human annotators to identify human- numan--referred descrip- tions. In this paper, we propose EmoPrefer, which leverages MLLMs achieve more cost-efficient preference annotation, offering a meaningful complement to existing evaluation strategies for de- scriptive emotions. To the best of our knowledge, this is the first work to explore the capabilities of MILMMs in emotion preference decoding.
B.2 LLM-BASED JUDGE
LLM-as-a a- Judge aims to leverage LLMs as evaluators to replace traditional human-driven assess- ments, offering cost-effective evaluation solution (Gu et al.). Beyond evaluation, these LLM-based judges can also serve as reward models in reinforcement learning (Christiano et al. 2017) or act as
verifiers to select the best-of-N responses from multiple candidates (Cobbe et al.) 2021 To assess
17

---

Published as a conference paper at ICLR 2026
the effectiveness of LLM-based judges, it is necessary to generated responses and human judgments. This necessitates a series of benchmark datasets with human annotations. For example, MT-Bench (Zheng et al., (023) consists of multi-turn questions designed to evaluate chatbot's conversational abilities and adherence to instructions. EvirEval [Wang al., (0244) provides human-annotated preferences for responses generated by ChatGPT and Vicuna. JudgeBench (Tan et al.J (225) evaluates LLM-based judges on challenging response pairs requiring advanced reasoning skills, such knowledge, math, and coding. These benchmarks primarily focus on text-only preference tasks. In contrast, MLLM-as- -ass-a-Judge (Chen et al.) 2024) extends such tasks to single-image or image-seque segge-eeuence preference evaluations, broadening the range the first multimodal preference of involved modalities. In this paper, we propose EmoPrefer-Dee dataset specifically designed for human emotions. Unlike previous benchmarks that focus on text- only, single-image, or image-sequence inputs, human emotions are often conveyed through subtle behaviors embedded in multimodal cues. To achieve this, EmoPrefer-De ported modalities to full multimodal scenarios (including audio, text, and video) but also serves as the first dataset centered on human emotions. Therefore, our work provides a valuable data resource for current research on LLM-based judges.
the agreement between LLM-
not only expands the sup-
C DETAILS OF EMOPREFER-DATIE
To construct EmoPrefer-Datee provide pairwise emotion descriptions for each video and then recruit multiple annotators to evaluate which description better aligns with the character's ' emotional state. This section provides further details about the construction and characteristies of our dataset.
C.I DATA VISUALIZATION
Figure 13 presents three examples from EmoPrefer-Data. The original videos capture real peo- ple's emotional expressions in uncontrolled settings. To preserve privacy, apply style-transfer technique to anonymize the individuals. The use of this dataset has been authorized by its owner. As shown in Figure 13 these videos primarily depict single front-facing individual and include complete audio recordings, ensuring the full expression of emotions for the target speaker.
Subritie: Why are you looking me that? So, long woman, does have have relationship with me?
Subtitle: Reading requires literacy! in movies, even the illitente can enjoy themselvvs, what really competes with you cross-talk.
Subtitle: Goodness, the relationships on our 22nd Hour are getting more and more complicated. seems like I'll have to speak loss in the future.
Figure 13: Data visualization. We cartoonize these video data eliminate privacy concerns.
C.2 DESCRIPTION STATISTICS
To ensure the quality description pairs, we leverage data in two benchmark datasets: MERR-Fine and MER-Caption+. To visualize their description richness, extract emotion words and nouns from these descriptions using Qwen2.5-7B with the following prompts:
Please assume the role of expert in the field of emotions. We provide clues that may be related to the emotions of the characters. Based on the provided clues, please identify the emotional states of the main characters. Please separate different emotional categories with
Emotion Words:
18

---

Published as a conference paper at ICLR 2026
commas and output only the clearty identifiable emotional categories in list format. If none identified, please output an empty list.
We provide clues that may be related to the emotions of the characters. Please extract all nouns from the provided clues. Please separate different words with commas and output a list format. If none identified, please output empty list.
Nouns:
C.3 ANNOTATION PLATFFOM LAYOUT
Figure 14 shows the layout of the annotation platform. During the annotation process, we provide two descriptions for each video. Annotators should select the preferred description, i.e., the one that better matches the character's emotional state, based on the video content. Annotators can choose from three options: (1) description better; (2) description 2 is better; (3) tie.
Descrition 1
whacing en oer accanpaeed by sreeing bas mair whici gia
Choose the etscriptto that teet ouptures the offrrcter's emooii states
Briu 2.r
Figure 14: Annotation platform layout. We provide pairwise descriptions for each video and ask annotators to select the one that better reflects the character's emotional state. In this figure, we apply cartoonization to the video data to eliminate privacy concerns.
D DETAILS OF MLLM-BASED STRATEGIES
Figures 15 17 illustrate the prompts for S1~S4. As shown in these figures, the length of the reasoning chains increases progressively from S1 to S4.
D.1 DETAILS OF S1
For S1, we provide the MLLM with a video and two descriptions, instructing to identify the better one. Figure 15 shows the prompt used in S1.
19

---

Published as a conference paper at ICLR 2026
We provide two descriptions for a given imput. Plcase determine which one is better aligned with the imput content. If both descriptions are equally aligned with the input content, please output 'tie". Directly output the answer without additional reasoning. Video: (Video)
DI: [Description 1}
D2: (Description 2)
Figure 15: Prompt for S1.
D.2 DETAILS S2 AND S3
$2 and S3 decompose S1 into two stages: description generation and preference prediction. Figure 16 illustrates the prompts for S2 and $3.
Please provide detailed description of a given video, with particular focus on emotions portrayed.
Vidco: (Vidco)
We provide a true description and two predicted descriptions. Please determine which one is better aligned with the true description. If both descriptions equally aligned with the true description, please output "tie". Directly output the answer without additional reasoning. Ground Truth: (Output of the First Step) DI: [Description_ !
D2: [Description_
Figure 16: Prompt for S2 and S3.
D.3 DETAILS S4
S4 decomposes S1 into three steps: description generation, preference reasoning, and answer ex- traction. Figure 17 provides details of the prompt used $4.
Please provide a detailed description of a given video, with particular focus on the emotions portrayed.
Video: (Video}
We provide true description and two predicted deseriptions. Plcase determine which one is better aligned with the true description. If both descriptions are equally aligned with the description, plcase output 'tic'. Please output the answer along with the reasoning process. Ground Truth: (Output of the First Step} D1: (Description 1
D2: (Deseription_2 2}
Based on provided description, please determine which one better aligned with the ground truth description. The output should be 'DI', 'D2', 'tie".
Description: (Output of the Second Step!
Figure 17: Prompt for $4.
20

---

Published as a conference paper at ICLR 2026
E PROMPT TEMPLATE FOR SWAP CONSISTENCY
Figure 18 visualizes the prompts for normal and swapped inputs. We require the model to be robust input order changes.
We provide two descriptions for a given input. Please determine which one better aligned with the imput content. If both descriptions are equally aligned with the input content, please output 'tie". Directly output answer without additional reasoning. Video: {Video)
DI: (Description_] _
D2: (Description_2 2]
Output: (D1)
We provide descriptions for given input. Please determine which is better aligned with the input content, If both descriptions are equally aligned with the input content, please output 'tie'. Directly output the answer without additional reasoning. Video: {Video)
DI: (Description_
D2: (Description_1)
Output: (D2)
Figure 18: Swap consistency. This figure illustrates an ideal case, where the model prefers consis- tent descriptions regardless of input order.
F DETAILS OF BRADLEY-TERRY ALGORITHM
Suppose there are M models, each denoted as m. In this algorithm, each model m, is associated with positive parameter 0g, which quantifies its relative advantage. The probabilities that model m, wins, loses, or ties against model my are defined as follows:
0,
0g
3
P(m: > m2) =
0, 0,+0,+ + P(m; < myy) = 0, +
P(m, ~ mj) 0, + +
(2)
where 33r controls the likelihood of ties. The parameters can then be estimated by maximizing the following likelihood function. Models with higher 0 values ranked higher.
8
winsy
losess
tioy
max ((0) = II
(3)
0,
+0
+
0, + 01+0,+0 +
0,
0, +
where win.y, lose,y. and tie,y denote the numbers of wins, losses, and ties between model m, and model my. respectively.
G MLLM PERFORMANCE ACROSS PROMPTING STRATEGIES
Table 3 presents the performance of MLLMs across four prompting techniques. Our analysis reveals that different MILMs favor distinct prompting strategies. Notably, high-performing MILMs multimodal tasks (such as Qwen2.5-Omm and Qwen2.5-VLJ favor S1 or $2, whereas other MLLMs generally achieve better performance with S3 or S4.
21

---

Published as a conference paper at ICLR 2026
Table 3: Performance of MLLMs under four prompting strategies.
Rec. (2-Class) WAF(1)
Rec. (3-Class)
Swap
Model
ACC(1)
WAF(1)
ACC(1)
Cons. (1)
S1
38.8111.68
50.18:1.51
36.62.11.00
43.55.10.70
13.5911.552
VidcoChat
S2
38.45=0.76
49.82 49.82:10.27
33,90=0.74
38.07:00.78
14.90:12.00
ei al. 2025b
S3
45.9411.66
51.60:10.22
33.29
27.70.11.92
41.99.12.44
51.79-1.21
52.31110.88
44.88
40.7711.57
40.8511183
31.98=0.08
48.67:10.18
19.81=103
18.29=20.87
46.69=0.17
Video Video-ChatGPP ChatGPT
32.06.0..00
48.85
2.83.10.46
3,40.10.26
93.12.10.44
Maaz al. 2024
S3
45.04=10.33
47.8711.27
37.99-0.38
35.7111.552
45.30:10.00
$4
02.94
52.13++0.98
43.97+0.44
40,42+00.35
45.91++1113
S1
32.06.10.00
48.85100.00
0.07.10.00
1,92.10000
100.00.10.00
Otter
S2
32.06=00.00
08.05
0.07:10.00
1.92:00.00
98.9511..00
Li el al. 2025a
S3
0..11
06.45
00,51
31.79
46.95-10..4
S4
12.12.1.110
52,49
43.41
18.15.11.74
40.08.10.96
S1
3..59+0.53
42.63+0.118
37.01
37.63+0.87
36.24+0.17
VideoChat2 Li et al. 2024b
S2
42.02.12.48
49 29.64
12.30
45, 45.12.11.57
21.60.0.117
45.31
48.3111160
37.73-11.62
34.8411005
38.85111.74
S4
52.38+0.100
52.49+0.099
45.88+0096
43.82+00.44
41.4++1774
39.13
49.20.11.07
33.32.13.08
33.62.12.779
22.13.12.61
mPLUG-Owl
S2
3..3
48.3110.118
31.71-0.33
33.0111148
18.47:1.70
Ye al. 2023
S3
48.41-0.224
01.33+00.36
42.87-+0.40
40.16++0.44
42.25+-5.31
S4
13.7610.117
53.82.10.18
46.90
44.34.10.44
42.33.11.39
SI
32.06+0..00
48.85++0.00
31.04+0.00
47.91+0.000
0.00++0.00
Video-L VA
$2
1..06
1..85
31.04-10.00
47.9110.000
0.09
(Lin et al. 2024
S3
48. 10
51.69110.18
30.81
3..63.11.13
39.5511..35
S4
54.53+0.81
54.62+00.80
43.61+0.82
38.94+0.44
42.86+3.31
S1
36.66
48 49.1....118
35.52.10.28
1..49
10.71
Chat-UniVV
35.2010.05
49.29910.09
33.92.10.21
47.30.10.26
6.1011017
Jin 2024
48.66+0.114
01.15+0.18
44.05+00.02
42.25+0.09
44.34+22.3
$4
54.97 64
55.06.1
47.03.12.32
44.69
43.90.10.52
35.03+0.00
50.27+0.00
34.04+0..00
49.30++0.00
2.79++0.00
LLaVA-Nex- Video al. 2024a
$2
33.23:+0.00
49.38
18:00000000
48.43-+0.00
2.26++0.00
S3
10.03
51.42.10.09
40.23
42.16.10.17
51.74
56.41+00988 S4
56,57+00.98
12.84+1.25
10.78++1.11
53.92+2..00
S1
33.03
49.29.10.09
31,99 10.19
48.34.10.09
0.61
LLaMA-VID
S2
32.25=0.20
08.93:#0.09
31.23=0.119
48.00=00.09
0.09=10.09
Li 2024e
S3
54.30+1.31
56,48+ 1.24
47.76+1.37
44.16+1.113
44.16+1.3
57.10.1..3
57.10.10.62
10.42.11.17
47.13
45.12:10.70
35.79+0.00
50,44+00.00
34.69+0.00
49.48-+0.00
5.05++0.00
PLLAVAA Xu el 2024
$2
32.84+0.00
49.20++0.00
31.80+0.00
48.26++0.00
1.39++0.00
S3
51.13
53.02-10.44
47.76
46.43.10.09
51.22
S4
57.29=0.118
57.55++0.18
54.02=00.05
52.61+0.00
13.83++1222
S1
48.00
48.85
47.44:10.00
47.91
70.73:10.00
VITA-1.5 Fu et al. 2025
$2
56.34=0.00
60.04+0.000
54.74=0.00
08.89
41.99+0.00
S3
56.38+0.37
57.28++0.27
52.85+0.70
51.83+0.78
58.54+0.17
S4
60.08 +0.61
0.61
57.08 +0.16
56.01100099
19.02.-1.22
SI
13.13.0.37
54.62+00.27
39,91=0.08
50.96==0.09
6.18++0.99
Qwen2 Audio
S2
50.52+0.224
54.44+0.44
44.18+0.04
48.17+0.44
14.81+0.117
Chu 2024
18.10.1..29
58 0.344
10.46
56.36.10.61
63.94-10.70
S4
63.17=0.19
63.23++0.18
60.15=0.76
59.32+0.78
61.50+0.177
6.84
65.28
62.60+0.84
64.02-+0.78
62.02++0.35
Owen2.5-Vi
$2
62.42=00.9
62.79-10.98
60.63=0.97
61.59
68.90=0.61
Bai et al. 2025
S3
61.79+1.41
1..61+1.33
19.38811.62
16.53+11.48
58.71+1.05
S4
63.37+33.47
63.59
60.1 1
29.23+2.44
58.19+
S1
63.36=0.00
63.4++0.00
61.54=0.00
62.20=00.00
73.87=0.00
Qwen2.5 SmOnni
82
67.21+0.00
67.32+0.00
65.29+00.00
66.03+0.00
79.09+0.000
Xu ct al. 2025
S3
61.5420.07
61.72-00.09
59.05
56.62-10.00
72.6540.52
65.69=0.29
65.81+0.27
63.48=0.29
62.54-0.35
22

---

Published as a conference paper at ICLR 2026
H RELATIONSHIP ANALYSIS BETWEEN HUMANS AND MLLMS
We analyze the correlation between MLLMs and humans using the examples mentioned Section Specifically, first recruit multiple human annotators perform preference annotations. Then, we examine this correlation from two perspectives: model-pair similarity and ranking similarity.
5
H.1 MODEL-PAIR SIMILARITY
Taking the comparison between m, and m, as an example. We denote the counts of wins, losses, and ties between m, and my for human annotators win.y, loss.j. and tie.j. and those for MLLMS win.y. lossij. and tie.j. To evaluate the similarity between humans and MILMs, first convert these counts into three categories:
win, win.j lossij
win, winej lossij
Vij
lose,
win lossij
Øūj lose,
winss lossij
(4)
tie, win,y lossij tic,
wingj = lossis
Figure presents the model-pair comparison results for humans and MLLMs.
win
lose
win
win
win
win
wini
lose
win
min
win
vin
1po0
losa
lose
lose
win
la:0
losa
lose
lose
tib
win
Vin
Wi Count
Win
Iose
lose
10ee
lose
lose
lose
lose
Fie
lose
win
Vin
lose
1050
lose
lose
Icsa
lase
kose
lase
lose
lose
lase
m00022000
los2
win
lose
tic
win
m220229400
los2
lose
lose
lose
win
(a) Human
(b) MLLM
Figure 19: Model-pair comparison results.
We then compute the similarity score between them:
Similarity ΣM Ej=it1 Thyn = Uts
(5)
ΣM Σ Dj=41
where TT.)) is the indicator function. Experimental analysis reveals an 80.95% agreement rate, indi- cating strong alignment between humans and MIL.Ms.
H.2 RANKING SIMILARITY
Using the Bradley- Bradley-Terry Terry algorithm (see Appendix(F), can derive model-level rankings from pair- wise comparisons in Figure 19 In this section, we further evaluate the ranking similarity between MILMM and humans. Specifically, similarity. Let Rank and R-nkg(.) be two ranking functions. For each model m,, its rank the two rankings is denoted as Rank (m); and Rank (mz). Spearman's 's p computed as:
Spearman's rank correlation coefficient p quantify this
G [Rank, a (m;) Rank z (mm)))
(6)
M - 1)
This coefficient p ranges from -1 to 1, with higher values indicating stronger positive correlation. Through experimental analysis, we observe that p reaches 0,8571, relatively high score, indicating good consistency between human and MI.LM in decoding emotion preferences.
23

---

Published as a conference paper at ICLR 2026
I RATIONALE FOR CHOOSING BINARY WAF AS THE DEFAULT METRIC
Theoretical Basis. First, we opt for binary classification rather than tri-class because ties contain higher ambiguity than clear preferences (e.g., when one description is distinctly better than another), as we discussed in Section Binary performance focuses on more definitively labeled comparisons, enabling more reliable conclusions. Second, chose WAF over ACC due to the inherent class imbalance in EmoPrefer- Data. As shown in Table 4 the d1 category slightly outnumbers the d2 category. In imbalanced datasets, ACC is dominated by the majority class, whereas WAF better accounts for class imbalance. Therefore, we adopted WAF as our primary metric. Combining these two considerations, we use binary WAF as the default evaluation metric.
Table 4: Class distribution in EmoPrefer-Diiee For a given video and two descriptions (ar, d1, d2), the preference label falls into one of three categories: d1, d2, or tie.
Preference Percentage
50.17%
d2
47.91 %
tie
1.92%
Experimental Analysis. Using the scores from Table we analyze the correlation between differ- ent metrics. As shown in Figure PCC scores between these metrics are relatively high, indicating strong inter-metric correlations. Meanwhile, we examine the ranking consistency across metries. Although the rankings from different metrics are not identical (see Table their overall trends are similar. For instance, Owen2.5-Ommn consistently performs best both binary and tri-class metrics, whereas VideoChat ranks relatively poorly across all metrics.
1.00
1.00
0.97
0.97
0.97
0.97
1.00
0.99
0.96
0.97
0.99
1.00
Recid
Figure 20: PCC scores between different metrics.
J INFERENCE COST COMPARISON
In this section, we fix the MLLM to O0mmn and report the inference costs across S1~S4. As shown in Table [5] S1 the fastest strategy due its fewer inference steps. Mcanwhile, S3/S4 runs faster than S2. The primary reason is that we leverage vLLM (Kwon et al.) 2023) to accelerate the inference process of external LLMs in S3/S4, resulting in reduced inference time.
Table 5: Inference cost analysis for different prompting strategies.
Promtping Strategy
Inference Time per Sample (seconds)
$1
1.3875
S2
5.4188
S3
4. 4.0442
S4
4.9370
24

---

Published as a conference paper at ICLR 2026
K VISUALIZATION OF INTERNAL OUTPUTS FOR S1~S4
We employ Video-ChatGPT as the MLLM and visualize its outputs under S1~S4 strategies.
Description In the viden, young displays facial expression characterized by widened eyes and arched lips, indicating surprise, shock, or even anger. Her negarive tone and intonation further support this inforence. Description
text, cuption content: "Our unfinished busimess." This sentence might someone's response to a particular situation dialogue, Rased the video cloes showing the woman's 's happiness, excitement, or curiosity, and her body langunge indicating of comfort and case, we can infer that this sentence might earry casual or playfal tone. However, based on the audio clues deseribing tension, urgency, and pressuree and the of frustration, anger, or anxicty. Therefore, this sentence may be a sarcastie or provocative response, expressing the complex emotional state of the person in the
the phrase "our husiness" indicating responsibility abligation,
also infer thar this sentence might carry
Strategy S7
We provide two descriptions for given input. Please determine which one is better aligned with the imput content. If both descriptions are equally aligned the input content, pleuse output Directly output answer without additional reasoning.
Video: [Video]
DI: [Description, 1]
D2: [Description
Output: Both descriptions are equally aligned with the input content, as they both describe the same scene with a young woman displaying facial expression characterized by widened eyes arched lips, indicating surprise, shock, or anger. audio clues the video suggest rension, urgency, and pressure, while the caption content "Our unfinished business" implies casual playful tonc. The woman's body language indicates comfort and ease, which further supports the ides that the sentence might carry a casual or playfal tone. However, the audio lead sarcastic provocative response. Overall, both deseriptions
clues also suggest that the situation could tense urgent, which provide plausible interpretations of the seme, making it difficult to determine which one is more accurate.
Strategy S2
Please provide detailed deseription of given video, with a particular focus the emotioon partrayed. Video: (Video)
young girl talking young woman, the girl crying. young woman appears to concerned and tries console the girl. The girl sitting couch and the young woman is standing front of her. The emotions displayed in the video of suchness and concern.
Output: video
We provide a true description and two predicted descriptions. Please determine which one is better aligned with the true description. If' both descriptions are equally aligned
true description, please output Directly ourput answer without additional reasoning. Ground Truth: (Output of the First Step)
DI: [Description. 1]
[Description Output: In the video, young woman displays a facial expression characterized by widened eyes and arched lips, indicating surprise, shock, or even anger. negative tone and intonation further support inference. young girl sitting standing front of her. The emotions displayed in the video sadness and concern. The young woman trying console the girl, which suggests a sense of empathy and care. The presence of a couch and the fact that the girl sitring on it imply casual and comfortable setting, The young woman's facial expression and body language indicate that trying help the girl feel better, which further supports idea that the situation is not serious urgent.
couch, and the young woman
Figure 21: Visualization of the internal outputs of and S2.
25

---

Published as a conference paper at ICLR 2026
Deseription In the video, young. woman displays a Beial expression characterized by widened eyes and arched lips, indicating supprise, Shock, anger. Her negative tons and intonation fürther support this inference.
Description 2: In the text, cuption content: "Our un unfinished business," This sentence might someone's resporsse to particular situation or dialogue. Based case, we can infer that this seneence might pressure,
happiness, excitement, curiesity. casual or playtil tone. However, based on the audio clues describing tensioe, urgency, and the phrase husiness indicating responsibility obligation, of frustration, anger, anxiety. Therefore, this sentence may surcastie provocative response, expressing the complex emotional of the person in the simuatio..
video clues showing
body language indicating
sentence might carry
Strategy
Please provide detailed description of given videa, with particular focus the emotions portrayed.
Video: [Video]
Output: vidco console the girl. The girl is sitting on couch and the young woman is standing in front of her. The emotions displayed in the video are of sadnesss and concern.
young girl talking a young woman, the girl is crying. young weman appears concerned and tries
We provide a true description and two predicted deseriptions. Please determine which one better aligned with the true deseription. If both description. please output Directly output
descriptions are equally aligned Ground Truth: (Output of the First Step]
additional reasoning.
DI: [Description
D2: [Description 2]
Output: DI
Figure 22: Visualization of the internal outputs of S3.
26

---

Published as a conference paper at ICLR 2026
Description I: In the viden, young woman displays facial expression characterized by widened eyes and arched lips, indicating surprise. shock, or even anger. Her nogative tone and intonation further support this inforence.
Description It the toxt, caption content: "Our unfinished business." This sentence might somcone's response to a particular situation or dialogue. Based on the video clues showing the woman's is happiness, excitement, or curiosity, and her body langunge indicating of comfort and case, we can infer that this sentence might carry casual or playtal tone, However, based on the audio clues deseribing tension, urgency, and pressure, and the use the phrase "our business" indicating responsibility or abligation, of frustration, anger, or anxiety. Therefore, this sentence may be a surcastic or provocative response, expressing the complex emotional state of the person in the situation.
also infer that this sentence might carry
Strategy S4
Please provide detailed deseription of a given video, with a particular focas on the emotions portrayed. Video: (Vidco)
Output: The video shows young girl talking to young woman, and the girl is crying. The young woman appears to be concerned and tries to consolc the girl. The girl sitting on couch and the young woman is standing in front of her. The emotions displayed in the vidco are of sadness and concern.
We provide truc description and two prodicted descriptions. Please determine which descriptions are equally aligned with the true description, please outpot 'tie'. Please output the answer ulong with the reasoning process.
aligned with the description. both
Ground Truth: (Output of the First Stop)
DI: [Description_] 1]
D2: [Description 2)
Output: Based on the provided ground truth description and two predicted descriptions, let's analyze each oneaatl Ground Truth Description: A young girl is crying. A young woman is concemed and trying to console the girl.- The gidl is sitting on couch. The young woman standing in front her- The emations displayed are sadness and concemm 4AH Predicted Descriptionsl. "'DI"s; "In the video, young woman displays facial expression charatterized by widened eyes and arched lips, indicating surprise, shock, begative tone and intonation further support this inference."" '2. **D2*s "In the text. caption content: Our unfinished business.' This sentence might someone's response particular situation or dialogue. Based
unger. Her
video clues showing the woman's happiness, excitement, or curiosity, and her body language indicating of comfort and ease, we can infer that this sentence might carry casual playful tone. However, bused on the audio clues deseribing tension. urgency, and pressure, and the use of the phrase 'our business' indicating responsibility or obligntion. also infer that this sentence might carry sense of frustration, anger, anxiety. Therefore, this sentence may he sarcastic or provocative respense, expressing the cemplex emetional state of the person in the situation. "WHe Analysis:- **D]** focuses on the young woman displaying expressions of surprise, shack, anger, which de not align and trying to console the crying girl.- **pz** contains completely different context and description ("Our unfinished business") that does not muich the seene described the ground truth at all. It talks about woman's emotional state being happy, excited, or curious, which contradicts the ground tnuth where she concemed and trying to console the girlaate Conclusion: Neither the provided predicted descriptions aligns well with the ground truth description. Since both descriptions contain significant discrepancies with the actual soene, the most appropriate response is: ""mmm" This indicates both descriptions are equally the mark and do accurately reflect the given ground truth.
ground where is descrihed being concerned
Based the provided description, please determine which one is better aligned with the ground truth description. The ousput should be 'DI'. "D2', or "tic".
Description: (Output the Second Stop)
Output: tie
Figure 23: Visualization of the internal outputs of S4.
27

---

Published as a conference paper at ICLR 2026
L EMOTION RANGE ANALYSIS OF EMOPREFER-DATA
This paper focuses on descriptive emotions, which differ from basic emotion recognition. In basic emotion recognition, restricts the label space few basic emotions. In contrast, descriptive emo- tions encompass open- vecebvocyy set of emotion words. For clarification, Figure 24 visualizes descriptive emotions, which include diverse emotional words in an open-vocabulary manner. Figure 25 presents the emotion word cloud for EmoPrefer- Data, demonstrating that our dataset includes wide range of emotions. Notably, we observe a skewed emotion distribution. For instance, anger appears more frequently than helplessness.
Subtitle: Why are you all looking at me like that? So, as long as it's woman, does she have have a relationship with me?
shows male churacter in indoor setting. At the beginning the video, his eyes wide open and his mouth is also open, indicating a surprised facial expression. In the following scenes, he looks around, seemingly explaining or narrating something to the people around him. Overall, emations are not positive or optimistic, stutter, which usally expresses feelings of nervousness, anxiety, unease. Combined with the text content, the character seems to be unhappy and ungry due the prejudice of the people around him. The subtitee in the text says. "Why are you all looking at me like that? So, as long as it's
Descriptive Emotion: In the video,
audio, the character speaks with
does she have have relationship with me?" This sentence expresses the male character's dissarisfaction and anger towards the people around him. Based on the surprised and negative facial expression of the male character in the video clues, as well as the stuttering speech in audio clues, we can infer
male churacter expressing feeling discatisfaction anger in this sentence. may feel troubled by the prejudice of the people around him and unhappy with this unfair reatment. Descriptive Emotion - Open-vocabulary Emution: surprise, nervaus dixsatiefied
Figure 24: Visualization of descriptive emotions, which include diverse emotional words in an open- vocabulary manner.
sUrp positive
wornied
PIM
conce e
neu
a1
sad
tens
iion
confusion
an
ste
e
nappy
S
sadness
concerned
axxousanxa
anxiet
anxie
di
dissati
Lety
anxious
ted
trustrat
trus trattlon
Ffrustrated
angry
disappointnent
tu
Figure 25: Emotion word cloud visualization of EmoPrefer-Data.
M IMPACT OF EMOTION DISTRIBUTION ON PREFERENCE
In this section, we investigate how emotion distribution influences human and MLLM preferences. Specifically, conduct an insightful experiment. Figure 26 visualizes the emotion word clouds of preferred and non-preferred descriptions from both humans and MLLMs. These figures reveal similar distributions. For example, anger and frustration frequently appear in all distributions. To quantitatively assess their similarity, we compute the Jensen- Shannon Distance (JSD) between dif- ferent distributions, where lower values indicate higher similarity. Since we do not restrict the emotion range in EmoPrefer-Data, we calculate this score only for overlapping emotions in two dis- tributions. Experimental results in Figure 27 demonstrate that these distributions exhibit relatively high similarity. Based on both quantitative and qualitative analyses, we conclude that preferred and non-preferred descriptions from humans and MLLMs share similar emotion distributions. other words, emotion distribution alone cannot predict human or MLLM preferences. For instance, the presence of anger does not reliably indicate preference. Therefore, we can conclude that emotion distribution does not introduce bias in MLLM's preference evaluation.
28

---

Published as a conference paper at ICLR 2026
concerned
anx
worry angry sadnes's sfrustrrtte
e
happy
worrled e
restig
anXious
Ohe
e
aaxiet
disappoiit
excited
positi
Frustration ange
e
trent
heurr ral
stress
neut
frusttrtee
USa
mataaa
Fear ful
a
Concer DIR
saa
on
desplar
dissa
sad
sfaction
wor
worried ried
The
a anx1ous concer
confidert
con fusion
tens
ion
angry
saidnes
mant reston regret
n tense
relaxed
Joyful
(a) Preferred (Human)
(b) Non- preferred (Human)
Non-
anx1ous
ano tension
usfuueed
happy
a
concern
O
sadness
concernet
pasiaaee
engry
worried
angry
WO
sad
itenson
surprised
joyfur
pateclh aed
sad
a
fear
neit
rel relaxed wnm
feariu
ge
atreen
anx1out
and Srenntii
hhappy
anxiety
aoooy K tems
confusson
concern
excated
an
fear
angry
f frustrat 1on
frustrated excled
ton
FustFaa
f
disappointment
Fie
verry
dissatisfactio
S
sadness
aiaioon
S
1on
of
worried
(c) Preferred (MLLM)
(d) Non- (MLLM)
Figure 26: Word clouds for preferred and non-preferred descriptions of humans and MLLMs.
0.00
0.28
0.11
0.23
NPHH
0.28
0.24
P(M)
0.1.
0.24
0.00
0.23
NP(M)
Р(H)
PIMI
NP(M)
Figure 27: ISD between different emotion distributions, where lower values indicate higher simi-
larity. P(H), NP(H), P(M), and NP(M) abbreviations for human-preferred, human- nonpreeenpe,erred MLLM-preferred, and MLLM-nonpreferree emotion distributions, respectively.
N IMPACT OF TEMPLATES ON SWAP CONSISTENCY
In Figure 28 we examine three different templates and analyze their influence on swap consistency. Experimental results are presented Table 6 where we use Qwen2.5-Omni as the MLLM. We observe that different templates lead to variations in swap consistency. Among them, template #2 achieves the best performance.
Table 6: Impact of prompt wording on swap consistency.
Prompt Template
Swap Consistency
Template #1
79.97=0.00 87.80=0.00
Template #2
Template #3
80.31 +0.00
29

---

Published as a conference paper at ICLR 2026
Template 1:
We provide two descriptions for given input. Please determine which one is better aligned with the input video. If both deseriptions are equally aligned with the inpul content, please outpul "tie". Directly output the answer without additional reasoning. Video: (Video]
DI: [Description_]]] 13
D2: [Description_2] 2]
We provide descriptions for given input. Please determine which better aligned with the input video. If both descriptions equally aligned with the input content, please output 'tic'. Directly oulput the answer without additional reasoning. Video: [Video]
DI: [Description
D2: [Description _]
Template 2:
Two descriptions are provided below, Please identify which one aligns better with the input video. they equally aligned, respond with 'tie'. Output only answers and no explanation needed. Video: [Video)
D1: [Description 13
D2: [Description_2]
Two descriptions are provided below, Please identify which one aligns better with the input video. they are equally aligned, respond with 'tic'. Output only answers and no explanation needed. Video: (Video)
D1: (Description 2}
D2: [Description 13
Template 3:
For the provided input video, we present two candidate descriptions. Determine which description is more aligned with the input content. If neither is preferable, return 'tic'. Your response should be either 'D'', or 'tie', with no further commentary. Video: (Video)
DI: (Description 13
D2: (Description 2}
provided input video, we present two candidate descriptions. Determine which description more aligned with the input content. If neither is preferable, return 'tic'. Your response should be either 'D1', 'D2', 'tic', with no further commentary. Video: (Video) D1: [Description 2
D2: (Description 1)
Figure 28: Visualization of different prompt templates for swap consistency.
O ABLATION STUDIES ON RECOGNITION PERFORMANCE
0.1 IMPACT OF INPUT MODALITY
In Table we analyze how different input modalities affect recognition performance. Experimen- tal results demonstrate that incorporating lexical modality consistently improves performance. For instance, audio-text inputs outperform audio-only inputs. However, using all modalities (audio- video-text) does not yield better results than using either audio-text or video-text inputs.
30

---

Published as a conference paper at ICLR 2026
Table 7: Impact of input modality on recognition performance.
Input Modality
Rec. (2-Class) WAF(1)
Rec. (3-Class)
Audio
Video
Text
ACC(1)
WAF(1)
ACC(1)
65.16:1:.00
65.19:10.00
63.30-1:.00
63.94-10.00
x
v
69.09:10.36
69.09-10.36
66.29-40.53
65.85-10.52
X
66.93:10.00
67.50 67.50:10.00
65.02:10.00
66.2011.00
X
V
v
68.92+00.1
69.18+0.62
66.27+00.6
65.2440.44
67.54:10.00
67.6710.00
65.61:10.00
66.38110.00
68.34+0000
68.38+0.00
66.38+00.0
67.07+0.00
O.2 IMPACT OF PROMPT TEMPLATES
To evaluate the model's sensitivity to prompt templates, we conduct experiments using three differ- ent templates for S4. These templates are illustrated in Figures 29 31
Template 1:
provide two descriptions for given input. Please determine which one better aligned with input video. both descriptions are equally aligned with the input content, please output "tie'. Directly output the answer without additional reasoning.
Video: [Video)
DI: (Description 1
D2: (Description_2 2|
Template 2:
Two descriptions provided below. Please identify which aligns better with the input video. If they equally aligned, respond with Output only answers and no explanation needed.
Vidco: [Vidco)
DI: (Description_
D2: (Description_2 2
Template 3:
For the provided input video, we present two candidate descriptions. Determine which description is more aligned with the input content. If neither preferable, return 'tie'. Your response should be either 'D1', 'D2', 'tie", with no further commentary.
Video: (Video)
DI: [Description 1)
D2: (Description 2)
Figure 29; Different prompt templates for S1.
31

---

Published as a conference paper at ICLR 2026
Template 1:
Please provide detailed description of a given vidco, with particular focus on the emotions portrayed. Video: (Video)
Template
Please provide a detailed deseription of the given video, with particular focus on the emotions expressed.
Video: (Video)
Template
Describe the given video in detail, paying special attention to the emotions shown.
Video: (Video)
Template
We provide truc description and two predicted descriptions. Please determinc which one better aligned with true description. If both descriptions are equally aligned with the true description, plcase output 'tic". Directly output the answer without additional reasoning. Ground Truth: {Output the First Step) DI: [Description 1}
D2: (Description 2]
Template 2:
The ground truth description and two predicted descriptions provided below. Please identify which predicted description aligns better with the ground truth. If both are equally aligned, output 'tie'. Your answer should be either D1', 'D2', or 'tie'. Do not include any reasoning. Ground Truth: (Output of the First Step) DI: (Deseription !}
D2: (Description 2)
Template 3:
We provide the correct description and two model-generated descriptions. Determine which of the two predicted descriptions is more consistent with the ground truth. If there is no clear difference in alignment, output 'tic". Please respond with 'DI', 'D2', or 'tie' only.
Ground Truth: (Output of the First Step)
DI: (Description_]. 1}
D2: [Description_22 2
Figure 30: Different prompt templates for S2 and S3.
32

---

Published as a conference paper at ICLR 2026
Template I:
Please provide detailed description of given video, with particular focus on the emotions portrayed. Video: [Video]
Template
Please provide a detailed description of the given video, with particular focus on the emotions expressed.
Video: (Video)
Template
Deseribe the given video in detail, paying special attention to the emotions shown. Video: (Video)
Template 1:
We provide true description and two predicted descriptions. Please determine which one is better aligned the true description. If both descriptions are equally aligned with the true description, please output "tic'. Please output the answer along with the reasoning process.
Ground Truth:
(Output the First Step)
DI: [Description. 1)
ID2: (Description 2|
Template 2:
The ground truth description and two predicted descriptions given. Please determine which of the two predicted descriptions is better aligned with the ground truth. If both are equally aligned, please output "tic'. Provide your final answer ('D1", 'D2', or 'tie" along with the reasoning process that leads to your choice. (Output of the First Step)
Ground Truth:
DI: (Description 1}
D2: [Description 2]
Template 3:
Given the ground truth and two predicted description. Determine which predicted description is better aligned with the ground truth. both are equally aligned, return 'tie'. Your output must include final answer ('DI', 'D2', or "tie") as well as clear explanation of your reasoning.
Ground Truth: (Output of the First Step)
DI: (Description 1}
D2: (Description_2 2
Template
Based on the provided description, please determine which one is better aligned with the ground truth description.
The output should be 'D1', 'D2', "D2', or 'tic". Description: (Output of the Second Step)
Template 2:
Based the provided description, please determine which of the two options, 'D1' or 'D2', is better aligned with ground truth description. The output should be either 'D1', 'D2', or "tie". Please directly output the answer without any reasoning.
Description: (Output the Second Step)
Template 3:
Given the description, decide which is more aligned with the ground truth: 'DI' or 'D2', If they are equally aligned, output "tic". Just output 'D1" or 'D2', or "tic", No reasoning needed. Deseription: (Output of the Second Step)
Figure 31: Differnet prompt templates for S4.
33

---

Published as a conference paper at ICLR 2026
As shown in Table the impact of different prompt templates is minimal. Different prompt tem-
plates yield similar performance in emotion preference decoding.
Table 8: Impact of prompt templates on recognition performance.
Rec. (2-Class)
Rec. (3-Class)
Template
WAF(f)
ACC(1)
WAF(t)
ACC(1)
#1
67.74:1.000
67.85=0.00
65.81=0.00
66.53:10.00
#2
68.02+0..0
68.03++0.00
66.07+0.00
66.72++0.00
66.46++0.00
66.61+0.00
64.57+0.00
65.33++0.0
O.3 IMPACT OF TEMPERATURE AND SEED
Tables 9 and 10 present the recognition performance under varying temperatures and random seeds.
For temperature, we test values the range 0.7 to 1.3, avoiding extreme settings. For random seeds, we compare three choices: 100, 200, and 300. Experimental results demonstrate that neither temperature nor seed affects recognition performance.
Table 9: Impact of temperature on recognition performance.
Rec. (2-Class) WAF(t)
Rec. (3-Class)
Temperature
ACC(1)
WAF(f)
ACC(1)
0.7
67.74=0.00
67.85=0.00
65.81=0.000
66.55=0.00
1.0
67.74110.00
67.85=00.00
65.81+10.00
66.55=0.00
67.74:10.00
67.85=00.00
65.81+10.00
66.55=0.00
Table 10: Impact of seed on recognition performance.
Rec. (2-Class) WAF(1)
Rec. (3-Class)
Seed
ACC(t)
WAF(1)
ACC(1)
100
67.7411.000
67.85.10.00
65.81-10.00
66.55110.00
200
67.74=00.00
67.85+0.00
65.01=0.00
66.55=00.00
300
07.74=00.0
67.85:+0.00
65.81=00.0
66.55=t0.00
34

---

Published as a conference paper at ICLR 2026
P EMOPREFER-DATEE V2
In EmoPrefer-Data, we recruited three annotators to label 1,368 pairs. this section, we expanded the annotator pool to 14 annotators and labeled an additional 2,096 samples, creating new dataset EmoPrefer-Data- V2. Table [T presents the performance of various MLLMs on both datasets. We observe that Qwen2.5-Ommi, which performed well EmoPrefer- -Dta, also achieves strong perfor- mance on EmoPrefer-Datte V2. To quantitatively assess their similarity, we calculate the PCC simi- larity between corresponding metrics of the two datasets. In Table [T] the PCC score reaches above 0.75, relatively high value. These results indicate that the conclusions drawn from Praterer-- are reflective of the MI..Ms* statistical performance in emotion preference decoding.
Table 11:
Main results on EmoPrefer-Daae and EmoPrefer-Data- V2. This table reports the two- class recognition performance of various MLLMs under their best-perg prompting strategies.
EmoPrefer-Date
Mmmorreer-DDta- ACC(1)
Model
WAF(1)
ACC(1)
WAF(1)
Open-source MLLMs
VideoChat (Li al.
2025b
11.79=1221
52.31=0.98
56.63:10.79
58.37=0.65
JddooChatGPT (Maaz
2024
52.10=0.94
52.13
55.03:10.12
54.74=1.009
Otter et al.
2025a
52.12+1.10
52.49+0.98
58.53++0.12
61.144+0115
VideoChaf2 (T.1 ct al.
2024h
52.38+0.10
52.49+0.09
56.77+0077
57.14+0.995
mPLUG-OWl Ye al.
2023
53.7610.117
53.82:10118
56.66:1..40
16.15
Video-LLaaA (Lin ct
2024
54.53+0.81
54.62+0.80
56.50+-0.25
56.98+0.25
Chat-UniVi JJin
2024
14.97+1.64
1..06+1.60
55.38+
55.35-+0.28
LLaVA-Next Video (Li
2024a
56.41
56.5711.98
57.28.10.49
56.86110.43
LLaMA-VID (Li et 20246
57.1010.63
$7.1010.62
1..91
58.43110.95
PLLAVA (Xu ct al.
2024
57.29=0.18
07.55
1..3
17.85
VITA-1.5 [Iu et al. 2025
60.08:10.6
60.12:10.22
63.59410.00
64.00
Qwen2-Audio (Chu al.
2024
63.17-0.19
63.2340..18
58.53
58.98
Qwen2.5-VL (Bar et al.
2025
64.43=0.87
65.2840.80
61.6610..1
12.12-0..5
Qwen2.5 500m (Xu et al.
2025
67.21
67.32.10.00
64.96:10.00
66.03+10.00
Cloved-source ML..S
GPT-4o OpenAl
2024
59.28+0008
59.41
64.54+0.00
64.62++0.00
Gemini-24 2.0-ii--.--Fahh
Google
2025a
59.80=0.88
60.39+0.89
63.53=+0.0
63.69+0055
Gemini-2.55 Faa-aahhh
(Gogge 202.5b
60.60-0066
61.1911.62
65.06:10.37
65.60.10.43
GPT-4.1 (Achiam al. 202.3
60.75:0.11
60.75-10.18
64.73-1..00
64.55-10.00
Gemini- 1.3-Pro [Team ci al.
2024
60.79-0.68
61.55-10.22
61. 61.9410.06
61.42+00.6
Gemini-1.5-Flaan Team et al.
2024
64.64-10.12
65.19
63.27+40.12
66.46
PCC Similarity Between Corresponding Metrics of Two Datasets
PCC Score
0.7754
0.7570
0.7754
0.7570
35

---

Published as a conference paper at ICLR 2026
Q HUMAN-ONLY VS. HUMAN-LLM COLLABORATION
In Figure we present an example illustrating the differences between human-only annotations and those generated through human-LLM collaboration. We observe that human-LLM collaboration produces more detailed and refined descriptions with the assistance of MLLMs.
Subeitte: Don't about feelings anymere.
Humae-only Annotation: In the video, she roared, speke passionately, spoke quickly, spoke loudly, and stared with wide eyes. Therefore,
inferred her emotional state anger.
with expressive mouth movements, seemingly expressing her views emotions more strongly. the audio, the voice expresses anger. The tome sharp and aggressive. There are no pauses, the speech is fiss, and the volume foclings anymore."
Human-LLM Collaboration: In the video,
the subtitic says, "'Don' talk something, Based on the video clue of the woman'' expressive mouth movements and the audio clue of ungry voice with sharp and aggressive tone, fast speoch, and high volume, we can infer very unhappy with other person's wards actions, and therefore expresses her emotions with aggressive tone.
bigh.
sentence may be the
response someone
sentence carries sense anger and dissatisfaction. The woman may be
Figure 32: Comparing annotation quality: human-only vs. human-LLM collaboration.
36


Reviews:
[{"review_id": "NEM0RZXN15", "reviewer": "Reviewer_ZiTv", "summary": "This paper explores an interesting topic of human emotion preference, aiming to determine whether large language models can understand human preferences on different emotional representations. To this end, they construct EmoPrefer and introduce a new dataset (EmoPrefer-Data) and benchmark (EmoPrefer-Bench), supported by extensive experiments that reveal the upper-bound performance of current models. This work not only provides evaluation metrics for descriptive emotions but also advances future research toward developing more emotion-intelligent LLMs.", "questions": "See wkes", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.This paper proposes EmoPrefer, dedicated to human emotion preference.\n\n2.This paper introduces a new dataset and a new benchmark, laying the foundations for this field.\n\n3.This paper conducts extensive experiments, revealing the upper-bound performance of current models, and proposes techniques to enhance the performance on emotion preference prediction.\n\n4.This paper has promising applications in descriptive emotion understanding and MLLM training.", "weaknesses": "1.\tThis paper aims to investigate whether MLLMs can replace humans in decoding emotion preferences. Besides experiments on EmoPrefer-Data, it would be beneficial to further discuss the relationship between MLLMs and humans in practical applications (mentioned in Figure 12).\n\n2.\tFor EmoPrefer-Data, the authors primarily use samples with unanimous preference annotations. To better reveal human preferences, an analysis of which descriptions humans prefer most—considering factors such as emotion richness, description length, and modality coverage—would be necessary.\n\n3.\tThe paper employs win/lose/tie counts and the Bradley-Terry algorithm to derive final rankings. To the best of our knowledge, alternative approaches exist—such as using binary 1/0 scores for win/lose without detailed counts. A discussion on why this ranking method was chosen in this paper should also be provided.\n\n4.\tFigure 10 shows that recognition performance is highly dependent on the choice of k. An explanation for this phenomenon would strengthen the analysis.\n\n5.\tFigure 7 indicates that the larger LLM (Qwen3-14B) underperforms compared to the smaller LLM (Qwen2.5-7B). This unexpected finding needs further explanation.\n\n6.\tThe paper evaluates MLLMs in human emotion decoding under a zero-shot setup, with current models achieving around 70% recognition performance. While the focus of this paper is on evaluation, it would be valuable to discuss potential solutions for improving model performance in this task. Such discussion should be included in the future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XjiNcI6vBi", "reviewer": "Reviewer_uypZ", "summary": "This paper addresses the challenges in Descriptive Multimodal Emotion Recognition (DMER), which uses free-form natural language to describe emotions for finer-grained representation. To tackle evaluation difficulties and reduce the manual effort in preference annotation, the authors propose EmoPrefer, the first exploration of leveraging multimodal large language models (MLLMs) for emotion preference decoding. They introduce EmoPrefer-Data, a high-quality expert-annotated preference dataset, and EmoPrefer-Bench, a benchmark to evaluate MLLMs and prompting techniques.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The results of the experiment are promising.", "weaknesses": "1. The dataset's sample source is too narrow and the scenarios are too limited. All original videos are from the MER2024 dataset, and the scenarios are restricted to \"a single character facing forward with complete audio,\" lacking diverse scenarios such as multi-person interaction and complex environmental background interference.\n2. This paper does not explain the theoretical basis for choosing binary WAF as the default metric. Further correlation analysis between binary and tri-class performance should be provided (e.g., whether the performance rankings of different MLLMs are consistent under both settings) to confirm that the default metric does not distort model comparisons.\n3. Exchange consistency only quantifies whether the results are consistent before and after the order is swapped, but it does not explore the root cause of order bias. For example, is this bias caused by the model's attention mechanism or the modality fusion logic?\n4. The paper designs four prompting strategies (S1-S4) with increasing reasoning chain complexity, but does not clarify why external LLMs improve performance in S3/S4. It only attributes it to \"MLLMs may have limited language understanding,\" but does not verify whether the improvement comes from the external LLM’s stronger language reasoning ability or the multi-step reasoning’s error reduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hoZOYrqM2k", "reviewer": "Reviewer_y5v9", "summary": "This paper proposed EmoPrefer, which presents a pioneering exploration into whether MLLMs can serve as cost-effective evaluators of human emotional understanding.\nThis work constructed EmoPrefer-Data, EmoPrefer-Bench, and evaluation metrics. These will contribute to the study of this area.\nThe experiments show that open-source models such as Qwen2.5-Omni can rival or surpass closed-source ones (GPT-4.1, Gemini 2.5), achieving 67.2% WAF and 79.1% swap consistency. Model-based crowdsourcing further improves both performance and robustness. The authors argue this lays groundwork for emotion-aware reward modeling and emotionally intelligent AI.", "questions": "1. Is “swap consistency” sensitive to prompt wording or model randomness?\n2. Could the authors elaborate on how cultural bias in EmoPrefer-Data might affect model generalization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is working on a novel research problem. It also presents a EmoPrefer-Data and EmoPrefer-Bench to support the research for this area.\n2. Experimental Setup is comprehensive with insightful analyses.", "weaknesses": "1. Limited Dataset Scale and Diversity. Only 1,368 annotated pairs. The videos are only sourced from MER2024.\n2. Lack of Downstream Evaluation. No demonstration of how emotion preference prediction could enhance practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2eh6L8OT2O", "reviewer": "Reviewer_evbf", "summary": "This paper addresses a gap in Descriptive Multimodal Emotion Recognition: the lack of cost-efficient, reliable evaluation methods for open-ended emotion descriptions. By proposing EmoPrefer—a framework that leverages Multimodal Large Language Models to decode human emotion preferences—the work makes three notable contributions: (1) EmoPrefer-Data, the first multimodal dataset for emotion preference annotation with expert consensus; (2) EmoPrefer-Bench, a benchmark to evaluate MLLMs/prompting strategies for preference prediction; and (3) empirical insights into MLLM behavior.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- High-Quality Dataset Construction. EmoPrefer-Data addresses common pitfalls in preference datasets.\n- EmoPrefer-Bench is well-designed.\n- Two complementary metrics—recognition performance (alignment with human annotations) and swap consistency (robustness to input order)—address both accuracy and reliability, a rare focus in LLM-as-judge work.", "weaknesses": "- The paper reports 1,368 video samples, but it is unclear how many final annotated pairs are retained after filtering for annotator consensus. For example, if ties (which have lower agreement, 59.23%) are excluded, the effective dataset size may be smaller—this impacts the statistical power of MLLM evaluations.\n- The paper does not specify the range of emotions in EmoPrefer-Data (e.g., is it balanced across positive/negative/neutral, or dominated by common emotions like anxiety/frustration?). A skewed emotion distribution could bias MLLM evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0oUBGHVFkz", "reviewer": "Reviewer_yfmT", "summary": "This paper introduces EmoPrefer, a benchmark designed to evaluate whether large language models can understand human emotion preferences.\nThe authors first construct a multimodal dataset, EmoPrefer-Data, by pairing two different emotion descriptions for the same video and collecting human preference annotations indicating which description better reflects the perceived emotion.\nBased on this dataset, they develop EmoPrefer-Bench, which assesses the ability of multimodal large language models to act as “judges” of emotional preference—that is, to decide which of two emotion descriptions aligns more closely with human judgment.\nThe benchmark measures both recognition accuracy and swap consistency, testing whether a model’s decision is invariant to input order.", "questions": "1. Inconsistent variable naming in the swap-consistency formula\nIn the definition of the swap-consistency metric (Equation (1)), the variables “onnormal” and “onswapped” are used inside the mathematical expression. These concatenated English words are non-standard and may confuse readers.\n\n2. Unmatched quotation marks in the prompt template\nIn Figure 17 (Appendix D.3, “Details of S4”), the word 'Tie’ is enclosed with mismatched quotation marks—one straight quote and one curly quote.\n\n3. Redundant wording in contribution statements\nThe sentence “Additionally, we introduce additional strategies ...” repeats the word “additional”.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Novelty and scope\n Proposes a first benchmark targeting emotion preference with MLLMs-as-judges, plus a new multimodal preference dataset and a clear pipeline (data, bench, metrics).\n\n2. Dataset curation\nUses overlapping videos from two descriptive MER sources, applies annotator screening and consensus filtering, and reports inter-annotator agreement; tie handling is explicitly discussed.\n\n3. Evaluation design\nDefines two complementary metrics—recognition performance (2-class/3-class WAF/ACC) and swap consistency—to capture both accuracy and order-invariance.", "weaknesses": "1. Potential LLM–LLM evaluation loop\nThe compared descriptions originate from model-generated datasets; humans provide pairwise preferences rather than authoring gold descriptions, risking measurement of model self-consistency more than human preference.\n\n2. Limited human supervision\nSmall, filtered annotator pool and consensus-only retention reduce diversity; the reported human upper bound is modest and ties lower agreement further, constraining ceiling estimates.\n\n3. Cost and scalability\nNo systematic reporting of inference cost across S1–S4, nor analysis of error accumulation vs. chain length or budgeted evaluation settings.\n\n4. Missing ablations\nNo systematic per-modality or prompt-sensitivity ablations (e.g., audio-off, video-off, text-only), and limited analysis of temperature/seed effects for judge stability.\n\nOverall, the paper addresses a novel and relevant topic that bridges affective computing and large model evaluation.\nIt provides a useful dataset and benchmark to analyze how well MLLMs capture subjective human emotional judgments.\nHowever, since the emotion descriptions used for annotation originate from other model-generated datasets, the study risks forming a closed LLM–LLM evaluation loop, where models are effectively judging content generated by other models rather than real human descriptions. I think the author can reconsider the data construction process. It is difficult for the community to be convinced by the purely synthetic data benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)