Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
Authors: Daiqing Wu
Dongbao Yang
Sicheng Zhao
Can Ma
Yu ZHOU
Venue: ICLR 2026 Poster (2026)
Keywords: MLLM
Visual Emotion Evaluation
Primary area: datasets and benchmarks

Abstract: Recently, Multimodal Large Language Models (MLLMs) have achieved exceptional performance across diverse tasks, continually surpassing previous expectations regarding their capabilities. Nevertheless, their proficiency in perceiving emotions from images remains debated, with studies yielding divergent results in zero-shot scenarios. We argue that this inconsistency stems partly from constraints in existing evaluation methods, including the oversight of plausible responses, limited emotional taxonomies, neglect of contextual factors, and labor-intensive annotations. To facilitate customized visual emotion evaluation for MLLMs, we propose an Emotion Statement Judgment task that overcomes these constraints. Complementing this task, we devise an automated pipeline that efficiently constructs emotion-centric statements with minimal human effort. Through systematically evaluating prevailing MLLMs, our study showcases their stronger performance in emotion interpretation and context-based emotion judgment, while revealing relative limitations in comprehending perception subjectivity. When compared to humans, even top-performing MLLMs like GPT4o demonstrate remarkable performance gaps, underscoring key areas for future improvement. By developing a fundamental evaluation framework and conducting a comprehensive MLLM assessment, we hope this work contributes to advancing emotional intelligence in MLLMs. Project page: https://github.com/wdqqdw/MVEI.

Published as a conference paper at ICLR 2026
CUSTOMIZING VISUAL EMOTION EVALUATION FOR MLLMS: AN OPEN-VOC,,,LARY, MULTIFACETED, AND SCALABLE APPROACH
Daiqing Wul.4
Dongbao Yang!. 1.s
Sicheng Zhao? Can Ma
Yu Zhou?
Institute of Information Engineering. Chinese Academy of Sciences
2VCIP & TMCC & DISSec, College of Computer Science, Nankai University
*Department Psychological and Cognitive Sciences, Tsinghua University 'University of Chinese Academy of Sciences
vudaigingéiie en yangdongbac@iie..e.. ac. en macandiie.ac.om
ABSTRACT
Recently, Multimodal Large Language Models (MLLMs) have achieved excep- tional performance across diverse tasks, continually surpassing previous expec- tations regarding their capabilities. Nevertheless, their proficiency in perceiving emotions from images remains debated, with studies yielding divergent results zero-shot scenarios. We argue that this inconsistency stems partly from constraints in existing evaluation methods, including the oversight of plausible responses, lim- ited emotional taxonomies, neglect of contextual factors, and labor-intensive anno- tations. To facilitate customized visual emotion evaluation for MLLMs, we propose an Emotion Statement Judgment task that overcomes these constraints. Comple- menting this task, we devise automated pipeline that efficiently constructs emotion-centric statements with minimal human effort. Through systematically evaluating prevailing MLLMs, our study showcases their stronger performance in emotion interpretation and context-based emotion judgment, while revealing relative limitations in comprehending perception subjectivity. When compared to humans, even top-performing MI.LMs like GPT4o demonstrate remarkable performance gaps, underscoring key areas for future improvement. By developing a fundamental evaluation framework and conducting a comprehensive MLLM assessment, we hope this work contributes to advancing emotional intelligence MLLMs. Project page:
INTRODUCTION
Perceiving emotional signals from visual stimuli is fundamental for humans to refine decision- decision-making making and build effective communication (Schutte al., 2001), and modeling this capability has led to the emergence of Affective Image Content Analysis (AICA) as a key research direction (Zhao et al., 2022). Recently, the advent of Multimodal Large Language Models (MLLMs) has revolutionized image understanding tasks (Yang et al., 2023c). However, their competence in AICA remains contested. Divergent findings underscore a paradox: while some studies (Xie al., 2024; Wu al., 2025) demonstrate MLLMs' limited emotion recognition performance, others successfully employ them as emotion annotators for data augmentation (Lian et al., 2025; Cheng al., 2024). We attribute this discrepancy to the incompatibility of conventional emotion evaluation approaches with MLLMss.
Specifically, current evaluation approaches can broadly categorized into emotion classification and emotion interpretation, as illustrated in Figure (a.b). In emotion classification, models are required to assign the affective state of an input image to a predefined set of emotion categories. Most benchmarks (You et al., 2016; Yang et al., 2023a) provide single label per image, while few (Kosti et al., 2017; Wei et al., 2020) incorporate multiple labels. contrast, emotion interpretation focuses understanding the underlying causes of emotions images. It encompasses two primary sub-tasks: explaining the causes of emotional states (Achlioptas et al., 2021; 2023) and identifying salient visual elements that contribute to emotional responses (Lin al., 2025).
When applied to MLLMs, these methods reveal four primary limitations. Firstly, their adoption of fixed trouhd-truth answers for open-ended questions imposes structural constraints that exclude

---

Published as a conference paper at ICLR 2026
(1),
Emotion Classification
(b),
Emotion Interpretation
Trok: What amotionly miget the image eacde?
Tisk: What reight cosse apecfii onotionalateiel?
Single lobbl Claxiilcation
Label: Contessecer
Emotion Explanation
Queation: Why might the inage exetee contestimntt winwer??
Label: Amuremen GPTic Prediction: Conterement
Ground Truth Explanation:
it don and gndino but the fousi has of choractess
GPT4p Exploration:
Human Bualustion: Arosonable
proceffl and rural serting
allipe countrysile
Ewaluation: Ressorable Bual
Mutti-ibel Clasufication
Emotional Trigger identilcction
Label Disooceccios
Question: die casse of the man itage Sehig sus?
Confeierr Disoorecction Diguicowont
Prodiction:
Ground Truth Answens
Stariiin noet to acooker possons
Cyyin rndoors af night
2.A
GPT4 Answer:
Roanturted mount, and sigatly clored eyes.
cod or harsh crsiruement
Human Exalustiore Acosomable
Selig confooool by anooler.
1. might expenacee on emotiinal dgficity
Morkic Bellustion Ircanect
Human Evaluation: Ressorable Metric Evalvation: Insarrett
Statement Sentiment Palority
(c).
Statementt Scane Contect
Label:
vinwing the image obseners despite endiedaall
Anolgtter bamiing
Synilly fror, daager,
crabe organcy
Interpretation Label: Comrect
smage might cvake braeery bscoure it dipiets /vvllat ner engoyed in exting boking fhe The fustyrew's protectie poar and the use of (re crasguaseer sugpest hightsscloff skV and cowrgge require tu handle such a siluttionn
Subjecthulty
Labet: Inromnc
Upon slmwing Image, Baarenn- make fonf
Emotion Statement Judgement (ES/)
to Reel fow compored to orpency
Rasic the atatement correct reparding the image?
Figure I: Comparison between current emotion evaluation approaches and the proposed ESJ.
other plausible responses. Emotion perception is inherently subjective (Zhao et al., 2016), as the same image may evoke divergent reactions across individuals, and emotional states permit varied interpretations. As demonstrated in Figure I, responses generated by GPT4o (Hurst et al., 2024) that seem reasonable to humans are judged as inaccurate under rigid evaluation metrics. Secondly, they are mostly constructed upon emotion theories with limited emotional taxonomies. Popular emotion classification and interpretation benchmarks, such as FI (You et al., 2016) and Artemis (Achlioptas et al., 2021), comprise only eight emotion categories. Such taxonomic gramularity fails to capture fine-grained affective variations between images. Thirdly, they focus solely on intrinsic image attributes while overlooking critical contextual dimensions. As recognized in established psychological literature, emotion perception can be influenced by extravisual factors (Barrett et al., 2011), I), including the scene context in which the image is set (Wieser & Brosch, 2012), as well as the viewer's identity and personality (Hamann & Canli, 2004). Fourthly, they predominantly rely on majority voting mechanisms to ensure label reliability in crowdsourced annotations (Li et al., 2017), which is labor-intensive, particularly for fine-grained annotation tasks. EMOTIC (Kosti et al., 2017), for instance, requires coordination with 23,788 annotators. This operational burden severely constrains dataset scalability in magnitude and generalization capacity across image domains.
To facilitate customized visual emotion evaluations for MLLMs, we propose compnnnt solution for these limitations: the Emotion Statement Judgment (ESJ) task, complemented by the INSETS (INtelligent ViSual Emotion Tagger and Statement Constructor) pipeline for efficient annotation. In designing the framework, we emphasize evaluation precision over complexity to establish reliable offline standard. With this aim, ESJ reformulates visual emotion evaluation by requiring MLLMs to validate emotion-centiog statements for a given image. It effectively mitigates ambiguity in open-ended questions while being highly extensible for evaluation depth and diversity. In parallel, INSETS annotates images with multiple vocabuocyyy emotion labels, significantly refining the emotional taxonomies. These labels are then utilized to construct multifaceted emotion- centric statements, covering both intrinsic image attributes and extrinsic contextual factors. Crucially, only minimal human intervention required, ensuring a high scalability of the approach.
Leveraging INSETS, we automatically construct INSETS-4613 a large-scale annotated ESJ corpus. Building on it, we curate MVEI benchmark (Multifaceted evaluation of Visual Emotion Intelligence) through careful human refinement. MVEI comprises 3,086 unique image-statement pairs designed to enable comprehensive evaluation of MI.LMs. Grounded established theories of affective cognition, it covers four complementary dimensions: sentiment polarity (Russell, 1980), emotion interpretation (Ekman& & Friesen, 1971), scene context (Barrett et al., 2011), and perception subjectivity (Hamann & Canli, 2004). Systematic evaluation reveals that recent MLLMs exhibit considerable proficiency but still lag behind humans, particularly in discerning emotional polarity and interpreting perception subjectivity. Further explorations indicate that the former likely be improved through targeted adaptation, whereas the latter is more tied to the models' inherent properties, highlighting potential directions for future research. In summary, the contributions of this paper are threefold:
2

---

Published as a conference paper at ICLR 2026
We identify four major limitations in existing visual emotion evaluations for MLLMs and introduce the customized Emotion Statement Judgement task to address them.
Complementing the ESJ task, we further develop the INSETS pipeline, offering scalable approach to annotating images with open- veea-vocabllary emotion labels and constructing multifaceted emotion-centric statements with minimal human effort.
Building on INSETS annotations with human refinement, we curate the MVEI benchmark, followed by systematic evaluation of recent MIL.Ms. Comprehensive results provide insights and foster further advancements in visual emotional intelligence.
2
RELATED WORKS
2.1 AICA BENCHIMARKS
Psychological researchers conceptualize emotion representation through two principal frameworks: the Categorical Emotion Space (CES), which discretizes affective states into predefined taxonomies, and the Dimensional Emotion Space (DES), which maps emotions onto continuous coordinations. For simplicity and better interpretability, most benchmarks adopt emotion classification evaluations based on discrete CES emotion taxonomies. This category encompasses both early small-scale benchmarks, such as IAPSa (Mikels et al., 2005) and Abstract (Machajdik & Hanbury, 2010), as well as later larger-scale benchmarks like FI (You et al., 2016) and WebEmo (Panda et al., 2018). Over time, benchmarks with enriched metadata have also been developed. Notable examples include EMOTIC (Kosti et al., 2017), which integrates multiple emotion categories. VAD values (Schlosberg, 1954), and human-related bounding boxes, and EmoSet (Yang et al., 2023a), which employs describable emotion attributes that cover different levels of visual information.
Some other benchmarks adopt emotion interpretation evaluations by extending CES-based taxonomics with additional emotional explanations, such Artemis (Achlioptas et al., 2021) and Affection (Achlioptas et al., 2023). ElBench (Lin et al., 2025) diverges slightly, shifting focus on identifying visual emotional triggers. Based these benchmarks, numerous expert models (Jia & Yang. 2022; Feng et al., 2023; Wu et al., 2024) have been developed, demonstrating strong performance under the fine-tuning and testing paradigm. In contrast, MLLMS are commonly pre-trained on web-scale data, without explicitly aligning with benchmark-specific knowledge. This discrepancy introduces multiple constraints when applying conventional benchmarks MLLMs, necessitating customized visual emotion evaluation approaches that account for their generalized knowledge structures.
2.2 EVALUATION OF MLLMS
Recent years have witnessed growing academic and industrial interest in MLLMs. Unlike specialized models, MLLMs demonstrate versatile competence across diverse tasks (Li & Lu, 2024), fueling expectations for their trajectory toward Artificial General Intelligence (Maruyama, 2020). To evaluate MLLMM, various benchmarks have been established, covering perception (Li al., 2023b; Liu et al., 2023), reasoning (Nie et al., 2024; Zhang et al., 2019), ethics (Qian al., 2024b; Guan et al., 2024), and specialized domains (Chen et al., 2024a; Qian et al., 2024a). Yet emotional intelligence remains conspicuously underexplored, particularly in the visual modality. In existing efforts. MM Bencc (Yang et al., 2023b) simply aggregates mainstream image-text benchmarks; MABA-Bennhl (Li et al., 2024) focuses primarily
facial expressions and actions; EmoBBnnc--y M (Hu al., 2025) and #Emmo-Bench (Gao et al., 2025), while largely extending task coverage, still insufficiently handle the ambiguity inherent in open-ended questions. To fill this gap, propose the E.S. task and the MVEI benchmark, aiming to customize and advance visual emotion evaluation of MLLMs.
3 EMOTION STATEMENT JUDGEMENT
ESJ aims to evaluate the competence of MLLMs in perceiving emotions from visual content. In each trial, MLLMs receive an image and a paired emotion-centric statement. MLLMs are then tasked to judge whether the statement is accurate in relation to the image. To ensure both breadth and depth in evaluation, we draw inspiration from cognitive research (Shuman et al., 2017) and AICA surveys (Zhao et al., 2023), and design emotion-centric statements from four complementary
3

---

Published as a conference paper at ICLR 2026
(a).
Require Costs
Require API Costs and Human Efforts
Dataset-wise
[b).
Filter Unquaiified Words
(c). Attach Open vocabulory Emotions to Parrott's Herrrchicl Emotion Model
MITE
Row Speittre Diwer
Dertary Grageite)
Fall
ftr
postalgia happiness
oy
Imsge Dataset
(d).
Parrott- based Open-vocauuaary Hierarchicall Model (POM)
Open-vocabulan Emotions
Hegpliness
Dapitort
Local MLLLMs inferences
Local MLLMss inforences
Sample-wise
Extract Open vocaluibrry Emotions
Vote for Consensus Emotions based on POM
Happ 2i14
Usve
prrapp)
Image Sample
Seconders Seyel
Figure 2: Illustration of the open- vecebboccyylary emotion tagging stage. We first extract all potential open- vocabulary emotions from the image dataset (a) and then attach these emotions to a well-established emotion model (b,c). Through this model (d), we identify and select open- npecbbocabulary emotions consistently recognized by multiple MILMS as the labels of each image (e).
dimensions: 1). Sentiment Polarity Statements require MLLMs to decide sentiment polarities without any additional clues, aiming to assess MLLMs' proficiency in directly identifying the basic emotional tone. 2). Emotion Interpretation Statements ask MLLMs to verify the consistency between affective explanations and corresponding emotional states. They mcasure MLLMs' affective reasoning capability given specific emotional triggers. 3). Scene Context Statements probe MI.LMs' comprehension of the dynamic interplay between the potential scene context where the image takes place, and image-evoked emotional responses. 4). Perception Subjectivity Statements task MILMs to predict the personalized emotional responses under assumptions of specific viewer identities, examining whether MLLMs can recognize how subjectivity shapes emotional perceptions.
Collectively, these dimensions establish holistic visual emotion evaluation framework for MLLMs. They cover both intrinsic image attributes emphasized in existing benchmarks and underexplored contextual factors critical for human emotional perception (Stemmler & Wacker, 2010).
ANNOTATION PIPELINE: INSETS
Complementing the ESJ task, we design an automated pipeline for constructing emotion-centric statements, termed INSETS (INtelligent ViSual Emotion Tagger and Statement Constructor). It operates through two stages: open-vocabulary emotion tagging and emotion statement construction, both of which build upon the well-established Parrott's Hierarchical emotion model (Parrott, 2001). This tree-structured taxonomy organizes emotions into primary, 25 secondary, and 113 tertiary categories (Appendix D), where the primary level includes three positive emotions (joy, love, surprise) and three negative emotions (anger, fear, sadness). Secondary emotions elaborate these categories with greater diversity, while tertiary emotions refine them into more specific affective states.
4.1 OPEN-VOCABULARY EMOTION TAGGING
At this stage, INSETS aims to assign vecabuacyyl emotion labels for images, laying a solid foundation for constructing meaningful emotion-centric statements, with its procedure depicted in Figure 2. According to (Cheng et al., 2024), MLLMs demonstrate promising capabilities in generating emotional descriptions from visual content and extracting underlying emotions from these
4

---

Published as a conference paper at ICLR 2026
(a). Prototype Statement Generation
Local Inferennce
[Emotion]
[MLLM]
Gencrtt Statements
NX W S3nnss song S000)
Salisbuity
Prautratie
Prototype Statements
[b].
Sentiment Polanity Statament Corstraction
(c).
Emotion Interpretation Statament Corstruction
OLrer Ihage
Insereet Scleect Eentraation
Soredt Matcewst Sestractien
1421
Mall bollbbieee
Visualy pannee Dasie
Statrnnne A4 Faushan Hinpertttion
Doody
be but Wh
Ofle of Postrre Inape
helthee
Coler Fulyy Angenter
legefii Lebels
(e).
Perception Subjectivity Statement Construction
(d).
Scene Contact Statement Comstruction
Exntraatin Paas
Repplir (83)
Incernet Statrmntt Constucsion
Cerrec Sttarent Contraccion
Upen Vivirgtte maas
Trepltr a1l)
Eaje
Nertens foo
Prelect compart
the Cress Fowwer
for atoc)
Slatomentr
kaige
Johne
Poreptionn Sah bettaaty
Correc
Iecarvel Sclewredd Contraaiin
Zeternett
Segitho Lobel
Sing pugerred Certor ALEENNTT
Tomphiie
Impalcy
SUTT VIr Comper Senterecc
Roe pelems
Figure 3: Illustration of the emotional statement construction stage. It begins with prototype statement generation (a) for each emotion label, which is distributed across multiple MLLMs. Then, based on the assigned emotion labels and the corresponding prototype statements, correct and incorrect emotion-centric statements are constructed from four dimensions: sentiment polarity (b), emotion interpretation (c), scene context (d), and perception subjectivity (e).
descriptions. However, challenges such as hallucinations (Bai et al., 2024), trustworthiness issues (Liu et 2024b), and inherent limitations in emotional perception can lead to inaccuracies in the extracted emotions. To enhance reliability, we devise an ensemble-based majority voting mechanism, aggregating outputs from multiple MLLMs to cross-validate and refine emotion label assignments.
Given an image sample, we first extract its potential open-voca emotions from multiple MLLMs. MLLMS are prompted to analyze the emotions evoked by the image (with #1 prompt in Table 7, abbreviated as "#1" in the following) and then extract emotions applicable to the image (#2) [Figure (a)]. This process is iteratively applied to all images in the dataset, aggregating potential emotions into an emotion pool. Next, we refine this pool by filtering out words unsuitable as emotion descriptors (#3), using GPT-4 (OpenAI, 2023) as the judge due to its superior linguistic emotional perception (Sabour et al., 2024) [b]. Once the filtered emotion pool is obtained, we attach the remaining emotions to Parrott's 's hierarchical emotion model [c]. GPT-4 is prompted to categorize each open-vocabulary emotion into the closest tertiary emotion in Parrott's model (#4), followed by manual refinement from a hired human expert. This process results in an extended version of Parrott's model, which we refer to as the Parrott- basett--baed Open-vocabulary Hicrarchical Model (POM) [d]. This unified framework enables multi-level tracing of affective states for each open- open-vocabulary emotion, facilitating more accurate and interpretable emotion tagging.
Subsequently, leveraging POM, the ensemble- based majority voting mechanism selects consensus open-vocabulary emotion labels for images [e]. Specifically, emotions extracted from multiple MLLMs are first mapped to secondary categories, where model voting allocates quotas. Within each category, candidate labels are ranked by frequency, and the top-ranked ones selected accordingly. This procedure enhances the reliability of annotations while retaining opea-vocabulary flexibility.
4.2 EMOTIONAL STATEMENT CONSTRUCTION
Building upon the assigned emotion labels, we construct automatically-annotiiio emotion-centtee centric statements, as illustrated in Figure 3. The pipeline initiates with prototype statement generation [a].
5

---

Published as a conference paper at ICLR 2026
Table 1: Statistics of the MLLMs employed in INSETS. For each MLLM, we report the number parameters, the average extracted emotions per image, the number selected emotion labels, and the proportion prototype statements it generates.
Table 2: Statistics of INSETS- 462k and MVEI.
INSETS-462K
Number of Images
17,716
#P
Extracted
Selected Emotion
Generated Statement
Number of Statements
462,369
MLLMs
Emotion Labels Per Imuge
(B)
Emotion
Distinct Emotion Labels
751
LLaVa-1 (Liu et al., 2024a)
7.6
8.3
2.4
9.8%
Statements Per Image
26.1
Mantis (Jiag et al., 2024)
12.6
2.9
13.1%
Average Length of Statements
39.0
mPLUG-OwL3 (Ye et al. 2024)
9.2
2.7
11.2%
MVEI
Idefics3 (I aurengon al., 2024)
10.0
2.9
Number of Images
3,086
Phi-3.5-Vision (Abdin et al., 2024)
2.8
11.7%
Number of Statements
3,086
Qwen2-VL (Wang al., 2024)
8.3
8.8
2.7
10.9%
Emotion Labels Per Image
5.2
Llama-3.2 2-iiin (Dubey et al., 2024)
10.7
7.2
2.3
9.3%
Distinct Emotion Labels
424
Molmo (Deitke et al.. 2024)
8.0
10.8
2.7
12.0%
Statements Per Image
LL2e (Chea et al., 2024b)
8.3
8.5
2.3
9,5%
Average Length of Statements
37.0
For each emotion label, we trace it back the MILM that extracts it, prompting the MILM to generate three prototype statements: 1). prototype interpretation of the emotion by inquiring about the cause of the emotion (#5); 2). prototype context that aligns with the emotion by requesting background story (#6); and 3). prototype character who would experience the emotion by questioning the possible identity of the viewer (#7). From the dataset perspective, the prototype generation distributed across multiple MLLMs, ensuring diversity in the subsequent statement construction.
Sentiment Polarity Statement Construction [b]: We classify the sentiment polarity of each image into three mutually exclusive categories according to POM: 1). Fully Positive when all labels reside the positive spectrum; 2). Fully Negative when all labels reside in the negative spectrum: 3). Mixed when positive and negative labels both exist. Next, the ground truth correctness of three predefined statements on sentiment polarity (48,9, 10) is determined accordingly.
Emotion Interpretation Statement Construction [c]: Each statement is constructed by combining a prototype interpretation with an emotional state (#11). Matched labels and prototype statements assigned correct, while mismatched ones considered incorrect. We design two disruption strategies for each image: 1). Inter-image disruption retrieves two images from the dataset-one one exhibiting visual similarity but emotional dissimilarity test whether MLLMs can comprehend the affective gap (Hanjalic, 2006), the other demonstrating emotional similarity but visual dissimilarity evaluate whether MLLMs can identify the emotional triggers images-and and substitute the current prototype interpretation using one of theirs. Visual similarity is measured by CLIP-score (Radford al., 2021), and emotional similarity is decided by tertiary emotions in POM. 2). Intra-image disruption exchanges interpretations between labels of contrasting polarity within the same image, probing whether MILMMs can establish precise causal linkages between triggers and specific emotions.
Scene Context Statement Construction [d]: Each statement is combined from a prototype context and an emotional conclusion (#12), where the construction of correct statements mirrors the previous case. For incorrect ones, adopt two strategies: 1). flip-polarity operation that replaces the label with a tertiary emotion randomly sampled from the opposite spectrum in POM, and 2). swapping prototype contexts between opposite-polarity labels within the same image.
Perception Subjectivity Statement Construction [e]: We combine a prototype character with their inclination toward one of two candidate emotions (#13) to form statement. For each character, the preferred emotion corresponds to its label, while the non-preferred emotion is obtained either from opposite-polie polarity labels within the same image via flip-polarity sampling. Correct statements adopt the canonical preference order, whereas incorrect ones are formed by reversing it.
4.3 CONSTRUCTION OF INSETS-462K AND MVEI
Given the high quality of EmoSet (Yang et al., 2023a), we select 17,716 images from as the image source for INSETS. We employ nine recent popular MLLMs with impressive performance (Contributors, 2023) for open- ngen-vocabllary emotion extraction and prototype statement generation. Their detailed participation is reported in Table 1. Observably, the final assigned emotion labels and prototype statements are evenly distributed across the MLLMs, ensuring diversity in the constructed data. In addition, psychology postgraduate with formal training is hired to refine the attachment of

---

Published as a conference paper at ICLR 2026
Perosgtinn Subjedtaity
(c)
Nenegh
(b)
Sentiment Polarity
Amtidpation
12.0%
Angry 1.7%
Scere Contect
17.3%
Emrtion Interpesation
6.
Excitement
Tear
Emotlon contric Statement
Label: Incorrect
Rae
Sadrass 17.7%
11.2%
Longth
arangsment
Werder
5.0%
by
Auerage Length: 37.0 Words
packoggnng can aution analett due overwhelning number of fhems and cluttered space.
Contort
5.8%
6.9%
7.7%
3,0%
Figure 4: A closer gaze MVEI. Illustrations of sample (a), the distribution of emotion labels (b), and the distribution of emotion-centric statements (c).
open-vocabulary labels, which takes approximately 15 hours in total. Collectively, INSETS produces automatically annotated ESJ corpus of 462K samples, namely INSETS-4621
Based on this corpus, we sample 3,164 distinct image-statement pairs for care- ful human refinement, Five graduate stu- dents, each provided with detailed task instructions, are recruited to assess the ac- curacy of automatically assigned annota- tions. The statistics of human refinement
Table 3:
Statistics of the human refinement process. Kappa represents Fleiss' Kappa.
MVEI
Sentiment Polarity
Emotion Interpretation
Scene Context
Perception Subjectivity
Total
Annotation Agreement (%)
5/5
61.0
42.5
78.1
44.0
54.0
are presented Table where annota-
4/5
33.2
46.6
15.9
43.7
tors achieve consistently high agreement
3/5
1.3
1.3
1.3
1.4
across the four task dimensions. For each pair, the annotation is deemed correct if at least four annotators reach consensus (5/5 or 4/5), incorrect if consensus is in the opposite direction (1/5 or 0/5), and ambiguous otherwise (3/5 or 2/5).
2/5
1.0
0.7
0.4
2.4
3.6
1.9
015
1.0
5.3
2.4
4.3
3.8
Kappa
0.68
0.51
0.81
0.52
0.61
Construction Accuracy (%)
86.2
94.6
Pairs
93.4
92.0
93.4
88.0
91.5
Overall, 90.6% the automated an- notations are judged accurate-89.795 % for correct statements and 91.5% for incorrect state- ments-validating the high reliability of INSETS. After retaining correct labels, rectifying errors, and discarding ambiguous cases, we derive the final MVEI benchmark (Multifaceted evaluation of Visual Emotion Intelligence). MVEI comprises 3,086 samples with over 400 distinct emotion labels, with detailed statistics provided Table Benefiting from the large-scale automatic construction of INSETS-462K, MVEI is far more labor-efficient than prior emotion evaluation benchmarks, requiring only about 100 person-houry for the subsequent refinement.
5 ANALYSIS AND EVALUATION
5.1 DETAILS OF MVEI
To further characterize MVEI, we provide its fine-grained statistics in Figure 4. A sample is shown in Figure 4 (a), which includes five emotion labels and an emotion statement. Figure 4 (b) illustrates the distribution of popular emotion labels, where the most frequent labels include Joy, Excitement, Nostalgia, Happiness, and Contentment. When mapped to the primary emotions in POM, Joy dominates (4,.3%) followed by Sadness (17.7%). Love (17.2%). Fear (13.4%), Surprise (7.7%), and Anger (3.7%). This distribution reflects broad coverage of affective states. Finally, Figure (c) presents statistics of the statements, showing a natural length distribution and balanced spread across the four evaluation dimensions as well as correct/incorrec labels.
5.2 EVALUATION PREPARATIONS
To evaluate MLLMs with ESJ, each model is given image-statenen statement pair and prompted to judge its correctness. The prompt is formulated as: "Based on the provided image and emotional statement, please determine whether the statement aligns with the content of the image. If it does, respond with
7

---

Published as a conference paper at ICLR 2026
Table 4: Evaluation of popular MLLMs MVEL. For fair comparison, we separate the MLLMs involved constructing INSETS-462K (the upper part) and the others (the lower part). The highest values in each section are marked in bold, underline, and wavy underline. respectively.
Accuracy (%)
Positive
Give-up Ratio
MLLMs
#Param
Sentiment Polarity
Emotion Interpretation
Scene Context
Perception Subjectivity
Total
1.6 (Lio al. 2024a)
7.6B
49.7
Mantis (Jiag al., 2024)
8.5B
61.2
65.9
67.2
61.2
64.4
84.4
0,1
mPLUG-Ow13 (Ye et al., 2024)
8.1B
73.9
79.3
81.7
75.0
78.1
67.3
Ideficx3 (Laurengon al., 2024)
8.5B
75.4
78.6
75.5
62.6
73.4
49.5
0,2
Phi-3.5-Vissoo (Abdin al., 2024)
4.1B
74.7
72.5
$2.6
74.8
75.9
64.1
Qwen2-VL et 2024)
8.3B
70.7
75.0
86.1
72.8
76.6
65.7
Llama-3.2 Vision (Dubey et al., 2024)
10.78
68.7
75.9
85.2
72.0
76.3
71.2
0.2
Molano (Deitke et 2024)
61.4
70.7
IntemV1.2.55 (Chen ol., 2024b)
8.3B
75.7
80.2
79.4
61.3
74.7
52.9
0.2
BI.IP2 ct. 2023c)
7.7B
$1.1
$2.8
55.4
52.5
53.2
96.8
2.5
InstrctBLIPP
al., 2023)
7,9B
29.8
40.5
33.9
37.8
36.8
43.8
37.5
Otter (Li ct al., 2023a)
8.2B
32.6
21.4
32.1
27.2
27.0
9.9
52.1
DeepSeck-VL. (1.u et 2024)
7.3B
68.7
70.8
81.1
73.2
73.7
73.1
Paligemma (Beyer et al., 2024)
2.91
50.6
46.3
49.3
45.7
47.4
49.4
5.5
MinCPMM (Yao et al., 2024)
20.4
70.5
Qwen2.5 VL. (Team, 2025)
8.3B
63.2
81.5
83.9
66.3
75.9
45.9
GPT4o-mini (Hurst et al., 2024)
62.5
80.0
78.9
71.8
75.4
49.5
GPT40 (Hurst 2024)
72.5
84.3
81.6
69.2
78.3
65.0
1.6
Correct. If it does not, respond with Incorrect." Each image-statement pair is queried three times per model, and the most frequent response is selected as the final decision. Accuracy serves as the primary evaluation metric. As identified in prior work (Li al., 2023d), some MLLMs may exhibit a strong bias toward either positive or negative responses, which may compromise accuracy- bassd evaluation validity. To address this, introduce the proportion of "Correct" among all responses; Give-up Ratio measures the proportion of cases where the MILM fails provide either judgment.
diagnostic metrics: Positive Ratio calculates
We evaluate a wide range of MLLMs on MVEI, including both open-source and closed-source ones. Besides the MLLMs employed in constructing INSETS-462K, we also incorporate: BLIP-2 (Li et al., 2023c), InstructBLIP (Dai et al., 2023), Otter (Li et al., 2023a), Deepseck- VL (Lu et al., 2024), Paligemma (Beyer al., 2024), MiniCPM (Yao al., 2024), Qwen2.5-VL VL (Team, 2025), GPT4o-mini, and GPT4o (Hurst et al., 2024).
5.3 RESULTS AND FINDINGS
Comparison of MLLMs (Table 4): Overall, recent MLLMs substantially outperform earlier ones, which often suffer from severe response biases or instruction-follooing failures. These results suggest that advancements in general visual tasks also benefit emotional perception. Among state- of-the-art MLLMs, their capabilities vary noticeably across different task dimensions, with no single MILMM achieving top performance in all categories. For instance, InternV1.2.5 and GPT4o excel at recognizing basic emotional tones and performing affective reasoning, yet exhibit relative shortcomings contextual and subjective emotion prediction. These results highlight the multifaceted challenges of visual emotion understanding and the need for continued targeted development.
Comparison with Human Performance (Table We evaluate 25 human participants alongside leading MLLMs on a 300-sample subset of MVEI. The results show that humans achieve an average overall accuracy of 91.6%, substantially surpassing both open-source and proprictary MLLMs. The performance gap is most evident in determining sentiment polarity and understanding perception subjectivity. Given that MILLMS perform comparatively well emotion interpretation, their limi- tations in polarity appear to stem from an overreliance on provided affective cues and difficulty in distinguishing boundaries between sentiment categories. the case of perception subjectivity, the gap seems more fundamental, reflecting MILMs' limited proficiency to capture individual differences in emotional perception. Collectively, these findings suggest that current MLLMs may not yet be sufficiently competent for LLM-as-a-judhh a-jdge applications in affective perception tasks. They underscore the need for more rigorous benchmarking foundational capabilities, while also pointing considerable potential for future advancement.
8

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation of humans on a 300-sample subset of MVEI. To ensure faimess, the results of partial leading MLLMs are also reported, adhering to the same partition as Table 4.
Accuracy (%)
Positive Ratio
Give-up Ratio
MILMs
#Param
Sentiment Polarity
Emotion Interpretation
Scene Context
Perception Subjectivity
Total
mPLUG- (Ye et al., 2024)
8.1B
74.6
80.4
82.9
77.2
79.5
67.3
Phi-3,5-Vision (Abdin et al., 2024)
4.1B
75.4
72.9
83,9
23.3
76.1
64.0
1ntemV1.2.5 (Chen et al,, 2024b)
8.1B
77.2
79,5
79.3
63.2
75.1
52.1
DeopSock-VL VL (Lu et al., 2024)
7.3B
20.2
70.5
80.2
73.7
73.7
73.5
0
MiniCPM (Yao
2024)
8.7B
20.2
78.9
82.4
72.4
27.1
65.4
Qwen2.5-VL (Team, 2025)
8.3B
64.0
81.5
83,3
68.0
76.4
47.4
4PTio-mini (OpenAl, 2023)
64.0
79.2
77.5
21.3
74.9
49.8
GPT4o (OpenAl, 2023)
73.7
84.5
81.2
71.1
79.0
64.6
0.6
Human Average
92.3
90.1
95.3
89.6
91.6
53.4
0
Human Best
97.4
95.8
98.7
94.7
95.2
Table 6: Evaluation of lightweight MLLMs adaptation techniques on MVEI.
Accuracy (5)
Qw (8.3B)
#Shot
Sentiment Polarity
Emotion Interpretation
Scene Context
Perception Subjectivity
Total
Direct Inference
63.2 67.4 (+4.2)
81.5 81.5 (+0.0)
83.9 84.6 (+0.7)
66.3
75.9
Chain-of Thought Reasoning
(7.0
76.6 (+0.8)
In Co-nteext Learning: Random Retrieval
2
66.3 (+3.1)
81 (+0.1)
84.8 (+0.9)
66.5 (+0.2)
76.9 (+1.0)
In Context Learning: Random Retrieval
4
68.8 (+5.6)
81.7 (+0.2)
(+1.1)
66.7 (+0.4)
77.1 (+1.2)
In-Context Learning: Random Retrieval
8
70.1 (+6.9)
81.7 (40.2)
84.9 (+1.0)
67.0 (+0.7)
77.3 (+1.4)
2022) Full Parameter Fine-Tuning (Freeze Vision) GRPO (Shao al., 2024)
LoRA Fine: Tuning
78.6 (+15.4)
84.7 (+3.2)
86.3 (+2.4)
70.3 (+4.0)
80.7 (+4.8)
84.3 (+21.I)
84.8 (43.3)
87.0 (43.1)
71.1 (+4.8)
81.9 (+6.0)
83.2 (+20.0)
82.5 (+1.0)
86.5 (+2.6)
71.1 (+4.8)
80.7 (+4.8)
Influence of MLLM adaptations (Table 6): To delve deeper into MLLMs' emotional intelligence and shed light on the potential influence of model adaptation, we adapt Qwen2.5-VL. using several popular techniques and evaluate their impact on MVEI. For in-context learning, demonstrations are randomly retrieved from the corresponding task dimensions of INSETS-462K, with overlapping MVEI samples excluded. For parameter- efficcient fine-tuning, we apply LoRA (Hu et al., 2022) on a 10k-sample subset of INSETS-4622 (excluding MVEI overlaps), using a learning rate of lc-5 and LoRA rank of 16. The full fall-parameter lne-tuning is conducted on the same subset with an identical learning rate, during which the vision encoder is frozen. Finally, for GRPO (Shao et al., 2024), we train on the same subset with a learning rate of le-6 and perform 4 rollouts per query.
As shown in Table 6, all applied techniques consistently improve MLLM performance, demonstrating the benefits of both in-context learning and task-specific fine-tuning. The most pronounced gains occur in sentiment polarity, indicating that MILMs possess the capability to capture overall emotional tone. Their previous deficiency is likely due to confusion between positive, negative, and mixed categories, and it can be effectively alleviated through few-shot demonstrations or lightweight fine-tuning. By contrast, perception subjectivity shows only modest improvement and remains the weakest dimension, reflecting more fundamental challenge that may require subjectivity-oriente pre-training objectives or specialized datasets. While targeted adaptation on INSETS-462K provides clear benefits, we view this work primarily as a foundational benchmark for advancing emotional intelligence in more general general-purpose purpose MLLMs. Rather than treating ESJ as a direct optimization target, we advocate its use as an evaluation metric and feedback signal to guide broader model development.
6 CONCLUSION
In this paper, we introduce the Emotion Statement Judgment task and the INSETS pipeline, which jointly address the incompatibility of conventional emotion evaluation approaches with MLLMs. Building these components, we construct the MVEI benchmark and the large-scale INSETS- 462K corpus in a labor-efficient manner, aiming to advance open- vocabblaa,,,,,, multifaceted, and scalable visual emotion evaluation in MLLMs. Grounded in psychological theory, MVEI evaluates four complementary dimensions of affective cognition: sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Comprehensive experiments on MVEI reveal that, while

---

Published as a conference paper at ICLR 2026
current MLLMs demonstrate certain competence in interpreting basic emotions, contextual cues, and the associations between triggers and affective states, they still fall substantially short of human performance. In particular, their limitations most evident in handling perception subjectivity, which remains a fundamental challenge even after targeted model adaptations. Taken together, this work establishes foundation for advancing the study of emotional intelligence in MILMs, aiming to foster future research in both MLLM development and AICA.
7 ACKNOWLEDGEMENT
This work is supported by the National Natural Science Foundation of China (Grant No. 62406318 62376266 & 62571294 & 62441614) and the Beijing Natural Science Foundation (L252009).
8 ETHICS STATEMENT
This study evaluates MLLMs in visual emotion comprehension through new task, pipeline, and benchmark. While we aim to advance research, several ethical considerations merit attention.
First, the dataset exhibits certain distributional imbalances the emotion label of images. The images used in this work originate from EmoSet, which ultimately traces back to user-generated posts on social media. Such posts naturally reflect platform-specific emotional biases, most notably, positive content typically appears more frequently than negative content (Niu et al., 2016). This characteristic carries over to our MVEI benchmark, where, as shown in Figure 4, images account for 65.2% of the data, compared to 34.8% for negative ones. While this skew partially mirrors real-world content distributions, it may influence model behavior or downstream analysis if not interpreted carefully. We therefore encourage users remain aware of these imbalances avoid biased or misleading conclusions.
Second, perception subjectivity statements may contain latent demographic biases arising from the automatically generated characters. Although extreme or inappropriate cases are filtered through human refinement, demographic attributes, such as age, gender, or cultural background, may still be unevenly reflected or stereotypically implied by the MLLMs. And since the characters are produced without explicit structural control, systematically quantifying their demographic statistics remains challenging. This limitation introduces the risk of subtle demographic skew being propagated or reinforced through the benchmark. Empirical quantification of these demographic patterns would enable finer-gre aner-grained evaluation and customization, and we regard this important direction for future development.
Third, MLLLM-generaal data may still exhibit celtural-perspectivee and aesthetie-pereeption
biases. Prior work has shown that cultural tendencies embedded in training corpora, such as language distribution and region-specifee viewpoints, can be amplified during LLM inference, and even multilingual models often fail to equitably represent diverse cultural values (Tao et al., 2024). In addition, L.LMs have been found to exhibit aesthetic preferences that may implicitly reinforce stereotypical standards (Kotek et al., 2023). Such tendencies could be potentially carried over into the constructed emotional labels and statements. Although the bias of any specific model can be alleviated through the proposed ensemble- based majority voting mechanism, these biases cannot be fully eliminated. Users should therefore interpret culturally sensitive results with caution and avoid overgeneralizing findings.
Fourth, annotation-level biases due cultural differences or personal experiences may persist. Emotion perception is inherently subjective, and these differences can shape annotation outcomes. While ESJ task formulation targets specifically for such issues, it can hardly guarantee the complete climination of these biases. Therefore, these concerns also warrant caution from users.
Although addressing these ethical concerns falls beyond the immediate scope of this study, we document them here to maintain transparency. We hope this clarifies the limitations of the benchmark and supports future efforts toward mitigation. All data used in this work are drawn from publicly available benchmarks, and no private information collected or disclosed. By acknowledging these considerations, we aim to promote responsible use of our data, mitigate potential risks, and support its positive impact on future research.
10

---

Published as a conference paper at ICLR 2026
9 REPRODUCIBILITY STATEMENT
To ensure reproducibility, the manuscript provides comprehensive documentation of the INSETS implementation, the human refinement process, and detailed statisties of both the INSETS-462K corpus and the MVEI benchmark. We release code and data on: https//gittub.com///ddd///WWWWW
REFERENCES
Marah I Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, and et al. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404.142 14219, 2024.
Panos Achlioptas, Maks Ovsjanikov, Kilichbek Haydarov, Mohamed Elhoseiny, and Leonidas J. Guibas. Artemis: Affective language for visual art. In CVPR, pp. 11569-11579, 2021.
Panos Achlioptas, Maks Ovsjanikov, Leonidas J. Guibas, and Sergey Tulyakov. Affection: Learning affective explanations for real-world visual data. CVPR, pp. 6641-6651, 2023.
Zechen Bai, Pichao Wang. Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. Hallucination of multimodal large language models: A survey. CoRR, abs/2404.1890 2024.
Lisa Feldman Barrett, Batja Mesquita, and Maria Gendron. Context emotion perception. Current Directions in Psychological Science, 20(5):286-299 2011.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, and et al. Paligemma: A versatile 3b VLM for transfer. CoRR, abs/2407.07726, 2024.
Pengcheng Chen, Jin Ye, Guoan Wang. Yanjum Li, Zhongying Deng, and et al. Gmai Ommaimmbennhh: A comprehensive multimodal evaluation benchmark towards general medical AL. In NeurlPS, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, and et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. CoRR, abs/2412.05271, 2024b.
Zebang Cheng, Zhi-Qi Cheng, Jun-Jannnn Yan He, Kai Wang, Yuxiang Lin, Zheng Lian, Xiaojiang Peng, and Alexander G. Hauptmann. Emotion-llama Multimodal emotion recognition and reasoning with instruction tuning. In NeurlPS, 2024.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: //g/ttttbubbb htw.thub.coooopooe ccom/opent compass /ppp//opppsssssseeoeo///om/ 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, and et al. Instructblip:
Towards general-purpose vision-language models with instruction tuning. In NeurIPS, 2023.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang. and al. Molmo and pixmo: Open weights and open data for state-of-the-an multimodal models. CoRR, abs/2409. 17114/169911166.... 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, and et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024.
Paul Ekman and Wallace V Friesen. Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2):144, 1971.
Tinglei Feng, Jiaxuan Liu, and Jufeng Yang. Probing sentiment-oriented pretraining inspired by human sentiment perception mechanism. In CVPR, pp. 2850-2860. 2023.
Lancheng Gao, Ziheng Jia, Yunhao Zeng, Wei Sun, Yiming Zhang, Wei Zhou, Guangtao Zhai, and Xiongkuo Min. Eemo-bench: benchmark for multi-modal large language models on image evoked emotion assessment. In ACM MM, pp. 7064-7073, 2025.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, and et al. Hallusionbench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In CVPR, pp. 44375-14385 2024.

---

Published as a conference paper at ICLR 2026
Stephan Hamann and Turhan Canli. Individual differences in emotion processing. Current opinion in neurobiology, 14(2):233-22 238, 2004.
Alan Hanjalic. Extracting moods from pictures and sounds: Towards truly personalized tv. IEEE Signal Processing Magazine, 23(2):90-100, 2006.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
He Hu, Yucheng Zhou, Lianzhong You, Hongbo Xu, Qianning Wang, Zheng Lian, Fei Richard Yu, Fei Ma, and Laizhong Cui. Emobench-m: Benchmarking emotional intelligence for multimodal large language models. arXiv preprint arXiv:2502.04424. 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Guoli Jia and Jufeng Yang. S2.ver: Semi-supervised visual emotion recognition. In ECCV, volume 13697 of Lecture Notes in Computer Science, pp. 493-509, 2022.
Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. MANTIS: interleaved multi-image instruction tuning. CoRR, abs/2405.01483, 2024.
Ronak Kosti, José M. Álvarez, Adrià Recasens, and Ágata Lapedriza. EMOTIC: emotions in context dataset. In CVPR Workshops, pp. 2309-2317, 2017.
Hadas Kotek, Rikker Dockum, and David Sun. Gender bias and stereotypes in large language models. In Proceedings of the ACM Collective Intelligence Conference, pp. 12-24, 2023.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better understand- ing vision-language models: Insights and future directions. CoRR, abs/2408. 12n022408.1663,, 2024.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. CoRR, abs/2305.03726, 2023a.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench. Bench- marking multimodal llms with generative comprehension. CoRR, abs/2307.16125, 2023b.
Jian Li and Weiheng Lu. A survey on benchmarks of multimodal large language models. CoRR, 0bb/2208.08662, 2024.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742, 2023c.
Shan Li, Weihong Deng, and Junping Du. Reliable crowdsourcing and deep locality- learning for expression recognition in the wild. In CVPR, pp. 2584-2593, 2017.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision- Ssiounlanguage models. In EMNLP, pp. 292-305, 2023d.
Yifan Li, Anh Dao, Wentao Bao, Zhen Tan, Tianlong Chen, Huan Liu, and Yu Kong. Facial affective behavior analysis with instruction tuning. In ECCV, volume 15076 of Lecture Notes Computer Science, pp. 165-186, 2024.
Zheng Lian, Haoyu Chen, Lan Chen, Haiyang Sun, Licai Sun, Yong Ren, Zebang Cheng, Bin Liu, Rui Liu, Xiaojiang Peng, et al. Affectgpt: A new dataset, model, and benchmark for emotion understanding with multimodal large language models. In ICML, 2025.
Yuxiang Lin, Jingdong Sun, Zhi-Qu Cheng, Jue Wang. Haomin Liang, Zebang Cheng, Yifei Dong, Jun Yan He, Xiaojiang Peng, and Xian-Sheer Hua. Why we feel: Breaking boundaries in emotional reasoning with multimodal large language models. In CVPR Workshops, pp. 5205-5215, June 2025.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NewrlPS, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee, Improved baselines with visual instruction tuning. In CVPR, pp. (1286-26296, 2024a.
Xin Liu, Yichen Zhu, Jindong Gu, Yunshi Lan, Chao Yang, and Yu Qiao. Mm-safetybench:: A benchmark for safety evaluation of multimodal large language models. In ECCV, volume 15114 of Lecture Notes Computer Science, pp. 386-403, 2024b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, and et al. Deepseek-vl: Towards real-world vision-language understanding. CoRR, abs/2403.05525, 2024.
Jana Machajdik and Allan Hanbury. Affective image classification using features inspired by psychology and art theory. In MM, pp. 83-92, 2010.
Yoshihiro Maruyama. The conditions of artificial general intelligence: Logie, autonomy, resilience, integrity, morality, emotion, embodiment, and embeddedness. In AGI. volume 12177 of Lecture Notes in Computer Science, pp. 242-251, 2020.
Joseph A Mikels, Barbara Fredrickson, Gregory R Larkin, Casey M Lindberg, Sam Maglio, and Patricia A Reuter-Lorenz. Emotional category data images from the international affective picture system. Behavior Research Methods, 37:626-630, 2005.
Jiahao Nie, Gongjie Zhang, Wenbin An, Yap-Peng Tan, Alex C. Kot, and Shijian Lu. Mmrel: A relation understanding dataset and benchmark in the MLLM era. CoRR, abs/2406.09121/ 2024.
Teng NIu, Shiai Zhu, Lei Pang, and Abdulmotaleb El-Saddik. Sentiment analysis on multi-view social data. In MMM, volume 9517 of Lecture Notes in Computer Science, pp. 15-27, 2016.
OpenAl. GPT-4 technical report. CoRR, abs/2303.08774, 2023.
Rameswar Panda, Jianming Zhang, Haoxiang Li, Joon- Joon-Young Lee, Xin Lu, and Amit Roy- Chowdhury. Contemplating visual emotions: Understanding and overcoming dataset bias. In ECCV, volume 11206, pp. 594-612, 2018.
W Gerrod Parrott. Emotions in Social Psychology: Essential Readings. Psychology Press, 2001.
Tianwen Qian, Jingjing Chen, Linhai Zhuo, Yang Jiao, and Yu-Gang Jiang. Nuscenes-qar A multi- modal visual question answering benchmark for autonomous driving scenario. In AAAI, pp. 4542-4550, 2024a.
Yusu Qian, Haotian Zhang, Yinfei Yang, and Zhe Gan. How easy is it fool your multimodal llms? an empirical analysis on deceptive prompts. CoRR, 1as/22402.13200, 2024b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, and et. al. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp.
2021.
James A Russell. A cireumplex model of affect. Journal of Personality and Social Psychology, 39(6): 1161, 1980.
Sahand Sabour, Siyang Liu, Zheyuan Zhang, June M. Liu, Jinfeng Zhou, and et al. Emobench: Evaluating the emotional intelligence of large language models. In ACL, pp. 5986-696669 2024.
Harold Schlosberg. Three dimensions of emotion. Psychological review, 61)))811 1954.
Nicola S Schutte, John M Malouff, Chad Bobik, Tracie D Coston, Cyndy Greeson, Christina Jedlicka, Emily Rhodes, and Greta Wendorf. Emotional intelligence and interpersonal relations. The Journal of Social Psychology, 141 141(4):523-366 2001.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits mathemat- ical reasoning in open language models. arXiv preprint 240.0030000003000, 2024.
13

---

Published as a conference paper at ICLR 2026
Vera Shuman, Elizabeth Clark-Polner, Ben Meuleman, David Sander, and Klaus R Scherer. Emotion perception from a componential perspective. Cognition and Emotion, 31(():47-56, 2017.
Gerhard Stemmler and Jan Wacker. Personality, emotion, and individual differences in physiological responses. Biological psychology. 84(3):541-551, 2010.
Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizileee. Cultural bias and cultural alignment of large language models. PNAS nexus, 39))p 2024.
Qwen Team. Qwen2.5-vl. https: /ww.lm/wwmmmmmmmmmmmmm.mm...mm. github. ht/blog///wnt... 5-v1/, 2025.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, and et al. Qwen2-v1: Enhancing vision-language model's perception of the world at any resolution. CoRR, 12b//249..1291, 2024.
Zijun Wei, Jiamming Zhang, Zhe Lin, Joon- Young Lee, Niranjan Balasubramanian, Minh Hoai, and Dimitris Samaras. Learning visual emotion representations from web data. CVPR, pp.
13106-13115, 2020.
Matthias J Wieser and Tobias Brosch. Faces in context: A review and systematization of contextual influences on affective face processing. Frontiers in psychology, 3:471, 2012.
Daiqing Wu, Dongbao Yang, Yu Zhou, and Can Ma. Bridging visual affective gap: Borrowing textual knowledge by learning from noisy image-text pairs. In ACM MM, pp. 602 621, 2024.
Daiqing Wu, Donghao Yang, Sicheng Zhao, Can Ma, and Yu Zhou. An empirical study configuring in-context learning demonstrations for unleashing milms' sentimental perception capability. In ICML, 2025.
Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Hung-Jen Chen, Chan-Feng Hsu, Hong-Han Shuai, and Wen-Huang Cheng. Emovit: Revolutionizing emotion insights with visual instruction tuning. In CVPR, pp. 26586-26595, 2024.
Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Emoset: A large-scale visual emotion dataset with rich attributes. In ICCV, pp. 20326-20337,
2023a.
Xiaocui Yang, Wenfang Wu, Shi Feng, Ming Wang, Daling Wang, and al. Mm-bigbench: Evaluat- ing multimodal models on multimodal content comprehension tasks, 2023b.
Zhengyuan Yang. Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision). CoRR, 17421,
2023c.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, and et al. Minicpm-v: A GPT-4V level MILMM on your phone. CoRR, abs/2408.01 00000 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, and et al. mplug-ow13, Towards long image-sequence understanding in modai large language models. CoRR, abs/2408. 0044400 2024.
Quanzeng You, Jiebo Luo, Hailin Jin, and Jianchao Yang. Building a large scale dataset for image emotion recognition: The fine print and the benchmark. In AAAI, pp. 108-311,, 2016.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chup Zhu. RAVEN: A dataset for relational and analogical visual reasoning. CVPR, pp. 5317-5321 2019.
Sicheng Zhao, Hongxun Yao, Yue Gao, Rongrong Ji, Wenlong Xie, Xiaolei Jiang, and Tat-Seng Chua. Predicting personalized emotion perceptions of social images. In MM, pp. 1385-1394, 2016.
Sicheng Zhao, Xingxu Yao, Jufeng Yang, Guoli Jia, Guiguang Ding, Tat-Seng Chua, Björ W. Schuller, and Kurt Keutzer. Affective image content analysis: Two decades review and new perspectives. Trans. Puttern Anal. Mach. Intell., (4(11))66299-611 66711, 2022.
Sicheng Zhao, Xiaopeng Hong, Jufeng Yang, Yanyan Zhao, and Guiguang Ding. Toward label- efficient emotion and sentiment analysis. Proceedings of the IEEE, 15911977 2023.
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS
Several limitations in this work can be further improved. First, our evaluation primarily focuses on MILMs with parameters under 10B due to computational constraints imposed by hardware. Although this covers practical deployment scenarios, it excludes larger-scale open-source MLLMs that may exhibit superior visual emotion perception capabilities. Second, the current implementation is limited to monolingual evaluation. Yet we highlight that adapting INSETS for multilingual construction would require relatively limited engineering effort, primarily involving adjustments in MLLM selection, prompt design, and template configuration. Moreover, while we explored lightweight model adaptations, more nuanced or advanced strategies remained underexplored.
B
LLM USAGE
This paper employs (M)LLMs for prompt engineering and data annotation. Additionally, they are also used during manuscript writing, mainly for grammar checking and refinement.
C PROMPTS AND STATEMENT TEMPLATES
The prompts and statement templates used in the INSETS pipeline are presented in Table 7.
Table 7: Prompts and statement templates employed in the INSETS pipeline.
Prompts and Statement Templates
You an Emotional Perception Expert. Please unalyze the emotions that might be evoked the given image. Your analysis should explore a wide range of visual attributes. such as brightness. colorfulness, depicted scenes. objects. human actions, and facial expressions, Additionally, provide detailed explanations linking these attributes the emotions they may trigger. applicable, discuss any potential cultural or psychological factors influencing these emotional responses.
#1
You an Emotional Perception Expert. Your task extract all applicable emotions as comprchensively possible hased on the image description. Focus on distinct emotions such as happiness, sadness, fear, anger, etc. Keep the list concise, with maximum of 10 distinct emotions.
#2
You are tusked with determining whether the word "[word]" describes specific emotional state. An emotional state is psychological condition involving feelings and reactions triggered by internal or external events. Respond with "Yes" the word aligns with this definition, or "No" otherwise. The output format should be ("word": "response").
#3
You are tasked with assigning the word "[word]" to the most closely related emotional categury from the following 115 predefined options: "[categories]". Consider brouder semantic connections und possible emotional nuances when making your judgment. If the word cannot reasonably fit any category, respond with "not applicable". Do not creute or assign new categuries outside of the provided list. Do not provide any explanations or reasons for your choice. The output format should be ("word": "response"), "I.
#4
#5
Brietly explain why this image might evoke "[emotion]" in viewers, without meutioning any other emotions.
Imagine a hackground story for the image that would evoke a sense of "[emotion]" in viewers. Respond in one sentence. Do not mention content in the image.
#6
Imagine a character who would feel "[emotion]" when viewing this image. Include details such as their age, gender. profession, and other relevant traits. Describe the churacter one concise sentence without further explanation.
#7
Upon viewing this image, observers, despite various individual contextual factors, are most likely to experience positive emotions.
#8
Upon viewing this image, observers, despite various individual or contextual factors, are most likely to experience negative emotions.
#9
Upon viewing this image, observers are equally likely to experience cither positive or nogative emotions, depending on individual or contextual factors.
#10
#11
Therefore, the image might evoke "[emotion]" in viewers.
#12
In the context of: "[context]", the image is likely to evoke sense of "[emotion]".
#13
Upon viewing the image, "[rote]" is more inclined to feel "[emotion]]" compared to "[emotion2]".
D DETAILS OF PARROTT's HIERARCHICAL MODEL
We present the complete emotion taxonomy of Parrott's hierarchical model in Table 8.
15

---

Published as a conference paper at ICLR 2026
Table 8: Emotion laxonomy of Parrott's hierarchical model.
Primary Fimotion
Secondary Finotion
Tertiary Emotion
Love
Aflection
Adoration, Fondnees, Liking, Attraction, Caring, Tendemesss, Compassion Sentimentality
Lust
Desire, Passion, Infatuation
Longing
Longing
Amusemens, Bliss, Gaiety, Gloc, Jolliness, Joviality, Joy, Delight, Enjoyment, Gladness, Happiness,
Joy
Cheerfulness
Jubilation, Elation, Satisfaction, Bestasy, Euphoria
Zest
Enthosiasm, Zeal, Excitement, Thrill, Exhilaration
Contentment
Pleasure
Pride
Triumph
Optimism
Fagereess, Hope
Enshrallment
Enthrallment, Rapture
Relief
Relief
Surprise
Surprise
Amazemens, Astonishment
Anger
Irritabllity
Agaravation, Agitation, Amooyance, Grouchy. Grumpy, Crosspatch
Exasperation
Frostration
Anger, Outrage, Fury, Wrath, Hostility, Ferocity, Bittemess, Hatred, Scorn, Spite, Vengefulneess
Rage
Dislike, Resentment
Disgust
Revulsion, Contempt, Loathing
Envy
Jealowsy
Torment
Sadness
SufTering
Agony, Anguish, Hurt
Depression, Despuir, Gloom, Glumness, Unhuppiness, Grief, Somow, Misery, Melancholy
Disuppointment
Dismay. Displeusure
Shame
Regret, Remorse
Alienation, Defentism, Dejection, Emharrassment, Homesickness, Humiliation, Insecurity, Insulr,
Neglect
Isolation, Loscliness, Rejection
Sympathy
Pity, Mono no aware, Sympathy
Fear
Homor
Alamm, Shock, Fright, Horror, Terror, Panic, Hysteria, Mortification
Nervousness
Anxiety, Suspense, Uneasiness, Apprehension, Warry, Distress, Dread
E FORMALIZATION OF THE MAJORITY- VOTING MECHANISM
In this section, we provide formalized definition of the majority-voting mechanism for clarification and transparency. Let image be processed by MLLMs. The set of open-vocabulary emotion labels generated by the i-th model is denoted by L, = { 6, ), where E {1, 2, n2..
Let P be the mapping from open-vocabulary emotion to the secondary emotion in POM. The ensemble-bard based majority-votinn procedure defined as follows.
1. Secondary-Emotion Quota. For secondary emotion é, define its quota Q(ē) as:
Q(e) max Σ l[ (P(e) L.
.0
where I[-] is the indicator function. This quantity is strictly positive only when is supported by majority of the models.
2. Candidate Pool Formation. The candidate pool of open-vocahulary labels for the secondary emotion ē is defined as:
S()) = (ess es, (1,2.... n); j ( m,]; P(eus) =
3. Consensus Selection. Let freq(e) denotes the frequency of an open-vocabulary label e in S(E): freq(e) = (e,, E S(E)le,, e}). The consensus labels for ē are defined as the top-Q(() unique labels in S(ě) ranked by freq(-), where ties are resolved uniformly at random. The final consensus label set for the image is obtained by taking the union all secondary emotions:
Leons
U Topq (S(e))
16

---

Published as a conference paper at ICLR 2026
Emotion
Emotion Labels
Emotion
Lobels
Labels
Companiorahip
Soothing
Warmeh
Curfous
Calmness
Playfulness
Comfort
Huppiness
Happineus
Scene Context Statteeent
Emotion Interpretation Statement
Correct
Sentiment Polority Stotement
Label:
sitting on green paasy fiuld, looking a bird flying overhead, the likeyy evole grumpy.
The image might evoke swe because the caterplibrrs
Upon slewing the iage, middi-aged woman more indined
Figure 5: Samples that are deemed ambiguous during the human refinement process.
F VISUALIZATION OF AMBIGUOUS SAMPLES
Figure presents three representative samples that are identified as ambiguous during the manual refinement process and thus excluded from the MVEI benchmark. In the left case, the contextual description considered an inappropriate or overextended inference from the visual content. the middle case, the emotional interpretation is overly brief, with cues that were too general or vague to support a reliable emotional conclusion. In the right case, the characterization lacked sufficient specificity, where two distinct emotions remain equally plausible.
attribute these ambiguities to two principal factors: the inherent limitations of MLLMs in visual perception, which manifest inaccurate descriptions superficial analyses, and the intrinsic challenges of emotion-related data construction, where contextual subjectivity can render multiple interpretations valid. These observations collectively highlight the critical necessity of human refinement in data construction pipelines involving MILMM.
G AGREEMENT BETWEEN ASSIGNED LABELS AND EMOSET LABELS
To further evaluate the automatically assigned vocabulayy labels, we perform a cross-validation based the original EmoSet labels. Since EmoSet is built upon the Mikels model (Mikels et al., 2005), which contains eight emotion categories (amusement, awe, contentment, excitement, anger, disgust, fear, and sadness), is naturally aligned with adopted Parrott hierarchical model.
However, we note that except for awe, the remaining seven Mikels categories are distributed across different levels of Parrott's model. To enable comparison, map them the primary-level emotions in Parrott's ' model: amusement + joy, contentment - joy, excitement + joy, anger + anger, disgust + anger, fear - fear, sadness - sadness. Finally, using the open- veoabuuaabrr attachment obtained from our constructed POM, we map awe + surprise. Based on this mapping, our analysis shows that automatically assigned open-vocabulary labels 97.3% of the 17,716 images in INSETS-462k overlap with their original EmoSet labels at the primary level of the Parrott model. This high level of consistency provides strong evidence for the validity of our automated annotation pipeline.
H ADDITIONAL RESULTS OF MLLMS ADAPTATION TECHNIQUES
Table reports results of more comprehensive MILLM adaptation techniques on MVEI, building upon those in Table 6. The 459k samples used for supervised fine-tuning correspond the full INSETS-462k dataset with MVEI excluded, and the 50k samples used for reinforcement learning sampled subset of the former. For inference and GRPO, the CoT prompt constructed by appending "think step-by- stepp" to the original prompt.
17

---

Published as a conference paper at ICLR 2026
Table Comprehensive Evaluation of MLLMs adaptation techniques on MVEL.
Accuracy (%)
Qwen2.5-VL (8.3B)
#Shot
#Sample
Sentiment Polarity
Emotion Interpretation
Scene Context
Perception Subjectivity
Total
Direct Inference
63.2
81.5
83,9
66.3
75.9
Chain-of- Thouuhght (CoT) Reasoning
67.4
81.5
84.6
67.0
76.6
Context Learning: Random Retrieval
66.3
81.6
84.8
66.5
76.6
In- Co-teeext Leaming: Random Retrieval
68.8
81.7
85.0
66.7
77.1
In Context Learning: Random Retrieval
70.1
81.7
84.9
67.0
77.3
LoRA Fine-Tuning (Hu al., 2022)
10k
78.6
84.7
86.3
70.3
80.7
LoRA Fine-Tuning (Hu 2022)
82.2
82.2
Full Parameter Tuning (Freeze Vision)
10k
84.3
84.8
87.0
71.1
81.9
Purameter Fine- Tunin (Freeze Vision)
73.3
83.4
(Shao al., 2024) (without
71.1
GRPO (Shao al., 2024) (without CoT)
50k
84.0
82.7
86.3
71.4
80.9
GRPO (Shao al., 2024) (with CaT)
10k
86.2
82.9
86.6
72.3
81.6
GRPO (Shao ol., 2024) (with CoT)
86.8
87.2
72.7
82.0
I LINKS OF MLLMS
We provide the links to the model cards of the MILMs we evaluated in the experiments.
LLa (Liu et al., 2024a)
Mantis at, 2024)
http:://aagggii.....o////aaaaaa2 11
mPLUG-OWI3 (Ye al., 2024)
Idefics3 et 2024)
(Abii al., 2024))
https://ccccai..ccccccccttttttsr Qwen2-VL a1, 2024)
Llama-3.2-Vision (Dubey al., 2024)
Inst
Molmo et,, 2024)
Inter (Chen et al., 2024b)
BLIP2 (Li al., 2023c)
InstructBLIP (Dai al., 2023)
Otter (Li et al., 2023a)
LA- InContext
Dee al., 2024)
18

---

Published as a conference paper at ICLR 2026
http://///ngggga....eeee//eeeeee Paligemma (Beyer al, 2024)
https:///haggingaaac..oooooog/// 100-pt-440 448
MiniCPM (Yao 2024)
http:://wwuughnneee...oooo/ppott (Tem,, 2025)
J
VISUALIZATION OF MVEI
More samples from MVEI are visualized in Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13.
Emotion
Emocion
Emotion
Labels
Labels
Labels
Wermth Amusement
Empathy
Cuncam
Determination
Anger
STOP THE PROPAGANDA!
Contentment
Diguut
Belasation
Sentiment Polarity Statement
Sentiment Polonity Statement
Conrect
Sentiment Polority Statement
Upon viewing this image, observers, despite var iuperience poshtiu emotinns.
Upon siewing this image, obsereess are equally Eiily
Upon viewing this image, dbservers, despite varlous
depending indieddull contestual fectors.
esperience negatiue emotion.
Figure 6: Sentiment polarity statements labeled as correct.
Labels
Labels
Labels
Discomoot
Exiiement
Warm
Confusion Fear
Confidence
Dread
Empowerenti
Enthusiasm
Confort
Passion
Contentment
Polority Statement
Polority Statement
Label:
Upon viewingtho image, observers, despite vor lous Individual contectual
Upon viewing this image, ebservers are equally Sikyy
Upon viewing this image, observers, despite various Individual contestual experience nogative emotions.
either posttive emations.
experience emotions.
deponding individual contestual
Figure 7: Sentiment polarity statements labeled as incorrect.
Labels
Labels
tabels
Excitement
Thril
Relaxation
Confidence
Courage
Awe
Ansiety
Excitment
Pride
Freedom
Interpretation
Interpretation
indivdduass are dressed recognicable unffrrm, surrounded by the calming blue water and white foam,
night
Image
The might
implaing professional untire suggest schievement Involvement meaningful activitios. create peaceful and serene atmosphere, which can be
same miacaaion viewen. The srfer's
is's colled position prodiiity roeks, which could be perceived as threatening shurtion.
gentle
soothing
Figure 8: Emotion interpretation statements labeled correct.
19

---

Published as a conference paper at ICLR 2026
Emation Labels
Emotion
Emotion
Labels
Labels
Esxitement
Exitement
Fascination
Determination
Senurity
Wonder
Passion
Concentration
Amusement
Disgust
Emotion Interpretation Statement Incorrect
Emotion Interpretation Stotement
Labet Incorect
Emation Interpratation Statement Incorrect
The image of fly might svolee wonder in viewers becsuse targe, bulbous and painted mouthparts
image of smiling woman holding a plass wine
The accordion, traditional and oftee vintage musical ovole nostaigla viewers with past experienoes.
besusiful oundoor setting. possiity patio or balkony.
happiness enjoyment.
threatoning, Additionaly, the fl's prosimity to the viewer's
seme dissmmortt
Figure 9: Emotion interpretation statements labeled as incorrect.
Emotion
GALAS
Appreciation
Gratitude
Joy
Soliude
Deternination:
Enthusiaum
Wander
Hope
Label: Correct
Label: Correct
Label: Conrect
place the world, Sarah finally discovered her passion
years struggling
wast esparie toy lundscapa, a lone figure trudges through the snow,
group space, untted a cause, holding signs
for stand comedy, and now she's taking the
leawing babind trail footprints that tell aleen bale
express their graniuuee and celebrane signtficam
beamiin
accomnlishmentt
soltary jounney, Image
milistonn',
evolic
the image is likly evoke sense joy.
solitude.
Figure 10: Scene context statements labeled as correct.
Emotion
Emotion Labels
Emotion Labels
Labels
Fear
Bravery
Curlosity
Wonder
Determiiation
Excitement
Scene Contest Statement
Scene Context Statement
Label: Incorrect
Scene Contest Statement
In the contect of: ' lonely mas wanders through desclate foreet, searching fora glimps unforgiving landscape', image
In the compent of ' firefighter rushes into bumiin building family faur, danger'. image
In the content of 'A family gathered togother at fair, loughing sharing the ecctement the calarful rides, the image likely evoke sense of isolation.
lkaly to evoke a pense of peace.
Figure 11: Scene context statements labeled incorrect.
Emotion
Emotion
Emotion
PAIMEE
Happiness
Nostaiga Peace
Nestalgaa
Avee
Abandonment
Maylilnes
Wonder
Sadness
Pescefulnes
Heppineas
Neplect
Excitement
Freedom
Contemplative
Perception Subjectivity Statement Label: Correct
Perception Subjectivity Statement
Label: Correct
Perception Subjectivity Statement
Label: Correct
image, alderly
viewing image, more inclined to feel sue compared to misery.
biologiat
Upon viewing image, young possibly a more inclined to feel wander companed nostsighs.
bakeery in the 2010s due the ecosomic downtur, more incined feel compared to delighe.
Figure 12: Perception subjectivity statements labeled as correct.
20

---

Published as a conference paper at ICLR 2026
Emodion
Emodion
Emottion
Contentment Warmth
Determination: Wonder Fear Apprectation
WAR
Solidarity Frustration
Connection
SAFT
WCHAANS
Heppiness
Perceation Subjectivity Statement Label: Incorrect
Perception Subjectivity Statement abel: Incorrect
Perception Subjectivity Statoment label: Incorrect
viewing middl
Upon viewing middls-ged male politicann
pediatrccinn who a mother two feel revalsion compared warmth.
more inclined fear compared determination.
poace
solidarity compared frustration.
Figure 13: Perception subjectivity statements labeled as incorrect.
21


Reviews:
[{"review_id": "cVD9WVqWwo", "reviewer": "Reviewer_nM4y", "summary": "This paper addresses the evaluation of visual emotional perception in Multimodal Large Language Models (MLLMs). The authors argue that prior evaluation methods—fixed label classification and coarse taxonomies—fail to capture the subjectivity, contextual dependence, and open-vocabulary nature of emotion perception. To remedy this, they propose the Emotion Statement Judgment (ESJ) task: MLLMs judge whether emotion-focused statements about an image are correct, enabling open-vocabulary, multifaceted assessment. They also introduce INSETS, an automated pipeline that generates emotion-centric labels and statement candidates with limited human intervention. Using INSETS they build a large automatically annotated corpus (INSETS-462k) and then refine a human-validated benchmark (MVEI) of 3,086 image–statement pairs covering four complementary dimensions: sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Systematic evaluation of many MLLMs shows that models have notable strengths (especially in emotion interpretation and scene context) but still trail humans—particularly on sentiment polarity and perception subjectivity. The paper also explores lightweight adaptation methods (in-context learning, LoRA, full fine-tuning, GRPO) and finds that adaptation improves performance—most dramatically for sentiment polarity—while perception subjectivity remains challenging. The authors release code and data and discuss ethical considerations around dataset biases and model outputs.Three main contributions:1. Task formulation: Introduces the Emotion Statement Judgment (ESJ) task, a flexible, open-vocabulary framework that reframes visual emotion evaluation as a statement verification problem, reducing issues from rigid ground-truth answers and enabling richer, multifaceted assessment.2. Scalable annotation pipeline and corpus: Proposes INSETS, an automated pipeline for generating emotion labels and statements with minimal human effort, and uses it to construct INSETS-462k (462k statements across ~17.7k images), significantly improving scalability over prior labor-intensive datasets.3. Benchmark and empirical analysis: Curates MVEI (3,086 human-refined image–statement pairs) covering four evaluation dimensions (sentiment polarity, emotion interpretation, scene context, perception subjectivity), and provides systematic benchmarking of many contemporary MLLMs plus adaptation studies that reveal strengths, weaknesses, and directions for improvement.Three main shortcomings / limitations:1. Model scale coverage: The evaluation is limited mainly to MLLMs with fewer than ~10B parameters because of computational constraints, excluding larger open-source or closed models that might perform differently; this limits conclusions about upper-bound capabilities.2. Monolingual focus and potential biases: The current implementation and benchmark are monolingual (presumably English) and the automatically generated INSETS-462k may inherit biases from pretraining data and MLLMs; despite human refinement some problematic or culturally specific labels could persist.3. Fundamental difficulty with subjectivity: While adaptation improves sentiment polarity substantially, perception subjectivity remains poorly handled by MLLMs and shows only modest gains from fine-tuning; this suggests deeper architectural or objective-level limitations that the current pipeline and data do not fully resolve.", "questions": "SEE WEAKNESS", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Task formulation: Introduces the Emotion Statement Judgment (ESJ) task, a flexible, open-vocabulary framework that reframes visual emotion evaluation as a statement verification problem, reducing issues from rigid ground-truth answers and enabling richer, multifaceted assessment.\n- Scalable annotation pipeline and corpus: Proposes INSETS, an automated pipeline for generating emotion labels and statements with minimal human effort, and uses it to construct INSETS-462k (462k statements across ~17.7k images), significantly improving scalability over prior labor-intensive datasets.\n- Benchmark and empirical analysis: Curates MVEI (3,086 human-refined image–statement pairs) covering four evaluation dimensions (sentiment polarity, emotion interpretation, scene context, perception subjectivity), and provides systematic benchmarking of many contemporary MLLMs plus adaptation studies that reveal strengths, weaknesses, and directions for improvement.", "weaknesses": "- Model scale coverage: The evaluation is limited mainly to MLLMs with fewer than ~10B parameters because of computational constraints, excluding larger open-source or closed models that might perform differently; this limits conclusions about upper-bound capabilities.\n- Monolingual focus and potential biases: The current implementation and benchmark are monolingual (presumably English) and the automatically generated INSETS-462k may inherit biases from pretraining data and MLLMs; despite human refinement some problematic or culturally specific labels could persist.\n- Fundamental difficulty with subjectivity: While adaptation improves sentiment polarity substantially, perception subjectivity remains poorly handled by MLLMs and shows only modest gains from fine-tuning; this suggests deeper architectural or objective-level limitations that the current pipeline and data do not fully resolve.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GenoDbIauG", "reviewer": "Reviewer_joCJ", "summary": "This paper addresses the incompatibility of conventional emotion evaluation methods with Multimodal Large Language Models (MLLMs) by proposing the Emotion Statement Judgment (ESJ) task and the INSETS annotation pipeline. The work constructs MVEI, a benchmark with 3,086 samples covering four dimensions: sentiment polarity, emotion interpretation, scene context, and perception subjectivity. The authors evaluate 18 MLLMs and demonstrate that even top-performing models like GPT4o fall substantially short of human performance.", "questions": "Please check the weakness part.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Benchmark Innovation and Scope: The ESJ task and MVEI benchmark fill a substantial gap in visual emotion evaluation for MLLMs, moving beyond rigid classification to nuanced, open-vocabulary, and multifaceted judgment tasks. This approach allows broader, context-aware, and fine-grained assessment.\n2. Automated scalable pipeline: The INSETS pipeline effectively reduces human labor by automating emotion tagging and statement generation while maintaining a reported 90.6 % accuracy after human refinement.\n3. Comprehensive empirical analysis: The benchmark evaluation spans >15 MLLMs, both open and proprietary, and compares against human baselines .\n4. Grounding in Theory: The mapping to Parrott’s hierarchical emotion model and the use of extensive prompts/protocols demonstrate that the benchmark is theoretically informed, not ad-hoc.", "weaknesses": "1. Potential circularity in evaluation: Some MLLMs used to construct INSETS also participate in evaluation, which could inflate results and reduce benchmark independence.\n2. Vague or Underspecified Mathematical/Labeling Procedures: The emotion label assignment, majority voting, and mapping to the hierarchical model are described mainly in prose and schematic diagrams, with some reliance on prompt outputs and “manual expert check.” The specific algorithms are insufficiently formalized.\n3. Human annotation reliability and bias: Although inter-annotator agreement statistics are reported, there is limited qualitative analysis of ambiguous cases or bias due to annotator demographics.\n4. Insufficient discussion of dataset bias and ethical risk: Bias acknowledgement is limited to a short ethics statement without empirical analysis of representation balance (e.g., emotional polarity, demographic context).\n5. Limited analysis of failure modes: Sec. 5.3 identifies that \"perception subjectivity shows only modest improvement\"  but provides no error analysis. What types of subjectivity statements are most challenging? Are errors systematic? Figure 12 shows examples but no quantitative breakdown.\n6. Absence of ablation or component analysis: There is no direct evidence quantifying how each component contributes to final benchmark quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V8V5VZZu4H", "reviewer": "Reviewer_Kehu", "summary": "This paper introduces a new evaluation paradigm for assessing multimodal large language models (MLLMs) on visual emotion understanding. Instead of traditional emotion classification, the authors propose an Emotion Statement Judgment (ESJ) task, where models evaluate whether a given emotion-related statement about an image is correct. The paper further presents an automated pipeline (INSETS) to construct large-scale emotion-centric statements and builds the MVEI benchmark covering four dimensions: sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Experiments show that current MLLMs demonstrate certain capabilities but still lag behind humans, especially in subjective and contextual emotional reasoning.", "questions": "Please refer to the weakness.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Comprehensive emotional dimensions, covering aspects that existing benchmarks ignore, such as scene context and perception subjectivity.\n\nExtensive experiments on a wide range of MLLMs with meaningful insights and human comparison results.", "weaknesses": "The dataset is partially constructed using existing MLLMs, which may introduce inherited biases. Although human refinement is applied, its proportion remains limited.\n\nThe distinction from EEmo-Bench: A Benchmark for Multi-modal Large Language Models on Image Evoked Emotion Assessment is not sufficiently articulated.\n\nThe dataset is built on a single source domain, and its generalizability to other visual contexts is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0sNQlGvMz2", "reviewer": "Reviewer_ck1T", "summary": "The paper introduces a new annotation framework for MLLMs in the task of visual emotion estimation. The paper identifies limitations in the evaluation of MLLMs (namely, inaccurate responses, limited taxonomies, a focus on intrinsic image attributes, and reliance on majority voting) and first introduces a new task to address these limitations. Then, an automated annotation pipeline leveraging an ensemble of MLLMs is used to annotate 400k images and create fine-grained annotations that are used to produce statements with GPT4 and curated by a human expert. The labels and statements are further processed to produce additional coarse annotations (in Parrott-based hierarchy and positive/negative polarity). Finally, several popular MLLMs are benchmarked on the test set of the dataset, including an in-context learning and a finetuned version of Qwen.", "questions": "- How do the annotations generated by the proposed method in step [a] compare to the ones from EmoSet for the subset used in this work? Are the generated emotions in agreement with the 8 categorical used there?\n- Using a small subset, is it possible to compare human open-vocabulary to MLLM open-vocabulary annotations? Are these in agreement or at least in the same region?\n- The polarity underperformance may be due to the prompting used. Have you tried CoT reasoning? Polarity is a derivative, not a primary label, hence the suggestion.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problems identified by the paper are valid. The uncertain nature of human emotions (invoked, perceived or otherwise) is one of the key limitations in achieving high accuracy in the task, opposed to object detection, for example. Cultural barriers, inconsistent hierarchies, etc, are all issues in affective computing (and psychology for that matter).\n- The proposed method is indeed scalable and can be replicated to extend the proposed dataset or correct the labels of existing benchmarks.\n- With the rise of MLLMs, there is a growing interest in applying them to downstream tasks; however, the lack of annotations in natural language makes things more complicated, which is something this method addresses.", "weaknesses": "- One major weakness is the contradiction in motivation. Specifically, two of the key issues identified in the intro are a) MLLM's inaccurate responses when compared to human judgment and b) majority voting to address disagreement. The method then proceeds to create annotations by taking the majority voting of MLLMs. This is reiterated in the discussion (lines 448-9): \"MLLMs may not yet be sufficiently competent for LLM-as-a-judge [...]\". If this is the case, how are they sufficient as annotators? The method is contradictory to the problem it is trying to solve and thus is not convincing in terms of annotation accuracy. The one human expert curating statements post-hoc is probably not enough, as: a) there is already unconscious bias in the process as the expert is shown annotations rather than producing annotations, b) the task is uncertain even for experts, therefore, is one person really enough?, c) the method does not account for cultural differences and biases (both in the expert and the process as a whole). We also don't have an indication of how the humans (expert or otherwise) perform on the open-vocabulary compared to the LLMs. In general, I am unconvinced about the validity and motivation of the method.\n- Looking at the human refinement process, the annotator agreement in Table 3 does not seem very strong, particularly for emotion interpretation and subjectivity, showing a (Cohen's?) Kappa of only 0.5. In fact, the only aspect where the human annotators agree is the scene context; for everything else, there is only weak agreement.\n- For the fine-tuned results in Table 6, only 10k out of 462k images are selected. This seems a bit of an odd choice, as most large-scale affective datasets (including EmoSet) are much larger; therefore, it seems unlikely that the finetuned version will be able to compete, but at least performance gains from fine-tuning are shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)