Title: Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach
Venue: NeurIPS 2025
Accepted: no
Keywords: MLLM

Visual Emotion Evaluation
Abstract: Recently, Multimodal Large Language Models (MLLMs) have achieved exceptional performance across diverse tasks, continually surpassing previous expectations regarding their capabilities. Nevertheless, their proficiency in perceiving emotions from images remains debated, with studies yielding divergent results in zero-shot scenarios. We argue that this inconsistency stems partly from constraints in existing evaluation methods, including the oversight of plausible responses, limited emotional taxonomies, neglect of extra-visual factors, and labor-intensive annotations. To facilitate customized visual emotion evaluation for MLLMs, we propose an Emotion Statement Judgment task that overcomes these constraints. Complementing this task, we devise an automated pipeline that efficiently constructs emotion-centric statements with minimal human effort. Through systematically evaluating prevailing MLLMs, our study showcases their stronger performance in emotion interpretation and context-based emotion judgment, while revealing relative limitations in direct determination of sentiment polarity and personalized emotion prediction. When compared to humans, even top-performing MLLMs like GPT-4o demonstrate remarkable performance gaps, underscoring key areas for future improvement. By developing a fundamental evaluation framework and conducting a comprehensive MLLM assessment, we hope this work contributes to advancing emotional intelligence in MLLMs. Codes and data will be released.

Abstract

Recently, Multimodal Large Language Models (MLLMs) have achieved exceptional performance across diverse tasks, continually surpassing previous expectations regarding their capabilities. Nevertheless, their proficiency in perceiving emotions from images remains debated, with studies yielding divergent results in zero-shot scenarios. We argue that this inconsistency stems partly from constraints in existing evaluation methods, including the oversight of plausible responses, limited emotional taxonomies, neglect of extra-visual factors, and labor-intensive annotations. To facilitate customized visual emotion evaluation for MLLMs, we propose an Emotion Statement Judgment task that overcomes these constraints. Complementing this task, we devise an automated pipeline that efficiently constructs emotion-centric statements with minimal human effort. Through systematically evaluating prevailing MLLMs, our study showcases their stronger performance in emotion interpretation and context-based emotion judgment, while revealing relative limitations in direct determination of sentiment polarity and personalized emotion prediction. When compared to humans, even top-performing MLLMs like GPT-4o demonstrate remarkable performance gaps, underscoring key areas for future improvement. By developing a fundamental evaluation framework and conducting a comprehensive MLLM assessment, we hope this work contributes to advancing emotional intelligence in MLLMs. Codes and data will be released.

1 Introduction

Perceiving emotional signals from visual stimuli is essential for humans to improve decision-making and build effective communication [1, 2]. To computationally model this capability, Affective Image Content Analysis (AICA) has emerged as a key research direction in computer vision [3], focusing on emotion perception through visual features [4]. Over the decades, advances in this field have given rise to various applications, including opinion mining [5], customized advertising [6], and mental health care [7]. Recently, the advent of Multimodal Large Language Models (MLLMs) [8, 9] has revolutionized image understanding tasks [10]. However, their effectiveness in AICA remains contested. Divergent findings underscore a paradox: while some studies [11, 12] demonstrate MLLMs' poor zero-shot emotion recognition performance, others successfully employ them as emotion annotators for training data augmentation [13, 14]. We attribute this discrepancy to the partial incompatibility of conventional emotion evaluation approaches with MLLMs.

Specifically, current evaluation approaches can be broadly categorized into emotion classification and emotion interpretation, as illustrated in Fig. 1 (a,b). In emotion classification, models are required to assign the emotional state of an input image to a predefined set of emotion categories, with most

Task: What emotion(s) might the image evoke?

Label:

Disconnection, Doubt

Confident, Disconnection, Disquietment

Task: What might cause a specific emotional state?

Label:

Sad

Question:

What is the cause of the man in the image being sad?

Ground Truth Answers:

1.

Crying.

2.

Indoors or at night.

3.

Standing next to another person.

4.

Being comforted by another.

GPT4o Answers:

1.

Downturned mouth, and slightly closed eyes.

2.

A possibly cold or harsh environment.

3.

He might experience an emotional difficulty.

Label:

Contentment

Question:

Why might the image evoke contentment in viewers?

Ground Truth Explanation:

It is dark and gloomy but the house has a lot of characters.

Its peaceful and rural setting depict a quiet village or countryside scene.

Single-label Classification

Label:

Amusement

Contentment

Multi-label Classification

Emotion Explanation

Human Evaluation:

Reasonable

Metric

Evaluation:

Incorrect

Emotional Trigger Identification

Human Evaluation:

Reasonable

Metric Evaluation:

Incorrect

Human Evaluation:

Reasonable

Metric Evaluation:

Incorrect

Human Evaluation:

Reasonable

Metric

Evaluation:

Incorrect

Emotion Statement Judgement (ESJ)

Task: Is the statement correct regarding the image?

Statement on Sentiment Polarity

Upon viewing the image, observers, despite individual or cont-

extual factors, are most likely to experience negative emotions.

Label:

Incorrect

Statement on Emotion Interpretation

The image might evoke 'bravery' because it depicts a firefigh-

ter engaged in extinguishing a fire. The firefighter's protective

gear and the use of a fire extinguisher suggest a high level of

skill and courage required to handle such a situation.

Label:

Correct

Statement on Scene Context

In the context of: 'A firefighter rushes into a burning forest to

save a family of four, despite the danger', the image is likely to

evoke a sense of urgency.

Label:

Correct

Statement on Perception Subjectivity

Label:

Incorrect

Upon viewing the image, a 35-year-old male firefighter is more

inclined to feel fear compared to urgency.

(a).

(b).

(c).

benchmarks [15, 16, 17, 18, 19] providing a single label per image, and a few [20, 21] incorporating 35 multiple labels. In contrast, emotion interpretation focuses on understanding the underlying causes of 36 emotions in images. It encompasses two primary sub-tasks: explaining the causes of emotional states 37 [22, 23] and identifying salient visual elements that contribute to the emotional response [24]. 38

We identify four primary limitations when applying these evaluation approaches to MLLMs. Firstly , 39 their adoption of fixed ground-truth answers for open-ended questions imposes structural constraints 40 that exclude other plausible responses. Emotion perception is inherently subjective [25, 26], as 41 the same image may evoke divergent reactions across individuals, and emotional states permit 42 varied interpretations. As demonstrated in Fig. 1 (a,b), responses generated by GPT-4o that seem 43 reasonable to humans are judged as inaccurate under rigid evaluation metrics. Secondly , they are 44 mostly constructed upon emotion theories with limited emotional taxonomies. Popular emotion 45 classification and interpretation benchmarks, such as FI [16] and Artemis [22], comprise only eight 46 emotion categories. Such taxonomic granularity fails to capture fine-grained affective variations 47 between images. Thirdly , they focus solely on intrinsic image attributes while neglecting extrinsic 48 contextual factors. According to recent studies [4, 27], emotion perception can also be shaped by 49 extra-visual factors, including the scene context where the image takes place, as well as the identity 50 and personality of the viewer [25]. Fourthly , they predominantly rely on majority voting mechanisms 51 to ensure label reliability in crowdsourced annotations [28], which is labor-intensive, particularly 52 for fine-grained annotation tasks. EMOTIC [20], for instance, requires coordination with 23,788 53 annotators to label 18,316 images. This operational burden severely constrains dataset scalability in 54 magnitude and generalization capacity across image domains. 55

To facilitate customized visual emotion evaluations for MLLMs, we propose a dual-component 56 solution that addresses these limitations: the Emotion Statement Judgment (ESJ) task, complemented 57 by the INSETS ( IN telligent Vi S ual E motion T agger and S tatement Constructor) pipeline for efficient 58 annotation. As a pioneering effort, we prioritize a precise over a complex evaluation design, aiming 59 to establish a fundamental offline standard. With this purpose, ESJ reformulates visual emotion 60 evaluation by requiring MLLMs to validate emotion-centric statements for a given image . It 61 effectively mitigates ambiguity in open-ended questions while being highly extensible for evaluation 62 depth and diversity. Meanwhile, INSETS annotates images with multiple open-vocabulary emotion 63 labels, significantly refining the emotional taxonomies. These labels are then utilized to construct 64 multifaceted emotion-centric statements, covering intrinsic image attributes like sentiment polarity 65 and emotion interpretation, as well as extrinsic contextual factors like scene context and perception 66 subjectivity. Crucially, only minimal human intervention is required, ensuring a high scalability of 67 the approach. An example of ESJ is illustrated in Fig. 1 (c). 68

Leveraging INSETS, we introduce two ESJ benchmarks: a large-scale INSETS-462K and its human69 refined subset INSETS-3K. Systematic evaluation demonstrates that recent MLLMs exhibit non-trivial 70 visual emotion perception capabilities, yet maintain non-negligible performance gaps compared to 71 humans, particularly in discerning sentiment polarity and comprehending perception subjectivity. In 72 summary, the contributions of this paper are three-fold: 73

74

75

This paper constitutes a pioneer effort to identify limitations in existing visual emotion evaluations for MLLMs and address them with a customized ESJ task.

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

Complementing the ESJ task, this paper designs the INSETS pipeline, providing a reliable approach to annotating images with multiple open-vocabulary emotion labels and constructing multifaceted emotion-centric statements with minimal human effort.

Utilizing the ESJ task and the INSETS pipeline, this paper conducts a systematic evaluation of recent MLLMs in visual emotion understanding, offering insights and fostering further developments of emotional intelligence in MLLMs.

2 Related Works

2.1 AICA Benchmarks

Psychological researchers conceptualize emotion representation through two principal frameworks: the Categorical Emotion Space (CES), which discretizes affective states into predefined taxonomies, and the Dimensional Emotion Space (DES), which maps emotions onto continuous 2D/3D coordinate systems (e.g., valence-arousal-dominance (V AD) axes [29]). For simplicity and better interpretability, most benchmarks adopt emotion classification evaluations based on discrete CES emotion taxonomies. This category encompasses both early small-scale benchmarks, such as IAPSa [30] and Abstract [15], as well as later larger-scale benchmarks like FI [16] and WebEmo [18]. Over time, benchmarks with enriched metadata have also been developed. Notable examples include EMOTIC [20], which integrates multiple emotion categories, V AD values, and human-related bounding boxes, and EmoSet [19], which employs describable emotion attributes that cover different levels of visual information.

Some other benchmarks adopt emotion interpretation evaluations by extending CES-based taxonomies with additional emotional explanations, such as Artemis [22] and Affection [23]. EIBench [24] diverges slightly, shifting focus on identifying and extracting visual emotional triggers. Based on these benchmarks, numerous expert models [31, 32, 33] have been developed, demonstrating strong performance under the fine-tuning and testing paradigm. In contrast to them, MLLMs are commonly pre-trained on web-scale data, without explicitly aligning with benchmark-specific knowledge. This discrepancy introduces multiple constraints when applying conventional benchmarks to MLLMs, necessitating customized visual emotion evaluation approaches that account for their generalized knowledge structures.

2.2 Evaluation of MLLMs

Recent years have witnessed surging academic and industrial interest in MLLMs. Unlike earlier models that are limited to specific domains, MLLMs demonstrate versatile competence across diverse tasks [34, 35], fueling expectations about their trajectory toward Artificial General Intelligence [36]. To evaluate MLLMs, various benchmarks have been established to examine their capabilities in areas such as perception [37, 9], reasoning [38, 39], ethics [40, 41], and specialized domains [42, 43]. However, emotional intelligence remains conspicuously underexplored. In existing efforts, FABABench [44] evaluates MLLMs' comprehension of facial expressions and actions, MM-BigBench [45] simply aggregates mainstream image-text benchmarks, and EmoBench [46] is confined to solely language modality. To fill this gap, we propose the ESJ task and corresponding benchmarks INSETS462K and INSETS-3K to advance more comprehensive visual emotion evaluation of MLLMs.

3 Emotion Statement Judgement

ESJ aims to evaluate the proficiency of MLLMs in perceiving emotions from visual content. In each evaluation trial, MLLMs receive an image and a paired emotion-centric statement. MLLMs are then required to judge the correctness of the statement in relation to the image by responding

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

with Correct or Incorrect . To ensure both depth and breadth in evaluation, we systematically design emotion-centric statements from four dimensions:

Sentiment Polarity Statements require MLLMs to decide sentiment polarities without any additional clues. They assess MLLMs' proficiency in directly identifying the basic emotional tone.

Emotion Interpretation Statements ask MLLMs to verify the consistency between affective explanations and corresponding emotional states. They measure MLLMs' affective reasoning capability given specific emotional triggers.

Scene Context Statements probe MLLMs' comprehension of the dynamic interplay between the external scene context where the image takes place, and image-evoked emotional responses.

Perception Subjectivity Statements task MLLMs to predict the personalized emotional responses under assumptions of specific viewer identities, examining whether MLLMs can recognize how subjectivity shapes emotional perceptions.

Collectively, these dimensions establish a holistic visual emotion evaluation framework for MLLMs. They cover both intrinsic image attributes emphasized in existing benchmarks and underexplored extrinsic contextual factors critical for human emotional perception [47, 48].

4 Annotation Pipeline: INSETS

Complementing the ESJ task, we design an automated pipeline for constructing emotion-centric statements, termed INSETS ( IN telligent Vi S ual E motion T agger and S tatement Constructor). It operates through two primary stages: open-vocabulary emotion tagging and emotion statement construction. The prompts and templates used in the process are listed in Appendix Table 5.

4.1 Preliminary: Parrott's Hierarchical Model

We first introduce a well-established emotion model, which provides essential context for understanding the subsequent stages. Parrott's Hierarchical model [49, 50] is a tree-structured emotion taxonomy, comprising 6 primary emotions, 25 secondary emotions, and 113 tertiary emotions. The primary category contains three positive emotions (joy, love, and surprise) and three negative emotions (anger, fear, and sadness). Secondary emotions offer more diverse emotional states, each categorized under a corresponding primary emotion, while tertiary emotions further refine secondary emotions into more specific affective states. The complete taxonomy is presented in Appendix Table 7.

4.2 Open-vocabulary Emotion Tagging

At this stage, INSETS aims to assign open-vocabulary emotion labels for images, laying a solid foundation for constructing meaningful emotion-centric statements, with its procedure depicted in Fig. 2. According to Cheng et al. [14], MLLMs demonstrate promising capabilities in generating emotional descriptions from visual content and extracting underlying emotions from these descriptions. However, challenges such as hallucinations [51], trustworthiness issues [52], and inherent limitations in emotional perception can lead to inaccuracies in the extracted emotions. To enhance reliability, we devise an ensemble-based majority voting mechanism, aggregating outputs from multiple MLLMs to cross-validate and refine emotion label assignments.

Given an image sample, we first extract its potential open-vocabulary emotions from multiple MLLMs. MLLMs are prompted to analyze the emotions evoked by the image (with #1 prompt in Table 5, abbreviated as '#1' in the following) and then extract emotions applicable to the image (#2) [Fig. 2 (a)]. This process is iteratively applied to all images in the dataset, aggregating potential emotions into an emotion pool. Next, we refine this pool by filtering out words unsuitable as emotion descriptors (#3), using GPT-4 [53] as the judge due to its superior linguistic emotional perception [46] [Fig. 2 (b)]. Once the filtered emotion pool is obtained, we attach the remaining emotions to Parrott's hierarchical emotion model [Fig. 2 (c)]. Specifically, GPT-4 is prompted to categorize each open-vocabulary emotion into the closest tertiary emotion in Parrott's model (#4), followed by manual refinement by a human expert. This process results in an extended version of Parrott's model, which we refer to as the Parrott-based Open-vocabulary Hierarchical Model (POM) [Fig. 2 (d)]. This unified framework

Extract Open-Vocabulary

Emotions

Image Dataset

Filter Unqualified Words

Filtered Emotion Pool

Attach Open-

vocabulary Emotions to Parrott's Hierarchical Emotion Model

Instruction:

Determine which emotional category from [

Tertiary Categories

] most aligns with [

Word

].

(#4)

GPT-4:

{[

Reverence

]: Adoration},  {[

Admiration

]: Adoration},  {[

Intimacy

]: Caring},  {[

Protectiveness

]: Caring},  {[

Warm

]: Tenderness},

[Word]

[Tertiary Categories]

Emotion Pool

Qualified [Word]

Parrott's Hierarchical Emotion Model

Polarity

Primary

Secondary

Tertiary

Positive,

Negative

Joy,

Love,

Surprise

Positive

Comprises:

Affection

Lust, Longing

Love

Comprises:

Caring, Tenderness,

Affection

Comprises:

......

Rough Relation

Instruction:

Determine whether [

Word

] describes an emotional state.

(#3)

GPT-4:

{[

Serenity

]:        },  {[

Calmness

]:        },  {[

Energy

]:        },

......

[Word]

......

Reverence

Manual Refinement

Open-vocabulary

Emotions

Emotions

Admiration

Intimacy

Protectiveness

Warm

Adoration

Caring

Fondness

Tenderness

Reverence

Open-vocabulary

Emotions

Admiration

Intimacy

Protectiveness

Warm

Adoration

Caring

Fondness

Liking

Instruction:

Analyze the emotions evoked by the <image>.

MLLM:

The individuals in the image are wearing matching

costumes that include feather boas, sequined tops, and

elaborate headpieces. The headpieces are adorned with …

Instruction:

Extract emotions applicable for the <image>.

MLLM:

Happiness, Excitement, Joy, Euphoria, Community,

Creativity, Festivity, Celebration, Enthusiasm, Satisfaction

Instruction:

Analyze the emotions evoked by the <image>.

MLLM:

The costumes are elaborate and festive, with

elements such as flowers, feathers, and possibly animal motifs.

These elements can evoke a sense of celebration, festivity …

Instruction:

Extract emotions applicable for the <image>.

MLLM:

Joy, Happiness, Excitement, Celebration, Community,

Cultural Appreciation

Instruction:

Analyze the emotions evoked by the <image>.

MLLM:

The scene is incredibly bright and full of vivid colors.

Pink, green, red, and blue dominate the palette, creating an

energetic and cheerful atmosphere. Bright and saturated …

Instruction:

Extract emotions applicable for the <image>.

MLLM:

Joy, Exuberance, Euphoria, Playfulness, Whimsy,

Celebration, Inclusion, Community

Instruction:

Analyze the emotions evoked by the [

Image

].

(#1)

MLLM:

The image depicts a vibrant and colorful scene, likely

from a festival or parade. The use of bright and vivid colors,

such as pink, green and blue, can evoke feelings of joy …

Instruction:

Extract emotions applicable for the [

Image

].

(#2)

MLLM:

Joy, Excitement, Happiness, Wonder, Amusement,

Anticipation, Celebration, Community, Engagement

Qwen2-VL

InternVL2.5

Llava-Next

Idefics3

Extract Open-vocabulary Emotions

Image Sample

[Image]

Vote for Consensus Emotions based on POM

Qwen2-VL

InternVL2.5

Llava-Next

Idefics3

Exuberance, Euphoria …

Joy, Cultural Appreciation …

Excitement, Enthusiasm …

Qwen2-VL

InternVL2.5

Llava-Next

Idefics3

Cheerfulness, Optimism …

Cheerfulness, Zest

Cheerfulness, Zest

Cheerfulness, Zest

Map Open-vocabulary Emotions

to Secondary Emotions

Vote at

Secondary-level

Cheerfulness

Optimism

Zest

Quota: 3

4/4 Votes

Quota: 0

1/4 Votes

Quota: 1

3/4 Votes

Happiness

Engagement …

Exuberance, Euphoria

…

Joy

Cultural Appreciation …

Excitement

Enthusiasm …

Cheerfulness

Zest

Select Top-

frequent Open-

vocabulary

Emotions

Parrott-based Open-vocabulary Hierarchical Model  (POM)

PM : Primary

OV

: Open-vocabulary

SC

: Secondary

PL

: Polarity

TT

: Tertiary

Positive

Negative

Joy

Love

Surprise

Fear

Anger

Sadness

Affection

Lust

Longing

Zest

Irritability

Exasperation

Rage

Reverence

Admiration

Reverent

OV

OV

OV

Warm

Closeness

Companionship

OV

OV

OV

Passion

Passionate

OV

OV

Intense

Emotional

Intensity

OV

OV

Frustration

Hostility

Dislike

Outrage

Outrage

OV

Unloving

Aversion

OV

OV

Surprise

Passion

Fondness

Adoration

Caring

Suffering

Horror

Unkind

Hostility

Aggression

Aggressive

Confrontation

OV

OV

OV

OV

OV

TT

TT

TT

TT

TT

TT

TT

TT

SC

SC

SC

SC

SC

SC

SC

SC

SC

PM

PM

PM

PM

PM

PM

PL

PL

Guide

Dataset-wise

Sample-wise

Open-vocabulary Emotions

Happiness

Joy

Euphoria

Excitement

Local MLLMs inferences

Local MLLMs inferences

Happiness

Joy

Euphoria

Excitement

Emotions

(a).

(b).

(c).

(e).

(a).

(d).

excitementorear

curiosity

tranquility

happiness

relaxationt

contentment

serenity

OV

curiosityoy

excitement.fear

peace

happiness

nostalgia

serenityatranquility

contentment

mondecsasness

000

enables multi-level tracing of affective states for each open-vocabulary emotion, facilitating more 166 accurate and interpretable emotion tagging. 167

Subsequently, leveraging POM, the ensemble-based majority voting mechanism selects reliable 168 open-vocabulary emotion labels for images [Fig. 2 (e)]. First, open-vocabulary emotions extracted 169 from multiple MLLMs are mapped to secondary emotion categories, with model voting determining 170 the quota of open-vocabulary emotions for each secondary category. Second, within each secondary 171 category, the open-vocabulary candidates are ranked based on their frequencies, and the top-ranked 172 ones are selected according to the allocated quota. This process guarantees label reliability while pre173 serving the open-vocabulary nature of the emotion labels, thereby achieving synergistic optimization 174 between annotation precision and semantic coverage. 175

176

177

178

179

180

181

182

183

184

185

186

4.3 Emotional Statement Construction

Building upon the assigned emotion labels, we construct both correct and incorrect emotion-centric statements from four dimensions: sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Its procedure is illustrated in Fig. 3.

The construction pipeline initiates with prototype statement generation [Fig. 3 (a)]. For each emotion label, we trace it back to the MLLM that extracts it, prompting the MLLM to generate three prototype statements: [1] . prototype interpretation of the emotion by inquiring about the cause of the emotion (#5); [2] . prototype context that aligns with the emotion by requesting a background story (#6); and [3] . prototype character who would experience the emotion by questioning the possible identity of the viewer (#7). From the dataset perspective, the prototype generation is distributed across multiple MLLMs, ensuring diversity in the subsequent statement construction.

Sentiment Polarity Statement Construction [Fig. 3 (b)]: Under the guidance of POM, we derive 187 sentiment polarity by mapping the labels to primary emotions. Each image's sentiment polarity is 188 classified into three mutually exclusive categories: 1). Fully Positive when all labels reside in the 189 positive spectrum; 2). Fully Negative when all labels reside in the negative spectrum; 3). Mixed 190 when positive and negative labels both exist. Next, the ground truth correctness of three predefined 191 statements on sentiment polarity (#8,9,10) is determined accordingly. 192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

Determination

Solidarity

Anger

Hope

Frustration

InternVL2.5

InternVL2.5

Qwen2-VL

Llava-Next

Idefics3

Generate Prototype Statements

Instruction:

Explain the cause of [

Emotion

].

(#5)

MLLM:

The image evokes hope through the presence of a diverse group of

people united in a peaceful protest, advocating for positive change and

expressing their opinions on important social issues.

Statements of [Emotion] are Generated by the Source [MLLM]

Instruction:

Imagine a context that would evoke [

Emotion

].

(#6)

MLLM:

In a bustling city, people gather to peacefully protest for a better

future, united in their quest for change and a brighter tomorrow.

Trace Back to Source MLLM

Instruction:

Image a character who feels [

Emotion

].

(#7)

MLLM:

A middle-aged female teacher, wearing a red cardigan and glasses,

recognizing the importance of advocating for healthcare and education.

Interpretation

Context

Character

[Emotion]

[MLLM]

Local MLLMs inferences

Interpretation

Context

Character

Hope:

Interpretation

Context

Character

Anger:

Interpretation

Context

Character

Solidarity:

Interpretation

Context

Character

Frustration:

Interpretation

Context

Character

Determination:

Prototype Statements

Prototype Statement Generation

Sentiment Polarity Statement Construction

Hope

POM

Map Open-vocabulary Emotions

to Sentiment Polarities

Anger

Solidarity

Frustration

Determination

Positive

Negative

Positive

Negative

Positive

Classify the Sentiment Polarity as Mixed

Emotions (Can be Both Positive and Negative)

Claim of Fully Positive Image.

(#8)

Statements on Sentiment Polarity

Sentiment Polarity

Fully Positive

Fully Negative

Mixed

√

Claim of Fully Negative Image.

(#9)

Claim of Mixed Image.

(#10)

Label

Incorrect

Incorrect

Correct

Guide

Emotion Interpretation Statement Construction

Statement on Emotion Interpretation

Label

Raw Statement of Emotion

Interpretation for Each Label.

Prototype Statements

Raw Statement of Emotion

Interpretation for Each Label.

Raw Statement of Emotion

Interpretation

Correct Statement Construction

Correct

Incorrect Statement Construction

Intra-image Disruption

Inter-image Disruption

Image Dataset

Query

Current Image

Interpretation

Interpretation

Substitute Current

Image's Interpretation

Emotionally

Similar but Visually

Dissimilar

Visually Similar

but Emotionally

Dissimilar

Interpretation

Emotions with Mismatch

Interpretation from Other Images

Emotion Statements with Mismatch

Interpretation from Other Images

Emotion Statement with Mismatch

Interpretation from Other Images

Incorrect

Determination

Frustration

Interpretation

Interpretation

Exchange Interpretations

between Positive and

Negative Labels

Emotions with Mismatch

Interpretation from Other Images

Emotion Statements with Mismatch

Interpretation from Other Images

Emotion Statement with Mismatch

Interpretation from the Current Image

Incorrect

Scene Context Statement Construction

Correct Statement Construction

Statements on Scene Context

Label

Templated Context Statement

Templated Context Statement

Correct

Incorrect

Incorrect

Incorrect Statement Construction

In the Context of

[Context], the Image is

Likely to Evoke [Emotion].

Context with Polarity-flipped Emotion

Template

(#12)

Hope

Context

between Positive and

Negative Labels

Anger

Context

Template

Context

Determination

Context

Determination

Context

Determination

Templated Context Statement

with Mismatched Context

Templated Context Statement

with Mismatched Context

Templated Context Statement

with Mismatched Context

Templated Context Statement

with Polarity-flipped Emotion

Hope

Context

Anger

Context

Anger

Context

Template

Zeal

Context

Flip

Polarity

Perception Subjectivity Statement Construction

Statements on

Perception Subjectivity

Label

Correct

Incorrect

Correct Statement Construction

Upon Viewing the Image,

[Role] is More Inclined to

Feel [Emotion1] Compared

to [Emotion2].

Incorrect Statement Construction

Template

(#13)

Correct Templated

Subjectivity Statement

Anger

Character

Correct Templated

Subjectivity Statement

Correct Templated

Subjectivity Statement

Correct Templated

Subjectivity Statement

Correct Templated

Subjectivity Statement

Incorrect Templated

Subjectivity Statement

[Emotion2]

Zeal

Solidarity

Construction of Emotion Pairs with Preference

Retrieve from

Opposite Polarity

Character

Anger

Emotion and the

Corresponding

Statement

Template

Non-

preferred

Preferred

Non-

preferred

Preferred

Non-preferred

[Emotion1]

Character

[Role

]

[Emotion1]

Template

Preferred

Non-preferred

[Emotion2]

Character

[Role]

Template

(#11)

Context

Determination

Interpretation

Determination

(a).

(b).

(c).

(d).

(e).

SforJobs

MIBASNOON

NEWWAR

Healthcarc

Schols

NotWar!

口

I1

口

I1

一

Emotion Interpretation Statement Construction [Fig. 3 (c)]: We combine a prototype interpretation with an emotional state (#11) to construct an emotion interpretation statement. Matched labels and prototype statements are assigned as correct statements, while unmatched ones are considered incorrect. To construct mismatched pairs, we design two strategies: inter-image and intra-image disruption. The former retrieves two images from the dataset - one exhibiting visual similarity but emotional dissimilarity to test whether MLLMs can comprehend the affective gap [54], and the other demonstrating emotional similarity but visual dissimilarity to evaluate whether MLLMs can identify the emotional triggers in images. Visual similarity is quantified using CLIP-score [55], whereas emotional similarity is determined by the correspondence at tertiary emotions of POM. The latter strategy exchanges interpretations between emotion labels of contrasting polarity within identical images, aiming to assess whether MLLMs can establish precise causal linkages between emotional triggers and specific emotions.

Scene Context Statement Construction [Fig. 3 (d)]: We combine a prototype context and an emotional conclusion (#12) to form a scene context statement, where the construction of correct statements mirrors the previous case. The strategy for incorrect statements differs slightly: in addition to exchanging prototype contexts between emotion labels of contrasting polarity within identical images, we devise a flip polarity operation. Specifically, the emotional label is substituted with a tertiary emotion randomly sampled from the opposing polarity spectrum in POM.

Perception Subjectivity Statement Construction [Fig. 3 (e)]: We combine a prototype character and their inclination toward one of two candidate emotions (#13) to form a perception subjectivity statement. We first create emotion pairs with preference bias. For each prototype character, its preferred emotion is the corresponding emotion label. The non-preferred emotion is derived by either retrieving opposite polarity emotions within the image or the flip polarity operation for the preferred emotion. Next, correct and incorrect statements are constructed by configuring emotion pairs into canonical or anomalous orders, respectively.

Anticipation

Satisfaction

Joy

Pleasure

Excitement

Label: Incorrect

The chaotic and disorganized arrangement of various

food packaging can evoke a sense of anxiety due to the

overwhelming number of items and cluttered space.

Emotion

Emotion-centric Statement

Joy

Excitement

Nostalgia

Happiness

Contentment

Warmth

Sadness

Fear

Curiosity

Playfulness

Anxiety

Celebration

Anticipation

Serenity

Awe

Concern

Wonder

Love

Comfort

Freedom

Unease

Joy

40.3%

28.5%

26.5%

23.8%

21.5%

21.3%

18.2%

15.7%

15.3%

14.8%

10.9%

9.8%

9.0%

8.4%

8.3%

8.1%

8.1%

8.0%

6.9%

6.9%

6.9%

6.8%

Love

17.2%

Surprise

7.7%

Sadness

17.7%

Fear

13.4%

Angry

3.7%

(a)

(b)

Sentiment

Polarity

12.6%

37.5%

24.9%

25.0%

Emotion

Interpretation

Scene

Context

Perception

Subjectivity

Correct

54.1%

Incorrect

45.9%

0

10

20

30

40

50

60

70

80

90

100

110

120

0.1%

3.8% 3.0% 2.8% 3.3% 2.1% 0.6%

6.3%

11.5%

15.8%

19.6%

31.2%

Average Length: 37.0 Words

Length Distribution

(c)

4.4 Evaluation Benchmarks 218

In this process, INSETS begins with an image dataset, relying primarily on local MLLM inferences to 219 assign reliable open-vocabulary emotion labels and construct meaningful emotion-centric statements. 220 For open-vocabulary emotion extraction and prototype statement generation, we employ nine recent 221 popular MLLMs with impressive performance [65]. As reported in Table 1, the final assigned emotion 222 labels and prototype statements are evenly sourced across the MLLMs, ensuring diversity in the 223 constructed benchmark. Given the high quality of Emoset [19], we select it as the image source. 224 From a subset of 17,716 images, we construct 462K emotion-centric statements, referred to as 225 INSETS-462K. 226

227

228

229

230

231

232

233

234

To further enhance accuracy, we sample 3,164 distinct image-statement pairs from INSETS-462K for human validation. Among them, annotators judge 218 pairs (6.9%) as inaccurate, 2,868 pairs (90.6%) as accurate, and 79 pairs (2.5%) as ambiguous. Accurate pairs are retained, incorrect labels are corrected, and ambiguous ones are discarded, yielding a high-quality subset of 3,086 distinct image-statement pairs, which we name INSETS-3K. Although the open-vocabulary emotion labels are not required in the ESJ task, we retain them in both INSETS-462K and INSETS-3K to enhance interpretability and facilitate future development of the benchmarks. The statistics of emotion labels and statements are reported in Table 2.

235

236

5 Analysis and Evaluation

5.1 Details of INSETS-3K

To gain deeper insights into the properties of INSETS-3K, we provide detailed statistics in Fig. 4. A 237 sample is shown in Fig. 4 (a), which includes five emotion labels and an emotion-centric statement. 238 Fig. 4 (b) illustrates the distribution of popular emotion labels, where the most frequent labels 239 include Joy, Excitement, Nostalgia, Happiness, and Contentment. When mapped to the primary 240 emotions in Parrott's model, Joy is the most dominant category (40.3%), followed by Sadness (17.7%), 241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

Love (17.2%), Fear (13.4%), Surprise (7.7%), and Anger (3.7%). This distribution suggests a rich representation of emotions, ensuring coverage of diverse affective states. Fig. 4 (c) presents statistics on the statements, which exhibit a natural length distribution and a well-balanced distribution across the four evaluation dimensions as well as correct/incorrect labels.

5.2 Evaluation Preparations

We evaluate MLLMs through the ESJ task. Specifically, we provide each MLLM with an imagestatement pair and prompt it to determine the correctness of the statement. The prompt is formulated as: 'Based on the provided image and emotional statement, please determine whether the statement aligns with the content of the image. If it does, respond with Correct . If it does not, respond with Incorrect . ' Each image-statement pair is queried three times per MLLM, and the most frequent response is selected as the final decision. After collecting responses for all images, we adopt accuracy as the primary evaluation metric. As identified in prior work [73], some MLLMs may exhibit a strong bias toward either positive or negative responses, which may compromise accuracy-based evaluation validity. To mitigate this, we introduce two diagnostic metrics: Positive Ratio calculates the proportion of positive responses out of all responses; Give-up Ratio measures the proportion of cases where the MLLM neither provides a positive nor a negative response.

To conduct a comprehensive evaluation, we adopt a diverse range of MLLMs, including both opensource and closed-source ones. Besides the MLLMs used to construct the benchmarks, we also incorporate the following MLLMs: BLIP-2 [66], InstructBLIP [67], Otter [68], Deepseek-VL [69], Paligemma [70], MiniCPM [71], Qwen2.5-VL [72], GPT4o-mini, and GPT4o [53]. All experiments are conducted on NVIDIA GeForce RTX 4090 GPUs.

5.3 Results and Findings

The evaluation results on INSETS-3K are reported in Table 3. Overall, more recent MLLMs outperform earlier MLLMs. The latter suffers from severe response biases or instructional failures. This indicates that advancements in general visual tasks also enhance emotional perception. However, no MLLM achieves optimal performance across all tasks. While InternVL2.5 and GPT4o demonstrate superior performance in identifying the basic emotional tone and performing affective reasoning, they exhibit comparative deficiencies in contextual and personalized emotion prediction. This underscores the multifaceted challenges of visual emotion understanding. Further evaluation on INSETS-462K is reported in Appendix Table 6.

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

Comparison with Human Performance : We sample 300 image-statement pairs from INSETS-3K and evaluate 25 human participants alongside leading MLLMs. As shown in Table 4, humans achieve nearperfect accuracy. In contrast, MLLMs exhibit notable performance gaps, particularly in determining sentiment polarity and understanding perception subjectivity. Given their comparatively high accuracy on emotion interpretation statements, we suggest that the affective reasoning from emotional clues to emotional states is essential for MLLMs to perceive emotions. Regarding perception subjectivity, we speculate that MLLMs may lack sufficient awareness of individual differences. Overall, ESJ is a fundamental task format, and the performance gap between MLLMs and humans highlights the considerable potential for improvement in MLLMs' visual emotional intelligence.

6 Limitations and Discussion

Several limitations in this work can be further improved. First , our evaluation primarily focuses on MLLMs with parameters under 10B due to computational constraints imposed by hardware. Although this covers practical deployment scenarios, it excludes larger-scale open-source MLLMs that may exhibit superior visual emotion perception capabilities. Second , the current implementation is limited to monolingual evaluation. Yet we highlight that adapting INSETS for multilingual construction would require relatively limited engineering effort, primarily involving adjustments in MLLM selection, prompt design, and template configuration. Moreover, while we explored basic reasoning strategies, advanced strategies such as in-context learning and chain-of-thought prompting remained underexplored. Systematically investigating these techniques can potentially reveal deeper insights into MLLMs' visual emotion perception mechanisms. Third , due to the lack of human validation, the INSETS-462K benchmarks inevitably incorporate certain noises and inaccuracies. Nevertheless, the validation process of the INSETS-3K benchmark suggests that over 85% imagestatement pairs retain high accuracy. This can be attributed to our efforts in ensuring reliability, including the ensemble-based majority voting mechanism and necessary human interventions.

Future work includes expanding MLLM coverage, extending INSETS to support multilingual construction and further refinement of the human-AI collaborative annotation process.

7 Conclusion

In this paper, we propose the ESJ task and a complemented automated pipeline, INSETS, to advance open-vocabulary, multifaceted, and scalable visual emotion evaluation in MLLMs. Through them, we provide a nuanced and comprehensive evaluation framework that covers sentiment polarity, emotion interpretation, scene context, and perception subjectivity. Our evaluation reveals that while MLLMs exhibit certain capabilities in visual emotion perception, they still lag behind humans, highlighting the necessity of developing emotion-oriented training objectives. We hope that INSETS-3K and INSETS-462K can serve as reliable benchmarks to advance future research, fostering the development of MLLMs with improved emotional reasoning and understanding.

307

308

309

310

311

312

References

N. S. Schutte, J. M. Malouff, C. Bobik, T. D. Coston, C. Greeson, C. Jedlicka, E. Rhodes, and G. Wendorf, 'Emotional intelligence and interpersonal relations,' The Journal of Social Psychology , vol. 141, no. 4, pp. 523-536, 2001.

J. Yang, J. Li, L. Li, X. Wang, and X. Gao, 'A circular-structured representation for visual emotion distribution learning,' in CVPR , 2021, pp. 4237-4246.

L. Zhang, S. Wang, and B. Liu, 'Deep learning for sentiment analysis: A survey,' WIREs Data 313 Mining Knowl. Discov. , vol. 8, no. 4, 2018. 314

S. Zhao, X. Yao, J. Yang, G. Jia, G. Ding, T. Chua, B. W. Schuller, and K. Keutzer, 'Affective 315 image content analysis: Two decades review and new perspectives,' Trans. Pattern Anal. Mach. 316 Intell. , vol. 44, no. 10, pp. 6729-6751, 2022. 317

S. Zhao, H. Yao, Y. Gao, R. Ji, W. Xie, X. Jiang, and T. Chua, 'Predicting personalized emotion 318 perceptions of social images,' in MM , 2016, pp. 1385-1394. 319

A. Shukla, S. S. Gullapuram, H. Katti, M. S. Kankanhalli, S. Winkler, and R. Subramanian, 320 'Recognition of advertisement emotions with application to computational advertising,' Trans. 321 Affect. Comput. , vol. 13, no. 2, pp. 781-792, 2022. 322

G. N. Yannakakis, 'Enhancing health care via affective computing,' 2018. 323

S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, 'A survey on multimodal large 324 language models,' 2023. 325

H. Liu, C. Li, Q. Wu, and Y. J. Lee, 'Visual instruction tuning,' in NeurIPS , 2023. 326

Z. Yang, L. Li, K. Lin, J. Wang, C. Lin, Z. Liu, and L. Wang, 'The dawn of lmms: Preliminary 327 explorations with gpt-4v(ision),' CoRR , vol. abs/2309.17421, 2023. 328

H. Xie, C. Peng, Y. Tseng, H. Chen, C. Hsu, H. Shuai, and W. Cheng, 'Emovit: Revolutionizing 329 emotion insights with visual instruction tuning,' in CVPR , 2024, pp. 26 586-26 595. 330

S. Bhattacharyya and J. Z. Wang, 'Evaluating vision-language models for emotion recognition,' 331 arXiv preprint arXiv:2502.05660 , 2025. 332

Z. Lian, H. Sun, L. Sun, J. Yi, B. Liu, and J. Tao, 'Affectgpt: Dataset and framework for 333 explainable multimodal emotion recognition,' CoRR , vol. abs/2407.07653, 2024. 334

Z. Cheng, Z. Cheng, J. He, K. Wang, Y. Lin, Z. Lian, X. Peng, and A. G. Hauptmann, 'Emotion335 llama: Multimodal emotion recognition and reasoning with instruction tuning,' in NeurIPS , 336 2024. 337

J. Machajdik and A. Hanbury, 'Affective image classification using features inspired by psy338 chology and art theory,' in MM , 2010, pp. 83-92. 339

Q. You, J. Luo, H. Jin, and J. Yang, 'Building a large scale dataset for image emotion recognition: 340 The fine print and the benchmark,' in AAAI , 2016, pp. 308-314. 341

L. Vadicamo, F. Carrara, A. Cimino, S. Cresci, F. Dell'Orletta, F. Falchi, and M. Tesconi, 342 'Cross-media learning for image sentiment analysis in the wild,' in ICCV Workshops , 2017, pp. 343 308-317. 344

R. Panda, J. Zhang, H. Li, J. Lee, X. Lu, and A. K. Roy-Chowdhury, 'Contemplating visual 345 emotions: Understanding and overcoming dataset bias,' in ECCV , vol. 11206, 2018, pp. 594346 612. 347

J. Yang, Q. Huang, T. Ding, D. Lischinski, D. Cohen-Or, and H. Huang, 'Emoset: A large-scale 348 visual emotion dataset with rich attributes,' in ICCV , 2023, pp. 20 326-20 337. 349

R. Kosti, J. M. Álvarez, A. Recasens, and À. Lapedriza, 'EMOTIC: emotions in context dataset,' 350 in CVPR Workshops , 2017, pp. 2309-2317. 351

Z. Wei, J. Zhang, Z. Lin, J.-Y. Lee, N. Balasubramanian, M. Hoai, and D. Samaras, 'Learning 352 visual emotion representations from web data,' in CVPR , 2020, pp. 13 106-13 115. 353

P. Achlioptas, M. Ovsjanikov, K. Haydarov, M. Elhoseiny, and L. J. Guibas, 'Artemis: Affective 354 language for visual art,' in CVPR , 2021, pp. 11 569-11 579. 355

P. Achlioptas, M. Ovsjanikov, L. J. Guibas, and S. Tulyakov, 'Affection: Learning affective 356 explanations for real-world visual data,' in CVPR , 2023, pp. 6641-6651. 357

E. Authors, 'Eibench: Assessing the emotion interpretation ability of vision large language 358 models,' https://github.com/Lum1104/EIBench, 2024. 359

S. Zhao, H. Yao, Y. Gao, R. Ji, W. Xie, X. Jiang, and T. Chua, 'Predicting personalized emotion 360 perceptions of social images,' in MM , 2016, pp. 1385-1394. 361

J. Yang, M. Sun, and X. Sun, 'Learning visual sentiment distributions via augmented conditional 362 probability neural network,' in AAAI , 2017, pp. 224-230. 363

364

365

S. Zhao, X. Hong, J. Yang, Y. Zhao, and G. Ding, 'Toward label-efficient emotion and sentiment analysis,' Proceedings of the IEEE , vol. 111, no. 10, pp. 1159-1197, 2023.

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

S. Li, W. Deng, and J. Du, 'Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild,' in CVPR , 2017, pp. 2584-2593.

H. Schlosberg, 'Three dimensions of emotion.' Psychological review , vol. 61, no. 2, p. 81, 1954.

J. A. Mikels, B. L. Fredrickson, G. R. Larkin, C. M. Lindberg, S. J. Maglio, and P. A. ReuterLorenz, 'Emotional category data on images from the international affective picture system,' Behavior Research Methods , vol. 37, pp. 626-630, 2005.

L. Xu, Z. Wang, B. Wu, and S. Lui, 'MDAN: multi-level dependent attention network for visual emotion analysis,' in CVPR . IEEE, 2022, pp. 9469-9478.

G. Jia and J. Yang, 'S 2 -ver: Semi-supervised visual emotion recognition,' in ECCV , ser. Lecture Notes in Computer Science, vol. 13697, 2022, pp. 493-509.

T. Feng, J. Liu, and J. Yang, 'Probing sentiment-oriented pretraining inspired by human sentiment perception mechanism,' in CVPR , 2023, pp. 2850-2860.

J. Li and W. Lu, 'A survey on benchmarks of multimodal large language models,' CoRR , vol. abs/2408.08632, 2024.

S. Tong, Z. Liu, Y. Zhai, Y. Ma, Y. LeCun, and S. Xie, 'Eyes wide shut? exploring the visual shortcomings of multimodal llms,' in CVPR . IEEE, 2024, pp. 9568-9578.

Y. Maruyama, 'The conditions of artificial general intelligence: Logic, autonomy, resilience, integrity, morality, emotion, embodiment, and embeddedness,' in AGI , ser. Lecture Notes in Computer Science, vol. 12177, 2020, pp. 242-251.

B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan, 'Seed-bench: Benchmarking multimodal 386 llms with generative comprehension,' CoRR , vol. abs/2307.16125, 2023. 387

388

389

J. Nie, G. Zhang, W. An, Y. Tan, A. C. Kot, and S. Lu, 'Mmrel: A relation understanding dataset and benchmark in the MLLM era,' CoRR , vol. abs/2406.09121, 2024.

390

391

392

393

C. Zhang, F. Gao, B. Jia, Y. Zhu, and S. Zhu, 'RA VEN: A dataset for relational and analogical visual reasoning,' in CVPR , 2019, pp. 5317-5327.

Y. Qian, H. Zhang, Y. Yang, and Z. Gan, 'How easy is it to fool your multimodal llms? an empirical analysis on deceptive prompts,' CoRR , vol. abs/2402.13220, 2024.

T. Guan, F. Liu, X. Wu, R. Xian, Z. Li, and et al., 'Hallusionbench: An advanced diagnostic 394 suite for entangled language hallucination and visual illusion in large vision-language models,' 395 in CVPR , 2024, pp. 14 375-14 385. 396

P. Chen, J. Ye, G. Wang, Y. Li, Z. Deng, and et al., 'Gmai-mmbench: A comprehensive 397 multimodal evaluation benchmark towards general medical AI,' in NeurIPS , 2024. 398

T. Qian, J. Chen, L. Zhuo, Y. Jiao, and Y. Jiang, 'Nuscenes-qa: A multi-modal visual question 399 answering benchmark for autonomous driving scenario,' in AAAI , 2024, pp. 4542-4550. 400

Y. Li, A. Dao, W. Bao, Z. Tan, T. Chen, H. Liu, and Y. Kong, 'Facial affective behavior analysis 401 with instruction tuning,' in ECCV , ser. Lecture Notes in Computer Science, vol. 15076, 2024, 402 pp. 165-186. 403

X. Yang, W. Wu, S. Feng, M. Wang, D. Wang, and et al., 'Mm-bigbench: Evaluating multimodal 404 models on multimodal content comprehension tasks,' 2023. 405

S. Sabour, S. Liu, Z. Zhang, J. M. Liu, J. Zhou, and et al., 'Emobench: Evaluating the emotional 406 intelligence of large language models,' in ACL , 2024, pp. 5986-6004. 407

G. Stemmler and J. Wacker, 'Personality, emotion, and individual differences in physiological 408 responses,' Biological psychology , vol. 84, no. 3, pp. 541-551, 2010. 409

L. F. Barrett, B. Mesquita, and M. Gendron, 'Context in emotion perception,' Current Directions 410 in Psychological Science , vol. 20, no. 5, pp. 286-290, 2011. 411

P. Shaver, J. Schwartz, D. Kirson, and C. O'connor, 'Emotion knowledge: Further exploration 412 of a prototype approach.' Journal of personality and social psychology , vol. 52, no. 6, p. 1061, 413 1987. 414

W. G. Parrott, Emotions in Social Psychology: Essential Readings . Psychology Press, 2001. 415

Z. Bai, P. Wang, T. Xiao, T. He, Z. Han, Z. Zhang, and M. Z. Shou, 'Hallucination of multimodal 416 large language models: A survey,' CoRR , vol. abs/2404.18930, 2024. 417

418

419

420

X. Liu, Y. Zhu, J. Gu, Y. Lan, C. Yang, and Y. Qiao, 'Mm-safetybench: A benchmark for safety evaluation of multimodal large language models,' in ECCV , ser. Lecture Notes in Computer Science, vol. 15114, 2024, pp. 386-403.

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

OpenAI, 'GPT-4 technical report,' CoRR , vol. abs/2303.08774, 2023.

A. Hanjalic, 'Extracting moods from pictures and sounds: Towards truly personalized tv,' IEEE Signal Processing Magazine , vol. 23, no. 2, pp. 90-100, 2006.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, and et. al., 'Learning transferable visual models from natural language supervision,' in ICML , ser. Proceedings of Machine Learning Research, vol. 139, 2021, pp. 8748-8763.

H. Liu, C. Li, Y . Li, and Y . J. Lee, 'Improved baselines with visual instruction tuning,' in CVPR , 2024, pp. 26 286-26 296.

D. Jiang, X. He, H. Zeng, C. Wei, M. Ku, Q. Liu, and W. Chen, 'MANTIS: interleaved multi-image instruction tuning,' CoRR , vol. abs/2405.01483, 2024.

J. Ye, H. Xu, H. Liu, A. Hu, M. Yan, and et al., 'mplug-owl3: Towards long image-sequence understanding in multi-modal large language models,' CoRR , vol. abs/2408.04840, 2024.

H. Laurençon, A. Marafioti, V. Sanh, and L. Tronchon, 'Building and better understanding vision-language models: Insights and future directions,' CoRR , vol. abs/2408.12637, 2024.

M. I. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, and et al., 'Phi-3 technical report: A highly capable language model locally on your phone,' CoRR , vol. abs/2404.14219, 2024.

P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, and et al., 'Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution,' CoRR , vol. abs/2409.12191, 2024.

A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, and et al., 'The llama 3 herd of models,' CoRR , vol. abs/2407.21783, 2024.

M. Deitke, C. Clark, S. Lee, R. Tripathi, Y. Yang, and et al., 'Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models,' CoRR , vol. abs/2409.17146, 2024.

Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, and et al., 'Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling,' CoRR , vol. abs/2412.05271, 2024.

O. Contributors, 'Opencompass: A universal evaluation platform for foundation models,' https://github.com/open-compass/opencompass, 2023.

J. Li, D. Li, S. Savarese, and S. C. H. Hoi, 'BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models,' in ICML , ser. Proceedings of Machine Learning Research, vol. 202, 2023, pp. 19 730-19 742.

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, and et al., 'Instructblip: Towards general-purpose vision-language models with instruction tuning,' in NeurIPS , 2023.

B. Li, Y. Zhang, L. Chen, J. Wang, J. Yang, and Z. Liu, 'Otter: A multi-modal model with in-context instruction tuning,' CoRR , vol. abs/2305.03726, 2023.

H. Lu, W. Liu, B. Zhang, B. Wang, K. Dong, and et al., 'Deepseek-vl: Towards real-world 456 vision-language understanding,' CoRR , vol. abs/2403.05525, 2024. 457

L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, and et al., 'Paligemma: A versatile 458 3b VLM for transfer,' CoRR , vol. abs/2407.07726, 2024. 459

Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, and et al., 'Minicpm-v: A GPT-4V level MLLM on 460 your phone,' CoRR , vol. abs/2408.01800, 2024. 461

Q. Team, 'Qwen2.5-vl,' https://qwenlm.github.io/blog/qwen2.5-vl/, 2025. 462

Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen, 'Evaluating object hallucination in 463 large vision-language models,' in EMNLP , 2023, pp. 292-305. 464

A Links of MLLMs 465

B Prompts and Statement Templates 499

The prompts and statement templates used in the INSETS pipeline are presented in Table 5. 500

501

502

503

C Details of Parrott's Hierarchical Model

We present the complete emotion taxonomy of Parrott's hierarchical model in Table 7.

D Further evaluation on INSETS-462K

Despite dataset noise, obvious task-specific disparities persist: MiniCPM surpasses Qwen2.5-VL 504 in judging scene context statements (86.8% vs. 76.4%) but trails in judging emotion interpretation 505 statements (77.1% vs. 81.3%). This finding reinforces the need for emotion-oriented training 506 objectives in the future MLLM developments. 507

E Visualization of INSETS-3K 508

More samples from INSETS-3K are visualized in Fig. 5, Fig. 6, Fig. 7, Fig. 8, Fig. 9, Fig. 10, Fig. 11, 509 Fig. 12. 510

Warmth

Joy

Amusement

Contentment

Relaxation

Label: Correct

Upon viewing this image, observers, despite various

individual or contextual factors, are most likely to

experience positive emotions.

Emotion

Sentiment Polarity Statement

Concern

Disgust

Anger

Sadness

Label: Correct

Upon viewing this image, observers, despite various

individual or contextual factors, are most likely to

experience negative emotions.

Emotion

Sentiment Polarity Statement

Empathy

Anger

Determination

Urgency

Concern

Label: Correct

Upon viewing this image, observers are equally likely to

experience either positive or negative emotions,

depending on individual or contextual factors.

Emotion

Sentiment Polarity Statement

Solidarity

STOP THE

BANFRACKING

PROPAGANDA!

FRACKDOWN

#GLOBAL

LBUAU/BAOKDO

Discomfort

Dread

Confusion

Horror

Anxiety

Label: Incorrect

Upon viewing this image, observers, despite various

individual or contextual factors, are most likely to

experience positive emotions.

Emotion

Sentiment Polarity Statement

Warm

Comfort

Contentment

Label: Incorrect

Upon viewing this image, observers, despite various

individual or contextual factors, are most likely to

experience negative emotions.

Emotion

Sentiment Polarity Statement

Excitement

Empowerment

Confidence

Enthusiasm

Passion

Label: Incorrect

Upon viewing this image, observers are equally likely to

experience either positive or negative emotions,

depending on individual or contextual factors.

Emotion

Sentiment Polarity Statement

Joy

Fear

Excitement

Confidence

Happiness

Pride

Label: Correct

The image might evoke confidence because the

individuals are dressed in recognizable uniforms,

implying a sense of belonging and success. The setting

and professional attire can suggest achievement and

involvement in meaningful activities.

Emotion

Emotion Interpretation Statement

Thrill

Awe

Relaxation

Excitement

Freedom

Label: Correct

The image of a surfer riding a wave in the ocean,

surrounded by the calming blue water and white foam,

can evoke a sense of relaxation in viewers. The surfer's

focused posture and the gentle motion of the wave

create a peaceful and serene atmosphere, which can be

calming and soothing to observe.

Emotion

Emotion Interpretation Statement

Fear

Anxiety

Awe

Label: Correct

The image might evoke anxiety in viewers due to the

snake's coiled position and the close proximity to the

rocks, which could be perceived as a threatening or

dangerous situation.

Emotion

Emotion Interpretation Statement

Courage

TECHNOLOGY

CALIFORNARER

Excitement

Passion

Determination

Joy

Label: Incorrect

The accordion, a traditional and often vintage musical

instrument, may evoke nostalgia in viewers due to its

association with past cultural and social experiences.

Emotion

Emotion Interpretation Statement

Excitement

Amusement

Joy

Label: Incorrect

The image of a smiling woman holding a glass of wine in

a beautiful outdoor setting, possibly a patio or a balcony,

might evoke happiness in viewers because it portrays a

moment of relaxation and enjoyment .

Emotion

Emotion Interpretation Statement

Fascination

Wonder

Fear

Label: Incorrect

The image of a fly might evoke wonder in viewers

because of its large, bulbous eyes and sharp, pointed

mouthparts, which could be perceived as menacing or

threatening. Additionally, the fly's proximity to the

viewer's face may create a sense of discomfort or unease,

as if it is about to land on them.

Emotion

Emotion Interpretation Statement

Serenity

Concentration

Disgust

Excitement

Joy

Enthusiasm

Label: Correct

In the context of: 'After years of struggling to find her

place in the world, Sarah finally discovered her passion

for stand-up comedy, and now she's taking the stage at

Comic-Con, beaming with pride and accomplishment',

the image is likely to evoke a sense of joy.

Emotion

Scene Context Statement

Stillness

Solitude

Wonder

Label: Correct

In the context of: 'In the vast expanse of the icy

landscape, a lone figure trudges through the snow,

leaving behind a trail of footprints that tell a silent tale

of a solitary journey', the image is likely to evoke a sense

of solitude.

Emotion

Scene Context Statement

Appreciation

Determination

Hope

Label: Correct

In the context of: 'A group of people gathered in a public

space, united by a common cause, holding up signs that

express their gratitude and celebrate a significant

milestone', the image is likely to evoke a sense of hope.

Emotion

Scene Context Statement

Gratitude

Solidarity

Peace

Melancholic

Label: Incorrect

In the context of: 'A lonely man wanders through a

desolate forest, searching for a glimpse of hope in the

midst of a bleak and unforgiving landscape', the image is

likely to evoke a sense of peace.

Emotion

Scene Context Statement

Admiration

Label: Incorrect

In the context of: 'A firefighter rushes into a burning

building to save a family of four, despite the danger', the

image is likely to evoke a sense of fear.

Emotion

Scene Context Statement

Joy

Wonder

Anticipation

Label: Incorrect

In the context of: 'A family gathered together at a fair,

laughing and sharing the excitement of the colorful

carnival rides, creating memories that will last a lifetime',

the image is likely to evoke a sense of isolation.

Emotion

Scene Context Statement

Thrill

Excitement

Fear

Bravery

Curiosity

Determination

Seriousness

Happiness

Peacefulness

Admiration

Excitement

Label: Correct

Upon viewing the image, a 35-year-old marine biologist

is more inclined to feel awe compared to misery.

Emotion

Perception Subjectivity Statement

Label: Correct

Upon viewing the image, an elderly woman who lost her

bakery in the 2010s due to the economic downturn, is

more inclined to feel sad compared to delight.

Emotion

Perception Subjectivity Statement

Label: Correct

Upon viewing the image, a young child, possibly a girl is

more inclined to feel wonder compared to nostalgia.

Emotion

Perception Subjectivity Statement

Awe

Playfulness

Nostalgia

Happiness

Excitement

Freedom

Peace

Wonder

Nostalgia

Neglect

Fear

Contemplative

Abandonment

Sadness

Contentment

Security

Love

Happiness

Label: Incorrect

Upon viewing the image, a 35-year-old female

pediatrician, who is a mother of two is more inclined to

feel revulsion compared to warmth.

Emotion

Perception Subjectivity Statement

Label: Incorrect

Upon viewing the image, a middle-aged male politician,

who is pro-war and anti-peace is more inclined to feel

solidarity compared to frustration.

Emotion

Perception Subjectivity Statement

Label: Incorrect

Upon viewing the image, a middle-aged male kayaker is

more inclined to feel fear compared to determination.

Emotion

Perception Subjectivity Statement

Warmth

Connection

Determination

Excitement

Appreciation

Thrill

Wonder

Fear

Hope

Empathy

Anger

Concern

Solidarity

Frustration

RBAN

RIDER

WAR

canmofmkeus

SAFE

AFGHANISTAN

NOW

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims include a new ESJ task for visual emotion evaluation of MLLMs and a complemented INSETS pipeline for efficient annotation. It accurately reflects the paper's contribution and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The potential limitations are discussed before the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We present the proposed annotation framework step-by-step, and all prompts and statement templates are listed in the Appendix. We will also release data and code.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide INSETS-3K benchmark in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Most of the experiments are evaluations of MLLMs. We follow their default settings provided by the model card, which are listed in the Appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We have not reported error bars. However, to ensure the reliability of results, in each evaluation trial, we queried three times per MLLM and selected the most frequent response as the final decision.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are conducted on NVIDIA GeForce RTX 4090 GPUs.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully checked the NeurIPS Code of Ethics and conformed in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The main positive societal impacts of this paper include the promotion of emotional intelligence in MLLMs. Since this paper mainly focuses on MLLMs' evaluation, there are limited negative societal impacts.

Guidelines: 716

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: This appropriately cites the original paper that produced the code package or dataset.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide details about the benchmarks we constructed. We will also provide documentation alongside the benchmarks upon releasing them.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: In benchmark construction, human interventions are performed by the authors of the paper. In human testing, volunteers are provided the same instructions as MLLMs, which are provided in the manuscript.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: Study participants are asked to judge whether emotion-centric statements are accurate in relation to images. All potential risks are disclosed to the volunteers.

Guidelines: 820

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: This paper focuses on the visual emotion evaluation of MLLMs and adopts MLLMs as tools for efficient annotations.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is proposing an evaluation benchmark for MLLMs in elicited emotion recognition. To produce the benchmark, images from EmoSet are passed to multiple MLLMs that generate annotations in terms of sentiment, elicited emotion, scene description and perception subjectivity. The captions are then constructed using the annotations and majority voting. Finally, human experts curate a subset of the image-caption sets to verify the caption accuracy and validity.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well written; the authors do a good job explaining the issue of label uncertainty in emotion recognition tasks (for humans and automated systems alike) and propose a reasonable method to address this when evaluating MLLMs. One potential issue with the use of MLLM ensemble and majority voting would be that of dataset bias from the MLLM pre-training affecting the output. It may be worth including all generated captions with the dataset, for validation purposes and so that researchers can investigate label uncertainty/bias further.\nOne thing that seems inaccurate in the text, is the context of the image being extrinsic or external (mentioned in the intro and line 126 of the method). Not sure is the scene context is extrinsic at all, as that would refer to the context at which the viewer sees the image (eg. on the news or their mobile phone). The scene context is still very much intrinsic to the image, even if sometimes it does not directly relate to the image subject (eg. the EMOTIC examples of Fig 1 (a)).\nIn terms of experiments, all of the experiments are zero-shot. It would be best to include an example of a fine-tuned model (maybe with LoRA). It would also be good to include some in-context learning experiments from the 3k human annotated subset. Both would be helpful benchmarks.\n\nQUESTIONS:\nThe method is clear so not too many questions, but few additions (specifically finetuned and in-context results) would significantly help show the importance of a task specific dataset.\n\nLIMITATIONS:\nMLLMs are trained on large amounts of publicly available, often non-curated data. As such, there is always a chance of dataset bias that can be particularly dangerous in sensitive tasks, including emotion recognition. In addition, as this is technically a benchmark for a generative task involving human emotion recognition, an ethics section should be included.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper redefines emotion evaluation for MLLMs through a new Emotion Statement Judgment (ESJ) task, where models judge whether a given emotion-related statement about an image is correct. To support this, the authors propose INSETS, a pipeline that automatically extracts open-vocabulary emotion labels using multiple MLLMs and generates emotion-centric statements across four dimensions. They release a large-scale dataset (462K samples) and a 3K subset validated by human annotators for benchmarking.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The paper introduces a novel reformulation of emotion evaluation in MLLMs via the ESJ task.\n- The proposed INSETS pipeline is described in thorough detail, covering open-vocabulary emotion extraction, taxonomy mapping, and multi-perspective statement construction.\n- The authors evaluate a broad suite of MLLMs across the ESJ (3k) benchmark, and release a human-verified subset for standardized comparison.\n\n### Weaknesses\n- The dataset relies on majority voting between existing MLLMs to generate emotion labels and statements. However, these very models are shown by the authors to have a large gap compared to humans.\n- The evaluation of a highly subjective task like emotion judgment lacks rigor without expert human annotations or deeper analysis of inter-subject variability. The annotators’ qualifications and agreement metrics are also not discussed.\n- As it stands, the work primarily offers a pipeline and benchmarking framework that is difficult to subjectively verify. \n- Given these limitations, particularly around label reliability and subjectivity, the paper falls short of the bar for acceptance in its current form.\n\nQUESTIONS:\nI am wondering if showing users the MLLM prediction and asking them to judge this evaluation is less reliable that having the human label the image first and then compare that to the MLLM result.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-29"}

comments: SUMMARY:
The paper introduces a new emotion evaluation benchmark for MLLMs. They propose a new task called ESJ which classifies whether an emotional sentence is correct relative to the input image or not. For that they create an automatic annotation pipeline to create sentences that cover aspects of sentiment polarity, emotion interpretation, scene context, and perception subjectivity. They benchmark existing MLLMs on the new ESJ task underscoring shortcomings of existing models.

STRENGTHS AND WEAKNESSES:
Strengths:
1.This paper is the first to identify key limitations in existing visual emotion evaluation frameworks for MLLMs, and addresses these shortcomings through the introduction of a novel task—Emotional Statement Judgment (ESJ).

2.The authors propose the Intelligent Visual Emotion Tagger and Statement Constructor (INSETS) pipeline, which enables efficient image annotation with diverse open-vocabulary emotion labels and generates multifaceted, emotion-centered statements while minimizing manual effort.

Weaknesses:
1.The INSETS pipeline offers only limited support for the broader task of evaluating image-based emotions in MLLMs. Framing the task as a binary judgment of whether a statement matches an image appears somewhat simplistic. A more impactful direction would be to explore how to evaluate the emotional statements generated by MLLMs themselves, rather than having MLLMs assess the correctness of predefined emotional statements.

2.The paper’s contributions are relatively modest, focusing mainly on the proposed benchmark and evaluation framework. Research on the ESJ task remains limited.

3.Too many acronums make the paper hard to read.

QUESTIONS:
Please refer to weakness

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an evaluation framework for subjective, taxonomy-rich task such as the emotion understanding task from visual context. Their core argument that conventional approaches like Emotion Classification and Emotion Interpretation are incompatible with modern MLLMs due to \"limited emotional taxonomies\" and \"rigid evaluation metrics\" is central to their thesis and is a salient one. Their proposed four dimensions of the \"ESJ\" task: Sentiment Polarity, Emotion Interpretation, Scene Context, and Perception Subjectivity is comprehensive and addresses various aspects of emotion perception. Benchmark dataset (with a human-validated subset) and results are compellinh.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper does a good job at framing the central problem: the core mismatch between existing visual emotion benchmarks and the capabilities of modern MLLMs. The introductory examples (Fig. 1 a,b) are particularly effective, clearly showing how plausible MLLM responses can be unfairly penalized by rigid, closed-vocabulary, or fixed ground-truth evaluation schemes.\n2. The solution framed as \"Emotion Statement Judgment\" instead of classification or generation allows the statements themselves to be complex and nuanced, but the eval metric to be less ambiguous relying on binary yes/no answers.\n3. Perhaps the most important, The evaluation of current MLLMs (Table 3, 4) shows that performance gap between top MLLMs like GPT-4o and humans provides a clear signal to the community that visual emotional \"intelligence\" is still an open problem. I find this insightful\n\nWeaknesses:\n1. Minor: The hierarchical model of the Parrott's theory doesn't allow for continuous mixed emotions (think arousal-valence model) but the \"perception subjectivity\" dimension address this somewhat\n\nQUESTIONS:\nMy question concerns the ground-truth generation, which effectively defines your scoring function. The INSETS pipeline anchors open-vocabulary terms to the fixed hierarchy of Parrott's model to determine the 'correctness' of a statement.\n\nIn this process, did you observe any inherent contradictions or challenging edge cases? For example, were there open-vocabulary emotions that did not fit into clean categorization within Parrott's rigid structure, or instances where an automatically generated 'incorrect' statement was, in fact, subjectively plausible? Given MLLMs can be inherently inconsistent?\n\nLIMITATIONS:\nNo major limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}