Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Authors: Yiyang Fang
Jian Liang
Wenke Huang
He Li
Kehua Su
Mang Ye
Venue: ICML 2025 spotlightposter (2025)
Keywords: Multimodal Large Language Models
Emotion Recognition
Training-Free
Primary area: deep_learning->large_language_models
TL;DR: We propose SEPM to enhance MLLMs' emotion recognition by refining classification through a two-stage inference and reducing visual redundancy, offering a scalable, resource-efficient solution.

Abstract: Multimodal large language models (MLLMs) have achieved impressive progress in tasks such as visual question answering and visual understanding, but they still face significant challenges in emotional reasoning. Current methods to enhance emotional understanding typically rely on fine-tuning or manual annotations, which are resource-intensive and limit scalability. In this work, we focus on improving the ability of MLLMs to capture emotions during the inference phase. Specifically, MLLMs encounter two main issues: they struggle to distinguish between semantically similar emotions, leading to misclassification, and they are overwhelmed by redundant or irrelevant visual information, which distracts from key emotional cues. To address these, we propose Sharpening Emotion Perception in MLLMs (SEPM), which incorporates a Confidence-Guided Coarse-to-Fine Inference framework to refine emotion classification by guiding the model through simpler tasks. Additionally, SEPM employs Focus-on-Emotion Visual Augmentation to reduce visual redundancy by directing the attention of models to relevant emotional cues in images. Experimental results demonstrate that SEPM significantly improves MLLM performance on emotion-related tasks, providing a resource-efficient and scalable solution for emotion recognition.

Catch Your Emotion:
Sharpening Emotion Perception in Multimodal Large Language Models
Yiyang Fang" !
Jian Liang" I
Wenke Huang" "
He Li
Kehua Su Mang Ye 12
Abstract
1. Introduction
Multimodal large language models (MLLMs) have gained increasing attention in various fields due to their powerful understanding and reasoning capabilities (Liu et al., 2023; Chen et al., 2024; Li et al., 2024b; Wang et al., 2024: Li et al., 2024a). In recent years, driven by the rapid advance- ment of deep learning (Huang et al., 2024; 2023), they have shown exceptional performance in traditional tasks such as visual question answering (Lee et al., 2024), visual under- standing and reasoning (Zhang et al., 2025). When it comes to the emotional domain, advanced MLLMs seem to lack the ability to accurately interpret and respond to emotional cues (Yang et al., 2024a). Although current MLLMs can generate basic emotional responses, they still struggle to fully understand and capture the complexity of emotions.
Multimodal large language models (MLLMs) have achieved impressive progress in tasks such as visual question answering and visual understand- ing, but they still face significant challenges in emotional reasoning. Current methods to enhance emotional understanding typically rely fine- tuning or manual annotations, which are resource- intensive and limit scalability. In this work, we fo- cus on improving the ability of MLLMs to capture emotions during the inference phase. Specifically, MLLMS encounter two main issues in the infer- ence stage: they struggle distinguish between semantically similar emotions, leading to misclas- sification, and they are overwhelmed by redun- dant or irrelevant visual information, which dis- tracts from key emotional cues. To address these, we propose a training-free free method named Sharp- ening Emotion Perception in MLLMs (SEPM). which incorporates Confidence- Guided to-Fine Inference framework to refine emotion classification by guiding the model through sim- pler tasks. Additionally. SEPM employs Focus- on-Emotion Visual Augmentation to reduce vi- sual redundancy by directing the attention of mod- els to relevant emotional cues in images. Exper- imental results demonstrate that SEPM signifi- cantly improves MLLM performance emotion- related tasks, providing resource-efficient and scalable solution for emotion recognition. Our code is available in https: /tttt/bb./ttttt..////ee thub. com/ fuyyyyy/SEPM.
Recently, some studies have attempted address the short- comings of MLLMs in capturing emotional cues (Xing et al., 2024; Yang et al., 2024a; Zhao et al., 2024a; Zhang et al., 2024a). These efforts can mainly be divided into fine-tuning methods and training- free methods. Regarding the former, different encoders are often introduced to interpret emo- tional information across various modalities (Cheng et al., 2024), or an additional MILM (Achiam et al., 2023) is em- ployed to generate corresponding textual descriptions (Xie et al., 2024). Fine-tuning emotion-related datasets is then carried out to enhance the model's 's ability to understand emo- tions. The additional fine-tuning in these methods incurs significant training costs, which can limit their scalability and practicality in applications. Therefore, training-fret methods serve as an alternative approach to enhance the ability of MILMs to capture emotions. Some studies guide MLLMs to pay more attention to imotion-related regions through visual prompting (Zhang et al., 2024b), but such methods require prior annotation of images, resulting in additional overhead. Building on this, we aim to enhance the ability of MLLMs to capture emotions solely during the inference stage, without relying on additional training or manual annotation. This not only significantly reduces resource consumption but also improves the efficiency and flexibility of the model in real-world applications.
-uidddee Coarse-
'Equal contribution National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan Univer- sity, Wuhan, China "Taikang Center for Life and Medical Sciences, Wuhan University, Wuhan, China. Correspondence to: Kehua Su <skh www.www...c..c..ommoom Mang Ye <yemang @ whh..cdu.cn>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
In the inference stage, MLLMs face unique challenges when handling emotion classification tasks. On the one hand,

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
address @. We guide the MLLM focus on emotional in- formation through Focus Emotion (FoE) textual prompts. The relevance of visual tokens to FoE is calculated based on the attention map obtained during the first inference. By dropping visual tokens unrelated to FoE, we reduce visual redundancy and enable the model to better capture and in- terpret emotional cues. Experimental results show that our approach significantly improves MLLM performance on emotion-related tasks without requiring additional training. The main contributions are as follows:
shich of follawing descriptions represented
ixage?
MLLM Q
Excitement
A Amsssmment
Enidon Carrgars
Key point
Witch
connected
To address inference confusion from semantically sim- ilar emotions, we propose Confidence- Guided Coarse- to-Fine Inference. The two-stage inference allows the model to focus on relatively simpler tasks at each stage, thereby improving inference accuracy.
Redundancy
Bootiae-relatea
Figure 1. Illustration of the challenges faced by Multimodal Large Language models (MLLMs) in emotion tasks. The upper half of the figure demonstrates that semantically similar emotion cate- gories are more prone to confused inference than distinctly differ- ent ones. The lower half points out that different patches of the image have varying importance when conveying emotion informa- tion, and some redundant information is present.
We propose Apcus-on-Emotionn Visual Augmentation to address redundant emotional information in images. We eliminate visual tokens unrelated to the Focus-on- Emotion prompt, improving the ability of MLLMs to detect and interpret emotions.
since MLLMS perform classification tasks based on category word meanings, semantically similar emotion categories often lead to confusion during inference, as the model strug- gles to differentiate between subtle variations in emotional cues. For example, distinguishing between "amusement" and "excitement" can be challenging for the model due to their similar emotional cues, as illustrated in Fig. 1. With- out an accurate understanding of context, the model might misclassify one as the other. On the other hand, @ the visual information conveyed when expressing emotions contains a certain degree of redundancy, which can prevent the model from effectively extracting key emotional cues. As Fig. illustrates, emotional information is often concentrated in specific local regions of an image. It's crucial to direct the model's 's focus to these regions and reduce irrelevant back- ground noise. Effectively leveraging more reliable textual input (Yang et al., 2024b) in combination with these visual cues can significantly enhance the model's 's ability to capture and interpret emotions accurately.
We conduct experiments on various emotion datasets demonstrate that our method improves the perfor- mance of MLLMs on emotion tasks, without requiring additional training or manual annotations.
2. Related work
2.1. Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) (Liu et al., 2023; Chen et al., 2024; Li et al., 2024b; Wang et al., 2024; Li et al., 2024a) have garnered significant attention for their strong reasoning capabilities (Liu al., 2025b; Jin et al., 2025; 2024) and ability to handle a variety of tasks across different modalities (Huang et al., 2025a;b; Liang et al., 2025). Most research in this field focuses on leveraging pre- trained models for general applications (Liu et al., 2024a; Zhao et al., 2024b; Han et al., 2024; Bai et al., 2025), in- cluding vasion-language tasks such as multimodal reasoning (Lu et al., 2022; Masry al., 2022; Wang et al., 2025), im- age captioning (Lin et al., 2014; Young et al., 2014), visual question answering (Hudson & Manning, 2019; Goyal et al., 2017; Singh et al., 2019; Huang et al., 2025c). MLLMS have demonstrated excellent performance in these general tasks (Bi et al., 2024), showcasing their ability to integrate and reason across multiple modalities.
In this paper, we propose Sharpening Emotion Perception in Multimodal Large Language Models (SEPM), which con- sists of two primary components. First, due to O, we design a two-stage inference framework named Confidence- Guided Coarse-to-Fine Inference. The first stage performs coarse- grained emotion classification, generating textual prompts for the next stage based on the results of positive or nega- tive emotion classification and their confidence levels. The second stage then completes fine-grin grained emotion classifi- cation based on this enhanced input. This approach allows the MLLM to focus on a relatively simpler task at each step, thereby improving the model's 's inference accuracy. Second, we introduce Focus-on- on-oonnioooo Visual Augmentation to
However, MLLMs often struggle with emotion-related tasks (Lian et al., 2025; Fang et al., 2025; Yang et al., 2025). To address this, several studies have focused on fine-tuning MLLMS using emotional datasets (Xing et al., 2024; Yang et al., 2024a: Zhao et al., 2024a; Zhang et al., 2024a). EMoVIT (Xie et al., 2024) leverages GPT-4 to generate
2

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
relevant textual descriptions, aiding models in better under- standing emotions and improving their ability to interpret emotional cues. Meanwhile, Emotion-LLaMA (Cheng et al., 2024) integrates specialized encoders that are specifically designed to capture and interpret emotional information across different modalities, thereby enhancing its capacity to understand and reason about emotions from diverse in- puts, including text, audio, and visual cues. These methods require fine-tuning, leading to high training costs. We aim to enhance the emotional recognition capabilities of MLLMs during inference, without relying on extra training, making emotional understanding more efficient and scalable.
capturing capability of MILMs by optimizing the inference process, eliminating the need for additional training.
3.2. Confidence-Guidee Coarse-to-Fine Inference (CCI)
Motivation. Considering the large number of emotion cate- gories, MLLMs performing emotion classification prone to confusion due to the influence of semantically similar emotion categories. However, requiring MLLMs to dis- tinguish between positive and negative emotions is signif- icantly easier. Further enhancing emotion classification by using positive and negative emotions as prompts can lead higher inference accuracy. Therefore, we design a two-stage inference process following Coarse-to-Fine ap- proach, which enables the model to concentrate simpler tasks during inference.
2.2. Training-Free in MLLMs
Due to the high training costs associated with MLLMs, raining-free methods have gained considerable attention as a fresh perspective (Zhu et al., 2024; Ding et al., 2024; Bi et al., 2025), particularly in tasks such as inference acceler- ation (Chen et al., 2025; Zhang et al.. 2024c; Bolya al., 2022; Choudhury et al., 2024) and mitigating MILM hallu- cinations (Kim et al., 2024b; Liu et al., 2025a; Kim et al., 2024a). Typical training-free methods involve techniques such as text prompt engincering to guide model behavior and improve outcomes without additional training. token merging to enhance processing efficiency by simplifying input data, and visual augmentation alongside multi-step in- ference to boost performance through visual enhancements and iterative processing. In emotion-related tasks, Zhang et al. (Zhang et al., 2024b) propose Set-of- Vision (SoV), method for identifying the locations of faces in images, thereby enhancing facial expression recognition in MLLMs. However, this approach brings the additional cost of image annotation and is limited to tasks that focus solely on facial features. Thus, we strive to develop a training-free method to eliminate the need for additional data modifications while effectively addressing a wider range of emotional cues.
Graise Stage. We pre-divide all emotion cate- gories into the positive emotion categories (PEC) and the negative emotion categories (NEC), and generate the follow- ing Coarse-grained Query (2c).
Coarse-grained Query:
Which of the following descriptions best represents the image?
A. Positive
B. Negative
Positive emotions include [PEC). Negative emotions include (NEC). Answer directly with the letter of the chosen option.
Then, we input this query along with the sample D into the MLLM for inference:
غ =M D);
(1)
where E represents the positive or negative emotion, and M represents the inference function of the MLLM.
Fine-Grained Stage. Based on the positive and negative emotion label È obtained from the graiied stage, we modify Fine-grained Queries (Qy) for different samples, to simplify the classification task.
3. The Proposed Method
3.1. Preliminary
The architecture of a multimodal large language model (MLLM) typically includes three components: a visual encoder f. such ViT, an LLM g, such as Vicuna and LLaMA, and a connector module yp. For a query instance, the input consists of visual image v. and a textual instruc- tion Z₁. The corresponding label is the language response y. Initially, visual features z" are extracted using (x="). Sub- sequently, the trainable projection is applied to transform z" into language embedding tokens h" = 2..2.. Meanwhile, the textual instruction is tokenized as h² Tokenize The visual and textual tokens are then combined and input into the LLM module g. which generates the language out- put ý g(hh,, AfI). In our work, we enhance the emotion-
Fine-grained Query: Which of the following descriptions best represents the image?
[PEC] [NEC]
Answer directly with the letter of the chosen option.
For samples with positive emotions, we retain only the options from the positive emotion categories, while for neg- ative emotions, we keep the negative ones. Similarly, we feed sample D and its corresponding query into the MLLM, which results in the specific emotion category E:
ع = =M D).
(2)
3

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Vision Token
Stage1
Dropping Map
Stage2
F
3
5
3
2
0
1
4
Mean
Drop
E
2
9
2
Attention
Emedion
MLLM
Vision Token
MI MLLM LM
Categery
Dropping Map
Confident?
logit
Descriptions
Focus on emotion
Prompts
Coarse-grained Query
Fine-grained Query
Figure Architecture illustration of SEPM involves two main stages. Stage performs coarse-grained emotion inference. The model applies the Cincus-on-Emotinne prompt optimization to establish the relationship between the visual token and emotions. Using the positive/negative emotions inferred by MLLM and the confidence calculated based on the model's logits output, the model refines and customizes the prompt optimization for the fine- tineggrained stage. In Stage 2, fine-grained emotion inference is performed. The model discards redundant information based on the importance of the visual token from the coarse- caasee-rained stage and leverages the optimized prompt achieve more precise emotion classification.
Confidence Assessment. What if the inference in the coarse- grained stage is incorrect? Due to the tendency of MLLMs to prefer textual information, biases in the coarse- grainnd stage can directly lead to errors in the final inference re- sult. Thus, mitigating the impact of mistakes in the coarse- grained stage model inference is a problem that needs to be urgently addressed.
3.3. Focus-on-Emotion Visual Augmentation
Motivation. Generally, MLLMs tend to prioritize textual in- formation over other modalities. To mitigate the redundancy in visual information that causes the model to overlook key emotional cues, we employ a text-based prompt to guide the optimization of visual augmentation (Liu al., 2024b), enabling the model to focus more effectively the most critical emotional cues.
We obtain the logits 2 before the model output in the coarse- grained stage and compute its probability distribution.
Focus-on-Emotion Prompts (FP). To improve the focus of MLLMs on emotional information, We introduce a straight- forward yet effective prompting technique called the Fo- cus on Emotion (FoE) prompt, which explicitly directs the MLLM prioritize emotional cues:
z = Mingiaa Qcy (Q.; D),
p = soft((()))
where Miogits represents the logits generator, and p denotes the output probability distribution. Since what we obtain is a probability distribution over the vocabulary, we find the indices corresponding the letters and B and calculate the probability variance between the options:
Please focus on emotion. [Original Query]
This method can seamlessly integrate into the current model queries, as criterion for selecting visual tokens in subse- quent steps. It is highly efficient and convenient.
(pa 112 + (ps 11)
(4)
2
Estimation of Visual Token Significance. Subsequently, we need understand the degree relevance between vi- sual tokens and emotions in order to identify which visual tokens are redundant or interfering. Therefore, it is natural to leverage the attention map (AA) from the VLM transforma- tion layers of the first stage as reference, since they already encapsulate the query results from language to vision:
where C is the confidence score, is the mean, and PAs, PB are the probabilities corresponding to letter A and B.
The confidence score indicates that the smaller the value of C, the closer the probabilities of the two options are. This suggests that the model lacks strong certainty during the reasoning process. Therefore, set threshold, denoted as where results with a confidence score lower than a are considered unreliable samples. For these, we retain all emotion categories in the fine-grained stage and add ambiguous-related descriptions reflecting the behavior from the first stage, enhancing the prompting effect.
A Mattn Qcy D),
(5)
Mattn the function that computes the attention map in the final layer of attention in the MLLM. Then we extract the FoE prompt set as anotion-relate text, leveraging the query-

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Algorithm 1 SEPM
importance. Thus, we considered that dropping redundant tokens would be an effective approach. We select the lower- ranked tokens in P to generate the dropping map R:
Input: Multimodal Large Language Models M, Crarse-Grained Query Cc. Sample
Output: Specific emotion category &
/a Stage
Coarse-Grained Inference
*/
R argmin k LBNoJ,
+ M(Qc, D)
> Eq.(1)
where argmin, (P) represents the indices of the k smallest values in P. Here, ,8 is the dropping ratio, and N, denotes the total number of visual tokens.
A + Mami D)
b Eq.(5)
PA:PD + softmax(Mazuu De; D))
(3)
/+ Confidence Asscssment //
C [(Pa - 11)² + (pD //2
Eq.(4)
For the visual input in the second stage, we discard redun- dant visual tokens based on the dropping map. The refined visual token set V is defined as:
ifC > a then
Fine-Grained Query 2
{
2 OPEC; E Positive,
0 QNEC, E = Negative.
V = {uz 15 ( No)).
(10)
else
2 < Qamii
where V fui, U2,...,, UN } represents the original set of visual tokens, with No being the total number of tokens. The dropping map R contains the indices of redundant tokens determined in the first stage.
end
/* Stage Fine-Grained Inference */
for j = 1,2, ..., Ly do
/x Emotion-Relas Renn-Relltte Sig. Estimation x/
The refined visual token set V and the textual prompts are combined into a new sample D', which is fed into the MLLM for fine- ane-ggained inference, in Eq. (2). This ap- proach enhances the focus of the model on critical emotional cues, resulting in more accurate and reliable outcomes.
Pfi.j] < A(i
b (q.
Plj] + +, Et. P(t,5)
> Eq)
end
k - [8N.l, R argmin (P) V'
(q)
{1y i $ RR j E 21, ... N.]}
(10
+ M(Q,.D"
(q.
return Fine-Grained Emotion Class E
4. Experiments
In this section, we observe the performance of SEPM from multiple perspectives, and also notice several notable issues:
dimension of textual logits and the key-dimension of the visual modality to construct the drop matrix P E R gl..xl,, LL, where L, and L. are the lengths of the text tokens (FoE) and visual tokens, respectively. The drop matrix P is defined as:
RQ1: Is there more redundant visual information in emotion-related tasks compared to general tasks? (Sec. 4.2)
RQ2: How should we choose which visual tokens to drop to improve inference accuracy? (Sec. 4.2, Sec. 4.3)
Pli, = J(i,j), E {a [x} e T),
(6)
RQ3: Is it reliable use the variance of the logits distribu- tion output by MLLMs basis for confidence? (Sec. 4.3)
jE E (v myj VV),
where Il denotes the set of all input tokens, including both textual and visual modalities, T denotes FoE prompt token set and V is visual tokens set, i.e.
RQ4: Do MLLMs exhibit biases when performing emotion classification by different prompts? (Sec. 4.3)
4.1. Experimental Setup
(T CI
(7)
Architecture. Following the multimodal large language model paradigm, we use two popular models as the foun- dation for our experiments to evaluate the effectiveness of our method: LLaVA (Liu et al., 2023) and VILA (Lin et al., 2024). Specifically, LLaVA uses a model with 7 billion parameters, while VILA employs model with 8 billion parameters as the test model.
To estimate the significance of the vision token, we compute the average score of all instruction tokens as:
1 L Pli,), j E (1,2....,NNK
Pb = T Σ
(8)
where we use P as the importance indicator for visual to- kens, with more significant tokens assigned higher values.
Datasets. We evaluate our framework on four emotion datasets, which are annotated across different scenarios and numbers of categories: Emotionó (Peng et al., 2015), EmoSet (Yang et al., 2023), WebEmo (Panda et al., 2018), and Abstract (Machajdik & Hanbury, 2010). The WebEmo
Visual Token Augmentation (VTA). For the importance indicator P obtained in Eq. (8), we intend for the model to focus more on the parts of the visual tokens with higher
5

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Table 1. Comparison with state-of-the-art on various emotion datasets. 33 is drop rate. The optimal results are denoted by boldface.
indicates an increase in accuracy for zero-shot inference with the same architecture, while . indicates decrease. Refer Sec. 4.2.
Dataset
Emotionó
EmoSet
WebEmo,
WehEmogs Abstract
Average
Class
6
25
8
LLVAA-7b
Zero-shot
48.32
52.77
25.56
15.71
27.86
34.04
Zeno-shot-ooT
48.65
51.67
22.93
15.52
26.43
33.04
SparseVLM
49,83
54.00
26.64
1,568
26,79
34,59
SEPM (3=0.1)
54.21 55.89
56.04
42.39
18.26-2.55
2
40.04.6.00
SEPM (3=0.2)
54.04-5.72
56.24
/
18.34 -2063
28.21
39.92.5.88
VILA-8b
Zero-shot
47.47
44.45
41.90
15.35
20.00
33.83
Zero-shot-Cof
44.28
43,20
41,08
13.57
21.07
32.64
SEPM (G=0.1)
51.35-3 -.355-3.88
53.11
44.30
17.36-2.01
25.71
38.37 14.54
SEPM (3=0.2)
52.02-4.55
53.11 -3.01
19.10
19.39-a.04
26.43,6643
38.01
-4.18
Table 2. Comparison of different dropping strategies in LLaVA.
ACC
ACC
3 is drop rate. Bold is the best. Refer to Sec. 4.2.
Finsdor
Feuder
Dataset
Emotion6
EmoSet
Emitiind
58
Enstlend
Drop rate
3=0.1
3=0.2
3=0.1
3=0.2
36
Random
53,20
51.85
55.99
55,91
Query-related
53.54
53.54
55.99
56.10
FoE-related 54.21
54.04
56.04
56.24
52
48
50
0.05
8.1
0.2
0.4
8.1
0.2
0.3
0.4
Table 3. Ablation study of the key components in SEPM. Bold is
B
the best. f is an increase over the baseline. Refer to See. 4.3.
Figure 3. Sensitivity analysis. The results are obtained by varying the value of the corresponding hyperparameter, while fixing the others to the values adopted the experiments. Refer to Sec. 4.3.
CCI
FP
VTA
Emotionó
EmoSet
48.32
52.77
51.68
13,98
51,52--.20
54,28-1.51
shot-CoT designs "let's think step by step" as prompt enhancement. However, compared to MILM zero-shot, the overall performance in emotion classification decreases. This suggests that simple prompt engineering does not effec- tively enhance the ability of MLLMs to capture emotions. Next, SparseVL.M primarily improves inference efficiency by discarding task-irrelevant visual tokens. Since it simi- lar our approach, we also include it for comparison here. We are pleasantly surprised to find that Sparse VLM shows slight improvement in overall performance on emotion tasks compared to zero-shot, which is in stark contrast to the general decrease in reasoning accuracy it typically expe- riences on more general tasks. This suggests that, compared to general tasks, MLLMs contain more redundant tokens in visual information when capturing emotions (RQ1). It also supports the rationale of our approach, where dropping redundant tokens helps highlight key information.
53.03
10.10
54.04-5.72
56.24 3.47
dataset includes emotion annotations at two levels of granu- larity, with 7 categories and 25 categories.
Implement Details. We utilize checkpoints from model pre-training and follow the template provided by the LLaVA model design basic question prompts for inference with- out further training. To evaluate the performance of MLLMs in solving emotion-related tasks, use answer ac- curacy (ACC) as the evaluation metric for our experiments. The confidence threshold a and drop rate 3 are set to 0.1 and 0.2 by default, respectively. All experiments are conducted on 8 NVIDIA 4090 GPUs, each with 24GB of memory.
4.2. Comparison Experiments
We conduct comparisons on multiple emotion datasets with MLLMs Zero-shot, shoro-ssoo-CoTTT (Kojima et al., 2022) and sparse (Zhang et al., 2024c). We adjust the Zero-shot- CoT prompt to better suit our task: "Let's 's think step by step, then answer with the option's 's letter from the given choices directly." We also evaluate different dropping strategies.
Finally, we compare SEPM under two different drop rates with previous methods and find that our approach outper- forms all others on each dataset. Specifically, under the LLaVA VA architecture, SEPM with a 20% drop rate improve by 17.19 compared to zero-shot, and it also shows significant improvements in other comparison groups. This indicates that SEPM greatly improves the emotion-capturing ability of MLLMs without requiring any training.
Comparison with State-of-the-ae the-art.. We compare SEPM with the zero-shot method of MLLMs with the same archi- tecture, as well as other training- free methods on MLLMs on various emotion datasets, as shown in Tab. Firstly, Zero-
Comparison of Different Dropping Strategies. We eval- uate three different dropping strategies under the LL
6

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Original
Drop 20%
Drop 40%
Original
Drop 20%
Drop 40%
Amusement
Disgust
Anger
Excitement
Awe
Fear
Contentment
Sadness
Figure 4. Visualization of dropped visual tokens for different emotions. The occluded represents dropped tokens. Refer to Sec. 4.3.
Threshold-0.1
inference inaccuracies, indicating that not all visual tokens are meaningless. Discarding useful information reduces inference accuracy, emphasizing the need for well-chosen dropping strategy. In conclusion, choosing which visual tokens to drop based on their relevance to emotion is the most reasonable and effective approach (RQ2).
1.1
1.0
0.54
0.05
0.97
3.99
0.9
0.8
4.3. Diagnostic Analysis
0.6
0.57
0.5
We conduct ablation and visualization experiments on the Emotionó and EmoSet datasets for in-depth analysis.
Figure 5. Visualization of confidence estimation. VAR is vari- ance, representing the confidence level of coarse-grained inference. Threshold is our default confidence threshold Refer to Sec. 4.3.
Key Component Analysis. We evaluate the effects of key components SEPM, including Confidence-Guidars Coarse-to-Fine Inference (CCI), (FP), and Visual Token Augmentation (VTA). As VTA de- pends other components, it not studied alone in ablation. The results are shown in Tab. 3, highlighting accu- racy changes across components. Using a single component modestly improves inference accuracy. Moreover, combin- ing CCI, FP, and VTA yields best results. These findings show the key role of components boosting performance.
Prompts
framework the Emotionó and EmoSet datasets. The first strategy is Random Drop, where the model randomly drops certain number of tokens. The second, Query-related Drop, drops tokens with the lowest scores based on their relevance to the overall query. The final strategy our method, FoE-related Drop, which drops redundant tokens prompt.
based on their relevance to the As shown in Tab. 2, FoE-related Drop outperforms the other two dropping strategies across different datasets, as this strat- egy focuses more on the relevance to emotion. In contrast, Query-related Drop may include more irrelevant informa- tion due to its broader textual reference. Notably, in the case of Random Drop, increasing the drop rate leads to more
Sensitivity Analysis. To demonstrate the robustness of SEPM, conduct sensitivity analysis of key hyperparam- eters. Focusing on overall model performance, we evaluate different settings for the loss parameters, confidence thresh- old a, and drop rate 3 on the Emotionó and EmoSet datasets.

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
90
70
13-21
50
30
10
Content
Excite
Anger
Disgust
Zero-shot (Llava)
SEPM (Llava)
Zero-shot
(vila)
SEPM
(vila)
Figure 6. Visualization of fine- tneeggrined emotion classification EmoSet. The left side is the positive emotion categories, and the right side is the negative ones. The dashed line represents the overall performance. Please see details Sec. 4.3.
time while keeping the others fixed. As shown in Fig. 3, model perfor- mance initially improves with increasing o, then declines. A low threshold fails to distinguish reliable inferences in the first stage, while high threshold misclassifies many unreliable predictions as reliable, weakening prompt en- hancement in the second stage. Additionally, performance steadily declines as B increases. A small drop rate reduces emotional redundancy and slightly boosts performance, but a high drop rate removes important visual information, de- grading final results. Based on these observations, we select appropriate values for a and 3 as the final settings.
Specifically, vary hyperparameter
tions are, the greater the likelihood of inference errors. This also means that the logits distribution effectively character- izes inference precision (RQ3). The confidence threshold we choose effectively avoids the variance intervals with poor accuracy, eliminates the interference from low-confidence regions, and enhances the overall performance of the model.
Visualization of Fine-grained Emotion Classification. To explore the preferences of MLLMs for different emotion cat- egories, we calculate and visualize the accuracy of zero-shot and SEPM for each category in the EmoSet dataset under different architectures, as shown in Fig. 6. Our method improves overall performance, and we observe a striking phenomenon: the accuracy for each category fluctuates sig- nificantly. In semantically similar categories, accuracy increase in one can cause decrease in others. Since our method enhances prompts, we find that MILMS exhibit dif- ferent preferences based on the textual prompts (RQ4). Cur- rently, we focus on improving overall performance, while addressing model preferences and selectively enhancing accuracy for each category an area for further exploration.
Visualization of Dropped Visual Tokens. To demonstrate the reliability of visual token dropping, visualize the dropping process of visual tokens in images across different emotions. In Fig. 4, each set of images from left to right consists of the original image, the image after a 20% drop rate, and the image after a 40% drop rate. As the drop rate increases, the amount of emotionally redundant information in the image decreases, with the remaining content primarily consisting of more significant emotional cues. This indi- cates that by selecting effective emotional cues based the correlation between visual tokens and the Focus-on- Emotion textual prompt, we are able to retain and highlight the most relevant emotional information while discarding less important, redundant visual elements. It also reinforces that selecting visual tokens to drop based on their relevance emotion is appropriate approach (RQ2).
5. Conclusion
In this paper, we address the challenges of emotion recogni- tion in Multimodal Large Language Models (MILMss,, par- ticularly the confusion caused by semantically similar emo- tion categories and visual redundancy. We propose Sharp- ening Emotion Perception in Multimodal Large Language Models (SEPM), which introduces a Confidence- Guided Coarse-to-Fine Inference framework and Focus-on-Emottion Visual Augmentation. These methods enhance emotion clas- sification accuracy without additional training or manual annotation. Our approach, validated through experiments on various emotion datasets, demonstrates its effectiveness. believe this work offers valuable insights for future research in MLLMs and emotion recognition.
Visualization of Confidence Estimation. To validate the rationale of calculating the coarse -raiined inference confi- dence based on the the logits distribution, we conduct an evaluation of inference accuracy at different variances the logits distribution in the first stage. As shown in Fig. 5, as the variance increases, the inference accuracy gradually improves, indicating that the closer the logits of different op-
8

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Acknowledge This work is supported by the National Key Research and Development Program of China (2024FCC3008000) and National Natural Science Foun- dation of China under Grant (62272354, 13361166629 62176188, 62225113, 62382080). The supercomputing system at the Supercomputing Center of Wuhan University supported the numerical calculations in this paper.
Choudhury, R., Zhu, G., Liu, S., Ninuma,, K., Kitani, K. M., and Jeni, L., Don't look twice: Faster video trans- formers with run-length tokenization. arXiv preprint arXi::2411.0522,, 2024.
Ding, Y., Li, B., and Zhang, Eta: Evaluating then aligning safety of vision language models inference time. arXiv preprint arXiv:2410.06623 2024.
Impact Statement
Fang, Y., Huang, W., Wan, G., Su, K., and Ye, M. Emoe: -sdality-specific enhanced dynamic emotion experts. In CVPR, 2025.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., and Parikh, D. Making the V in VQA matter: Elevating the role of image understanding in Visual Question An- swering. In CVPR, 2017.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint, 2023.
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K.,, Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. In CVPR, pp. 26584- 26595, 2024.
Huang, W., Ye, M., Shi, Z., Li, H., and Du, B. Rethinking federated learning with domain shift: A prototype view. In CVPR, 2023.
Bai, Y., Ji, Y., Cao, M., Wang, J., and Ye, M. Chat-based per- son retrieval via sallogue-refined cross- moosl alignment. In CVPR, 2025.
Huang, W., Yc, M., Shi, Z., Wan, Gi, H., Du, B., and Yang, Q. federated learning for generalization, robust- ness, fairness: A survey and benchmark. IEEE PAMI, 2024.
Bi, J., Wang, Y., Chen, H., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. Visual instruction tuning with 500x fewer pa- rameters through modality linear representation-sterrig arXiv preprint arXiv:2412.1233 2024.
Huang, W., Liang, J., Guo, X., Fang, Y., Wan, G., Rong, X., Wen, C., Shi, Z., Li, Q., Zhu, D., et al. Keeping yourself important in downstream tuning multimodal large language model. arXiv preprint arX1v:2503.04543, 2025a.
Bi, J., Wang, Y., Yan, D., Xiao, X., Hecker, A., Tresp, V., and Ma, Y. Prism: Self-pruning intrinsic selection method for training-free free multimodal data selection, arXiv preprint 2rXiv22502.12119 2025.
Bolya, D., Fu, C.-Y., Dai, X., Zhang, P., Feichtenhofer, C., and Hoffman, J. Token merging: Your vit but faster. arXiv preprint arXiv:2210.0944
Huang, W., Liang, J., Shi, 7.,, Zhu, D., Wan, G., Li, H., Du, B., Tao, D., and Ye, M. Learn from downstream and be yourself in multimodal large language model fine-tuning. In ICML, 2025b.
2022.
Chen, I.., Zhao, H., Liu, T., Bai, S., Lin, J., Zhou, C., and Chang, B. An image is worth 1/2 tokens after layer 2: and-play inference acceleration for large vision- language models. In ECCV, pp. 19-35. Springer, 2025.
Huang, W., Liang, J., Wan, G., Zhu, D., Li, H., Shao, J., Ye, M., Du, B., and Tao, D. Be confident: Uncovering overfitting in milm multi-task tuning. ICML, 2025c.
Hudson, D. A. and Manning, C. D. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
Chen, Z., WU, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In CVPR, pp. 24185- 24198, 2024.
Jin, C., Che, T., Peng, H., Li, Y., Metaxas, D., and Pavone, M. Learning from teaching regularization: Generalizable correlations should be easy to imitate. Advances in Neural Information Processing Systems, 37:966-994 2024.
Cheng. Z., Cheng, Z.-Q., He, J.-Y., Sun, J., Wang, K., Lin, Y., Lian, Z., Peng. X., and Hauptmann, A. Emotion- Ilama: Multimodal emotion recognition and reasoning with instruction tuning. In NeurlPS, 2024.
Jin, C., Peng, H., Zhang, Q., Tang, Y., Metaxas, D. N., and Che, T. Two heads are better than one: Test-time scaling
9

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
of multi-agent collaborative reasoning. arXiv preprint arXiv:2504.09772, 2025.
Liu, S.-Y., Wang. C.-Y., Yin, H., Molchanov, P., Wang. Y.C. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXi:24002.0935. 2024a.
Kim, J., Yconju, K., and Ro, Y. M. What if...?: Thinking counterfactual keywords helps to mitigate hallucination in large multi-modal models. In EMNLP, pp. 10672-10689, 2024a.
Liu, X., Wang, W., Yuan, Y., Huang, J.-t., Liu, Q., He, P., and Tu, Z. Insight over sight? exploring the vision- knowledge conflicts in multimodal 1lms. arXiv preprint arXiv:2410.08/4 2024b.
Kim, S., Cho, B., Bae, S., Ahn, S., and Yun, S.-Y. Vacode: Visual augmented contrastive decoding. arXiv preprint arXiv:2408 053377 2024b.
Liu, Y., Zhai, S., Du, M., Chen, Y., Cao, T., Gao, H., Wang, C., Li, X., Wang, K., Fang, J., et al. Guardreasoner- vl: Safeguarding vlms via reinforced reasoning, arXiv preprint arXiv:2505.11045 2025b.
Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. NewrIPS, 35:22199 222113, 2022.
Lu, P., Mishra, S., Xia, T., Qiu, L.., Chang, K.-W., Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 2022.
Lee, J., Cha, S., Lee, Y., and Yang, C. Visual question answering instruction: Unlocking multimodal large lan- guage model to domain specifcc visual multitasks. arXiv preprint ar XxXx:2442.008666 2024.
Machajdik, and Hanbury, A. Affective image classifica- tion using features inspired by psychology and art theory. In ACM MM, pp. 83-92, 2010.
Li, B., Zhang, Y., Guo, D., Zhang, R., Li, F., Zhang, H., Zhang, K., Li, Y., Liu, Z., and Li, C. Llava- onevision: Easy visual task transfer. arXiv preprint arXiv:2408.033266 2024a.
Masry, A., Long, D., Tan, J. Q., Joty, S., and Hoque, ChartQA: benchmark for question answering about charts with visual and logical reasoning. In ACL, 2022.
Li, Z., Yang, B., Liu, Q., Ma. Z., Zhang, S., Yang, J., Sun, Y., Liu, Y., and Bai, X. Monkey: Image resolution and text label are important things for large modal models. In CVPR, pp. 26763-26773, 2024b.
Panda, R., Zhang, J., Li, H., Lee, J.-Y., Lu, X., and Roy- Chowdhury, A. Contemplating visual emotions: Un- derstanding and overcoming dataset bias. In ECCV, pp. 579-595, 2018
Lian, Z., Chen, H., Chen. L., Sun, H., Sun, L., Ren, Y., Cheng, Z., Liu, B., Liu, R., Peng, X., et al. Affectgpt: A new dataset, model, and benchmark for emotion under- standing with multimodal large language models. arXiv preprint 250.:2500.666666 2025.
Peng, K.-C., Chen, T., Sadovnik, A.. and Gallagher, C. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In CVPR, pp. 860-868, 2015.
Singh, A., Natarajan, V., Shah, M., Jiang, Y. Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In CVPR, pp. 8317-8326, 2019..
Liang, J., Huang, W., Wan, G., Yang, Q., and Ye, M. Lo- rasculpt: Sculpting lora for harmonizing general and spe- cialized knowledge in multimodal large language models. In CVPR, 2025.
Wang, C., Liu, Y., Li, B., Zhang, D., Li, Z., and Fang, J. Safety in large reasoning models: A survey. arXiv preprint arXiv: 17704, 2025.
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. Vila: On pre-training for visual language models. In CVPR, pp. 26689-26699, 2024.
Wang. P., Bai, S., Tan, S., Wang, S., Fan, Z.,, Bai, J., Chen, K., Liu, X., Wang. J., Ge, W., et al. Qwen2-vl: vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zimick, C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755, 2014.
Xie, H., Peng, C.-J,, Tseng, Y.W.,, Chen, H.-J,, Hsu, C.-F., Shuai, H.-H., and Cheng, W.-H. Emovit: Revolutionizing emotion insights with visual instruction tuning. In CVPR, pp. 26596-26605, 2024.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In CVPR, 2023.
Xing, B., Yu, Z., Liu, X., Yuan, K., Ye, Q., Xie, W., Yue, H., Yang, J., and Kälviäinen, H. Emo-Ilama: Enhancing facial emotion understanding with instruction tuning. arXiv preprint arXiv:2408. /424, 2024.
Liu, S., Zheng, K., and Chen, W. Paying more attention to image: A training-free free method for alleviating hallucina- tion in Iyims. In ECCV, pp. 125-140. Springer, 2025a.
10

---

Catch Your Emotion: Sharpening Emotion Perception in Multimodal Large Language Models
Yang, J., Huang, Q. Ding, T., Lischinski, D., Cohen-Or, D., and Huang, H. Emoset: A large-scale visual emotion dataset with rich attributes. In ICCV, pp. 20383-20394, 2023.
Yang, Q. Ye, M., and Du, B. Emollm: Multimodal emo- tional understanding meets large language models. arXiv preprint ar arXiv:2406.1647
2024a.
Yang, Q. Shi, Q. Wang, T., and Ye, M. Uncertain multi- modal intention and emotion understanding in the wild. In CVPR, 2025.
Yang, T., Jia, J., Zhu, X., Zhao, W., Wang, B., Cheng, Y., Li, Y., Liu, S., Chen, Q. Jiang, P. et al. Enhanc- ing foretoonn capability of visual-language models by reducing image redundancy. arXiv preprint arXiv:2411. 241.::2311.1555,, 2024b.
Young, P., Lai, A., Hodosh, M., and Hockenmaier, J. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014.
Zhang, L., Luo, Z. Wu, S., and Nakashima, Microemo: time-sensitive multimodal emotion recognition with sub- lle clue dynamics in video dialogues. In ACM MM Work- shop, pp. 1 110-115. 2024a.
Zhang, Q. Wang, Z., Zhang, D., Niu, W. Caldwell, S., Gedeon, T., Liu, Y., and Qin, Z. Visual prompting llms for enhancing emotion recognition. EMNLP, 2024b.
Zhang, S., Chen, A., Sun, Y., Gu, J., Zheng, Y.-Y., Koniusz, P., Zou, K., Hengel, A. v. d., and Xue, Open eyes, then reason: Fine graingra visual mathematical understanding in milms. arXiv preprint arXiv:2501 Xiv:22501.06430, 2025.
Zhang, Y., Fan, C.-K., Ma, J., Zheng, W., Huang, T., Cheng, K., Gudovskiy, D., Okuno, T., Nakata, Y., Keutzer, K., et al. Sparsevim: Visual token sparsification for effi- cient lasion-language model inference. arXiv preprint arXiv:2410.04447,, 2024c.
Zhao, H., Liu, Z., Liu, Y., Qin, Z., Liu, J., and Gedeon, T. Facephi: Lightweight multimodal large language model for facial landmark emotion recognition. In ICLR Work- shop, 2024a.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory-efficient I1m train- ing by gradient low-rank projection. arXiv preprint arXiv:2403.0352 03507, 2024b.
Zhu,1 T. Liu, Q., Wang, F., Tu, 7.,, and Chen, M. Unravel- ing mrosstmodality knowledge conflicts in large vision- language models. arXiv preprint 470.v22400.03559 2024.


Reviews:
[{"review_id": "VhObaZtjnS", "reviewer": "Reviewer_wn8T", "summary": "This paper proposes a training-free approach for emotion classification using Multimodal Large Language Models (MLLMs). They find that MLLMs (1) struggles to distinguish between semantically similar emotions, and (2) are overwhelmed by redundant visual information. To address these challenges, they propose a Coarse-to-Fine inference framework to refine emotion classification and a Focus-on-Emotion Viusal Augmentation approach to reduce visual redundancy. Experimental results on multiple benckmarks demonstrate the effectiveness of the proposed method.\n\n## update after rebuttal\n\nI read authors' response and other reviewers' comments. I appreciate the clarifications and new results, which have fully addressed my concerns. Therefore, I will raise my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer wn8T:\n\nWe sincerely appreciate your time and effort in reviewing our paper. We have provided further clarification on each of the issues you raised. We hope the detailed responses below fully address your concerns, and we would be grateful if you would consider updating your score.\n\n**Q1: Different with SPARSEVLM & innovations** (Relation To Broader Scientific Literature & Other Strengths And Weaknesses & Other Comments Or Suggestions)\n\nThank you for raising this valuable question. Compared to general tasks, emotion recognition is often susceptible to interference from redundant visual information (noise), leading to inference biases. While SparseVLM accelerates inference for general tasks by retaining only the most crucial information in the visual modality, our method, in contrast, designs a specialized Focus-on-Emotion prompt, effectively identifying emotion-related visual redundancy through prompt-guided relevance. By discarding such redundant information, we can significantly enhance the accuracy of the model in emotion inference. Additionally, in emotion recognition tasks with a large number of categories, semantic similarities between emotions often lead to confusion. Therefore, we introduce Confidence-Guided Coarse-to-Fine Inference, enabling the model to handle simpler subtasks incrementally, thereby improving overall performance.\n\n**Q2: Reason for not comparing with other emotion-related methods** (Other Strengths And Weaknesses)\n\nTo the best of our knowledge, we are the **first** to explore training-free optimization in general emotion recognition tasks. Therefore, we do not identify suitable emotion-related methods for a fair and meaningful comparison. Although the SoV [1] method mentioned in the paper is also training-free, it is specifically designed for facial emotion images and requires additional manual bounding box annotations. Moreover, since its code has not been released, we do not have the capacity to adapt it to our model architecture and more general emotion recognition tasks in a short period of time. We will try to include a comparison with SoV in the final version, and will continue to follow similar work in the future. Thank you for your valuable suggestion.\n\n**Q3: More comparative experiment** (Other Comments Or Suggestions)\n\nIn the absence of suitable emotion-related baselines for direct comparison,  we introduced a comparative experiment with PDrop [2] based on its official code, which aims to eliminate visual redundancy to optimize the model. The experimental results in the table show that PDrop yields a slight improvement over the zero-shot setting by mitigating visual redundancy, supporting the hypothesis that emotional visual redundancy may interfere with the judgment of the model. Moreover, the results highlight the superior effectiveness of our proposed method.\n\n*Table: Comparison with PDrop on various emotion datasets.*\n\n|   Dataset   | Emotion6  |  EmoSet   | WebEmo7 | WebEmo25 | Abstract  |  Average  |\n| :---------: | :-------: | :-------: | :--------: | :---------: | :-------: | :-------: |\n|    LLaVA    |   48.32   |   52.77   |   25.56    |    15.71    |   27.86   |   34.04   |\n|    PDrop    |   51.24   |   53.53   |   27.50    |    16.02    |   27.50   |   35.16   |\n| SEPM (ours) | **54.21** | **56.04** | **42.39**  |  **18.26**  | **29.29** | **40.04** |\n\n**Thank you for your valuable advice. We are deeply grateful and appreciate your guidance.**\n\n[1] Visual Prompting in LLMs for Enhancing Emotion Recognition, EMNLP, 2024\n\n[2] PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction, CVPR, 2025", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "no", "experimental_designs_or_analyses": "no", "relation_to_broader_scientific_literature": "This paper propose a training free approach for emotion perception using MLLMs, offering a new well-performed solution. However, one of the core contributions of this paper, i.e., Focus-on-Emotion Visual Augmentation, as the author claimed,  has been proposed in prior work ([**SPARSEVLM'25**](http://export.arxiv.org/pdf/2410.04417)), which focus on more common VQA tasks. Although they achieves best overall results in emotion perception task, the contributions of this paper are less significant to the community under a broader background.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "strengths\n1. This paper propose a training-free approach for zero-shot emotion perception using MLLMs, providing an effective and efficient solution in this area.\n\n2. Experimental results on multiple benchmarks and extensive diagnostic analysis demonstrate the effectiveness.\n\n3. The writing is accessible, making the paper easy to follow.\n\nweaknesses\n\n\n1. The innovations of this paper is concerning. In particular, the proposed Focus-on-Emotion Visual Augmentation is methodologically idential to the adaptive sparsification in [**SPARSEVLM'25**](http://export.arxiv.org/pdf/2410.04417), which employ the same approach to drop irrelavant visual tokens.\n\n2. The author does not compare their method to those in the emotion-related field (e.g., [**SoV'24**](https://arxiv.org/pdf/2410.02244), which is also zero-shot), making the evaluation less convinced. At least, the rationale behind this choice should be clarified.", "other_comments_or_suggestions": "1. I recommend the authors further improve the methodology and highlight the innovations.\n\n2. More methods could be included for comparison to strengthen the comprehensiveness of the evaluation.", "questions_for_authors": "see weaknesses", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ct5Siw0bDQ", "reviewer": "Reviewer_jWJJ", "summary": "This paper presents Sharpening Emotion Perception in MLLMs (SEPM), a training-free method to enhance emotional reasoning in multimodal large language models. SEPM improves emotion classification by using a Confidence-Guided Inference framework and Focus-on-Emotion Visual Augmentation to reduce distractions. Experimental results show significant performance improvements in emotion-related tasks, offering a scalable, resource-efficient solution.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jWJJ:\n\nThank you again for your thoughtful and constructive suggestions. In the following responses, we address each of your points thoroughly, providing additional explanations and supporting evidence to strengthen the clarity of our methods.\n\n**Q1: Discussion on generalizability of SEPM** (Other Strengths And Weaknesses)\n\nThank you for highlighting this issue. It should be clarified that 1) the two-stage inference process in our method is specifically designed to address the progressive categorization from coarse-grained to fine-grained levels inherent in emotion recognition tasks, and 2) the visual enhancement component of our method is motivated by the interference caused by redundant visual information in emotion-related tasks. Consequently, both modules are specifically tailored for emotion recognition and are not directly transferable to other tasks. Nonetheless, we have explored applying the Focus-on-X prompt to other domains and conducted experiments on a medical dataset (VQA-RAD [1]). As shown in the table, models across various architectures all exhibited improved performance, indicating that Focus-on-X can effectively serve as a generalized prompt enhancement.\n\n*Table: Generalizability experiment on the VQA-RAD Dataset (Medical Domain)*\n\n|     Dataset      | LLaVA-7b  |  VILA-8b  |\n| :--------------: | :-------: | :-------: |\n|    Zero-shot     |   37.47   |   38.58   |\n| Focus-on-Medical | **39.25** | **40.58** |\n\n**Q2: Experiment on higher drop rate** (Other Strengths And Weaknesses)\n\nWe conducted ablation experiments with higher drop rates (beta), as shown in the table below. As the drop rate increases, the performance of the model declines to some extent but still maintains a certain inference capability. We believe discarding the least relevant redundant information helps the model better focus on emotion-related content, thereby improving performance. However, as the drop rate continues to increase, the model begins discarding some moderately important information, leading to a reduction in inference accuracy. Nonetheless, since the most critical emotional information is preserved, the model's performance does not completely collapse.\n\n*Table: Ablation experiment on drop rate*\n\n| Drop rate |  0.1  |    0.2    |  0.3  |  0.4  |  0.5  |  0.6  |  0.7  |  0.8  |\n| :-------: | :---: | :-------: | :---: | :---: | :---: | :---: | :---: | :---: |\n|    ACC    | 56.04 | **56.24** | 55.97 | 55.96 | 53.95 | 53.82 | 53.81 | 52.51 |\n\n**Q3: Lack of clarity in Equation 6** (Other Strengths And Weaknesses)\n\nSorry for the confusion. We have revised the relevant part of Equation 6: \n\nThen we extract the FoE prompt set as emotion-related text, leveraging the query-dimension of textual logits and the key-dimension of the visual modality to construct the drop matrix $P \\in \\mathbb{R}^{L_t \\times L_v}$, where $L_t$ and $L_v$ are the lengths of the text tokens (FoE) and visual tokens, respectively. The drop matrix $P$ is defined as:\n$$\nP[i, j] = \\mathcal{A}(i, j)\n$$\n$i \\in$ { $x \\mid \\mathbb{I}[x] \\in \\mathbb{T}$ },  $\\quad j \\in$ { $y \\mid \\mathbb{I}[y] \\in \\mathbb{V}$ },\n\nwhere $\\mathbb{I}$ denotes the set of all input tokens, including both textual and visual modalities, $\\mathbb{T}$ denotes FoE prompt token set and $\\mathbb{V}$ is visual tokens set, i.e.\n$$\n(\\mathbb{T} \\cup \\mathbb{V}) \\subseteq \\mathbb{I}\n$$\nWe will update the above content in the final version.\n\n**Q4: Problem on Figure 6** (Other Strengths And Weaknesses)\n\nThank you for tips. We will update in the final version. \n\n[1] A dataset of clinically generated visual questions and answers about radiology images, Scientific data, 2018", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. The method is relatively applicable.", "theoretical_claims": "The paper presents a simple and easy-to-understand approach, but provides limited discussion on the theory.", "experimental_designs_or_analyses": "The experimental results are comprehensive. The author conducts various experiments with different relative method in various downstream tasks.", "relation_to_broader_scientific_literature": "Based on the issues in existing work, the author proposes a new approach.", "essential_references_not_discussed": "There is none.", "other_strengths_and_weaknesses": "Pros: \n• The paper is well-structured and clearly written.\n• SEPM is able to enhance emotion recognition in MLLMs without requiring additional training or manual annotations. \n\nCons: \n• The generalizability of SEPM is not fully established, as its performance and suitability for other multimodal tasks have not been adequately validated, indicating the need for additional experiments and data validation.\n• Why does the acc in Fig. 3 show little change with \\beta in on the EmoSet dataset? Additionally, how do larger \\beta values affect the results? Further discussion is needed.\n• The meaning of Eq. (6) is unclear; further explanation would be appreciated.\n• The numbers on the right side of the dashed line in Fig. 6 are too small, resulting in poor readability.", "other_comments_or_suggestions": "It is suggested to correct the minor errors in the paper.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VzTvBnsoyi", "reviewer": "Reviewer_6rbK", "summary": "This paper proposes SEPM to tackle emotion recognition challenges in multimodal models. It focuses on issues like confusing similar emotions and visual noise. SEPM introduces a two-stage inference process: a coarse-to-fine approach to improve confidence in emotion classification and a focus on relevant emotional cues to reduce visual redundancy. The method improves performance without needing extra training, offering a scalable and efficient solution for emotion recognition tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6rbK:\n\nThank you very much for your valuable comments and constructive feedback. Below, we carefully respond to each of your concerns point-by-point, providing detailed explanations and supplementary evidence to further clarify our approach and demonstrate its effectiveness.\n\n**Q1:  Attention map computation** (Other Strengths And Weaknesses)\n\nSorry for any inconvenience caused. Within the transformer architecture of MLLM, each block generates an attention map that reflects the model focus at that layer. Notably, attention maps from deeper layers tend to encode more abstract, high-level semantic representations, aligning more closely with the model learned conceptual understanding of the input. Therefore, we use the attention map from the final transformer block as the basis for token dropping in the second stage. Additionally, to enhance the accuracy of emotional focus, we retain only the attention maps between visual tokens and the Focus-on-Emotion prompt.\n\n**Q2:  More comprehensive ablation experiment** (Other Strengths And Weaknesses)\n\nSince we perform a classification task using an MLLM, the metric Accuracy serves as an appropriate indicator of performance. We conducted further ablation studies on the EmoSet dataset. As shown in the table, the effectiveness of each module is further demonstrated. More details will be provided in the final version.\n\n*Table: Ablation experiments on additional datasets*\n\n|     CCI      |      FP      |     VTA      | Emotion6 (ACC) | EmoSet (ACC) |\n| :----------: | :----------: | :----------: | :------------: | :----------: |\n|              |              |              |     48.32      |    52.77     |\n| $\\checkmark$ |              |              |     51.68      |    53.98     |\n|              | $\\checkmark$ |              |     51.52      |    54.28     |\n|              |              | $\\checkmark$ |     53.03      |    56.10     |\n| $\\checkmark$ | $\\checkmark$ | $\\checkmark$ |   **54.04**    |  **56.24**   |\n\n**Q3:  Discussion on inference efficiency** (Other Strengths And Weaknesses)\n\nAlthough the two-stage inference process introduces a certain increase in inference time, the overall inference latency for emotion recognition tasks remains low, due to the relatively lightweight nature of the task, making this additional overhead negligible in practical use. Additionally, we effectively reduced part of the inference time by discarding tokens. Overall, the slight extra time cost results in improved inference performance.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. The method is well evaluated.", "theoretical_claims": "The theory used in the paper is clear and easy to understand.", "experimental_designs_or_analyses": "The experimental results are comprehensive.", "relation_to_broader_scientific_literature": "The author introduces an innovative method that enhances the sentiment reasoning ability of MLLM through a two-stage inference process and visual augmentation.", "essential_references_not_discussed": "NaN.", "other_strengths_and_weaknesses": "Strengths：\n1.The method is novel, and the experiments are comprehensive.\n2.The paper is easy to understand.\n\nWeaknesses： \n1.The paper does not clearly specify how the attention map is obtained in the method, and it would be helpful to provide more details on this. \n2.The experiments are relatively thorough, but the ablation study includes fewer metrics. It would be helpful to provide more information. \n3.I would like to see an analysis of the inference efficiency.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Refer to weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VBs0KgpqHy", "reviewer": "Reviewer_W74m", "summary": "This paper proposes a method called Sharpening Emotion Perception in Multimodal Large Language Models (SEPM) to improve emotion recognition in MLLMs. It addresses two challenges: confusion between semantically similar emotions and the visual redundancy that distracts from emotional cues. SEPM incorporates a two-stage inference process and focuses on emotion-related visual cues, enhancing classification accuracy. The approach is training-free, requiring no additional manual annotations or fine-tuning, and significantly improves MLLM performance on emotion tasks, offering a resource-efficient and scalable solution for emotion recognition.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer W74m:\n\nWe are deeply grateful for your positive feedback on our work and the insightful suggestions. We have carefully reviewed each point and provided detailed responses accordingly.\n\n**Q1:  Performance under different prompts** (Other Strengths And Weaknesses)\n\nTo explore the performance under different prompts, we designed a prompt augmentation semantically similar to Focus-on-Emotion: *Identify the predominant emotion conveyed*, and conducted corresponding experiments. As presented in the table, the performance of the new prompt is noticeably inferior to that of Focus-on-Emotion. We hypothesize that more direct prompts tend to yield better results. We plan to further explore the impact of various prompt designs on performance in future work.\n\n*Table: Ablation experiments on different prompts*\n\n|                Prompt type                | Drop rate = 0.1 | Drop rate = 0.2 |\n| :---------------------------------------: | :-------------: | :-------------: |\n| Identify the predominant emotion conveyed |      50.53      |      50.77      |\n|             Focus on emotion              |    **56.04**    |    **56.24**    |\n\n**Q2:  More analysis on removing visual information** (Other Strengths And Weaknesses)\n\nCompared to general tasks, visual redundancy in emotion-related tasks is more likely to interfere with the inference bias. Therefore, dropping a small amount of the least relevant visual information can help improve the model’s performance. In Figure 3 of the original paper, we conduct a sensitivity analysis under different drop rates and observe that discarding a small amount of redundant information leads to performance gains. However, excessive dropping results in the loss of some informative content, causing a decline in performance to some extent. Additionally, Figure 4 of the paper visualizes the specific content being dropped, showing that at lower drop rates, the removed tokens are mostly irrelevant, which allows the model to focus more on salient information and enhances its reasoning capability.\n\n**Q3:  The confidence-based prompt selection process in the first stage** (Other Strengths And Weaknesses)\n\nSorry for the confusion. To encourage the model to focus on relatively simpler problems at each stage, we introduce prompt enhancement for fine-grained inference based on the outcomes of coarse-grained inference. To mitigate the risk of error propagation from the first stage adversely impacting the second-stage inference, we assess the confidence of the model using the variance of the output logits, rather than relying solely on predicted labels. Specifically, when the variance among logits is high (indicating strong confidence), we apply direct positive/negative prompt augmentation. Conversely, when the variance is low (suggesting uncertainty), we incorporate ambiguity-related descriptions to characterize the model decision process in the first stage, which are then used to guide the second stage. The selection of the variance threshold is discussed in the experimental section, where the effectiveness of the proposed strategy is demonstrated.\n\n**Q4:  Inappropriate expressions** (Other Strengths And Weaknesses)\n\nWe will revise the inappropriate expressions in the final version. Thanks for your suggestions.", "claims_and_evidence": "YES", "methods_and_evaluation_criteria": "Yes, the proposed method can effectively enhance the model's ability in sentiment reasoning.", "theoretical_claims": "The approach is logically clear and the ideas are innovative.", "experimental_designs_or_analyses": "The experimental results are comprehensive. The method has been tested on different datasets, and the results are shown to be reliable through ablation studies and visualization experiments.", "relation_to_broader_scientific_literature": "The author references the latest related work and proposes a method that takes a different approach compared to existing studies.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Pros:\nSEPM improves emotion recognition in MLLMs without requiring additional training or manual annotations, making it a resource-efficient and scalable solution for real-world applications.\n\nCons: \n\n-\tThe Focus-on-Emotion prompt designed in the paper leads to significant performance improvements. I'm curious to know if this prompt is the only one that works, or if changing the prompt would have an impact on performance.\n\n-\tIn theory, removing more visual information could mean losing important emotional cues needed for the model to make accurate predictions, but this is not clearly reflected in the paper's sensitivity analysis.\n\n-\tThe description of the selection process based on the confidence threshold in the first stage is too simplistic in the paper.\n\n-\tThe paper contains some inappropriate expressions.", "other_comments_or_suggestions": "The paper mentions that the model is highly sensitive to different prompts. I hope future work can explore this aspect further.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces a training-free framework to enhance emotion recognition in MLLMs. Empirical evaluations on multiple emotion recognition benchmarks show that SEPM offers notable improvements over baseline zero-shot settings and some alternative methods (such as PDrop). The authors also include an exploratory study showing potential generalizability to other domains (e.g., medical VQA), though the main focus remains on emotion recognition. There is some disagreement among the reviewers. One reviewer expressed reservations (leaning towards weak rejection) mainly due to methodological overlap with existing work and limited baseline comparisons. However, the remaining reviewers lean toward acceptance, with scores ranging from Accept to Strong Accept.

Given the strong empirical results, clear scalability and efficiency benefits, and the authors’ responsiveness in addressing the core concerns, the overall recommendation is to accept the paper, with the expectation that the final version will incorporate the clarifications and additional comparisons where possible.
Decision: Accept (spotlight poster)