AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Authors: Zheng Lian
Haoyu Chen
Lan Chen
Haiyang Sun
Licai Sun
Yong Ren
Zebang Cheng
Bin Liu
Rui Liu
Xiaojiang Peng
Jiangyan Yi
Jianhua Tao
Venue: ICML 2025 oral (2025)
Keywords: multimodal emotion recognition
AffectGPT
MER-Caption
MER-UniBench
Primary area: applications

Abstract: The emergence of multimodal large language models (MLLMs) advances multimodal emotion recognition (MER) to the next level—from naive discriminative tasks to complex emotion understanding with advanced video understanding abilities and natural language description. However, the current community suffers from a lack of large-scale datasets with intensive, descriptive emotion annotations, as well as a multimodal-centric framework to maximize the potential of MLLMs for emotion understanding. To address this, we establish a new benchmark for MLLM-based emotion understanding with a novel dataset (MER-Caption) and a new model (AffectGPT). Utilizing our model-based crowd-sourcing data collection strategy, we construct the largest descriptive emotion dataset to date (by far), featuring over 2K fine-grained emotion categories across 115K samples. We also introduce the AffectGPT model, designed with pre-fusion operations to enhance multimodal integration. Finally, we present MER-UniBench, a unified benchmark with evaluation metrics tailored for typical MER tasks and the free-form, natural language output style of MLLMs. Extensive experimental results show AffectGPT's robust performance across various MER tasks. We have released both the code and the dataset to advance research and development in emotion understanding: https://github.com/zeroQiaoba/AffectGPT.

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Zheng Lian'
Haoyu Chen 2
Lan Chen'
Haiyang Sun Licai Sun Yong Ren' Zebang Cheng* Bin Liu'
Rui Liu
Xiaojiang Peng" Jiangyan Yi?
Jianhua Tao 78
Abstract
1. Introduction
The emergence of multimodal large language models (MLLMs) advances multimodal emotion recognition (MER) to the next level-from naive discriminative tasks complex emotion under- standing with advanced video understanding abil- ities and natural language description. However, the current community suffers from a lack of large-scale datasets with intensive, descriptive emotion annotations, as well as a multimodal- centric framework to maximize the potential of MLLMs for emotion understanding. To address this, establish based emotion understanding with novel dataset (MER (MER-Caption) and a new model (AffectGPT). Utilizing our model-based crowd-sourcing data collection strategy, we construct the largest de- scriptive emotion dataset to date (by far), fea- turing over 2K fine-grained emotion categories across 115K samples. We also introduce the Af- fectGPT model, designed with pre-fusion opera- tions to enhance multimodal integration. Finally, we present MER-UniBence a unified bench- mark with evaluation metrics tailored for typi- cal MER tasks and the free-form, natural lan- guage output style of MLLMs. Extensive exper- imental results show AffectGPT's robust perfor- mance across various MER tasks. We have re- leased both the code and the dataset to advance research and development in emotion understand- ing:
Emotions encapsulate human intentions, and accurately ognizing emotional states is essential for enhancing human- computer interaction (Minsky, 1988). Emotions can be con- veyed through various human behaviors in different forms (Chen et al., 2023; Pfister et al., 201 1), giving rise to the task of multimodal emotion recognition (MER), which integrates multimodal information to evaluate human emotional states. As a critical area in artificial intelligence, MER has broad applications, ranging from education (Schutz, 2007) and psychological counseling (Liu et al., 2021a) to empathic embodied robots (Spezialetti et al., 2020).
benchmark for MLLM-
Traditional methods primarily rely on discriminative models that map human emotions to the most likely categories from predefined emotion taxonomies. The most widely used tax- onomy is Ekman's 's theory (Ekman & Kelmer, 1970), which classifies all emotions into six basic categories: sadness, happiness. fear, anger, surprise, and disgust. However, such categorical frameworks exhibit some limitations in model- ing human affective states. For example, our emotional expressions are diverse and nuanced due to culture-specific idioms (Matsumoto, 2001), context-dependent metaphors (Kövecses, 2003), and highly personalized behavioral pat- terns (Izard et al., 1993). Current closed-set classification paradigms fail to capture the rich diversity of emotional expressions in real-world scenarios (Plutchik, 1980). Mean- while, the rigid emotion taxonomies oversimplify the contin- uous spectrum of emotional experiences by forcing discrete labels (e.g., anger or surprise) onto nuanced affective states that often coexist (Cowen & Keltner, 2017). Illustrations are provided in Figure 1, where the diverse and coexisting issues are presented in Figs. (a) and (b).
Instiittee of Automation, Chinese Academy of Scienees 2 CMVS, University of Oulu 'Shanghai Jíao Tong University "Shnnzenn University 'Inner Mongolia University "Shenzhen Tech- nology University Department of Automation, Tsinghua Uni- versity "Beijing National Research Center for Information Sci- ence and Technology, Tsinghua University. Correspondence to: Zheng Lian <lianzheng20166 @aa..c..c>, Jianhua Tao <jh- t tsinghua. tsinghua.edu.cn>>. edu.en>.
Recent advances in multi-modal large language models (MLLMs) enable emotion understanding to move beyond traditional discriminative approaches, embracing a more generative framework (Liang al., 2024). This shift allows models to describe complex, coexisting emotional states in natural language. With the vast vocabulary. MLLMs can generate diverse, descriptive emotion categories be- yond basic emotions, offering new opportunities for emo- tional understanding. However, recent research highlights
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
(a) Diversity
Deseription: The sentence might be the woman's comment reaction to someone nearby. Given the audio cues of steady tone with humor and the woman's smile and glance the left, we can infer that her words carry mocking or sarcastle tonc. Thus, this sentence likely expresses the woman's mockery of the other person's lack of understanding or recognition of something, conveyed humorous manner.
(b) Coexistence
Description: the vidco, his eyes wide open and his mouth is also open, indicating surprised facial expression. the audio, the charucter speaks with stutter, which usually expresses feelings of nervousness, anxiety. character seems be unhoppy and angry due the prejudice of the people around him.
Combined with the text content, the
Figure I: Emotion complexity analysis. Human emotions are often diverse and coexist simultaneously. Such complex emotional states are difficult to describe using discriminative frameworks. However, MIL.Ms can generate emotional descriptions, offering new possibilities for complex emotion modeling. Since the original videos contain real people, to address copyright concerns, we first use DemoAl to remove personal information and then proceed with visualization.
that MILMs still face limitations in emotion understanding (Lian et al., 2024a;d). To address these challenges, this pa- per aims to advance emotional understanding from two key perspectives: the dataset and the model. Finally, we estab- lish a unified benchmark tailored to the free-form, natural language output style of MLLMs.
Models. Existing MLLMs typically consist of three key components: modality encoder that converts audio and video into low-level hidden features, a connector that trans- forms these features into a format more suitable for LLMs, and an LLM--ased generator that produces responses based on the given instructions. While the results of MILMs are promising, existing models generally leave everything of multimodal fusion LLMs, which is insufficient for MER that emphasizes multimodal characteristics. This paper in- troduces the AffectGPT model, designed with pre-fusion operation to emphasize multimodal integration.
Dataset. The current community still suffers from a lack of large-scale datasets with intensive, descriptive emotion annotations to realize the potential of MLLMs. The annota- tion strategies for constructing descriptive emotion datasets can be classified into three types: model-based, human- based, and human-model collaborative strategies. The human-based strategy is the most common way to con- struct emotion datasets with rich descriptive annotations. However, it's costly to conduct crowd-sourcing to scale up the dataset size with this purely manual annotation manner. Besides, humans tend to focus on main cues, resulting in brief and incomplete descriptions (Liu et al., 2022a). Thus, researchers propose model-based automatic annotation ap- proaches. However, due to the lack of human proofreading, this approach may result in insufficient label quality (Cheng et al., 2024). Recently, Lian et al. (2024a) propose human- model collaborative strategy, in which models provide pre- labeled cues and humans conduct multiple rounds of checks, which can be seen as a human-led. model-assisted strategy. Although this approach offers more comprehensive descrip- tions, it is costly and difficult scale the dataset. To balance label quality and dataset size, we introduce novel annota- tion strategy that conducts model-based crowd crowd-sourcing la- beling with human priors, named model-led human-asaen assisted, to construct a lcage-scale emotion descriptive dataset with diverse emotional categories.
Benchmark. Although it's desirable to generate tional descriptions in free-form, natural language style (see Appendix D), this poses challenges for quantitative comparison. To address this, we propose metrics specif- ically designed for this output style. Additionally, to en- sure fair and comprehensive evaluation, we introduce MER- UniBench, a benchmark that incorporates three typical tasks: fine-grained emotion recognition, basic emotion recogni- tion, and sentiment analysis. We believe this work can enhance the emotion understanding capabilities of MIL.Ms and open possibilities for complex emotion modeling. The main contributions of this paper are summarized as follows:
We construct a large-scale emotional description dataset MER-Caption, which adopts model-led, human-assisted annotation strategy strike a balance between label quality and dataset size.
We develop AffectGPT, which uses additional pre- fusion operations to enhance multimodal integration, thereby improving emotion understanding.
2

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Table 1: Dataset comparison. "A", "V", and "T" stand for image, audio, video, and text, respectively. We observe that deseriptive datasets contain more diverse labels, providing the potential for modeling complex emotions.
Dataset
Modality
# Samples
Description
Emotions
Annotation Manner
RAF-DB (Li et al., 2017)
I
29,672
7
Human
AffectNet (Mollahosseimi et al., 2017)
450,000
X
Human
EmoDB (Burkhardt et al., 2005)
A 535
7
Human
Categorical Dataset
MSP-Podcast (Lotfinn & Busso, 2017)
73,042
8
Human
DFEW (Jiang et al., 2020)
11,697
7
Human
FERV39K (Wang al., 2022)
V
38,935
7
Human
MER2023 (aa et al., 2023)
A,V,T
5,030
X
Human
MELD (Poria et al., 2019)
A,V,T
13,708
7
Human
Emo (Xic al., 2024)
51,200
988
Model
MERR-Coarse (Cheng et 2024)
A,V,T
28,618
113
Model
MAEW (Liu et al., 2022a)
A.V.T
10,045
399
Human
Descriptive Dataset
OV-MERD (Lian et al., 2024a)
A.V.T
332
236
Human- Jod-Mtsddesssiiad
MERR-Fine (Cheng et al., 2024)
A,V.T
4,487
484
Human- seatn---ed---tddd assimed
MER- CeR-Caption
A.V.T
115,595
2,932
Model-led-Himee assisted
MER-Captn Captiin++
A,V.T
31,327
1,972
Model-led -edd----e----miiiaa
We build MER-UniBenenn which encompasses typical MER tasks with tailored metrics. This benchmark can offer comprehensive evaluation results for MLLM- based emotion understanding.
and faces challenges in scaling the dataset. In this paper, review these annotation methods and introduce model- led, human-assisted strategy. As shown in Figure 2, we leverage human priors to guide description generation and sample filtering, ultimately achieving automatic annotation for unlabeled data. Using this strategy, we construct the MER-Caption dataset, which includes 115K coarse-labeled samples and 31K fine-labeled samples, making a significant contribution to current descriptive datasets. The raw data in MER- MER-Caption is sourced from the unlabeled portions of MER2024 (Lian et al., 2024b), with explicit permission from the dataset owners. Therefore, this paper does not involve the collection of new data but provides additional annotations for existing datasets. Appendix H provides more comparisons with existing datasets.
Extensive experiments demonstrate the effectiveness of AffectGPT, which achieves over 9% performance improvement compared existing MILMs.
2. MER-Caption: Dataset Construction
Table I summarizes existing emotion datasets, which can be broadly classified into categorical and descriptive datasets. The former directly provides emotion labels (e-g., happy), while the latter offers textual descriptions related emo- tions. We first conduct preliminary experiments to extract emotion labels from descriptive datasets (see Appendix E). As shown in Table 1. descriptive datasets contain more diverse labels, offering the potential to capture complex emotions. Thus, this paper focuses on descriptive datasets.
2.1. Description Generation
The choice of base models is critical for generating accurate deseriptions. Unlike previous work that relied solely on experience (Cheng al., 2024), we guide model selection using human priors. Specifically, set of samples for preliminary experiments. In this phase, we annotate riee-grained labels for each sample, allowing annotators to assign any emotions they deem appropriate, thus providing more diverse and precise labels. Based on the results in preliminary experiments (see Appendix F), we employ SALMONN (Tang et al., 2023) the audio LLM (ALLM) to generate audio cues, Chat- Chat-UniVi (Jin al., 2024) as the video LLM (VLLM) to extract visual cues, and GPT-3.5 (OpenAI, 2022) to merge the audio and video cues with text content. Then, to further reduce annotation costs, we experimented with replacing GPT-3.5 with other open-source LLMs but ob- served a drop in performance. The primary reason that multimodal fusion in MER is inherently complex, often
Based on the annotation manner, descriptive datasets can be categorized into model-based, human-based, and human- model collaborative strategies (see Table 1). Although the model-based approach makes it easy to expand the dataset size, it mainly relies on experience to select models and lacks human intervention, resulting in insufficient label qual- ity (Cheng et al., 2024). To enhance label quality, Liu et al. (2022a) relied on human annotators to generate emotion descriptions. However, humans tend to focus on primary clues, easily leading to incomplete descriptions. To this end, Lian et al. (2024a) proposed a human-led, model-assisted strategy. Specifically, the model first provides pre-labeled descriptions, and then multiple annotators perform multi- round checks. Although this strategy produces more com- prehensive descriptions, comes with high annotation costs
first select small sub-
3

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
i|le. Audio
Madeblad Harman-dniiie devisted Description Geueration
MER-Caae Capt-Cntionn 115k
Modokfeal Flumatt
MER- Cept-Cption+
Sample Filtering
31k
Video
Low- filtering
High-lesd flloerng
Crowdhouncing Sentiment Claoiffars
Seatiment
Text
Mirnarchdd Andio and Video
Tist
Prodiction
Are they consitear??
Video
Unwddwereng Emoilio Classdees
Emnoton
Asdio
ALLM
GIPE Afrrod
Predecson
Yes
Scatiment Pradictio
Vilco
YELMM
Captian
MERR
LLM
Entoiion Pradiction
cumsisient?
Abmormal L eaggt MaRli-Captiee
Hurran Priors ALLM VLLM Selection
Model-itanol Cosssbnncing for Semple Selection
X
Figure 2: Dataset construction pipeline. To create a large-scale dataset with guaranteed label quality, we propose a model-led, human-assisted annotation strategy. In this approach, we leverage human priors to guide description generation and sample filtering, ultimately achieving automatic annotation for unlabeled data.
countering issues such as modality conflict, where inconsis- tencies or contradictions arise between different modalities (see Appendix G). This places high demands on the L.LM's reasoning capabilities. Then, we adopt the above strategy for automatic annotation and create the MER- EER-Caption dataset.
errors. For instance, when ALLM and VLLM (in Section 2.1) fail to generate responses, the resulting descriptions tend to be short. As a result, we further remove descriptions located at both ends of the distribution.
High-level Filtering. In addition to low-level filterin, we propose a model-based crowdsourcing technique for high- level filtering. Specifically, we train multiple multimodal emotion and sentiment classifiers using human-ane auman-annotated categorical datasets. Guided by MERBench (Lian al., 2024c), we use CLIP ViT-L (Radford et al., 2021) as the visual encoder and HUBERT-L (Hsu al., 2021) as the acoustic encoder, followed by an attention- based fusion strategy to make final emotion and sentiment predictions. These pre-trained models are then used to predict labels for unlabeled data, generating multiple predictions for each sample. To mitigate potential prediction errors, we apply majority voting to determine the final label, ensuring more reliable results. We refer to this process model-based crowdsourcing, Alternatively, emotions and sentiments can also be predicted based on the descriptions using the strat- egy outlined in Appendix E. If the labels extracted from the descriptions differ from those obtained through model- based crowdsoureing, we consider these descriptions be of low quality and remove them. Through this process, we can extract knowledge from multiple human- datasets to guide sample selection. After applying multi-level filter- ing, we obtain the MER-Caption detailed comparisons between our dataset and existing ones, highlighting that our dataset is the largest multimodal emo- tion description dataset with diverse emotion categories.
2.2. Sample Filtering
Since the descriptions generated by the above process have not been manually verified, MER- DER-Caption inevitably con- tains some errors. To this end, we implement two-level filtering process to enhance the label quality.
Low-level Filtering. First, we observe that some samples contain mismatched audio and video. As shown in Fig- ure 2, the visible person is not speaking, while the audio comes from an invisible person. This setup differs from our task, where aim to analyze a person's 's emotions based their audio, video, and text content. Mismatched data com- plicates this task, shifting the focus to understanding how the interlocutor's actions may influence the target person's 's emotions. Therefore, we remove such data and plan to ad- dress this issue in future work. To automatically determine whether the visible person speaking, we use TalkNet (Tao al., 2021). Preliminary experiments indicate that this tool achieves over 90% accuracy in identifying the speaking in- dividual. Then, we remove samples with mismatched audio and video. Second, the length distribution of the generated descriptions roughly follows a Gaussian distribution (see Figure 2). Preliminary experiments reveal that descriptions at both ends of the distribution are more likely to contain
dataset. Table presents
4

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
LLM
LLM A HH Tokenizer
LLM
LLM
HL AH
AH, Projector
AH,
A H, Tokenizer
HL Projector
HH,
Au Takenizer
Projector
Tokenizer
Projector
Projector
Pre-fusion & Projector
Projecton
7,
X,
Z,
Z,
Z,
Andio Encoder
Video Encoder
Audio Encoder
Video Encader
Audio Encader
Video Encoder
A Video X.
A
A Video
Audio X.
Prompe
Prompt
Audio
Video
Prompt
Audio X.
Prompt
Andio LLM (ALLM)
Video LLM (VLLM)
Audio- LLM (AV-LLM)
AffectGPT (Quzx)
Figure 3: Model comparison. ALLM and VLLM primarily use modality-speciffi encoders and align them with the LLM through projection layers. AV-LLM mainly facilitates cross-modal interaction within the language model. In AffectGPT, we move the cross-modal interaction outside the language model and use a pre-fusion operation to enhance multimodal integration. these figures, can determined based on the requirement of whether include X,-
3. AffectGPT: Model Design
represent the response as R (ri where L, is the number of tokens. Then, Eq. is transformed into:
Our primary goal is to map audio-video-text inputs to emotion-related descriptions. In this section, we first re- view the current mainstream architectures. We then intro- duce AffectGPT, model specifically designed highlight multimodal characteristics in emotion understanding.
Lr max II P Xt. Q Rc).
(2)
In this equation, is the current token to be predicted. and Ret = (r. (ri) is the previously generated tokens, which serve as additional conditioning during training.
3.1. Mainstream Architecture
MILMM aims understand multimodal input and generate appropriate responses based on the input and user instruc- tions. Unlike pure-text LLMs, the primary challenge for MILMSs lies in enabling the model to perceive multimodal input, i.c. providing the model with "eyes" and "ears". In existing models, the most common approach is to first ex- tract sonddalit--specfff embeddings and then align them with the LLM through projection layers. For audio-vides joint tasks, Audio-Vidie Video LLMs (AV-LLMs) typically facilitate modsl interaction within the language model. Figure 3 illustrates the current mainstream architecture.
3.2. Pre-fusion Operation
Mainstream AV-LLMS leave everything of cross-modal in- teraction to the LLMs (in Figure 3), which is insufficient for handling MER with multimodal characteristics. To ad- dress this, propose pre-fusion operation that moves the cross-modal interaction outside the LLMs, further hancing multimodal integration. We refer to this model as AffectGPT. This paper introduces two types of pre-fusion operations: Q-Former- based and attention By default, we apply this operation to Z, E RC, xd and Za E R We also experimented with H, and Ha, but this choice led decrease in performance.
pre-fusion.
Formally, for each sample X, we represent its video, audio, and text content as X,, X,, and X,, respectively. Given an instruction Q, the goal is to output the correct response R. video expert encode
For the visual input X., it into latent space Zy, then apply projector (-) to generate visual tokens H, = G. (Z.).. Similarly, for the acoustics input X., we use an audio expert and a projector to generate the acoustic embeddings Za and tokens Hg. For the instruction Q and text content X,, we use a template to merge them into a prompt P. and then map them to the corresponding tokens through the tokenizer and embedding layer in the language model. After obtaining these tokens, we concatenate them and feed them into the LLM decoder. The primary objective is to maximize the likelihood of the target response R. conditioned on multimodal content (X.,
In this module, we preserve the temporal in- formation in the vision features Z, and audio features Z and utilize Q-Former (Li al., 2023b) for multimodal fu- sion. Specifically, to compress the multimodal content, first create K learnable query tokens Zg E R xd, Then, we interact Zg with the concatenated Z, and Z₂ through cross-attention, thereby distilling the knowledge from the multimodal content into the query tokens. Formally, this process can be represented as:
Q-Former.
Zav = Concat (Za, Zv),
(3)
Xa. X₂) and user instruction Q:
Z₁ Q-Former (Zg. Zav + PE (Znv)),
(4)
max P (R) X,, X,, Q)
(1)
where Zav E ((te+t))xd with the concatenation operation applied along the temporal dimension. Here, Ze E kxdd and PE(-) represents the positional encoding.
The above formula is optimized in an autoregressiie man- ner, consistent with the objective function of L.LMs. We
5

---

AffectGPT: A New Datasct, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Attention. Unlike Q-Former which preserves temporal information, we propose simpler architecture that di- rectly compresses temporal information and applies atten- tion mechanisms for multimodal fusion. This simplified module is inspired by MERBench (Lian et al., 2024c), which proves that in MER tasks, features with temporal information do not always lead to better performance than compressed features. Formally, we first apply average pool- ing to compress unimodal features. Then, we calculate the attention weights to emphasize important modalities:
Level 3. Emotion wheel provides natural grouping information, with core emotions displayed in the inner part and more nuanced labels in the outer part (Plutchik, 1980). Since there is no consensus on the emotion wheel, we use K emotion wheels (see Appendix K). For each sector of the emotion wheel Wk. k E [1, K), we map all outer labels to the corresponding inner labels. This mapping function is denoted as Fig (.).
The above grouping functions can be summarized as:
2,- Pooling (Z.), Zy = Pooling (Zo);
(5)
Gwa( = (Fl, (Fi, ()), E [1,
(8)
Zuv = Concat (2.: 2
(6) For each sample, the number of labels is variable. There- fore, we define a set-based evaluation metric. Specifically, suppose the dataset contains N samples. For sample 24, the true labels are Y,- {u. yー= and the predieted labels are The evaluation metric is defined as follows:
ZF 2 (W- Flatten (av) )Zmv))
(7)
Y = (0 A 1=1"
where 2. E RR, Zx R2, Zav E R2xd, and W R2x2d Finally, we obtain the fused features Zr Rd.
N
Gun (Y:) n G (Yi)
Regarding computational efficiency, the pre-fusion opera- tion relies on Q-Former or attention mechanisms, which are significantly less computationally intensive than LLMs. Theoretically, the Q-Former enables cross-modal interaction by distilling multimodal content into query tokens, whereas the attention mechanism achieves this by dynamically com- puting attention weights based on multimodal inputs.
k Precision
(9)
N
Gua (Y
1=1
N
Gwa (Y,) Gase (81)
Recall =
(10)
N
Guuz (Y:)
Precision & x Recall
F = 2 x
(11)
Precisions + Recall
4. MER-UniBench: Evaluation Benchmark
Finally, we compute the average results across different emotion wheels for ranking. Take Fs as an example:
We introduce MER- HIER-UniBench, a comprehensive evaluation benchmark designed to cover typical MER tasks. Given the free-form, natural language output style of MLLMs (see Appendix D), also design specialized evaluation metrics. More details can be found in Appendix J.
K
Fs
FS.
(12)
K
k=
Here, Precisions indicates the number of correctly predicted labels, and Recalls indicates whether the prediction covers all ground truth. Fs is a harmonic mean of two metrics. Since Fs considers both accuracy and completeness, we use it as the primary metric, with Precisions and Recalls serving as secondary metrics. To extract the predicted emotions Y:, we employ the strategy mentioned in Appendix E.
Fine-grained Emotion Recognition. This task enables the prediction of fine-grained emotions, extending beyond basic categories. OV-MERD (Lian et al., 2024a) is a typical dataset for this task. To improve the reliability of the eval- uation results, we expand its dataset size, referring to it as OV-MERD+. For the evaluation metrics, we draw inspira- tion from previous work (Lian et al., 2024a) and calculate results in two steps: eliminating the impact of synonyms and using set-level metrics. First, we apply three-level grouping strategy to mitigate the impact of synonyms:
Basic Emotion Recognition. This task is a key branch of MER, whose main goal is to select the most likely label from a fixed set of basic emotions. For this task, we select four widely used benchmark datasets: MER2023 (Lian et al., 2023), MER2024 (Lian et al., 2024b), IEMOCAP (Busso et al., 2008), and MELD (Poria et al., 2019). However, the output of MILMss, Y, = (i,² J==1- contains variable number of labels, while the dataset only provides one true label y;- In this case, traditional metrics (such as accuracy) are not suitable for performance evaluation. To address this, we propose new metric, hit rate, which is set to 1
We map different forms of emotion words to their base form. For example, we map happier and happiness to happy. This function is denoted as Fi, (.).
Level 1.
Level 2.
We map synonyms to a unified label. For example, we map happy and joyful to happy. This mapping function represented as Fl, (+).
6

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Table 2: Main results. This table presents the results for the primary metrics, with Section 4 outlining the primary metrics for each task. The values for other metrics can be found in Appendix L. In this table, "MOSI", "MOSEI", "SIMS", and "SIMS v2" refer to CMU-MOSI, CMU-MOSHI, CH-SIMS, and CH-SIMS v2, respectively. The last column shows the wase mean score, i.e., the average score across all datasets.
Modality
Basic
Sentiment
Fine-grained OV-MERD+
Mean
MER2023
MER2024
MELD
IEMOCAP
MOSI
MOSEI
SIMS
SIMS v2
OueLLM
X
25,52
17.21
28.32
33.44
64.01
54,09
63.39
61.98
22.25
41.14
SECap
V
V
40,95
52.46
25.56
36.92
55.76
54.18
59.51
57.41
36.97
46,
PandaGPT
V
X
V
33.57
39.04
31.91
36.55
66.06
61.33
62.93
31.33
46.84
Qwen-Audio
V
X
V
41.85
31.61
49.09
35.47
70.09
46.90
70.73
65.26
32.36
49.26
SALMONN
55.53
45.38
45.62
46.84
81.00
67.03
68.69
65.93
45,00
57.89
AffectGPT
V
V
72.94
73.41
56.63
55.68
83.46
80.74
82.99
83.75
59.98
72.18
Outer
X
V
16.41
14.65
22.57
29.08
52.89
50.44
57.56
53.12
16.63
34.82
Video-LLaVA
V
36,93
30.25
30.73
38.95
56.37
61. 64
53.28
57.45
34.00
44.40
PandaGPT
39.13
47.16
38.33
47.21
58.50
64.25
62.07
65.25
35.07
50.77
Video-ChatGPT
44.86
46.80
37.33
56.83
54.42
63.12
64.82
65.80
39.80
52.64
VideoChar2
X
33.67
54.50
36.64
48.70
66.84
54,32
69.49
70.66
39.21
$2.67
LLaMA-VID
50.72
57.60
42.75
46.02
61.78
63.89
69.35
67.48
45.01
56.07
VideoChat
48.73
57.30
41.11
48.38
65.13
63.61
69.52
72.14
44.52
56.71
Vhat-UniVi
X
57.62
65.67
45.61
52.37
54,53
63.18
68.15
66.36
48.00
$7.94
mPL.G-Owl
X
V
56.86
$9.89
49,11
55.54
72.40
72.91
72.13
75.00
48. 18
62,45
AffecGPPT
V
74.58
75.29
57.63
62.19
82.39
81.57
87.20
86,29
61.65
74.31
PandaGPT
V
40.21
51.89
37.88
44.04
61.92
67.61
68.38
67.23
37.12
52.92
Emotion-T.LaMe
59.38
73.62
46.76
55.47
66.13
67.66
78.32
77.23
$2.97
64.17
AffecGPPT
78.54
78.80
55.65
60.54
81.30
80.90
88,49
86.18
62.52
74.77
when y/ E Y, and 0 otherwise. Considering that Y, is in free-form and y, belongs to basic emotions V, we may encounter cases where Di & y. To this end, we use the mapping function Gwa( (-) and define the metric follows:
datasets: CMU-MOSI (Zadeh et al., 2017), CMU-MOSEI (Zadeh et al., 2018), CH-SIMS (Yu et al., 2020), and CH- SIMS v2 (Liu et al., 2022b). For these benchmark datasets, the original labels are floating-poin values, ranging from 3]. We map scores of < to negative sen- timent and scores of >>>>>>>>:: 0 to positive sentiment. To extract sentiment labels from the MLLM's output, we employ the strategy outlined in Appendix E. Following previous work (Zadeh et al., 2017; 2018), we evaluate performance using accuracy (ACC) and weighted average F-score (WAF). Due to the inherent label imbalance, we choose WAF as the primary metric and ACC as the secondary metric.
[-1, or [-3,3].
N HIT
Σ I Gu E Giou (Y.),
(13)
N
where I[-] is an indicator function. The motivation for this metric stems from the fact that basic emotion recognition tasks typically provide majority-voted labels ys, which are generally reliable. However, emotion descriptions produce free-form outputs Y, that may contain multiple labels, in- cluding fine-grainen ones beyond basic emotions. Therefore, we use the hit rate as the metric, ensuring that the basic label y: should be at least in Y,
5. Results and Discussion
In this section, we present the experimental results and provide an in-depth analysis. Detailed implementation in- formation can be found in Appendix B.
During the design of this metric, we also explored the possi- bility evaluating potentially incorrect labels Y, How- ever, the labels in Y, that differ from the basic label y are not necessarily incorrect they may represent some fine- grained emotions not covered by basic categories. Since basic emotion recognition tasks lack fine-grained reference labels, we have not yet established appropriate evaluation metrics for this purpose. This remains an important research direction for our future work.
Main Results. We compare the performance of Affect- GPT with other MLLMS on MER-UniBench. Since our imputs include audio. video, and text content, we only select MILMs that support at least audio video. For models that support both audio and video, we test different modality combinations. Model cards are provided in Appendix C. To ensure a fair comparison, we use their official weights and input corresponding multimodal content, asking them to in- fer the emotional state. In Table 2, AffectGPT significantly outperforms existing MLL.Ms. This can be attributed the
Sentiment Analysis. This task is more fundamental than the two tasks mentioned above, aiming to predict the sen- timent polarity. For this task, we select four benchmark

---

AffectGPT: A New Datasct, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Table 3: Dataset comparison. We only change the training dataset, keeping all other aspects consistent. This table re- ports the mean score across all datasets in NER-UniBenchh
Table 4: Necessity of filtering, Besides the results on MER- UniBench, we also provide task-level results. "E" and "S" are abbreviations for emotion and sentiment, respectively.
Dataset
Filtering
MER-UniBench
Low
High
Fine-grained
Basic
Sentiment
Vi-R-UniBench
MiniGPT4
31.74
X
58.42
64.36
76.09
68.91
35,53
X
58.61
62.78
79.04
69.54
VideoChat
x
37.16
E
61.72
67.83
82.74
73.78
General
37.63
S 61.00
66.33
85.04
74.05
Instruction
LLaVA
x
46.69
E+S
62.52
68.38
84,22
74.77
46.27
21.65
WavCaps
Table 5: Role of pre-fusion operation.
37.91
EmoVIT
51.05
MAFW
58.16
Pre-fusion
Fine-grained
Basic
Sentiment
MER-UniBende
Emotion
MERR-Coarse
49.85
x
61.21
66.28
82.57
72.95
Description
MERR-Fine
64.55
Q-Former
62.65
66.89
84.30
74.16
MER-C MER-Caption
68.91
Attention
62.52
68.38
84.22
74.77
MER-Caption+
74.77
of an automatic annotation strategy without manual checks. However, the experimental results in Table show that MER-Captions achieves significantly better performance than the manually annotated MAFW dataset. The main reason is that humans tend to focus on major clues, which can easily lead to incomplete descriptions. These results confirm that, despite the lack of manual checks MER- Caption+, we can still ensure the quality of the labels. the future, we will investigate other post-filtering techniques to further improve MER- SEER-Caption+'s annotation quality.
fact that current instruction datasets pay little attention to MER tasks. Additionally, existing models place the entire multimodal fusion within the LLM, which is insufficient for MER tasks that require effective multimodal integra- tion. By leveraging our newly proposed dataset and model, we provide a promising approach to enhancing emotion understanding capability in MILMMs. Meanwhile, for differ- ent datasets, increasing the input modality does not always improve performance, as may also introduce irrelevant information that interferes with emotional understanding.
Ablation Study on MER-Caption. As shown in Table 4, compared to the results without filtering or with only low- level filtering, our two-level filtering leads to a performance improvement, further verifying the effectiveness of our fil- tering technique. These findings underscore that dataset quality is as critical as quantity, and fewer training samples do not necessarily lead to worse performance. Please see Appendix M for more details.
Effectiveness of MER-Caption. Table 3 compares the performance of MER-Caption with existing datasets. For a fair comparison, we use the same model architecture and experimental settings and only change the training data. For general instruction datasets, we further conduct filtering experiments to remove samples without emotion-related content, emphasizing emotion-related subsets. Specifically, we use the prompt in Appendix and extract emotion labels from each instruction-anseee answer pair. Samples yiclding empty emotion outputs are removed.
A Ablation Study on Model. Table 5 compares different architectures and examines the impact of pre-fusion opera- tions. Our results show that pre-fusion operations generally improve performance. This highlights the importance of treating cross-modal interactions as separate modules to more effectively capture multimodal characteristics.
In Table 3, the excellent performance of MER-Caption proves the limitations of current datasets in addressing MER. On the one hand, general instruction datasets pay insuffi- cient attention to emotion- related tasks. On the other hand, emotional description datasets often suffer from inadequate dataset scales or insufficient annotation quality. Therefore, our dataset can serve as an important complement to exist- ing datasets. Meanwhile, for the general instruction datasets, the filtering approach is less effective on the LLaVA and VideoChat datasets. We hypothesize that the detailed de- scriptions in non-emotion subsets may also provide valuable cues for inferring emotional states in some scenarios.
nnlyssi of Input Impact. Table 6 reveals the impact of different inputs. The distinction between "face" and "frame" lies in whether an additional face extractor used to extract faces from frames. We observe a general trend: multimodal results outperform unimodal results. These find- ings suggest that humans express emotions through multiple modalities, and integrating them leads to improved perfor- mance. Additionally, face inputs slightly outperform frame inputs, and their combination does not result in further im- provement. This suggests that current MER datasets mainly
Furthermore, we would like to acknowledge that MER- Caption+ may contain inaccurate descriptions due to the use
8

---

AffectGPT: A New Datasct, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Table 6: Input impact analysis. The difference between "face" and "frame" is whether an additional face extractor is used extract faces from frames.
6800
Input
MER-UniBench
audio
face
frame
LLM
Audio Encoder
(c) Video Encoder
x
60.08
unimodal
X
v
x
x
60.47
X
x
V
x
59.47
Figure 4: Ablation studies on LLMs, audio encoders, and video encoders.
X
x
x
67.44
74.77
multimodal
x
73.39
74.60
Table 8: Impact of rank value in LoRA. In this table, we count the increase in trainable parameters when using LoRA for the LLM branch. The first row represents the model without the LoRA module.
Table 7: User study.
MER-Caption+
Dataset
Wins
Losses
Ties
Rank
Increased Parameters
MER-UniBench
MERR-Cuars
0.86
0.04
0,10
0
73.30
MERR-Find
0.59
0.27
0.14
8
30,185,088
74.65
16
40,370,176
74.77
32
80,740,352
74.92
focus on people, with limited emotional information con- veyed through the environment. As a result, in this paper, we default to using audio, face, and text as the inputs.
content understanding tasks (e.g., ASR), where audio con- tent plays a critical role in emotion recognition. Similarly, for video encoders (Figure 4(c)), CLIP VIT marginally out- performs EVA_CLIP and DINOv2, aligning with findings from MERBench (Lian et al., 2024c), a unified benchmark for traditional categorical frameworks. These results sug- gest that insights derived from traditional categorical frame- works, such as encoder selection, may also be applicable to HLLM-based descriptive frameworks.
User Study. We conduct a user study to evaluate the qual- ity of our proposed dataset. Since MERR-Fine and MERR- Coarse (Cheng et al., 2024) share some samples with our dataset, randomly select 20 overlapping samples. We then hire four expert annotators and present them with two descriptions for each sample: one from our dataset and one from the other datasets. The annotators are asked to watch the video and select the more accurate description. As shown in Table 7, our dataset provides more accurate descriptions than both the model-based MERR-Coars and the human-filtered MERR-Fine, thereby validating the ef- fectiveness of our proposed annotation strategy.
Role of LoRA in LLMs. In Table 8, we count the in- crease in trainable parameters when using LoRA (Hu et al., 2022) for the LLM branch. The first represents the model without the LoRA module. In Table 8, fine-tuning the LLM with LoRA improves performance compared to models without LoRA. However, increasing the rank for loRA-based models does not yield significant performance gains and instead increases computational costs.
Choice of LLMs. This paper adopts Qwen2.5 the de- fault LLM. In Figure 4(a), we further explore the impact of different LLMs. Experimental results show that the perfor- mance difference brought by LLM is limited. These results verify that the superior performance of AffectGPT over the existing MLLMs does not come from LLM but from our proposed emotion description dataset and model.
6. Conclusion
This paper aims to enhance the emotional understanding of MLLMs from three aspects: (1) the dataset MER- MEa-CCation, which uses a model-led human-assisted strategy to create a large-scale dataset with guaranteed quality; (2) the model AffectGPT, which enhances multimodal fusion by moving cross-modal interactions outside of the LLM; and (3) the benchmark, which provides comprehensive evaluation met- rics tailored to the free-form, natural language output style of MLLMs. Extensive experiments validate the effective- ness of our model and dataset. This work lays the foundation for building MLLMs with emotional understanding, con- tributing to the advancement of emotion AI.
Choice of Audio and Video Encoders. In Figures 4(b) and 4(c), the choice of audio and video encoder has a min- imal impact on performance. This underscores that Af-
fecIGPT's exceptional performance is primarily driven by our proposed high-quality. large-scale dataset and effective framework, rather than the specific acoustic or visual en- coders used, For audio encoders (Figure 4(b)), ImageBind exhibits slightly inferior performance compared to other audio encoders. This may be attributed to the fact that other audio encoders are predominantly utilized audio

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Cheng, Z., Cheng, Z.-Q.. He, J.-Y., Wang, K., Lin, Y., Lian, This work is supported by the Excellent Youth Program of State Key Laboratory of Multimodal Artificial Intelli- gence Systems (MAIS202431D), the National Natural Sci- ence Foundation of China (62201572, 62322120, 61831022, 62276259, U21B2010, 62271083, 62306316, 62176165, 62206136, 62476146), the Stable Support Projects for Shen- zhen Higher Education Institutions (202207 181 10918001), the Young Elite Scientists Sponsorship Program by CAST (20240NC001), and the University of Oulu& Research Council of Finland Profi (352788).
Acknowledgments
Z., Peng, X., and Hauptmann, A. Emotion-llamar Multi- modal emotion recognition and reasoning with instruction tuning. Advances in Neural Information Processing Sys- tems, 37:1 17:110805-1 10853, 2024.
Chu, Y., Xu, J., Zhou, X., Yang, Q. Zhang, S., Yan, Z. Zhou, C., and Zhou, J. Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models. arXiv preprint arXiv:2311.07911 2023.
Cowen, A. S. and Keltner, D. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the national academy of sciences, 114 (88):E7900-E7900 2017.
Impact Statement
Social Impact. Emotion plays an important role in hu- man communication, conveying human intentions and deep thoughts. As Minsky (Minsky, 1988) stated: The question not whether intelligent machines can have any emotions, but whether machines can be intelligent without any emo- tions. The development of MER technology can enhance the human-computer interaction experience.
Demszky, D., Movshovitz D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 4040-4054, 2020.
Du, S., Tao, Y., and Martinez, A. M. Compound facial expressions of emotion. Proceedings of the national academy of sciences, 111(15)EE144-E1445 2014.
Ethics Statement. This paper does not involve the collec- tion of new data. The original data comes from the unlabeled part of MER2024 (Lian et al., 2024b). with permission from the dataset owners. The annotation process does not involve hiring external annotators, and no ethical issues are associ- ated with this process. Additionally, this dataset under the license of CC BY-NC 4.0, requiring researchers to use our dataset responsibly. Therefore, no ethical concerns are raised in this paper.
Ekman, P. and Keltner, D. Universal facial expressions of emotion. Califormia mental health research digest, 8(4): 151-158, 1970.
restrict the use of
Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., et al. Challenges in representation learn- ing: A report on three machine learning contests. In Neural information processing: 20th international con- ference, ICONIP 2013, daegu, korea, november 3-7, 2013. Proceedings, Pari III 20, pp. 117-124. Springer, 2013.
References
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F,, Weiss, B., et al. A database of german emotional speech. In Interspeech, volume 5, pp. 1517-1520, 2005.
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., and Marsic, I. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, pp. 2225 2235, 2018
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. S. Iemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335-
359, 2008
Chen, H., Liu, Li, X., Shi, H., and Zhao, G. Analyze spontaneous gestures for emotional stress state recog- nition: A micro-gesture dataset and analysis with deep learning. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1-8. IEEE, 2019.
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26584-26595, 2024.
Hazarika, D., Zimmermann, R., and Poria, S. Misa: Modality and-specific representations for mul- timodal sentiment analysis. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 1122- 1131, 2020.
Chen, H., Shi, H., Liu, X., X., and Zhao, G. Smg: A micro- micro-gesture dataset towards spontaneous body gestures for emotional stress state analysis. International Journal of Computer Vision, 131(6):1344) (6:6)):336-11366 1366, 2023.
10

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Hsu, C.-C., Chen, S.-Y., Kuo, C.-C., K. Huang, T.-H., and Ku, L.-W. Emotionlines: An emotion corpus of multi- party conversations. In Proceedings of the Eleventh Inter- national Conference on Language Resources and Evalua- fion, pp. 1597-1601, 2018.
Mybench: A comprehensive multi-modal video under- standing benchmark. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Li, S., Deng, W., and Du, J. Reliable crowdsourcing and deep
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked pre- diction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-3460, 2021.
learning for expression recogni- tion in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2852- 2861, 2017.
Li, Y., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2024b.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2)3, 2022.
Lian, Z., Sun, H., Sun, I.., Chen, K., Xu, M., Wang, K., Xu, K., He, Y., Li, Y., Zhao, J., et al. Mer 2023: Multi- label learning, modality robustness, and semi-supervised learning. In Proceedings of the 31stACM ACM International Conference on Multimedia, pp. 9610-9614, 2023.
Huang, Z., Zhao, J., and Jin, Q. Ecr-chain: Advancing generative language models to better emotion-caut cause rea- soners through reasoning chains. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI, pp. 6288-6296, 2024.
Lian, Z., Sun, H., Sun, L., Chen, L., Chen, H., Gu, H., Wen, Z., Chen, S., Zhang, S., Yao, H., et al. Open-vocabulary multimodal emotion recognition: Dataset, metric, and benchmark. arXiv preprint arXiv:2410.01495, 20244,
Izard, C. E., Libero, D. Z., Putnam, P., and Haynes, O. M. Stability of emotion experiences and their relations to traits of personality. Journal of personality and social psychology, 64(5):847, 1993.
2024a.
Lian, Z., Sun, H., Sun, L., Wen, Z., Zhang, S., Chen, S., Gu, H., Zhao, J., Ma, Z., Chen, X., et al. Mer 2024: Semi-supervised learning, noise robustness, and open- vocabulary multimodal emotion recognition. In Proceed- ings of the 2nd International Workshop Multimodal and Responsible Affective Computing, pp. 41-48, 2024b.
Jang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C.,, and Liu, J. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM international conference on multimedia, pp. 2881-288188 2020.
Jin, P., Takanobu, R., Zhang, W., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition, pp. 13700-13710, 2024.
Lian, Z., Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., and Tao, J. Merbench: A unified evaluation bench- mark for multimodal emotion recognition. arXiv preprint arXiv: 2401 240.0444 2024c
Lian, Z., Sun, L., Sun, H., Chen, K., Wen, Z., Gu, H., Liu, B., and Tao, J. Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition. Information Fusion, 108: 1 02367, 2024d.
Kövecses, Z. Metaphor and emotion: Language, culture, and body in human feeling. Cambridge University Press, 2003.
Li, B., Zhang, Y., Chen, I.., Wang, J., Yang, J., and Liu, 7. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint PrXiv:2305.03726 2023a.
Liang, Z., Xu, Y., Hong, Y., Shang, P., Wang. Q., Fu, Q., and Liu, K. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Com- puter, Artificial Intelligence and Control Engineering, pp. 405 409, 2024.
Li, Li, D., Savarese, S., and Hoi, S. Blip-2: Bootstrapping Language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pp. 1-13, 2023b.
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-Ilava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 5971-5984, 2024.
Li, K., He, Y., Wang. Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L., and Qiao, Y. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023c.
Liu, S., Zheng, C., Demasi, O., Sabour, S., Li, Y., Yu, Z., Jiang, Y., and Huang, M. Towards emotional support dia- log systems. In Proceedings of the 59th Annual Meeting
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., Wang, L.,, and Qiao, Y.

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
of the Association for Computational Linguistics and the 1th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 3469-3483, 2021a.
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., and Morency, L..P.. Context-dependent sentiment anal- ysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties, volume 1, pp. 873-883, 2017.
Liu, X., Shi, H., Chen, H., Yu, Z., Li, X., and Zhao, G. imigue: An identity-free video dataset for micro-gesture understanding and emotion analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 10631-10642, 2021b.
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. Meld: A multimodal multi-party dataset for emotion recognition in conversations. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, pp. 527-536, 2019.
Liu, Y., Dai, W., Feng, C., Wang, W., Yin, G., Zeng. J., and Shan, Mafw: A large-scale, multi-modal, compound af- fective database for dynamic facial expression recognition in the wild. In Proceedings of the 30th ACM International Conference Multimedia, pp. 24 32. 2022a.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Liu, Y., Yuan, Z., Mao, H., Liang. Z.,, Yang. W., Qiu, Y., Cheng, T., Li, X., Xu, H., and Gao, K. Make acous- tic and visual cues matter: Ch-sims v2. 0 dataset and av-mixup consistent module. In Proceedings of the In- ternational Conference on Multimodal Interaction, pp- 247-258, 2022b.
Schutz, P. A. Emotion in education, 2007.
Spezialetti, M., Placidi, G., and Rossi, S. Emotion recogni- tion for human-robot interaction: Recent advances and fu- ture perspectives. Frontiers Robotics and A1, 7:532279, 2020.
Lotfian, R. and Busso, C. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions on Affective Computing, 2017.
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instruction-follow them all. In Proceedings of the 1st Workshop on Taming Large Lan- guage Models: Controllability the era of Interactive Assistants, pp. 11-23, 2023.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties Vollme 1: Long Papers), pp. 12585-12602, 2024.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., MA, Z., and Zhang, C. Salmonn: Towards generic hearing abilities for large language models. In The Twelfth International Conference Learning Representations, 2023.
Tao, R., Pan, Z., Das, R. K., Qian, X., Shou, M. Z., and Li, H. Is someone speaking? exploring long-term temporal features for audio-visuda visual active speaker detection. In Proceedings of the 29th ACM international conference on multimedia, pp. 3927-3935, 2021.
Matsumoto, D. Culture and emotion. The handbook of culture and psychology, pp. 171-194, 2001.
Minsky, M. Society of mind. Simon and Schuster, 1988.
Mollahosseini, A., Hasani, B., and Mahoor, M. H. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18-33 31,, 2017.
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, pp. 6558-6569, 2019.
OpenAl. Chatgpt, 2022. URL
com/ Pfister, T., Li, X., Zhao, G., and Pietikäinen, M. Recognising spontaneous facial micro-expressions. In 20/ interna- tional conference on computer vision, pp. 1449-1456. IEEE, 2011.
Wang. Y., Sun, Y., Huang, Y., Liu, Z., Gao, S., Zhang, W., Ge, W., and Zhang, W. Ferv39k: A large-scale multi- scene dataset for facial expression recognition in videos. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 20922 20922-20931, 20931, 2022.
Xie, H., Peng, C.-J., Tseng, Y.-W., Chen, H.-J., Hsu, C.-F., Shuai, H.-H., and Cheng, W.-H. Emovit: Revolutioniz- ing emotion insights with visual instruction tuning. In
Plutchik, R. A general psychoevolutionary theory of emo- tion. Emotion: Theory, research, and experience, 1, 1980.
12

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26596-26605, 2024.
Xu, Y., Chen, H., Yu, J., Huang, Q., Wu, Z., Zhang, S.- X., Li, G., Luo, Y., and Gu, R. Secap: Speech emotion captioning with large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, pp- 19323-19331, 2024.
Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024.
Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A., Shi, P., Shi, Y., et mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint arXiv:2304. 1..004144177 14178, 2023.
Yu, W., Xu, H., Meng. F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. Ch-sims: chinese multimodal sentiment analysis dataset with fine- ane-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 3718-3727, 2020.
Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the Conference Empiri- cal Methods in Natural Language Processing, pp. 1103- 1114, 2017.
Zadeh, A. B., Liang, P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fu- sion graph. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics ( Volume 1: Long Papers), pp. 2236-2266, 2018
Zhang, Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video understand- ing. In Proceedings of the 2023 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pp. 543-553, 2023.
13

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
A. Related Works
This paper focuses on constructing datasets and designing models to enhance the emotional understanding capability of MLLMs. In this section, we mainly review related work in these two aspects.
A.1. Emotion Dataset
Emotion datasets are the foundation for building MER systems (Wang et al., 2022; Chen et al., 2023; Liu et al., 2021b; Chen al., 2019). Most research has focused on building categorical datasets, where basic emotions are first defined, and annotators are asked to select the most likely one (Goodfellow et al., 2013) or multiple (Li et al., 2017) labels from basic emotions. However, emotions are often diverse (Demszky et al., 2020) and can coexist (Du et al., 2014), making it challenging for categorical datasets to fully capture these complex emotions.
To address this, recent studies have shifted from categorical datasets to descriptive datasets, as emotion descriptions provide greater flexibility and enable the deseription of complex emotions in natural language. To construct such datasets, Liu et al. (2022a) used human-based annotation strategy to capture the environment, body movements, facial expressions, and other emotion-related cues. However, the high annotation cost limits the scalability of these datasets. With the development of MLLMs, Cheng al. (2024) used a more cost-effective automatic annotation method, where MLLMs used to extract emotion-relatee descriptions from audio, facial expressions, and visual objects. However, they lacked pre-experimentationn on MILM selection, relying on empirical model choices, leading to insufficient label quality. In this paper, we propose a solution to balance label quality and dataset size. By leveraging high-quality human- baaed datasets to guide description generation and sample filtering, we achieve a auslity- automatic annotation process and ultimately construct -aatiinn
A.2. Emotion Models
Emotion models are closely related to the training corpus. For categorical datasets, researchers often build classifiers to map multimodal human information to corresponding emotion labels. Apart from choosing the architecture (such as CNN, RNN, or Transformers), most research focuses on how to align and fuse multimodal information. For example, Hazarika et al. (2020) introduced decomposition module split features into modality-specific and modality-invariant representations. Gu et al. (2018 () aligned different modalities at the word level and then learned time-dependent cross- moosll interactions. Tsai et al. (2019) proposed using cross-attention to align features in the latent space. More recently, Lian et al. (2024c) conducted fair comparison of various fusion and alignment strategies, showing that cemporal-preserviee features do not always outperform time-compressed features, suggesting that MER may be more suitable to solve from a global perspective.
For descriptive datasets, due to their natural language style output, the framework needs to shift from traditional discriminative methods to generative methods. With the development of LLMs and MILMM, researchers have started to build models based on them. For example, Huang et al. (2024) used Vicuna as the language model, jointly training emotion labels and descriptions. Xie et al. (2024) used the instruction-awarc Q-Former module to learn the mapping between input images and emotional descriptions. Cheng et al. (2024) integrated different encoders to understand multimodal inputs and used LLaMA-2 as an LLM decoder. However, current models either only focus on unimodal information (Huang et al., 2024; Xie et al., 2024) or leave all cross-modal interactions to the LLM (Cheng et al., 2024), which is insufficient for solving MER tasks with multimodal characteristics. To this end, we introduce the AffectGPT model in this paper.
B. Implementation Details
Our choice of unimodal encoders is guided by previous research (Lian et al., 2024c), using CLIP VIT-L (Radford et al., 2021) as the visual encoder and HUBERT-L (Hsu et al., 2021) as the acoustic encoder. Given the remarkable performance of Qwen-2.5 ( (Yang et al.. 2024), we choose it as the LLM. To ensure training efficiency, we only fine-tune an extra LoRA module (in the LLM), projector, and fre-fusion branch, while freezing the weights of the LLM and unimodal encoders (see Figure 3). We default to setting the rank the LoRA module to 16. This approach reduces GPU memory usage and speeds up training. Additionally, through preliminary experiments, we found that pre-training on other instruction datasets followed by second-stage training on MER-Caption did not lead to performance improvements. The primary reason is the large scale of our dataset and the limited focus on MER in current instruction datasets. Therefore, we do not perform multi- training in our experiments. All models are implemented in PyTorch and conducted training and inference on 80GB NVIDIA Tesla A A100 GPU. During training, we set the maximum number of epochs to 60, each epoch contains 5000 iterations, and the
14

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
batch size of each iteration is 3. To optimize all trainable parameters, the AdamW optimizer and the learning rate le-5. For more implementation details, please refer to the code provided in http://gttubbcm//eeeeoaaaaaa/att
C. Details about MLLMs
Table 9 provides model cards for different MLLMs, including reference papers, supported modalities, and links to pre-trained weights.
Table 9: Model cards for MLLMs.
Supported Modality
Link
Otter (Li ct al., 2023a)
Video, Text
https://tthbb..om//uucccaaaa///
VideoChal (Li et al., 2023c)
Video, Text
chat
VideoChat2 (Li et al., 2024a)
Video, Text
chat
Video-LLaVA (Lin al., 2024)
Video, Text
Video-LLaMA (Zhang al., 2023)
Video, Text
Video-ChatGPT (Manz al., 2024)
Video. Text
LLaMA-VID al., 2024b)
Video, Text
mPLUG-Owl (Ye al., 2023)
Video, Text
Chat-UniVi (Jin al., 2024)
Video, Text
SALMONN (Tang et al., 2023)
Audio, Text
Qwen-Audio (Chu el al., 2023)
Audio, Text
SECap (Xu et al., 2024)
Audio, Text
http://gttabb.com//tnncssSSCCCC
Onel.IM (Han et al., 2024)
Audio, Viden, Text hutp:://shhaaaccmmmssss////olM
PandaGPT (Su et al., 2023)
Audio, Viden, Text http://ghhhaaccmmmyyuuuu///ddddr
Emotion-LLaMee (Cheng et al., 2024)
Audio, Video. Text
https:/ https///iibbbboooo/b/o/////////n LLMAA
15

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
D. Visualization of MILM Outputs
Figure 5 provides an example to visualize the outputs of different MLLMs. These outputs contain varying numbers of emotions, with emotion labels that are open-enden and not restricted to any predefined taxonomy. Therefore, traditional classification metrics, such as accuracy and Fl score, are not suitable for performance evaluation, In this paper, we propose evaluation metries specifically tailored for the fore-form, natural language output style of MLLMs.
Subtitle: Why are you all looking at me like that? So, as long as it's a woman, does she have to have a relationship with me?
In the video, a man is seen speaking to the camera while making facial expressions and gestures. The man's body language and expressions suggest that he might be trying to convey a message or share his thoughts with the audience. The environment appears to be a simple setting, with no significant background details. The man's emotional state is not entirely clear from the video alone, as depends on the context of his words and actions. However, based on his facial expressions and body movements, we can infer that he might be feeling passionate, expressive, even slightly nervous while addressing the camera. The video's content and the man's demeanor invite the viewer to interpret his emotions and message further.
Chat-UniVi-mann:
trying
LLaMA-VID-man The man in the video appears to be in a state of confusion or uncertainty, as he looks around and appears to be unsure of what to do. His facial expressions and body movements suggest that is trying to process information or make decision. The environment in which speaking, which is laboratory, may also contribute to his emotional state, as it may suggest that dealing with a difficult or complex problem. Overall, difficult predict the exact emotional state of individual based the information provided, but is clear that he experiencing some level of confusion or uncertainty.
SALMONN:
It is difficult to predict the emotional state of the individual in the audio without more context. However, based on the given information, it is possible that the individual is feeling frustrated or misunderstood. They may be feeling like they are being judged or scrutinized by others, and this could causing them to feel upset or angry. Altematively, they may be feeling defensive or protective of themselves, and this could causing them to become aggressive or confrontational. Ultimately, without more information, is impossible to accurately predict the emotional state of the individual in the audio.
Video-I.La VA: The man in the video is sitting in front of a camera and talking. He appears to be in relaxed state, with his hands resting on his lap. His facial expressions are calm und composed, and his body language suggests that he is comfortuble in front of the camera. The environment around him is well-lit, and there are no visible signs of distractions or interruptions. The subtitle content of video does not provide any specific information about individual's emotions or the context the video. However, based on the visual cues, it can be inferred that man is likely in positive emotional state.
Cheeo-ChatGP:: In the video, a man is seen talking to the camera while woman looks on. The man is wearing white shirt and tie, and he sppears to be engaged in a conversation with the woman. The woman's facial expression is not elearly visible, but her body language suggests that she is listening attentively to the man. The environment appears to be an office setting, with a desk and a computer visible in the background. The acoustic information, such as the sound of the man speaking, is not provided. The subtities in the video indicate that the man is asking the woman she has relationship with him, and the woman responds by saying that she is not interested in him. Based on these visual and auditory cues, difficult determine the exact emotional state of the individual. However, clear that the man is engaged in conversation with the woman, and the woman is listening attentively to him.
The video docsn't seem to have any information related to the given subtitle content. It would be best to provide more details or context to help me understand the context and answer the question accurately.
ChatoChat2:
SECap: feeling astonished
Figure 5: Visualization of MLLM outputs.
E. Prompt for Label Extraction
To extract emotion labels from MILM outputs, we use Qwen2.5 and apply the following prompt:
Please assume the role of an expert in the field of emotions. We provide clues that may be related to the emotions of the characters. Based on the provided clues, please identify the emotional states of the main character. Please separate different emotional categories with commas and output only the clearly identifiable emotional categories in a list format. If none are identified, please output an empty list.
For sentiment analysis, we use the multi-step prediction process. Specifically, we first extract emotion labels using the prompt above, and then apply the following prompt for sentiment analysis:
Please act as an expert in the field of emotions. We provide a few words to describe the emotions of a character. Please choose the most likely sentiment from the given condidates: [positive, negative, neutral].
16

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
F. Choice of Description Generation Strategy
This section aims to determine the optimal strategy for generating descriptions. In Table 10, we present the results of preliminary experiments. First, we evaluate the performance of different ALLMs and VLLMs. Then, we investigate whether combining these models leads improved performance. To do this, use GPT-3.5 integrate audio and video cues, extracted by the ALLM and VLLM, with text content. As shown in Table 10, observe that these combinations generally outperform the use of either ALLM or VLLM alone. Based on these findings, we select SALMONN as the ALLM for generating audio cues, Chat-UniVi as the VLLM for generating visual cues, and GPT-3.5 to combine the audio, video, and text cues, resulting in the final descriptions.
We would like clarify that, in this paper, we do not use combined results for model selection. Instead, we rely on the performance of individual models. Specifically, for VLLM, Chat-UniVi outperforms mPLUG-Owl and :reeo-ChttPPT; for ALLM, SALMONN outperforms SECap. As a result, we employ the combination of Chat-UniVi and SALMONN for description generation. The combination experiments cues can enhance performance. In future work, we will conduct additional experiments where combined results are used for model selection. For example, leveraging the combination of SALMONN and Chat-UniVi for description generation.
primarily designed to demonstrate that integrating multimodal
Table 10: Preliminary experiments. We choose Fs the primary metric, this metric considers both accuracy and completeness.
Model
Fs(1)
Precisions(1)
Recalls(t)
SECap (Xu et al., 2024)
45.72
54.52.0.15
39.37
SALMONN (Tang al., 2023)
47.9640.04
50.20
45.9240.04
Video-ChaIGPT (Maaz et al., 2024)
50.52
54.03
47.44.0.07
mPLUG-Owl (Ye et al., 2023)
52.73
54.54
51.04
Chat-UniVi (Jin al., 2024)
53.08 20.01
53.68
52.50
SECap mPLUGGOWw
56.69+0.08
50.05:10.3
65.3840.33
SFCap + Vidco-ChatGPP
56.90
52.0340.04
62.79
SECap Chat-UniVi
57.34
48.85.1029
69.41
SALMONN + Video-ChatGPP
20.19
53.1640.17
64.2640.31
SALMONN + Chat-UniVi
58.43 20.06
51.62220.00
67.31
SALMONN + mPLUG-OWl
58.70=0.01
51.77
67.76
17

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
G. Prompt for Clue Merge
To merge multimodal clues, we use GPT-3.5 and apply the following prompt:
Please act as an expert in the field of emotions. We provide acoustic and visual clues that may be related the character's emotional state, along with the original subtitle of the video. Please analyze which parts infer the emotional state and explain the reasons. During the analysis, please inlegrate the textual, audio, and visual clues.
Even when modality conflicts exist (i.e., the emotions conveyed by audio, video, and text are not the same, as shown in Figure 6), GPT-3.5 can provide reasonable responses, primarily due to its powerful reasoning ability.
Text:
Why you all looking me like that? So, long woman, does she have relationship with me?
male character an indoor setting. the beginning the video, eyes are wide open his following scenes, looks around, scemingly explaining or narrating something to the people around him. Overall, his emotions are not positive or optimistic.
Visual Clue: the video, screen
mouth also open, indicating a surprised facial expression.
Acoustic Clue: In audio, the character spcaks with a stutter, which usually expresses feelings of nervonsness, anxiery, or unease.
The subtitle expresses the male character's dissatisfaction and anger towards the people around him. Based on the surprised and negative facial expression of the male character in the video clues, as well as the stuttering speech in the audio clues, we can infer that the male character is expressing feeling of dissatisfaction and anger in this sentence. He may feel troubled by the prejudice of the people around him is unhappy with this unfair treatment.
Merged Clue:
Figure 6: Example of modality conflict.
H. Dataset Comparison
Figure 7 compares the distribution of description lengths and the number of emotions per sample. We observe that dataset provides detailed descriptions and rich emotion labels for each sample.
(a) EITVIT
(b) MERR-Fine
(c) HERR-Coarse
(d) MAFW
(c) OV-MERD
(f) MER- BER-Caption
(g) EIT
(h) MERR- Fine
(i) MERR-CAO Coarse
i MAFW
(k) OV-MERD
(I) MER EEERCCntionn
Figure 7: Dataset comparison. The first row compares the lengths the descriptions, while the second row compares the number labels per sample.
I. Video Duration Distribution
Figure presents the video duration distribution of the MER-Caption dataset. We observe that the majority samples have durations ranging from seconds.
18

---

AffectGPT: A New Datasct, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Frequency
0.1
68
60
3.5
5.6
2.5
10.0
12.5
15.0
17.5
Duration
Figure 8: Video duration distribution.
J. BuRn Details
is comprehensive evaluation benchmark covering three typical tasks in MER, including fine-grained emotion recognition, basic emotion recognition, and sentiment analysis. Different tasks involve different datasets, and we provide their statistical information in Table 11. In this paper, we intentionally focus on single-person videos, as this allows us to eliminate interference from other speakers and reduce task difficulty. Multi-person MER belongs to another research topic and will addressed our future work.
Table 11: Dataset statistics in MER- HER-Unniench. All datasets in our study focus on single- perle- videos.
Dataset
Chosen Set
# Samples
Label Description
Data Source
unfixed categories and diverse numher of labels per sample
Fine-grained Emotion
OV-MERD+
All
532
movies, TV series
MER2023
HER-MULTI
411
most likely label among six candidates
movies, TV series movies, TV series actor's performance "Friends" TV series
MER2024
MER-SEMI
1,169
most likely label among six candidates
Basic Emotion
IEMOCAP
SessionS
1,241
most likely label among four candidates
MELD
Test
2.610
most likely label among seven candidates
CMU-MOSI
Test
686
sentiment intensity, ranging from [-3,
opinion videos in YouTube opinion videos YouTube movies, TV series, and shows movies, TV series, and shows
CMU-MOSHI
Test
4,659
sentiment intensity, ranging from [-3.3]
Sentiment Analysis
CH-SIMS
Test
sentiment intensity, ranging from [-1, I]
CH-SIMS
Test
1,034
sentiment intensity. ranging from [-1,
OV-MERD+ is our newly collected dataset, extended version of the previous OV-MERD (Lian et al., 2024a). Unlike traditional datasets, which select a single label from basic emotions, OV-MERD is fine-grained emotion dataset that allows each sample to have a variable number of emotions, using any emotion not restricted to predefined taxonomies. OV-MERD initially contains 332 samples, and we further expand its dataset size, obtaining OV-MERD+.
MER2023 (Lian et al., 2023) and MER2024 (Lian et al., 2024b) are widely used in Chinese MER research, with MER2024 being an extended version of MER2023. The original data in both datasets comes from movies and TV shows. They various techniques to segment video clips, ensuring that each clip has only one person, with their speech content being relatively complete. To ensure annotation quality, they hire multiple annotators, each selecting the most likely label from six candidate emotions: worry, happy, neutral, angry, surprised, and sad. The final label is determined through majority voting,
IEMOCAP (Busso et al., 2008) is one of the most widely used emotion datasets. contains five sessions, each with a male and a female actor in a laboratory environment. The dataset includes the following emotion labels: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and others. Following previous research (Poria al., 2017), we choose the last session for testing, and use the first four emotions, and merge surprise and happiness into happiness.
MELD (Poria et al., 2019) is extension of the text-centered EmotionLines dataset (Hsu et al., 2018), adding audio and video content. The raw data is derived from the Friends TVV series. The dataset has seven emotion labels, and each sample is assigned to one of the most likely labels: jnger.joy, sadness, neutral, disgust, and surprise.
CMU-MOSI (Zadeh et al., 2017) and CMU-MOSEI (Zadeh et al., 2018) consist of opinion videos collected from online platforms. CMU-MOSEI is an extended version of CMU-MOSI, with more samples and wider range of topics. In these
19

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
datasets, each sample is labeled with a sentiment intensity score ranging from -3 to +3, where -3 represents extremely negative emotion and +3 represents extremely positive emotion.
CH-SIMS (Yu et al., 2020) and CH-SIMS v2 (Liu et al., 2022b) differ from the English-centeres CMU-MOSI and CMU-MOSEI by focusing on emotions within the Chinese culture. The original data comes from movies, TV series, and shows. Similar to CMU-MOSI and CMU-MOSEL, these datasets also annotate sentiment intensity, but with different range [-1, I], where 11 represents extremely negative emotion and represents extremely positive emotion.
K. Emotion Wheel
Since there is no universal definition of the emotion wheel, we follow previous work (Lian et al., 2024a) and use five emotion wheels in this paper.
(3) WI
(b) W2
(c) W3
SYOTE AVERAION cull T WIIIIII. CIUNIITY LiBFRATIONN
3
AILIT Maree LSOLATION GRIPP
(d) W4
(e) w5
Figure 9: Emotion wheel. We use five emotion wheels, all of which are derived from previous research (Lian et al., 2024a).
L. Main Results
Table 12 reports the complete results, with several metries for each dataset, and the primary metrics are highlighted in gray. In the last column, we report the average value of the primary metrics. These results verify the effectiveness of our AffectGPT in multimodal emotion understanding.
M. Ablation Study on MER-Caption
Table 13 compares the performance across different datasets. To ensure a fair comparison, we keep the model architecture and experimental setup unchanged, only altering the training dataset. Experimental results in Table 13 demonstrate the effectiveness of our MER-Ca CER-Caption dataset for emotion understanding. It addresses the issue of existing datasets either giving insufficient attention to emotion tasks or lacking high-quality emotion descriptions.
20

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
Table 12: Main results. In this table, "A", "V",, and "T" represent audio, video, and text, indicating the input information used by each MLLM during inference. The gray-highlihtedd columns represent the primary metric for each dataset, while the "Mean" column reports the average score of the primary metrics across all datasets.
MER2023
MER2024
MELD
IEMOCAP
CMU-MOSI
CMU-MOSEI
A
V
T
HIT(()
HIT(t)
HIT(t)
HπT(1)
WAF(1)
ACC())
WAF(T)
ACC(1)
Otter
X
16.41
14.65
22.57
29.08
52.89
54.27
50.44
50.77
OneLLM
V
25.52
17.21
28.32
33.44
64.01
64.48
54.09
54.18
Video-LLaVA
X
36,93
30.25
30.73
38.95
56.37
57.62
61.64
64,20
SECap
40.95
52.46
25.56
36.92
55,76
56.71
54.18
53.85
PandaGPT
33.57
39.04
31.91
36.55
66.06
65.85
61.33
60.73
Qwen-Audio
41.85
31.61
49.09
35.47
70.09
71.49
46,90
51.16
PandaGPT
39.13
47.16
38.33
47.21
58.50
60.21
64.25
65.55
Video-ChatGPP
44.86
46.80
37.33
56.83
54.42
$7.77
63.12
65.66
VideoChat2
33.67
54.50
36.64
66.84
67.23
54.82
PandaGIT
40.21
51.89
37.88
61.92
62.80
67.61
68.82
LLMMA-VID
50.72
57.60
42.75
46.02
61.78
62.65
63.89
66.21
VideoChat
48.73
$7.30
41.11
48.38
65.13
65.09
63.61
63.02
SALMONN
55.53
45.38
45.62
46.84
81.00
81.25
67.03
66.90
Chat-UmiVi
57.62
65.67
45.61
52.37
54,53
57.62
63.18
67.47 .7,4
mPLUG-OWl
56.86
59.89
49.11
55.54
72,40
72.26
72.91
73.17
Fmotion-T.Iane aMA
59.38
73.62
46.76
55.47
66.13
66.31
67.66
67.25
AffectGPT
78.54
78.80
55.65
60.54
81.30
81.25
80.90
80.68
CH-SIMS
CH-SIMS v2
OV-MERD+
Mean
WAF(T)
ACC(t)
WAF(1)
ACC(t)
Fs
Precisiong(11
Recall,(1)
Otter
57.56
60.57
53.12
56.20
16.63
17.67
15.74
34,82
OneLLM
63.39
63.92
61.98
62.46
22.25
24.49
20.41
41.14
Vidco-LLaVA
53.28
54.64
$7.45
59.28
34.00
36.48
31.86
44.40
SECap
59.51
62.89
57.41
60.92
36.97
43.51
32.17
46.64
PandaGPT
62.37
58.88
31.33
33.08
29.77
Qwen-Audio
70.73
73.45
65.26
68.17
32.36
27.91
49,26
PandaGPT
62.07
61.60
65.25
65.31
35.07
37.86
32.67
50.77
Video-ChatGPT
64.82
64.43
65.80
66.85
39,80
43.12
36.97
52.64
VidcoChat2
68.38 
69. 49
69.59
70.66
71.13
39.21
42.85
36.16
52.67
PandaGPT
3
67.78
67. 27
67.40
37.12
39.64
34,91
52.92
LLaMA-VID
69.35
68.8
67.4 48
67.73
45.01
46.83
43.32
56.07
VidcoChat
69.52
69.33
74.14
72.12
44.52
44.55
44.49
56.71
SALMONN
68.69
69.85
65.93
67.07
45.00
43.57
46.61
57.89
Chat-UniVi
68.15
67.78
66.36
67.18
48.00
48.20
47.81
57.94
mPLUG-OWl
71.65
74.97
18.15
47.91
48.47
62.45
Emotion-LLaMA
78.61
77.23
77.39
52.97
54.85
51.22
64.17
AffectGPT
88.49
88.40
86.18
86.17
62.52
62.21
63.00
74.77
Table 13: Dataset comparison.
CMU-
CMU- MOSEI
CH-
CH-
OV. MERD+
Dataset
Filtering
MER2023
MER2024
MELD IEMOCAP
Mean
MOST
SIMS
SIMS v2
11.56
12.91
18.89
16.06
53.57
45.98
57.66
55.16
13.86
31.74
MiniGPT4
17.57
16.65
22.60
30.18
52.58
56.50
$2.36
$1.19
20.16
35.53
24.87
22.42
21.56
32.91
50.13
56.17
50.07
51.71
24.56
37.16
VidcoChat
General Instruction
/
27.66
39.46
45.45
56.86
43.68
47.05
26.09
37.63
LLaVA
x
42.21
41.54
32.97
49,96
54.48
56.42
$2.04
54.80
35.75
46.69
V
41.56
42.30
32.61
46.21
52.82
57.72
52.78
$3.44
36.96
46.27
5.75
7.71
4.35
4.99
45.59
22.76
53.04
45.68
4.95
21.65
WavCaps
V
23.72
26.97
23.39
27.30
54.67
49,54
58.12
55.93
21.57
37.91
EmoVIT
39.31
50.24
32.36
48.24
$3.40
61.53
69.72
66.53
38.09
$1.05
MAFW
52.67
55.99
40.85
57.60
66.11
62.27
75.20
70.02
42.75
58.16
Emotion Description
MERR-Coarse
35.34
36.60
29.37
36.94
65.10
63.27
75.12
73.76
33.18
49.85
MERR-Fine
69.00
72.84
47.38
$4.49
66.21
60.03
79.90
78.54
52.56
64.55
MER-Caption
72.12
74.21
$4.69
56.41
75.10
70.97
80.21
78.06
58.42
68.91
MER-Caption+
78.54
78.80
55.65
60.54
81.30
80.90
88.49
86.18
62.52
74.77
21

---

AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
N. Impact of Sampling Frames in Video Branch
This paper defaults to sampling 8 frames per video. But if we change the number sampled frames, will significantly impact performance? To answer this, we conducted additional experiments in this section. Specifically, we compare two types of inputs: (1) face-only and (2) face-text combinations, and evaluate model performance across different sampling frame counts, ranging from 2 to 64. In Figure 10, observe that using too few frames (e.g., fewer than 2) results in noticeable decline in performance, indicating that insufficient frames lead to information loss. However, further increasing the number of sampling frames (e.g., more than 16) does not yield significant performance improvements. This can be attributed to the fact that MER tasks typically use short-duration videos with relatively stable facial expressions.
61,26
61,49
74.31
73,54
74,31
33,51
74,2074.23
72.94
MER-UniBench
61.03
MER-UniBench
72.94
60
76
60.56
60,47
71.56
60.10
70.19
59.63
68.81
59
67 44
59.17
67.44
58.71
66.07
16
32
64
# of frames
of ffarrmss
(a) Face-only Input
(b) Face-text Input
Figure 10: Impact of sampling frames.
22


Reviews:
[{"review_id": "hlL4sR6bf6", "reviewer": "Reviewer_9mmC", "summary": "This paper introduces a new dataset, pre-fusion model, and evaluation benchmark to advance multimodal, natural language-based emotion understanding. It proposes a model-led, human-assisted strategy to minimize human effort while constructing the largest multimodal emotion dataset to date. The model features a pre-fusion architecture that integrates audio and video signals before combining them with text, enhancing multimodal fusion. Additionally, the paper presents a comprehensive benchmark covering fine-grained emotion recognition, basic emotion classification, and sentiment analysis, with metrics designed for free-form text outputs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our work's significance and your acknowledgment of our contributions—including the novel dataset, model architecture, and comprehensive benchmark for descriptive emotion understanding. These innovations enable richer, more flexible emotion representation that extends beyond conventional predefined-label paradigms.\n\n**Q1:** Lack of Baseline Comparisons on MER-Caption+: To better isolate the impact of the new dataset vs. the new model design, it would be helpful to compare baseline models trained on MER-Caption+ against AffectGPT trained on MER-Caption+. This would clarify whether the performance boost comes primarily from the dataset or the model architecture.\n\n**A1:** **In our experimental design, we systematically validate the effectiveness of both the new dataset and model using the control-of-variables method.** Specifically, in Table 3, we demonstrate the impact of our MER-Caption+ dataset by maintaining identical model architecture while varying only the training data. In Table 4, we verify the necessity of AffectGPT by keeping the training data constant while modifying only the model structure. **These carefully designed ablation studies can already verify the effectiveness of the new dataset and the new model.**\n\n\n**Q2:** Dataset Coverage and Diversity: The paper does not specify the scenarios included in the dataset (e.g., daily conversations, news reports, or movie reviews). Additionally, it is unclear whether the dataset primarily focuses on first-person speaking or also includes multi-person videos, which could affect its generalizability.\n\n**A2:** Thanks for your suggestion, and we will incorporate the following additional information in our revised manuscript. In this paper, we intentionally focus on single-person videos, as this allows us to eliminate interference from other speakers and reduce task difficulty. Multi-person MER belongs to another research topic and will be discussed in our future work.\n\n| Category              | Dataset       | Chosen Set   | # Samples | Label Description                          | Data Source                     |\n|-----------------------|--------------|-------------|----------|--------------------------------------------|---------------------------------|\n| ​**Fine-grained Emotion** | OV-MERD+     | All         | 532      | Unfixed categories and diverse number of labels per sample | movies, TV series               |\n| ​**Basic Emotion**     | MER2023      | MER-MULTI   | 411      | Most likely label among six candidates     | movies, TV series               |\n|                       | MER2024      | MER-SEMI    | 1,169    | Most likely label among six candidates     | movies, TV series               |\n|                       | IEMOCAP      | Session5    | 1,241    | Most likely label among four candidates    | actor's performance             |\n|                       | MELD         | Test        | 2,610    | Most likely label among seven candidates   | \"Friends\" TV series             |\n| ​**Sentiment Analysis** | CMU-MOSI     | Test        | 686      | Sentiment intensity, ranging from [-3, 3] | opinion videos in YouTube       |\n|                       | CMU-MOSEI    | Test        | 4,659    | Sentiment intensity, ranging from [-3, 3] | opinion videos in YouTube       |\n|                       | CH-SIMS      | Test        | 457      | Sentiment intensity, ranging from [-1, 1] | movies, TV series, and shows    |\n|                       | CH-SIMS v2   | Test        | 1,034    | Sentiment intensity, ranging from [-1, 1] | movies, TV series, and shows    |\n\n**Q3:** Impact of Frame Signals on Performance: The results suggest that using face signals alone outperforms frame signals, raising the question of whether frame signals introduce more noise. It would be valuable to explore whether this issue can be mitigated through data cleaning or improved visual signal alignment with other modalities.\n\n**A3:** Thanks for your comment. We agree that using frame signals may introduce additional noise, resulting in performance degradation. **Your suggestions for further verification are insightful, and we plan to explore these aspects through additional experiments in future work.**", "claims_and_evidence": "N/A", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. Lack of Baseline Comparisons on MER-Caption+: To better isolate the impact of the new dataset vs. the new model design, it would be helpful to compare baseline models trained on MER-Caption+ against AffectGPT trained on MER-Caption+. This would clarify whether the performance boost comes primarily from the dataset or the model architecture.\n\n2. Dataset Coverage and Diversity: The paper does not specify the scenarios included in the dataset (e.g., daily conversations, news reports, or movie reviews). Additionally, it is unclear whether the dataset primarily focuses on first-person speaking or also includes multi-person videos, which could affect its generalizability.\n\n3. Impact of Frame Signals on Performance: The results suggest that using face signals alone outperforms frame signals, raising the question of whether frame signals introduce more noise. It would be valuable to explore whether this issue can be mitigated through data cleaning or improved visual signal alignment with other modalities.", "relation_to_broader_scientific_literature": "This paper introduces a comprehensive dataset with detailed natural language annotations, enabling a richer representation of emotions beyond predefined labels. It adopts a model-led, human-assisted annotation strategy, effectively balancing scale and label quality.\n\nAdditionally, the paper presents a rigorous evaluation pipeline, featuring in-depth ablation studies on the dataset, model, and LLMs. Notably, it also includes a user study to assess and compare dataset annotations, further validating its approach.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JzqnYAnRu9", "reviewer": "Reviewer_imjz", "summary": "This paper presents a new video content description dataset with emotional words and highlights a novel annotation method for the dataset. Additionally, it proposes a model that enhances multimodal emotion recognition. The primary innovation of this model lies in its pre-fusion strategy for multimodal inputs. Finally, the paper introduces a new evaluation benchmark for assessing the performance of emotion recognition models.\n\n## update after rebuttal\nI would like to keep my rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of our fair comparative experiments, innovative dataset, effective key components, and comprehensive evaluation benchmark.\n\n**Q1:** In Appendix F, experiments have shown that using a combination of SALMONN and mPLUG-Owl results in better performance. However, the authors still chose the combination of SLAMONN and Chat-UniVi. An explanation for this choice is needed.\n\n**A1:** Thanks for your careful review. In this paper, we do not use the combined results for model selection but instead rely on the performance of individual models. For example, for VLLM, Chat-UniVi outperforms mPLUG-Owl, and for ALLM, SALMONN outperforms SECap. Therefore, we use the combination of Chat-UniVi and SALMONN for description generation. The combination experiments are primarily designed to demonstrate that integrating multimodal cues can lead to better performance. Your suggestion of using the combined results for model selection is insightful, and we will conduct more experiments in this direction.\n\n**Q2:** The origin of the raw data used in the dataset construction is not provided by the authors.\n\n**A2:** **The raw data comes from the unlabeled portions of MER2024, which is used with permission from the dataset owners.** In this work, we annotate each unlabeled sample with emotion descriptions. The complete annotation procedure is illustrated in Figure 2.\n\n**Q3:** A large amount of data was filtered out during the dataset construction process, which limits the application scenarios of the dataset.\n\n**A3:** We would like to highlight that **dataset quality is equally important as quantity**. It is verified by the experimental results in Table 4: Given the same testing set (with diverse sources/scenarios), the model trained on the filtered datasets performs better than the one trained on the noisy dataset.\n\n**Q4:** The evaluation metric used in the benchmark for basic emotion recognition tasks may have the risk of being inaccurate. The current evaluation metric only calculates the number of correctly matched emotion words in the output of the multimodal emotion recognition model. However, if the model outputs entirely incorrect results, the impact of these errors is not accounted for.\n\n**A4:** Thanks for your comment. **In fact, we have taken this into account during the metric design process.** Basic emotion recognition tasks provide the majority-voted labels $y$, which are generally reliable. However, emotion descriptions produce free-form outputs $\\mathbf{\\hat{Y}}$ that may contain multiple labels, including fine-grained ones beyond basic emotions. **Therefore, we use the HIT rate as the metric, ensuring that the basic label $y$ should be at least in $\\mathbf{\\hat{Y}}$.**\n\n**Meanwhile, we attempt to design metrics for evaluating potentially incorrect labels in $\\mathbf{\\hat{Y}}$. However, the labels in $\\mathbf{\\hat{Y}}$ that differ from the basic label $y$ are not necessarily incorrect - they may represent some fine-grained emotions not covered by basic categories. But since basic emotion recognition tasks lack fine-grained reference labels, we have not yet established appropriate evaluation metrics for this purpose. This remains an important research direction for our future work.**\n\n\n**Q5:** In Table 3, the performance of the LLaVA dataset decreased after applying the filtering strategy. The authors did not explain what the filtering strategy was. If it is the strategy mentioned in the paper, it suggests that this filtering strategy does not work well for the LLaVA dataset. The authors should analyze the reason.\n\n**A5:** Thanks for your comment. Regarding general instruction datasets, we use a filtering process to retain only emotion-related instruction-answer pairs. Specifically, **we use the prompt in Appendix E and extract emotion labels from each instruction-answer pair. Samples yielding empty emotion outputs are removed.** As shown in Table 3, this filtering approach proves less effective for the LLaVA dataset. We hypothesize that the detailed descriptions in non-emotion subsets may also provide valuable cues for inferring emotional states in some scenarios.\n\n\n**Q6:** Although the paper proposes a new multimodal emotion recognition model with significant performance improvement, the model's innovation compared to previous work is limited, as it mainly adds a pre-fusion mechanism between multiple modalities.\n\n**A6:** **The model architecture novelty is only a part of our work, and our contributions extend significantly beyond this.** Besides the model architecture, we also use a model-led, human-assisted strategy to minimize human effort while constructing the largest multimodal emotion dataset to date. Also, we present a comprehensive benchmark covering fine-grained emotion recognition, basic emotion classification, and sentiment analysis, with metrics designed for free-form text outputs.", "claims_and_evidence": "The claims proposed in this paper are mostly supported by experimental evidence or have been mentioned in previous studies, and the reasoning is relatively sound.", "methods_and_evaluation_criteria": "The method proposed in this paper effectively helps address the challenges in current emotion recognition tasks. The dataset construction method reduces the difficulty of dataset creation. The model, which adopts a new strategy, also demonstrates improved performance. Finally, the benchmark introduced in the paper provides an effective means to evaluate the performance of multimodal emotion recognition models.", "theoretical_claims": "I have indeed verified the correctness of the theories proposed in the paper. The paper includes numerous ablation experiments on the proposed methods. The results of these experiments validate the correctness of the proposed approaches. For example, Section 5 of the paper focuses on presenting various experimental results.", "experimental_designs_or_analyses": "The paper clearly describes the design details of each experiment. When presenting comparative experiments, it frequently emphasizes \"fairness.\" I also believe that the authors have made every effort to conduct fair comparative experiments.", "relation_to_broader_scientific_literature": "The paper builds upon previous multimodal emotion recognition research by summarizing and enhancing existing work. The authors propose new contributions, which are validated through experiments. The dataset introduced in the paper is also an innovation based on the experiences of previous studies. Finally, the benchmark presented serves as a comprehensive evaluation tool for assessing the capabilities of multimodal emotion recognition models.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The main advantages of this paper are as follows:\n\n1. The dataset proposed in the paper is large in scale and of high quality. The relevant experiments are thorough, and a user study has been conducted. While ensuring that the dataset can be automatically annotated, the quality of the data is also maintained. The proposed annotation and filtering methods have been experimentally validated for effectiveness.\n2. The model proposed for multimodal emotion recognition performs well. The effectiveness of the key component, the multimodal pre-fusion strategy, has also been validated through experiments.\n3. The paper introduces a new benchmark for evaluating the performance of multimodal emotion recognition models. This benchmark can comprehensively assess the model's ability to handle various multimodal emotion recognition tasks.\n\nAlthough the paper has many advantages, there are still some details that need attention:\n\n1. In Appendix F, experiments have shown that using a combination of SALMONN and mPLUG-Owl results in better performance. However, the authors still chose the combination of SLAMONN and Chat-UniVi. An explanation for this choice is needed.\n2. The origin of the raw data used in the dataset construction is not provided by the authors.\n3. A large amount of data was filtered out during the dataset construction process, which limits the application scenarios of the dataset.\n4. The evaluation metric used in the benchmark for basic emotion recognition tasks may have the risk of being inaccurate. The current evaluation metric only calculates the number of correctly matched emotion words in the output of the multimodal emotion recognition model. However, if the model outputs entirely incorrect results, the impact of these errors is not accounted for.\n5. In Table 3, the performance of the LLaVA dataset decreased after applying the filtering strategy. The authors did not explain what the filtering strategy was. If it is the strategy mentioned in the paper, it suggests that this filtering strategy does not work well for the LLaVA dataset. The authors should analyze the reason.\n6. Although the paper proposes a new multimodal emotion recognition model with significant performance improvement, the model's innovation compared to previous work is limited, as it mainly adds a pre-fusion mechanism between multiple modalities.", "other_comments_or_suggestions": "The paper is overall well-written, and I hope the authors can provide explanations for the various detailed issues mentioned above.", "questions_for_authors": "The specific issues can be found in the weakness mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2twKkSejqu", "reviewer": "Reviewer_LFYp", "summary": "This paper introduces a new dataset for the multimodal emotion recognition (MER) task. The dataset is constructed using a model-driven, human-assisted approach. Initially, a coarse-grained dataset is generated through data description, followed by fine-grained data refinement through both low-level and high-level filtering. Additionally, a novel MLLMs-based method, \"AffectGPT,\" is proposed, which enhances model performance by integrating audio and video feature vectors (pre-fusion) before feeding them into the LLM. This paper also presents a new MER evaluation standard to assess model performance in MER tasks. Several experiments validate the effectiveness of the dataset, the proposed method, and the evaluation standard.\n\n## update after rebuttal: I will keep my score as it is, even after the rebuttal, which addresses all of my comments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** Limited discussion on computational costs of pre-fusion operations. The pre-fusion mechanism lacks theoretical analysis of modality interaction dynamics. The structural innovation of AffectGPT is relatively insufficient, and it does not fully explain why the \"pre-fusion\" operation can serve as a solution for cross-modal interaction.\n\n**A1:** **Regarding computational efficiency, the pre-fusion operation only involves Q-Former or attention operations, which are significantly less computationally intensive than LLMs. Theoretically, the Q-Former facilitates cross-modal interaction by distilling knowledge from multimodal content into query tokens, while the attention mechanism achieves this through dynamically predicted attention weights based on multimodal inputs.**\n\n**The model architecture innovation represents one aspect of our work, and our contributions extend significantly beyond this.** Specifically, we introduce the largest-scale emotion description dataset to date, constructed using an efficient model-led, human-assisted approach; we establish a comprehensive benchmark encompassing three key tasks: fine-grained emotion recognition, basic emotion classification, and sentiment analysis; and we develop specialized evaluation metrics designed for assessing free-form text outputs.\n\n\n**Q2:** Regarding the ablation study of the model, since the video and audio features have already been fused, why are these features still fed separately into the LLMs? Have experiments been conducted using only the fused features?\n\n**A2:** Thanks for your valuable comment. Following your suggestion, we conducted additional experiments to investigate this aspect. **Our results demonstrate that incorporating raw audio and video features alongside the fused features yields modest performance improvements compared to using fused features alone.**\n\n| Input Type                              | MER-UniBench Score |\n|-----------------------------------------|-------------------:|\n| Audio, video features                   |              72.95 |\n| Audio-video fused features              |              74.14 |\n| ​Audio, video, and audio-video fused features | ​74.77|\n\n**Q3:** What is the purpose of designing the evaluation metric $F_s$, or in other words, what specific aspect does this metric reflect?\n\n**A3:** In this paper, $Precision_s$ indicates the number of correctly predicted labels, and $Recall_s$ indicates whether the prediction covers all ground truth. **$F_s$ is a harmonic mean of two metrics $Precision_s$ and $Recall_s$.**\n\n\n**Q4:** Is it possible that MER-Caption+ also contains some instances where the data does not match the descriptions?\n\n**A4:** Thanks for your valuable comment. **Yes, there may be inaccurate descriptions in MER-Caption+ because we used an automatic annotation strategy without manual checks.** However, the experimental results in Table 3 demonstrate that MER-Caption+ achieves significantly better performance than the manually annotated MAFW dataset. The main reason is that humans tend to focus on major clues, which can easily lead to incomplete descriptions. **These results confirm that, despite the lack of manual checks in MER-Caption+, we can still ensure the quality of the labels. In the future, we will investigate other post-filtering techniques to further improve MER-Caption+'s annotation quality.**", "claims_and_evidence": "The claims have been supported by experimental analysis. For example: \"We propose a pre-fusion operation that moves the cross-modal interaction outside the LLMs, further enhancing multimodal integration.\"\nAdditionally, I agree that the current community still lacks large-scale datasets with detailed, descriptive emotion annotations necessary to fully realize the potential of MLLMs.", "methods_and_evaluation_criteria": "- Mainstream AV-LLMs leave everything of cross-modal interaction to the LLMs, which is insufficient for handling MER with multimodal characteristics. >> confirmed by experiments.\n\nStrengths\n- Novel two-level filtering (low-level modality alignment + high-level model crowdsourcing)\n- Multi-level emotion grouping metrics (Level 1-3) handle free-form outputs effectively\n\nWeaknesses\n- Limited discussion on computational costs of pre-fusion operations", "theoretical_claims": "The pre-fusion mechanism lacks theoretical analysis of modality interaction dynamics.", "experimental_designs_or_analyses": "Yes, it is valid.", "relation_to_broader_scientific_literature": "The affective computing community is vast, and this dataset is currently the largest available, with a well-justified data collection procedure. The proposed method also demonstrates strong performance on existing datasets, and the suggested metrics appear reasonable. Therefore, I believe this work will be a valuable contribution to the community.\n\nMoreover, the data collection procedure could serve as a guideline for other human studies aiming to collect large-scale datasets. This makes it potentially useful not only for the affective computing field but also for other research areas.", "essential_references_not_discussed": "There is extensive research on these topics, but in assessing this paper, I did not find any major references missing.", "other_strengths_and_weaknesses": "Strengths:\n1. This paper introduces a new MER dataset along with a method for collecting and constructing the dataset. Compared to other related datasets, it offers higher quality, a larger volume of data, and more diverse classifications.\n2. This paper proposes a new MER evaluation standard, \"MER-UniBench,\" which provides a comprehensive evaluation framework for MLLMs-based MER emotion understanding tasks.\n3. Extensive experiments have been conducted to validate the dataset and evaluation standard.\n\nWeaknesses:\n1. The structural innovation of AffectGPT is relatively insufficient, and it does not fully explain why the \"Pre-fusion\" operation can serve as a solution for cross-modal interaction.", "other_comments_or_suggestions": "Originality\n- Novel model-led annotation paradigm\n- Pre-fusion mechanism builds on established cross-modal attention\n\nSignificance\n- Large-scale MLLM-focused emotion dataset\n- Practical metrics for free-form emotion analysis", "questions_for_authors": "1. Regarding the ablation study of the model, since the video and audio features have already been fused, why are these features still fed separately into the LLMs? Have experiments been conducted using only the fused features?\n\n2. What is the purpose of designing the evaluation metric FskF^k_s, or in other words, what specific aspect does this metric reflect?\n\n3. Is it possible that MER-Caption+ also contains some instances where the data does not match the descriptions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "slRSTs4fE0", "reviewer": "Reviewer_vBqq", "summary": "This paper focuses on descriptive emotion understanding. Compared with discriminative emotion recognition, descriptive emotion understanding provides the possibility of modeling complex emotions. To promote the development of this field, they proposed new datasets (MER-Caption and MER-Caption+) and new models (AffectGPT). They also established a benchmark (MER-UniBench) for MLLM-based descriptive emotion understanding solutions. Experimental results verify the effectiveness of their proposed framework and dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** It would be beneficial to also explore and discuss the impact of different audio and video encoders.\n\n**A1:** **(1) Impact of Audio encoders.** The choice of audio encoder does not significantly impact performance. This confirms that AffectGPT's remarkable performance is primarily attributed to our proposed high-quality, large-scale dataset and effective framework rather than acoustic encoders. Meanwhile, we note that ImageBind achieves slightly worse performance compared to other audio encoders. One possible explanation for this is that other audio encoders are widely applied in audio content understanding tasks (such as ASR), where audio content plays a crucial role in emotion recognition.\n\n|Audio Encoder|MER-UniBench|\n|-|-|\n|ImageBind|73.81|\n|Data2vec_Base|74.48|\n|WavLM_Large|74.39|\n|​HUBERT_Large|​74.77|\n\n**(2) Impact of Video Encoder.** The choice of video encoder has a limited impact on performance. Interestingly, CLIP\\_VIT slightly outperforms EVA\\_CLIP and DINOv2, consistent with the findings in MERBench [1], a unified benchmark for traditional categorical frameworks. These results suggest that conclusions drawn from traditional categorical frameworks, such as encoder selection, may also be applicable to MLLM-based descriptive frameworks.\n\n|Video Encoder|MER-UniBench|\n|-|-|\n|EVA_CLIP|73.42|\n|DINOv2|73.12|\n|SigLIP|74.63|\n|​CLIP_VIT|​74.77|\n\n[1] Lian, Zheng, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. \"Merbench: A unified evaluation benchmark for multimodal emotion recognition.\" arXiv preprint arXiv:2401.03429 (2024).\n\n\n**Q2:** Please further discuss the influence of rank selection in LoRA.\n\n**A2:** In the following table, we count the increase in trainable parameters when using LoRA for the LLM branch. The first row represents the model without the LoRA module. Experimental results demonstrate that fine-tuning the LLM with LoRA improves performance compared to models without LoRA. However, increasing the rank for LoRA-based models does not yield significant performance gains and instead increases computational costs.\n\n|Rank|Increased Parameters|MER-UniBench|\n|-|-|-|\n|-|0|73.30|\n|8|20,185,088|74.65|\n|16|40,370,176|74.77|\n|32|80,740,352|​74.92|\n\n\n**Q3:** How are videos sampled for the video branch?\n\n**A3:** We uniformly sample 8 frames per video by default.\n\n**Q4:** The impact of varying the number of sampled frames on the model's performance.\n\n**A4:** In the following table, we compare two types of inputs: (1) face-only and (2) face-text combinations, and evaluate model performance across different sampling frame counts, ranging from 2 to 64. We observe that using too few frames (e.g., fewer than 2) results in a noticeable decline in performance, indicating that insufficient frames lead to information loss. However, further increasing the number of sampling frames (e.g., more than 8) does not yield significant performance improvements. This can be attributed to the fact that MER tasks typically use short-duration videos with relatively stable facial expressions. Therefore, we default to sampling 8 frames per video in this paper.\n\n|Input Modality|0|2|4|8|16|32|64|\n|-|-|-|-|-|-|-|-|\n|Face-only|-|59.17|60.76|60.47|61.26|61.10|61.49|\n|Face-text|67.44|72.94|73.54|74.31|73.51|74.10|​74.23|\n\n**Q5:** The dataset construction pipeline could benefit from further explanation.\n\n**A5:** We adopt a **model-led, human-assisted** annotation strategy. This approach leverages human priors to guide description generation and sample filtering, ultimately enabling automatic annotation for unlabeled data.\n\n**(1) Description Generation.** During the description generation process, we first conduct preliminary experiments. In this phase, a small subset of samples is selected, and annotators are asked to assign fine-grained emotional labels to each sample. Based on the insights gained from these preliminary experiments, we perform a model selection process to ensure the quality of the automatically generated descriptions.\n\n**(2) Sample Filtering.** During sample filtering, we employ a two-stage filtering technique. In the first stage, we remove samples with mismatched audio and video, as well as those with abnormal description lengths. In the second stage, we use *model-based crowdsourcing* to generate relatively reliable emotion labels. If the labels derived from the descriptions differ significantly from those obtained through *model-based crowdsourcing*, the descriptions are considered to be of low quality and are then removed.\n\nIn summary, we integrate human priors into both description generation and sample filtering to ensure the high quality of the generated descriptions in MER-Caption+.\n\n\n**Q6:** The details of the filtering technique.\n\n**A6:** For general instruction datasets, we utilize the prompt provided in Appendix E to extract emotion labels from each instruction-answer pair. Samples that result in empty emotion outputs are removed.", "claims_and_evidence": "Yes. This paper contains extensive experiments to support its conclusions.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This is not a theoretical paper.", "experimental_designs_or_analyses": "Yes. Their experimental design and analysis are clear and reasonable. This paper first reports the performance of different MLLMs on MER-UniBench, demonstrating the effectiveness of their proposed AffectGPT framework. Then, they compare different instruction datasets under the same framework and training settings, demonstrating the effectiveness of their proposed dataset (MER-Caption+). At the same time, the ablation study of dataset filtering techniques, the necessity of pre-fusion operations, the discussion on the impact of input modality, and the user study of dataset quality assessment also demonstrate the effectiveness of each proposed module in this paper.", "relation_to_broader_scientific_literature": "The key contribution of this paper is advancing the field of descriptive emotion understanding through the introduction of new datasets, architectures, and benchmarks.", "essential_references_not_discussed": "No, essential related work has been correctly cited.", "other_strengths_and_weaknesses": "Despite the extensive experiments conducted in this paper, there are several additional experiments and discussions that could further enhance the paper's comprehensiveness:\n1. While the paper discusses the choice of LLMs (as shown in Figure 4), it would be beneficial to also explore and discuss the impact of different audio and video encoders.\n2. Please further discuss the influence of rank selection in LoRA.\n3. It appears that this paper does not discuss how videos are sampled for the video branch.\n4. Additionally, the paper should discuss the impact of varying the number of sampled frames on the model's performance.\n5. In Figure 2, the dataset construction pipeline could benefit from further explanation.\n6. In Table 3, please explain the details of the filtering technique you used for the general instruction dataset.", "other_comments_or_suggestions": "Please refer to my comments on the weakness part.", "questions_for_authors": "Please refer to my comments on the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "Not needed."}}]

Author rebuttal:

Meta-review: The paper received three Accepts and one Weak Accept. The reviewers note a rigorous evaluation pipeline, clear descriptions, and strong contribution to the field.  This is a high-quality contribution to ICML.
Decision: Accept (oral)