Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Authors: Zebang Cheng
Zhi-Qi Cheng
Jun-Yan He
Kai Wang
Yuxiang Lin
Zheng Lian
Xiaojiang Peng
Alexander G Hauptmann
Venue: NeurIPS 2024 poster (2024)
Keywords: Instruction Tuning
Multi-modal Emotion Recognition
Multi-modal Emotion Reasoning
Primary area: human-AI_interaction

Abstract: Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling.
However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset.

Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Zebang Chengl 12+ Zhi-Qi Cheng Jun-Yan He Jingdong Sun
Kai Wang Yuxiang Lin' Zheng Lian Xiaojiang Pengl21 Alexander G. Hauptmann"
Shenzhen Technology University Shenzhen University Carnegie Mellon University 'Alibaba Group National University of Singapore
Institute of Automation, Chinese Academy of Sciences
Project: https:: stathub..o//motiion-LLaaa
Demo: http://gg.......oo./oc//////aaaa LLaMA
Abstract
Accurate emotion perception is crucial for various applications, including human- computer interaction, education, and counseling. However, traditional single- modality approaches often fail capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recog- nizing subtle facial nicro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated sam- ples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-worke applications. Furthermore, we propose Mmnotion-LLaMa a model that seamlessly integrates audio, visual, and textual inputs through emotion emotion-specific encoders. By aligning features into a shared space and employing a modified I.LMMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reason- ing capabilities. Extensive evaluations show Emotion-LLaMMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an Fl score of 0.9036 on MER2023-SEMP challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset.
Introduction
Emotion perception plays vital role in applications such human-computer interaction [17-19, 80), educational assistance [43], and psychological counseling [7, 42]. While single-modality approaches, including facial expression recognition [45, 57, 70, 87], text emotion analysis [25, 49, 41], and audio emotion recognition [31, 39, 47], have shown effectiveness, real-world emotional data is often multimodal, integrating text, audio, and images.
Despite extensive multimodal fusion methods having achieved promising improvements [13, 14, 16, 56, 58. 59, 77, 91, 96, 100, 103], they mainly focus feature interaction and modality completion, remaining under-explored for knowledge-leve interaction which is essential for emotional reasoning of humans. Recently, Multimodal Large Language Models (MLLMs) have excelled in tasks such as visual-language understanding [50, 65], visual question answering [104], and video understanding [35, 63, 86]. However, for emotion recognition [45], models like GPT-4 with Vision (GPT-4V) still face two main challenges: the inability to process audio and the failure to recognize micro-expressions.
Equual contribution, listed in random order, completed during Zebang Cheng's internship CMU. Conresponding author (zhiqic@ es cmu.edu, pengxiaojiang @ sztu.edu.cn)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

We argue that the lack of specialized multimodal emotion instruction datasets is the main factor limiting MLLMs' effectiveness. These issues stem from the inability of previous methods to ef- fectively integrate audio inputs, which are crucial for capturing vocal tones and auditory cues, and their difficulty in recognizing subtle facial micco-expressioos. These limitations lead to sub-optimal performance in real-world scenarios.
To address these challenges, we introduce the MERR dataser (Sec. 3.1), which enables multimodal large models and supports instruction tuming to learn from diverse scenarios and generalize to real-world applications. We also propose the Emotion- audio, visual, and textual inputs through emotion-specifee encoders. By employing instruction tuning (Sec. 3.3). Emotion-LLaMA significantly enhances both the accuracy of emotional recognition and the depth of emotional reasoning, setting a new benchmark for multimodal emotion analysis. Extensive experiments and evaluations (Sec. 4) demonstrate Emotion-I.J LaM'ss superiority, achieving top scores on EMER, MER2023', MER2024", and DFEW datasets. Our main contributions are as follows:
model (Sec. 3.2), which integrates
We constructed the MERR dataset, which includes 28,618 coarse-grained and 4,487 fine- grained annotated samples, covering wide range of emotional categories such "doub" and "contempt". Unlike previous datasets, MERR's ' diverse emotional contexts allow models to learn from varied scenarios and generalize to real-world applications, serving as valuable resource for advancing large-scale multimodal emotion model training and evaluation.
We developed the Pmnotion-LaMMs model, which incorporates HuBERT for audio process- ing and multiview visual encoders (MAE, VideoMAE, EVA) for capturing facial details, dynamics, and context. By aligning these features into a modified LLaMA language model, Emotion-LLaMA enhances emotional recognition and reasoning capabilities.
Extensive experiments demonstrate that Emotion-LLaMa significantly outperforms other MLLMs across multiple datasets, establishing it as the current state-of-the-art model in public competitions. It achieved top scores on the EMER dataset (Clue Overlap: 7.83, Label Overlap: 6.25) and attained F1 scores of 0.9036 on MER2023-SEMI! and 0.8452 on MERR204-NOISEE? Fmotion-I. also surpassed ChatGPT-4V in zero-shot evaluations, including DFEW (+4.37%) and MFR2024-OV (+8.52%).
2 Related Work
To highlight our contributions, we review existing multimodal large language models and instruction tuning methods, emphasizing their limitations in emotional understanding.
Multimodal Large Language Models (MLLMs). MLLMs [1, 5, 11, 20, 73, 88] have gained substantial attention due to their powerful inferential capabilities. Research primarily focuses on leveraging pretrained models like CLIP [75], Q-Foreer [53], and ImageBind [34] for general
domain applications [10, 97, 104]. However, even advanced models like GPT-4V [61] struggle with understanding audio emotional cues and recognizing facial micro-expressions due to the lack of specialized training on multimodal emotional datasets and emotion knowledge. Recently, researchers have begun training MILMs on multimodal emotional datasets to identify emotion- triggering utterances in dialogues [2, 14], though these studies often lack detailed explanations. In contrast, our proposed Moottion-LLMas employs emotion-specii encoders extract multimodal features, enhancing emotional recognition and reasoning capabilities.
Instruction Tuning. Language instructions have been widely used across diverse NLP tasks [6, 21, 76, 101, 95]. Studies like InstructionGPT [72], FLAN [23], and OPT-IML [44] have explored instruction- tonning methods [89, 90] that significantly enhance the zero-shot and few-shot capabilities of LLMs. The vision field has also embraced language instructions for various vision-language tasks [1, 27, 40, 46, 99]. LLaVA [65] converted image-text pairs into cestruction-folloviii using a language-only model, while EmoVIT [92] generated visual emotion instruction data using paired annotations. However, these approaches often lack audio information, which is crucial for understanding human emotions. Due to high annotation costs, AffectGPT [62] manually annotated only 100 samples with emotion clues. To address the scarcity of emotionn-elatee instructtin:follloiing data, our approach generates multimodal descriptions using prior knowledge.
data
Inttp://merrchallenge wtwp:/eeennnncccce///2222 cn/mer2023
hattpe:://eeroqtaabbaa wit.bb....ttt.........g. 1ttp::u20..eeM022--000000- websitee

---

3
Methodology
This section presents our proposed Emotion-LLaMA model, which consists of three key components: the MERR dataset construction (Sec. 3.1), the Multimodal Emotion-LLaMA model architecture (Sec. 3.2), and the training procedures (Sec. 3.3).
3.1 MERR Dataset Construction
The MERR dataset is constructed through comprehensive process of emotion annotation in video data, outlined in Algorithm 1 and Figure I. First, human faces are extracted from each video frame using the OpenFace toolkit, which detects and scores Action Units (AUs) [28, 69] to identify the frame with the maximum cumulative intensity:
Ieaa arg max
Σ S
(1)
where Sau, represents the intensity of each AU. These AUs are mapped to facial expression descrip- tions Cred (Tables 10 and 11) to accurately depict facial movements. Next, MiniGPT-v2' analyzes the peak frame to extract contextual information Good. such activities and environment (Figure 1), fa- cilitating the identification of latent emotional elements within the background context. Qwen-Audio processes audio segments to extract nuances in speech and vocal tone, generating emotion-relatee descriptions Catd- Visual and audio information are concatenated into a raw multimodal description, integrating sensory inputs enhance the contextual supplementation for lexical subtitles. Lexical sub- titles Ct. are integrated into the multimodal description, providing textual context that complements the audio and visual data. LLaMA-33 refines these annotations by aggregating unimodal descriptions (Cueds Cvod Card. C1.) into a detailed multimodal description Cma. following instructions and examples in Table 12. Finally, the comprehensive description Cad is used to annotate the peak frame, ensuring the video is annotated with detailed emotional descriptors.
Visual Expression Description (Cor)):
Description (Car):
woman video speaks with an excited voice."
AU-05: 0.36
AU-26: 1.05
Lexical Subtitle (Cs): "Oh my!"
"Eyes widened, Wide mouthed."
Multi- modal Description (Coal:
Visual Objective Description (Cour)
"In the video, woman is conversing with Her facial expressions, with eyes widened and mouth wide open, clearly show surprise. She amplifies this visual cue with an excited tone she exclaims, "Oh my." This combination of voice and expression indicates that she experiencing surprise, likely response to unexpected news shared during the conversation."
"The woman in the video is talking man, possibly discussing some- thing important or sharing her thoughts and foelings."
Rament hh Rebel
Classification Label (C..): "Surprise"
Raw Video & Audio
Figure 1: Example of the MERR dataset: It includes audio tone description, lexical subtitle. visual objective description, visual expression description, classification label, and multimodal description.
The MERR dataset extends the range of emotional categories and annotations beyond those found in existing datasets (Table 15). Each sample is annotated with emotion label and described in terms of its emotional expression. The dataset was initially auto-annotated with coarse- coarse-grained labels for 28,618 samples from large pool of unannotated data using LLaMA-3, and was later refined to include 4,487 samples with fine-grained annotations, carefully selected by experts. Figure 5 shows that, compared to other datasets, MERR encompasses wider range of emotional categories. More details of MERR dataset construction are provided in the project homepage and the Appendix A.
3.2 Multimodal Emotion-LLaMM Model
The proposed Multimodal Emotion Large Language Model
LLaMA) architecture, depicted in Figure 2, comprises an audio encoder gaud a visual encoder Evis and multimodal large language
"https://gitte
v2.
'nttpa://wwa www.mmodop......ommmoo http://cppp..eoooaaa//////e/e//e an //uumarry
'nttpe://hugginggace co/meta- Maaaaaaaaa-aaaaaaaaaaaaaa Llama- 3- 8B- Instruct

---

Algorithm Multimodal Emotion Annotation Procedure
Require: Video frames = f1, Sn, Audio stream A
Ensure:
Annotated video with comprehensive emotional descriptors for the peak emotional expres-
sion frame
Initialize Ieea f
2: Initialize F Prammepeak t- 0
3: for each frame f& in V do
4: Detect AUs and compute TAU t- Ei St.
5: if IAU > .peak then
6:
Ipeak f IAU Pramereak f- SN:
7:
8: end if
9: end for
10: Analyze Frameceak with OpenFace to obtain Coed
11: Analyze Frameegakk with MiniGPT-v2 to obtain Cood
12: Analyze audio A with Qwen-Audio to obtain Catd
13: Integrate Cts, Cueds Cood and Cadd to synthesize context
14: Generate comprehensive description C.dd using LLaMA-3
15: return Cmdd
model Φb Given an input tuple (Audio, Video, Prompt), Emotion-LLaMA formulated as:
Ô (d) E, P)
Φ(Eund E
E (Prompt))
where St and & denote the LLaMA language model [85], vision pre-processor, and multimodal encoder, respectively. Ô represents the formatted output text result. The multimodal encoder & consists of audio, vision, and text prompt encoders. Input V Video is pre-processed to construct the frame sequence V and Frameppak (Sec. 3.1).
Multimodal Prompt Template. To address the intricate needs of emotional understanding, we craft structured multimodal prompt template incorporating descriptive captions and emotion flags (as detailed in Table 16 and 17), directing the LLM to decipher latent correlations between emotional states and corresponding visual or auditory content. The template is denoted as:

 <VideoFeature>> <AudioFeature> [Task Identifier] Prompt [/NST]
Multiview Multimodal Encoder. To capture emotional cues in audio and visual modalities, leverage the HuBERT [39] model as our audio encoder godd and a multiview visual encoder gvis HuBERT extracts comprchensive auditory representation and from the input audio signal, exhibiting remarkable performance in emotion recognition tasks.
We use a vision preprocessor to unify vision modalities, including facial sequences and peak frame extracted from the input video. Three visual encoders guts Cuis Evis guis Stemp are employed to comprehensively extract complementary multi-view visual emotional features:
Local Encoder: A ViT-structured model pre-trained by the MAE scheme [82] extracts static facial expression features. A facial sequence is fed into the local encoder, and the output frame-wise features are fused by average pooling, producing the local visual feature uoc = AVG (V)).
Temporal Encoder: A VideoMAE [84] model, produces the temporal feature utemp (V) of a facial sequence, learning facial dynamics that indicate emotional states and offering temporal dynamic view of human emotion.
guss (tenp(()
Global Encoder: A ViT-structured model, EVA [32], initialized with official pre-trained weights, produces the visual feature uni Egto (Faame capturing only facial expressions but also background context.
Multimodal Integration and Tokenization. We use the LLaMA tokenizer, employing a byte-pair encoding (BPE) model based on SentencePiece [48], to address open vocabulary challenges and

---

facilitate efficient processing of textual inputs. For multimodal emotional reasoning, a modified generate method iteratively selects the most probable tokens, producing contextually appropriate and emotionally nuanced responses.
To integrate audio and visual features with text tokens, introduce a linear projection mechanism that transforms these features into a common dimensional space. This involves trainable linear mappings o, which include o gaud for the audio token, and o uis Specifically, we apply to convert multimodal feature into language embedding tokens T:
glo loe and temp for the visual tokens.
T α U, with u = u and vie glor Vooc Utemp
u
vis
The resulting multimodal tokens T comprise a single audio token (Tund), three visual tokens (Tyle). (TTge), and (Temmp)) and a sequence of text tokens (Tha) ..... "T These tokens are fused through the inner cross- coossattention mechanism of Emotion-LLaad
enabling it to capture and reason about the
emotional content in the multimodal input.
By employing this linear projection and multimodal token representation, Emotion-LLaMA processes and integrates information from various modalities, leveraging the strengths of the underlying LLLMMA model while incorporating essential emotional cues from audio and visual sources. Further details of the Emotion-LLaMA Model are provided in the code repository.
Visual Token
TaRE
Audio Token
Text
LLaMA2
LoRA
Frozen Weight
7
T
T
Learnable Weight
Linear
Linear
Linear
Linear
Tokenizer
Audio Encoder
Global Encoder
Local Encoder
Temporal Encoder
Prompt
t Audio
Frame
Video
Video
Figure 2: Architecture of Emotion-LLaMA, which integrates audio, visual, and text inputs for
multimodal emotional recognition and reasoning.
3.3 Training of Emotion-L.LaMA Model
We design a multi-task learning scheme to simultaneously supervise the model in learning emotional reasoning and recognition. The ground truth output and labels converted and concatenated as standard text by formatted template for autoregressive loss calculation [99]. Iterative random instruction sampling (see Table 16 and 17 for full list of instructions) for emotional reasoning and recognition tasks during training guides the model to develop a comprehensive understanding of emotions. Typically, Emotion-LLaMA is trained in a coarse-to-fine manner, consisting of the Pre-training and Multimodal Instruction Tuning:
Stage 1: Pretraining. Initially, the model is trained 28.618 coarse gaause-rained samples from the MERR dataset. Distinct tasks help the model grasp emotions from multiple perspectives. This phase involves simple descriptions classifications, facilitating the rapid alignment of multimodal feature tokens ((Taad), ) (Tyie), (TTze), and (Teim to the word embedding space [10, 97].
Stage 2: Multimodal Instruction Tuning. The pretrained Aanootion-LLMas model is then refined using fine- mne-ggriied instructional datasets to enhance its capacity for emotion recognition and reasoning. This stage utilizes multimodal instruction tuning datasets, incorporating 4,487 fine fine-graine grained annotated descriptions for comprehensive reasoning from the MERR dataset. The tuning process is extended to diverse sources, including MER2023 [59] and DFEW [45], which feature precisely annotated emotional categories. This phase ensures that the model not only identifies emotions accurately but also understands the underlying context and reasoning behind each emotion. More details are in the code repository and the Appendix B.
5

---

Experiments
4.1 Experimental Setup
To verify the effectiveness of Emotion-LLaMA, we conducted extensive evaluations across four different datasets: MER2023 [59]. MER2024 [60]. DFEW [45], and EMER [62]. Notably, we utilized the MERR dataset for pre-training the model and then fine-tuned it on target datasets for evaluation.
Emotion Recognition Evaluation. We performed instruction tuning on the MER2023 and DFEW datasets, allowing the model to integrate the emotional knowledge acquired during pretraining. To test the generalizability of our model, we used three datasets: MER2023, MER2024, and DFEW. These datasets multimodal emotion recognition datasets composed of movie and TV series clips, each annotated with various emotion categories. For fair comparisons, we evaluated EmnttoonLLaMMe on MER2023-SEMI and MERS024-NOISE using the Fl score. We also compared with other MLLMs and state-of state-of-the-arr (SOTA) methods using unweighted average recall (UAR) and weighted average recall (WAR) on the DFEW dataset. Additionally, we used the average of accuracy and recall scores as evaluation metrics on the MER2024-OV dataset.
Emotion Reasoning Evaluation. The EMER dataset differs from traditional emotion datasets by including emotion trigger labels, such as facial micro-expressions, tone of speech, and video context information, in addition emotion categories. To assess the emotional reasoning capabilities of different MLLMs on the EMER dataset, we employ ChatGPT to score their predictions, focusing on three key aspects: (I) the degree of overlap between emotion-related clues, (2) the degree of overlap between summarized emotional states, and (3) the completeness of the reasoning process across modalities. This multi-faceted evaluation provides a rigorous and in-depth assessment of the models' ability to understand and explain emotions in a multimodal context.
4.2
Implementation Details
For the global visual encoder, employ the EVA model with full images sized at 448 x 448 pixels as input. For the local and temporal visual encoders, we first crop and align the faces within the images, then hierarchical sample 16 facial images inputs for the MAE and VideoMAF. models. The audio is handled by the HuBERT-Chhneee large model. The extracted emotional features are transformed into a 4096-dimensional space via linear layers before being concatenated with text tokens.
During the tuning process, we froze the visual and audio backbones, focusing on training the linear projection layer. For the language model (LLM), we utilize LLaMA2-chat (7B) equipped with LoRA for parameter-efficient tuning. Following the Minigpt-v2 approach, we fine-tune the query and value projection matrices (W₂ and W.) by setting r 64 and = 16. Consequently, the trainable totaled only 34 million, representing a mere 0.495% of the overall
parameters of parameter count. We train 4* A100 GPUs for 300,000 steps, which takes around 20 hours. Detailed information can be found on the project homepage and in the code repository.
4.3 Comparison with State-of-the-Art Methods
To comprehensively evaluate the performance of Emotion-LLaMa, we compared it with several state-of-the-art (SOTA) methods across different datasets.
Multimodal Emotion Reasoning Results.
Table 1:
Comparison of multimodal emotion reason- ing results on the EMER dataset. Clue Overlap and Label Overlap scores range from 0 10.
We compared Emotion-L_LaM with contem- porary MLLMs such as Video-LLaMA. Video- ChatGPT, PandaGPT, VideoChat, and Valley, presenting the results in Table 1. VideoChat demonstrates that aligning visual data directly with textual embedding space (VideoChat- Embed) significantly outperforms converting it into textual format (VideoChat-To) Text), sup- porting our method of mapping audio and vi- sual features to textual embedding space. No- tably, other MILMss that accept audio inputs, like PandaGPT and Video-LLaMA, show standout performance, suggesting inefficiencies in
Models
Clue Overlap
Label Overlap
VideoChai [54]
6.42
3.94
Video-LLaMe [97]
6.64
4.89
[68]
6.95
5.74
PandaGPT [81]
7.14
5,51
VideoChat- [54]
7.15
5.65
Valley [67]
7.24
5.77
Emotion- LLaMA (ours)
7.83
6,25

---

extracting rich emotional content from audio. Emotion-LLanee excels beyond these models across both Clue Overlap and Label Overlap evaluation metrics, highlighting our model's unparalleled ability to extract direct emotional features and engage in logical emotional reasoning. The scoring criteria and cases are presented the Appendix C.I.
Multimodal Emotion Recognition Results. Table 2 presents the comparison results on the DFEW dataset. In the zero-shot scenario, -LaMM demonstrates superior capabilities compared to all other MLLMs, showcasing its strong generalization ability, Notably, the majority of MLLMs scored zero in the disgust category, with GPT-4V achieving only 10.34%. This may be attributed to safety constraints on the term "disgust" within large language models, indicating need for further exploration. Additionally, different MLLMs tend to favor predicting specific emotion category, resulting in higher scores for those categories but lower recall scores overall. In contrast, Emotion-L_IaMs maintains a more balanced prediction across all categories, ultimately achieving the highest WAR score of 59.37%. After fine-tuning. AmottonnLLLaMi achieves the highest Unweighted Average Recall (UAR) and Weighted Average Recall (WAR) scores, further indicating exceptional performance in emotion recognition tasks. These results highlight the effectiveness of our model in adapting to new datasets and accurately identifying emotions across various modalities. Overall, the results of recognizing emotions from multimodal data.
performance highlight the effectiveness of our approach in accurately
Table 2: Comparison of multimodal emotion recognition results on DFEW. The upper part shows
zero-shot performance, while the lower part shows results after fine-tuning.
Method
Hap
Sad
Neu
Ang
Sur
Dis
Fea
UAR
WAR
Zero-Shot
Owen-Audio [22]
25.97
12.93
67.04
29.20
6.12
0.00
35.36
25.23
31.74
LLaVA-NEXT [64]
57.46
79.42
38.95
0.00
0.00
0.00
0.00
25.12
33.75
MiniGPT-v2 [10]
84.25
47.23
22.28
20.69
2.04
0.00
0.55
25.29
34.47
Vidco-LLaVA [63]
37.09
27.18
26.97
58.85
12.97
0.00
3.31
20.78
31.10
Valeo-LLaVAAviider [63]
51.94
39.84
29.78
58.85
0.00
0.00
2.76
26.17
35.24
Video-Llama [97]
20.25
67.55
80.15
5.29
4.76
0.00
26.77
35.75
GPT-4V [61]
62.35
70.45
56.18
50.69
32.19
10.34
51.11
47.69
54.85
Emotion- L.MMA (ours)
71.98
76.25
61.99
71.95
33.67
0.00
3.31
45.59
59.37
Fine-tuning
EC-STFI [45]
79.18
49.05
57.85
60.98
46.15
2.76
21.51
45.35
56.51
Former-DFER [102]
84.05
62.57
67.52
70.03
56.43
3.45
31.78
53.69
65.70
IAL [52]
87.95
67.21
70.10
76.06
62.22
0.00
26.44
55.71
69.24
MAE-DFER [82]
92.92
77.46
74.56
76.94
60.99
18.62
42.35
63.41
74.43
VideoMAE [84]
93.09
78.78
71.75
78.74
63.44
17.93
41.46
63.60
74.60
S2D [12]
93.62
80.25
77.14
81.09
64.53
1.38
34.71
61.82
76.03
Emotion Emotion-LLaMA LLaMA (ours)
93.05
79.42
72.47
84.14
72.79
3.45
44.20
64.21
77.06
4.4 Multimodal Emotion Recognition Challenge
To further validate the effectiveness of our proposed Emotion-LLaMe model, we conducted ex- periments on the MER2023 and MER2024 Challenge, comparing it with previous state-of sate--of-th--n methods. The results, presented in Table 3, demonstrate that our model, which maps audio and visual features to the textual space, achieves the highest Fl score across various modalities. This approach significantly enhances the context of the textual modality by providing a more comprehensive under- standing of the information, thereby outperforming other models. By integrating audio, visual, and textual data, Aaaotion-LLaMa, can better capture the nuances emotional expression, leading to more accurate and reliable emotion recognition.
The MER2024 Challenge introduced new Open- Vocabuloyy Multimodal Emotion Recognition (MER-OV) task. Unlike traditional tasks, MER-OV focuses on recognizing any number of labels across diverse categories, aiming for more nuanced and precise description of emotional states. As shown in Table 4, Emotion-LLaMA outperforms other mainstream multimodal models, yielding an 8.52% improvement in average accuracy and recall compared to GPT-4V, and achieving the highest zero-shot score among all participating large multimodal models. These results showcase the robustness and versatility of our approach in handling complex multimodal data for emotion recognition tasks, making it promising solution for real-world applications.

---

Table Comparison of multimodal emotion recognition results on MER2023. The table shows the performance of different models across various modalities, with the highest F1 scores achieved by our proposed method.
Table 4: Performance (%) Multimodal Large
Language Models on MER2024 Challenge track 3:
MER-OV. The "avg" column represents the average of "Accuracys" and "Recalls"
Model
Accuracy,
Recalls
Avg
Method
Modality
F1 Score
Empty
0.00
0.00
0.00
wav2vec 2.0 [4]
A
0,4028
Random
13.42
24.85
19.13
VGGish [38]
A
0.5481
Ground Truth
93.37
52.51
72.94
HuBERT [39]
A
0.8511
ResNet [37]
V
0,4132
Valley [67]
20.16
13.26
16.71
Otter [5]]
29.64
23.04
26.34
MAE [36]
V
0.5547
PandaGPT [81]
35.75
31.57
33.66
VideoMAE [84]
0.6068
RoBERTa [66]
T
0.4061
Video-LLaMe [97]
31.08
32.26
31.67
BERT [25]
0,4360
VideoChat [54]
43.17
44,92
44.05
MacBERT [24]
T
0,4632
VideoChat2 [55]
46.91
34.78
40.85
MER2023-Base Baseline [59]
A. V
0.8675
Video-ChatGPT [68]
46.20
39.33
42.77
SER2003-Basclinn [59]
A. V.
0.8640
SALMONN [83]
42.20
44.75
43.47
Transformer [8]
A. V.
0.8853
Qwen-Aude Audio [22]
55.12
32,91
44.02
FBP [13]
A, V.
0,8855
mPLUG-Owl [94]
44.80
46.54
45.67
VAT [26]
A. V
0.8911
AffectGPT [62]
66.14
46.56
56.35
Emotion-LLame LLaMA
A. V 0.8905
GPT-4V [71]
56.19
58.97
57.58
Emotion- Emotion-LLaMA LLaMA
A. V.T
0.9036
Emotion-I.LaMA
69.61
62.59
66.10
Table 5: An example of multimodal emotion reasoning comparing Emotion-LLaMA with other
MLLMs. Incorrect reasoning is marked in red, correct reasoning in blue, and hallucinations in gray.
Example Multimodal Emotion Reasoning
Subtitle (En)
you really stupid pretending? Do about spending this money? just joke.
Ground Truth
In the viden, although she had smile on her face, she questioned the other person, "Are you really stupid or just pretending?" expressing dissatisfaction and anger towards the other person's of pretending be foolish. Therefore, can inferred that his emotional state is angry.
PandaGiPT
emotional state this person appears positive and relaxed. They are smiling laughing. indicating that they find the situation amusing or enjoyable. The woman's 's reaction to the man's joke suggosts she is not offended the individuals friendly,
angry finds funny or amusing. tonse or mocomfortablc.
that
VideoChai Embed
video, lady appears quite upsel angry. discusses financial losses boyfriend soems indifferent to her feelings. She expresses desáre spend money. The lady' 's facial expressions and gestures,
frowning and clutching parse tightly..
Valley
From the video, we can infer that the woman might be in neutral mixed emotional state. In some scenes, she is huppy
smiling or making amusing faces, suggesting
Emotion-LLaMA
In the vidco. the woman's is facial expressions, including a furrowed brow, lifted eyebrows, and a slightly lowered lip. suggest a mix emotions. Her tone and intonation convey a sense of frustration, and her words reveal that she is questioning whether the other person silly, indicating sense of annoyance. Therefore, inferred that her emotional anger.
4.5 Qualitative Analysis of Emotion Reasoning
To illustrate the qualitative performance of Emotion-L.LaMA, present a detailed comparison of emotion reasoning results across different models. Table displays the emotion reasoning results of the four highest-scoring models. The video shows a person smiling while questioning another individual, an expression of dissatisfaction that suggests an angry emotional state. Accurate emotion reasoning for this sample necessitates integrating information from multiple modalities. PandaGPT and Valley captured the correct visual features but failed to incorporate information from other modalities, incorrectly classifying the emotion as happy. In contrast, VideoChat-Em eventually reached the correct inference, but its reasoning was compromised by hallucinations. Emotion-LLaMA went a step further by recognizing the tone of the person and combining subtle facial expressions with multimodal information for accurate emotion reasoning. This example demonstrates the superiority of our model in understanding and integrating emotional cues from various modalities, resulting in more precise and contextually relevant emotion recognition.

---

would like meet with you to discuss malter.
No appointment and going Impossible. vidao
Surpriso
vidco
10
H00P7
Angyy
Angly
Is necessary increase to 110V?
Sorry to trouble everyone.
4.5
Neutral
Worrod
Warrie
Figure 3: Visualization of the output probability distribution for multimodal emotion recognition by different models. Each sample is represented by two bar graphs: the left graph displays the results from other models, and the right graph shows the results from Einti.nnLLaMA.
Figure displays the recognition results of Emotion-LLaMA in comparison to other models. The left samples show that even when characters exhibit subtle emotional fluctuations lacking distinet emotional features, our Emotion-LLaMA model accurately discerns the true intentions behind text. The right samples demonstrate that, unlike other language models, Emotion-LLaMA can extract multimodal emotional features to enhance text, thereby accurately understanding the true emotions conveyed by the text. These qualitative results further illustrate the effectiveness of our model capturing and interpreting emotional nuances across different modalities, leading to a more comprchensive and accurate understanding of human emotions.
4.6 Ablation Evaluation
We conducted series of ablation experiments explore the effectiveness of each component of the proposed Emotion-LLaMA. More ablation experiments, which examine factors affecting instruction- tuning performance, including data quantity and quality well hyperparameters, are presented in Appendix C.
Investigation of Encoders. We explore various combinations of encoders. As shown in Table 6, the combinations listed in the upper part are fused by the part of the modalities, while the rest capture all the modalities. The best combination of the audio and multiview visual encoders the which obtained 0.891 F1 score. Two observations are worth noting: 1) multi- ple modalities, including audio, static, and dynamic vision, are compensated for in the emotion capturing process; 2) the spa- tial, spatial-context, and temporal infor- mation are fully considered by the mul- tiview visual module, which exhibits a significant improvement over single-view approaches. These findings highlight the importance of incorporating diverse modalities and considering multiple aspects of visual information for accurate emotion recognition.
Table 6: Ablation Study Results for Different Encoders.
Audio Encoder
Visual Encoder
Fl Score
Wav2Vec VGGish whisper HuBERT
0.4893
0.5944
0,5324
0.8394
MAE
0.6366
VideoMAE
0.6762
EVA
0.6635
MAE, VideoMAE, EVA
0.7122
HuBERT
MAE
0.8800
HuBERT
VideoMAE
0,8805
HuBERT
EVA
0.8757
HuBERT
MAE, VideoMAE
0.8880
HuBERT
MAE, EVA
0.8896
HuBERT
VidcoMAE, EVA
0.8802
HuBERT
MAE, VideoMAE, EVA
0.8910

---

In Table 7. we present the impact of different instruction data on the instruction-tune of Emotion- LLaMA. 'Raw' refers to the direct concatenation of visual and audio descriptions as instructions for training Emotion-L.LaMA. which yielded the poorest performance. When trained using the coarse- grained set from the MERR dataset, Emotion-L. label overlap, respectively. This marks an improvement of 1.87 and 1. 1.25 over the 'Raw' approach, demonstrating that coarse-grained annotations generated by the LLaMA-3 model effectively integrate emotional cues to capture genuine emotional expressions. Notably, further instruction-tuning using the fine-grained set from the MERR dataset resulted in additional gains of 0.42 and 0.69 in clue and label overlap. respectively, indicating that fine- theegga annotations offer higher quality data and further enhance the performance of instruction -ottuutttootuning...
achieved scores of 7.41 and 5.56 for clue and
To better understand the effect of the sample selection strategy, we compared our strategy against traditional semi-supervised approaches, as shown in Table 8. Due to the limited size of the MER2023 training dataset, which contains only 3,373 samples, pre-training on this dataset can lead to difficulties in fitting models with transformer structures, resulting in a low F1 score of 0.7977. We then compared traditional semi-supervised approaches, which involve assigning pseudo-labels to all unlabeled samples or selectively giving pseudo-labels to those samples that exhibit high softmax scores during inference. There are 73.148 and 36,490 samples selected by these two strategies, respectively. During the pre-training phase, a substantial increase in data volume can significantly enhance performance, even introducing considerable noise. Ultimately, the model tuning our automatically annotated MERR dataset achieves the best model performance. This demonstrates the effectiveness and robustness of the proposed Emotion-LLaMA in leveraging large- scale, diverse datasets for improved emotion recognition and reasoning.
Table 8: Ablation Study Results for Different
Table 7: Ablation Study Results for Different Stages of the MERR Dataset Instructions.
Training Strategies.
Strategy
F1 Score
Stage
Clue Overlap
Label Overlap
Standard Training
0.7917
Raw
5.54
4.31
High- ConfiCCnneeecee Training
0.8831
Coarse-grained
7.41
5.56
Pseudo-Label Training
0.8950
Fine- Fine-grained
7.83
6.25
Instruction Tuning
0.9036
5 Ethics and Conclusion
Ethics. All datasets used in this study are governed by signed usage agreements, strictly restricting their use to academic research. The MERR dataset is derived from MER2023 and includes over 70,000 unannotated samples from diverse movies and TV series. We have obtained the necessary End User License Agreements (EULA) and explicit permissions from the original data providers. The open-source MERR dataset contains only emotion description JSON files, intentionally excluding source videos. Researchers must apply directly to the original providers and fully comply with the EULA to access the dataset. We ensure that the MERR dataset exclusively comprises multimodal emotion descriptions without any discriminatory or biased content.
Conclusion. We introduced Einotion-LLaMM novel multimodal large language model designed to accurately recognize and interpret human emotions in real-world scenarios. Utilizing robust open- source tools, we automated the selection and annotation of the Multimodal Emotion Recognition and Reasoning (MERR) dataset for pre-training purposes. Instruction-tunie tuning on comprehensive datasets such as MER2023 and DFEW enabled to achieve -tate-of-the-art performance metrics. Com- parative analyses with other advanced multimodal large language models (MLLMs) demonstrated Emotion-LLaMA''s superior generalization capabilities in emotion recognition and reasoning tasks.
Acknowledgments. This work was partially supported by the National Natural Science Foundation of China (Grant No. 62176165), Stable Support Projects for Shenzhen Higher Education Institutions (Grant No. 202207181 111007711119111100 10911000) (0091001), and the Natural Science Foundation of Top Talent at SZTU (Grant No. GDRC202131). Additional support was provided by NSF CISE (Grant No. 1937998), the Air Force Research Laboratory (Grant No. FA8750-19-2-0200 19-2-0200), U.S. Department of Commerce (Grant No. 60NANB 17D156), IARPA (Grant No. D17PC00340), and DARPA (Grant No. HR001 I 1990063). The research findings and conclusions presented in this paper are solely those of the authors and do not necessarily represent the views of the funding agencies.
10

---

References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736 2022.
[2] Mohammed Abbas Ansari, Chandni Saxena, Tanvir Ahmad, al. Jmi semeval 2024 task 3: Two-step approach for multimodal ecac using in-context learning with gpt and instruction- tuned Ilama models. arXiv preprint arXiv:2403. 04798, 2024.
[3] Anas Awadalla, Irena Gao. Joshua Gardner, Jack Hessel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, March 2023.
[4] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:1 33:12449- 12460, 2020.
[5] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. /2966, 2023.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
[7] Min Cao and Zhendong Wan. Psychological counseling and character analysis algorithm based on image emotion. IEEE Access, 2020.
[8] Haifeng Chen, Chujia Guo, Yan Li, Peng Zhang, and Dongmei Jiang. Semi-supervised multimodal emotion recognition with class-balanced pseudo-labeling, In Proceedings of the 3t ACM International Conference on Multimedia, pages 9556-9566 2023.
[9] Haifeng Chen, Xi Li, Dongmei Jiang, Lei Xie, et al. Improving multimodal emotion recognition by leveraging acoustic adaptation and visual alignment. arXiv preprint arXiv:2409.0555 05015, 2024.
[10] Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478 2023.
[11] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.15/92 2023.
[12] Yin Chen, Jia Li, Shiguang Shan, Meng Wang, and Richang Hong. From static to dynamic: Adapting landmark aaware awwre image models for facial expression recognition in videos. IEEE Transactions on Affective Computing, 2024.
[13] Zebang Cheng, Yuxiang Lin, Zhaoru Chen, Xiang Li, Shuyi Mao, Fan Zhang, Daijun Ding, Bowen Zhang, and Xiaojiang Peng. Semi-supervised multimodal emotion recognition with expression mac. In Proceedings of the 31st ACM International Conference on Multimedia, pages 9436-9440, 2023.
[14] Zebang Cheng, Fuqiang Niu, Yuxiang Lin, Zhi-Qi Cheng, Bowen Zhang, and Xiaojiang Peng. Mips semeval-2024 task 3: Multimodal emotion-cause pair extraction in conversations with multimodal language models. arXiv preprint arXiv:2404.005 20224
[15] Zebang Cheng, Shuyuan Tu, Dawei Huang, Minghan Li, Xiaojiang Peng, Zhi-Qi Cheng, and Alexander G Hauptmann. Sztu-emu at mer2024: Improving emotion-llama with conv-attention for multimodal emotion recognition. arXiv preprint 2r8.v: 2024.
11

---

[16] Zhi-Qi Cheng, Qi Dai, Siyao Li, Teruko Mitamura, and Alexander Hauptmann. Gsrformer: Grounded situation recognition transformer with alternate semantic attention refinement. In Proceedings of the 30th ACM International Conference on Multimedia, pages 3272-3281 2022.
[17] Zhi-Qi Cheng, Yang Liu, Xiao Wu, and Xian-Sheng Hua. Video ecommerce: Towards online video advertising. In Proceedings of the 24th ACM international conference Multimedia, pages 1365-1374, 2016.
[18] Zhi-Qi Cheng, Xiao Wu, Yang Liu, and Xian-Sheng Hua. Video ecommerce++: Toward large scale online video advertising. IEEE transactions multimedia, (996):1170-111 2017.
[19] Zhi-Qi Cheng, Xiao Wu, Yang Liu, and Xian-Sheng Hua. Video2shop: Exact matching clothes in videos to online shopping images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4048-4056, 2017.
[20] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[21] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham. Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
[22] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang. Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919 2023.
[23] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling catructiio-finetuune language models. arXiv preprint ar arXiv:2210.1144 1416, 2022.
[24] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. Revisiting pre-trained models for chinese natural language processing. arXiv preprint arXiv:2004. 2020.
[25] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 2018.
[26] Chaoyue Ding, Daoming Zong, Baoxiang Li, Ken Zheng, Dinghao Zhou, Jiakui Li, and Qunyan Zhou. Learning aligned audiovisual representations for multimodal sentiment analysis. In Proceedings of the 1st International Workshop on Multimodal and Responsible Affective Computing, pages 21-28, 2023.
[27] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.033777 2023.
[28] Paul Ekman and Wallace V Friesen. Facial action coding system. Environmental Psychology
& Nonverbal Behavior, 1978.
[29] Qi Fan, Yutong Li, Yi Xin, Xinyu Cheng, Guanglai Gao, and Miao Ma. Leveraging contrastive learning and self-training for multimodal emotion recognition with limited labeled samples. arXiv preprint arXiv:2409. 04447, 2024.
[30] Qi Fan, Haolin Zuo, Rui Liu, Zheng Lian, and Guanglai Gao. Learning noise-robust joint representation for multimodal emotion recognition under realistic incomplete data scenarios. arXiv preprint arXiv:2311.161 14, 2023.
[31] Weiquan Fan, Xiangmin Xu, Xiaofen Xing, Weidong Chen, and Dongyan Huang. Lssed: a large-scale dataset and benchmark for speech emotion recognition. In ICASSP 202/-202/ IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 641-645. IEEE, 2021.
12

---

[32] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang. Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 19358- 19369, 2023.
[33] Mengying Ge, Dongkai Tang, and Mingyang Li. Video emotion ooen-voca recognition based on multimodal large language model. arXiv preprint arXiv:2408. /286, 2024.
[34] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180-15190, 2023.
[35] Yuxiang Guo, Faizan Siddiqui, Yang Zhao, Rama Chellappa, and Shao-Yuu Lo. Stimuvar: Spatiotemporal stimuli-aware video affective reasoning with multimodal large language models. arXiv preprint 2xXv:2449.003044 2024.
[36] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 16000-16009, 2022.
[37] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[38] Shawn Henshey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. Con architectures for large-scale audio classification. 2017 ieee international conference on acoustics, speech and signal processing (icassp), pages 131-135. IEEE, 2017.
[39] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdi- nov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3466 2021.
[40] Shaohan Huang, Li Dong, Wenhui Wang. Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302. 2e.e222222.445555 2023.
[41] Lai Po Hung and Suraya Alias. Beyond sentiment analysis: A review of recent trends in text based sentiment analysis and emotion detection. Journal of Advanced Computational Intelligence and Intelligent Informatics, 27(1):84 2023.
[42] Ashley Hutchison and Larry Gerstein. Emotion recognition, emotion expression, and cultural display rules: Implications for counseling. Journal of Asia Pacific Counseling, 7(1), 2017.
[43] Maryam Imani and Gholam Ali Montazer. A survey of emotion recognition methods with emphasis on e-learning environments. Journal of Network and Computer Applications, 147:102423, 2019.
[44] Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. Opt-iml: Scaling lan- guage model instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12001 2022.
[45] Xingxun Jiang, Yuan Zong, Wenming Zheng, Chuangao Tang, Wanchuang Xia, Cheng Lu, and Jiateng Liu. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM international conference on multimedia, pages 2881 2881-2889, 2020.
[46] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. arXiv preprint a7X1v:2301.13823, 2023.
13

---

[47] Vladimir Kondratenko, Artem Sokolov, Nikolay Karpov, Oleg Kutuzov, Nikita Savushkin, and Fyodor Minkin. Large raw emotional dataset with aggregation mechanism. arXiv preprint arXiv:2212. /2266, 2022.
[48] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint 1808. 06226, 2018.
[49] Shanglin Lei, Guanting Dong, Xiaoping Wang, Keheng Wang, and Sirui Wang. Instructere: Reforming emotion recognition in conversation with a retrieval multi-task Ilms framework. arXiv preprint arXiv:2309.11911 2023.
[50] Yuxuan Lei, Dingkang Yang, Zhaoyu Chen, Jiawei Chen, Peng Zhai, and Lihua Zhang. Large vision-language models as emotion recognizers in context awareness, arXiv preprint urXiv:2407.11300 2024.
[51] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306.055 05425, 2023.
[52] Hanting Li, Hongjing Niu, Zhaoqing Zhu, and Feng Zhao. Intensity-aware loss for dynamic facial expression recognition the wild. Proceedings of the AAAI Conference on Artificial Inielligence, volume 37, pages 67-75, 2023.
[53] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. B2ip- Bootstrapping language- image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[54] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0633 2023.
[55] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22195-22206, 2024.
[56] Xiang Li, Zhi- Zhi-Qi Cheng, Jun Jun-Yan He, Xiaojiang Peng, and Alexander G Hauptmann. Mm-tts: A unified framework for multimodal, prompt-induced emotional text-to-speech synthesis. arXiv preprint arXiv:2404.18399 2024.
[57] Xinpeng Li. Teng Wang, Jian Zhao, Shuyi Mao, Jinbao Wang, Feng Zheng, Xiaojiang Peng, and Xuelong Li. Two in one go: Single-stage emotion recognition with decoupled subject- context transformer. arXiv preprint arXiv:2404. 2024.
[58] Yong Li, Yuanzhi Wang, and Zhen Cui. Decoupled multimodal distilling for emotion recogni- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6631-6640, 2023.
[59] Zheng Lian, Haiyang Sun, Licai Sun, Kang Chen, Mngyu Xu, Kexin Wang, Ke Xu, Yu He, Ying Li, Jinming Zhao, et al. Mer 2023: Multi-label learning, modality robustness, and semi- supervised learning. In Proceedings of the 31 ACM International Conference on Multimedia, pages 9610-9614, 2023.
[60] Zheng Lian, Haiyang Sun, Licai Sun, Zhuofan Wen, Siyuan Zhang, Shun Chen, Hao Gu, Jinming Zhao, Ziyang Ma, Xie Chen, al. Mer 2024: Semi-supervised learning, noise robust- ness, and open-vocabulary multimodal emotion recognition. arXiv preprint arXiv:2404.17113. 2024.
[61] Zheng Lian, Licai Sun, Haiyang Sun, Kang Chen, Zhuofan Wen, Hao Gu, Bin Liu, and Jianhua Tao. Gpt-4v with emotion: zero-shot benchmark for generalized emotion recognition. Information Fusion, page 102367, 2024.
14

---

[62] Zheng Lian, Licai Sun, Mingyu Xu, Haiyang Sun, Ke Xu, Zhuofan Wen, Shun Chen, Bin Liu, and Jianhua Tao. Explainable multimodal emotion reasoning. arXiv preprint "Xiv:2306.15401 2023.
[63] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311. 10112 2023.
[64] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
[65] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024.
[66] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.1106 2019.
[67] Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint
2023.
[68] Muhammad Maaz. Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306. 05424, 2023.
[69] Daniel McDuff, Rana Kaliouby, Thibaud Senechal, May Amr, Jeffrey Cohn, and Rosalind Picard. Affectiva-mit facial expression dataset (am-fed); Naturalistic and spontaneous facial expressions collected. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 881 881-888,72 2013.
[70] Jia Le Ngwe, Kian Ming Lim, Chin Poo Lee, and Thian Song Ong. Patt-lite: Lightweight patch and attention mobilenet for challenging facial expression recognition. arXiv preprint arXiv:2306.0966 2023.
[71] OpenAI. Gpt-4v(ision) system card, 2023.
[72] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022.
[73] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models the world. arXiv preprint arXiv:2306.1448 2023.
[74] Anbin QI, Zhongliang Liu, Xinyong Zhou, Jinba Xiao, Fengrun Zhang, Qi Gan, Ming Tao, Gaozheng Zhang, and Lu Zhang. Multimodal emotion recognition with vasion-language prompting and modality dropout. arXiv preprint www.i:2409907078. 2024.
[75] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 3748-8763. PMLR, 2021.
[76] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
[77] Nicolas Richet, Soufiane Belharbi, Haseeb Aslam, Meike Emilie Schadt, Manuela González- González, Gustave Cortal, Alessandro Lameiras Koerich, Marco Pedersoli, Alain Finkel, Simon Bacon, et Text-and feature-based models for compound multimodal emotion recognition in the wild. arXiv preprint arXiv:2407.12922 2024.
15

---

[78] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07900 07909, 2015.
[79] Pujin Shi and Fei Gao. Audio-guided fusion techniques for multimodal emotion analysis. arXiv preprint arXiv:2409.05007 2024.
[80] Gaurav Sinha, Rahul Shahi, and Mani Shankar. Human computer interaction. In 2010 3rd International Conference on Emerging Trends in Engineering and Technology, pages 1-4. IEEE, 2010.
[81] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang. and Deng Cai. Pandagpt: One model to instruction-folee them all. arXiv preprint arXiv:2305. /6355, 2023.
[82] Licai Sun, Zheng Lian, Bin Liu, and Jianhua Tao. Mae-dfer: Efficient masked autoencoder for self-supervised dynamic facial expression recognition. In Proceedings of the 31st ACM International Conference on Multimedia, pages 61 6110-6121, 2023.
[83] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13388) 2023.
[84] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078-10093, 2022.
[85] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and finettuedd chat models. arXiv preprint 13xiv:2307.092888 2023.
[86] Shuyuan Tu, Qi Dai, Zuxuan Wu, Zhi-Qi Cheng, Han Hu, and Yu-Gang Jiang. Implicit temporal modeling with learnable alignment for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19936-19947, 2023.
[87] Kai Wang, Xiaojiang Peng, Jianfei Yang, Shijian Lu, and Yu Qiao. Suppressing uncertainties for large-scale facial expression recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 6897-6906, 2020.
[88] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305. // 175, 2023.
[89] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instruc- tions. arXiv preprint arXiv:22 /2. I0560, 2022.
[90] Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint 22KY::2204.07700. 2022.
[91] Yuanzhi Wang, Yong Li, and Zhen Cui. Incomplete multimodality emotion recogni- tion. Advances in Neural Information Processing Systems, 36, 2024.
[92] Hongxia Xie, Chu-Jun Peng, Yu-Wen Tseng, Hung-Jen Chen, Chan-Feng Hsu, Hong-Han Shuai, and Wen-Huang Cheng. Emovit: Revolutionizing emotion insights with visual instruc- tion tuning. arXiv preprint arXiv:2404.166 16670, 2024.
[93] Jingyuan Yang, Qirui Huang, Tingting Ding, Dani Lischinski, Danny Cohen-Or, and Hui Huang. Emoset: large-scale visual emotion dataset with rich attributes. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 00383-20394, 2023.
[94] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint 230i::2300.111770 2023.
16

---

[95] Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning, arXiv preprint arXiv:2309.05660 2023.
[96] AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis- Pouis-Philippe Morency. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-22466 2018.
[97] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint urXiv:2306.02888, 2023.
[98] Liyun Zhang. Microemo: Time-sensitive multimodal emotion recognition with micro- expression dynamics in video dialogues. arXiv preprint arXiv:2407.16652,, 2024.
[99] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient tin-ttnning of language models with zero-init attention. arXiv preprint arXiv:2303.16199 2023.
[100] Sitao Zhang, Yimu Pan, and James Z Wang. Learning emotion representations from verbal and nonverbal communication. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18993-19002 2023.
[101] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
[102] Zengqun Zhao and Qingshan Liu. Former-dfer Dynamic facial expression recognition transformer. In Proceedings of the 29th ACM International Conference on Multimedia, pages 1553-1561, 2021.
[103] Hengshun Zhou, Debin Meng, Yuanyuan Zhang, Xiaojiang Peng. Jun Du, Kai Wang, and Yu Qiao. Exploring emotion features and fusion strategies for audio-video emotion recognition. In 2019 International conference on multimodal interaction, pages 562-566, 2019.
[104] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
17

---

A MERR Dataset Details
A.1 Emotion Categories and Annotations
The Multimodal Emotion Recognition and Reasoning (MERR) dataset covers diverse range of emotion categories, including some that are often overlooked or challenging to distinguish, as shown in Figure 4 and Figure 5. The dataset includes nine emotion categories: neutral, happy, angry, worried, surprise, sad, fear, doubt, and contempt. While the first seven categories are commonly addressed in most emotion datasets, MERR stands out by also focusing on doubt and contempt. These two categories are often underrepresented due to the difficulty in collecting sufficient samples and the potential for confusion with other emotions. Doubt, for example, can easily be mistaken for worry, as both emotions involve sense of uncertainty and concern. However, there are subtle differences in facial expressions and contextual cues that can help distinguish between the two. Doubt often involves a more questioning or skeptical facial expression, with raised eyebrows and a slight frown, whereas worry tends to have a more anxious or apprehensive appearance, with furrowed brows and a downturned mouth. Contempt, on the other hand, is frequently misclassified as happiness due to the presence of a smile. However, the smile associated with contempt is often a scornful or dismissive one, accompanied by a slight sneer or a raised upper lip. The context and manner in which the smile is displayed can help differentiate between genuine happiness and contemptuous expression.
To accurately categorize these challenging emotions, the MERR dataset relies on rich multimodal descriptions that provide a comprehensive understanding of the emotional state and its context. These descriptions go beyond simple categorical labels and offer detailed insights into the facial expressions, body language, vocal cues, and situational factors that contribute to the emotional interpretation. Table 12 presents a template for the multimodal descriptions used in MERR, showcasing the different components that are captured for each sample. The descriptions include a visual expression compo- nent that focuses on the specific facial movements and action units associated with the emotion, visual objective component that describes the overall scene and context, an audio tone component that captures the vocal cues and intonation, and textual component that provides the transcribed speech or dialogue. To further illustrate the value of these detailed annotations, Tables 13 and 14 provide specific examples of annotated samples for doubt and contempt, respectively.
A.2 Data Filtering and Pseudo-Labeling
To create a high-quality dataset for multimodal emotion recognition and reasoning, we employed a data filtering and pseulo-labeling process. This process aimed to identify video segments with strong emotional expressions and assign them initial emotion labels based on facial cues.
First, we used OpenFace to extract faces from the video segments. OpenFace is lfate-of-the-ar tool for facial behavior analysis that can detect and track facial landmarks, head pose, eye gaze, and facial action units (AUs). AUs widely used system for describing facial muscle movements, with each AU corresponding to a specific muscle or group of muscles. After extracting the faces, we aligned them aligned faces to identify the presence and intensity of various AUS throughout each video segment. Next, we utilized the detected AUS to assign pseudo-labele to the video segments. As shown in Table 10, certain combinations of AUs are strongly correlated with specific emotions. For example, the combination of AU05 (upper lid raiser) and AU26 (jaw drop) is often associated with the emotion of surprise. Similarly, the presence of AU04 (brow lowerer) and AU15 (lip corner depressor) is indicative of sadness. By leveraging these known AU combinations, we created a rule-based system to assign pseudo-labels to the video segments. If a segment exhibited specific combination of AUs with sufficient intensity, was assigned the corresponding emotion label. This process allowed us to identify samples that displayed strong emotional expression characteristics based on facial cues alone. Through this data filtering and pseudo-labeling process, we selected a total of 28,618 samples from the initial pool of video segments. These samples were assigned pseudo-labels corresponding to the nine emotion categories in the MERR dataset: neutral, happy, angry, worried, surprise, sad, fear, doubt, and contempt. It is important note that while the lseudo-labels provided valuable starting point for annotation, they were not relied upon as ground truth. In subsequent stages of the dataset construction process, human annotators reviewed and refined these labels, taking into account additional context from the visual, audio, and textual modalities.
canonical pose facilitate accurate AU detection. OpenFace then analyzed the
18

---

A.3 Instruction Collection and Multimodal Annotation
To provide rich, multimodal annotations for the MERR dataset, we collected instructions and descriptions from various sources, focusing on four key aspects: visual expression, visual context, audio tone, and multimodal integration.
Visual Expression Description. In videos, natural actions such as blinking and speaking can lead to different combinations of Action Units (AUs) being extracted from various frames. To accurately represent the current emotion, it is crucial to determine the most relevant AUs. As illustrated in Figure 6, our approach involves analyzing the amplitude values of the AUs to identify the "emotional peak frame", which captures the most intense emotional expression.
The specific steps for identifying the emotional peak frame are as follows:
1. Identify the AUs present in all frames of the video segment.
2. Sum the amplitude values of these AUs for each frame.
3. Determine whether the combinations of these AUS match the pseudo-label.
4. Select the frame with the highest total amplitude as the emotional peak frame.
Once the emotional peak frame is determined, its corresponding AUS are mapped to visual expression descriptions using the guidelines provided in Table 11. These descriptions provide a detailed account of the facial movements and expressions associated with the emotion displayed in the peak frame.
Visual Objective Description. To provide comprehensive understanding of the emotional context in each video, we utilize the MiniGPT-v2 [10] model to generate descriptions of the visual scene. By inputting the complete emotional peak frame, which captures the most intense emotional expression, MiniGPT-v2 can analyze and describe various aspects of the video, such as the environment, character actions, and object interactions. These visual objective descriptions offer valuable insights into the situational context surrounding the emotional expression. For example, if a character is shown in a dimly lit room with a concerned facial expression, the model might generate a description like "The scene takes place in a dark, shadowy room. The character appears to be sitting alone, with a worried look on their face, fidgeting with their hands." " This description provides additional information about the setting and the character's ' body language, which can help in interpreting their emotional state. Moreover, the model can also identify and describe relevant objects or elements in the scene that may contribute to the emotional context. For instance, if a character is holding letter and appears upset, the model might mention the presence of the letter in its description, suggesting that the content of the letter could be related to the character's 's emotional response.
Audio Tone Description. In addition visual cues, audio plays crucial role in conveying emotional information. The tone, intonation, and prosodic features of a speaker's 's voice can provide valuable insights into their emotional state, often revealing subtle nuances that may not be apparent from visual cues alone. To capture these audio cues, we employ the Qwen- Audio [22] model, which is specifically designed to analyze and describe the emotional content of speech. By processing the audio track of each video segment, it can generate detailed descriptions of the speaker's 's tone and intonation. For example, if a character is speaking with a trembling voice and high pitch, the model might generate a description like "The speaker's s voice is shaky and high-pitched, indicating sense of fear or anxiety. There are noticeable pauses and hesitations in their speech, suggesting uncertainty or distress." " This description captures the emotional nuances conveyed through the speaker's 's vocal delivery, providing additional context for understanding their emotional state. Moreover, Owen-Audic can also identify and describe other relevant audio cues, such as sighs, laughter, or changes in speaking rate, which can further contribute to the emotional interpretation. For instance, if a character is speaking rapidly and laughing, the model might generate description like "The person in the video speaks with cheerful tone."
Multimodal Description. To generate initial coarse- gaarseegrained emotional descriptions, we concatenate the information obtained from all modalities (visual expression, visual context, audio tone, and textual content). These surface-level descriptions for the 28,618 pseudo-labeled samples are used in the first stage of pre-training to help the model align emotional features with the textual semantic space. We input all the collected emotional clues into the LLaMA-3 model for further refinement. I.LMM-33 sifts through the clues, identifies the most relevant ones, and combines them to generate a comprehensive emotional description. This process helps to filter out any erroncous contradictory
19

---

descriptions that may have been present in the initial set of emotional clues. Additionally. we remove duplicate or overabundant samples to ensure a balanced and diverse dataset. Through these refinement processes, the final MERR dataset contains 4,487 samples, each accompanied by a detailed multimodal description. Table 12 presents an example of the annotation format used for each sample in the dataset. Finally, four domain experts manually review the refined descriptions and use a voting process to select fine-grained samples, assessing whether the video descriptions are reasonable and the emotional reasoning is accurate. By collecting and integrating instructions and descriptions from multiple modalities, we have created a rich and informative dataset that captures the complexities of emotional expressions in real-world scenarios. The multimodal annotations in MERR enable models to learn more comprehensive and nuanced representations of emotions, leading to improved performance on emotion recognition and reasoning tasks.
A.4 Data Statistics and Comparisons
Video Duration Distribution. Figure 7 presents the distribution of video durations in the MERR dataset. The majority of the samples have a length between 2 and 4 seconds, which aligns with the typical duration of short, emotionally expressive video clips. This duration range strikes a balance between capturing sufficient context for emotion recognition and maintaining a manageable data size for processing and annotation. Shorter clips may lack the necessary context to fully understand the emotional state, while longer clips can be more challenging to annotate and may contain multiple or changing emotions. The concentration of samples in the 2-4 second range also reflects the natural temporal dynamics of emotional expressions. Most emotions intense bursts of facial movements, vocalizations, and body language. By focusing on this duration range, the MERR dataset captures the core expressive moments while minimizing the inclusion of neutral or ambiguous segments. Furthermore, the consistent duration range across the dataset facilitates the development of emotion recognition models that can operate on fixed-length input sequences. This consistency simplifies the data preprocessing and model architecture design, as the models can be optimized for the specific temporal scale of the emotional expressions.
conveyed through relatively brief,
Comparison with Previous Datasets. To highlight the unique features and contributions of the MERR dataset, compare it with several related datasets in Table 15. The MER2023 [59] and DFEW [45] datasets primarily focus on discrete emotion category labels, providing classification- oriented perspective on emotion recognition. While these datasets models that can predict specific emotion categories, they lack the detailed descriptions and contextual information necessary for deeper emotion understanding. On the other hand, datasets like EmoSet [93] and Emo VIT [92] offer visual descriptions of emotional expressions, capturing the facial cues and body language associated with different emotions. However, these datasets do not include information from other modalities, such as audio or text, which can provide crucial insights into the emotional context and help disambiguate complex or subtle expressions. The EMER [62] dataset stands out as one of the few existing datasets that contain multimodal emotion descriptions, incorporating information from visual, audio, and textual modalities. However, due to the high cost and effort involved in manual annotation, EMER is limited to only 100 samples, which may not be sufficient for training robust and generalizable emotion recognition models.
valuable for developing
In contrast, the MERR dataset offers a comprehensive and large-scale resource for multimodal emotion recognition and reasoning. With 28,618 coarse-grained and 4,487 fine-grained annotated samples, MERR provides a diverse range of emotional expressions across nine categories, including challenging ones like doubt and contempt. The extensive multimodal descriptions in MERR, en- compassing visual expressions, visual context, audio tone, and textual information, enable holistic understanding of emotional states and their situational context. Moreover, the MERR dataset's inclusion of detailed emotion reasoning annotations sets it apart from other datasets. By capturing the thought process and rationale behind the emotion labels, these annotations facilitate the development of models that can not only recognize emotions but also explain and justify their predictions. This level of interpretability is crucial for building trust and transparency in human- human-compure computer interaction scenarios. As such, MERR has the potential to advance the field of affective computing and contribute to the development of more intelligent and empathetic human-computer interaction systems.
20

---

Assessing the Quality of Dataset Instructions. We conducted a human evaluation of the annotation process by randomly selecting 20 video samples from each of the nine emotion categories. We then randomly shuffled the fine-grained annotations. Five volunteers evaluated the consistency and relevance of the video descriptions by scoring them. Each volunteer rated a total of 180 descriptions on a scale from 0 to 5. The evaluation criteria included:
Accuracy of the visual modality description.
2. Accuracy of the audio modality description.
3. Accuracy of the textual modality description.
4. Correctness of the reasoning process.
5. Correctness of the reasoning result.
As shown in the Table 9, the average score for the human evaluation of the MERR dataset is 4.258, indicating that the annotations are of high quality and align with real-world logic. Secondly, the "Neutral" category received the lowest score among all categories, suggesting that when character's emotion is neutral, the facial and audio cues are weaker, making automatic annotation more challenging. We will address these findings and discuss related limitations in future work. Details of the human evaluation, including the code and assessment results, can be accessed through the code repository.
Table 9: The scores from manual evaluation of the MERR datasets for fine mme-grained annotation. The scores range from 0 to 5, with the Mean representing the average score across all categories.
Volunteer
Angry
Happy
Surprise
Fear
Sad
Worry
Neutral
Doubt
Contempt
Mean
Human-1 4.23
4.33
4.31
4.19
4.33
4.04
4.55
4.79
4.34
Human-2 4.54
4,78
4.88
4.81
4.60
4.62
4.65
4.55
4,68
4.67
Human-3
3.92
4.11
4.25
4,50
4.00
4.24
3.78
4.27
4.05
4.12
Human-4
3.69
3.94
3.62
3.69
3.93
4.19
3.39
4.00
4.53
3.90
Human-5
3.92
4.72
4.25
4.06
4.00
4.19
4.17
4.32
4.58
4.26
21

---

Table 10: Correspondence between facial expression labels and Action Units (AUs). Each expression label associated with unique combination of AUs, allowing for accurate mapping between facial movements and emotional categories.
"happy" "AAU0"" "AU12", "AU14"]
"angry": [ "AU04", "AU05", "AU07", "AU23", "AU10"; "AU17"1
"worried": ("AU28", "AU20"]
"surprise": ["AU0I", "AU02", "AU05", "AU26"]
"sad": "AAU0"" "AU01", "AU14", "AU15")
"fear": [ "AAUOI", "AU02", "AU0"", "AU04", "AU05", "AU07", "AU20", "AU26""
"doubt": ["AU25"]
contempt": (AAU12, "AU10", "AU15", "AU17"]
Table 11: Mapping of Action Units (AUs) to their corresponding textual descriptions. Each AU repre- sents specific facial muscle movement, and the textual descriptions provide human-interpretable explanation of the visual cues associated with each AU.
"AU01": ["Inner brow raiser", "Frown", "Eyebrow raised", "Head lifting wrinkles", "Liif
eychrows"]
"AU02":
("Outer brow raiser", "Outer brow lift", "Elevate outer brow", "Outer brow arch"]
"AU04": ["Brow Lowerer", "Frowns furrowed", "Lower eyebrows", "A look disapprroval")
"AU05": U"ppper Lid Raiser", "Pupil enlargement", "Eyes widened", "Liif upper eyelids", "Raise upper eyelids"]
"AU06": '"Check Raiser", "smile, Pleasure", "Slight decrease in eycbrows", "Eyes narrowing", "Slightly lower eyebrows"]
"AU07": ["Lid Tightener", "Facial tightness", "Tightening of eyelids"]
"AU09": N*Nose Wrinkler", "Wrinkle the nose", "Curl the nose", "Make a face", "Pucker the nose"]
"AU10": U"Upper Lip Raiser", "Curl the lips upwards". "Upper lip lift", "Lips apart showing teeth"]
"AU12":
['Lip Comer Puller", "Toothy smile", "Grinning", "Big smile". "Show teeth"] "AU14": ("Dimpler, "Cheek dimple", "Indentation when smiling", "Hollow on the face when smiling"]
"AU15": ["Lip Corner Depressor", "Downturned corners of the mouth", "Downward mouth curvature", "Lower Lip Depressor")
"AU17": ["Chin Raiser", "Lift the chin", "Chin held high", "Lips arching", "Lips forming upward curve"]
"AU20": ['Lip stretcher", "Tense lips stretched",, "Anxiously stretched lips", "Nasal flaring".
"Nostrils enlarge"!
"AU23": ["Lip Tightener", "Tighten the lips,' 'Purse the lips". "Press the lips together")
"AU25": ["Lips part", "Open the lips". "Slightly puzzled", "lips slightly parted")
"AU26": ["Jaw Drop", "Mouth Stretch". "Open mouth wide". "Wide-mouthed", "Lips elon- gated"]
"AU28": ["Lip Suck", "Purse lips", "Pucker lips". "Draw in lips", "Bring lips together")
22

---

Action Units Distribution MERR Dataset
AU04
3284
AU07
1646
AU10
1624
AU25
1445
AU14
1371
AU17
1262
1108
Action Units
AU28
1047
AU06
1010
AU01
866
AU20
856
AU15
853
AU02
825
AU23
852
AU09
462
AU28
183
500
1000
1500
2000
2500
3000
$500
Quantity
Distribution
Action Distribution 'Happy'
Action 'Angry'
907
665
292
263
Quentty
Quantty
Quanty
485
385
379
312
274
AU10
AUM
AU10
AU17
AU25
Action Units
Action Units
Action Units
Action
'Worried"
1200
Action
'Supprise'
938
Quarity
Quanniy
167
95
125
118
322
262
AU14
AU07
Action Units
Action Units
Action Units
Units Distribution 'Fear"
Action Units Distribution 'Doubt'
40
Unite Distribution 'Contempt"
56
Quamity
AUZS
AUDS
AU04
AU10
AUJ7
Action Units
Action Units
Action Units
Figure 4: Distribution and analysis of Action Units (AUs) in the MERR dataset. The top part displays the frequency of different AUS across all samples, highlighting the most prevalent facial movements. The bottom part presents a statistical breakdown of the top five most frequent AUs for each of the nine facial expression categories, providing insights into the facial patterns associated with each emotion.
23

---

Lubel Distribution in OFEW Datanet
Label Distribtionn in MER2021 Dataset
Label Distribtion Across Dalasets
Teral:
DFEV
4000
MER2023
1600
EMER
3530
1000
Gergre
Organt
19/00
Label
Label
Dateset
Tobal: 100
lotat 4487
32.314
20
140
Label
Figure 5: Comparative analysis of label distributions across the DFEW, MER2023, EMER, and MERR datasets. The left side illustrates the individual label distributions for each dataset, while the right side presents side-by-side comparison of the label distributions, highlighting the similarities and differences in emotional category representation among the datasets.
1.02
AU04
0.36
AU09
0.28
0.1
0.02
AU26
Ramavaty Rebel
(a) Face landmark detection.
(b) Face texture feature extraction.
(c) Action units classification.
AUDS
Frame 01
AU26
AU05: 0.36
12-
AU26: 1.02
"Eyes widened" "Wide-mouthed"
Frame 07
(d) Action unit intensity by frame index. (e) Comparing emotional peak frames. (f) Visual expression description
Figure 6: Overview of the video expression annotation process using Action Units (AUs). The top section illustrates the sampling of frames from a video, extraction of aligned faces, and measurement of AU intensities in each frame. The bottom section depicts the identification of the emotional peak frame based on the highest total AU intensity. The AUS of the peak frame are then translated into corresponding facial expression descriptions, providing detailed characterization of the emotional expression at the most salient moment in the video.
24

---

Table 12: Example of an dnstructionfoolowinn data sample and its corresponding instruction template. The template provides a structured format for presenting the multimodal data, including visual, audio, and textual information, along with the target instruction and output. This structured representation facilitates the training and evaluation of the Emotion-LLaMa model on instruction- following tasks.
Lexical subtitle:
"Oh my!"
Audio tone description:
"The woman in the video speaks with an excited voice."
Visual expression description:
Raman Rebe
"Eyes widened, Wide-mouthed."
Visual objective description:
"The woman the video talking to man, possibly discussing something important or sharing
her thoughts and feelings."
Coarse-grained annotation synthesis: Template:
"This person the video is (Visual objective description). The person's 's expression actions include (Visual expression description). and the person's 's tone is [Auditory tone description). Saying: [Lexical subtitle
Example: "The woman in the video is talking man, possibly discussing something important sharing her thoughts and feelings. The woman's 's expression and actions include eyes widened and mouth wide open, and the woman in the video speaks with an excited voice. Saying: Oh my!"
Fine-grained description generation by LLaMA-3:
System: are an emotion analysis expert. Please infer emotion label based on the given the emotional features.
possibly discussing something important sharing her thoughts and feelings. The person's expression action include eyes widened, wide-mouthed, and speaking with happy voice, saying: "Oh my." Please sort out the correct emotional clues and infer why the person in the video feels surprise.
Question: The woman in the video is talking
Her facial expressions, with eyes widened mouth wide open, clearly show surprise. She amplifies visual cue with an excited tone she exclaims, "Oh my." This combination of voice and expression indicates that she is experiencing surprise, likely in response to unexpected news shared during the conversation.
Answer: In the video,
is conversing with
Table 13: Example the emotion category "doubt". The table presents multimodal data sample as- sociated with the "doubt" category, including the visual expression description, audio tone description, and textual content. This example highlights the subtle and complex nature of the "doubt" emotion and the importance of considering multiple modalities for accurate recognition and reasoning.
Lexical subtitle:
"That must his real name."
Audio tone deseription:
"The woman's 's tone sounds natural."
Visual expression description:
"Lips slightly parted."
Visual objective description:
"The person in the vidco is a woman wearing traditional Chinese clothing and gold headpiece."
Multimodal description:
"In the video, the woman's 's lips slightly parted. Her tone is natural, but her words express skepticism as she says, 'That must not be real name." Her questioning the authenticity of the name suggests that she experiencing doubt."
25

---

Table 14: Example of the emotion category "contempt". The table showcases multimodal data sample corresponding to the "contempt" emotion, encompassing the visual expression description, audio tone description, and textual content. This example illustrates the nuanced and challenging characteristics of the "contempt" category, emphasizing the need for a comprehensive multimodal approach capture its subtleties.
Lexical subtitle:
"Isn't t who is?"
Audio tone description:
"The person in the video speaks in normal tone."
Visual expression description:
"Downturned comers of the mouth, lips apart showing teeth,
lip Corner Puller, chin held high."
Visual objective description:
"The person in the video is with long black hair and is wearing a white shirt and a brown vest."
Multimodal description:
"In the video, the person's 's facial expressions (downturned mouth corners, lips apart showing teeth, and lip corner puller) and sarcastic smirk before speaking suggest sense of superiority disdain. The words, 'Isn't he who he is?", convey mockery ridicule."
Table 15: Comparison of emotional datasets. The table presents a comparative analysis of several key emotional datasets, including DFEW, MER2023, EMER, and MERR. It highlights the unique features and contributions of each dataset, such as the range emotion categories, availability of multimodal annotations, and dataset size. This comparison underscores the significance of the MERR dataset advancing multimodal emotion recognition and reasoning research.
Sufficicot Quantity
Audio Description
Visual Objective Description
Visual Exprcssion Description
Classification Label
Multimodal Description
EmoSet
[92]
DFEW [45]
X
MER2023 [59]
X
EMER [62]
x
MERR (ours)
1003
200
20
Figure 7: Distribution of video durations in the MERR dataset. The histogram illustrates the range and frequency of video lengths, providing insights into the temporal characteristics of the emotional expressions captured in the dataset. The majority videos fall within the 2-4 second range, striking balance between sufficient context and manageable data size for annotation and processing.
26

---

B Training and Implementation Details
B.1 Instructions for Multimodal Emotion Recognition
In the multimodal emotion recognition task, the goal is to predict the emotional category label for a given video sample. To guide the model, we use the task identifier [emotion] and provide set of instructions that prompt the model to classify the emotion displayed in the video. Table 16 presents the list of instructions used for the emotion recognition task. These instructions are designed to be clear, concise, and varied their phrasing to encourage the model to learn a robust understanding of the task. The instructions ask the model determine the specific emotion portrayed in the video, choosing from predefined set of categories: happy, sad, neutral, angry, worried, surprise, fear, contempt, and doubt. By using multiple instructions with slight variations in wording, we aim to prevent the model from overfitting to specific patterns or phrases and instead focus on the underlying task of identifying the emotional state based on the multimodal cues present in the video. During training, these instructions are randomly sampled and paired with the corresponding video samples and their associated annotations. The model learns to attend to the relevant visual, audio, and textual features and map them to the appropriate emotion label based on the provided instruction.
Table 16: List of instructions for multimodal emotion recognition. The table presents of carefully crafted instructions that guide the Emotion-LLaMA model in performing emotion recognition tasks. These instructions cover range of prompts and variations, ensuring the model's ability understand and respond to different formulations of the emotion recognition problem.
"Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise, fear, contempt, doubt."
"Identify the displayed emotion in the video: is it happy, sad, neutral, angry, worried, surprise, fear, contempt, doubt?"
"Determine the emotional state shown in the video, choosing from happy, sad, neutral, angry, worried, surprise, fear, contempt or doubt."
"Please ascertain the specific emotion portrayed in the video, whether it be happy, sad, neutral, angry, worried, surprise, fear, contempt or doubt."
"Assess and label the emotion evident in the video: could be happy, sad, neutral, angry, worried, surprise, fear, contempt, doubt?"
B.2 Instructions for Multimodal Emotion Reasoning
In addition to recognizing the emotional category, also want the model to be able to reason about the emotional state based on the available multimodal cues. This task requires a deeper understanding of how the different modalities contribute to the overall emotional interpretation and the ability to explain the reasoning behind the predicted emotion label. To support this task, we use the task identifier [reason] and provide set of instructions that prompt the model analyze the multimodal cues and provide a rationale for the predicted emotion. Table 17 shows the list of instructions used for the emotion reasoning task. These instructions ask the model integrate information from various modalities, such as facial expressions, vocal tone, and the intended meaning behind the spoken words, to infer the emotional state of the person the video. The model is expected to not only predict the emotion label but also provide a coherent explanation of how the different cues contribute that prediction. By training the model with these reasoning instructions, we aim to develop its ability to understand the complex interplay between the different modalities and to generate human-inte interpretable explanations for its predictions. This reasoning capability crucial for building trust and transparency in the emotion recognition system and facilitating more natural and engaging human-computer interactions. During training, the reasoning instructions are randomly sampled and paired with the video samples and their associated multimodal annotations. The model learns to attend to the relevant cues across modalities, combine them meaningful way, and generate a reasoning trace that justifies the predicted emotion label.
27

---

Table 17: List of Instructions for multimodal emotion reasoning. The table showcases a curated set of instructions designed to elicit emotional reasoning capabilities from the Emotion-LLaMA model. These instructions prompt the model to analyze multimodal cues, generate explanations, and justify its emotional predictions, fostering the development of deep and coherent emotion understanding.
Please analyze all the clues in the video and reason out the emotional label of the person the video."
"What is the emotional state of the person in the video? Please tell me the reason." "
"What the facial expressions and vocal tone used in the video? What is the intended meaning behind his words? Which emotion does this reflect?"
"Please integrate information from various modalities to infer the emotional category of the person in the video."
"Could you describe the emotion-related features of the individual in the video? What emotional category do they fall into?"
B.3 Implementation Emotion-LLaMM Model
The implementation of the Emotion-LLaMA model involves several key components and design choices that enable it to effectively learn and reason about emotions from multimodal data. this section, provide detailed overview of the main implementation details, including the training approach, model architecture, and inference process.
B.3.1 Multi-Task Learning Approach
One of the core aspects of the Emotion-LLaMA model is its ability to simultaneously learn and perform multiple tasks related to emotion understanding. Specifically, the model is trained using a multi-task learning approach, where the emotion recognition and emotion reasoning tasks are learned in parallel. In the emotion recognition task, the model Iearns to predict the appropriate emotion label for a given multimodal input, such trained to map the input features to one of the predefined emotion categories, such as happy, sad, angry, neutral. On the other hand, the emotion reasoning task focuses on generating numan-interpretabll explanations for the predicted emotion labels. Given a multimodal input, the model learns to analyze the various cues and generate natural language explanation that justifies the predicted emotion based on the available evidence. By training the model to perform both tasks simultaneously. we allow it to share representations and learn complementary skills across tasks. This multi-task learning approach has several benefits. First, it enables the model to develop a more comprehensive understanding of emotions by learning to recognize the emotional state and explain the reasoning behind it. Second, it encourages the model to learn more robust and generalizable representations by leveraging the commonalities and differences between the two tasks. During training, for each video sample input into the model, we randomly select either the emotion recognition or emotion reasoning task to enhance the model's 's generalization and robustness.
video clip accompanied by audio and text. The model is
B.3.2 Coarse-to-Fie Training Strategy
To facilitate the learning of nuanced and detailed emotional representations, we employ a coarse- to-fine training strategy. This strategy involves two main stages: pre-training on coarse- goarse-rrained annotations and fine-tuning on fine-grained annotations. In the pre-training stage, the model is trained on the coarse-grained annotations from the MERR dataset. These annotations provide simple description of emotions, such visual expression, audio tone. By training on these coarse- grained annotations, the model learns to capture the general emotional tone and develop an initial understanding of the emotional content in the multimodal data. After the pre-training stage, we proceed to the fine-tuning stage, where the model exposed to fine-grained annotations from the MERR dataset. These annotations offer more detailed and specific emotion descriptions, such as environmental information, body movements, and the emotional nuances inferred from tone and textual subtitles. By fine-tuning the model on these fine- fine-grained descriptions, enhance
28

---

its emotional understanding and enable it to capture more nuanced expressions and variations in scenarios that closely resemble real-world contexts. The coarse-to-fine training strategy offers several advantages. First, it allows the model to gradually learn more complex and detailed emotional representations, starting from a solid foundation of general emotional understanding. Second, it helps to prevent overfitring and improves the model's s generalization ability by providing a hierarchy of emotional annotations to learn from. During the fine-tuning stage, we typically use a smaller learning rate and a shorter training duration compared to the pre-training stage. This allows the model to adapt its representations to the fine-grained labels without drastically changing the learned features from the pre-training stage.
B.3.3 Modality-Speciitt Encoders
To effectively process and integrate information from multiple modalities, the Emotion EinotionLLLaMA model employs modallty-sppecffi encoders. These encoders designed to extract meaningful features and representations from each modality, capturing the unique characteristics and cues relevant to emotion understanding. For the visual modality, we use combination of three encoders: MAE (Masked Autoencoders) [36], VideoMAE [84], and EVA (Efficient Video Analysis) [32]. The MAE encoder focuses on capturing local facial features and expressions, which crucial for recognizing emotions. learns to reconstruct masked patches of the input frames, enabling it to capture fine- ine-grained details of facial movements and micro-expressions. The VideoMAE encoder, on the other hand, is designed to capture the temporal dynamics and motion patterns in the video data. It learns to predict the masked frames in a video sequence, allowing model the evolution of emotional expressions over time. By capturing the temporal context, VideoMAE helps the model to understand the progression and transitions between different emotional states. Finally, the EVA encoder is used capture the global scene understanding and contextual information in the visual data. It processes the entire video frames and learns to extract high-level features that represent the overall scene and environment. This global context is important for interpreting the emotional state of individuals in relation to their surroundings and the situational factors. For the audio modality, we employ the HuBERT (Hidden- Unit BERT) [39] encoder, which is a state-of-te thetthr model for speech representation learning. HuBERT is trained on a large corpus of unlabeled speech data and learns to predict the hidden units of a pre-trained teacher model. By doing so, it captures rich vocal representations that encode prosodic features, such as intonation, stress, and rhythm, which are indicative of emotional states. Finally, for the textual modality, use the LLaMA tokenizer to handle the transcribed speech dialogue. The LLaMA tokenizer [85] is based on a byte-level Byte Pair Encoding (BPE) [78] algorithm, which allows for efficient and flexible tokenization of the input text. The tokenized text is then passed through the LLaMA model, which learns to capture the semantic and emotional content in the language data.
B.3.4 Unified Representation Space
To enable the Emotion-LLaMA model to reason about emotions across modalities, we project the outputs from the modality-specificc encoders into a unified representation space. This is achieved by applying linear transformation to the features extracted by each encoder, mapping them to common embedding dimension. In addition to the projected features, we also introduce learned special tokens that are concatenated with the multimodal embeddings. These special tokens serve as task-specific indicators and provide additional context for the model to distinguish between the emotion recognition and emotion reasoning tasks. The resulting unified representation is then fed into the LLaMA model, which uses its self-attention mechanism to attend to and reason about the multimodal cues in a holistic manner. The LLaMA model Icarns to capture the interactions and dependencies between the different modalities, enabling it to make informed predictions and generate coherent explanations. By operating unified representation space, the Amotion-LLaMM, model can effectively integrate and align information from multiple modalities, allowing for cross-modal reasoning and understanding. This unified space facilitates the model's ability to capture the complex dynamics of emotional expressions and generate meaningful insights based on the combined evidence from visual, audio, and textual cues.
B.3.5 Training Objective and Optimization
The training objective of the Einotion-LLaMe model for both the emotion recognition and emotion reasoning tasks based on the language modeling loss. This loss function, commonly used in natural language generation tasks, measures the likelihood of generating the ground-truth tokens given the
29

---

input multimodal data. By minimizing this loss, the model learns to produce coherent and contextually relevant tokens that align with the provided reasoning examples or emotion categories, effectively capturing the underlying patterns and relationships in the data. To optimize the model parameters during training. we employ the Adam optimizer, a widely adopted and efficient optimization algorithm in deep learning. Adam adapts the learning rate for each parameter based on historical gradients, providing a robust and adaptive optimization process. By leveraging the adaptive learning rates, Adam helps the model converge faster and find better local minima, leading to improved performance and generalization. To train the large language model component of Emotion-LLaMe efficiently, utilize the Low-Rank Adaptation (LoRA) technique. LoRA significantly reduces the number of trainable parameters while preserving the model's pre-existing world knowledge. By adapting only small set of low-rank matrices, LoRA allows the model to acquire domain-spectin knowledge related to emotions, such as nuances in tone of speech and facial expressions, without overwriting losing the valuable information learned during pre-training. This technique strikes balance between efficiency and effectiveness, enabling the model to specialize the emotion domain while retaining its general language understanding capabilities. For more detailed of the training objective, optimization process, and the use of LoRA, please refer to the source code?.
http:://ggggttbbooooo////////n// Ltt.:chhhhenemmiioooooaeaaae LLaMA
30

---

Experiments & Demonstration
C.I Evaluation Metries
To comprehensively assess the performance of the Emotion-L.LaMA model on various datasets, we employ range of evaluation metrics tailored to the specific characteristics and requirements of each dataset. These metrics designed to provide fair and thorough evaluation of the model's emotion recognition and reasoning capabilittis, taking into account factors such as class imbalance and the importance of both majority and minority classes.
C.I.1 DFEW Dataset
For the DFEW dataset, we utilize two main evaluation metrics: Weighted Average Recall (WAR) and Unweighted Average Recall (UAR). These metrics are particularly well-suited for imbalanced datasets, where the distribution of samples across different emotion categories is not uniform. WAR focuses on the model's ability to recognize the majority classes, which have larger number of samples in the dataset. is calculated as the weighted sum of the recall values for each emotion category, where the weights are determined by the proportion of samples in each category:
WAR Σ N Recall,)
where N is the total number of emotion categories, is the number of samples in category i, and Recall, the recall value category On the other hand, UAR ensures that the model's capability to identify the minority classes, which have fewer samples, is not overlooked. It is calculated as the unweighted average of the recall values across emotion categories:
UAR Σ Recall;
(5)
By considering both WAR and UAR, we can assess the model's performance on the majority and minority classes independently, providing more comprehensive evaluation. A high WAR indicates that the model is effective at recognizing the most prevalent emotions, while high UAR suggests that the model is capable of identifying even the less frequent emotions accurately. The combination of these metrics offers a balanced assessment, ensuring that can handle the class imbalance present in the DFEW dataset and provide reliable emotion recognition across all categories.
C.1.2 MER2023 Dataset
For the MER2023 dataset, we employ the Weighted Average F-score (WAF) as the primary evaluation metric. WAF is composite metric that combines precision and recall, providing a single value that reflects the model's overall performance. The F-score for each emotion category calculated as the harmonic mean of precision and recall:
F2 (Precision; Recall,)
(6)
Precision; + Recall;
where Precision; and Recall; are the precision and recall values for emotion category i, respectively, The WAF is then computed as the weighted average of the F-scores across emotion categories, with the weights determined by the proportion of samples in each category:
WAF
F
where N is the total number of emotion categories, is the number of samples in category i, and F the F-score for category By incorporating both precision and recall, WAF provides balanced measure of the model's iss ability to correctly identify emotions while minimizing false positives and false negatives. takes into account the class imbalance present the dataset, giving more weight to the performance on the majority classes while still considering the minority classes. WAF offers several advantages over using recall or accuracy alone. It reduces the impact of class imbalance on the evaluation results, ensuring that the model's performance the minority classes is not overshadowed
31

---

by its performance on the majority classes. Additionally, WAF is more robust to noise and outliers, as it considers both the true positive and false positive rates. By using WAF as the evaluation metric for the MER2023 dataset, we can obtain a comprehensive assessment of the Emotion-L.LaMe model, taking into account the precision and recall of emotion recognition across categories.
C.1.3 EMER Dataset
For the EMER dataset, we employ a unique evaluation approach that leverages the reasoning capabilities of ChatGPT (i.e., gpt-3.5-turbo- 35-turbo-16k-06133 large language model. The evaluation focuses assessing the quality and coherence the emotional reasoning provided by the Emotion- LLaMA model. We follow the evaluation prompt template shown in Table 18, where (gt_reason) represents the ground truth emotional description provided in the EMER dataset, and (pred_reasee represents the emotional reasoning generated by the AmottinnLLLMMe model or other baseline models. The evaluation prompt is designed to elicit an objective score from ChatGPT based on a set of predefined scoring rules. These rules assess the quality of the generated reasoning in terms of its alignment with the ground truth description, the coherence and logicality of the reasoning process, and the extent which it captures the relevant emotional cues and evidence.
To ensure fair and unbiased evaluation, ChatGPT is provided with clear set of scoring guidelines. These guidelines instruct ChatGPT to consider factors such as the overlap between the predicted and ground truth reasoning, the presence of contradictory or irrelevant information, and the overall clarity and coherence of the generated explanation. ChatGPT then assigns a score to each generated reasoning based on these guidelines, providing a quantitative measure of the model's 's performance. Along with the numeric score, ChatGPT also provides a brief justification for the assigned score, highlighting the strengths and weaknesses of the generated reasoning.
By leveraging ChatGPT's language understanding and reasoning capabilities, we can obtain a more comprehensive and nuanced evaluation of the Emotion-LLaMA model's performance on the EMER dataset. This approach goes beyond simple metrics like accuracy or F-score and assesses the model's ability generate coherent and meaningful explanations for the predicted emotions.
Table 18: Prompt for calculating the degree of overlap between emotion-related clues. The table provides structured prompt template that guides the evaluation of the EmAA model's reasoning capabilities. By comparing the model's generated explanations with the ground truth, the prompt enables the quantitative assessment of the model's ability to identify and articulate relevant emotion-related cues.
Below, the "Actual Description" and "Predicted Description" of character are given. Please follow the steps below to calculate the score for the "Predicted Description", The score should range from 10. In the end, only output the numerical value of the predicted score along with the reasoning.
1. Summarize the emotional state description of the character from the "Actual De- scription".
2. Summarize the emotional state description of the character from the "Predicted Description".
3. Calculate the overlap between the "Predicted Description" and the "Actual Descrip- tion". The higher the overlap, the higher the score.
4. Output format:
'Predicted Score": Predicted Score; 'Reason': Reason
Input:
"Actual Description": [gt r
"Predicted Description": (pred_ reason]
Output:
32

---

C.2 Additional Results and Analysis
In this section, we present additional results and analysis to further explore the performance and characteristics of the Emotion-LLLaM model. We examine the confusion matrices for the MER2023 and DFEW datasets, discuss the challenges posed by class imbalance, and provide detailed scoring cases from the EMER dataset to showcase the model's 's emotion reasoning capabilities.
C.2.1 Confusion Matrices
Figure 8 presents the confusion matrices for the MER2023 and DFEW datasets, providing insights into the model's performance across different emotion categories. In both datasets, the categories "angry," "happy," and "sad" are the most common and distinctive emotions encountered in real- world scenarios. These emotions often have clear and pronounced facial expressions, vocal cues, and language patterns, making them relatively easier for the Emotion-LLaMr, model to learn and recognize accurately. The confusion matrices show that the model achieves high accuracy for these prevalent emotion categories. The majority of the samples belonging to "angry," "happy," and "sad" are correctly classified, with minimal confusion between them. This indicates that the model has successfully learned the discriminative features and patterns associated with these emotions and can effectively distinguish between them.
Angry
Angry 0.84
0.00
0.02
0.01
0.08
0.02
0.03
0.01
0.90
0.04
0.02
0.02
0.01
Disgust
0.24
0.03
0.10
0.00
0.18
0.03
0.10
Fear
0.8
0.06
0.06
0:81
0.04
0.01
0.02
0.02
0.00
0.00
0.93
0.03
0.01
0.01
Sed
0.00
0.00
0.00
0.01
0.04
0.72
0.06
0.08
0.00
0.14
0.07
0.29
Sad
0.04
0.02
0.03
0.2
0.09
0.02
0.07
0.04
0.02
8.76
Surprise
Arroy
Hapmy
Nestral
5ed
Supprise
Worried
Disgust
Fear
Happy
Nestral
Sed
Surprise
(a) MER2023-SEMI
(b) DFEW
Figure 8: Confusion matrices for multimodal emotion recognition datasets. The figure presents
confusion matrices that visualize the performance of the Emotion-LLaMA model on benchmark datasets such as MER2023 and DFEW. The matrices provide insights into the model's classification accuracy, highlighting the challenges and successes in distinguishing between different emotional categories. [Zoom in to view]
However, the confusion matrices also highlight the challenges posed by less frequent emotion categories, such as "disgust" and "fear." Due to the inherent class imbalance present in multimodal emotion recognition datasets, these categories have significantly fewer training samples compared to the more common emotions. The limited availability of training data for "disgust" and "fear" makes more difficult for the model to learn and generalize the patterns and cues specific to these emotions. As result, the model may struggle to accurately recognize and classify samples belonging to these categories, leading to higher confusion rates and lower performance compared to the more prevalent emotions. The confusion matrices reveal that notable proportion of "disgust" and "fear" samples are misclassified as other emotions, such as "angry" or "sad." This suggests that the model may be relying overlapping ambiguous features that shared between these emotions, leading to confusion and misclassification.
Note that addressing the class imbalance issue is a crucial challenge in multimodal emotion recogni- tion. Further research and exploration are needed to develop techniques that can effectively handle the imbalanced distribution of emotion categories and improve the model's performance on less frequent emotions. Potential approaches mitigate class imbalance include data augmentation techniques, such oversampling the minority classes generating synthetic samples, to increase the represen-
33

---

tation of underrepresented emotions in the training data. Additionally, employing class-weighted loss functions or adaptive learning strategies that focus on the minority classes during training can help the model learn more robust and generalizable features for these emotions. By tackling the class imbalance problem and improving the model's s ability to recognize and classify less frequent emotions accurately, we can enhance the overall performance and practicality of the Emotion-L_LaM model real-world applications.
C.2.2 Ablation Study of Hyperaarameters
Figure 9 presents additional ablation study results, providing further insights into the impact of different hyperparameters and data settings on the performance of the Emotion-LLaa LLaMA model.
One of the key factors investigated in the ablation study is the learning rate. The learning rate determines the step size at which the model's 's parameters are updated during the training process. plays a crucial role in the model's 's convergence and generalization ability. The results in Figure 9 compare the performance of the model across different learning rates. is observed that the choice of learning rate has significant impact on the model's performance. Too high a learning rate can lead to unstable training and suboptimal convergence, while too low a learning rate can result slow convergence and potential overfitting.
The ablation study helps identify the optimal learning rate range for the -LaMA model, striking a balance between convergence speed and generalization performance. By carefully tuning the learning rate, we can ensure that the model effectively learns the underlying patterns and features from the training data while avoiding overfitting or underfitting. Another aspect explored in the ablation study is the effect of data proportions on the model's performance. The results compare the model's 's performance when trained on different subsets of the training data, ranging from smaller fractions to the full dataset. The findings highlight the importance of sufficient training data for the Laationn-LLaMA model to achieve optimal performance. As the proportion of training data increases, the model's performance generally improves, indicating ability to learn more comprehensive and robust representations of emotions.
Moreover, the ablation study also reveals that the performance gains gradually diminish as the data proportion approaches the full dataset. This suggests that there may be saturation point beyond which adding more training data yields diminishing returns in terms of performance improvement. Understanding the relationship between data quantity and model performance is crucial for efficient resource allocation and practical deployment considerations. The ablation study results provide guidance the minimum data requirements for the Emotion-LLaMe model to achieve satisfactory performance and help identify the optimal trade-off between data size and computational resources.
0.59
0.50
0.00
Scoor
0.75
Score
0.87
0.70
0.55
0.86
Category
0.60
I:1e-5
0.25
10 30%
45%
42%
80%
100%
Epoch
Data Ratio
(b)
Figure 9: Ablation study results. (a) illustrates the impact of different learning rates on the model's performance, while (b) presents the effects of varying data ratios. These experiments provide valuable insights into the optimal hyperparameter settings and data requirements for training the Emotion- LLaMA model effectively. [Zoom in to view]
34

---

C.2.3 MER2024 Challenge
The MER2024 Challenge [60] has recently garnered significant attention for its focus on integrating multimodalities to identify human emotional states. This initiative aims address the limitations of existing technologies, which often struggle to meet the demands of practical applications. The challenge seeks to advance research in areas such as multimodal modeling of human affect, modality robustness in affect recognition, low-resource affect recognition, human affect synthesis in multimedia, privacy affective computing, and applications in health, education, and entertainment. The challenge's s theme has attracted large number of researchers from around the globe to discuss recent advancements and future directions for robust multimodal emotion recognition, generating many new insights [9, 29, 74, 79].
The MER2024 Challenge has introduced three distinet tracks: (1) MER-SEMI, which provides over 110,000 unlabeled samples for sumi-supervised or self-supervised learning; (2) MER-NOISE, which adds noise test videos to simulate more realistic variations in modality conditions such as background noise and blurry videos, thereby evaluating system robustness; (3) MER-OV, which requires models to extract richer and more subtle emotions an open- open- vocabulary manner, aiming to mitigate the intrinsic subjectivity in the emotion annotation process and the inherent ambiguity of MER. Therefore, the MER-NOISE and MER-OV tracks represent the most realistic expressions of emotions and data distributions, presenting both challenges and practical applications.
track emphasizes enhancing noise robustness in emotion recognition systems, as noise is ubiquitous in practical settings. Ensuring clear audio streams and high-resolution video frames can challenging. This track specifically targets two prevalent types of noise: audio additive noise and image blur, encouraging participants to utilize data augmentation techniques and other in- novative methods bolster the resilience of emotion recognition systems [30]. Due to the integration of multimodal information for reasoning, our proposed Emotion-L.L aMMA significantly reduces the impact of noise, resulting outstanding performance in the MER-NOISE track with a weighted average F-score (WAF) of 84.52%, surpassing all other teams. Furthermore, utilized the predic- tions from Einotion-LLMMe as pseudo-labels to enhance the performance Ultimately, our team leveraged the robust capabilities of Emotion-LLaMA to secure first place in the MER-NOISE track, exceeding the second and third place teams by 1.47% and 1.65%, respectively. Detailed rankings and scores shown in Figure 10.
The
Conv-Attention [15].
Ous
45
Toan
Tean Team
Weightdd avarago --c33r
-44
Team
81
50:52
Rank
Figure 10: Ranking and Scores of Teams in the MER-Noise Track of MER2024.
The MER-OV track introduces the concept of open-vocabulary emotion recognition, addressing the subjectivity and ambiguity inherent in emotion labeling. Traditional datasets often limit label spaces to few discrete categories, relying on multiple annotators and majority voting, which can overlook valid but non-candidate or minority labels. Participants generate a diverse array of labels, aiming for more nuanced and accurate representation of emotional states. [33, 98]. As shown in Table 4, Emotion-L.L.M.. demonstrates greater generalization capabilities compared to other multimodal large language models.
35

---

C.2.4 Detailed Scoring Cases
To showcase the emotion reasoning capabilities of the Emotion-LLaMA model, we present detailed scoring cases from the EMER dataset in Table and Table 20.
In these tables, we compare the emotion reasoning results generated by the Emotion-LLaMs model with those of other multimodal large language models. The evaluation is performed using the prompts from Table 18, which assess the quality and coherence of the generated reasoning against the ground truth emotional descriptions. The scoring cases demonstrate the superiority of the Mnnoiion-LLaAA model in accurately identifying and interpreting the emotional cues present in the multimodal data. The model successfully captures the facial details, vocal tone, and linguistic content, and integrates them to arrive at the correct emotion labels.
In Table 19, the Emotion- LLLMA model accurately identifies the subtle facial expressions and the slightly raised voice of the speaker, indicating a sense of doubt or uncertainty. By combining these cues with the content of the spoken dialogue, the model generates coherent and logical reasoning that aligns well with the ground truth description. Similarly, in Table 20, the Emotion-LLaMi model of raised eyebrows and slight frown, suggesting state of
correctly recognizes the facial surprise or disbelief. The model also takes into account the emphatic tone of the speaker and the context of the conversation infer the underlying emotion accurately.
In contrast, the other multimodal large language models struggle to provide accurate and coherent reasoning in these cases. They may focus irrelevant details, generate inconsistent explanations, or fail to capture the nuances of the emotional expressions. The detailed scoring cases highlight the effectiveness of the Mootion-LLaMa. model in performing multimodal emotion reasoning.
36

---

Table 19: Comparison of reasoning results, scores, and evaluation criteria between EmAM and other popular MILMS for _0pppee_000248".. The table presents detailed analysis of the emotional reasoning capabilities of different models on a specific data sample. It highlights the superiority of Emotion-LLaMa in generating accurate, coherent, and well-justified explanations, as evidenced by the higher scores and more comprehensive evaluation criteria.
An Example of Multimodal Emotion Reasoning
Subiite (En)
Ab. Wei Wei prognant.
Ground Truth
In the viden, she found out that her friend is pregnant and he very surprised. She spoke faster, raised her voice, and suddenly sat up from the sola. She opened her mouth wide, widened her eyes, and raised ber eyebrows. Therefore, it can be inferred that her emotional state surprise.
PandaGPT
This person's 's emotional state appears to be positive and happy, as they are smiling while siming on the bod. The presence pregnancy subtitle indicates that they expecting child, which is likely the source of their happiness and excitement.
Criteria
The prediction mentions the character silling the bed smiling, and subtille about pregnancy indicates that they are expecting child, suggesting positive and happy emotional state. Although the prediction does not explicilly mention the character's 's surprised emotional state. there some overlap between the described positive and happy cmotions and the surprised emotion in the actual description. leading to a higher prodiction score.
Score
7.0
VilenChat-Fmhed
Based on the viden, can be inferred that this person clearly unhappy, as they are on the phone, shaking their head, and displaying pained expression. The fact that they are crying indicates they deeply troubled or under significant stress. The video mentions a pregnant woman, so this person may be upset about recently born baby. However, without more information about the phone conversation or context, is difficult determine the exact emotional state.
Criteria
The "Predicted Description" about the character's emotional state has some overlap with the "Actual Description", as both mention their emotional condition. However, "Predicted Description" emational states, such as unhuppiness and puin, which do not fully align with the surprise described in the "Actual Description" Therefore, the scure
mentions other possible
Score
6.0
Valley
From the video, we can infer that this woman in a calm and relaxed emotional state. She sits on the bod, occasionally smiling while talking on the phone. Her facial expressions and body language indicate that she foels confortable and enjoying conversation. Additionally, woman wearing a white shirt, which contribates to casual and relaxed appearance.
Criteria
The "'Actall Description" describes the male protagonist's emotional state surprised, whereas the "Predicted Description" describes the female protagonist's emooiional state as calm and relaxed The emotionna states described completely difterent, with very low overlap, therefore the prediction score low.
Score
3.0
Viden-CharGPP
From viden, we can see that individual is happy and excited about news of the slight pregnancy. The woman is on the phone with smile on her face, which indicates her happiness and excitement. The "Actual Description" describes describes it happy and excited. Although the cmotional states are not exactly the same, both are positive emotional and based observations from the video. Therefore, overlap between the two, and the prediction score is 6. 6.0
Criteria
character's emotional state surprised, while "Predicted Description"
considered that there some degree of
Score
Video-LLaMA
Thee video shows that she young Asian female wearing white dress. The lady is looking the is on a call. She appears to be laughing or smiling. indicating that she is in good mood. The background features white cciling with black clock, suggosting that the lady is comfortable setting. Overall, the video portrays happy woman enjoying phone conversation while relaxing on the sofa.
woman sitting on sofa, talking on her mobile phone. From the description. we can see
which suggests that she
Criteria
According the "Actual Description." " the character displays surprised emotional state, whereas the "Predicted Description" describes the character's is emotional state as happy and relaxed. Although there no obvious overlup between the based on the descriptions, can speculated that the character might experience pleasant surprises during the phone conversation. Therefore, a prediction score of 6 is given. 6.0
Score
Emotion LLMM
In the video, a woman seen sitting on couch, displaying facial expressions and body language that suggest surprise and alertness. Her rassed eychrows, strong emotional tone, and quickened breathing further reinforce this interpretation. Therefore, inferred that her emotionnl siate is surprise. The predicted description closely matches the actual description regarding the emotional state of the person, including consistent descriptions of facial expressions, tonc of voice, and breathing. Therefore, the soore is high. 9.0
Criteria
Score
37

---

Table 20: Comparison of reasoning results, scores, and evaluation criteria between Emotion-LLaMA and other popular MLLMs for "sample_00006955 000006570.. Similar to the previous table, this table showcases the comparative performance of Emotion-LLaMe and other MLLMs on another data sample. It demonstrates the model's ability to capture nuanced emotional cues and provide compelling explana- tions, outperforming other approaches in reasoning quality and alignment with human judgments.
An Example of Multimodal Emotion Reasoning
Subtitle (Ein)
have to criticize you. Anyway, Mr. Wang is your teacher. How can you talk to Mr. Wang like that?
Ground Truth
In the video, he dissatistied and angry with the other person's disrespectful behavior towards Teacher Wang. and is currently criticizing the other person. The volume has been raised and the speech speed has also increased in the backend of the video. Therefore, it is infenred that his emotional state is anger.
PandaCiPT
The individuul uppears to be depressed or frustrated as they sit in a chair and talk on the phone with someome, hand on their forchead. The subitles suggest they are criticizing their teaceer. Mr. Wang, and expressing dissatisfaction with his teaching methods. This implies that the person may be facing some challenges or disagreements with their teachor. leading to their negative emetional state.
Criteria
The predicted description of the emotional state overlaps somewhat with the actual description of the emotional state, both mentioning dissatisfaction and criticism towards the teacher. Although the predicted description refers to foelings of depression or frustration, which differ from the anger mentioned in the actual description, both express dissatixfaction with the teaching methods. Therefore, the prediction of the emotional state in the predicted description can considered accurate. Due high degree overlup, a high scure is awarded.
Score
7,0
VideoChat Eimbed
From the video, we can The individual appears confident and assertive during the conversation with the teacher, indicating that they are neither overly anxious nor uncomfortable. On the other hand, teacher feels frustrated by the student's lack of couperation and professionalis during the dialogue.
person's emotional state by observing their facial expressions tone voice.
Criteria
The predicted description mentions that individual appears confident and assertive during the conversation with the teacher, which partially overlaps with the actual description where the character is dissatisied and angry with the disrespecttul behavior towards Mr. Wang. Additionally, predicted description also netes teacher's frustration due to the student's 's lack of cooperation and profeasionalism in the dialogue, which aligns to some extent with the emotional state descrited the actual context. Therefare, given the considerable overlap between the predicted and aciual descriptions, a score of 7 is assigned.
Score
7.0
Valley
From the vidco, we can infer that the man is likely in a professional setting. possibly dector's 's office, as he is dressed in white shirt and tic. He also looking clipboard, which suggests he might reviewing medical records or notes. Moreover, the video shows the man conversing with a woman, who could he patient colleague, and there man't 's emotional state is likely calm, focused, and professional, as he is engaged in a conversation and appears to he hundling medical work related tasks.
computer and chair in the
Based on these visual cues, can deduce that the
Criteria
The prodicted description of the emotional state partially overlaps with the actual description. as both mention the man's 's emotional state. In the actual description, he angry, while in the predicted description, be is calm, focused, and professional. Although the two descriptions are not eatirely consistent, both involve emotional states, so cortain score can be justified.
Score
Video-ChatGPP
The person in the video appears to be an older man who is conversing with younger woman. He is dressed while coal and seated across from her. The man talks to the woman in a calm manner, while she listens attentively. Based on these visual cues, we infer that the man likely dector professional, and the woman cither his patient or colleague. The man converses with the woman in professional manner, indicating that is providing her with medical advice or discussing professional matters. Solely hased on visual clues, it is not possible to determine man's emational state, but clear Based on the emational state described in the "Actual Description," " where the character feels dissutisfied and angry, and "Predicted Description." which does mention the character's iss emotioona state, impossible to determine the character's true emotional state. The overlap is minimal, hence a score of 5 is assigned.
is engaging in a conversation with the woman.
Criteria
Score
Video-LLaMA
This video showcases man and woman in professional office setting. The woman is speaking to the man, who appears doctor, discussing a case she handling. They are seated next table The woman seems unhuppy or stressed while speaking. Throughout the video, the sound of a clock ticking can be heard background. The seene is shot a medium frame with both individuals sitting by the table..
with documents.
Criteria
Accurding to the "Actual Description"" the character is angry, whereas the "Predicted Descriptin" does not clearly deline the character's emotional state merely describes their conversation in a professional office setting. Therefore, there is a low degree of overlap berween the prodicted and actual descriptions. resulting in score of 6. 6.0
Score
Emotion-LLLMA
In the video, a doctor sits at a dask, displaying facial expressions- a furrowed brow, raised upper eyelids- that suggest anger or frustration. Along with tome that implies anmuyance or iniitaioon words, "How can you talk to Mr. Wang like that?" convey a sense of disapproval and disgpoinntmnt. This indicates that the person in the video is experiencing anger due to the other person's misbehavior. Therefore, it inferred that his emational state anger. The predicted description of the character's emotional state highly overlaps with the actual description. including aspects such as facial expressions, tonc, and words, all conveying feelings of anger. Due this significant overlap. a high prediction score is awarded.
Criteria
Score
9.0
38

---

C.3 Demonstration of Emotion-LLaMA
To showcase the capabilities and potential applications of the Emotion-LLaMA model, we have developed interactive demo that allows users experience its emotion recognition and reasoning functionalities firsthand. The demo provides a user-friendly interface for inputring multimodal data, such images, videos, and text, and receiving real-time emotion predictions and explanations.
C.3.1 General Task Performance
Figure 12 illustrates the demo's performance general tasks, such as face detection and question answering. These tasks demonstrate the versatility and robustness of the Emotion-LLaMA model beyond its core emotion understanding capabilities. In the face detection task, the demo takes an input image and accurately identifies and localizes the faces present in the image. The model's ability to detect faces is crucial for subsequent emotion recognition, as it allows the system to focus the relevant regions of interest and extract facial features effectively. The demo also showcases the model's 's ceaistionnanseeinn capabilities. Given a question and an associated image video, the Emotion-LLaMA model can generate accurate and contextually relevant answers. By leveraging its multimodal understanding, the model can reason about the visual content and provide informative responses to user queries. 'hese general task examples highlight the Emotion-L ability to handle wide range of tasks that involve visual perception, language understanding, and reasoning. The model's performance on these tasks demonstrates its potential to be integrated into various applications, such as intelligent assistants and human-computer interaction interfaces.
model's
C.3.2 Multimodal Emotion Recognition and Reasoning
Figure 13 focuses on the core functionalities of the Emotion-LLaMA model: multimodal emotion recognition and reasoning. The demo allows users to input various combinations of visual, audio, and textual data and receive real-time emotion predictions and explanations. In the emotion recognition task, the user can provide an image or video depicting person's ' facial expressions, body language, and contextual cues. The Emotion-LLaMA model processes the visual input, extracting relevant features and patterns, and predicts the most likely emotion category based on its trained knowledge. The demo displays the predicted emotion label along with the corresponding confidence score, indicating the model's level of certainty in its prediction. This information helps users interpret the model's output and assess the reliability of the emotion recognition result.
In addition to emotion recognition, the demo enables users to explore the model's emotion reasoning capabilities. By providing multimodal input, such video clip accompanied by audio and text, users can request the Emotion-LLan -LaMA model to generate a natural language explanation for its predicted emotion. The model analyzes the multimodal data, considering the facial expressions, vocal cues, and linguistic content, and generates a coherent and human- like explanation for the identified emotion. The generated explanation highlights the specific cues and patterns that contributed to the model's prediction, providing insights into its reasoning process. The emotion reasoning feature of the demo is particularly valuable for applications that require transparent and interpretable emotion understanding. By providing explanations alongside the predicted emotions, the Emotion-L_LaN LaMA model enables users gain deeper understanding of the factors influencing the model's decisions and fosters trust in the system's 's outputs.
The demo also allows users to compare the Emotion-LLaMr. model's performance with other baseline models or human annotations. By presenting the emotion predictions and explanations from multiple sources side by side, with human judgments. This comparative analysis feature of the demo facilitates the evaluation and validation of the Emotion-LLaMA model's performance in real-world scenarios. It provides platform for researchers, developers, and end-users to explore the model's 's strengths, identify areas for improvement, and gather insights for further refinement and adaptation.
assess the model's accuracy, coherence, and alignment
C.3.3 Potential Applications and Impact
The Emotion-L_LaNe demo serves as powerful showcase of the model's capabilities and highlights its potential applications across various domains. Some of the key areas where the EmoA model can make a significant impact include:
39

---

1. Affective Computing:
The Emotion-LLaMa, model can be integrated into affective com- puting systems to enable more natural and empathetic human- computer interactions. By recognizing and responding to users' emotions, these systems can provide personalized and emotionally intelligent experiences.
2. Mental Health Assessment: The model's ability recognize and reason about emotions from multimodal data can be leveraged in mental health assessment tools. It can assist mental health professionals in analyzing patient data, identifying emotional patterns, and providing objective insights to support diagnosis and treatment planning.
3. Sentiment Analysis:
The Emotion-LLLMMA model can be applied to sentiment analysis tasks, such social media monitoring, customer feedback analysis, and brand reputation management. By accurately identifying the emotions expressed user-gemert user-generated content, businesses gain valuable insights into public opinion and make data-driven decisions.
4. Emotion-Awark Robotics: The model can be integrated into robotics systems to enable more natural and engaging human-robot interactions. By recognizing and responding to hu- man emotions, emotionally intelligent robots can provide more empathetic and personalized assistance in various settings, such as healthcare, education, and customer service.
5. Emotional Intelligence Training: The Aonotion-LLaMee model can be used to develop emotional intelligence training programs and tools. By exposing individuals to a wide range emotional expressions and providing explanations for the underlying cues and patterns, the model can help users enhance their emotional awareness and empathy skills.
These are just few examples of the potential applications and impact of the Emotion-L_J LaMA model. As the field of multimodal emotion understanding continues to advance, the model's capabilities can be further extended and adapted to address new challenges and opportunities across various domains.
For hands-on experience with the Emotion-LLaMA model, you can explore the online demo on Hugging Face", which allows you to input multimodal data samples and observe the model's emotion recognition and reasoning capabilities in action. The demo provides interactive interface to visualize the input data, the predicted emotion labels, and the generated explanations, giving you comprehensive understanding continues to evolve and the Emotion-LLaiie adoption and integration into real- world systems, empowering harness the power of multimodal emotion understanding for wide range of purposes.
the model's performance and potential applications. As the demo model is further refined, we envision its increasing
The peranie vidro aays Dhno, phone vedieeieten are my kep [smotoe) eoterine wft indtin vide mecontt happy cad sounal werted, susprie.
proeate
Shall tisk aviaal d
Figure 11: Online demo interface of the Emotion-LLaMA model. The figure presents screenshot of the interactive web-based demo, showcasing the model's capabilities in multimodal emotion recognition and reasoning. Users can input various combinations of visual, audio, and textual data and receive real-time emotion predictions and explanations, facilitating intuitive exploration and evaluation of the model's performance.
wwwnnnnggnnntiioioneonenooooooet LLaMA
40

---

what state is the person in the video, say the following: "Do you really think so?"
Is a boy or a girl in the video?
Boy.
In the video, the person is sitting on chair, looking at someone or something, and their facial expre- ssion neutral, with their eyebr- ows slightly raised the tone of their voice questioning, say "do you really think so?" This suggests that they engaged conversation or discussion, possibly asking for clarification or seeking confirmation about something.
What color clothes is the boy wearing in the video?
Red.
[detection] face.
they
What expression?
The boy in the video is wearing red jacket and has surprised expression on his face, with his mouth open and eyes wide.
[visual] What are the emotions of the woman in the video?
In the video, the woman's facial expressions and actions convey sense of happiness and contentment. She is standing in bathroom, likely enjoying her time alone or engaging in personal grooming, and her smile suggests that she feeling relaxed.
[audio] Analyze the speaker's voice in the video.
In the video, the speaker's tone sounds normal.
Figure 12: Detailed examples of general tasks performed by the Emotion LLaMA model. The figure illustrates the model's 's versatility and robustness in handling tasks beyond emotion recognition, such as face detection and question answering. These examples highlight the model's ability process and understand visual and textual information, enabling its application in wide range of scenarios.
41

---

The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label the video represents: happy, sad, neutral, angry, worried, surprise.
The person video says: Won't you? Impossible! Fan Xiaomei is not such a person. [reason] What facial expressions and vocal tone used in the video? What the intended meaning behind his words? Which emotion does this reflect?
the
Worried.
the video, the person's facial expr- essions and tone convey sense of surprise, as they say "won't you?" with a hint of disbelief. Her brows are slightly raised, and her lips are slightly parted. The combination of these cues suggests that the person taken aback by the suggestion or idea being presented, indicating that she find it unlikely or unbelievable.
The person video says: Oh my phone and wallet are in my bag. [emotion] Please determine which emotion label the video represents: happy, sad, neutral, angry, worried, surprise.
Sad.
The person in video says: Bastard! Boss, you don't choose, you prefer. [reason] Please integrate infor- mation from various modalities to infer the emotional category of the person in the video
In the video, a man stands in an office setting, looking downward with stern expression, his eyebrows furrowed and his lips pursed the tone his voice angry, as he speaks in harsh manner, using strong language, such as "bastard". This combination of facial expressions and tone suggests that the person is feeling frustrated.
The person video says: Me: I'm not going in anymore, scared. [emotion] Identify the displayed emotion in the video: is it happy, sad, neutral, angry, fear, contempt, doubt, worried, surprise?
Fear.
Figure 13: Detailed examples of multimodal emotion recognition and reasoning performed by the Emotion-LLaMA model. The figure showcases the model's core capabilities in accurately identifying emotions from multimodal data and generating human-like explanations for its predictions. These examples demonstrate the model's proficiency in capturing subtle emotional cues, integrating information across modalities, and providing meaningful insights into its decision-man makiing process.
42

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The framework proposed in the paper has been utilized to annotate emotion- related instruction-follww data. Models that are instruction-tunet on this dataset demonstrate robust generalization capabilities. This aligns with the main claims made in the abstract and introduction, accurately reflecting the paper's contributions and scope.
Justification:
Guidelines: Sec. 3.1, Table 2
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper addresses the limitations associated with the generation of instruction-follow data, specifically noting that small subset of the samples exhibits mismatches between emotional cues.
Guidelines: Sec. 3.1, Fiture 4
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
43

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper includes formula derivation of the model's trainable parameters and the principles of model training. The question regarding theoretical results is marked as not applicable (NA) because the paper primarily focuses on how to construct instruction-follow data and how to construct instructions during training.
Guidelines: Sec. 3.1, Sec. 3.3
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The paper fully discloses all information necessary to reproduce the main experimental results. It includes detailed explanations of the experimental procedures, covering aspects such as hyperparameters and instruction templates.
Justification:
Guidelines: Sec. 3.3, Sec. 4.2
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
44

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: This work provides open access to both the collected instruction-follow data and the trained model parameter files. Detailed descriptions of the necessary steps for reproducing the experiments faithfully replicate the main experimental results as described in the supplemental material. This facilitates transparency and reproducibility of the research findings.
included in the paper, ensuring that researchers can
Guidelines: Sec. 4.2
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. ca/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: /tpppscccppbllcc/cegee http://ullcggggudeeeeee /CodeSummm ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides detailed account of the training process, including how training was conducted in phases and all relevant training details. This includes specifics data splits, hyperparameters, the type of optimizer used, and the instruction templates employed. Such thorough documentation essential for understanding and validating the results presented the paper.
Guidelines: Sec. 3.3, Sec. 4.2
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
45

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: The question regarding the reporting of error bars or statistical significance is marked not applicable (NA) because the experiments were designed with reproducibility in mind, employing fixed random seeds. Multiple iterations of the experiments consistently yielded results that align with those reported in the paper, ensuring reliability without the explicit use of error bars or detailed statistical tests. This approach ensures that the findings are stable and replicable.
Guidelines: Sec. 4
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative enror rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper provides detailed information about the computing resources used in the experiments. It specifies the types of compute workers (e.g., GPU models), the memory requirements, and the execution times for each training step.
Justification:
Guidelines: Sec. 4.2
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //neuriiss... cwtrriis.ccppppe/e wttpccccpcccceeeeeeeeecc Ceaeliii///iiiiiee http:/tthhcouuieeeeeeeeeeee
Answer: [Yes]
46

---

Justification: This work adheres fully to the NeurIPS Code of Ethics. All datasets employed in the research conform standards on Copyright and Fair Use, ensuring ethical compliance in terms of data acquisition and utilization. This careful adherence supports the integrity and ethical standards expected in scientific research.
Guidelines: Sec. 4.1
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses the potential positive societal impacts of the research, particularly in the fields of human-computer interaction and intelligent education. It explores how the advancements in these areas can enhance user experience and educational outcomes, reflecting on the benefits that the technology can bring to society.
Guidelines: Sec.
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., degloymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: this work, while generating emotion-related descriptions for unlabeled samples, samples associated with political content and those potentially containing discrimi- natory material were intentionally filtered out to prevent misuse and negative impacts.
Guidelines: Sec. 3.1
The answer NA means that the paper poses such risks.
47

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In this work, all open-source models and datasets utilized adhere to their respective licenses. These resources are appropriately credited within the paper, with explicit mentions of their licenses and terms of use.
Guidelines: Sec. 3.2, Sec. 4.1
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: In this work, the newly generated instruction-follon data is thoroughly docu- mented, including details about its structure, purpose, and the methods used for generation and collection. Comprehensive statistical analyses have been conducted, and all relevant documentation has been made openly available.
Guidelines: Sec. 3.1
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
48

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
The question regarding crowdsourcing experiments and research with human subjects is marked as not applicable (NA) because the focus of this work is studying facial expressions using datasets collected and released by other institutions. These datasets comply with all relevant ethical standards. Additionally, this research was conducted under an approved application that adheres to the required regulations for academic studies involving such data.
Justification:
Guidelines: Sec. 4.1
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information
supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The study focuses the analysis of facial expressions using datasets that were collected and released by other institutions, which had already received IRB approvals. Additionally, this project has obtained the necessary approvals for academic research, ensuring compliance with all relevant regulations.
Guidelines: Sec. 4.1
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
49


Reviews:
[{"review_id": "PVqYyrjGZS", "reviewer": "Reviewer_3ouG", "summary": "The paper presents the Emotion-LLaMA model, a multimodal emotion recognition and reasoning system that integrates audio, visual, and textual inputs.\n\n- The authors constructed the MERR dataset, which includes 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories, enabling models to learn from varied scenarios and generalize to real-world applications.\n- The Emotion-LLaMA model incorporates specialized encoders for audio, visual, and textual inputs, aligning the features into a modified LLaMA language model and employing instruction tuning to enhance both emotional recognition and reasoning capabilities.\n- Extensive evaluations show that Emotion-LLaMA outperforms other multimodal large language models, achieving top scores on the EMER, MER2023, and DFEW datasets.\n\nThe main contributions are:\n- The MERR dataset, a valuable resource for advancing large-scale multimodal emotion model training and evaluation.\n- The Emotion-LLaMA model, which excels in multimodal emotion recognition and reasoning through the innovative use of instruction tuning.\n- Establishing Emotion-LLaMA as the current state-of-the-art model in public competitions for multimodal emotion analysis.", "questions": "1. In section 4.2, the author mentioned that they used the HuBERT-Chinese large model for audio modality input processing. Regarding this part, I would like to know the following questions:\n\n    -  Are the experimental results sensitive to language? I hope the author can provide more experimental results to illustrate this point.\n\n    - In the ablation study section (Tab5), the author seems to have only conducted ablation on the visual encoder. Why didn't they further conduct ablation experiments on the audio encoder, considering that there are many alternatives to Hubert?\n\n    - Based on the previous question, does the author believe that the audio modality is not important in this task? I would like to see more ablation results on the modality scale.\n2. As I mentioned in the weaknesses part, could the author provide more details of fine-tuning on the target datasets?\n3. Is Multimodal Emotion Recognition a classification task? How do the authors explain the Dis column in Table 2?", "limitations": "Although the author mentioned in the checklist that they discussed the limitations of the article, they did not explicitly discuss them in the text. \nMeanwhile, I noticed that the author did not mention their data sources in the article, and I am concerned whether this might involve data copyright issues.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper is well structured and well presented. The author organizes the article into five sections: introduction, related work, methodology, experiments, and conclusion. They clearly describe how they conduct data annotation and model design, provide a good introduction to the experimental setup and analysis of the experimental results, and present a relatively clear conclusion.\n- Model details are thorough. The authors have provided a good description of their model and training methods, allowing me to clearly understand how the model is designed and trained. I believe their results are reproducible.\n- This work is valuable . The authors provided a paradigm for emotion annotation of multimodal data and an annotated dataset. They also offered a clear explanation of the annotation process, which will contribute to the development of the related field.", "weaknesses": "- Insufficient experiments: Although the author has conducted some comparative and ablation experiments, it is obviously insufficient for such a complex multimodal LLM.\n- Missing details of experimental setup: The authors mentioned that they fine-tuned on several target datasets, but the details of the fine-tuning (including data volume, dataset division, and fine-tuning setup) were not included in the article. This can lead to a decrease in the credibility of their results. \n- Lack of result analysis: Although the proposed model surpasses existing models in many metrics, the authors only list their experimental results in the results section without further analysis and explanation of the results and some phenomena. The lack of proper explanation and analysis can make some results confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We have addressed each of your points below and will incorporate these improvements in our revised manuscript.\n\n**Q1: Are the experimental results sensitive to language? How does the choice of HuBERT-Chinese affect performance?**\n\nA1: Yes, Emotion-LLaMA is sensitive to language. This sensitivity stems from its foundation on LLaMA2-chat (7B), which processes input instructions and outputs in English. Given that most samples in the EMER, MER2023, and DFEW datasets are in Chinese, we translated all text subtitles to English.\n\nOur choice of HuBERT-Chinese as the audio encoder was informed by MERBench [1], showing that language-matching encoders achieve better performance. Our experiments confirmed this, with HuBERT-Chinese achieving the highest scores among single-modal models, underscoring the significant role of the audio modality in multimodal emotion recognition.\n\nWe tested other audio models, including Whisper, Wav2Vec, and VGGish, but they performed poorly in multimodal fusion. Consequently, we selected HuBERT-Chinese and focused on different visual encoders.\n\nOur ablation experiments show the following results:\n\n| **Audio Encoder**  | **Visual Encoder**  | **F1 Score** |\n|--------------------|---------------------|--------------|\n| Wav2Vec            | -                   | 48.93        |\n| Wav2Vec            | MAE, VideoMAE, EVA  | 71.92        |\n| VGGish             | -                   | 59.44        |\n| VGGish             | MAE, VideoMAE, EVA  | 73.89        |\n| Whisper            | -                   | 53.24        |\n| Whisper            | MAE, VideoMAE, EVA  | 70.38        |\n| HuBERT-Chinese     | -                   | 83.94        |\n| HuBERT-Chinese     | MAE, VideoMAE, EVA  | 89.10        |\n\nThese results highlight the importance of using language-matching encoders for audio modalities in multimodal emotion recognition tasks. We will discuss these findings and their implications in the revised manuscript, particularly in the limitations section.\n\n**Q2: Details of Fine-Tuning on Target Datasets**\n\nA2: Please refer to **Sec. 4.2** of our submitted paper. Implementation details, including code (train.py, train_configs/Emotion-LLaMA_finetune.yaml) and setup instructions (README.md: Setup, Training), are available in our anonymous repository. Further tuning process details are in our response to Reviewer V8m9: Q1-A1.\n\n**Q3: Is Multimodal Emotion Recognition a classification task? How do the authors explain the Dis column in Table 2?**\n\nA3: Multimodal Emotion Recognition is a classification task, aiming to classify input samples into different emotional categories. In **Tab. 2**, the 'Dis' column represents the accuracy score for the 'Disgust' emotion category. Most existing MLLMs perform poorly in recognizing 'disgust', often with zero accuracy. This is likely due to the scarcity of multimodal samples for 'disgust' in current datasets. Additionally, LLMs may have safety restrictions related to 'disgust', contributing to the low accuracy. We plan to collect more samples to enrich the MERR dataset and further explore this issue.\n\n**Q4: The author did not explicitly discuss limitations in the text and raised concerns about data copyright issues.**\n\nA4: The MERR dataset is sourced from MER2023. Our team signed the relevant End User License Agreements (EULA) and obtained permission from the original data providers. We acknowledge the need for an ethics review concerning data privacy, copyright, and consent. We have followed all ethical guidelines and included a detailed statement on ethical considerations in the revised manuscript.\n\n**Q5: Insufficient experiments.**\n\nA5: We demonstrated Emotion-LLaMA's capabilities through extensive experiments, achieving SOTA scores on the EMER, MER2023, and DFEW datasets, and conducted ablation studies to validate its components and the MERR dataset.\n\nWe also performed additional experiments:\n\n- **Audio Modality Ablation**: As detailed in Q1, we conducted extensive ablation studies on different audio encoders.\n- **MER2024 Competition Results**: Recently, we participated in the [MER2024] competition, widely regarded as one of the most authoritative benchmarks in the field of multimodal emotion recognition. Emotion-LLaMA excelled in two tracks:\n\n   a) **Noise Robustness Track (MER-Noise)**:  \n   Emotion-LLaMA achieved the highest score of 85.30%, surpassing the second and third-place scores by 1.47% and 1.65%, respectively.\n\n   | **Anonymous team** | **F1 Score** |\n   |----------------|----------|\n   | team 6         | 80.66    |\n   | team 5         | 81.28    |\n   | team 4         | 82.71    |\n   | team 3         | 83.65    |\n   | team 2         | 83.83    |\n   | team 1 (ours)  | 85.30    |\n\n   b) **Open-Vocabulary Track (MER-OV)**:  \n   Our application of Emotion-LLaMA for open-vocabulary annotation improved the average accuracy and recall by 8.52% compared to GPT-4V.\n\n   | **Model**       | **Accuracy** | **Recall** | **Avg**  |\n   |----------------|----------|--------|------|\n   | Video-LLaMA    | 31.08    | 32.26  | 31.67|\n   | Video-ChatGPT  | 46.20    | 39.33  | 42.77|\n   | mPLUG-Owl      | 44.80    | 46.54  | 45.67|\n   | AffectGPT      | 66.14    | 46.56  | 56.35|\n   | GPT-4V         | 56.19    | 58.97  | 57.58|\n   | Emotion-LLaMA  | 69.61    | 62.59  | 66.10|\n\nThese additional experiments demonstrate Emotion-LLaMA's robustness and effectiveness.\n\nIn our revised manuscript, we will:\n1. Include a comprehensive presentation of our experimental results.\n2. Provide a deeper analysis of the results, discussing implications for noisy conditions and open-vocabulary tasks.\n3. Elaborate on how these results contribute to multimodal emotion recognition and potential real-world applications.\n\nIf you have further questions or need additional clarification, please let us know. We value your feedback and are committed to providing thorough responses.\n\n[MER2024]: https://zeroqiaoba.github.io/MER2024-website/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GsLWQPNAS4", "reviewer": "Reviewer_iNoC", "summary": "The paper presents a new multi-modal instruction tuning dataset for emotion recognition. The authors also present results of training on this dataset with a multi-modal architecture based on LLaMa 2. They show evaluation results on DFEW and MER2023.", "questions": "-", "limitations": "Limitations are not discussed in enough detail. There is no separate limitations section. The authors also do not lay out the limitations concerning e.g. (citing from the checklist):\n\n\"The authors should reflect on the scope of the claims made, e.g., if the approach was580\nonly tested on a few datasets or with a few runs. In general, empirical results often581\ndepend on implicit assumptions, which should be articulated.\"\n\n\"The authors should reflect on the factors that influence the performance of the approach.583\nFor example, a facial recognition algorithm may perform poorly when image resolution584\nis low or images are taken in low lighting. Or a speech-to-text system might not be585\nused reliably to provide closed captions for online lectures because it fails to handle586\ntechnical jargon.\"", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Evaluating multi-modal emotion recognition approaches based on LLMs are a very relevant research topic.\nThe results shown by the authors are convincing w.r.t. the performance on emotion recognition datasets.", "weaknesses": "-------------\n\nThe argument presented in lines 31 and 33 is not convincing. Why does an inability in methods lead to a lack of datasets?\n\n\nRelated Work:\n-------------\n\nAs the dataset is one of the claimed contributions of the paper, there should be a discussion on previous emotion recognition datasets. It is important to lay out the reasons why these previous datasets cannot be used (or not easily be used) for instruction tuning of LLMs.\n\n\nMethodology:\n------------\n\nWhat videos is the MERR dataset based on? I could not find details in the paper on the selection process. Judging from the screenshots, it seems to be movie clips. This has important implications on the concept of \"emotion\" that is addressed in the paper and needs to be clarified.\n\nThe chosen methodology lacks justification.\nFor example, what is the reasoning behind selecting the frames with the highest sum of AU activations across all AUs included in OpenFace? It seems to me that there would be the danger of having a strong bias towards moments when the person is speaking, as this often leads to high AU activations, especially AUs related to the lower half of the face. \n\nThe mapping of AUs to facial expression labels needs more explanation. E.g. is \"happy\" assigned if the combination of AU06, AU12, AU14 is active, or is it also assigned if only some of those AUs are active? Figure 1 does not show a facial expression description according to Table 7 at all, only descriptions according to Table 8. Concerning Table 8, there are several descriptions given for each AU. Are they all used at the same time, or is only one of them chosen? In Figure 1 it appears that only one of them is chosen, but how is this decided?\n\nSeveral further steps are not clearly defined, e.g. how does LLaMA-3 \"refine\" the annotations by aggregating the multimodal description? How is the instruction following data constructed of which a single example is presented in Table 9. Is it done manually? If yes, how and how many samples are created?\n\nIn 3.1 it is not clear, how the dataset is \"auto-annotated\" with emotion labels. It is also not clear how these annotations are refined (with human involvement)? In general the concept of \"emotion\" used in the paper remains unclear. Is it about emotional displays, about internal states,...?\n\nIn later parts of the method section it seems authors are targeting (internal?) emotional states. It would be important to know how they were annotated.\n\nDo the instructions for multimodal emotion recognition (Table 11) refer to different tasks? Some of these instructions seem to target displayed emotions, some target internal states.\n\n\nTraining details are unclear. With the description provided in the paper it is difficult to understand the training procedure.\n\n\n\nEvaluation:\n-----------\n\nThe authors employ chatGPT to evaluate emotion reasoning. It is not clear to what extent this approach leads to a valid evaluation. E.g. ChatGPT was shown to be biased on emotion-related tasks [1].\nIn the end, evaluating the emotion understanding capabilities of one language model using another language model is circular.\n\nThe impact of using the proposed MERR dataset for pre-training needs to be evaluated.\n\n\n[1] R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Transactions on Affective Computing, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We've addressed your concerns as follows:\n\n**Q1: The argument presented in lines 31 and 33 is not convincing. Why does an inability in methods lead to a lack of datasets?**\n\nA1: The *issues* in line 33 refer to challenges faced by current multimodal large language models (MLLMs), particularly their difficulty in processing audio and recognizing micro-expressions. This stems from the lack of specialized multimodal emotion instruction datasets (line 31), crucial for training these tasks. Without these datasets, developing methods to integrate audio and recognize subtle expressions is difficult. \n\n**Q2: There should be a discussion on previous emotion recognition datasets and why they cannot be used for instruction tuning of LLMs.**\n\nA2: Previous emotion recognition datasets provide discrete emotion labels, unsuitable for MLLM instruction tuning:\n\n1. *EmoVIT*: Lacks audio data crucial for comprehensive emotion recognition. Our MERR dataset includes audio features for robust multimodal analysis.\n2. *EMER*: Only 100 annotated samples, insufficient for tuning. MERR offers 28,618 coarse-grained and 4,487 fine-grained samples, providing a larger, diverse set.\n3. *Other datasets (e.g., AFEW, DFEW)*: Useful for traditional tasks but not for MLLM tuning. They lack the detailed, instruction-based annotations of MERR.\n\nWe will highlight MERR's unique features - size, multimodal nature, and instruction-based annotations - that make it ideal for training MLLMs in emotion recognition.\n\n**Q3: What videos is the MERR dataset based on? Judging from the screenshots, it seems to be movie clips.**\n\nA3: The MERR dataset is indeed sourced from MER2023, which includes over 70,000 unannotated samples primarily derived from movies and TV series. These sources offer rich and diverse emotional expressions, more representative of real-world scenarios. Our team signed the relevant End User License Agreements (EULA) and obtained permission from the original data providers.\n\n**Q4 & Q5: The chosen methodology lacks justification. The mapping of AUs to facial expression labels needs more explanation.**\n\nA4 & A5: We extract the most expressive facial frame by summing the highest AU activations. This approach mitigates biases. For example, high values in AU05 (Upper Lid Raiser) and AU26 (Jaw Drop) indicate surprise or fear. Speaking often results in high AU activations, but the MERR dataset balances AUs between the upper and lower face (Fig. 4), addressing speech bias. For mapping AUs, 'happy' is assigned if AU06, AU12, and AU14 are active, or even if only some are active. Fig. 1 shows AU combinations from Tab. 7 and descriptions from Tab. 8. The top right of Fig. 1 shows the combination for 'surprise' (AU-05: 0.36, AU-26: 1.03). We will clarify this in the revised manuscript.\n\n**Q6 & Q7: It is not clear how the dataset is \"auto-annotated\" with emotion labels and how these annotations are refined.**\n\nA6 & A7: In **Sec. 3.1**, we explain the auto-annotation process. We use MiniGPT-v2 for Visual Objective Descriptions, Action Units (AUs) for Visual Expression Descriptions, and Qwen-Audio for Audio Tone Descriptions. By combining these multimodal descriptions with Lexical Subtitles, we generate coarse-grained descriptions. Then, LLaMA-3 refines these annotations to provide in-depth understanding of expressions and speech content, corresponding to internal states. Finally, we remove erroneous annotations and have four experts select annotations that align with human preferences.\n\n**Q8: Do the instructions for multimodal emotion recognition refer to different tasks?**\n\nA8: Yes, **Tab. 11** lists instructions for different tasks where the model outputs emotion category labels. Emotion-LLaMA integrates external cues (facial expressions, audio tones) and internal states to accurately determine and output the appropriate emotion category.\n\n**Q9: Training details are unclear.**\n\nA9: Please refer to our anonymous repository for training details. Further details about the tuning process can be found in our response to Reviewer *V8m9: Q1-A1*.\n\n**Q10: The authors employ ChatGPT to evaluate emotion reasoning. It is not clear to what extent this approach leads to a valid evaluation.**\n\nA10: Previous work [2, 3] shows ChatGPT can be used in emotion-related tasks. In our approach, ChatGPT evaluates the similarity between our model's outputs and the ground truth based on its reasoning capabilities. Tasks such as emotion reasoning [4] and open-vocabulary emotion recognition [5] benefit from ChatGPT’s reasoning skills. Similarly, in video understanding [6], ChatGPT’s reasoning is used for assessment. This mitigates circular evaluation by focusing on the model's ability to match outputs with the ground truth, not ChatGPT's direct emotional understanding. We will include this explanation in the revised manuscript.\n\n**Q11: The impact of using the proposed MERR dataset for pre-training needs to be evaluated.**\n\nA11: In **Tab. 6**, we compare Emotion-LLaMA's performance when trained on the MERR dataset versus other datasets. Due to the limited size of the MER2023 training dataset (3,373 samples), pre-training poses challenges for models with transformer structures, resulting in a low F1 score of 79.17%. Pre-training with larger pseudo-labeled datasets (73,148 and 36,490 samples) significantly improves performance. Tuning the model on our automatically annotated MERR dataset achieves the best performance, improving the F1 score by 11.19%, demonstrating the richness and quality of the MERR dataset's annotations. Detailed information about the MERR dataset is provided in Fig. 4 and Fig. 5, with comparisons to other datasets in **Tab. 10**.\n\nAdditionally, Emotion-LLaMA, trained on the MERR dataset, excelled in the recent [MER2024] competition. Detailed information about the competition and results can be found in our response to Reviewer *3ouG: Q5-A5*.\n\n[MER2024]: https://zeroqiaoba.github.io/MER2024-website/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U04JCvdSGR", "reviewer": "Reviewer_V8m9", "summary": "The paper introduces a multimodal large language model, named Emotion-LLaMA for emotional state understanding. The authors use open-source tools to collect and annotate a dataset, named MERR for model pre-training. Then they perform instruction-tuning on downstream datasets for emotion recognition and emotion reasoning. Extensive experiments are conducted and demonstrate the promising performance of the proposed approach.", "questions": "1. How do you pre-train Emotion-LLaMA? Is it supervised learning with coarse-grained emotion labels?\n\n2. Do you plan to release the MERR dataset?\n\n3. Table 2 fine-tuning, why does Emotion-LLaMA get worse performance on Disgust than MAE-DFER and VideoMAE?", "limitations": "Please refer to weaknesses", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive experiments are conducted and the method achieves SOTA performance on various dataset for emotion recognition and reasoning.\n2. Straightforward visualizations are presented.", "weaknesses": "1. The paper claims the MERR dataset as a core contribution. However, there is no systematic evaluation for the label quality of the dataset. I understand that pre-training on this dataset improves the downstream performance and thus its validity can be shown to some extent. However, this may be due to the diversity of the unlabeled data instead of the automatically generated label.\n\n2. Table 3, with audio and video inputs, Emotion-LLaMA’s performance is worse/close to the baseline (VAT).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful review. Your feedback has been valuable in improving our work. We've addressed each point below and will incorporate these enhancements in our revision.\n\n**Q1: How do you pre-train Emotion-LLaMA? Is it supervised learning with coarse-grained emotion labels?**\n \nA1: Yes, we use automatically generated labels and descriptions as answers for supervised learning. Instruction tuning of Emotion-LLaMA involves two main stages, utilizing both coarse-grained and fine-grained data from our MERR dataset. Here's a detailed breakdown of our training process:\n \n1. **Data Preparation:**\n   - Coarse-grained data: 28,618 samples\n   - Fine-grained data: 4,487 samples\n \n2. **Feature Extraction:**\n   - Audio features: Extracted using HuBERT-Chinese\n   - Visual features: Extracted using a combination of MAE, VideoMAE, and EVA2\n   - Text tokens: Processed directly by BERT’s Tokenizer\n \n3. **Prompt Construction:**\n   We construct prompts relevant to the training task, examples of which can be found in **Tables 11 and 12** of our paper. \n \n4. **Instruction Template:**\n   We use the following template to form our instructions:\n   ```\n   
 < AudioFeature > < VideoFeature > [Task Identifier] Prompt 
\n   ```\n   Where:\n   - `< AudioFeature >` and `< VideoFeature >` are the extracted features\n   - `[Task Identifier]` specifies whether it's a recognition or reasoning task\n   - `Prompt` is the constructed prompt for the specific task\n \n5. **Answer Preparation:**\n   For each instruction, we prepare corresponding answers:\n   - For recognition tasks: Emotion category labels\n   - For reasoning tasks: Detailed emotion descriptions\n \n6. **Training Procedure:**\n\n   a. **Stage 1 - Coarse-grained Pretraining:**\n   - Data: 28,618 coarse-grained samples\n   - Epochs: 30\n   - Batch size: 4\n   - Learning rate: 1e-5 with cosine decay\n \n   b. **Stage 2 - Fine-grained Tuning:**\n   - Data: 4,487 fine-grained samples\n   - Epochs: 10\n   - Batch size: 4\n   - Learning rate: 1e-6 with cosine decay\n \n7. **Implementation:**\n   The specific code (train.py, train_configs/Emotion-LLaMA_finetune.yaml) and more implementation details (README.md: Setup, Training) are available in our anonymous repository (https://anonymous.4open.science/r/Emotion-LLaMA). Detailed descriptions of the training process and feature extraction can be found in **Sec. 4.2** of our paper.\n\n**Q2: Do you plan to release the MERR dataset?**\n\nA2: Yes, the MERR dataset is available on GitHub as part of our submission. You can access and view the MERR dataset (README.md: MERR Dataset) in the anonymous repository mentioned in the paper.\n\n**Q3: The improvement might be due to the diversity of unlabeled data rather than the automatically generated labels.**\n\nA3: While the MER2023 dataset contains many unlabeled video samples, MLLMs require instruction datasets for training and cannot be directly trained with unlabeled samples. We built the MERR dataset with 28,618 coarse-grained and 4,487 fine-grained annotated samples with rich emotional descriptions. Instruction tuning based on these labels and descriptions significantly enhanced Emotion-LLaMA's emotional understanding capability. The improvement is due not only to the diversity of the unlabeled data but also to the quality of the automatically generated labels and descriptions.\n\n**Q4: Why does Emotion-LLaMA perform worse on 'Disgust' than MAE-DFER and VideoMAE?**\n\nA4: As shown in **Tab. 2**, almost all existing MLLMs perform poorly in recognizing 'disgust', often with zero accuracy. This is likely due to the scarcity of 'disgust' samples in current datasets. We also suspect LLMs may have safety restrictions related to 'disgust', contributing to the low accuracy. We plan to collect more samples to enrich the MERR dataset and further explore this issue to improve Emotion-LLaMA's ability to recognize new categories.\n\n**Q5: In Table 3, with audio and video inputs, Emotion-LLaMA’s performance is worse/close to the baseline (VAT).**\n\nA5: **Tab. 3** shows F1 Scores for MER2023-Baseline and VAT models. The MER2023-Baseline is the competition's baseline model, and VAT is the first-place winner. They found the text modality to be weaker in emotion prediction, so they focused on visual and auditory modalities. Their results showed that adding text for fusion lowered scores due to the lack of dialogue content and contextual background in the video samples.\n\nHowever, the textual modality is crucial in multimodal emotion recognition tasks. Emotion-LLaMA maps audio and visual features to the textual space, using them as contextual information. Even without the text modality, Emotion-LLaMA achieves scores close to the highest in the MER2023 competition, demonstrating robustness. More importantly, when the text modality is added, there is a significant improvement in the F1 Score, proving that Emotion-LLaMA can understand the emotional content in the subtitles.\n\nIf you have further questions or need additional clarification, please let us know. We value your feedback and are committed to providing thorough responses.\n\n[MER2024]: https://zeroqiaoba.github.io/MER2024-website/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G2ypiUs4Gv", "reviewer": "Reviewer_646t", "summary": "This paper implements multimodal emotion recognition and reasoning by fine-tuning the LLaMA model with instructions. It is trained on a large-scale dataset, fine-tuned, and tested on three datasets.", "questions": "1. The complex interaction relationship between modalities is not considered.\n\n2. The process is complicated, and the computing resource requirements are high.\n\n3. The effect of Prompt Tuning and Instruction Tuning is highly dependent on the design of prompt words and instructions. If the prompt words or instructions are not accurate or comprehensive enough, it may impact the performance of the model.\n\n4. The lack of reproducibility: the anonymous Github link provided by the author is empty.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The global, temporal and local features of the video modality are considered, and LoRA fine-tuning (all tokens), Prompt fine-tuning (text modality) and supervised fine-tuning (linear layer) are used at the same time.", "weaknesses": "The innovation is limited and only existing methods are used. The key problems are not solved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We've addressed your points carefully and will incorporate these clarifications in our revision.\n \n**Q1: The complex interaction relationship between modalities is not considered.**\n\nWe appreciate your attention to the interaction between modalities. However, we respectfully suggest that the premise of this question may not fully align with our approach and findings:\n\n1. Early works in multimodal large language models (MLLMs) often used the Q-Former proposed in BLIP2, involved complex two-stage pre-training, which could lead to information loss. \n\n2. Direct feature mapping: Emotion-LLaMA directly maps visual and audio features into the textual semantic space. This approach preserves unique characteristics from each modality (e.g., facial micro-expressions, vocal tones) while allowing for their interaction in the unified semantic space.\n\n3. Experimental validation: Our experiments in the MER2024 competition demonstrate the effectiveness of this approach:\n\n   | Modality Combination | Pre-alignment F1 | Post-alignment F1 |\n   |----------------------|------------------|-------------------|\n   | Audio only (HuBERT)  | 0.7277           | 66.18             |\n   | Video only (CLIP)    | 0.6673           | 69.07             |\n   | Text only (Baichuan) | 0.5429           | 56.85             |\n   | All modalities       | 80.91            | 73.01             |\n\nThese results suggest that complex alignment methods may benefit weaker modalities but can also lead to information loss (a 7.9% decrease in fusion scores post-alignment).\n\n**Q2: The process is complicated, and the computing resource requirements are high.**\n \nWe appreciate your concern about computational efficiency. However, we believe our method is relatively resource-efficient:\n \n1. Parameter-efficient tuning: We use LoRA, resulting in only 34 million trainable parameters (0.495% of the total).\n2. Hardware requirements: All work was completed using only 4 A100 (40G) GPUs, which is modest compared to many LLM pre-training efforts.\n3. Inference efficiency: Emotion-LLaMA requires only a single A10 or A100 GPU for rapid inference.\n\n**Q3: Dependence on Prompt and Instruction Design**\n \nA3: Designing effective prompts and instructions is indeed crucial for harnessing the powerful reasoning abilities of LLMs. In this work, we have made significant contributions by designing various training tasks and different instructions for Emotion-LLaMA to enhance its robustness and generalization capabilities. We continuously refine our prompts and instructions based on iterative testing and validation to ensure they are accurate and comprehensive. The demo in the anonymous repository showcases Emotion-LLaMA's excellent performance, demonstrating that it can provide correct answers regardless of whether the instructions are previously learned or new, highlighting the robustness and innovation of our design.\n \n**Q4: The lack of reproducibility: the anonymous GitHub link provided by the author is empty.**\n \nA4: We apologize for the inconvenience you experienced. This issue may have been due to a temporary network problem. We have verified that the anonymous repository is accessible and contains all necessary files for reproducibility. Our Emotion-LLaMA has been successfully reproduced by other researchers, who have praised its performance in the demo. We encourage you to access the repository again to review the open-source MERR dataset, the training process, and the code for Emotion-LLaMA. We are committed to ensuring that all materials are available and easily accessible for reproducibility.\n \n**Q5: The innovation is limited, and only existing methods are used. The key problems are not solved.**\n \nA5: We respectfully disagree with this assessment. Our work addresses critical challenges in multimodal emotion recognition and reasoning:\n \n1. **Data scarcity**: Existing datasets predominantly consist of image-text pairs, lacking dynamic expression descriptions and audio components. Manual annotation of real-world multimodal samples is prohibitively expensive. To address this, we introduced the MERR dataset, comprising 28,618 coarse-grained and 4,487 fine-grained automatically annotated samples across diverse emotion categories. This dataset significantly advances the field by providing rich, multimodal emotion-related instruction-following data.\n \n2. **Real-world applicability**: Existing MLLMs struggle with accurate emotion understanding in real-world scenarios. Emotion-LLaMA, instructed-tuned on the MERR dataset, has demonstrated excellent performance in both controlled and real-world conditions. Our first-place win in the MER2024 competition's MER-Noise track underscores its robustness in noisy, real-world environments.\n \n3. **Generalization**: In the MER-OV track of the [MER2024 competition][1], Emotion-LLaMA significantly outperformed other MLLMs, including GPT-4V, by improving average accuracy and recall by 8.52%. This showcases its superior generalization capabilities across various emotion understanding tasks.\n \n4. **Comprehensive evaluation**: Extensive experimental results demonstrate Emotion-LLaMA's outstanding multimodal emotion understanding capabilities across multiple benchmarks and real-world scenarios.\n \n5. **Reproducibility and accessibility**: We have open-sourced both the MERR dataset and the code for Emotion-LLaMA, and provided an online demo in our [anonymous repository][2]. This facilitates further research and practical applications in the field of multimodal emotion recognition and reasoning.\n \nWhile we acknowledge that there is always room for improvement, we believe these contributions offer valuable insights into addressing challenges in multimodal emotion understanding. We welcome further discussion on how we can enhance our approach to better solve key problems in the field.\n\n[1]: https://zeroqiaoba.github.io/MER2024-website/\n[2]: https://anonymous.4open.science/r/Emotion-LLaMA/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the thoughtful feedback and constructive criticism from all reviewers. Your insights have been instrumental in refining our work. Below, we summarize the key changes and improvements we have made in response to your comments.

### Key Changes and Improvements

1. **Clarification of Dataset and Methodology**:
    - We have provided a detailed explanation of the MERR dataset, including its sources, annotation process, and the unique features that make it suitable for multimodal emotion recognition and instruction tuning.
    - We clarified how the dataset is auto-annotated with emotion labels and refined with expert input, ensuring high-quality annotations.

2. **Experimental Details**:
    - We added comprehensive details about our pre-training and fine-tuning processes, including specific datasets, sample sizes, feature extraction methods, prompt construction, and instruction templates.
    - We included additional ablation studies and experimental results, particularly focusing on the audio modality and its impact on performance.

3. **Model Evaluation**:
    - We elaborated on the evaluation metrics and provided a thorough analysis of the results, including the rationale behind using ChatGPT for emotion reasoning evaluation and how it mitigates circular evaluation issues.
    - We compared Emotion-LLaMA’s performance with other state-of-the-art models across multiple benchmarks and real-world scenarios.

4. **Addressing Limitations**:
    - We explicitly discussed the limitations of our work, including potential biases, data privacy, and ethical considerations. We have also outlined the steps taken to address these issues.
    - We acknowledged the challenges related to the 'disgust' emotion category and our plans to enhance the MERR dataset with more diverse samples.

5. **Resource Efficiency and Innovation**:
    - We addressed concerns about computational efficiency, highlighting our use of parameter-efficient tuning methods and modest hardware requirements.
    - We emphasized the innovative aspects of our approach, including the design of effective prompts and instructions that enhance the robustness and generalization capabilities of Emotion-LLaMA.

6. **Reproducibility and Accessibility**:
    - We ensured that our anonymous repository is accessible and contains all necessary files for reproducibility, including the MERR dataset, training process, and code for Emotion-LLaMA.

### References and Links

[1] MERBench: A unified evaluation benchmark for multimodal emotion recognition, arXiv 2024.  
[2] The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection, Affective Computing 2022.  
[3] GPT-4V with emotion: A zero-shot benchmark for generalized emotion recognition, Information Fusion 2024.  
[4] Explainable multimodal emotion reasoning, arXiv 2023.  
[5] MER 2024: Semi-Supervised Learning, Noise Robustness, and Open-Vocabulary Multimodal Emotion Recognition, arXiv 2024.  
[6] TempCompass: Do Video LLMs Really Understand Videos?  ACL 2024.  

**Repository Links:**
- [MER2024 Competition Website](https://zeroqiaoba.github.io/MER2024-website/)
- [Anonymous Repository for Emotion-LLaMA](https://anonymous.4open.science/r/Emotion-LLaMA/)

We hope these changes address your concerns and effectively demonstrate the robustness and novelty of our work. Thank you once again for your valuable feedback and support.

Meta-review: Before the rebuttal phase, the paper received ratings of 2 borderline reject, 1 reject, and 1 borderline accept. After the rebuttal phase, one of the reviewers raised their rating to borderline accept, resulting in 2 borderline accepts, 1 borderline reject, and one reject. The authors gave a significant rebuttal answering many questions about their work. Reviewer engagement was less noticeable during the rebuttal phase, with lack of engagement with the authors. Many of the questions and concerns from the reviewers were answered by the authors, or they were originally available in the paper, or supplemental material. When available in the paper, the authors pointed this out in their rebuttal, including relevant figures and/or tables. After going through and reading the paper, these details were available as noted by the authors.

After reading through the paper, the reviews, and the author rebuttal, many of the reviewer concerns were addressed adequately and this is an interesting paper including the new dataset and the proposed approach. The topic of multimodal emotion recognition is timely and relevant. The results are encouraging showing the efficacy of the proposed approach. The dataset is interesting, and is a good contribution compared to current state-of-the-art emotion-based datasets (e.g., labels and descriptions extend state of the art). 

Although many of the reviewer concerns were addressed in the rebuttal, multiple reviewers note some lack of details such as on training in Section 3.3. I agree that the paper lacks these details. The paper, and the authors in their rebuttal, state those details are in the code repository. After going through the repository, I agree that there are training details available there, however, I don't agree that this is sufficient for the paper or as a response to the reviewers. More details regarding training should be included in the paper, and in their response to the reviewers.

Currently, after the author's response to the reviewers, the strengths outweigh the weaknesses. This paper can be accepted.
Decision: Accept (poster)