Title: Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Instruction Tuning

Multi-modal Emotion Recognition

Multi-modal Emotion Reasoning
Abstract: Accurate emotion perception is crucial for various applications, including human-computer interaction, education, and counseling.
However, traditional single-modality approaches often fail to capture the complexity of real-world emotional expressions, which are inherently multimodal. Moreover, existing Multimodal Large Language Models (MLLMs) face challenges in integrating audio and recognizing subtle facial micro-expressions. To address this, we introduce the MERR dataset, containing 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories. This dataset enables models to learn from varied scenarios and generalize to real-world applications. Furthermore, we propose Emotion-LLaMA, a model that seamlessly integrates audio, visual, and textual inputs through emotion-specific encoders. By aligning features into a shared space and employing a modified LLaMA model with instruction tuning, Emotion-LLaMA significantly enhances both emotional recognition and reasoning capabilities. Extensive evaluations show Emotion-LLaMA outperforms other MLLMs, achieving top scores in Clue Overlap (7.83) and Label Overlap (6.25) on EMER, an F1 score of 0.9036 on MER2023-SEMI challenge, and the highest UAR (45.59) and WAR (59.37) in zero-shot evaluations on DFEW dataset.


Peer reviews:
comments: SUMMARY:
This paper implements multimodal emotion recognition and reasoning by fine-tuning the LLaMA model with instructions. It is trained on a large-scale dataset, fine-tuned, and tested on three datasets.

STRENGTHS:
The global, temporal and local features of the video modality are considered, and LoRA fine-tuning (all tokens), Prompt fine-tuning (text modality) and supervised fine-tuning (linear layer) are used at the same time.

WEAKNESSES:
The innovation is limited and only existing methods are used. The key problems are not solved.

QUESTIONS:
1. The complex interaction relationship between modalities is not considered.

2. The process is complicated, and the computing resource requirements are high.

3. The effect of Prompt Tuning and Instruction Tuning is highly dependent on the design of prompt words and instructions. If the prompt words or instructions are not accurate or comprehensive enough, it may impact the performance of the model.

4. The lack of reproducibility: the anonymous Github link provided by the author is empty.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-22

comments: SUMMARY:
The paper introduces a multimodal large language model, named Emotion-LLaMA for emotional state understanding. The authors use open-source tools to collect and annotate a dataset, named MERR for model pre-training. Then they perform instruction-tuning on downstream datasets for emotion recognition and emotion reasoning. Extensive experiments are conducted and demonstrate the promising performance of the proposed approach.

STRENGTHS:
1. Extensive experiments are conducted and the method achieves SOTA performance on various dataset for emotion recognition and reasoning.
2. Straightforward visualizations are presented.

WEAKNESSES:
1. The paper claims the MERR dataset as a core contribution. However, there is no systematic evaluation for the label quality of the dataset. I understand that pre-training on this dataset improves the downstream performance and thus its validity can be shown to some extent. However, this may be due to the diversity of the unlabeled data instead of the automatically generated label.

2. Table 3, with audio and video inputs, Emotion-LLaMA’s performance is worse/close to the baseline (VAT).

QUESTIONS:
1. How do you pre-train Emotion-LLaMA? Is it supervised learning with coarse-grained emotion labels?

2. Do you plan to release the MERR dataset?

3. Table 2 fine-tuning, why does Emotion-LLaMA get worse performance on Disgust than MAE-DFER and VideoMAE?

LIMITATIONS:
Please refer to weaknesses
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-10

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new multi-modal instruction tuning dataset for emotion recognition. The authors also present results of training on this dataset with a multi-modal architecture based on LLaMa 2. They show evaluation results on DFEW and MER2023.\n\nSTRENGTHS:\nEvaluating multi-modal emotion recognition approaches based on LLMs are a very relevant research topic.\nThe results shown by the authors are convincing w.r.t. the performance on emotion recognition datasets.\n\nWEAKNESSES:\n-------------\n\nThe argument presented in lines 31 and 33 is not convincing. Why does an inability in methods lead to a lack of datasets?\n\n\nRelated Work:\n-------------\n\nAs the dataset is one of the claimed contributions of the paper, there should be a discussion on previous emotion recognition datasets. It is important to lay out the reasons why these previous datasets cannot be used (or not easily be used) for instruction tuning of LLMs.\n\n\nMethodology:\n------------\n\nWhat videos is the MERR dataset based on? I could not find details in the paper on the selection process. Judging from the screenshots, it seems to be movie clips. This has important implications on the concept of \"emotion\" that is addressed in the paper and needs to be clarified.\n\nThe chosen methodology lacks justification.\nFor example, what is the reasoning behind selecting the frames with the highest sum of AU activations across all AUs included in OpenFace? It seems to me that there would be the danger of having a strong bias towards moments when the person is speaking, as this often leads to high AU activations, especially AUs related to the lower half of the face. \n\nThe mapping of AUs to facial expression labels needs more explanation. E.g. is \"happy\" assigned if the combination of AU06, AU12, AU14 is active, or is it also assigned if only some of those AUs are active? Figure 1 does not show a facial expression description according to Table 7 at all, only descriptions according to Table 8. Concerning Table 8, there are several descriptions given for each AU. Are they all used at the same time, or is only one of them chosen? In Figure 1 it appears that only one of them is chosen, but how is this decided?\n\nSeveral further steps are not clearly defined, e.g. how does LLaMA-3 \"refine\" the annotations by aggregating the multimodal description? How is the instruction following data constructed of which a single example is presented in Table 9. Is it done manually? If yes, how and how many samples are created?\n\nIn 3.1 it is not clear, how the dataset is \"auto-annotated\" with emotion labels. It is also not clear how these annotations are refined (with human involvement)? In general the concept of \"emotion\" used in the paper remains unclear. Is it about emotional displays, about internal states,...?\n\nIn later parts of the method section it seems authors are targeting (internal?) emotional states. It would be important to know how they were annotated.\n\nDo the instructions for multimodal emotion recognition (Table 11) refer to different tasks? Some of these instructions seem to target displayed emotions, some target internal states.\n\n\nTraining details are unclear. With the description provided in the paper it is difficult to understand the training procedure.\n\n\n\nEvaluation:\n-----------\n\nThe authors employ chatGPT to evaluate emotion reasoning. It is not clear to what extent this approach leads to a valid evaluation. E.g. ChatGPT was shown to be biased on emotion-related tasks [1].\nIn the end, evaluating the emotion understanding capabilities of one language model using another language model is circular.\n\nThe impact of using the proposed MERR dataset for pre-training needs to be evaluated.\n\n\n[1] R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Transactions on Affective Computing, 2022.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\nLimitations are not discussed in enough detail. There is no separate limitations section. The authors also do not lay out the limitations concerning e.g. (citing from the checklist):\n\n\"The authors should reflect on the scope of the claims made, e.g., if the approach was580\nonly tested on a few datasets or with a few runs. In general, empirical results often581\ndepend on implicit assumptions, which should be articulated.\"\n\n\"The authors should reflect on the factors that influence the performance of the approach.583\nFor example, a facial recognition algorithm may perform poorly when image resolution584\nis low or images are taken in low lighting. Or a speech-to-text system might not be585\nused reliably to provide closed captions for online lectures because it fails to handle586\ntechnical jargon.\"", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents the Emotion-LLaMA model, a multimodal emotion recognition and reasoning system that integrates audio, visual, and textual inputs.\n\n- The authors constructed the MERR dataset, which includes 28,618 coarse-grained and 4,487 fine-grained annotated samples across diverse emotional categories, enabling models to learn from varied scenarios and generalize to real-world applications.\n- The Emotion-LLaMA model incorporates specialized encoders for audio, visual, and textual inputs, aligning the features into a modified LLaMA language model and employing instruction tuning to enhance both emotional recognition and reasoning capabilities.\n- Extensive evaluations show that Emotion-LLaMA outperforms other multimodal large language models, achieving top scores on the EMER, MER2023, and DFEW datasets.\n\nThe main contributions are:\n- The MERR dataset, a valuable resource for advancing large-scale multimodal emotion model training and evaluation.\n- The Emotion-LLaMA model, which excels in multimodal emotion recognition and reasoning through the innovative use of instruction tuning.\n- Establishing Emotion-LLaMA as the current state-of-the-art model in public competitions for multimodal emotion analysis.\n\nSTRENGTHS:\n- The paper is well structured and well presented. The author organizes the article into five sections: introduction, related work, methodology, experiments, and conclusion. They clearly describe how they conduct data annotation and model design, provide a good introduction to the experimental setup and analysis of the experimental results, and present a relatively clear conclusion.\n- Model details are thorough. The authors have provided a good description of their model and training methods, allowing me to clearly understand how the model is designed and trained. I believe their results are reproducible.\n- This work is valuable . The authors provided a paradigm for emotion annotation of multimodal data and an annotated dataset. They also offered a clear explanation of the annotation process, which will contribute to the development of the related field.\n\nWEAKNESSES:\n- Insufficient experiments: Although the author has conducted some comparative and ablation experiments, it is obviously insufficient for such a complex multimodal LLM.\n- Missing details of experimental setup: The authors mentioned that they fine-tuned on several target datasets, but the details of the fine-tuning (including data volume, dataset division, and fine-tuning setup) were not included in the article. This can lead to a decrease in the credibility of their results. \n- Lack of result analysis: Although the proposed model surpasses existing models in many metrics, the authors only list their experimental results in the results section without further analysis and explanation of the results and some phenomena. The lack of proper explanation and analysis can make some results confusing.\n\nQUESTIONS:\n1. In section 4.2, the author mentioned that they used the HuBERT-Chinese large model for audio modality input processing. Regarding this part, I would like to know the following questions:\n\n    -  Are the experimental results sensitive to language? I hope the author can provide more experimental results to illustrate this point.\n\n    - In the ablation study section (Tab5), the author seems to have only conducted ablation on the visual encoder. Why didn't they further conduct ablation experiments on the audio encoder, considering that there are many alternatives to Hubert?\n\n    - Based on the previous question, does the author believe that the audio modality is not important in this task? I would like to see more ablation results on the modality scale.\n2. As I mentioned in the weaknesses part, could the author provide more details of fine-tuning on the target datasets?\n3. Is Multimodal Emotion Recognition a classification task? How do the authors explain the Dis column in Table 2?\n\nLIMITATIONS:\nAlthough the author mentioned in the checklist that they discussed the limitations of the article, they did not explicitly discuss them in the text. \nMeanwhile, I noticed that the author did not mention their data sources in the article, and I am concerned whether this might involve data copyright issues.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}