Open-vocabulary Multimodal Emotion Recognition: Dataset, Metric, and Benchmark

Authors: Zheng Lian, Haiyang Sun, Licai Sun, Lan Chen, Haoyu Chen, Hao Gu, ZHUOFAN WEN, Shun Chen, Zhang Siyuan, Hailiang Yao, Mingyu Xu, Kang Chen, Bin Liu, Rui Liu, Shan Liang, Ya Li, Jiangyan Yi, Jianhua Tao
Primary area: datasets and benchmarks
Keywords: Open-vocabulary Multimodal Emotion Recognition, Dataset, Metric, Benchmark

Abstract:
Multimodal Emotion Recognition (MER) is an important research topic. This paper advocates for a transformative paradigm in MER. The rationale behind our work is that current approaches often rely on a limited set of basic emotion labels, which do not adequately represent the rich spectrum of human emotions. These traditional and overly simplistic emotion categories fail to capture the inherent complexity and subtlety of human emotional experiences, leading to limited generalizability and practicality. Therefore, we propose a new MER paradigm called Open-vocabulary MER (OV-MER), which encompasses a broader range of emotion labels to reflect the richness of human emotions. This paradigm relaxes the label space, allowing for the prediction of arbitrary numbers and categories of emotions. To support this transition, we provide a comprehensive solution that includes a newly constructed database based on LLM and human collaborative annotations, along with corresponding metrics and a series of benchmarks. We hope this work advances emotion recognition from basic emotions to more nuanced emotions, contributing to the development of emotional AI.

Decision: reject
Reviewer scores: 8, 5, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1729196499264

thread_type:
official_review

<thread_object>
creator: reviewer_gtfb

<content>
summary:
The paper introduces a new task in the realm of MER, termed Open-vocabulary MER (OV-MER), which contains a broader and more diverse range of emotion labels. The authors propose an LLM-based framework to generate labels for OV-MER, along with corresponding metrics and benchmarks for evaluation.

strengths:
1. Extensive experiments with plenty of SOTA LLM methods are conducted.
2. The visualizations are clear and straightforward, making the content easy to follow.
3. The idea of extending MER to OV-MER is novel and adds a fresh perspective to the field.

weaknesses:
1. The dataset scale is not mentioned or compared in Sec 2 or Tab 1.
2. There is no quality evaluation for the generated labels (e.g., comparison with manual annotations), and their validity has not been assessed (e.g., by using the dataset for pre-training or other downstream tasks). Although there are comparisons with humans regarding clue length, label count, and word cloud, the label accuracy is never evaluated or discussed. This raises concerns about the reliability and overall usefulness of the dataset for the study.
3. The setup in Table 2 and 3 presents a data leakage issue. The ground truth label is generated using SALMONN and GPT-4V, but both of these methods are also being evaluated.

I am willing to raise my score if the authors could address my second concern.

questions:
1. Sec 2.1, pre-annotation, what is the average duration of the videos? I'm concerned that using only three frames as input may miss a significant amount of visual info, especially since the paper later notes that there are errors, duplicates, and missing info related to visual clues.
2. Sec 2.1, manual check, could you please clarify why ALLM struggles to capture emotion-related acoustic features? From my understanding, paralinguistic information in the audio modality contains meaningful emotion-related features, which should be useful in this context.
3. Table 1, CMU-MOSEI has discrete emotion labels.
4. In Sec 3.1, the GPT-based grouping is described as costly, but if the entire process only costs $1, it doesn't seem to be a significant expense. Could you clarify the reason?
5. Fig 4 (a), typo in video branch: ALLM -> VLLM.
6. Sec 4.1, CLUE-A, could you explain why text is used to generate acoustic clues when the audio already contains the necessary content information? It seems redundant.
7. It is not clear what fusion method you use in Tab 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729487746494

thread_type:
official_review

<thread_object>
creator: reviewer_sjue

<content>
summary:
This paper introduces a novel paradigm in multimodal emotion recognition called Open-vocabulary Multimodal Emotion Recognition (OV-MER), which aims to address the limitations of traditional methods that rely on a fixed set of emotion labels. The OV-MER aims to better capture the subtle variations in human emotions by allowing predictions across a broader range of emotion categories. To support this method, the authors developed a new dataset named OV-MERD, using a collaborative annotation strategy between humans and large language models (LLMs) to generate detailed emotion-related descriptions (CLUE), which enrich the variety and depth of emotion labels. The paper defines new evaluation metrics based on emotional similarity grouping and conducts extensive experiments with various baseline models to highlight the challenges and limitations of this task. The study also emphasizes the advantages of human-LLM collaboration in generating more comprehensive emotion descriptions, as opposed to human-only annotations.

strengths:
1. The paper introduces a novel paradigm, which extends beyond traditional methods that rely on a fixed set of emotion labels. 
2. The development of the OV-MERD dataset, which includes a wide range of emotion categories, is a significant contribution. Its rich diversity allows models to learn from a broader spectrum of emotional expressions, which may be used to improve the generalizability of emotion recognition systems.
3. The paper conducts comprehensive experiments and benchmarks, demonstrating the effectiveness of human-LLM collaboration-based  CULE-Multi.

weaknesses:
1. The baseline results rely on CLUE-Multi/A/V/T descriptions that include human checks. In real-world scenarios, the human intervention is impractical and the actual performance may be lower.
2. The dataset’s cartoonish style could hinder the model’s ability to generalize to real-world data, potentially limiting its effectiveness in practical applications.
3. There is a lack of information on the total number of samples and the train/validation/test split.

questions:
1. The models in the paper are trained using CLUE-Multi/A/V/T data that includes manual checks, which ensures high-quality and detailed emotion descriptions. However, in real-world applications, it is likely that only LLMs-generated CLUEs without any human oversight will be available. Would this discrepancy lead to significant performance degradation in models trained using CLUE-Multi/A/V/T data?
2. Would the cartoonish style of the dataset cause a significant domain shift problem? Specifically, how would a model pre-trained on this dataset perform when tested or fine-tuned on other datasets like DFEW and IEMOCAP? Do you consider any adaptation strategies to mitigate potential domain shifts?
3. What is the sample size of the dataset, and how are the train/validation/test sets divided?
4. The paper mentioned, "This is because, in MER, text is emotionally ambiguous; without integrating other modalities, we cannot accurately understand the emotions conveyed through text." However, as far as I know, there are datasets like MOSI and MOSEI where the text/language modality contributes more than other modalities, as evidenced by relevant experimental results [1-4]. Is this explanation of yours appropriate, or is it specific to this particular dataset?

Reference

[1] Hazarika, Devamanyu, Roger Zimmermann, and Soujanya Poria. "Misa: Modality-invariant and-specific representations for multimodal sentiment analysis." in ACM MM 20.

[2] Yu, Wenmeng, et al. "Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis." in AAAI 2021.

[3] Zhang, Haoyu, et al. "Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis." in EMNLP 2023.

[4] Feng, Xinyu, et al. "Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis." in EMNLP 2024.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730142133452

thread_type:
official_review

<thread_object>
creator: reviewer_tzua

<content>
summary:
Traditional emotion recognition tasks typically focus on predicting a single label for a given data point, such as "angry," "happy," or "surprised." This work argues that more nuanced emotions cannot be captured by this limited set of labels. Therefore, it proposes an open-vocabulary approach (OV-MER), where the number of categories is greatly expanded, thus facilitating the development of emotion AI. Overall, the main contributions include the development of datasets, new metrics, and a new task.

strengths:
1. The authors have effectively expanded upon a previous dataset by implementing thoughtfully designed metrics grounded in fundamental emotional theories.

2. The problem formulation is clear, and the motivation is intuitive.

3. The experiments are well-conducted and organized.

4. Significant effort and comprehensive content have been included to ensure reproducibility.

weaknesses:
1. While it's true that nuanced emotional labels can't always be conveyed by a single adjective, like "surprised," as it may stem from either positive or negative experiences, the VAD (valence, arousal, and dominance) framework can help mitigate this issue. This is why emotion recognition tasks are generally categorized as either discrete label-based or dimensional-based. By using VAD scores, we can better distinguish whether "surprise" is elicited by positive or negative emotions. Discrepancies do exist within each subject's understanding and annotators's agreement. However, it is not clear whether the issue was addressed by the proposed approach. In Appendix Section G, the authors state, 'Specifically, in the first round, we randomly select four annotators to check the clues and labels; in the second round, we merge the clues and labels reviewed by the first four annotators and ask another four annotators to perform a second round of checks.' How exactly are they aligned, and what has been removed/kept?

2. Section 2, which focuses on the construction of the 'OV-MERD dataset', is one of the most significant contributions of this work. However, the authors first discuss feature extractions and annotations, leaving readers unclear about the original data source until Section 2.3. This structure is not well-designed; as a reader, it became frustrating to navigate from the first page to Figure 2 and then through Sections 2.1 and 2.2 without understanding the origins (where and what) of the original data until Section 2.3 on page 4. This makes the paper difficult to follow.

3. In Section 2.3, the phrase "evenly select samples with further annotation" is vague. Does this refer to the original dataset (MER2023), which (I can only assume) has an even number of samples under each category? Or is it discussing the newly constructed dataset? I interpret it as referring to the original dataset based on Section E in the Appendix, but this should be stated more clearly.

4. Another concern is the grouping strategy. The authors utilized LLM and Putchik's emotional wheels to group several emotions into single labels. However, the order of discrete emotions in each group—such as "surprise," "nervous," and "dissatisfied"—is crucial, as different arrangements can lead to varied interpretations of the data points. This perspective is not addressed or mentioned anywhere in the paper.

5. Additionally, what are the visual and acoustic cues mentioned? Are there any low-level details? Or are they just text output from LLM?

6. A key issue in emotion recognition and affective computing is the cultural differences that influence emotional interpretation. People from different linguistic and cultural backgrounds can interpret and express emotions, like anger, in distinct ways. Although the authors primarily used Chinese and English in this work, they have not discussed these potential differences, especially considering they propose new metrics and benchmarks. Further background on the 'expert annotators' is not detailed enough.

7. The discussion regarding "Limitations" is quite vague. What does "more effective" model mean? How do the authors intend to "broaden the scope of your evaluations"?

Minor Issues:
Does 'MLLM' first mentioned in the third contribution stand for Multimodal LLM? It needs to be specified.

The dataset comparison can be more comprehensive by including more recently released datasets in 2023 and 2024.

Overall, I appreciate the authors' efforts in constructing a new dataset and introducing new perspectives and metrics to this task. However, substantial changes are needed to improve the quality of the work.

questions:
Please see the 'weaknesses.'
</content>
</thread_object>

</thread>
<thread>
creation date:
1730403220861

thread_type:
official_review

<thread_object>
creator: reviewer_5yry

<content>
summary:
This work focuses on introducing a new perspective on emotion annotation by not restricting it to classic categories and taking advantage of large language models in collaboration with humans for a more nuanced translation into labels. The proposed method increases the label space of emotion labels with a more detailed multi-label perspective, which can promote efficiency and understanding of emotions by systems. Furthermore, using an LLM over the output from multimodal LLMs can promote a detailed, closer-to-machine understanding of these labels. By grouping labels and defining metrics for sound evaluation, the authors provide a deeper view of label similarity to assess the system’s true understanding of emotion.

strengths:
The work presents an improvement over conventional emotion labeling by proposing a richer annotation which can enhance machine’s understanding of emotion. Additionally, the study focuses on the ambiguities and overlaps in emotional labels—a vital contribution to the field. Leveraging LLMs along human experts bridges the gap between human intuition and machine learning, making the model's emotional comprehension more transferable and adaptable.

weaknesses:
dependency of this approach to a certain versions of large language model definitely has its own limitations. From known weaknesses of this models to potential future risks of such approach. Furthermore, attempts to control this is quite expensive and risky. The bigger concern that comes to my mind is the effect of mirrors reflection in learning. How can we be sure that from this circle of learning, a limited and biased understanding is not created for machine learning?
A broader concern is the issue of model feedback loops—how can we be confident that machine learning isn't developing a narrow, biased interpretation of emotions from these circular, LLM-driven annotations? This feedback cycle may compromise the depth of emotional understanding in machine learning.

questions:
What is meant by the "short duration of video"? How can you be sure that a few frames (3) can thoroughly describe and capture facial motions indicating emotions? As you know, previous work in the field highlights the importance of facial movement features in video inputs, and as humans, we can imagine how fast movements can show excitement, which may not necessarily be captured by three frames. Furthermore, have you experimented to see if increasing the frame count could improve levels of hallucination in VLLM? Could the information gap be pushing the model?

As you mentioned, the role of VLLM control annotators includes adding missing content. How did the human annotators who evaluated the VLLM output capture visual data? Did they view the entire video or the same inputs as the VLLM?

What is the reason behind the difference in performance between matching-based and LLM-based metrics? I would prefer a more analytical study rather than just reporting results. I also suggest a deeper study on the effectiveness of multimodal fusion, as this title suggests an important question that has not been investigated as deeply as it could be.

Did you consider human evaluation of the combined output of ALLM and VLLM? How capable do you think this combination is of handling ambiguous and contradictory cases? Furthermore, how can we ensure that our fusion is unbiased?

I see that no annotation control has been used for the merging step. Do you have a control measure for this step? How can we be sure the merge is effective enough?

Regarding the language test, I appreciate that you included it, but one question that comes to mind is the purpose of comparing the similarity of the label sets from Yce to Yee. I believe these sets are not from the same data, as the former comes from data originally in Chinese (translated to English) and the latter from data in English. Given this, I understand the importance of comparing Yec to Yee, which can ensure that the model is not biased by language in the same data, but not Yce to Yee. In any case, I believe you could include more details on language.

In a comparison of human-only and human-LLM collaboration in clue generation, I couldn't understand why you relate the longer length of clues in human-LLM collaboration to informativeness regarding minor details. I don't question the assumption, as I see this fact more clearly in the comparison of label counts; perhaps elaborate more on that.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730568410673

thread_type:
official_review

<thread_object>
creator: reviewer_xyya

<content>
summary:
This paper centers on the task of multimodal emotion recognition. Unlike prior work, which primarily assigns a single label and fixes the label space, this study expands upon earlier tasks, enabling the prediction of multiple labels across various categories. This approach allows for a more nuanced representation of a person’s emotional state. In addition to establishing this new task, the paper also presents a newly constructed dataset, defines evaluation metrics and baseline systems, and undertakes foundational work within this area. Furthermore, the experimental sections offer insights and guidelines for advancing solutions to this task in future research.

strengths:
This paper introduces a novel task, OV-MER, which extends traditional multimodal emotion recognition (MER) to more comprehensively capture individuals' emotional states. The authors have conducted foundational research in this area by constructing a dataset, defining evaluation metrics, and proposing initial solutions. Overall, the paper is well-structured, and this advancement may enhance the application of MER in downstream tasks such as human-computer interaction. Additionally, the paper includes discussions on ethics and reproducibility.

weaknesses:
1. The color scheme in Figure 5 does not align with the main visual style of the paper. It is recommended to redraw this figure for consistency.
2. In Section 6, please provide additional elaboration on the recommendations for framework design, as this could be valuable for future researchers.
3. Traditional facial emotion recognition methods often utilize AU units and assign a single label per face. However, this paper’s OV-MER assigns multiple labels to a video. Could you clarify why videos in multimodal scenarios are suited to convey multiple emotions?
4. Table 1 includes only a limited selection of emotion datasets. Please consider expanding it to include additional datasets, such as MSP-Podcast.
5. In Table 2, please further clarify the distinctions between the labels extracted from CLUE-Multi and the final ground truth.

questions:
Please refer to my comments on weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732100889541

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
Dear Reviewers, Area Chairs, and Program Chairs:

We would like to express our gratitude to all the reviewers for taking their valuable time to review our paper. We sincerely appreciate the reviewers' recognition of the importance of our proposed task, the novelty of our method, and the richness of our experiments. Meanwhile, we hope the reviewers can value the originality of this work. While there are some minor issues with the expression, we have made every effort to clarify them. We believe this work is an important extension of the traditional MER task and will contribute to the development of this field.

At the same time, we appreciate the reviewers for pointing out the shortcomings. Your valuable comments help us improve this paper. We kindly ask the reviewers to take the above clarifications into account when considering score adjustments. We welcome any further discussion with the reviewers.

Best regards,

Paper5870 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732662400850

thread_type:
official_comment

<thread_object>
creator: area_chair_ikpq

<content>
title:
Questions from the AC

comment:
Dear Authors,

I have been following your paper and the discussions with interest, and would like to ask a few follow-up questions.

1. How does the proposed framework/paradigm, namely “open-vocabulary,” differ from established concepts such as “multi-label learning” and “partial label learning”? Both of these areas have been explored extensively, including in affective computing and other domains of human-machine systems/computing.

2. While the label space is more relaxed in your framework, it remains finite. This raises the question of whether the term “open” is accurate. Could you clarify your perspective on this?

3. Additionally, the claim that this paradigm “broadens” the “range of emotion labels” seems debatable. From my understanding, this approach primarily enables capturing simultaneous emotions rather than introducing a fundamentally broader range or type of emotions. Could you elaborate on this point?

4. Finally, are any of the baselines explicitly designed for multi-label learning? Given the existence of established methods in this area, comparisons to such approaches seem essential for a fair and complete evaluation.

Best regards,\
AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733294866758

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Reviewer Feedback and Rebuttal Response

comment:
Dear Area Chair and Reviewers,

We sincerely appreciate AC and all reviewers for their valuable insights, which greatly improved the quality and clarity of our work. We are encouraged by the recognition of our contributions, which will motivate us to work continuously on this task.

In our rebuttal, we addressed the key concerns raised during the review process, including:

1. **Task clarification**: We provided a detailed explanation of how OV-MER differs from existing tasks (MLL and PLL) in terms of task definition, solutions, and evaluations.

2. **Validation of OV labels**: We conducted a *user study*, demonstrating that OV labels provide more information than basic emotions, helping to more accurately describe a person's emotional state.

3. **Clarifications**: We clarified our pre-labeling and manual checking process, added more experimental analysis, and corrected some statements.

Finally, we would like to thank the AC and reviewers for initiating insightful and in-depth discussions during this rebuttal. We believe our intensive and high-quality discourse itself already proves the huge potential of OV-MER and highlights the emerging demand for further development of the OV-MER task.

We acknowledge that OV-MER is in its nascent stages and that some challenges remain. However, we are confident that this emerging perspective offers a fresh lens for approaching emotion recognition tasks and could inspire further innovation in emotion AI. By transitioning from basic to more nuanced emotional representations, OV-MER aligns with contemporary research trends both for the machine learning field and the cognitive science field, setting the stage for the future of emotion AI.

We sincerely wish reviewers consider the potential positive impact this work could make in the community. We hope this work can be visible so that it draws more attention from other researchers, thus joint efforts in the community can be made to push forward the renovation of the current emotion AI field by refining the OVL task. We envision this work as a seed for transformative change, not just within emotion AI but across the broader landscape of human-centric technology, and we humbly invite the community to join us in nurturing its growth into something extraordinary.

Thank you once again for your support and consideration. 

Best regards,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a new paradigm for multimodal emotion recognition by expanding the label space based on LLMs. Additionally, it includes a new database and benchmarks for this new approach. The paper is quite interesting and out-of-the-box. The overall concept introduced in the paper is quite thought-provoking and could result in interesting new directions in affective computing. Additionally, the paper is easy to understand and follow. However, the paper unfortunately lacks in several areas. In particular, the validity of the newly proposed/identified labels, questionable positions given existing areas such as multi-label learning, lack of long-term stability given the rapidly varying state of LLMs, and a few others.