OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Authors: Zheng Lian
Haiyang Sun
Licai Sun
Haoyu Chen
Lan Chen
Hao Gu
Zhuofan Wen
Shun Chen
Zhang Siyuan
Hailiang Yao
Bin Liu
Rui Liu
Shan Liang
Ya Li
Jiangyan Yi
Jianhua Tao
Venue: ICML 2025 poster (2025)
Keywords: multimodal emotion recognition
OV-MER
dataset
benchmark
Primary area: applications

Abstract: Multimodal Emotion Recognition (MER) is a critical research area that seeks to decode human emotions from diverse data modalities. However, existing machine learning methods predominantly rely on predefined emotion taxonomies, which fail to capture the inherent complexity, subtlety, and multi-appraisal nature of human emotional experiences, as demonstrated by studies in psychology and cognitive science. To overcome this limitation, we advocate for introducing the concept of *open vocabulary* into MER. This paradigm shift aims to enable models to predict emotions beyond a fixed label space, accommodating a flexible set of categories to better reflect the nuanced spectrum of human emotions. To achieve this, we propose a novel paradigm: *Open-Vocabulary MER (OV-MER)*, which enables emotion prediction without being confined to predefined spaces. However, constructing a dataset that encompasses the full range of emotions for OV-MER is practically infeasible; hence, we present a comprehensive solution including a newly curated database, novel evaluation metrics, and a preliminary benchmark. By advancing MER from basic emotions to more nuanced and diverse emotional states, we hope this work can inspire the next generation of MER, enhancing its generalizability and applicability in real-world scenarios. Code and dataset are available at: https://github.com/zeroQiaoba/AffectGPT.

OV-MER: Towards Open-Vocabulayy Multimodal Emotion Recognition
Zheng Lian' Haiyang Sun? Licai Sun Haoyu Chen Siyuan Zhang'
Lan Chen
Hao Gu Zhuofan Wen Shun Chen
Hailiang Yao Bin Liu' Rui Liu
Shan Liang?
Ya Li Jiangyan Yi? Jianhua Tao
Abstract
1. Introduction
Multimodal Emotion Recognition (MER) is crit- ical research area that seeks decode human emotions from diverse data modalities. How- ever, existing machine learning methods predom- inantly rely on predefined emotion taxonomies, which fail to capture the inherent complexity, sub- tlety, and multi-appraisal nature of human emo- tional experiences, as demonstrated by studies in psychology and cognitive science. To over- come this limitation, we advocate for introducing the concept of open vocabulary into MER. This paradigm shift aims to enable models to predict emotions beyond a fixed label space, accommo- dating a flexible set of categories to better reflect the nuanced spectrum of human emotions. To achieve this, we propose novel paradigm: Open- Vocabulary MER (OV-MER), which enables emo- tion prediction without being confined prede- fined spaces. However, constructing dataset that encompasses the full range of emotions for OV- MER is practically infeasible; hence, we present a comprehensive solution including newly cu- rated database, novel evaluation metrics, and a preliminary benchmark. By advancing MER from basic emotions to more nuanced and di- verse emotional states, hope this work can inspire the next generation of MER, enhancing its generalizability and applicability in real-world scenarios. Code and dataset are available at: http://gtttbbbooomeeeeoo/////btf
Research on emotions has history spanning two centuries. As early the 19th century, Charles Darwin conducted pio- neering research about the evolutionary origins and possible purposes of emotions, explaining the emotional expressions of humans and animals (Darwin, 1872). In 1884, James re- vealed the process of emotion generation, noting that stimuli trigger activities
autonomic nervous system, which in turn produces an emotional experience in the brain (James, 1884). With the rapid development of AI, emotions have garnered increasing attention (Minsky, 1988).
The basis of Multimodal Emotion Recognition (MER) lies in the effective modeling of emotions. Current emotion mod- els are primarily categorized into two types: dimensional and discrete models. Dimensional models, particularly those based on psychological theories such as the Cireum- plex Model of Affect (Russell, 1980), represent emotions within a continuous, multi-dimensiona. space. The most widely adopted framework uses two or three primary dimen- continuum), arousal (the -ewvttiooneaactivatiion level), and dominance (the degree control perceived). These dimensions allow for the quantification of emotional states into measurable numerical values (Warriner al., 2013). However, this so- phisticated numerical representation demands specialized psychological expertise for accurate interpretation, making it abstract and less descriptive to the general public. This abstraction can result in inconsistencies among different annotators, particularly in complex emotional states that fall between the primary dimensions, thereby complicating sub- sequent applications and potentially affecting the reliability of emotion recognition systems.
sions: valence (the pleasantness-uppleassattt
'Institute of Automation, Chinese Academy of Sci- ences "Shanghai Jiao Tong University PCMVS, University of Oulu *Inner Mongolia University "Xi'an Jiaotong-Liverpool University "Beijing University of Posts and Telccommuni- cationsy 'Department of Automation, Tsinghua University "Beijing National Research Center for Information Science and Technology, Tsinghua University. Correspondence to: Zheng Lian <lianzheng2016@ia.cc ia..a........................ Jianhua Tao <jh- t tsinghua.cdu.cn>>.
Discrete models, which categorize emotions into distinct classes, tend to mirror the way people naturally perceive and express emotions in daily life. Ekman (1992) proposed the basic emotion theory, suggesting that there are six ba- sie emotions: anger, disgust.feat happiness, sadness, and surprise. This theory is widely used in MER, where re- searchers typically limit the label space these basic emo- tions and use multiple annotators to select the most likely label through majority voting. We refer to this task as One- hot MER (OH-MER). Considering that emotions can be
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

OV-MER: Towards
Multimodal Emotion Recognition
One-hot MER
Multi-label MER
OV-MER
Label
Space
mosia, iitimie
what
Oot Labeli
Baic Exmedions
Bati Eometinas
Aky Enstie
Description:
indoas with bright vides, backgmandd for facal equesecee cakn, with hand tiked to the right and her mouth sights open, as the ongigeel in lgihteantd corrersation As tine poo by. her arale becertes mort precosseet and she altaity staces her acad to the lett jaansing dae lell if refentty to semoase or seteciing our of vtew. Sowasis dhe esc the vidion aape aniling lows feoardt and opene her north dightss comitegl enphanzzng aadie, charader's too scatiddy seads, rotoolble emploaaic atcentios
windown and
Label
Number
sonceingg the peraes
Uee Label
Mattgee har Liaated
Any Nurther
intoration, wlile "Hending requirs lnersst competes with you is cros-tikk wilk " This sectesce night be the womers cormnnr rnsction to scrconse acatty Giver for audi cues ot stady saze with harser and fie worsats smile and ganee let we cas wir fhe her warls carry modking or coraaii ieee. This the seniase lidely capasa womnals moskey of other pascr's lock of undernarding on Iecograton OV lalack: modley,
The sabtillo real. fhrradiee bez wist nally
illeert
Annotation Manner
I.M
comogge
Cooperatin
Task comparison
(b) Label comparison
Figure 1: Comparison. (a) Task Comparison: We compare the differences among three tasks (one-hot MER, multi-label MER, and OV-MER) across three aspects (label space, label number, and annotation manner). An in-depth comparison is provided in the Appendix A; (b) Label Comparison: We provide an example to visualize the one-hot and OV labels. More examples are provided in Appendix F. Since the original video contains real people, we use DemoAl to remove personal information to address copyright concerns. In this paper, we use amotion-related descriptions as a bridge to extract OV labels. We observe that OV labels offer more insightful understanding of the emotional state.
compound, researchers further propose Multi-label MER (ML-MER), allowing each sample to have multiple labels (Li et al., 2017). However, both OH-MER and ML-MMER generally have limited label spaces. Plutchik (2001) pointed out that humans can express approximately 34,000 distinct emotions. Although some efforts have been made to expand label spaces with more emotional categories, current ap- proaches still fail to capture emotional diversity, inevitably overlooking some of these nuanced emotions.
emotions are expressed in our real-life interactions, leading to more accurate and numan--entere MER. As illustrated in Figure (b)), labeling an emotion solely as happy is not sufficiently informative. In contrast, OV-MER provides emotions like mockery, offering a more comprehensive and insightful understanding of the emotional state. Therefore, OV-MER facilitates the transition from basic to nuanced emotion recognition, advancing the development of emo- tion AI. Appendix C provides more detailed motivation. In summary, we make the following key contributions:
In this paper, we introduce a new MER paradigm, open- vocabulary MER (OV-MER), by introducing the concept of open-vocabulary into MER, enabling the prediction of arbitrary emotion categories. Figure I(a) provides a compar- ison between different tasks, and in-depth comparison is provided in Appendix A. To support this shift, we build dataset, define evaluation metrics, and develop solutions. (1) Dataset: we propose a human-LLM collaboration strategy to construct the dataset. Compared to human-only annota- tion, our strategy can leverage L.M to enhance the label richness; (2) Metrics: since there is no fixed label space, the model may predict closely related but differently expressed emotions (c.g.. joyful and happy). To provide more reli- able evaluation results, we first group similar emotions and specifically design metrics for this task; (3) Solutions: tra- ditional discriminative classifiers rely on fixed label spaces. However, OV-MER does not restrict the label space, neces- sitating the definition of new solutions.
Paradigm. We propose new paradigm in MER, called OV-MER. This paradigm transitions from tradi- tional MER to a framework that enables the prediction of any number and category of emotions, thereby ad- vancing emotion AI toward real-world applicability by capturing the full spectrum of human emotions.
Groundwork. We lay the groundwork for OV-MER by constructing datasets, defining evaluation metrics, and proposing solutions. Our dataset enhances label rich- ness through human-LLM collaboration. Mcanwhile, we introduce new evaluation metrics that leverage emo- tional relevance to achieve more reliable results.
Benchmark.
We build zero-shot benchmarks for OV- MER through extensive experiments and detailed anal- ysis. This task can serve as an important evaluation benchmark for multimodal LLMs (MLLMs), challeng- ing their ability to integrate multimodal clues and cap- ture subtle temporal variations emotional expression.
A natural question arises: why is OV-MER so important?
A simple answer is that it naturally aligns with the way
2

---

OV-MER: Towards Vpen-Vocabulary Multimodal Emotion Recognition
(a) CLUE-Multi Generation
m & analysis
prelabel
check X2
CLUE-Multi: In the video... In the audio... Based on the surprised and negative facial expression of the male character
ALLM
LLM
audio
the video clues, as well as the stuttering speech the audio clues, we can infer that
pretabel
check X2
VLLM
video
(b) Ground-truth OV Label Extraction
merge & check
CLUE-Multi (English)
OV Label (English)
OV Label (Chinese) YEC
B
Similarity
Ycc
Yεc
0.82
YEE
GT Labels: surprise, nervous, dissatisfied
YFE
YGg
0.82
X2
YFF
Yee
0.78
Yoe OV Label (Chinese)
YCB OV Label (English)
Yec
YCB
0.73
CLUE-Multi (Chinese)
OV Extract
LLM
Translate
Figure 2: Dataset construction. (a) CLUE-Multi Generation: For audio and video, we use audio LLM (ALLM) and video LLM (VLLM) to extract initial clues, followed by two rounds of manual checks to eliminate errors and duplicates while adding missing content. Each round involves multiple annotators, with no overlap between annotators in the two rounds. Finally, merge the checked clues with
generate CLUB-Multi. (b) Ground-truth OV Label Extraction: There certain differences in the labels extracted from different languages. To eliminate language influence and achieve consensus labels, we merge these labels and conduct manual checks. These checked labels are regarded as the ground truth.
Experiments. Our intensive experimental results not only demonstrate the strength of our methods but also prove that OV-MER can effectively enhance the pre- sentation ability of emotions and user experience.
ers. We randomly selected a subset of MER2023 for further annotation to construct our OV-MERD dataset. Additional details about MER2023 can be found in Appendix E.
2.1. CLUE-Multi Generation
2. The OV-MERD Dataset Construction
During the annotation process, we observe that human- LLM collaboration yields more detailed descriptions than the human-only strategy (see Section 5). In this section, we provide detailed overview of our strategy. For manual checks, to maintain high-quality annotations, all annota- tors must pass a preliminary test. This test evaluates their performance on 12 samples, each of which was previously annotated by five annotators with full agreement. Annota- tors who perform poorly are removed from the annotator pool. More annotation details can be found Appendix L.
Although the concept of OV-MER is intuitive and holds great promise, its practical implementation faces significant challenges. The main difficulty lies in the broad and subtle range of human emotions, making comprehensive labeling complex task. Traditional annotation methods are limited by their predefined emotion categories, which often in- sufficient for the needs of OV-MER. In Figure 2, we propose human-LLie collaboration strategy that consists of two steps: CLUE-Multi generation and emotion label extraction. Ultimately, we create dataset, OV-MERD, which offers a richer of emotions compared to existing datasets (see Ta- ble 1). This dataset an extension of MER2023 (Lian al., 2023). Specifically, MER2023 is collected from movies and TV series, with most samples consisting of single-person videos featuring relatively complete speech content. The use of this dataset has been approved by the dataset own-
Initially, we attempt annotate visual and acoustic clues directly. However, the descriptions ob- tained in this way cannot cover all information. Therefore, we explore using other models for pre-annotation. (1) For video, given the strong visual understanding capabilities of GPT-4V 4-ipt---vision- -resiinnprevvew"), we use as VLLM for pre-annotation, Since GPT-4V only supports image in-
Pre-annotation.
3

---

OV-MER: Towards Open-Vocabulas Multimodal Emotion Recognition
Table 1: Dataset comparison. See Appendix H for a comprehensive comparison.
Dataset
Modality
Annotation Type
# Categories
# Labels per Sample
MOUD (Pérez-Rosan al., 2013)
A.
Dimensional Emotion
CMU-MOSI (Zadeh et 2017)
A,V.
Dimensional Emotion
CH-SIMS (Yu et al., 2020)
A,
Dimensional Emotion
CH-SIMS v2 (Liu al., 2022b)
A,V.T
Dimensional Emotion
SEMAINE Kewwn et al., l)
A,V,T
Dimensional Emotion
MSP-IMPROV (Busso el al., 2016)
A.
Discrete Emotion
IEMOCAP (Busso et al., 2008)
V.
Discrete Emotion
10
MELD (Poria al., 2019)
A.V.T
Discrete Emotion
MER2023 (Lian ct al., 2023)
A,V.T
Discrete Emotion
6
MER2024 (Lian al., 2024a)
A.V.T
Discrete Emotion
6
236
1~99, most 2-
OV-MERD (Ours)
A,V,T
Discrete Emotion
(arbitrary label)
(arbitrary number)
put, we uniformly sample three frames from each video and input them into GPT-4V. We discuss the reasons for sampling three frames Appendix K. (2) For audio, we use the open-source SALMONN (Tang et al., 2023) ALLM for pre-annotation, as GPT-4V does not support audio input.
Appendix K, we discuss the details of this merging process and the reasons behind The above annotation pipeline reflects the collaboration between humans and LLMs.
2.2. Ground-truth OV Label Extraction
Label Extraction. After that, we use the LLM extract emotion labels from CLUE-Multi. This process relies on GPT-3.5, which request identify emotional states based on the provided descriptions without restricting the label space. See Appendix K for more details.
Manual Check. As part of our quality assurance pro- cedures, perform detailed examination of the pre- annotated results. For visual clues, GPT-4V may generate hallucinated responses, i.e., clues that do not actually ex- ist. Additionally, there are repeated expressions and some temporal association clues are missing. Therefore, we hire annotators to eliminate errors and duplicates, as well as add missing content. For acoustic clues, ALLM struggles capture emotion-related paralinguistic features. The main reason is that current ALLM mainly focuses tasks like ASR or audio event detection (Tang et al., 2023), with less emphasis on paralinguistic information. Hence, we hire multiple annotators to focus on the speaker's intonation and other emotion-related paralinguistic clues. To reduce subjective bias, we conduct two rounds of manual checks. Ultimately, these checked clues accurately reflect the video content. Appendix L provides the annotation guide- line and layout of the annotation platform.
Language Impact. We further explore the language im- pact. In Figure 2, first extract OV labels from English and Chinese descriptions, obtaining YEE and Ycc. Then, we translate them into the other language, yielding YεC and YCE- Next, we measure the similarity between different sets and report results Figure In Appendix N, we detail our metric calculation process. observe that the labels extracted from different languages exhibit some differences. For example, the similarity score between YEE and Yeu is 0.82, which may be due to the varying definitions of emo- tions in different languages. To eliminate language influence and achieve consensus labels, we merge the labels extracted from both languages and conduct manual checks. These checked labels are regarded as the ground truth.
CLUE-Multi Generation. We leverage the reasoning pabilities of LLM to merge all clues. Specifically, we use GPT-3.5 ("gpt-3.5-turbo -ur.--ttooo--1--0613333 16k-0613") as the LLM and ask it to merge textual, acoustic, and visual clues. The output is an emotion-related description, denoted as CLUE-Multi (see Figure 2). worth noting that we not perform addi- tional manual checks of the generated CLUE-Multi, as GPT- 3.5 consistently produced reasonable and logical results. This reliability likely stems from the GPT-series models' exceptional performance in reading comprehension (Brown et al., 2020) (close to human performance), where multi- clue integration a core functionality. Therefore, we skip the manual inspection of CLUE-M striking a balance between dataset reliability and construction efficiency.
2.3. OV-MERD Dataset
Finally, we construct a dataset called OV-MERD. This dataset an extension of MER2023 (Lian et al., 2023), from which we randomly select portion of samples for fur- ther annotation. Table compares OV-MERD with existing datasets. We observe that our OV-MERD dataset contains 236 emotion categories, and most samples have 2 to 4 labels, far exceeding those in current datasets. In Appendix I, we observe that OV-MERD encompasses a broader range of emotions, including some that have been rarely discussed in previous research, such as shy, nervous, and grateful.

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
3. Evaluation Metric
i.e., (s(Ww) {v).... ...g... Uf the E EW), where /() is function that maps each label yo to its synonym yf. We also define its inverse function f'(.), which maps different synonyms yy back to their base label Ye.
Defining evaluation metrics for OV-MER presents signifi- cant challenges: (1) OV-MER supports predicting emo- tions of any category. Thus, the model may predict closely related but differently expressed emotions. To provide more reliable evaluation results, we first group the emotions based on their similarities. (2) OV-MER allows for the predic- tion of an arbitrary number of labels. Thus, traditional evaluation metrics designed for a fixed number of labels may not be applicable. In this section, we propose set-based evaluation metrics specifically tailored for this task.
To eliminate the influence of word forms (e.g.. happy and happiness), we further ask GPT-3.5 multiple times to gen- erate different forms for each label. The prompt used is as follows: Please output different forms of the following word in a list format. After that, obtain EW-SF, i.e., {g(Wr) = Luy.... Vg }, y/ E EW-S). where g(-) is func- tion that maps each label yy to its different forms yg. We also define its inverse function g' (.), which maps different labels Bg back to their base form Vy- Finally, we define different types of metrics:
3.1. Grouping
We propose two grouping strategies: one based on GPT and the other based on the emotion wheel (EW) (Plutchik, 1980). In the experiments, we use GPT-based grouping by default.
(I) M1. We use g'(+) to map each label its yf.
(2) M2. We use (-)) to map each label to its yur
GPT-based Grouping. The most direct approach is to use GPT-335 to group all labels based on their similarity: Please assume the role of an expert in the field of emotions. We provide a set of emotions. Please group the emotions, with each group containing emotions with the same meaning. Directly output the results. The output format should be list containing multiple lists. However, the evaluation results may be affected by the API version. For example, if OpenAl deprecates an old API, the results based on that API will become difficult to reproduce. Additionally, this process is costly (see Appendix O). Therefore, we attempt to find a replacement for GPT-based grouping.
(3) M3. We use the emotion wheel during metric calculation. Specifically, we first use (g(g(.) (.)) to map each label to its You Then, we define two grouping functions, LI and L2. For LI, we map all labels to their corresponding L...:
ywu m (Bwr) m2=1 (
if yu E Lmn
if y E L2 if yau E I33
(1)
For L2, we map all labels to their corresponding D
select one label in m ( if you E Liv ywu (
if you E Li (2) if you E D
Ea-based Grouping. EW is a psychological model that categorizes emotions in a structured manner. The inner part shows core emotions, while moving to the outer part reveals more nuanced emotions. Therefore, EW naturally provides emotion grouping information. Since there is no consensus on EWW we select five typical wheels (see Appendix P).
3.2. Metric Definition
Then, we convert the above emotion grouping informa- tion into a function G(-), which can map each label to its group ID. Specifically, suppose ) and ( N the ground truth and predictions, where M and N are the number of labels. We first map each label into its group ID: y = [z E (w)M)) and 5 = [G(=)]æ E (0:) Then, we design set-based metrics for performance eval- uation. Specifically, Precisions indicates the number of correctly predicted labels; Recalls indicates whether the pre- diction covers all ground truth; Fs the harmonic mean of two metrics, which is used for the final ranking:
Before calculating the metrics, we define some symbols. We group the labels by their levels from the innermost to the outermost as 1 T2 and I.,., Next, we define function m.)) that maps the labels in I to the cor- responding labels in L' From inner to outer (i < j), (-) is a many-to-on to- mapping; from outer to inner (i >>> f), m (-) is a one-to-many mapping. We collect all the labels from these emotion wheels and represent them as EW, i.e. [L2,, s 5, S j S 3). We denote the labels in EW as yu-
are
D n'n [D]
p'n
Precisions
Recalls
(3)
Considering that the emotional categories in EW are still lim- ited, we perform some label expansion operations. Specif- ically, we repeatedly call GPT-3.5, asking it to generate synonyms for each label. The prompt used is as follows: Please retrieve the synonyms for the following words and output them in a table format. Then, we generate EW-S,
[>]
Fs = 2 Precisions x Recalls
(4)
Precisions + Recalls
It is important to note that changing the label order in y and 5 does not result in any score change. The subscript
5

---

LLM. We OV-MER: Towards Multimodal Emotion Recognition
(a) Preliminary
(b) CLUE Generation
(c) Metric Calculation
English Branch
Brunch
text
CLUE (English)
CLUE (Chinese)
mrbbo
Acoustic Clue (Prelabeled)
V
CLUE- MLLM
Acoustic Clue (Prelabeled)
ALEM
Visual Clue (Prelabeled)
audio
OV Label (English)
Label (Chinese)
dlchk
Acoustic Clue (Checked)
Acoustic Clue (Checked)
CLUE-
Label (English)
LIX
Audio
ALEM
Visual Clue (Prelabeled)
video
Visual Clue (Checked)
LIM
CLUE-
Video
GT Labels
Visual Clue
LLM
CLUE-
(Checked)
Text
Trandate
Fxtroc
Menio
Figure 3: Baselines. (a) Preliminary: We begin by defining some preliminary symbols. (b) CLUE Generation: CLUE- Video and CLUF-Audio use manually-cheoked clues; CLUE-Text relies solely on text; CLUE-MILLM does not involve manual checks and directly uses the outputs from ALLM or VLLM. (c) Metric Calculation: We rely on CLUE to predict emotion labels. Due variations in labels extracted from different languages, we report results across different languages.
"s" indicates that these metrics are set-based, distinguishing them from the traditional single-label metrics.
MILMs, we use their 7B version by default. All models are implemented in PyTorch, and all inference processes are executed on a 32GB NVIDIA Tesla V100 GPU.
4. Baselines for OV-MER
4.2. Metric Calculation
4.1. CLUE Generation
As shown in Figure 2, there are certain differences in the labels extracted from different languages. Therefore, we report the results for both English and Chinese descriptions. In Figure 3, for the Chinese branch, we first extract OV labels and then translate them into English; for the English branch, we directly extract OV labels. Finally, we compute the evaluation metrics with the ground truth. is worth noting that the OV labels extracted from the monolingual CLUE-Multi differ from the ground truth. Our ground truth combines the labels extracted from different languages and undergoes further manual checks (see Figure 2).
Figure illustrates the generation process of CLUE-Multi, where we combine text with checked visual and acoustic clues. In this section, further introduce some variants.
CLUE-A/T/V. To reveal the modality impact, propose three variants of CLUE-Multi: CLUE-Audio, CLUE-Text, and CLUE-Vide Video. In Figure 3, we illustrate their genera- tion process. (1) CLUE-Audio: We observe that ALLM cannot fully leverage the text, and using an additional LLM to emphasize the text can further improve performance, which is also verified in Section 5, Therefore, we merge the checked acoustic clues with text using additional LLM; (2) CLUE-Tex:: We only use the text to infer emotional states; (3) ELUE-Vide:: Since the visual content does not contain audio and text, we only use the checked visual clues. See Appendix Q for more examples.
5. Results and Discussion
In this section, we default using GPT-based grouping and employ GPT-3.5 generally report evaluation results in both languages, but if no specific language is mentioned, we default to reporting results for the English branch. To mitigate the impact of randomness, we conduct each experiment twice and report the average scores and standard deviations. In addition to MLLM-bassed generative models, also report the perfor- mance of discriminative models in Appendix U.
CLUE-MLLM. MLLMs can address various multimodal tasks. Since emotion recognition relies on temporal informa- tion, we choose models that support at least video audio. To generate CL.ME-MLJM, we first use ALLMM or VL.LM to extract emotion-related descriptions, and then combine these descriptions with text using LLM. Compared with CLUE- Multi, this process does not use manually checked clues. Appendix provides model cards and relevant prompts. For
Main Results on CLUE-M/A/TVV For CLUE-M/A/TIV,
most baselines use manually checked clues, serving as per-
6

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Table 2: Main results. Figure 3 illustrates the metric calculation process. The primary distinction between CLUE-MLLM (Baselines) and CLUE-M/A/T/V (Upper-Boc Performance) is whether manually verified clues are utilized.
English Precisions t
Chinese Precisions f
Model
L
V
A
Fs 1
Recalls 1
Fs 1
Recalls
Heuristic Baseline
Random
X
X
x
17.42.0.01
24.85 10.15
13.4210.04
16.59 10.5910.00
24.7010.00
12.48
CLUE-MLLM (Baselines)
Qwen-Audio
x
38. 10.00
49.4
31.0440.00
41. 41.14
53.71 =0.00
33.3440000
OneLLM
X
42.84
45.92.0.05
40.1510.06
46.17
52.07 1:000
41.47 10.08.0.00
Otter
43.5 10.09
50.71
38.09
46.22.10.01
52.65.10.16
41.18
Video-LLaMA
44.73
44.14 40.33
45.34
47.26
47.98+00.07
46.56,20.01
VideoChat
45.53
42.90+0.27
48,49+0.10
45.57
47.20
44.05
SECap
45.72.10099
14.52
10.05
45.57 1:013
55.55.00.0
38.64.00008
PandaGPT
45.89 10.20
50.03
42.38.10.3
4..33
53.01 1:0.08
42.75.10.11
Video-ILLaV
47.07
48.08
45.6640.29
49.21 20006
53.95
45.23
SALMONN
47. 47.964
50.20+t0.44
45.92+0.04
48. 48.24
52.24
44.82++0.0
VideoChat2
49.07 10.28
54.72.0.41
44.47
48. 18.86
57.1210.008
42.68 1.004
Video-ChatGPT
50.52
54.03
47.4440.07
54.73.40.00
61.15+0.10
49.52.40.0
OneLLM
50.52
55.93
46,06.00.6
51.44
56.43200.44
47.26
LLaMA- VID
51.25 10.09
52.71
49.87 10.00
52.01
57.30
47.61 1:000
mPLUG-OWl
52.73
10.14
51.04
50.9510.06
56.40
46.47.0.11
Chat-UniVi
53.08
53.68 #0.00
52.50
53.86
58.54
49.86=0.08
GPT-4V
55.51
48.52 40.52
64.86
57.21 +0.01
54.61 #0.00
=0.07
CLUE-M/A/T/V (Upper- Bapper-Buudd Bound Performance)
CLUE--Txt
46.00.0.08
54.41 10.15
39.8410.01
43. 1:025
50.69 10026
37.50
CLUE- Video
V.
60.55
63. 40.0s
58.05
61.73
66.47 =0.11
57.62.00.08
CLUE-Audio
65.35.0.04
67.5 07.5410.06
63.3010.00
68.56
70.10...06
67.07.0.00
CLUE--uUti
80.05 10.24
80.03.1.55
80.07 10.10
85.1610.08
87.09 10.00
83.31 10.05
formance upper bounds of different modality combinations. In Table 2, we observe that CLUE-Multi performs the best, highlighting the importance of multimodal information in MER. Meanwhile, CLUE-VIE Video outperforms CLUE-Text, consistent with the nature of our OV-MERD dataset. To be specific, OV-MERD is derived from MER2023, where the textual modality contributes less than the visual modality in emotion recognition (Lian et al., 2024b). Relying solely on text makes it difficult to recognize emotions accurately. Furthermore, CLUE-Audio achieves superior performance over both CLUE-Text and CLUE-Video, suggesting that although textual expressions may be ambiguous for emotion recognition, combining them with audio cues can effectively resolve these ambiguities.
angry sa
worry
happy
sabness
(a) Length (H)
(b) #Label (H)
(c)
Word cloud (H)
iffied
excited
happy angry
Jooy Ma
(d) Length (Hel.)
(c) #Lahel (H+l.)
Word (H+T.)
Figure 4: Human-onja (H) vs. Human- (H+L) strategy.
Main Results on CLUE-MLLM. In Table 2, we intro- duce heuristic baseline called Random, where we ran- domly select a label from basic emotions. This baseline reflects the lower bound. We observe that MILM generally outperforms Random, indicating that MLLM can partially address the OV-MER task. However, the performance of MLLM remains unsatisfactory, highlighting the limitations of existing MILMss and the challenges of OV-MER. Further- more, models that perform well in Chinese often perform well in English, suggesting that the impact of language differences on rankings is limited. Appendix S presents quantitative analysis results on language differences.
Human-only vs. Human-LLM Collaboration. To verify the effectiveness our human-LLM strategy, we addition- ally introduce baseline using human-only annotation. In Figure 4, we compare two strategies from three aspects: the length distribution of generated descriptions, the distri- bution of sample- wise label numbers, and the word cloud. In Figure 4, we observe that through human-L.LM collab- oration, we can obtain longer descriptions, provide more diverse labels for each sample, and generate a broader range of emotions. These results demonstrate that human-only annotation generally focuses on primary emotions while

---

OV-MER: Towards
Multimodal Emotion Recognition
54.33
65.34
$6.22
45.t5
20.15
05.14
15.0
4035
37.7
45.1
30:3
20.4
15.32
20.33
27.55
24.00
26.39
25.42
(a) PandaGPT
(b) Vidco-ChatGPT
(c) Vidco-LLaMA
(d) VidcoChat
(c) VideoChar2
34.54
20.42
43.32
5172
2143
64.37
44.38
4852
66.30
56.15
31.00
0.30
4630
41.38
3150
56.02
15.00
22.70
23.33
3453
(f) mPLUG-Owl
(g) Qwen-Audde
(h)
ii LLaMA-VID
UhittUUniii
Figure 5: Performance comparison of different strategies for generating CLUE-MLLM.
neglecting minor ones. With the pre-annotation and seman- tic reasoning capabilities of LLMs, we can obtain richer emotional labels. These results validate the effectiveness of our human-LLM collaborative strategy. Meanwhile, these results suggest that the LLM-driven approach does not lead to narrow or biased interpretation of emotions, but rather helps uncover more subtle emotional nuances. We provide additional analysis Appendix
CLUI-MMali (Reglihh
CLiI--Maie (Chineso)
CLUEMLLL (Fagian
CLUE-NILMM (Chlamm
Label (lopila
OV Label (Chlumd)
Label (Feglan)
G21
GTbbl
Ablation Study on CLUE-MLLM.
We reveal the impact of different CLUE-MLLM generation strategies.
(a) Metric Calculation
(b) Correlation Analysis
Figure 6 introduces three methods:
1) SO does not use text and inputs the video into MILM; 2) S1 inputs both text and video into MILM; 3) S2 first uses MILM to extract de- scriptions and then combines with text using another LLM, same with the strategy in Figure In Figure 5, SI and S2 generally outperform SO,
Figure GPT- vs. Matching- based metrics.
matching- based metrics, with BLEU,, BLEU,, METEOR, and ROUGE, being the most widely used. To investigate this, we use CLUE-MLLM as imput and calculate both GPT- based and matching- report their Pearson Correlation Coefficient (PCC) scores in Figure 7(b). Experimental results reveal several inter- esting observations. First, the same metric across different languages typically shows high correlations. However, the correlation between GPT-based and matching-based metrics is relatively weak. For instance, the highest PCC score be- tween "F(E)" and matching- metrics only 0.77. This discrepancy arises because matching- based metrics focus on low-level word-level matches, whereas emotion understand- ing more complex, high-level perceptual task. Appendix V provides more detailed explanation of these findings.
Figure 6: Ablation.
metrics (see Figure 7()), and
indicating the importance the text content in OV-MER. Moreover, S2 typically performs better than $1. The reason that inputting video and text into the MLLM simultane- ously increases the task difficulty, and current MLLMs may struggle to handle complex prompts. S2 divides this pro- cess into two steps, reducing task complexity and achieving better performance. Therefore, we adopt S2 as the default strategy. More results are provided in Appendix R.
GPT-based vs. Matching-based Metrics. In Table 2, CLUE-Multi demonstrates the best performance, leading to the hypothesis: Do sentences that are more similar to CLUE-Mulh yield better emotion recognition performance? The most common way to measure "similarity" is through
GPT-based vs. EW-based Grouping. We propose two grouping strategies: GPT-based and EW-based grouping. In this section, we explore the relationship between them.
8

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Table 3: GPTT-based vs. EW-based grouping. We calculate the PCC score to reveal their correlation.
M3-WI
M3-W2
Model
GPT
M1
M2
L2
L2
Qwen-Audio
38.13-0.05
20).49 =0.01
23.37
43.85
26.60=0.01
41.52.20.28
26.68+00.1
Otter
43.51 00.9
22.21
10.06
27.99 10.02
49.75.10111
23.50
49.93
13.06
Video-LLaMA
44.73 20.14 -0.14
23.56
20.08
28.39 #0.17
52.90
36.08 10.13
53.60
25.33
VideoChat
45.53
=0.10
22.15
=0.00
26.24 20.08
47.79
=2.64
47.76
32.14
SECap
4..72_0.09
26.52
32.88
52.2610.03 0.03
20.55
12.11
10008
Video-LLaVA
47.07_0.16
25.47 20.12
30.73
54.65 10.10
37.65 1.0.44
1..54
10.22
SALMONN
20.04
23.57
28.83
54.90
38.93
54.29 0..06
37.79
VideoChat2
49.07=0.26
26.92=0.09
31.40.00100
52.38
36.44=0.11
53.56.0018
=6.91
Video-ChaIGPT
50.52=0.08
2..9
20.05
57.66 04
41.48
50.37
40.95
LLaMA VID
0.009
28.28 2.00
10.03
10.59.0.04
41.22..0.0
57.49 0.005
10.09
mPLUG-Owl
52.73 0011
27.47 20.17
32.47 10019
57.60
41.32
56.32
40.83
Chat-UniVi
53.08=0.01
28.89 -0.02
33.23+0.08
57.00=0.06
42.25=0004
57.50=0.0s
42.43++008
PCC score
0.887
0.857
0.911
0.940
0.913
0.942
M3-W3
M3-W4
M3-W5
Model
M-avg
L1
L2
LI
L2
LI
L2
Qwen-Audio
39,46_0.24 20.28
30.65 =0.01
36.64
27.33 00.0
35.89 20.08
29.66
31.84
Otter
51.03_0.00
37.12
47.54
34.77 .0
50.51 10.03
35.54
39.41
Vidco-LLaMA
47.50
36.50_..25
10.07.0.00
35.78
20.12
34.77.0.23
40.31
VideoChat
46.78 0011
34.37 0..03
49.53
32.82
45.93
32.85 0.004
37.58
SECap
50.77
40.49 20.03
50.43
38.21 20.03
-9.97700.00
40.25 0.00
42.43
Vidco-LLaVA
52.29
40.58
52.45
39,91 39.91 10
=2.97
39.69
43.27
SALMONN
56.25
43.01 10.02
50.53
38.54
53.65
42.09
43.53
VideoChat2
52.14
40.57
50.63
39.64
51.37
39.89
42.65
Vidco-ChutGPT
55.50_o.13
14.15
55.24
42.42.20.03 05.0
52.93
41.54_0014
46.02
ALaMA-VID
55.12_0.05
44.06
16.62
42,42 03
53.03
41.65
45,81
mPLUG-Owl
55.67
43.71
55.06
40.67 10 19
54.44_0.13
42.00
45.63
Chat-UniVe UniVi
56.80
45.66
55.86
41.97 +0.0 09
55.81
43.61 -0.66
46.75
PCC score
0.904
0.927
0.899
0.922
0.885
0.894
0.942
Table 3 reports F's for different EW-based strategies, as this metric is used for the final ranking, and we compute the PCC scores between different metries. We observe that the PCC scores between GPT-based and EW-based groupings relatively high, indicating that EW-based metrics can serve an alternative to GPT-based metrics. Meanwhile, we observe that M3-1.2 is always more correlated with the GPT-based metrics than M3-L1. M3-LI emphasizes coarse- grained clustering information, whereas M3-L2 emphasizes fine-grained clustering information. The higher correla- tion between M3-L2 and GPT-based metries suggests that GPT-based metrics primarily rely on fine-grained emotion clustering during the metric calculation.
Aligmment with Human Perception. To evaluate how well OV-MER aligns with human perception, we conducted an additional user study. Specifically, we recruited nine an- notators and randomly selected 20 samples from our dataset. Each annotator was presented with (sample, OV-MERD label) pairs and asked to judge their alignment with hu- man perception using a binary (Yes/No) response format. To ensure annotation quality, we included inspection data consisting of (sample, incorrect label) pairs. The results show that 96% of the annotations confirmed the alignment between OV-MERD labels and human perception. Consid- ering potential annotator errors, this result demonstrates that our OV-MERD labels align well with human perception.
Informative Comparison. We conducted user study to evaluate whether our annotation manner provides greater informativeness compared to traditional basic emotions. Specifically, we recruited four annotators and randomly selected 20 samples from our dataset. For each sample, we presented both the basic emotion label and the OV-MERD label. Annotators were instructed follows: Which la- bel provides greater informativenes?? ? The label with more information was marked as 1, and the other lubel as O. Experimental results show that 97.5% of the annotations favored our OV-MERD labels, confirming their superiority in informativeness over basic emotions and verifying the effectiveness of our OV-MERD in emotion representation.
6. Conclusion
This paper extends traditional MER to OV-MER, allowing for the prediction of arbitrary numbers and types of emo- tions. For this task, we build dataset, define metrics, and propose solutions. We observe that current MILMS struggle to achieve satisfactory results, as this task requires consid- eration of multimodal clues and subtle temporal changes, placing higher demands on MILMs. Additionally, EW- based metrics can replace GPT-based metrics, thus reducing evaluation costs while ensuring reproducibility. This paper advances current research from basic to nuanced emotion recognition, which is crucial for emotion AL.

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Acknowledgments
Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
This work is supported by the Excellent Youth Program of State Key Laboratory of Multimodal Artificial Intelli- gence Systems (MAIS202431D,, the National Natural Sci- ence Foundation of China (62201572, 62322120, 61831022, 62276259, U21B2010, 62271083, 62306316, 62176165, 62206136, 62476146), the Internal Research Project of Xi'an Jiaotong- Liveero-- University (RDF-24-01-016), the Young Elite Scientists Sponsorship Program by CAST (20240NR001), and the University of Oulu& Research Council of Finland Profi (352788).
Burkhardt, F., Paeschke, A., Rolfes, M., Sendlmeier, W. F., Weiss, B., et al. A database of german emotional speech. In Interspeech, volume 5, pp. 1517-1520, 2005.
Busso, C., Bulut, M., Lee, C.-C.,, Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., and Narayanan, S. S. lemocap: Interactive emotional dyadic motion capture database. Language Resources and Evaluation, 42:335-
359, 2008.
Impact Statement
Busso, C., Parthasarathy, S., Burmania, A., AbdelWahab, M., Sadoughi, N., and Provost, E. M. Msp-improv: An acted corpus of dyadic interactions to study emotion per- ception. IEEE Transactions on Affective Computing, 8 (1):67-80,, 2016.
Ethies Statement. The raw data of the OV-MERD dataset comes from MER2023, from which we select some samples with further annotation. Therefore, we do not collect new data; we just re-annotate existing data. This annotation pro- cess has received consent from the dataset owners and has passed our internal review. During the annotation process, we generously pay each annotator approximately Y3,000 (around $280). which is considered high. After proofread- ing our annotation results, we find that the annotations focus on the multimodal clues present in the videos, without any discriminatory annotations. Additionally, we restrict the use of the OV-MERD dataset to non-commereial purposes under the CC BY-NC 4.0 license. This license clearly outlines the correct and responsible use of our dataset.
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., and Verma, R. Crema-d: Crowd-sourced emotional multimodal actors dataset. IEEE Transactions on Affective Computing, 5(4):377-390, 2014.
Chen, H., Liu, X., Li, X., Shi, H., and Zhao, G. Analyze spontameous gestures for emotional stress state recog- nition: A micro- geero-g dataset and analysis with deep learning. 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1-8. IEEE, 2019.
Proper Use. MER is widely discussed research topic. In this paper, we extend traditional MER by providing more accurate emotion annotations that go beyond the fixed emo- tion taxonomy. In the license we provide, we restrict the use of this dataset to academic research; commercial usage is prohibited. Meanwhile, this dataset can only be used in non- sensitive compuerr interaction scenarios enhance the machine's s ability understand human emotions and respond appropriately. It cannot be used in sensitive areas. For example, in interrogation scenarios, emotion recognition results should not be used to determine whether a criminal has committed a crime; Emotion recognition results should also not be used in recruitment and loan approval processes, as this may lead to unfair treatment of certain groups and affect their employment and financial opportunities; In the field of education, emotion recognition results should not be used to evaluate the performance of teachers and students.
Chen, H., Shi, H., Liu, X., Li, X., and Zhao, G. Smg: A micro-gesture dataset towards spontaneous body gestures for emotional stress state analysis. International Journal of Computer Vision, 3666661346-136 2023.
Chu, Y., Xu, J., Zhou, X., Yang, Q., Zhang, S., Yan, Z., Zhou, C., and Zhou, J. Qwen-audio: Advancing univer- sal audio understanding via unified large-scale audio- language models. arXiv preprint arXiv:2311.079199 2023.
Costantini, G., Iaderola, I., Paoloni, A., Todisco, M., et al. Emovo corpus: an italian emotional speech database. In Proceedings of the ninth international conference on language resources and evaluation (LREC' 14), pp. 3501-3500 European Language Resources Association (ELRA), 2014.
Cour, T., Sapp, B., Jordan, C., and Taskar, B. Learning from ambiguously labeled images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 919-926, 2009.
References
Bishop, C. M. Pattern recognition and machine learning. Springer, 2006.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Darwin, C. The Expression of Emotions in Man and Animals. Penguin Classics, 1872.
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
10

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Demszky, D., Movshovitz-Atiaa, D., Ko, J., Cowen, A., Nemade, G., and Ravi, S. Goemotions: A dataset of fine-grained emotions. In Proceedings of the 58th Annual Meeting of the Association. for Computational Linguistics, pp. (040-4054, 2020.
D., Lee, D.-H., al. Challenges in representation learn- ing: A report on three machine learning contests. In Neural information processing: 20th international con- ference, ICONIP 2013, daegu, korea, november 3-7, 2013. Proceedings, Part III 20, pp. 117-124. Springer, 2013.
Deng, and Ren, Multi-label emotion detection via -ppciiie- feature extraction and emotion correla- tion learning. IEEE Transactions Affective Computing, 14(1):475-488) 2020.
Grimm, M., Kroschel, K., and Narayanan, S. The vera mittag german audio-visual emotional speech database. In IEEE International Conference on Multimedia and Expo, pp. 865-868. IEEE, 2008
Dhall, A., Ramana Murthy, O., Goecke, R., Joshi, J., and Gedeon, T. Video and image based emotion recognition challenges in the wild: Emotiw 2015. Proceedings of the 2015 ACM on international conference on multimodal interaction, pp. 423 426,422,222 2015.
Gu, Y., Yang, K., Fu, S., Chen, S., Li, X., and Marsic, I. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics, pp. 2225 2235, 2018.
Dhall, A., Goecke, R., Ghosh, S., Joshi, J., Hoey, J., and Gedeon, T. From individual to group-level emotion recog- nition: Emotiw 5.0. In Proceedings of the 19th ACM international conference on multimodal interaction, pp. 524-528, 2017.
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26584-26595, 2024.
Hazarika, D., Zimmermann, R., and Poria, S. Misa: Modality-invariian and-specific representations for mul- timodal sentiment analysis. In Proceedings of the 28th ACM International Conference Multimedia, pp. 1122- 1131, 2020.
Duville, M. M., Alonso-Valerdi, L. M., and Ibarra-Zarate, D. I. The mexican emotional speech database (mesd): claboration and assessment based on machine learning. In 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 1644-1647, IEEE, 2021.
He, H. and Xia, R. Joint binary neural network for multi- label learning with applications emotion classification. In Natural Language Processing and Chinese Computing: 7th CCF International Conference, NLPCC 2018, Hohhot, China, August 26-30, 2018, Proceedings, Part I 7, pp. 250-259. Springer, 2018.
Ekman, P. An argument for basic emotions. Cognition &
emotion, 6(3-4):169-200 1992.
Fabian Benitez-Quiroz, C., Srinivasan, R., and Martinez, A. M. Emotionet: An accurate, real-time algorithm for the automatic annotation of a million facial expressions in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5562-5570, 2016.
Huang, C., Trabelsi, A., Qin, X., Farruque, N., Mou, L.,, and Zaiane, O. R. Seq2emo: A sequence to multi-label emotion classification model. In Proceedings of the 2021 conference of the North American chapter of the asso- ciation for computational linguistics: human language technologies, pp. 4717-4724, 2021.
Feng, I.., Lv, J., Han, B., Xu, M., Niu, G., Geng, X., An, B., and Sugiyama, M. Provably consistent partial-label learning. In Proceedings of the Advances in Neural Infor- mation Processing Systems, pp. 10948-10999 2020.
Jackson, P. and Haq, S. Surrey audio-visual expressed emo- tion (savee) database. University of Surrey: Guildford. UK, 2014.
Ghiasi, G., Gu, X., Cui, Y., and Lin, T.-Y. Scaling open- vocabulary image segmentation with image-level labels. In European Conference on Computer Vision, pp. 540- 557. Springer, 2022.
James, J.. Tian, L., and Watson, C. An open source emo- tional speech corpus for human robot interaction applica- tions. Interspeech 2018, 2018
Godbole, S. and Sarawagi, S. Discriminative methods for multi-labeled classification. In Pacific-Asia conference knowledge discovery and data mining, pp. 22-30. Springer, 2004.
James, W. What is emotion? Mind, 9(34):188 -205, 1884.
Jiang, X., Zong, Y., Zheng, W., Tang, C., Xia, W., Lu, C., and Liu, J. Dfew: A large-scale database for recognizing dynamic facial expressions in the wild. In Proceedings of the 28th ACM International Conference on Multimedia, pp. 2881-2888 2020.
Goodfellow, J.. Erhan, D.,. Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler,

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Jin, P., Takanobu, R., Zhang, W., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700-13710, 2024.
Lian, Z.,, Sun, L., Ren, Y., Gu, H., Sun, H., Chen, L., Liu, B., and Tao, J. Merbench: A unified evaluation bench- mark for multimodal emotion recognition. arXiv preprint arXiv:24011 201.::2001.3342,, 2024b.
Lian, Z. Sun, L., Sun, H.,, Chen, K., Wen, Z., Gu, H., Liu, B., and Tao, J. Gpt-4v with emotion: A zero-shot benchmark for generalized emotion recognition. Information Fusion, 108: 108:102367, 2024c.
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., and Pantic, M. Af database for valence and arousal estimation in-the-wild, wild. Image and Vision Computing, 65:23-36, 2017.
Lin, B., Ye, Y., Zhu, B., Cui, J., Ning, M., Jin, P., and Yuan, L. Video-Ilava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 5971-599788 2024.
Kossaifi, J., Walecki, R., Panagakis, Y., Shen, J., Schmitt, M., Ringeval, F., Han, J., Pandit, V., Toisoul, A., Schuller, B., et al. Sewa db: A rich database for audio-visual emotion and sentiment research in the wild. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 43 (3):1022-1040, 2019.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Li, B., Weinberger, K. Q.. Belongie, S., Koltun, V., and Ranftl, R. Language-driven semantic segmentation. In International Conference on Learning Representations, 2021.
Liu, R., Zuo. H., Lian, Z., Xing, X., Schuller, B. W., and Li, H. Emotion and intent joint understanding in multimodal conversation: A benchmarking dataset. arXiv preprint wwxiv:240.00275. 2024.
Li, B., Zhang, Y., Chen, L., Wang, J., Yang, J., and Liu, Z. Otter: A multi-modal model with in- in-context instruction tuning. Xrxiv preprint WwXiv:2305.03726. 2023a.
Li, K., He, Y., Wang. Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, I.,, and Qiao, Videochat: Chat-centric video understanding. arXiv preprint
Liu, X., Shi, H., Chen, H., Yu, Z., Li, X., and Zhao, G. imigue: An identity-free video dataset for micro- sacro-gesture understanding and emotion analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 10631-10642, 2021.
Ntl.:2305006355, 2023b. Li, K., Wang, Y. He, Y., Li. Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., Wang, L.,, and Qiao, Y. Mvbench: A comprehensive multi-modal video under- standing benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Liu, Y., Dai, W., Feng, C., Wang, W., Yin, G., Zeng, J., and Shan, S. Mafw: A large-scale, multi-modal, compound af- fective database for dynamic facial expression recognition in the wild. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 24 32, 2022a.
Li, S., Deng, W., and Du, J. Reliable crowdsourcing and deep locality-preed preeeering learning for expression recogni- tion in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition. pp. 2852- 2861, 2017.
Liu, Y., Yuan, Z., Mao, H., Liang, Z., Yang, W., Qiu, Y., Cheng, T., Li, X., Xu, H., and Gao, K. Make acous- tic and visual cues matter: Ch-sims v2. O dataset and av-mixup consistent module. In Proceedings of the In- ternational Conference on Multimodal Interaction, pp. 247-258, 2022b.
Li, Y., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323-340. Springer, 2024b.
Lian, Z., Sun, H., Sun, L., Chen, K., Xu, M.. Wang. K., Xu, K., He, Y., Li, Y., Zhao, J., et al. Mer 2023: Multi- label learning, modality robustness, and semi-supervised learning. In Proceedings of the 31 ACM International Conference Multimedia, pp. 9610-9614, 2023.
Liu, Z. Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Zadeh, A., and Morency, L.-P. Efficient low-rank multi- modal fusion with modality-specifie factors. In Proceed- ings of the 56th Annual Meeting of the Association for Computational Linguistics, pp. 2247-2256, 2018.
Lian, Z., Sun, H., Sun, L.,, Wen, Z., Zhang, S., Chen, S., Gu, H., Zhao, J., Ma, Z., Chen, X., et al. Mer 2024: Semi-super learning, noise robustness, and open- vocabulayy multimodal emotion recognition. arXiv preprint Xrv4:22004. 131/3, 2024a.
Livingstone, S. R. and Russo, F. A. The ryerson audio- visual database of emotional speech and song (ravdess): A dynamic, multimodal set of facial and vocal expressions in north american english. PloS One, 1315)e9019639,, 2018.
12

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Lotfian, R. and Busso, C. Building naturalistic emotionally balanced speech corpus by retrieving emotional speech from existing podcast recordings. IEEE Transactions Affective Computing, 10(4):471-43) 2017.
Pfister, T., Li, X., Zhao, G., and Pietikäinen, M. Recognising spontaneous facial miero-expressions. In 20/ interna- tional conference on computer vision, pp. 1449-1456. IEEE, 2011.
Lv, J., Xu, M., Feng, L.. Niu, G., Geng, X., and Sugiyama, M. Progressive identification of true labels for partial- label learning. In international conference machine learning, pp. 6500-650 6510. PMLR, 2020.
Plutchik, R. A general psychoevolutionary theory of emo- tion. Emotion: Theory, research, and experience, 1, 1980.
Plutchik, R. The nature of emotions: Human emotions have deep evolutionary roots, fact that may explain their complexity and provide tools for clinical practice. American Scientist, 8944)3344-350, 2001.
Maas, A., Daly, R. E., Pham, P. Huang, D., Ng. A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 142-150, 2011.
Poria, S., Hazarika, D., Majumder, N., Naik, G., Cambria, E., and Mihalcea, R. Meld: multimodal multi-party dataset for emotion recognition in conversations. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, pp. 527-536, 2019.
Maaz, M., Rasheed, H., Khan, S., and Khan, S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pp. 12585-12602, 2024.
Russell, J. A. A circumplex model of affect. Journal of personality and social psychology, 39(6); 161, 1980.
Martin, O., Kotsia, I., Macq. B., and Pitas, I. The enter- face' 055 audio-visual emotion database. In Proceedings of the 22nd International Conference on Data Engineering Workshops, pp. 1-8. IEEE, 2006.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng. A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, 2013.
McKeown, G., Valstar, M., Cowie, R., Pantic, M., and Schroder, M. The semaine database: Annotated mul- timodal records of emotionally colored conversations be- tween a person and limited agent. IEEE Transactions on Affective Computing, 3(1):5-17, 2011.
Su, Y., Lan, T., Li, H., Xu, J., Wang, Y., and Cai, D. Pandagpt: One model to instruction-follow them all. In Proceedings of the 1st Workshop Taming Large Lan- guage Models: Controllability in the era of Interactive Assistants, pp. 11-23, 2023.
Minsky, M. Society of mind. Simon and Schuster, 1988.
Sun, I.., Lian, Z.,, Liu, B., and Tao, J. Hiemae: Hierarchical contrastive masked autoencoder for self superssee audio- visual emotion recognition. Information Fusion, 108: 102382, 2024.
Mollahosseini, A., Hasani, B., and Mahoor, M. H. Affectnet: A database for facial expression, valence, and arousal computing in the wild. IEEE Transactions on Affective Computing, 10(1):18 31, 2017.
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., MA, Z., and Zhang, C. Salmonn: Towards generic hearing abilities for large language models. In Proceed- ings of the Twelfth International Conference on Learning Representations, 2023.
Morency, L., Mihalcea, R., and Doshi, P. Towards multi- modal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th International Conference Multimodal Interfaces, pp. 169-176, 2011.
OpenAI. URL
Gpt-4v(ision) system card, 2023. pettps:/oopente com/ rewo/eeeeacc/// SLL-Av-system--a card.
Tkachenko, M., Malyuk, M., Holmanyuk, A., and Liubimov, N. Label Studio: Data labeling
software, 2020. URL.
Pang, B., Lee, L., and Vaithyanathan, S. Thumbs up? sen- timent classification using machine learning techniques. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 79- 2002.
heartexlabs /aa.bllleelsui...... Open source soft- ware available from https://github.a https://gitub..om//eettte////// how.h///.w...eeettttaab/aabbbbb. studio.
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z.,, Morency, L.-P., and Salakhutdinov, R. Multimodal transformer for unaligned multimodal language sequences. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, pp. 6558-6569, 2019a.
Pérez-Rosas, V., Mihalcea, R., and Morency, L.-P. Utterance-level multimodal sentiment analysis. Pro- ceedings of the 51st Annual Meeting of the Association for Computational waaimmallinngiinn Linguistics, pp. 973-982, 2013.
13

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Tsai, Y.-H. H., Liang, P. P. Zadeh, A., Moreney, L.-P., and Salakhutdinov, R. Learning factorized multimodal repre- sentations. In Proceedings of the 7th International Con- ference on Learning Representations, pp. 1-20, 2019b.
Zadeh, A. B., Liang, P. P., Poria, S., Cambria, E., and Morency, L.-P. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fu- sion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 2236-2246, 2018b.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Zareian, A., Rosa, K. D., Hu, D. H., and Chang, S.-F. Open- vocabulary object detection using captions. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14393-14402, 2021.
Warriner, A. B., Kuperman, V., and Brysbaert, M. Norms of valence, arousal, and dominance for 13,915 english lem- mas. Behavior research methods, 45:1191-1207, 2013,
Zhang, H.,, Li, X.,, and Bing, L. Video-Ilama: An instruction- tuned audio-visual language model for video understand- ing. In Proceedings of the Conference on Empirical Meth- ods in Natural Language Processing: System Demonstra- tions, pp. 543-553, 2023.
Wöllmer, M., Weninger, F., Knaup. T., Schuller, B., Sun, C.,, Sagae, K., and Morency, I..P.. Youtube movie reviews: Sentiment analysis in an audio-visual context. IEEE Intelligent Systems, 28(3):46-53, 2013.
Zhang, S., Yang, Y., Chen, C.,, Zhang, X., Leng, Q. and Zhao, X. Deep learning- based multimodal emotion recog- nition from audio, visual, and text modalities: A system- atic review of recent advancements and future prospects. Expert Systems with Applications, 237: 121922, 2024.
Wu, C.-H., Lin, J.-C., and Wei, W.-L. Survey on audiovi- sual emotion recognition: databases, features, and data fusion strategies. APSIPA Transactions on Signal and Information Processing, 3, 2014.
Wu, J., Li, X., Xu, S., Yuan, H., Ding, H., Yang, Y., Li, X., Zhang, J., Tong, Y., Jiang, X., et al. Towards open vocabulary learning: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Zhang, Z., Luo, P., Loy, C. C., and Tang, X. From facial ex- pression recognition to interpersonal relation prediction. International Journal of Computer Vision, 126:550-569, 2018.
Xu, Y., Chen, H., Yu, J., Huang, Q., Wu, Z.. Zhang, S.- X., Li, G., Luo, Y., and Gu, R. Secap: Speech emotion captioning with large language model. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 19323-19331, 2024.
Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A., Shi, P., Shi, Y., et al. mplug-owl: Modulariza- tion empowers large language models with multimodality. arXiv preprint arXiv:2304. 200.2334444...... 2023.
Yu, W., Xu, H., Meng, F., Zhu, Y., Ma, Y., Wu, J., Zou, J., and Yang, K. Ch-sims: chinese multimodal sentiment analysis dataset with Ane-grained annotation of modality. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 3718-3727, 2020.
Zadeh, A., Chen, M., Poria, S., Cambria, E., and Morency, L.-P. Tensor fusion network for multimodal sentiment analysis. In Proceedings of the Conference on Empiri- cal Methods in Natural Language Processing, pp. 1103- 1114, 2017.
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cam- bria, E., and Morency, L.-P. Memory fusion network for multi-view sequential learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5634 5641, 2018
14

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
A. Task Comparison
There are various MER tasks. In this section, we provide an in-depth analysis of the differences between our proposed OV-MER and existing tasks.
One-Hot MER (OH-MER) is most widely discussed in the community. In this task, each sample is labeled with only one emotion. We should design a framework to predict the most likely label from a predefined emotional taxonomy (Zhang et al., 2024). Current research primarily focuses on the framework design and multimodal fusion strategy. For the former, research has shifted from traditional classifiers (e.g., SVM (Bishop, 2006)) to deep models (e.g., Transformers (Vaswani et al., 2017)). For the latter, researchers mainly focus on how to align heterogeneous features (e.g.. word alignment (Gu et al., 2018) and implicit alignment (Tsai et al., 2019a)) for subsequent multimodal fusion.
Multi-Label MER (ML-MER) considers the complexity of emotions (i.e., multiple emotions often occur simultaneously) and allows the model to predict multiple labels. The most straightforward solution is to use binary classifiers for each emotion (Godbole & Sarawagi, 2004; He & Xia, 2018). However, this approach is based on the assumption of emotion independence, ignoring the correlations between different emotions. Therefore, Huang et al. (2021) and Deng & Ren (2020) proposed a sequence-to-emotion model, transforming the multi-label emotion recognition task into an emotion sequence prediction task, to further consider the correlations between emotions.
Partial Label Learning MER (PLL-MER) differs from the tasks mentioned above. In PLL-MER, each sample is associated with a set of candidate labels, only one of which is correct (Feng et al.. 2020). Research on PLL-MER can be roughly approaches. The former assumes that each
divided into two categories: average- based methods and identification-base candidate label has an equal probability of being the ground truth (Cour al., 2009), while the latter directly identifies the ground truth and maximizes its estimated probability (Lv et al., 2020).
Fine-Grained MER (FG-MER) differs from the above tasks in the label space. Previous tasks typically rely on coarse- grained emotion taxonomies, such as the widely accepted basic emotion taxonomies, like Ekman (Ekman, 1992) or Plutchik (Plutchik, 1980) emotions. Differently, FG-MER aims to use fine- -re-grained emotion taxonomies to capture more subtle emotions (Demszky et al., 2020). After expanding the label space, FG-MER follows the typical solution of OH-MER.
OV-MER shares some similarities with existing tasks, such as allowing the prediction of multiple labels, like ML-MER. However, OV-MER is fundamentally different from previous tasks:
Task Definition. The main distinction of OV-MER from other tasks lies in its focus on generalizing to unseen or new labels, whereas other tasks operate within a predefined taxonomy, either coarse-grained or fine-grained taxonomies. This is also the reason why use the term "open" in the task name.
Emotional Complexity. Human emotional states are diverse and nuanced. As psychologist Plutchik pointed out, humans can express approximately 34,000 different emotions (Plutchik, 1980). Predefined taxonomies that categorize the full spectrum of emotions into a limited set of labels inevitably overlook some subtle emotional states. In contrast, OV-MER allows the model to understand and predict any emotion, enabling a more accurate modeling of the complex nature of human emotions.
Relationship Between OV-MER and ML-MER. Theoretically, ML-MER can be converted to OV-MER by spanning the label space complete set that includes all the emotion labels in the language (e.g., around 34,000 different emotions (Plutchik, 1980)). However, it is not feasible in practice to construct such a kind of dataset, i.e., annotators need to label each sample with 34,000 different emotions, and it's hard to collect sufficient samples for every emotion category, let alone multiple labels).
Evaluation Metries. In OV-MER, we can use any emotion word describe a person's 's emotional state. Therefore, the test set may contain new emotion labels that are not seen during training. In contrast, traditional methods require the label space in both training and test sets to be strictly consistent. This is why we propose new evaluation metrics for OV-MER.
Solution. Previous tasks rely on predefined label spaces, meaning that the predicted output is a fixed-size M- dimensional vector, where M is the size of the label space. Therefore, previous tasks can be solved using discriminative methods. However, in OV-MER, we do not constrain the label space, making discriminative methods unsuitable. Therefore, we adopt a generative approach, leveraging the rich output vocabulary of LLMSs to construct our solution.
15

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
B. Limitations and Future Work
Firstly, the main contribution of this paper is the definition of a new task and the conduct of foundational research. In the future, we plan to design more effective frameworks to solve OV-MER. Specifically, we will incorporate more emotion- related instruction datasets to fine-tune MLLMs, thereby enhancing their emotion recognition ability. Meanwhile, how to integrate subtitle information and fuse multimodal inputs also plays erucial role. We will also consider these aspects in the framework design. Secondly, we evaluate some MLLMs, but not all models are covered. In the future, we will expand the scope of evaluation to cover more MLLMs to enrich our benchmark. Thirdly, this paper does not involve cultural differences. Specifically, our original data is in Chinese, and the annotators we hired are also native Chinese speakers. In the future, we will also try to extend our method to other cultures and further analyze cultural differences. Fourth, we will focus on fairness-awes aware and unbiased modeling future work. Fifth, OV-MERD is derived from MER2023, which sourced from high-rated movies and TV shows. The high ratings serve as an implicit validation of the actors' performances, ensuring spontaneous and realistic emotional expressions. Currently, this type of dataset is the mainstream in the MER research community, as it provides cost-effective means to expand the dataset scale. In the future, we plan to apply for additional funding to collect data featuring spontaneous, real-life emotional expressions by recruiting participants. Furthermore, we will employ domain adaptation techniques (e.g., domain adversarial neural networks) to address potential domain gaps between different data sources.
C. Detailed Motivation
Video Emotion vs. Facial Emotion. Video emotion is more complex than facial emotion. This is because, in videos, we need to capture subtle changes in the temporal dimension and integrate multimodal clues. Take Figure 1(b) an example. In the temporal dimension, we need to infer a person's 's nervousness based on his stuttering; in the multimodal dimension, we need to combine information from different modalities to gain a more comprehensive understanding of emotion. Due to the complexity of video emotion, using a single label is limiting, and more discrete labels are required to better describe video emotion. This also the motivation behind our OV-MER task.
Label Importance. In OV-MER, we do not assign different levels of importance to each label. Every emotion holds equal significance, and neglecting anyone can impact the performance of downstream tasks. For example, if a human-computer interaction system overlooks any emotion, it may fail to generate appropriate responses.
D. Related Work
Multimodal Emotion Recognition. MER has rapidly developed in recent years (Wu et al., 2014). Current research mainly focuses on building more efficient architectures to achieve higher accuracy on benchmark datasets (Sun et al., 2024). For example, Zadeh al. (2017) proposed a tensor fusion network that addressed the MER task by leveraging interactions among unimodal, bimodal, and trimodal inputs. Tsai et al. (2019a) introduced Transformer-basee model that learned implicit alignment between different modalities and achieved promising results. Lian et al. (2024b) further established Bench involving various features, fusion strategies, and datasets. In emotion recognition, benchmark datasets usually limit the label space to basic emotions and use majority voting to determine the most likely one or more labels (Lian et al., 2023; Li et al., 2017). However, emotional categories extend far beyond basic emotions. Restricting the label space will inevitably overlook some nuanced emotions. To address this issue, we extend traditional MER to OV-MER, which allows for the prediction of any number and category of emotions.
Open Vocabulary Learning. Its main goal is to identify categories beyond the annotated label space (Wu et al., 2024), which has been applied in various fields, such object detection (Zareian et al., 2021), segmentation (Ghiasi et al., 2022), and scene understanding (Li et al., 2021). For example, the object detection dataset COCO (Lin et al., 2014) contains 80 categories, while objects in the real world are nearly infinite. highlighting the importance of open vocabulary learning. This paper makes the first attempt to address MER in an open-vocabulary manner. Compared to other tasks, MER is more difficult as it requires considering multimodal clues and subtle temporal variations.
16

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
E. MER2023 Details
Our dataset utilizes videos sourced from MER2023 (Lian et al., 2023), and the use of this dataset has been consented to by the dataset owners. During the construction of this dataset, MER2023 employs a voice activity detection (VAD) tool to split videos based on the presence or absence of human speech. Subsequently, uses tool to measure speaker similarity and merge consecutive clips from the same speaker, thereby ensuring relatively complete content for each video clip. Afterward, multiple filters are applied to remove videos with inappropriate lengths those with multiple speakers, Finally, most samples in MER2023 are single-person videos with relatively complete speech content.
F. More Examples
Figures 8- 10 provide more examples to visualize the difference between one-hot and OV labels. This paper uses emotion- related descriptions as a bridge for OV label extraction. Since the original video contains real people, we use DemoAl to remove personal information to address copyright concerns. Our OV-MERD dataset is derived from the MER2023 dataset (Lian et al., 2023) with further annotations. Therefore, for the original data, please download the MER2023 dataset.
Subtitle: Goodness, the relationships our 22nd floor are getting more and more complicated. It seems like I'll have to speak less in the future.
One-hot Label: surprise
In the video, there the camera, while the lady on the right is facing the camera. We mainly analyze the emotional state the lady on the right. At the beginning of the video, her facial expression is not positive, and she seems a bit nervous and uneasy. In the following scenes, her eyebrows are raised, indicating that she emphasizing something. the end of the video, she lowers her head and looks down, not looking at other person while speaking. This behavior may stem from inner pressure, anxicty, uncasiness, or frustration, causing her to tend to hide her emotions or avoid communication with others. In the audio, the volume is low, the tone is low, and there is sense of pressure the voice. the text, the subtitle says, "Gooddness, the relationships on our 22nd floor are getting more more complicated. seems like I'll have to speak less in future." Based on the tense and uneasy facial expression, the emphasized movement of raising eyebrows, and the behavior of lowering her head and looking down, it can be inferred that she may be feeling pressure, anxiety, uneasiness, or fiustration. Additionally, based the audio clue describing low volume, low tone, and a sense of pressure the voice, it can be further confirmed that her emotional state is likely negative. Therefore, this sentence expresses her exclamation about the increasing complexity of the relationship on the 22nd floor and her decision speuk less in future, implying her uneasiness and pressure. Description - OV labels: pressure, negative, uneasy, surprised
Description:
ladies in the frame. The lady the left has her back
Figure 8: Examplel.
17

---

OV-MER: Towards Open- Veen-Vocabulary Multimodal Emotion Recognition
Why are you all looking at me like that? So, as long as it's woman, does she have to have a relationship with me?
Subtitle:
One-hot Label: surprise
the video, the sereen shows a male character in an indoor setting, the beginning of the video, his eyes are wide open and his mouth is also open, indicating a surprised facial expression. In the following scenes, he looks around, scemingly explaining narrating something to the people around him. Overall, his emotions audio, the character speaks with a stutter, which usually expresses feelings of nervousness, anxicty, or unease. Combined with the text content, the character seems to be unhappy and angry due the prejudice of the people around him. The subtitle in the text says, "Why you all looking at like that? So, long it's woman, does she have have relationship with me?" This sentence expresses the male character's dissatisfaction and anger towards the people around him. Based on the surprised and negative facial expression of the male character in the video clues, well as the stuttering speech in the audio clues, we character is expressing feeling dissatisfaction and anger in this sentence. may feel troubled by the prejudice of the people around him and is unhappy with this unfair treatment. Description - OV labels: surprise, nervous, dissatisfied
Description:
positive optimistic. In the
infer that the male
Figure 9: Example2.
Subtitle:
Luyuan will never fall into your hands, because he has found high-paying joh.
One-hot Label: angry
Deseription: In the video, the frame includes two women, The woman the left has her back the camera, while the woman the right facing the camera. We mainly analyze the emotional state the woman on the right, At the beginning of the video, she slightly furrows her brows, opens her mouth, and her facial expression appears angry, as if she is engaged in heated argument with the other person. At the same time, she points her finger at the other person and makes motion towards them, which may indicate that she accusing the other person or emphasizing her own viewpaint, Overall, she may be going through a debate or intense conversation, and her emotional state may be one of excitement and anger. She seems to be accusing and expressing her dissatisfaction. In the audio, the tone is aggressive and the character's emotions
excited. Combined with text content, the sense of threat, In the text, the subtitle says, "Luyuan will never fall into your hands, because he has found a high-paying job." This sentence may be an accusation or threat from the woman on the right the woman the left. Based the angry and angry emotions displayed by the woman on the right in the video clues, as well other person, it can be inferred that she accusing the other person or emphasizing her own viewpoint. At the same time, based on the aggressive tone and excited emotions described in the audio clues, as well as the mention in the subtitle that Luyuan has found high-paying job, can inferred that threatening the woman on the left not to interfere with or harm Luyuan. Therefore, this sentence expresses the woman on the right's anger and threatening emotions. Description OV labels: warning, angry, threat
seems to carry
pointing finger and motion towards
sentence may carry sense threat, and the woman on the right may be
Figure 10: Example3.
18

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
G. Summary of Abbreviations
Table 4, we summarize the main abbreviations and their meanings.
Table 4: Summary of main abbreviations.
Category
Abbreviation
Explanation
OH label
One-hot label, the most likely lahel in a limited set of basic emotions.
Label
OV labels
Open-vocabulary labels, a set labels an unlimited label space.
Multimodal Emotion Recognition, which aims to recognize the one-hat emotion label.
MER
Task
OV-MER
Open-vocahulary MER, which aims to identify the OV emotion lahels.
Dataset
OV-MERD
This a dataset we huilt for the OV-MER task.
EW
Emotion Wheel.
Metric
M1, M2, M3
Different grouping strategies based on the emotion wheel.
Precisions, Recalls, Fs
Metrics defined for OV-MER.
LLM
Large Language Model. Large-scale models and only process text.
ALLM
Audio L.M.M Different from L.M.M it can also process audio input.
Model
VLLM
Video LLM. Different from LLM, it can also process video input.
Multimodal LLM. Unlike LLM, it can process at least one more modality (e.g., audio or video). Thus. MLLM includes ALLM and VLLM.
MLLM
Mult--Muti
It uses the checked acoustic and visual clues to generate descriptions.
CLUE-Audio
Different from CLUE-Mult, it only uses checked acoustic clues.
CLUE-Vides Video
Different from CLUE-Multi it only uses checked visual clues.
Text
It only relies on text to generate descriptions.
Description
CLUE-A/T/V
Any of CLUE-Audio, CLUE-Tex and CLUE-Videe
CLUEE-MA/TIV
Any CLUE-Multi, CLUR-Audio, Teut and CLUE-Video.
CLUE-MLLM
It uses the output from MLLM without any manual checking process.
SO, S1, S2
Different CLUE-MILM generation strategies.
19

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
H. Dataset Comparison
This paper introduces a new task, OV-MER, and constructs dataset for this task called OV-MERD, Table 5 compares OV-MERD with existing datasets. The annotation types of these datasets can be broadly categorized into two types: dimensional emotions and discrete emotions. We classify sentiment analysis datasets (e.g., CMU-MOSI) as dimensional datasets because the definition of sentiment intensity overlaps with the valence in dimensional emotions. We observe that OV-MERD contains 236 emotion categories, with most samples having 2 to 4 labels, significantly exceeding the number of labels in existing datasets. In the future, as the scale of the dataset increases, the number of candidate labels can be further expanded. Meanwhile, we would like to emphasize that our OV-MERD is the first dataset that uses the human-LLM collaborative annotation strategy, aiming to provide richer labels to capture more nuanced emotions. We believe this work is an important extension of traditional MER and will contribute to the development of the field.
Table 5: Dataset comparison. In this table, "T", "A", "V", and "T" are abbreviations for image, audio, video, and text, respectively. Some datasets (such as CMU-MOSEI and MSP- (SSP-Podcast) contain both discrete and dimensional emotions.
Datasel
Modality
Annotation Type
Calegories
Labels per Sample
MSP-Podcast (Lotfian & Busso, 2017)
A
Dimensional
3
SST (Socher al., 2013)
T
Dimensional
Cornell (Pang et al., 2002)
T
Dimensional
Large Movie (Maas et al., 2011)
T
Dimensional
ICT-MMMO (Wöllmer ul., 2013)
A.V.T
Dimensional
YouTube (Morency et al., 2011)
V.V.T
Dimensional
MOUD (Pérez-Rosas et al., 2013)
A,V.T
Dimensional
CMU-MOSI (Zadeh ul., 2017)
A.
Dimensional
CMU-MOSEI (Zadeh al., 2018b)
A. A.V.T
Dimensional
CH-SIMS (Yu et al., 2020)
A,
Dimensional
CH-SIMS v2 (Liu al., 20226)
A,
Dimensional
VAM (Grimm et al., 2008)
A.V.T
Dimensional
SEMAINE (McKeown et al., 2011)
A.
Dimensional
5
AFEW-VA (Kossaifi et al., 2017)
A,V.T
Dimensional
2
SEWA(Kossaifi al., 2019)
A.V.T
Dimensional
MSP-Podcast (Lotfian & Busso, 2017)
A
Discrete
8
JL-Corpus (James al., 2018)
Discrete
10
EmoDB (Burkhardi et al., 2005)
A
Discrete
EMOVO (Costantini et al., 2014)
A
Discrete
7
MESD (Duville et al., 2021)
A
Discrete
6
SFEW (Dhall al., 2015)
Discrete
FER-2013 (Goodfellow et al., 2013)
Discrete
7
EmotioNet (Fabian Benitez- Qunreez--uuzzzeeee al., 2016)
Discrete
23
AffectNet (Mollahosseini al., 2017)
Discrete
ExpW (Zhang et al., 2018)
Discrete
7
RAF-DB (Li al., 2017)
Discrete
19
1~2
CMU-MOSEI (Zadeh al., 2018b)
A,V.T
Discrete
eNTERFACE (Martin et al., 2006)
A.V.T
Discrete
6
SAVEE (Jackson & Haq, 2014)
A.V.T
Discrete
7
AFEW 7.0 (Dhall et al., 2017)
A.V.T
Discrete
MAFW (Liu et al., 2022a)
A.V.T
Discrete
11
DFEW (Jiang al., 2020)
A.V.T
Discrete
7
CREMA-D (Cao et al., 2014)
A.V.T
Discrete
MSP- IMPROV (Busso et al., 2016)
A.V.T
Discrete
4
RAVDESS Livingstone & Russo (2018)
A.
Discrete
8
IEMOCAP (Busso
2008)
A,
Discrete
10
MELD (Poria al., 2019)
A.V.T
Discrete
7
MC-EIU (Liu al., 2024)
V.V.T
Discrete
7
MER2023 (Lian al., 2023)
A,
Discrete
MER2024 (Lian al., 2024a)
A.V.T
Discrete
236
1--9, most
OV-MERD (Ours)
A,V,T
Discrete
(arbitrary label)
(arbitrary number)
20

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
I. One-hot vs. OV Labels
This section provides a deeper comparison between the one-hot labels the MER2023 dataset and the OV labels in the OV-MERD dataset. Figure 11 shows the word cloud and label number distribution of OV labels. In Figure [a)), we observe that OV labels cover a wider variety of emotions, some of which (such as shy, nervous, and grateful) are rarely discussed in previous datasets. In Figure 11(b), notice that most samples have about to 4 labels, much more than the traditional task where each sample is assigned only one emotion. Therefore, OV-MER provides richer labels.
care
Hively
grateful S
sad
sadnes
SS
exce
ted
mad
advice nisitvitty t
can
Certint
accuse
a
Slessure
surp 1-1
sea
eereeee Peart tor
wor
ydissa
ydissatist ceaatsfaction
Dlane
posittiv
uestimg Sast
frequency
super
concern compleen
ple sant
geit
ner erruus
resiecconceenee
fur
Sad
fontration
willi
g
disappeintes
accusation
apppiinee
0.5
worr
anger.
fou warning
ldd
ooy uu
del
delighted
negat.
foyfur
a
cearthrent
ques
ques
contused
Sughtty
S
serion ous
anxiety
friendly y
DOS
0.10
803
displessh
displeasure
displeased
e elaxed
000
satisfied amiiedduut
label number
(a) Word cloud
(b) Label number distribution
Figure 11:
Word cloud and label number distribution of OV labels.
In Table 6, report the performance of one-hot labels in OV-MER. We observe that one-hot labels have high precisions but low recalls, indicating that one-hot labels are correct but not comprehensive. Due to the limited label space and the constrained number of labels, one-hot labels cannot cover all emotions, highlighting the limitations of traditional MER and the importance of OV-MER. Additionally, these results reflect the necessity to use Fs for the final ranking, which can balance accuracy and completeness.
Table 6: Performance of one-hot labels OV-MER.
Language
Fs
Precisions
Recalls 1
English
65.71 0.00
92.17
51.05
Chinese
66. 16.16.0000
93.07+0.00
51.32+0.03
Figure 12 shows the emotion distribution of OV labels. We observe that the number of samples for different emotions follows a long-tail distribution. These results indicate that OV labels not only cover some major labels but also capture subtle emotions that occur infrequently.
0.88
frequency
0.00
0.84
0.6
0.00
emotion label
Figure 12: Emotion distribution of OV labels.
21

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
J. Duration Distribution of OV-MERD
In Figure 13, we analyze the duration distribution of the OV-MERD dataset. We observe that the majority of the samples have durations ranging from 1 to 4 seconds. This distribution is consistent with that of the MER2023 dataset, which was used the original dataset for constructing OV-MERD (see Section 2.3 for details).
0.35
Frequency
0.00
Durat
Figure 13: Duration distribution of the OV-MERD dataset.
K. Details in Dataset Construction
Prompts. Figure 2 presents our dataset construction process. In Table 7, we provide prompts and corresponding models used this process, considering various emotion cues (Chen et al., 2023; Liu et al., 2021; Chen et al., 2019; Pfister et al., 2011).
Table 7: Prompts and corresponding models used in the dataset construction process.
Function (Model)
Prompt
As expert in the field of emotions, please focus on facial expressions, body language, environmental cues, and events the video and predict the emotional of the character. Please ignore the character's 's identity. We uniformly sample frames from this video. Please consider the temporal relationship between these frames and provide a complete description this vidco. Avoid using descriptions like "the first image" and "the second image", and instead use terms like "heginning". "middle", and "end" to denote the progression of time. As an expert in the field of emotions, please focus the acoustic information in the audio to discern clues related the emotions of the individual. Please provide a detailed description and ultimately predict the emotional state of the individual.
#1 Pre-label visual clue (VLLM)
#2 Pre-label acoustic clue (ALLM)
Please act expert the field of emotions. We provide acoustic and visual clues that may be related to the character's s emotional state, along with the original subtitle of video, Please analyze which parts can infer the emotional state and explain the reasons. During the analysis, please integrate the textual, audio, and visual clues.
#3 Merge (LLM)
Translation (I.LM)
Chinese=Engliad Please translate the following sentence from Chinese into English. Biglish-Chinese Please translate the following sentence from English into Chinese.
Please assume the role related to the emotions of the characters. Based on the provided clues, please identify the emotional states of the main characters. Please separate different emotional categories with commas and output only the clearly identifiable emotional categories in list format. If none are identified, please output an empty list.
expert the field of emotions. We provide clues that may be
#5 OV label extraction
(L.LM)
Number of Sampled Frames in Pre-annotation To generate pre- pre-annotated visual clues, sample three frames from each video and input them into GPT-4V. In this section, we discuss the rationale behind the choice of the number of sampled frames. Specifically, we categorize visual clues into two types: (1) visual clues with relatively long durations; and (2) visual clues with fast movements, such as eye movements, head movements, and micro-expressions. For the first type, since the duration of most videos is between and seconds (see Appendix J) and the video content is usually continuous with only minor differences between adjacent frames (see Appendix F), uniformly sampling three frames are sufficient to capture this
22

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
information; For the second type, we observe that current MILMs (including the GPT-4V used in this paper) struggle capture these fast movements. Increasing the number of sampled frames does not address this issue. Previous research has also shown that GPT-4V cannot recognize micro-expressions (Lian et al., 2024c). To capture these fast movements, we employ multiple professional annotators to manually add this information.
Merging Process. In this paper, we rely on the powerful reasoning capabilities of LLM for multimodal fusion. Specifically, as shown in Table 7, we ask LLM to integrate textual, acoustic, and visual clues to infer the emotional state. From the output, we observe that LLM can produce reasonable analytical results. However, ambiguities and contradictions in multimodal fusion are inevitable, which is challenging and open problem. In this paper, we simply use LLM address this problem, providing a practical solution for the OV-MER task. However, more effective strategies may exist, such as using more complex prompts or incorporating modality control measures. Therefore, plan to explore this area in the future.
Visualization. In Figure 14, we visualize the output of the main steps.
Subtitle: Why are you all looking me like that? So, long as it's woman, does she have to have relationship with me?
Visual Clue (Prelabeled): In the opening scene, confident, with a slight smile on his lips and eyes directly facing the cumeru, as if he explaining or narrating something. The bac hacckround features computers and office equipment, suggesting that this might be au office or laboratory setting, As the scene progresses to the middle part, the man's 's head turns slightly to one side, and his gaze shifts sideways, seemingly interacting with someone next to him. His smile broadens, and his eyebrows lift slightly, indicating that he might be having a friendly conversution or sharing light-hearted moment. the next scene, the mun's 's expression becomes more exaggerated; his mouth opens wide if expressing surprise emphasizing a point. This kind of faciul expression and body language typically associated with sharing surprising to something. In the final seene. he returns to facing the camera directly, his expression calm once ugain with slight smile, giving the impression that he might have just finished explaining narrating something and feels satisfied or confident. Overall, based on these scenes, we can infer that the man in the video likely conducting some sort of demonstration or explanation. He interacts with the audience, shares information, and possibly expresses surprise at some point. His emotional state throughout video appears be positive, exhibiting confidence and friendly demeanor.
man wearing white lab coat. He looks
reacting strongly
Visual Clue (Checked); In the video. the sereen shows a male character in an indoor setting. At the beginning of the video, his eyes are wide open and his mouth is also open, indicating a surprised facial expression. In the following seenes, he looks around, seemingly explnining or narrating something to the people around him. Overall, his emotions are not positive or optimistic.
Acoustic Clue (Prelabeled); The character in the voice is expressing frustration and annoyance. The tone sarcastic and mocking, and the intonation rising the end of each sentence. There puuses between each sentence, and the stress is certain words to emphasize the frustration and annoyance.
Acoustic Clue (Checked): In the audio, the charucter speaks with a stutter, which usually expresses feelings of nervousness, anxiety, or unease. Combined with the text content, the character soems to be unhappy and angry due to the prejudice
people around him.
CLEE-Mulli: the video, the screen shows male character an indoar setting, At the beginning of the video, his eyes are wide open and his mouth is also open, indicating supprised facial expression. In the following scenes, he looks around, scemingly explaining or narrating something to the people around him. Overall, his emotions are not positive or optimistic. In the audio, the charucter speaks with stutter, which usually expresses feelings of nervousness, anxiety, or unease. Combined with the text content, the character seems to be unhappy and angry due the prejudice the people around him. The subtitle in the says, "Why are you looking at me like that? So, as long it's woman, does she have to have relationship with me?" This sentence expresses the male churacter's dissatisfaction and anger towards the people around him. Based on the surprised and negative facial expression of the male character in the video clues, as well as the stuttering speech in the audio clues, we can infer thut the male character is expressing a feeling of dissatisfaction and anger in this sentence. He may feel troubled by the prejudice of the people around him and is unhappy with this unfair treatment.
Figure 14: An example to visualize the output of the main steps.
23

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
L. Annotation Details
This section presents our annotation guidelines and the layout of the annotation platform. Our annotation process relies on the Label Studio (Tkachenko et al., 2020) toolkit. As shown in Figure 2, there are two parts that require manual checking: 1) the pre-annotated acoustic and visual clues; 2) the merged open-vocabulary labels. To reduce subjective bias, we hire eight annotators who are experts in affective computing and familiar with the definitions of emotions. To maintain high-quality annotations, all annotators must pass a rigorous preliminary test. This test evaluates their performance on 12 samples, each of which was previously annotated by five annotators with full agreement. Annotators who perform poorly are removed from the annotator pool. Additionally, we conduct two rounds of checks with no overlap among annotators in each round. Specifically, in the first round, we randomly select four annotators to check the clues and labels; in the second round, we merge the clues and labels reviewed by the first four annotators and ask another four annotators to perform a second round of checks. Ultimately, we find that these checked clues and labels are well-aligned with the video content.
Figure 15 shows the layout of the annotation platform used for manually checking acoustic and visual clues. During the annotation process, we use the following instructions: We provide pre-labeled acoustic and visual clues. Please manually check these clues, remove errors, and add missing information. On the annotation platform, we design an interface with a time slider, allowing annotators to start playing the video from any frame. This enables annotators to view the entire video during the manual check, helping them better annotate the details that may be missed in pre-annotation.
Nesd zove
M sk.
aoountic
pne-lpaeed visaa sha
The
riglt raght Lunga danig
demm comalie amail
Figure 15: Layout of the annotation platform used for manually checking acoustic and visual clues.
24

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Figure 16 displays the layout of the annotation platform used for manually checking emotional labels. During annotation. use the following instructions: Please select all labels that match the character's emotional state in the "c "candidate emotions". If the provided candidate labels cannot perfectly describe the character's 's emotional state, you can also manually add new labels to the "other emotions" part, Specifically, annotators need to label two parts. First, we list all candidate labels from which annotators can choose what they believe to be the correct labels; second, when the candidate labels cannot perfectly describe the emotions, annotators can manually add additional labels in the "other emotions" part.
Caber
7 SL
Figure 16: Layout of the annotation platform used for manually checking emotional labels.
To illustrate which labels are removed or kept, we provide two examples, each with labels from multiple annotators. To ensure annotation quality, we hire professional annotators who are experts in affective computing and familiar with the definition of emotions. Some of these annotators are members of our team who specialize in affective computing. In Figure 17, as the character doesn't 't know which doctor to see, most annotators provide labels such as confused puzzled. Based on his tone and expression, some annotators further provide labels like anxious and serious. In Figure 18, most annotators notice his disapproval based on the textual content. Combining other modalities, some annotators further note his blame and accuse of what others are planning to do. From these examples, we can observe that these annotators provided relatively reliable labels. However, some annotators may focus only on the most relevant labels and overlook some details. To ensure the comprehensiveness of the annotation results, we merge the labels checked by four annotators. For example, in Figure 17, the final merged labels are troubled, focused, puzzled, anxious, worried, confused, and serious. In the next round, we invite another four annotators for a second check. Through this process, we can ensure that each preserved label is confirmed by at least one annotator in each round, thereby ensuring the comprehensiveness and accuracy of annotation results.
25

---

OV-MER: Towards Vocablocy Multimodal Emotion Recognition
Subtitle: Huh? Which director?
A1: troubled, focuse, puzzled
A2: anxious, worried, confused
A3: confused, puzzled
A4: puzzled, anxious, confused, troubled, serious
Figure 17: Examplel with labels from multiple annotators.
Subtitle: It's not right to do this, even it's for the child's good!
A1: surprised, dislike, disapprove, unexpected, worry
A2: disapprove, serious, worry
A3: surprised, serious, amazed, shocked
A4: disapprove, serious, accuse, blame
Figure 18: Example2 with labels from multiple annotators.
26

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
Inter-annotator Agreement. In this section, caleulate the inter-annotator agreement for two-round checks. Unlike the traditional single-label-based annotation method with fixed label space, OV-MER employs multi-label-based annotation method without fixed label space. Therefore, cannot directly compute the Kappa value between different annotators. To this end, we draw inspiration from Section N and utilize the Jaccard similarity coefficient to measure the inter-annotator agreement. Specifically, assume there are N samples and K annotators. For each pair of annotators Am and An their annotation results for each sample e; are denoted as Ym and Y; respectively. Here, y, and Y contain a set emotion labels. We calculate the agreement score between annotators Am and A, as:
N
Y
Y Y
Similarity m,,
(5)
N
In our annotation process, we hired 8 annotators and conducted two rounds checks, with no overlap among annotators in each round. Table 8 presents the inter-annotate agreement for the first round, and Table presents the inter-annotator agreement for the second round. We observe that through multi-round checks, the inter-annotator agreement gradually increases. These results demonstrate the necessity of multi-round checks, which help enhance label reliability.
Table 8: Inter-annotator agreement in the first round of annotation.
A1
A2
A3
A4
A
1.00
0.57
0.47
0.51
A2
0.57
1.00
0.49
0.48
Ag
0.47
0.49
1.00
0.46
A4
0.51
0.48
0.46
1.00
Table 9: Inter-annotator agreement in the second round of annotation.
Aş
Ag
AA
Ag
A5
1.00
0.66
0.71
0.77
A6
0.66
1.00
0.64
0.67
A7
0.71
0.64
1.00
0.69
As
0.77
0.67
0.69
1.00
27

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
M. CLUE-Multi Analysis
In this section, we further analyze the reliability and comprehensiveness of CLUE-Multi from three aspects: discrete emotion recognition, dimensional emotion recognition, and visual clue statistics. Table 10 provides prompts and models for each part of the analysis.
Table 10: Prompts and corresponding models used in CLUE-Multi analysis.
Function (Model)
Prompt
Please assume the role of an expert in the emotional domain. We provide clues that may be related the emotions of the character. Based the provided clues, identify the emotional states the main characters. We provide set emotional candidates, please rank them in order of likelihood from high to low. The candidate set is [happy, angry. worried, sad, surprise, neutral).
#1 Discrete Emotion Recognition (GPT-3.5)
As an expert in the emotional domain, we provide clues that may be related to the emotions of characters. Bused on the provided clues, please identify the overall pusitive or negative emotional polarity of the main characters. The oulput should be a floating point number ranging from -5 to +5. Here, -5 indicates extremely negative emotions, 0 indicates neutral emotions, and +5 indicates extremely positive emotions. Larger numbers indicate more positive emotions, while smaller numbers indicate more negative emotions. Please provide your judgment as a poatingg-poit number with two decimal places, directly outputting the numerical result without including the analysis process.
#2 Valence Estimation (GPT-3.5)
Please assume the role of an expert in the field of emotions. We provide clues related to the emotions of the characters in the video. Please output the facial movements and body gestures involved in the description, separated by commas. The output format should be in list form.
#3 Visual Clue Analysis (GPT-3.5)
Discrete Emotion Recognition. Our dataset is based on MER2023, which provides relatively reliable one-hot labels. Therefore, we attempt to determine whether these one-hot labels can be identified from CLUE-Multi. This part of the analysis aims to verify whether CLUE-Multi can cover the traditional one-hot emotion recognition task. Experimental results indicate that the top-1 and top-2 scores can reach 93.48 and 96.89, respectively. Further analysis shows that the prediction errors are primarily due to the limitations of one-hot labels. For example, in Figure 1, the character shows a compound emotional state, including surprised, nervous, and unsatisfied. However, when we rank the candidate emotions, the output is: angry, surprised, worried, neutral, sad, happy. The top-1 label is angry, which differs from surprise in MER2023, leading to prediction error. These results reveal the limitations of traditional one-hot labels in describing emotions.
Valence Estimation. Besides discrete labels, MER2023 also provides relatively reliable valence scores. Therefore, we attempt to verify whether CLUE-Multi can be used for valence estimation. Through experimental analysis, we observe that the PCC score between predictions and annotations can reach 0.88, indicating that CLUE-Multi also contains clues for dimensional emotion recognition.
Visual Clue Analysis. Following that, we attempt to analyze the diversity of visual clues in CLUE-Multi. Through experimental analysis, we observe that each sample has an average of 4.95 visual clues. Therefore, we conclude that CLUE- Multi contains a wealth of clues that can help address discrete emotion recognition and valence estimation. Additionally, these results validate the completeness and reliability of CLUE-Multi.
28

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
N. Details of Language Impact Experiments
Experimental Design. In Figure 2, we analyze from two perspectives: 1) the impact of descriptive language (Clue-Multi), and 2) the impact of abstract language (OV labels). The YE to YEC (or Yee to Ycr) experiment aims to keep the descriptive language consistent analyze the effect of abstract language, while the YcE YEE (or Y9C Ycc) experiment aims keep the abstract language consistent analyze the effect of descriptive language.
Jaccard Similarity Coeflicient. Figure 2 uses the Jaccard similarity coefficient to measure the similarity between two sets, which is slightly different from the evaluation metrics defined in Section 3. Specifically, in Section 3, we use the following
metries:
D n |DI
[) N D Fs 2 Precisions + Recalls
Precisions x Recalls
Precisions
Recalls
The motivation for the above metrics is that ) represents the ground truth, while Y represents the prediction. However, in Figure 2, the two sets of emotions considered equally important. As result, we use the Jaccard similarity coefficient to measure the similarity. This metric evaluates the similarity between two sets by comparing the size of their intersection to the size of their union:
|yn
Similarity,
O. Cost of GPT-based Metrics
This paper reports zero-shot performance, focusing only on the inference process. The cost of evaluating our OV-MERD dataset is approximately $1 per evaluation, which may not seem high. However, for future work aimed at training frameworks to better address the OV-MER task, this cost will become prohibitive. For example, if we plan to train a model for 100 epochs, the evaluation cost will rise to $1 x 100 epochs $100. If we intend to test N different parameter combinations and M different frameworks, the evaluation cost will increase to $100 M N. Moreover, we plan expand the OV-MERD dataset in the future. This cost will further increase. Therefore, this paper explores alternatives to GPT-based metries.
29

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
P. Emotion Wheel
The emotion wheel provides psychologically based emotion grouping information. this paper, we select five representative emotion wheels (WI~W5) and use their grouping information for metric calculation. Figure provides more details.
(8) W1
(b) W2
(c) W3
SWAME AVERSION GUI CIIUNITTY LLLSERTALISRAAONNNN INTUITION COURAGE
Hane
Sig
reea
SPAYE
VUUNERABRITY LNTION GRISF EEA CEGGGGG
PRRVIN
powertal
(d) W4
(c) W5
Figure 19: Emotion wheels. This paper selects five representative emotion wheels (please zoom in clearly view the emotional hierarchy): (a) WI (b) W2 (c) W3 (d) W4 (e) W5
30

---

OV-MER: Towards Open-Vocabulay Multimodal Emotion Recognition
Visualization of CLUE-M/A/T/V
Figure 20 provides an example and visualizes CLUE-M/A/TIV.
Why are you looking at me like that? So, as long as woman, does she have have a relationship with me?
Subtitle:
CLUE-Multi: In the video, the screen shows a male character in an indoor setting. At the beginning of the video, his eyes wide open and his mouth is also open, indicating surprised facial expression. In the following seenes, he looks around, seemingly explaining or narrating something the people around him. Overall, his emotions are not positive or optimistic. In the audio, the character speaks with a stutter, which usually expresses feelings of nervousness, anxiety, or unease. Combined with the text content, the character seems to be unhappy and angry due the prejudice of the people around him. The subtitle in the text says, "Why are you looking at me like that? So, as long as woman, does she have to have a relationship with me?" This sentence expresses the male character's dissatisfaction and anger towards the people around him. Based the surprised and negative facial expression of the male character the video clues, as well as stuttering speech the audio clues, we can infer the male character is expressing feeling of dissatisfaction and anger in this sentence. He may feel troubled by the prejudice of the people around him and is unhappy with this unfair treatment.
CLUE-Audio: In the audio, the character speaks with stutter, which usually expresses feelings of nervousness, anxiery, or unease. Combined with the text content, the character seems to be unhappy and angry due to the prejudice of the people around him. In the text, the subtitle content: "Why are you like that? So, long it's woman, does she have have a relationship with me?" This sentence expresses the character's dissatisfaction and anger. The stuttering described in the audio clues, which is usually associated with emotions like nervousness, anxicty, or unease, combined with the subtitle content suggests that the character feels dissatisfied and angry. The character feels dissatisfied with the prejudices those around them and believes that they have certain expectations or demands of them. Therefore, combining the audio clues and the subtitle content, we can infer that the character's emotional state is one of nervousness, anxicty, unease, dissatisfaction, and anger.
all looking
like that? So, long it's woman, does she have to have relationship with me?" From this piece of text, we can infer that the character's emotional state is one of dissatisfaction or anger. The reasoning is as follows: 1. "Why are you all looking like that?" This question expresses the character's dissatisfaction with being stared implies the charucter feels uncomfortable or displeased with others watching him. 2. "So, as long as it's a woman, does she have to have relationship with me?" This statement conveys the character's frustration with others' expectations assumption that there should be a special relationship between him and woman simply because she female. In summary, the text suggests that the charucter is experiencing emotions of dissatisfaction or anger.
CLUE-Text: Subtitle: "Why are you all looking
prejudices. The character feels annoyed or angry about the
In the video, the screen shows male character in indoor setting. At the beginning of the video, his eyes are wide open and his mouth is also open, indicating surprised facial expression. In the following scenes, he looks around, seemingly explaining or narrating something to the people around him. Overall, his emotions not positive optimistic.
CLUE-Video:
Figure 20: Visualization of CLUE-M/A/TI.
31

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
R. Details of CLUE-MLLM
CLUE-MLLM directly utilizes the output from MLLM without any manual checking process. Table 11 provides model cards for different MLLMs. For each MLLM, we provide two types of prompts (see Table 12): one that ignores text and another that considers text. To ensure a fair comparison, use similar prompts for audio, video, and audio-video LLMs.
Table 11: Model cards for MLLMs.
Model
Link
SECap (Xu et al., 2024)
https://giihub..om/fuuuuu www.thhcssshchhcccccppppp
SALMONN (Tang al., 2023)
Qwen-Audio (Chu et ul., 2023)
Otter (Li et al., 2023a)
https//ggitub..om//uuuddaaaaaa
OneLLM (Han al., 2024)
http://ggrhub.coomm/sssaaa//LLLV
PandaGPT (Su et al., 2023)
VideoChat (Li et al., 2023b)
VideoChat2 (Li et al., 2024a)
Video-LLaMA (Zhang et al., 2023)
Video-LLaVA (Lin et al., 2024)
Video-ChatGPT (Maaz al., 2024)
http://ggfhtbbbccommmmmmmaa./ooe CharPP
LLaMA-VID al., 2024b)
mPLL'GOwl (Ye et al., 2023)
Chat-UniVi (Jin al., 2024)
https//gittub.com/PKU- http://tw...ooooottaaaanonnnnaaa
GPT-4V (OpenAI, 2023)
https://openai.com
Table 12: Prompts for extracting emotion-related descriptions using MLLMs.
Model
Text
Prompt
As expert the field of emotions, please focus the acoustic information in the audio to discern clues related to the emotions the individual. Please provide detailed description and ultimately predict the emotional state of the individual.
w/o
Audio LLM
Subtitle content of the audio: (subtitle): As expert in the field of emotions, please focus on the acoustic information and subtitle content in the audio to discem clues related to the emotions of the individual. Please provide detailed description and ultimately predict the emotional state of the individual the audio.
w/
As an expert in the field of emotions, please focus on the facial expressions. body movements, environment, etc., in the video to discern clues related to the emotions of the individual. Please provide detailed description and ultimately predict the emotional state the individual the vidco.
w/o
Video L.M
Subtitle content of the video: (subtitle): As an expert in the field of emotions, please focus the facial expressions. body movements, environment, subtitle content, the video discem clues related to the emotions the individual. Please provide a detailed description and ultimately predict the emotional state of the individual.
w/
As an expert the field of emotions, please focus on the facial expressions, body movements, environment, acoustic information, etc., in the video to discern clues related w/o
to the emotions of the individual. Please provide detailed description and ultimately predict the emotional
of the individual the video.
Audio-Von Vidio L.M
Subtitle content of the video: (subtitle]; an expert in field of emotions, please focus on the facial expressions. body movements, environment, acoustic information, the video discern clues related to the emotions of the individual.
w/
subtitle content, Please provide detailed description and ultimately predict the emotional state the individual in the video.
32

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
This paper tests different CLUE-MLLM generation strategies: SO, S1, and S2. Experimental results shown in Table 13. We observe that $2 generally outperforms both SO and S1. Therefore, adopt S2 as the default strategy.
Table 13: Performance comparison of different strategies for generating CLUE-MLLM.
English
Chinese Precisions
Model
Strategy
Fs
Precisions
Recalls
Fs
Recalls
SO
34.75
40.41
30.48
31.08
35.71
27.51 +0.07
Otter
22.54 05
26.05
19. 86 04
25.06 20.00
29. 29.03
20.99
S2
43.51
50.71
38. 08.009=..om =0000
46.22200.1
52.65
41.18
S0
26.9920.01
29.18.0.08
25. 10
28.
28.70
30.95
26. 26.7620.00
PandaGPT
30.75
36.77
32. 24.14
34.74
37.27 20.15
32.53
S2
45.89+0.20
50.03
42.38
47.33
53.01 40.08
42.75
SO
34.77 0.004
37.66
32.30 20.03
37.62
40.33 20.05
35.25 20025
Video-ChatGPT
S1
41.74
45.59
38. 20.23
40.81 10.05
45.07 40.00
37.2 37.28 +7.288:0.00.
S2
50.52.a0.00
54.03
47.44 47.4422000
54.73
61.15
49.52
28.174:0.26
28.64
27.72
20.10
30.09 20.14
31.34
Eido-LLLMa
S1
34.43
35.82.2020
33. 10
34.01 20..00..0
35.1620.22
32.94
S2
44.73
44.14
25.34
47.26
47.98
46.56+0.01
31.95
31.73
32.17 10.100
34.53.0.02
33.53
35.60
VideoChat
S1
45.10
46.24-20.5
44.01 10.10
44.25
20.09
44.76.0.02
43.75=0016
S2
40
42.90
48.49 20.10
45.57 20.08
47.20
44.05
35.70 06
43.08
30.47.0.00
=0.27
41.16
30.86
VideoCha12
S1
37.56
44.62
32.43
38.7 1:010
45.14
33.88
S2
49.07 20.26
54.72.0.41
44.47 10.15
48. 48.86-0.06
57.12
42.68
SO
39.21 1:0.14
40.56.0.15
37.9 37.94.0.11
20.33
40.4440.24
40.62..0.48
mPLUG-Owl
S1
45.80
47.4940.04
44.22-10.07
47. 47.97
49.33
46.69
S2
52.73
54.54
51.04
50.95 20.06
56.40
46.47:20.18
SO
40.71 40.71
41.38
40.07 20.04
43,45
43.24
43.66+0.16
SALMONN
S1
39.79
39.54
40.05
41 10.13
41.1 10.03
41.76+0.22
S2
47.96 20004
50.20
45.92
48. 48.24
52.24
44.82..0.05
SO
30.64
41.9220000
24. 24.1420000
30.50
40.84
24.33+0.08
Qwen-Audio
S1
35.23
46.69
28.29 10.08
44.09 10.00
58.08 10.00
35.53
38.13
49.42.10.18
31 1:.00000
41. 41.14
53.71 10.00
33.34-20.00
SO
32.6410.00
33.31 33.31
32.00
32.76
33.1910.08
32.33
Video-LLaVA
S1
30.19+0.00
34.10.0.08
27.08 20.06
31. 21.93
33.40
30.58
47.07-20.10
48.58
45. 45.66.2.22
49.2 20.06
53.95
45.23++013
SO
35.14
36.71+0.15
33.69
33.30
33.1240.06
33.48
LLaMA-VID
S1
42.37:20.03
43.97.20.04
40.89 19..9..00
42.56
43.28 10.11
41.8640.00
51.2510.09
52.71
49.8 49.87 10000
52.01 4:0.02
57.30
47.61 1..03
SO
39.89
42.32
37.72+2.18
36.83
37.7440.27
35.96+0.38
Chat-UniVi
S1
47.94 19.19
50.96
45.26 1..11
47. 47.02....00
48.07 10.00
46.01 1.000
52
53.08
53.68
52.50
53.86
58.5410.01
49.8610.03
33

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
S. Cross-linguistic Correlation
Table 14, we leverage the results from Table 2 to compute PCC scores between the English and Chinese results for each metric. Experimental results demonstrate that all metrics exhibit strong cross- moos--inguiisic correlations.
Table 14: PCC between the English and Chinese results for each metric.
Fs
Precisions
Recalls
PCC scores
0.9896
0.9738
0.9817
T. Relationship between Description Length and Label Numbers
This section further discusses the relationship between description length and the number of labels per sample, i.e., whether longer descriptions correlate with more labels. To this end, we compute their PCC scores. We observe that, for the human-only strategy, the PCC score is 0.3416, and for the human-LLM collaboration strategy, the PCC score is 0.2939, Therefore, although from the dataset level, the length of descriptions related to the richness labels (see Figure 4), these two metrics do not show strong correlation at the sample level.
U. Performance of Discriminative MER Methods
This paper primarily focuses on MLLM-based generative models for OV-MER. Traditional discriminative methods are not applied due to fundamental differences in our experimental setup. Specifically, discriminative methods require identical label spaces Y' for both training and testing sets. They cannot predict unseen emotions, i.e., y & y. However, we use an open-vocabulary annotation manner OV-MER, which inherently cannot guarantee alignment between training and testing label spaces (i.e., Virain + Vrest). MLLM-based generative methods offer greater flexibility in emotion prediction, making them better suited for our task. Consequently, we primarily leverage MLLM-based solutions. If forced to discriminative approaches, these models could only predict labels within their training label space Vtrain-
In Table 15, we follow the zero-shot experimental setup commonly used in generative models and report results for discriminative models. Specifically, we train on the IEMOCAP (Busso et al., 2008) (or MELD (Poria et al., 2019)) dataset and evaluate on OV-MERD. For discriminative models, we use CLiP-Large for visual features, HUBERT-Large for acoustic features, and Baichuan- for lexical features, comparing the performance of different classifiers. In this table, the "Attention" model refers to foundation model architecture in MERBench (Lian et al., 2024b). Specifically, let le E Rd, f E Rd., and f! d denote the acoustic, visual, and lexical features for sample I,, respectively. This model first converts all inputs into the same dimension and then computes importance scores for each modality. Subsequently, it employs weighted fusion to obtain multimodal features 2,, which are utilized for emotion prediction.
h = ReLU Wf + 6lh)) m E (a,
(8)
h; Concat (he h1 h?)
(9)
0, = softmax (h, Wa + ba).
(10)
2; h;06.
(11)
Here, Whh E Idd.. xh 6h Rh, Wa e gax1 and by e 3 are trainable parameters. For the output, we have hT" e Rh h4 R 03 R3x1, and E Raa Experimental results in Table 15 show that while discriminative models can be adapted to solve OV-MER, they generally perform worse than MLLM-based generative models.
34

---

OV-MER: Towards Vpen-Vocabulary Multimodal Emotion Recognition
Table 15: Zero-shot performance traditional discriminative models and MILLM-based generative models.
M3-WI
M3-W2
M3-W3
M3-W4
M3-W5
Model
L1
L2
L1
L2
L1
L2
L1
L2
L1
L2
Traditional Discriminative Models
MELD + MFM (Tsai et al., 2019b)
22.28
13.51
21.77
13.51
19.59
17.67
22.10
18.20
16.72
14.82
MELD + MISA (Hazarika ct al., 2020)
28.72
21.75
27.59
22.43
34.31
28.50
26.19
21.80
34.79
29.24
MELD GMFN (Zadeh al., 2018b)
34.28
22.16
33.77
22.47
32.40
29.16
33.43
28.18
29.43
26.50
MELD + MFN (Zadeh et al., 2018a)
31.19
21.57
30.66
21.66
31.26
28.02
32.42
25.54
27.97
24.89
MELD MulT (Tsai et al., 20)
30.74
17.76
30.67
18.45
28.08
23.58
29.89
23.68
24.72
20.79
MELD LMF (Liu al., 2018)
41.47
27.70
40.86
28.43
42.29
37.36
38.54
32.83
40.05
35.16
MELD + TFN (Zadeh et al., 2017)
31.91
20.54
20.56
31.15
26.75
28.41
29.68
25.36
MELD + Attention (1 ian et al., 2024b)
33.61
23.16
32.27
23.42
35.17
30.41
30.88
25.75
33.72
29.53
IEMOCAP MFM (Tsai al., 2019h)
45.46
32.86
47.55
33.12
46.37
39.90
43.03
36.97
43.97
39.28
IEMOCAP MISA (Hazarika al., 2020)
49.14
35.98
48.80
36.53
43.86
47.31
39.82
48.21
43.37
IEMOCAP GMPN (Zadeh et al., 2018b)
49.35
35.85
49.57
36.09
49.18
43.29
46.72
39.28
47.30
42.71
IEMOCAP MPN (Zadeh al., 2018a)
50.56
36.82
50.86
36.72
49.97
44.70
48.69
40.55
48.97
44.11
IEMOCAP MulT (Tsai al., 2019a)
42.67
30.27
43.50
30.79
42.10
37.21
40.75
34.31
41.00
36,55
IEMOCAP + LMF (Liu et al., 2018)
46.34
32.44
46.42
32.94
44.19
39.22
44.23
36.78
43.57
38.57
IEMOCAP TFN (Zadeh al., 2017)
46.13
33.45
46.66
33,91
46.27
41.27
42.31
35.95
45.82
40.69
IEMOCAP Attention (l.ian al., 2024h)
45.64
32.23
46.18
32.31
44.42
39.23
43.40
36.67
43.65
38.49
MLLL-hhased
Generative Models
Qwen-Audio (Chu al., 2023)
43.85
26.60
41.52
26.68
39.46
30.65
36.64
27.33
35.89
29.66
Otter (Li et al., 2023a)
49.75
33.50
49.93
33.04
51.03
37.12
47.54
34.77
50.51
35.54
Video-I.LaMA (Zhang et al., 2023)
52.90
36.08
53.60
35.33
47.50
36.50
52.97
35.78
46.39
34.77
VideoChat (Li al., 20236)
47.79
32.64
47.76
32.14
46.78
34.37
49.53
32.82
45.93
32.85
SHCap (Xu et al., 2024)
52,26
37.55
52.11
37.71
50.77
40.49
50.43
38.21
49.97
40.25
Video-I.LaVA (I.in ct al., 2024)
54.65
37.65
54.54
38.25
52.29
40.58
$2.45
39.91
52.97
39.69
SALMONN (Tang al., 2023)
$4,90
38.93
54.29
37.79
56.25
43.01
50.53
38.54
53.65
42.09
VideoChat2 al., 2024a)
$2.38
36.44
53.56
36,91
52.14
40.57
$0.63
39.64
51.37
39.89
Video-ChatGPT (Maaz et al., 2024)
57.66
41.48
57.37
40.95
55.50
44.15
55.24
42.42
52.93
41.54
I.AMA-VID (I.i al., 2024b)
56.59
41.22
57.49
40.39
55.12
44.06
56.62
42.42
53.03
41.65
mPLUG-OWl et al., 2023)
$7.60
41.32
56.32
40.83
55.67
43.71
$5.06
40.67
54.44
42.00
Chat-UniVi (Jin et al., 2024)
57.00
42,25
57.50
42,43
56,80
45.66
55.86
41.97
55.81
43.61
35

---

OV-MER: Towards Open-Vocabulary Multimodal Emotion Recognition
V. GPTTbased vs. Matching-based Metrics
Table 16 provides raw scores for GPT- and matching- metrics. See Section for more analysis.
Table 16: GPT-based vs. matching-basee based metries. "Ps", "Rs", "B1", "B1", "M', and "R/" are abbreviations for Precisions, Recalls, BLEU1, BLEU4, METEOR, and ROUGE, respectively.
English
Chinese
MILM
L V A
GPT-based
Matching-based
GPT-based
Matching-based
Fs
B1
M
Fs
Rs
B
B₂
R₁
Owen-Audio
X
38.13
49.42
31.04
21.87
06.55
21.65
20.81
41.14
53.71
35.34
27.64
12.07
26.09
25.24
OneLLM
42.84
45.92
40.15
33.81
08.54
28.00
22.46
46.17
52.07
41.47
42.75
16,60
34.42
26.81
Otter
43.51
50.71
38.09
27.26
07.55
23.42
21.05
46.22
52.65
41.18
35.35
14.41
29.34
25.91
Video-LLaMA
44.73
44.14
45.34
28.76
06.41
31.22
20.41
47.26
47.98
46.56
34.88
12.13
37.61
24.25
VideoChat
45.53
42.90
48.49
26.44
05.41
30.58
19.11
45.57
47.20
44.05
31.36
10.86
37.48
22.57
PandaGPT
45.89
50.03
42.38
33.69
07.64
30.29
22.07
47.33
53.01
42.75
43.02
15.83
37.94
26.87
Video-LLLaV
47.07
48.58
45.66
33.48
08.25
29.68
22.34
49.21
53.95
45.23
42.72
15.97
36.87
26.90
SALMONN
47.96
50.20
45.92
31.89
07.19
28.42
20.99
48.24
52.24
44.82
39.00
14.00
35.12
25.35
VideoChat2
49.07
54.72
44.47
31.60
08.10
26.61
21.65
48.86
42.68
41.18
16.15
33.54
26.80
Video-ChaGGPT
X
50.52
54.03
47.44
32.64
07.65
30.25
22.01
54.73
61.15
49.52
41.96
15.50
38.18
26.35
OneLLM
50.52
55.93
46.06
32.19
08.10
28.44
22.25
51.44
56.43
47.26
41.31
15.15
35.15
25.98
ALMMA-VID
V
51.25
52.71
49.87
33.81
08.26
30.31
22.36
52.01
57.30
47.61
43.01
16.23
37.92
27.20
mPLUG-OWl
V X
52.73
54.54
51.04
33.04
07.75
30.24
21.75
50.95
56.40
46.47
41.69
15.16
37.81
26.39
Chat-UniVi
X
53.08
53.68
52.50
32.80
07.83
31.12
22.15
53.86
58.54
49.86
40.76
15.05
38.75
26.43
GPT-4V
V
55.51
48.52
64.86
39.40
18.41
43.67
32.60
57.21
54.61
60.07
45.45
29.08
53.76
40.37
In Table 16, observe that there is strong correlation between the GPTT-based metrics and the matching- metrics. To clarify this point, we use the following three sentences examples:
#1. The clue is "the weather is great". His emotion is "happy".
#2. The clue "the weather bad". His emotion is "sad""
His emotion is "happy".
For matching-based metrics, BLEU, score between #1 and #3 is 0. 0738. Therefore, based the BLEU, score, #1 is closer to #2. For LLM-based metrics, we first extract the emotion labels and compare their similarity, so #1 is closer to #3. This demonstrates that matching-base based metrics are not suitable for evaluating emotion recognition performance.
BLEU, example. The BLEU, score between #1 and #2 is 0.8181, while the
36


Reviews:
[{"review_id": "sUy3C8OCDJ", "reviewer": "Reviewer_J1xd", "summary": "This paper proposes a novel paradigm by integrating the open-vocabulary concept into Multimodal Emotion Recognition (MER), which facilitates emotion prediction without relying on predefined categories. Specifically, the authors introduce a new dataset generated via their proposed CLUE-Multi Generation method, accompanied by novel evaluation metrics and preliminary benchmarks designed to improve MER applicability in real-world scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** CLUE-Multi achieves higher scores than GPT-4V. This performance gap suggests potential methodological issues and undermines the credibility of the results.\n\n**A1:** We believe the reviewer may have misunderstood the results in Table 2. As explained in Section 4.1 and illustrated in Figure 2, CLUE-Multi is the ground truth derived from **manually verified visual and acoustic clues**. In contrast, as shown in Figure 3, GPT-4V belongs to the CLUE-MLLM category, where **it does not utilize manually verified clues**. Therefore, **CLUE-Multi is the upper-bound performance, while GPT-4V is a baseline.** To enhance the clarity, we will revise the manuscript as follows: (1) The caption for the second part will be updated from \"CLUE-MLLM\" to \"CLUE-MLLM (Baselines)\". (2) The caption for the third part will be revised from \"CLUE-M/A/T/V\" to \"CLUE-M/A/T/V (Upper-Bound Performance)\".\n\n**Q2:** Comparison with conventional MER methods.\n\n**A2:** Traditional MER methods are not applied due to fundamental differences in our experimental setup. Specifically, conventional methods require identical label spaces $\\mathcal{Y}$ for both training and testing sets. They cannot predict unseen emotions, i.e., $y \\notin \\mathcal{Y}$. However, we use an open-vocabulary annotation manner for each sample, which inherently cannot guarantee alignment between training and testing label spaces. **Since MLLM-based methods give us more freedom in emotion prediction, our work primarily leverages MLLM-based solutions.** \n\n**If we force the traditional MER approach, these models can only predict labels that belong to their training label space**. To address your concerns, we train on the IEMOCAP (or MELD) dataset and test on OV-MERD, following the zero-shot experimental setup for a fair comparison. For the model architecture, we evaluate both the attention model and MulT.\n\n||M3-W1 L1|M3-W1 L2|M3-W2 L1|M3-W2 L2|M3-W3 L1|M3-W3 L2|M3-W4 L1|M3-W4 L2|M3-W5 L1|M3-W5 L2|\n|-|-|-|-|-|-|-|-|-|-|-|\n|​**Traditional Discriminative Models**|||||||||||\n|MELD+MulT|30.74|17.76|30.67|18.45|28.08|23.58|29.89|23.68|24.72|20.79|\n|MELD+Attention|33.61|23.16|32.27|23.42|35.17|30.41|30.88|25.75|33.72|29.53|\n|IEMOCAP+MulT|42.67|30.27|43.50|30.79|42.10|37.21|40.75|34.31|41.00|36.55|\n|IEMOCAP+Attention|45.64|32.23|46.18|32.31|44.42|39.23|43.40|36.67|43.65|38.49|\n|​**MLLM-based Generative Models**|||||||||||\n|Chat-UniVi|57.00|42.25|57.50|42.43|56.80|45.66|55.86|41.97|55.81|43.61|\n\n**Q3:** Manual checks were performed only for ALLM/VLLM outputs, not for texts generated by LLMs.\n\n**A3:** During our pipeline design, we observed noticeable errors and hallucinations in the outputs from ALLM/VLLM. In contrast, when it comes to the text merging task, given that GPT-series models exhibit impressive performance in reading comprehension [1] (close to human performance) and considering that multi-clue merging is a fundamental function in reading comprehension, we directly adopt the merging results from GPT-series models. This decision balances dataset reliability and construction cost. The user study in our response to Reviewer 2UWh further validates the high quality of LLM-based MER outputs.\n\n[1] Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.\n\n**Q4:** Training, validation, and testing are absent. This omission raises concerns regarding potential overfitting due to insufficient data.\n\n**A4:** We believe the reviewer may have misunderstood some aspects of our work. This paper focuses on establishing a **zero-shot benchmark** for OV-MER. Consequently, all data is used for inference, meaning every sample serves as part of the testing set. **Since our method does not involve any training process, the overfitting problem does not exist at all.** In this paper, our primary contributions are to propose a new task (OV-MER) and lay the groundwork for OV-MER. In our follow-up work (as discussed in Section B. Limitations), we plan to develop more effective frameworks to better address OV-MER.\n\n**Q5:** Meaning of the subscript \"s\".\n\n**A5:** The subscript ``s'' indicates that these metrics are **set-based**, distinguishing them from the traditional **single-label** metrics.\n\n**Q6:** Lack theoretical parts. More suitable for the NeurIPS Datasets and Benchmarks track.\n\n**A6:** This paper goes beyond merely proposing a dataset or establishing benchmarks within existing paradigms. Instead, it represents **a significant task-wise innovation** in the MER field by introducing a novel paradigm that enables more accurate emotion modeling. Additionally, this paper employs a **psychological theory-based method** to address the challenging evaluation problem of OV-MER. Given these substantial contributions to the MER research community, we firmly believe this work meets ICML's standards.", "claims_and_evidence": "Strengths:\n1. The paper is comprehensive and well-organized.\n2. The figures are clear, rich and concise.\n3. Extensive experimental evaluations have been conducted.\n\nLimitations:\n1. The authors introduce a new task, Open-Vocabulary MER (OV-MER), accompanied by a self-constructed dataset and evaluation metrics. However, the details regarding data collection and processing lack sufficient transparency. The authors compare their results, obtained on their dataset, against several general multimodal large language models (MLLMs) without fine-tuning, raising concerns about fairness and validity. For example, GPT-4V achieves scores of 55.51, 48.52, 64.86, 57.21, 54.61, and 60.07 (Table 2), whereas CLUE-Multi obtains significantly higher scores of 80.05, 80.03, 80.07, 85.16, 87.09, and 83.31, respectively. This substantial performance gap suggests potential methodological issues and undermines the credibility of the reported results.\n2. Despite advocating for an open-vocabulary framework, the authors still categorize emotions into 236 classes (Table 5). While understandable for task evaluation purposes, the existence of finite categories means that traditional MER methods (such as [1-2]) could also be evaluated on the constructed dataset. However, the authors have not provided comparative experiments with conventional MER methods. Furthermore, all baselines listed in the main experiments are MLLM-based, neglecting the traditional MER approaches. Such omission significantly limits the robustness and comprehensiveness of the presented analysis.\n\n[1] Multimodal transformer for unaligned multimodal language sequences. ACL 2019.\n\n[2] Decoupled multimodal distilling for emotion recognition. CVPR 2023.\n\n3. During dataset construction, the authors perform manual verification only for labels generated by ALLM/VLLM (\"check×2\"), while textual data generated by LLM (through \"merge & analysis\") undergo no manual checks, as explicitly stated by the authors. Considering the hallucination problem commonly observed in LLM-generated texts, an addition of manual verification for these texts is necessary. The lack of this critical step casts doubt on the reliability and rationality of the dataset construction process.\n\n4. The authors state, \"This dataset is an extension of MER2023 (Lian et al., 2023), from which we randomly select a portion of samples for further annotation.\" However, the description of dataset scale—including dataset size and proportions for training, validation, and testing—is absent both in the dataset description and in Table 1. This omission raises concerns regarding potential overfitting due to insufficient data, thus weakening confidence in the reported experimental outcomes. The vague dataset description fails to convincingly support the reliability of the experimental results.\n\n5.  Regarding the definition of evaluation metrics, the authors introduce a subscript \"s,\" but do not explain its meaning or significance. Clarification of this notation is necessary to ensure clear understanding and reproducibility.\n6. Although I acknowledge the comprehensive nature of this paper and appreciate its overall completeness, I find the heavy reliance on large language models for both dataset and methodological construction lacking sufficient theoretical underpinning—an essential criterion for ICML submissions. Given its current state, this work may be more suitable for submission to the NeurIPS Datasets and Benchmarks track, as it does not fully meet the rigorous theoretical standards expected for ICML (and of course this needs to be evaluated by AC and PC as well).", "methods_and_evaluation_criteria": "Please see the Limitation 1 and 2", "theoretical_claims": "I cannot evaluate the Theoretical Claims of this paper. This is because there are no theoretical claims in this paper.", "experimental_designs_or_analyses": "Please see the Limitation 1 and 2. The experimental designs and results are not convincing.", "relation_to_broader_scientific_literature": "n/a", "essential_references_not_discussed": "please see Limitation 2", "other_strengths_and_weaknesses": "no other Strengths And Weaknesses", "other_comments_or_suggestions": "no other comments or suggestions", "questions_for_authors": "please see Limitations.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Discrimination / Bias / Fairness Concerns", "Inappropriate Potential Applications & Impact  (e.g., human rights concerns)", "Privacy and Security", "Legal Compliance (e.g., GDPR, copyright, terms of use)", "Responsible Research Practice (e.g., IRB, documentation, research ethics, participant consent)"]}}, {"review_id": "WeDc2D35Jo", "reviewer": "Reviewer_2UWh", "summary": "The paper presents a novel paradigm for Open-Vocabulary Multimodal Emotion Recognition (OV-MER), addressing the limitations of existing MER systems that rely on predefined emotion taxonomies. The key contributions include:\n1.\tA new MER paradigm (OV-MER): Unlike traditional MER, which limits emotions to a fixed set of labels, OV-MER enables models to predict emotions beyond predefined categories, allowing for a more nuanced and flexible representation of human emotions.\n2.\tThe OV-MERD dataset: A newly curated multimodal emotion dataset that supports open-vocabulary annotation, leveraging a human-LLM collaboration strategy to improve label richness.\n3.\tNew evaluation metrics: Since OV-MER allows flexible labeling, the paper proposes set-based evaluation criteria, including similarity-based grouping methods and modified precision-recall metrics.\n4.\tBenchmarking and analysis: The paper provides extensive experiments evaluating state-of-the-art multimodal large language models (MLLMs) on the OV-MER task, showing that current models struggle to handle the complexity of open-vocabulary emotion recognition.\nThe study suggests that OV-MER can significantly improve the generalizability of MER systems and facilitate a more human-like emotional understanding in AI applications.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our work. We greatly appreciate your recognition that OV-MER represents a significant advancement in MER systems, enhancing their generalizability and enabling more human-like emotional understanding in AI applications.\n\n**Q1:** While human reviewers refine dataset labels, there is no separate human assessment of final model outputs. A user study evaluating how well OV-MER aligns with human perception would be beneficial.\n\n**A1:** Thanks for your insightful comment. To assess how well OV-MER aligns with human perception, we conducted a user study. Specifically, we hired 9 annotators and randomly selected 20 samples from our dataset. Each annotator was presented with (sample, OV-MERD label) pairs and asked to judge their alignment with human perception using a binary (Yes/No) response format. To ensure annotation quality, we also included inspection data consisting of (sample, incorrect label) pairs. **We observe that 96\\% of the annotations confirm the alignment between OV-MERD labels and human perception. Considering the potential annotator errors, this result demonstrates that our OV-MERD labels align well with human perception.**\n\n\n**Q2:** Ablation studies on the impact of different modalities (text, audio, video) are limited. While results indicate multimodal inputs improve performance, further analysis on their individual contributions would be valuable. \n\n**A2:** Thanks for your suggestion. In Table 2, we observe that CLUE-Video outperforms CLUE-Text, consistent with the nature of our OV-MERD dataset. To be specific, OV-MERD is derived from MER2023, where the textual modality contributes less than the visual modality in emotion recognition [1]. Meanwhile, CLUE-Audio achieves superior performance over both CLUE-Text and CLUE-Video, suggesting that although textual expressions may be ambiguous for emotion recognition, combining them with audio cues can effectively resolve these ambiguities, leading to better performance.\n\n\n[1] Lian, Zheng, Licai Sun, Yong Ren, Hao Gu, Haiyang Sun, Lan Chen, Bin Liu, and Jianhua Tao. \"Merbench: A unified evaluation benchmark for multimodal emotion recognition.\" arXiv preprint arXiv:2401.03429 (2024).\n\n\n**Q3:** The dataset is primarily sourced from movies and TV series, which may not fully represent spontaneous, real-life emotional expressions. A discussion on potential domain adaptation strategies would strengthen the work.\n\n**A3:** OV-MERD is derived from MER2023, which is sourced from high-rated movies and TV shows. **The high ratings serve as an implicit validation of the actors' performances, ensuring spontaneous and realistic emotional expressions.** Currently, this type of dataset is the mainstream in the MER research community, as it provides a cost-effective means to expand dataset scale. **In the future, we plan to apply for additional funding to collect data featuring spontaneous, real-life emotional expressions by recruiting participants. Furthermore, we will employ domain adaptation techniques (e.g., Domain-Adversarial Neural Networks, DANN) to address potential domain gaps between different data sources.** These will be incorporated into the future work section of the revised manuscript.", "claims_and_evidence": "The claims made in the paper are generally well-supported by experiments and analysis.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are well-justified for the problem, particularly:\n•\tDataset Construction: The OV-MERD dataset introduces a novel human-LLM hybrid annotation process, which improves label diversity.\n•\tEvaluation Metrics: The study introduces set-based precision-recall metrics tailored for open-vocabulary tasks, which is appropriate given the nature of OV-MER.\n•\tExperimental Design: The evaluation compares multiple state-of-the-art MLLMs, making the benchmarks robust.\nA potential limitation is that human evaluations (e.g., expert reviews on model-generated labels) are not included, which could provide a more qualitative assessment of OV-MER’s effectiveness.", "theoretical_claims": "The paper does not focus on formal theoretical proofs but rather on empirical evaluation. Therefore, there are no formal proofs to verify.", "experimental_designs_or_analyses": "The experimental design is robust and well-executed, with appropriate baselines. However, two areas could be improved:\n1.\tAblation studies on the impact of different modalities (text, audio, video) are limited. While results indicate multimodal inputs improve performance, further analysis on their individual contributions would be valuable.\n2.\tGeneralization to real-world data is not extensively discussed—datasets are primarily sourced from movies and TV shows, which may not fully capture spontaneous human emotions.", "relation_to_broader_scientific_literature": "The paper builds upon existing MER work but extends it toward open-vocabulary recognition, aligning with recent advances in LLM-driven perception models.", "essential_references_not_discussed": "The paper provides a solid literature review on multimodal emotion recognition (MER) and open-vocabulary learning.", "other_strengths_and_weaknesses": "Strengths\nNovelty: The introduction of open-vocabulary emotion recognition is a significant departure from traditional MER and provides greater flexibility in capturing nuanced human emotions.\nDataset Quality: The OV-MERD dataset is well-constructed, with human-LLM collaboration ensuring diverse and high-quality annotations.\nEvaluation Metrics: The paper carefully designs set-based evaluation metrics, which are more appropriate for open-vocabulary settings than traditional classification metrics.\nComprehensive Benchmarking: The study evaluates state-of-the-art MLLMs and provides thorough performance comparisons, offering valuable insights into current limitations in emotion AI.\nWeaknesses\nLimited real-world validation: The dataset is primarily sourced from movies and TV series, which may not fully represent spontaneous, real-life emotional expressions. A discussion on potential domain adaptation strategies would strengthen the work.\nNo explicit human evaluation of model predictions: While human reviewers refine dataset labels, there is no separate human assessment of final model outputs. A user study evaluating how well OV-MER aligns with human perception would be beneficial.", "other_comments_or_suggestions": "No other comments or suggestions.", "questions_for_authors": "I recommend conducting an ablation study to analyze the individual contributions of text, audio, and video modalities in OV-MER. This would help clarify the relative importance of each modality and provide insights into how multimodal integration enhances emotion recognition.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P2bYon1S7q", "reviewer": "Reviewer_yWC4", "summary": "This paper extends traditional MER and introduces a novel task called open-vocabulary MER (OV-MER). The primary motivation behind this is to expand the scope of emotion recognition to encompass more fine-grained emotion labels. Since OV-MER is a newly proposed task lacking datasets, metrics, and baselines, the authors further construct a dataset (OV-MERD), define metrics (set-level metrics based on GPT and the emotion wheel), and establish baselines (baselines based on LLMs). In summary, this paper extends traditional MER tasks to OV-MER, offering a new research direction in this field.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback and recognition of our contributions to advancing MER research. Your comments on our work are truly valuable to us.\n\n**Q1:** In Table 2, some baseline models include both 7B and 13B versions. Please specify which version is used in the leaderboard.\n\n**A1:** Thank you for your suggestion. In this paper, **we use the 7B model by default**, and we will clarify this in the revised manuscript.\n\n**Q2:** Please explain the reason for reporting both English and Chinese results in Table 2.\n\n**A2:** In Figure 2, we observe that there are certain differences in the labels extracted from different languages. **To study the impact of language differences**, we report results for both English and Chinese descriptions in Table 2.\n\n\n**Q3:** Please discuss the correlation between English and Chinese results.\n\n**A3:** We utilize the results from Table 2 to compute Pearson correlation coefficients (PCC) between the English and Chinese results for each metric. As illustrated in the following table, **all metrics demonstrate strong cross-linguistic correlations.**\n\n| Metric       | Fₛ     | Precisionₛ | Recallₛ |\n|--------------|--------|------------|--------|\n| PCC scores   | 0.9896 | 0.9738     | 0.9817 |\n\n**Q4:** Please explain how the OV-MERD dataset is annotated. Traditionally, due to the fixed label space, MER datasets usually employ multiple annotators and use majority voting to determine the final label. In OV-MER, please explain the method for determining the final label.\n\n**A4:** We appreciate your comment regarding our annotation process. **As detailed in Appendix K, our labeling procedure involved eight annotators familiar with emotion definitions and utilized a two-round verification pipeline.** In the first round, four randomly selected annotators performed independent annotations. In the second round, the labels reviewed by the first group of annotators were merged, and the remaining four annotators conducted a second round of checks. This approach ensures that each preserved label receives confirmation from at least one annotator per round, thereby guaranteeing both the comprehensiveness and accuracy of the annotation results.\n\n\n**Q5:** Please discuss the reasons behind why the GPT-based scores are highly correlated with the EW-based metrics, but less correlated with the matching-based scores?\n\n**A5:** GPT-based and EW-based metrics are two grouping techniques used in our work, focusing on calculating emotion label similarity. In contrast, matching-based scores emphasize word-level matching between two descriptions, including non-emotional words. Consequently, matching-based metrics exhibit lower similarity compared to GPT-based and EW-based metrics.\n\n\n**Q6:** In Table 12, it seems that M*-L2 is always more correlated with the GPT-based metrics than M*-L1. Please explain this further.\n\n**A6:** M*-L1 emphasizes coarse-grained clustering information, whereas M*-L2 emphasizes fine-grained clustering information. The higher correlation between M*-L2 and GPT-based metrics suggests that **GPT-based metrics primarily rely on fine-grained emotion clustering during the metric calculation.**", "claims_and_evidence": "Yes. This paper contains extensive experiments to support its conclusions.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "This is not a theoretical paper.", "experimental_designs_or_analyses": "Yes. Their experiment designs and analyses are clear and sound. This paper first presents the baseline results on OV-MER. Subsequently, it verifies the impact of different annotation methods, different baseline generation strategies, the correlation among different metrics, and the rationality of labels in OV-MERD. Meanwhile, there are extensive experiments in the appendix to explore more aspects.", "relation_to_broader_scientific_literature": "The key contribution of this paper lies in extending traditional MER, which has a fixed label space, to OV-MER. OV-MER encompasses a more diverse range of emotion labels, thereby facilitating a more accurate description of emotions.", "essential_references_not_discussed": "No, essential related work has been correctly cited.", "other_strengths_and_weaknesses": "1. In Table 2, some baseline models include both 7B and 13B versions. Please specify which version is used in the leaderboard?\n2. Please explain the reason for reporting both English and Chinese results in Table 2.\n3. Please discuss the correlation between English and Chinese results.\n4. Please explain how the OV-MERD dataset is annotated. Traditionally, due to the fixed label space, MER datasets usually employ multiple annotators and use majority voting to determine the final label. In OV-MER, please explain the method for determining the final label.\n5. Please discuss the reasons behind why the GPT-based scores are highly correlated with the EW-based metrics, but less correlated with the matching-based scores?\n6. In Table 12, it seems that M*-L2 is always more correlated with the GPT-based metrics than M*-L1. Please explain this further.", "other_comments_or_suggestions": "Please refer to my comments on the weakness part.", "questions_for_authors": "Please refer to my comments on the weakness part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "Not needed."}}, {"review_id": "C03Hhap4Ub", "reviewer": "Reviewer_qUo8", "summary": "This paper proposes a new platform for emotion recognition studies. It extends the previously released MER2023 dataset, where GPT-3.5 is heavily utilized to group emotions meaningfully. For comparison, the evaluation benchmark includes many existing LMMs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** This research aims to overcome the limitations of previous studies that rely on predefined taxonomies to capture more complex, subtle emotions. However, the grouping and emotion recognition performance results are not so different from those of the existing studies. The methods and evaluation criteria are sound and reasonable but limited to identifying L1-level groups, which is not very different from the present classification framework.\n\n**A1:** It seems the reviewer may have some misunderstandings about our work. The L1-level group is just one of several grouping techniques used in this paper. Specifically, Table 2 presents results using GPT-based grouping, Table 3 provides results using L1-level grouping, and Table 12 shows results using L2-level grouping.\n\nIn this paper, our prediction approach follows an open-vocabulary setting, which allows for the recognition of unseen emotions. This is fundamentally different from traditional MER, which relies on fixed taxonomies. However, this flexibility introduces new evaluation challenges. Since there is no predefined label space, the model may predict closely related but differently expressed emotions (e.g., joyful and happy). To address this issue, we grouped similar emotions before computing evaluation metrics. **Thus, the grouping operation is not intended to restrict emotion recognition to L1-level groups. Instead, to align with our open-vocabulary setup, we designed grouping techniques to facilitate performance evaluation. Of course, other evaluation metrics that do not rely on grouping techniques could also be used.**\n\n**Meanwhile, due to the open-vocabulary setting of our OV-MER, we need to design not only new evaluation metrics but also novel dataset construction methods and baselines. These innovations are also key contributions of our work. Therefore, OV-MER is fundamentally different from traditional MER in terms of the task, dataset, evaluation metrics, and solution approach.**\n\n**Q2:** As the authors claim, human emotion includes many different elements. Therefore, emotion recognition should be a \"multi-label\" problem. I strongly recommend that the proposed framework be extended in this direction.\n\n**A2:** Thanks for your comment. **We would like to clarify that OV-MER has already attempted to address MER in a multi-label manner.** As shown in Figure 2 and Appendix E, a single instance can be associated with multiple emotion labels simultaneously. Furthermore, we argue that our proposed OV-MER task encompasses the concept of a multi-label problem but extends beyond it by incorporating flexible and expandable emotion expressions. Consequently, conventional evaluation protocols (e.g., L2 loss) cannot be directly applied to the OV-MER task.\n\n**Q3:** I wonder how much agreement there is among annotators (Kappa value) in the annotation framework.\n\n**A3:** Unlike the traditional single-label-based annotation method with a fixed label space, OV-MER employs a multi-label-based annotation method without a fixed label space. Therefore, we cannot directly compute the Kappa value between different annotators. **Therefore, we draw inspiration from Section M and utilize the Jaccard similarity coefficient to measure the inter-annotator agreement.**\n\nSpecifically, assume there are $N$ samples and $K$ annotators. For each pair of annotators $A_m$ and $A_n$, their annotation results for each sample $x_i$ are denoted as $Y_m^i$ and $Y_n^i$, respectively. Here, $Y_m^i$ and $Y_n^i$ contain a set of emotion labels. We calculate the agreement score between annotators $A_m$ and $A_n$ as:\n\n$$\nSimilarity_{m,n} = \\frac{1}{N}\\sum_{i=1}^{N}\\frac{|Y_m^i \\cap Y_n^i|}{|Y_m^i \\cup Y_n^i|}\n$$\n\nIn our annotation process, we hired 8 annotators and conducted two rounds of checks, with no overlap among annotators in each round (see Appendix K). For the first round, the inter-annotator agreement is shown as follows:\n\n|      | A₁  | A₂  | A₃  | A₄  |\n|------|-----|-----|-----|-----|\n|**A₁**   | 1.00 | 0.57 | 0.47 | 0.51 |\n| **A₂**  | 0.57 | 1.00 | 0.49 | 0.48 |\n| **A₃**   | 0.47 | 0.49 | 1.00 | 0.46 |\n| **A₄**  | 0.51 | 0.48 | 0.46 | 1.00 |\n\nFor the second round, the inter-annotator agreement is shown as follows:\n\n|      | A₅  | A₆  | A₇  | A₈  |\n|------|-----|-----|-----|-----|\n| ​**A₅** | 1.00 | 0.66 | 0.71 | 0.77 |\n| ​**A₆** | 0.66 | 1.00 | 0.64 | 0.67 |\n| ​**A₇** | 0.71 | 0.64 | 1.00 | 0.69 |\n| ​**A₈** | 0.77 | 0.67 | 0.69 | 1.00 |\n\nWe observe that through multi-round checks, the inter-annotator agreement gradually increases. These results demonstrate the necessity of multi-round checks, which help enhance label reliability.", "claims_and_evidence": "This research aims to overcome the limitations of previous studies that rely on predefined taxonomies to capture more complex, subtle emotions. However, the grouping and emotion recognition performance results are not so different from those of the existing studies. LLMs may have learned limited taxonomies (such as those in Emotion Wheels) from literature. While this research provides a more detailed vocabulary set of emotions, it is still unclear what the new findings are in this paper.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are sound and reasonable, but limited to identifying L1-level groups, which is not very different from the present classification framework.", "theoretical_claims": "This paper is theoretically sound.", "experimental_designs_or_analyses": "The experiments are well-designed and the ablation studies are good.", "relation_to_broader_scientific_literature": "This paper challenges universal symbol grounding problems, which are significant for analyzing human behavior. Thus, it relates to the border scientific literature.", "essential_references_not_discussed": "I believe the survey in this paper is thorough.", "other_strengths_and_weaknesses": "This paper proves that collaborative annotation between humans and AI is feasible and fruitful.", "other_comments_or_suggestions": "As the authors claim, human emotion includes many different elements. Therefore, emotion recognition should be a \"multi-label\" problem. I strongly recommend that the proposed framework be extended in this direction.\n\n## update after rebuttal ##\nThank you for the rebuttal.\nI did not change my score.", "questions_for_authors": "I wonder how much agreement there is among annotators (Kappa value) in the annotation framework.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper extends traditional MER to the so-called open-vocabulary MER (OV-MER), in order to expand the scope of emotion recognition to encompass more fine-grained emotion labels. For this new task, the authors further construct a dataset (OV-MERD), define metrics (set-level metrics based on GPT and the emotion wheel), and establish baselines (baselines based on LLMs), thus offering a new research direction in this field.
Decision: Accept (poster)