A Unimodal Valence-Arousal Driven Contrastive Learning Framework for Multimodal Multi-Label Emotion Recognition

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 04:41:51

Abstract: Multimodal Multi-Label Emotion Recognition (MMER) aims to identify one or more emotion categories expressed by an utterance of a speaker. Despite obtaining promising results, previous studies on MMER represent each emotion category using a one-hot vector and ignore the intrinsic relations between emotions. Moreover, existing works mainly learn the unimodal representation based on the multimodal supervision signal of a single sample, failing to explicitly capture the unique emotional state of each modality as well as its emotional correlation between samples. To overcome these issues, we propose a $\textbf{Uni}$modal $\textbf{V}$alence-$\textbf{A}$rousal driven contrastive learning framework (UniVA) for the MMER task. Specifically, we adopt the valence-arousal (VA) space to represent each emotion category and regard the emotion correlation in the VA space as priors to learn the emotion category representation. Moreover, we employ pre-trained unimodal VA models to obtain the VA scores for each modality of the training samples, and then leverage the VA scores to construct positive and negative samples, followed by applying supervised contrastive learning to learn the VA-aware unimodal representations for multi-label emotion prediction. Experimental results on two benchmark datasets MOSEI and M$^3$ED show that the proposed UniVA framework consistently outperforms a number of existing methods for the MMER task.

Main paper: 
 related work 2.1 emotion recognition: 
 Single-Label Emotion Recognition (SLER) is an important task in the field of affective computing. According to input sources, SLER is divided into textual SLER and multimodal SLER. For textual SLER, modeling contextual dependencies has become a widely discussed topic for emotion recognition in conversations [42, 52] . Some works [40, 51] also attempt to model speaker dependencies. Moreover, researchers [17, 34] are interested in improving performance by introducing commonsense and analyzing the speaker's mental states. With the emergence of large language models (LLMs) like ChatGPT [58, 73] , there has been a series of works that combine these LLMs [31, 38, 71] . Recently, the development of multimedia has drawn attention to multimodal SLER [49] . Some researchers focus on the importance of different modalities [8, 27, 63] and challenge of multimodal fusion [22, 25, 46] . Also, some works focus on the field of conversation [18, 23, 32] , and some researchers are focusing on proposing robust approaches [21, 33, 35] .Multi-Label Emotion Recognition (MLER) is a task of identifying one or more emotions in a given text or video. Existing studies can be divided into Textual MLER and Multimodal MLER. Firstly, for Textual MLER [12, 77] , considering intrinsic relations between emotions, Wang and Zong [62] and Huang et al. [24] model emotional dependencies within text representations. Meanwhile, such as Fei et al. [13] and Ma et al. [41] , focus on distinguishing similar labels and learning distinct semantic representations for different labels. Furthermore, Fei et al. [14] consider the prior emotion distribution in sentences and capture the context information relevant to those emotions. Recently, some researches, such as [26, [68] [69] [70] , have begun to delve into Multimodal MLER. Akhtar et al. [1] design a multi-task learning approach to enhance performance of model. Anand et al. [3] propose multimodal distillation loss to improve the generalization ability. Srivastava et al. [53] design multimodal method to understand emotions and mental states of characters in movie scenes.In the field of affective computing, application of multi-dimensional valence and arousal [20, 78] is increasingly widespread due to its ability to provide a more detailed understanding of emotions, compared to discrete emotion categories. To further explore the correlation between discrete and dimensional emotions, several studies have introduced datasets for multi-task learning, such as IEMO-CAP [7] and MER2023 [36] . With these datasets, many multi-task learning methods have been proposed by [9, 47] . Considering the broad application prospects of continuous emotion prediction in real scenarios, various workshops and competitions have been introduced, such as AVEC [50, 59] , MuSe [2] , and ABAW [29] . Moreover, some works have employed the NRC-VAD lexicon [44] as an external knowledge base for the emotion recognition task [64, 75] and the empathetic response generation task [10, 76] . 
methodology: 
 In this section, we first introduce the task definition and the overview of our UniVA framework. We then describe the details of each module in UniVA. Figure 2 shows the overview of UniVA that contains three key modules, i.e., VA Scores Acquisition, VA-Driven Contrastive Learningbased Unimodal Representation, and Multi-Label Prediction with VA-Driven Emotion Correlation Priors. Specifically, we adopt the widely-used VA space [5] , and use either a NRC-VAD lexicon [44] or pre-trained VA models to obtain the VA scores for each emotion category and each modality of the training samples. The second module then leverages the VA scores to construct positive and negative sample pairs for each sample, which are then used to train a supervised constrastive learning model to obtain the VA-aware unimodal representations. Lastly, the third module integrates the unimodal representations and incorporates the correlation prior between emotion categories in the VA space as a regularization term for multi-label emotion prediction.Given an utterance 𝑢 𝑖 and its multi-label annotation 𝑦 𝑖 , we obtain the VA scores for 𝑦 𝑖 and three modalities 𝑢 𝑖 𝑡 , 𝑢 𝑖 𝑣 , 𝑢 𝑖 𝑎 as follows: Label. Given an emotion 𝑦 𝑗 𝑖 of 𝑦 𝑖 , we directly obtain the valence and arousal scores (V 𝑗 𝑒 , A 𝑗 𝑒 ) from the NRC-VAD lexicon [44] , which provides reliable human ratings of valence, arousal, and dominance for 20,000 English terms.Text. We fine-tune a RoBERTabase model [37] on the EmoBank dataset [6] and feed the textual input 𝑢 𝑖 𝑡 into the model for inference. We then obtain a valence score V 𝑖 𝑡 ∈ [-1, 1] and an arousal scoreA 𝑖 𝑡 ∈ [-1, 1]. Vision.For the visual input 𝑢 𝑖 𝑣 , we first extract its facial sequence 𝒔 𝒊 and feed it into the EmoFAN model [56] , which has been trained on the AffectNet dataset [45] . We then obtain a valence score and an arousal score for each face, and average the valence and arousal scores across the facial sequence to derive the overall valence scoreV 𝑖 𝑣 ∈ [-1, 1] and arousal score A 𝑖 𝑣 ∈ [-1, 1]. Audio.For the acoustic input 𝑢 𝑖 𝑎 , we feed it into the Wav2Vec2-Large-Robust model [61] , which was fine-tuned on the MSP-Podcast dataset [39] and has been shown to exhibit excellent generalization and robustness, to obtain a valence scoreV 𝑖 𝑎 ∈ [-1, 1] and an arousal score A 𝑖 𝑎 ∈ [-1, 1].In this subsection, we introduce the details of learning the unimodal representation based on VA-Driven Contrastive Learning.For an utterance 𝑢 𝑖 = 𝑢 𝑖 𝑡 , 𝑢 𝑖 𝑣 , 𝑢 𝑖 𝑎 , we employ existing feature extraction methods to obtain the textual, visual, and acoustic features, i.e., X 𝑡 ∈ R 𝑙 𝑡 ×𝑘 𝑡 , X 𝑣 ∈ R 𝑙 𝑣 ×𝑘 𝑣 , and X 𝑎 ∈ R 𝑙 𝑎 ×𝑘 𝑎 . Here 𝑙 𝑚∈ {𝑡,𝑣,𝑎} denotes the sequence length of each modality, and 𝑘 𝑚∈ {𝑡,𝑣,𝑎} is the feature dimension.Specifically, for the textual input 𝑢 𝑖 𝑡 , we utilize either Glove [48] or RoBERTa [37] to obtain the word representation. For Glove, we directly input 𝑢 𝑖 𝑡 to obtain the text representation X 𝑡 . For RoBERTa, we first concatenate the text from all clips in the current video by inserting special tokens ⟨/𝑠⟩, and then feed them into the model to obtain X 𝑡 .For the audio input 𝑢 𝑖 𝑎 sampled at 16kHz, Wav2Vec2.0 model [11] is utilized to extract low-level acoustic features X 𝑎 .For the video clip 𝑢 𝑖 𝑣 , we first employ the method [74] to extract facial sequence 𝒔 𝒊 = 𝑠 𝑖 1 , 𝑠 𝑖 2 , ..., 𝑠 𝑖 𝑞 , where 𝑞 denotes the total number of faces. These facial images are then fed to the Inception-ResNetv1 model [54] to obtain frame-level visual features X 𝑣 . Intra-Modal Interaction. For the textual modality, we employ a fully connected (FC) layer and additive attention (AddAtt) mapping [4] to obtain the utterance-level representation:EQUATIONwhere H 𝑡 ∈ R 𝑑 𝑡 and 𝑑 𝑡 is the hidden dimension.For visual and acoustic modalities, we respectively feed X 𝑣 and X 𝑎 into two separate Self-Attention (SAT) layers, followed by the additive attention mapping to obtain the utterance-level visual and acoustic representations as follows:EQUATIONwhere H 𝑣 ∈ R 𝑑 𝑣 and H 𝑎 ∈ R 𝑑 𝑎 . 3.But I don't want to shove it down my kid's throat and then... Positive and Negative Sample Construction. For an anchor sample 𝒙 𝑖 , the key question in supervised contrastive learning is how to obtain samples semantically similar to (or different from) 𝒙 𝑖 , which are called positive samples 𝒙 + 𝑖 (or negative samples 𝒙 - 𝑖 ). In previous studies, since SupCon is applied in the single-label classification task, we can obtain positive samples 𝒙 + 𝑖 and negative samples 𝒙 - 𝑖 based on the labels of samples. However, since there are many co-occurred emotions in the MMER task, it is hard to construct 𝒙 + 𝑖 and 𝒙 - 𝑖 based on the emotion labels. The VA space provides a rich, continuous spectrum of emotional states, allowing for a more precise and meaningful categorization of emotional similarity and difference. Therefore, we propose to utilize the VA scores of each modality to construct the positive and negative samples. In this way, positive samples 𝒙 + 𝑖 are not merely those sharing the same categorical label with the anchor, but rather those whose VA scores indicate a close emotional proximity. Conversely, negative samples 𝒙 - 𝑖 are identified through significant divergences in their VA scores from the anchor, reflecting a fundamental emotional disparity. This method acknowledges the multidimensional nature of emotions, recognizing that two samples could share a label (e.g., happy) while embodying different intensities or nuances of that emotion.Specifically, for any modality 𝑚 ∈ {𝑡, 𝑣, 𝑎}, assuming the batch size is 𝐵, we are given two samples 𝑢 𝑖 𝑚 and 𝑢 EQUATIONBased on the similarity score 𝑑, we then determine whether the two samples form a positive or negative pair with a predefined threshold 𝛿. If 𝑑 < 𝛿, 𝑢 𝑖 𝑚 and 𝑢 𝑗 𝑚 are considered as a positive pair; otherwise, they are deemed as a negative pair.To prevent the scenario where a batch consists entirely of negative pairs, we duplicate H 𝑚∈ {𝑡,𝑣,𝑎} and obtain multi-view unimodal representations H𝑚 = [H 𝑚 , H 𝑚 ]. Finally, for each anchor sample 𝒙 𝑖 ∈ 𝑿 ≡ {𝒙 1 , 𝒙 2 , . . . , 𝒙 𝐵 }, the loss of the VA-driven contrastive learning is defined as follows:EQUATIONEQUATIONwhere 𝑃 (𝒙 𝑖 ) = {𝒙 𝑗 ∈ 𝐴(𝒙 𝑖 ) | 𝑑 (𝑢 𝑖 , 𝑢 𝑗 ) < 𝛿, 𝑗 ≠ 𝑖} represents the set of all positive samples paired with anchor 𝒙 𝑖 , 𝐴(𝒙 𝑖 ) ≡ 𝑿 \ {𝒙 𝑖 }, and 𝜏 ∈ R + is a scalar temperature parameter.After obtaining the VA-aware unimodal representations, we concatenate them as the multimodal representation H 𝑚𝑢𝑙𝑡𝑖 , and then feed H 𝑚𝑢𝑙𝑡𝑖 into a softmax layer to obtain the emotion distribution ŷ for multi-label emotion prediction:EQUATIONEQUATIONwhere W and b are learnable parameters.VA-Driven Emotion Correlation Priors. To further capture the intrinsic relationship between emotion categories, we introduce a VA-driven emotion correlation prior 𝜂 for ŷ. Specifically, assuming we have 𝐶 emotions, we first calculate their similarity in the VA space to obtain the emotion similarity matrix M ∈ R 𝐶 ×𝐶 as follows:EQUATIONwhere V 𝑗 𝑒 , A 𝑗 𝑒 and V 𝑙 𝑒 , A 𝑙 𝑒 respectively denote the VA scores of the 𝑗-th emotion and the 𝑙-th emotion. We then incorporate the emotion correlation prior into our model with the following loss 𝜂:EQUATIONwhere 𝑁 represents the number of total samples in the training set.During the training process, we aim to minimize this 𝜂 with the goal of making label predictions on similar emotion positions more similar, and those on dissimilar emotion positions more distinct.For the main MMER task, we use KL divergence [30] as the multilabel loss function:EQUATIONwhere ŷ denotes the model prediction, y denotes the ground truth distribution. The full objective function of our UniVA framework is a combination of the contrastive learning loss, the main task loss, and the emotion correlation prior as follows:EQUATIONwhere 𝜆 is a trade-off parameter. shows the statistics of the samples with multiple labels of both datasets. Moreover, we introduce the three datasets used during the VA scores acquisition phase. EmoBank is a corpus focused on social media, consisting of 10,000 English sentences, each annotated with valence and arousal. AffectNet is a large facial imagery dataset containing over a million images, each face annotated with valence and arousal scores. MSP-Podcast is a speech emotional dataset containing over 150,000 speech segments from podcast recordings, with each segment annotated for valence and arousal scores. Implementation Details. For our UniVA framework, we employ either Glove-300d or RoBERTa-base as the textual encoder. For M 3 ED, we use RoBERTa-base in Chinese 1 . The visual encoder In-ceptionResNet was fine-tuned on the CASIA-WebFace dataset. For the acoustic modality, the acoustic encoder Wav2vec-English 2 used for MOSEI was fine-tuned on the Common Voice 6.1 dataset. Similarly, Wav2vec-Chinese 3 employed for M 3 ED was fine-tuned using the Common Voice 6.1, CSS10, and ST-CMDS datasets. Given an utterance in M 3 ED, since our textual VA model is trained on English corpus, we translate the text into English using DeepL API 4 , and then feed it into the VA model to obtain the textual VA scores. Regarding time overhead of VA models, the textual VA model requires approximately 40 minutes for training on an NVIDIA RTX3090 GPU and then performs inference on the target dataset. As for visual and acoustic VA models, we directly perform model inference, hence incurring negligible overhead. 
experiments: 
 The batch size for MOSEI and M 3 ED is set to 12 and 22, respectively. The learning rate and the hidden size in each modality are set to 5𝑒 -5 and 768. The threshold 𝛿 for Euclidean distance of contrastive learning is set to 0.1. During inference, we set an inference threshold 𝜁 to 0.18 so that the emotion with scores higher than 𝜁 is predicted as 1. Following previous works [69] , we adopt multi-label Accuracy (Acc), Hamming Loss (HL), Micro-F1 (miF1), and Macro-F1 (maF1) scores as our evaluation metrics. We optimize parameters with the AdamW optimizer and train our model on 4 NVIDIA RTX3090 GPUs.We compare the proposed framework UniVA with the following systems: MuIT [57] is a multimodal fusion algorithm that does not require modality-aligned inputs and captures inter-modal interactions with Cross-Modal Transformer. M3ER [43] uses canonical correlational analysis and multiplicative fusion for multimodal emotion 1 https://huggingface.co/hfl/chinese-roberta-wwm-ext 2 https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-english 3 https://huggingface.co/jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn 4 https://www.deepl.com/pro-api?cta=header-pro-api Table 2 : Comparison results of different methods on the MOSEI and M 3 ED datasets. The baselines tagged with ♠ utilize Glove as textual encoders, while those tagged with ♣ employ RoBERTa as textual encoders. Moreover, the baseline tagged with ★ only uses textual and visual modalities, while other models use three modalities. The best results are marked in bold, while the second best results are underlined. recognition. HHMPN [69] models the feature-to-label, modalityto-label, and label-to-label dependencies via heterogeneous graph message passing. TAILOR [70] enhances the multimodal diversity with adversarial learning to obtain the shared and private representations of each modality. RobMMR [16] introduces two adversarial training strategies, temporal masking and parameter perturbation, to learn a more robust multimodal representation. MDI [72] considers emotional dependency of context in dialogues and proposes a dialogue-aware interaction framework. FacialMMT [74] improves the importance of visual modality by extracting the facial sequence of the real speaker in conversations. Gemini 5 [55] is a large multimodal model that exhibits remarkable capabilities in multimodal understanding.EQUATIONNote that since MDI and FacialMMT is designed for the singlelabel emotion recognition task, we replace the Cross-Entropy loss used in these methods with the same KL loss as used in our approach. For Gemini, we first extract five video frames from each video clip, and then feed these into Gemini-Vision to obtain video captions with emotion, which are then concatenated with the textual and spoken content and fed into Gemini to generate one or more emotions from the pre-defined emotion list. The prompt fed to Gemini is shown in supplementary materials.In Table 2 , we report the results of UniVA and all comparison methods on the two datasets.First, we can find that the performance of multimodal fusion methods such as MuIT and M3ER is relatively poor due to their insufficient consideration of the dependencies among emotions. HHMPN and TAILOR achieve better results, because of modeling the both modality-emotion and emotion-emotion dependencies in their models. Moreover, RobMMR which focuses on the model robustness attains on the best performance on the HL metric, while MDI and FacialMMT achieve significant improvements on metrics like Acc and miF1. In addition, the performance of Gemini is rather limited, 5 In this work, the Pro version is used.revealing that existing large multimodal models may not be suitable for the multi-label emotion recognition task due to the complexity of the task. Lastly, it is clear that UniVA-RoBERTa consistently achieves the best performance across all four metrics on both datasets, which demonstrates the effectiveness of our proposed model. Additionally, we find that using Glove instead of RoBERTa as the textual encoder leads to a decrease in performance. When compared with baselines that utilize Glove for text encoding, although slightly inferior to RobMMR and HHMPN on the MOSEI dataset in the HL and maF1 metrics respectively, the proposed UniVA-Glove still achieves certain advantages overall.Effect of Each Component. Firstly, we conduct ablation studies on two main components proposed in UniVA. As shown in Table 3 , removing the VA-driven contrastive learning (VA-CL) results in an average reduction of 1.04 percentage points, especially in Acc and miF1, with an average decrease of 1.55 percentage points and 1.35 percentage points, respectively. It indicates that by capturing the unique emotional state of each modality and the emotional correlations between samples, VA-CL enhances the emotion recognition capability. Furthermore, removing the VA-driven Emotion Correlation Prior (VA-ECP) leads to a decline in performance across all metrics, particularly on the maF1, where there is a decrease of 0.7 percentage points on the MOSEI and 2 percentage points on the M 3 ED. It suggests that VA-ECP strengthens the intrinsic relations between different emotion categories. Effect of Each Modality. We report the results of UniVA of removing each modality in Table 4 . It is evident that removing one or two modalities consistently leads to the performance drop, indicating that each modality is indispensable for emotion prediction. Among the three modalities, we find that the textual modality is much more important than the other two on both datasets. More detailed results are provided in supplementary materials. Effect of Different Contrastive Learning. We compared our proposed VA-CL with Supervised Contrastive Learning (Sup-CL) and Self-supervised Contrastive Learning (Self-CL). In Sup-CL, we 8) on the two datasets. For instance, emotions such as happy and surprise, as well as disgust and anger, exhibit a relatively high positive correlation, while happy and sad, along with neutral and surprise, show a significantly high negative correlation. This aligns with our commonsense understanding of these emotional relationships.Sensitivity Study of Threshold 𝛿. Hyper-parameter 𝛿 determines whether the sample pair is positive or negative. As shown in the Figure 5 , our UniVA achieves the best performance when 𝛿 is set to 0.1; moreover, it is observed that at values of 0.05 and 0.15, the model's performance is approximately the same; additionally, the performance of UniVA gradually decreases as the value of 𝛿 increases beyond 0.15.To better demonstrate the reasonability of the obtained VA scores for each modality, we present four test examples along with predictions from different methods. In Table 6 (a), due to the complexity of the ground-truth emotion labels, both TAILOR and FacialMMT missed one emotion and gave the incorrect prediction; for examples (b) and (c), the emotional tendency displayed by the textual modality is exceedingly apparent. As a result, FacialMMT only accurately predicted the dominant emotion reflected by the text in example (b), while TAILOR made a similar error in example (c); for example (d), all three models predicted correctly. In all cases, our uniVA correctly classified the multi-label emotion categories, which shows the advantage of our framework by leveraging the VA scores from each modality to design unimodal VA driven contrastive learning. Moreover, with VA scores, we can clearly visualize the contribution of each modality to the multimodal multi-label prediction.  
conclusion: 
 In this paper, we proposed a Unimodal Valence-Arousal driven contrastive learning framework (UniVA) for the MMER task. Specifically, UniVA employs pre-trained VA models to obtain VA scores for each modality of all training samples, which are used to construct positive and negative samples for contrastive learning to obtain VA-aware unimodal representations. UniVA then integrates the unimodal representations and incorporates the emotion correlation prior in the VA space for emotion prediction. Experimental results on two datasets show the effectiveness of our UniVA model.  
: 
 expressed by an utterance of a speaker such as happy and angry [16, 26] .Existing studies typically model the MMER task as a multi-label classification problem. One line of work focuses on designing different inter-modal interaction mechanism to obtain the multimodal representation and capturing its dependency on each emotion category [16, 69, 70] . Another line of work focuses on modeling the dependency between the emotion categories by proposing a labelaware Transformer decoder [26] or different multi-label loss functions [3, 16] .Despite obtaining promising results on several benchmark datasets for the MMER task, most existing studies still suffer from several limitation. First, most existing MMER studies [26, 68] represent each emotion category using a one-hot vector and regard it as an independent label, ignoring the intrinsic relationship between different emotion categories. For example, happiness and surprise are often encoded as distinct positive emotions, without considering their shared characteristic of conveying high emotional intensity. Similarly, sadness and boredom are both encoded as distinct negative labels, ignoring their commonality in terms of lower emotional intensity. Second, some studies [19, 66, 70] have recognized the importance of learning a modality-specific representation for each modality, e.g., Yu et al. [66] utilize multimodal annotations to generate unimodal labels. However, they primarily learn the representation of each modality based on the multimodal supervision signals, failing to explicitly capture the unique emotional state of each modality. For example, in Sample 1 of Fig. 1 , although the multi-label ground truth is (disgust, happy) and the visual modality clearly displays happy, the emotion displayed by the textual modality tends towards neutral. Such unimodal emotional state, used to mitigate the polarity of emotions expressed by other modality and to prevent prediction biases caused by overreliance on the polarized emotional modality, is difficult to obtain solely relying on multimodal supervision signals. The work [65] attempted to manually annotate each unimodal label for the single-label emotion recognition task. Nevertheless, it leads to a high cost. Lastly, existing methods mainly focus on learning the multimodal representation with the supervision signal of a single sample, ignoring the emotion correlations between different samples. For instance, as shown in the acoustic modality of Fig. 1 , if two samples have similar emotional states in one modality, their representations in that modality tend to be similar.To address the aforementioned limitations, we propose a Unimodal Valence-Arousal driven contrastive learning framework named UniVA for the MMER task. Specifically, to capture the intrinsic relationship between emotion categories, we adopt the widely-used valence-arousal (VA) space [5] to represent each emotion category with a dimensional valence score and a dimensional arousal score. As illustrated in Figure 1 (a), the valence measures the positivity or negativity of an emotion and the arousal indicates its intensity, which well map emotions in a manner that reflects their inherent similarities or differences. Thus, we derive the correlation between emotion categories from the VA space and use it as priors to learn the emotion category representation. Secondly, to explicitly model the emotional state of each modality, we propose to obtain the valence and arousal scores for each modality based on the unimodal models that are pre-trained on existing VA datasets. By utilizing VA scores, we can gain a detailed understanding of the emotional dynamics among different modalities and how each modality contributes to and influences the multimodal prediction. Moreover, to consider the emotion correlations among different samples, we first measure the similarity between each modality of a pair of training samples based on their unimodal VA scores, and then leverage the similarity score to construct positive and negative sample pairs for each sample. With the positive and negative sample pairs, we apply a supervised contrastive learning model to obtain the VA-aware unimodal representations, and integrate them as the multimodal representation for multi-label emotion prediction.The main contributions in this work can be summarized as follows:• We propose to represent each emotion category with the valencearousal (VA) space to capture the correlation between emotion categories and use it as priors to learn the emotion category representation for the MMER task. • We design a unimodal VA-driven contrastive learning algorithm, which first obtains the VA scores for each modality based on pre-trained models, and then utilize these VA scores to construct positive and negative samples for supervised contrastive learning.  
 
 Appendix:  


Decision: this paper investigates the Multimodal Multi-Label Emotion Recognition (MMER) problem and argues that treating the emotion categories as independent of each other might not be the best practice. in this case, the authors introduce a new framework called UniVA that considers the valence-arousal (VA) space to represent emotion categories and their correlation. on multiple benchmarks, the proposed method achieves competitive performance. 

the reviewers gave initial positive ratings of WA, WA, WA, and A, across all reviewers. there are concerns like the related work, the need for more discussions, especially on directly mapping categorical emotions to dimensional emotions, quality and training data of the pre-trained VA models, impact of the optimization process for the UniVA, visualizations, and additional experiments. with that said, the reviewers are overall happy with the analysis on the current one-hot approach and the new VA framework, the experimental results, and the writing. during rebuttal, the authors mostly addressed the concerns of the reviewers, and the final ratings maintained positive across the board at WA, WA, WA, A. 

since the AC was not able to finish the meta-review in time, the PC stepped in and went through all the reviews and the author feedback. after careful consideration, the PC recommends Accept with Oral presentation.

Reviews: review: paper_summary: paper_summary: It engages to resolve the problems that common emotion recognition works on one-hot labelling but overlook the essential relationships between emotions and overlook the corelations among modalities.
strength_weakness: strength_weakness: The theory is correct technically. The evaluation includes Sota comparisons and ablation study which is also adequate.
limitations: limitations: In my opinion, the references are relatively homogenized such as many references are from ACM MM. Additionally, there are many references which are in pre-print status, which may be considered not very formal.
paper_summary: paper_summary: In this study, the authors propose a Unimodal Valence-Arousal driven contrastive learning framework prompting the development of MMER. It utilizes valence-arousal to capture correlation between emotion classes different from the previous methods.
strength_weakness: strength_weakness: 1. The writing is clear, with accurate descriptions of the proposed framework and precise formulas.
2. A new paradigm for addressing the MMER problem is provided.
3. The experimental results indicate that the proposed method offers a promising performance enhancement.
limitations: limitations: 1.The accuracy of directly mapping categorical emotions to dimensional emotions should be discussed. For some datasets, "Neutral" is classified as low valence, while in others, it is classified as high valence. These inconsistencies and gaps in mapping categorical emotions to arousal and valence should be addressed.
2.Can the experimental comparisons be extended to include methods based on single modalities?
3.Without introducing emotion correlation priors, I am curious about how the feature visualizations would change.
4.Regarding the visualization section, the authors mainly present the representation differences between different modalities. I would like to see the t-SNE visualization of the emotion distribution.
paper_summary: paper_summary: The paper introduces UniVA, a novel framework for Multimodal Multi-Label Emotion Recognition (MMER). The authors argue that existing methods for MMER, which treat each emotion category as independent, often fail to capture the intrinsic relationships between emotions. The UniVA framework addresses this limitation by representing each emotion category in the valence-arousal (VA) space, which allows for the capture of emotion correlations. The framework employs pre-trained VA models to obtain scores for each modality, which are then used in a supervised contrastive learning approach to learn VA-aware unimodal representations. These representations are integrated with a VA-driven emotion correlation prior to multi-label emotion prediction. The authors demonstrate the effectiveness of UniVA on two benchmark datasets, MOSEI and M3ED.
strength_weakness: strength_weakness: (1) The paper presents a unique approach to MMER by utilizing the VA space to represent emotion categories and capture their intrinsic relationships, which is a significant advancement over existing methods.
(2) The use of pre-trained VA models to obtain scores for each modality is an innovative application of contrastive learning for improving emotion recognition.
(3) The framework's design, which includes a VA-driven emotion correlation prior, is a thoughtful integration that considers the complex dependencies between emotion categories.
(4) The authors provide a comprehensive evaluation of UniVA on two benchmark datasets, showing improved performance over several existing methods.
(5) The paper includes detailed ablation studies and in-depth analysis, such as the visualization of VA-aware unimodal representations and the sensitivity study of the threshold parameter, which contribute to a thorough understanding of the framework's performance.
limitations: limitations: (1) What is the impact of the quality and training data of the pre-trained VA models on the final emotion recognition performance?
(2) How does the framework handle imbalanced datasets or emotions with limited samples, and what strategies are employed to address these challenges?
(3) Could the authors provide more details on the computational complexity of the UniVA framework and its scalability to larger datasets or real-time applications?
paper_summary: paper_summary: This paper proposes the method to address the multimodal multi-label emotion recognition problem. The authors propose to use the unimodal valence-arousal label to help the model learn the emotion category representation, which can capture the correlation between samples and emotions. The experimental results can prove the performance.
strength_weakness: strength_weakness: 1. The writing is good, making it easy for readers to understand.
2. In emotion recognition tasks, one-hot label often struggles to adequately represent emotions. Therefore, the work in this paper is highly significant.
3. The experimental section is thorough, effectively validating the proposed methods.
limitations: limitations: The paper presents a clear overall exposition. There are  some further comments:

1. The main analysis of the article focuses on the assistance of valence-arousal in emotion recognition. Yet, MOSEI itself is widely used as a regression dataset. 

1\) Why didn't the authors utilize more commonly used emotion recognition datasets such as IEMOCAP? 

2\) What would happen if use the regression labels of MOSEI? Or when using the method in this paper, what would happen in regression metrics for MOSEI?

2. Given that this paper is addressing a classification task, would using KL divergence as the task loss have any particular effects?

3. 1\)In practical recognition scenarios, if there are clear conflicts between the emotional representations of two modalities, especially if one modality is inconsistent with both the labels and other modalities, can this modality be considered as noise? 

2\) What impact does the optimization process have on this modality within the algorithm of this paper? 

3\) If some denoising methods are adopted or this modality is not used, would it adversely affect the multi-label prediction?