# M2 ER: Multimodal Emotion Recognition Based on Multi-Party Dialogue Scenarios
## 1 Introduction  
Emotions are unique and important forms of human expression [1]. When conducting early research on emotions, [PERSON] [2] classified people's basic emotions according to their needs. In 1977, [PERSON] proposed the concept of emotional computing [3], aiming to equip computers with the ability to recognize, understand, express, and adapt to human emotions. An important direction in emotional computing research is emotion recognition, which can create more intelligent and harmonious user entities for applications such as lie detection, audiovisual monitoring, online conferences, and human-computer interaction (HCI) [4]. Researchers often rely on unimodal emotion recognition [5]. Recently, significant progress has been made in the research of unimodal approaches for text, audio, and video. Particularly, facial emotion recognition (FER) technology has a wide range of applications, including HCI, emotional chat, psychological diagnosis, and other tasks [6]. AffectNet [7] is a widely recognized corpus for video modality emotion recognition. Currently, the top three models in terms of accuracy for seven-class emotion recognition on AffectNet are POSTER++ (67.49%) [8], Emotion-GCN (66.46%) [9], and EmoAffectNet (66.37%) [10]. Other studies related to FER are as follows: [PERSON] et al. [11] created a real-time system that can recognize human faces, assess human emotions, and recommend music to users. [PERSON] et al. [12] proposed a facial image sentiment analysis model based on a CNN. It is discovered that more convolution layers, a strong dropout, a large batch size, and many epochs can obtain better effects. [PERSON] [13] studied lightweight convolutional neural networks (CNNs) for FER task learning and verified the effectiveness of CNNs for FER. [PERSON] et al. [14] utilized Inception-v3, along with additional deep features, to enhance image categorization performance. A CNN-based Inception-v3 architecture was used for emotion detection and classification. In a study by [PERSON] [15], they found that CNNs are highly effective for image recognition tasks due to their ability to capture spatial features using numerous filters. They proposed a model consisting of six convolutional layers, two max-pooling layers, and two fully connected layers, which performed better than decision trees and feed-forward neural networks on the FER-2013 dataset. [PERSON] [16] used a CNN, which extracts geometric and appearance features, and LSTM, which captures temporal and contextual information on facial expressions. This CNN-LSTM architecture allows for a more comprehensive representation of facial expressions by combining spatial and temporal information. [PERSON] et al. [17] presented a facial expression recognition method that included an attention mechanism based on a CNN and LSTM. This model was able to effectively extract information on important regions, better than general CNN-LSTM-based models. [PERSON] [18] focused on reducing intra-class variation in facial expression depth features and introduced a dense convolutional network [19] for the FER task. There has been an increase in the combination of transformers in various FER methods. [PERSON] [20] was the first to use the vision transformer for FER and achieved state-of-the-art results. VTF [21] excels in dealing with facial expression recognition tasks in the wild due to its feature fusion. [PERSON] et al. [22] introduced CrossViT, which uses dual branches to combine image patches of different sizes to produce more reliable features. [PERSON] et al. [23] examined the benefits of pooling layers in ViT, similar to their advantages in CNNs. However, in real-world scenarios, the video modality often presents complex data formats. For example, multiple faces often appear in the same frame in multi-party dialogue scenarios, and the presence of non-speaking individuals' faces can interfere with the final emotion recognition. This is the reason why most of the existing research on multimodal emotion recognition in multi-party dialogues has not utilized the video modality.