A Facial Expression-Aware Multimodal Multi-task Learning Framework for Emotion Recognition in Multi-party Conversations
Year: 2023

Abstract:
Multimodal Emotion Recognition in Multiparty Conversations (MERMC) has recently attracted considerable attention. Due to the complexity of visual scenes in multi-party conversations, most previous MERMC studies mainly focus on text and audio modalities while ignoring visual information. Recently, several works proposed to extract face sequences as visual features and have shown the importance of visual information in MERMC. However, given an utterance, the face sequence extracted by previous methods may contain multiple people’s faces, which will inevitably introduce noise to the emotion prediction of the real speaker. To tackle this issue, we propose a two-stage framework named Facial expressionaware Multimodal Multi-Task learning (FacialMMT). Specifically, a pipeline method is first designed to extract the face sequence of the real speaker of each utterance, which consists of multimodal face recognition, unsupervised face clustering, and face matching. With the extracted face sequences, we propose a multimodal facial expression-aware emotion recognition model, which leverages the frame-level facial emotion distributions to help improve utterance-level emotion recognition based on multi-task learning. Experiments demonstrate the effectiveness of the proposed FacialMMT framework on the benchmark MELD dataset. The source code is publicly released at https: //github.com/NUSTM/FacialMMT.

heading: 1 Introduction
text: Multimodal Emotion Recognition in Multi-party Conversations (MERMC) is a challenging task in the field of multimodal research. The complexity of the task arises from the dynamic and spontaneous nature of human communication in multi-party conversations, which often involves multiple people expressing a variety of emotions simultaneously. In this task, the use of multiple modalities (e.g.,
∗Corresponding authors.
text, audio, and vision) is essential as it allows for a more comprehensive understanding of the emotions being expressed. Among different modalities, visual information usually plays a crucial role as it often provides direct clues for emotion prediction. For example, in Figure 1, without the information from the visual modality, it is hard to determine the anger emotion of the real speaker, i.e., Chandler.
In the literature, most previous MERMC studies primarily focus on the text and audio modalities (Poria et al., 2019; Liang et al., 2020; Mao et al., 2021; Chen et al., 2021), because the visual context in MERMC often involves many people and complex environmental scenes, which may bring much noise to emotion recognition of the real speaker. Owing to the indispensable role of visual modalities, a number of studies explored the potential of visual information in MERMC (Mai et al., 2019; Wang et al., 2022a; Li et al., 2022b; Hu et al., 2022a), which employ 3D-CNNs (Ji et al., 2010; Tran et al., 2015) to extract video features and model the interaction and dependency between consecutive video frames. However, the visual information extracted by these methods still contains much noise from environmental scenes.
To alleviate the visual noise from environmental scenes, several recent studies (Dai et al., 2021; Liang et al., 2021; Hu et al., 2021; Zhao et al.,
15445
2022b) propose to detect all the faces in an utterance based on face detection tools such as MTCNN (Zhang et al., 2016), OpenFace (Baltrušaitis et al., 2016) or pre-trained active speaker detection models (Tao et al., 2021). However, given an utterance, the face sequence extracted by these methods may still contain multiple people, which may mislead the emotion prediction of the real speaker. For example, in Figure 1, there are two persons, Joey and Chandler, with distinct facial expressions, i.e., disgust and anger. Previous methods use the face sequence containing both persons’ faces as visual features, which will inevitably have a negative impact on predicting Chandler’s emotion. Therefore, to fully leverage the visual modality for emotion recognition, it is crucial to extract the face sequence of the real speaker of each utterance.
To this end, we propose a two-stage multimodal multi-task learning framework named FacialMMT for the MERMC task. In the first stage, we design a pipeline solution to obtain the face sequence of the real speaker, which contains three steps: 1) Extract the face sequence containing all possible speakers based on the combination of multimodal rules and an active speaker detection model (Tao et al., 2021); 2) Identify the number of face clusters in the face sequence based on an unsupervised clustering algorithm named InfoMap (Rosvall and Bergstrom, 2008); 3) Perform face matching and choose the face sequence with the highest confidence as the face sequence of the real speaker. Table 1 illustrates the differences between our method and previous
methods.
Based on the extracted face sequence, in the second stage, we further propose a Multimodal facial expression-aware multi-task learning model named MARIO. MARIO first resorts to an auxiliary framelevel facial expression recognition task to obtain the emotion distribution of each frame in the face sequence. Next, the emotion-aware visual representation is then integrated with textual and acoustic representations via Cross-Modal Transformer (Tsai et al., 2019) for utterance-level emotion recognition.
Our main contributions can be summarized as follows:
• To obtain the face sequence of the real speaker in an utterance, we propose a face sequence extraction method, which consists of three steps, i.e., multimodal face recognition, unsupervised face clustering, and face matching.
• We propose a Multimodal facial expressionaware multi-task learning model named MARIO for the MERMC task, which leverages an auxiliary frame-level facial expression recognition task to obtain the frame-level emotion distribution to help utterance-level emotion recogntion.
• Experimental results on a benchmark dataset MELD demonstrate the superiority of our proposed FacialMMT framework over the SOTA systems. Moreover, FacialMMT outperforms a number of SOTA systems with a significant margin when only visual modality is used.

heading: 2 Method

heading: 2.1 Task Formulation
text: Given an MERMC corpus D, let us use {X1, X2, . . . , X|D|} to denote a set of samples in the corpus. Each sample contains a multimodal dialogue with n utterances d = {u1, u2, . . . , un}, in which each utterance ui = {uil, uia, uiv} contains information from three modalities, i.e., text, audio, and vision, denoted by {l, a, v}. The goal of the MERMC task is to classify each utterance ui into one of C pre-defined emotion types yi, and predict a label sequence y = {y1, y2, . . . , yn} for d. Note that each utterance is only annotated with one speaker’s identity (real speaker) and his/her emotion is annotated as the emotion of the current utterance.

heading: 2.2 Framework Overview
text: As shown in Figure 2, our FacialMMT framework contains two stages. To obtain the face sequence of the real speaker in each utterance, the first stage introduces a pipeline method to perform multimodal face recognition and unsupervised clustering, followed by face matching. With the extracted face sequences, the second stage resorts to an auxiliary frame-level facial expression recognition task to generate the emotion distribution for each frame in the face sequence, and then employs Cross-Modal Transformer to integrate the emotion-aware visual representations with text and acoustic representations for multimodal emotion recognition.
We will present the details of the two stages in the following two subsections.

heading: 2.3 Face Sequence Extraction
text: As shown in the left side of Figure 2, the first stage extracts the face sequence of the real speaker based on the following three steps:
Multimodal Face Recognition. First, we propose to combine multimodal rules and an active speaker detection (ASD) model to extract face sequences of all possible speakers. Specifically, given a video utterance, we use a pre-trained ASD model TalkNet (Tao et al., 2021) to combine visual and audio information for speaker detection. However, TalkNet often fails to identify speakers for videos with short duration or complex scenes (e.g., for a video with multiple people, someone is laughing or making noise instead of speaking). To obtain the face sequence in these videos, we further design several multimodal rules, including the opening and closing frequency of mouth, movement of different people’s mouths between video frames, and the alignment between the mouth movement and audio signals. The details of these multimodal rules are described in Appendix A.1.
Unsupervised Clustering. Based on the raw face sequence, we apply an unsupervised clustering algorithm InfoMap (Rosvall and Bergstrom, 2008) to identify the number of face clusters in the sequence as follows: • We first employ the K-Nearest Neighbors algo-
rithm to construct a graph of all potential speakers’ faces, and then calculate the similarity between faces, followed by using the normalized as the weight of edges.
• Random walks are then conducted on the graph
to generate different face sequences.
• Lastly, we hierarchically encode the face sequences, and minimize the minimum average encoding length to obtain the clustering result.
The minimization process includes minimizing the average encoding length of classes, as well as the average encoding length of each class’s in-class objects. The formulation is defined as follows:
argmin K,Y
L (P,K, Y ) = q↷ ( − K∑
i=1
qi↷ q↷ log qi↷ q↷
)
+ K∑
i=1
pi⟲ ( −qi↷ pi⟲ log qi↷ pi⟲ − ∑
α∈i
pα pi⟲ log pα pi⟲
) (1)
where Y is the predicted face sequence category, K represents the number of face sequences, qi↷ represents the probability of the occurrence of category i, q↷ = ∑K i=1 qi↷, pα represents the probability of the occurrence of a face image α, and pi⟲ = qi↷ + ∑K α∈i pα.
Face Matching. Finally, we construct a face library to determine the face sequence of the real speaker. Because the benchmark dataset for the MERMC task, i.e., MELD (Poria et al., 2019), contains six leading roles occurring frequently in the dataset, we manually select 20 different face images for each leading role based on the raw face sequence extracted in Multimodal Face Recognition and regard these 120 images as the face library. Next, we use a ResNet-50 model (He et al., 2016) pre-trained on a face recognition dataset MS-Celeb1M (Guo et al., 2016) to extract visual features for the images in the library and in different face clusters. As each utterance provides the real speaker’s identity, who is either one of six leading roles or a passerby, we match the images in each face cluster with six leading roles’ images in the library by calculating the cosine similarity between their visual representations. Specifically, if the identity of the real speaker is one of the six leading roles, the face sequence with the highest similarity is regarded as the real speaker’s face sequence; otherwise, we regard the face sequence with the lowest similarity as the real speaker’s face sequence.

heading: 2.4 A Multimodal Facial Expression-Aware Multi-Task Learning Model
text: After obtaining the real speaker’s face sequence in each utterance, we further propose a Multimodal facial expression-aware multi-task learning model (MARIO), as shown in the right side of Figure 2.
Next, we introduce the details of MARIO, including unimodal feature extraction, emotion-aware visual representation, and multimodal fusion.

heading: 2.4.1 Unimodal Feature Extraction
text: In the MERMC task, given an utterance ui, we extract unimodal features from three modalities {uil, uia, uiv} to obtain the text, audio, and visual representations as follows:
• Text: To efficiently utilize the dialogue context and speaker’s emotional dynamics, we concatenate the input utterance and all its contextual utterances as input, and feed it into a pre-trained language model (e.g., BERT) for fine-tuning. We then take out the hidden representation of the first token as the text representation El ∈ Rdl , where dl = 512 is the size of text features.
• Audio: We obtain the word-level audio representation based on the Wav2vec2.0 model (Baevski et al., 2020) pre-trained on the Librispeech-960h dataset (Panayotov et al., 2015), denoted by Ea ∈ Rda , where da = 768 is the dimension of audio features.
• Vision: Given the real speaker’s face sequence of the input utterance, we use an InceptionResNetv1 model (Szegedy et al., 2017) pre-trained on the CASIA-WebFace dataset (Yi et al., 2014) to obtain the frame-level visual representation Ev ∈ RL×dv , where L is the face sequence length and dv = 512 is the size of visual features.

heading: 2.4.2 Emotion-Aware Visual Representation
text: Because the goal of MERMC is to predict the emotion of all the utterances in a dialogue, we propose to enhance the frame-level visual representation with the emotion distribution of each frame. To achieve this, we introduce an auxiliary framelevel facial expression recognition task, which is known as Dynamic Facial Expression Recognition (DFER) in the computer vision community (Li and Deng, 2020). Formally, let Ds be another set of samples for the DFER task. Each sample is a face sequence containing m faces, denoted by s = {s1, s2, . . . , sm}. The goal of DFER is to predict the label sequence z = {z1, z2, . . . , zm}, where each label zi belongs to one of C pre-defined facial expressions (i.e., emotion categories).
Auxiliary DFER Module. As shown in the top right of Figure 2, we employ a well-known SwinTransformer model (Liu et al., 2021) pre-trained
on the Ms-Celeb-1M dataset (Guo et al., 2016) to obtain the representation of each frame in the face sequence as follows:
Hs = {hs1, · · · ,hsm} = Swin-Transformer(s) (2) where Hs ∈ Rm×du is the generated facial features. Next, we feed Hs into a multi-layer perceptron (MLP) layer for facial expression recognition. During the training stage, we use cross-entropy loss to optimize the parameters for the DFER task:
p(zi) = softmax(MLP(h s i )) (3)
LDFER = − 1 M
M∑
i=1
m∑
j=1
log p(zij) (4)
where M is the number of samples in Ds. Facial Expression Perception for MERMC. Based on the auxiliary DFER module, a direct solution to obtain the emotion-aware visual representation is to convert the predicted emotion of each frame to an one-hot vector and concatenate it with its original representation as the frame-level visual representation. However, as we all know, the one-hot vector derived from the argmax function is not differentiable, which will affect the parameter optimization in our multi-task learning framework.
To tackle this issue, we apply Gumbel Softmax Jang et al. (2017), which has a continuous relaxed categorical distribution, to obtain an approximated emotion distribution for each frame. By using softmax as the differentiable approximation of argmax and adding a temperature function τ , it can achieve gradient updates during backpropagation:
gi = softmax((g + h s i )//τ) (5)
where gi ∈ RC , g = − log(− log(u)) is a noise sampled from the Gumbel distribution, and u ∼ Uniform(0, 1). As τ → 0, the softmax computation smoothly approaches the argmax, and the sample vectors approximate one-hot vectors.
Moreover, if the emotion distribution of the i-th frame in the face sequence concentrates on certain emotion, it shows that this frame reflects the clear emotion; otherwise if the emotion distribution is uniform, it implies the emotion in this frame is blurred and may bring noise to our MERMC task. To alleviate the noise from the emotion-blurred frames, we design a gating mechanism to dynamically control the contribution of each frame in the face sequence for the MERMC task. Specifically,
the emotion clarity of the i-th frame can be computed as δi = gi · g⊤i , where · denotes the dot product. Based on this, we can obtain the emotion clarity of all the frames in the face sequence:
δ = {δ1, δ2, · · · , δm} (6) We then apply δ to the original visual representation Ev to filter out the emotion-blurred frames, in which δi is less than a predetermined threshold.
Finally, we concatenate the filtered visual representation E ′ v and the emotion distributions of all the frames Ee to obtain the emotion-aware visual representation as follows:
Êv = E ′ v ⊕Ee, Ee = {g1, · · · ,gm′} (7)
where Êv ∈ Rm ′×(dv+C), m ′ is the number of filtered frames, and ⊕ is the concatenation operator.

heading: 2.4.3 Multimodal Fusion
text: Intra-Modal Interactions. We feed Ea and Êv to two separate self-attention Transformer layers (Vaswani et al., 2017) to model the intra-modal interactions within audio features and visual features as follows:
Ha = Transformer(Ea),Hv = Transformer(Êv)
Inter-Modal Interactions. To achieve interactions between different modalities, we apply the Cross-Model Transformer (CMT) layer (Tsai et al., 2019). Firstly, we fuse the text and audio modalities, alternating the two modalities as the query vector, then concatenating them to obtain the textaudio fused representation Hl−a. Similarly, Hl−a is then fused with visual modality to obtain the utterance-level text-audio-visual fused representation Hl−a−v below:
Hl−a = CM-Transformer(El,Ha) (8)
Hl−a−v = CM-Transformer(Hl−a,Hv) (9)
Finally, Hl−a−v is fed to a softmax layer for emotion classification:
q(y) = softmax(W⊤Hl−a−v + b) (10)
The standard cross-entropy loss is used to optimize the parameters for the MERMC task:
LMERMC = − 1 N
N∑
i=1
log q(yi) (11)
where N is the number of utterance samples. The pseudocode for training the MARIO model is provided in Appendix A.2.

heading: 3 Experiments and Analysis

heading: 3.1 Dataset
text: To verify the effectiveness of our FacialMMT framework, we conduct experiments with two datasets. One is the dataset for the main MERMC task, and the other is the dataset for the auxiliary DFER task. The descriptions are as follows: Dataset for MERMC: We use the MELD dataset (Poria et al., 2019), which is a publicly available dataset for MERMC. MELD contains 13,707 video clips extracted from the sitcom Friends, which contain information such as utterance, audio, video, and speaker identity. It also provides emotion annotations on each utterance with seven classes, including neutral, surprise, fear, sadness, joy, disgust, and anger. Dataset for DFER: For the auxiliary DFER task, we use the Aff-Wild2 dataset (Kollias and Zafeiriou, 2019; Kollias, 2022), which contains 548 video clips collected from YouTube in realworld environments. Each clip has several frames of aligned faces and each frame is annotated with a facial expression. It has eight classes of emotions (six basic emotions, neutral, and other). Because the goal is to leverage Aff-Wild2 to guide the emotion prediction on our main dataset, we removed samples annotated with the other emotion.

heading: 3.2 Compared Systems
text: We compare FacialMMT against the following systems: DialogueRNN (Majumder et al., 2019) models the speaker identity, historical conversation, and emotions of previous utterances with RNNs. ConGCN (Zhang et al., 2019) proposes a Graph Convolutional Network (GCN)-based model, which constructs a heterogeneous graph based on context-sensitive and speaker-sensitive dependencies. MMGCN (Hu et al., 2021) builds both long-distance dependency and dependencies between speakers with GCNs. DialogueTRM (Hu et al., 2021) proposes to consider the temporal and spatial dependencies and models local and global context information. DAG-ERC (Shen et al., 2021) models the information flow between the conversation background and its surrounding context. MMDFN (Hu et al., 2022a) introduces a dynamic fusion module to fuse multimodal context features. EmoCaps (Li et al., 2022b) extracts the emotional tendency and fuses modalities through an emotion capsule. UniMSE (Hu et al., 2022b) unifies multimodal sentiment analysis and ERC tasks with a
unified framework based on T5 (Raffel et al., 2020). GA2MIF (Li et al., 2023) proposes a graph and attention based two-stage multi-source multimodal fusion approach.

heading: 3.3 Implementation
text: For our FacialMMT framework, we employ either BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) as the textual encoder and use tiny version of Swin Transformer 1. The maximum length of input text is set to 512. The intercept operation is to remove the last word of the longest utterance in a dialogue and loop until the condition is met. The maximum length of visual and audio is set to the average plus 3 times the standard deviation. The batch sizes for MERMC task and DFER task is set to 1 and 150, respectively. The size of hidden layers is 768. The number of heads in self-attention layers and cross-modal transformer layers is 12, and the learning rates for MERMC and DFER is set to 7e-6 and 5e-5, respectively. The dropout rate is 0.1. The threshold for filter out the emotion-blurred frames is set to 0.2.
Following previous works, we use the weighted average F1-score as the evaluation metric for the MERMC task. For the DFER task, the macro F1score on the validation set is reported. Our model is trained on a GeForce RTX 3090Ti GPU and parameters are optimized through an AdamW optimizer.

heading: 3.4 Main Results on the MERMC task
text: We report the results of different methods on the MERMC task in Table 2 and Table 4. The results of baselines are retrieved from previous studies.
First, we compare the multimodal emotion recognition results of each method. As shown in Table 2, FacialMMT-RoBERTa outperforms all the compared systems with a significant margin, indicating the effectiveness of our proposed approach. Additionally, we find that using BERT instead of RoBERTa as the text encoder leads to a slight decrease in performance. Although it performs relatively than the T5-based UniMSE model, it still outperforms all the other baseline systems that either use BERT or RoBERTa as the text encoder.
Moreover, we compare the emotion recognition results in a single visual modality. As shown in Table 4, previous methods such as EmoCaps and MM-DFN directly employ 3D-CNN to extract the visual features, which introduce environmental
1https://github.com/JDAI-CV/FaceX-Zoo
noise and thus obtain relatively poor emotion recognition results. By extracting the face sequence of all possible speakers in a video, MMGCN achieves the best performance among the baseline systems. Moreover, we can observe our FacailMMT framework significantly outperforms all the compared systems, mainly due to the accurate extraction of the real speaker’s face sequence.
Lastly, we conduct ablation studies of Face Sequence Extraction in Section 2.3. First, after removing unsupervised clustering (UC) and face matching (FM), the emotion recognition result decreases by 2.12%, which demonstrates the usefulness of the two modules. Furthermore, if all the three steps are removed, meaning that video frames are directly used as visual features, the performance drops significantly.

heading: 3.5 Results on the DFER task
text: Table 3 shows the comparison of our method and one of the state-of-the-art methods (Savchenko, 2022) on the DFER task. For a fair comparison, we re-implement the compared system and run experiments based on the same setting as ours. In Table 3, we can clearly observe that our framework outperforms the compared system by 1.52 absolute percentage points on the Aff-Wild2 dataset, which demonstrates the effectiveness of our model on the auxiliary DFER task.

heading: 3.6 Ablation Study
text: We conduct ablation studies of FacialMMT, and show the results in Table 5. It is obvious that any removal of one or two modality leads to a performance drop, indicating that any modality plays an essential role in emotion recognition. Specifically, we can infer that the visual modality plays a more important role than the audio modality, which differs from the observations in previous studies. This suggests that enhancing multimodal emotion recognition from the perspective of visual representation is effective. Moreover, removing the auxiliary DFER module also drops the performance, indicating that introducing frame-level facial expression supervision signals can indeed provide important clues for utterance-level emotion recognition.

heading: 3.7 Case Study
text: To better understand the two main contributions of our work, we present two examples in Figure 3. As our work is the first to use enhanced visual representations to help the MERMC task, we only compare FacialMMT with its variants: 1) Toolkitbased FacialMMT represents using face detection
toolkits to detect the face sequence, as in previous methods; 2) FacialMMT -w/o UC, FM represents face sequences extracted by multimodal face recognition. As shown in Figure 3, the face sequences extracted by two variants of our model contain much noise, which may mislead the emotion prediction of the input utterance. In contrast, our FacialMMT model correctly extracts the face sequence of the real speaker in both cases, and leverages the framelevel emotion distribution to help correctly predict the utterance-level emotions.

heading: 4 Related Work

heading: 4.1 Emotion Recognition in Conversations
text: Recently, Emotion Recognition in Conversations (ERC) has gradually become a hot topic in the field of emotion analysis. According to the input form, ERC is classified into text-based ERC and multimodal ERC. Text-based ERC mainly focuses on research in modeling context, modeling speaker relationships, and incorporating commonsense knowledge (Majumder et al., 2019; Li et al., 2020; Shen et al., 2021; Liu et al., 2022c; Li et al., 2022a; Ong et al., 2022).
To better mimic the way of human thinking, mul-
timodal ERC has rapidly developed in recent years. Multimodal ERC mainly focuses on multimodal feature extraction, interaction, and fusion. First, some studies (Mao et al., 2021; Joshi et al., 2022; Li et al., 2022a) consider context information in conversations and utilize pre-trained language models such as BERT (Devlin et al., 2019) and BART (Lewis et al., 2020) to obtain dialogue-level text representations. Some works (Dai et al., 2021; Liang et al., 2021; Hu et al., 2021; Zhao et al., 2022b) also extract facial representations using various tools, such as MTCNN (Zhang et al., 2016). For multimodal interactions, exiting studies (Tsai et al., 2019; Lv et al., 2021) propose a Cross-Modal Transformer model and a progressive modality reinforcement approach for unaligned multimodal sequences. For modality fusion, Jin et al. (2020) propose a localness and speaker aware transformer to capture local context and emotional inertia. Li et al. (2022b) design an emotion capsule to fuse sentence vectors through multimodal representations, and Zou et al. (2022) propose to use a main modal Transformer to improve the effectiveness of multimodal fusion. In this work, due to the specific nature of multi-party conversations, we extract the face sequence of the real speaker from a video, and use frame-level facial expressions to help utterancelevel emotion recognition.

heading: 4.2 Dynamic Facial Expression Recognition
text: The value of understanding facial expressions lies in collecting direct impressions from others during a conversation. Thus, there has been a significant amount of research conducted on the Dynamic Fa-
cial Expression Recognition (DFER) task. Early DFER datasets were mainly collected from laboratory environments, such as CK+ (Lucey et al., 2010), MMI (Valstar et al., 2010), Oulu-CASIA (Zhao et al., 2011). Since 2013, Emotion Recognition in the Wild (EmotiW) competition has been held, researchers have begun to shift their focus from laboratory-controlled environments to more realistic and complex wild scenarios. Some works (Sümer et al., 2021; Delgado et al., 2021; Mehta et al., 2022) focus on predicting student engagement, while others focus on mental health issues (Yoon et al., 2022; Amiriparian et al., 2022; Liu et al., 2022a). Moreover, there are also several studies proposing new datasets or methods for facial expression recognition of characters in movies and TV shows (Jiang et al., 2020; Zhao and Liu, 2021; Toisoul et al., 2021; Wang et al., 2022b; Liu et al., 2022b).

heading: 5 Conclusion
text: In this paper, we proposed a two-stage framework named Facial expression-aware Multimodal MultiTask learning (FacialMMT) for the MERMC task. FacialMMT first extracts the real speaker’s face sequence from the video, and then leverages an auxiliary frame-level facial expression recognition task to obtain the emotion-aware visual representation through multi-task learning, followed by multimodal fusion for the MERMC task. Experiments on the MELD dataset show the effectiveness of FacialMMT.
Limitations
Our work has the following limitations. First, our proposed FacialMMT approach is a two-stage framework that is not fully end-to-end. We plan to propose an end-to-end framework in the future, which integrates face sequence extraction and multimodal emotion recognition in a joint learning manner. Second, this work primarily focuses on the visual modality, and has not yet delved into other aspects of the MERMC task. Therefore, in the future, we plan to leverage the extracted face sequences to explore better cross-modal alignment and multimodal fusion mechanisms to improve the performance of the MERMC task.
Ethics Statement
We would like to thank Poria et al. (2019) and Kollias and Zafeiriou (2019) for their valuable work
in constructing and sharing the MELD and AffWild2 datasets. MELD is licensed under the GNU General Public License v3.02. For the Aff-Wild2, we have signed an End User License Agreement3. Since MELD is built on the sitcom Friends, we manually annotate 20 different face images occurring in the sitcom for each of the six leading roles without accessing to any personal information. We do not share personal information and do not release sensitive content that can be harmful to any individual or community.
If applying our framework to real-world scenarios in the future, it could potentially involve some ethical issues such as user privacy and ethical biases, as pointed out by Stark and Hoey (2021) and Stark and Hutson (2021). While the ethical issues faced in emotion recognition are common, we will engage with the concerns raised about emotion recognition in the references and strictly comply with relevant regulations and ethical standards. Specifically, our work is based on publicly available datasets, and if we construct new MERMC datasets in the future, we will carefully consider user privacy issues, anonymize or obfuscate facial data, and ensure that the framework is only used in contexts where explicit consent for facial data processing has been obtained. Moreover, we will refer to the recommendations in Stark and Hoey (2021) and develop a comprehensive ethical framework that guides our research process. We are also committed to being transparent about our research methods, data sources, and potential limitations. Regarding potential biases, we plan to evaluate our framework on more diverse datasets in the future, and propose appropriate solutions to alleviate the bias issues.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their insightful comments. This work was supported by the Natural Science Foundation of China (62076133 and 62006117), and the Natural Science Foundation of Jiangsu Province for Young Scholars (BK20200463) and Distinguished Young Scholars (BK20200018).
2https://github.com/declare-lab/MELD/blob/ master/LICENSE
3https://ibug.doc.ic.ac.uk/resources/ cvpr-2022-3rd-abaw/

heading: A Appendix
text: A.1 Multimodal Rules
We design several multimodal rules to obtain possible speakers’ face sequences from a video. The detailed steps are as follows: 1) using the FFmpeg 4 tool to sample frame-level images from a video; 2) using the OpenFace 5 library to detect all the people in the frame-level images, and obtain different FaceID, confidence, 68 feature landmarks, and aligned facial images; 3) using the FFmpeg tool to extract the audio from the video; 4) determining the number of possible speakers in the current video based on three rules as follows:
• “Mouth open-close” count. For different FaceID candidates, count their mouth open and close times respectively. If the sum of the distance between the upper and lower lips of a person is greater than a certain threshold at a certain time, we will record that the mouth of this FaceID is open at the current time.
• Mouth movement. Count which person’s mouth moves the most during the time period by considering the movement of the lips between two consecutive frames of facial images, the difference in width between the inner corners of the mouth between these two frames, and the difference in height between the upper and lower inner lips between these two frames.
• Voice Activity Detection algorithm. Following Zhao et al. (2022a), we identify which frames in the current video have sound by considering whether the visual movement of the lips matches the audio signal. The better the matching is, the higher the probability that it is the speaker. 4https://ffmpeg.org/ 5https://github.com/TadasBaltrusaitis/OpenFace
A.2 Pseudo-code of MARIO We provide the pseudocode for training the proposed MARIO model, where θSwin, θT , θself−attn, and θCMT represent the parameters of SwinTransformer, the text encoder, self-attention Transformer, and Cross-Modal Transformer, respectively.
Algorithm 1: Multitask training procedure of MARIO Input: DFER dataset; MERMC dataset. Output: θSwin; θT ; θself−attn; θCMT .
1 repeat 2 for all batches in the DFER dataset do 3 Forward face sequences through Swin-Transformer ; 4 Compute loss LDFER ; 5 Finetune θSwin using ∇LDFER 6 for all batches in the MERMC dataset do 7 Forward text through text encoder ; 8 Forward face sequences through Swin-Transformer ; 9 Obtain facial expression-aware
visual representation ; 10 Audio and vision are sent to
self-attention Transformer layer respectively ;
11 Conduct cross-modal fusion of text and audio ; 12 Conduct cross-modal fusion of text-audio and vision ; 13 Compute loss LMERMC ; 14 Update θself−attn and θCMT and
finetune θSwin and θT using ∇LMERMC ;
15 until epoch reaches its maximum;
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section Abstract and Section Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We use several pre-training language models, which are referenced and briefly introduced in Section Method

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section Method

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section Ethics Statement

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? For the scientific artifacts we use, we use them as expected; The scientific artifact we have created is described in the Section Introduction and its intended use.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section Ethics Statement

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We analyze our FacialMMT framework in Section Experiments and Analysis.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We show the details of the datasets we used in section Experiments and Analysis.
C 3 Did you run computational experiments? Section Experiments and Analysis.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We describe them in the experimental setting of Section 3.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We describe them in the experimental setting of Section 3.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We describe them in Section 3.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? we report them in Section 3.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section Method
7 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? We ourselves annotated 20 different face images for each of the six leading occurring in the sitcom
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? We ourselves annotated 20 different face images for each of the six leading occurring in the sitcom
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The data we use is publicly available
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We use publicly available datasets and have added a response to ethics review in the camare-ready submission and look forward to it being approved.
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? We ourselves annotated data

heading: Future Work
text: We plan to propose an end-to-end framework in the future, which integrates face sequence extraction and multimodal emotion recognition in a joint learning manner. Second, this work primarily focuses on the visual modality, and has not yet delved into other aspects of the MERMC task. Therefore, in the future, we plan to leverage the extracted face sequences to explore better cross-modal alignment and multimodal fusion mechanisms to improve the performance of the MERMC task