Topic and Style-aware Transformer for Multimodal Emotion Recognition
Year: 2023

Abstract:
Understanding emotion expressions in multimodal signals is key for machines to have a better understanding of human communication. While language, visual and acoustic modalities can provide clues from different perspectives, the visual modality is shown to make minimal contribution to the performance in the emotion recognition field due to its high dimensionality. Therefore, we first leverage the strong multimodality backbone VATT to project the visual signal to the common space with language and acoustic signals. Also, we propose contentoriented features Topic and Speaking style on top of it to approach the subjectivity issues. Experiments conducted on the benchmark dataset MOSEI show our model can outperform SOTA results and effectively incorporate visual signals and handle subjectivity issues by serving as content "normalization".

heading: 1 Introduction
text: Emotion recognition is intrinsic for social robots to interact with people naturally. The ability to tell emotional change and propose timely intervention solutions can help maintain people’s mental health and social relations. Though the traditional task of sentiment analysis is purely based on text (Wang et al., 2020; Ghosal et al., 2020; Shen et al., 2021), humans express emotions not only with spoken words but also through non-verbal signals such as facial expressions and the change of tones. Therefore, following the current trend of multimodal emotion recognition (Delbrouck et al., 2020; Zadeh et al., 2017; Rahman et al., 2020; Gandhi et al., 2022), we focus on addressing problems of understanding the expressed emotions in videos along with their audio and transcripts.
In this work, we tackle the problem of the multimodal emotion recognition task from two major issues: Minimal contribution of visual modality, and emotional subjectivity. Previous works which have used multimodal approaches (Rahman et al.,
2020; Joshi et al., 2022; Delbrouck et al., 2020) have shown that text+audio outperforms the results of combining all three modalities. While facial and gesture signals contain abundant information, they tend to introduce more noise to the data due to its high dimensionality. In order to increase the contribution from visual modality , we propose to take advantage of the strong multimodal backbone VATT (Akbari et al., 2021) that can project features of different granularity levels into a common space. On the other hand, the expression of emotion is subjective. People’s emotion judgment can be influenced by enclosed scenarios. As shown in the left two columns in Figure 1, though the two examples are all labeled as “happy”, the signals we use to detect “happy” may not be the same. In a public speech, showing gratitude may mean a positive sentiment while in movie reviews, we may focus more on sentiment words like good or bad. Also, subjectivity may come from individual differences in their own emotional intensity. As the examples shown in the right three columns in Figure 1, the sadness and happiness of the person in the excited style are more distinguishable through his face while the person in the calm style always adopts a calm face that makes sad and happy less recognizable. Therefore, we introduce content-oriented features: topic and speaking style serving as a content “normalization”
2074
for each person. Our work makes the following contribution: 1) We propose to leverage the multimodal backbone to reduce the high dimensionality of visual modality and increase its contribution to the emotion recognition task.
2) We incorporate emotion-related features to handle modeling issues with emotional subjectivity
3) Experiments conducted on the benchmark dataset MOSEI show our model can outperform SOTA results and effectively incorporate visual signals and handle subjectivity issues.

heading: 2 Related Work
text: Emotion recognition using a fusion of input modalities such as text, speech, image, etc is the key research direction of human-computer interaction. Specific to the area of sentiment analysis, Multimodal Transformer applies pairwise crossattention to different modalities (Tsai et al., 2019). The Memory Fusion Network synchronizes multimodal sequences using a multi-view gated memory that stores intra-view and cross-view interactions through time (Zadeh et al., 2018). TFN performs the outer product of the modalities to learn both the intra-modality and inter-modality dynamics(Sahay et al., 2018). (Rahman et al., 2020) begins the endeavor to take BERT (Devlin et al., 2018) as a strong backbone pretrained on large scale corpus. (Arjmand et al., 2021) follows the direction and combines Roberta with a light-weighed audio encoder to fuse the text and audio features. A recent work (Yang et al., 2022a) presents a self-supervised framework to pretrain features within a single modality and across different modalities. Other frameworks include context and speaker-aware RNN (Shenoy and Sardana, 2020; Wang et al., 2021), graph neural networks modeling knowledge graphs and inter/intra relations between videos (Joshi et al., 2022; Fu et al., 2021; Lian et al., 2020), while (Zhu et al., 2021) has used topic information to improve emotion detection

heading: 3 Method

heading: 3.1 Overview
text: Our model aims to predict the presence of different emotions given an utterance-level video input along with its audio and transcripts. Figure 2 shows the overall structure of our model. To first get a better alignment of features from different modalities,
the raw video input will be fed into our backbone VATT and we can get the corresponding projected features for visual, acoustic, and textual signals separately. Meanwhile, our high-level content module will extract the corresponding topic and style representation. Queried by the video context, the topic and style features are further merged by a crossattention layer. Then both low-level and high-level features are concatenated and put into the final classification layer.

heading: 3.2 Backbone
text: Video-Audio-Text Transformer (VATT) is a framework for learning multimodal representations that takes raw signals as inputs. For each modality encoder, VATT appends an aggregation head at the beginning of the input sequence. The corresponding latent feature will serve as the projection head for this modality. For pretraining, contrastive loss is applied to align features from different modalities in a common projected space. Details can be found in (Akbari et al., 2021).

heading: 3.3 Content-oriented Features

heading: 3.3.1 Topic
text: For each utterance input, we will first predict the topic of this utterance and feed the corresponding topic embedding into the model. Since we don’t have the ground truth label for topics, we use Latent Dirichlet Allocation (LDA) (Blei et al., 2003) model to cluster all the text from the training set into 3 topics. The number of topics is decided by grid search.

heading: 3.3.2 Speaking Style
text: We define speaking style based on the expression coefficient and the projection parameters of a 3DMM model (Blanz and Vetter, 1999). In a 3DMM model, the face shape is represented as an affine model of facial expression and facial identity: S = S̄+Bidα+Bexpβ. This 3D face will be
projected into a 2D image by translation and rotation p. Since there are multiple video frames, the expression coefficient β and the project parameter p will become time series β(t) and p(t). For a detailed analysis of the relations between the 3DMM parameters and the talking styles, (Wu et al., 2021) collected a dataset consisting of 3 talking styles: excited, tedious, and solemn. They find out that the standard deviation of the time series features and the gradient of these features are closely related to the styles. The final style code are denoted as σ(β(t)) ⊕ σ(∂β(t)∂t ) ⊕ σ( ∂p(t) ∂t ), ⊕ signifies the vector concatenation.

heading: 3.3.3 Aggregating Different Features
text: Given each data input with its corresponding video ID, we collect all the transcripts with the same video ID as the context, and the context feature will be extracted from the text encoder of VATT. To adapt general topic and style features to the current speaker, we treat them as the feature sequence of length 2 and use an additional cross-attention layer to aggregate these features queried by the video context. Then this information along with the context and aligned features will be concatenated and fed into the final linear classifier.

heading: 4 Dataset
text: We conduct our experiments on CMU-Multimodal Opinion Sentiment and Emotion Intensity (CMUMOSEI (Bagher Zadeh et al., 2018)) dataset. The dataset contains more than 23,500 sentence utterance videos from more than 1000 online YouTube speakers. Each sentence is annotated for a sentiment intensity from highly negative (-3) to highly positive (+3) and for 6 emotion classes: happiness, sadness, anger, fear, disgust, and surprise. The number of utterances for train/test/dev is 16327/4662/1871 separately. Label distribution of the training set is shown in Table 3

heading: 5 Experiments

heading: 5.1 Setup
text: We train our models on 8 V100 GPU for 8 hours using the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 1e− 4 and a mini-batch size of 64. The total number of parameters of our model is 155M. For topic clustering, we adopt the scikitlearn LDA library (Pedregosa et al., 2011). We extract the style code for each video using https: //github.com/wuhaozhe/style_avatar. The final model is selected based on validation accuracy on the development set.
Task We evaluate the performance of our model on two tasks: 1) Multi-label emotion recognition: the model needs to classify whether each of the 6 emotion classes presents or not. 2) Sentiment anal-
ysis: the model is tested on both 2-class (sentiment is positive or negative) and 7-class (a scale from -3 to +3) classification.
Evaluation Since the labels in MOSEI are unbalanced, we use the weighted F1 score for each emotion as the evaluation metric. We compare the performance with Multilogue-Net (Shenoy and Sardana, 2020) that adopted context and speaker-aware RNN , TBJE (Delbrouck et al., 2020), a state-ofthe-art method using cross-attention for modality fusion and MESM (Dai et al., 2021), who were the first to introduce a fully end-to-end trainable model for the multimodal emotion recognition task . There are two recent works on emotion recognition, COGMEN (Joshi et al., 2022) and i-Code (Yang et al., 2022b). Since COGMEN adopted a structural representation that can exploit more relational information from other data samples and i-Code did not report the same metrics and is not opensourced, we will not compare with them in this paper.

heading: 5.2 Emotion Recognition
text: Table 1 shows our quantitative results. Compared with other SOTA methods in the first three rows, our full model achieves the best performance on recognizing happy, sad and angry. We reckon that it is due to very limited data for surprise and fear to train the big backbone, our model does not gain much improvement (shown in Table 3). To further analyze the contribution of each component of our model design, we also conduct a detailed ablation study: 1) We first remove the aligned features from the backbone each at a time. We can see from the results in the second block that combining all three modalities in our full model outperforms the bimodality input. Especially contrasting rows with and without video input, their comparative performance validates that our model can learn effectively from visual modalities. 2) In the third block, we report the performance when we simply concatenate aligned features as the input to the emotion classification layer without high-level features. The degraded performance reveals the efficacy of our content feature design. 3) Lastly, we investigate the influence of each content feature and the aggregation using context. To remove the context, we directly apply a self-attention layer to the feature sequence and use a linear layer to project the outputs into the aggregate feature dimension. For topic and style, we just remove the corresponding
feature from the input. As shown in the last block, removing any part will result in a performance drop. Overall, our full model in comparison yields the best performance.

heading: 5.3 Sentiment Analysis
text: To further validate our methods, we run our model on the other subtask, sentiment analysis. For each data sample, the annotation of sentiment polarity is a continuous value from -3 to 3. -3 means extremely negative, and 3 means extremely positive. Our model is trained to regress the sentiment intensity. Then we ground the continuous value into 2 or 7 classes to calculate the accuracy. Contrasting 2-class and 7-class results in Table 2, our model works better for more fine-grained classification.

heading: 6 Qualitative Results
text: We first show that our model can correctly recognize emotions under different topics. As shown in Figure 3, for movie reviews, finance or commercial advertisements, the model can use different cues to predict the emotion as happy or sad. In Figure 4, our model can distinguish between excited/calm speaking styles and recognize the slight emotional change within each person. (all example videos can be found in supp).

heading: 7 Conclusion and Future Work
text: This study employs the powerful multimodal backbone VATT to facilitate feature alignment across various modalities. Moreover, content-specific features are introduced to mitigate the influence
of individual subjectivity. The experimental outcomes demonstrate that the model can effectively assimilate visual information with reduced dimensions. Furthermore, the incorporation of sentimentoriented features yields further improvements in the model’s performance, helping beat state of the art models on CMU-MOSEI dataset

heading: 8 Limitations
text: For modeling simplicity, we adopt the classic LDA methods to get the topic ID for each video segment. We plan to investigate more advanced topic clustering methods and check how it can be applied to multilingual cases. Also, we propose a twostage framework that first extract topic and style features, based on which the emotion classifier will be trained. In the future, we hope to extend this work to learn features in an end-to-end manner.

heading: A Appendix
text: A.1 Topic visualization We first show the final topic clustering results. The second column shows the top 20 high frequency words in this topic and the third column shows some examples under this topic. The first topic is more related to movie reviews, the second covers business and finance, and the third one seems to associate with commercial and instruction videos.
A.2 Style Code In Fig 5, we can see that styles have a distinctive embedding based on emotion which confirms our hypothesis that style code can add a meaningful input to our multimodal approach.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
7 A2. Did you discuss any potential risks of your work? We do not consider any risks in our work
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? model:3, 5.1 data: 4
3 B1. Did you cite the creators of artifacts you used? model:3, 5.1 data: 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 3.2, 4
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data is anonymized and discussed in the original paper
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5.1
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5.1
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we hope to extend this work to learn features in an end-to-end manner