SlowGuess/NAACL_2021_OCR huggingface dataset:
Fear
original
Figure 3: Case study of MESM on six basic emotion categories (happy, sad, angry, surprised, fear, disgusted). From left to right, we show the original image and the Nucleus Sampling  $(p = 0.6)$  result over points in each attention layer. Red regions represent the points that are computed for the next layer.
layer 3
Disgusted
original
Attention over Mel-Spectrum
original
Figure 4: Visualization of cross-modal attention of the acoustic modality. We only show the highest  $10\%$  value of mel-spectrogram in each image. From left to right, we show the original image and the Nucleus Sampling  $(p = 0.6)$  results over points in each attention layer. Red regions represent the active points that will go to the next sparse CNN layer.
dataset and the results are shown in Table 4. We observe similar trends on this dataset.
6.2 Case Study
To improve the interpretability and gain more insights from our model, we visualize the attention maps of our sparse cross-modal attention mechanism on the six basic emotions: happy, sad, angry, surprised, fear, and disgusted. As shown in Figure 3, in general, the models attend to several regions of interest such as the mouth, eyes, eyebrows, and facial muscles between the mouth and
the eyes. We verify our method by comparing the regions that our model captures based on the facial action coding system (FACS) (Ekman, 1997). Following the mapping of FACS to human emotion categories (Basori, 2016; Ahn and Chung, 2017), we conduct empirical analysis to validate the sparse cross-modal attention on each emotion category. For example, the emotion happy is highly influenced by raising of the lip on both ends, while sad is related to a lowered lip on both ends and downward movement of the eyelids. Angry is determined from a narrowed gap between the eyes and thinned lips, while surprised is expressed with an open mouth and raising of the eyebrows and eyelids. Fear is indicated by a rise of the eyebrows and upper eyelids, and also an open mouth with the ends of the lips slightly moving toward the cheeks. For the emotion disgusted, wrinkles near the nose area and movement of the upper lip region are the determinants.
Based on the visualization of the attention maps on the visual data in Figure 3, the MESM can capture most of the specified regions of interest for the six emotion categories. For the emotion angry, the sparse cross-modal attention can retrieve the features from the lip region quite well, but it sometimes fails to capture the gap between the eyes. For surprised, the eyelids and mouth regions can be successfully captured by MESM, but sometimes the model fails to consider the eyebrow regions. For the acoustic modality, it is hard to analyse the attention in terms of emotion labels. We show a general visualization of the attention maps over the audio data in Figure 4. The model attends to
Figure 5: The trend line of the Top: Weighted Accuracy and Bottom: FLOPs  $(\mathrm{x10^9})$  of the MESM with different top-p values used in the Nucleus Sampling. represents performance of MESM, while represents performance of the FE2E model
the regions with high spectrum values in the early attention layer, and more points are filtered out after going through further cross-modal attention layers. More visualized examples are provided in Appendix B.
6.3 Effects of Nucleus Sampling
To have an in-depth understanding of the effects of Nucleus Sampling on the MESM, we perform more experiments with different top-p values ranging from 0 to 1, with a step of 0.1. As shown in Figure 5, empirically, the amount of computation is reduced consistently with the decrease of the top-p values. In terms of performance, with a top-p value from 0.9 to 0.5, there is no significant drop in the evaluation performance. Starting from 0.5 to 0.1, we can see a clear downgrade in the performance, which means some of the useful information for recognizing the emotion is excluded. The inflection point of this elbow shaped trend line can be an indicator to help us make a decision on the value of the top-p. Specifically, with a top-p of 0.5, the MESM can achieve comparable performance to the FE2E model with around half of the FLOPs in the feature extraction.
Model	Mods.	Avg. Acc	Avg. F1
FE2E	TAV	84.5	58.5
TA	83.7	54.0
TV	82.8	55.7
VA	81.2	54.4
T	80.8	50.0
A	73.3	44.9
V	78.2	49.8
MESM	TAV	84.4	57.3
TA	83.6	56.7
TV	82.1	56.0
Table 5: Results of the ablation study of our fully end-to-end model (FE2E) and multimodal end-to-end sparse model (MESM) on the IEMOCAP dataset. In the Mods. (modalities) column, the T/A/V indicates the existence of the textual (T), acoustic (A), and visual (V) modalities.
We conduct a comprehensive ablation study to further investigate how the models perform when one or more modalities are absent. The results are shown in Table 5. Firstly, we observe that the more modalities the more improvement in the performance. TAV, representing the presence of all three modalities, results in the best performance for both models, which shows the effectiveness of having more modalities. Secondly, with only a single modality, the textual modality results in better performance than the other two, which is similar to the results of previous multimodal works. This phenomenon further validates that using textual (T) to attend to acoustic (A) and visual (V) in our cross-modal attention mechanism is a reasonable choice. Finally, with two modalities, the MESM can still achieve a performance that is on par with the FE2E model or is even slightly better.
In this paper, we first compare and contrast the two-phase pipeline and the fully end-to-end (FE2E) modelling of the multimodal emotion recognition task. Then, we propose our novel multimodal end-to-end sparse model (MESM) to reduce the computational overhead brought by the fully end-to-end model. Additionally, we reorganize two existing datasets to enable fully end-to-end training. The empirical results demonstrate that the FE2E model has an advantage in feature learning
and surpasses the current state-of-the-art models that are based on the two-phase pipeline. Furthermore, MESM is able to halve the amount of computation in the feature extraction part compared to FE2E, while maintaining its performance. In our case study, we provide a visualization of the cross-modal attention maps on both visual and acoustic data. It shows that our method can be interpretable, and the cross-modal attention can successfully select important feature points based on different emotion categories. For future work, we believe that incorporating more modalities into the sparse cross-modal attention mechanism is worth exploring since it could potentially enhance the robustness of the sparsity (selection of features).
This work is funded by MRP/055/18 of the Innovation Technology Commission, the Hong Kong SAR Government.
Duck-ki Ahn and Jean-Hun Chung. 2017. A study on character's emotional appearance in distinction focused on 3d animation "inside out". Journal of Digital Convergence, 15:361-368.
Md Shad Akhtar, Dushyant Singh Chauhan, Deepanway Ghosal, Soujanya Poria, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Multi-task learning for multi-modal emotion recognition and sentiment analysis. arXiv preprint arXiv:1905.05812.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2018. Multimodal machine learning: A survey and taxonomy. IEEE transactions on pattern analysis and machine intelligence, 41(2):423-443.
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pages 59–66. IEEE.
T. Baltrusaitis, M. Mahmoud, and P. Robinson. 2015. Cross-dataset learning and person-specific normalisation for automatic action unit detection. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), volume 06, pages 1-6.
Ahmad Hoirul Basori. 2016. Emotional facial expression based on action units and facial muscle. International Journal of Electrical and Computer Engineering (IJECE), 6:2478-2487.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335.
Long Chen, Hanwang Zhang, Jun Xiao, L. Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. 2017. Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 6298-6306.
Wenliang Dai, Zihan Liu, Tiezheng Yu, and Pascale Fung. 2020a. Modality-transferrable emotion embeddings for low-resource multimodal emotion recognition. ArXiv, abs/2009.09629.
Wenliang Dai, Tiezheng Yu, Zihan Liu, and Pascale Fung. 2020b. Kungfupanda at semeval-2020 task 12: Bert-based multi-task learning for offensive language detection. ArXiv, abs/2004.13432.
Paul Ekman, Wallace V Freisen, and Sonia Ancoli. 1980. Facial signs of emotional experience. Journal of personality and social psychology, 39(6):1125.
Rosenberg Ekman. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.
Benjamin Graham and Laurens van der Maaten. 2017. Submanifold sparse convolutional networks. arXiv preprint arXiv:1706.01307.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Lianian Harold Li, Mark Yatskar, Da Yin, C. Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. ArXiv, abs/1908.03557.
Seyedmahdad Mirsamadi, E. Barsoum, and C. Zhang. 2017. Automatic speech emotion recognition using recurrent neural networks with local attention. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2227-2231.
Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: Harvesting opinions from the web. In Proceedings of the 13th international conference on multimodal interfaces, pages 169-176.
Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Tadas Baltrušaitis, and Louis-Philippe
Morency. 2016. Deep multimodal fusion for persuasiveness prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, pages 284-288.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pages 8026-8037.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 973-982.
Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020. Integrating multimodal information in large pretrained transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2359-2369.
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Edmund Tong, Amir Zadeh, Cara Jones, and Louis-Philippe Morency. 2017. Combating human trafficking with multimodal deep models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1547-1556.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the conference. Association for Computational Linguistics. Meeting, volume 2019, page 6558. NIH Public Access.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. arXiv preprint arXiv:1806.06176.
J. C. Vásquez-Correa, P. Klumpp, J. R. Orozco-Arroyave, and E. Noth. 2019. Phonet: A tool based on gated recurrent neural networks to extract phonological posteriors from speech. In INTERSPEECH.
Juan Camilo Vásquez-Correa, J. R. Orozco-Arroyave, T. Bocklet, and E. Nöth. 2018. Towards an automatic evaluation of the dysarthria level of patients
with parkinson's disease. Journal of communication disorders, 76:21-36.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2017. Select-additive learning: Improving generalization in multimodal sentiment analysis. In 2017 IEEE International Conference on Multimedia and Expo (ICME), pages 949–954. IEEE.
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7216-7223.
Peng Xu, Zihan Liu, Genta Indra Winata, Zhaojiang Lin, and Pascale Fung. 2020. Emograph: Capturing emotion correlations using graph networks. *ArXiv*, abs/2008.09378.
Zhou Yu, J. Yu, Yuhao Cui, D. Tao, and Qi Tian. 2019. Deep modular co-attention networks for visual question answering. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6274-6283.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multi-view sequential learning. arXiv preprint arXiv:1802.00927.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018b. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2236-2246.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499-1503.
L. Zhang and B. Liu. 2017. Sentiment analysis and opinion mining. In Encyclopedia of Machine Learning and Data Mining.
IEMOCAP	CMU-MOSEI
FE2E	MESM	FE2E	MESM
Batch size	8	8	8	8
Learning rate	5e-5	5e-5	5e-5	5e-5
Dim	64	64	64	64
#Heads	4	4	4	4
#Layers	4	4	4	4
Max text len	50	100	50	100
N	-	3	-	3
Table 6: The best hyper-parameters used in training for the two datasets.
B Case Study on Acoustic Modality
We provide more visualized examples of the sparse cross-modal attention maps of the acoustic modality in Figure 6.
Figure 6: Case study of the sparse cross-modal attention maps on six basic emotion categories (happy, sad, angry, surprised, fear, disgusted) on the audio modality. From the left to right, we show the original image and the Nucleus Sampling results over feature points in each attention layer. Red regions represent the active points that will be computed in the next sparse layer.