SlowGuess/ACL_2020_OCR huggingface dataset:
We perform two different evaluation tasks on CMU-MOSI dataset: i) Binary Classification, and ii) Regression. We formulate it as a regression problem and report Mean-absolute Error (MAE) and the correlation of model predictions with true labels. Besides, we convert the regression outputs into categorical values to obtain binary classification accuracy (BA) and F1 score. Higher value means better performance for all the metrics except MAE. We use two evaluation metrics for BA and F1, one used in (Zadeh et al., 2018d) and one used in (Tsai et al., 2019).
Table 1 shows the results of the experiments in this paper. We summarize the observations from the results in this table as following:
6.1 Performance of MAG-BERT
In all the metrics across the CMU-MOSI dataset, we observe that performance of MAG-BERT is superior to state-of-the-art multimodal models that use BERT word embeddings. Furthermore, MAG-BERT also performs superior to fine-tuned BERT. This essentially shows that the MAG component is allowing the BERT model to adapt to multimodal information during fine-tuning, thus achieving superior performance.
6.2 Performance of MAG-XLNet
A similar performance trend to MAG-BERT is also observed for MAG-XLNet. Besides superior performance than baselines and fine-tuned XLNet, MAG-XLNet achieves near-human level performance for CMU-MOSI dataset. Furthermore, we train MulT using the fine-tuned XLNet embeddings and get the following performance: 83.6/85.3, 82.6/84.2, 0.810, 0.759 which is lower than both MAG-XLNet and XLNet. It is notable that the p-value for student t-test between MAG-XLNet and XLNet in Table 1 is lower than  $10e - 5$  for all the metrics.
The motivation behind the experiments reported in Table 1 is as follows: we extracted word embeddings from pre-trained BERT and XLNet models and trained the baseline models using those embeddings. Since BERT and XLNet are often perceived to provide better word embeddings than Glove, it is not fair to compare MAG-BERT/MAG-XLNet with previous models trained with Glove embeddings. Therefore, we retrain previous works us
Task Metric	BA↑	F1↑	MAE↓	Corr↑
Original (glove)
TFN	73.9/-	73.4/-	0.970/-	0.633/-
MARN	77.1/-	77.0/-	0.968/-	0.625/-
MFN	77.4/-	77.3/-	0.965/-	0.632/-
RMFN	78.4/-	78.0/-	0.922/-	0.681/-
LFN	76.4/-	75.7/-	0.912/-	0.668/-
MulT	- /83.0	- /82.8	- /0.871	- /0.698
BERT
TFN	74.8/76.0	74.1/75.2	0.955	0.649
MARN	77.7/78.9	77.9/78.2	0.938	0.691
MFN	78.2/79.3	78.1/78.4	0.911	0.699
RMFN	79.6/80.7	78.9/79.1	0.878	0.712
LFN	79.1/80.2	77.3/78.1	0.899	0.701
MulT	81.5/84.1	80.6/83.9	0.861	0.711
BERT	83.5/85.2	83.4/85.2	0.739	0.782
MAG-BERT	84.2/86.1	84.1/86.0	0.712	0.796
XLNet
TFN	78.2/80.1	78.2/78.8	0.914	0.713
MARN	78.3/79.5	78.8/79.6	0.921	0.707
MFN	78.3/79.9	78.4/79.1	0.898	0.713
RMFN	79.1/81.0	78.6/80.0	0.901	0.703
LFN	80.2/82.9	79.1/81.6	0.862	0.701
MulT	81.7/84.4	80.4/83.1	0.849	0.738
XLNet	84.7/86.7	84.6/86.7	0.676	0.812
MAG-XLNet	85.7/87.9	85.6/87.9	0.675	0.821
Human	85.7/-	87.5/-	0.710	0.820
Table 1: Sentiment prediction results on CMU-MOSI dataset. Best results are highlighted in bold. MAGBERT and MAG-XLNet achieve superior performance than the baselines and their language-only finetuned counterpart. BA denotes binary accuracy (higher is better, same for F1), MAE denotes Mean-absolute Error (lower is better), and Corr is Pearson Correlation (higher is better). For BA and F1, we report two numbers: the number on the left side of “/” is measures calculated based on (Zadeh et al., 2018c) and the right side is measures calculated based on (Tsai et al., 2019). Human performance for CMU-MOSI is reported as (Zadeh et al., 2018a).
Model	E	1	4	6	8	12	A	⊕	⊙
MAG-XLNet	80.1	85.6	84.1	84.1	83.8	83.6	64.0	60.0	55.8
Table 2: Results of variations of XLNet model: MAG applied at different layers of the XLNet model, input-level concatenation and addition of all modalities. "E" denotes application of MAG immediately after embedding layer of the XLNet and "A" denotes applying MAG after the embedding layer and all the subsequent Encoding layers.  $\oplus$  and  $\odot$  denote input-level addition and concatenation of all modalities respectively. MAG applied at initial layers performs better overall.
ing BERT/XLNet embeddings to establish a more
#	Spoken words + acoustic and visual behaviors	Ground Truth	MAG-XLNet	XLNet
1	"And it really just lacked what made the other movies more enjoyable." Frustrated and disappointed tone	+	-1.4	-1.41	-0.9
2	"But umm I liked it." + Emphasis on tone positive shock through sudden eyebrow raise	+	1.8	1.9	1.2
3	"Except their eyes are kind of like this welcome to the polar express." tense voice + frown expression	+	-0.6	-0.6	0.8
4	"Straight away miley cyrus acting miley cyrus, or lack of, she had this same expression throughout the entire film" + sarcastic voice frustrated facial expression	+	-1.0	-1.2	0.2
Table 3: Examples from the CMU-MOSI dataset. The ground truth sentiment labels are between strongly negative (-3) and strongly positive (+3). For each example, we show the Ground Truth and prediction output of both the MAG-XLNet and XLNet. XLNet seems to be replicating language modality mostly while MAG-XLNet is integrating the non-verbal information successfully.
fair comparison between proposed approach in this paper, and previous work. Based on the information from Table 1, we observe that MAG-BERT/MAG-XLNet models outperforms various baseline models using BERT/XLNet/Glove models substantially.
6.3 Adaptation at Different Layers
We also study the effect of applying MAG at different encoder layers of the XLNet. Specifically, we first apply the MAG to the output of the embedding layer. Subsequently, we apply the MAG to the layer  $j \in \{1, 4, 6, 8, 12\}$  of the XLNet. Then, we apply MAG at all the XLNet layers. From Table 2, we observe that earlier layers are more suitable for application of MAG.
We believe that earlier layers allow for better integration of the multimodal information, as they allow the word shifting to happen from the beginning of the network. If the semantics of words should change based on the nonverbal accompaniments, then initial layers should reflect the semantic shift, otherwise, those layers are only working unimodally. Besides, the higher layers of BERT learn more abstract and higher-level information about the syntactic and semantic structure of linguistic features (Coenen et al., 2019). Since, the acoustic and visual information present in our model corresponds to each word in the utterance, it will be more difficult for the MAG to shift the vector extracted from a later layer since that vector's information will be very abstract in nature.
6.4 Input-level Concatenation and Addition
From Table 2, we see that both input-level concatenation and addition of modalities perform poorly. For Concatenation, we simply concatenate all the modalities. For Addition, we add the audio and visual information to the language embedding after mapping both of them to the language dimension. These results demonstrate the rationale behind using an advanced fusion mechanism like MAG.
6.5 Results on Comparable Datasets
We also perform experiments on the CMU-MOSEI dataset (Zadeh et al., 2018d) to study the generalization of our approach to other multimodal language datasets. Unlike CMU-MOSI which has sentiment annotations at utterance level, CMU-MOSEI has sentiment annotations at sentence level. The experimental methodology for CMU-MOSEI is similar to the original paper. For the sake of comparison, we suffice<sup>1</sup> to comparing the binary accuracy and f1 score for the top 3 models in Table 1. In BERT category, we compare the performance of MulT (with BERT embeddings), BERT and MAG-BERT which are respectively as follows: [83.5, 82.9] for MulT, [83.9, 83.9] for BERT, and [84.7, 84.5] for MAG-BERT. Similarly for XLNET category, the results for MulT (with XLNet embeddings), XLNet and MAG-XLNet are as follows: [84.1, 83.7] for MulT, [85.4, 85.2] for XLNet and [85.6, 85.7] for MAG-XLNet. Therefore, superior performance of
<sup>1</sup>Since Transformer based models take a long time to train for CMU-MOSEI
MAG-BERT and MAG-XLNet also generalizes to CMU-MOSEI dataset.
6.6 Fine-tuning Effect
We study whether or not the superior performance of the MAG-BERT and MAG-XLNet is related to successful finetuning of the models, or related to other factors e.g. any transformer with architecture like BERT or XLNet would achieve superior performance regardless of being pretrained. By randomly initializing the weights of BERT and XLNet within MAG-BERT and MAG-XLNet, we get the following performance on BA for the CMU-MOSI: 70.1 and 70.7 respectively. This indicates that the success of the MAG-BERT and MAG-XLNet is due to successful fine-tuning. Even on the larger CMU-MOSEI dataset we get BA of 76.8 and 78.4 for MAG-BERT and MAG-XLNet, which further substantiates the fact that fine-tuning is successful using MAG framework.
6.7 Qualitative Analysis
In Table 3, we present some examples where MAG-XLNet adjusted sentiment intensity properly by taking into account nonverbal information. The examples demonstrate that MAG-XLNET can successfully integrate the non-verbal modalities with textual information.
In both Example-1 and Example-2, XLNet correctly predicted the polarity of the displayed emotion. However, additional information was present in the acoustic and visual domain which XLNet could not utilize. Given those information, MAGXLNet could better predict the magnitude of emotion displayed in both cases.
Although the emotion in the text of Example-3 can be portrayed as a bit positive, the tense voice and frown expression helps MAG-XLnet reverse the polarity of predicted emotion. Similarly, the text in Example-4 is mostly neutral, but MAG-XLNet can predict the negative emotion through the sarcastic vocal and frustrated facial expression.
In this paper, we introduced a method for efficiently finetuning large pre-trained Transformer models for multimodal language. Using a proposed Multimodal Adaptation Gate (MAG), BERT and XLNet were successfully fine-tuned in presence of vision and acoustic modalities. MAG essentially poses the nonverbal behavior as a vector with a trajectory
and magnitude, which is subsequently used to shift lexical representations within the pre-trained Transformer model. A unique characteristic of MAG is that it makes no change to the original structure of BERT or XLNet, but rather comes as an attachment to both models. Our experiments demonstrated the superior performance of MAG-BERT and MAG-XLNet. The code for both MAG-BERT and MAG-XLNet are publicly available here
This research was supported in part by grant W911NF-15-1-0542 and W911NF-19-1-0029 with the US Defense Advanced Research Projects Agency (DARPA) and the Army Research Office (ARO). Authors AZ and LM were supported by the National Science Foundation (Awards #1750439 #1722822) and National Institutes of Health. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of US Defense Advanced Research Projects Agency, Army Research Office, National Science Foundation or National Institutes of Health, and no official endorsement should be inferred.
Minghai Chen, Sen Wang, Paul Pu Liang, Tadas Baltrusaitis, Amir Zadeh, and Louis-Philippe Morency. 2017. Multimodal sentiment analysis with word-level fusion and reinforcement learning. In Proceedings of the 19th ACM International Conference on Multimodal Interaction, pages 163-171. ACM.
Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. 2019. Visualizing and measuring the geometry of bert. arXiv preprint arXiv:1906.02715.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep—a collaborative voice analysis repository for speech technologies. In 2014 IEEE international conference on acoustics, speech and signal processing (icassp), pages 960-964. IEEE.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep
$^{2}$ https://github.com/WasifurRahman/BERT-multimodal_transformer
bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Devamanyu Hazarika, Soujanya Poria, Amir Zadeh, Erik Cambria, Louis-Philippe Morency, and Roger Zimmermann. 2018. Conversational memory network for emotion recognition in dyadic dialogue videos. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 2122-2132.
iMotions. 2017. Facial expression analysis.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Paul Pu Liang, Ziyin Liu, Amir Zadeh, and LouisPhilippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. arXiv preprint arXiv:1808.03920.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111-3119.
Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagee, and Louis-Philippe Morency. 2014. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal prediction approach. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 50-57. ACM.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365.
Hai Pham, Paul Pu Liang, Thomas Manzini, Louis-Philippe Morency, and Barnabas Poczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. arXiv preprint arXiv:1812.07809.
Soujanya Poria, Amir Hussain, and Erik Cambria. 2018. Multimodal Sentiment Analysis, volume 8. Springer.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws.com/openai-assetss/researchcovers/languageunsupervised/language understanding paper.pdf.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. arXiv preprint arXiv:1904.01766.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. arXiv preprint arXiv:1906.00295.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. arXiv preprint arXiv:1806.06176.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. arXiv preprint arXiv:1811.09362.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. Journal of the Acoustical Society of America, 123(5):3878.
Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250.
Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018a. Memory fusion network for multiview sequential learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
Amir Zadeh, Paul Pu Liang, Louis-Philippe Morency, Soujanya Poria, Erik Cambria, and Stefan Scherer. 2018b. Proceedings of grand challenge and workshop on human multimodal language (challenge-hml). In Proceedings of Grand Challenge and Workshop on Human Multimodal Language (ChallengeHML).
Amir Zadeh, Paul Pu Liang, Soujanya Poria, Prateek Vij, Erik Cambria, and Louis-Philippe Morency. 2018c. Multi-attention recurrent network for human communication comprehension. In Thirty-Second AAAI Conference on Artificial Intelligence.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Mosi: multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos. arXiv preprint arXiv:1606.06259.
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018d. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2236-2246.