SlowGuess/NAACL_2021_OCR huggingface dataset:
In Figure 5, we show an example of the asymmetric bi-modal relations between vision and text. We observe that our model picks on meaningful relations between words such as "I really enjoyed" and facial expressions such as raising eyebrow, highlighted in the red dashed boxes in Figure 5a. Our model can also learn long-range correlation between "I really enjoyed" and head nodding. Interestingly, we discover that strong relations that are not detected by vision-to-text edges can be recovered by the text-to-vision edges. This advocates the design of the multi-type edges, which allows the model to learn different relations independently that can complement one another.
Figure 1 gives a holistic view of the attention weights among all three modalities. We observe a pattern where almost all edges involve the text modality. A possible explanation for this observation is that the text is the dominant modality with respect to the sentiment analysis task. This hypothesis is verified by the ablation study in Sec. 5.3. Meanwhile, there appears to be very small amount of edges connecting directly between vision and audio, indicating that there might be little meaningful correlation between them. This resonates with our ablation studies in Table 5, where vision and audio combined produce the lowest bi-modal performance. Under such circumstance, our MTAG learns to kill direct audio-vision relations and instead fuse their information indirectly using the text modality as a proxy, whereas previous methods such as MulT keeps audio-vision attentions alive along the way, introducing possible spurious relations that could distract model learning.
We conduct ablation study using unalgined CMUMOSI dataset. MTAG Full Model implements multimodal temporal edge types, adopts TopK
Ablation	Acc2↑	F1↑	MAE↓
Edge Types
No Edge Types	82.4	82.5	0.937
Multimodal Edges Only	85.6	85.7	0.859
Temporal Edges Only	85.2	85.2	0.887
Pruning
Random Pruning Keep 80%	75.5	74.5	1.080
No Pruning	84.7	84.7	0.908
Modalities
Language Only	81.5	81.4	0.911
Vision Only	57.0	57.1	1.41
Audio Only	58.1	58.1	1.37
Vision, Audio	62.0	59.2	1.360
Language, Audio	85.9	85.7	0.915
Language, Vision	86.6	86.6	0.896
Full Model, All Modalities	87.0	87.0	0.859
Table 5: Ablation on unaligned CMU-MOSI validation set.
edge pruning that keeps edges with top  $80\%$  edge weights, and includes all three modalities as its input. Table 5 shows the performance. We present research questions (RQs) as follows and discuss how ablation studies address them.
5.1 RQ1: Does using 27 edge types help?
We first study the effect of edge types on our model performance. As we incrementally add in multimodal and temporal edge types, our model's performance continues to increase. The model with 27 edge types performs the best under all metrics. By dedicating one attention vector  $\mathbf{a}^{\phi_{ji},\tau_{ji}}$  to each edge, MTAG can model each complex relation individually, without having one relation interfering another. As shown in Figure 5 and Table 5, such design enhances multimodal fusion and alignment, helps maintain long-range dependencies in multimodal sequences, and yields better results.
5.2 RQ2: Does our pruning method help?
We compare our TopK edge pruning to no pruning and random pruning to demonstrates its effectiveness. We find that TopK pruning exceeds both no pruning and random pruning models in every aspect. It is clear that, by selectively keeping the top  $80\%$  most important edges, our model learns more
(a) Text-to-vision edge attention weights.
(b) Vision-to-text edge attention weights.
Figure 5: We visualized the asymmetric attention weights of the text-to-vision and vision-to-text edges for one of the validation sequence in CMU-MOSI dataset. The visualized attention weights are from layer 3 of MTAG. Note that the edge types are not shown here for visual clarity.
meaningful representations than randomly keeping  $80\%$ . Our model also beats the one where no pruning is applied, which attests to our assumption and observation from previous work (Lee et al., 2019; Knyazev et al., 2019) that spurious correlations do exist and can distract model from focusing on important interactions. Therefore, by pruning away the spurious relations, the model learned a better representation of the interactions, while using significantly fewer computation resources.
5.3 RQ3: Are all modalities helpful?
Lastly, we study the impact of different modality combinations used in our model. As shown in Table 5, we find that adding a modality consistently brings performance gains to our model. Through the addition of individual modalities, we find that adding the text modality gives the most significant performance gain, indicating that text may be the most dominant modality for our task. This can also be qualitative confirmed by seeing the concentrated edge weights around text modality in Figure 1. This observation also conforms with the observations seen in prior works (Tsai et al., 2019a; Pham et al., 2019). On the contrary, adding audio only brings marginal performance gain. Overall, this ablation study demonstrates that all modalities are beneficial for our model to learn better multimodal representations.
In this paper, we presented the Modal-Temporal Attention Graph (MTAG). We showed that MTAG is an interpretable model that is capable of both fusion and alignment. It achieves similar to SOTA performance on two publicly available datasets for emotion recognition and sentiment analysis while utilizing substantially lower number of parameters than a transformer-based model such as MulT.
We thank Jianing Qian, Xiaochuang Han and Haoping Bai at CMU and the anonymous reviewers at NAACL for providing helpful discussions and feedbacks. This material is based upon work partially supported by BMW of North America, the National Science Foundation and National Institutes of Health. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of BMW of North America, National Science Foundation or National Institutes of Health, and no official endorsement should be inferred.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional
dyadic motion capture database. Language resources and evaluation, 42(4):335.
Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarep—a collaborative voice analysis repository for speech technologies. In 2014 IEEE international conference on acoustics, speech and signal processing (icassp), pages 960–964. IEEE.
Sri Harsha Dumpala, Imran Sheikh, Rupayan Chakraborty, and Sunil Kumar Kopparapu. 2019. Audio-visual fusion for sentiment classification using cross-modal autoencoder. In Proc. Neural Inf. Process. Syst.(NIPS), pages 1-4.
Marco Gori, Gabriele Monfardini, and Franco Scarselli. 2005. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pages 729-734. IEEE.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376.
Yue Gu, Kangning Yang, Shiyu Fu, Shuhong Chen, Xinyu Li, and Ivan Marsic. 2018. Multimodal affective analysis using hierarchical attention strategy with word-level alignment. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2225-2235, Melbourne, Australia. Association for Computational Linguistics.
Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. In Advances in neural information processing systems, pages 1024-1034.
iMotions. 2017. Facial expression analysis.
Mahmoud Khademi. 2020. Multimodal neural graph memory networks for visual question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7177-7188, Online. Association for Computational Linguistics.
Aparna Khare, Srinivas Parthasarathy, and Shiva Sundaram. 2020. Multi-modal embeddings using multitask learning for emotion recognition. *Interspeech* 2020.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Boris Knyazev, Graham W Taylor, and Mohamed Amer. 2019. Understanding attention and generalization in graph neural networks. In Advances in Neural Information Processing Systems, pages 4202-4212.
Angeliki Lazaridou, Nghia The Pham, and Marco Baroni. 2015. Combining language and vision with a multimodal skip-gram model. arXiv preprint arXiv:1501.02598.
J. Lee, Inyeop Lee, and Jaewoo Kang. 2019. Self-attention graph pooling. In ICML.
Paul Pu Liang, Ziyin Liu, Amir Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. arXiv preprint arXiv:1808.03920.
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y Ng. 2011. Multimodal deep learning. In ICML.
Andrei Nicoliciou, Iulia Duta, and Marius Leordeanu. 2019. Recurrent space-time graph neural networks. In Advances in Neural Information Processing Systems, pages 12838-12850.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Hai Pham, Paul Pu Liang, Thomas Manzini, Louis Philippe Morency, and Barnabás Póczos. 2019. Found in translation: Learning robust joint representations by cyclic translations between modalities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6892-6899.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–883, Vancouver, Canada. Association for Computational Linguistics.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. 2008. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2017. Modeling relational data with graph convolutional networks. arXiv preprint arXiv:1703.06103.
Chuan Shi, Yitong Li, Jiawei Zhang, Yizhou Sun, and S Yu Philip. 2016. A survey of heterogeneous information network analysis. IEEE Transactions on Knowledge and Data Engineering, 29(1):17-37.
S. Siriwardhana, Andrew Reis, Rivindu Weerasekera, and Suranga Nanayakkara. 2020. Jointly fine-tuning "bert-like" self supervised models to improve multimodal speech emotion recognition. In INTER-SPEECH.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019a. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558-6569, Florence, Italy. Association for Computational Linguistics.
Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019b. Learning factorized multimodal representations. ICLR.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. International Conference on Learning Representations. Accepted as poster.
Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. 2019a. Heterogeneous Graph Attention Network. In The World Wide Web Conference on - WWW '19, pages 2022-2032, San Francisco, CA, USA. ACM Press.
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019b. Words can shift: Dynamically adjusting word representations using nonverbal behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7216-7223.
Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019. MMGCN: Multi-modal Graph Convolution Network for Personalized Recommendation of Micro-video. In Proceedings of the 27th ACM International Conference on Multimedia, pages 1437-1445, Nice France. ACM.
Kaicheng Yang, Hua Xu, and Kai Gao. 2020. Cmbert: Cross-modal bert for text-audio sentiment analysis. In Proceedings of the 28th ACM International Conference on Multimedia, MM '20, page 521-528, New York, NY, USA. Association for Computing Machinery.
Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo. 2020. A novel graph-based multi-modal fusion encoder for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025-3035, Online. Association for Computational Linguistics.
Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Morency. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. IEEE Intelligent Systems, 31(6):82-88.
Chen Zheng, Quan Guo, and Parisa Kordjamshidi. 2020. Cross-modality relevance for reasoning on language and vision. ACL 2020.
A.1 Positional Embedding
$$
P E _ {(p o s, 2 i)} = \sin (p o s / 1 0 0 0 0 ^ {2 i / d _ {e m b}}) \tag {6}
$$
$$
P E _ {(p o s, 2 i + 1)} = \cos (p o s / 1 0 0 0 0 ^ {2 i / d _ {e m b}}) \tag {7}
$$
A.2 Details regarding pseudo-alignment
Figure 7: Convergence comparison between MTAG and MulT on validation set of unaligned CMU-MOSI dataset.
(b) Pseudo-Alignment example with more vision nodes
Figure 6: Examples of pseudo-align heuristic to coarsely define past, present and future relationships between two unaligned modalities. We try to spread and match the two modalities as much as uniformly possible (the top figure). When the shorter modality contains more and more nodes, we align as many nodes from the shorter sequence as possible with a minimum alignment window size of 2 to the longer sequence, and the rest nodes from the shorter sequence are aligned with window size of 1 (the bottom figure).
For node  $v_{i}$  in  $\pi_{i}$ , in order to determine the "present" nodes  $\mathbf{V}_{\text{present}}$  in a different modality, we draw an analogy from 1D convolution operation. We are given two sequences of different lengths, and we can treat the longer sequence as input and shorter sequence as output to a Conv1D operation. Our goal is to find a feasible stride and kernel size that aligns the input and output. The kernel size defines how many nodes from the longer sequence to be aligned as "present" to each node from the shorter sequence. The stride size defines how far away such alignments should spread in time. We
Unalinged MOSI Validation MAE
do not consider any padding and have the following equation in Conv1D operation:
$$
\frac {M - W}{S} + 1 = N \tag {8}
$$
where  $M$  and  $N$  are the sequence lengths of the output and input to a Conv1D operator, respectively.  $W$  is the kernel size and  $S$  the stride size. From Eq. 8, we can further write the relationship between  $W$  and  $S$  as  $W = M - (N - 1)*S$ . It is clear that the minimum stride size is 1 to a Conv1D operation, and the maximum is  $\left\lfloor \frac{M}{N - 1} \right\rfloor$  in order to keep  $W$  positive. We take the average of the minimum and maximum possible values of  $S$  as our stride size. In case that  $N > \frac{M}{2}$ , we set window size as 2 and stride as 2. We then find the maximal number of nodes from  $N$  that can have kernel size of 2, and the rest of the nodes will have kernel size of 1. Eq. 9 shows our kernel size and stride size calculation and Figure. 6 illustrates our pseudoalignment heuristic.
$$
\left\{ \begin{array}{l l} S = \lceil \mathbf {a v g} (1, \lfloor \frac {M}{N - 1} \rfloor) \rceil , & \text {i f} N \leq \frac {M}{2} \\ W = M - (N - 1) * S & \\ S = 2, W = 2 & \text {o t h e r w i s e} \end{array} \right. \tag {9}
$$
A.3 Model Efficiency
Number of Parameters. We compare the parameter efficiency of our model against the SOTA model, the Multimodal Transformer (MulT) (Tsai et al., 2019a). We first look at the total number of parameters used by the two models. Table 4
Hyperparameter	CMU-MOSI	IEMOCAP
Batch Size	64	64
Initial Learning Rate	1e-3	1e-3
Optimizer	Adam	Adam
Number of MTAG Layers	6	2
Number of Attention Heads	4	8
Node Embedding Dimension	64	64
Edge Pruning Keep Percentage	80%	80%
Total Epochs	20	35
Table 6: Hyperparameter settings on CMU-MOSI and IEMOCAP benchmarks.
Model\Dataset	CMU-MOSI	IEMOCAP
MulT	27.2±2.33	56.0±4.59
MTAG(ours)	24.4±0.95	44.4±0.55
Table 7: Training time per epoch (in seconds) comparison. Run time is averaged over 5 training epochs, with the subscript denoting standard deviation. Both models use batch_size=32, num_layers=6 and are run on the same machine with a single GPU of Nvidia GTX 1080 Ti. Both benchmarks used are the unaligned version.
illustrates that our model uses 0.14 million parameters, only  $6.3\%$  of those in MulT, which has 2.24 million parameters, and yet still achieves state-of-the-art performance. We attribute this result to the effective early fusion of multiple (more than 2) modalities using the MTAG component. In MulT, trimodal fusion happens at a very late stage of the architecture, since each cross-modal Transformer can only model bi-modal interactions. This late fusion regime requires earlier layers to preserve more original information, and thus resulting in a need for more model parameters.
Convergence. Figure 7 gives a comparison between the convergence speed of our model and MulT. Both models are trained with batch_size=32 and lr=1e-3, with the default (best) hyperparameters used for both models. We use the unaligned CMU-MOSI for this study. We observe MTAG converges much faster at epoch 12, comparing with MulT at epoch 17. In addition, we see that our validation MAE curve on the unaligned MOSI validation set goes consistently below MulT's curve. This faster convergence performance could be explained by the small amount of parameters MTAG
Model	# Parameters
MulT (previous SOTA)	2,240,921
MTAG (ours)	142,363
Table 8: Number of model parameters comparison.
uses - MTAG has a much smaller parameter search space for the optimizer, resulting in faster training and earlier convergence.
Training time. We also compare how fast our model runs against MulT. Specifically, under the same condition, we calculate the time it takes for each model to run training for 1 epoch. Table 7 shows the details. We can see that our model runs significantly faster than MulT on both benchmarks, which can be attributed to our light-weight model design (as shown in Table 8). Meanwhile, our edge pruning also reduces the number of computation by throwing away edges that are deemed less important by the model, thus improving the run-time of our model.
Overall Efficiency. From the perspective of training time, number of parameters used, and convergence analysis, it is clear that our model is capable of achieving better results while using much smaller amount of computational resources than the previous state of the art.
We elaborate on the technical details including hyperparameter settings in Table 6. We conduct a basic grid search to find good hyperparameters such as initial learning rate, number of MTAG layers etc. We use Adam as our optimizer and decays the
learning rate by half whenever the validation loss plateaus. Notice that we are using a design that roughly yields a model with a similar structure as in previous works such as MulT. Nevertheless, we still manage to use far less number of parameters during optimization. We use one NVIDIA GTX 1080 Ti for training and evaluation. In addition, the model and hyperparameters we use for ablation study are the same as the ones used for the main experiment, both of which are conducted on CMU-MOSI.
A.5 Number of Parameters Comparison
For a fair comparison on number of parameters between MTAG and MulT, we use the same number of layers and attention heads for both models (i.e. 6 layers of MulT with 4 attention heads). A detailed comparison is shown in Table 8.