SlowGuess/NAACL_2021_OCR huggingface dataset:
Jianing Yang $^{1*}$ , Yongxin Wang $^{1*}$ , Ruitao Yi $^{1}$ , Yuying Zhu $^{1}$ , Azaan Rehman $^{1}$ , Amir Zadeh $^{1}$ , Soujanya Poria $^{2}$ , Louis-Philippe Morency $^{1}$
$^{1}$ Carnegie Mellon University  $^{2}$ Singapore University of Technology and Design
{jianing3, yongxinw, ruitaoy, yuyingz, arehman, abagherz} @cs.cmu.edu, sporia@sutd.edu.sg, morency@cs.cmu.edu
Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a graph with heterogeneous nodes and edges that captures the rich interactions across modalities and through time. Then, a novel graph fusion operation, called MTAG fusion, along with a dynamic pruning and read-out technique, is designed to efficiently process this modal-temporal graph and capture various interactions. By learning to focus only on the important interactions within the graph, MTAG achieves state-of-the-art performance on multimodal sentiment analysis and emotion recognition benchmarks, while utilizing significantly fewer model parameters.<sup>1</sup>
With recent advances in machine learning research, analysis of multimodal sequential data has become increasingly prominent. At the core of modeling this form of data, there are the fundamental research challenges of fusion and alignment. Fusion is the process of blending information from multiple modalities. It is usually preceded by alignment, which is the process of finding temporal relations between the modalities. An important research area that exhibits this form of data is multimodal language analysis, where sequential modalities of
Figure 1: Example visualization of tri-modal Modal-Temporal Attention learned by our proposed model. Each circle represents a node from video/text/audiomodalities, and the blue lines denote the learned attention weights (i.e. the thicker and darker a blue line is, the larger the attention weight). We observe high intensities between semantically correlated graph entities, such as "Really Enjoy" and the raise in eyebrow, which indicate positive sentiment. Note that our graph-based model learns multimodal interactions without prior alignment, and captures diverse types of interactions across multiple modalities all the same time. Edge types are not shown for visual clarity.
language, vision, and acoustic are present. These three modalities carry the communicative information and interact with each other through time; e.g. positive word at the beginning of an utterance may be the cause of a smile at the end. When analyzing such multimodal sequential data, it is crucial to build models that perform both fusion and alignment accurately and efficiently by a) aligning arbitrarily distributed asynchronous modalities in an interpretable manner, b) efficiently accounting for short and long-range dependencies, c) explicitly modeling the inter-modal interactions between
*Equal contribution
 1Code is available at https://github.com/jedyang97/MTAG.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1009-1021 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 2: The 3-stage MTAG framework: Node Construction, Edge Construction and Fusion+Pruning. [Node Construction] Each modality's features are first passed through a distinct Feed-Forward-Network to be mapped into the same embedding size. Then, a positional embedding is added to each transformed feature based on its position in its own modality, so that temporal information are encoded. The features are now nodes in the graph. [Edge Construction] We then apply an algorithm to construct edges among these nodes by appropriately indexing each edge with a modal type and a temporal type. [Fusion+Pruning] Finally, we pass the graph into the MTAG module to learn interactions across modality and time. The output graph with updated node embeddings and pruned edges can be passed to downstream modules, e.g. a Multi-layer Perceptron, to complete specific tasks such as regression or classification.
the modalities while simultaneously accounting for intra-modal dynamics.
In this paper, we propose MTAG (Modal-Temporal Attention Graph). MTAG is capable of both fusion and alignment of asynchronously distributed multimodal sequential data. Modalities do not need to be pre-aligned, nor do they need to follow similar sampling rate. MTAG can capture interactions of various types across any number of modalities all at once, comparing to previous methods that model bi-modal interactions at a time (Tsai et al., 2019a). At its core, MTAG utilizes an efficient trimodal-temporal graph fusion operation. Coupled with our proposed dynamic pruning technique, MTAG learns a parameter-efficient and interpretable graph. In our experiments, we use two unaligned multimodal emotion recognition and sentiment analysis benchmarks: IEMOCAP (Busso et al., 2008) and CMU-MOSI (Zadeh et al., 2016). The proposed MTAG model achieves state-of-the-art performance with far fewer parameters. Subsequently, we visualize the learned relations between modalities and explore the underlying dynamics of multimodal language data. Our model incorporates all three modalities in both alignment and fusion, a fact that is also substantiated in our ablation studies.
Human Multimodal Language Analysis Analyzing human multimodal language involves learning from data across multiple heterogeneous sources that are often asynchronous, i.e. language, visual, and acoustic modalities that each uses a different sampling rate. Earlier works assumed multimodal sequences are aligned based on word boundaries (Lazaridou et al., 2015; Ngiam et al., 2011; Gu et al., 2018; Dumpala et al., 2019; Pham et al., 2019) and applied fusion methods for aligned sequences. To date, modeling unaligned multimodal language sequences remains understudied, except for (Tsai et al., 2019a; Khare et al., 2020; Zheng et al., 2020), which used cross-modal Transformers to model unaligned multimodal language sequences. However, the cross-modal Transformer module is a bi-modal operation that only account for two modalities' input at a time. In Tsai et al. (2019a), the authors used multiple cross-modal Transformers and applies late fusion to obtain trimodal features, resulting in a large amount of parameters needed to retain original modality information. Other works that also used cross-modal Transformer architecture for include Yang et al. (2020); Siriwardhana et al. (2020). In contrast to the existing works, our proposed graph method,
with very small amount of model parameters, can aggregate information from multiple (more than 2) modalities at early stage by building edges between the corresponding modalities, allowing richer and more complex representation of the interactions to be learned.
Graph Neural Networks Graph Neural Network (GNN) was introduced in (Gori et al., 2005; Scarselli et al., 2008) with an attempt to extend deep neural networks to handle graph-structured data. Since then, there has been an increasing research interest on generalizing deep neural network's operations such as convolution (Kipf and Welling, 2016; Schlichtkrull et al., 2017; Hamilton et al., 2017), recurrence (Nicoliciou et al., 2019), and attention (Veličković et al., 2018) to graph.
Recently, several heterogeneous GNN methods (Wang et al., 2019a; Wei et al., 2019; Shi et al., 2016) have been proposed. The heterogeneous nodes referred in these works consist of uni-modal views of multiple data generating sources (such as movie metadata node, audience metadata node, etc.), whereas in our case the graph nodes represent multimodal views of a single data generating source (visual, acoustic, textual nodes from a single speaking person). In the NLP domain, multimodal GNN methods (Khademi, 2020; Yin et al., 2020) on tasks such as Visual Question Answering and Machine Translation. However, these settings still differ from ours because they focused on static images and short text which, unlike the multimodal video data in our case, do not exhibit long-term temporal dependencies across modalities.
Based on these findings, we discovered there has been little research using graph-based methods for modeling unaligned, multimodal language sequences, which includes video, audio and text. In this paper, we demonstrate our proposed MTAG method can effectively model such unaligned, multimodal sequential data.
3 MTAG
In this section, we describe our proposed framework: Modal Temporal Attention Graph (MTAG) for unaligned multimodal language sequences. We describe how we formulate the multimodal data into a graph  $\mathcal{G}(\mathcal{V},\mathcal{E})$ , and the MTAG fusion operation that operates on  $\mathcal{G}$ . In essence, our graph formulation by design alleviates the need for any hard alignments, and combined with MTAG fusion, allows nodes from one modality to interact
Notation	Explanation
vi	Node i
eij	Edge from vi to vj
Ni	Neighbor nodes incident into vi
xi	Initial node feature for vi
x′i	Transformed node feature for vi
πi	Node type for vi
φij	Edge modality type for eij
τij	Edge temporal type for eij
Mπi	Node type specific transformation matrix
aφij,τij	Edge type specific learnable attention vector
βi,j	Raw attention score of node pair (vi,vj)
αi,j	Attention weight of node pair (vi,vj), normalized over Ni
zi	Node output feature for vi
k	Prune percentage
h	Index of multi-head attention head
H	Number of total attention heads
Table 1: Terminologies used in this paper.
freely with nodes from all other modalities at the same time, breaking the limitation of only modeling pairwise modality interactions in previous works. Figure 2 gives a high-level overview of the framework.
As illustrated in Figure 2, each modality's input feature vectors are first passed through a modality-specific Feed-Forward-Network. This allows feature embeddings from different modalities to be transformed into the same dimension. A positional embedding (details in Appendix A) is then added (separately for each modality) to each embedding to encode temporal information. The output of this operation becomes a node  $v_{i}$  in the graph. Each node is marked with a modality identifier  $\pi_{i}$ , where  $\pi_{i} \in \{\text{Audio}, \text{Video}, \text{Text}\}$  in our case.
In this section, we describe our design of modality edges and temporal edges. For a given node of a particular modality, its interactions with nodes from different modalities should be considered differently. For example, given a Video node, its interaction with an Audio node should be different from that with a Text node. In addition, the temporal order of the nodes also plays a key role in multimodal analysis (Poria et al., 2017). For example, a transition from a frown to a smile  $(\odot \rightarrow \odot \rightarrow \odot)$  may imply a positive sentiment, whereas a transition from a smile to a frown  $(\odot \rightarrow \odot \rightarrow \odot)$  may imply a negative sentiment. Therefore, interactions
between nodes that appear in different temporal orders should also be considered differently. In GNNs, the edges define how node features are aggregated within a graph. In order to encapsulate the diverse types of node interactions, we assign edge types to each edge so that information can be aggregated differently on different types of edges. By indexing edges with edge types, different modal and temporal interactions between nodes can be addressed separately.
Multimodal Edges. As we make no assumption about prior alignment of the modalities, the graph is initialized to be a fully connected graph. We use  $e_{ij}$  to represent an edge from  $v_i$  to  $v_j$ . We assign  $e_{ij}$  with a modality type identifier  $\phi_{ij} = (\pi_i \rightarrow \pi_j)$ . For example, an edge pointing from a Video node to a Text node will be marked with type  $\phi_{ij} = (\text{Video} \rightarrow \text{Text})$ .
Temporal Edges. In addition to  $\phi_{ij}$ , we also assign a temporal label  $\tau_{ij}$  to each  $e_{ij}$ . Depending on the temporal order of  $v_{i}$  and  $v_{j}$  connected by  $e_{ij}$ , we determine the value of  $\tau_{ij}$  to be either of  $\{\text{past, present, future}\}$ . For nodes from the same modality, the temporal orders can be easily determined by comparing their order of occurrences. To determine the temporal orders for nodes across different modalities, we first roughly align the two modalities with our pseudo-alignment. Then the temporal order can be simply read out.
Pseudo-Alignment. As mentioned above, it is simple to determine the temporal edge types for nodes in a single modality. However, there is no clear definition of "earlier" or "later" across two modalities, due to the unaligned nature of our input sequences. To this end, we introduce the pseudo-alignment heuristic that coarsely defines the past, present and future connections between nodes across two modalities. Given a node  $v_{i}$  from one modality  $\pi_{i}$ , our pseudo-alignment first determines a set of nodes  $\mathcal{V}_{i, \text{present}}$  in the other modality that can be aligned to  $v_{i}$  and considered as "present". All nodes in the other modality that exist after  $\mathcal{V}_{i, \text{present}}$  are considered "future"  $\mathcal{V}_{i, \text{future}}$ , and all those before are considered  $\mathcal{V}_{i, \text{past}}$ . Once the coarse temporal order is established, the cross-modal temporal edge types can be easily determined. Figure 3 shows an example of such pseudo-alignment, and more details regarding the calculations can be found in Appendix A.2.
Figure 3: An example of the pseudo-alignment between two unaligned sequences. We first align the longer sequence to the shorter one as uniformly as possible. Then the aligned nodes from the longer sequence become the  $\nu_{i, \text{present}}$  for node  $v_i$  in the shorter sequence.  $\nu_{i, \text{past}}$  and  $\nu_{i, \text{future}}$  can then be determined accordingly.
3.3 Fusion and Pruning
3.3.1 MTAG Fusion
With our formulation of the graph, we design the MTAG fusion operation that can digest our graph data with various node and edge types, and thus model the modal-temporal interactions. An algorithm of our method is shown in Algorithm 1 and a visual illustration is given in Figure 4. Specifically, for each neighbor node  $v_{j}$  that has an edge incident into a center node  $v_{i}$ , we compute a raw attention score  $\beta_{[h],i,j}$  based on that edge's modality and temporal type:
$$
\beta_ {[ h ], i, j} = \text {L e a k y R e l u} \left(\mathbf {a} _ {[ h ]} ^ {\phi_ {j i}, \tau_ {j i}} \cdot \left[ \mathbf {x} _ {i} ^ {\prime} \| \mathbf {x} _ {j} ^ {\prime} \right]\right) \tag {1}
$$
where  $[\cdot ||\cdot ]$  denotes the concatenation of two column vectors into one long column vector. The  $[h]$  index is used to distinguish which multi-head attention head is being used. Note that  $\mathbf{a}_{[h]}^{\phi_{ji},\tau_{ji}}$  depends on both the modality and temporal edge types of  $e_{ji}$ . This results in 27 edge types (9 types of modality interaction  $\times 3$  types of temporal interaction).
We normalize the raw attention scores over all neighbor nodes  $v_{j}$  with Softmax so that the normalized attention weight sums to 1 to preserve the scale of the node features in the graph.
$$
\alpha_ {[ h ], i, j} = \frac {\exp \left(\beta_ {[ h ] , i , j}\right)}{\sum_ {k \in \mathcal {N} _ {i}} \exp \left(\beta_ {[ h ] , i , k}\right)} \tag {2}
$$
Then, we perform node feature aggregation for each node  $v_{i}$  following:
$$
\mathbf {z} _ {i} = \underset {h = 1} {\operatorname {c o n c a t}} ^ {H} \left(\sum_ {j \in \mathcal {N} _ {i}} \alpha_ {[ h ], i, j} \mathbf {x} _ {j} ^ {\prime}\right) \tag {3}
$$
Algorithm 1: MTAG with edge pruning
1 Feature transformation  $\mathbf{x}_i^{\prime}\gets \mathbf{M}_{\pi_i}\mathbf{x}_i,\forall i$
2 for  $h = 1\dots H$  do
$\pmb{\mathfrak{o}}$  calculate node output feature
$$
\mathbf {z} _ {i} = \underset {h = 1} {\overset {H} {\operatorname {c o n c a t}}} \left(\sum_ {j \in \mathcal {N} _ {i} \cup i} \alpha_ {[ h ], i, j} \mathbf {x} _ {j} ^ {\prime}\right)
$$
7 calculate average attention weight across all
heads  $\overline{\alpha}_{i,j} = \frac{1}{H}\sum_{h = 1}^{H}(\alpha_{[h],i,j})$
8 sort  $\overline{\alpha}_{i,j}$  and delete the edges with the smallest  $k\%$  average attention weight from
$\mathcal{N}_i \cup i$ , obtaining  $\mathcal{N}_i'$
9 return  $\mathbf{z}_i, \mathcal{N}_i'$ $\forall i$
where  $\mathcal{N}_i$  defines the neighbors of  $v_{i}$  and hyperparameter  $H$  is the number of total attention heads.  $\mathbf{z}_i$  now becomes the new node embedding for node  $v_{i}$ . After aggregation,  $v_{i}$  transformed from a node with unimodal information into a node encoding the diverse modal-temporal interactions between  $v_{i}$  and its neighbors (illustrated by the mixing of colors of the nodes in Figure 2).
We design the operation to have  $H$  multi-head attention heads because the heterogeneous input data of the multimodal graph could be of different scales, making the variance of the data high. Adding multi-head attention could help stabilize the behavior of the operation.
3.3.2 Dynamic Edge Pruning
Our graph formulation models interactions for all 27 edge types. This design results in a very large number of edges in the graph, making the computation graph difficult to fit into GPU memories. More importantly, when there are so many edges, it is hard to avoid some of these edges from inducing spurious correlations and distracting the model from focusing on the truly important interactions (Lee et al., 2019; Knyazev et al., 2019). To address these challenges, we propose to dynamically prune edges as the model learns the graph. Specifically, after each layer of MTAG, we have the attention weight  $\alpha_{[h],i,j}$  for each attention head  $h$  and for
Figure 4: Visualization of the MTAG operation around a single node. The text on each edge indicates which attention vector is used for that edge. Purple triangle represents a video node, green circle represents a text node and blue square represents an audio node.
each edge  $e_{ij}$ . We take the average of the attention weights over the attention heads:
$$
\bar {\alpha} _ {i j} = \frac {1}{H} \sum_ {h = 1} ^ {H} \left(\alpha_ {[ h ], i, j}\right) \tag {4}
$$
Then, we sort  $\overline{\alpha}_{ij}$  and delete  $k\%$  edges with the smallest attention weights, where  $k$  is a hyperparameter. These deleted edges will no longer be calculated in the next MTAG fusion layer. Our ablation study in Section 5.2 empirically verifies the effectiveness of this approach by comparing to no pruning and random pruning.
3.4 Graph Readout
At the end of the MTAG fusion process, we need to read out information scattered in the nodes into a single vector so that we can pass it through a classification head. Recall that the pruning process drops edges in the graph. If all edges incident into a node have been dropped, then it means that node was not updated based on its neighbors. In that case, we simply ignore that node in the readout process.
$$
\boldsymbol {\mathcal {V}} ^ {\prime} = \left\{v _ {i} \mid v _ {i} \in \boldsymbol {\mathcal {V}} \text {a n d c o u n t} \text {i n c i d e n t} \text {e d g e} \left(v _ {i}\right) > 0 \right\} \tag {5}
$$
We readout the graph by averaging all the surviving nodes' output features into one vector. This vector is then passed to a 3-layer Multi-Layer-Perceptron (MLP) to make the final prediction.
Model \ Emotion	Happy	Sad	Angry	Neutral
(Unaligned) IEMOCAP Emotions.
CTC + EF-LSTM	75.7	70.5	67.1	57.4
LF-LSTM	71.8	70.4	67.9	56.2
CTC + RAVEN	76.8	65.6	64.1	59.5
CTC + MCTN	77.5	71.7	65.6	49.3
MulT	81.9	74.1	70.2	59.7
MTAG (ours)	86.0	79.9	76.7	64.1
We empirically evaluate MTAG model on two datasets: IEMOCAP (Busso et al., 2008) and CMUMOSI (Zadeh et al., 2016); both are well-known datasets used by prior works (Liang et al., 2018; Pham et al., 2019; Tsai et al., 2019b,a) to benchmark multimodal emotion recognition and sentiment analysis.
4.1 Dataset and Metrics
IEMOCAP IEMOCAP is a multimodal emotion recognition dataset consisting of 10K videos. The task we chose is the 4-way multilabel emotion classification, classifying into happy, sad, angry and neutral. For train split, the positive/negative label ratio for each emotion is 954:1763, 338:2379, 690:2027 and 735:1982. For the test split, the ratio is 383:555, 135:803, 193:745 and 227:711. Due to this unbalanced distribution of the labels, we use F1 score as a better metric for comparison.
CMU-MOSI CMU Multimodal Opinion Sentiment Intensity is a multimodal sentiment analysis dataset with 2,199 movie review video clips. Each video clip is labeled with real-valued sentiment score within  $[-3, +3]$ , with  $+3$  being a very positive sentiment and  $-3$  a very negative one. Following previous works (Tsai et al., 2019a), we report five metrics: 7-class classification accuracy  $(\mathrm{Acc}_7)$ , binary classification accuracy  $(\mathrm{Acc}_2)$ , positive/negative sentiments), F1 score, Mean Absolute Error (MAE) and the correlation of the model's prediction with human.
We follow prior works (Tsai et al., 2019a) to evaluate on both of the above unaligned datasets, where original audio and video features are used, resulting in variable sequence lengths across modalities. For both datasets, the multimodal features are extracted from the textual (GloVe word embeddings
Table 2: F1 Scores on unaligned IEMOCAP. Higher is better.
Model\Metirc	Acc7↑	Acc2↑	F1↑	MAE↓	Corr ↑
(Unaligned) CMU-MOSI Sentiment
CTC+EF-LSTM	31.0	73.6	74.5	1.078	0.542
LF-LSTM	33.7	77.6	77.8	0.988	0.624
CTC+MCTN	32.7	75.9	76.4	0.991	0.613
CTC+RAVEN	31.7	72.7	73.1	1.076	0.544
MulT	39.1	81.1	81.0	0.889	0.686
MTAG (ours)	38.9	82.3	82.1	0.866	0.722
Table 3: Results on unaligned CMU-MOSI.  $\uparrow$  means higher is better and  $\downarrow$  means lower is better.
Model	# Parameters
MulT (previous SOTA)	2.24 M
MTAG (ours)	0.14 M
Table 4: Number of model parameters (M = Million).
(Pennington et al., 2014)), visual (Facet (iMotions, 2017)), and acoustic (COVAREP (Degottex et al., 2014)) data modalities.
For basleine evaluations, we use Early Fusion LSTM (EF-LSTM) and Late Fusion LSTM (LF-LSTM) (Tsai et al., 2019a) as baselines. In addition, we compare our model against similar methods as in previous works (Tsai et al., 2019a), which combine a Connectionist Temporal Classification (CTC) loss (Graves et al., 2006) with the pre-existing methods such as EF-LSTM, MCTN (Pham et al., 2019), RAVEN (Wang et al., 2019b).
Shown in Table 2 and Table 3, MTAG substantially out-performs previous methods on unaligned IEMOCAP benchmark and CMU-MOSI benchmark on most of the metrics. MTAG also achieves on-par performance on the Acc7 metric on CMU-MOSI benchmark. With an extremely small number of parameters, our model is able to learn better alignment and fusion for multimodal sentiment analysis task. Details regarding our model and hyper-parameters can be found in the Appendix A.
Parameter Efficiency (MTAG vs MulT) We discover that MTAG is a highly parameter-efficient model. A comparison of model parameters between MTAG and MulT (Tsai et al., 2019a) (previous state of the art) is shown in Table 4. The hyperparameter used for this comparison can be found
in the Appendix. With only a fraction (6.25%) of MulT's parameters, MTAG is able to achieve on-par, and in most cases superior performance on the two datasets. This demonstrates the parameter efficiency of our method.
Qualitative Analysis The attention weights on the graph edges forms a natural way to interpret our model. We visualize the edges to probe what MTAG has learned. The following case study is a randomly selected video clip from the CMU-MOSI validation set. We observe the phenomena shown below is a general trend.