Visualization of Attentions	We visually display how each attention is sensitive to different dimensions of LSTHM outputs in Figure 3 . Each column of the figure denoted by $a^k$ shows the behavior of the $k$ th attention on a sample video from CMU-MOSI. The left side of $a^k$ is $t=1$ and the right side is $t=20$ , since the sequence has 20 words. The $y$ axis shows what modality the dimension belongs to. Dark blue means high coefficients and red means low coefficients. Our observations (O) are detailed below: O1: By comparing each of the attentions together, they show diversity on which dimensions they are sensitive to, indicating that each attention is sensitive to different cross-view dynamics. O2: Some attention coefficients are not active (always red) throughout time. These dimensions carry only view-specific dynamics needed by that modality and not other modalities. Hence, they are not needed for cross-view dynamics and will carry no weight in their formation. O3: Attentions change their behaviors across time. For some coefficients, these changes are more drastic than the others.

Multi-attention Block	At each timestamp $t$ , various cross-view dynamics across the modalities can occur simultaneously. For example, the first instance can be the connection between a smile and positive phrase both happening at time $t$ . A second instance can be the occurrence of the same smile at time $t$ being connected to an excited voice at time $t-4$ , that was carried to time $t$ using the audio LSTHM memory. In both of these examples, cross-view dynamics exist at time $t$ . Therefore, not only do cross-view dynamics span across various modalities, they are scattered across time forming asynchronous cross-view dynamics. The Multi-attention Block is a network that can capture multiple different, possibly asynchronous, cross-view dynamics and encode all of them in a neural cross-view dynamics code $z_t$ . In the most important step of the Multi-attention Block, different dimensions of LSTHM outputs $h^m_t$ are assigned attention coefficients according to whether or not they form cross-view dynamics. These attention coefficients will be high if the dimension contributes to formation of a cross-view dynamics and low if they are irrelevant.

The MARN resembles the mechanism of our brains for understanding communication, where different regions independently process and understand different modalities  ,  – our LSTHM – and are connected together using neural links for multimodal information integration  – our MAB. We benchmark MARN by evaluating its understanding of different aspects of human communication covering sentiment of speech, emotions conveyed by the speaker and displayed speaker traits. We perform extensive experiments on 16 different attributes related to human communication on public multimodal datasets. Our approach shows state-of-the-art performance in modeling human communication for all datasets.

Multimodal Computational Descriptors	All the datasets consist of videos where only one speaker is in front of the camera. The descriptors we used for each of the modalities are as follows: Language All the datasets provide manual transcriptions. We use pre-trained word embeddings (glove.840B.300d)  to convert the transcripts of videos into a sequence of word vectors. The dimension of the word vectors is 300. Vision Facet  is used to extract a set of features including per-frame basic and advanced emotions and facial action units as indicators of facial muscle movement. Acoustic We use COVAREP  to extract low level acoustic features including 12 Mel-frequency cepstral coefficients (MFCCs), pitch tracking and voiced/unvoiced segmenting features, glottal source parameters, peak slope parameters and maxima dispersion quotients. Modality Alignment To reach the same time alignment between different modalities we choose the granularity of the input to be at the level of words. The words are aligned with audio using P2FA  to get their exact utterance times. Time step $t$ represents the $t$ th spoken word in the transcript.

Baseline Models	We compare the performance of our MARN to the following state-of-the-art models in multimodal sentiment analysis, speaker trait recognition, and emotion recognition. All baselines are trained for datasets for complete comparison. TFN (Tensor Fusion Network)  explicitly models view-specific and cross-view dynamics by creating a multi-dimensional tensor that captures unimodal, bimodal and trimodal interactions across three modalities. It is the current state of the art for CMU-MOSI dataset. BC-LSTM (Bidirectional Contextual LSTM)  is a model for context-dependent sentiment analysis and emotion recognition, currently state of the art on the IEMOCAP and MOUD datasets. MV-LSTM (Multi-View LSTM)  is a recurrent model that designates special regions inside one LSTM to different views of the data. C-MKL (Convolutional Neural Network (CNN) with Multiple Kernel Learning)  is a model which uses a CNN for visual feature extraction and multiple kernel learning for prediction. THMM (Tri-modal Hidden Markov Model)  performs early fusion of the modalities by concatenation and uses a HMM for classification.

We suspect that the less drastic the change in an attention dimension over time, the higher the chances of that dimension being part of multiple cross-view dynamics. Thus more attentions activate this important dimension. O4: Some attentions focus on cross-view dynamics that involve only two modalities. For example, in $a^3$ , the audio modality has no dark blue dimensions, while in $a^1$ all the modalities have dark blue dimensions. The attentions seem to have residual effects. $a^1$ shows activations over a broad set of variables while $a^4$ shows activation for fewer sets, indicating that attentions could learn to act in a complementary way.

Conclusion	In this paper we modeled multimodal human communication using a novel neural approach called the Multi-attention Recurrent Network (MARN). Our approach is designed to model both view-specific dynamics as well as cross-view dynamics continuously through time. View-specific dynamics are modeled using a Long-short Term Hybrid Memory (LSTHM) for each modality. Various cross-view dynamics are identified at each time-step using the Multi-attention Block (MAB) which outputs a multimodal neural code for the hybrid memory of LSTHM. MARN achieves state-of-the-art results in 6 publicly available datasets and across 16 different attributes related to understanding human communication.

It is differs from the second and third category since we explicitly model view-specific dynamics using a LSTHM for each modality as well as cross-view dynamics using the MAB. Finally, MARN is different from the fourth category since it explicitly models view-specific dynamics and proposes more advanced temporal modeling of cross-view dynamics.

SVM (Support Vector Machine)  a SVM is trained on the concatenated multimodal features for classification or regression  ,  ,  . To compare to another strong non-neural baseline we use RF (Random Forest)  using similar multimodal inputs. SAL-CNN (Selective Additive Learning CNN)  is a model that attempts to prevent identity-dependent information from being learned by using Gaussian corruption introduced to the neuron outputs. EF-HCRF: (Hidden Conditional Random Field)  uses a HCRF to learn a set of latent variables conditioned on the concatenated input at each time step. We also implement the following variations: 1) EF-LDHCRF (Latent Discriminative HCRFs)  are a class of models that learn hidden states in a CRF using a latent code between observed concatenated input and hidden output. 2) MV-HCRF: Multi-view HCRF  is an extension of the HCRF for Multi-view data, explicitly capturing view-shared and view specific sub-structures. 3) MV-LDHCRF: is a variation of the MV-HCRF model that uses LDHCRF instead of HCRF.

We additionally compare to a stronger recurrent baseline that uses early fusion while maintaining the factor of time. A shortcoming of these models is the lack of detailed modeling for view-specific dynamics, which in turn affects the modeling of cross-view dynamics, as well as causing overfitting on input data  . Late Fusion: Late fusion methods learn different models for each modality and combine the outputs using decision voting  ,  . While these methods are generally strong in modeling view-specific dynamics, they have shortcomings for cross-view dynamics since these inter-modality dynamics are normally more complex than a decision vote. As an example of this shortcoming, if a model is trained for sentiment analysis using the vision modality and predicts negative sentiment, late fusion models have no access to whether this negative sentiment was due to a frowning face or a disgusted face. Multi-view Learning: Extensions of Hidden Markov Models  and Hidden Conditional Random Fields  ,  have been proposed for learning from multiple different views (modalities)  ,  . Extensions of LSTMs have also been proposed in a multi-view setting  . MARN is different from the first category since we model both view-specific and cross-view dynamics.

Multi-attention Recurrent Network for Human Communication Comprehension	Human face-to-face communication is a complex multimodal signal. We use words (language modality), gestures (vision modality) and changes in tone (acoustic modality) to convey our intentions. Humans easily process and understand face-to-face communication, however, comprehending this form of communication remains a significant challenge for Artificial Intelligence (AI). AI must understand each modality and the interactions between them that shape human communication. In this paper, we present a novel neural architecture for understanding human communication called the Multi-attention Recurrent Network (MARN). The main strength of our model comes from discovering interactions between modalities through time using a neural component called the Multi-attention Block (MAB) and storing them in the hybrid memory of a recurrent component called the Long-short Term Hybrid Memory (LSTHM). We perform extensive comparisons on six publicly available datasets for multimodal sentiment analysis, speaker trait recognition and emotion recognition. MARN shows state-of-the-art performance on all the datasets.

The first dimension of $\widetilde{h}_t$ contains information needed for the first cross-view dynamic highlighted using $a^1_t$ , the second dimension of $\widetilde{h}_t$ contains information for the second cross-view dynamic using $a^2_t$ , and so on until $K$ . $\widetilde{h}_t$ is high dimensional but ideally considered sparse due to presence of dimensions with zero value after element-wise multiplication with attentions. Therefore, $\widetilde{h}_t$ is split into $m$ different parts – one for each modality $m$ – and undergoes dimensionality reduction using $\mathcal {C}_m : \mathbb {R}^{K \times {d^m_{mem}}} \mapsto \mathbb {R}^{{d^m_{local}}}, \forall m \in M$ with $a^1_t$0 as the target low dimension of each modality split in $a^1_t$1 . The set of networks $a^1_t$2 maps the attended outputs of each modality $a^1_t$3 to the same vector space.

Without these attentions, MARN is not able to accurately model the cross-view dynamics. RQ3: In our experiments the MARN with only one attention (like conventional attention models) under-performs compared to the models with multiple attentions. One could argue that the models with more attentions have more parameters, and as a result their better performance may not be due to better modeling of cross-view dynamics, but rather due to more parameters. However we performed extensive grid search on the number of parameters in MARN with one attention. Increasing the number of parameters further (by increasing dense layers, LSTHM cellsizes etc.) did not improve performance. This indicates that the better performance of MARN with multiple attentions is not due to the higher number of parameters but rather due to better modeling of cross-view dynamics. RQ4: Different tasks and datasets require different number of attentions. This is highly dependent on each dataset's nature and the underlying interconnections between modalities.

Properties of Attentions	To better understand the effects of attentions, we pose four fundamental research questions (RQ) in this section as RQ1: MARN (no MAB): whether the cross-view dynamics are helpful. RQ2: MARN (no $\mathcal {A}$ ): whether the attention coefficients are needed. RQ3: MARN: whether one attention is enough to extract all cross-view dynamics. RQ4: whether different tasks and datasets require different numbers of attentions. RQ1: MARN (no MAB) model can only learn simple rules among modalities such as decision voting or simple co-occurrence rules such as Tensor Fusion baseline. Across all datasets, MARN (no MAB) is outperformed by MARN. This indicates that continuous modeling of cross-view dynamics is crucial in understanding human communication. RQ2: Whether or not the presence of the coefficients $a_t$ are crucial is an important research question. From the results tables, we notice that the MARN (no $\mathcal {A}$ ) baseline severely under-performs compared to MARN. This supports the importance of the attentions in the MAB.

Experimental Methodology	In this paper we benchmark MARN's understanding of human communication on three tasks: 1) multimodal sentiment analysis, 2) multimodal speaker traits recognition and 3) multimodal emotion recognition. We perform experimentations on six publicly available datasets and compare the performance of MARN with the performance of state-of-the-art approaches on the same datasets. To ensure generalization of the model, all the datasets are split into train, validation and test sets that include no identical speakers between sets, i.e. all the speakers in the test set are different from the train and validation sets. All models are re-trained on the same train/validation/test splits. To train the MARN for different tasks, the final outputs $h_T$ and neural cross-view dynamics code $z_T$ are the inputs to another deep neural network that performs classification (categorical cross-entropy loss function) or regression (mean squared error loss function). The code, hyperparameters and instruction on data splits are publicly available at https://github.com/A2Zadeh/MARN. Following is the description of different benchmarks.

Results on CMU-MOSI dataset	We summarize the results on the CMU-MOSI dataset in Table 1 . We are able to achieve new state-of-the-art results for this dataset in all the metrics using the MARN. This highlights our model's capability in understanding sentiment aspect of multimodal communication.

Results on ICT-MMMO, YouTube, MOUD Datasets	We achieve state-of-the-art performance with significant improvement over all the comparison metrics for two English sentiment analysis datasets. Table 2 shows the comparison of our MARN with state-of-the-art approaches for ICT-MMMO dataset as well as the comparison for YouTube dataset. To assess the generalization of the MARN to speakers communicating in different languages, we compare with state-of-the-art approaches for sentiment analysis on MOUD, with opinion utterance video clips in Spanish. The final third of Table 2 shows these results where we also achieve significant improvement over state-of-the-art approaches.

Comparison Metrics	Different datasets in our experiments have different labels. For binary classification and multiclass classification we report accuracy A $^C$ where $C$ denotes the number of classes, and F1 score. For regression we report Mean Absolute Error MAE and Pearson's correlation $r$ . For all the metrics, higher values denote better performance, except MAE where lower values denote better performance.

4) EF-HSSHCRF: (Hierarchical Sequence Summarization HCRF)  is a layered model that uses HCRFs with latent variables to learn hidden spatio-temporal dynamics. 5) MV-HSSHCRF: further extends EF-HSSHCRF by performing Multi-view hierarchical sequence summary representation. The best performing early fusion model is reported as EF-HCRF $_{(\star )}$ while the best multi-view model is reported as MV-HCRF $_{(\star )}$ , where $\star \in \lbrace  \textrm {h, l, s}\rbrace $ to represent HCRF, LDCRF and HSSCRF respectively. DF (Deep Fusion)  is a model that trains one deep model for each modality and performs decision voting on the output of each modality network. EF-LSTM (Early Fusion LSTM) concatenates the inputs from different modalities at each time-step and uses that as the input to a single LSTM. We also implement the Stacked, (EF-SLSTM) Bidirectional (EF-BLSTM) and Stacked Bidirectional (EF-SBLSTM) LSTMs for stronger baselines.

Out of 79 videos in the dataset, 49 are used for training, 10 for validation and 20 for testing.

Introduction	Humans communicate using a highly complex structure of multimodal signals. We employ three modalities in a coordinated manner to convey our intentions: language modality (words, phrases and sentences), vision modality (gestures and expressions), and acoustic modality (paralinguistics and changes in vocal tones)  . Understanding this multimodal communication is natural for humans; we do it subconsciously in the cerebrum of our brains everyday. However, giving Artificial Intelligence (AI) the capability to understand this form of communication the same way humans do, by incorporating all involved modalities, is a fundamental research challenge. Giving AI the capability to understand human communication narrows the gap in computers' understanding of humans and opens new horizons for the creation of many intelligent entities. The coordination between the different modalities in human communication introduces view-specific and cross-view dynamics  . View-specific dynamics refer to dynamics within each modality independent of other modalities. For example, the arrangement of words in a sentence according to the generative grammar of the language (language modality) or the activation of facial muscles for the presentation of a smile (vision modality).

Multimodal Sentiment Analysis	 CMU-MOSI The CMU-MOSI dataset  is a collection of 2199 opinion video clips. Each opinion video is annotated with sentiment in the range [-3,3]. There are 1284 segments in the train set, 229 in the validation set and 686 in the test set. ICT-MMMO The ICT-MMMO dataset  consists of online social review videos that encompass a strong diversity in how people express opinions, annotated at the video level for sentiment. The dataset contains 340 multimodal review videos, of which 220 are used for training, 40 for validation and 80 for testing. YouTube The YouTube dataset  contains videos from the social media web site YouTube that span a wide range of product reviews and opinion videos. Out of 46 videos, 30 are used for training, 5 for validation and 11 for testing. MOUD To show that MARN is generalizable to other languages, we perform experimentation on the MOUD dataset  which consists of product review videos in Spanish. Each video consists of multiple segments labeled to display positive, negative or neutral sentiment.

MARN Model	In this section we outline our pipeline for human communication comprehension: the Multi-attention Recurrent Network (MARN). MARN has two key components: Long-short Term Hybrid Memory and Multi-attention Block. Long-short Term Hybrid Memory (LSTHM) is an extension of the Long-short Term Memory (LSTM) by reformulating the memory component to carry hybrid information. LSTHM is intrinsically designed for multimodal setups and each modality is assigned a unique LSTHM. LSTHM has a hybrid memory that stores view-specific dynamics of its assigned modality and cross-view dynamics related to its assigned modality. The component that discovers cross-view dynamics across different modalities is called the Multi-attention Block (MAB). The MAB first uses information from hidden states of all LSTHMs at a timestep to regress coefficients to outline the multiple existing cross-view dynamics among them. It then weights the output dimensions based on these coefficients and learns a neural cross-view dynamics code for LSTHMs to update their hybrid memories. Figure 1 shows the overview of the MARN. MARN is differentiable end-to-end which allows the model to be learned efficiently using gradient decent approaches.

For each modality $m \in M$ , the input to the $m$ th LSTHM is of the form $\mathbf {X}^m=\lbrace {x}_{1}^m, {x}_{2}^m, {x}_{3}^m, \cdots , {x}_{T}^m \ ; {x}_{t}^m \in \mathbb {R}^{d_{in}^m} \rbrace $ , where ${x}^m_{t}$ is the input at time $t$ and $d^m_{in}$ is the dimensionality of the input of modality $m$ . For example if $m=l \textrm {(language)}$ , we can use word vectors with $M$0 at each time step $M$1 . $M$2 is the dimensionality of the memory for modality $M$3 . $M$4 is the (hard-)sigmoid activation function and $M$5 is the tangent hyperbolic activation function. $M$6 denotes vector concatenation and $M$7 denotes element-wise multiplication.

Cross-view dynamics refer to dynamics between modalities and are divided into synchronous and asynchronous categories. An example of synchronous cross-view dynamics is the simultaneous co-occurrence of a smile with a positive sentence and an example of asynchronous cross-view dynamics is the delayed occurrence of a laughter after the end of sentence. For machines to understand human communication, they must be able to understand these view-specific and cross-view dynamics. To model these dual dynamics in human communication, we propose a novel deep recurrent neural model called the Multi-attention Recurrent Network (MARN). MARN is distinguishable from previous approaches in that it explicitly accounts for both view-specific and cross-view dynamics in the network architecture and continuously models both dynamics through time. In MARN, view-specific dynamics within each modality are modeled using a Long-short Term Hybrid Memory (LSTHM) assigned to that modality. The hybrid memory allows each modality's LSTHM to store important cross-view dynamics related to that modality. Cross-view dynamics are discovered at each recurrence time-step using a specific neural component called the Multi-attention Block (MAB). The MAB is capable of simultaneously finding multiple cross-view dynamics in each recurrence timestep.

Multimodal Emotion Recognition	 IEMOCAP The IEMOCAP dataset  consists of 151 videos of recorded dialogues, with 2 speakers per session for a total of 302 videos across the dataset. Each segment is annotated for the presence of 9 emotions (angry, excited, fear, sad, surprised, frustrated, happy, disappointed and neutral) as well as valence, arousal and dominance. The dataset is recorded across 5 sessions with 5 pairs of speakers. To ensure speaker independent learning, the dataset is split at the level of sessions: training is performed on 3 sessions (6 distinct speakers) while validation and testing are each performed on 1 session (2 distinct speakers).

We treat speech pause as a word with vector values of all zero across dimensions. The visual and acoustic modalities follow the same granularity. We use expected feature values across the entire word for vision and acoustic since they are extracted at a higher frequency (30 Hz for vision and 100 Hz for acoustic).

Results on POM Dataset	We experiment on speaker traits recognition based on observed multimodal communicative behaviors. Table 3 shows the performance of the MARN on POM dataset, where it achieves state-of-the-art accuracies on all 11 speaker trait recognition tasks including persuasiveness and credibility.

The best performing model is reported as EF-LSTM $_{(\star )}$ , $\star \in \lbrace  \textrm {-, s, b, sb}\rbrace $ denoting vanilla, stacked, bidirectional and stacked bidirectional LSTMs respectively. Majority performs majority voting for classification tasks, and predicts the expected label for regression tasks. This baseline is useful as a lower bound of model performance. Human performance is calculated for CMU-MOSI dataset which offers per annotator results. This is the accuracy of human performance in a one-vs-rest classification/regression. Finally, MARN indicates our proposed model. Additionally, the modified baseline MARN (no MAB) removes the MAB and learns no dense cross-view dynamics code $z$ . This model can be seen as three disjoint LSTMs and is used to investigate the importance of modeling temporal cross-view dynamics. The next modified baseline MARN (no $\mathcal {A}$ ) removes the $\mathcal {A}$ deep network and sets all $K$ attention coefficients $a^k_t = 1$ ( $h^k_t = \tilde{h}^k_t$ ).

Results on IEMOCAP Dataset	Our results for multimodal emotion recognition on IEMOCAP dataset are reported in Table 4 . Our approach achieves state-of-the-art performance in emotion recognition: both emotion classification as well as continuous emotion regression except for the case of correlation in dominance which our results are competitive but not state of the art.

Similar to the LSTM, $M$8 is the input gate, $M$9 is the forget gate, and $m \in M$0 is the output gate. $m \in M$1 is the proposed update to the hybrid memory $m \in M$2 at time $m \in M$3 . $m \in M$4 is the time distributed output of each modality. The neural cross-view dynamics code $z_{t}$ is the output of the Multi-attention Block at the previous time-step and is discussed in detail in next subsection. This neural cross-view dynamics code $z_{t}$ is passed to each of the individual LSTHMs and is the hybrid factor, allowing each individual LSTHM to carry cross-view dynamics that it finds related to its modality. The set of weights $W^m_*$ , $U^m_*$ and $V^m_*$ respectively map the input of LSTHM $x^m_t$ , output of LSTHM $h^m_t$ , and neural cross-view dynamics code $z_{t}$ to each LSTHM memory space using affine transformations. [b!]

Discussion	Our experiments indicate outstanding performance of MARN in modeling various attributes related to human communication. In this section, we aim to better understand different characteristics of our model.

Related Work	Modeling multimodal human communication has been studied previously. Past approaches can be categorized as follows: Non-temporal Models: Studies have focused on simplifying the temporal aspect of cross-view dynamics  ,  ,  in order to model co-occurrences of information across the modalities. In these models, each modality is summarized in a representation by collapsing the time dimension, such as averaging the modality information through time  . While these models are successful in understanding co-occurrences, the lack of temporal modeling is a major flaw as these models cannot deal with multiple contradictory evidences, eg. if a smile and frown happen together in an utterance. Furthermore, these approaches cannot accurately model long sequences since the representation over long periods of time become less informative. Early Fusion: Approaches have used multimodal input feature concatenation instead of modeling view-specific and cross-view dynamics explicitly. In other words, these approaches rely on generic models (such as Support Vector Machines or deep neural networks) to learn both view-specific and cross-view dynamics without any specific model design. This concatenation technique is known as early fusion  ,  . Often, these early fusion approaches remove the time factor as well  ,  .

Multimodal Speaker Trait Recognition	 POM Persuasion Opinion Multimodal (POM) dataset  contains movie review videos annotated for the following speaker traits: confidence, passion, dominance, credibility, entertaining, reserved, trusting, relaxed, nervous, humorous and persuasive. 903 videos were split into 600 were for training, 100 for validation and 203 for testing.

This dimensionality reduction produces a dense code $a^1_t$4 for the $a^1_t$5 times attended dimensions of each modality. Finally, the set of all $a^1_t$6 attended modality outputs, $a^1_t$7 , are passed into a deep neural network $a^1_t$8 to generate the neural cross-view dynamics code $a^1_t$9 at time $\widetilde{h}_t$0 .

Long-short Term Hybrid Memory	Long-short Term Memory (LSTM) networks have been among the most successful models in learning from sequential data  . The most important component of the LSTM is a memory which stores a representation of its input through time. In the LSTHM model, we seek to build a memory mechanism for each modality which in addition to storing view-specific dynamics, is also able to store the cross-view dynamics that are important for that modality. This allows the memory to function in a hybrid manner. The Long-short Term Hybrid Memory is formulated in Algorithm 1. Given a set of $M$ modalities in the domain of the data, subsequently $M$ LSTHMs are built in the MARN pipeline.

This comparison shows whether explicitly outlining the cross-view dynamics using the attention coefficients is required. For MARN and MARN (no $\mathcal {A}$ ), $K$ is treated as a hyperparamter and the best value of $K$ is indicated in parenthesis next to the best reported result.

Multi-attention Recurrent Network (MARN), Long-short Term Hybrid Memory (LSTHM) and Multi-attention Block (MAB) Formulation [1] $\textrm {MARN}$ $\mathbf {X}^m$ $c_0,h_0,z_0 \leftarrow \mathbf {0}$ $t = 1, . T$ : $h_t \leftarrow \textrm {LSTHM\_Step} (\bigcup _{m \in M} \lbrace  x^m_t\rbrace , z_{t-1})$ $z_t \leftarrow \textrm {MAB\_Step} (h_t)$ $h_T, z_T$   $\textrm {LSTHM\_Step}$ $\bigcup _{m \in M} \lbrace {x}^m_t\rbrace $ , $z_{t-1}$ $m \in M$ : $\triangleleft $ for all the $M$ modalities $i

The coefficient assignment is performed multiple times due to the existence of possibly multiple such cross-view dynamics across the outputs of LSTHM. The Multi-attention Block is formulated in Algorithm 1. We assume a maximum of $K$ cross-view dynamics to be present at each timestamp $t$ . To obtain the $K$ attention coefficients, $K$ softmax distributions are assigned to the concatenated LSTHM memories using a deep neural network $\mathcal {A} : \mathbb {R}^{{d_{mem}}} \mapsto \mathbb {R}^{K \times {d_{mem}}}$ with ${d_{mem}} = \sum _{m \in M} {d^{m}_{mem}}$ . At each timestep $t$ , the output of LSTHM is the set $\lbrace h^m_t : m \in M, h^m_t \in \mathbb {R}^{{d^m_{mem}}}\rbrace $ .

Acknowledgements	This project was partially supported by Oculus research grant. We thank the reviewers for their valuable feedback.

_t^m \leftarrow \sigma (W_i^m\ x^m_t+U^m_i\ h^m_{t-1}+V^m_i\ z_{t-1}+b^m_{i})$ $f^m_t \leftarrow \sigma (W^m_{f}\ x^m_t + U^m_{f}\ h^m_{t-1} + V^m_f\ z_{t-1}+b^m_{f})$ $o^k_t \leftarrow \sigma (W^m_{o}\ x^m_t + U^m_{o}\ h^m_{t-1} + V^m_o\ z_{t-1}+b^m_{o})$

$\bar{c}_t^m \leftarrow W_{\bar{c}}^m\ x^m_t + U_{\bar{c}}^m\ h^m_{t-1} + V_{\bar{c}}^m\ z_{t-1} + b^m_{\bar{c}}$ $\bigcup _{m \in M} \lbrace {x}^m_t\rbrace $0 $\bigcup _{m \in M} \lbrace {x}^m_t\rbrace $1 $\bigcup _{m \in M} \lbrace {x}^m_t\rbrace $2   $

h_t$   $\textrm {MAB\_Step}$ $h_t$ $a_t \leftarrow \mathcal {A}(h_t; \theta _{\mathcal {A}})$ $\triangleleft $ $K$ output coefficients $\widetilde{h}_t \leftarrow a_t \odot \langle \Uparrow _K h_t \rangle $ $m \in M$ : $\triangleleft $ calculate cross-view dynamics $s^m_{t} \leftarrow \mathcal {C}_m (\widetilde{h}^m_{t}; \theta _{\mathcal {C}_m})$ $s_t \leftarrow \bigoplus _{m \in M} s^m_{t}$ $h_t$0   $z_t$

$h^m_t$0 takes the concatenation of LSTHM outputs $h^m_t$1 as input and outputs a set of $h^m_t$2 attentions $h^m_t$3 with $h^m_t$4 , $h^m_t$5 . $h^m_t$6 has a softmax layer at top of the network which takes the softmax activation along each one of the $h^m_t$7 dimensions of its output $h^m_t$8 . As a result, $h^m_t$9 which forms a probability distribution over the output dimensions. $K$0 is then broadcasted (from $K$1 to $K$2 ) and element-wise multiplied by the $K$3 to produce attended outputs $K$4 , $K$5 . $K$6 denotes broadcasting by parameter $K$7 .

In the next subsection, we first outline the Long-short Term Hybrid Memory. We then proceed to outline the Multi-attention Block and describe how the two components are integrated in the MARN.

Question: What modalities are being used in different datasets?
Answer: Language, Vision, Acoustic