Title: FACTORIZED MULTIMODAL TRANSFORMER FOR MULTIMODAL SEQUENTIAL LEARNING

Abstract: The complex world around us is inherently multimodal and sequential (continuous). Information is scattered across different modalities and requires multiple continuous sensors to be captured. As machine learning leaps towards better generalization to real world, multimodal sequential learning becomes a fundamental research area. Arguably, modeling arbitrarily distributed spatio-temporal dynamics within and across modalities is the biggest challenge in this research area. In this paper, we present a new transformer model, called the Factorized Multimodal Transformer (FMT) for multimodal sequential learning. FMT inherently models the intramodal and intermodal (involving two or more modalities) dynamics within its multimodal input in a factorized manner. The proposed factorization allows for increasing the number of self-attentions to better model the multimodal phenomena at hand; without encountering difficulties during training (e.g. overfitting) even on relatively low-resource setups. All the attention mechanisms within FMT have a full time-domain receptive field which allows them to asynchronously capture long-range multimodal dynamics. In our experiments we focus on datasets that contain the three commonly studied modalities of language, vision and acoustic. We perform a wide range of experiments, spanning across 3 well-studied datasets and 21 distinct labels. FMT shows superior performance over previously proposed models, setting new state of the art in the studied datasets.

1. INTRODUCTION:

In many naturally occurring scenarios, our perception of the world is multimodal. For example, consider multimodal language (face-to-face communication), where modalities of language, vision and acoustic are seamlessly used together for communicative intent (Kottur et al., 2019) . Such scenarios are widespread in everyday life, where continuous sensory perceptions form multimodal sequential data. Each modality within multimodal data exhibits exclusive intramodal dynamics, and presents a unique source of information. Modalities are not fully independent of each other. Relations across two (bimodal) or more (trimodal, . . . ) of them form intermodal dynamics; often asynchronous spatio-temporal dynamics which bind modalities together (Zadeh et al., 2017) .

Learning from multimodal sequential data has been an active, yet challenging research area within the field of machine learning (Baltrušaitis et al., 2018) . Various approaches relying on graphical models or RNNs have been proposed for multimodal sequential learning. Transformer models are a new class of neural models that rely on a carefully designed non-recurrent architecture for sequential modeling (Vaswani et al., 2017) . Their superior performance is attributed to a self-attention mechanism, which is uniquely capable of highlighting related information across a sequence. This self-attention is a particularly appealing mechanism for multimodal sequential learning, as it can be modified into a strong neural component for finding relations between different modalities (the cornerstone of this paper). In practice, numerous such relations may simultaneously exist within multimodal data, which would require increasing the number of attention units (i.e. heads). Increasing the number of attentions in an efficient and semantically meaningful way inside a transformer model, can boost the performance in modeling multimodal sequential data.

In this paper, we present a new transformer model for multimodal sequential learning, called Factorized Multimodal Transformer (FMT) . FMT is capable of modeling asynchronous intramodal and intermodal dynamics in an efficient manner, within one single transformer network. It does so by specifically accounting for possible sets of interactions between modalities (i.e. factorizing based on combinations) in a Factorized Multimodal Self-attention (FMS) unit. We evaluate the performance of FMT on multimodal language: a challenging type of multimodal data which exhibits idiosyncratic and asynchronous spatio-temporal relations across language, vision and acoustic modalities. FMT is compared to previously proposed approaches for multimodal sequential learning over multimodal sentiment analysis (CMU-MOSI) (Zadeh et al., 2016) , multimodal emotion recognition (IEMOCAP) (Busso et al., 2008) , and multimodal personality traits recognition (POM) (Park et al., 2014) .

2. RELATED WORKS:

The related works to studies in this paper fall into two main areas.

2.1. MULTIMODAL SEQUENTIAL LEARNING:

Modeling multimodal sequential data is among the core research areas within the field of machine learning. In this area, previous work can be classified into two main categories.

The first category of models, and arguably the simplest, are models that use early or late fusion. Early fusion uses feature concatenation of all modalities into a single modality. Subsequently, the multimodal sequential learning task is treated as a unimodal one and tackled using unimodal sequential models such as Hidden Markov Models (HMMs) (Baum & Petrie, 1966) , Hidden Conditional Random Fields (HCRFs) (Quattoni et al., 2007; Morency et al., 2007) , and RNNs (e.g. LSTMs Hochreiter & Schmidhuber (1997) ). While such models are often successful for real unimodal data (i.e. not feature concatenated multimodal data), they lack the necessary components to deal with multimodal data often causes suboptimal performance (Xu et al., 2013) . Contrary to early fusion which concatenates modalities at input level, late fusion models have relied on learning ensembles of weak classifiers from different modalities (Snoek et al., 2005; Vielzeuf et al., 2017; Nojavanasghari et al., 2016) . Hybrid methods have also been used to combine early and late fusion together (Wu et al., 2019; Nguyen & Okatani, 2018; Lazaridou et al., 2015; Hu et al., 2017) .

The second category of models comprise of models specifically designed for multimodal data. Multimodal variations of graphical models have been proposed, including Multi-view HCRFs where the potentials of the HCRF are changed to facilitate multiple modalities Song et al. (2012; 2013) . Multimodal models based on LSTMs include Multi-view LSTMs Rajagopalan et al. (2016) , Memory Fusion Network (Zadeh et al., 2018a) with its recurrent and graph variants (Liang et al., 2018; Zadeh et al., 2018c) , as well as Multi-attention Recurrent Networks (Zadeh et al., 2018b) . Studies have also proposed generic fusion techniques that can be used in various models including Tensor Fusion (Zadeh et al., 2017) and its approximate variants (Liang et al.; Liu et al., 2018) , as well as Compact Bilinear Pooling (Gao et al., 2015; Fukui et al., 2016; Kim et al., 2016) .

Many of these models, from both first and second categories, are used as baselines in this paper.

2.2. TRANSFORMER MODEL:

Transformer is a non-recurrent neural architecture designed for modeling sequential data (Vaswani et al., 2017) . It has shown superior performance across multiple NLP tasks when compared to RNN-based or convolutional architectures (Devlin et al., 2018; Vaswani et al., 2017) . This superior performance of Transformer model is largely credited to a self-attention; a neural component that allows for efficiently extracting both short and long-range dependencies within its input sequence space. Transformer models have been successfully applied to various areas within machine learning including NLP and computer vision (Yang et al., 2019; Parmar et al., 2018; Alsentzer et al., 2019) . Extending transformer to multimodal domains, specially for structured multimodal sequences is relatively understudied; with the previous works mainly focusing on using transformer models for modality alignment using cross-modal links between single transformers for each modality (Tsai et al., 2019) .

3. FACTORIZED MULTIMODAL TRANSFORMER (FMT) MODEL:

In this section, we outline the proposed Factorized Multimodal Transformer1 (FMT). Figure 1 shows the overall structure of the FMT model. The input first goes through an embedding layer, followed by multiple Multimodal Transformer Layers (MTL). Each MTL consists of multiple Factorized Multimodal Self-attentions (FMS). FMS explicitly accounts for intramodal and intermodal factors within its multimodal input. S1 and S2 are two summarization networks. They are necessary components of FMT which allow for increasing the number of attentions efficiently, without overparameterization of the FMT. Consider a multimodal sequential dataset with constituent modalities of language, vision and acoustic. The modalities are denoted as {L, V, A} from hereon for abbreviation. After resampling using a reference clock, modalities can follow the same frequency (Chen et al., 2017) . Essentially, this resampling is often based on word timestamps (i.e. word alignment). Subsequently, the dataset can be denoted as:

3.1. INPUT EMBEDDING:

D = x i = x (t,i) = l (t,i) , v (t,i) , a (t,i) t=Ti t=1 , y i N i=1

x i ∈ R Ti×dx , y i ∈ R dy are the inputs and labels.

x (t,i) = l (t,i) , v (t,i) , a (t,i

) is a triplet of language, visual and audio inputs for timestamp t in i-th datapoint. N is the total number of samples within the dataset, and T i the total number of timestamps within i-th datapoint. Zero paddings (on the left) can be used to unify the length of all sequences to a desired fixed length T . At the first step within the FMT model, each modality is passed to a unimodal embedding layer with the operation

d x = d L + d V + d A

E M∈{L,V,A} (•); R d M∈{L,V,A} → R e M∈{L,V,A} . In turn, E M takes as input m (t,i) ; m ∈ {l, v, a}.

Positional embeddings are also added to the input at this stage. The output of the embeddings collectively form x0 = l0 , v0 , â0 . We denote the dimensionality of this output as e x = e L + e V + e A .

3.2. MULTIMODAL TRANSFORMER LAYER (MTL):

After the initial embedding, FMT now consists of a stack of Multimodal Transformer Layers (MTL). MTL 1) captures factorized dynamics within multimodal data in parallel, and 2) aligns the timeasynchronous information both within and across modalities. Both of these are achieved using multiple Factorized Multimodal Self-attentions (FMS), each of which has multiple specialized selfattentions inside. The high dimensionality of the intermediate attention outputs within MTL and FMS is controlled using two distinct summarization networks. The continuation of this section provides detailed explanation of the inner-operations of MTL.

Let •,i) denote the input to the k-th MTL. We assume a total of K MTLs in a FMT (indexed 0 . . . K -1), with k = 0 being the output of the embedding layer (input to k = 0 MTL). The input of MTL, immediately goes through one/multiple2 Factorized 

EQUATION

[L] [V] [A] [L,V] [L,A] [V,A] [L,V,A] . . . . . . . . . . . . . . . . . . . . . SA L,V,A SA V,A SA L,A SA L,V SA A SA V SA L Add & Norm 𝑆1 # 𝑥 (&) ( ) 𝑙 (+,&) ( # 𝑣 (+,&) ( # 𝑎 (+,&) ( ) 𝑙 (/,&) ( # 𝑣 (/,&) ( # 𝑎 (/,&) ( ) 𝑙 (𝒯,&) ( # 𝑣 (𝒯,&) ( # 𝑎 (𝒯,&) ( Factorized Multimodal

Self-attention (FMS)

Output (to MTL)

Figure 2 : Best viewed in color. Overview of a single Factorized Multimodal Self-attention (FMS) in k-th MTL. SA is self-attention (Vaswani et al., 2017) with full time-domain receptive field (0 . . . T ).

The grayed areas are for demonstration purposes, and not a part of the implementation.

Multimodal Self-attentions (FMS). The operations inside a single Factorized Multimodal Selfattention is demonstrated in Figure 2 . For 3 modalities3 , there exist 7 distinct attentions inside a single FMS unit. Each attention has a unique receptive field with respect to modalities f ∈ F = {L,V,A,LV,LA,VA,LVA}; essentially denoting the modalities visible to the attention. Using this factorization, FMS explicitly accounts for possible unimodal, bimodal and trimodal interactions existing within the multimodal input space. All attentions within a FMS extend to the length of the sequence, and therefore can extract asynchronous relations within and across modalities. For f ∈ F , each attention within a single FMS unit is controlled by the Key K f , Query Q f , and Value V f all with dimensionality R T ×T ; parameterized respectively using affine maps W K f , W Q f , and W V f . After the attention is applied using Key, Query and Value operations (Vaswani et al., 2017) , the output of each of the attentions goes through a residual addition with its perceived input (input in the attention receptive field), followed by a normalization.

The output of the FMS contains the aligned and extracted information from the unimodal, bimodal and trimodal factors. This output is high-dimensional; essentially R 4×T ×ex (each dimension within input of shape T ×e x is present in 4 factors). Our goal is to reduce this high-dimensional data using a mapping from R 4×T ×ex → R T ×ex . Without overparameterizing the FMS, in practice, we observed this mapping can be efficiently done using a simple 1D convolutional network S1 M∈{L,V,A} (•); R4 → R. Internally, S1(•) maps its input to multiple layers of higher dimensions and subsequently to R. Using language as an example, S1 L moves across language modality dimensions e L for t = 1 . . . T and summarizes the information across all the factors. The output of this summarization applied on all modality dimensions and timesteps, is the output of FMS, which has the dimensionality R T ×ex .

In practice, there can be various possible unimodal, bimodal or trimodal interactions within a multimodal input. For example, consider multiple sets of important interactions between L and V (e.g. smile + positive word, as well as eyebrows up + excited phrase), all of which need to be highlighted and extracted. A single FMS may not be able to highlight all these interactions without diluting its intrinsic attentions. Multiple FMS can be used inside a MTL to efficiently extract diverse multimodal interactions existing in the input data 4 . Consider a total of U FMS units inside a MTL. The output of each FMS goes through a feedforward network (for each timestamp t of the FMS output).

Model \Metric BA F1 MAE Corr MV-LSTM (Rajagopalan et al., 2016) 73.9/-74.0/-1.019 0.601 TFN (Zadeh et al., 2017) 73.9/-73.4/-1.040 0.633 MARN (Zadeh et al., 2018b) 77.1/-77.0/-0.968 0.625 MFN (Zadeh et al., 2018a) 77.4/-77.3/-0.965 0.632 RMFN (Liang et al., 2018) 78.4/-78.0/-0.922 0.681 RAVEN (Wang et al., 2018) 78.0/--/-0.915 0.691 MulT (Tsai et al., 2019) -/83.0 -/82.8 0.87 0.698 FMT (ours) 81.5/83.5 81.4/83.5 0.837 0.744

Table 1 : FMT achieves superior performance over baseline models for CMU-MOSI dataset (multimodal sentiment analysis). We report BA (binary accuracy) and F1 (both higher is better), MAE (Mean-absolute Error, lower is better), and Corr (Pearson Correlation Coefficient, higher is better).

For BA and F1, we report two numbers: the number on the left side of "/" is calculated based on approach taken by Zadeh et al. (2018b) , and the right side is by Tsai et al. (2019) .

The output of this feedfoward network is residually added with its input, and subsequently normalized. The feedforward network is the same across all U FMS units and timestamps t. Subsequently, the dimensionality of the output of the normalizations collectively is R U ×T ×ex . Similar to operations performed by S1, a secondary summarization network S2 M∈{L,V,A} (•); R U → R can be used here. S2 is also a 1D convolutional network that moves across modality dimensions and different timesteps to map R U ×T ×ex to R T ×ex . The output of the secondary summarization network is the final output of MTL, and denoted as xk+1 i .

Let

xK i = lK (•,i) , vK (•,i) , âK (•,i)

be the output of last MTL in the stack. For supervision, we feed this input one timestamp at a time as input to a Gated Recurrent Unit (GRU) (Cho et al., 2014) . The prediction is conditioned on output at timestamp t = T of the GRU, using an affine map to d y .

4. EXPERIMENTAL METHODOLOGY:

In this section, we discuss the experimental methodology including tasks, datasets, computational descriptors, and comparison baselines.

4.1. TASKS AND DATASETS:

The following inherently multimodal tasks (and accompanied datasets) are studied in this paper. All the tasks are related to multimodal language: a complex and idiosyncratic sequential multimodal signal, where semantics are arbitrarily scattered across modalities (Holler & Levinson, 2019) .

Multimodal Sentiment Analysis: The first benchmark in our experiments is multimodal sentiment analysis, where the goal is to identify a speaker's sentiment based on the speaker's display of verbal and nonverbal behaviors. We use the well-studied CMU-MOSI (CMU Multimodal Opinion Sentiment Intensity) dataset for this purpose (Zadeh et al., 2016) . There are a total of 2199 data points (opinion utterances) within CMU-MOSI dataset. The dataset has real-valued sentiment intensity annotations in the range [-3, +3] . It is considered a challenging dataset due to speaker diversity (1 video per distinct speaker), topic variations and low-resource setup.

Multimodal Emotion Recognition::

The second benchmark in our experiments is multimodal emotion recognition, where the goal is to identify a speaker's emotions based on the speaker's verbal and nonverbal behaviors. We use the well-studied IEMOCAP dataset (Busso et al., 2008) . IEMOCAP consists of 151 sessions of recorded dialogues, of which there are 2 speaker's per session for a total of 302 videos across the dataset. We perform experiments for discrete emotions (Ekman, 1992) of Happy, Sad, Angry and Neutral (no emotions) -similar to previous works (Tsai et al., 2019; Wang et al., 2018) (Zadeh et al., 2017) 24.1 31.0 31.5 34.5 24.6 25.6 27.6 29.1 MARN (Zadeh et al., 2018b) 29.1 33.0 --31.5 ---MFN (Zadeh et al., 2018a) 34.5 35.5 37.4 41.9 34.5 36.9 36.0 37.9 RMFN (Liang et al., 2018) 37.4 38.4 37.4 -37.4 38.9 38.9 -MulT (Tsai et al., 2019) 34.5 34.5 36.5 38.9 37.4 36.9 37.9 (Zadeh et al., 2017) 30.5 38.9 35.5 37.4 33.0 42.4 27.6 33.0 MARN (Zadeh et al., 2018b) 36.9 -52.2 --47.3 31.0 44.8 MFN (Zadeh et al., 2018a) 38.4 57.1 53.2 46.8 47.3 47.8 34.0 47.3 RMFN (Liang et al., 2018) 39.4 -53.7 -48.3 48.3 35.0 46.8 MulT (Tsai et al., 2019) 41.4 60.6 54.2 43.3 49.3 46.3 33.5 43.3 FMT 44.8 61.1 57.6 51.7 51.7 51.2 40.4 48.3 Table 3 : FMT achieves superior performance over baseline models in POM dataset (multimodal personality traits recognition). For label abbreviations please refer to Section 4.3. MA(5,7) denotes multi-class accuracy for (5,7)-class personality labels (higher is better).

Multimodal Speaker Traits Recognition: The third benchmark in our experiments is speaker trait recognition based on communicative behavior of a speaker. It is a particularly difficult task, with 16 different speaker traits in total. We study the POM dataset which contains 1,000 movie review videos (Park et al., 2014) . Each video is annotated for various personality and speaker traits, specifically: Confident (Con), Passionate (Pas), Voice Pleasant (Voi), Dominant (Dom), Credible (Cre), Vivid (Viv), Expertise (Exp), Entertaining (Ent), Reserved (Res), Trusting (Tru), Relaxed (Rel), Outgoing (Out), Thorough (Tho), Nervous (Ner), Persuasive (Per) and Humorous (Hum). The short form of these speaker traits is indicated inside the parentheses and used for the rest of this paper.

4.2. MULTIMODAL COMPUTATIONAL DESCRIPTORS:

The following computational descriptors are used by FMT and baselines (all the baselines use the same descriptors in their original respective papers).

Language: P2FA forced alignment model (Yuan & Liberman, 2008) is used to align the text and audio at word level. From the forced alignment, the timing of words and sentences are extracted. Word-level alignment is used to unify the modality frequencies (Chen et al., 2017) . GloVe embeddings (Pennington et al., 2014) are subsequently used for word representation.

Visual: For the visual modality, the Emotient FACET (iMotions, 2017) is used to extract a set of visual features including Facial Action Units (Ekman et al., 1980) , visual indicators of emotions, and sparse facial landmarks.

Acoustic: COVAREP (Degottex et al., 2014) is used to extract the following features: fundamental frequency, quasi open quotient (Kane & Gobl, 2013) , normalized amplitude quotient, glottal source parameters (H1H2, Rd, Rd conf) (Drugman et al., 2012) , Voiced/Unvoiced segmenting features (VUV) (Drugman & Alwan, 2011) , maxima dispersion quotient (MDQ), the first 3 formants, parabolic spectral parameter (PSP), harmonic model and phase distortion mean (HMPDM 0-24) and deviations (HMPDD 0-12), spectral tilt/slope of wavelet responses (peak/slope), Mel Cepstral Coefficients (MCEP 0-24).

4.3. BASELINE MODELS AND PERFORMANCE MEASURES:

The 2019)). There are fundamental distinctions between FMT and MulT, chief among them: 1) MulT consists of 6 transformers, 3 cross-modal transformers and 3 unimodal. Naturally this increases the overall model size substantially. FMT consists of only one transformer, with components to avoid overparameterization. 2) FMT sees interactions as undirected (unlike MulT which has L → V and V → L), and therefore semantically combines two attentions in one. 3) MulT has no trimodal factors (which are important according to Section 5). 4) MulT has no direct unimodal path (e.g. only L), as input to unimodal transformers are outputs of cross-modal transformers. 5) All FMT attentions have full time-domain receptive field, while MulT splits the input based on the heads.

In their original publication, all the models report6 the performance over the datasets in Section 4.1, using the same descriptors discussed in Section 4.2. The models in this paper are compared using the following performance measures (depending on the dataset): (BA) denotes binary accuracyhigher is better, (MA5,MA7) are 5 and 7 multiclass accuracy -higher is better, (F1) denotes F1 score -higher is better, (MAE) denotes the Mean-Absolute Error -lower is better, (Corr) is Pearson Correlation Coefficient -higher is better. The hyperparameter space search for FMT (and baselines if retrained) is discussed in Appendix A.1.

5. RESULTS AND DISCUSSION:

The results of sentiment analysis experiments on CMU-MOSI dataset are presented in Table 1 .

FMT achieves superior performance than the previously proposed models for multimodal sentiment analysis. We use two approaches for calculating BA and F1 based on negative vs. non-negative sentiment (Zadeh et al., 2018b) on the left side of /, and negative vs. positive (Tsai et al., 2019) on the right side. MAE and Corr are also reported. For multimodal emotion recognition, experiments on IEMOCAP are reported in Table 2 . The performance of FMT is superior than other baselines for multimodal emotion recognition (with the exception of Happy emotion). The results of experiments for personality traits recognition on POM dataset are reported in Table 3 . We report MA5 and MA7, depending on the label. FMT outperforms baselines across all personality traits.

We study the importance of the factorization in FMT. We first remove the unimodal, bimodal and trimodal attentions from the FMT model, resulting in 3 alternative implementations of FMT. demonstrates the results of this ablation experiment over CMU-MOSI dataset. Furthermore, we use only one modality as input for FMT, to understand the importance of each modality (all other factors removed). We also replace the summarization networks with simple vector addition operation. All factors, modalities, and summarization components are needed for achieving best performance.

We also perform experiments to understand the effect of number of FMT units within each MTL.

Table 5 shows the performance trend for different number of FMT units. The model with 6 number of FMS (42 attentions in total) achieves the highest performance (6 is also the highest number we experimented with). Tsai et al. (2019) reports the best performance for CMU-MOSI dataset is achieved when using 40 attentions per cross-modal transformer (3 of each, therefore 120 attention, without counting the subsequent unimodal transformers). FMT uses fewer number of attentions than MulT, yet achieves better performance. We also experiment with number of heads for original transformer model (Vaswani et al., 2017) and compare to FMT (Appendix A.3).

6. CONCLUSION:

In this paper, we presented the Factorized Multimodal Transformer (FMT) model for multimodal sequential learning. Using a Factorized Multimodal Self-attention (FMS) within each Multimodal Transformer Layer (MTL), FMT is able to model the intra-model and inter-modal dynamics within asynchronous multimodal sequences. We compared the performance of FMT to baselines approaches over 3 publicly available datasets for multimodal sentiment analysis (CMU-MOSI, 1 label), emotion recognition (IEMOCAP, 4 labels) and personality traits recognition (POM, 16 labels). Overall, FMT achieved superior performance than previously proposed models across the studied datasets.

A APPENDIX 

A.2 NUMBER OF MTL:

We study the effect of number of MTL on FMT performance. Table 6 shows the results of this experiment. The best performance is achieved using 8 MTL layers (which was also the maximum layers we tried in our hyperparameter search).

A.3 NUMBER OF ATTENTION HEADS FOR ORIGINAL TRANSFORMER MODEL:

In this section, we discuss the effect of increasing the number of heads on the original transformer model (OTF, Vaswani et al. (2017) ). Please note that we implement the OTF to allow for all attention heads to have full input receptive field (from 1 . . . T ), similar to FMT. We increase the attention heads from 1 to 35 (after 35 does not fit on a Tesla-V100 GPU with batchsize of 20). Table 7 shows the results of increasing number of attention heads for both models. We observe that achieving superior performance is not a matter of increasing the attention heads. Even using 1 FMS unit, which leads to 7 total attention, FMT achieves higher performance than counterpart OTF.

A.4 TRAINING REMARKS FOR MORE THAN 3 MODALITIES:

In many scenarios in nature, as well as what is currently pursued in machine learning, the number of modalities goes as high as 3 (mostly language, vision and acoustic, as studied in this paper). This leads to 7 attentions within each FMS, well manageable for successful training of FMT as demonstrated in this paper. However, as the number of modalities increases, the underlying multimodal phenomena becomes more challenging to model. This causes complexities for any competitive multimodal model, regardless of their internal design. While studying these cases are beyond the scope of this paper, due to rare nature of having more than 3 main modalities modalities, for FMT, the complexity can be managed due to the factorization in FMS. We propose two approaches: 1) for high number of modalities, the involved factors can be reduced based on domain knowledge, the nature of the problem, and the assumed dependencies between modalities (e.g. removing factors between modalities that are deemed weakly related inter-modality dependencies, a greedy approach may be taken for adding factors; an approach similar to stepwise regression (Kleinbaum et al., 1988) , iteratively adding the next most important factor. Using these two methods, the model can cope with higher number of modalities with a controllable compromise between performance and overparameterization.

:

Code: github.com/removed-for-blind-review, Public Data: https://github.com/A2Zadeh/CMU-MultimodalSDK

2 Multiple FMS have the same time-domain receptive field, which is equal to the length of the input. This is contrary to the implementations of the transformer model that split the sequence based on number of attention heads.

Remarks on more than 3 modalities is in Appendix A.4.

We study the impact of number of FMS units inside MTL in Section

We use the aligned variant of MulT model, which has shown better performance than unaligned version in the original paper.

With the exception of MulT for POM dataset, which is not reported in original paper. It is trained in this paper using authors' provided github code with hyperparameter search in Appendix A.1.

Figures and Tables:

Figure 1: Overview of the proposed Factorized Multimodal Transformer (FMT) model.

denotes the dimensionality of input at each timestep, which in turn is equal to the sum of dimensionality of each modality. d y denotes the dimensionality of the associated labels of a sequence.

following strong baselines are compared to FMT: MV-LSTM (Multi-view LSTM, Rajagopalan et al. (2016)), TFN (Tensor Fusion Network, Zadeh et al. (2017)), MARN (Multi-attention rent Network, Zadeh et al. (2018b)), MFN (Memory Fusion Network, Zadeh et al. (2018a)), RAVEN (Recurrent Attended Variation Embedding Network, Wang et al. (2018)), MulT 5 (Multimodal Transformer for [Un]aligned Sequences, Tsai et al. (

. FMT achieves superior performance over baseline models (with the exception of Happy emotion) for discrete emotions in IEMOCAP (multimodal emotion recognition). We report BA (binary accuracy) and F1 (both higher is better).

Table content: <table><tr><td>Model \ Emotion</td><td colspan="2">Happy</td><td>Sad</td><td/><td colspan="2">Angry</td><td colspan="2">Neutral</td></tr><tr><td>Metric</td><td>BA</td><td>F1</td><td>BA</td><td>F1</td><td>BA</td><td>F1</td><td>BA</td><td>F1</td></tr><tr><td colspan="9">MV-LSTM (Rajagopalan et al., 2016) 85.9 81.3 80.4 74.0 85.1 84.3 67.0 66.7</td></tr><tr><td>MARN (Zadeh et al., 2018a)</td><td colspan="8">86.7 83.6 82.0 81.2 84.6 84.2 66.8 65.9</td></tr><tr><td>MFN (Zadeh et al., 2018a)</td><td colspan="8">86.5 84.0 83.5 82.1 85.0 83.7 69.6 69.2</td></tr><tr><td>RMFN (Liang et al., 2018)</td><td colspan="8">87.5 85.8 82.9 85.1 84.6 84.2 69.5 69.1</td></tr><tr><td>RAVEN (Wang et al., 2018)</td><td colspan="8">87.3 85.8 83.4 83.1 87.3 86.7 69.7 69.3</td></tr><tr><td>MulT (Tsai et al., 2019)</td><td colspan="8">90.7 88.6 86.7 86.0 87.4 87.0 72.4 70.7</td></tr><tr><td>FMT</td><td colspan="8">88.8 87.2 88.0 87.7 89.7 89.5 74.0 73.8</td></tr><tr><td>Model \ Trait</td><td colspan="8">Con Pas MA7 MA7 MA7 MA7 MA7 MA7 MA7 MA7 Voi Dom Cre Viv Exp Ent</td></tr><tr><td colspan="9">MV-LSTM (Rajagopalan et al., 2016) 25.6 28.6 28.1 34.5 25.6 32.5 32.5 29.6</td></tr><tr><td>TFN</td><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>



Table content: <table><tr><td>Model \Metric</td><td>BA</td><td>F1</td><td>MAE</td><td>Corr</td></tr><tr><td>FMT [UNI]</td><td colspan="2">80.6/82.8 80.5/82.8</td><td colspan="2">0.868 0.719</td></tr><tr><td>FMT [BI]</td><td colspan="2">81.2/81.7 81.2/81.6</td><td colspan="2">0.877 0.706</td></tr><tr><td>FMT [TRI]</td><td colspan="2">80.2/81.6 80.2/81.5</td><td colspan="2">0.874 0.705</td></tr><tr><td>FMT [L]</td><td colspan="2">77.7/79.6 77.7/79.6</td><td colspan="2">0.935 0.666</td></tr><tr><td>FMT [A]</td><td colspan="2">62.5/62.7 62.6/73.2</td><td colspan="2">1.338 0.306</td></tr><tr><td>FMT [V]</td><td colspan="2">59.3/59.3 59.4/72.7</td><td colspan="2">1.357 0.218</td></tr><tr><td>FMT [S]</td><td colspan="2">80.3/82.0 80.3/81.9</td><td colspan="2">0.860 0.734</td></tr><tr><td>FMT</td><td colspan="2">81.5/83.5 81.4/83.5</td><td colspan="2">0.837 0.744</td></tr></table>

FMT ablation studies on CMU-MOSI dataset. UNI, BI, TRI denote removing all unimodal, bimodal and trimodal factors respectively. L, A, V denote using only language, audio, and visual factors respectively. S denotes the model with summarization networks replaced by simple addition. All factors, modalities, and components are needed for achieving best performance.

Table content: <table><tr><td>Model \Metric</td><td>BA</td><td>F1</td><td>MAE</td><td>Corr</td></tr><tr><td>FMT [1]</td><td colspan="2">80.0/82.0 79.4/81.1</td><td colspan="2">0.864 0.712</td></tr><tr><td>FMT [2]</td><td colspan="2">79.7/82.2 79.7/82.2</td><td colspan="2">0.863 0.725</td></tr><tr><td>FMT [3]</td><td colspan="2">79.2/80.9 79.1/80.8</td><td colspan="2">0.905 0.698</td></tr><tr><td>FMT [4]</td><td colspan="2">79.4/81.1 79.4/81.0</td><td colspan="2">0.855 0.733</td></tr><tr><td>FMT [5]</td><td colspan="2">81.5/82.6 81.5/82.5</td><td colspan="2">0.886 0.711</td></tr><tr><td>FMT [6]</td><td colspan="2">81.5/83.5 81.4/83.5</td><td colspan="2">0.837 0.744</td></tr></table>

Multimodal sentiment analysis experiments with different number of FMS units inside MTL. The number in the square bracket indicates the number of FMS. The total number of attention is 7 times the number in bracket.

Table content: <table/>

A.1 TRAINING REMARKS AND HYPERPARAMETER SPACE SEARCHThe hyperparameters of FMT include the Adam(Kingma & Ba, 2014) learning rate ({0.001, 0.0001}), structure of summarization network (randomly picked 5 architectures from {1, 2, 3} layers of conv, with kernel shapes of {2, 5, 10, 15, 20}), number of MTL layers ({4, 6, 8} except for ablation experiments which was 2 . . . 8), number of FMT units ({4, 6}, except for ablation experiment which was 1 . . . 6), e M ∈{L,V,A} ({20, 40}), dropout (0, 0.1). The same parameters (when applicable) are used for training MulT for POM dataset (e.g. num encoder layers same as number of MTL). Furthermore, for MulT specific hyperparameters, we use similar values as Table5in the original paper. All models are trained for a maximum of 200 epochs. The hyperparameter validation is similar toZadeh et al. (2018b).

Table content: <table/>

). Alternatively, without making assumptions about Multimodal sentiment analysis experiments with different number of MTL layers. The number in the square bracket indicates the number of MTL layers.

Table content: <table><tr><td>Model \Metric</td><td>MAE</td><td>Corr</td></tr><tr><td>FMT [2]</td><td colspan="2">0.881 0.720</td></tr><tr><td>FMT [3]</td><td colspan="2">0.876 0.727</td></tr><tr><td>FMT [4]</td><td colspan="2">0.871 0.723</td></tr><tr><td>FMT [5]</td><td colspan="2">0.876 0.724</td></tr><tr><td>FMT [6]</td><td colspan="2">0.852 0.730</td></tr><tr><td>FMT [7]</td><td colspan="2">0.859 0.732</td></tr><tr><td>FMT [8]</td><td colspan="2">0.837 0.744</td></tr></table>

Results of experiments with different number of heads for OTF. The number in the square bracket indicates the number of heads.

Table content: <table/>

Peer review (iclr2020):
Summary: This work presents a *factorized multimodal transformer* (FMT) as well as the concept of *factorized multimodal self-attention* (FMS). The basic idea behind these approaches is to structure the self-attention layers in a transformer to explicitly account for the fact that there are different modalities in the input, i.e., attention is computed across modalities as well as within modalities in a structured way---rather than just naively applying a transformer to the concatenated multi-modal input. Numerous implementation adjustments are made in order to ensure the model is not too overparameterized, with the primary strategy being to use 1D CNNs to reduce the dimensionality of the internal representations after applying FMS layers. One multi-modal language benchmarks the proposed approach outperforms a number of strong and recent baselines. Assessment: Overall, this is a borderline paper. It is a very incremental contribution in terms of methodology, but the empirical results are strong. The authors themselves acknowledge that there proposed approach is very similar to Tsai et al. 2019, which also proposes a multi-modal variant of the transformer model. The key distinctions---as the authors point out---is (a) the use of 1D CNNs to reduce the dimensionality of the intermediate layers of the network and (b) the use of trimodal interactions, and (c) the use of unimodal self-attention in addition to intermodal attention, and (d) the fact that this approach uses the full time-domain as input. These differences seem to lead to better performance. However, they are not major conceptual changes or advancements and essentially amount to minor architectural differences. As another minor comment, the introduction frames the contribution as being generally relevant for multi-modal learning. However, the proposed architecture has many design decisions made specifically for the task of multimodal language modeling, and the model also assumes a time-varying input. This should be acknowledged in the paper and the framing should perhaps be adjusted. Finally, the paper is very lacking in detail (e.g., detailed equations and derivations), and the paper does not appear reproducible based on the paper description alone (i.e., without code). If the code were not linked, this would be a major issue. The paper is only understandable by reference to the original transformer work and---due to the lack technical detail---does not stand alone as a research contribution. Indeed, I believe I understand the workings of the model, but there are certain aspects that would require me to investigate the code to fully understand, which indicates that the paper could substantially benefit from improved technical writing and more detailed descriptions of the methodology. Reasons to accept: - Strong empirical results - Thorough treatment of baselines and related work Reasons to reject: - Incremental contribution (especially compared to Tsai et al., 2019) - Writing lacks sufficient technical detail