Venue: NAACL 2019

# Strong and Simple Baselines for Multimodal Utterance Embeddings

Paul Pu Liang\({}^{*}\), Yao Chong Lim\({}^{*}\),

**Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency School of Computer Science, Carnegie Mellon University {pliang,yaochonl,yaohungt,rsalakhu,morenyc}@cs.cmu.edu**

###### Abstract

Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intomations. Learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information. Recent advances in multimodal learning have followed the general trend of building more complex models that utilize various attention, memory and recurrent components. In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a conditional factorization of the utterance into unimodal factors. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the embedding. We show that the optimal embedding can be derived in closed form by taking a weighted average of the unimodal features. In order to capture richer representations, our second baseline extends the first by factorizing into unimodal, bimodal, and trimodal factors, while retaining simplicity and efficiency during learning and inference. From a set of experiments across two tasks, we show strong performance on both supervised and semi-supervised multimodal prediction, as well as significant (10 times) speedups over neural models during inference. Overall, we believe that our strong baseline models offer new benchmarking options for future research in multimodal learning.

+
Footnote †: authors contributed equally

## 1 Introduction

Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intomations Streeck and Knapp (1992). At the heart of many multimodal modeling tasks lies the challenge of learning rich representations of spoken utterances from multiple modalities Papo et al. (2014). However, learning representations for these spoken utterances is a complex research problem due to the presence of multiple heterogeneous sources of information Baltrusaitis et al. (2017). This challenging yet crucial research area has real-world applications in robotics Montalvo et al. (2017); Noda et al. (2014), dialogue systems Johnston et al. (2002); Rudnicky (2005), intelligent tutoring systems Mao and Li (2012); Banda and Robinson (2011); Pham and Wang (2018), and healthcare diagnosis Wentzel and van der Geest (2016); Lisetti et al. (2003); Sonntag (2017). Recent progress on multimodal representation learning has investigated various neural models that utilize one or more of attention, memory and recurrent components Yang et al. (2017); Liang et al. (2018). There has also been a general trend of building more complicated models for improved performance.

In this paper, we propose two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a factorization of the utterance into unimodal factors conditioned on the joint embedding. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the utterance embedding. We derive a coordinate-ascent style algorithm Wright (2015) to learn the optimal multimodal embeddings under our model. We show that, under some assumptions, maximum likelihood estimation for the utterance embedding can be derived in closed form and is equivalent to computing a weighted average of the language, visual and acoustic features. Only a few linear transformation parameters need to be learned. In order to capture bimodal and trimodal representations, our second baseline extends the first one by assuming a factorization into unimodal, bimodal, and trimodal factors Zadeh et al. (2017). To summarize, our simple baselines 1) consist primarily of linear functions, 2) have few parameters, and 3) can be approximately solved in a closed form solution. As a result, they demonstrate simplicity and

efficiency during learning and inference.

We perform a set of experiments across two tasks and datasets spanning multimodal personality traits recognition (Park et al., 2014) and multimodal sentiment analysis (Zadeh et al., 2016). Our proposed baseline models 1) achieve competitive performance on supervised multimodal learning, 2) improve upon classical deep autoencoders for semi-supervised multimodal learning, and 3) are up to 10 times faster during inference. Overall, we believe that our baseline models offer new benchmarks for future multimodal research.

## 2 Related Work

We provide a review of _sentence embeddings_, _multimodal utterance embeddings_, and _strong baselines_.

### Language-Based Sentence Embeddings

Sentence embeddings are crucial for down-stream tasks such as document classification, opinion analysis, and machine translation. With the advent of deep neural networks, multiple network designs such as Recurrent Neural Networks (RNNs) (Rumelhart et al., 1986), Long-Short Term Memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997), Temporal Convolutional Networks (Bai et al., 2018), and the Transformer (Vaswani et al., 2017) have been proposed and achieve superior performance. However, more training data is required for larger models (Peters et al., 2018). In light of this challenge, researchers have started to leverage unsupervised training objectives to learn sentence embedding which showed state-of-the-art performance across multiple tasks (Devlin et al., 2018). In our paper, we go beyond unimodal language-based sentence embeddings and consider multimodal spoken utterances where additional information from the nonverbal behaviors is crucial to infer speaker intent.

### Multimodal Utterance Embeddings

Learning multimodal utterance embeddings brings a new level of complexity as it requires modeling both intra-modal and inter-modal interactions (Liang et al., 2018). Previous approaches have explored variants of graphical models and neural networks for multimodal data. RNNs (Elman, 1990; Jain and Medsker, 1999), LSTMs (Hochreiter and Schmidhuber, 1997), and convolutional neural networks (Krizhevsky et al., 2012) have been extended for multimodal settings (Rajagopalan et al., 2016; Lee et al., 2018). Experiments on more advanced networks suggested that encouraging correlation between modalities (Yang et al., 2017), enforcing disentanglement on multimodal representations (Tsai et al., 2018), and using attention to weight modalities (Gulrajani et al., 2017) led to better performing multimodal representations. In our paper, we present a new perspective on learning multimodal utterance embeddings by assuming a conditional factorization over the language, visual and acoustic features. Our simple but strong baseline models offer an alternative approach that is extremely fast and competitive on both supervised and semi-supervised prediction tasks.

### Strong Baseline Models

A recent trend in NLP research has been geared towards building simple but strong baselines (Arora et al., 2017; Shen et al., 2018; Wieting and Kiela, 2019; Denkowski and Neubig, 2017). The effectiveness of these baselines indicate that complicated network components are not always required. For example, Arora et al. (2017) constructed sentence embeddings from weighted combinations of word embeddings which requires no trainable parameters yet generalizes well to downstream tasks. Shen et al. (2018) proposed parameter-free pooling operations on word embeddings for document classification, text sequence matching, and text tagging. Wieting and Kiela (2019) discovered that random sentence encoders achieve competitive performance as compared to larger models that involve expensive training and tuning. Denkowski and Neubig (2017) emphasized the importance of choosing a basic neural machine translation model and carefully reporting the relative gains achieved by the proposed techniques. Authors in other domains have also highlighted the importance of developing strong baselines (Lakshminarayanan et al., 2017; Sharif Razavian et al., 2014). To the best of our knowledge, our paper is the first to propose and evaluate strong, non-neural baselines for multi-modal utterance embeddings.

## 3 Baselines for Multimodal Learning

### Notation

Suppose we are given video data where each utterance segment is denoted as \({\bf s}\). Each segment contains individual words \(w\) in a sequence \({\bf w}\), visual features \(v\) in a sequence \({\bf v}\), and acoustic features \(a\) in a sequence \({\bf a}\). We aim to learn a representation 

\(m_{\mathbf{s}}\) for each segment that summarizes information present in the multimodal utterance.

### Background

Our model is related to the work done by Arora et al. (2016) and Arora et al. (2017). In the following, we first provide a brief review of their method. Given a sentence, Arora et al. (2016) aims to learn a sentence embedding \(c_{s}\). They do so by assuming that the probability of observing a word \(w_{t}\) at time \(t\) is given by a log-linear word production model Mnih and Hinton (2007) with respect to \(c_{s}\):

\[\mathbb{P}[w_{t}|c_{s}]=\frac{\exp\left(\langle v_{wt_{i}},c_{s}\rangle\right) }{Z_{c_{s}}}, \tag{1}\]

where \(c_{s}\) is the sentence embedding (context), represents the word vector associated with word \(w_{t}\) and \(Z_{c_{s}}=\sum_{w\in V}\exp\left(\langle w_{v},c_{s}\rangle\right)\) is a normalizing constant over all words in the vocabulary. Given this posterior probability, the desired sentence embedding \(c_{s}\) can be obtained by maximizing Equation (1) with respect to \(c_{s}\). Under some assumptions on \(c_{s}\), this maximization yields a closed-form solution which provides an efficient learning algorithm for sentence embeddings.

Arora et al. (2017) further extends this model by introducing a "smoothing term" \(\alpha\) to account for the production of frequent stop words or out of context words independent of the discourse vector. Given estimated unigram probabilities \(p(w)\), the probability of a word at time \(t\) is given by

\[\mathbb{P}[w_{t}|c_{s}]=\alpha p(w_{t})+(1-\alpha)\frac{\exp\left(\langle v_{ w_{t}},c_{s}\rangle\right)}{Z_{c_{s}}}. \tag{2}\]

Under this model with the additional hyperparameter \(\alpha\), we can still obtain a closed-form solution for the optimal \(c_{s}\).

### Baseline 1: Factorized Unimodal Model

In this subsection, we outline our method for learning representations of multimodal utterances. An overview of our proposed baseline model is shown in Figure 1. Our method begins by assuming a factorization of the multimodal utterance into unimodal factors conditioned on the joint utterance embedding. Next, each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the utterance embedding. Finally, we incorporate positional encodings to represent temporal information in the features. We first present the details of our proposed baseline before deriving a coordinate ascent style optimization algorithm to learn utterance embeddings in our model.

**Unimodal Factorization:** We use \(m_{\mathbf{s}}\) to represent the multimodal utterance embedding. To begin, we simplify the composition of \(m_{\mathbf{s}}\) by assuming that the segment \(\mathbf{s}\) can be conditionally factorized into words (\(\mathbf{w}\)), visual features (\(\mathbf{v}\)), and acoustic features (\(\mathbf{a}\)). Each factor is also associated with a

Figure 1: Our baseline model assumes a factorization of the multimodal utterance into unimodal factors conditioned on the joint utterance embedding. Each unimodal factor is modeled using the simple form of a likelihood function obtained via a linear transformation of the utterance embedding. We show that, under some assumptions, maximum likelihood estimation for the utterance embedding can be derived in closed form and is equivalent to taking a weighted average of the language, visual and acoustic features.

 

temperature hyperparameter (\(\alpha_{\mathbf{w}}\), \(\alpha_{\mathbf{v}}\), \(\alpha_{\mathbf{a}}\)) that represents the contribution of each factor towards the multimodal utterance. The likelihood of a segment \(\mathbf{s}\) given the embedding \(m_{\mathbf{s}}\) is therefore

\[\mathbb{P}[\mathbf{s}|m_{\mathbf{s}}]=\mathbb{P}[\mathbf{w}|m_{ \mathbf{s}}]^{\alpha_{\mathbf{w}}}\left(\mathbb{P}[\mathbf{v}|m_{\mathbf{s}}] \right)^{\alpha_{\mathbf{v}}}\mathbb{P}[\mathbf{a}|m_{\mathbf{s}}]^{\alpha_{ \mathbf{a}}}\] \[=\prod_{w\in\mathbf{w}}\mathbb{P}[w|m_{\mathbf{s}}]^{\alpha_{ \mathbf{w}}}\prod_{v\in\mathbf{v}}\mathbb{P}[v|m_{\mathbf{s}}]^{\alpha_{ \mathbf{v}}}\prod_{a\in\mathbf{a}}\mathbb{P}[a|m_{\mathbf{s}}]^{\alpha_{ \mathbf{a}}}. \tag{3}\]

Choice of Likelihood FunctionsAs suggested by Arora et al. (2017), given \(m_{\mathbf{s}}\), we model the probability of a word \(w\) using Equation (2). In order to analytically solve for \(m_{\mathbf{s}}\), a lemma is introduced by Arora et al. (2016, 2017) which states that the partition function \(Z_{m_{\mathbf{s}}}\) is concentrated around some constant \(Z\) (for all \(m_{\mathbf{s}}\)). This lemma is also known as the "self-normalizing" phenomenon of log-linear models Andreas and Klein (2015); Andreas et al. (2015). We use the same assumption and treat \(Z_{m_{\mathbf{s}}}\approx Z\) for all \(m_{\mathbf{s}}\).

Unlike discrete text tokens, the visual features are continuous. We assume that the visual features are generated from an isotropic Gaussian distribution. In section 5.1, we visually analyze the distribution of the features for real world datasets and show that these likelihood modeling assumptions are indeed justified. The Gaussian distribution is parametrized by simple linear transformations \(W_{v}^{\mu},W_{v}^{\sigma}\in\mathbb{R}^{|v|\times|m_{\mathbf{s}}|}\) and \(b_{v}^{\mu},b_{v}^{\sigma}\in\mathbb{R}^{|v|}\):

\[v|m_{\mathbf{s}} \sim\mathcal{N}(\mu_{v},\sigma_{v}^{2}), \tag{4}\] \[\mu_{v} =W_{v}^{\mu}m_{\mathbf{s}}+b_{v}^{\mu},\] (5) \[\sigma_{v} =\text{diag}\left(\exp\left(W_{v}^{\sigma}m_{\mathbf{s}}+b_{v}^{ \sigma}\right)\right). \tag{6}\]

Similarly, we also assume that the continuous acoustic features are generated from a different isotropic Gaussian distribution parametrized as:

\[a|m_{\mathbf{s}} \sim\mathcal{N}(\mu_{\alpha},\sigma_{a}^{2}), \tag{7}\] \[\mu_{a} =W_{a}^{\mu}m_{\mathbf{s}}+b_{a}^{\mu},\] (8) \[\sigma_{a} =\text{diag}\left(\exp\left(W_{a}^{\sigma}m_{\mathbf{s}}+b_{v}^{ \sigma}\right)\right). \tag{9}\]

Positional EncodingsFinally, we incorporate positional encodings Vaswani et al. (2017) into the features to represent temporal information. We use \(d\)-dimensional positional encodings with entries:

\[PE_{pos,2i} =\sin\left(pos/10000^{2i/d}\right), \tag{10}\] \[PE_{pos,2i+1} =\cos\left(pos/10000^{2i/d}\right). \tag{11}\]

where \(pos\) is the position (time step) and \(i\in[1,d]\) indexes the dimension of the positional encodings. We call this resulting model Multimodal Baseline 1 (**MMB1**).

### Optimization for Baseline 1

We define our objective function by the log-likelihood of the observed multimodal utterance \(\mathbf{s}\). The maximum likelihood estimator of the utterance embedding \(m_{\mathbf{s}}\) and the linear transformation parameters \(W\) and \(b\) can then be obtained by maximizing this objective

\[\mathcal{L}(m_{\mathbf{s}},W,b;\mathbf{s})=\log\mathbb{P}[\mathbf{ s}|m_{\mathbf{s}};W,b], \tag{12}\]

where we use \(W\) and \(b\) to denote all linear transformation parameters.

Coordinate Ascent Style AlgorithmSince the objective (12) is not jointly convex in \(m_{\mathbf{s}}\), \(W\) and \(b\), we optimize by alternating between: 1) solving for \(m_{\mathbf{s}}\) given the parameters \(W\) and \(b\) at the current iterate, and 2) given \(m_{\mathbf{s}}\), updating \(W\) and \(b\) using a gradient-based algorithm. This resembles the coordinate ascent optimization algorithm which maximizes the objective according to each coordinate separately Tseng (2001); Wright (2015). Algorithm 1 presents our method for learning utterance embeddings. In the following sections, we describe how to solve for \(m_{\mathbf{s}}\) and update \(W\) and \(b\). Solving for \(m_{\mathbf{s}}\): We first derive an algorithm to solve for the optimal \(m_{\mathbf{s}}\) given the log likelihood objective in (12), and parameters \(W\) and \(b\).

```
1:procedureBaseline 1
2: Initialize \(m_{\mathbf{s}},W,b\).
3:foreach iterationdo
4: Fix \(W^{(k)},b^{(k)}\), compute \(m_{\mathbf{s}}^{(k)}\) by (13).
5: Fix \(m_{\mathbf{s}}^{(k)}\), compute \(\nabla_{W}\mathcal{L}\) by (21-22).
6: Fix \(m_{\mathbf{s}}^{(k)}\), compute \(\nabla_{b}\mathcal{L}\) by (23-24).
7: Update \(W^{(k+1)}\) from \(W^{(k)}\) and \(\nabla_{W}\mathcal{L}\).
8: Update \(b^{(k+1)}\) from \(b^{(k)}\) and \(\nabla_{b}\mathcal{L}\).
```

**Algorithm 1** Baseline 1 

_where \(\otimes\) denotes Hadamard (element-wise) product and the weights \(\psi\)'s are given as follows:_

\[\psi_{w}=\frac{\alpha_{\mathbf{w}}(1-\alpha)/(\alpha Z)}{p(w)+(1- \alpha)/(\alpha Z)}, \tag{16}\] \[\psi^{(1)}_{v}=\text{diag}\left(\frac{\alpha_{\mathbf{v}}}{\exp{(2 b_{v}^{\sigma})}}\right),\] (17) \[\psi^{(2)}_{v}=\text{diag}\left(\frac{\alpha_{\mathbf{v}}}{\exp{(2 b_{v}^{\sigma})}}-\alpha_{\mathbf{v}}\right),\] (18) \[\psi^{(1)}_{a}=\text{diag}\left(\frac{\alpha_{\mathbf{a}}}{\exp{(2 b_{a}^{\sigma})}}\right),\] (19) \[\psi^{(2)}_{a}=\text{diag}\left(\frac{\alpha_{\mathbf{a}}}{\exp{(2 b_{a}^{\sigma})}}-\alpha_{\mathbf{a}}\right). \tag{20}\]

Proof.: The proof is adapted from Arora et al. (2017) and involves computing the gradients \(\nabla_{m_{\mathbf{s}}}\log\mathbb{P}[\cdot|m_{\mathbf{s}}]^{\alpha}\). We express \(\log\mathbb{P}[\cdot|m_{\mathbf{s}}]\) via a Taylor expansion approximation and we observe that \(\log\mathbb{P}[\cdot|m_{\mathbf{s}}]\approx c+\langle m_{\mathbf{s}},g\rangle\) for a constant \(c\) and a vector \(g\). Then, we can obtain \(m_{\mathbf{s}}^{*}\) by computing \(\arg\max_{m_{\mathbf{s}}^{*}}\mathcal{L}(m_{\mathbf{s}},W,b;\mathbf{s})\) which yields a closed-form solution. Please refer to the supplementary material for proof details. 

Observe that the optimal embedding \(m_{\mathbf{s}}^{*}\) is a weighted average of the word features \(w\) and the (shifted and transformed) visual and acoustic features, \(\tilde{v}\) and \(\tilde{a}\). Our choice of a Gaussian likelihood for the visual and acoustic features introduces a squared term \((v-b_{v}^{\mu})\otimes(v-b_{v}^{\mu})\) to account for the \(\ell_{2}\) distance present in the pdf. The transformation matrix \(W^{\top}\) transforms the visual and acoustic features into the multimodal embedding space. Regarding the weights \(\psi\), note that: 1) the weights are proportional to the global temperatures \(\alpha\) assigned to that modality, 2) the weights \(\psi_{w}\) are inversely proportional to \(p(w)\) (rare words carry more weight), and 3) the weights \(\psi_{v}\) and \(\psi_{a}\) scale each feature dimension inversely by their magnitude.

**Updating \(W\) and \(b\):** To find the optimal linear transformation parameters \(W\) and \(b\) to maximize the objective in (12), we perform gradient-based optimization on \(W\) and \(b\) (in Algorithm 1 line 5-8).

**Proposition 1**.: _[Updating \(W\) and \(b\)] The gradients \(\nabla_{W}\mathcal{L}(m_{\mathbf{s}},W,b)\) and \(\nabla_{b}\mathcal{L}(m_{\mathbf{s}},W,b)\), in each dimension, are:_

\[\nabla_{W^{e}_{x,ij}}\mathcal{L}(m_{\mathbf{s}},W,b)=\\ -\frac{\alpha_{\mathbf{v}}}{2}\mbox{tr}\Big{[}\big{(}\sigma_{v}^ {-2}-\sigma_{v}^{-2}(v-\mu_{v})(v-\mu_{v})^{\top}\sigma_{v}^{-2}\big{)}^{\top }\sigma_{vii}m_{\mathbf{s}j}\Big{]}, \tag{22}\]

\[\nabla_{b_{v}^{\mu}}\mathcal{L}(m_{\mathbf{s}},W,b)=\\ \nabla_{b_{v}^{\mu}}\mathcal{L}(m_{\mathbf{s}},W,b)=\\ -\frac{\alpha_{\mathbf{v}}}{2}\mbox{tr}\Big{[}\big{(}\sigma_{v}^ {-2}-\sigma_{v}^{-2}(v-\mu_{v})(v-\mu_{v})^{\top}\sigma_{v}^{-2}\big{)}^{\top }\sigma_{vi}m_{\mathbf{s}j}\Big{]}, \tag{24}\]

Proof.: The proof involves differentiating the log likelihood of a multivariate Gaussian with respect to \(\mu\) and \(\sigma\) before applying the chain rule to \(\mu=W^{\mu}m_{\mathbf{s}}+b^{\mu}\) and \(\sigma=\text{diag}\left(\exp{(W^{\sigma}m_{\mathbf{s}}+b^{\sigma})}\right)\). 

### Baseline 2: Incorporating Bimodal and Trimodal Interactions

So far, we have assumed the utterance segment \(s\) can be independently factorized into unimodal features. In this subsection, we extend the setting to take account for bimodal and trimodal interactions. We adopt the idea of early-fusion (Srivastava and Salakhutdinov, 2012), which means the bimodal and trimodal interactions are captured by the concatenated features from different modalities. Specifically, we define our factorized model as:

\[\begin{split}&\mathbb{P}[\mathbf{s}|m_{\mathbf{s}}]=\mathbb{ P}[\mathbf{w}|m_{\mathbf{s}}]^{\alpha_{\mathbf{w}}}\mathbb{P}[\mathbf{v}|m_{ \mathbf{s}}]^{\alpha_{\mathbf{w}}}\mathbb{P}[\mathbf{a}|m_{\mathbf{s}}]^{ \alpha_{\mathbf{a}}}\\ &\mathbb{P}[(\mathbf{w}\oplus\mathbf{v})\,|m_{\mathbf{s}}]^{ \alpha_{\mathbf{w}\mathbf{w}}}\mathbb{P}[(\mathbf{w}\oplus\mathbf{a})\,|m_{ \mathbf{s}}]^{\alpha_{\mathbf{w}\mathbf{a}}}\\ &\mathbb{P}[(\mathbf{v}\oplus\mathbf{a})\,|m_{\mathbf{s}}]^{ \alpha_{\mathbf{w}\mathbf{a}}}\mathbb{P}[(\mathbf{w}\oplus\mathbf{v}\oplus \mathbf{a})\,|m_{\mathbf{s}}]^{\alpha_{\mathbf{w}\mathbf{a}}},\end{split} \tag{25}\]

where \(\oplus\) denotes vector concatenation for bimodal and trimodal features. Each of the individual probabilities factorize in the same way as Equation (3) (i.e. \(\mathbb{P}[\mathbf{a}|m_{\mathbf{s}}]^{\alpha_{\mathbf{a}}}=\prod_{a\texttt{a} \texttt{a}}\mathbb{P}[a|m_{\mathbf{s}}]^{\alpha_{\mathbf{a}}}\)). Similar to baseline 1, we assume a log-linear likelihood (2) for \(\mathbb{P}[w|m_{\mathbf{s}}]\) and a Gaussian likelihood (4) for all remaining terms. We call this Multimodal Baseline 2 (**MMB2**).

### Optimization for Baseline 2

The optimization algorithm derived in section 3.4 can be easily extended to learn \(m_{\mathbf{s}}\), \(W\) and \(b\) in Baseline 2. We again alternate between the 2 steps of 1) solving for \(m_{\mathbf{s}}\) given the parameters \(W\) and \(b\) at the current iterate, and 2) given \(m_{\mathbf{s}}\), updating \(W\) and \(b\) using a gradient-based algorithm.

**Solving for \(m_{\mathbf{s}}\):** We state a result that derives the closed-form of \(m_{\mathbf{s}}\) given \(W\) and \(b\) 



**Corollary 1**.: _[Solving for \(m_{\mathbf{s}}\)] Assume that the optimal \(m_{\mathbf{s}}\) lies on the unit sphere (i.e. \(\|m_{\mathbf{s}}\|^{2}_{2}=1\)). The closed-form (in Algorithm 1 line 4) for \(m_{\mathbf{s}}\) is:_

\[m^{*}_{\mathbf{s}}=\sum_{w\in\mathbf{w}}\psi_{w}w\] \[+\sum_{v\in\mathbf{v}}\Big{(}W^{\mu\intercal}_{v}\tilde{v}^{(1)} \psi^{(1)}_{v}+W^{\sigma\intercal}_{v}\tilde{v}^{(2)}\psi^{(2)}_{v}\Big{)}\] \[+\sum_{a\in\mathbf{a}}\Big{(}W^{\mu\intercal}_{a}\tilde{a}^{(1)} \psi^{(1)}_{a}+W^{\sigma\intercal}_{a}\tilde{a}^{(2)}\psi^{(2)}_{a}\Big{)}\] \[+\sum_{\begin{subarray}{c}\mathbf{f}\in\{\mathbf{w}\oplus v, \mathbf{w}\oplus a\}\\ \mathbf{v}\oplus a,\mathbf{w}\oplus\mathbf{v}\oplus\mathbf{a}\end{subarray}} \sum_{f\in\mathbf{f}}\left(W^{\mu\intercal}_{f}\tilde{f}^{(1)}\psi^{(1)}_{ f}+W^{\sigma\intercal}_{f}\tilde{f}^{(2)}\psi^{(2)}_{f}\right) \tag{26}\]

_where the shifted (and squared) visual features are:_

\[\tilde{v}^{(1)}=v-b^{\mu}_{v},~{}\tilde{v}^{(2)}=(v-b^{\mu}_{v})\otimes(v-b^{ \mu}_{v}), \tag{27}\]

_(and analogously for \(\tilde{f}^{(1)},\tilde{f}^{(2)},f\in\{a,w\oplus v,w\oplus a,v\oplus u\}\)). The weights \(\psi\)'s are:_

\[\psi_{w} =\frac{\alpha_{\mathbf{w}}(1-\alpha)/(\alpha Z)}{p(w)+(1-\alpha) /(\alpha Z)}, \tag{28}\] \[\psi^{(1)}_{v} =\text{diag}\left(\frac{\alpha_{\mathbf{v}}}{\exp{(2b^{\sigma}_{ v})}}\right),\] (29) \[\psi^{(2)}_{v} =\text{diag}\left(\frac{\alpha_{\mathbf{v}}}{\exp{(2b^{\sigma}_{ v})}}-\alpha_{\mathbf{v}}\right). \tag{30}\]

_(and analogously for \(\psi^{(1)}_{f},\psi^{(2)}_{f},f\in\{a,w\oplus v,w\oplus a,v\oplus a\}\))._

Proof.: The proof is a symmetric extension of Theorem 1 to take into account the Gaussian likelihoods for bimodal and trimodal features. 

**Updating \(W\) and \(b\):** The gradient equations for updating \(W\) and \(b\) are identical to those derived in Proposition 1, Equations (21-24).

### Multimodal Prediction

Given the optimal embeddings \(m_{\mathbf{s}}\), we can now train a classifier from \(m_{\mathbf{s}}\) to labels \(y\) for multimodal prediction. \(m_{\mathbf{s}}\) can also be fine-tuned on labeled data (i.e. taking gradient descent steps to update \(m_{\mathbf{s}}\) with respect to the task-specific loss functions) to learn task-specific multimodal utterance representations. In our experiments, we use a fully connected neural network for our classifier.

## 4 Experimental Setup

To evaluate the generalization of our models, we perform experiments on multimodal speaker traits recognition and multimodal sentiment analysis. The code for our experiments is released at [https://github.com/yaochie/multimodal-baselines](https://github.com/yaochie/multimodal-baselines), and all datasets for our experiments can be downloaded at [https://github.com/A2Zadeh/CMU-MultimodalSDK](https://github.com/A2Zadeh/CMU-MultimodalSDK).

### Datasets

All datasets consist of monologue videos where the speaker's intentions are conveyed through the language, visual and acoustic modalities. The multimodal features are described in the next subsection.

**Multimodal Speaker Traits Recognition** involves recognizing speaker traits based on multimodal utterances. **POM**Park et al. (2014) contains 903 videos each annotated for speaker traits: confident (con), voice pleasant (voi), dominance (dom), vivid (viv), reserved (res), trusting (tru), relaxed (rel), outgoing (out), thorough (tho), nervous (ner), and humorous (hum). The abbreviations (inside parentheses) are used in the tables.

**Multimodal Sentiment Analysis** involves analyzing speaker sentiment based on video content. Multimodal sentiment analysis extends conventional language-based sentiment analysis to a multimodal setup where both verbal and non-verbal signals contribute to the expression of sentiment. We use **CMU-MOSI**Zadeh et al. (2016) which consists of 2199 opinion segments from online videos each annotated with sentiment from strongly negative (\(-3\)) to strongly positive (\(+3\)).

### Multimodal Features and Alignment

GloVe word embeddings Pennington et al. (2014), Facet iMotions (2017) and COVAREP Degottex et al. (2014) are extracted for the language, visual

Figure 2: Histogram visualizations of the visual (top) and acoustic (bottom) features in some CMU-MOSI multimodal utterances. Many of the features converge to a Gaussian distribution across the time steps in the utterance, justifying our parametrization for the visual and acoustic likelihood functions.



and acoustic modalities respectively1. Forced alignment is performed using P2FA Yuan and Liberman (2008) to obtain the exact utterance times of each word. The video and audio features are aligned by computing the expectation of their features over each word interval Liang et al. (2018).

Footnote 1: Details on feature extraction are in supplementary.

### Evaluation Metrics

For classification, we report multiclass classification accuracy A(\(c\)) where \(c\) denotes the number of classes and F1 score. For regression, we report Mean Absolute Error (MAE) and Pearson's correlation (\(r\)). For MAE lower values indicate better performance. For all remaining metrics, higher values indicate better performance.

## 5 Results and Discussion

### Gaussian Likelihood Assumption

Before proceeding to the experimental results, we perform some sanity checks on our modeling assumptions. We plotted histograms of the visual and acoustic features in CMU-MOSI utterances to visually determine if they resemble a Gaussian distribution. From the plots in Figure 2, we observe that many of the features indeed converge approximately to a Gaussian distribution across the time steps in the utterance, justifying the parametrization for the visual and acoustic likelihood functions in our model.

### Supervised Learning

Our first set of experiments evaluates the performance of our baselines on two multimodal prediction tasks: multimodal sentiment analysis on CMU-MOSI and multimodal speaker traits recognition on POM. On CMU-MOSI (right side of Table 1), our model **MMB2** performs competitively against many neural models including early fusion deep neural networks Nojavanasghari et al. (2016), several variants of LSTMs stacked, bidirectional etc. Hochreiter and Schmidhuber (1997); Schuster and Paliwal (1997), Multi-view LSTMs Rajagopalan et al. (2016), and tensor product recurrent models (TFN) Zadeh et al. (2017). For multimodal personality traits recognition on POM (left side of Table 1), our baseline is able to additionally outperform more complicated memory-based recurrent models such as MFN Zadeh et al. (2018) on several metrics. We view this as an impressive achievement considering the simplicity of our model and the significantly fewer parameters that our model contains. As we will later show, our model's strong performance comes with the additional benefit of being significantly faster than the existing models.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline Dataset & \multicolumn{8}{c}{**POM Personality Trait Recognition, measured in MAE**} \\ Task & Con & Vioi & Dom & Viv & Res & Tru & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Net & Hum \\  & & Com & Vic & Res & Tri & Rel & Out & Tho & Het & Hum \\ 



### Semi-supervised Learning

Our next set of experiments evaluates the performance of our proposed baseline models when there is limited labeled data. Intuitively, we expect our model to have a lower sample complexity since training our model involves learning fewer parameters. As a result, we hypothesize that our model will generalize better when there is limited amounts of labeled data as compared to larger neural models with a greater number of parameters.

We test this hypothesis by evaluating the performance of our model on the CMU-MOSI dataset with only 40%, 60%, 80%, and 100% of the training labels. The remainder of the train set now consists of unlabeled data which is also used during training but in a semi-supervised fashion. We use the entire train set (both labeled and unlabeled data) for unsupervised learning of our multimodal embeddings before the embeddings are fine-tuned to predict the label using limited labeled data. A comparison is performed with two models that also learn embeddings from unlabeled multi-modal utterances: 1) deep averaging autoencoder (**AE**) [25, 37] which averages the temporal dimension before using a fully connected autoencoder to learn a latent embedding, and 2) sequence to sequence autoencoder (**seq2seq**) [9] which captures temporal information using a recurrent neural network encoder and decoder. For each of these models, an autoencoding model is used to learn embeddings on the entire training set (both labeled and unlabeled data) before the embeddings are fine-tuned to predict the label using limited labeled data. The validation and test sets remains unchanged for fair comparison.

Under this semi-supervised setting, we show prediction results on the CMU-MOSI test set in Table 2. Empirically, we find that our model is able to outperform deep autoencoders and their recurrent variant. Our model remains strong and only suffers a drop in performance of about 3% (75.1% \(\rightarrow\) 72.9% binary accuracy) despite having access to only 40% of the labeled training data.

### Inference Timing Comparisons

To demonstrate another strength of our model, we compare the inference times of our model with existing baselines in Table 3. Our model achieves an inference per second (IPS) of more than 10 times the closest neural model (EF-LSTM). We attribute this speedup to our (approximate) closed form solution for \(m_{\text{s}}\) as derived in Theorem 1 and Corollary 1, the small size of our model, as well as the fewer number of parameters (linear transformation parameters and classifier parameters) involved.

### Ablation Study

To further motivate our design decisions, we test some ablations of our model: 1) we remove the modeling capabilities of the visual and acoustic

\begin{table}
\begin{tabular}{l c|c|c|c} \hline \hline Dataset & & \multicolumn{2}{c}{**CMU-MOSI**} \\ Task & & & \multicolumn{2}{c}{**Sentiment**} \\ Model & PE & FT & A(2) & F1 \\ \hline MMB2, language only & ✓ & ✓ & 72.3 & 73.7 \\ MMB2 & ✗ & ✗ & 74.1 & 73.9 \\ MMB2 & ✗ & ✓ & 74.6 & 74.6 \\ MMB2 & ✓ & ✗ & 74.6 & 74.6 \\ MMB2 & ✓ & ✓ & 75.2 & **75.1** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Ablation studies on CMU-MOSI test set. Incorporating nonverbal (visual and acoustic) features, positional encodings (PE), and task-specific fine tuning (FT) are important for good prediction performance.

\begin{table}
\begin{tabular}{r l r r} \hline \hline  & Dataset & \multicolumn{2}{c}{**CMU-MOSI**} \\ \% Labels & Task & \multicolumn{2}{c}{**Sentiment**} \\  & Metric & A(2) & F1 \\ \hline  & AE & 55.4 & 54.7 \\
40\% & seq2seq & 56.4 & 49.3 \\
**MMB2** & **72.9** & **72.8** \\ \cline{2-3}  & AE & 55.2 & 54.2 \\
60\% & seq2seq & 56.3 & 51.5 \\
**MMB2** & **73.6** & **73.5** \\ \hline  & AE & 55.2 & 54.8 \\
80\% & seq2seq & 55.7 & 54.7 \\  & **MMB2** & **74.1** & **74.1** \\ \hline  & AE & 55.2 & 53.2 \\
100\% & seq2seq & 57.0 & 54.1 \\
**MMB2** & **75.1** & **75.1** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Semi-supervised sentiment prediction results on CMU-MOSI. Our model outperforms deep autoencoders (AE) and their recurrent variant (seq2seq), remaining strong despite limited labeled data.

\begin{table}
\begin{tabular}{l c c} \hline \hline Method & ‘Average Time (s)” & **Interser Per Second (IPS)** \\ \hline DF & \(\downarrow\) & 0.305 & 1850 \\ EF-LSTM & \(\downarrow\) & 0.022 & 31200 \\ MV-LSTM & \(\downarrow\) & 0.490 & 1400 \\ BC-LSTM & \(\downarrow\) & 0.210 & 3270 \\ TFN & \(\downarrow\) & 2.058 & 333 \\ MPN & \(\downarrow\) & 0.144 & 4760 \\ \hline MMB1 & **0.00163** & **421000** \\ MMB2 & \(\downarrow\) & **0.00219** & **313000** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Average time taken for inference on the CMU-MOSI test set and Inferences Per Second (IPS) on a single Nvidia GeForce GTX 1080 Ti GPU, averaged over 5 trials. Our proposed baselines are more than 10 times faster than the closest neural model (EF-LSTM).



modalities, instead modeling only the language modality, 2) we remove the positional encodings, and 3) we remove the fine tuning step. We provide these results in Table 4 and observe that each component is indeed important for our model. Although the text only model performs decently, incorporating visual and acoustic features under our modeling assumption improves performance. Our results also demonstrate the effectiveness of positional encodings and fine tuning without having to incorporate any additional learnable parameters.

## 6 Conclusion

This paper proposed two simple but strong baselines to learn embeddings of multimodal utterances. The first baseline assumes a factorization of the utterance into unimodal factors conditioned on the joint embedding while the second baseline extends the first by assuming a factorization into unimodal, bimodal, and trimodal factors. Both proposed models retain simplicity and efficiency during both learning and inference. From experiments across multimodal tasks and datasets, we show that our proposed baseline models: 1) display competitive performance on supervised multimodal prediction, 2) outperform classical deep autoencoders for semi-supervised multimodal prediction and 3) attain significant (10 times speedup during inference. Overall, we believe that our strong baseline models provide new benchmarks for future research in multi-modal learning.

## Acknowledgements

PPL and LM were partially supported by Samsung and NSF (Award 1750439). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Samsung and NSF, and no official endorsement should be inferred. YHT and RS were supported in part by the NSF IIS1763562, Office of Naval Research N000141812861, and Google focused award. We would also like to acknowledge NVIDIA's GPU support and the anonymous reviewers for their constructive comments on this paper.

## References

* Volume 1_, NIPS'15, pages 1783-1791, Cambridge, MA, USA. MIT Press. Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to pmi-based word embeddings. _Transactions of the Association for Computational Linguistics_, 4:385-399. Arora et al. (2017) Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In _International Conference on Learning Representations_, ICLR. Bai et al. (2018) Shaojie Bai, J Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. _arXiv preprint arXiv:1803.01271_. Baltrusaitis et al. (2017) Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2017. Multimodal machine learning: A survey and taxonomy. _arXiv preprint arXiv:1705.09406_. Banda and Robinson (2011) Ntombiakayise Banda and Peter Robinson. 2011. Multimodal affect recognition in intelligent tutoring systems. In _Affective Computing and Intelligent Interaction_, pages 200-207, Berlin, Heidelberg. Springer Berlin Heidelberg. Degottex et al. (2014) Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarepa collaborative voice analysis repository for speech technologies. In _Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on_, pages 960-964. IEEE. Denkowski and Neubig (2017) Michael Denkowski and Graham Neubig. 2017. Stronger baselines for trustable results in neural machine translation. In _Proceedings of the First Workshop on Neural Machine Translation_, pages 18-27. Association for Computational Linguistics. Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_. Elman (1990) Jeffrey L Elman. 1990. Finding structure in time. _Cognitive science_, 14(2):179-211. Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017. Improved training of wasserstein gans. _arXiv preprint arXiv:1704.00028_. Hinton and Salakhutdinov (2006) G. E. Hinton and R. R. Salakhutdinov. 2006. Reducing the dimensionality of data with neural networks. _Science_, 313(5786):504-507
 

* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._, 9(8):1735-1780.
* iMotions (2017) iMotions. 2017. Facial expression analysis.
* Iyyer et al. (2015) Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1681-1691. Association for Computational Linguistics.
* Jain and Medsker (1999) L. C. Jain and L. R. Medsker. 1999. _Recurrent Neural Networks: Design and Applications_, 1st edition. CRC Press, Inc., Boca Raton, FL, USA.
* Johnston et al. (2002) Michael Johnston, Srinivas Bangalore, Gunaranjan Vasireddy, Amanda Stent, Patrick Ehlen, Marilyn Walker, Steve Whittaker, and Preetam Maloor. 2002. Match: An architecture for multimodal dialogue systems. In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_, ACL '02, pages 376-383, Stroudsburg, PA, USA. Association for Computational Linguistics.
* Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 25_, pages 1097-1105. Curran Associates, Inc.
* Lakshminarayanan et al. (2017) Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In _Advances in Neural Information Processing Systems_, pages 6402-6413.
* Lee et al. (2018) Chan Woo Lee, Kyu Ye Song, Jihoon Jeong, and Woo Yong Choi. 2018. Convolutional attention networks for multimodal emotion recognition from speech and text data. _arXiv preprint arXiv:1805.06606_.
* Liang et al. (2018) Paul Pu Liang, Ziyin Liu, Amir Zadeh, and Louis-Philippe Morency. 2018. Multimodal language analysis with recurrent multistage fusion. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, EMNLP.
* Lisetti et al. (2003) C. Lisetti, F. Nasoz, C. LeRouge, O. Ozyer, and K. Alvarez. 2003. Developing multimodal intelligent affective interfaces for tele-home health care. _Int. J. Hum.-Comput. Stud._, 59(1-2):245-255.
* Mao and Li (2012) Xia Mao and Zheng Li. 2012. Multimodal intelligent tutoring systems. In Elvis Pontes, Anderson Silva, Adisson Guelfi, and Sergio Takeo Kofuji, editors, _Learning_, chapter 6. IntechOpen, Rijeka.
* Mnih and Hinton (2007) Andriy Mnih and Geoffrey Hinton. 2007. Three new graphical models for statistical language modelling. In _Proceedings of the 24th International Conference on Machine Learning_, ICML '07, pages 641-648, New York, NY, USA. ACM.
* Montalvo et al. (2017) Melissa Montalvo, Eduardo Calle-Ortiz, and Juan Chica. 2017. A multimodal robot based model for the preservation of intangible cultural heritage. In _Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_, HRI '17, pages 213-214, New York, NY, USA. ACM.

* Nojavanasghari et al. (2016) Behnaz Nojavanasghari, Deepak Gopinath, Jayanth Koushik, Takas Baltrusaitis, and Louis-Philippe Morency. 2016. Deep multimodal fusion for persuasureness prediction. In _Proceedings of the 18th ACM International Conference on Multimodal Interaction_, pages 284-288. ACM.
* Papo et al. (2014) D. Papo, G. Vigliocco, P. Pernis, R.L. Thompson, D. Vinson, and Royal Society (Great Britain). 2014. _Language as a Multimodal Phenomenom: Implications for Language Learning, Processing and Evolution ; Papers of a Theme Issue_. Philosophical Transactions of the Royal Society: Biological Sciences. Royal Society Publishing.
* Park et al. (2014) Sunghyun Park, Han Suk Shim, Motireya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2014. Computational analysis of persuasureness in social multimedia: A novel dataset and multimodal prediction approach. In _Proceedings of the 16th International Conference on Multimodal Interaction_, ICMI '14, pages 50-57, New York, NY, USA. ACM.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_, volume 14, pages 1532-1543.
* Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. _arXiv preprint arXiv:1802.05365_.
* Pham and Wang (2018) Phuong Pham and Jingtao Wang. 2018. Predicting learners' emotions in mobile mooc learning via a multimodal intelligent tutor. In _Intelligent Tutoring Systems_, pages 150-159, Cham. Springer International Publishing.
* Rajagopalan et al. (2016) Shyam Sundar Rajagopalan, Louis-Philippe Morency, Tadas Baltrusaitis, and Goecke Roland. 2016. Extending long short-term memory for multi-view structured learning. In _European Conference on Computer Vision_.
* Rudnicky (2005) Alexander I. Rudnicky. 2005. _Multimodal Dialogue Systems_, pages 3-11. Springer Netherlands, Dordrecht.

 

David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back-propagating errors. _nature_, 323(6088):533.
* Schuster and Paliwal [1997] M. Schuster and K.K. Paliwal. 1997. Bidirectional recurrent neural networks. _Trans. Sig. Proc._, 45(11):2673-2681.
* Razavian et al. [2014] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. 2014. Cnn features off-the-shelf: an astounding baseline for recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_, pages 806-813.
* Shen et al. [2018] Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018. Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 440-450. Association for Computational Linguistics.
* A multimodal multisensory interactive cognitive assessment tool. _CoRR_, abs/1709.01796.
* Srivastava and Salakhutdinov [2012] Nitish Srivastava and Ruslan R Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In _Advances in neural information processing systems_, pages 2222-2230.
* Streeck and Knapp [1992] Jurgen Streeck and Mark L. Knapp. 1992. The interaction of visual and verbal features in human communication. _Advances in Non-Verbal Communication: Sociocultural, clinical, esthetic and literary perspectives_.
* Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Tsai et al. [2018] Yao-Hung Hubert Tsai, Paul Pu Liang, Amir Zadeh, Louis-Philippe Morenev, and Ruslan Salakhutdinov. 2018. Learning factorized multimodal representations. _arXiv preprint arXiv:1806.06176_.
* Tseng [2001] P. Tseng. 2001. Convergence of a block coordinate descent method for nondifferentiable minimization. _J. Optim. Theory Appl._, 109(3):475-494.
* Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems 30_, pages 5998-6008. Curran Associates, Inc.
* Wentzel and van der Geest [2016] Jobke Wentzel and Thea van der Geest. 2016. Focus on accessibility: Multimodal healthcare technology for all. In _Proceedings of the 2016 ACM Workshop on Multimedia for Personal Health and Health Care_, MMHealth '16, pages 45-48, New York, NY, USA. ACM.
* Wieting and Kiela [2019] John Wieting and Douwe Kiela. 2019. No training required: Exploring random encoders for sentence classification. In _International Conference on Learning Representations_.
* Wright [2015] Stephen J. Wright. 2015. Coordinate descent algorithms. _Math. Program._, 151(1):3-34.
* Yang et al. [2017] Xiao Yang, Ersin Yumer, Paul Asente, Mike Kraley, Daniel Kifer, and C. Lee Giles. 2017. Learning to extract semantic structure from documents using multimodal fully convolutional neural networks. In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Yuan and Liberman [2008] Jiahong Yuan and Mark Liberman. 2008. Speaker identification on the scotus corpus. _Journal of the Acoustical Society of America_, 123(5):3878.
* Zadeh et al. [2017] Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Moreney. 2017. Tensor fusion network for multimodal sentiment analysis. In _Empirical Methods in Natural Language Processing, EMNLP_.
* Zadeh et al. [2018] Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Moreney. 2018. Memory fusion network for multi-view sequential learning. _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_.
* Zadeh et al. [2016] Amir Zadeh, Rowan Zellers, Eli Pincus, and Louis-Philippe Moreney. 2016. Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages. _IEEE Intelligent Systems_, 31(6):82-88.