Venue: NAACL 2019

# Multi-task Learning for Multi-modal

Emotion Recognition and Sentiment Analysis

Md Shad Akhtar\({}^{\dagger}\), Dushyant Singh Chauhan\({}^{\dagger}\), Deepanway Ghosal\({}^{\dagger}\), Soujanya Poria\({}^{+}\), Asif Ekbal\({}^{\dagger}\) and Pushpak Bhattacharyya\({}^{\dagger}\)

\({}^{\dagger}\) Department of Computer Science & Engineering

Indian Institute of Technology Patna, India

{shad.pcs15, 1821CS17, deepenway.ee14, asif, pb}@iitp.ac.in

\({}^{+}\) School of Computer Science and Engineering, Nanyang Technological University, Singapore

sporia@ntu.edu.sg

###### Abstract

Related tasks often have inter-dependence on each other and perform better when solved in a joint framework. In this paper, we present a deep multi-task learning framework that jointly performs sentiment and emotion analysis both. The multi-modal inputs (i.e., _text_, _acoustic_ and _visual frames_) of a video convey diverse and distinctive information, and usually do not have equal contribution in the decision making. We propose a context-level inter-modal attention framework for simultaneously predicting the sentiment and expressed emotions of an utterance. We evaluate our proposed approach on CMU-MOSEI dataset for multi-modal sentiment and emotion analysis. Evaluation results suggest that multi-task learning framework offers improvement over the single-task framework. The proposed approach reports new state-of-the-art performance for both sentiment analysis and emotion analysis.

## 1 Introduction

With the rapid growth of social media video platforms such as Youtube, Vimeo, users now tend to upload videos on these platforms. Such video platforms offer users an opportunity to express their opinions on any topic. Videos usually consist of _audio_ and _visual_ modalities, and thus can be considered as a source of multi-modal information. Although videos contain more information than text, fusing multiple modalities is a major challenge. A common practice in sentiment analysis and emotion recognition or affective computing, in general, is to analyze textual opinions. However, in recent days multi-modal affect analysis has gained a major attention Poria et al. (2016, 2017). In these works, in addition to the _visual frames_, other sources of information such as _acoustic_ and _textual_ (transcript) representation of the spoken languages are also incorporated in the analysis. Multi-modal analysis (e.g. sentiment analysis Zadeh et al. 2018c, emotion recognition Poria et al. 2016, question-answering Teney et al. 2017 etc.) is an emerging field of study, that utilizes multiple information sources for solving a problem. These sources (e.g., text, visual, acoustic, etc.) offer a diverse and often distinct piece of information that a system can leverage on. For example, _'text'_ carries semantic information of the spoken sentence, whereas _'acoustic'_ information reveals the emphasis (pitch, voice quality) on each word. In contrast, the _'visual'_ information (image or video frame) extracts the gesture and posture of the speaker.

Traditionally, _'text'_ has been the key factor in any Natural Language Processing (NLP) tasks including sentiment and emotion analysis. However, with the recent emergence of social media platforms and their available multi-modal contents, an interdisciplinary study involving _text_, _acoustic_ and _visual_ features have drawn significant interest among the research community. Effectively fusing this diverse information is non-trivial and poses several challenges to the underlying problem.

In our current work, we propose a multi-task model to extract both sentiment (i.e. _positive_ or _negative_) and emotion (i.e. _anger_, _disguest_, _fear_, _happy_, _sal_ or _surprise_) of a speaker in a video. In multi-task framework, we aim to leverage the inter-dependence of these two tasks to increase the confidence of individual task in prediction. For e.g., information about _anger_ emotion can help in prediction of _negative_ sentiment and vice-versa.

A speaker can utter multiple utterances (a unit of speech bounded by breathes or pauses) in a single video and these utterances can have different sentiments and emotions. We hypothesize that the sentiment (or, emotion) of an utterance often has inter-dependence on other contextual utterances i.e. the knowledge of sentiment (or, emo

tion) for an utterance can assist in classifying its neighbor utterances. We utilize all three modalities (i.e. _text, acoustic_ and _visual_) for the analysis. Although all these sources of information are crucial, they are not equally beneficial for each individual instance. Few examples are presented in Table 1. In the first example, _visual frames_ provide important clues than _textual_ information for finding the sentiment of a sarcastic sentence "_Thanks for putting me on hold! I've all the time in the world._". Similarly, the _textual_ representation of second example "_I'm fine._' does not reveal the exact emotion of a _sad_ person. For this particular case, _acoustic_ or _visual_ information such as low tone voice, facial expression etc. have bigger role to play for the classification.

## 2 Problem Definition

Multi-task learning paradigm provides an efficient platform for achieving generalization. Multiple tasks can exploit the inter-relatedness for improving individual performance through a shared representation. Overall, it provides three basic advantages over the single-task learning paradigm a). it helps in achieving generalization for multiple tasks; b). each task improves its performance in association with the other participating tasks; and c). offers reduced complexity because a single system can handle multiple problems/tasks at the same time.

Sentiments Pang et al. (2005) and emotions Ekman (1999) are closely related. Most of the emotional states have clear distinction of being a positive or negative situation. Emotional states e.g. _'anger'_, _'fear'_, _'disgust'_, _'sad'_ etc. belong to negative situations, whereas _'happy'_ and _'surprise'_ reflect the positive situations. Motivated by the association of sentiment & emotion and the advantages of the multi-task learning paradigm, we present a multi-task framework that jointly learns and classifies the sentiments and emotions in a video. As stated earlier, contextual-utterances and/or multi-modal information provide important cues for the classification. Our proposed approach applies attention over both of these sources of information simultaneously (i.e., contextual utterance and inter-modal information), and aims to reveal the most contributing features for the classification. We hypothesize that applying attention to contributing neighboring utterances and/or multi-modal representations may assist the network to learn in a better way.

Our proposed architecture employs a recurrent neural network based contextual inter-modal attention framework. In our case, unlike the previous approaches, that simply apply attention over the contextual utterance for classification, we take a different approach. Specifically, we attend over the contextual utterances by computing correlations among the modalities of the target utterance and the context utterances. This particularly helps us to distinguish which modalities of the relevant contextual utterances are more important for the classification of the target utterance. The model facilitates this modality selection process by attending over the contextual utterances and thus generates better multi-modal feature representation when these modalities from the context are combined with the modalities of the target utterance. We evaluate our proposed approach on the recent benchmark dataset of CMU-MOSEI Zadeh et al. (2018c). It is the largest available dataset (approx. 23K utterances) for multi-modal sentiment and emotion analysis (c.f. Dataset Section). The evaluation shows that contextual inter-modal attention framework attains better performance than the state-of-the-art systems for various combinations of input modalities.

The main contributions of our proposed work are three-fold: **a)**_we leverage the inter-dependence of two related tasks (i.e. sentiment and emotion) in improving each others performance using an effective multi-modal framework_; **b)**_we propose contextual inter-modal attention mechanism that facilitates the model to assign weightage to the contributing contextual utterances and/or to different modalities simultaneously_. Suppose, to classify an utterance '_u1_' of 5 utterances video, visual features of '_u2_' & '_u4_', acoustic features of '_u3_' and textual features of '_u1_', '_u3_' & '_u5_' are more important than others. Our attention model is capable of highlighting such diverse contributing features; and **c)**_we present the state-of-the-arts for both sentiment and emotion predictions_.

\begin{table}
\begin{tabular}{l|c c c}

- & ✓ \\ \hline \multicolumn{4}{l|}{_Tm fine._} & Emotion (Sal) &
- & ✓ & ✓ \\ \end{tabular}
\end{table}
Table 1: Contributing modalities for different scenario. Tick represents the most contributing information.
 



## 3 Related Work

A survey of the literature suggests that multi-modal sentiment prediction is a relatively new area as compared to textual based sentiment prediction (Morency et al., 2011; Poria et al., 2017b; Zadeh et al., 2018a). A good review covering the literature from uni-modal analysis to multi-modal analysis is presented in (Poria et al., 2017a).

Zadeh et al. (2016) introduced the multi-modal dictionary to understand the interaction between facial gestures and spoken words better when expressing sentiment. In another work, Zadeh et al. (2017) proposed a Tensor Fusion Network (TFN) model to learn the intra-modality and inter-modality dynamics of the three modalities (i.e., text, visual and acoustic). Authors reported improved accuracy using multi-modality on the CMU-MOSI dataset. These works did not take contextual information into account. Poria et al. (2017b) proposed a Long Short Term Memory (LSTM) based framework for sentiment classification that leverages the contextual information to capture the inter-dependencies between the utterances. Zadeh et al. (2018a) proposed multi-attention blocks (MAB) to capture the information across the three modalities (text, visual and acoustic) for predicting the sentiments. Authors evaluated their approach on the different datasets and reported improved accuracies in the range of 2-3% over the state-of-the-art models. Blanchard et al. (2018) proposed a multi-modal fusion model that exclusively uses high-level visual and acoustic features for sentiment classification.

An application of multi-kernel learning based fusion technique was proposed in (Poria et al., 2016), where the authors employed deep convolutional neural network (CNN) for extracting the textual features and fused it with other modalities (_visual_ & _acoustic_) for emotion prediction. Ranganathan et al. (2016) proposed a convolutional deep belief network (CDBN) models for multi-modal emotion recognition. The author used CDBN to learn salient multi-modal (acoustic and visual) features of low-intensity expressions of emotions. Hazarika et al. (2018) introduced a self- attention mechanism for multi-modal emotion detection by feature level fusion of text and speech. Recently, Zadeh et al. (2018c) introduced the CMU-MOSEI dataset for multi-modal sentiment analysis and emotion recognition. They effectively fused the tri-modal inputs through a dynamic fusion graph and also reported competitive performance _w.r.t._ various state-of-the-arts on MOSEI dataset for both sentiment and emotion classification.

The main difference between the proposed and existing methods is contextual inter-modal attention. Systems (Poria et al., 2016; Zadeh et al., 2016, 2017; Blanchard et al., 2018) do not consider context for the prediction. System (Poria et al., 2017b) uses contextual information for the prediction but without any attention mechanism. In contrast, (Zadeh et al., 2018a) uses multi-attention blocks but did not account for contextual information. Our proposed model is novel in the sense that our approach applies attention over multi-modal information of the contextual utterances in a single step. Thus, it ensures to reveal the contributing features across _multiple modalities_ and _contextual utterances_ simultaneously for sentiment and emotion analysis. Further, to the best of our knowledge, this is the first attempt at solving the problems of multi-modal sentiment and emotion analysis together in a multi-task framework.

The contextual inter-modal attention mechanism is not much explored in NLP domains as such. We found one work that accounts for bi-modal attention for visual question-answering (VQA) (Teney et al., 2017). However, its attention mechanism differs from our proposed approach in the following manner: a) VQA proposed question guided image-attention, but our attention mechanism attends multi-modalities; b) attention is applied over different positions of the image, whereas our proposed approach applies attention over multiple utterances and two-modalities at a time; c). our proposed attention mechanism attends a sequence of utterances (text, acoustic or visual), whereas VQA applies attention in the spatial domain. In another work, Ghosal et al. (2018) proposed an inter-modal attention framework for the multi-modal sentiment analysis. However, the key differences with our current work are as follows: a) Ghosal et al. (2018) addressed only sentiment analysis, whereas, in our current work, we address both the sentiment and emotion analysis; b) Ghosal et al. (2018) handles only sentiment analysis in single task learning framework, whereas our proposed approach is based on multi-task learning framework, where we solve two tasks, i.e., sentiment analysis and emotion analysis, together in a single network; c) we perform detailed com

parative analysis over the single-task vs. multi-task learning; and d) we present state-of-the-art for both sentiment and emotion analysis.

## 4 Multi-task Multi-modal Emotion Recognition and Sentiment Analysis

In our proposed framework, we aim to leverage multi-modal and contextual information for predicting sentiment and emotion of an utterance simultaneously in a multi-task learning framework. As stated earlier, a video consists of a sequence of utterances and their semantics often have inter-dependencies on each other. We employ three bi-directional Gated Recurrent Unit (biGRU) network for capturing the contextual information (i.e., one for each modality). Subsequently, we introduce pair-wise inter-modal attention mechanism (i.e. _visual-text_, _text-acoustic_ and _acoustic-visual_) to learn the joint-association between the multiple modalities & utterances. The objective is to emphasize on the contributing features by putting more attention to the respective utterance and neighboring utterances. Motivated by the residual skip connection He et al. (2016) the outputs of pair-wise attentions along with the representations of individual modalities are concatenated. Finally, the concatenated representation is shared across the two branches of our proposed network- corresponding to two tasks, i.e., sentiment and emotion classification for prediction (one for each task in the multi-task framework). Sentiment classification branch contains a _softmax_ layer for final classification (i.e. _positive & negative_), whereas for emotion classification we use sigmoid layer. The shared representation will receive gradients of error from both the branches (sentiment & emotion) and accordingly adjust the weights of the models. Thus, the shared representations will not be biased to any particular task, and it will assist the model in achieving generalization for the multiple tasks. Empirical evidences support our hypothesis (c.f. Table 4).

### Contextual Inter-modal (CIM) Attention Framework

Our contextual inter-modal attention framework works on a pair of modalities. At first, we capture the cross-modality information by computing a pair of matching matrices \(M_{1},M_{2}\in\mathbb{R}^{u\times u}\), where '\(u\)' is the number of utterances in the video. Further, to capture the contextual dependencies, we compute the probability distribution scores (\(N_{1},N_{2}\in\mathbb{R}^{u\times u}\)) over each utterance of cross-modality matrices \(M_{1},M_{2}\) using a softmax function. This essentially computes the attention weights for contextual utterances. Subsequently, we apply soft attention over the contextual inter-modal matrices to compute the modalitiy-wise attentive representations (\(O_{1}\&O_{2}\)). Finally, a multiplicative gating mechanism Dhingra et al. (2016) (\(A_{1}\&A_{2}\)) is introduced to attend the important components of multiple modalities and utterances. The concatenated attention matrix of \(A_{1}\&A_{2}\) then acts as the output of our contextual inter-modal attention framework. The entire process is repeated for each pair-wise modalities i.e. _text-visual_, _acoustic-visual_ and _text-acoustic_. We illustrate and summarize the proposed methodology in Figure 1 and Algorithm 1, respectively.

```
procedureMTMM-ES(\(t,v,a\)) \(d\gets 100\)\(\triangleright\) GRU dimension \(T\gets biGRU_{T}(t,d)\) \(V\gets biGRU_{V}(v,d)\) \(A\gets biGRU_{A}(a,d)\) \(Att_{TV}\leftarrow\textit{CIMI-Attention}(T,V)\) \(Att_{AV}\leftarrow\textit{CIMI-Attention}(A,V)\) \(Att_{TA}\leftarrow\textit{CIMI-Attention}(T,A)\) \(Rep\leftarrow[Atn_{TV},Att_{AV},Att_{TA},T,V,A]\) \(polarity\gets Sentiment(Rep)\) \(emotion\gets Emotion(Rep)\) return\(polarity,emotion\)
```

**Algorithm 1** Multi-task Multi-modal Emotion and Sentiment (MTMM-ES)

```
procedureTIM-Attention(\(X,Y\)) /*Cross-modality information*/ \(M_{1}\gets X.Y^{T}\) \(M_{2}\gets Y.X^{T}\) /*Contextual Inter-modal attention*/ for\(i,j\in 1,...,u\)do\(\triangleright\)\(u=\#utterances\) \(N_{1}(i,j)\leftarrow\frac{e^{M_{1}(i,j)}}{\sum_{k=1}^{K}e^{M_{1}(i,k)}}\) \(N_{2}(i,j)\leftarrow\frac{e^{M_{2}(i,j)}}{\sum_{k=1}^{K}e^{M_{2}(i,k)}}\) \(O_{1}\gets N_{1}.Y\) \(O_{2}\gets N_{2}.X\) /*Multiplicative gating*/ \(A_{1}\gets O_{1}\odot X\)\(\triangleright\) Element-wise mult. \(A_{2}\gets O_{2}\odot Y\)return[\(A_{1},A_{2}\)]
```

**Algorithm 2** Multi-task Multi-modal Emotion and Sentiment (MTMM-ES) 



## 5 Datasets, Experiments, and Analysis

In this section, we describe the datasets used for our experiments and report the results along with necessary analysis.

### Datasets

We evaluate our proposed approach on the benchmark datasets of sentiment and emotion analysis, namely CMU Multi-modal Opinion Sentiment and Emotion Intensity (CMU-MOSEI) dataset [23]. CMU-MOSEI dataset consists of 3,229 videos spanning over 23,000 utterances from more than 1,000 online YouTube speakers. The training, validation & test set comprises of 16216, 1835 & 4625 utterances, respectively.

Each utterance has six emotion values associated with it, representing the degree of emotion for _anger_, _disgust_, _fear_, _happy_, _sad_ and _surprise_. Emotion labels for an utterance are identified as all non-zero intensity values, i.e. if an utterance has three emotions with non-zero values, we take all three emotions as multi-labels. Further, an utterance that has no emotion label represents the absence of emotion. For experiments, we adopt 7-classes (\(6\)_emotions_\(+\)\(1\)_no emotion_) and pose it as multi-label classification problem, where we try to optimize the binary-cross entropy for each of the class. A brief statistics for multi-label emotions is presented in Table 3. In contrast, the sentiment values for each utterance are disjoint, i.e. _value_\(<0\) and _value_\(\geq 0\) represent the negative and positive sentiments, respectively. A detailed statistics of the CMU-MOSEI dataset is shown in Table 2.

Figure 1: Overall architecture of the proposed framework. Contextual inter-modal (CIM) attention computation between _visual_ and _text_ modality.

\begin{table}
\begin{tabular}{|l|c c c|} \hline
**Statistics** & Train & Dev & Test \\ \hline \hline _\#Videos_ & 2250 & 300 & 679 \\ _\#Unterance_ & 16216 & 1835 & 4625 \\ _\#Positive_ & 11499 & 1333 & 3281 \\ _\#Negative_ & 4717 & 502 & 1344 \\ _\#Anger_ & 3506 & 334 & 1063 \\ _\#Disgust_ & 2946 & 280 & 802 \\ _\#Fear_ & 1306 & 163 & 381 \\ _\#Happy_ & 8673 & 978 & 2484 \\ _\#Sad_ & 4233 & 511 & 1112 \\ _\#Surprise_ & 1631 & 194 & 437 \\ _\#Speakers_ & & 1000 & \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 2: Dataset statistics for CMU-MOSEI. Each utterance contains multi-modal information.





### Feature extraction

We use the CMU-Multi-modal Data SDK1 for downloading and feature extraction. The dataset was pre-tokenized and a feature vector was provided for each word in an utterance. The _textual_, _visual_ and _acoustic_ features were extracted by _GloVe_Pennington et al. (2014), _Facets2 & CovaRep_Degottex et al. (2014), respectively. Thereafter, we compute the average of _word-level_ features to obtain the _utterance-level_ features.

Footnote 1: [https://github.com/AZadeh/CMJ-MultimodalDataSDK](https://github.com/AZadeh/CMJ-MultimodalDataSDK)

Footnote 2: [https://pair-code.github.io/facets/](https://pair-code.github.io/facets/)

### Experiments

We evaluate our proposed approach on the datasets of CMU-MOSEI. We use the Python based Keras library for the implementation. We compute _F1-score_ and _accuracy values_ for sentiment classification and _F1-score_ and _weighted accuracy_Tong et al. (2017) for emotion classification. Weighted accuracy as a metric is chosen due to unbalanced samples across various emotions and it is also in line with the other existing works Zadeh et al. (2018c). To obtain multi-labels for emotion classification, we use 7 sigmoid neurons (corresponds to 6 emotions + 1 no-emotion) with binary cross-entropy loss function. Finally, we take all the emotions whose respective values are above a \(threshold\). We optimize and cross-validate both the evaluation metrics (i.e. F1- score and weighted accuracy) and set the _threshold_ as \(0.4\) & \(0.2\) for F1-score and weighted accuracy, respectively. We show our model configurations in Table 5.

As stated earlier, our proposed approach requires at least two modalities to compute bi-modal attention. Hence, we experiment with bimodal and tri-modal input combinations for the proposed approach i.e. taking _text-visual_, _text-acoustic_, _acoustic-visual_ and _text-visual-acoustic_ at a time. For completeness (i.e., uni-modal inputs), we also experiment with a variant of the proposed approach where we apply self-attention on the utterances of each modality separately. The self-attention unit utilizes the contextual information of the utterances (i.e., it receives _u\(\times d\)_ hidden representations), applies attention and forward it to the output layer for classification. We report the experimental results of both single-task (STL) and multi-task (MTL) learning framework in Table 4. In the single-task framework, we build separate systems for sentiment and emotion analysis, whereas in multi-task framework a joint-model is learned for both of these problems. For sentiment classification, our single-task framework reports an F1-score of 77.67% and accuracy value of 79.8% for the tri-modal inputs. Similarly, we obtain 77.71% F1-score and 60.88% weighted accuracy for emotion classification.

Comparatively, when both the problems are learned and evaluated in a multi-task learning framework, we observe performance enhancement for both sentiments as well as emotion classification. MTL reports 78.86% F1-score and 80.47% accuracy value in comparison to 77.67% and 79.8% of STL with tri-modal inputs, respectively. For emotion classification, we also observe an improved F-score (78.6 (MTL) vs. 77.7 (STL)) and weighted accuracy (62.8 (MTL) vs. 60.8 (STL))

\begin{table}
\begin{tabular}{l|c||c c|c c|c c|c c|c c|c c|c} \multicolumn{3}{c|}{**Flsce**} & \multicolumn{3}{c|}{**Acc (Sentiment) \& \multicolumn{3}{c|}{**Weighted-Acc (Emotion)**} \\ \cline{3-14}  & T & A & V & T+V & T+A & A+V & T+A+V & T & A & V & T+V & T+A & A+V & T+A+V \\ \hline \hline \multirow{2}{*}{Sent} & STL & 75.1 & 67.9 & 66.3 & 77.0 & 76.5 & 69.6 & 77.6 & 78.2 & 74.8 & 75.8 & 79.4 & 79.7 & 76.6 & 79.8 \\  & MTL & 77.5 & 72.1 & 69.1 & 78.7 & 78.6 & 75.8 & 78.8 & 79.7 & 75.7 & 76.5 & 80.4 & 80.2 & 77.4 & 80.5 \\ \hline \multirow{2}{*}{Emo} & STL & 75.9 & 72.3 & 73.6 & 77.5 & 76.8 & 76.0 & 77.7 & 58.0 & 56.7 & 53.7 & 60.1 & 59.6 & 58.0 & 60.8 \\  & MTL & 76.9 & 74.6 & 75.4 & 78.5 & 77.6 & 77.0 & 78.6 & 60.2 & 56.2 & 57.5 & 62.5 & 60.5 & 59.3 & 62.8 \\ \end{tabular}
\end{table}
Table 4: Single-task learning (STL) and Multi-task (MTL) learning frameworks for the proposed approach. T: Text, V: Visual, A: _Acoustic_. Weighted accuracy as a metric is chosen due to unbalanced samples across various emotions and it is also in line with the other existing works Zadeh et al. (2018c).

\begin{table}
\begin{tabular}{p{113.8pt}|r}
**Parameters** & **Values** \\ \hline \hline Bi-GRU & 2\(\times\)200 _neurons_, _dropout=0.3_ \\ Dense layer & 100 _neurons_, _dropout=0.3_ \\ Activations & _ReLu_ \\ Optimizer & _Adam_ (_lr=0.001_) \\ Output & _Softmax_ (Sent) \& _Sigmoid_ (Emo) \\ Loss & _Categorical cross-entropy_ (Sent) \\ \multicolumn{3}{c}{_Binary cross-entropy_ (Emo)} \\ Threshold & 0.4 (F1) \& 0.2 (W-Acc) for multi-label Batch & _16_ \\ Epochs & _50_ \\ \end{tabular}
\end{table}
Table 5: Model configurations

in the multi-task framework. It is evident from Figure 2 that multi-task learning framework successfully leverages the inter-dependence of both the tasks in improving the overall performance in comparison to single-task learning. The improvements of MTL over STL framework is also statistically significant with _p_-value \(<0.05\) (c.f. Table 7).

We also present attention heatmaps of the multi-task learning framework in Figure 3. For illustration, we take the video of the first utterance of Table 6. It has total six utterances. We depict three pair-wise attention matrices of \(2\times(6\times 6)\) dimension-one each for _text-visual_, _text-acoustics_ and _acoustics-visual_. Solid lines in between represent the boundary of the two modalities, e.g. left

Figure 3: (a), (b) & (c): Pair-wise softmax attention weights \(N_{1}\) & \(N_{2}\) of _visual-text_, _acoustic-visual_ & _text-acoustic_ for multi-task learning framework. Solid line at the center represents boundary of \(N_{1}\) & \(N_{2}\). The heatmaps represent attention weights of a particular utterance with respect to other utterances in \(N_{1}\) & \(N_{2}\). Each cell (\(i,j\)) of the heatmap signifies the weights of utterance ‘\(j\)’ for the classification of utterance ‘\(i\)’ of the pair-wise modality matrices, hence, assists in predicting the labels concisely by incorporating contextual inter-modal information.

\begin{table}
\begin{tabular}{l||c|c|c|c|c}  & \multicolumn{2}{c||}{**Sentiment**} & \multicolumn{2}{c}{**Emotion**} \\ \cline{3-4} \multicolumn{1}{c||}{**Utterances**} & **Actual** & **MTL** & **Actual** & **MTL** \\ \hline
1 & _line hello my name is sarah and i will be doing my video opinion on the movie shall we dance uhh starting jamifer_ & Pos & Pos & Anger & Anger \\ \hline
2 & _richard gere and susan num you i really didn’t enjoy this movie at all it kinda boring for_ & Neg & Neg & Anger, Disgust, & Anger, Disgust, \\ \hline
3 & _for unum ikinda felt as if there were parts in there they_ & Pos & Neg & No class & Anger, Disgust, \\ \hline
4 & _they just put in there to kinds pass the time on basically the movie is about num richard character and him being a_ & Pos & Pos & Happy & Anger, Disgust, \\ \hline
5 & _num looking for some some stutter extra size to add into his life he meets up with a dance instructor who is played by jemifer hope;and basically she convinces him to sign up for some ballroom he gets into it he enjoys it a lot but still a secret from his family_ & Pos & Onager & Anger & Disgust, \\ \hline
6 & _family he is trying to cope with having this shutter_ & Neg & Neg & Anger, Disgust & Anger, Disgust, \\ \end{tabular}
\end{table}
Table 6: Example video for heatmap analysis of the contextual inter-modal (CIM) attention mechanism of the proposed MTMM-ES framework. Figure 3 depicts the heatmaps for the above video.

Figure 2: Single-task learning (STL) and Multi-task (MTL) learning frameworks for the proposed approach.

 

side of Figure 3a represents _text_ modality and right side represents the _visual_ modality. The heatmaps represent the contributing features for the classification of utterances. Each cell (_i,j_) of Figure 3 signifies the weights of utterance '_j_' for the classification of utterance '_i_' of the pair-wise modality matrices. For example, for the classification of utterance '_u4_' in Figure 3a, model puts more focus on the textual features of '_u2_' and '_u6_' than others and more-or-less equal focus on the visual features of all the utterances.

### Comparative Analysis

We compare our proposed approach against various existing systems [19, 50, 78] that made use of the same datasets. A comparative study is shown in Table 7. We report the results of the top three existing systems (as reported in Zadeh et al.2018c) for each case. In emotion classification, the proposed multi-task learning framework reports the best F1-score of 78.6% as compared to the 76.3% and Weighted Accuracy of 62.8% as compared to the 62.3% of the state-of-the-art. Similarly, for sentiment classification, the state-of-the-art system reports 77.0% F1-score and 76.9% accuracy value in the multi-task framework. In comparison, we obtain the best F1-score and accuracy value of 78.8% and 80.4%, respectively, i.e., an improvement of 1.8% and 3.5% over the state-of-the-art systems.

During analysis, we make an important observation. Small improvements in performance do not reveal the exact improvement in the number of instances. Since there are more than 4.6K test samples, even the improvement by one point reflects that the system improves its predictions for 46 samples.

We also perform test-of-significance (_T_-test) and observe that the obtained results are statistically significant _w.r.t._ the state-of-the-art and proposed single-task results with _p_-values\(<0.05\).

### STL v/s MTL framework

In this section, we present our analysis _w.r.t._ single-task and multi-task frameworks. Table 8 lists a few example cases where the proposed multi-task learning framework shows how it yields better performance for both sentiment and emotion, while the single-task framework finds it non-trivial for the classification. For example, first utterance has gold sentiment label as negative which was misclassified by STL framework. However, the MTL framework improves this by correctly predicting '_positive_'. Similarly, in emotion analysis STL predicts three emotions i.e. _disgust_, _happy_ and _sad_, out of which only one emotion (_disgust_) matches the gold emotions of _anger_ and _disgust_. In comparison, MTL predicts four emotions (i.e. _anger_, _disgust_, _happy_ and _sad_) for the same utterance. The precision (2/4) and recall (2/2) for MTL framework is better than the precision (1/3) and recall (1/2) for the STL framework. These analyses suggest that the MTL framework, indeed, captures better evidences than the STL framework.

In the second example, knowledge of sentiment helps in identifying the correct emotion label in the MTL framework. For the gold sentiment (_positive_) and emotion (_happy_ and _sad_) labels, STL correctly classifies one emotion (i.e. _sad_), but fails to predict the other emotion (i.e. _happy_). In addition, it misclassifies another emotion (i.e. _anger_). Since, gold label _happy_ corresponds to the _posi

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c c|c c|c c}  & \multicolumn{2}{c|}{_Aager_} & \multicolumn{2}{c|}{_Diquest_} & \multicolumn{2}{c|}{_Four_} & \multicolumn{2}{c|}{_Happy_} & \multicolumn{2}{c|}{_Sad_} & \multicolumn{2}{c|}{_Surprise_} & \multicolumn{2}{c}{_Average_’} \\ System & _F1_ & _W-Acc_ & _F1_ & _W-Acc_ & _F1_ & _W-Acc_ & _F1_ & _W-Acc_ & _F1_ & _W-Acc_ & _F1_ & _W-Acc_ & _F1_ & _W-Acc_ & _F1_ & _Acc_ \\ \hline Blanchard et al. Blanchard:2018: & - & - & - & - & - & - & - & - & - & - & - & - & _62.1_ & _60.0_ \\ Zaghd et al. Zadeh:2018b\({}^{*}\) & - & - & 71.4 & 65.2 & **89.9** & - & - & - & 60.8 & - & 854.4 & 53.3 & - & - & 76.0 & 76.0 \\ Nojiavasphani et al. Ngoviavasphani:2016: & 71.4 & - & 67.0 & - & - & - & - & - & - & - & - & - & - & - & - \\ Rajeguela et al. Rajeguela:2016: & - & 56.0 & - & - & - & - & - & - & - & - & - & - & - & 76.4 & 76.4 \\ EF+LSTM Zadeh et al. Rajeguela:2016: & - & - & - & - & - & 56.7 & - & 57.8 & - & 59.2 & - & - & - & - & - \\ TNZadeh et al. TAN 

tive_ scenario and predicted label _anger_ is related to negative, knowledge of sentiment is a crucial piece of information. Our MTL framework identifies this relation and leverage the predicted sentiment for the classification of emotion i.e. _positive_ sentiment assists in predicting _happy_ emotion. This is an example of inter-dependence between the two related tasks and the MTL framework successfully exploits it for the performance improvement.

We also observe that the system puts comparatively more focus on some classes in MTL framework than the STL framework. As an instance, MTL predicts '_anger_' class for 1173 utterances, whereas STL predicts it for 951 utterances (1063 anger utterances in the gold dataset). Further, we observe contrasting behavior for the '_sad_' class, where MTL predicts 1618 utterances as '_sad_' compared to the 2126 utterances of STL. For '_disgust_' and '_happy_' classes, both STL and MTL frameworks predict the approximately equal number of utterances.

Further, we observe that MTL performs poorly for the '_fear_' and '_surprise_' classes, where it could not predict a significant number of utterances. A possible reason would be the under-representation of these instances in the given dataset.

## 6 Conclusion

In this paper, we have proposed a deep multi-task framework that aims to leverage the inter-dependence of two related tasks, i.e., multi-modal sentiment and emotion analysis. Our proposed approach learns a joint-representation for both the tasks as an application of GRU based inter-modal attention framework. We have evaluated our proposed approach on the recently released benchmark dataset on multi-modal sentiment and emotion analysis (MOSEI). Experimental results suggest that sentiment and emotion assist each other when learned in a multitask framework. We have compared our proposed approach against the various existing systems and observed that multi-task framework attains higher performance for all the cases.

In the future, we would like to explore the other dimensions to our multi-task framework, e.g., Sentiment classification & intensity prediction, Emotion classification & intensity prediction and all the four tasks together.

## Acknowledgement

Asif Ekbal acknowledges the Young Faculty Research Fellowship (YFRF), supported by Visvesvaraya Ph.D. scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia).

## References

* Blanchard et al. (2018) Nathaniel Blanchard, Daniel Moreira, Aparna Bharati, and Walter Scheirer. 2018. Getting the subtext without the text: Scalable multimodal sentiment classification from visual and acoustic modalities. In _Proceedings of Grand Challenge and Workshop on Human Multimodal Language_, pages 1-10, Melbourne, Australia.
* Breiman (2001) Leo Breiman. 2001. Random forests. _Machine Learning_, 45(1):5-32.
* Degottex et al. (2018) G. Degottex, J. Kane, T. Drugman, T. Raitio, and

\begin{table}
\begin{tabular}{|l|p{142.3pt}|c|c|c|c|c|c|}  & **Utterances** & \multicolumn{2}{c|}{**Sentiment**} & \multicolumn{2}{c|}{**Emotion**} \\ \cline{3-10} \multirow{-2}{*}{} & **Actual** & **STL** & **MTL** & **Actual** & **STL** & **MTL** & **MTL** \\ \hline
1 & _richual genre and assant pass you really didn’t enjoy this movie_ & Neg & Pos & Neg & Auger, Disgust & Disgust, Happy, Sad & Auger, Disgust, Happy, Sad \\ \cline{2-10} \multirow{-2}{*}{} & _at el la la la la de la la petitre y depressione with the nose government as a work to make progress on audio genre of image vishelting further character formats promotion of human rights account_ & Pos & Pos & Happy; Sad & Auger, Sad & Happy; Sad \\ \hline
2 & _hugher and expressive all all three things_ & Pos & Neg & Pos & Happy & Happy; Suprise & Happy \\ \hline
3 & _hugher and expressive all three things_ & Neg & Pos & Neg & Anger, Disgust, Sad & Disgust, Happy; Sad & Auger, Disgust, Ssd \\ \hline
4 & _a n love with another others even you know the story_ & Neg & Pos & Neg & Ungust, Sat, Surprise & Anger, Happy, Sad & Auger, Disgust, Happy, Sad \\ \hline
5 & _can say unfortunately don’t think t’s a serious program_ & Neg & Pos & Neg & Magnet, Sagut, Bad & Anger, Happy & Auget, Dagust, Happy, Sad \\ \hline
6 & _the last mainstream brought longst stu are such as this time_ & Neg & Pos & Neg & Algoer, Disgust, Happy & Auger, Happy & Auger, Diggust, Happy \\ \hline
7 & _any best born of a risk or just its socially acceptable_ & Neg & Pos & Neg & Algoer, Disgust, Happy & Auger, Happy & Auger, Diggust, Happy \\ \hline
8 & _had a rock barredike having nice your major bars made_ & Pos & Pos & Pos & Harry & Hurry, Sad. No chase & Happy \\ \hline \end{tabular}
\end{table}
Table 8: Comparison with multi-task learning and single-task learning frameworks. Few error cases where multi-task learning framework performs better than the single-task framework. _First utterance_: Improved MTL (_Pre: 0.5, Rec: 1.0)_ performance over STL (_Pre: 0.3, Rec: 0.5). _Second utterance_: Sentiment (i.e. _Pos_) assists in emotion classification (i.e. _Happy_). Red color represents error in classification.

 

S. Scherer (2014) Covarep - a collaborative voice analysis repository for speech technologies. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 960-964. Cited by: SS3.
* B. Dhingra, H. Liu, Z. Yang, W. V. Cohen, and R. Salakhutdinov (2016)Gated-attention readers for text comprehension. arXiv preprint arXiv:1606.01549. Cited by: SS1.
* P. Ekman (1999)Basic emotions. The handbook of cognition and emotion. External Links: Link Cited by: SS1.
* D. Ghosal, M. S. Akhtar, D. Chauhan, S. Poria, A. Ekbal, and P. Bhattacharyya (2018)Contextual inter-modal attention for multi-modal sentiment analysis. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Cited by: SS1.
* D. Hazarika, S. Gorantla, S. Poria, and R. Zimmermann (2018)Self-attentive feature-level fusion for multimodal emotion detection. In 2018 IEEE Conference on Multimedia Information Processing and Retrieval, pp. 196-201. Cited by: SS1.
* K. He, X. Zhang, S. Ren, and J. Sun (2016)Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778. Cited by: SS1.
* L. Morency, R. Mihalcea, and P. Doshi (2011)Towards multimodal sentiment analysis: harvesting opinions from the web. In Proceedings of the 13th International Conference on Multimodal Interfaces (ICMI-2011\({},pages169-176,Alicante,Spain\)... \(-\)176,Alicante,Spain\).. Cited by: SS1.
* E. Nojavanasghari, D. Gopinath, J. Koushik, T. Baltrusaitis, and L. Morency (2016)Deep multimodal fusion for persuasivers prediction. In Proceedings of the 18th ACM International Conference on Multimodal Interaction, New York, NY, USA, pp. 284-288. Cited by: SS1.
* B. Pang, and L. Lee (2005)Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL, pp. 115-124. Cited by: SS1.
* J. Pennington, R. Socher, and C. D. Manning (2014)Glove: global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543. Cited by: SS1.
* S. Poria, E. Cambria, R. Bajpai, and A. Hussain (2017)A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion37, pp. 98-125. Cited by: SS1.
* S. Poria, E. Cambria, R. Bajpai, and A. Hussain (2017)A review of affective computing: from unimodal analysis to multimodal fusion. Information Fusion37, pp. 98-125. Cited by: SS1.
* S. Poria, E. Cambria, D. Hazarika, N. Majumder, A. Zadeh, and L. Morency (2017)D Context-dependent sentiment analysis in user-generated videos. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1, pp. 873-883. Cited by: SS1.
* S. Poria, I. Chaturvedi, E. Cambria, and A. Hussain (2016)Convolutional mkl based multimodal emotion recognition and sentiment analysis. In Data Mining (ICDM), 2016 IEEE 16th International Conference on, pp. 439-448. Cited by: SS1.
* S. S. Rajagopalan, L. Morency, T. Baltrusaitis, and R. Goecke (2016)Extending long short-term memory for multi-view structured learning. In European Conference on Computer Vision (ECCV-2016), pp. 338-353. Cited by: SS1.
* H. Ranganathan, S. Chakraborty, and S. Panchanathan (2016)Multimodal emotion recognition using deep learning architectures. In Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, pp. 1-9. Cited by: SS1.
* D. Teney, P. Anderson, X. He, and A. van den Hengel (2017)Tips and tricks for visual question answering: learnings from the 2017 challenge. CoRRabs/1708.02711. Cited by: SS1.
* E. Tong, A. Zadeh, C. Jones, and L. Morency (2017)Combating human trafficking with Multimodal Deep Models. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada. Cited by: SS1.
* 5649. Cited by: SS1.
* A. Zadeh, R. Zellers, E. Picus, and L. P. Moreno (2016)Multimodal Sentiment Intensity Analysis in Videos: facial Gestures and verbal Messages. IEEE Intelligent Systems31 (6), pp. 82-88. Cited by: SS1.
* A. Zadeh, M. Chen, S. Poria, E. Cambria, and L. Morency (2017)Tensor fusion network for multimodal sentiment analysis. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 1103-1114. Cited by: SS1.
* A. Zadeh, P. Pu Liang, N. Mazumder, S. Poria, E. Cambria, and L. Morency (2018)Memory fusion network for multi-view sequential learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, pp. 2236-2246. Cited by: SS1.