Venue: NAACL 2018

# Conversational Memory Network

for Emotion Recognition in Dyadic Dialogue Videos

Devamanyu Hazarika

School of Computing,

National University of Singapore

hazarika@comp.nus.edu.sg &Soujanya Poria

Artificial Intelligence Initiative,

A*STAR, Singapore

sporia@ihpc.a-star.edu.sg &Amir Zadeh

Language Technologies

Institute, CMU, USA

abagherz@cs.cmu.edu &Erik Cambria

School of Computer Science and Engineering, NTU, Singapore cambria@ntu.edu.sg &Louis-Philippe Morecy

Language Technologies Institute, CMU, USA

morency@cs.cmu.edu &Roger Zimmermann

School of Computing,

National University of Singapore rogerz@comp.nus.edu.sg

###### Abstract

Emotion recognition in conversations is crucial for the development of empathetic machines. Present methods mostly ignore the role of inter-speaker dependency relations while classifying emotions in conversations. In this paper, we address recognizing utterance-level emotions in dyadic conversational videos. We propose a deep neural framework, termed conversational memory network, which leverages contextual information from the conversation history. The framework takes a multimodal approach comprising audio, visual and textual features with gated recurrent units to model past utterances of each speaker into memories. Such memories are then merged using attention-based hops to capture inter-speaker dependencies. Experiments show an accuracy improvement of \(3-4\%\) over the state of the art.

## 1 Introduction

Development of machines with emotional intelligence has been a long-standing goal of AI. With the increasing infusion of interactive systems in our lives, the need for empathetic machines with emotional understanding is paramount. Previous research in affective computing has looked at dialogues as an essential basis to learn emotional dynamics Sidnell and Stivers (2012); Poria et al. (2017); Zhou et al. (2017).

Since the advent of Web 2.0, dialogue videos have proliferated across the internet through platforms like movies, webinars, and video chats. Emotion detection from such resources can benefit numerous fields like counseling De Choudhury et al. (2013), public opinion mining Cambria et al. (2017), financial forecasting Xing et al. (2018), and intelligent systems such as smart homes and chatbots Young et al. (2018).

In this paper, we analyze emotion detection in videos of dyadic conversations. A dyadic conversation is a form of a dialogue between two entities. We propose a conversational memory network (CMN), which uses a multimodal approach for emotion detection in utterances (a unit of speech bound by breathes or pauses) of such conversational videos.

Emotional dynamics in a conversation is known to be driven by two prime factors: self and inter-speaker emotional influence Morris and Keltner (2000); Liu and Maitlis (2014). Self-influence relates to the concept of _emotional inertia_, i.e., the degree to which a person's feelings carry over from one moment to another Koval and Kuppens (2012). Inter-speaker emotional influence is another trait where the other person acts as an influencer in the speaker's emotional state. Conversely, speakers also tend to mirror emotions of their counterparts Navarretta et al. (2016). Figure 1 provides an example from the dataset showing the presence of these two traits in a dialogue.

Existing works in the literature do not capitalize on these two factors. Context-free systems infer emotions based only on the current utterance in the conversation Bertero et al. (2016). Whereas, state-of-the-art context-based networks like Poria et al. (2017), use long short-term memory (LSTM) networks to model speaker-based context that suffers from incapability of long-range summarization and unweighted influence from context, leading to model bias.

Our proposed CMN incorporates these factors by using emotional context information present in the conversation history. It improves speaker-based emotion modeling by using memory networks which are efficient in capturing long-term

dependencies and summarizing task-specific details using attention models (Weston et al., 2014; Graves et al., 2014; Young et al., 2017).

Specifically, the memory cells of CMN are continuous vectors that store the context information found in the utterance histories. CMN also models interplay of these memories to capture inter-speaker dependencies.

CMN first extracts multimodal features (audio, visual, and text) for all utterances in a video. In order to detect the emotion of a particular utterance, say \(u_{i}\), it gathers its histories by collecting previous utterances within a context window. Separate histories are created for both speakers. These histories are then modeled into memory cells using gated recurrent units (GRUs).

After that, CMN reads both the speaker's memories and employs attention mechanism on them, in order to find the most useful historical utterances to classify \(u_{i}\). The memories are then merged with \(u_{i}\) using an addition operation weighted by the attention scores. This is done to model inter-speaker influences and dynamics. The whole cycle is repeated for multiple hops and finally, this merged representation of utterance \(u_{i}\) is used to classify its emotion category.

The contributions of this paper can be summarized as follows:

1. We propose an architecture, termed CMN, for emotion detection in a dyadic conversation that considers utterance histories of both the speaker to model emotional dynamics. The architecture is extensible to multi-speaker conversations in formats such as textual dialogues or conversational videos.
2. When applied to videos, we adopt a multimodal approach to extract diverse features from utterances. It also makes our model robust to missing information.
3. CMN provides a significant increase in accuracy of \(3-4\%\) over previous state-of-the-art networks. One variant called \(\text{CMN}_{self}\) which does not consider the inter-speaker relation in emotion detection also outperforms the state of the art by a significant margin.

The remainder of the paper is organized as follows: Section 2 provides a brief literature review; Section 3 formalizes the problem statement; Section 4 describes the proposed method in detail; experimental results are covered in Section 5; finally, Section 6 provides concluding remarks.

## 2 Related Works

Over the years, emotion recognition as an area of research has seen contributions from researchers across varied fields like signal processing, machine learning, cognitive and social psychology, natural language processing, etc. (Picard, 2010). Ekman, 1993, provided initial findings that related facial expressions as universal indicators of emotions. Datcu and Rothkrantz, 2008, 2011, showed the importance of acoustic cues in affect modeling.

A large section of researchers approaches emotion recognition from a multimodal learning perspective. Hence, many works used visual and audio features together for detecting affect (Busso et al., 2004; Castellano et al., 2008; Ranganathan et al., 2016). An in-depth review of the literature in these systems is provided by D'mello and Kory, 2015. Our work, which performs context-sensitive recognition (Wollmer et al., 2010) uses three modalities: audio, visual and text. Recently, this combination of modalities has provided the best performance in affect recognition systems (Poria et al., 2017b; Wang et al., 2017; Tzirakis et al., 2017), thus motivating the use of a multimodal approach.

Previous works have focused on conversations as a resourceful event for emotion analysis. Ruusuvuori, 2013, provides an in-depth analysis on how emotions affect social interactions and conversations. In fact, significant works have attributed emotional dynamics as an interactive phe

Figure 1: An abridged dialogue from the dataset. Person A (wife) is leaving B (husband) for a work assignment. Initially both A and B are emotionally driven by their own _emotional inertia_. In the end, _emotional influence_ can be seen when B, despite being sad, reacts angrily to A’s angry statement.

 

nomenon, rather than being within-person and one-directional (Richards et al., 2003; Hareli and Rafaeli, 2008). Such emotional dynamics are modeled by observing transition properties. Yang et al., 2011, study patterns for emotion transitions and show the evidence of emotional inertia. Xiaolan et al., 2013, use finite state machines to model transitions using stimuli and personality characteristics. Our work also tries to model emotional transitions using multimodal features. Unlike these works, however, we use memory networks to achieve the same.

The use of memory networks have been instrumental in the progress of multiple research problems, e.g., question-answering (Weston et al., 2014; Sukhbaatar et al., 2015; Kumar et al., 2016), machine translation (Bahdanau et al., 2014), speech recognition (Graves et al., 2014), and commonsense reasoning (Cambria et al., 2018). The repeated read and write to their memory cells is often coupled with attention modules, thus allowing it to filter only relevant memories.

Our model is loosely inspired from Sukhbaatar et al., 2015. Unlike their model, which directly encodes sentences into memories, we perform temporal sequence processing on our utterance histories using GRUs. We also extend their architecture to handle two speakers while keeping the possibility to add more. Finally, our model is different in the fact that we use multimodal features for input and processing.

## 3 Task Definition

Our goal is to infer the emotion of utterances present in a dyadic conversation. Let us define a dyadic conversation to be an asynchronous exchange of utterances between two persons \(P_{a}\) and \(P_{b}\). Both the speakers speak a sequence of utterances \(U_{a}\) and \(U_{b}\), respectively. Here, \(U_{\lambda}=(s^{1}_{\lambda},s^{2}_{\lambda},...,s^{l_{\lambda}}_{\lambda})\) is ordered temporally, where \(s^{i}_{\lambda}\) is the \(i^{th}\) utterance by \(P_{\lambda}\) and \(l_{\lambda}\) is the total number of utterances spoken by person \(P_{\lambda},~{}\lambda\in\{a,b\}\). Overall, the utterances by both speakers can be linearly ordered based on temporal occurrence as \((u_{1},u_{2},....u_{l_{a}+b})\) , where, \(u_{j}\in U_{a}\) or \(U_{b}\).

Our model takes as input an utterance \(u_{i}\) whose emotion category (Section 5.1) needs to be classified. To get its history, preceding \(K\) utterances of each person are separately collected as \(hist_{a}\) and \(hist_{b}\). Here, \(K\) serves as the length of the context window for history of \(u_{i}\). Thus, for \(\lambda\in\{a,b\}\):

\[hist_{\lambda}=\{u_{j}\mid u_{j}\in U_{\lambda},~{}j<i\}~{},~{}|~{} hist_{\lambda}|\leq K \tag{1}\]

\(hist_{\lambda}\) is also ordered temporally. At the beginning of the conversation, histories would have lesser than \(K\) utterances, i.e., \(|~{}hist_{\lambda}|<K\).

In the remaining sections, for brevity, we explain the processes using a subscript \(\lambda\) which can instantiate to either \(a\) or \(b\), i.e., \(\lambda\in\{a,b\}\).

## 4 Approach

We start by detailing the multimodal feature extraction scheme for all utterances followed by the mechanism to model emotional context using memory networks.

### Multimodal Feature Extraction

The first phase of CMN is to extract multimodal features of all utterances in the conversations. The dyadic conversations are present in the form of videos. Each utterance of a particular conversation is thus a small segment of the full video. For each utterance, we extract features for the modes: audio, visual and text. The process of feature extraction for each mode is described below.

#### 4.1.1 Textual Features Extraction

We extract features from the transcript of an utterance video using convolutional neural networks (CNNs). CNNs are effective in learning high level abstract representations of sentences from constituting words or n-grams (Kalchbrenner et al., 2014). To get our sentence representation, we use a simple CNN with one convolutional layer followed by max-pooling (Kim, 2014; Poria et al., 2016).

Specifically, the convolution layer consists filters of sizes \(3,4\) and \(5\) with \(50\) feature maps each. Max-pooling is employed on these feature maps with a pooling window of size \(2\). Finally, a fully connected layer is used with \(100\) neurons. The activations of this layer form our sentence representation \(t_{u}\).

#### 4.1.2 Audio Feature Extraction

To extract audio features we use openSMILE (Eyben et al., 2010). It is an open-source software which provides high dimensional audio vectors. These vectors comprise of features like loudness, Mel-spectra, MFCC, pitch, etc. Audio features play a significant role in providing information on the emotional state of a speaker (Song et al., 2004) 

In fact, the literature shows that there exists a high correlation between many statistical measures of speech with speakers' emotion. For example, high pitch and fast speaking rate often denote anger while sadness associates low standard deviation of pitch and slow speech rate (Dellaert et al., 1996; Amir, 1998). In this work, we use the _IS13_ComParE\({}^{1}\)_ config file which extracts a total of 6373 features for each utterance video. Z-standardization is performed for voice normalization and dimension of the audio vector is reduced to \(100\) using a fully-connected neural layer. This provides the final audio feature vector \(a_{u}\).

#### 4.1.3 Visual Feature Extraction

Facial expressions and visual surrounding provide rich emotional indicators. We use a 3D-CNN to capture these details from the utterance video. Apart from the benefits of extracting relevant features from each image frame, 3D-CNN also extracts spatiotemporal features across frames (Tran et al., 2015). This leads to the identification of emotional expressions like a smile or brown.

The working of a 3D-CNN is identical to its 2D counterpart with an input being a video \(v\) of dimension: \((3,f,h,w)\). Here, \(3\) represents the RGB channels and \(f,h,w\) are the number of frames, height, and width of each frame, respectively. For the convolution operation, a 3D filter \(f_{l}\) of dimension \((f_{m},3,f_{d},f_{h},f_{w})\) is used where, \(f_{[m/d]hf/f}\) represents number of feature maps, depth, height and width of the filter, respectively. Max-pooling is applied to the output of this convolution across a 3D sliding window of dimension (\(m_{p},m_{p},m_{p}\)).

In our model, we use \(128\) feature maps for 3D filters of size \(5\). For pooling, we set \(m_{p}\) to be \(3\) whose output is fed to a fully connected layer with \(100\) neurons. All the values are decided using hyperparameter tuning (see Section 5). For the input utterance, the activations of this layer form the video representation \(v_{u}\).

Fusion:We perform feature level fusion to map the individual modalities to a joint space. This is done through a simple feature concatenation. Thus, the extracted features \(t_{u},a_{u}\) and \(v_{u}\) are joined to form the utterance representation \(u=[t_{u};a_{u};v_{u}]\) of dimension \(d_{in}=300\). This multimodal representation is generated for all utterances in a conversation.

Literature consists of numerous fusion techniques for multimodal data (Atrey et al., 2010; Zadeh et al., 2017; Poria et al., 2017c). Exploring these on CMN, however, is beyond the scope of this paper and left as a future work.

### Conversational Memory Network

For classifying the emotion of an utterance \(u_{i}\), its corresponding histories (\(hist_{a}\) and \(hist_{b}\)) are taken. Each history \(hist_{\lambda}\) contains the preceding \(K\) utterances by person \(P_{\lambda}\) (see Section 3). Here, both \(u_{i}\) and utterances in the histories are represented using their multimodal feature vectors of dimension \(\mathcal{R}^{d_{in}}\) (Figure 2).

The histories are first modeled into memory cells using GRUs. This provides the memories with context information summarized by the GRU. We call this step as memory representation. Following cognitive evidence of self-emotional dynamics, we model separate memory cells for each person. Thus, identical but separate computations are performed on both histories. From these memories, content relevant to utterance \(u_{i}\) is then filtered out using attention mechanism over multiple input/output hops. At each hop, both memories are accumulated and merged with \(u_{i}\) to model interspeaker emotional dynamics. First, we describe our model as a single layer memory network which runs one hop operation on the memories.

#### 4.2.1 Single Layer

Here, we explain the representation scheme of the memories for both histories and the input/output operations on them along with attention mechanism. The memory representation for each history is generated using a GRU for modeling emotion transitions. First, we define the GRU cell.

**Gated Recurrent Unit:** GRUs are a gating mechanism in recurrent neural networks introduced by Cho et al. (2014). Similar to an LSTM Hochreiter and Schmidhuber (1997), GRU provides a simpler computation with similar performance. At any timestep \(t\), it utilizes two gates \(r_{t}\) (_reset gate_) and \(z_{t}\) (_update gate_) to control the combination criteria with current input utterance \(u_{t}\) and previous hidden state \(s_{t-1}\).

The new state \(s_{t}\) is computed as:

\[z_{t} =\sigma(V^{z}.u_{t}+W^{z}.s_{t-1}+b^{z})\] (2) \[r_{t} =\sigma(V^{r}.u_{t}+W^{r}.s_{t-1}+b^{r})\] (3) \[h_{t} =tanh(V^{h}.u_{t}+W^{h}.(s_{t-1}\otimes r_{t})+b^{h})\] (4) \[s_{t} =(1-z_{t})\otimes h_{t}+z_{t}\otimes s_{t-1}\] (5) 

Here, \(V,W\) and \(b\) are parameter matrices and vector and \(\otimes\) represents element-wise multiplication. The above equations can be summarized as: \(s_{t}=GRU_{\lambda}(s_{t-1},u_{t})\).

Memory Representation:For each \(\lambda\in\{a,b\}\), a memory representation \(M_{\lambda}=[m^{1}_{\lambda},...,m^{K}_{\lambda}]\) for \(hist_{\lambda}\) is generated using a GRU. To grasp the temporal context, the \(K\) utterances in \(hist_{\lambda}\) are framed as a sequence (starting from the oldest one) and fed to the GRU\({}_{\lambda}\). At each timestep \(t\in[1,K]\), the GRU\({}_{\lambda}\)'s internal state \(s_{t}\) (equation 5) forms the \(t^{th}\) memory cell \(m^{t}_{\lambda}\) of memory representation \(M_{\lambda}\).

Memory Input:This step takes the memory representation \(M_{\lambda}\) and performs an attention mechanism on it, resulting in an attention vector \(p_{\lambda}\in\mathcal{R}^{K}\). First, the current utterance \(u_{i}\) is embedded into a vector \(q_{i}\) of dimension \(\mathcal{R}^{d}\) using a projection matrix \(B\in\mathcal{R}^{dvd_{in}}\). To find the relevance of each memory \(m^{t}_{\lambda}\)'s context with \(q_{i}\), a match between both is computed. We do this by taking an inner product as follows:

\[q_{i}=B.u_{i} \tag{6}\] \[p^{t}_{\lambda}=softmax(q^{T}_{i}.m^{t}_{\lambda}) \tag{7}\]

Here, \(softmax(x_{i})=e^{x_{i}/\sum_{j}e^{x_{j}}}\) and attention vector \(p_{\lambda}=\left\{p^{t}_{\lambda}\right\}\) is a probability distribution over the input memories \(M_{\lambda}=\left\{m^{t}_{\lambda}\right\}\) for \(t\in[1,K]\).

Memory Output:First a new set of memories are created using another GRU\({}^{{}^{\prime}}_{\lambda}\) to get new memory representation \(M^{{}^{\prime}}_{\lambda}=\left\{(m^{t}_{\lambda})^{{}^{\prime}}\right\}\). An output representation \(o_{\lambda}\in\mathcal{R}^{d}\) is then generated using the weighted sum of attention vector \(p_{\lambda}\) and new memory \(M^{{}^{\prime}}_{\lambda}\) as follows:

\[o_{\lambda}=\sum_{t}p^{t}_{\lambda}.(m^{t}_{\lambda})^{{}^{\prime}}=M^{{}^{ \prime}}_{\lambda}.p_{\lambda} \tag{8}\]

Thus, the output representation \(o_{\lambda}\) contains weighted contextual summary accumulated from the memory.

Final Prediction:To generate the predictions for the current utterance \(u_{i}\), we combine the output representations of both persons: \(o_{a}\) and \(o_{b}\) with \(u_{i}\)'s representation \(q_{i}\) and perform an affine transformation using matrix \(W_{o}\). Softmax is applied to this final vector to get the emotion predictions,

\[\hat{y}=softmax(W_{o}.(q_{i}+o_{a}+o_{b})) \tag{9}\]

Categorical cross-entropy is used as the loss:

\[Loss=\frac{-1}{N}\underset{i=1}{\overset{N}{\sum}}\underset{j=1}{\overset{C} {\sum}}y_{i,j}~{}log_{2}(\hat{y}_{i,j}) \tag{10}\]

Here, \(N\) denotes total utterances across all videos and \(C\) is the number of emotion categories. \(y_{i}\) is the one-hot vector ground truth of \(i^{th}\) utterance from the training set and \(\hat{y}_{i,j}\) is its predicted probability of belonging to class \(j\).

Figure 2: Overall architecture of proposed model: CMN. First, multimodal representations are extracted for each utterance and then the previous \(K=4\) utterances by both persons are used to model the histories using GRUs. For each person, \(R+1\) different GRUs are used to represent \(M^{(r)}_{\lambda}\) for all \(R\) hops. Then, attention based filtering using multiple memory hops is performed. Finally, Person A’s utterance \(u_{i}\) is classified to predict its emotion category.

 



#### 4.2.2 Multiple Layers

Many recent works on memory networks adopt a multiple hop scheme in their network. This repeated input and output cycle on the memories along with a soft attention module, leads to a refined representation of the memories (Sukhbaatar et al., 2015; Kumar et al., 2016). Motivated by these works, we extend our model to perform \(R\) hops on the memories. This is done by stacking the single hop layers (Section 4.2.1) as follows:

* At a particular hop \(r\), the output memory of the previous hop \({M_{\lambda}^{{}^{\prime}}}^{(r-1)}\) is used as the input memory of the current hop \({M_{\lambda}^{(r)}}\). Output memory of current \(r^{th}\) hop is generated using a new \(GRU_{\lambda}^{(r)}\). This constraint of sharing parameters adjacently between layers is added for reduction in total parameters and ease of training.
* At every hop, the query utterance \(u_{i}\)'s representation \(q_{i}\) is updated as: \[q_{i}^{(r+1)}=q_{i}^{(r)}+o_{a}^{(r)}+o_{b}^{(r)}\] (11) \(o_{\lambda}^{(r)}\) is calculated as per equation 8 using \({M_{\lambda}^{{}^{\prime}}}^{(r)}\).
* After \(R\) hops, the final prediction is done using equation 9 as: \(\hat{y}=softmax(W_{o}.(q_{i}^{(R+1)}))\). Algorithm 1 summarizes the overall CMN network.

```
1:procedureCMN(\(u_{i},hist_{a},hist_{b},K,R\)) \(\triangleright\) predict the emotion of u\({}_{i}\)
2:\(q_{i}^{(1)}\gets B.u_{i}\)
3:\({M_{\lambda}^{{}^{\prime}}}^{(0)}\gets GRU_{\lambda}^{(0)}(hist_{ \lambda})\)
4:for\(z\)in \([1,\mathbb{R}]\)do\(\triangleright\) Multi-hop memory I/O
5:\(M_{\lambda}^{(r)}\leftarrow M_{\lambda}^{{}^{\prime}(r-1)}\)
6:\(M_{\lambda}^{{}^{\prime}(r)}\gets GRU_{\lambda}^{(r)}(hist_{\lambda})\)
7:\(p_{\lambda}\gets softmax(~{}(q_{i}^{r})^{T}.M_{\lambda}^{(r)}~{})\)\(\triangleright\) Memory in
8:\(o_{\lambda}^{(r)}\leftarrow{M_{\lambda}^{{}^{\prime}}}^{(r)}p_{\lambda}\)\(\triangleright\) Memory out
9:\(q_{i}^{(r+1)}\gets q_{i}^{(r)}+o_{a}^{(r)}+o_{b}^{(r)}\)\(\triangleright\) Query update
10:return\(\hat{y}\gets softmax(W_{o}.(q_{i}^{(R+1)}))\)\(\triangleright\) Prediction
```

**Algorithm 1** Conversational Memory Network

## 5 Experiments

### Dataset

We perform experiments on the IEMOCAP dataset2Busso et al. (2008). It is a multimodal database of 10 speakers (5 male and 5 female) involved in two-way dyadic conversations. A pair of speakers is given multiple conversation scenarios which are grouped in a single session. All the conversations are segmented into utterances. Each utterance is annotated using the following emotion categories: anger, happiness, sadness, neutral, excitement, frustration, fear, surprise, and other. However, in our experiments, we consider the first four categories. This is done to compare our method with state-of-the-art frameworks Poria et al. (2017); Rozgic et al. (2012). The dataset provides rich video and audio samples for all the utterances along with transcriptions.

Footnote 2: [http://sail.usc.edu/iemocap/](http://sail.usc.edu/iemocap/)

Apart from these emotional states, we also investigate the valence and arousal degrees of each utterance. IEMOCAP provides labels for both these attributes on a 5-point Likert scale. Following Aldeneh et al. (2017), we convert the attributes into 3 categories, namely, _low_ (\(\leq 2\)), _medium_ (\(>2\) and \(<4\)) and _high_ (\(\geq 4\)). The dataset configuration for the experiments is obtained from Poria et al. (2017). The first 8 speakers (Session 1 - 4) compose the training fold while the last session is used as the testing fold. Overall, the training and testing set comprises of \(4290\) utterances (\(120\) conversational videos) and \(1208\) utterances (\(31\) conversational videos), respectively. There is no speaker overlap in the training and testing set to make the model person-independent.

### Emotional Influence Patterns

In this section, we perform dataset exploration to check the existence of emotional influences. Figure 2(a) presents the emotion sequence of two videos

Figure 3: Each block represents an utterance and the blocks are ordered as per temporal occurrence. Color scheme identifies their corresponding emotions. The arrows denote emotional influence directions.



sampled from the dataset. Both videos show the presence of self and inter-speaker emotional influences. Visual exploration of videos from the dataset reveal significant existence of such instances in the conversations. To provide quantitative evidence of the emotional influence patterns, we curate a non exhaustive list of possible cases of influence. For all utterances in the dataset, we sample their histories by setting \(K=5\), i.e., five previous utterances (as per availability) from both speakers.

Cases \(1\) and \(2\) (Figure 3) represent scenarios when the emotion of current utterance is influenced by self or the other person respectively. In case \(3\), the utterance has relevant content in the histories that do not precede immediately. An effective attention mechanism provides the capability to capture this pattern. Finally case \(4\) presents the situation when the utterance is independent of the history. Such situations are indicative from the content of the utterance which often deviates from the previous topic of discussion or introduces a new information. Table 1 presents a statistical summary of these cases present in the dataset. From the table it can be seen that a large section of the dataset demonstrate these influence patterns. This provides motivation to explicitly model these patterns. We thus hypothesize that models that are able to capture these cases would have superior emotion inference capabilities.

This passive exploration is a label-based analysis which is performed as a sanity check. Needless to say, existence of some false positive patterns at the label level is imminent. On the other hand, our model CMN is content-based which enables it to mine intricate patterns from the utterance histories.

#### 5.2.1 Training Details

We use \(10\%\) of the training set as a held-out validation set for hyperparameter tuning. To optimize the parameters, we use Stochastic Gradient Descent (SGD) optimizer, starting with an initial learning rate (lr) of \(0.01\). An annealing approach halves the lr every \(20\) epochs and termination is decided using an early-stop measure with a patience of \(12\) by monitoring the validation loss. Gradient clipping is used for regularization with a norm set to \(40\). Hyperparameters are decided using a Random Search Bergstra and Bengio (2012). Based on validation performance, context window length \(K\) is set to be \(40\) and the number of hops \(R\) is fixed at \(3\) hops. If \(K\) previous utterances are unavailable, then null utterances are added at the beginning of the history sequence. The dimension size of the memory cells \(d\) is set as \(50\).

#### 5.2.2 Baselines

We compare CMN with the following baselines:

**SVM-ensemble:** A strong context-free benchmark model which uses similar multimodal approach on an ensemble of trees. Each node represents binary support vector machines (SVM) Rozgic et al. (2012).

**bc-LSTM:** A bi-directional LSTM equipped with hierarchical fusion, proposed by Poria et al. (2017b). It is the present state-of-the-art method. The model uses context features from unimodal LSTMs and its concatenation is fed to a final LSTM for classification. For fair comparison in an end-to-end learning paradigm, we remove the penultimate SVM of this model. The model doesn't accommodate inter-speaker dependencies.

**Memn2n:** The original memory network as proposed by Sukhbaatar et al. (2015). Contrasting to CMN, the model generates the memory representations for each historical utterance using an embedding matrix \(B\) as used in equation 7, without sequential modeling. Thus for utterance \(u_{i}\), both memories are created as \(M_{\lambda}\) using \(\{m^{t}_{\lambda}=B.u_{t}~{}|~{}u_{t}\in hist_{\lambda}\text{ and }t\in[1, K]\}\) for \(\lambda\in\{a,b\}\).

**CMN\({}_{Self}\):** In this baseline, we use only self history for classifying emotion of utterance \(u_{i}\). Thus, if \(u_{i}\) is spoken by person \(P_{a}\), then only \(hist_{a}\) is considered. Clearly, this variant is also incapable of modeling inter-speaker dependencies.

**CMN\({}_{NA}\):** Single layer variant of the CMN with no attention module. Thus, its output \(o_{\lambda}\) (equation 8) is generated using a uniform probability distribution \(p_{\lambda}\), i.e., \(\{p^{t}_{\lambda}=\frac{1}{K}\}_{t=1}^{K}\).

### Results

Table 2 presents the performances of CMN and its variants along with the state-of-the-art mod

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline  & _Case 1_ & _Case 2_ & _Case 3_ & _Case 4_ \\ \cline{2-6} Percentage & 63.77 & 40.44 & 30.97 & 16.24 \\ \hline \end{tabular}
\end{table}
Table 1: Percetange of occurrence of different cases in the dataset as mentioned in Section 5.2. All cases are analyzed with \(K=5\). Utterances whose history has aletaest \(3\) similar emotion labels in either own history or the history of the other person, is counted in case 1 or 2, respectively. Case 3 is considered when the utterance’s emotion is found in atleast \(3\) utterances which occur before the second past-utterance of each history. Case 4 is considered when no history has the emotion label of the current utterance.

 

els. CMN succeeds over both neural (Poria et al., 2017b) and SVM-based (Rozgic et al., 2012) methods by 3.3% and 8.12%, respectively. Improvement in performance is seen for all emotions over the ensemble-SVM based method. A similar trend is seen with bc-LSTM (Poria et al., 2017b), where our model does explicitly well for the active emotions _happiness_ and _anger_. This trend suggests that CMN is capable of capturing inter-speaker emotional influences which are often seen in the presence of such active emotions.

The importance of sequential processing of the histories using a recurrent neural network (in our case, a GRU) is evidenced by the poorer performance of Memn2n with respect to CMN. This suggests that gathering contexts temporally through sequential processing is indeed a superior method over non-temporal memory representations. CMN\({}_{self}\) which uses only single history channel also provides lesser performance when compared to CMN. This signifies the role of inter-speaker influences that often moderate the emotions of the current utterance. Overall, predictions on valence and arousal levels also show similar results which reinforce our hypothesis of CMN's ability to model emotional dynamics.

Hyperparameters:Figure 4 provides a summary of the performance trend of our model for different values of the hyperparameters K (context window length) and Q (number of hops). In the first graph, as \(K\) increases, more past-utterances are provided to the model as memories. The performance maintains a positive correlation with \(K\). This trend supplements our intuition that the historical context acts as an essential resource to model emotional dynamics. Given enough history, the performance saturates. The second graph shows that multiple hops on the histories indeed lead to an improvement in performance. The attention-based filtering in each hop provides a refined context representation of the histories. Models with hops in the range of \(3-10\) outperform the single layer variant. However, each added hop contributes a new set of parameters for memory representation, leading to an increase in total parameters of the model and making it susceptible to overfitting. This effect is evidenced in the figure where higher hops lead to a dip in performance.

Multimodality:Table 3 summarizes the performance of unimodal and multimodal variants of the baselines along with CMN. As seen in the table, text modality performs best out of the three. This is in contrast to Rozgic et al. 2012 where audio provides the best performance. A possible reason for

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c|c} \hline \multirow{2}{*}{Models} & \multicolumn{2}{c|}{_unimodal_} & \multicolumn{2}{c|}{_unimodal_} & \multicolumn{2}{c|}{_unimodal_} & \multicolumn{2}{c}{_trimodal_} \\  & \multicolumn{2}{c|}{_audio_} & \multicolumn{2}{c|}{_visual_} & \multicolumn{2}{c|}{_text_} & \multicolumn{2}{c|}{_text_} & \multicolumn{2}{c}{_trimal_} \\ \hline SVM-ensemble & 60.8 & 51.5 & 48.5 & 69.5\({}^{\ddagger}\) \\ bc-LSTM & 62.2 & 56.1 & 72.5 & 74.3\({}^{\ddagger}\) \\ Memm2n & 63.0 & 61.8 & 72.6 & 75.0\({}^{\ddagger}\) \\ CMN\({}_{self}\) & 63.1 & 62.5 & 73.0 & 76.5\({}^{\ddagger}\) \\ CMN\({}_{NA}\) & 62.4 & 60.9 & 74.1 & 75.7 \\ CMN & **65.3** & **64.2** & **74.2** & **77.6\({}^{\ddagger}\)** \\ \hline t: significantly better than unimodals (p ¡ 0.05) & & & & & & & \\ \end{tabular}
\end{table}
Table 3: Comparison of CMN to all the baselines in different modalities. Weighted accuracy is used as the metric.

Figure 4: Performance trends of our model with different values of K (history length) and Q (number of hops). While K is varied, Q is set to be \(3\). Similarly, \(K=20\) when Q varies.

 

this shift is the improved representational scheme of the textual modality. Text tends to have lesser noisy signals as opposed to audio-visual sources, thus providing better features in the joint representation. Overall, multimodal systems outperform the unimodal variants justifying the design of CMN as a multimodal system.

Table 3 also showcases the superiority of CMN and its variants over bc-LSTM. The proposed model achieves better performance over the state of the art in all the unimodal and multimodal segments. This asserts the importance of the memory-network framework and its ability to effectively store context information.

Role of Attention:Attention module plays a vital role in memory refinement. This is also observed in Table 2, where \(\text{CMN}_{NA}\) provides inferior performance over CMN. With the uniform weight, all the memory cells in both memories \(M_{a}\) and \(M_{b}\) equally contribute to the output representation. This incorporates irrelevant information from the perspective of emotional context.

Case Study:We perform qualitative visualization of the attention module by applying it on the testing set. Figure 4(a) represents a conversation where both the speakers are in an excited and jolly mood. Person A, in particular, drives the dialogue with less influence from Person B. To classify the test utterance of A, the attention module of CMN successfully focuses on the utterances \(1,3,5\) which had triggered the speaker's positive mood in the video. This shows CMN's capacity to model speaker-based emotions. Also, at the textual level, utterances \(3\) and \(6\) do not seem to depict a happy mood. However, audio and visual sources provide contrasting evidence which helps CMN to correctly model them as utterances spoken with happiness. This shows the advantage of a multimodal system.

In Figure 4(b) we reiterate through the dialogue presented in Figure 1. As shown, Person A converses in a sad mood (utterances \(1,3,5\) in Fig 4(b)), bounded by the grief of his wife's departure. But when he expresses his inhibitions, his wife B reacts in an angry and sarcastic manner (utterance \(7\)). This ignites an emotional shift for A who then replies angrily. In this example, CMN is able to focus on utterance \(7\) spoken by B to anticipate A's test utterance to be an angry statement, thus showing its ability to model inter-speaker influences. However, there are cases where our model fails, e.g., in the absence of historical utterances as this forces attention to focus on null memories.

## 6 Conclusion

In this paper, we presented a deep neural framework that identifies emotions for utterances in dyadic conversational videos. Our results suggest that leveraging context information from utterance histories and representing them as memories indeed helps to better recognize emotions. Performing speaker-specific modeling and considering inter-speaker influences also helps in capturing emotional dynamics.

This work also showed the importance of attention mechanism in filtering relevant contextual information from utterance histories and, hence, paved the path to the development of more efficient and human-like dialogue systems.

## Acknowledgement

This research was supported in part by the National Natural Science Foundation of China under Grant no. 61472266 and by the National University of Singapore (Suzhou) Research Institute, 377 Lin Quan Street, Suzhou Industrial Park, Jiang Su, People's Republic of China, 215123.

Figure 5: Average attention vectors across \(3\) hops for both memories for a given test utterance.

 



## References

* Aldeneh et al. (2017) Zakaria Aldeneh, Soheil Khorram, Dimitrios Dimitriadis, and Emily Mower Provost. 2017. Pooling acoustic and lexical features for the prediction of valence.
* Amir (1998) Noam Amir. 1998. Towards an automatic classification of emotions in speech. _ICSLP_, pages 699-702.
* Atrey et al. (2010) Pradeep K Atrey, M Anwar Hossain, Abdulmotaleb El Saddik, and Mohan S Kankanhandili. 2010. Multimodal fusion for multimedia analysis: a survey. _Multimedia systems_, 16(6):345-379.
* Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. _arXiv preprint arXiv:1409.0473_.
* Bergstra and Bengio (2012) James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. _Journal of Machine Learning Research_, 13(Feb):281-305.
* Bertero et al. (2016) Dario Bertero, Farhad Bin Siddique, Chien-Sheng Wu, Yan Wan, Ricky Ho Yin Chan, and Pascale Fung. 2016. Real-time speech emotion and sentiment recognition for interactive dialogue systems. In _EMNLP_, pages 1042-1047.
* Busso et al. (2008) Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungokle Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. _Language resources and evaluation_, 42(4):335.
* Busso et al. (2004) Carlos Busso, Zhigang Deng, Serdar Yildirim, Murtaza Bulut, Chul Min Lee, Abe Kazemzadeh, Sungbok Lee, Ulrich Neumann, and Shrikanth Narayanan. 2004. Analysis of emotion recognition using facial expressions, speech and multimodal information. In _ICMI_, pages 205-211.
* Cambria et al. (2017) Erik Cambria, Soujanya Poria, Alexander Gelbukh, and Mike Thelwall. 2017. Sentiment analysis is a big suitcase. _IEEE Intelligent Systems_, 32(6):74-80.
* Cambria et al. (2018) Erik Cambria, Soujanya Poria, Devamanyu Hazarika, and Kenneth Kwok. 2018. SenticNet 5: Discovering conceptual primitives for sentiment analysis by means of context embeddings. In _AAAI_.
* Castellano et al. (2008) Ginevra Castellano, Loic Kessous, and George Caridakis. 2008. Emotion recognition through multiple modalities: face, body gesture, speech. _Affect and emotion in human-computer interaction_, pages 92-103.
* Cho et al. (2014) Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. _arXiv preprint arXiv:1406.1078_.
* Datcu and Rothkrantz (2008) Dragos Datcu and L Rothkrantz. 2008. Semantic audio-visual data fusion for automatic emotion recognition. _Euromedia'2008_.
* Datcu and Rothkrantz (2011) Dragos Datcu and Leon JM Rothkrantz. 2011. Emotion recognition using bimodal data fusion. In _International Conference on Computer Systems and Technologies_, pages 122-128. ACM.
* De Choudhury et al. (2013) Munmun De Choudhury, Michael Gamon, Scott Counts, and Eric Horvitz. 2013. Predicting depression via social media. _ICWSM_, 13:1-10.
* Dellaert et al. (1996) Frank Dellaert, Thomas Polzin, and Alex Waibel. 1996. Recognizing emotion in speech. In _ICSLP_, volume 3, pages 1970-1973.
* D'mello and Kory (2015) Sidney K D'mello and Jacqueline Kory. 2015. A review and meta-analysis of multimodal affect detection systems. _ACM Computing Surveys (CSUR)_, 47(3):43.
* Ekman (1993) Paul Ekman. 1993. Facial expression and emotion. _American psychologist_, 48(4):384.
* Eyben et al. (2010) Florian Eyben, Martin Wollmer, and Bjorn Schuller. 2010. Opensmile: the munch versatile and fast open-source audio feature extractor. In _International Conference on Multimedia_, pages 1459-1462. ACM.
* Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. _arXiv preprint arXiv:1410.5401_.
* Hareli and Rafaieli (2008) Shlomo Hareli and Anat Rafaieli. 2008. Emotion cycles: On the social influence of emotion in organizations. _Research in organizational behavior_, 28:35-59.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Kalchbrenner et al. (2014) Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In _ACL 2014_, volume 1, pages 655-665.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _EMNLP 2014_, pages 1746-1751.
* Koval and Kuppenas (2012) Peter Koval and Peter Kuppenas. 2012. Changing emotion dynamics: individual differences in the effect of anticipatory social stress on emotional inertia. _Emotion_, 12(2):256.
* Kumar et al. (2016)Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In _ICML_, pages 1378-1387.
* Liu and Matilis (2014) Feng Liu and Sally Matilis. 2014. Emotional dynamics and strategizing processes: A study of strategic conversations in top team meetings. _Journal of Management Studies_, 51(2):202-234

Michael W Morris and Dacher Keltner. 2000. How emotions work: The social functions of emotional expression in negotiations. _Research in organizational behavior_, 22:1-50.
* Navarretta et al. (2016) Costanza Navarretta, K Choukri, T Declerck, S Goggi, M Grobelnik, and B Maegaard. 2016. Mirroring facial expressions and emotions in dyadic conversations. In _LREC_.
* Picard (2010) Rosalind W Picard. 2010. Affective computing: from laughter to ieee. _IEEE Transactions on Affective Computing_, 1(1):11-17.
* Poria et al. (2017a) Soujanya Poria, Erik Cambria, Rajiv Bajpai, and Amir Hussain. 2017a. A review of affective computing: From unimodal analysis to multimodal fusion. _Information Fusion_, 37:98-125.
* Poria et al. (2017b) Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017b. Context-dependent sentiment analysis in user-generated videos. In _ACL 2017_, volume 1, pages 873-883.
* Poria et al. (2017c) Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louisiana-Philippe Morency. 2017c. Multi-level multiple attentions for contextual multimodal sentiment analysis. In _ICDM 2017_, pages 1033-1038. IEEE.
* Poria et al. (2016) Soujanya Poria, Erik Cambria, Devamanyu Hazarika, and Prateek Vij. 2016. A deeper look into sarcastic tweets using deep convolutional neural networks. In _COLING 2016_, pages 1601-1612.
* Ranganathan et al. (2016) Hirnamayi Ranganathan, Shayak Chakraborty, and Sethuraman Panchanathan. 2016. Multimodal emotion recognition using deep learning architectures. In _WACV_, pages 1-9. IEEE.
* Richards et al. (2003) Jane M Richards, Emily A Butler, and James J Gross. 2003. Emotion regulation in romantic relationships: The cognitive consequences of concealing feelings. _Journal of Social and Personal Relationships_, 20(5):599-620.
* Rosenberg (2012) Andrew Rosenberg. 2012. Classifying skewed data: Importance weighting to optimize average recall. In _INTERSPEECH 2012_.
* Rozgic et al. (2012) Viktor Rozgic, Sankaranarayananan Ananthakrishnan, Shirin Saleem, Rohit Kumar, and Rohit Prasad. 2012. Ensemble of svm trees for multimodal emotion recognition. In _APSIPA ASC_, pages 1-4.
* Ruusuvuori (2013) Johanna Ruusuvuori. 2013. Emotion, affect and conversation. _The handbook of conversation analysis_, pages 330-349.
* Sidnell and Stivers (2012) Jack Sidnell and Tanya Stivers. 2012. _The handbook of conversation analysis_, volume 121. John Wiley & Sons.
* Song et al. (2004) Mingli Song, Jiajun Bu, Chun Chen, and Nan Li. 2004. Audio-visual based emotion recognition-a new approach. In _CVPR_, volume 2.
* Sukhbaatar et al. (2015) Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In _NIPS 2015_, pages 2440-2448.
* Tran et al. (2015) Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. 2015. Learning spatiotemporal features with 3d convolutional networks. In _ICCV 2015_, pages 4489-4497.
* Tzirakis et al. (2017) Panagiotis Tzirakis, George Trigeorgis, Mihalis A Nicolaou, Bjorn W Schuller, and Stefanos Zafeiriou. 2017. End-to-end multimodal emotion recognition using deep neural networks. _IEEE JSTPSP_, 11(8):1301-1309.
* Wang et al. (2017) Haohan Wang, Aaksha Meghawat, Louis-Philippe Morency, and Eric P Xing. 2017. Select-additive learning: Improving generalization in multimodal sentiment analysis. In _ICME_, pages 949-954.
* Weston et al. (2014) Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. _arXiv preprint arXiv:1410.3916_.
* Wallmer et al. (2010) Martin Wallmer, Angeliki Metallinou, Florian Eyben, Bijorn Schuller, and Shrikanth S Narayanan. 2010. Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional lstm modeling. In _INTERSPEECH 2010_.
* Xiaolan et al. (2013) Peng Xiaolan, Xie Lun, Liu Xin, and Wang Zhiliang. 2013. Emotional state transition model based on stimulus and personality characteristics. _China Communications_, 10(6):146-155.
* Xing et al. (2018) Frank Xing, Erik Cambria, and Roy Welsch. 2018. Natural language based financial forecasting: A survey. _Artificial Intelligence Review_.
* Yang et al. (2011) Liang Yang, Hong-fei LIN, and Wei GUO. 2011. Text-based emotion transformation analysis. _Computer Engineering & Science_, 9:026.
* Young et al. (2018) Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Augmenting end-to-end dialog systems with commonsense knowledge. In _AAAI_.
* Young et al. (2017) Tom Young, Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. 2017. Recent trends in deep learning based natural language processing. _arXiv preprint arXiv:1708.02709_.
* Zadeh et al. (2017) Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. In _EMNLP 2017_, pages 1103-1114.
* Zhou et al. (2017) Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. 2017. Emotional chatting machine: Emotional conversation generation with internal and external memory. _arXiv:1704.01074_