SlowGuess/ACL_2020_OCR huggingface dataset:
Tulika Saha,\* Aditya Prakash Patra,\* Sriparna Saha, Pushpak Bhattacharyya  
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India (sahatulika15, aditya.prakash.patra1997)@gmail.com (sriparna.saha, pushpakbh)@gmail.com
The task of Dialogue Act Classification (DAC) that purports to capture communicative intent has been studied extensively. But these studies limit themselves to text. Non-verbal features (change of tone, facial expressions etc.) can provide cues to identify DAs, thus stressing the benefit of incorporating multi-modal inputs in the task. Also, the emotional state of the speaker has a substantial effect on the choice of the dialogue act, since conversations are often influenced by emotions. Hence, the effect of emotion too on automatic identification of DAs needs to be studied. In this work, we address the role of both multi-modality and emotion recognition (ER) in DAC. DAC and ER help each other by way of multi-task learning. One of the major contributions of this work is a new dataset- multimodal Emotion aware Dialogue Act dataset called EMOTyDA, collected from open-sourced dialogue datasets. To demonstrate the utility of EMOTyDA, we build an attention based (self, inter-modal, inter-task) multi-modal, multi-task Deep Neural Network (DNN) for joint learning of DAs and emotions. We show empirically that multimodality and multi-tasking achieve better performance of DAC compared to uni-modal and single task DAC variants.
Dialogue Act Classification (DAC) is concerned with deciding the type i.e., communicative intention (question, statement, command etc.) of the speaker's utterance. DAC is very important in the context of discourse structure, which in turn supports intelligent dialogue systems, conversational speech transcription and so on. Considerable works have been done on classical Machine Learning (ML) based DAC (Jurafsky et al., 1997), (Stolcke et al., 2000), (Verbree et al., 2006), etc. and Deep
Learning (DL) based DAC (Kalchbrenner and Blunsom, 2013), (Papalampidi et al., 2017), (Liu et al., 2017), (Ribeiro et al., 2019), (Ortega et al., 2019), (Saha et al., 2019) etc.
Humans are emotional entities. A speaker's emotional state considerably influences or affects its intended content or its pragmatic content (Barrett et al., 1993). An utterance such as "Okay sure" or "Ya right" (say) can be considered as "agreement" or- in case of sarcasm- "disagreement". For expressive DAs such as "greeting", "thankling", "apologizing" etc., the speaker's feeling or emotion can assist in recognizing true communicative intent and vice-versa. Thus, it is important to consider the speaker's emotion when deciding on the DA.
There is considerable work on ER (Cowie et al., 2001), (Jain et al., 2018), (Zhang et al., 2018), etc. and adapting the Virtual Agents (VAs) to act accordingly (Huang et al., 2018), (Zhou et al., 2018), (Fung et al., 2018), etc. But very little research has been done, that addresses the impact of emotion while deciding the DA of an utterance (Novielli and Strapparava, 2013), (Bosma and Andre, 2004). As DAs primarily dictate the flow of any dialogue conversation (be it human-human or human-computer), such synergy of ER and DAC is required. Research too has shown the benefit of utilizing the combination of text and nonverbal cues (Poria et al., 2017b), (Poria et al., 2017a) etc., for solving various Natural Language Processing (NLP) tasks. The main advantage of integrating other modalities to text is the usage of behavioral signs present in acoustic (vocal modulations) and visual (facial expression) modalities. In addition, the various modalities offer important signals to better identify the speaker's communicative intention and emotional state. This will in effect help create sturdy and more reliable DAC models.
In this paper, we study the influence of emotion on the identification of DAs, by utilizing the com
The authors have contributed equally.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4361-4372 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
bination of text, vocal modulations and facial expressions for task-independent conversations. DAC is our primary task, assisted by Emotion Recognition (ER) as an auxiliary task. We implement an attention based multi-modal, multi-tasking DNN to do joint modeling of DAC and ER. Also, we introduce a new dataset to help advance research in multi-modal DAC.
The key contributions of this paper are as follows:  $i$ . We curate a new dataset called EMO-TyDA for facilitating multi-modal DAC research with high-quality annotations, including emotionally aided cues and conversational context features. We believe this dataset will advance research in multi-modal DAC;  $ii$ . We point to different scenarios where discrepancy in DAC is evident across different modalities, thus, showing the importance of multi-modal approaches to DAC;  $iii$ . We show using various instances, the usefulness of considering the emotional state of the user while identifying DAs. Consequently, we deduce that EMO-TyDA will lead to a novel sub-task for future research: emotion aware DAC;  $iv$ . We propose an attention based (self, inter-modal, inter-task) multitask, multi-modal DNN for jointly optimizing the DAC and ER task and show its benefit over single task DAC variants. Through this, we also establish that multi-modal DAC performs significantly better than uni-modal DAC.
The tasks of ER and DAC are extensively explored.
Dialogue Act Frameworks: DAC has been investigated since late 90s (Reithinger and Klesen, 1997), (Stolcke et al., 1998) and early 2000's (Stolcke et al., 2000), (Grau et al., 2004). Much of this research, however, uses chat transcripts with only the text mode, due partly due to unavailability of multi-modal open-source dataset. In (Khanpour et al., 2016), authors apply stacked LSTM to classify speech acts. In (Kumar et al., 2018), the author developed a Hierarchical Network based approach using Bi-LSTMs and the CRF. A contextual self-attention system fused with hierarchical recurrent units was proposed by the authors of (Raheja and Tetreault, 2019) to develop a sequence label classifier. The authors of (Yu et al., 2019) proposed a method for the capture of long-range interactions that span a series of words using a Convolutional Network based approach. In (Saha et al., 2019), authors proposed several ML and DL based
approaches such as Conditional Random Fields, clustering and word embeddings to identify DAs. However, all these works identify DAs by utilizing solely the textual modality without the use of emotional cues.
Emotion aware DAs. Within a multi-modal setting, little work is available in the literature to study the impact of emotional state in the evaluation of DAs. The effect of integrating facial features as a way of identifying emotion to classify DAs was examined by authors in (Boyer et al., 2011). They exhibited their work for tutorial dialogue session typically task-oriented and applied logistic regression to identify DAs. But they studied only the cognitive-affecting states such as confusion and flow as the emotional categories to learn DAs. In (Novielli and Strapparava, 2013), authors examined the impact of affect analysis in DA evaluation for an unsupervised DAC model. The authors made use of lexicon based features from WordNet Affect and SentiWordNet to map them with emotion labels to model the DAs in a LSA based approach. Authors of (Ihasz and Kryssanov, 2018), also inspected the impact of emotions mediated with intention or DAs for an in-game Japanese dialogue. Their goal was to construct DA-emotion combinations from the pre-annotated corpus. However, such stringent associations or dis-associations amongst DA-emotion pairs may not truly hold for real life conversations.
To facilitate and enhance the research in multimodal DAC assisted with user emotion, we introduce a new dataset (EMOTyDA) consisting of short videos of dialogue conversations manually annotated with its DA along with its pre-annotated emotions.
To gather potentially emotion rich conversations to explore its affect on DAC, we scanned the literature for existing multi-modal ER dataset. During our initial search, we obtained several multi-modal ER datasets which include Youtube (Morency et al., 2011), MOUD (Pérez-Rosas et al., 2013), IEMOCAP (Busso et al., 2008), ICT-MMMO (Wöllmer et al., 2013), CMU-MOSI (Zadeh et al., 2016), CMU-MOSEI (Zadeh et al., 2018) and MELD (Poria et al., 2019) etc. However, we zeroed down on IEMOCAP and MELD datasets for the further
investigations of our problem statement. The reason behind this choice was that remaining all the datasets mentioned above were particularly monologues involving opinions and product reviews. Whereas our research requires task-independent dyadic or multi-party conversations to analyze its full potential. Both these available datasets are not annotated for their corresponding DAs.
Also, benchmark DAC datasets such as Switchboard (SWBD) (Godfrey et al., 1992), ICSI Meeting Recorder (Shriberg et al., 2004) consist of text and audio-based conversations whereas TRAINS (Heeman and Allen, 1995) consist of solely text-based conversations with no emotional tags. HCRC Map Task corpus (Anderson et al., 1991) additionally encompasses audio modality with the transcripts but the corpus itself has task-oriented conversations and is not annotated for its emotion tags. It is to be noted that task-oriented conversations generally restrict the presence of diverse tags which are commonly encountered in task-independent conversations.
To the best of our knowledge, at the time of writing, we were unaware of any sizable and open-access DA and emotion annotated multi-modal dialogue data. Thus, MELD and IEMOCAP datasets have been manually annotated for the corresponding DAs to encourage and promote novel research on multi-modal DACs to build a multi-tasking system that allows DA and emotion for an utterance to be learned jointly.
Over the years, SWBD-DAMSL tag-set comprising of 42 DAs developed by (Jurafsky, 1997) has been used widely for the task of DAC for task-independent dyadic conversation such as SWBD corpus. Thus, we use SWBD-DAMSL tag-set as the base for conceiving tag-set for the EMO-TyDA dataset since both these datasets contain task-independent conversations. Of the 42 SWBD-DAMSL tags, 12 most commonly occurring tags have been used to annotate utterances of the EMO-TyDA dataset. The choice of 12 tags is because of the limited length of the EMO-TyDA dataset in comparison to the SWBD corpus. It stems from the fact that it is highly likely that many of the tags of the SWBD-DAMSL tag-set will never appear in the EMO-TyDA dataset due to lesser number of utterances and lower diversity of occurrence of such fine-grained tags. The 12 most commonly occur
ring chosen tags are Greeting (g), Question (q), Answer (ans), Statement-Oinion (o), Statement-Non-Oinion (s), Apology (ap), Command (c), Agreement (ag), Disagreement (dag), Acknowledge (a), Backchannel (b) and Others (oth).
For the current work, we have selected a subset of 1039 dialogues from MELD amounting to 9989 utterances and the entire IEMOCAP dataset of 302 dialogues amounting to 9376 utterances to curate EMOTyDA dataset. Details of the original MELD and IEMOCAP datasets are provided in the Appendix 6. Three annotators who were graduate in English linguistics were accredited to annotate the utterances with the appropriate DAs out of the 12 chosen tags. They were asked to annotate these utterances by only viewing the video available considering the dialogue history without the information of the pre-annotated emotion tags. This was done so as to assure that the dataset does not get biased by specific DA-emotion pairs. The inter-annotator score over  $80\%$  was considered as reliable agreement. It was determined based on the count that for a given utterance more than two annotators agreed on a particular tag. To remove the discrepancy in the number of emotion tags for IEMOCAP and MELD datasets, we mapped the joy tag of the MELD to the happy tag of the IEMOCAP to finally settle on 10 tags from the IEMOCAP for the EMOTyDA dataset.
3.3 Emotion-DA Dataset: EMOTyDA
The EMOTyDA dataset<sup>1</sup> now comprises of 1341 dyadic and multi-party conversations resulting in a total of 19,365 utterances or annotated videos with the corresponding DA and emotion tags considering the dialogue history. The dataset contains approximately 22 hours of recordings. Source distribution and major speakers statistics of the dataset are shown in Figures 3a and 3b, respectively. Since DAC and ER tasks are known to exploit the contextual features, i.e., dialogue history (Yu et al., 2019) so, utterances in the dataset are accompanied with their corresponding contextual utterances, which are typically preceding dialogue turns by the speakers participating in the dialogue. Each of the utterances contains three modalities: video, audio, and text. All the utterances are even followed by their speaker identifiers. Table 1 shows few utterances along with the corresponding DAs and emotion la
<sup>1</sup>The dataset with its DA and emotion tags will be made publicly available to the research community.
Speaker	Utterance	DA	Emotion
M_4	Adders don’t snap, they sting.	dag	ang
Rachel	Well, I just checked our messages and Joshua didn’t call.	s	sad
M_1	That’s very amusing indeed.	dag	ang
Chandler	Come on, pick up, pick up	c	fear
Table 1: Example utterances from the EMOTyDA dataset with its corresponding DA and emotion categories
1) Phoebe: Finel Then you tell Roger because he was really looking forward to this!
Text: suggests agreement or opinion Audio: commanding tone Video: furious
2) M_1: That's very amusing indeed.
Text: agreement Audio:sarcastic tone Video: slight anger
Figure 1: Statistics across the datasets : (a) Distribution of DA labels, (b) Distribution of emotion labels.
1) Monica: I can't leave it! You gouged a hole in my dingy floor.
DA: disagreement
2) M_2: Well, you know I appreciate you coming over and talking to me, I mean it definitely helps.
DA: acknowledge
Figure 2: (a) Incongruent modalities in DAC, (b) Importance of emotion in DAC.
(b)
Figure 3: Statistics : (a) Source across the dataset, (b) Overall speaker distribution.
bels from the proposed dataset. Distributions of DA and emotion labels across the source datasets are shown in Figure 1a and 1b, respectively.
3.4 Qualitative Aspects
In the current work, we seek to analyze the affect of emotion in classifying DAs. Also, DAC in text usually involves extra information that can be benefitted from associated modalities. Below, we analyze some samples that require emotion aided and multi-modal reasoning. We exemplify using few instances from our proposed dataset in order to support our claim of DA often being expressed in a multi-modal way along with exploiting the
emotional state of the speaker.
Role of Emotion. In Figure 2b, we present two instances from the dataset where the emotional state of the user seems beneficial in deciding the DA of an utterance. In the first example, the reference to the sad and dismal state of the speaker directs it to acknowledge the presence of the hearer. In the second case, the angry emotional state of the speaker forces her to disagree with people's opinion or suggestion involved in the conversation. The examples above illustrate the importance of having emotional information as emotions affect the communicative intention or DA of the speaker discussed above. The presence of emotion in our dataset caters the models with the ability to use additional information while reasoning about DA.
Role of Multi-modality. Figure 2a shows two cases where DA is articulated through incongruity between modalities. In the first instance, the facial modality implies anger or fury. Whereas the textual modality lacks any visible sign of displeasure, on the contrary it indicates an agreement. So, the textual claims does not validate the facial features. In the second case, the textual modality hints pure
agreement. Whereas the audio modality expresses a sarcastic appeal. In both these cases, there exists inconsistency between modalities, which acts as a strong indicator that multi-modal information is also important in providing additional cues for identifying DAs. The availability of complementary information across multiple modalities improves the model's ability to learn discriminatory patterns that are responsible for this complex process.
This section describes the proposed multi-task, multi-modal approach followed by the implementation details.
4.1 Multi-modal Feature Extraction
Here, we discuss, the process of multi-modal feature extraction.
Textual Features. The transcriptions available for each video forms the source of the textual modality<sup>2</sup>. To extract textual features, pretrained GloVe (Pennington et al., 2014) embeddings of dimension 300 have been used to obtain representation of words as word vectors. The resultant word embeddings of each word are concatenated to obtain a final utterance representation. While it is indeed possible to use more advanced textual encoding techniques (for e.g., convolutional or recurrent neural network), we decided to use the same pre-trained extractive strategy as in the case of other modalities.
Audio Features. To elicit features from the audio, openSMILE (Eyben et al., 2010), an open source software has been used. The features obtained by openSMILE include maxima dispersion quotients (Kane and Gobl, 2013), glottal source parameters (Drugman et al., 2011), several low-level descriptors (LLD) such as voice intensity, voice quality (for eg., jitter and shimmer), MFCC, voiced/unvoiced segmented features (Drugman and Alwan, 2011), pitch and their statistics (for eg., root quadratic mean, mean etc.), 12 Mel-frequency coefficients etc. All the above features are then concatenated together to form a  $d_{q} = 256$  dimensional representation for each window. The final audio representation of each utterance  $(A)$  is obtained by concatenating the obtained  $d_{q}$  for every window
i.e.,  $A\in \mathbb{R}^{w\times d_q}$  where  $w$  represents total window segments.
Video Features. To elicit visual features for each of the  $f$  frames from the video of an utterance, we use a pool layer of an ImageNet (Deng et al., 2009), pretrained ResNet-152 (He et al., 2016) image classification model. Initially, each of the frames is preprocessed which includes resizing and normalizing. So, the visual representation of each utterance  $(F)$  is obtained by concatenating the obtained  $d_{f} = 4096$  dimensional feature vector for every frame, i.e.,  $F \in \mathbb{R}^{f \times d_{f}}$  (Castro et al., 2019), (Illendula and Sheth, 2019), (Poria et al., 2017b), (Poria et al., 2017a).
The proposed network consists of three main components: (i) Modality Enocoders (ME) which primarily takes as input the uni-modal features (extracted above) and produce as outputs the individual modality encodings, (ii) Triplet Attention Subnetwork (TAS) that encompasses self, inter-modal and inter-task attention and (iii) classification layer that contains outputs of both the tasks (DAC and ER).
4.2.1 Modality Encoders
In this section, we discuss how different modalities are encoded in the architectural framework.
Textual Modality. The obtained utterance representation (U) from the extracted textual features (discussed above) is then passed through three different Bi-directional LSTMs (Bi-LSTMs) (Hochreiter and Schmidhuber, 1997) to sequentially encode these representations into hidden states and learn different semantic dependency based features pertaining to different task, i.e., DAC and ER. One Bi-LSTM learns DAC features that are tuned in accordance with the emotion features. Second learns features for the ER task regulated by the learning of DA features. The third Bi-LSTM learns private features for the task of DAC which is not influenced by the features learnt from emotion.
$$
\overrightarrow {h _ {i}} = L S T M _ {f d} \left(u _ {i}, \overrightarrow {h} _ {i - 1}\right), \tag {1}
$$
$$
\overleftarrow {h _ {i}} = L S T M _ {b d} \left(u _ {i}, \overleftarrow {h} _ {i + 1}\right). \tag {2}
$$
For each of these word features, its corresponding forward and backward hidden states  $\overrightarrow{h_i}$ ,  $\overleftarrow{h_i}$ , respectively, from the forward  $LSTM_{fd}$  and the backward  $LSTM_{bd}$  are concatenated to obtain a
2Original dataset with its video and transcript are downloaded from: https://github.com/ SenticNet/MELD, https://sail.usc.edu/ iemocap/iemocap_release.htm
Figure 4: The architectural diagram of the proposed network. SA, IMA, ITA represent self, inter-modal and inter-task attentions, respectively.
single hidden state  $h_i$ . The complete hidden state matrix is obtained as,
$$
H = \left[ h _ {1}, h _ {2}, \dots , h _ {n} \right], \tag {3}
$$
where  $H \in \mathbb{R}^{n \times 2d}$ .  $d$  represents the number of hidden units in each LSTM and  $n$  is the sequence length. Thus, the obtained three hidden state matrices correspond to three Bi-LSTMs, i.e.,  $H_{1}, H_{2}, H_{3}$ . These representations are then passed through three fully connected layers, each of dimension say  $d_{c}$  to learn attention of different variants.
Audio and Video Modalities. The audio and video features  $(A$  and  $F)$  extracted are also passed through three fully connected layers, each of dimension say  $d_{c}$ , to learn attention of different variants.
4.2.2 Triplet Attention Subnetwork
We use a similar concept as in (Vaswani et al., 2017), where the authors proposed to compute attention as mapping a query and a set of key-value pairs to an output. The output is estimated as a weighted sum of the values, where the weight assigned to each value is calculated by a compatibility function of the query with its corresponding key. So, the representations obtained from each of the modality encoders above which are passed through three fully-connected layers each are termed as queries and keys of dimension  $d_{k} = d_{c}$  and values of dimension  $d_{v} = d_{c}$ . We now have five triplets
of  $(Q,K,V)$  as:  $(Q_{1},K_{1},V_{1})$ ,  $(Q_{2},K_{2},V_{2})$ ,  $(Q_{3},K_{3},V_{3})$ ,  $(Q_{a},K_{a},V_{a})$ ,  $(Q_{v},K_{v},V_{v})$  where first three triplets are from the textual modality encoder (one each for DA_shared, DA_private and Emotion_shared)<sup>3</sup> followed by one from audio and video encoder each. These triplets are then used in different combinations to compute attention scores meant for specific purposes that includes self attention, inter-modal attention and inter-task attention.
Self Attention. We compute self attention  $(SA)$  for all these triplets by computing the matrix multiplication of all its corresponding queries to its corresponding keys.
$$
S A _ {i} = Q _ {i} K _ {i} ^ {T} \tag {4}
$$
where  $SA \in \mathbb{R}^{n \times n}$  for  $SA_1, SA_2, SA_3, SA \in \mathbb{R}^{n \times w}$  for  $SA_a, SA \in \mathbb{R}^{n \times f}$  for  $SA_v$ .
Inter-modal Attention. We compute inter-modal attention (IMA) amongst triplets of all the modalities for the multi-task by computing the matrix multiplication of combination of queries and keys of different modalities using Equation 4. In this manner, we obtain five IMA scores as  $IMA_{v1} \in \mathbb{R}^{f \times n}$ ,  $IMA_{v3} \in \mathbb{R}^{f \times n}$ ,  $IMA_{a1} \in \mathbb{R}^{w \times n}$ ,  $IMA_{a3} \in \mathbb{R}^{w \times n}$  and  $IMA_{va} \in \mathbb{R}^{f \times w}$ .
<sup>3</sup>Subscript 1, 2 and 3 represent DA_shared, DA_private and Emotion_shared representations, respectively.
This is done in order to identify significant contributions amongst different modalities to learn optimal features for an utterance.
Inter-task Attention. We compute inter-task attention (ITA) amongst triplets of different tasks from the textual modality by computing the matrix multiplication of combinations of queries and keys of different tasks using Equation 4. In this manner, we obtain three  $ITA_{12} \in \mathbb{R}^{n \times n}$ ,  $ITA_{21} \in \mathbb{R}^{n \times n}$  and  $ITA_{31} \in \mathbb{R}^{n \times n}$ . This is done in order to learn joint features of an utterance for identification of DAs and emotions.