SlowGuess/ACL_2020_OCR huggingface dataset:
Fusion of Attentions. We then obtain softmax of all these computed different attention scores to squash them in a range of [0,1] so that the ones having maximum contribution gets the highest probability values and vice-versa. We then compute the matrices of attention outputs for different tasks and modalities from the different attention scores as:
$$
A = \operatorname {s o f t m a x} \left(Q _ {i} K _ {j} ^ {T}\right) V _ {i} \tag {5}
$$
where  $A\in \mathbb{R}^{n\times d_c}$ . So, we obtain 13 different attention outputs from its corresponding attention scores which are  $SA\in \mathbb{R}^{n\times d_c}$  for  $SA_{1}$ ,  $SA_{2}$ ,  $SA_{3}$ ,  $SA\in \mathbb{R}^{w\times d_c}$  for  $SA_{a}$ ,  $SA\in \mathbb{R}^{f\times d_c}$  for  $SA_{v}$ ,  $IMA_{v1}\in \mathbb{R}^{f\times d_c}$ ,  $IMA_{v3}\in \mathbb{R}^{f\times d_c}$ ,  $IMA_{a1}\in \mathbb{R}^{w\times d_c}$ ,  $IMA_{a3}\in \mathbb{R}^{w\times d_c}$ ,  $IMA_{va}\in \mathbb{R}^{f\times d_c}$ ,  $ITA_{12}\in \mathbb{R}^{n\times d_c}$ ,  $ITA_{21}\in \mathbb{R}^{n\times d_c}$  and  $ITA_{31}\in \mathbb{R}^{n\times d_c}$ .
Next, we obtain mean of different attention outputs in varying combinations to finally obtain representations for each of the modalities and tasks as  $M_1^{DA}$ ,  $M_2^{DA}$ ,  $M^E$ ,  $M^v$  and  $M^a$ .
$$
M _ {1} ^ {D A} = \operatorname {m e a n} \left(S A _ {1}, I M A _ {v a}, I T A _ {1 2}\right) \tag {6}
$$
$$
M _ {2} ^ {D A} = \operatorname {m e a n} \left(S A _ {2}, I T A _ {2 1}\right) \tag {7}
$$
$$
M ^ {E} = \operatorname {m e a n} \left(S A _ {3}, I T A _ {3 1}\right) \tag {8}
$$
$$
M ^ {v} = \operatorname {m e a n} \left(S A _ {v}, I M A _ {v 1}, I M A _ {v 3}\right) \tag {9}
$$
$$
M ^ {a} = \operatorname {m e a n} \left(S A _ {a}, I M A _ {a 1}, I M A _ {a 3}\right) \tag {10}
$$
where  $M \in \mathbb{R}^{1 \times d_c}$ . Next, we focus on learning appropriate weights to combine these representations to obtain final sentence representation for each of the tasks to be optimized jointly.
$$
W _ {1} = M _ {1} ^ {D A} * M _ {2} ^ {D A} \tag {11}
$$
$$
W _ {2} = M _ {1} ^ {D A} * M ^ {E} \tag {12}
$$
IEMOCAP	MELD
# Utterance	# Dialogue	# Utterance	# Dialogue
Train	7497	242	7489	831
Test	1879	60	2500	208
Table 2: Statistics of the train and test set of the EMO-TyDA dataset from different sources
where  $*$  represents dot product of two vectors. Finally, we obtain sentence representation  $(S)$  for each of the tasks as follows:
$$
S _ {D A} = M _ {1} ^ {D A} + W _ {1} * M _ {2} ^ {D A} + W _ {2} * M ^ {E} \tag {13}
$$
$$
S _ {E} = M ^ {E} * M ^ {v} * M ^ {a} \tag {14}
$$
4.2.3 Classification Layer
The output, i.e., sentence representation for each of the tasks ( $S_{DA}$  and  $S_{E}$ ) from the TAS are connected to a fully-connected layer which in turn consists of the output neurons for both the tasks (DAC and ER). The errors computed from each of these channels are back-propagated jointly to the successive prior layers of the model in order to learn the joint features of both the tasks thereby, allowing them to benefit from the TAS layer. As the main aim of this study is to learn DA with the help of emotion, the performance of the DAC task also banks on the quality of features learned for the ER task with useful and better features assisting the collective learning process and vice-versa.
EMOTyDA dataset was divided into two parts of  $80\%$  -  $20\%$  split for train and test set respectively. The statistics of the train and test set are shown in Table 2. For all the experiments conducted, same train and test sets were employed to allow a fair distinction between all approaches. For encoding the textual modality, a Bi-LSTM layer with 200 memory cells was used followed by a dropout rate of 0.1. Fully-connected layer of dimension 300 was used in all the subsequent layers. The first and the second channel contain 12 and 10 output neurons, respectively, for the DA and the emotion tags. Categorical crossentropy loss function is used in both the channels. A learning rate of 0.01 was found to be optimum. Adam optimizer was used in the final experimental setting. All these values are selected after a thorough sensitivity analysis of the parameters.
EMOTyDA contains dialogues pertaining to dyadic and multi-party speakers, so, we performed experi
Dataset
EMOTyDA:dyadic	EMOTyDA:multiparty	EMOTyDA
DA	DA + ER	DA	DA + ER	DA	DA + ER
Modality	Acc.	F1-score	Acc.	F1-score	Acc.	F1-score	Acc.	F1-score	Acc.	F1-score	Acc.	F1-score
Text (T)	63.75	60.67	65.23	62.35	46.20	39.23	48.90	41.10	53.56	49.17	53.02	50.22
Audio (A)	32.06	24.95	35.42	38.92	25.76	19.45	26.58	21.01	27.13	23.09	28.65	24.87
Video (V)	35.94	29.71	36.88	30.34	27.23	20.26	28.12	21.03	30.16	26.85	32.09	27.73
T + A	65.43	60.67	66.98	62.08	47.17	40.30	49.42	41.69	54.12	50.00	56.62	51.99
A + V	38.59	34.98	40.07	36.00	27.91	22.76	28.95	23.89	32.09	28.86	33.76	29.13
T + V	67.12	64.14	70.55	68.12	49.80	41.90	51.00	44.52	57.31	53.20	60.88	57.96
T + A + V	66.35	62.30	69.45	67.00	49.02	41.00	50.65	44.00	56.77	52.09	59.86	56.05
T + V (emotional cue)	65.26	60.20	-	-	46.88	39.70	-	-	54.31	50.02	-	-
Table 3: Results of all the baselines and the proposed models in terms of accuracy and F1-score. All the reported results are statistically significant
Model	EMOTyDA (DA + ER)
Acc.	F1-score
Feature level (early fusion)	51.20	48.09
Hidden-state level (late fusion)	53.27	49.80
Hypothesis level	50.93	47.31
T + V (SA)	56.76	49.84
T + V (IMA)	56.62	52.79
T + V (ITA)	56.99	52.23
T + V (SA + IMA)	56.62	51.70
T + V (SA + ITA)	58.48	52.62
T + V (IMA + ITA)	57.74	52.85
T + V	60.88	57.96
Table 4: Results of various baseline models for the multi-task framework for the EMOTyDA dataset
ments segregating dyadic and multi-party conversations as well in addition to the whole dataset for the multi-task framework along with different modalities. Additionally, we also provide results of the multi-task framework with its varying combinations of different attentions applied to provide analysis on the effectiveness of each attention for the entire EMOTyDA dataset. Along with this, we also include results of some simple baselines such as feature level, hidden state level and hypothesis level concatenation. It is to be noted that the purpose of the current work is to examine the effect of emotion while deciding the DA of an utterance from multiple modalities. We, therefore, do not focus on enhancements or analysis of the ER task and view it as an auxiliary task aiding the primary task, i.e., DAC. In regards to this, the results and findings are reported with respect to only the DAC task and its different combinations.
Table 3 shows the results of all the various models. As visible, the textual modality provides the best results amongst the uni-modal variants. The addition of audio and visual features individually improves this uni-modal baseline. The combination of visual and textual features achieves the best score throughout all the combinations of the dataset. The tri-modal variant is not able to attain the best score supposedly because of suboptimal
Figure 5: The visualization of the attention scores for 5 sample utterances for the tri-modal variant.  $V, A$  and  $T$  represent attention scores of video, audio and textual features, respectively. Sample utterance -  $u_{1}$ : "I am not in the least bit drunk.",  $u_{2}$ : "There's a lot of people looking for jobs.",  $u_{3}$ : "It was ridiculous. Completely ridiculous.",  $u_{4}$ : "You don't have to explain.",  $u_{5}$ : "No, Rachel doesn't want me to..."
performance of the audio modality. Though it still improves the performance compared to all the unimodal baselines. Figure 5 shows the heatmap visualization of the tri-modal variant to highlight the contributions of different modalities.
As is also evident from the results, the multi-task variant performs consistently well throughout all the experiments compared to its single task DAC variant. As a baseline, we also show that using emotion as a feature in the single task DAC counterpart doesn't outperform the proposed multi-task variant. This shows that the joint optimization of both these tasks boosts the performance of DAC. Table 4 shows the results of few simple baselines along with the ablation study of different attentions used in the proposed framework to highlight the importance and effectiveness of each of the attentions used for the whole EMOTyDA dataset. As seen from the table, the combinations of all three attention mechanisms, i.e., SA, IMA and ITA, yields the best results, thus, stressing the roles of incorporating across-task and across-modal relationships.
Figure 6 shows the visualization of the learned weights of different words for a sample utterance for the single task DAC as well as the multi-task model to highlight the importance of incorpora
Utterance	True Label	MT(T+V)	ST (T+V)
She is not Larry's girl	dag	dag	s
I know, it was amazing! I mean, we totally nailed it, it was beautiful.	ag	ag	o
Then why is she still single?,New York is full of men.,Why hasn't she married?	o	s	q
Probably a hundred people told her she's foolish, but she's waited.
God, I,feel so guilty about Ross.	ap	ap	s
Table 5: Sample utterances with its predicted labels for the best performing multi-task (MT)  $(\mathrm{T} + \mathrm{V})$  model and its single task (ST) DAC variants; These examples show that ER as an auxiliary task helps DAC for better performance in MT.
Figure 6: The visualization of the learned weights for an utterance -  $u_{1}$ : "Oh yes, yes I am, you can't stop me." for the best performing model (T+V), single task DAC (baseline) and multi-task DAC+ER (proposed) model
ing ER as an auxiliary task. The true DA label of the utterance in Figure 6 is disagreement with emotion as anger. With the multi-task approach, the attention is laid on appropriate disagreement bearing words whereas with single task, attention is learnt on agreement words such as yes which here has just been used in a sarcastic way to disagree. It is also observed that the experiments with dyadic conversations attain better results as compared to multi-party conversations. This is supposedly due to the constant change of speakers in multi-party conversations that misleads the classifier to learn suboptimal features, thus, stressing on the role of using speaker information as valuable cues for DAC.
Error Analysis. Plausible reasons behind the faults in the DA prediction are as follows: (i) Skewed dataset: The occurrence of most of the tags in the proposed dataset is very less, i.e., the dataset is skewed as shown in Figure 1a. This consistently conforms with real time task-independent conversations where some tags occur less frequently as compared to others; (ii) Composite and longer length utterance: Most of the utterances in the dataset are longer in length and is also composite in nature encompassing diversified intentions in a single utterance. In such cases, it becomes difficult to learn features for discrete DAs; (iii) Mis-classification of emotion labels: Misclassification of the DAs can be attributed to the mis-classification of the emotions for that partic
ular utterance. Some examples for the same are shown in Table 5.
In this paper, we investigate the role of emotion and multi-modality in determining DAs of an utterance. To enable research with these aspects, we create a novel dataset, EMOTyDA, that contains emotion-rich videos of dialogues collected from various open-source datasets manually annotated with DAs. Consequently, we also propose an attention based (self, inter-modal, inter-task) multi-modal, multi-task framework for joint optimization of DAs and emotions. Results show that multi-modality and multi-tasking boosted the performance of DA identification compared to its unimodal and single task DAC variants. In future, conversation history, speaker information, fine-grained modality encodings can be incorporated to predict DA with more accuracy and precision.
Dr. Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya PhD scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia) for carrying out this research.
Anne H Anderson, Miles Bader, Ellen Gurman Bard, Elizabeth Boyle, Gwyneth Doherty, Simon Garrod, Stephen Isard, Jacqueline Kowtko, Jan McAllister, Jim Miller, et al. 1991. The hcrc map task corpus. Language and speech, 34(4):351-366.
Lisa Feldman. Barrett, Michael Lewis, and Jeannette M. Haviland-Jones. 1993. Handbook of emotions. The Guilford Press.
Wauter Bosma and Elisabeth Andre. 2004. Exploiting emotions to disambiguate dialogue acts. In Proceed-
ings of the 9th international conference on Intelligent user interfaces, pages 85-92. ACM.
Kristy Elizabeth Boyer, Joseph F Grafsgaard, Eun Young Ha, Robert Phillips, and James C Lester. 2011. An affect-enriched dialogue act classification model for task-oriented dialogue. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1190-1199. Association for Computational Linguistics.
Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. Iemocap: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42(4):335.
Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019. Towards multimodal sarcasm detection (an _obviously_ perfect paper). In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4619-4629.
Roddy Cowie, Ellen Douglas-Cowie, Nicolas Tsapatsoulis, George Votsis, Stefanos Kollias, Winfried Fellenz, and John G Taylor. 2001. Emotion recognition in human-computer interaction. IEEE Signal processing magazine, 18(1):32-80.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248-255.
Thomas Drugman and Abeer Alwan. 2011. Joint robust voicing detection and pitch estimation based on residual harmonics. In Twelfth Annual Conference of the International Speech Communication Association.
Thomas Drugman, Mark Thomas, Jon Gudnason, Patrick Naylor, and Thierry Dutoit. 2011. Detection of glottal closure instants from speech signals: A quantitative review. IEEE Transactions on Audio, Speech, and Language Processing, 20(3):994-1006.
Florian Eyben, Martin Wöllmer, and Björn Schuller. 2010. Opensmile: the munich versatile and fast open-source audio feature extractor. In Proceedings of the 18th ACM international conference on Multimedia, pages 1459-1462. ACM.
Pascale Fung, Dario Bertero, Peng Xu, Ji Ho Park, Chien-Sheng Wu, and Andrea Madotto. 2018. Empathetic dialog systems. In *The International Conference on Language Resources and Evaluation*. European Language Resources Association.
John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, volume 1, pages 517-520. IEEE.
Sergio Grau, Emilio Sanchis, Maria Jose Castro, and David Vilar. 2004. Dialogue act classification using a bayesian approach. In 9th Conference Speech and Computer.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778.
Peter A Heeman and James F Allen. 1995. The trains 93 dialogues. Technical report, ROCHESTER UNIV NY DEPT OF COMPUTER SCIENCE.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Chenyang Huang, Osmar Zaiane, Amine Trabelsi, and Nouha Dziri. 2018. Automatic dialogue generation with expressed emotions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 49-54.
Peter Lajos Ihasz and Victor Kryssanov. 2018. Emotions and intentions mediated with dialogue acts. In 2018 5th International Conference on Business and Industrial Research (ICBIR), pages 125-130. IEEE.
Anurag Illendula and Amit Sheth. 2019. Multimodal emotion classification. In Companion Proceedings of The 2019 World Wide Web Conference, pages 439-449.
Neha Jain, Shishir Kumar, Amit Kumar, Pourya Sham-solmoali, and Masoumeh Zareapoor. 2018. Hybrid deep neural networks for face emotion recognition. Pattern Recognition Letters, 115:101-106.
Dan Jurafsky. 1997. Switchboard swbd-damsl shallow-discourse-function annotation coders manual. *Institute of Cognitive Science Technical Report*.
Daniel Jurafsky, Rebecca Bates, Noah Coccaro, Rachel Martin, Marie Meteer, Klaus Ries, Elizabeth Shriberg, Andreas Stolcke, Paul Taylor, and Carol Van Ess-Dykema. 1997. Automatic detection of discourse structure for speech recognition and understanding. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 88-95. IEEE.
Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. arXiv preprint arXiv:1306.3584.
John Kane and Christer Gobl. 2013. Wavelet maxima dispersion for breathy to tense voice discrimination. IEEE Transactions on Audio, Speech, and Language Processing, 21(6):1170-1179.
Hamed Khanpour, Nishitha Guntakandla, and Rodney Nielsen. 2016. Dialogue act classification in domain-independent conversations using a deep recurrent neural network. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2012-2021.
Harshit Kumar, Arvind Agarwal, Riddhiman Dasgupta, and Sachindra Joshi. 2018. Dialogue act sequence labeling using hierarchical encoder with crf.
Yang Liu, Kun Han, Zhao Tan, and Yun Lei. 2017. Using context information for dialog act classification in dnn framework. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2170-2178.
Louis-Philippe Morency, Rada Mihalcea, and Payal Doshi. 2011. Towards multimodal sentiment analysis: harvesting opinions from the web. In Proceedings of the 13th International Conference on Multimodal Interfaces, ICMI 2011, Alicante, Spain, November 14-18, 2011, pages 169-176.
Nicole Novielli and Carlo Strapparava. 2013. The role of affect analysis in dialogue act identification. IEEE Transactions on Affective Computing, 4(4):439-451.
Daniel Ortega, Chia-Yu Li, Gisela Vallejo, Pavel Denisov, and Ngoc Thang Vu. 2019. Context-aware neural-based dialog act classification on automatically generated transcriptions. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7265-7269. IEEE.
Pinelopi Papalampidi, Elias Iosif, and Alexandros Potamianos. 2017. Dialogue act semantic representation and classification using recurrent neural networks. SEMDIAL 2017 SaarDial, page 104.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Verónica Pérez-Rosas, Rada Mihalcea, and Louis-Philippe Morency. 2013. Utterance-level multimodal sentiment analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, 4-9 August 2013, Sofia, Bulgaria, Volume 1: Long Papers, pages 973-982.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017a. Context-dependent sentiment analysis in user-generated videos. In Proceedings of the
55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873-883.
Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Mazumder, Amir Zadeh, and Louis Philippe Morency. 2017b. Multi-level multiple attentions for contextual multimodal sentiment analysis. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1033-1038. IEEE.
Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. 2019. MELD: A multimodal multi-party dataset for emotion recognition in conversations. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 527-536.
Vipul Raheja and Joel Tetreault. 2019. Dialogue act classification with context-aware self-attention. arXiv preprint arXiv:1904.02594.
Norbert Reithinger and Martin Klesen. 1997. Dialogue act classification using language models. In Fifth European Conference on Speech Communication and Technology.
Eugenio Ribeiro, Ricardo Ribeiro, and David Martins de Matos. 2019. A multilingual and multidomain study on dialog act recognition using character-level tokenization. Information, 10(3):94.
Tulika Saha, Saurabh Srivastava, Mauajama Firdaus, Sriparna Saha, Asif Ekbal, and Pushpak Bhattacharyya. 2019. Exploring machine learning and deep learning frameworks for task-oriented dialogue act classification. pages 1-8.
Elizabeth Shriberg, Raj Dhillon, Sonali Bhagat, Jeremy Ang, and Hannah Carvey. 2004. The icsi meeting recorder dialog act (mrda) corpus. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteor. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339-373.
Andreas Stolcke, Elizabeth Shriberg, Rebecca Bates, Noah Coccaro, Daniel Jurafsky, Rachel Martin, Marie Meteor, Klaus Ries, Paul Taylor, Carol Van Ess-Dykema, et al. 1998. Dialog act modeling for conversational speech. In AAAI Spring Symposium on Applying Machine Learning to Discourse Processing, pages 98-105.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural