Conclusion and Future Work	In this work, we presented a method to extend conversational multi-modal emotion datasets with dialogue act labels. We successfully show this on two well-established emotion datasets: IEMOCAP and MELD, which we labeled with dialogue acts and made publicly available for further study and research. As a first insight, we found that many of the dialogue acts and emotion labels follow certain relations. These relations can be useful to learn about the emotional behaviours with dialogue acts to build a natural dialogue system and for deeper conversational analysis. The conversational agent might benefit in generating an appropriate response when considering both emotional states and dialogue acts in the utterances. In future work, we foresee the human in the loop for the annotation process along with a pool of automated neural annotators. Robust annotations can be achieved with very little human effort and supervision, for example, observing and correcting the final labels produced by ensemble output labels from the neural annotators. The human-annotator might also help to achieve segmented-utterance labelling of the dialogue acts. We also plan to use these datasets for conversational analysis to infer interactive behaviours of the emotional states with respect to dialogue acts.

EDAs Analysis	We can see emotional dialogue act co-occurrences with respect to emotion labels in Figure FIGREF12 for both datasets. There are sets of three bars per dialogue act in the figure, the first and second bar represent emotion labels of IEMOCAP (IE) and MELD (ME), and the third bar is for MELD sentiment (MS) labels. MELD emotion and sentiment statistics are interesting as they are strongly correlated to each other. The bars contain the normalized number of utterances for emotion labels with respect to the total number of utterances for that particular dialogue act category. The statements without-opinion (sd) and with-opinion (sv) contain utterances with almost all emotions. Many neutral utterances are spanning over all the dialogue acts. Quotation (⌃q) dialogue acts, on the other hand, are mostly used with `Anger' and `Frustration' (in case of IEMOCAP), however, some utterances with `Joy' or `Sadness' as well (see examples in Table TABREF21).

The last utterance, “I'm so sorry!", has been completely disagreed by all the five annotators. Similar apology phrases are mostly found with `Sadness' emotion label's, and the correct dialogue act is Apology (fa). However, they are placed either in the sd or in ba dialogue act category. We believe that with human annotator's help those labels of the utterances can be corrected with very limited efforts.

Action Directive (ad) dialogue act utterances, which are usually orders, frequently occur with `Anger' or `Frustration' although many with `Happy' emotion in case of the MELD dataset. Acknowledgements (b) are mostly with positive or neutral, however, Appreciation (ba) and Rhetorical (bh) backchannels often occur with a greater number in `Surprise', `Joy' and/or with `Excited' (in case of IEMOCAP). Questions (qh, qw, qy and qy⌃d) are mostly asked with emotions `Surprise', `Excited', `Frustration' or `Disgust' (in case of MELD) and many are neutral. No-answers (nn) are mostly `Sad' or `Frustrated' as compared to yes-answers (ny). Forward-functions such as Apology (fa) are mostly with `Sadness' whereas Thanking (ft) and Conventional-closing or -opening (fc or fp) are usually with `Joy' or `Excited'. We also noticed that both datasets exhibit a similar relation between dialogue act and emotion.

Enriching Existing Conversational Emotion Datasets with Dialogue Acts using Neural Annotators.	The recognition of emotion and dialogue acts enrich conversational analysis and help to build natural dialogue systems. Emotion makes us understand feelings and dialogue acts reflect the intentions and performative functions in the utterances. However, most of the textual and multi-modal conversational emotion datasets contain only emotion labels but not dialogue acts. To address this problem, we propose to use a pool of various recurrent neural models trained on a dialogue act corpus, with or without context. These neural models annotate the emotion corpus with dialogue act labels and an ensemble annotator extracts the final dialogue act label. We annotated two popular multi-modal emotion datasets: IEMOCAP and MELD. We analysed the co-occurrence of emotion and dialogue act labels and discovered specific relations. For example, Accept/Agree dialogue acts often occur with the Joy emotion, Apology with Sadness, and Thanking with Joy. We make the Emotional Dialogue Act (EDA) corpus publicly available to the research community for further study and analysis.

Annotation of Emotional Dialogue Acts ::: Dialogue Act Tagset and SwDA Corpus	There have been many taxonomies for dialogue acts: speech acts  refer to the utterance, not only to present information but to the action at is performed. Speech acts were later modified into five classes (Assertive, Directive, Commissive, Expressive, Declarative) . There are many such standard taxonomies and schemes to annotate conversational data, and most of them follow the discourse compositionality. These schemes have proven their importance for discourse or conversational analysis . During the increased development of dialogue systems and discourse analysis, the standard taxonomy was introduced in recent decades, called Dialogue Act Markup in Several Layers (DAMSL) tag set. According to DAMSL, each DA has a forward-looking function (such as Statement, Info-request, Thanking) and a backwards-looking function (such as Accept, Reject, Answer) . The DAMSL annotation includes not only the utterance-level but also segmented-utterance labelling. However, in the emotion datasets, the utterances are not segmented, as we can see in Figure FIGREF2 first or fourth utterances are not segmented as two separate.

It is important to notice that the dialogue act annotation is based on the given transcripts, however, the emotional expressions are better perceived with audio or video . We report some examples where we mark the utterances with an determined label (xx) in the last row of Table TABREF21. They are skipped from the final annotation because of not fulfilling the conditions explained in Section SECREF14 It is also interesting to see the previous utterance dialogue acts (P-DA) of those skipped utterances, and the sequence of the labels can be followed from Figure FIGREF6 (utt-l1, utt-l2, con1, con2, con3). In the first example, the previous utterance was b, and three DANA models produced labels of the current utterance as b, but it is skipped because the confidence values were not sufficient to bring it as a final label. The second utterance can be challenging even for humans to perceive with any of the dialogue acts. However, the third and fourth utterances are followed by a yes-no question (qy), and hence, we can see in the third example, that context models tried their best to at least perceive it as an answer (ng, ny, nn).

The research in emotion recognition is growing very rapidly and many datasets are available, such as text-based, speech- or vision-level, and multimodal emotion data. Emotion expression recognition is a challenging task and hence multimodality is crucial . However, few conversational multi-modal emotion recognition datasets are available, for example, IEMOCAP , SEMAINE , MELD . They are multi-modal dyadic conversational datasets containing audio-visual and conversational transcripts. Every utterance in these datasets is labeled with an emotion label. In this work, we apply an automated neural ensemble annotation process for dialogue act labeling. Several neural models are trained with the Switchboard Dialogue Act (SwDA) Corpus ,  and used for inferring dialogue acts on the emotion datasets. We ensemble five model output labels by checking majority occurrences (most of the model labels are the same) and ranking confidence values of the models. We have annotated two potential multi-modal conversation datasets for emotion recognition: IEMOCAP (Interactive Emotional dyadic MOtion CAPture database)  and MELD (Multimodal EmotionLines Dataset) .

There are about 3% in IEMOCAP and 5% in MELD (BM). Finally, when none the above conditions are fulfilled, we leave out the label with an unknown category. This unknown category of the dialogue act is labeled with `xx' in the final annotations, and they are about 7% in IEMOCAP and 11% in MELD (NM). The statistics of the EDAs is reported in Table TABREF13 for both datasets. Total utterances in MELD includes training, validation and test datasets.

Figure FIGREF2, shows an example of dialogue acts with emotion and sentiment labels from the MELD dataset. We confirmed the reliability of annotations with inter-annotator metrics. We analysed the co-occurrences of the dialogue act and emotion labels and discovered a key relationship between them; certain dialogue acts of the utterances show significant and useful association with respective emotional states. For example, Accept/Agree dialogue act often occurs with the Joy emotion while Reject with Anger, Acknowledgements with Surprise, Thanking with Joy, and Apology with Sadness, etc. The detailed analysis of the emotional dialogue acts (EDAs) and annotated datasets are being made available at the SECURE EU Project website.

In our recent work, where we used dialogue acts to build a dialogue system for a social robot, we find this study and dataset very helpful. For example, we can extend our robotic conversational system to consider emotion as an added linguistic feature to produce natural interaction.

Annotation of Emotional Dialogue Acts ::: Data for Conversational Emotion Analysis	There are two emotion taxonomies: (1) discrete emotion categories (DEC) and (2) fined-grained dimensional basis of emotion states (DBE). The DECs are Joy, Sadness, Fear, Surprise, Disgust, Anger and Neutral; identified by Ekman et al. ekman1987universalemos. The DBE of the emotion is usually elicited from two or three dimensions , , . A two-dimensional model is commonly used with Valence and Arousal (also called activation), and in the three-dimensional model, the third dimension is Dominance. IEMOCAP is annotated with all DECs and two additional emotion classes, Frustration and Excited. IEMOCAP is also annotated with three DBE, that includes Valance, Arousal and Dominance . MELD , which is an evolved version of the Emotionlines dataset developed by , is annotated with exactly 7 DECs and sentiments (positive, negative and neutral).

Annotation of Emotional Dialogue Acts ::: Reliability of Neural Annotators	The pool of neural annotators provides a fair range of annotations, and we checked the reliability with the following metrics . Krippendorff's Alpha ($\alpha $) is a reliability coefficient developed to measure the agreement among observers, annotators, and raters, and is often used in emotion annotation . We apply it on the five neural annotators at the nominal level of measurement of dialogue act categories. $\alpha $ is computed as follows: where $D_{o}$ is the observed disagreement and $D_{e}$ is the disagreement that is expected by chance. $\alpha =1$ means all annotators produce the same label, while $\alpha =0$ would mean none agreed on any label. As we can see in Table TABREF20, both datasets IEMOCAP and MELD produce significant inter-neural annotator agreement, 0.553 and 0.494, respectively. A very popular inter-annotator metric is Fleiss' Kappa score, also reported in Table TABREF20, which determines consistency in the ratings.

Annotation of Emotional Dialogue Acts ::: Ensemble of Neural Annotators	First preference is given to the labels that are perfectly matching in all the neural annotators. In Table TABREF11, we can see that both datasets have about 40% of exactly matching labels over all models (AM). Then priority is given to the context-based models to check if the label in all context models is matching perfectly. In case two out of three context models are correct, then it is being checked if that label is also produced by at least one of the non-context models. Then, we allow labels to rely on these at least two context models. As a result, about 47% of the labels are taken based on the context models (CM). When we see that none of the context models is producing the same results, then we rank the labels with their respective confidence values produced as a probability distribution using the $softmax$ function. The labels are sorted in descending order according to confidence values. Then we check if the first three (case when one context model and both non-context models produce the same label) or at least two labels are matching, then we allow to pick that one.

Acknowledgements	We would like to acknowledge funding from the European Union's Horizon 2020 research and innovation programme under the Marie Sklodowska Curie grant agreement No 642667 (SECURE).

The fourth utterance, it could be segmented to have two dialogue act labels, for example, a statement (sd) and a question (qy). That provides very fine-grained DA classes and follows the concept of discourse compositionality. DAMSL distinguishes wh-question (qw), yes-no question (qy), open-ended (qo), and or-question (qr) classes, not just because these questions are syntactically distinct, but also because they have different forward functions . For example, yes-no question is more likely to get a “yes" answer than a wh-question (qw). This also gives an intuition that the answers follow the syntactic formulation of question, providing a context. For example, qy is used for a question that, from a discourse perspective, expects a Yes (ny) or No (nn) answer. We have investigated the annotation method and trained our neural models with the Switchboard Dialogue Act (SwDA) Corpus , . SwDA Corpus is annotated with the DAMSL tag set and it is been used for reporting and bench-marking state-of-the-art results in dialogue act recognition tasks , ,  which makes it ideal for our use case.

Context 2 DANA is another context model that uses three utterances similar to the Context 1 DANA model, but the utterances are composed as the mean of the word embeddings over each utterance, similar to the Utt-level 2 model ($mean$ passed to context model in Figure FIGREF10 with con2 line). Hence, the Context 2 DANA model is composed of one RNN layer with three input vectors, finally topped with the softmax function for computing the probability distribution of the dialogue acts. This model achieved 75.97% accuracy on the test set of the SwDA corpus. Context 3 DANA is a context model that uses three utterances similar to the previous models, but the utterance representations combine both features from the Context 1 and Context 2 models (con1 and con2 together in Figure FIGREF10). Hence, the Context 3 DANA model combines features of almost all the previous four models to provide the recognition of the dialogue acts. This model achieves 75.91% accuracy on the SwDA corpus test set.

The Switchboard DAMSL Coders Manual can be followed for knowing more about the dialogue act labels.

Annotation of Emotional Dialogue Acts ::: Neural Model Annotators	We adopted the neural architectures based on Bothe et al. bothe2018discourse where two variants are: non-context model (classifying at utterance level) and context model (recognizing the dialogue act of the current utterance given a few preceding utterances). From conversational analysis using dialogue acts in Bothe et al. bothe2018interspeech, we learned that the preceding two utterances contribute significantly to recognizing the dialogue act of the current utterance. Hence, we adapt this setting for the context model and create a pool of annotators using recurrent neural networks (RNNs). RNNs can model the contextual information in the sequence of words of an utterance and in the sequence of utterances of a dialogue. Each word in an utterance is represented with a word embedding vector of dimension 1024. We use the word embedding vectors from pre-trained ELMo (Embeddings from Language Models) embeddings . We have a pool of five neural annotators as shown in Figure FIGREF6. Our online tool called Discourse-Wizard is available to practice automated dialogue act labeling.

In this tool we use the same neural architectures but model-trained embeddings (while, in this work we use pre-trained ELMo embeddings, as they are better performant but computationally and size-wise expensive to be hosted in the online tool). The annotators are: Utt-level 1 Dialogue Act Neural Annotator (DANA) is an utterance-level classifier that uses word embeddings ($w$) as an input to an RNN layer, attention mechanism and computes the probability of dialogue acts ($da$) using the softmax function (see in Figure FIGREF10, dotted line utt-l1). This model achieved 75.13% accuracy on the SwDA corpus test set. Context 1 DANA is a context model that uses 2 preceding utterances while recognizing the dialogue act of the current utterance (see context model with con1 line in Figure FIGREF10).

It uses a hierarchical RNN with the first RNN layer to encode the utterance from word embeddings ($w$) and the second RNN layer is provided with three utterances ($u$) (current and two preceding) composed from the first layer followed by the attention mechanism ($a$), where $\sum _{n=0}^{n} a_{t-n} = 1$. Finally, the softmax function is used to compute the probability distribution. This model achieved 77.55% accuracy on the SwDA corpus test set. Utt-level 2 DANA is another utterance-level classifier which takes an average of the word embeddings in the input utterance and uses a feedforward neural network hidden layer (see utt-l2 line in Figure FIGREF10, where $mean$ passed to $softmax$ directly). Similar to the previous model, it computes the probability of dialogue acts using the softmax function. This model achieved 72.59% accuracy on the test set of the SwDA corpus.

The kappa $k$ can be defined as, where the denominator $1 -\bar{P}_e$ elicits the degree of agreement that is attainable above chance, and the numerator $\bar{P} -\bar{P}_e$ provides the degree of the agreement actually achieved above chance. Hence, $k = 1$ if the raters agree completely, and $k = 0$ when none reach any agreement. We got 0.556 and 0.502 for IEOMOCAP and MELD respectively with our five neural annotators. This indicated that the annotators are labeling the dialogue acts reliably and consistently. We also report the Spearman's correlation between context-based models (Context1 and Context2), and it shows a strong correlation between them (Table TABREF20). While using the labels we checked the absolute match between all context-based models and hence their strong correlation indicates their robustness.

Introduction	With the growing demand for human-computer/robot interaction systems, detecting the emotional state of the user can heavily benefit a conversational agent to respond at an appropriate emotional level. Emotion recognition in conversations has proven important for potential applications such as response recommendation or generation, emotion-based text-to-speech, personalisation, etc. Human emotional states can be expressed verbally and non-verbally , , however, while building an interactive dialogue system, the interface needs dialogue acts. A typical dialogue system consists of a language understanding module which requires to determine the meaning of and intention in the human input utterances , . Also, in discourse or conversational analysis, dialogue acts are the main linguistic features to consider . A dialogue act provides an intention and performative function in an utterance of the dialogue. For example, it can infer a user's intention by distinguishing Question, Answer, Request, Agree/Reject, etc. and performative functions such as Acknowledgement, Conversational-opening or -closing, Thanking, etc. The dialogue act information together with emotional states can be very useful for a spoken dialogue system to produce natural interaction .

Question: What other relations were found in the datasets?
Answer: Quotation (⌃q) dialogue acts, on the other hand, are mostly used with `Anger' and `Frustration', Action Directive (ad) dialogue act utterances, which are usually orders, frequently occur with `Anger' or `Frustration' although many with `Happy' emotion in case of the MELD dataset, Acknowledgements (b) are mostly with positive or neutral, Appreciation (ba) and Rhetorical (bh) backchannels often occur with a greater number in `Surprise', `Joy' and/or with `Excited' (in case of IEMOCAP), Questions (qh, qw, qy and qy⌃d) are mostly asked with emotions `Surprise', `Excited', `Frustration' or `Disgust' (in case of MELD) and many are neutral, No-answers (nn) are mostly `Sad' or `Frustrated' as compared to yes-answers (ny)., Forward-functions such as Apology (fa) are mostly with `Sadness' whereas Thanking (ft) and Conventional-closing or -opening (fc or fp) are usually with `Joy' or `Excited'