Venue: NAACL 2021

# Towards Sentiment and Emotion aided Multi-modal Speech Act Classification in Twitter

Tulika Saha, Apoorva Upadhyaya, Sriparna Saha and Pushpak Bhattacharyya

Department of Computer Science and Engineering, Indian Institute of Technology Patna, India (sahatulikal5,sriparna.saha,pushpakhb)@gmail.com

###### Abstract

Speech Act Classification determining the communicative intent of an utterance has been investigated widely over the years as a standalone task. This holds true for discussion in any fora including social media platform such as Twitter. But the emotional state of the tweeter which has a considerable effect on the communication has not received the attention it deserves. Closely related to emotion is sentiment, and understanding of one helps understand the other. In this work, we firstly create a new multi-modal, emotion-TA ('TA' means tweet act, i.e., speech act in Twitter) dataset called _EmoTA_ collected from open-source Twitter dataset. We propose a Dyadic Attention Mechanism (DAM) based multi-modal, adversarial multi-tasking framework. DAM incorporates intra-modal and inter-modal attention to fuse multiple modalities and learns generalized features across all the tasks. Experimental results indicate that the proposed framework boosts the performance of the primary task, i.e., TA classification (TAC) by benefiting from the two secondary tasks, i.e., Sentiment and Emotion Analysis compared to its uni-modal and single task TAC (tweet act classification) variants.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5727–5737 June 6–11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Identification of speech acts is one of the preliminary means of determining the communicative intent or pragmatics of a speaker (for example, statement, request, question etc.). This is true for dialogue system, speech transcription, social media such as Twitter, MySpace etc. Twitter is one of the leading micro-blogging services. By 2019, 330 million users were active monthly and 500 million tweets were sent per day1. Identification of tweet acts (TAs- speech acts in Twitter) is highly beneficial for Twitter as well as tweezers. For Twitter, it helps decipher a particular subject in terms of speech acts and discrepancy identification. It also helps in social media monitoring by analysing topic alteration or spamming. It assists the followers in monitoring and scanning the subject with the most advantageous speech acts based on their needs. This helps reduce their search space and encourages them to obtain useful information from out of millions of tweets. It gives the tweeter a greater sense of the content, mood and trend.

Footnote 1: [https://www.omnicoreagency.com/twitter-statistics/](https://www.omnicoreagency.com/twitter-statistics/)

A person's emotional state and sentiment greatly impacts its intended content Barrett et al. (1993). Often sentiment and emotion are treated as two different problems Do et al. (2019), Soleymani et al. (2017), Albanie et al. (2018), Hossain and Muhammad (2019), Majumder et al. (2019). However, sentiment and emotion are are closely related. For example, emotions such as _happy_ and _joy_ are inherently related to a _positive_ sentiment. But emotion is much more nuanced and fine-grained compared to sentiment Kumar et al. (2019). Emotion along with sentiment provides better understanding of the state of mind of the tweeter. For example, a _question_ or _statement_ is associated with _anticipation_. An _opinion_ is many times associated with _anger_ or _disgust_. The close association between emotion and sentiment motivates considering tweeter's sentiment along with emotion while deciphering the tweet acts. For expressive TAs such as "expression", "request", "threat" etc., the tweeter's sentiment and emotion can aid in classifying true communicative intent and vice-versa.

Additionally, multi-modal inputs, i.e., the combination of text and other nonverbal cues (emoijs in tweets) Felbo et al. (2017) help create reliable classification models aiding the identification of emotional state and sentiment of the tweeter which in turn help in determining correct TAs.

In this paper, we leverage the relationships as delineated above to predict TAs of tweets in a multimodal framework. In this multi-task framework, TAC is treated as the primary task and Sentiment

Analysis (SA) and Emotion Recognition (ER) as auxiliary (i.e., secondary) tasks.

Contributions of this paper are as follows : **i.** We create a new dataset called _EmoTA_ consisting of tweets with high-quality annotations of TAs, including emotionally aided and multi-modal cues; **ii.** We establish the need for considering the sentiment and emotional state of the tweeter while identifying TAs. **iii.** We propose a _Dyadic Attention Mechanism_ (DAM) based multi-task adversarial learning framework for multi-modal TAC, SA and ER. In DAM, we incorporate _intra-modal_ and _inter-modal_ attention to integrate information across multiple modalities and learn generalized features across multiple tasks; **iv.** We illustrate performance gains by jointly optimizing TAC, SA and ER. Multi-modal and multi-task TAC performs significantly better than its uni-modal and single task TAC variants.

## 2 Related Works

There exist plenty of works which address the task of TAC as a standalone problem. In [35], [19], authors proposed Machine Learning based approaches for TAC namely Support Vector Machines (SVM), Logistic Regression etc. In saha2020learning, authors proposed a first ever public dataset for the identification of speech acts in Twitter followed by a capsule based network built on top of BERT for TAC. In vosoughi2015deep, authors highlighted the importance of identification of tweet acts and established it to be one of the elementary steps for detection of rumours in Twitter. In saha2020learning, authors proposed an attention based model built on top of the Transformer for predicting TAs. In saha2020learning, authors proposed a capsule based network built on top of BERT for TAC. All these works utilized only the textual modality to identify TAs without any sentiment or emotional correlation of the tweeter. In cerisara2018learning, authors proposed a LSTM based study for jointly optimizing SA and TAC in a decentralized social media platform called Mastodon. However, they modelled their task as a multi-party conversation pretty different in essence to that of Twitter analysis. In jeong2009attention, authors presented a semi-supervised approach to identify speech acts in emails and different forums. These works, however, use datasets that comprise of face-to-face or telephone data that can not directly aid in advancing work on endless data in electronic mode such as micro-blogging networks, instant-messaging, etc.

Apart from these, identification of speech acts has been studied extensively for dialogue conversations starting from early 2000's with stolcke2000language being one of the benchmark works where the authors presented varieties of approaches such as Hidden Markov Models, Neural Networks and Decision Trees to identify dialogue acts on a benchmark dialogue data known as the Switchboard swBDGodfrey2014, sidafarey2021towards dataset. In saha2021towards, authors studied the role of emotion in identifying dialogue acts for a dyadic conversation by considering thee textual and the audio modality of the utterances in the conversation. In saha2020learning, authors proposed studying the role of emotion in determining dialogue acts on a dyadic and multi-party conversational dataset in a multi-modal framework (incorporating text, audio and video). However, tweets are unstructured and noisy communications with spelling mistakes, random coinages with limitations in expression because of character constraint per tweet. This makes it very different from face-to-face or other conversations.

## 3 Dataset

Here, we discuss the details of the newly created dataset, _EmoTA_.

### Data Collection

To begin with, we scanned the literature for the latest SA and ER dataset for Twitter in order to gather potentially emotionally rich tweets to explore its impact on TAC. Initially, we came across several SA and ER datasets for Twitter such as oleri2016using, moohammad2018semi, semEval-2018 moohammad2018semi, btabwal2017semeval, cbcher2017semeval, sbsmamad2018semi, sbsmamad20 

(Vosoughi and Roy, 2016), Saha et al. (2020a) etc. However, the tweets in these datasets were devoid of nonverbal cues such as emojis which are quite excessively used in Twitter.

### Data Annotation

To the best of our knowledge, we were unaware of any sizable and open sourced Twitter dataset annotated for its TA and emotion labels. Hence, the SemEval-2018 dataset has been manually annotated for its TA categories. Unlike dialogic conversations, there isn't a standard TA tag-set available for annotating tweets. However, we made use of 7 TA categories of Saha et al. (2020a) for annotating SemEval-2018 dataset as opposed to 5 and 6 TA categories of Zhang et al. (2011) and Vosoughi and Roy (2016), respectively. The 7 TA tags are "Statement" (sta), "Expression" (exp), "Question" (que), "Request" (req), "Suggestion" (sug), "Threat" (tht) and "Others" (oth). For the current work, we selected a subset of SemEval-2018 dataset amounting to 6810 tweets to create _EmoTA_ dataset. Three annotators who were graduate in English linguistics were accredited to annotate the tweets with the appropriate TA tags. They were asked to annotate these tweets individually by only viewing the tweet available without the information of the pre-annotated emotion tags. This was done so as to assure that the dataset does not get biased by specific TA-emotion pairs. The conflicting annotations were resolved through discussions and mutual agreements. The inter-annotator score over 80% was considered as reliable agreement. It was determined based on the count that for a given tweet more than two annotators agreed on a particular tag.

For annotating the dataset with sentiment labels, we followed a semi-supervised approach instead of manual annotation which is cost intensive. We used the IBM Watson Sentiment Classifier2, an open-sourced API readily available for obtaining silver standard sentiment label of the tweets categorized into 3 tags namely "Positive", "Negative" and "Neutral".

Footnote 2: [https://cloud.ibm.com/apidocs/natural-language-understanding#seminent](https://cloud.ibm.com/apidocs/natural-language-understanding#seminent)

### Emotion-Tweet Act Dataset : _EmoTA_

The _EmoTA_ dataset3 now comprises of 6810 tweets with the corresponding gold standard TA and multi-label emotion tags. Each of the tweet contains its Tweet ID and two modalities: text and emoji. Few sample tweets along with the corresponding TA, sentiment and emotion labels from the proposed dataset are shown in Table 1. Distributions of TA,

\begin{table}
\begin{tabular}{|c|l|c|c|c|} \hline  & **Tweet** & **TA** & **Emotion** & **Sentiment** \\ \hline _And it pisses me off more they killed people who sermedered. Hands up and all. If hands visible you shouldn’t be faring for your life_ & exp & anger, disgust, fear & negative \\ \hline _We’re going to get City in the next round for a reverse_ & tht & anger & negative \\ \hline _Ryan Gosling and Eva Mendes finally; B joforilan atony/dont boss/dont argue/do everything with kids/go on mini car trip/focus on love_ & sug & joy, love & positive \\ \hline @MendipHilkAONB do we think the swallows and swifts have gone? Photo d 3 nights ago, not seen since. \#Anatomy_ & que & pessimism, sadess & negative \\ \hline \end{tabular}
\end{table}
Table 1: Sample tweets from the EmoTA dataset with its corresponding true TA, Emotion and Sentiment labels

Figure 1: (a) Distribution of tweet act labels, (b) Distribution of emotion labels.

Figure 2: Distribution of sentiment labels



emotion and sentiment labels across the dataset are shown in Figure 0(a), 0(b) and 2, respectively.

### Qualitative Aspects

Below, we analyze using some samples from the dataset that require sentiment-emotion aided and multi-modal reasoning.

Role of Sentiment and Emotion.In Figure 4(b), we demonstrate using two examples from the dataset to establish our hypothesis that sentiment and emotional states of the tweeter can aid the identification of TAs. In the first instance, the tweeter questions about the impending doom supposedly because of a pessimistic expectation arising due to the negative sentiment. Similarly, in the second instance, because of a joyous emotion emerging due to positive sentiment, the tweeter shares an optimistic suggestion with the readers. The above examples highlight the need for incorporating these additional user behavior, i.e., sentiment and emotion while reasoning about TAs. Thus, stressing the requirement of addressing such synergy amongst TAC, SA and ER.

Role of Multi-modality.In Figure 4(a), we present two examples from the dataset to highlight the importance of including other nonverbal features such as emoji present in the tweet along with the text for several tweet analysis tasks. In the first example tweet, the text represents an overall negative sentiment with emotion such as anger and disgust. However, the presence of an emoji face with tears of joy gives it an emotion of joy along with the other emotions. Similarly, in the second example tweet, the text represents the emotional state of the tweeter as sad, whereas the ok, celebration and heart emojis depict the feeling of joy. These instances show that the presence of complementary information in the form of emojs aids the process of any twitter analysis task including TAC.

## 4 Proposed Methodology

The proposed multi-tasking, multi-modal approach and implementation details are outlined in this section.

### Feature Extraction

The procedure for feature extraction across multiple modalities is discussed below.

Textual Features.To extract textual features of a tweet \(U\) having \(n_{u}\) number of words, the representation of each of the words, \(w_{1},...,w_{u}\), where \(w_{i}\in\mathbb{R}^{d_{u}}\) and \(w_{i}\)'s are obtained from BERT Devlin et al. (2019) which is a multi-layered attention aided bidirectional Transformer Encoder model based on the original Transformer model Vaswani et al. (2017) where \(d_{u}=768\).

Emoji Features.To extract emoji features from a tweet, we use _emoji_, a python based library for eliciting the pictorial image of an emoji (primarily that of a face, object or symbols). A total of 1816 kind of emojis are available along with its different types. We then use _emoji2vec_Eisner et al. (2016), which provides \(d_{v}=300\) dimensional vector representation for each of the emojis present in the tweet. Let's say a tweet contains \(n_{v}\) number of emoji. Thus, we obtain the final emoji representation \(V\) for a tweet as \(V\in\mathbb{R}^{n_{v}\times d_{v}}\).

### Network Architecture

The proposed network consists of four main components : (i) _Modality Encoders_ (ME) produces respective modality encodings by taking as input the uni-modal features extracted above, (ii) _Dyadic Attention Mechanism_ (DAM) that comprises dual attention mechanisms such as _intra-modal_ and _intermodal_ attentions, (iii) _Adversarial Loss_ to make the feature spaces of task-specific and shared layers of each task mutually exclusive, (iv) _Classification Layer_ that contains output channels for the three tasks at hand (TAC, SA and ER) to learn generalized representations across all the tasks.

Figure 3: (a) Importance of emoji in analysis of tweets, (b) Importance of emotion and sentiment in TAC.

 



#### 4.2.1 Modality Encoders

In this section we discuss how the architectural framework encodes different modalities.

Text and Emoji Modalities.The features \(U\) and \(V\) obtained from each of the modalities corresponding to a tweet (discussed above) are then passed through two discrete Bi-directional LSTMs [6] sequentially encode these representations and learn complementary semantic dependency based features into hidden states from these modalities. In case of textual modality (say), the final hidden state matrix of a tweet is obtained as \(H_{u}\in\mathbb{R}^{n_{u}\times 2d_{l}}\). \(d_{l}\) represents the number of hidden units in each LSTM and \(n_{u}\) is the sequence length. In the similar way, a representation of corresponding emoji modality encoding as \(H_{v}\in\mathbb{R}^{n_{v}\times 2d_{l}}\) is obtained. The number of representations from modality encoders vary depending on the variant of the multitask learning framework used (e.g., fully shared (FS) or shared-private model (SP)). In a FS variant, two representations are obtained one for text and another for emoji cumulatively for optimizing all the three tasks. However, for a SP model, six encoding representations are obtained. Three for text and the remaining for emoji forming a pair of text-emoji representations for each of the three tasks.

#### 4.2.2 Dyadic Attention Mechanism

We use a similar concept as in [30], where the authors proposed to compute attention as mapping a query and a set of key-value pairs to an output. So, the representations obtained from the modality encoders above are passed through three fully-connected layers each termed as queries and keys of dimension \(d_{k}=d_{f}\) and values of dimension \(d_{v}=d_{f}\). For a FS model, we have two triplets of (\(Q,K,V\)) as : \((Q_{u},K_{u},V_{u})\) and \((Q_{v},K_{v},V_{v})\). Similarly for a SP model, we have six such triplets as : \((Q_{u1},K_{u1},V_{u1})\), \((Q_{v1},K_{v1},V_{v1})\), \((Q_{uz},K_{u2},V_{u2})\), \((Q_{v2},K_{v2},V_{v2})\), \((Q_{u3},K_{u3},V_{u3})\), \((Q_{v3},K_{v3},V_{v3})\) where pair of two triplets are from the textual and emojI modality encoders for each of the tasks4. These triplets are then used to compute attention values for different purposes in various combinations which include intra attention and inter-modal attention.

Footnote 4: Subscript \(1\), \(2\) and \(3\) represent TAC, ER and SA task, respectively.

Intra-modal Attention.We compute intra-modal attention (IA) for all these individual modalities in order to learn the interdependence between the current words and the preceding part of the tweet. In a way, we aim to relate different positions of a single sequence to estimate a final representation of the same sequence for individual modalities [30]. Thus, the IA scores for individual modalities are calculated as :

\[IA_{j}=softmax(Q_{j}K_{j}^{T})V_{j} \tag{1}\]

Figure 4: The architectural diagram of the proposed network

where \(IA\in{\mathbb{R}^{n_{u}\times d_{f}}}\) for \(IA_{u}\), \(IA\in{\mathbb{R}^{n_{v}\times d_{f}}}\) for \(IA_{v}\) for FS model and six such \(IA\) scores for SP model.

Inter-modal Attention.The IA scores obtained above are then used to compute inter-modal attention (IRA). We re-iterate the same process (explained above) to now form triplets of \((Q,K,V)\) for these IA scores and then compute IRA scores amongst triplets of all IA scores by computing the matrix multiplication of combination of queries and keys of different IA modality scores using Equation 1. In this manner, we obtain one \(IRA\) score as \(IRA_{uv}\in{\mathbb{R}^{n_{u}\times d_{f}}}\) for FS variant and three \(IRA\) scores for SP model as \(IRA_{w1}\), \(IRA_{uv2}\) and \(IRA_{uv3}\). This is done to distinguish important contributions between various modalities to achieve optimal representation of a tweet.

Attention Fusion.Next, we concatenate each of these computed \(IA\) and \(IRA\) vectors as :

\[C=concat(IRA_{uv},IA_{u},IA_{v}),\text{for FS} \tag{2}\]

\[C_{1}=concat(IRA_{uv1},IA_{u1},IA_{v1}),\text{for SP} \tag{3}\]

\[C_{2}=concat(IRA_{uv2},IA_{u2},IA_{v2}),\text{for SP} \tag{4}\]

\[C_{3}=concat(IRA_{uv3},IA_{u3},IA_{v3}),\text{for SP} \tag{5}\]

Next, we obtain mean of these three different concatenated attention vectors for the SP variant or directly use the obtained \(C\) attention vector for the FS variant to obtain the final representation of a tweet.

\[M=mean(C_{1},C_{2},C_{3}) \tag{6}\]

Shared Layer.Additionally, for the SP model, other than having task-specific layers, we allow a shared layer to learn task invariant features. Here, the shared layer is in the form of a fully-connected layer of dimension \(d_{f}\). The inputs to the shared layer are the hidden representations of three IRA vectors : \(IRA_{uv1}\), \(IRA_{uv2}\) and \(IRA_{uv3}\). Thus for a given tweet, the loss of the shared layer is minimized if the model correctly classifies the tasks of each of the tweets in the input. This helps learn domain invariant feature space for different tasks.

Adversarial Loss.The goal of this adversarial loss function is to tune the weights of the shared layer so that it learns a representation that misleads the task discriminator. The adversarial loss \(l_{adv}\), aims to make the feature space of shared and task-specific layers to be mutually exclusive Liu et al. (2017). We follow the similar strategy as that of Liu et al. (2017), where a task discriminator \(D\) (say) maps the shared feature to its original task. Thus, on a correct prediction when the loss at the shared layer decreases, the adversarial loss increases and vice-versa. Alternatively, the shared layer is tuned to work in an adversarial way, thereby prohibiting the discriminator to predict one of the three tasks. The adversarial loss is computed as :

\[l_{adv}=\min_{F}(\max_{D}(\sum_{n=1}^{N}\sum_{k=1}^{K}d_{k}^{n}\log[D(F(x_{k}^ {n}))])) \tag{7}\]

where \(d_{k}^{m}\) represents the true label amongst the type of the tasks, \(N\), and \(x_{k}^{n}\) is the \(k^{th}\) example for task \(n\). The min-max optimization problem is addressed by the gradient reversal layer Ganin and Lempitsky (2015).

#### 4.2.3 Classification Layer

The final representation of the tweet obtained from the DAM module is shared across three channels pertaining to the three tasks, i.e., TAC, SA and ER (for FS model) and three DAM representations for three individual tasks are subjected to individual output layer (for SP model). The task-specific loss (\(l_{t}\)), shared loss (\(l_{s}\)) and adversarial loss (\(l_{adv}\)) are used as

\[l_{f}=l_{t}+\alpha l_{s}+\gamma l_{adv},\text{for SP model} \tag{8}\]

\[l_{f}=l_{s}+\gamma l_{adv},\text{for FS model} \tag{9}\]

where \(\alpha\) and \(\gamma\) are hyper-parameters.

### Experimentation Details

Hyper-parameters.80% of the tweets of the _EmoTA_ dataset were used for training and the remaining 20% were used for testing the models. The same training and testing data were used for all the experiments in order to ensure fair comparison of models. To encode different modalities, a Bi-LSTM layer with 100 memory cells was used. Dense layers of dimensions 100 were used for \(d_{f}\). The three channels contain 7, 3 and 11 output neurons, for TA, sentiment and emotion tags, respectively. _Categorical crossentropy_ loss is used for TA and sentiment channels and _Binary crossentropy_ loss function is used for emotion channel. A learning rate of 0.01 and _Adam_ optimizer were used in the final experimental setting. _All these values of the parameters were selected after a careful sensitivity analysis._

Pre-processing.We employ NLTK based _TweetTokenizer_ to tokenize tweets. UrIs were removed. User mentions were replaced by <user> token. Numbers occurring in the tweet were replaced by 

\(<\)number\(>\) token. _Ekphrasis_[2] was used to extract hashtags by segmenting long string into its constituent words. All the characters of the tweet were lower-cased. Since the dataset is under-represented for most of the TA tags, we over-sample 80% of the tweets used for training as : the mediocrely represented tags (e.g., sug, que and oth) are over-sampled to be equally represented as the most represented tags (e.g., sta and exp). Similarly, the highly under-represented classes (e.g., req and tht) are over-sampled to be equally represented as the mediocrely represented tags in the _EmoTA_ dataset. All the results reported below are on the 20% test data without any over-sampling.

## 5 Results and Analysis

A series of experiments were conducted for evaluating the proposed approach. Experiments were conducted for single task and several combinations of multi-task framework with TAC being the pivotal task along with varying modalities. A thorough ablation study is performed to analyze the importance of each of the attention mechanisms of the proposed architectural framework along with several variations of multi-task learning (e.g., FS, SP etc.). _Note that we aim to enhance the performance of TAC with the help of other two auxiliary tasks. Following this, we report results and analysis with TAC strictly being the pivotal task in all the task combinations._ Since, the dataset is unbalanced for all the task categories, we report results for different dimensions of TAC in the following set-up:

* **Five-class** Classification : This includes the top 5 highly occurring TA tags namely sta, exp, que, sug and oth.
* **Seven-class** Classification : This includes all the 7 categories of TAs used in the annotation process.

Table 3 and 2 illustrate the results of single task TAC and varying combinations of multi-task proposed models for different set-up (as mentioned

Figure 5: The visualization of the learned weights for a tweet from \(IA_{u}\) layer: \(u_{1}\): “I lost a couple niggas I want revenge so put me in coach.” for single task TAC (baseline), multi-task TAC+SA+ER (proposed) models

\begin{table}
\begin{tabular}{|p{49.8pt}||p{26.8pt}|p{26.8pt}|p{26.8pt}|p{26.8pt}|p{26.8pt}|p{26.8pt}|p{26.8pt}|p{26.8pt}|} \hline  & \multicolumn{5}{c|}{**SA \& ER**} \\ \cline{2-11} \multirow{2}{*}{**Model**} & \multicolumn{5}{c|}{**Five-Class**} & \multicolumn{5}{c|}{**Five-Class**} & \multicolumn{5}{c|}{**Serve-Class**} \\ \cline{2-11}  & \multicolumn{2}{c|}{**Test**} & \multicolumn{2}{c|}{**Test**} & \multicolumn{2}{c|}{**Test**} & \multicolumn{2}{c|}{**Test**} & \multicolumn{2}{c|}{**Test**} & \multicolumn{2}{c|}{**Length**} \\ \cline{2-11}  & Acc. & F1 & Acc. & F1 & Acc. & F1 & Acc. & F1 & Acc. & F1 \\ \hline Single Task SA & 87.26 & 86.05 & 88.52 & 87.20 & 88.85 & 87.30 & 90.10 & 89.00 \\ \hline Simple Task H& 81.57 & 60.77 & 84.63 & 64.32 & 80.07 & 73.51 & 81.85 & 76.63 \\ S/A-T SAC & 99.31 & 87.85 & 90.47 & 89.09 & 88.86 & 87.51 & 95.56 & 90.35 \\ \hline ER+ TAC & 83.52 & 65.58 & 86.09 & 67.02 & 81.37 & 75.21 & 84.21 & 78.30 \\ SA + HE & 92.30 & 91.06 & 93.02 & 92.00 & 90.33 & 88.65 & 92.73 & 90.37 \\ \hline SA + ER & 84.61 & 68.77 & 87.37 & 70.19 & 82.72 & 70.00 & 88.30 & 72.04 \\ \hline SA + ER + TAC & 92.06 & 91.13 & 93.19 & 92.38 & 92.49 & 90.53 & 93.68 & 91.82 \\ (for sk) & & & & & & & & \\ \hline SA + ER + TAC & 85.39 & 70.04 & 88.31 & 72.77 & 83.26 & 79.66 & 86.01 & 81.05 \\ \hline \end{tabular}
\end{table}
Table 4: Results of the proposed model for the single and multi-task SA and ER 

above). As evident, the addition of non-verbal cues in the form of emojis improves the uni-modal textual baseline consistently. This improvement implies that the proposed architecture utilizes the interaction among the input modalities very effectively. This highlights the importance of incorporating multi-modal features for different Twitter analysis tasks. We also report result for utilizing emoji as textual feature instead of treating it as a different modality in the single task TAC framework. Also, the five-class set-up gave better results than the seven-class set up. This is pretty obvious, as with 5-class set-up, the model needs to distinguish and identify lesser fine-grained features compared to the 7-class set-up. Additionally, the under-representation of two tags in the _EmoTA_ dataset for the 7-class set-up also effects its performance.

As seen in Table 2, the multi-task framework with all the three tasks (i.e., TAC + SA + ER) consistently gave better results as compared to single task TAC. In the bi-task variant, TAC+SA, shows little improvement in different metrics as opposed to TAC+ER over and above the single task TAC. This gain is rather intuitive as sentiment alone is sometimes unable to convey complete information of the tweeter's state of mind. E.g., a _negative_ sentiment can occur because of various emotions such as _disgust_, _fear_, _sadness_ etc. Similarly, a _positive_ sentiment can take place because of emotions such as _happiness_, _surprise_ etc. Thus, with sentiment alone, sometimes this discreteness or fine differences in the state of mind cannot be completely determined and conveyed. To illustrate this, in Figure 5, we provide a visualization of the learned weights of a tweet from the \(IA_{u}\) layer (as this layer contains word-wise attention scores). For this particular tweet, its true TA label is _tht_. With the multi-task framework, the importance of warning bearing words are learnt well such as _lost, reverse_ compared to the single-task TAC where attention is laid on expression bearing word such as _put me_. Additionally, we also report results for cases where sentiment and emotion were directly used as features in the single task TAC models to leverage from instead of deploying a multi-task based approach in Table 3.

As stated above, we treat SA and ER as auxiliary tasks aiding the primary task, i.e., TAC. However, we report the performance of SA and ER tasks on the proposed model for single as well as multi-task frameworks in Table 4 for further investigations. However, we do not make any explicit effort to enhance their performance.

Comparison amongst Different Multi-task Architecture.In terms of varying ways of multi-tasking such as FS, SP along with adversarial loss (adv), it was observed that SP model gave better results compared to FS model. Additionally, incorporating adversarial loss further boosted the performance of different multi-task models. Intuitively, as TAC shares lesser amount of correlation with SA and ER compared to SA and ER themselves, FS model was not sufficient enough to learn diverse features across different tasks. This observation is in conformity with the existing literature. We also demonstrate the importance of different attentions used for the best performing multi-task model, i.e., SP+Adv. Furthermore, we also report results by replacing BERT model to extract textual representation with Glove embeddings Pennington et al. (2014). Results indicate that each of these aspects contributed significantly to aid the performance of the proposed multi-tasking framework. _All the reported results here are statistically significant Welch (1947)._

Comparison with the State of the Art Models.We also compare our proposed approach with the recent state of the art models for single task TAC as we are unaware of any other work which jointly optimized tweet act, emotion and sentiment in Twitter. In Table 6, we report the results for the same by re-implementing those on the _EmoTA_ dataset. As

\begin{table}
\begin{tabular}{|c|c|c|} \hline  & **Tree** & **TAC** & **TAC+SA** & **TAC+ER** \\ \hline \#BreeyWeckes have berey, you wantang give me some of that coffee you posted an your soap?? please & rep & exp & req & req \\ \hline \#we going to get City in the next round for a average & tht & sta & exp & tht \\ \hline \#vagemnagine, did you put team from @FreePeople ’ viral insult to ballet? Stop trying to wrongfully & sug & que & exp & eug & sug \\ \hline \multicolumn{3}{|r|}{_I wonder if Corey will vote for Nickel? Finance dbk’s pandemic within glateball/animal_} & que & que & que & que & que \\ \hline \end{tabular}
\end{table}
Table 5: Sample tweets from the EmoTA dataset with its corresponding ground truth and predicted labels for different single and multi-task models 

evident, the proposed model outperformed these SOTA approaches.

Error AnalysisAn in-depth analysis revealed several scenarios as to why the proposed model faltered which are as follows : _i._**Imbalanced Dataset** : As visible in Figure 0(a), except for "sta" and "exp" tags, all the classes are under-represented in the _EmoTA_ dataset. Even though we apply oversampling to partially counter this issue but still the tags such as "req" and "ht" contain very little tweets for the model to learn fine differences amongst different categories. In accordance with this, we observe that five-class performs exceptionally better than the seven-class classification set-up; _ii._**Fine-grained tags** : It was also observed that the tweets which were mis-classified were subset of each other. For instance, tweet such as "_don't get discouraged! it's early on; it can get overwhelming. keep reading; use cue cards it'll get better!!_" is wrongly predicted as "exp" rather than "sup" which in the superficial way is a subset of the former tag; _iii._**Miscellaneous** : Tweets belonging to "oth" tag was also majorly mis-classified as there was no fixed pattern of tweets belonging to this category. To counter this, even more fine-grained categories of TAs needs to be identified and modelled. _Sample utterances for the error analysis are shown in Table 5._

## 6 Conclusion and Future Work

In this paper, we studied the role of sentiment and emotion in speech act classification in Twitter. We curate a novel dataset _EmoTA_, that contains pre-annotated tweets with emotions collected from open-source dataset and annotated with TAs and sentiment categories. We propose a Dyadic Attention Mechanism based multi-modal (emojis and text), adversarial multi-task framework for joint optimization of TAs, sentiment and emotions. The DAM (dyadic attention mechanism) module employs intra-modal and inter-modal attention to fuse multiple modalities and learn generalized features across all the tasks. Results show that multi-modality and multi-tasking boosted the performance of TA identification compared to its uni-modal and single task TAC variants.

In future, attempts will be made to predict TAs with more precision by incorporating fine-grained modality encodings and also identifying which other NLP tasks (e.g., named entity recognition) might assist TAC as a task.

## Acknowledgments

Dr. Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia) for carrying out this research.

## References

* Albanie et al. (2018) Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. 2018. Emotion recognition in speech using cross-modal transfer in the wild. In _Proceedings of the 26th ACM international conference on Multimedia_, pages 292-301.
* Feldman et al. (1993) Lisa Feldman. Barrett, Michael Lewis, and Jeannette M. Haviland-Jones. 1993. _Handbook of emotions_. The Guilford Press.
* Baziotis et al. (2017) Christos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. Datastories at semeval-2017 task 4: Deep lstm with attention for message-level and topic-based sentiment analysis. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 747-754, Vancouver, Canada. Association for Computational Linguistics.
* Cerisara et al. (2018) Christophe Cerisara, Somayeh Jafaritzaehjani, Adedayo Oluokun, and Hoa Le. 2018. Multi-task dialog act and sentiment recognition on mastodon. _arXiv preprint arXiv:1807.05013_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_, pages 4171-4186.
* Da et al. (2019) Hai Ha Do, PWC Prasad, Angelika Maag, and Abeer Alsaloon. 2019. Deep learning for aspect-based sentiment analysis: a comparative review. _Expert Systems with Applications_, 118:272-299.
* Eisner et al. (2016) Ben Eisenr, Tim Rocktaschel, Isabelle Augenstein, Matko Bosnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In _Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, SocialNLP@EMNLP 2016, Austin, TX, USA, November 1, 2016_, pages 48-54. Association for Computational Linguistics.
* Ekman (1999) Paul Ekman. 1999. Basic emotions. _Handbook of cognition and emotion_, 98(45-60):16 

Bjarke Felbo, Alan Mislove, Anders Sogaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_, pages 1615-1625. Association for Computational Linguistics.
* Ganin and Lempitsky (2015) Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In _International conference on machine learning_, pages 1180-1189. PMLR.
* Go et al. (2009) Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. _CS224N project report, Stanford_, 1(12):2009.
* Godfrey et al. (1992) John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In _Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on_, volume 1, pages 517-520. IEEE.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Hossain and Muhammad (2019) M Shamim Hossain and Ghulam Muhammad. 2019. Emotion recognition using deep learning approach from audio-visual emotional big data. _Information Fusion_, 49:69-78.
* Jeong et al. (2009) Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In _Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3-Volume 3_, pages 1250-1259. Association for Computational Linguistics.
* Kumar et al. (2019) Abhishek Kumar, Asif Ekbal, Daisuke Kawahra, and Sadao Kurohashi. 2019. Emotion helps sentiment: A multi-task model for sentiment and emotion analysis. In _2019 International Joint Conference on Neural Networks (IJCNN)_, pages 1-8. IEEE.
* August 4, Volume 1: Long Papers_, pages 1-10. Association for Computational Linguistics.
* Majumder et al. (2019) Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 6818-6825.
* Mohammad (2012) Saif Mohammad. 2012. # emotional tweets. In
* _SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_, pages 246-255.
* Mohammad et al. (2018) Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In _Proceedings of the 12th international workshop on semantic evaluation_, pages 1-17.
* Mohammad and Kiritchenko (2018) Saif Mohammad and Svetlana Kiritchenko. 2018. Understanding emotions: A dataset of tweets to study interactions between affect categories. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
* Mohammad and Turney (2013) Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word-emotion association lexicon. _Computational Intelligence_, 29(3):436-465.
* Oleri and Karagoz (2016) Obrahim Oleri and Pinar Karagoz. 2016. Detecting user emotions in twitter through collective classification. In _KDIR_, pages 205-212.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_, pages 1532-1543.
* Plutchik (1980) Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In _Theories of emotion_, pages 3-33. Elsevier.
* Saha et al. (2021) Tulika Saha, Dhawal Gupta, Sriparna Saha, and Pushpak Bhattacharyya. 2021. Emotion aided dialogue act classification for task-independent conversations in a multi-modal framework. _Cognitive Computation_, 13(2):277-289.
* Saha et al. (2020a) Tulika Saha, Srivaras Ramesh Jayashree, Sriparna Saha, and Pushpak Bhattacharyya. 2020a. Bertcaps: A transformer-based capsule network for tweet act classification. _IEEE Transactions on Computational Social Systems_, 7(5):1168-1179.
* Saha et al. (2020b) Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020b. Towards emotion-aided multi-modal dialogue act classification. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4361-4372.
* Saha et al. (2020c) Tulika Saha, Aditya Prakash Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020c. A transformer based approach for identification of tweet acts. In _2020 International Joint Conference on Neural Networks (IJCNN)_, pages 1-8. IEEE.
* Saha et al. (2019) Tulika Saha, Sriparna Saha, and Pushpak Bhattacharyya. 2019. Tweet act classification: A deep learning based classifier for recognizing speech acts in twitter. In _2019 International Joint Conference on Neural Networks (IJCNN)_, pages 1-8. IEEE 

* Gholipour Shahraki and Zaiane (2017) Ameneh Gholipour Shahraki and Osmar R Zaiane. 2017. Lexical and learning-based emotion mining from text. In _Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing_, volume 9, pages 24-55.
* Soleymani et al. (2017) Mohammad Soleymani, David Garcia, Brendan Jou, Bjorn Schuller, Shih-Fu Chang, and Maja Pantic. 2017. A survey of multimodal sentiment analysis. _Image and Vision Computing_, 65:3-14.
* Stolcke et al. (2000) Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. _Computational linguistics_, 26(3):339-373.
* Thelwall et al. (2012) Mike Thelwall, Kevan Buckley, and Georgios Palatoglou. 2012. Sentiment strength detection for the social web. _Journal of the American Society for Information Science and Technology_, 63(1):163-173.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_, pages 5998-6008.
* Vosoughi (2015) Soroush Vosoughi. 2015. _Automatic detection and verification of rumors on Twitter_. Ph.D. thesis, Massachusetts Institute of Technology.
* Vosoughi and Roy (2016) Soroush Vosoughi and Deb Roy. 2016. Tweet acts: A speech act classifier for twitter. In _Tenth International AAAI Conference on Web and Social Media_.
* Wang et al. (2012) Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P Sheth. 2012. Harnessing twitter" big data" for automatic emotion identification. In _2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing_, pages 587-592. IEEE.
* Welch (1947) Bernard L Welch. 1947. The generalization ofstudent's problem when several different population variances are involved. _Biometrika_, 34(1/2):28-35.
* Zhang et al. (2011) Renxian Zhang, Dehong Gao, and Wenjie Li. 2011. What are tweeters doing: Recognizing speech acts in twitter. _Analyzing Microtext_, 11(05).