SlowGuess/NAACL_2021_OCR huggingface dataset:
Department of Computer Science and Engineering, Indian Institute of Technology Patna, India
Speech Act Classification determining the communicative intent of an utterance has been investigated widely over the years as a standalone task. This holds true for discussion in any fora including social media platform such as Twitter. But the emotional state of the tweeter which has a considerable effect on the communication has not received the attention it deserves. Closely related to emotion is sentiment, and understanding of one helps understand the other. In this work, we firstly create a new multi-modal, emotion-TA ('TA' means tweet act, i.e., speech act in Twitter) dataset called EmoTA collected from open-source Twitter dataset. We propose a Dyadic Attention Mechanism (DAM) based multi-modal, adversarial multi-tasking framework. DAM incorporates intra-modal and inter-modal attention to fuse multiple modalities and learns generalized features across all the tasks. Experimental results indicate that the proposed framework boosts the performance of the primary task, i.e., TA classification (TAC) by benefitting from the two secondary tasks, i.e., Sentiment and Emotion Analysis compared to its uni-modal and single task TAC (tweet act classification) variants.
Identification of speech acts is one of the preliminary means of determining the communicative intent or pragmatics of a speaker (for example, statement, request, question etc.). This is true for dialogue system, speech transcription, social media such as Twitter, MySpace etc. Twitter is one of the leading micro-blogging services. By 2019, 330 million users were active monthly and 500 million tweets were sent per day<sup>1</sup>. Identification of tweet acts (TAs- speech acts in Twitter) is highly beneficial for Twitter as well as tweeters. For Twitter, it helps decipher a particular subject in terms
of speech acts and discrepancy identification. It also helps in social media monitoring by analysing topic alteration or spamming. It assists the followers in monitoring and scanning the subject with the most advantageous speech acts based on their needs. This helps reduce their search space and encourages them to obtain useful information from out of millions of tweets. It gives the tweeter a greater sense of the content, mood and trend.
A person's emotional state and sentiment greatly impacts its intended content (Barrett et al., 1993). Often sentiment and emotion are treated as two different problems (Do et al., 2019), (Soleymani et al., 2017), (Albanie et al., 2018), (Hossain and Muhammad, 2019), (Majumder et al., 2019). However, sentiment and emotion are closely related. For example, emotions such as happy and joy are inherently related to a positive sentiment. But emotion is much more nuanced and fine-grained compared to sentiment (Kumar et al., 2019). Emotion along with sentiment provides better understanding of the state of mind of the tweeter. For example, a question or statement is associated with anticipation. An opinion is many times associated with anger or disgust. The close association between emotion and sentiment motivates considering tweeter's sentiment along with emotion while deciphering the tweet acts. For expressive TAs such as "expression", "request", "threat" etc., the tweeter's sentiment and emotion can aid in classifying true communicative intent and vice-versa.
Additionally, multi-modal inputs, i.e., the combination of text and other nonverbal cues (emojis in tweets) (Felbo et al., 2017) help create reliable classification models aiding the identification of emotional state and sentiment of the tweeter which in turn help in determining correct TAs.
In this paper, we leverage the relationships as delineated above to predict TAs of tweets in a multimodal framework. In this multi-task framework, TAC is treated as the primary task and Sentiment
<sup>1</sup>https://www.omnicoreagency.com/twitter-statistics/
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5727-5737 June 6-11, 2021. ©2021 Association for Computational Linguistics
Analysis (SA) and Emotion Recognition (ER) as auxiliary (i.e., secondary) tasks.
Contributions of this paper are as follows: i. We create a new dataset called EmoTA consisting of tweets with high-quality annotations of TAs, including emotionally aided and multi-modal cues; ii. We establish the need for considering the sentiment and emotional state of the tweeter while identifying TAs. iii. We propose a Dyadic Attention Mechanism (DAM) based multi-task adversarial learning framework for multi-modal TAC, SA and ER. In DAM, we incorporate intra-modal and inter-modal attention to integrate information across multiple modalities and learn generalized features across multiple tasks; iv. We illustrate performance gains by jointly optimizing TAC, SA and ER. Multi-modal and multi-task TAC performs significantly better than its uni-modal and single task TAC variants.
There exist plenty of works which address the task of TAC as a standalone problem. In (Zhang et al., 2011), (Vosoughi and Roy, 2016), authors proposed Machine Learning based approaches for TAC namely Support Vector Machines (SVM), Logistic Regression etc. In (Saha et al., 2020a), authors proposed a first ever public dataset for the identification of speech acts in Twitter followed by a capsule based network built on top of BERT for TAC. In (Vosoughi, 2015), authors highlighted the importance of identification of tweet acts and established it to be one of the elementary steps for detection of rumours in Twitter. In (Saha et al., 2020c), authors proposed an attention based model built on top of the Transformer for predicting TAs. In (Saha et al., 2020a), authors proposed a capsule based network built on top of BERT for TAC. All these works utilized only the textual modality to identify TAs without any sentiment or emotional correlation of the tweeter. In (Cerisara et al., 2018), authors proposed a LSTM based study for jointly optimizing SA and TAC in a decentralized social media platform called Mastodon. However, they modelled their task as a multi-party conversation pretty different in essence to that of Twitter analysis. In (Jeong et al., 2009), authors presented a semi-supervised approach to identify speech acts in emails and different forums. These works, however, use datasets that comprise of face-to-face or telephone data that can not directly aid in advancing work on endless data in electronic mode such as
micro-blogging networks, instant-messaging, etc.
Apart from these, identification of speech acts has been studied extensively for dialogue conversations starting from early 2000's with (Stolcke et al., 2000) being one of the benchmark works where the authors presented varieties of approaches such as Hidden Markov Models, Neural Networks and Decision Trees to identify dialogue acts on a benchmark dialogue data known as the Switchboard (SWBD) (Godfrey et al., 1992) dataset. In (Saha et al., 2021), authors studied the role of emotion in identifying dialogue acts for a dyadic conversation by considering the textual and the audio modality of the utterances in the conversation. In (Saha et al., 2020b), authors proposed studying the role of emotion in determining dialogue acts on a dyadic and multi-party conversational dataset in a multi-modal framework (incorporating text, audio and video). However, tweets are unstructured and noisy communications with spelling mistakes, random coinages with limitations in expression because of character constraint per tweet. This makes it very different from face-to-face or other conversations.
Here, we discuss the details of the newly created dataset, EmoTA.
To begin with, we scanned the literature for the latest SA and ER dataset for Twitter in order to gather potentially emotionally rich tweets to explore its impact on TAC. Initially, we came across several SA and ER datasets for Twitter such as (Oleri and Karagoz, 2016), (Mohammad and Kiritchenko, 2018), SemEval-2018 (Mohammad et al., 2018), BTD (Wang et al., 2012), TEC (Mohammad, 2012), CBET (Shahraki and Zaiane, 2017), STS-Gold (Mohammad and Turney, 2013), STS (Go et al., 2009), SS-Twitter (Thelwall et al., 2012) etc. However, we chose to use SemEval-2018 dataset for further investigation of our task at hand. The reason behind this choice was that most of the ER datasets were annotated with only six Eckman's (Ekman, 1999) or eight Plutchik's (Plutchik, 1980) emotion categories. Whereas SemEval-2018 dataset contains tweets annotated with multi-label 11 emotion categories which aids the diversity of the problem statement. Intuitively, it was indeed possible to go the other way round and search for Twitter dataset annotated with TAs such as (Zhang et al., 2011),
Tweet	TA	Emotion	Sentiment
And it pisses me off more they killed people who surrendered. Hands up and all. If hands visible you shouldn’t be fearing for your life	exp	anger, disgust, fear	negative
We’re going to get City in the next round for a revenge	tht	anger	negative
Ryan Gosling and Eva Mendes finally; B joyful an funny/dont boss/dont argue/do everything with kids/go on mini car trips/ focus on love	sug	joy, love	positive
@MendipHillsAONB do we think the swallows and swifts have gone? Photo’d 3 nights ago, not seen since. #sad #Autumn	que	pessimism, sadness	negative
Table 1: Sample tweets from the EmoTA dataset with its corresponding true TA, Emotion and Sentiment labels
Figure 1: (a) Distribution of tweet act labels, (b) Distribution of emotion labels.
Figure 2: Distribution of sentiment labels
(Vosoughi and Roy, 2016), (Saha et al., 2020a) etc. However, the tweets in these datasets were devoid of nonverbal cues such as emojis which are quite excessively used in Twitter.
To the best of our knowledge, we were unaware of any sizable and open sourced Twitter dataset annotated for its TA and emotion labels. Hence, the SemEval-2018 dataset has been manually annotated for its TA categories. Unlike dialogic conversations, there isn't a standard TA tag-set available for annotating tweets. However, we made use of 7 TA categories of (Saha et al., 2020a) for annotating SemEval-2018 dataset as opposed to 5 and 6 TA categories of (Zhang et al., 2011) and (Vosoughi and Roy, 2016), respectively. The 7 TA tags are "Statement" (sta), "Expression" (exp), "Question" (que), "Request" (req), "Suggestion" (sug), "Threat" (tht) and "Others" (oth). For the current work, we selected a subset of SemEval-2018 dataset amounting to 6810 tweets to create EmoTA dataset. Three annotators who were graduate in English linguistics were accredited to annotate the tweets with
the appropriate TA tags. They were asked to annotate these tweets individually by only viewing the tweet available without the information of the pre-annotated emotion tags. This was done so as to assure that the dataset does not get biased by specific TA-emotion pairs. The conflicting annotations were resolved through discussions and mutual agreements. The inter-annotator score over  $80\%$  was considered as reliable agreement. It was determined based on the count that for a given tweet more than two annotators agreed on a particular tag.
For annotating the dataset with sentiment labels, we followed a semi-supervised approach instead of manual annotation which is cost intensive. We used the IBM Watson Sentiment Classifier $^2$ , an open-sourced API readily available for obtaining silver standard sentiment label of the tweets categorized into 3 tags namely "Positive", "Negative" and "Neutral".
3.3 Emotion-Tweet Act Dataset : EmoTA
The EmoTA dataset<sup>3</sup> now comprises of 6810 tweets with the corresponding gold standard TA and multi-label emotion tags. Each of the tweet contains its Tweet ID and two modalities: text and emoji. Few sample tweets along with the corresponding TA, sentiment and emotion labels from the proposed dataset are shown in Table 1. Distributions of TA,
$^{2}$ https://cloud.ibm.com/apidocs/natural-language-understanding#sentiment  
 $^{3}$ The dataset with its TA and emotion tags will be made publicly available to the research community.
1. All the young people are so bitter about how the older contestants probably know how to make Bakewell Tarts
Text: anger, disgust, sadness
- Emoji: joy
2. @asjoshtaylor sadly I don't think I'll see you on tour, but have fun, you're gonna rock!
Text :sadness
- Emoji: joy
Figure 3: (a) Importance of emoji in analysis of tweets, (b) Importance of emotion and sentiment in TAC.
1. Do you think humans have the sense for recognizing impending doom?
TA : question
2. Be happy not because everything is good, but because you can see the good side of everything #optimism
TA : suggestion
anticipation, pessimism
joy, optimism
emotion and sentiment labels across the dataset are shown in Figure 1a, 1b and 2, respectively.
Below, we analyze using some samples from the dataset that require sentiment-emotion aided and multi-modal reasoning.
Role of Sentiment and Emotion. In Figure 3b, we demonstrate using two examples from the dataset to establish our hypothesis that sentiment and emotional states of the tweeter can aid the identification of TAs. In the first instance, the tweeter questions about the impending doom supposedly because of a pessimistic expectation arising due to the negative sentiment. Similarly, in the second instance, because of a joyous emotion emerging due to positive sentiment, the tweeter shares an optimistic suggestion with the readers. The above examples highlight the need for incorporating these additional user behavior, i.e., sentiment and emotion while reasoning about TAs. Thus, stressing the requirement of addressing such synergy amongst TAC, SA and ER.
Role of Multi-modality. In Figure 3a, we present two examples from the dataset to highlight the importance of including other nonverbal features such as emoji present in the tweet along with the text for several tweet analysis tasks. In the first example tweet, the text represents an overall negative sentiment with emotion such as anger and disgust. However, the presence of an emoji face with tears of joy gives it an emotion of joy along with the other emotions. Similarly, in the second example tweet, the text represents the emotional state of the tweeter as sad, whereas the ok, celebration and heart emojis depict the feeling of joy. These instances show that the presence of complementary information in the form of emojis aids the process of any twitter analysis task including TAC.
The proposed multi-tasking, multi-modal approach and implementation details are outlined in this section.
The procedure for feature extraction across multiple modalities is discussed below.
Textual Features. To extract textual features of a tweet  $U$  having  $n_u$  number of words, the representation of each of the words,  $w_{1},\ldots ,w_{u}$  where  $w_{i}\in \mathbb{R}^{d_{u}}$  and  $w_{i}$ 's are obtained from BERT (Devlin et al., 2019) which is a multi-layered attention aided bidirectional Transformer Encoder model based on the original Transformer model (Vaswani et al., 2017) where  $d_{u} = 768$
emoji Features. To extract emoji features from a tweet, we use `emoji`, a python based library for eliciting the pictorial image of an emoji (primarily that of a face, object or symbols). A total of 1816 kind of emojis are available along with its different types. We then use `emoji2vec` (Eisner et al., 2016), which provides  $d_v = 300$  dimensional vector representation for each of the emojis present in the tweet. Let's say a tweet contains  $n_v$  number of emojis. Thus, we obtain the final emoji representation  $V$  for a tweet as  $V \in \mathbb{R}^{n_v \times d_v}$ .
The proposed network consists of four main components: (i) Modality Encoders (ME) produces respective modality encodings by taking as input the uni-modal features extracted above, (ii) Dyadic Attention Mechanism (DAM) that comprises dual attention mechanisms such as intra-modal and intermodal attentions, (iii) Adversarial Loss to make the feature spaces of task-specific and shared layers of each task mutually exclusive, (iv) Classification Layer that contains output channels for the three tasks at hand (TAC, SA and ER) to learn generalized representations across all the tasks.
Figure 4: The architectural diagram of the proposed network
In this section we discuss how the architectural framework encodes different modalities.
Text and Emoji Modalities. The features  $U$  and  $V$  obtained from each of the modalities corresponding to a tweet (discussed above) are then passed through two discrete Bi-directional LSTMs (BiLSTMs) (Hochreiter and Schmidhuber, 1997) to sequentially encode these representations and learn complementary semantic dependency based features into hidden states from these modalities. In case of textual modality (say), the final hidden state matrix of a tweet is obtained as  $H_{u} \in \mathbb{R}^{n_{u} \times 2d_{l}}$ .  $d_{l}$  represents the number of hidden units in each LSTM and  $n_{u}$  is the sequence length. In the similar way, a representation of corresponding emoji modality encoding as  $H_{v} \in \mathbb{R}^{n_{v} \times 2d_{l}}$  is obtained. The number of representations from modality encoders vary depending on the variant of the multitask learning framework used (e.g., fully shared (FS) or shared-private model (SP)). In a FS variant, two representations are obtained one for text and another for emoji cumulatively for optimizing all the three tasks. However, for a SP model, six encoding representations are obtained. Three for text and the remaining for emoji forming a pair of text-emoji representations for each of the three tasks.
4.2.2 Dyadic Attention Mechanism
We use a similar concept as in (Vaswani et al., 2017), where the authors proposed to compute
attention as mapping a query and a set of key-value pairs to an output. So, the representations obtained from the modality encoders above are passed through three fully-connected layers each termed as queries and keys of dimension  $d_{k} = d_{f}$  and values of dimension  $d_{v} = d_{f}$ . For a FS model, we have two triplets of  $(Q,K,V)$  as:  $(Q_{u},K_{u},V_{u})$  and  $(Q_{v},K_{v},V_{v})$ . Similarly for a SP model, we have six such triplets as:  $(Q_{u1},K_{u1},V_{u1})$ ,  $(Q_{v1},K_{v1},V_{v1})$ ,  $(Q_{u2},K_{u2},V_{u2})$ ,  $(Q_{v2},K_{v2},V_{v2})$ ,  $(Q_{u3},K_{u3},V_{u3})$ ,  $(Q_{v3},K_{v3},V_{v3})$  where pair of two triplets are from the textual and emoji modality encoders for each of the tasks<sup>4</sup>. These triplets are then used to compute attention values for different purposes in various combinations which include intra attention and inter-modal attention.
Intra-modal Attention. We compute intramodal attention (IA) for all these individual modalities in order to learn the interdependence between the current words and the preceding part of the tweet. In a way, we aim to relate different positions of a single sequence to estimate a final representation of the same sequence for individual modalities (Vaswani et al., 2017). Thus, the IA scores for individual modalities are calculated as:
$$
I A _ {j} = \operatorname {s o f t m a x} \left(Q _ {j} K _ {j} ^ {T}\right) V _ {j} \tag {1}
$$
4Subscript 1, 2 and 3 represent TAC, ER and SA task, respectively.
where  $IA \in \mathbb{R}^{n_u \times d_f}$  for  $IA_u, IA \in \mathbb{R}^{n_v \times d_f}$  for  $IA_v$  for FS model and six such  $IA$  scores for SP model.
Inter-modal Attention. The IA scores obtained above are then used to compute inter-modal attention (IRA). We re-iterate the same process (explained above) to now form triplets of  $(Q, K, V)$  for these IA scores and then compute IRA scores amongst triplets of all IA scores by computing the matrix multiplication of combination of queries and keys of different IA modality scores using Equation 1. In this manner, we obtain one IRA score as  $IRA_{uv} \in \mathbb{R}^{n_u \times d_f}$  for FS variant and three IRA scores for SP model as  $IRA_{uv1}$ ,  $IRA_{uv2}$  and  $IRA_{uv3}$ . This is done to distinguish important contributions between various modalities to achieve optimal representation of a tweet.
Attention Fusion. Next, we concatenate each of these computed  $IA$  and  $IRA$  vectors as:
$$
C = \operatorname {c o n c a t} \left(I R A _ {u v}, I A _ {u}, I A _ {v}\right), \text {f o r F S} \tag {2}
$$
$$
C _ {1} = \operatorname {c o n c a t} \left(I R A _ {u v 1}, I A _ {u 1}, I A _ {v 1}\right), \text {f o r S P} \tag {3}
$$
$$
C _ {2} = \operatorname {c o n c a t} \left(I R A _ {u v 2}, I A _ {u 2}, I A _ {v 2}\right), \text {f o r S P (4)}
$$
$$
C _ {3} = \operatorname {c o n c a t} \left(I R A _ {u v 3}, I A _ {u 3}, I A _ {v 3}\right), \text {f o r S P (5)}
$$
Next, we obtain mean of these three different concatenated attention vectors for the SP variant or directly use the obtained  $C$  attention vector for the FS variant to obtain the final representation of a tweet.
$$
M = \operatorname {m e a n} \left(C _ {1}, C _ {2}, C _ {3}\right) \tag {6}
$$
Shared Layer. Additionally, for the SP model, other than having task-specific layers, we allow a shared layer to learn task invariant features. Here, the shared layer is in the form of a fully-connected layer of dimension  $d_f$ . The inputs to the shared layer are the hidden representations of three IRA vectors:  $IRA_{uv1}$ ,  $IRA_{uv2}$  and  $IRA_{uv3}$ . Thus for a given tweet, the loss of the shared layer is minimized if the model correctly classifies the tasks of each of the tweets in the input. This helps learn domain invariant feature space for different tasks.
Adversarial Loss. The goal of this adversarial loss function is to tune the weights of the shared layer so that it learns a representation that misleads the task discriminator. The adversarial loss  $l_{adv}$ , aims to make the feature space of shared and task-specific layers to be mutually exclusive (Liu et al., 2017). We follow the similar strategy as that of (Liu et al., 2017), where a task discriminator  $D$  (say) maps the shared feature to its original task. Thus, on a correct prediction when the loss at the shared
layer decreases, the adversarial loss increases and vice-versa. Alternatively, the shared layer is tuned to work in an adversarial way, thereby prohibiting the discriminator to predict one of the three tasks. The adversarial loss is computed as:
$$
l _ {a d v} = \min  _ {F} \left(\max  _ {D} \left(\sum_ {n = 1} ^ {N} \sum_ {k = 1} ^ {K} d _ {k} ^ {n} \log \left[ D \left(F \left(x _ {k} ^ {n}\right)\right) \right]\right)\right) \tag {7}
$$
where  $d_k^n$  represents the true label amongst the type of the tasks,  $N$ , and  $x_{k}^{n}$  is the  $k^{th}$  example for task  $n$ . The min-max optimization problem is addressed by the gradient reversal layer (Ganin and Lempitsky, 2015).
The final representation of the tweet obtained from the DAM module is shared across three channels pertaining to the three tasks, i.e., TAC, SA and ER (for FS model) and three DAM representations for three individual tasks are subjected to individual output layer (for SP model). The task-specific loss  $(l_{t})$ , shared loss  $(l_{s})$  and adversarial loss  $(l_{adv})$  are used as
$$
l _ {f} = l _ {t} + \alpha l _ {s} + \gamma l _ {a d v}, \text {f o r S P m o d e l} \tag {8}
$$
$$
l _ {f} = l _ {s} + \gamma l _ {a d v}, \text {f o r F S m o d e l} \tag {9}
$$
where  $\alpha$  and  $\gamma$  are hyper-parameters.
4.3 Experimentation Details
Hyper-parameters.  $80\%$  of the tweets of the EmoTA dataset were used for training and the remaining  $20\%$  were used for testing the models. The same training and testing data were used for all the experiments in order to ensure fair comparison of models. To encode different modalities, a Bi-LSTM layer with 100 memory cells was used. Dense layers of dimensions 100 were used for  $d_{f}$ . The three channels contain 7, 3 and 11 output neurons, for TA, sentiment and emotion tags, respectively. Categorical crossentropy loss is used for TA and sentiment channels and Binary crossentropy loss function is used for emotion channel. A learning rate of 0.01 and Adam optimizer were used in the final experimental setting. All these values of the parameters were selected after a careful sensitivity analysis.