SlowGuess/NAACL_2021_OCR huggingface dataset:
Pre-processing. We employ NLTK based TweetTokenizer to tokenize tweets. URLs were removed. User mentions were replaced by <user> token. Numbers occurring in the tweet were replaced by
Model	TAC + SA	TAC + ER	TAC + SA + ER
Five-Class	Seven-Class	Five-Class	Seven-Class	Five-Class	Seven-Class
Text	Text+Emoji	Text	Text+Emoji	Text	Text+Emoji	Text	Text+Emoji	Text	Text+Emoji	Text	Text+Emoji
Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
FS	72.06	69.87	74.73	72.02	62.25	59.66	66.85	64.35	73.72	71.05	76.60	74.32	63.58	61.00	68.73	66.20	78.01	75.85	78.16	76.01	71.29	68.85	75.62	73.20
FS + Adv	73.92	71.05	75.61	73.32	63.67	61.27	69.54	67.03	75.57	73.05	77.35	75.00	65.11	62.80	71.24	69.02	80.01	77.59	81.34	79.08	72.90	70.51	76.21	73.95
SP	73.41	70.91	76.81	74.52	62.71	60.25	67.62	65.28	75.05	72.85	77.12	74.93	64.63	62.35	69.30	67.02	78.41	76.00	80.68	78.28	72.02	69.90	76.50	74.33
SP + Adv(without DAM)	74.73	72.06	75.86	73.33	64.13	61.75	70.32	68.04	76.11	73.80	77.57	75.20	65.80	63.16	71.86	69.60	80.32	78.00	81.49	79.14	73.24	70.90	77.60	75.28
SP + Adv(Glove)	73.82	71.22	77.27	75.00	66.71	64.46	69.94	67.61	75.61	73.28	78.42	76.05	68.81	66.36	72.26	69.83	79.35	77.15	81.79	79.46	73.31	70.90	78.17	76.00
SP + Adv(only IA)	76.21	73.85	78.62	76.35	69.73	67.30	71.75	69.50	77.64	75.21	80.68	78.37	71.07	68.95	73.05	71.00	81.64	79.27	83.04	81.16	75.62	73.35	79.95	77.62
SP + Adv(only IRA)	-	-	78.75	76.30	-	-	72.17	70.05	-	-	80.82	78.55	-	-	73.59	71.29	-	-	83.49	81.15	-	-	80.10	78.02
SP + Adv(with DAM)	76.21	73.85	79.37†	77.01	69.73	67.30	72.90†	70.63	77.64	75.21	80.97†	78.70	71.07	68.95	74.08†	72.00	81.64	79.27	84.08†	81.85	75.62	73.35	80.32†	78.16
Table 2: Results of all the baselines and the proposed multi-task models in terms of accuracy and weighted F1-score. † indicates that the reported results are statistically significant
Model	Single Task TAC
Five-Class	Seven-Class
Text	Text+Emoji	Text	Text+Emoji
Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
FS (without DAM)	70.31	68.60	72.60	70.25	59.86	57.20	64.21	62.10
FS (only IA)	72.61	70.30	73.35	71.17	64.02	62.90	67.16	65.00
FS (only IRA)	-	-	73.74	71.26	-	-	67.70	65.28
FS (with DAM)	72.61	70.30	74.73	72.15	64.02	62.90	68.57	66.15
FS (Glove)	71.05	68.74	72.23	69.90	60.68	58.62	66.17	64.00
FS (emoji as text)	74.09	71.91	-	-	66.66	64.12	-	-
FS (with DAM) (sentiment as feature)	74.16	71.75	76.35	74.20	66.24	64.15	69.73	67.25
FS (with DAM) (emotion as feature)	75.35	73.34	78.50	75.82	68.03	65.83	71.46	69.19
FS (with DAM) (sentiment & emotion as features)	77.42	74.81	79.05	76.74	68.83	66.12	72.21	70.03
<number> token. Ekphrasis (Baziotis et al., 2017) was used to extract hashtags by segmenting long string into its constituent words. All the characters of the tweet were lower-cased. Since the dataset is under-represented for most of the TA tags, we over-sample  $80\%$  of the tweets used for training as: the mediocrly represented tags (e.g., sug, que and oth) are over-sampled to be equally represented as the most represented tags (e.g., sta and exp). Similarly, the highly under-represented classes (e.g., req and tht) are over-sampled to be equally represented as the mediocrly represented tags in the EmoTA dataset. All the results reported below are on the  $20\%$  test data without any over-sampling.
A series of experiments were conducted for evaluating the proposed approach. Experiments were conducted for single task and several combinations of multi-task framework with TAC being the pivotal task along with varying modalities. A thorough ablation study is performed to analyze the importance of each of the attention mechanisms of the proposed architectural framework along with several variations of multi-task learning (e.g., FS, SP etc.). Note that we aim to enhance the performance
Figure 5: The visualization of the learned weights for a tweet from  $IA_{u}$  layer-  $u_{1}$ : "I lost a couple niggas I want revenge so put me in coach." for single task TAC (baseline), multi-task TAC+SA+ER (proposed) models
Table 3: Results of the single task TAC models in varying combinations
Model	SA & ER
Five-Class	Seven-Class
Text	Text+Emoji	Text	Text+Emoji
Acc.	F1	Acc.	F1	Acc.	F1	Acc.	F1
Single Task SA	87.26	86.05	88.52	87.20	88.85	87.30	90.10	89.00
Single Task ER	81.57	60.77	84.63	64.32	80.07	73.51	81.58	76.63
SA + TAC	89.31	87.85	90.74	89.09	89.60	88.75	91.55	90.35
ER + TAC	83.52	65.86	86.09	67.02	81.37	75.21	84.21	78.30
SA + ER(for SA)	92.30	91.06	93.02	92.00	90.33	88.65	92.73	90.37
SA + ER(for ER)	84.61	68.77	87.37	70.19	82.72	70.00	85.30	72.04
SA + ER + TAC(for SA)	92.06	91.13	93.19	92.38	92.49	90.53	93.68	91.82
SA + ER + TAC(for ER)	85.39	70.04	88.31	72.77	83.26	79.66	86.01	81.05
Table 4: Results of the proposed model for the single and multi-task SA and ER
of TAC with the help of other two auxiliary tasks. Following this, we report results and analysis with TAC strictly being the pivotal task in all the task combinations. Since, the dataset is unbalanced for all the task categories, we report results for different dimensions of TAC in the following set-up:
- Five-class Classification : This includes the top 5 highly occurring TA tags namely sta, exp, que, sug and oth.
- Seven-class Classification : This includes all the 7 categories of TAs used in the annotation process.
Table 3 and 2 illustrate the results of single task TAC and varying combinations of multi-task proposed models for different set-up (as mentioned
Tweet	True	TAC	TAC+SA	TAC+ER	TAC+SA+ER
@BreezyWeekes hey breezy, you wanna give me some of that coffee you posted on your snap?? please	req	exp	req	req	req
We're going to get City in the next round for a revenge	tht	sta	exp	tht	tht
@voguemagazine, did you not learn from @FreePeople 's viral insult to ballet? Stop trying to wrongfully stick models into pointe shoes	sug	que	exp	exp	sug
I wonder if Corey will vote for Nicole?? #snacole #bb18 #paulsgonnawin #finale #halfamill	que	que	que	que	que
Table 5: Sample tweets from the EmoTA dataset with its corresponding ground truth and predicted labels for different single and multi-task models
Model	Acc.	F1
JointDAS (TAC + SA) (Cerisara et al., 2018)	59.05	57.60
CNN-SVM (TAC) (Saha et al., 2019)	61.32	59.75
Transformer (TAC) (Saha et al., 2020c)	65.46	63.65
Bert-Caps (TAC) (Saha et al., 2020a)	67.10	65.00
Proposed (TAC)	68.57	66.15
Table 6: Comparative Analysis with the state of the art models
above). As evident, the addition of non-verbal cues in the form of emojis improves the uni-modal textual baseline consistently. This improvement implies that the proposed architecture utilizes the interaction among the input modalities very effectively. This highlights the importance of incorporating multi-modal features for different Twitter analysis tasks. We also report result for utilizing emoji as textual feature instead of treating it as a different modality in the single task TAC framework. Also, the five-class set-up gave better results than the seven-class set up. This is pretty obvious, as with 5-class set-up, the model needs to distinguish and identify lesser fine-grained features compared to the 7-class set-up. Additionally, the under-representation of two tags in the EmoTA dataset for the 7-class set-up also effects its performance.
As seen in Table 2, the multi-task framework with all the three tasks (i.e., TAC + SA + ER) consistently gave better results as compared to single task TAC. In the bi-task variant, TAC+SA, shows little improvement in different metrics as opposed to TAC+ER over and above the single task TAC. This gain is rather intuitive as sentiment alone is sometimes unable to convey complete information of the tweeter's state of mind. E.g., a negative sentiment can occur because of various emotions such as disgust, fear, sadness etc. Similarly, a positive sentiment can take place because of emotions such as happiness, surprise etc. Thus, with sentiment alone, sometimes this discreteness or fine differences in the state of mind cannot be completely determined and conveyed. To illustrate this, in Figure 5, we provide a visualization of the learned weights of a tweet from the  $IA_{u}$  layer (as this layer contains word-wise attention scores). For this particular tweet, its true TA label is  $tht$ . With the multi-task framework, the importance of warning
bearing words are learnt well such as lost, revenge compared to the single-task TAC where attention is laid on expression bearing word such as put me. Additionally, we also report results for cases where sentiment and emotion were directly used as features in the single task TAC models to leverage from instead of deploying a multi-task based approach in Table 3.
As stated above, we treat SA and ER as auxiliary tasks aiding the primary task, i.e., TAC. However, we report the performance of SA and ER tasks on the proposed model for single as well as multi-task frameworks in Table 4 for further investigations. However, we do not make any explicit effort to enhance their performance.
Comparison amongst Different Multi-task Architecture. In terms of varying ways of multitasking such as FS, SP along with adversarial loss (adv), it was observed that SP model gave better results compared to FS model. Additionally, incorporating adversarial loss further boosted the performance of different multi-task models. Intuitively, as TAC shares lesser amount of correlation with SA and ER compared to SA and ER themselves, FS model was not sufficient enough to learn diverse features across different tasks. This observation is in conformity with the existing literature. We also demonstrate the importance of different attentions used for the best performing multi-task model, i.e.,  $\mathrm{SP + Adv}$ . Furthermore, we also report results by replacing BERT model to extract textual representation with Glove embeddings (Pennington et al., 2014). Results indicate that each of these aspects contributed significantly to aid the performance of the proposed multi-tasking framework. All the reported results here are statistically significant (Welch, 1947).
Comparison with the State of the Art Models. We also compare our proposed approach with the recent state of the art models for single task TAC as we are unaware of any other work which jointly optimized tweet act, emotion and sentiment in Twitter. In Table 6, we report the results for the same by re-implementing those on the EmoTA dataset. As
evident, the proposed model outperformed these SOTA approaches.
Error Analysis. An in-depth analysis revealed several scenarios as to why the proposed model faltered which are as follows: i. Imbalanced Dataset : As visible in Figure 1a, except for "sta" and "exp" tags, all the classes are under-represented in the EmoTA dataset. Even though we apply oversampling to partially counter this issue but still the tags such as "req" and "tht" contain very little tweets for the model to learn fine differences amongst different categories. In accordance with this, we observe that five-class performs exceptionally better than the seven-class classification set-up; ii. Fine-grained tags : It was also observed that the tweets which were mis-classified were subset of each other. For instance, tweet such as "don't get discouraged! it's early on; it can get overwhelming. keep reading; use cue cards it'll get better!!" is wrongly predicted as "exp" rather than "sug" which in the superficial way is a subset of the former tag; iii. Miscellaneous : Tweets belonging to "oth" tag was also majorly mis-classified as there was no fixed pattern of tweets belonging to this category. To counter this, even more fine-grained categories of TAs needs to be identified and modelled. Sample utterances for the error analysis are shown in Table 5.
In this paper, we studied the role of sentiment and emotion in speech act classification in Twitter. We curate a novel dataset EmoTA, that contains pre-annotated tweets with emotions collected from open-source dataset and annotated with TAs and sentiment categories. We propose a Dyadic Attention Mechanism based multi-modal (emojis and text), adversarial multi-task framework for joint optimization of TAs, sentiment and emotions. The DAM (dyadic attention mechanism) module employs intra-modal and inter-modal attention to fuse multiple modalities and learn generalized features across all the tasks. Results show that multi-modality and multi-tasking boosted the performance of TA identification compared to its uni-modal and single task TAC variants.
In future, attempts will be made to predict TAs with more precision by incorporating fine-grained modality encodings and also identifying which other NLP tasks (e.g., named entity recognition) might assist TAC as a task.
Dr. Sriparna Saha gratefully acknowledges the Young Faculty Research Fellowship (YFRF) Award, supported by Visvesvaraya Ph.D. Scheme for Electronics and IT, Ministry of Electronics and Information Technology (MeitY), Government of India, being implemented by Digital India Corporation (formerly Media Lab Asia) for carrying out this research.
Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. 2018. Emotion recognition in speech using cross-modal transfer in the wild. In Proceedings of the 26th ACM international conference on Multimedia, pages 292-301.
Lisa Feldman. Barrett, Michael Lewis, and Jeannette M. Haviland-Jones. 1993. Handbook of emotions. The Guilford Press.
Christos Baziotis, Nikos Pelekis, and Christos Doukkeridis. 2017. Datastories at semeval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747-754, Vancouver, Canada. Association for Computational Linguistics.
Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa Le. 2018. Multi-task dialog act and sentiment recognition on mastodon. arXiv preprint arXiv:1807.05013.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Hai Ha Do, PWC Prasad, Angelika Maag, and Abeer Alsadoon. 2019. Deep learning for aspect-based sentiment analysis: a comparative review. Expert Systems with Applications, 118:272-299.
Ben Eisner, Tim Roktaschel, Isabelle Augenstein, Matko Bosnjak, and Sebastian Riedel. 2016. emoji2vec: Learning emoji representations from their description. In Proceedings of The Fourth International Workshop on Natural Language Processing for Social Media, SocialNLP@EMNLP 2016, Austin, TX, USA, November 1, 2016, pages 48-54. Association for Computational Linguistics.
Paul Ekman. 1999. Basic emotions. Handbook of cognition and emotion, 98(45-60):16.
Bjarke Felbo, Alan Mislove, Anders Søgaard, Iyad Rahwan, and Sune Lehmann. 2017. Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 1615-1625. Association for Computational Linguistics.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR.
Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter sentiment classification using distant supervision. CS224N project report, Stanford, 1(12):2009.
John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, volume 1, pages 517-520. IEEE.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
M Shamim Hossain and Ghulam Muhammad. 2019. Emotion recognition using deep learning approach from audio-visual emotional big data. Information Fusion, 49:69-78.
Minwoo Jeong, Chin-Yew Lin, and Gary Geunbae Lee. 2009. Semi-supervised speech act recognition in emails and forums. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3-Volume 3, pages 1250-1259. Association for Computational Linguistics.
Abhishek Kumar, Asif Ekbal, Daisuke Kawahra, and Sadao Kurohashi. 2019. Emotion helps sentiment: A multi-task model for sentiment and emotion analysis. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 1-10. Association for Computational Linguistics.
Navonil Majumder, Soujanya Poria, Devamanyu Hazarika, Rada Mihalcea, Alexander Gelbukh, and Erik Cambria. 2019. Dialoguernn: An attentive rnn for emotion detection in conversations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6818-6825.
Saif Mohammad. 2012. # emotional tweets. In * SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings
of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 246-255.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 task 1: Affect in tweets. In Proceedings of the 12th international workshop on semantic evaluation, pages 1-17.
Saif Mohammad and Svetlana Kiritchenko. 2018. Understanding emotions: A dataset of tweets to study interactions between affect categories. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436-465.
Obrahim Oleri and Pinar Karagoz. 2016. Detecting user emotions in twitter through collective classification. In KDIR, pages 205-212.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In *Theories of emotion*, pages 3-33. Elsevier.
Tulika Saha, Dhawal Gupta, Sriparna Saha, and Pushpak Bhattacharyya. 2021. Emotion aided dialogue act classification for task-independent conversations in a multi-modal framework. Cognitive Computation, 13(2):277-289.
Tulika Saha, Srivatsa Ramesh Jayashree, Sriparna Saha, and Pushpak Bhattacharyya. 2020a. Bert-caps: A transformer-based capsule network for tweet act classification. IEEE Transactions on Computational Social Systems, 7(5):1168-1179.
Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020b. Towards emotion-aided multi-modal dialogue act classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4361-4372.
Tulika Saha, Aditya Prakash Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020c. A transformer based approach for identification of tweet acts. In 2020 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Tulika Saha, Sriparna Saha, and Pushpak Bhattacharyya. 2019. Tweet act classification: A deep learning based classifier for recognizing speech acts in twitter. In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE.
Ameneh Gholipour Shahraki and Osmar R Zaiane. 2017. Lexical and learning-based emotion mining from text. In Proceedings of the International Conference on Computational Linguistics and Intelligent Text Processing, volume 9, pages 24-55.
Mohammad Soleymani, David Garcia, Brendan Jou, Björn Schuller, Shih-Fu Chang, and Maja Pantic. 2017. A survey of multimodal sentiment analysis. Image and Vision Computing, 65:3-14.
Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteor. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339-373.
Mike Thelwall, Kevan Buckley, and Georgios Paltoglou. 2012. Sentiment strength detection for the social web. Journal of the American Society for Information Science and Technology, 63(1):163-173.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pages 5998-6008.
Soroush Vosoughi. 2015. Automatic detection and verification of rumors on Twitter. Ph.D. thesis, Massachusetts Institute of Technology.
Soroush Vosoughi and Deb Roy. 2016. Tweet acts: A speech act classifier for twitter. In Tenth International AAAI Conference on Web and Social Media.
Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P Sheth. 2012. Harnessing twitter" big data" for automatic emotion identification. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 587-592. IEEE.
Bernard L Welch. 1947. The generalization of student's' problem when several different population variances are involved. Biometrika, 34(1/2):28-35.
Renxian Zhang, Dehong Gao, and Wenjie Li. 2011. What are tweeters doing: Recognizing speech acts in twitter. Analyzing Microtext, 11(05).