SlowGuess/ACL_2020_OCR huggingface dataset:
Pre-trained Language Models. We fine-tune base versions of BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) using the Hugging-Face Transformers library (Wolf et al., 2019). We
<sup>6</sup>We also experimented with keeping all negative samples as opposed to sampling an equal amount. Each binary task had around 5-7x more negative samples; this significantly hurt model performance. Even with a class imbalance penalty, the models almost never predicted positive samples. Note that although, in aggregate, the number of positive and negative samples match, they do not necessarily match in the train, validation, and test splits.
5295
AGR	OPT	LOV	SBM	AWE	DSP	RMR	CNT	AVG
Logistic Reg.	49.8	74.7	50.9	50.6	48.9	49.7	48.3	46.8	52.5
Char CNN	50.2	74.3	43.0	47.2	44.7	47.1	47.4	48.8	50.3
Word CNN	43.6	74.5	44.7	45.4	44.2	47.0	46.9	43.9	48.8
GRU	48.4	74.7	54.0	50.9	50.1	49.9	48.9	49.2	53.3
BERT	67.6	75.0	54.0	67.4	68.3	55.7	58.5	66.8	64.1
RoBERTa	59.7	74.7	54.0	62.3	56.0	50.9	49.7	56.4	58.0
Table 5: Plutchik-8 binary task accuracies, including aggressiveness (agr), optimism (opt), love (lov), submission (sbm), awe (awe), disapproval (dsp), remorse (rmr), contempt (cnt). We also report an average (avg) across all binary tasks. Best results are bolded.
use the sentence representations embedded in the [CLS] token, then project it with a weight matrix  $\mathbf{W} \in \mathbb{R}^{d \times 2}$ . The language model and classification parameters are jointly fine-tuned.
Table 5 presents our classification results. We make the following observations:
BERT consistently outperforms other models on most emotion tasks. BERT shows strong performance across all 8 binary tasks in comparison to traditional neural models and RoBERTa. Unlike most traditional neural models, its accuracy never falls below random chance, showing it captures at least some of the complex phenomena present in our dataset. However, our tasks remain challenging for both types of models alike. For traditional models, word embeddings alone do not provide enough representational power to model our emotional contexts. Although GRUs perform well on EMONET (Abdul-Mageed and Ungar, 2017), we suspect that they simply memorize emotion lexicons ( $\S 4.1$ ), which is not a notable strategy for capturing implicit emotions. Nevertheless, BERT only obtains an average accuracy of about  $64\%$ . This leaves plenty of room for future work; we perform a comprehensive error analysis as a step towards this goal ( $\S 5.3$ ).
"Better" pre-trained models (e.g., RoBERTa) do not necessarily help performance. Unlike popular benchmarks such as GLUE (Wang et al., 2018) where more pre-training monotonically increases performance, rather encouragingly, we do not observe the same trend. RoBERTa's average performance is around  $5\%$  better than GRU's, but still around  $6\%$  worse than BERT's. We hypothesize that this drop in performance is attributed to pre-training  $\rightarrow$  fine-tuning domain discrepancies. That is, RoBERTa's (additional) pre-training data (e.g., CC-News) may be too distant from Twitter
data, which is known for its short contexts and unique vernacular (Ritter et al., 2011). We encourage practitioners to avoid applying state-of-the-art models without augmenting them with task-guided pre-training objectives, as we explore later ( $\S 6$ ).
Using our BERT model, we sample 25 test errors from each of the 8 emotion tasks, yielding a total of 200 errors. We group the errors into the following categories: lexical and syntactic cues (45%), insufficient context (24%), entity mentions (15%), subjective labeling (10%), and unknown reasons (6%). The top three categories are discussed below:
Lexical and Syntactic Cues. BERT often relies on surface-level lexical features to make predictions, as do most emotion prediction models. This bias also extends to certain syntactic features, such as punctuation. In "pls be safe everyone!!!", BERT associates the exclamation mark with a positive emotion, but here, the speaker is more concerned.
Insufficient Context. Users often comment on events, public policies, or linked content that, by themselves, do not carry features for supervised learning. This type of error is not necessarily a shortcoming of BERT, but rather our dataset. For example, in "for [tracy mcgrady]1, [hall induction]2 muted by effects of [hurricane harvey]3 at home", one use external knowledge to reason between the noun phrases and discern the latent emotions.
Entity Mentions. BERT also makes erroneous predictions in the presence of certain entity mentions. For example, BERT classifies this tweet as AGGRESSIVENESS: "nytimesworld: mexico offered aid to texas after harvey. but after an earthquake and hurricane, it says all help is needed at home." Here, the user is merely quoting a
AGR	OPT	LOV	SBM	AWE	DSP	RMR	CNT	AVG
NO-PRETRAIN	67.6	75.0	54.0	67.4	68.3	55.7	58.5	66.8	64.1
Supervised Transfer
EMONET	73.5	75.2	55.2	68.8	67.5	53.1	60.0	71.7	65.6
SENTIMENT	72.8	75.8	62.7	71.0	65.6	53.4	57.0	67.3	65.7
Unsupervised Transfer
EMONET	72.1	75.1	54.0	61.0	65.1	54.2	60.7	69.4	63.9
SENTIMENT	69.1	74.9	53.6	66.2	67.3	54.3	57.9	64.4	63.5
HURRICANEEXT	73.6	75.4	69.8	68.9	69.7	57.9	60.2	70.2	68.2
Table 6: Task-guided pre-training accuracies (abbreviations defined in Table 5). Displayed in order of supervised (middle) and unsupervised (bottom) pre-training. Results are highlighted with blue  $(\uparrow)$  and red  $(\downarrow)$  with respect to NO-PRETRAIN. Best viewed in color.
news statement as opposed to formulating opinions regarding NY Times' discourse. Because the sentiment towards NY Times is negative in our datasets overall (due to public backlash on its stories), BERT likely capitalizes on this mention-emotion bias.
6 Task-Guided Pre-training
To improve upon our baselines, we explore pretraining as a means of implicitly incorporating an inductive bias into our BERT model. Our hope is that these pre-training tasks will not only make BERT more robust in the Twitter domain, but also provide useful (albeit abstract) knowledge for the end emotion prediction tasks. For brevity, we chiefly focus on BERT, although our methods can be generalized to other pre-trained models.
Setup. We explore, in isolation, supervised and unsupervised pre-training tasks. For the supervised setting, we pre-train on a multi-class emotion task (EMONET) (Abdul-Mageed and Ungar, 2017) and binary sentiment analysis task (SENTIMENT) (Go et al., 2009). For the unsupervised setting, we pretrain on dynamic masked language modeling (Liu et al., 2019) on (unlabeled) samples from EMONET, SENTIMENT, and HURRICANEEXT ( $\S 3.1$ ). For both types of tasks, we further pre-train BERT for a fixed number of epochs, then fine-tune it on a HURRICANEEMO task. We compare these results to NO-PRETRAIN, namely the BERT results verbatim from Table 5. We report mean performance across 10 pre-training  $\rightarrow$  fine-tuning runs. Further training details, including samples sizes for the pre-training tasks, are available in Appendix D.
Results. Table 6 shows the pre-training results. Supervised pre-training significantly helps with 3-
4 emotions, but degrades overall performance on 2-4 emotions. We posit SENTIMENT aids emotions with highly predictive features. For example, "wtf" in "it's literally the size of texas. wtf" is correlated with AGGRESSIVENESS, but no such lexical cues exist in "not all heros wear capes  $< 3$  thank you stanley - homeless #hurricane evacuee grooms lost pets," which is an AWE sample.
The unsupervised pre-training results also show a couple trends. First, EMONET largely hurts downstream performance, especially reducing SUBMISSION accuracy by  $-6\%$ . Second, SENTIMENT (in its unlabeled form) yields no noticeable benefits. This implies sentiment information is much more valuable, but of course, subject to the fact that the emotion task is heavily aligned with the original sentiment task. Third, we obtain encouraging results with HURRICANEEXT pre-training. The gains are most noticeable on AGGRESSIVENESS and LOVE, but this objective adds  $+1 - 2\%$  accuracy for tasks on which supervised pre-training suffered.
7 Fine-Grained Unsupervised Domain Adaptation
When new disasters emerge, it is likely we may not have emotion annotations, as alluded to previously (§2). Nevertheless, these annotations would be valuable for organizations trying to understand the emotional profile of users during a crisis (Fraustino et al., 2012). In this section, we explore ways to leverage supervision from large-scale emotion datasets (e.g., EMONET (Abdul-Mageed and Ungar, 2017)) in providing labels for our hurricane emotion datasets. We frame this problem as unsupervised domain adaptation; EMONET is the labeled source domain and our hurricane datasets are the unlabeled target domain. Below, we elaborate
AGR	OPT	LOV	SBM	AWE	DSP	RMR	CNT	AVG
SRC-ONLY	53.3	42.2	43.4	47.1	54.7	49.8	62.5	56.5	51.2
PRETRAIN-SRC	54.8	43.2	45.1	47.8	54.4	50.4	63.3	57.1	52.0
PRETRAIN-TRG	55.0	44.2	46.2	48.0	55.5	49.9	63.7	60.5	52.9
PRETRAIN-JOINT	52.7	44.2	45.5	47.8	54.8	49.9	61.6	56.3	51.6
TRG-ONLY	67.6	75.0	54.0	67.4	68.3	55.7	58.5	66.8	64.1
Table 7: Unsupervised domain adaptation accuracies (abbreviations defined in Table 5). Results are highlighted with blue (↑) and red (↓) with respect to SRC-ONLY. Best viewed in color.
on our methods.
Framework. EMONET was conceived as a multiclass classification task for Plutchik-8 emotions (Abdul-Mageed and Ungar, 2017). In contrast, we introduce binary classification tasks, one for each Plutchik-8 emotion. We split the EMONET multiclass task into 8 binary tasks; this creates a one-to-one alignment between each source and target domain task. We separately perform unsupervised domain adaptation for each binary task.
Methods. We use our BERT model (without task-guided pre-training) as the underlying classifier. Following Han and Eisenstein (2019), we chiefly focus on using strategic pre-training techniques that enable effective transfer between disparate domains. The systems for comparison are: (1) SRC-ONLY: BERT is trained in the source domain and evaluated in the target domain; (2) TRG-ONLY: BERT is trained and evaluated in the target domain. These results are borrowed verbatim from Table 5; (3) PRETRAIN-\*: BERT undergoes dynamic masked language modeling pre-training using data from domain  $*$ , is trained in the source domain, and finally evaluated in the target domain (Han and Eisenstein, 2019). PRETRAIN-SRC only uses pre-training samples from the source domain, PRETRAIN-TRG only uses samples from the target domain, and PRETRAIN-JOINT uses samples from both the source and target domains. We report mean performance across 10 pre-training  $\rightarrow$  fine-tuning runs.
Results. Table 7 shows the unsupervised domain adaptation results. Overall, we do not find a significant increase in performance over the SRC-ONLY baseline. Pre-training consistently adds  $+1\%$  in average accuracy, but still leaves a large gap between PRETRAIN-SRC and TRG-ONLY. Re
gardless, we have a few observations. First, we do not see a (relatively) large increase in performance for SUBMISSION, AWE, DISAPPROVAL, and REMORSE. These emotions may need more explicit strategies to enable domain adaptation. This is also supported by our previous results ( $\S 6$ ), where we also do not see a (relatively) large benefit from task-guided pre-training. Second, PRETRAIN-JOINT performs worse than both PRETRAIN-SRC and PRETRAIN-TRG. We posit that, for our emotion tasks, pre-training with a mixture of domains yields a noisier training signal compared to a parameter bias towards the target domain.
We present HURRICANEEMO, an annotated dataset of perceived emotions spanning 15,000 tweets from multiple hurricanes. Tweets are annotated with fine-grained Plutchik-24 emotions, from which we analyze implicit and explicit emotions and construct Plutchik-8 binary classification tasks. Comprehensive experiments demonstrate our dataset is a challenging benchmark, even for large-scale pre-trained language models. We release our code and datasets as a step towards facilitating research in disaster-centric domains.
Thanks to Katrin Erk for reviewing an early version of this manuscript, Yasumasa Onoe for discussions on masked language model pre-training, and the anonymous reviewers for their helpful comments. This work was partially supported by the NSF Grants IIS-1850153, IIS-1912887, and IIS-1903963.
Muhammad Abdul-Mageed and Lyle Ungar. 2017. EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks. In Proceedings
PRETRAIN-JOINT is conceptually similar to ADAPT-ABERT in Han and Eisenstein (2019), however, we dynamically generate pre-training data (Liu et al., 2019).
of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 718-728, Vancouver, Canada. Association for Computational Linguistics.
Firoj Alam, Shafiq Joty, and Muhammad Imran. 2018. Domain Adaptation with Adversarial Training and Graph Embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1077-1087, Melbourne, Australia. Association for Computational Linguistics.
Saima Aman and Stan Szpakowicz. 2007. Identifying Expressions of Emotion in Text". In Text, Speech and Dialogue, pages 196-205, Berlin, Heidelberg. Springer Berlin Heidelberg.
Ron Artstein and Massimo Poesio. 2008. Inter-coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555-596.
Jonathan Belles. 2017. 2017 Atlantic Hurricane Season Recap: 17 Moments We'll Never Forget. Weather.com.
Felipe Bravo-Marquez, Marcelo Mendoza, and Barbara Poblete. 2014. Meta-level Sentiment Models for Big Social Data Analysis. Knowledge-Based Systems, 69:86-99.
Cornelia Caragea, Adrian Silvescu, and Andrea H. Tapia. 2016. Identifying informative messages in disaster events using convolutional neural networks. In Proceedings of the 13th International Conference on Information Systems for Crisis Response and Management (ISCRAM).
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
Michael Robert Dennis, Adrienne Kunkel, Gillian Woods, and Paul Schrodt. 2006. Making Sense of New Orleans Flood Trauma Recovery: Ethics, Research Design, and Policy Considerations for Future Disasters. *Analyses of Social Issues and Public Policy*, 6(1):191-213.
Shrey Desai, Hongyuan Zhan, and Ahmed Aly. 2019. Evaluating Lottery Tickets under Distributional Shifts. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 153–162, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference
of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Paul Ekman. 1992. An Argument for Basic Emotions. Cognition and Emotion, 6(3-4):169-200.
Julia Daisy Fraustino, Brooke Fisher Liu, and Yan Xian Jin. 2012. Social Media Use During Disasters: A Review of the Knowledge Base and Gaps. National Consortium for the Study of Terrorism and Responses to Terrorism.
Charles Fritz and Eli Marks. 1954. The NORC Studies of Human Behavior in Disaster. Journal of Social Issues, 10(3):26-41.
Jessica Fritze, Grant Blashki, Susie Burke, and John Wiseman. 2008. Hope, Despair and Transformation: Climate Change and the Promotion of Mental Health and Wellbeing. International Journal of Mental Health Systems, 2(1):13.
Maria Giatsoglou, Manolis Vozalis, Konstantinos Diamantaras, Athena Vakali, George Sarigiannidis, and Konstantinos Chatzisavvas. 2017. Sentiment Analysis Leveraging Emotions and Word Embeddings. Expert Systems with Applications, 69:214-224.
Alec Go, Richa Bhayani, and Lei Huang. 2009. Twitter Sentiment Classification using Distant Supervision. Stanford University CS224N Project Report.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4237-4247, Hong Kong, China. Association for Computational Linguistics.
Muhammad Imran, Carlos Castillo, Fernando Diaz, and Sarah Vieweg. 2015. Processing Social Media Messages in Mass Emergency: A Survey. Association for Computing Machinery (ACM) Computing Surveys, 47(4):67:1-67:38.
Muhammad Imran, Shady Elbassuoni, Carlos Castillo, Fernando Diaz, and Patrick Meier. 2013. Practical Extraction of Disaster-relevant Information from Social Media. In Proceedings of the 22Nd International Conference on World Wide Web, WWW 2013 Companion, pages 1021-1024, New York, NY, USA. Association for Computing Machinery (ACM).
Muhammad Imran, Prasenjit Mitra, and Jaideep Srivastava. 2016. Cross-Language Domain Adaptation for Classifying Crisis-Related Short Messages. In 13th Proceedings of the International Conference on Information Systems for Crisis Response and Management, Rio de Janeiro, Brasil, May 22-25, 2016.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep Unordered Composition Rivals Syntactic Methods for Text Classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1681–1691, Beijing, China. Association for Computational Linguistics.
Phil Katz, Matthew Singleton, and Richard Wicentowski. 2007. SWAT-MP: The SemEval-2007 Systems for Task 5 and Task 14. In 4th International Workshop on Semantic Evaluations, pages 308-313.
Hamed Khanpour and Cornelia Caragea. 2018. Fine-grained emotion detection in health-related online posts. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1160-1166, Brussels, Belgium. Association for Computational Linguistics.
Yoon Kim. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Warren Kinston and Rachel Rosser. 1974. Disaster: Effects on Mental and Physical State. Journal of Psychosomatic Research, 18(6):437-456.
Hongmin Li, Doina Caragea, and Cornelia Caragea. 2017. Towards Practical Usage of a Domain Adaptation Algorithm in the Early Hours of a Disaster. In Proceedings of the 14th International Conference on Information Systems for Crisis Response and Management (ISCRAM).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Navonil Majumder, Soujanya Poria, Alexander Gelbukh, and Erik Cambria. 2017. Deep Learning-Based Document Modeling for Personality Detection from Text. IEEE Intelligent Systems, 32(2):74-79.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, NIPS 2013, pages 3111-3119, USA. Curran Associates Inc.
Saif Mohammad. 2012. #Emotional Tweets. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages
246-255, Montreal, Canada. Association for Computational Linguistics.
Saif Mohammad and Svetlana Kiritchenko. 2015. Using Hashtags to Capture Fine Emotion Categories from Tweets. Computational Intelligence, 31(2):301-326.
Saif Mohammad and Peter Turney. 2013. Crowdsourcing a Word-Emotion Association Lexicon. Computational Intelligence, 29(3):436-465.
Dat Nguyen, Kamela Ali Al Mannai, Shafiq Joty, Hassan Sajjad, Muhammad Imran, and Prasenjit Mitra. 2017. Robust Classification of Crisis-Related Data on Social Networks Using Convolutional Neural Networks. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM 2017).
Emily Ohman, Kaisla Kajava, Jörg Tiedemann, and Timo Honkela. 2018. Creating a Dataset for Multilingual Fine-grained Emotion-detection Using Gamification-based Annotation. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 24-30, Brussels, Belgium. Association for Computational Linguistics.
Leysia Palen and Kenneth Anderson. 2016. Crisis Informatics—New Data for Extraordinary Times. Science, 353(6296):224-225.
Rebecca Passonneau. 2004. Computing Reliability for Coreference Annotation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC 2004), Lisbon, Portugal. European Language Resources Association (ELRA).
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke