SlowGuess/ACL_2020_OCR huggingface dataset:
Shrey Desai<sup>1</sup> Cornelia Caragea<sup>2</sup> Junyi Jessy Li<sup>1</sup>
<sup>1</sup>The University of Texas at Austin <sup>2</sup>University of Illinois at Chicago {shreydesai@, jessy@austin.}utexas.edu cornelia@uic.edu
Natural disasters (e.g., hurricanes) affect millions of people each year, causing widespread destruction in their wake. People have recently taken to social media websites (e.g., Twitter) to share their sentiments and feelings with the larger community. Consequently, these platforms have become instrumental in understanding and perceiving emotions at scale. In this paper, we introduce HURRICANEEMO, an emotion dataset of 15,000 English tweets spanning three hurricanes: Harvey, Irma, and Maria. We present a comprehensive study of fine-grained emotions and propose classification tasks to discriminate between coarse-grained emotion groups. Our best BERT (Devlin et al., 2019) model, even after task-guided pre-training which leverages unlabeled Twitter data, achieves only  $68\%$  accuracy (averaged across all groups). HURRICANEEMO serves not only as a challenging benchmark for models but also as a valuable resource for analyzing emotions in disaster-centric domains.
Natural disasters cause thousands of deaths and displace hundreds of millions each year (Ritchie and Roser, 2020). These catastrophic events not only induce material destruction but also stir an integral part of being human: our emotions. Disasters adversely affect individuals' mental states (Fritz and Marks, 1954; Kinston and Rosser, 1974), and therefore it is no surprise that many take to social media (e.g., Twitter) to share their feelings. Social media websites, as a result, have become an essential platform for understanding the expression and perception of emotions at a significantly larger scale (Mohammad, 2012; Wang et al., 2012; Mohammad and Kiritchenko, 2015; Volkova and Bachrach, 2016; Abdul-Mageed and Ungar, 2017), with far reaching potential influences from academic re
search to public policy (Dennis et al., 2006; Fritze et al., 2008; Fraustino et al., 2012).
While natural language processing methods have been effective for emotion detection (Strapparava and Mihalcea, 2007), existing resources struggle in disaster-centric domains, in part due to distributional shifts. Emotion detection in natural disasters (e.g., hurricanes) requires implicit reasoning not available as surface-level lexical information. For example, in "of course, [we] still have the [storm surge] coming," given the context, we can reasonably infer discontent towards the "storm surge" despite the absence of polarizing words. Therefore, distantly supervised techniques largely based on lexical units (Mohammad and Turney, 2013; Abdul-Mageed and Ungar, 2017) fail to capture this type of deeper semantic phenomena.
Our paper presents a comprehensive investigation into perceived emotions in hurricane disasters. To this end, we introduce HURRICANEEMO, a dataset of 15,000 disaster-related tweets (in English) streamed during Hurricanes Harvey, Irma, and Maria, which were devastating tropical storms occurring in the 2017 Atlantic hurricane season (Belles, 2017). Our samples are annotated with fine-grained emotions derived from the Plutchik Wheel of Emotions (Plutchik, 2001), a well-defined ontology of emotion classes commonly used in computational social science (Abdul-Mageed and Ungar, 2017). To measure inter-annotator agreement on fine-grained emotion labels, we conceptualize the Plutchik Emotion Agreement (PEA) metric ( $\S 3$ ). PEA is intuitively grounded; our human evaluation shows workers agree with PEA's rankings  $88\%$  of the time. Furthermore, we perform insightful analyses on implicit and explicit emotions in hurricane tweets ( $\S 4$ ). Quite surprisingly, our results show that the two datasets are very similar.
<sup>1</sup>Specifically, we use Plutchik-8 and Plutchik-24 emotions. We refer readers to Plutchik (2001) for an in-depth discussion on their conception.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5290-5305 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ingly, we find consistencies in Plutchik-24 emotion distributions across Hurricanes Harvey, Irma, and Maria.
HURRICANEEMO also serves as a challenging new benchmark for large-scale, pre-trained language models. We establish baselines for a coarser Plutchik-8 emotion detection task using BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) ( $\S 5$ ). Our experiments reveal: (1) BERT only achieves  $64\%$  (averaged) accuracy; and (2) using "better" pre-trained models (e.g., RoBERTa) does not help, which is a strikingly different trend than most leaderboards (Wang et al., 2018). To better understand their pitfalls, in particular BERT, we conduct a comprehensive error analysis of 200 incorrectly predicted samples. In addition, we incorporate stronger inductive biases into BERT via pretraining on related tasks, which culminates in (averaged, absolute)  $+4\%$  accuracy ( $\S 6$ ). Finally, we propose unsupervised domain adaptation to bridge the domain gap between existing large-scale emotion datasets (e.g., EMONET (Abdul-Mageed and Ungar, 2017)) and HURRICANEEMO ( $\S 7$ ). Our code and datasets are made publicly available.
Emotion detection has been extensively studied in news headlines (Strapparava and Mihalcea, 2007; Katz et al., 2007), blog posts (Aman and Szpakowicz, 2007), health-related posts (Khanpour and Caragea, 2018), and song lyrics (Strapparava et al., 2012), but only recently, in social media websites (e.g., Twitter, Facebook) (Mohammad, 2012; Wang et al., 2012; Mohammad and Kiritchenko, 2015; Volkova and Bachrach, 2016; Abdul-Mageed and Ungar, 2017). However, emotion detection in disaster-centric domains, despite its practical importance, is limited. Schulz et al. (2013) (singlehandedly) annotate 2,200 Hurricane Sandy tweets using Ekman-6 emotions (Ekman, 1992). In contrast, we introduce 15,000 annotated tweets from multiple hurricanes with (much more fine-grained) Plutchik-24 emotions. Unlike Abdul-Mageed and Ungar (2017), we focus on readers' perceived emotions rather than writers' intended emotions.
Furthermore, in disaster-centric domains, the lack of labeled data required to train reliable models precludes the use of supervised learning techniques. Several works propose to use labeled data
from prior (source) disasters to learn classifiers for new (target) disasters (Verma et al., 2011; Nguyen et al., 2017; Imran et al., 2013, 2016; Caragea et al., 2016). However, due to the unique nature of each disaster (e.g., type, geographical location, season, cultural differences among the affected population), the source disaster may not accurately reflect the characteristics of the target disaster (Palen and Anderson, 2016; Imran et al., 2015). Domain adaptation techniques address these challenges by efficiently using large amounts of unlabeled target domain data, consequently outperforming the aforementioned supervised techniques (Alam et al., 2018; Li et al., 2017). Our work contributes to disaster-centric emotion detection in three ways by: (1) introducing a dataset large enough to train supervised classifiers; (2) exploring various forms of pre-training to instill strong inductive biases; and (3) establishing domain adaptation baselines by leveraging emotive samples obtainable via distant supervision.
In this section, we present HURRICANEEMO, an annotated dataset of 15,000 English tweets from Hurricanes Harvey, Irma, and Maria. We detail each component, including the initial preprocessing (§3.1), annotation procedures (§3.2), and the formulation and calculation of inter-annotator agreement (§3.3).
3.1 Preprocessing
Ray Chowdhury et al. (2019) release a repository of large-scale Twitter datasets consisting of tweets streamed during the Harvey, Irma, and Maria hurricanes, which we will refer to as HURRICANEEXT (i.e., extended). We use their tweets as a starting point for the construction of our dataset. We perform two types of preprocessing. First, we replace usernames and links with <USER> and <URL>, respectively, then eliminate duplicate tweets. Second, we use filtering techniques to ensure the resulting tweets contain emotive content.
We assume a lexical prior over emotion tweets, that is, requiring that an emotive tweet consist of at least one word derived from EMOLEX (Mohammad and Turney, 2013). EMOLEX consists of 14,182 crowdsourced words associated with several emotion categories. Critically, these words appear in emotional contexts, but are not necessarily emotion words themselves. For example, "payback" is
$^{2}$ https://github.com/shreydesai/hurricane
5291
related to the emotion "anger," but is also used extensively in finance. Significant past work (Bravo-Marquez et al., 2014; Majumder et al., 2017; Giatsoglou et al., 2017) has used this lexicon to bootstrap their emotion datasets, since the alternatives are (1) using unlabeled tweets as-is or (2) using a model to classify emotional tweets. Initially, we started with (1) and did no emotion-related preprocessing. However, the dataset contained many spurious tweets, such as snippets of news articles, that had little to do with emotions. The level of noise rendered the data prohibitively costly to annotate. For (2), there is simply no such large-scale data to train on, and existing resources like EMONET manifest an even stronger prior where tweets are only included if they explicitly contain an emotion hashtag (e.g., #sad, #angry, #happy).
We randomly sample 5,000 tweets each for annotation from the filtered datasets for Harvey, Irma, and Maria; in total, this yields 15,000 annotations. We request workers on Amazon Mechanical Turk to label tweets with a list of Plutchik-24 emotions. Furthermore, to enable fine-grained emotion analysis, we do not crowdsourced Plutchik-8 emotions directly. We require that workers reside in the US and have completed  $500+$  HITs with an acceptance rate  $\geq 95\%$ . Each HIT is completed by 5 workers.
3.3 Inter-Annotator Agreement
In this section, we elaborate on our PEA metric for computing inter-annotator agreement with fine-grained emotion labels.
Challenges. Fine-grained emotion annotation presents several challenges for evaluating inter-annotator agreement. First, because a tweet can convey multiple emotions, we allow workers to select more than one Plutchik-24 emotion. This implies an agreement metric must support scoring sets of categorical values. Passonneau (2004) use set distance metrics for capturing agreement between coreference cluster annotations. Similarly, Wood et al. (2018) incorporate Jaccard's similarity in Krippendorff's alpha. However, these methods would penalize fine-grained emotions equally, which is not ideal. For the Plutchik wheel, the proximity of any two emotions represents their relatedness. For example, TRUST and ADMIRATION belong to the same emotion group while LOATHING and ADMIRATION are orthogonal to each other.
Figure 1: Visualization of the PEA metric. The unit circle is superimposed on the Plutchik wheel, and each Plutchik-8 emotion is assigned a radian value. In this example, the (normalized) distance between the emotions corresponding to  $\frac{3\pi}{2}$  and  $\frac{\pi}{4}$  is 0.25.
PEA Scores. We introduce the Plutchik Emotion Agreement—hereafter referred to as PEA—to address these challenges. We superimpose a unit circle onto the Plutchik wheel, representing each Plutchik-8 emotion as a polar coordinate (e.g.,  $\mathrm{DISAPPROVAL} = \left(\frac{\sqrt{2}}{2},\frac{-\sqrt{2}}{2}\right)$ ). Intuitively, the angles between Plutchik-8 emotions represent how similar or dissimilar they are. If two Plutchik-24 annotations belong to the same Plutchik-8 group, we do not penalize them (e.g., JOY and ECSTASY incur no penalty). Otherwise, we enforce a linear penalty based on how radially separate the annotations are (e.g., ECSTASY and GRIEF incur the highest penalty). Higher PEA scores imply more agreement.
Example. Figure 1 visualizes our metric. In this example, two annotators select emotions with radians  $\frac{3\pi}{2}$  and  $\frac{\pi}{4}$ , respectively. The  $|f(e_x^{(i)}) - f(e_y^{(j)})|$  term evaluates to  $\frac{5\pi}{4}$ . Then, it is normalized using  $\frac{1}{\pi}$ , yielding  $\frac{5}{4} = 1.25$ . Finally, we subtract to obtain the agreement score:  $|1 - 1.25| = 0.25$ . Intuitively, this makes sense as the decisions are only slightly better than being in complete disagreement (i.e., orthogonal).
Formulation. For clarity, we introduce notation. Let  $w_{x}$  and  $w_{y}$  denote workers with (categorical) annotation sets  $\{e_{x}^{(i)}\}_{i = 1}^{n}$  and  $\{e_{y}^{(j)}\}_{j = 1}^{m}$ , respectively. The pairwise agreement  $d(w_{x},w_{y})$  between the workers is computed as:
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \max _ {j} \left(| 1 - \frac {1}{\pi} | f (e _ {x} ^ {(i)}) - f (e _ {y} ^ {(j)}) | |\right)
$$
Hurricane	Vocabulary	Features (%)
Orig.	Filt.	#	@	//
Harvey	20.6 K	14.4 K	48.1	27.4	85.3
Irma	14.6 K	8.8 K	41.4	22.5	81.7
Maria	21.6 K	15.8 K	36.5	30.3	78.3
Table 1: Per-hurricane dataset statistics. In the vocabulary section, Orig. shows vocabulary counts (obtained through whitespace tokenization) and Filt. shows counts after <USER> and <URL> preprocessing. In the features section, we show the percentage of tweets with hashtags (#), user mentions (@), and links (/).
where  $\frac{1}{\pi}$  is a normalizing constant and  $f:\Omega \to \mathbb{R}$  is a map from Plutchik-8 emotions to radians. Given a collection of workers that annotated a tweet, we obtain per-worker PEA scores by averaging over all possible pairwise agreements. For example, if workers  $w_{1 - 3}$  annotated the same tweet,  $\mathrm{PEA}(w_1) = \frac{1}{2} (d(w_1,w_2) + d(w_1,w_3))$  .For quality control, we filter annotations from workers with  $\mathrm{PEA}\leq 0.55$  . This threshold is determined through manual inspection of 50 workers and their annotations. The (averaged, per-worker) PEA scores for each hurricane are: Harvey (65.7), Maria (67.3), and Irma (70.3).
Human Evaluation. We perform a human evaluation with our proposed metric, which is absent in previous work for measuring inter-annotator agreement for emotion annotations (Wood et al., 2018; Ohman et al., 2018). Crowdsourced workers are asked to determine the agreement between two annotation pairs constructed from three annotators, that is, A:  $(e_1,e_2)$  and B:  $(e_1,e_3)$ . They choose between three options: (1) A has higher agreement than B; (2) A and B have (roughly) the same agreement; and (3) B has higher agreement than A.  $88.2\%$  of the worker rankings match with PEA's rankings, pointing towards strong human agreement. The workers themselves in this study also show good agreement according to Krippendorff's alpha  $(\alpha = 74.0)$  (Artstein and Poesio, 2008).
4.1 Dataset Overview
Table 1 presents several statistics of HURRICANEEMO. We make three observations. First, the
Mexico helped us during Houston, lets return the favor!	joy, admiration, pensiveness
Hurricane Irma is hitting Florida. Ever-
yone evacuated Here I am, still in
Florida bring it on Irma, bring it on.	acceptance, an-
ticipation, vigi-
lance
puerto rico should be the ONLY
THING in American News. <URL>	anger, annoyance, interest
Table 2: Samples from HURRICANEEMO. Each sample is annotated with multiple Plutchik-24 emotions.
vocabularies across all datasets are large considering there are only 5,000 tweets per hurricane. The vocabularies do decrease by about  $30\%$  after preprocessing, although the resulting sizes still suggest users use a myriad of words to express their emotions. Second, only about  $50\%$  of Harvey tweets and  $40\%$  of Irma/Maria tweets contain hashtags. Hashtags are a unique marker of Twitter discourse (Ritter et al., 2011), but in our dataset specifically, hashtags are used to tag particular entities, spread disaster-relief awareness, and create trending content. This phenomena alone makes our tweets different from those collected through distant supervision (Abdul-Mageed and Ungar, 2017). Third, roughly  $80 - 85\%$  of tweets contain links to third-party content. Users commonly use links to share news articles, resources for humanitarian aid, and other miscellaneous multimedia.
Table 2 shows three samples from HURRICANEEMO. Unlike EMONET (Abdul-Mageed and Ungar, 2017), our dataset does not have the strong assumption that only one emotion can be expressed in a tweet. For example, the first tweet lexically points towards the expression of more than one emotion. The predicate "helped us" implies the user admires Mexico for providing aid, and the exclamation mark is indicative of JOY. In addition, our samples contain a mix of implicit and explicit emotions, which lexical information alone cannot resolve. In the third tweet, there are no particular words that point towards ANGER and ANNOYANCE, but we can infer the user is upset that the media is not prioritizing Hurricane Maria.
Finally, our emotion prediction tasks cannot be solved by simply retrofitting pre-trained word embeddings (Mikolov et al., 2013; Pennington et al., 2014) or contextualized representations (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019), which we also empirically show in our experiments ( $\S 5$ ). These methods work best for explicit emotion detection as they largely overfit to sparse lex
${}^{3}$  A reasonable interpretation of PEA scores may be as follows: 0-25 (no agreement), 25-50 (poor agreement), 50-75 (moderate agreement), 75-100 (high agreement).  ${}^{4}$  See Appendix B for details on our procedures.
Plutchik-8	Plutchik-24
Emotion	Abbrv.	Emotion	Abbrv.
aggressiveness	agrsv	rage	rage
anger	anger
annoyance	anyce
optimism	optsm	vigilance	vglnc
anticipation	antcp
interest	inrst
love	love	ecstasy	ecsty
joy	joy
serenity	srnty
submission	sbmsn	admiration	admrn
trust	trust
acceptance	acptn
awe	awe	terror	trror
fear	fear
apprehension	aprhn
disapproval	dspvl	amazement	amzmt
surprise	srpse
distraction	dstrn
remorse	rmrse	grief	grief
sadness	sadns
pensiveness	psvne
contempt	cntmp	loathing	lthng
disgust	dsGST
boredom	brdom
Table 3: Plutchik-8 (left) and Plutchik-24 (right) abbreviations used throughout this paper.
ical features. Rather, in order to capture implicit emotions, models must carry an inductive bias that appropriately reasons over the context (e.g., what event(s) occurred?) and semantic roles (e.g., what happened to whom?) while balancing the aforementioned features.
4.2 Fine-Grained Emotions
We begin to analyze the fine-grained emotions present in our datasets. We ask the following questions: What is the general distribution of emotions? Are certain emotion groups highlighted more than others? How does the distribution change across hurricanes?
Figure 2 shows Plutchik-24 emotion distributions for Hurricanes Harvey, Irma, and Maria. From these plots, a couple of trends emerge. First, the Plutchik-24 emotion counts are within the ballpark of each other with the notable exceptions of ADMIRATION and FEAR. This suggests that, on average, hurricane disasters evoke a similar spread of implicit and explicit emotions among most emotion categories. Second, users tend to post more optimistic content during hurricane disasters. We hy
Figure 2: Per-hurricane emotion counts where each box's Plutchik-8 emotion is broken down into its respective Plutchik-24 emotions. Plutchik-24 emotions are abbreviated using the codes in Table 3.
pothesize that users use Twitter as a social platform to spread awareness of the hurricanes themselves or post-disaster relief efforts, commonly using hashtags like #prayfortexas, #floridaevacuation, and #donationdrive. It is encouraging to see that although users do express natural emotions such as fear, sadness, and anger, many seek to help others in the face of adversity. Third, sharp changes in emotion counts between Harvey and Irma may be tied to their history. In the 2017 Atlantic hurricane season, Harvey materialized as a Cat-4 hurricane, and Irma followed around two weeks later as a Cat-5 hurricane. Through side-by-side comparisons of both hurricanes' tweets, we found the Irma tweets had more descriptions of destruction and its aftermath. These changes in discourse potentially explain shifts between the emotion distributions.
4.3 Emotion Co-Occurrence
Thus far, we have analyzed each Plutchik-24 emotion in isolation. In this section, we ask the following questions: How do Plutchik-8 emotion groups co-occur with one another? Do co-occurrence patterns change across hurricanes?
Figure 3 shows co-occurrence heatmaps for each hurricane. Intuitively, we see strong correlations between polarized emotions, that is, emo
5 Abbreviations for Category-  $x$  . This refers to the SaffirSimpson scale for classifying hurricanes based on sustained wind speed, which ranges from 1-5 in order of severity.
Figure 3: Per-hurricane Plutchik-8 emotion co-occurrences. The matrices are symmetric across the diagonal, so we mask the upper diagonal of the matrix for clarity. Plutchik-8 emotions are abbreviated using the codes in Table 3.
tions categorized as positive and negative. For example, ( LOVE, AGGRESSIVENESS) does not appear as frequently as ( LOVE, OPTIMISM) or ( CONTEMPL, AGGRESSIVENESS). However, this premise does not always hold; the pairs (\{ DISAPPROVAL, REMORSE \}, OPTIMISM) also co-occur across all hurricanes. Representative of this phenomenon is the tweet: "I'm raising money for Hurricane Maria Destroyed Everything. Click to Donate: <URL> via <USER)." The user indicates disapproval towards the hurricane by evoking pathos, but also shows optimism by donating money to a relief effort. Finally, similar to our previous observations (§4.2), we notice an increase in co-occurrence frequencies from Harvey → Irma. This increase is, somewhat surprisingly, most apparent with ( AWE, OPTIMISM), although (\{ DISAPPROVAL, REMORSE \}, AWE) frequencies also exhibit a noticeable gain. Once again, we posit that users may be expressing their sadness regarding the Cat-4 → Cat-5 jump, but at the same time, offering solidarity to those affected by the hurricanes.
5 Baseline Modeling
We now turn to modeling the emotions in HURRICANEEMO. Because Plutchik-24 emotion counts are heavily imbalanced, we group them into Plutchik-8 emotions and consequently create 8 binary classification tasks.
The tweets are assorted into their respective label buckets; because tweets may be labeled with more than one emotion, each belongs to one or more buckets. These buckets represent positive samples (i.e., tweets labeled with that emotion). To create negative samples, we sample an equal amount from
Plutchik-8 Emotion	Train	Valid	Test
Aggressiveness	4,209	526	527
Optimism	11,902	1,488	1,488
Love	2,569	321	322
Submission	6,092	762	762
Awe	7,324	916	916
Disapproval	5,931	741	742
Remorse	7,732	967	967
Contempt	3,763	470	471
Table 4: Train, validation, and test splits for each Plutchik-8 emotion.
other buckets. From here, we shuffle the positive and negative samples and perform an 80/10/10 split to create the train, validation, and test sets.<sup>6</sup> Table 4 enumerates the splits.
We consider both traditional neural models and pretrained language models. We implement our models in PyTorch (Paszke et al., 2019) and perform all experiments on an NVIDIA Titan V GPU. Training and optimization hyperparameters are detailed in Appendix C. We report mean performance across 10 runs, each with a different random initialization. Below, we elaborate on our models:
Traditional Neural Models. Each is equipped with 200D GloVe embeddings pre-trained on 2B tweets (Pennington et al., 2014): (1) Logistic Regression: We average the word embeddings of each token in the sequence (Iyyer et al., 2015); (2) CNN: A word-level CNN (Kim, 2014) with 100 filters of size [3, 4, 5] obtains representations. They are max-pooled and concatenated row-wise. We also experiment with a character-level CNN with filter sizes [5, 6, 7]; (3) GRU: A one-layer, unidirectional GRU (Cho et al., 2014) with a hidden dimension of 100 obtains features, which are mean pooled. For all models, penultimate representations are projected with a weight matrix  $\mathbf{W} \in \mathbb{R}^{d \times 2}$ .