SlowGuess/ACL_2020_OCR huggingface dataset:
Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Robert Plutchik. 2001. The Nature of Emotions: Human emotions have deep evolutionary roots, a fact that may explain their complexity and provide tools for clinical practice. American Scientist, 89(4):344-350.
Jishnu Ray Chowdhury, Cornelia Caragea, and Doina Caragea. 2019. Keyphrase Extraction from Disaster-related Tweets. In *The World Wide Web Conference*, WWW 2019, pages 1555–1566, New York, NY, USA. Association for Computing Machinery (ACM).
Hannah Ritchie and Max Roser. 2020. Natural Disasters. Our World in Data.
Alan Ritter, Sam Clark, Musam, and Oren Etzioni. 2011. Named Entity Recognition in Tweets: An Experimental Study. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1524-1534, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Axel Schulz, Tung Dang Thanh, Heiko Paulheim, and Immanuel Schweizer. 2013. A Fine-Grained Sentiment Analysis Approach for Detecting Crisis Related Microposts. In Information Systems for Crisis Response and Management (ISCRAM).
Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 Task 14: Affective Text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70-74, Prague, Czech Republic. Association for Computational Linguistics.
Carlo Strapparava, Rada Mihalcea, and Alberto Battocchi. 2012. A Parallel Corpus of Music and Lyrics Annotated with Emotions. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC 2012), pages 2343-2346, Istanbul, Turkey. European Language Resources Association (ELRA).
Sudha Verma, Sarah Vieweg, William Corvey, Leysia Palen, James Martin, Martha Palmer, Aaron Schram, and Kenneth Anderson. 2011. Natural Language Processing to the Rescue? Extracting "Situational Awareness" Tweets During Mass Emergency. In Proceedings of the International AAAI Conference on Web and Social Media (ICWSM 2017).
Svitlana Volkova and Yoram Bachrach. 2016. Inferring Perceived Demographics from User Emotional Tone and User-Environment Emotional Contrast. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long
Papers), pages 1567-1578, Berlin, Germany. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit Sheth. 2012. Harnessing Twitter "Big Data" for Automatic Emotion Identification. In Proceedings of the 2012 ASE/IEEE International Conference on Social Computing and 2012 ASE/IEEE International Conference on Privacy, Security, Risk and Trust, SOCIALCOM-PASSAT 2012, pages 587-592, Washington, DC, USA. IEEE Computer Society.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's Transformers: State-of-the-art Natural Language Processing. arXiv preprint arXiv:1910.03771.
Ian Wood, John McCrae, Vladimir Andryushechkin, and Paul Buitelaar. 2018. A Comparison Of Emotion Annotation Schemes And A New Annotated Data Set. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Figure 4: Top 1000 (common) wordpiece densities for EMONET (left) and HURRICANEEMO (right). Densities are calculated by counting wordpiece occurrences and normalizing by the total number of occurrences.
A Domain Shifts
Following the methodology outlined in Desai et al. (2019), we use the Jenson-Shannon Divergence (JSD) between the vocabulary distributions in EMONET and HURRICANEEMO to quantify the domain divergence. The JSD is 0.199, approximately 1e5 larger than those reported in Desai et al. (2019). Figure 4 shows the densities of the top 1000 common wordpieces between both domains. The striking visual differences, even among common wordpieces, indicates a large discrepancy in the input distributions.
B Plutchik Emotion Agreement
Interpretable Scale. To assign PEA scores an interpretable scale, we compare randomly generated annotations against our obtained annotations. We detail the process to create random annotations. First, we compute the average number of emotions a worker assigns to a tweet, which evaluates to 3 for all hurricanes. Second, we sample 3 random emotions from the Plutchik-8 wheel for 5000 total annotations. Figure 5 compares the two types of annotations. The per-worker PEA scores for the random annotations collect around the mean (0.5), which is expected due to the law of large numbers. In contrast, the per-worker PEA scores for our annotations are shifted towards the right, indicating better agreement than the random baseline. Therefore, we interpret our annotations as showing "moderate agreement" under the PEA metric.
Human Evaluation. Using our worker annotations across all three hurricanes, we create two annotation pairs for three workers, that is, A:  $(w_{1}, w_{2})$  and B:  $(w_{1}, w_{3})$ , where A and B have a shared worker  $w_{1}$ . This format lends a total of 73,418 A/B total pairs. We sample 500 A/B pairs from this
pool, initialize each HIT with 10 pairs, and assign 5 total workers per HIT.
C Baseline Modeling
Table 8 shows the hyperparameters. For our pretrained models (e.g., BERT and RoBERTa), we use the default dropout rate (0.1) on the self-attention layers, but do not use additional dropout on the top linear layer. Furthermore, we use gradient accumulation to enable training with larger mini-batches.
D Task-Guided Pre-training
Masked Language Modeling. Following Devlin et al. (2019), we select  $15\%$  of inputs uniformly at random (except for [CLS] and [SEP]) as prediction targets for the masked language modeling task. From the corresponding inputs,  $80\%$  are set to [MASK],  $10\%$  are set to random tokens, and  $10\%$  are set to the original tokens. However, we follow Liu et al. (2019) in creating pre-training data dynamically, rather than statically. This merely leads to slower convergence times as it becomes more difficult to fit the data. We fine-tune on the pre-training data for 10 epochs using a batch size of 16 and learning rate of 2e-5. Once pre-training concludes, we initialize a BERT model with these weights and fine-tune it on our emotion tasks using the hyperparameters in Table 8 with a learning rate of 3e-5.
Pre-training Corpus. Our pre-training corpus is created by concatenating a collection of (shuffled) tweets  $x_{1}, x_{2}, \dots, x_{n}$  together, each separated by [SEP]. The corpus is split into segments of size 512 with [CLS] prepended to each one. For clarity, each batch consisting of tokens  $x_{i}, \dots, x_{j}$  is constructed as [CLS]  $x_{i}$  [SEP] ... [SEP]  $x_{j}$  [SEP]. We elaborate on two design decisions. First, prepending [CLS] to each batch, as opposed to each tweet, leads to better results. Second, largely due to computational reasons, we pack disparate tweets together in the same batch.
E Extended Pre-training Experiments
E.1 EmoNet Binary Task Pre-training
In Section 6, we pre-trained on a EMONET multiclass classification task. In this section, we explore a fine-grained pre-training scheme. We create Plutchik-8 binary tasks from EMONET, then fine-tune each emotion model separately on their respective HURRICANEEMO tasks. Table 9 shows
Figure 5: Histograms corresponding to PEA score distributions for random annotations (top) and our annotations (bottom).
Logistic Reg.	Word CNN	Char CNN	GRU	BERT	RoBERTa
Epochs	5	5	5	5	3	3
Batch Size	64	64	64	64	16	16
Learning Rate	1e-4	1e-3	5e-5	1e-4	2e-5	2e-5
Weight Decay	0	0	0	0	0	1e-3
Dropout	0	0.5	0.7	0.7	-	-
Table 8: Hyperparameters for the baseline modeling experiments (§5).
the results. EMONET-BINARY performs markedly worse than EMONET-MULTI and leads to a  $-2\%$  reduction in averaged accuracy. Therefore, multiclass pre-training creates better representations for downstream evaluation, although they are still not as effective as other pre-training methods (e.g., masked language modeling).
E.2 Varying Amounts of Pre-training Data
The SENTIMENT and HURRICANEEXT datasets contain significantly more samples than currently used. In this section, we study the effects of using varying amounts of pre-training data on downstream HURRICANEEMO performance. For both pre-training datasets, we use 1.6M samples. Table 10 shows the supervised SENTIMENT results. Tables 11 and 12 show the unsupervised SENTIMENT and HURRICANEEXT results, respectively. For both types of pre-training tasks, there is no noticeable benefit to using more pre-training data. The supervised SENTIMENT and unsupervised HURRICANEEXT results both saturate around 200K samples, which is what we report in our paper. The results for unsupervised HURRICANEEXT pre-training are especially compelling because they show that, without any labeled data, we can achieve strong down
stream results. Finally, the unsupervised SENTIMENT task yields almost no gains for most emotions, showing that the type of data used for masked language modeling matters. Through side-by-side comparisons, we notice that the SENTIMENT samples are shorter in length and the HURRICANEEXT samples contain more relevant content, such as hurricane-specific hashtags.
AGR	OPT	LOV	SBM	AWE	DSP	RMR	CNT	AVG
NO-PRETRAIN	67.6	75.0	54.0	67.4	68.3	55.7	58.5	66.8	64.1
Multi	73.5	75.2	55.2	68.8	67.5	53.1	60.0	71.7	65.6
Binary	67.7	74.9	53.7	64.7	67.5	54.5	55.8	63.6	62.8
Table 9: Pre-training using multi-class and binary EMONET tasks. See Table 6 for styling considerations.
AGR	OPT	LOV	SBM	AWE	DSP	RMR	CNT	AVG
NO-PRETRAIN	67.6	75.0	54.0	67.4	68.3	55.7	58.5	66.8	64.1
50 K	73.5	75.3	60.7	69.7	67.1	51.3	55.2	66.3	64.9
100 K	72.8	75.8	62.7	71.0	65.6	53.4	57.0	67.3	65.7
200 K	73.4	75.6	69.1	69.8	66.5	53.3	57.1	69.8	66.8
400 K	73.1	75.4	67.2	70.1	65.7	53.2	57.2	67.4	66.2
800 K	73.5	75.3	56.2	69.4	65.1	54.4	57.1	68.2	64.9
1600 K	71.2	75.2	64.8	68.8	64.7	55.1	56.1	70.7	65.8
Table 10: Pre-training using 50-1600K labeled samples from SENTIMENT. See Table 6 for styling considerations.
AGR	OPT	LOV	SBM	AWE	DSP	RMR	CNT	AVG
NO-PRETRAIN	67.6	75.0	54.0	67.4	68.3	55.7	58.5	66.8	64.1
50 K	70.7	74.9	54.6	66.3	67.0	53.9	59.3	65.8	64.0
100 K	71.6	75.0	54.0	66.3	68.6	55.1	57.4	62.3	63.8
200 K	69.1	74.9	53.6	66.2	67.3	54.3	57.9	64.4	63.5
400 K	70.0	74.9	53.8	69.0	68.8	54.5	60.1	64.5	64.5
800 K	70.5	74.9	55.1	66.2	69.0	53.3	59.4	63.4	64.0
1600 K	69.1	74.9	55.3	66.5	67.2	54.6	59.3	65.0	64.0
Table 11: Pre-training using 50-1600K unlabeled samples from SENTIMENT. See Table 6 for styling considerations.
AGR	OPT	LOV	SBM	AWE	DSP	RMR	CNT	AVG
NO-PRETRAIN	67.6	75.0	54.0	67.4	68.3	55.7	58.5	66.8	64.1
50 K	72.7	75.0	60.0	67.2	69.0	56.4	60.4	72.2	66.6
100 K	71.8	75.1	57.4	69.1	70.3	55.2	62.4	65.3	65.8
200 K	73.6	75.4	69.8	68.9	69.7	57.9	60.2	70.2	68.2
400 K	71.4	75.2	59.7	69.7	68.8	55.2	60.7	63.6	65.5
800 K	71.4	75.3	58.9	69.4	69.6	54.0	60.3	71.3	66.3
1600 K	73.3	75.7	50.7	68.3	65.5	55.8	61.0	64.1	64.3
Table 12: Pre-training using 50-1600K unlabeled samples from HURRICANEXT. See Table 6 for styling considerations.
Figure 6: Visualization of BERT's self-attention on a Hurricane Irma sample. In particular, this head captures the entities "hurricane irma," "florida," "everyone" and the verb phrase "crane collapses."
5305