SlowGuess/ACL_2020_OCR huggingface dataset:
admiration	amusement	approval	caring	anger	annoyance	disappointment	disapproval	confusion
great (42)	lol (66)	agree (24)	you (12)	fuck (24)	annoying (14)	disappointing (11)	not (16)	confused (18)
awesome (32)	haha (32)	not (13)	worry (11)	hate (18)	stupid (13)	disappointed (10)	don't (14)	why (11)
amazing (30)	funny (27)	don't (12)	careful (9)	fucking (18)	fucking (12)	bad (9)	disagree (9)	sure (10)
good (28)	lmao (21)	yes (12)	stay (9)	angry (11)	shit (10)	disappointment (7)	nope (8)	what (10)
beautiful (23)	hilarious (18)	agreed (11)	your (8)	dare (10)	dumb (9)	unfortunately (7)	doesn't (7)	understand (8)
desire	excitement	gratitude	joy	disgust	embarrassment	fear	grief	curiosity
wish (29)	excited (21)	thanks (75)	happy (32)	disgusting (22)	embarrassing (12)	scared (16)	died (6)	curious (22)
want (8)	happy (8)	thank (69)	glad (27)	awful (14)	shame (11)	afraid (16)	rip (4)	what (18)
wanted (6)	cake (8)	for (24)	enjoy (20)	worst (13)	awkward (10)	scary (15)	why (13)
could (6)	wow (8)	you (18)	enjoyed (12)	worse (12)	embarrassment (8)	terrible (12)	how (11)
ambitious (4)	interesting (7)	sharing (17)	fun (12)	weird (9)	embarrassed (7)	terrifying (11)	did (10)
love	optimism	pride	relief	nervousness	remorse	sadness	realization	surprise
love (76)	hope (45)	proud (14)	glad (5)	nervous (8)	sorry (39)	sad (31)	realize (14)	wow (23)
loved (21)	hopefully (19)	pride (4)	relieved (4)	worried (8)	regret (9)	sadly (16)	realized (12)	surprised (21)
favorite (13)	luck (18)	accomplishment	relieving (4)	anxiety (6)	apologies (7)	sorry (15)	realised (7)	wonder (15)
loves (12)	hoping (16)	(4)	relief (4)	anxious (4)	apologize (6)	painful (10)	realization (6)	shocked (12)
like (9)	will (8)	worrying (4)	guilt (5)	crying (9)	thought (6)	omg (11)
Table 3: Top 5 words associated with each emotion (positive, negative, ambiguous). The rounded  $z$ -scored log odds ratios in the parentheses, with the threshold set at 3, indicate significance of association.
why we release all 58K examples with all annotators' ratings.
Grouping emotions. We create a hierarchical grouping of our taxonomy, and evaluate the model performance on each level of the hierarchy. A sentiment level divides the labels into 4 categories - positive, negative, ambiguous and Neutral - with the Neutral category intact, and the rest of the mapping as shown in Figure 2. The Ekman level further divides the taxonomy using the Neutral label and the following 6 groups: anger (maps to: anger, annoyance, disapproval), disgust (maps to: disgust), fear (maps to: fear, nervousness), joy (all positive emotions), sadness (maps to: sadness, disappointment, embarrassment, grief, remorse) and surprise (all ambiguous emotions).
5.2 Model Architecture
We use the BERT-base model (Devlin et al., 2019) for our experiments. We add a dense output layer on top of the pretrained model for the purposes of finetuning, with a sigmoid cross entropy loss function to support multi-label classification. As an additional baseline, we train a bidirectional LSTM.
When finetuning the pre-trained BERT model, we keep most of the hyperparameters set by Devlin et al. (2019) intact and only change the batch size and learning rate. We find that training for at least 4 epochs is necessary for learning the data, but training for more epochs results in overfitting. We also find that a small batch size of 16 and learning rate of 5e-5 yields the best performance.
For the biLSTM, we set the hidden layer dimensionality to 256, the learning rate to 0.1, with a
decay rate of 0.95. We apply a dropout of 0.7.
Table 4 summarizes the performance of our best model, BERT, on the test set, which achieves an average F1-score of .46 (std=.19). The model obtains the best performance on emotions with overt lexical markers, such as gratitude (.86), amusement (.8) and love (.78). The model obtains the lowest F1-score on grief (0), relief (.15) and realization (.21), which are the lowest frequency emotions. We find that less frequent emotions tend to be confused by the model with more frequent emotions related in sentiment and intensity (e.g., grief with sadness, pride with admiration, nervousness with fear) — see Appendix G for a more detailed analysis.
Table 5 and Table 6 show results for a sentiment-grouped model (F1-score = .69) and an Ekman-grouped model (F1-score = .64), respectively. The significant performance increase in the transition from full to Ekman-level taxonomy indicates that this grouping mitigates confusion among inner-group lower-level categories.
The biLSTM model performs significantly worse than BERT, obtaining an average F1-score of .41 for the full taxonomy, .53 for an Ekman-grouped model and .6 for a sentiment-grouped model.
6 Transfer Learning Experiments
We conduct transfer learning experiments on existing emotion benchmarks, in order to show our data generalizes across domains and taxonomies. The goal is to demonstrate that given little labeled data in a target domain, one can utilize GoEmotions as baseline emotion understanding data.
Figure 3: Transfer learning results in terms of average F1-scores across emotion categories. The bars indicate the  $95\%$  confidence intervals, which we obtain from 10 different runs on 10 different random splits of the data.
Emotion	Precision	Recall	F1
admiration	0.53	0.83	0.65
amusement	0.70	0.94	0.80
anger	0.36	0.66	0.47
annoyance	0.24	0.63	0.34
approval	0.26	0.57	0.36
caring	0.30	0.56	0.39
confusion	0.24	0.76	0.37
curiosity	0.40	0.84	0.54
desire	0.43	0.59	0.49
disappointment	0.19	0.52	0.28
disapproval	0.29	0.61	0.39
disgust	0.34	0.66	0.45
embarrassment	0.39	0.49	0.43
excitement	0.26	0.52	0.34
fear	0.46	0.85	0.60
gratitude	0.79	0.95	0.86
grief	0.00	0.00	0.00
joy	0.39	0.73	0.51
love	0.68	0.92	0.78
nervousness	0.28	0.48	0.35
neutral	0.56	0.84	0.68
optimism	0.41	0.69	0.51
pride	0.67	0.25	0.36
realization	0.16	0.29	0.21
relief	0.50	0.09	0.15
remorse	0.53	0.88	0.66
sadness	0.38	0.71	0.49
surprise	0.40	0.66	0.50
macro-average	0.40	0.63	0.46
std	0.18	0.24	0.19
6.1 Emotion Benchmark Datasets
We consider the nine benchmark datasets from Bostan and Klinger (2018)'s Unified Dataset, which vary in terms of their size, domain, qual
Table 4: Results based on GoEmotions taxonomy.
Sentiment	Precision	Recall	F1
ambiguous	0.54	0.66	0.60
negative	0.65	0.76	0.70
neutral	0.64	0.69	0.67
positive	0.78	0.87	0.82
macro-average	0.65	0.74	0.69
std	0.09	0.10	0.09
Table 5: Results based on sentiment-grouped data.
Ekman Emotion	Precision	Recall	F1
anger	0.50	0.65	0.57
disgust	0.52	0.53	0.53
fear	0.61	0.76	0.68
joy	0.77	0.88	0.82
neutral	0.66	0.67	0.66
sadness	0.56	0.62	0.59
surprise	0.53	0.70	0.61
macro-average	0.59	0.69	0.64
std	0.10	0.11	0.10
Table 6: Results using Ekman's taxonomy.
ity and taxonomy. In the interest of space, we only discuss three of these datasets here, chosen based on their diversity of domains. In our experiments, we observe similar trends for the additional benchmarks, and all are included in the Appendix H.
The International Survey on Emotion Antecedents and Reactions (ISEAR) (Scherer and Wallbott, 1994) is a collection of personal reports on emotional events, written by 3000 people from different cultural backgrounds. The dataset contains 8k sentences, each labeled with a single emotion. The categories are anger, disgust, fear, guilt, joy, sadness and shame.
EmoInt (Mohammad et al., 2018) is part of the SemEval 2018 benchmark, and it contains crowdsourced annotations for 7k tweets. The labels are
intensity annotations for anger, joy, sadness, and fear. We obtain binary annotations for these emotions by using .5 as the cutoff.
Emotion-Stimulus (Ghazi et al., 2015) contains annotations for  $2.4\mathrm{k}$  sentences generated based on FrameNet's emotion-directed frames. Their taxonomy is anger, disgust, fear, joy, sadness, shame and surprise.
6.2 Experimental Setup
Training set size. We experiment with varying amount of training data from the target domain dataset, including 100, 200, 500, 1000, and  $80\%$  (named "max") of dataset examples. We generate 10 random splits for each train set size, with the remaining examples held as a test set.
We report the results of the finetuning experiments detailed below for each data size, with confidence intervals based on repeated experiments using the splits.
Finetuning. We compare three different finetuning setups. In the BASELINE setup, we finetune BERT only on the target dataset. In the FREEZE setup, we first finetune BERT on GoEmotions, then perform transfer learning by replacing the final dense layer, freezing all layers besides the last layer and finetuning on the target dataset. The NOFREEZE setup is the same as FREEZE, except that we do not freeze the bottom layers. We hold the batch size at 16, learning rate at 2e-5 and number of epochs at 3 for all experiments.
The results in Figure 3 suggest that our dataset generalizes well to different domains and taxonomies, and that using a model using GoEmotions can help in cases when there is limited data from the target domain, or limited resources for labeling.
Given limited target domain data (100 or 200 examples), both FREEZE and NOFREEZE yield significantly higher performance than the BASELINE, for all three datasets. Importantly, NOFREEZE results show significantly higher performance for all training set sizes, except for "max", where NOFREEZE and BASELINE perform similarly.
We present GoEmotions, a large, manually annotated, carefully curated dataset for fine-grained emotion prediction. We provide a detailed data
analysis, demonstrating the reliability of the annotations for the full taxonomy. We show the generalizability of the data across domains and taxonomies via transfer learning experiments. We build a strong baseline by fine-tuning a BERT model, however, the results suggest much room for future improvement. Future work can explore the cross-cultural robustness of emotion ratings, and extend the taxonomy to other languages and domains.
Data Disclaimer: We are aware that the dataset contains biases and is not representative of global diversity. We are aware that the dataset contains potentially problematic content. Potential biases in the data include: Inherent biases in Reddit and user base biases, the offensive/vulgar word lists used for data filtering, inherent or unconscious bias in assessment of offensive identity labels, annotators were all native English speakers from India. All these likely affect labeling, precision, and recall for a trained model. The emotion pilot model used for sentiment labeling, was trained on examples reviewed by the research team. Anyone using this dataset should be aware of these limitations of the dataset.
We thank the three anonymous reviewers for their constructive feedback. We would also like to thank the annotators for their hard work.
Muhammad Abdul-Mageed and Lyle Ungar. 2017. EmoNet: Fine-Grained Emotion Detection with Gated Recurrent Neural Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 718-728.
Cecilia Ovesdotter Alm, Dan Roth, and Richard Sproat. 2005. Emotions from text: Machine learning for text-based emotion prediction. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 579-586, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Laura-Ana-Maria Bostan and Roman Klinger. 2018. An analysis of annotated corpora for emotion classification in text. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2104-2119.
Luke Breitfeller, Emily Ahn, David Jurgens, and Yulia Tsvetkov. 2019. Finding microaggressions in the
wild: A case for locating elusive phenomena in social media posts. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1664-1674.
Sven Buechel and Udo Hahn. 2017. Emobank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 578-585.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37-46.
Alan Cowen, Disa Sauter, Jessica L Tracy, and Dacher Keltner. 2019a. Mapping the passions: Toward a high-dimensional taxonomy of emotional experience and expression. Psychological Science in the Public Interest, 20(1):69-90.
Alan S Cowen, Hillary Anger Elfenbein, Petri Laukka, and Dacher Keltner. 2018. Mapping 24 emotions conveyed by brief human vocalization. American Psychologist, 74(6):698-712.
Alan S Cowen, Xia Fang, Disa Sauter, and Dacher Keltner. in press. What music makes us feel: At least thirteen dimensions organize subjective experiences associated with music across cultures. Proceedings of the National Academy of Sciences.
Alan S Cowen and Dacher Keltner. 2017. Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proceedings of the National Academy of Sciences, 114(38):E7900-E7909.
Alan S Cowen and Dacher Keltner. 2019. What the face displays: Mapping 28 emotions conveyed by naturalistic expression. American Psychologist.
Alan S Cowen, Petri Laukka, Hillary Anger Elfenbein, Runjing Liu, and Dacher Keltner. 2019b. The primacy of categories in the recognition of 12 emotions in speech prosody across two cultures. Nature Human Behaviour, 3(4):369.
CrowdFlower. 2016. https://www_figure-eight.com/data/sentiment-analysis-emotion-text/.
Rosario Delgado and Xavier-Andoni Tibau. 2019. Why cohen's kappa should be avoided as performance measure in classification. PloS one, 14(9).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In 17th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Maeve Duggan and Aaron Smith. 2013.  $6\%$  of online adults are reddit users. Pew Internet & American Life Project, 3:1-10.
Paul Ekman. 1992a. Are there basic emotions? Psychological Review, 99(3):550-553.
Paul Ekman. 1992b. An argument for basic emotions. Cognition & Emotion, 6(3-4):169-200.
Diman Ghazi, Diana Inkpen, and Stan Szpakowicz. 2015. Detecting Emotion Stimuli in Emotion-Bearing Sentences. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 152-165. Springer.
Chao-Chun Hsu and Lun-Wei Ku. 2018. SocialNLP 2018 EmotionX challenge overview: Recognizing emotions in dialogues. In Proceedings of the Sixth International Workshop on Natural Language Processing for Social Media, pages 27-31, Melbourne, Australia. Association for Computational Linguistics.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
Chen Liu, Muhammad Osama, and Anderson De Andrade. 2019. Dens: A dataset for multi-class emotion analysis. arXiv preprint arXiv:1910.11769.
Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 english words. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 174-184.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 task 1: Affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1-17, New Orleans, Louisiana. Association for Computational Linguistics.
Saif M Mohammad. 2012. # emotional tweets. In Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation, pages 246-255. Association for Computational Linguistics.
Saif M Mohammad, Xiaodan Zhu, Svetlana Kiritchenko, and Joel Martin. 2015. Sentiment, emotion, purpose, and style in electoral tweets. Information Processing & Management, 51(4):480-499.
Shruthi Mohan, Apala Guha, Michael Harris, Fred Popowich, Ashley Schuster, and Chris Priebe. 2017. The impact of toxic language on the health of reddit communities. In *Canadian Conference on Artificial Intelligence*, pages 51-56. Springer.
Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. 2008. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372-403.
Emily Ohman, Kaisla Kajava, Jorg Tiedemann, and Timo Honkela. 2018. Creating a dataset for multilingual fine-grained emotion-detection using gamification-based annotation. In Proceedings of the 9th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 24-30.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL).
Rosalind W Picard. 1997. Affective Computing. MIT Press.
Inna Pirina and Cagrì Cöltekin. 2018. Identifying depression on reddit: The effect of training data. In Proceedings of the 2018 EMNLP Workshop SMM4H: The 3rd Social Media Mining for Health Applications Workshop & Shared Task, pages 9-12.
Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In *Theories of emotion*, pages 3-33. Elsevier.
James A Russell. 2003. Core affect and the psychological construction of emotion. Psychological review, 110(1):145.
Klaus R Scherer and Harald G Wallbott. 1994. Evidence for Universality and Cultural Variation of Differential Emotion Response Patterning. Journal of personality and social psychology, 66(2):310.
Hendrik Schuff, Jeremy Barnes, Julian Mohme, Sebastian Padó, and Roman Klinger. 2017. Annotation, modelling and analysis of fine-grained emotions on a stance and sentiment detection corpus. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 13-23.
Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 task 14: Affective text. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 70-74, Prague, Czech Republic. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT Rediscover the Classical NLP Pipeline. In Association for Computational Linguistics.
Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and Practical BERT Models for Sequence Labeling. In EMNLP 2019.
Wenbo Wang, Lu Chen, Krishnaprasad Thirunarayan, and Amit P Sheth. 2012. Harnessing twitter 'big data' for automatic emotion identification. In 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing, pages 587-592. IEEE.
Cebrian-M. Yanardag, P. and I. Rahwan. 2018. Norman: World's first psychopath ai.
A Emotion Definitions
admiration Finding something impressive or worthy of respect.
amusement  $\odot$  Finding something funny or being entertained.
anger A strong feeling of displeasure or antagonism.
annoyance  $\odot$  Mild anger, irritation.
approval Having or expressing a favorable opinion.
caring Displaying kindness and concern for others. confusion Lack of understanding, uncertainty.
curiosity A strong desire to know or learn something.
desire A strong feeling of wanting something or wishing for something to happen.
disappointment Sadness or displeasure caused by the nonfulfillment of one's hopes or expectations.
disapproval Having or expressing an unfavorable opinion.
disgust Revulsion or strong disapproval aroused by something unpleasant or offensive.
embarrassment Self-consciousness, shame, or awkwardness.
excitement Feeling of great enthusiasm and eagerness.
fear  $\odot$  Beingafraidorworried.
gratitude A feeling of thankfulness and appreciation.
grief Intense sorrow, especially caused by someone's death.
joy A feeling of pleasure and happiness.
love  $\because$  A strong positive emotion of regard and affection.
nervousness Apprehension, worry, anxiety.
optimism Hopefulness and confidence about the future or the success of something.
pride Pleasure or satisfaction due to ones own achievements or the achievements of those with whom one is closely associated.
realization Becoming aware of something.
relief Reassurance and relaxation following release from anxiety or distress.
remorse Regret or guilty feeling.
sadness Emotional pain, sorrow.
surprise Feeling astonished, startled by something unexpected.
B Taxonomy Selection & Data Collection
We selected our taxonomy through a careful multi-round process. In the first pilot round of data col
lection, we used emotions that were identified to be salient by Cowen and Keltner (2017), making sure that our set includes Ekman's emotion categories, as used in previous NLP work. In this round, we also included an open input box where annotators could suggest emotion(s) that were not among the options. We annotated 3K examples in the first round. We updated the taxonomy based on the results of this round (see details below). In the second pilot round of data collection, we repeated this process with 2k new examples, once again updating the taxonomy.
While reviewing the results from the pilot rounds, we identified and removed emotions that were scarcely selected by annotators and/or had low interrater agreement due to being very similar to other emotions or being too difficult to detect from text. These emotions were boredom, doubt, heartbroken, indifference and calmness. We also identified and added those emotions to our taxonomy that were frequently suggested by raters and/or seemed to be represented in the data upon manual inspection. These emotions were desire, disappointment, pride, realization, relief and remorse. In this process, we also refined the category names (e.g. replacing ecstasy with excitement), to ones that seemed interpretable to annotators. This is how we arrived at the final set of 27 emotions + Neutral. Our high interrater agreement in the final data can be partially explained by the fact that we took interpretability into consideration while constructing the taxonomy. The dataset is we are releasing was labeled in the third round over the final taxonomy.
C Cohen's Kappa Values
In Section 4.1, we measure agreement between raters via Spearman correlation, following considerations by Delgado and Tibau (2019). In Table 7, we report the Cohen's kappa values for comparison, which we obtain by randomly sampling two ratings for each example and calculating the Cohen's kappa between these two sets of ratings. We find that all Cohen's kappa values are greater than 0, showing rater agreement. Moreover, the Cohen's kappa values correlate highly with the interrater correlation values (Pearson  $r = 0.85$ ,  $p < 0.001$ ), providing corroborative evidence for the significant degree of interrater agreement for each emotion.
Emotion	Interrater Correlation	Cohen's kappa
admiration	0.535	0.468
amusement	0.482	0.474
anger	0.207	0.307
annoyance	0.193	0.192
approval	0.385	0.187
caring	0.237	0.252
confusion	0.217	0.270
curiosity	0.418	0.366
desire	0.177	0.251
disappointment	0.186	0.184
disapproval	0.274	0.234
disgust	0.192	0.241
embarrassment	0.177	0.218
excitement	0.193	0.222
fear	0.266	0.394
gratitude	0.645	0.749
grief	0.162	0.095
joy	0.296	0.301
love	0.446	0.555
nervousness	0.164	0.144
optimism	0.322	0.300
pride	0.163	0.148
realization	0.194	0.155
relief	0.172	0.185
remorse	0.178	0.358
sadness	0.346	0.336
surprise	0.275	0.331
Table 7: Interrater agreement, as measured by interrater correlation and Cohen's kappa
D Sentiment of Reddit Subreddits