SlowGuess/NAACL_2021_OCR huggingface dataset:
As for BERT-based models, we used the implementation in Transformers $^{17}$  (Wolf et al., 2020). We used the whole-word-masking model with a batch size of 32, a dropout rate of 0.1, a learning rate of 2e-5, and Adam (Kingma and Ba, 2015) for optimization. The training stopped after 3 epochs without improvement in the validation loss.
In the evaluation of subjective labels, the personality of the writers is considered in the Subj. BERT in the following two ways.
-  $\mathbf{w} / \mathbf{Pc}$ : Feature extraction is performed with  $h_c = [u; v]W^c$  in consideration of personality. Here,  $\mathbf{v}$  is a 768-dimensional text representation obtained from the [CLS] token of
BERT, and  $\pmb{u}$  is a representation of the Big Five personality indicators given by linearly transforming the five indicator values into a 768-dimensional vector. When estimating the emotional intensity,  $h_c$  is used instead of  $\pmb{h}$ .
-  $\mathbf{w} / \mathbf{Pa}$ : Feature extraction is performed with  $h_a =$  attention  $(\pmb{u}\pmb{W}^Q,\pmb{v}\pmb{W}^K,\pmb{v}\pmb{W}^V)$  in consideration of personality. That is, in the calculation of the attention mechanism, the personality representation  $\pmb{u}$  is used as the query, and the text representation  $\pmb{v}$  is used as both the key and the value.  $h_a$  is used instead of  $\pmb{h}$  for emotional intensity estimation.
The performance of each model on subjective and objective labels is shown in Tables 6 and 7, respectively. Regardless of the method, the evaluation of subjective label estimation gets a larger mean absolute error than the evaluation of objective labels. In our previous discussion, we have stated that it is difficult for readers to estimate the emotions of writers; this also applies to machine learning models.
5.2.1 Evaluation with Subjective Labels
In the evaluation of subjective labels, the traditional models of BoW+LogReg and fastText+SVM achieved lower mean absolute errors than the Random baseline, but were inferior to the Modal Class baseline. The BERT methods achieved a mean absolute error lower than the Modal Class baseline. Surprisingly, Obj. BERT trained with objective labels, rather than Subj. BERT trained with subjective labels, achieved the highest performance.
$^{16}$ https://scikit-learn.org/  
 $^{17}$ https://github.com/huggingface/transformers
Objective labels	MAE	QWK
Joy	Sadness	Anticipation	Surprise	Anger	Fear	Disgust	Trust	Overall	Overall
Random	1.353	1.333	1.333	1.291	1.516	1.354	1.391	1.450	1.378	0.001
Modal Class	0.595	0.459	0.713	0.518	0.044	0.420	0.383	0.026	0.395	0.000
BoW+LogReg	0.560	0.460	0.606	0.525	0.064	0.432	0.412	0.057	0.390	0.277
fastText+SVM	0.595	0.459	0.713	0.518	0.059	0.406	0.383	0.051	0.398	0.040
Subj. BERT	0.489	0.446	0.685	0.518	0.044	0.414	0.381	0.039	0.377	0.209
Obj. BERT	0.403	0.411	0.475	0.442	0.044	0.386	0.348	0.024	0.317	0.442
Reader 1	0.222	0.246	0.277	0.276	0.035	0.233	0.226	0.053	0.196	0.830
Reader 2	0.224	0.264	0.268	0.349	0.045	0.277	0.269	0.027	0.215	0.769
Reader 3	0.237	0.241	0.332	0.360	0.046	0.346	0.310	0.021	0.237	0.808
Table 7: Evaluation of MAE and QWK in estimating objective emotional intensity.
Since it is difficult to estimate subjective labels, which are the emotion of the writer, a simple model may not provide sufficient performance.
Therefore, we examined Subj. BERT w/ Pc and Subj. BERT w/ Pa to assist training using the personality information of the writer. As a result, Subj. BERT w/ Pc, which simply concatenates the personality representation and the text representation, was not effective, but Subj. BERT w/Pa, which considers personality representation with weighting, achieved higher performance than simple Subj. BERT. The evaluation by QWK also shows the usefulness of using the personality information of the writer. However, even with personality information, the performance is not comparable with that of Obj. BERT. Improving methods for accurate estimation of subjective emotions is our future work.
Below the dotted line in Table 6, the performance of the human readers is shown for comparison. Estimating the emotional intensity of writers is difficult for both human readers and machine learning models.
5.2.2 Evaluation with Objective Labels
In the evaluation of objective labels (Table 7), the traditional models of BoW+LogReg and fast-Text+SVM were comparable to the Modal Class baseline. Similar to the evaluation in the subjective labels, the BERT-based models achieved mean absolute errors lower than the Modal Class baseline, and Obj. BERT achieved the highest performance.
Below the dotted line in Table 7, the performance of the human readers is shown for comparison. Note that the objective labels are the average of each of these readers. Compared to each reader, Obj. BERT does not reach human performance.
We introduce a new dataset, WRIME, for Japanese emotional intensity estimation. Our dataset is based on Plutchik's eight emotions (Plutchik, 1980), labeling both the writer's subjective emotional intensity and the reader's objective one in SNS posts.
Overall, the readers tend to underestimate the writer's emotions. Even the strong emotions of the writer cannot be detected by the reader, especially in the emotions of anger and trust.
Experimental results on emotional intensity estimation show that it is more difficult to estimate the writer's subjective labels than the readers'. The large gap between the subjective and objective emotions imply the complexity of the mapping from a text to the subjective emotional intensities, which also leads to a lower performance with machine learning models.
Estimating the writer's subjective emotions with higher accuracy is future work. We have shown the possibility of improving the performance of subjective emotional intensity estimation by considering the personality of the writer. It may be worth considering the writer's meta information, including personality, and the writer's past posting history.
We ensure that our work is conformant to the ACM Code of Ethics.
This work was supported by Innovation Platform for Society 5.0 from Japan Ministry of Education, Culture, Sports, Science and Technology.
Francisca Adoma Acheampong, Chen Wenyu, and Henry Nunoo-Mensah. 2020. Text-based Emotion Detection: Advances, Challenges, and Opportunities. Engineering Reports, 2(7):1-24.
Saima Aman and Stan Szpakowicz. 2007. Identifying Expressions of Emotion in Text. In International Conference on Text, Speech and Dialogue, pages 196-205.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, 5:135-146.
Laura-Ana-Maria Bostan, Evgeny Kim, and Roman Klinger. 2020. GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1554-1566.
Laura-Ana-Maria Bostan and Roman Klinger. 2018. An Analysis of Annotated Corpora for Emotion Classification in Text. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2104-2119.
Sven Buechel and Udo Hahn. 2017. EmoBank: Studying the Impact of Annotation Perspective and Representation Format on Dimensional Emotion Analysis. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 578-585.
Jacob Cohen. 1968. Weighted Kappa: Nominal Scale Agreement Provision for Scaled Disagreement or Partial Credit. Psychological Bulletin, 70(4):213-220.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171-4186.
Paul Ekman. 1992. An Argument for Basic Emotions. Cognition and Emotion, 6(3-4):169-200.
Lewis R. Goldberg. 1992. The Development of Markers for the Big-Five Factor Structure. Psychological Assessment, 4(1):26-42.
Nobuhiro Kaji and Masaru Kitsuregawa. 2006. Automatic Construction of Polarity-Tagged Corpus from HTML Documents. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 452-459.
Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations.
Taku Kudo, Kaoru Yamamoto, and Yuji Matsumoto. 2004. Applying Conditional Random Fields to Japanese Morphological Analysis. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 230-237.
Saif Mohammad and Felipe Bravo-Marquez. 2017a. Emotion Intensities in Tweets. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics, pages 65-77.
Saif Mohammad and Felipe Bravo-Marquez. 2017b. WASSA-2017 Shared Task on Emotion Intensity. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 34-49.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SemEval-2018 Task 1: Affect in Tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 1-17.
Saif Mohammad and Svetlana Kiritchenko. 2018. Understanding Emotions: A Dataset of Tweets to Study Interactions between Affect Categories. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, pages 198-209.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournaepau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85):2825-2830.
Robert Plutchik. 1980. A General Psychoevolutionary Theory of Emotion. Theories of Emotion, 1:3-31.
James A. Russell. 1980. A Circumplex Model of Affect. Journal of Personality and Social Psychology, 39(6):1161-1178.
Takako Saito, Tomoyasu Nakamura, Toshihiko Endo, and Madoka Yokoyama. 2001. Standardization of Big Five Scales Using the Adjective Check List. Kyushu University Psychological Research, 2:135-144.
Klaus R. Scherer and Harald G. Wallbott. 1994. Evidence for Universality and Cultural Variation of Differential Emotion Response Patterning. Journal of Personality and Social Psychology, 66(2):310-328.
Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment
Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642.
Stefan Stieglitz and Linh Dang-Xuan. 2013. Emotions and Information Diffusion in Social Media — Sentiment of Microblogs and Sharing Behavior. Journal of Management Information Systems, 29(4):217-248.
Carlo Strapparava and Rada Mihalcea. 2007. SemEval-2007 Task 14: Affective Text. In Proceedings of the Fourth International Workshop on Semantic Evaluations, pages 70-74.
Yu Suzuki. 2019. Filtering Method for Twitter Streaming Data Using Human-in-the-Loop Machine Learning. Journal of Information Processing, 27:404-410.
Ryoko Tokuhisa, Kentaro Inui, and Yuji Matsumoto. 2008. Emotion Classification Using Massive Examples Extracted from the Web. In Proceedings of the 22nd International Conference on Computational Linguistics, pages 881-888.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45.