SlowGuess/ACL_2020_OCR huggingface dataset:
Aycan Kapucu, Aslı Kılıç, Yildiz Özkılıç, and Bengisu Sarıbaz. 2018. Turkish emotional word norms for arousal, valence, and discrete emotion categories. Psychological Reports, pages 1-22. [Available online Dec 4, 2018].
Sopan Khosla, Niyati Chhaya, and Kushal Chawla. 2018. AFF2VEC: Affect-enriched distributional word representations. In COLING 2018 - Proceedings of the 27th International Conference on Computational Linguistics, pages 2204-2218, Santa Fe, New Mexico, USA, August 20-26, 2018.
Diederik Kingma and Jimmy Ba. 2015. ADAM: A method for stochastic optimization. In ICLR 2015 — Proceedings of the 3rd International Conference on Learning Representations, San Diego, California, USA, May 7–9, 2015.
Svetlana Kiritchenko and Saif M. Mohammad. 2016. Capturing reliable fine-grained sentiment associations by crowdsourcing and best-worst scaling. In *NAACL-HLT* 2016 — Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811–817, San Diego, California, USA, June 12–17, 2016.
Maximilian Köper and Sabine Schulte im Walde. 2016. Automatically generated affective norms of abstractness, arousal, imageability and valence for 350,000 German lemmas. In LREC 2016 — Proceedings of the 10th International Conference on Language Resources and Evaluation, pages 2595–2598, Portoorž, Slovenia, May 23–28, 2016.
Agnieszka Landowska. 2018. Towards new mappings between emotion representation models. Applied Sciences, 8(2):#274.
Nicolas Leveau, Sandra Jhean-Larose, Guy Denhière, and Ba-Linh Nguyen. 2012. Validating an interlingual metanorm for emotional analysis of texts. Behavior Research Methods, 44(4):1007-1014.
Minglei Li, Qin Lu, Yunfei Long, and Lin Gui. 2017. Inferring affective meanings of words from word embedding. IEEE Transactions on Affective Computing, 8(4):443-456.
Ying Li, Tomas Engelthaler, Cynthia S. Q. Siew, and Thomas T. Hills. 2019. The MACROSCOPE: A tool for examining the historical structure of language. Behavior Research Methods, 51(4):1864-1877.
Saif Mohammad. 2018. Obtaining reliable human ratings of valence, arousal, and dominance for 20,000 English words. In ACL 2018 — Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 1: Long Papers, pages 174–184, Melbourne, Victoria, Australia, July 15–20, 2018.
Saif Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 Shared Task on Emotion Intensity. In WASSA 2017 — Proceedings of the 8th Workshop
on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis @ EMNLP 2017, pages 34-49, Copenhagen, Denmark, September 8, 2017.
Saif Mohammad, Felipe Bravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. 2018. SEMEval-2018 Task 1: Affect in Tweets. In SemEval 2018 — Proceedings of the 12th International Workshop on Semantic Evaluation @ NAACL-HLT 2018, pages 1–17, New Orleans, Louisiana, USA, June 5–6, 2018.
Saif M. Mohammad and Peter D. Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436-465.
Catherine Monnier and Arielle Syssau. 2008. Semantic contribution to verbal short-term memory: Are pleasant words easier to remember than neutral words in serial recall and serial recognition? Memory & Cognition, 36(1):35-42.
Maria Montefinese, Ettore Ambrosini, Beth Fairfield, and Nicola Mammarella. 2014. The adaptation of the Affective Norms for English Words (ANEW) for Italian. Behavior Research Methods, 46(3):887-903.
Agnes Moors, Jan De Houwer, Dirk Hermans, Sabine Wanmaker, Kevin van Schie, Anne-Laura Van Harmelen, Maarten De Schryver, Jeffrey De Winne, and Marc Brysbært. 2013. Norms of valence, arousal, dominance, and age of acquisition for 4,300 Dutch words. Behavior Research Methods, 45(1):169-177.
Roberto Navigli and Simone Paolo Ponzetto. 2012. BABELNET: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence, 193:217-250.
Elisavet Palogiannidi, Polychronis Koutsakis, Elias Iosif, and Alexandros Potamianos. 2016. Affective lexicon creation for the Greek language. In LREC 2016—Proceedings of the 10th International Conference on Language Resources and Evaluation, pages 2867–2872, Porto-rož, Slovenia, May 23–28, 2016.
Sungjoon Park, Jiseon Kim, Jaeyeol Jeon, Heeyoung Park, and Alice Oh. 2019. Toward dimensional emotion detection from categorical emotion annotations. arXiv:1911.02499 [cs, eess].
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournaepau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. SCIKIT-LEARN: Machine learning in Python. Journal of Machine Learning Research, 12(85):2825-2830.
Ana P. Pinheiro, Marcelo Dias, João Pedrosa, and Ana P. Soares. 2017. Minho Affective Sentences (MAS): Probing the roles of sex, mood, and empathy in affective ratings of verbal stimuli. Behavior Research Methods, 49(2):698-716.
Robert Plutchik. 1980. A general psychoevolutionary theory of emotion. In Robert Plutchik and Henry Kellerman, editors, Emotion: Theory, Research and Experience, volume 1: Theories of Emotion, pages 3-33. Academic Press, New York, NY, USA.
Jaime Redondo, Isabel Fraga, Isabel Padrón, and Montserrat Comesaña. 2007. The Spanish adaptation of ANEW (Affective Norms for English Words). Behavior Research Methods, 39(3):600-605.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of LSTM-networks for sequence tagging. In EMNLP 2017 — Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 338–348, Copenhagen, Denmark, September 9–11, 2017.
Monika Riegel, Małgorzata Wierzba, Marek Wypych, Łukasz Žurawski, Katarzyna Jednoróg, Anna Grabowska, and Artur Marchewka. 2015. Nencki Affective Word List (NAWL): The cultural adaptation of the Berlin Affective Word List—Reloaded (BAWL-R) for Polish. Behavior Research Methods, 47(4):1222-1236.
Sara Rosenthal, Preslav I. Nakov, Svetlana Kiritchenko, Saif M. Mohammad, Alan Ritter, and Veselin Stoyanov. 2015. SEMEval 2015 Task 10: Sentiment Analysis in Twitter. In SemEval 2015 — Proceedings of the 9th International Workshop on Semantic Evaluation @ NAACL-HLT 2015, pages 451-463, Denver, Colorado, USA, June 4-5, 2015.
Sascha Rothe, Sebastian Ebert, and Hinrich Schütze. 2016. Ultradense word embeddings by orthogonal transformation. In *NAACL* 2016 — Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 767–777, San Diego, California, USA, June 12–17, 2016.
David S. Schmidtke, Tobias Schröder, Arthur M. Jacobs, and Markus Conrad. 2014. ANGST: Affective Norms for German Sentiment Terms, derived from the Affective Norms for English Words. Behavior Research Methods, 46(4):1108-1118.
João Sedoc, Daniel Preoiciuc-Pietro, and Lyle H. Ungar. 2017. Predicting emotional word ratings using distributional representations and signed clustering. In EACL 2017 — Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 564–571, Valencia, Spain, April 3–7, 2017.
Samira Shaikh, Kit Cho, Tomek Strzalkowski, Laurie Feldman, John Lien, Ting Liu, and George Aaron
Broadwell. 2016. ANEW+: Automatic expansion and validation of Affective Norms of Words lexicons in multiple languages. In LREC 2016 - Proceedings of the 10th International Conference on Language Resources and Evaluation, pages 1127-1132, Porto Roz, Slovenia, May 23-28, 2016.
Agnes Sianipar, Pieter van Groenestijn, and Ton Dijkstra. 2016. Affective meaning, concreteness, and subjective frequency norms for Indonesian words. Frontiers in Psychology, 7:#1907.
Ana Paula Soares, Montserrat Comesana, Ana P. Pinheiro, Alberto Simões, and Carla Sofia Frade. 2012. The adaptation of the Affective Norms for English Words (ANEW) for European Portuguese. Behavior Research Methods, 44(1):256-269.
Hans Stadthagen-González, Pilar Ferré, Miguel A. Pérez-Sánchez, Constance Imbault, and José Antonio Hinojosa. 2018. Norms for 10,491 Spanish words for five discrete emotions: Happiness, disgust, anger, fear, and sadness. Behavior Research Methods, 50(5):1943-1952.
Hans Stadthagen-Gonzalez, Constance Imbault, Miguel A. Perez-Sanchez, and Marc Brysbart. 2017. Norms of valence and arousal for 14,031 Spanish words. Behavior Research Methods, 49(1):111-123.
Ryan A. Stevenson, Joseph A. Mikels, and Thomas W. James. 2007. Characterization of the Affective Norms for English Words by discrete emotional categories. Behavior Research Methods, 39(4):1020-1024.
Carlo Strapparava and Alessandro Valitutti. 2004. WORDNET-AFFECT: An affective extension of WORDNET. In LREC 2004 - Proceedings of the 4th International Conference on Language Resources and Evaluation, pages 1083-1086, Lisbon, Portugal, May 24-30, 2004.
Enrica Troiano, Sebastian Padó, and Roman Klinger. 2019. Crowdsourcing and validating event-focused emotion corpora for German and English. In ACL 2019 - Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4005-4011, Florence, Italy, July 28 - August 2, 2019.
Peter D. Turney and Michael L. Littman. 2003. Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 21(4):315-346.
Melissa L.-H. Võ, Markus Conrad, Lars Kuchenke, Karolina Urton, Markus J. Hofmann, and Arthur M. Jacobs. 2009. The Berlin Affective Word List Reloaded (BAWL-R). Behavior Research Methods, 41(2):534-538.
Jin Wang, Liang-Chih Yu, K. Robert Lai, and Xuejie Zhang. 2016. Community-based weighted graph model for valence-arousal prediction of affective
words. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(11):1957-1968.
Amy Beth Warriner, Victor Kuperman, and Marc Brysbert. 2013. Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods, 45(4):1191-1207.
Małgorzata Wierzba, Monika Riegel, Marek Wypych, Katarzyna Jednoróg, Paweł Turnau, Anna Grabowska, and Artur Marchewka. 2015. Basic emotions in the Nencki Affective Word List (NAWL BE): New method of classifying emotional stimuli. PLoS ONE, 10(7):#e0132305.
Zhao Yao, Jia Wu, Yanyan Zhang, and Zhenhong Wang. 2017. Norms of valence, arousal, concreteness, familiarity, imageability, and context availability for 1,100 Chinese words. Behavior Research Methods, 49(4):1374-1385.
Liang-Chih Yu, Lung-Hao Lee, Shuai Hao, Jin Wang, Yunchao He, Jun Hu, K. Robert Lai, and Xuejie Zhang. 2016. Building Chinese affective resources in valence-arousal dimensions. In *NAACL-HLT* 2016 — Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 540–545, San Diego, California, USA, June 12–17, 2016.
Liang-Chih Yu, Jin Wang, K. Robert Lai, and Xuejie Zhang. 2017. Refining word embeddings for sentiment analysis. In EMNLP 2017 - Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 534-539, Copenhagen, Denmark, September 9-11, 2017.
Feng Zhou, Shu Kong, Charless C. Fowlkes, Tao Chen, and Baiying Lei. 2020. Fine-grained facial expression analysis using dimensional emotion model. Neurocomputing. [Available online Jan 23, 2020].
The exact design of the Source train-dev-test split is as follows: All entries (words plus ratings) from all splits are taken from Warriner et al. (2013). The data was then partitioned based on the overlap with the two precursory versions by Bradley and Lang (1999) (the original ANEW) and Bradley and Lang (2010) (an early extended version of ANEW roughly twice as large). Source-test was built by intersecting the lexicon from Warriner et al. (2013) with the original ANEW. A similar process was applied for Source-dev: we intersected the words from Warriner et al. (2013) and Bradley and Lang (2010) and removed the ones present in Source-test. Lastly, Source-train is made up by all words from Warriner et al. (2013) which are neither in Source-test nor in Source-dev. The reason why the ratings in Source are taken exclusively from Warriner et al. (2013) is that these are distributed under a more permissive license compared to their precursors.
We removed multi-token entries (e.g., boa constructor) and entries with upper case characters (e.g., Budweiser) from all data splits of Source, thus restricting the lexicon to single-token, nonproper noun entries to make it more suitable for word embedding-based research. All splits combined have 13,791 entries (train: 11,463, dev: 1,296, test: 1,032), thus removing less than  $1\%$  from the original lexicon.
Regarding the remaining gold standards, the only cases which needed additional preparation or cleansing steps were zh1 (Yu et al., 2016) and zh2 (Yao et al., 2017). zh1 was created and is distributed using traditional Chinese characters, whereas the embedding model by Grave et al. (2018) employs simplified ones. Therefore, we converted zh1 into simplified characters using GOOGLE TRANSLATE prior to evaluation.
While manually examining the zh2 lexicon, we noticed several cases where the ratings seemed rather counter-intuitive (e.g., seemingly positive words which received very negative ratings). We contacted the authors who confirmed the problem and sent us a corrected version. We did not find any such problems in the second version. We consulted
with a Chinese native speaker for both of these procedures regarding the zh1 and zh2 lexicons.
A.2 Model Training and Implementation
Training of the MTLFFN model closely followed the procedure specified by Buechel and Hahn (2018b): For each language, the model was trained for roughly 15k iterations (exactly 168 epochs) with a batch size of 128 using the Adam optimizer (Kingma and Ba, 2015) with learning rate  $10^{-3}$ , and .5 dropout on the hidden layers and .2 on the input layer. As nonlinear activation function we used leaky ReLU with "leakage" of 0.01.
Embedding vectors are the only model input. They have 300 dimensions for every language, independent of their respective training data size (Grave et al., 2018). Since the automatic translation of Source is not guaranteed to result in single-word translations, we use the following workaround to derive embedding vectors for multi-token translations: If the translation as a whole cannot be found in the embedding model, the multi-token term gets split up into its constituent parts, using spaces, apostrophes or hyphens as separators. Each substring is looked up in the embedding model, the averaged vector is taken as input. If no substring is recognized, we use the zero vector instead. We also use the zero vector for single-token entries in TargetMT that are missing in the embeddings.
Since Buechel and Hahn (2018b) considered only VAD but not BE5 datasets, we conducted a development experiment on the TargetMT-dev sets for all 91 languages where we assessed whether MTL is advantageous for BE5 variables as well, or for a combination of VAD and BE5 variables. We found that MTL improved performance when applied separately among all VAD and BE5 variables. Yet, when jointly learning all eight emotion variables, the results were somewhat inconclusive. Performance increased for BE5, but decreased for VAD. Hence, for lexicon creation, we took a cautious approach and trained two separate models per language, one for VAD, the other for BE5. An analysis of MTL across VAD and BE5 is left for future work.
The MTLFFN model is implemented in PYTORCH, adapting part of the TENSORFLOW code from Buechel and Hahn (2018b). The ridge regression baseline model is implemented with SCIKIT-LEARN (Pedregosa et al., 2011) using default parameters.
5The data split is available at: https://github.com/JULIELab/XANEW
In this case the regular Web application, not the API, was used: https://translate.google.com/
Table 8: Overview of generated emotion lexicons with silver evaluation results; sorted by Mean performance over the eight emotional variables.