SlowGuess/ACL_2020_OCR huggingface dataset:
Embeddings. We use the fastText embedding models from Grave et al. (2018) trained for 157 languages on the respective WIKIPEDIA and the respective part of COMMONCRAWL. These resources not only greatly facilitate our work but also increase comparability across languages. The restriction to "only" 91 languages comes from intersecting the ones covered by the vectors with the languages covered by the translation service.
Models. Since our proposed methodology is ag- nostic towards the chosen word emotion model, we will re-use models from the literature. In particular, we will rely on the multi-task learning feed-forward network (MTLFFN) worked out by Buechel and Hahn (2018b). This network constitutes the current state of the art for monolingual emotion lexicon creation (expanding an existing lexicon for a given language) for many of the datasets in Table 2.
The MTLFFN has two hidden layers of 256 and 128 units, respectively, and takes pre-trained embedding vectors as input. Its distinguishing feature is that hidden layer parameters are shared between the different emotion target variables, thus constituting a mild form of multi-task learning (MTL). We apply MTL to VAD and BE5 variables individually (but not between both groups), thus training two distinct emotion models per language, following the outcome of a development experiment. Details are given in Appendix A.2 together with the remainder of the model specifications.
Being aware of the infamous instability of neural approaches (Reimers and Gurevych, 2017), we also employ a ridge regression model, an  $L_{2}$  regularized version of linear regression, as a more robust, yet also powerful baseline (Li et al., 2017).
The size of the resulting lexicons (a complete list is provided in Table 8 in the Appendix) ranges from roughly  $100\mathrm{k}$  to more than 2M entries mainly depending on the vocabulary of the respective embeddings. We want to point out that not every single entry should be considered meaningful because of noise in the embedding vocabulary caused by typos and tokenization errors. However, choosing the "best" size for an emotion lexicon necessarily translates into a quality-coverage trade-off for which there is no general solution. Instead, we release the full-size lexicons and leave it to prospective users to apply any sort of filtering they deem appropriate.
Silver Evaluation. Figure 2 displays the results of our silver evaluation. Languages (x-axis) are sorted by their average performance over all variables (not shown in the plot; tabular data given in the Appendix). As can be seen, the evaluation results for English are markedly better than for any other language. This is not surprising since no (potentially error-prone) machine translation was performed. Apart from that, performance remains relatively stable across most of the languages and
4https://cloud.google.com/translate/
Figure 2: Silver evaluation results in Pearson's  $r$ . Languages (x-axis) are sorted according to mean correlation.
starts degrading more quickly only for the last third of them. In particular, for Valence—typically the easiest variable to predict—we achieve a strong performance of  $r > .7$  for 56 languages. On the other hand, for Arousal—typically, the most difficult one to predict—we achieve a solid performance of  $r > .5$  for 55 languages. Dominance and the discrete emotion variables show performance trajectories swinging between these two extremes. We assume that the main factors for explaining performance differences between languages are the quality of the translation and embedding models which, in turn, both depend on the amount of available text data (parallel or monolingual, respectively).
Comparing MTLFFN and ridge baseline, we find that the neural network reliably outperforms the linear model. On average over all languages and variables, the MTL models achieve  $6.7\%$  -points higher Pearson correlation. Conversely, ridge regression outperforms MTLFFN in only 15 of the total 728 cases (91 languages  $\times 8$  variables).
Gold Evaluation. Results for VAD variables on gold data are given in Table 3. As can be seen, our lexicons show a good correlation with human judgment and do so robustly, even for less-resourced languages, such as Indonesian (id), Turkish (tr), or Croatian (hr), and across affective variables. Perhaps the strongest negative outliers are the Arousal results for the two Chinese datasets (zh), which are likely to result from the low reliability of the gold ratings (see below).
ID	Shared	(%)	Val	Aro	Dom
en1	1032	100	.94 (.87)	.76 (.67)	.88 (.76)
en2	1034	100	.92 (.92)	.71 (.73)	.78 (.82)
es1	612	59	.91 (.88)	.71 (.70)	.82 (.83)
es2	7685	54	.79 (.82)	.64 (.74)	—
es3	363	41	.91	.73	—
de1	677	67	.89 (.87)	.78 (.80)	.68 (.74)
de2	2329	80	.75	.64	—
de3	916	91	.80	.67	—
pl1	2271	46	.83 (.74)	.74 (.70)	.60 (.69)
pl2	1381	47	.82	.61	—
zh1	1685	60	.84 (.85)	.56 (.63)	—
zh2	701	63	.84	.44	—
it	660	58	.89 (.86)	.63 (.65)	.76 (.75)
pt	645	62	.89 (.86)	.71 (.71)	.75 (.73)
nl	2064	48	.85 (.79)	.58 (.74)	—
id	696	46	.84 (.80)	.64 (.60)	.63 (.58)
el	633	61	.86	.50	.74
tr1	721	35	.75	.57	—
hr	1331	44	.81	.66	—
Mn (all)	.85	.65	.74
Mn (vs. monolingual)	.87 (.84)	.68 (.70)	.74 (.74)
Table 3: Gold evaluation results for VAD (Valence, Arousal, Dominance) in Pearson's  $r$ . Parentheses give comparative monolingual results from Buechel and Hahn (2018b). Shared words between TargetGold and TargetPred-test; (\%) : percentage relative to TargetGold; Mn (all): mean over all datasets; Mn (vs. monolingual): mean over datasets with comparative results.
We compare these results against those from Buechel and Hahn (2018b) which were acquired on the respective TargetGold dataset in a monolingual fashion using 10-fold cross-validation (10-
ID	Shared	(%)	Joy	Ang	Sad	Fea	Dis
en3	1033	99	.89	.83	.80	.82	.78
es4	363	41	.86	.84	.84	.84	.76
es5	6096	58	.64	.72	.72	.72	.63
es6	992	43	.80	.74	.71	.72	.68
de4	848	43	.80	.66	.52	.68	.42
pl3	1381	47	.78	.71	.66	.69	.71
tr2	721	35	.77	.69	.71	.70	.65
Mean	.79	.74	.71	.74	.66
CV). We admit that those results are not fully comparable to those presented here because we use fixed splits rather than 10-CV. Nevertheless, we find that the results of our cross-lingual set-up are more than competitive, outperforming the monolingual results from Buechel and Hahn (2018b) in 17 out of 30 cases (mainly for Valence and Dominance, less often for Arousal). This is surprising since we use an otherwise identical model and training procedure. We conjecture that the large size of the English Source lexicon, compared to most TargetGold lexicons, more than compensates for error-prone machine translation.
Table 4 shows the results for BE5 datasets which are in line with the VAD results. Regarding the ordering of the emotional variables, again, we find Valence to be the easiest one to predict, Arousal the hardest, whereas basic emotions and Dominance take a middle ground.
Comparison against Human Reliability. We base this analysis on inter-study reliability (ISR), a rather strong criterion for human performance. ISR is computed, per variable, as the correlation between the ratings from two distinct annotation studies (Warriner et al., 2013). Hence, this analysis is restricted to languages where more than one gold lexicon exists per emotion format. We intersect the entries from both gold standards as well as the respective TargetPred-test set and compute the correlation between all three pairs of lexicons. If our lexicon agrees more with one of the gold standards than the two gold standards agree with each other, we consider this as an indicator for superhuman reliability (Buechel and Hahn, 2018b).
As shown in Table 5, our lexicons are often competitive with human reliability for Valence (especially for English and Chinese), but outperform
Table 4: Gold evaluation results for BE5 (Joy, Anger, Sadness, Fear, Disgust) in Pearson's  $r$ . Shared words between TargetGold and TargetPred-test; (%): percentage relative to TargetGold; Mean over all datasets.
Gold1	Gold2	Shared	Emo	G1vsG2	G1vsPr
G1vsPr	G2vsPr
en1	en2	1032	V	.953	.941	.922
A	.760	.761	.711
D	.794	.879	.782
es1	es2	610	V	.976	.905	.912
A	.758	.714	.725
es2	es3	222	V	.976	.906	.907
A	.710	.724	.691
de2	de3	498	V	.963	.806	.812
A	.760	.721	.663
pl1	pl2	445	V	.943	.838	.852
A	.725	.764	.643
zh1	zh2	140	V	.932	.918	.898
A	.482	.556	.455
Table 5: Comparison against human performance. Correlation between two gold standards, Gold1 and Gold2, with each other (G1vsG2), as well as with our lexicons TargetPred-test (G1vsPr and G2vsPr) relative to Emotional variable and Shared number of words.
human reliability in 4 out of 6 cases for Arousal, and in the single test case for Dominance. There are no cases of overlapping gold standards for BE5.
6 Methodological Assumptions Revisited
This section investigates patterns in prediction quality across languages, validating design decisions of our methodology.
Translation vs. Prediction. Is it beneficial to predict new ratings for the words in TargetMT rather than using them as final lexicon entries straight away? For each TargetGold lexicon (cf. Table 2), we intersect its word material with that in TargetMT and TargetPred. Then, we compute the correlation between TargetPred and TargetMT with the gold standard. This analysis was done on the respective train sets because using TargetMT rather than TargetPred is only an option for entries known at training time.
Table 6 depicts the results of this comparison averaged over all gold lexicons. As hypothesized, the TargetPred lexicons agree, on average, more with human judgment than the TargetMT lexicons, suggesting that the word emotion model acts as a value-adding post-processor, partly mitigating rating inconsistencies introduced by mere translation of the lexicons. The observation holds for each individual emotion variable with particularly large benefits for Arousal, where the post-processed TargetPred lexicons are on average
Val	Aro	Dom	Joy	Ang	Sad	Fea	Dis
Pred	.871	.652	.733	.767	.734	.692	.728	.650
MT	.796	.515	.613	.699	.677	.636	.654	.579
Diff	.076	.137	.119	.068	.057	.056	.074	.071
14%-points better compared to the translation-only TargetMT lexicons. This seems to indicate that lexical Arousal is less consistent between translational equivalents compared to other emotional meaning components like Valence and Sadness, which appear to be more robust against translation.
Gold vs. Silver Evaluation. How meaningful is silver evaluation without gold data? We compute the Pearson correlation between gold and silver evaluation results across languages per emotion variable. For languages where we consider multiple datasets during gold evaluation, we first average the gold evaluation results for each emotion variable. As can be seen from Table 7, the correlation values range between  $r = .91$  for Joy and  $r = .27$  for Disgust. This relatively large dispersion is not surprising when we take into account that we correlate very small data series (for Valence and Arousal there are just 12 languages for which both gold and silver evaluation results are available; for BE5 there are only 5 such languages). However, the mean over all correlation values in Table 7 is .64, indicating that there is a relatively strong correlation between both types of evaluation. This suggests that the silver evaluation may be used as a rather reliable proxy of lexicon quality even in the absence of language-specific gold data.
Table 6: Quality of TargetMT vs. TargetPred in terms of average Pearson correlation over all languages and gold standards. Diff := Pred - MT.
Val	Aro	Dom	Joy	Ang	Sad	Fea	Dis
#Lg	12	12	8	5	5	5	5	5
r	.54	.57	.52	.91	.85	.57	.87	.27
Table 7: Agreement between gold and silver evaluation across languages in Pearson's  $r$  relative to the number of applicable languages ("#Lg").
Emotion lexicons are at the core of sentiment analysis, a rapidly flourishing field of NLP. Yet, despite large community efforts, the coverage of existing lexicons is still limited in terms of languages, size,
and types of emotion variables. While there are techniques to tackle these three forms of sparsity in isolation, we introduced a methodology which allows us to cope with them simultaneously by jointly combining emotion representation mapping, machine translation, and embedding-based lexicon expansion.
Our study is "large-scale" in many respects. We created representationally complex lexicons—comprising 8 distinct emotion variables—for 91 languages with up to 2 million entries each. The evaluation of the generated lexicons featured 26 manually annotated datasets spanning 12 diverse languages. The predicted ratings showed consistently high correlation with human judgment, compared favorably with state-of-the-art monolingual approaches to lexicon expansion and even surpassed human inter-study reliability in some cases.
The sheer number of test sets we used allowed us to validate fundamental methodological assumptions underlying our approach. Firstly, the evaluation procedure, which is integrated into the generation methodology, allows us to reliably estimate the quality of resulting lexicons, even without target language gold standard. Secondly, our data suggests that embedding-based word emotion models can be used as a repair mechanism, mitigating poor target-language emotion estimates acquired by simple word-to-word translation.
Future work will have to deepen the way we deal with word sense ambiguity by way of exchanging the simplifying type-level approach our current work is based on with a semantically more informed sense-level approach. A promising direction would be to combine a multilingual sense inventory such as BABELNET (Navigli and Ponzetto, 2012) with sense embeddings (Camacho-Collados and Pilehvar, 2018).
We would like to thank the anonymous reviewers for their helpful suggestions and comments, and Tinghui Duan, JULIE LAB, for assisting us with the Chinese gold data. This work was partially funded by the German Federal Ministry for Economic Affairs and Energy (funding line "Big Data in der makroökonomischen Analyse" [Big data in macroeconomic analysis]; Fachlos 2; GZ 23305/003#002).
Mohamed Abdalla and Graeme Hirst. 2017. Crosslingual sentiment analysis without (good) translation. In IJCNLP 2017 - Proceedings of the 8th International Joint Conference on Natural Language Processing, volume 1: Long Papers, pages 506-515, Taipei, Taiwan, November 27 - December 1, 2017.
Muhammad Abdul-Mageed and Lyle H. Ungar. 2017. EmoNet: Fine-grained emotion detection with gated recurrent neural networks. In ACL 2017 - Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, volume 1: Long Papers, pages 718-728, Vancouver, British Columbia, Canada, July 30 - August 4, 2017.
Soraia M. Alarcão and Manuel J. Fonseca. 2017. Identifying emotions in images from valence and arousal ratings. Multimedia Tools and Applications, 77(13):17413-17435.
Stefano Baccianella, Andrea Esuli, and Fabrizio Sebastiani. 2010. SENTIWORDNET 3.0: An enhanced lexical resource for sentiment analysis and opinion mining. In LREC 2010 — Proceedings of the 7th International Conference on Language Resources and Evaluation, pages 2200–2204, La Valletta, Malta, May 17–23, 2010.
Jeremy Barnes, Roman Klinger, and Sabine Schulte im Walde. 2018. Bilingual sentiment embeddings: Joint projection of sentiment across languages. In ACL 2018 — Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, volume 1: Long Papers, pages 2483–2493, Melbourne, Victoria, Australia, July 15–20, 2018.
Jeremy Barnes, Samia Touileb, Lilja Øvrelid, and Erik Velldal. 2019. Lexicon information in neural sentiment analysis: a multi-task learning approach. In NoDaLiDa 2019 — Proceedings of the 22nd Nordic Conference on Computational Linguistics, pages 175–186, Turku, Finland, September 30 – October 2, 2019.
Yves Bestgen. 2008. Building affective lexicons from specific corpora for automatic sentiment analysis. In LREC 2008 — Proceedings of the 6th International Conference on Language Resources and Evaluation, pages 496-500, Marrakesh, Morocco, May 28-30, 2008.
Laura-Ana-Maria Bostan and Roman Klinger. 2018. An analysis of annotated corpora for emotion classification in text. In COLING 2018 - Proceedings of the 27th International Conference on Computational Linguistics, pages 2104-2119, Santa Fe, New Mexico, USA, August 20-26, 2018.
Margaret M. Bradley and Peter J. Lang. 1994. Measuring emotion: The Self-Assessment Manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1):49-59.
Margaret M. Bradley and Peter J. Lang. 1999. Affective norms for English words (ANEW): Stimuli, instruction manual and affective ratings. Technical Report C-1, The Center for Research in Psychophysiology, University of Florida, Gainesville, Florida, USA.
Margaret M. Bradley and Peter J. Lang. 2010. Affective norms for English words (ANew): Stimuli, instruction manual and affective ratings. Technical Report C-2, University of Florida, Gainesville, Florida, USA.
Benny B. Briesemeister, Lars Kuchinke, and Arthur M. Jacobs. 2011. Discrete Emotion Norms for Nouns: Berlin Affective Word List (DENN-BAWL). Behavior Research Methods, 43(2):#441.
Sven Buechel and Udo Hahn. 2016. Emotion analysis as a regression problem: Dimensional models and their implications on emotion representation and metrical evaluation. In ECAI 2016 — Proceedings of the 22nd European Conference on Artificial Intelligence, pages 1114–1122, The Hague, The Netherlands, August 29 – September 2, 2016.
Sven Buechel and Udo Hahn. 2017. EMOBANK: Studying the impact of annotation perspective and representation format on dimensional emotion analysis. In *EACL 2017 - Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics*, volume 2: Short Papers, pages 578-585, Valencia, Spain, April 3-7, 2017.
Sven Buechel and Udo Hahn. 2018a. Emotion representation mapping for automatic lexicon construction (mostly) performs on human level. In COLING 2018 - Proceedings of the 27th International Conference on Computational Linguistics, pages 2892-2904, Santa Fe, New Mexico, USA, August 20-26, 2018.
Sven Buechel and Udo Hahn. 2018b. Word emotion induction for multiple languages as a deep multi-task learning problem. In *NAACL-HLT* 2018 — Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1: Long Papers, pages 1907–1918, New Orleans, Louisiana, USA, June 1–6, 2018.
Sven Buechel, Johannes Hellrich, and Udo Hahn. 2016. Feelings from the past: Adapting affective lexicons for historical emotion analysis. In LT4DH 2016 — Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities @ COLING 2016, pages 54–61, Osaka, Japan, December 11, 2016.
Rafael A. Calvo and Sunghwan Mac Kim. 2013. Emotions in text: Dimensional and categorical models. Computational Intelligence, 29(3):527-543.
Jose Camacho-Collados and Mohammad Taher Pilehvar. 2018. From word to sense embeddings: A survey on vector representations of meaning. Journal of Artificial Intelligence Research, 63:743-788.
Yanqing Chen and Steven Skiena. 2014. Building sentiment lexicons for all major languages. In ACL 2014 — Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 2: Short Papers, pages 383–389, Baltimore, Maryland, USA, June 23–25, 2014.
Ta-Chung Chi and Yun-Nung Chen. 2018. CLUSE: Cross-Lingual Unsupervised Sense Embeddings. In EMNLP 2018 - Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 271-281, Brussels, Belgium, October 31 - November 4, 2018.
Paul Cook and Suzanne Stevenson. 2010. Automatically identifying changes in the semantic orientation of words. In LREC 2010 - Proceedings of the 7th International Conference on Language Resources and Evaluation, pages 28-34, La Valletta, Malta, May 17-23, 2010.
Bojana Coso, Marc Guasch, Pilar Ferre, and José Antonio Hinojosa. 2019. Affective and concreteness norms for 3,022 Croatian words. Quarterly Journal of Experimental Psychology, 72(9):2302-2312.
Per Davidson and Åse Innes-Ker. 2014. Valence and arousal norms for Swedish affective words. *Lund Psychological Reports*, 14:#2.
Luna De Bruyne, Pepa Atanasova, and Isabelle Augenstein. 2019. Joint emotion label space modelling for affect lexica. arXiv:1911.08782 [cs].
Steven Du and Xi Zhang. 2016. Aicyber's system for IALP 2016 Shared Task: Character-enhanced word vectors and boosted neural networks. In IALP 2016 - Proceedings of the 2016 International Conference on Asian Language Processing, pages 161-163, Tainan, Taiwan, November 21-23, 2016.
Paul Ekman. 1992. An argument for basic emotions. Cognition and Emotion, 6(3-4):169-200.
Pilar Ferre, Marc Guasch, Natalia Martínez-García, Isabel Fraga, and José Antonio Hinojosa. 2017. Moved by words: Affective ratings for a set of 2,266 Spanish words in five discrete emotion categories. Behavior Research Methods, 49(3):1082-1094.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In LREC 2018 — Proceedings of the 11th International Conference on Language Resources and Evaluation, pages 3483-3487, Miyazaki, Japan, May 7-12, 2018.
William L. Hamilton, Kevin Clark, Jure Leskovec, and Dan Jurafsky. 2016. Inducing domain-specific sentiment lexicons from unlabeled corpora. In EMNLP
2016 - Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 595-605, Austin, Texas, USA, November 1-5, 2016.
Vasileios Hatzivassiloglou and Kathleen R. McKeown. 1997. Predicting the semantic orientation of adjectives. In ACL-EACL 1997 - Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics & 8th Conference of the European Chapter of the Association for Computational Linguistics, pages 174-181, Madrid, Spain, July 7-12, 1997.
Johannes Hellrich, Sven Buechel, and Udo Hahn. 2018. JESEME: Interleaving semantics and emotions in a Web service for the exploration of language change phenomena. In COLING 2018 — Proceedings of the 27th International Conference on Computational Linguistics, volume System Demonstrations, pages 10–14, Santa Fe, New Mexico, USA, August 20–26, 2018.
Jose Antonio Hinojosa, Natalia Martínez-García, Cristina Villalba-García, Uxia Fernández-Folgueiras, Alberto Sánchez-Carmona, Miguel Angel Pozo, and Pedro R. Montoro. 2016. Affective norms of 875 Spanish words for five discrete emotional categories and two emotional dimensions. Behavior Research Methods, 48(1):272-284.
Holger Hoffmann, Andreas Scheck, Timo Schuster, Steffen Walter, Kerstin Limbrecht-Ecklundt, Harald C. Traue, and Henrik Kessler. 2012. Mapping discrete emotions into the dimensional space: An empirical approach. In SMC 2012 - Proceedings of the 2012 IEEE International Conference on Systems, Man, and Cybernetics, pages 3316-3320, Seoul, Korea, October 14-17, 2012.
Markus J. Hofmann, Lars Kuchenke, Sascha Tamm, Melissa L.-H. Vo, and Arthur M. Jacobs. 2009. Affective processing within 1/10th of a second: High arousal is necessary for early facilitative processing of negative but not positive words. Cognitive, Affective, & Behavioral Neuroscience, 9(4):389-397.
Clayton J. Hutto and Eric Gilbert. 2014. VADER: A parsimonious rule-based model for sentiment analysis of social media text. In ICWSM 2014 — Proceedings of the 8th International AAAI Conference on Weblogs and Social Media, pages 216–225, Ann Arbor, Michigan, USA, June 1–4, 2014.
Kamil K. Imbir. 2016. Affective Norms for 4900 Polish Words Reload (ANPW_R): Assessments for valence, arousal, dominance, origin, significance, concreteness, imageability and, age of acquisition. Frontiers in Psychology, 7:#1081.
Philipp Kanske and Sonja A. Kotz. 2010. Leipzig Affective Norms for German: A reliability study. Behavior Research Methods, 42(4):987-991.