SlowGuess/ACL_2020_OCR huggingface dataset:
Sven Buechel, Susanna Rucker, and Udo Hahn
{sven.buechel|susanna.ruecker|udo.hahn}@uni-jena.de
Jena University Language and Information Engineering (JULIE) Lab
Friedrich-Schiller-Universität Jena, Jena, Germany
https://julielab.de
Emotion lexicons describe the affective meaning of words and thus constitute a centerpiece for advanced sentiment and emotion analysis. Yet, manually curated lexicons are only available for a handful of languages, leaving most languages of the world without such a precious resource for downstream applications. Even worse, their coverage is often limited both in terms of the lexical units they contain and the emotional variables they feature. In order to break this bottleneck, we here introduce a methodology for creating almost arbitrarily large emotion lexicons for any target language. Our approach requires nothing but a source language emotion lexicon, a bilingual word translation model, and a target language embedding model. Fulfilling these requirements for 91 languages, we are able to generate representationally rich high-coverage lexicons comprising eight emotional variables with more than 100k lexical entries each. We evaluated the automatically generated lexicons against human judgment from 26 datasets, spanning 12 typologically diverse languages, and found that our approach produces results in line with state-of-the-art monolingual approaches to lexicon creation and even surpasses human reliability for some languages and variables. Code and data are available at github.com/JULIELab/MeoLon archived under DOI 10.5281/zenodo.3779901.
An emotion lexicon is a lexical repository which encodes the affective meaning of individual words (lexical entries). Most simply, affective meaning can be encoded in terms of polarity, i.e., the distinction whether an item is considered as positive, negative, or neutral. This is the case for many well-known resources such as WORDNET-AFFECT (Strapparava and Valitutti, 2004), SENTIWORDNET (Baccianella et al., 2010), or VADER (Hutto
and Gilbert, 2014). Yet, an increasing number of researchers focus on more expressive encodings for affective states inspired by distinct lines of work in psychology (Yu et al., 2016; Buechel and Hahn, 2017; Sedoc et al., 2017; Abdul-Mageed and Ungar, 2017; Bostan and Klinger, 2018; Mohammad, 2018; Troiano et al., 2019).
Psychologists, on the one hand, value such lexicons as a controlled set of stimuli for designing experiments, e.g., to investigate patterns of lexical access or the structure of memory (Hofmann et al., 2009; Monnier and Syssau, 2008). NLP researchers, on the other hand, use them to augment the emotional loading of word embeddings (Yu et al., 2017; Khosla et al., 2018), as additional input to sentence-level emotion models so that the performance of even the most sophisticated neural network gets boosted (Mohammad and Bravo-Marquez, 2017; Mohammad et al., 2018; De Bruyne et al., 2019), or rely on them in a keyword-spotting approach when no training data is available, e.g., for studies dealing with historical language stages (Buechel et al., 2016).
As with any kind of manually curated resource, the availability of emotion lexicons is heavily restricted to only a few languages whose exact number varies depending on the variables under scrutiny. For example, we are aware of lexicons for 15 languages that encode the emotional variables of Valence, Arousal, and Dominance (see Section 2). This number leaves the majority of the world's (less-resourced) languages without such a dataset. In case such a lexicon exists for a particular language, it is often severely limited in size, sometimes only comprising some hundreds of entries (Davidson and Innes-Ker, 2014). Yet, even the largest lexicons typically cover only some ten thousands of words, still leaving out major portions of the emotion-carrying vocabulary. This is especially true for languages with complex morphology or
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1202-1217 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
productive compounding, such as Finnish, Turkish, Czech, or German. Finally, the diversity of emotion representation schemes adds another layer of complexity. While psychologists and NLP researchers alike find that different sets of emotional variables are complementary to each other (Stevenson et al., 2007; Pinheiro et al., 2017; Barnes et al., 2019; De Bruyne et al., 2019), manually creating emotion lexicons for every language and every emotion representation scheme is virtually impossible.
We here propose an approach based on crosslingual distant supervision to generate almost arbitrarily large emotion lexicons for any target language and emotional variable, provided the following requirements are met: a source language emotion lexicon covering the desired variables, a bilingual word translation model, and a target language embedding model. By fulfilling these preconditions, we can automatically generate emotion lexicons for 91 languages covering ratings for eight emotional variables and hundreds of thousands of lexical entries each. Our experiments reveal that our method is on a par with state-of-the-art monolingual approaches and compares favorably with (sometimes even outperforms) human reliability.
Representing Emotion. Whereas research in NLP has focused for a very long time almost exclusively on polarity, more recently, there has been a growing interest in more informative representation structures for affective states by including different groups of emotional variables (Bostan and Klinger, 2018). Borrowing from distinct schools of thought in psychology, these variables can typically be subdivided into dimensional vs. discrete approaches to emotion representation (Calvo and Mac Kim, 2013). The dimensional approach assumes that emotional states can be composed out of several foundational factors, most noticeably Valence (corresponding to polarity), Arousal (measuring calmness vs. excitement), and Dominance (the perceived degree of control in a social situation); VAD, for short (Bradley and Lang, 1994). Conversely, the discrete approach assumes that emotional states can be reduced to a small, evolutionary motivated set of basic emotions (Ekman, 1992). Although the exact division of the set has been subject of hot debates, recently constructed datasets (see Section 4) most often cover the categories of Joy, Anger, Sadness, Fear, and Disgust; BE5, for
short. Plutchik's Wheel of Emotion takes a middle ground between those two positions by postulating emotional categories which are yet grouped into opposite pairs along different levels of intensity (Plutchik, 1980).
Another dividing line between representational approaches is whether target variables are encoded in terms of (strict) class-membership or scores for numerical strength. In the first case, emotion analysis translates into a (multi-class) classification problem, whereas the latter turns it into a regression problem (Buechel and Hahn, 2016). While our proposed methodology is agnostic towards the chosen emotion format, we will focus on the VAD and BE5 formats here, using numerical ratings (see the examples in Table 1) due to the widespread availability of such data. Accordingly, this paper treats word emotion prediction as a regression problem.
Val	Aro	Dom	Joy	Ang	Sad	Fea	Dis
sunshine	8.1	5.3	5.4	4.2	1.2	1.3	1.3	1.2
terrorism	1.6	7.4	2.7	1.2	2.9	3.3	3.9	2.5
nuclear	4.3	7.3	4.1	1.4	2.2	1.9	3.2	1.6
ownership	5.9	4.4	7.5	2.1	1.4	1.2	1.4	1.3
Table 1: Sample entries from our English source lexicon described via eight emotional variables: Valence, Arousal, Dominance [VAD], and Joy, Anger, Sadness, Fear, and Disgust [BE5]. VAD uses 1-to-9 scales (“5” encodes the neutral value) and BE5 1-to-5 scales (“1” encodes the neutral value).
Building Emotion Lexicons. Usually, the ground truth for affective word ratings (i.e., the assignment of emotional values to a lexical item) is acquired in a questionnaire study design where subjects (annotators) receive lists of words which they rate according to different emotion variables or categories. Aggregating individual ratings of multiple annotators then results in the final emotion lexicon (Bradley and Lang, 1999). Recently, this workflow has often been enhanced by crowdsourcing (Mohammad and Turney, 2013) and best-worst scaling (Kiritchenko and Mohammad, 2016).
As a viable alternative to manual acquisition, such lexicons can also be created by automatic means (Bestgen, 2008; Köper and Schulte im Walde, 2016; Shaikh et al., 2016), i.e., by learning to predict emotion labels for unseen words. Researchers have worked on this prediction problem for quite a long time. Early work tended to focus on word statistics, often in combination with linguistic rules (Hatzivassiloglou and McKeown,
1997; Turney and Littman, 2003). More recent approaches focus heavily on word embeddings, either using semi-supervised graph-based approaches (Wang et al., 2016; Hamilton et al., 2016; Sedoc et al., 2017) or fully supervised methods (Rosenthal et al., 2015; Li et al., 2017; Rothe et al., 2016; Du and Zhang, 2016). Most important for this work, Buechel and Hahn (2018b) report on near-human performance using a combination of FASTTEXT vectors and a multi-task feed-forward network (see Section 4). While this line of work can add new words, it does not extend lexicons to other emotional variables or languages.
A relatively new way of generating novel labels is emotion representation mapping (ERM), an annotation projection that translates ratings from one emotion format into another, e.g., mapping VAD labels into BE5, or vice versa (Hoffmann et al., 2012; Buechel and Hahn, 2016, 2018a; Alarcão and Fonseca, 2017; Landowska, 2018; Zhou et al., 2020; Park et al., 2019). While our work uses ERM to add additional emotion variables to the source lexicon, ERM alone can neither increase the coverage of a lexicon, nor adapt it to another language.
Translating Emotions. The approach we propose is strongly tied to the observation by Leveau et al. (2012) and Warriner et al. (2013) who found—comparing a large number of existing emotion lexicons of different languages—that translational equivalents of words show strong stability and adherence to their emotional value. Yet, their work is purely descriptive. They do not exploit their observation to create new ratings, and only consider manual rather than automatic translation.
Making indirect use of this observation, Mohammad and Turney (2013) offer machine-translated versions of their NRC Emotion Lexicon. Also, many approaches in cross-lingual sentiment analysis (on the sentence-level) rely on translating polarity lexicons (Abdalla and Hirst, 2017; Barnes et al., 2018). Perhaps most similar to our work, Chen and Skiena (2014) create (polarity-only) lexicons for 136 languages by building a multilingual word graph and propagating sentiment labels through that graph. Yet, their method is restricted to high frequency words—their lexicons cover between 12 and 4,653 entries, whereas our approach exceeds this limit by more than two orders of magnitude.
Our methodology also resembles previous work which models word emotion for historical language stages (Cook and Stevenson, 2010; Hamilton et al.,
2016; Hellrich et al., 2018; Li et al., 2019). Work in this direction typically comes up with a set of seed words with assumingly temporally stable affective meaning (our work assumes stability against translation) and then uses distributional methods to derive emotion ratings in the target language stage. However, gold data for the target language (stage) is usually inaccessible, often preventing evaluation against human judgment. In contrast, we here propose several alternative evaluation set-ups as an integral part of our methodology.
3 A Novel Approach to Lexicon Creation
Our methodology integrates (1) cross-lingual generation and expansion of emotion lexicons and (2) their evaluation against gold and silver standard data. Consequently, a key aspect of our workflow design is how data is split into train, dev, and test sets at different points of the generation process. Figure 1 gives an overview of our framework including a toy example for illustration.
Lexicon Generation. We start with a lexicon (Source) of arbitrary size, emotion format<sup>1</sup> and source language which is partitioned into train, dev, and test splits denoted by Source-train, Source-dev, and Source-test, respectively. Next, we leverage a bilingual word translation model between source and desired target language to build the first target-side emotion lexicon denoted as TargetMT. Source words are translated according to the model, whereas target-side emotion labels are simply copied from the source to the target (see Section 2). Entries are assigned to train, dev, or test set according to their source-side assignment (cf. Figure 1). The choice of our translation service (see below) ensures that each source word receives exactly one translation.
TargetMT is then used as the distant supervisor to train a model that predicts word emotions based on target-side word embeddings. TargetMT-train and TargetMT-dev are used to fit model parameters and optimize hyperparameters, respectively, whereas TargetMT-test is held out for later evaluation. Once finalized, the model is used to predict new labels for the words in TargetMT, resulting in a second target-side emotion lexicon denoted TargetPred. Our rationale for doing so is that a reasonably trained model should generalize well
This encompasses not only VA(D) and BE5, but also any sort of (real-valued) polarity encodings.
Figure 1: Schematic view on the methodology for generating and evaluating an emotion lexicon for a given target language based on source language supervision. Included is a toy example starting with an English VA lexicon (sunshine, nuclear, terrorism and the associated numerical scores for Valence and Arousal) and resulting in an extended German lexicon which incorporates translated entries with altered VA scores and additional entries originating from the embedding model with newly learned scores.
over the entire TargetMT lexicon because it has access to the target-side embedding vectors. Hence, it may mitigate some of the errors which were introduced in previous steps, either by machine translation or by assuming that source- and target-side emotion are always identical. We validate this assumption in Section 6. We also predict ratings for all the words in the embedding model, leading to a large number of new entries.
The splits are defined as follows: let  $MT_{\mathrm{train}}$ ,  $MT_{\mathrm{dev}}$ , and  $MT_{\mathrm{test}}$  denote the set of words in train, dev, and test split of TargetMT, respectively. Likewise, let  $P_{\mathrm{train}}$ ,  $P_{\mathrm{dev}}$ , and  $P_{\mathrm{test}}$  denote the splits of TargetPred and let  $E$  denote the set of words in the embedding model. Then
$$
P _ {\text {t r a i n}} := M T _ {\text {t r a i n}}
$$
$$
P _ {\mathrm {d e v}} := M T _ {\mathrm {d e v}} \backslash M T _ {\mathrm {t r a i n}}
$$
$$
P _ {\text {t e s t}} := \left(M T _ {\text {t e s t}} \cup E\right) \backslash \left(M T _ {\text {d e v}} \cup M T _ {\text {t r a i n}}\right)
$$
The above definitions help clarify the way we address polysemy. Ambiguity on the target-side
may result in multiple source entries translating to the same target-side word. $^3$  This circumstance leads to "partial duplicates" in TargetMT, i.e., groups of entries with the same word type but different emotion values (because they were derived from distinct Source entries). Such overlap could do harm to the integrity of our evaluation since knowledge may "leak" from training to validation phase, i.e., by testing the model on words it has already seen during training, although with distinct emotion labels. The proposed data partitioning eliminates such distortion effects. Since partial duplicates receive the same embedding vector, the prediction model assigns the same emotion value to both, thus merging them in TargetPred.
Evaluation Methodology. The main advantage of the above generation method is that it allows us to create large-scale emotion lexicons for languages
a subject and the response (rating) is recorded. The absence of sense-level annotation simplifies the mapping between lexicon and embedding entries. While sense embeddings form an active area of research (Camacho-Collados and Pilehvar, 2018; Chi and Chen, 2018), to the best of our knowledge, type-level embeddings yield state-of-the-art performance in downstream applications.
${}^{3}$  Source-side polysemy,in contrast to its target-side counterpart, is less of a problem, because we receive only a single candidate during translation. This may result in cases where the translation misaligns with the copied emotion value in TargetMT. Yet, the prediction step partly mitigates such inconsistencies (see Section 6).
2In short, our work evades this problem by dealing with lexical entries exclusively on the type- rather than the senselevel. From a lexicological perspective, this may seem like a strong assumption. From a modeling perspective, however, it appears almost obvious as it aligns well with the major components of our methodology, i.e., lexicons, embeddings, and translation. The lexicons we work with follow the design of behavioral experiments: a stimulus (word type) is given to
for which gold data is lacking. But if that is the case, how can we assess the quality of the generated lexicons? Our solution is to propose two different evaluation scenarios—a gold evaluation which is a strict comparison against human judgment, meaning that it is limited to languages where such data (denoted TargetGold) is available, and a silver evaluation which substitutes human judgments by automatically derived ones (silver standard) which is feasible for any language in our study. The rationale is that if both, gold and silver evaluation, strongly agree with each other, we can use one as proxy for the other when no target-side gold data exists (examined in Section 6).
Note that our lexicon generation approach consists of two major steps, translation and prediction. However, these two steps are not equally important for each generated entry in TargetPred. Words, such as German Sonnenschein for which a translational equivalent already exists in the Source ("sunshine"; see Figure 1), mainly rely on translation, while the prediction step acts as an optional refinement procedure. In contrast, the prediction step is crucial for words, such as Erdbeben, whose translational equivalents ("earthquake") are missing in the Source. Yet, these words also depend on the translation step for producing training data.
These considerations are important for deciding which words to evaluate on. We may choose to base our evaluation on the full TargetPred lexicon, including words from the training set—after all, the word emotion model does not have access to any target-side gold data. The problem with this approach is that it merges words that mainly rely on translation, because their equivalents are in the Source, and those which largely depend on prediction, because they are taken from the embedding model. In this case, generalizability of evaluation results becomes questionable.
Thus, our evaluation methodology needs to fulfill the following two requirements: (1) evaluation must not be performed on translational equivalents of the Source entries to which the model already had access during training (e.g., Sonnenschein and nuklear in our example from Figure 1); but, on the other hand, (2) a reasonable number of instances must be available for evaluation (ideally, as many as possible to increase reliability). The intricate cross-lingual train-dev-test set assignment of our generation methodology is in place so that we meet these two requirements.
ID	Encoding	Size	Citation
en1	VAD	1032	Warriner et al. (2013)
en2	VAD	1034	Bradley and Lang (1999)
en3	BE5	1034	Stevenson et al. (2007)
es1	VAD	1034	Redondo et al. (2007)
es2	VA	14031	Stadthagen-González et al. (2017)
es3	VA	875	Hinojosa et al. (2016)
es4	BE5	875	Hinojosa et al. (2016)
es5	BE5	10491	Stadthagen-González et al. (2018)
es6	BE5	2266	Ferré et al. (2017)
de1	VAD	1003	Schmidtke et al. (2014)
de2	VA	2902	Võ et al. (2009)
de3	VA	1000	Kanske and Kotz (2010)
de4	BE5	1958	Briesemeister et al. (2011)
pl1	VAD	4905	Imbir (2016)
pl2	VA	2902	Riegel et al. (2015)
pl3	BE5	2902	Wierzba et al. (2015)
zh1	VA	2794	Yu et al. (2016)
zh2	VA	1100	Yao et al. (2017)
it	VAD	1121	Montefinese et al. (2014)
pt	VAD	1034	Soares et al. (2012)
nl	VA	4299	Moors et al. (2013)
id	VAD	1487	Sianipar et al. (2016)
el	VAD	1034	Palogiannidi et al. (2016)
tr1	VA	2029	Kapucu et al. (2018)
tr2	BE5	2029	Kapucu et al. (2018)
hr	VA	3022	Ácoso et al. (2019)
Table 2: Lexicons used for gold evaluation. IDs consist of the respective ISO 639-1 language code plus a cardinal number to distinguish different datasets, if needed; the format of emotion Encoding is specified and Size gives the number of lexical entries per lexicon.
In particular, for our silver evaluation, we intersect TargetMT-test with TargetPred-test and compute the correlation of these two sets individually for each emotion variable. Pearson's  $r$  will be used as correlation measure throughout this paper. Establishing a test set at the very start of our workflow, Source-test, assures that there is a relatively large overlap between the two sets and, by extension, that our requirements for the evaluation are met.
The gold evaluation is a somewhat more challenging case, because we can, in general, not guarantee that the overlap of a TargetGold lexicon with TargetPred-test will be of any particular size. For this reason, the words of the embedding model are added to TargetPred-test (see above), maximizing the expected overlap with TargetGold. In practical terms, we intersect TargetGold with TargetPred-test and compute the variable-wise correlation between these sets, in parallel to the silver evaluation. A complementary strategy for maximizing overlap, by exploiting dependencies between published lexicons, is described below.
Gold Lexicons and Data Splits. We use the English emotion lexicon from Warriner et al. (2013) as first part of our Source dataset. This popular resource comprises about 14k entries in VAD format collected via crowdsourcing. Since manually gathered BE5 ratings are available only for a subset of this lexicon (Stevenson et al., 2007), we add BE5 ratings from Buechel and Hahn (2018a) who used emotion representation mapping (see Section 2) to convert the existing VAD ratings, showing that this is about as reliable as human annotation.
As apparent from the previous section, a crucial aspect for applying our methodology is the design of the train-dev-test split of the Source because it directly impacts the amount of words we can test our lexicons on during gold evaluation. In line with these considerations, we choose the lexical items which are already present in ANEW (Bradley and Lang, 1999) as Source-test set. ANEW is the precursor to the version later distributed by Warriner et al. (2013); it is widely used and has been adapted to a wide range of languages. With this choice, it is likely that a resulting TargetPred-test set has a large overlap with the respective TargetGold lexicon. As for the TargetGold lexicons, we included every VA(D) and BE5 lexicon we could get hold of with more than 500 entries. This resulted in 26 datasets covering 12 quite diverse languages (see Table 2). Note that we also include English lexicons in the gold evaluation. In these cases, no translation will be carried out (Source is identical to TargetMT) so that only the expansion step is validated. Appendix A.1 gives further details on data preparation.
Translation. We used the Google CLOUD TRANSLATION API to produce word-to-word translation tables. This is a commercial service, total translation costs amount to 160 EUR. API calls were performed in November 2019.