UniSent: Universal Adaptable Sentiment Lexica for 1000+ Languages

Abstract: In this paper, we introduce UniSent a universal sentiment lexica for 1000 languages created using an English sentiment lexicon and a massively parallel corpus in the Bible domain. To the best of our knowledge, UniSent is the largest sentiment resource to date in terms of number of covered languages, including many low resource languages. To create UniSent, we propose Adapted Sentiment Pivot, a novel method that combines annotation projection, vocabulary expansion, and unsupervised domain adaptation. We evaluate the quality of UniSent for Macedonian, Czech, German, Spanish, and French and show that its quality is comparable to manually or semi-manually created sentiment resources. With the publication of this paper, we release UniSent lexica as well as Adapted Sentiment Pivot related codes. method.

Sentiment classification is an important task which requires either word level or document level sentiment annotations. Such resources are available for at most 136 languages BIBREF0 , preventing accurate sentiment classification in a low resource setup. Recent research efforts on cross-lingual transfer learning enable to train models in high resource languages and transfer this information into other, low resource languages using minimal bilingual supervision BIBREF1 , BIBREF2 , BIBREF3 . Besides that, little effort has been spent on the creation of sentiment lexica for low resource languages (e.g., BIBREF0 , BIBREF4 , BIBREF5 ). We create and release Unisent, the first massively cross-lingual sentiment lexicon in more than 1000 languages. An extensive evaluation across several languages shows that the quality of Unisent is close to manually created resources. Our method is inspired by BIBREF6 with a novel combination of vocabulary expansion and domain adaptation using embedding spaces. Similar to our work, BIBREF7 also use massively parallel corpora to project POS tags and dependency relations across languages. However, their approach is based on assignment of the most probable label according to the alignment model from the source to the target language and does not include any vocabulary expansion or domain adaptation and do not use the embedding graphs.
Our method, Adapted Sentiment Pivot requires a sentiment lexicon in one language (e.g. English) as well as a massively parallel corpus. Following steps are performed on this input.
Our goal is to evaluate the quality of UniSent against several manually created sentiment lexica in different domains to ensure its quality for the low resource languages. We do this in several steps.
As the gold standard sentiment lexica, we chose manually created lexicon in Czech BIBREF11 , German BIBREF12 , French BIBREF13 , Macedonian BIBREF14 , and Spanish BIBREF15 . These lexica contain general domain words (as opposed to Twitter or Bible). As gold standard for twitter domain we use emoticon dataset and perform emoticon sentiment prediction BIBREF16 , BIBREF17 .
We use the (manually created) English sentiment lexicon (WKWSCI) in BIBREF18 as a resource to be projected over languages. For the projection step (Section SECREF1 ) we use the massively parallel Bible corpus in BIBREF8 . We then propagate the projected sentiment polarities to all words in the Wikipedia corpus. We chose Wikipedia here because its domain is closest to the manually annotated sentiment lexica we use to evaluate UniSent. In the adaptation step, we compute the shift between the vocabularies in the Bible and Wikipedia corpora. To show that our adaptation method also works well on domains like Twitter, we propose a second evaluation in which we use Adapted Sentiment Pivot to predict the sentiment of emoticons in Twitter.
To create our test sets, we first split UniSent and our gold standard lexica as illustrated in Figure FIGREF11 . We then form our training and test sets as follows:
(i) UniSent-Lexicon: we use words in UniSent for the sentiment learning in the target domain; for this purpose, we use words INLINEFORM0 .
(ii) Baseline-Lexicon: we use words in the gold standard lexicon for the sentiment learning in the target domain; for this purpose we use words INLINEFORM0 .
(iii) Evaluation-Lexicon: we randomly exclude a set of words the baseline-lexicon INLINEFORM0 . In selection of the sampling size we make sure that INLINEFORM1 and INLINEFORM2 would contain a comparable number of words.
In Table TABREF13 we compare the quality of UniSent with the Baseline-Lexicon as well as with the gold standard lexicon for general domain data. The results show that (i) UniSent clearly outperforms the baseline for all languages (ii) the quality of UniSent is close to manually annotated data (iii) the domain adaptation method brings small improvements for morphologically poor languages. The modest gains could be because our drift weighting method (Section SECREF3 ) mainly models a sense shift between words which is not always equivalent to a polarity shift.
In Table TABREF14 we compare the quality of UniSent with the gold standard emoticon lexicon in the Twitter domain. The results show that (i) UniSent clearly outperforms the baseline and (ii) our domain adaptation technique brings small improvements for French and Spanish.
Using our novel Adapted Sentiment Pivot method, we created UniSent, a sentiment lexicon covering over 1000 (including many low-resource) languages in several domains. The only necessary resources to create UniSent are a sentiment lexicon in any language and a massively parallel corpus that can be small and domain specific. Our evaluation showed that the quality of UniSent is closed to manually annotated resources.
 

Figure and table captions:
Figure 1: Neighbors of word ’sensual’ in Spanish, in bible embedding graph (a) and twitter embedding graph (b). Our unsupervised drift weighting method found this word in Spanish to be the most changing word from bible context to the twitter context. Looking more closely at the neighbors, the word sensual in the biblical context has been associated with a negative sentiment of sins. However, in the twitter domain, it has a positive sentiment. This example shows how our unsupervised method can improve the quality of sentiment lexicon.
Figure 2: Data split used in the experimental setup of UniSent evaluation: Set (C) is the intersection of the target embedding space words (Wikipedia/Emoticon) and the UniSent lexicon as well as the manually created lexicon. Set (A) is the intersection of the target embedding space words and the UniSent lexicon, excluding set (C). Set (B) is the intersection of the target embedding space words and the manually created lexicon, excluding set (C).
Table 1: Comparison of manually created lexicon performance with UniSent in Czech, German, French, Macedonians, and Spanish. We report accuracy and the macro-F1 (averaged F1 over positive and negative classes). The baseline is constantly considering the majority label. The last two columns indicate the performance of UniSent after drift weighting.
Table 2: Comparison of domain adapted and vanilla UniSent for Emoticon sentiment prediction using monlingual twitter embeddings in German, Italian, French, and Spanish.

Questions about this paper:
how is quality measured?

Answers:
Accuracy and the macro-F1 (averaged F1 over positive and negative classes) are used as a measure of quality.