Targeted Sentiment Classification
Opinions are everywhere in our lives. Every time we open a book, read the newspaper, or look at social media, we scan for opinions or form them ourselves. We are cued to the opinions of others, and often use this information to update our own opinions Asch1955,Das2014. This is true on the Internet as much as it is in our face-to-face relationships. In fact, with its wealth of opinionated material available online, it has become feasible and interesting to harness this data in order to automatically identify opinions, which had previously been far more expensive and tedious when the only access to data was offline.
Sentiment analysis, sometimes referred to as opinion mining, seeks to create data-driven methods to classify the polarity of a text. The information obtained from sentiment classifiers can then be used for tracking user opinions in different domains Pang2002,Socher2013b,Nakov2013, predicting the outcome of political elections wang2012demo,bakliwal2013, detecting hate speech online Nahar2012,hartung-EtAl:2017:WASSA2017, as well as predicting changes in the stock market Pogolu2016.
Sentiment analysis can be modeled as a classification task, especially at sentence- and document-level, or as a sequence-labeling task at target-level. Targeted sentiment analysis aims at predicting the polarity expressed towards a particular entity or sub-aspect of that entity. This is a more realistic view of sentiment, as polarities are directed towards targets, not spread uniformly across sentences or documents. Take the following example, where we mark the sentiment target with green, positive sentiment expressions with blue, and negative sentiment expressions with red.:
The café near my house has great coffee but I
never go there because the service is terrible.
In this sentence, it is not stated what the sentiment towards the target “café” is, while the sentiment of the target “coffee” is positive and that of “service” is negative. In order to correctly classify the sentiment of each target, it is necessary to (1) detect the targets, (2) detect polarity expressions, and (3) resolve the relations between these.
In order to model these relationships and test the accuracy of the learned models, hand-annotated resources are typically used for training machine learning algorithms. Resource-rich languages, e. g., English, have high-quality annotated data for both classification and sequence-labeling tasks, as well as for a variety of domains. However, under-resourced languages either completely lack annotated data or have only a few resources for specific domains or sentiment tasks. For instance, for aspect-level sentiment analysis, English has datasets available in the news domain Wiebe2005, product review domain HuandLiu2004,Ding2008,Pontiki2014,Pontiki2015, education domain Welch2016, medical domain Grasser2018, urban neighborhood domain Saeidi2016, and financial Maia2018 domain. Spanish, on the other hand, has only three datasets Agerri2013,Pontiki2016, while Basque and Catalan only have one each for a single domain Barnes2018a. The cost of annotating data can often be prohibitive as training native-speakers to annotate fine-grained sentiment is a long process. This motivates the need to develop sentiment analysis methods capable of leveraging data annotated in other languages.
Cross-Lingual Approaches to Sentiment Analysis
Previous work on cross-lingual sentiment analysis (CLSA) offers a way to perform sentiment analysis in an under-resourced language that does not have any annotated data available. Most methods relied on the availability of large amounts of parallel data to transfer sentiment information across languages. Machine translation (MT), for example, has been the most common approach to cross-lingual sentiment analysis Banea2013,Almeida2015,Zhang2017. Machine translation, however, can be biased towards domains Hua2008,Bertoldi2009,Koehn2017, does not always preserve sentiment Mohammad2016, and requires millions of parallel sentences Gavrila2011,Vaswani2017, which places a limit on which languages can benefit from these approaches. The following example illustrates that MT does not preserve sentiment (hotel review in Basque, automatically translated via translate.google.com):
Hotel $^{1}$ txukuna da, nahiko berria. Harreran zeuden langileen arreta $^{2}$ ez zen onena izan. Tren geltoki bat $^{3}$ du 5 minutura eta kotxez $^{4}$ berehala iristen da baina oinez $^{5}$ urruti samar dago.
The hotel $^{1}$ is tidy, quite new. The care of the workers at reception $^{2}$ was not the best. It's 5 minutes away from a train station $^{3}$ and it's quick to reach the car $^{4}$ , but it's a short distance away.
While the first two sentences are mostly well translated for the purposes of sentiment analysis, in the third, there are a number of reformulations and deletions that lead to a loss of information. It should read “It has a train station five minutes away and by car you can reach it quickly, but by foot it's quite a distance.” We can see that one of the targets has been deleted and the sentiment has flipped from negative to positive. Such common problems degrade the results of cross-lingual sentiment systems that use MT, especially at target-level.
Although high quality machine translation systems exist between many languages and have been shown to enable cross-lingual sentiment analysis, for the vast majority of language pairs in the world there is not enough parallel data to create these high quality MT systems. This lack of parallel data coupled with the computational expense of MT means that approaches to cross-lingual sentiment analysis that do not require MT should be preferred. Additionally, most cross-lingual sentiment approaches using MT have concentrated on sentence- and document-level, and have not explored targeted or aspect-level sentiment tasks.
Bilingual Distributional Models and the Contributions of this Paper
Recently, several bilingual distributional semantics models (bilingual embeddings) have been proposed and provide a useful framework for cross-lingual research without requiring machine translation. They are effective at generating features for bilingual dictionary induction Mikolov2013translation,Artetxe2016,Lample2017, cross-lingual text classification Prettenhofer2011b,Chandar2014, or cross-lingual dependency parsing Sogaard2015, among others. In this framework, words are represented as $n$ -dimensional vectors which are created on large monolingual corpora in order to (1) maximize the similarity of words that appear in similar contexts and use some bilingual regularization in order to (2) maximize the similarity of translation pairs. In this work, we concentrate on a subset of these bilingual embedding methods that perform a post-hoc mapping to a bilingual space, which we refer to as embedding projection methods. One of the main advantages of these methods is that they make better use of small amounts of parallel data than MT systems, even enabling unsupervised machine translation Artetxe2018,Lample2018.
With this paper, we provide the first extensive evaluation of cross-lingual embeddings for targeted sentiment tasks. We formulate the task of targeted sentiment analysis as classification, given the targets from an oracle. The question we attempt to address is how to infer the polarity of a sentiment target in a language that does not have any annotated sentiment data or parallel corpora with a resource-rich language. In the following Catalan sentence, for example, how can we determine that the sentiment of “servei” is negative, while that of “menjar” is positive if we do not have annotated data in Catalan or parallel data for English-Catalan?
El servei al restaurant va ser péssim. Al menys el menjar era bo.
Specifically, we propose an approach which requires (1) minimal bilingual data and instead makes use of (2) high-quality monolingual word embeddings in the source and target language. We take an intermediate step by first testing this approach on sentence-level classification. After confirming that our approach performs well at sentence-level, we propose a targeted model with the same data requirements. The main contributions are that we
compare projection-based cross-lingual methods to MT,
extend previous cross-lingual approaches to enable targeted cross-lingual sentiment analysis with minimal parallel data requirements,
compare different model architectures for cross-lingual targeted sentiment analysis,
perform a detailed error analysis, and detailing the advantages and disadvantages of each method,
and, finally, deploy the methods in a realistic case-study to analyze their suitability beyond applications on (naturally) limited language pairs.
In addition, we make our code and data publicly available at https://github.com/jbarnesspain/targeted_blse to support future research. The rest of the article is organized as follows: In Section "Previous Work" , we detail related work and motivate the need for a different approach. In Section "Projecting Sentiment Across Languages" , we describe both the sentence-level and targeted projection approaches that we propose. In Section "Experiments" , we detail the resources and experimental setup for both sentence and targeted classification. In Section "Results" , we describe the results of the two experiments, as well as perform a detailed error analysis. In Section "Case Study: Real World Deployment" , we perform a case study whose purpose is to give a more qualitative view of the models. Finally, we discuss the implications of the results in Section "Conclusion" .
Previous Work
Sentiment analysis has become an enormously popular task with a focus on classification approaches on individual languages, but there has not been as much work on cross-lingual approaches. In this section, we detail the most relevant work on cross-lingual sentiment analysis and lay the basis for the bilingual embedding approach we propose later.
Machine Translation Based Methods
Early work in cross-lingual sentiment analysis found that machine translation (MT) had reached a point of maturity that enabled the transfer of sentiment across languages. Researchers translated sentiment lexicons Mihalcea2007,Meng2012 or annotated corpora and used word alignments to project sentiment annotation and create target-language annotated corpora Banea2008,Duh2011a,Demirtas2013,Balahur2014d.
Several approaches included a multi-view representation of the data Banea2010,Xiao2012 or co-training Wan2009,Demirtas2013 to improve over a naive implementation of machine translation, where only the translated version of the data is considered. There are also approaches which only require parallel data Meng2012,Zhou2016,Rasooli2017, instead of machine translation.
All of these approaches, however, require large amounts of parallel data or an existing high quality translation tool, which are not always available. To tackle this issue, Barnes2016 explore cross-lingual approaches for aspect-based sentiment analysis, comparing machine translation methods and those that instead rely on bilingual vector representations. They conclude that MT approaches outperform current bilingual representation methods.
Chen2016 propose an adversarial deep averaging network, which trains a joint feature extractor for two languages. They minimize the difference between these features across languages by learning to fool a language discriminator. This requires no parallel data, but does require large amounts of unlabeled data and has not been tested on fine-grained sentiment analysis.
Bilingual Embedding Methods
Recently proposed bilingual embedding methods Hermann2014,Chandar2014,Gouws2015 offer a natural way to bridge the language gap. These particular approaches to bilingual embeddings, however, also require large parallel corpora in order to build the bilingual space, which gives no advantage over machine translation. Another approach to creating bilingual word embeddings, which we refer to as Projection-based Bilingual Embeddings, has the advantage of requiring relatively little parallel training data while taking advantage of larger amounts of monolingual data. In the following, we describe the most relevant approaches.
Mikolov2013translation find that vector spaces in different languages have similar arrangements. Therefore, they propose a linear projection which consists of learning a rotation and scaling matrix. Artetxe2016,Artetxe2017 improve upon this approach by requiring the projection to be orthogonal, thereby preserving the monolingual quality of the original word vectors.
Given source embeddings $S$ , target embeddings $T$ , and a bilingual lexicon $L$ , Artetxe2016 learn a projection matrix $W$ by minimizing the square of Euclidean distances
$$\operatornamewithlimits{arg\,min}_W \sum _{i} ||S^{\prime }W-T^{\prime }||_{F}^{2}\,,$$   (Eq. 13)
where $S^{\prime } \in S$ and $T^{\prime } \in T$ are the word embedding matrices for the tokens in the bilingual lexicon $L$ . This is solved using the Moore-Penrose pseudoinverse $S^{\prime +} = (S^{\prime T}S^{\prime })^{-1}S^{\prime T}$ as $ W = S^{\prime +}T^{\prime }$ , which can be computed using SVD. We refer to this approach as VecMap.
Lample2017 propose a similar refined orthogonal projection method to Artetxe2017, but include an adversarial discriminator, which seeks to discriminate samples from the projected space $WS$ , and the target $T$ , while the projection matrix $W$ attempts to prevent this making the projection from the source space $WS$ as similar to the target space $T$ as possible.
They further refine their projection matrix by reducing the hubness problem Dinu2015, which is commonly found in high-dimensional spaces. For each projected embedding $Wx$ , they define the $k$ nearest neighbors in the target space, $\mathcal {N}_{T}$ , suggesting $k = 10$ . They consider the mean cosine similarity $r_{T}(Wx)$ between a projected embedding $Wx$ and its $k$ nearest neighbors
$$r_{T}(Wx) = \frac{1}{k} \sum _{y \in \mathcal {N}_{T}(Wx) } \cos (Wx,y)$$   (Eq. 15)
as well as the mean cosine of a target word $y$ to its neighborhood, which they denote by $r_{S}$ .
In order to decrease similarity between mapped vectors lying in dense areas, they introduce a cross-domain similarity local scaling term (CSLS)
$$\textrm {CSLS}(Wx,y) = 2 \cos (Wx,y) - r_{T}(Wx) - r_{S}(y)\,,$$   (Eq. 16)
which they find improves accuracy, while not requiring any parameter tuning.
Gouws2015taskspecific propose a method to create a pseudo-bilingual corpus with a small task-specific bilingual lexicon, which can then be used to train bilingual embeddings (Barista). This approach requires a monolingual corpus in both the source and target languages and a set of translation pairs. The source and target corpora are concatenated and then every word is randomly kept or replaced by its translation with a probability of 0.5. Any kind of word embedding algorithm can be trained with this pseudo-bilingual corpus to create bilingual word embeddings.
Sentiment Embeddings
Maas2011 first explored the idea of incorporating sentiment information into semantic word vectors. They proposed a topic modeling approach similar to latent Dirichlet allocation in order to collect the semantic information in their word vectors. To incorporate the sentiment information, they included a second objective whereby they maximize the probability of the sentiment label for each word in a labeled document.
Tang2014 exploit distantly annotated tweets to create Twitter sentiment embeddings. To incorporate distributional information about tokens, they use a hinge loss and maximize the likelihood of a true $n$ -gram over a corrupted $n$ -gram. They include a second objective where they classify the polarity of the tweet given the true $n$ -gram. While these techniques have proven useful, they are not easily transferred to a cross-lingual setting.
Zhou2015 create bilingual sentiment embeddings by translating all source data to the target language and vice versa. This requires the existence of a machine translation system, which is a prohibitive assumption for many under-resourced languages, especially if it must be open and freely accessible. This motivates approaches which can use smaller amounts of parallel data to achieve similar results.
Targeted Sentiment Analysis
The methods discussed so far focus on classifying textual phrases like documents or sentences. Next to these approaches, others have concentrated on classifying aspects HuandLiu2004,Liu2012,Pontiki2014 or targets Zhang2015,Zhang2016,Tang2016 to assign them with polarity values.
A common technique when adapting neural architectures to targeted sentiment analysis is to break the text into left context, target, and right context Zhang2015,Zhang2016, alternatively keeping the target as the final/beginning token in the respective contexts Tang2016. The model then extracts a feature vector from each context and target, using some neural architecture, and concatenates the outputs for classification.
More recent approaches attempt to augment a neural network with memory to model these interactions Chen2017,Xue2018,Wang2018,Liu2018. Wang2017 explore methods to improve classification of multiple aspects in tweets, while Akhtar2018 attempt to use cross-lingual and multilingual data to improve aspect-based sentiment analysis in under-resourced languages.
As mentioned before, MT has traditionally been the main approach for transferring information across language barriers BIBREF0 . But this is particularly problematic for targeted sentiment analysis, as changes in word order or loss of words created during translation can directly affect the performance of a classifier Lambert2015.
Projecting Sentiment Across Languages
In this section, we propose a novel approach to incorporate sentiment information into bilingual embeddings, which we first test on sentence-level cross-lingual sentiment classification. We then propose an extension in order to adapt this approach to targeted cross-lingual sentiment classification. Our model, Bilingual Sentiment Embeddings (Blse), are embeddings that are jointly optimized to represent both (a) semantic information in the source and target languages, which are bound to each other through a small bilingual dictionary, and (b) sentiment information, which is annotated on the source language only. We only need three resources: (1) a comparably small bilingual lexicon, (2) an annotated sentiment corpus in the resource-rich language, and (3) monolingual word embeddings for the two involved languages.
Sentence-level Model
In this section, we detail the projection objective, the sentiment objective, and finally the full objective for sentence-level cross-lingual sentiment classification. A sketch of the full sentence-level model is depicted in Figure 1 .
We assume that we have two precomputed vector spaces $S = \mathbb {R}^{v \times d}$ and $T = \mathbb {R}^{v^{\prime } \times d^{\prime }}$ for our source and target languages, where $v$ ( $v^{\prime }$ ) is the length of the source vocabulary (target vocabulary) and $d$ ( $d^{\prime }$ ) is the dimensionality of the embeddings. We also assume that we have a bilingual lexicon $L$ of length $n$ which consists of word-to-word translation pairs $L$ = $\lbrace (s_{1},t_{1}), (s_{2},t_{2}),\ldots , (s_{n}, t_{n})\rbrace $ which map from source to target.
In order to create a mapping from both original vector spaces $S$ and $T$ to shared sentiment-informed bilingual spaces $\mathbf {z}$ and $\mathbf {\hat{z}}$ , we employ two linear projection matrices, $M$ and $M^{\prime }$ . During training, for each translation pair in $L$ , we first look up their associated vectors, project them through their associated projection matrix and finally minimize the mean squared error of the two projected vectors. This is similar to the approach taken by Mikolov2013translation , but includes an additional target projection matrix.
The intuition for including this second matrix is that a single projection matrix does not support the transfer of sentiment information from the source language to the target language. Without $M^{\prime }$ , any signal coming from the sentiment classifier (see Section UID27 ) would have no affect on the target embedding space $T$ , and optimizing $M$ to predict sentiment and projection would only be detrimental to classification of the target language. We analyze this further in Section UID63 . Note that in this configuration, we do not need to update the original vector spaces, which would be problematic with such small training data.
The projection quality is ensured by minimizing the mean squared error
$$\textrm {MSE} = \dfrac{1}{n} \sum _{i=1}^{n} (\mathbf {z_{i}} - \mathbf {\hat{z}_{i}})^{2}\,,$$   (Eq. 26)
where $\mathbf {z_{i}} = S_{s_{i}} \cdot M$ is the dot product of the embedding for source word $s_{i}$ and the source projection matrix and $\mathbf {\hat{z}_{i}} = T_{t_{i}} \cdot M^{\prime }$ is the same for the target word $t_{i}$ .
We add a second training objective to optimize the projected source vectors to predict the sentiment of source phrases. This inevitably changes the projection characteristics of the matrix $M$ , and consequently $M^{\prime }$ and encourages $M^{\prime }$ to learn to predict sentiment without any training examples in the target language.
In order to train $M$ to predict sentiment, we require a source-language corpus $C_{\textrm {source}}= \lbrace (x_{1}, y_{1}), (x_{2}, y_{2}), \ldots , (x_{i}, y_{i})\rbrace $ where each sentence $x_{i}$ is associated with a label $y_{i}$ .
For classification, we use a two-layer feed-forward averaging network, loosely following Iyyer2015 . For a sentence $x_{i}$ we take the word embeddings from the source embedding $S$ and average them to $\mathbf {a}_{i} \in \mathbb {R}^{d}$ . We then project this vector to the joint bilingual space $\mathbf {z}_{i} = \mathbf {a}_{i} \cdot M$ . Finally, we pass $\mathbf {z}_{i}$ through a softmax layer $P$ to obtain the prediction $\hat{y}_{i} = \textrm {softmax} ( \mathbf {z}_{i} \cdot P)$ .
To train our model to predict sentiment, we minimize the cross-entropy error of the predictions
$$H = - \sum _{i=1}^{n} y_{i} \log \hat{y_{i}} - (1 - y_{i}) \log (1 - \hat{y_{i}})\,.$$   (Eq. 29)
In order to jointly train both the projection component and the sentiment component, we combine the two loss functions to optimize the parameter matrices $M$ , $M^{\prime }$ , and $P$ by
$$J =\hspace{-14.22636pt}\sum _{(x,y) \in C_{\textrm {source}}}\hspace{2.84526pt}\sum _{(s,t) \in L}\hspace{0.0pt}\alpha H(x,y) + (1 - \alpha ) \cdot \textrm {MSE}(s,t)\,,$$   (Eq. 31)
where $\alpha $ is a hyperparameter that weights sentiment loss vs. projection loss.
For inference, we classify sentences from a target-language corpus $C_{\textrm {target}}$ . As in the training procedure, for each sentence, we take the word embeddings from the target embeddings $T$ and average them to $\mathbf {a}_{i} \in \mathbb {R}^{d}$ . We then project this vector to the joint bilingual space $\mathbf {\hat{z}}_{i} = \mathbf {a}_{i} \cdot M^{\prime }$ . Finally, we pass $\mathbf {\hat{z}}_{i}$ through a softmax layer $P$ to obtain the prediction $\hat{y}_{i} = \textrm {softmax} ( \mathbf {\hat{z}}_{i} \cdot P)$ .
Targeted Model
In our targeted model, we assume that the list of sentiment targets as they occur in the text is given. These can be extracted previously either by using domain knowledge Liu2005, by using a named entity recognizer Zhang2015 or by using a number of aspect extraction techniques Zhou2012. Given these targets, the task is reduced to classification. However, what remains is how to represent the target, to learn to subselect the information from the context which is relevant, how to represent this contextual information, and how to combine these representations in a meaningful way that enables us to classify the target reliably.
Our approach to adapt the Blse model to targeted sentiment analysis, which we call Split (depicted in Figure 2 ), is similar to the method proposed by Zhang2016 for gated recurrent networks. For a sentence with a target $a$ , we split the sentence at $a$ in order to get a left and right context, $\textrm {con}_\ell (a)$ and $\textrm {con}_r(a)$ respectively.
Unlike the approach from Zhang2016, we do not use recurrent neural networks to create a feature vector, as Atrio2019 showed that, in cross-lingual setups, they overfit too much to word order and source-language specific information to perform well on our tasks. Therefore, we instead average each left context $\textrm {con}_\ell (a_i)$ , right context $\textrm {con}_r(a_i)$ , and target $a_{i}$ separately. Although averaging is a simplified approach to create a compositional representation of a phrase, it has been shown to work well for sentiment Iyyer2015,Barnes2017. After creating a single averaged vector for the left context, right context, and target, we concatenate them and use these as input for the softmax classification layer $T \in \mathbb {R}^{d \times 3}$ , where $d$ is the dimensionality of the input vectors. The model is trained on the source language sentiment data using $M$ to project, and then tested by replacing $M$ with $M^{^{\prime }}$ , similar to the sentence-level model.
Experiments
In this section, we describe the resources and datasets, as well as the experimental setups used in both the sentence-level (Experiment 1 in Subsection "Setting for Experiment 1: Sentence-level Classification" ) and targeted (Experiment 2 in Subsection "Setting for Experiment 2: Targeted Classification" ) experiments.
Datasets and Resources
The number of datasets and resources for under-resourced languages are limited. Therefore, we choose a mixture of resource-rich and under-resourced languages for our experiments. We treat the resource-rich languages as if they were under-resourced by using similar amounts of parallel data.
To evaluate our proposed model at sentence-level, we conduct experiments using four benchmark datasets and three bilingual combinations. We use the OpeNER English and Spanish datasets Agerri2013 and the MultiBooked Catalan and Basque datasets BIBREF1 . All datasets contain hotel reviews which are annotated for targeted sentiment analysis. The labels include Strong Negative ( $--$ ), Negative ( $-$ ), Positive ( $+$ ), and Strong Positive ( $++$ ). We map the aspect-level annotations to sentence level by taking the most common label and remove instances of mixed polarity. We also create a binary setup by combining the strong and weak classes. This gives us a total of six experiments. The details of the sentence-level datasets are summarized in Table 1 .
For each of the experiments, we take 70 percent of the data for training, 20 percent for testing and the remaining 10 percent are used as development data for tuning meta-parameters.
We use the following corpora to set up the experiments in which we train on a source language corpus $C_{S}$ and test on a target language corpus $C_{T}$ . Statistics for all of the corpora are shown in Table 3 . We include a binary classification setup, where neutral has been removed and strong positive and strong negative have been mapped to positive and negative, as well as a multiclass setup, where the original labels are used.
OpeNER Corpora: The OpeNER corpora Agerri2013 are composed of hotel reviews, annotated for aspect-based sentiment. Each aspect is annotated with a sentiment label (Strong Positive, Positive, Negative, Strong Negative). We perform experiments with the English and Spanish versions.
MultiBooked Corpora: The MultiBooked corpora Barnes2018a are also hotel reviews annotated in the same way as the OpeNER corpora, but in Basque and Catalan. These corpora allow us to observe how well each approach performs on low-resource languages.
SemEval 2016 Task 5: We take the English and Spanish restaurant review corpora made available by the organizers of the SemEval event Pontiki2016. These corpora are annotated for three levels of sentiment (positive, neutral, negative).
USAGE Corpora: The USAGE corpora Klinger2014a are Amazon reviews taken from a number of different items, and are available in English and German. Each aspect is annotated for three levels of sentiment (positive, neutral, negative). As the corpus has two sets of annotations available, we take the annotations from annotator 1 as the gold standard.
For Blse, VecMap, Muse, and MT, we require monolingual vector spaces for each of our languages. For English, we use the publicly available GoogleNews vectors. For Spanish, Catalan, and Basque, we train skip-gram embeddings using the Word2Vec toolkit with 300 dimensions, subsampling of $10^{-4}$ , window of 5, negative sampling of 15 based on a 2016 Wikipedia corpus (sentence-split, tokenized with IXA pipes Agerri2014 and lowercased). The statistics of the Wikipedia corpora are given in Table 2 .
For Blse, VecMap, Muse, and Barista, we also require a bilingual lexicon. We use the sentiment lexicon from HuandLiu2004 (to which we refer in the following as Hu and Liu) and its translation into each target language. We translate the lexicon using Google Translate and exclude multi-word expressions. This leaves a dictionary of 5700 translations in Spanish, 5271 in Catalan, and 4577 in Basque. We set aside ten percent of the translation pairs as a development set in order to check that the distances between translation pairs not seen during training are also minimized during training.
Setting for Experiment 1: Sentence-level Classification
We compare Blse (Sections UID23 – UID30 ) to VecMap, Muse, and Barista (Section "Previous Work" ) as baselines, which have similar data requirements and to machine translation (MT) and monolingual (Mono) upper bounds which request more resources. For all models (Mono, MT, VecMap, Muse, Barista), we take the average of the word embeddings in the source-language training examples and train a linear SVM. We report this instead of using the same feed-forward network as in Blse as it is the stronger upper bound. We choose the parameter $c$ on the target language development set and evaluate on the target language test set.
Upper Bound Mono. We set an empirical upper bound by training and testing a linear SVM on the target language data. Specifically, we train the model on the averaged embeddings from target language training data, tuning the $c$ parameter on the development data. We test on the target language test data.
Upper Bound MT. To test the effectiveness of machine translation, we translate all of the sentiment corpora from the target language to English using the Google Translate API. Note that this approach is not considered a baseline, as we assume not to have access to high-quality machine translation for low-resource languages of interest.
Baseline Unsup We compare with the unsupervised statistical machine translation approach proposed by artetxe2018emnlp. This approach uses a self-supervised method to create bilingual phrase embeddings which then populates a phrase table. Monolingual n-gram language models and an unsupervised variant of MERT are used to create a MT model which is improved through iterative backtranslation. We use the Wikipedia corpora from Section UID42 to create the unsupervised SMT system between English and the target languages and run the training proceedure with default parameters. Finally, we translate all test examples in the target languages to English.
Baseline VecMap. We compare with the approach proposed by Artetxe2016 which has shown promise on other tasks, e. g., word similarity. In order to learn the projection matrix $W$ , we need translation pairs. We use the same word-to-word bilingual lexicon mentioned in Section UID23 . We then map the source vector space $S$ to the bilingual space $\hat{S} = SW$ and use these embeddings.
Baseline Muse. This baseline is similar to VecMap but incorporates and adversarial objective as well as a localized scaling objective, which further improve the orthogonal refinement so that the two language spaces are even more similar.
Baseline Barista. The approach proposed by Gouws2015taskspecific is another appropriate baseline, as it fulfills the same data requirements as the projection methods. The bilingual lexicon used to create the pseudo-bilingual corpus is the same word-to-word bilingual lexicon mentioned in Section UID23 . We follow the authors' setup to create the pseudo-bilingual corpus. We create bilingual embeddings by training skip-gram embeddings using the Word2Vec toolkit on the pseudo-bilingual corpus using the same parameters from Section UID42 .
Our method: BLSE. Our model, Blse, is implemented in Pytorch Pytorch and the word embeddings are initialized with the pretrained word embeddings $S$ and $T$ mentioned in Section UID42 . We use the word-to-word bilingual lexicon from Section UID46 , tune the hyperparameters $\alpha $ , training epochs, and batch size on the target development set and use the best hyperparameters achieved on the development set for testing. ADAM Kingma2014a is used in order to minimize the average loss of the training batches.
Ensembles. In order to evaluate to what extent each projection model adds complementary information to the machine translation approach, we create an ensemble of MT and each projection method (Blse, VecMap, Muse, Barista). A random forest classifier is trained on the predictions from MT and each of these approaches.
Setting for Experiment 2: Targeted Classification
For the targeted classification experiment, we compare the same models mentioned above, but adapted to the setting using the Split method from Section "Targeted Model" .
A simple majority baseline sets the lower bound, while the MT-based model serves as an upper bound. We assume our models to perform between these two, as we do not have access to the millions of parallel sentences required to perform high-quality MT and particularly aim at proposing a method which is less resource-hungry.
We hypothesize that cross-lingual approaches are particularly error-prone when evaluative phrases and words are wrongly predicted. In such settings, it might be beneficial for a model to put emphasis on the target word itself and learn a prior distribution of sentiment for each target independent of the context. For example, if you assume that all mentions of Steven Segal are negative in movie reviews, it is possible to achieve good results Bird2009. On the other hand, it may be that there are not enough examples of target-context pairs, and that it is better to ignore the target and concentrate only on the contexts.
To analyze this, we compare our model to two simplified versions. In addition, this approach enables us to gain insight in the source of relevant information. The first is Target-only, which means that we use the model in the same way as before but ignore the context completely. This serves as a tool to understand how much model performance originates from the target itself.
In the same spirit, we use a Context-only model, which ignores the target by constraining the parameters of all target phrase embeddings to be the same. This approach might be beneficial over our initial model if the prior distribution between targets was similar and the context actually carries the relevant information.
As the baseline for each projection method, we assume all targets in each sentence respectively to be of the same polarity (Sent). This is generally an erroneous assumption, but can give good results if all of the targets in a sentence have the same polarity. In addition, this baseline provides us with the information about whether the models are able to handle information from different positions in the text.
Experiment 1: Sentence-level Classification
In Table 4 , we report the results of all four methods. Our method outperforms the other projection methods (the baselines VecMap, Muse, and Barista) on four of the six experiments substantially. It performs only slightly worse than the more resource-costly upper bounds (MT and Mono). This is especially noticeable for the binary classification task, where Blse performs nearly as well as machine translation and significantly better than the other methods. Unsup also performs similarly to Blse on the binary tasks, while giving stronger performance on the 4-class setup. We perform approximate randomization tests Yeh2000 with 10,000 runs and highlight the results that are statistically significant (*p $<$ 0.01) in Table 4 .
In more detail, we see that MT generally performs better than the projection methods (79–69 $\text{F}_1$ on binary, 52–44 on 4-class). Blse (75–69 on binary, 41–30 on 4-class) has the best performance of the projection methods and is comparable with MT on the binary setup, with no significant difference on binary Basque. VecMap (67–46 on binary, 35–21 on 4-class) and Barista (61–55 on binary, 40–34 on 4-class) are significantly worse than Blse on all experiments except Catalan and Basque 4-class. Muse (67–62 on binary, 45–34 on 4-class) performs better than VecMap and Barista. On the binary experiment, VecMap outperforms Barista on Spanish (67.1 vs. 61.2) and Catalan (60.7 vs. 60.1) but suffers more than the other methods on the four-class experiments, with a maximum $\text{F}_1$ of 34.9. Barista is relatively stable across languages. Unsup performs well across experiments (76–65 on binary, 49–39 on 4-class), even performing better than MT on both Catalan tasks and Spanish 4-class.
The Ensemble of MT and Blse performs the best, which shows that Blse adds complementary information to MT. Finally, we note that all systems perform worse on Basque. This is presumably due to the increased morphological complexity of Basque, as well as its lack of similarity to the source language English (Section UID102 ).
We analyze three aspects of our model in further detail: 1) where most mistakes originate, 2) the effect of the bilingual lexicon, and 3) the effect and necessity of the target-language projection matrix $M^{\prime }$ .
In order to analyze where each model struggles, we categorize the mistakes and annotate all of the test phrases with one of the following error classes: vocabulary (voc), adverbial modifiers (mod), negation (neg), external knowledge (know) or other. Table 5 shows the results.
Vocabulary: The most common way to express sentiment in hotel reviews is through the use of polar adjectives (as in “the room was great”) or the mention of certain nouns that are desirable (“it had a pool”). Although this phenomenon has the largest total number of mistakes (an average of 72 per model on binary and 172 on 4-class), it is mainly due to its prevalence. MT performed the best on the test examples which according to the annotation require a correct understanding of the vocabulary (81 $\text{F}_1$ on binary /54 $\text{F}_1$ on 4-class), with Blse (79/48) slightly worse. Muse (76/23), VecMap (70/35), and Barista (67/41) perform worse. This suggests that Blse is better than Muse, VecMap and Barista at transferring sentiment of the most important sentiment bearing words.
Negation: Negation is a well-studied phenomenon in sentiment analysis Pang2002,Wiegand2010,Zhu2014,Reitan2015 . Therefore, we are interested in how these four models perform on phrases that include the negation of a key element, for example “In general, this hotel isn't bad". We would like our models to recognize that the combination of two negative elements “isn't" and “bad" lead to a Positive label.
Given the simple classification strategy, all models perform relatively well on phrases with negation (all reach nearly 60 $\text{F}_1$ in the binary setting). However, while Blse performs the best on negation in the binary setting (82.9 $\text{F}_1$ ), it has more problems with negation in the 4-class setting (36.9 $\text{F}_1$ ).
Adverbial Modifiers: Phrases that are modified by an adverb, e. g., the food was incredibly good, are important for the four-class setup, as they often differentiate between the base and Strong labels. In the binary case, all models reach more than 55 $\text{F}_1$ . In the 4-class setup, Blse only achieves 27.2 $\text{F}_1$ compared to 46.6 or 31.3 of MT and Barista, respectively. Therefore, presumably, our model does currently not capture the semantics of the target adverbs well. This is likely due to the fact that it assigns too much sentiment to functional words (see Figure 6 ). Muse performs poorly on modified examples (20.3 $\text{F}_1$ ).
External Knowledge Required: These errors are difficult for any of the models to get correct. Many of these include numbers which imply positive or negative sentiment (350 meters from the beach is Positive while 3 kilometers from the beach is Negative). Blse performs the best (63.5 $\text{F}_1$ ) while MT performs comparably well (62.5). Barista performs the worst (43.6).
Binary vs. 4-class: All of the models suffer when moving from the binary to 4-class setting; an average of 26.8 in macro $\text{F}_1$ for MT, 31.4 for VecMap, 22.2 for Barista, 34.1 for Muse, and 36.6 for Blse. The vector projection methods (VecMap, Muse, and Blse) suffer the most, suggesting that they are currently more apt for the binary setting.
We analyze how the number of translation pairs affects our model. We train on the 4-class Spanish setup using the best hyper-parameters from the previous experiment.
Research into projection techniques for bilingual word embeddings Mikolov2013translation,Lazaridou2015,Artetxe2016 often uses a lexicon of the most frequent 8–10 thousand words in English and their translations as training data. We test this approach by taking the 10,000 word-to-word translations from the Apertium English-to-Spanish dictionary. We also use the Google Translate API to translate the NRC hashtag sentiment lexicon Mohammad2013 and keep the 22,984 word-to-word translations. We perform the same experiment as above and vary the amount of training data from 0, 100, 300, 600, 1000, 3000, 6000, 10,000 up to 20,000 training pairs. Finally, we compile a small hand translated dictionary of 200 pairs, which we then expand using target language morphological information, finally giving us 657 translation pairs. The macro $\text{F}_1$ score for the Hu and Liu dictionary climbs constantly with the increasing translation pairs. Both the Apertium and NRC dictionaries perform worse than the translated lexicon by Hu and Liu, while the expanded hand translated dictionary is competitive, as shown in Figure 3 .
While for some tasks, e. g., bilingual lexicon induction, using the most frequent words as translation pairs is an effective approach, for sentiment analysis, this does not seem to help. Using a translated sentiment lexicon, even if it is small, gives better results.
The main motivation for using two projection matrices $M$ and $M^{\prime }$ is to allow the original embeddings to remain stable, while the projection matrices have the flexibility to align translations and separate these into distinct sentiment subspaces. To justify this design decision empirically, we perform an experiment to evaluate the actual need for the target language projection matrix $M^{\prime }$ : We create a simplified version of our model without $M^{\prime }$ , using $M$ to project from the source to target and then $P$ to classify sentiment.
The results of this model are shown in Figure 4 . The modified model does learn to predict in the source language, but not in the target language. This confirms that $M^{\prime }$ is necessary to transfer sentiment in our model.
Additionally, we provide an analysis of a similar model to ours, but which uses $M = \mathbb {R}^{d, o}$ and $M^{\prime } = \mathbb {R}^{d^{\prime }, o}$ , where $d$ ( $d^{\prime }$ ) is the dimensionality of the original embeddings and $o$ is the label size, to directly model crosslingual sentiment, such that the final objective function is
$$J =\hspace{-14.22636pt}\sum _{(x,y) \in C_{\textrm {source}}}\hspace{2.84526pt}\sum _{(s,t) \in L}\hspace{0.0pt}\alpha \cdot H(x, y) + (1 - \alpha ) \cdot || M \cdot s - M^{\prime } \cdot t ||$$   (Eq. 66)
thereby simplifying the model and removing the $P$ parameter. Table 6 shows that Blse outperforms this simplified model on all tasks.
In order to understand how well our model transfers sentiment information to the target language, we perform two qualitative analyses. First, we collect two sets of 100 positive sentiment words and one set of 100 negative sentiment words. An effective cross-lingual sentiment classifier using embeddings should learn that two positive words should be closer in the shared bilingual space than a positive word and a negative word. We test if Blse is able to do this by training our model and after every epoch observing the mean cosine similarity between the sentiment synonyms and sentiment antonyms after projecting to the joint space.
We compare Blse with VecMap and Barista by replacing the Linear SVM classifiers with the same multi-layer classifier used in Blse and observing the distances in the hidden layer. Figure 5 shows this similarity in both source and target language, along with the mean cosine similarity between a held-out set of translation pairs and the macro $\text{F}_1$ scores on the development set for both source and target languages for Blse, Barista, and VecMap. From this plot, it is clear that Blse is able to learn that sentiment synonyms should be close to one another in vector space and antonyms should have a negative cosine similarity. While the other models also learn this to some degree, jointly optimizing both sentiment and projection gives better results.
Secondly, we would like to know how well the projected vectors compare to the original space. Our hypothesis is that some relatedness and similarity information is lost during projection. Therefore, we visualize six categories of words in t-SNE, which projects high dimensional representations to lower dimensional spaces while preserving the relationships as best as possible Vandermaaten2008: positive sentiment words, negative sentiment words, functional words, verbs, animals, and transport.
The t-SNE plots in Figure 6 show that the positive and negative sentiment words are rather clearly separated after projection in Blse. This indicates that we are able to incorporate sentiment information into our target language without any labeled data in the target language. However, the downside of this is that functional words and transportation words are highly correlated with positive sentiment.
Finally, in order to analyze the sensitivity of the alpha parameter, we train Blse models for 30 epochs each with $\alpha $ between 0 and 1. Figure 7 shows the average cosine similarity for the translation pairs, as well as macro $\text{F}_1$ for both source and target language development data.
Values near 0 lead to poor translation and consecuently poor target language transfer. There is a rather large “sweet spot” where all measures perform best and finally, the translation is optimized to the detriment of sentiment prediction in both source and target languages with values near 1.
The experiments in this section have proven that it is possible to perform cross-lingual sentiment analysis without machine translation, and that jointly learning to project and predict sentiment is advantageous. This supports the growing trend of jointly training for multiple objectives Tang2014,Klinger2015,Ferreira2016.
This approach has also been exploited within the framework of multi-task learning, where a model learns to perform multiple similar tasks in order to improve on a final task Collobert2011a. The main difference between the joint method proposed here and multi-task learning is that vector space projection and sentiment classification are not similar enough tasks to help each other. In fact, these two objectives compete against one another, as a perfect projection would not contain enough information for sentiment classification, and vice versa.
Experiment 2: Targeted Classification
Table 7 shows the macro $\text{F}_1$ scores for all cross-lingual approaches (Blse, VecMap, Muse, Barista, MT, Unsup) and all targeted approaches (Sent, Split, Context-only, and Target-only). The final column is the average over all corpora. The final row in each setup shows the macro $\text{F}_1$ for a classifier that always chooses the majority class.
Blse outperforms other projection methods on the binary setup, 63.0 macro averaged $\text{F}_1$ across corpora versus 59.0, 57.9, and 51.4 for VecMap, Muse, and Barista, respectively. On the multiclass setup, however, Muse (32.2 $\text{F}_1$ ) is the best, followed by VecMap (31.0), Barista (28.1) and Blse (23.7). Unsup performs well across all experiments, achieving the best results on OpeNER ES (73.2 on binary and 42.7 on multiclass) and SemEval binary (77.1). VecMap is never the best nor the worst approach. In general, Barista performs poorly on the binary setup, but slightly better on the multiclass, although the overall performance is still weak. These results are similar to those observed in Experiment 1 for sentence classification.
The Split approach to ABSA improves over the Sent baseline on 33 of 50 experiments, especially on binary (21/25), while on multiclass it is less helpful (13/25). Both Sent and Split normally outperform Context-only or Target-only approaches. This confirms the intuition that it is important to take both context and target information for classification. Additionally, the Context-only approach always performs better than Target-only, which indicates that context is more important than the prior probability of an target being positive or negative.
Unlike the projection methods, MT using only the Sent representation performs well on the OpeNER and MultiBooked datasets, while suffering more on the SemEval and USAGE datasets. This is explained by the percentage of sentences that contain contrasting polarities in each dataset: between 8 and 12% for the OpeNER and Multibooked datasets, compared to 29% for SemEval or 50% for USAGE. In sentences with multiple contrasting polarities, the Sent baseline performs poorly.
Finally, the general level of performance of projection-based targeted cross-lingual sentiment classification systems shows that they still lag 10+ percentage points behind MT on binary (compare MT (72.9 $\text{F}_1$ ) with Blse (63.0)), and 6+ percentage points on multiclass (MT (38.8) versus Muse (32.2)). The gap between MT and projection-based approaches is therefore larger on targeted sentiment analysis than at sentence-level.
We perform a manual analysis of the targets misclassified by all systems on the OpeNER Spanish binary corpus (see Table 8 ), and found that the average length of misclassified targets is slightly higher than that of correctly classified targets, except for with VecMap. This indicates that averaging may have a detrimental effect as the size of the targets increases.
With the MT upperbounds, there is a non-negligible amount of noise introduced by targets which have been incorrectly translated (0.05% OpeNER ES, 6% MultiBooked EU, 2% CA, 2.5% SemEval, 1% USAGE). We hypothesize that this is why MT with Context-only performs better than MT with Split. This motivates further research with projection-based methods, as they do not suffer from translation errors.
The confusion matrices of the models on the SemEval task, shown in Figure 8 , show that on the multilabel task, models are not able to learn the neutral class. This derives from the large class imbalance found in the data (see Table 3 ). Similarly, models do not learn the Strong Negative class on the OpeNER and MultiBooked datasets.
Motivation
The performance of machine learning models on different target languages depends on the amount of data available, the quality of the data, and characteristics of the target language, e. g., morphological complexity. In the following, we analyze these aspects. There has been previous work that has observed target-language specific differences in multilingual dependency parsing Zeljko2016, machine translation Johnson2017, and language modeling Cotterell2018,Gerz2018. We are not aware of any work in cross-lingual sentiment analysis that explores the relationship between target language and performance in such depth and aim at improving this situation in the following.
Additionally, the effect of domain differences when performing cross-lingual tasks has not been studied in depth. Hangya2018 propose domain adaptation methods for cross-lingual sentiment classification and bilingual dictionary induction. They show that creating domain-specific cross-lingual embeddings improves the classification for English-Spanish. However, the source-language training data used to train the sentiment classifier is taken from the same domain as the target-language test data. Therefore, it is not clear what the effect of using source-language training data from different domains would be. We analyzed the model presented in Section "Sentence-level Model" in a domain adaptation setup, including the impact of domain differences Barnes2018c. The main result was that our model performs particularly well on more distant domains, while other approaches Chen2012,Ziser2017 performed better when the source and target domains were not too dissimilar.
In the following, we transfer this analysis to the target-based projection model in a real-world case study which mimics a user searching for the sentiment on touristic attractions. In order to analyze how well these methods generalize to new languages and domains, we deploy the targeted Blse, Muse, VecMap and MT models on tweets in ten Western European languages with training data from three different domains. Additionally, we include experiments with the Unsup models for a subset of the languages. English is the source language in all experiments, and we test on each of the ten target languages and attempt to answer the following research questions:
How much does the amount of monolingual data available to create the original embeddings effect the final results?
How do features of the target language, i. e. similarity to source language or morphological complexity, affect the performance?
How do domain mismatches between source-language training and target-language test data affect the performance?
Section "Discussion" addresses our findings regarding these questions and demonstrates that 1) the amount of monolingual data does not correlate with classification results, 2) language similarity between the source and target languages based on word and character n-gram distributions predicts the performance of Blse on new datasets, and 3) domain mismatch has more of an effect on the multiclass setup than binary.
Experimental Setup
We collect tweets directed at a number of tourist attractions in European cities using the Twitter API in 10 European languages, including several under-resourced languages (English, Basque, Catalan, Galician, French, Italian, Dutch, German, Danish, Swedish, and Norwegian). We detail the data collection and annotation procedures in Section UID85 . For classification, we compare MT the best performing projection-based methods (Blse, Muse, VecMap) using the Split method, detailed further in Section UID94 . As we need monolingual embeddings for all projection-based approaches, we create skipgram embeddings from Wikipedia dumps, detailed in Section UID91 .
As an experimental setting to measure the effectiveness of targeted cross-lingual sentiment models on a large number of languages, we collect and annotate small datasets from Twitter for each of the target languages, as well as a larger dataset to train the models in English. While it would be possible to only concentrate our efforts on languages with existing datasets in order to enable evaluation, this could give a distorted view of how well these models generalize. In order to reduce the possible ambiguity of the tourist attractions, we do not include those that have two or more obvious senses, e. g., Barcelona could refer either to the city or the football team.
In order to obtain a varied sample of tweets with subjective opinions, we download tweets that contain mentions of these tourist attractions as well as one of several emoticons or keywords. This distant supervision technique has been used to create sentiment lexicons Mohammad2016, semi-supervised training data Felbo2017, and features for a classifier Turney2003. We then remove any tweets that are less than 7 words long or which contain more than 3 hashtags or mentions. This increases the probability that a tweet text contains sufficient information for our use case setting.
We manually annotate all tweets for its polarity toward the target to insure the quality of the data. Note that we only annotate the sentiment towards the predefined list of targets, which leads to a single annotated target per tweet. Any tweets that have unclear polarity towards the target are assigned a neutral label. This produces the three class setup that is commonly used in the SemEval tasks Nakov2013,Nakov2016. Annotators were master's and doctoral students between 27 and 35 years old. All had either native or C1 level fluency in the languages of interest. Finally, for a subset of tweets in English, Catalan, and Basque two annotators classify each tweet. Table 11 shows three example tweets from English.
Table 10 depicts the number of annotated targets for all languages, as well as inter-annotator agreement using Cohen's $\kappa $ . The neutral class is the largest in all languages, followed by positive, and negative. These distributions are similar to those found in other Twitter crawled datasets Nakov2013,Nakov2016. We calculate pairwise agreement on a subset of languages using Cohen's $\kappa $ . The scores reflect a good level of agreement (0.62, 0.60, and 0.61 for English, Basque, and Catalan, respectively).
We collect Wikipedia dumps for ten languages; namely, Basque, Catalan, Galician, French, Italian, Dutch, German, Danish, Swedish, and Norwegian. We then preprocess them using the Wikiextractor script, and sentence and word tokenize them with either IXA pipes Agerri2014 (Basque, Galician, Italian, Dutch, and French), Freeling Padro2010 (Catalan), or NLTK Loper2002 (Norwegian, Swedish, Danish).
For each language we create Skip-gram embeddings with the word2vec toolkit following the pipeline and parameters described in Section UID42 . This process gives us 300 dimensional vectors trained on similar data for all languages. We assume that any large differences in the embedding spaces derive from the size of the data and the characteristics of the language itself. Following the same criteria laid out in Section UID46 , we create projection dictionaries by translating the Hu and Liu dictionary HuandLiu2004 to each of the target languages and keeping only translations that are single word to single word. The statistics of all Wikipedia corpora, embeddings, and projection dictionaries are shown in Table 12 .
Since we predetermine the sentiment target for each tweet, we can perform targeted experiments without further annotation. We use the Split models described in Section "Targeted Model" . Our model is the targeted Blse models described in Section "Targeted Model" . Additionally, we compare to the targeted Muse, VecMap, and MT models, as well as an Ensemble classifier that uses the predictions from Blse and MT before taking the largest predicted class for classification (see Section "Setting for Experiment 1: Sentence-level Classification" for details). Finally, we set a majority baseline by assigning the most common label (neutral) to all predictions. All models are trained for 300 epochs with a learning rate of 0.001 and $\alpha $ of 0.3.
We train the five models on the English data compiled during this study, as well as on the USAGE, and SemEval English data (the details can be found in Table 3 ) and test the models on the target-language test set.
Results
Table 13 shows the macro $\text{F}_1$ scores for all cross-lingual targeted sentiment approaches (Blse, Muse, VecMap, MT) trained on English data and tested on the target-language using the Split method proposed in "Targeted Model" . The final column is the average over all languages. Given the results from the earlier experiments, we hypothesize that MT should outperform Muse, VecMap and Blse for most of the languages.
On the binary setup, Blse outperforms all other cross-lingual methods including MT and Unsup, with 56.0 macro averaged $\text{F}_1$ across languages versus 48.7, 49.4, and 48.9 for Muse, VecMap, and MT respectively (54.1 across Basque and Catalan versus 46.0 for Unsup). Blse performs particularly well on Catalan (54.5), Italian (63.4), Swedish (65.3), and Danish (68.3). VecMap performs poorly on Galician (33.3), Italian (38.2), and Danish (43.4), but outperforms all other methods on Basque (56.4), Dutch (55.2) and Norwegian (59.0). MT performs worse than Blse and VecMap, although it does perform best for Galician (56.5). Unlike experiments in Section "Sentence-level Model" , the ensemble approach does not perform better than the individual classifiers and Muse leads to the classifier with the lowest performance overall. Unsup performs better than MT on both Basque and Catalan.
On the multiclass setup, however, MT (36.6 $\text{F}_1$ ) is the best, followed by VecMap (34.1), Blse (32.6), and Muse (26.1). Compared to the experiments on hotel reviews, the average differences between models is small (2.5 percentage points between MT and VecMap, and 1.5 between VecMap and Blse). Unsup performs better than MT on Basque (40.1), but worse on Catalan (28.5). Again, all methods outperform the majority baseline.
On both the binary and multiclass setups, the best overall results are obtained by testing and training on data from the same domain (56.0 $\text{F}_1$ for Blse and 36.6 $\text{F}_1$ for MT). Training MT, Muse, and VecMap on the SemEval data performs better than training on USAGE, however.
An initial error analysis shows that all models suffer greatly on the negative class. This seems to suggest that negative polarity towards a target is more difficult to determine within these frameworks. A significant amount of the tweets that have negative polarity towards a target also express positive or neutral sentiment towards other targets. The averaging approach to create the context vectors does not currently allow any of the models to exclude this information, leading to poor performance on these instances.
Finally, compared to the experiments performed on hotel and product reviews in Section "Experiments" , the noisy data from Twitter is more difficult to classify. Despite the rather strong majority baseline (an average of 40.5 Macro $\text{F}_1$ on binary), no model achieves more than an average of 56 Macro $\text{F}_1$ on the binary task. A marked difference is that Blse and VecMap outperform MT on the binary setup. Unlike the previous experiment, Muse performs the worst on the multiclass setup. The other projection methods obtain multiclass results similar to the previous experiment (32.6–34.1 $\text{F}_1$ here compared to 23.7–31.0 $\text{F}_1$ previously).
Discussion
In this section, we present an error analysis. Specifically, Table 14 shows examples where Blse correctly predicts the polarity of a tweet that MT and Unsup incorrectly predict, and vice versa, as well as examples where all models are incorrect.
In general, in examples where Blse outperforms MT and Unsup, the translation-based approaches often mistranslate important sentiment words, which leads to prediction errors. In the first Basque tweet, for example, “#txindoki igo gabe ere inguruaz goza daiteke... zuek joan tontorrera eta utzi arraroei gure kasa...”, Unsup incorrectly translates the most important sentiment word in the tweet “goza” (enjoy) to “overlook” and subsequently incorrectly predicts that the polarity towards txindoki is negative.
Tweets that contain many out-of-vocabulary words or non-standard spelling (due to dialectal differences, informal writing, etc.), such as the third tweet in Table 14 , “kanpora jun barik ehko asko: anboto, txindoki”, are challenging for all models. In this example “jun” is a non-standard spelling of “joan” (go), “barik” is a Bizcayan Basque variant of “gabe” (without) , and “ehko” is an abbreviation of “Euskal Herriko” (Basque Country's). These lead to poor translations for MT and Unsup, but pose a similar out-of-vocabulary problem for Blse.
In order to give a more qualitative view of the targeted model, Figure 9 shows t-sne projections of the bilingual vector space before and after training on the Basque binary task, following the same proceedure mentioned in Section UID68 . As in the sentence-level experiment, there is a separation of the positive and negative sentiment words, although it is less clear for targeted sentiment. This is not surprising, as a targeted model must learn not only the prior polarity of words, but how they interact with targets, leading to a more context-dependent representation of sentiment words.
Finally, we further analyze the effects of three variables that are present in cross-lingual sentiment analysis: a) availability of monolingual unlabeled data, b) similarity of source and target languages, and c) domain shift between the source language training data and the target language test data.
We pose the question of what the relationship is between the amount of available monolingual data to create the embedding spaces and the classification results of the models. If the original word embedding spaces are not of high quality, this could make it difficult for the projection-based models to create useful features. In order to test this, we perform ablation experiments by training target-language embeddings on varying amounts of data ( $1 \times 10^{4}$ to $5 \times 10^{9}$ tokens) and testing the models replacing the full target-language embeddings with these. We plot the performance of the models as a function of available monolingual data in Figure 10 .
Figure 10 shows that nearly all models, with the exception of Norwegian, perform poorly with very limited monolingual training data ( $1\times 10^{4}$ ) and improve, although erratically, with more training data. Interestingly, the models require little data to achieve results comparable to using the all tokens to train the embeddings. A statistical analysis of the amount of unlabeled data available and the performance of Blse, Muse, VecMap (Pearson's $r$ = $-0.14$ , $-0.27$ , $0.08$ , respectively) reveals no statistically significant correlation between them. This seems to indicate that all models are not sensitive to the amount of monolingual training data available in the target language.
One hypothesis to different results across languages is that the similarity of the source and target language has an effect on the final classification of the models. In order to analyze this, we need a measure that models pairwise language similarity. Given that the features we use for classification are derived from distributional representations, we model similarity as a function of 1) universal POS-tag n-grams which represent the contexts used during training, and 2) character n-grams, which represent differences in morphology. POS-tag n-grams have previously been used to classify genre Fang2010, improve statistical machine translation Lioma2005, and the combination of POS-tag and character n-grams have proven useful features for identifying the native language of second language writers in English Kulmizev2017. This indicates that these are useful features for characterizing a language. In this section we calculate the pairwise similarity between all languages and then check whether this correlates with performance.
After POS-tagging the test sentences obtained from Twitter using the universal part of speech tags Petrov2012, we calculate the normalized frequency distribution $P_{l}$ for the POS-tag trigrams and $C_{l}$ for character trigrams for each language $l$ in $L = \lbrace \textrm {Danish, Swedish, Norwegian, Italian, Basque, Catalan, French, Dutch, Galician,}$
$\textrm {German, English}\rbrace $ . We then compute the pairwise cosine similarity between $\cos (A, B) = \frac{A \cdot B}{||A|| \: ||B||} $ where $A$ is the concatenation of $P_{l_{i}}$ and $C_{l_{i}}$ for language $l_{i}$ and $B$ is the concatenation of $P_{l_{j}}$ and $C_{l_{j}}$ for language $l_{j}$ .
The pairwise similarities in Figure 11 confirm to expected similarities, and language families are clearly grouped (Romance, Germanic, Scandinavian, with Basque as an outlier that has no more than 0.47 similarity with any language). This confirms the use of our similarity metric for our purposes. We plot model performance as a function of language similarity in Figure 12 . To measure the correlation between language similarity and performance, we calculate Pearson's $r$ and find that for Blse there is a strong correlation between language similarity and performance, $r = 0.76$ and significance $p < 0.01$ . Muse, VecMap and MT do not show these correlations ( $r$ = 0.41, 0.24, 0.14, respectively). For MT this may be due to robust machine translation available in less similar languages according to our metric, e. g., German-English. For Muse and VecMap, however, it is less clear why it does not follow the same trend as Blse.
In this section, we determine the effect of source-language domain on the cross-lingual sentiment classification task. Specifically, we use English language training data from three different domains (Twitter, restaurant reviews, and product reviews) to train the cross-lingual classifiers, and then test on the target-language Twitter data. In monolingual sentiment analysis, one would expect to see a drop when moving to more distant domains.
In order to analyze the effect of domain similarity further, we test the similarity of the domains of the source-language training data using Jensen-Shannon Divergence, which is a smoothed, symmetric version of the Kullback-Leibler Divergence, $D_{KL}(A||B) = \sum _{i}^{N} a_{i} \log \frac{a_{i}}{b_{i}}$ . Kullback-Leibler Divergence measures the difference between the probability distributions $A$ and $B$ , but is undefined for any event $a_{i} \in A$ with zero probability, which is common in term distributions. Jensen-Shannon Divergence is then $ D_{JS}(A,B) = \frac{1}{2} \Big [ D_{KL}(A||B) + D_{KL}(B||A) \Big ]\,. $
Our similarity features are probability distributions over terms $t \in \mathbb {R}^{|V|}$ , where $t_{i}$ is the probability of the $i$ -th word in the vocabulary $V$ . For each domain, we create frequency distributions of the most frequent 10,000 unigrams that all domains have in common and measure the divergence with $D_{JS}$ .
The results shown in Table 15 indicate that both the SemEval and USAGE datasets are relatively distinct from the Twitter data described in Section UID85 , while they are more similar to each other. Additionally, we plot the results of all models with respect to the training domain in Figure 13 .
We calculate Pearson's $r$ on the correlation between domain and model performance, shown in Table 16 . On the binary setup, the results show a negligible correlation for Blse (0.32), with no significant correlation for Muse, VecMap or MT. This suggests that the models are relatively robust to domain noise, or rather that there is so much other noise found in the approaches that domain is less relevant. On the multiclass setup, however, there is a significant effect for all models. This indicates that the multiclass models presented here are less robust than the binary models.
Both the SemEval and USAGE corpora differ equally from the Twitter data given the metric defined here. The fact that models trained on SemEval tend to perform better than those trained on USAGE, therefore, seems to be due to the differences in label distribution, rather than to differences in domain. These label distributions are radically different in the multiclass setup, as the English Twitter data has a 30/50/20 distribution over Positive, Neutral, and Negative labels (67/1/32 and 68/4/28 for USAGE and SemEval, respectively). Both undersampling and oversampling help, but the performance is still worse than training on in-domain data.
The case study which we presented in this section showed results of deploying the models from Section "Projecting Sentiment Across Languages" to real world Twitter data, which we collect and annotate for targeted sentiment analysis. The analysis of different phenomena revealed that for binary targeted sentiment analysis, Blse performs better than machine translation on noisy data from social media, although it is sensitive to differences between source and target languages. Finally, there is little correlation between performance on the cross-lingual sentiment task and the amount of unlabeled monolingual data used to create the original embeddings spaces which goes against our expectations.
Unlike the experiments in Section "Sentence-level Model" , the ensemble classifier employed here was not able to improve the results. We assume that the small size of the datasets in this experiment does not enable the classifier to learn which features are useful in certain contexts.
One common problem that appears when performing targeted sentiment analysis on noisy data from Twitter is that many of the targets of interest are ambiguous, which leads to false positives. Even with relatively unambiguous targets like “Big Ben”, there are a number of entities that can be referenced; Ben Rothlisberger (an American football player), an English language school in Barcelona, and many others. In order to deploy a full sentiment analysis system on Twitter data, it will be necessary to disambiguate these mentions before classifying the tweets, either as a preprocessing step or jointly.
In sentiment analysis, it is not yet common to test a model on multiple languages, despite the fact that current state-of-the-art models are often theoretically language-agnostic. This section shows that good performance in one language does not guarantee that a model transfers well to other languages, even given similar resources. We hope that future work in sentiment analysis will make better use of the available test datasets.
Conclusion
With this article, we have presented a novel projection-based approach to targeted cross-lingual sentiment analysis. The central unit of the proposed method is Blse which enables the transfer of annotations from a source language to a non-annotated target language. The only input it relies on are word embeddings (which can be trained without manual labeling by self-annotation) and a comparably small translation dictionary which connects the semantics of the source and the target language.
In the binary classification setting (automatic labeling of sentences or documents), Blse constitutes a novel state of the art on several language and domain pairs. For a more fine-grained classification to four sentiment labels, Barista and Muse perform slightly better. The predictions in all settings are complementary to the strong upper bound of employing machine translations: in an ensemble, even this resource-intense approach is inferior.
The transfer from classification to target-level analysis revealed additional challenges. The performance is lower, particularly for the 4-class setting. Our analyses show that mapping of sentence predictions to the aspects mentioned in each sentence with a machine translation model is a very challenging empirical upper bound – the difference in performance compared to projection-based methods is greater here than for the sentence-classification setting. However, we showed that in resource-scarce environments, Blse constitutes the current state of the art for binary target-level sentiment analysis when incorporated in a deep learning architecture which is informed about the aspect. Muse performs better in the same architecture for the 4-class setting.
Our analysis further showed that the neural network needs to be informed about both the aspect and the context – limiting the information to a selection of these sentence parts strongly underperforms the combined setting. That also demonstrates that the model does not rely on prior distributions of aspect mentions.
The final experiment in the paper is a real-world deployment of the target-level sentiment analysis system in multilingual setting with 10 languages, where the assumption is that the only supervision is available in English (which is not part of the target languages). We learned here that it is important to have access to in-domain data (even for cross-lingual projection), especially in the multiclass setting. Binary classification however, which might often be sufficient for real-world applications, is more robust to domain changes. Further, machine translation is less sensitive to language dissimilarities, unlike projection-based methods. The amount of available unlabeled data to create embeddings plays a role in the final performance of the system, although only to a minor extent.
The current performance of the projection-based techniques still lags behind state-of-the-art MT approaches on most tasks, indicating that there is still much work to be done. While general bilingual embedding techniques do not seem to incorporate enough sentiment information, they are able to retain the semantics of their word vectors to a large degree even after projection. We hypothesize that the ability to retain the original semantics of the monolingual spaces leads to Muse performing better than MT on multiclass targeted sentiment analysis. The joint approach introduced in this work suffers from the degradation of the original semantics space, while optimizing the sentiment information. Moving from a similarity-based loss to a ranking loss, where the model must predict a ranked list of most similar translations could improve the model, but would require further resource development cross-lingually, as a simple bilingual dictionary would not provide enough information.
One problem that arises when using bilingual embeddings instead of machine translation is that differences in word order are no longer handled BIBREF2 . Machine translation models, on the other hand, always include a reordering element. Nonetheless, there is often a mismatch between the real source language word order and the translated word order. In this work, we avoided the problem by using a bag-of-embeddings representation, but Barnes2017 found that the bag-of-embeddings approach does not perform as well as approaches that take word order into account, e. g., Lstms or Cnns. We leave the incorporation of these classifiers into our framework for future work.
Unsupervised machine translation Artetxe2018,Lample2018,artetxe2018emnlp shows great promise for sentence-level classification. Like MT, however, it performs worse on noisy data, such as tweets. Therefore, users who want to apply targeted cross-lingual approaches to noisy data should consider currently consider using embedding projection methods, such as Blse. Future work on adapting unsupervised machine translation to noisy text may provide another solution for low-resource NLP.
The authors thank Patrik Lambert, Toni Badia, Amaia Oliden, Itziar Etxeberria, Jessie Kief, Iris Hübscher, and Arne Øhm for helping with the annotation of the resources used in this research. This work has been partially supported by the DFG Collaborative Research Centre SFB 732 and a SGR-DTCL Predoctoral Scholarship.

Question: what baseline do they compare to?
Answer: VecMap, Muse, Barista