Venue: EMNLP 2017

# Towards a Universal Sentiment Classifier in Multiple languages

Kui Xu

Xiaojun Wan

Institute of Computer Science and Technology, Peking University

The MOE Key Laboratory of Computational Linguistics, Peking University

{kuixu, wanxiaojun}@pku.edu.cn

###### Abstract

Existing sentiment classifiers usually work for only one specific language, and different classification models are used in different languages. In this paper we aim to build a universal sentiment classifier with a single classification model in multiple different languages. In order to achieve this goal, we propose to learn multilingual sentiment-aware word embeddings simultaneously based only on the labeled reviews in English and unlabeled parallel data available in a few language pairs. It is not required that the parallel data exist between English and any other language, because the sentiment information can be transferred into any language via a pivot languages. We present the evaluation results of our universal sentiment classifier in five languages, and the results are very promising even when the parallel data between English and the target languages are not used. Furthermore, the u-universal single classifier is compared with a few cross-language sentiment classifiers relying on direct parallel data between the source and target languages, and the results show that the performance of our universal sentiment classifier is very promising compared to that of different cross-language classifiers in multiple target languages.

## 1 Introduction

Nowadays, a large amount of user-generated content (UGC) appears online everyday, such as tweets, comments and product reviews. Sentiment classification on these data has become a popular research topic over the past few years Pang et al. (2002); Blitzer et al. (2007); Agarwal et al. (2011); Li-u (2012). Distributed representations of words or word embeddings have been widely explored, and have proved its great usability for the sentiment classification task Tang et al. (2014); Zhou et al. (2015); Xu et al. (2015); Bollegala et al. (2016); Ferreira et al. (2016).

Most existing sentiment classifiers rely on labeled training data and the data are usually language-dependent. In other words, a sentiment classifier is learned from a labeled dataset in a specific language and this sentiment classifier can be used for sentiment classification in this language. However, labeled training data for sentiment classification are not available or not easy to obtain in many languages in the world (e.g., Malaysian, Mongolian, Uighur). Without reliable labeled data, it is hard to build a sentiment classifier in these resource-poor languages.

Fortunately, there are a few studies investigating the task of cross-language sentiment classification Banea et al. (2008); Wan (2009); Meng et al. (2012); Xiao and Guo (2013); Gao et al. (2015); Chen et al. (2015); Zhou et al. (2015); Li et al. (2017); Zhou et al. (2016), which aims to make use of the labeled data in a source language (English in most cases) to build a sentiment classifier in a target language. However, cross-language sentiment classification methods rely on parallel data between the source and target languages1 In a resource-poor language, the parallel data between this language and the source language may not be available or is not easy to obtain. In this circumstance, previous cross-language sentiment classification meth

ods will fail to work.

Another shortcoming of previous cross-language sentiment classification researches is that we have to build an individual cross-language sentiment classifier for each target language, even when we want to perform sentiment classification in a couple of languages at the same time.

In this study, instead of building a sentiment classifier for each target language, we aim to build a universal sentiment classifier in multiple languages and this universal sentiment classifier only learns one single sentiment classification model and it can be applied for sentiment classification in many languages.

In order to achieve this goal, we propose an approach to learn multilingual sentiment-aware word embeddings simultaneously based only on the labeled reviews in English and unlabeled parallel data available in a few language pairs. As mentioned earlier, in some resource-poor languages, there do not exist direct parallel data between these languages and the source English language. In order to address this problem, we propose a pivot-based model to transfer the sentiment information from the source language to any resource-poor language via pivot languages. Finally, a universal sentiment classifier can be built because the multilingual word embeddings are in the same semantic space.

We build three different models (Bilingual Model, Pivot-Driven Bilingual Model and Universal Multilingual Model) and compare them empirically in order to answer two questions in this paper: 1) Can pivot-based models learn bilingual sentiment-aware word embeddings effectively? 2) Can an effective universal sentiment classifier be built for multiple languages?

Without loss of generality, we present and compare the evaluation results of the models in five languages. Evaluation results show that pivot-driven bilingual models perform as well as the bilingual model using direct parallel data, which lays the solid foundation of our universal model. Moreover, it is very promising that our universal sentiment classifier can work well in five languages, and it can achieve very promising classification results as compared to several typical cross-language sentiment classification models.

The main contributions of our study in this paper are summarized as follows:

* We are the first to build a universal sentiment classifier in multiple languages by learning multilingual sentiment-aware word embeddings, which can not be addressed by previous researches on cross-language sentiment classification.
* We propose pivot-based models to bridge two languages in which there do not exist parallel data, and thus the sentiment information can be transferred to any target language.
* Evaluation results on five languages demonstrate the efficacy of our proposed pivot-based models and the universal sentiment classifier.

## 2 Our Approach

In order to build a universal sentiment classifier, we propose an approach to learn multilingual sentiment-aware word embeddings simultaneously, and then train a universal sentiment classification model in the embedding space by averaging the word embeddings in a document as the document representation. Note that in this study, we focus on only using the labeled data in English and do not make use of any labeled data in other languages, which makes the task more challenging2. Formally, we aims to build a single sentiment classifier which can perform sentiment classification in many languages \(\{S,T_{1},T_{2},...,T_{N}\}\), where \(S\) refers to English language, and \(T_{1}\) to \(T_{N}\) refer to other \(N\) languages.

Footnote 2: Note that the labeled data in other languages can be easily used by our approach in the same way as the English labeled data, and we believe more labeled data will eventually improve the performance of the sentiment classifier.

In our approach, the multilingual sentiment-aware word embeddings play the key role in building the universal sentiment classifier, and now the question is how to learn the multilingual sentiment-aware word embeddings? Inspired by previous studies on cross-lingual sentiment classification and bilingual word embedding learning, we can leverage the labeled data in \(S\) (i.e., English) and unlabeled parallel data between \(S\) and language \(T\) to learn bilingual sentiment-aware word embeddings in both English and \(T\) languages with a bilingual model. However, such unlabeled parallel data are not always easy to obtain for all other languages. For a specific language \(T\), if the unlabeled parallel data between \(T\) and \(S\) do not exist, the bilingual model cannot be applied. In order to address this problem, we propose a pivot-driven bilingual model to 

to bridge \(T\) and \(S\). We choose a pivot language \(P\) where the parallel data between \(P\) and \(S\), and the parallel data between \(P\) and \(T\) are easy to obtain, and then leverage them to learn the multilingual sentiment-aware word embeddings in the three languages \(P\), \(T\) and \(S\). Furthermore, we can leverage more parallel data between multiple languages, some of which are parallel data between \(S\) and other languages, and some of which are parallel data within other languages, to build an universal multilingual model. The sentiment information will be directly or indirectly transfered to each language as well, and thus we obtain multilingual sentiment-aware word embeddings in many languages.

The bilingual model, pivot-driven bilingual model and universal multilingual model will be described in next sections, respectively.

### Bilingual Model

The bilingual model tries to induce bilingual word embeddings from a parallel corpus, and in the meantime make similar words from the two languages share adjacent vector representations in the same vector space.

Formally, we assume a source language \(S\) with \(|S|\) words and a target language \(T\) with \(|T|\) words. We use \(s\) and \(t\) to represent a word from \(S\) and \(T\), respectively. Given the bilingual parallel corpus \({\cal C}\) between language \(S\) and \(T\), it can be divided into a corpus \({\cal C_{S}}\) in language \(S\) and a corpus \({\cal C_{I}}\) in language \(T\). And we use a notation \(S-T\) to indicate a parallel corpus between languages \(S\) and \(T\).

Previous studies have proposed some bilingual models for learning bilingual word embeddings, so we extend the well-behaved BiSkip model Luong et al. (2015) to Bilingual Model (**BM**). This model requires word alignment information, and in this study word alignment is automatically obtained from parallel sentences by using a word alignment tool.

In our bilingual model, every word \(s\) in language \(S\) is required to predict the adjacent words of itself and the aligned word \(t\) in the target language \(T\). For corpus \({\cal C_{S}}\), the monolingual constraint on itself (\({\cal C_{S}}\rightarrow{\cal C_{S}}\)) is:

\[Obj({\cal C_{S}}|{\cal C_{S}})=\sum_{s\in{\cal C_{S}}}\sum_{w\in adj(s)}\log p (w|s), \tag{1}\]

and the cross-lingual constraint on \({\cal C_{T}}\) (\({\cal C_{S}}\rightarrow{\cal C_{T}}\)) is:

\[Obj({\cal C_{T}}|{\cal C_{S}})=\sum_{s\in{\cal C_{S}}}\sum_{w\in adj(t),s \leftrightarrow t}\log p(w|s) \tag{2}\]

where \(s\mbox{$\leftrightarrow$}t\) means word \(s(\in~{}{\cal C_{S}})\) is aligned to word \(t(\in{\cal C_{T}})\) and \(adj(s)\) or \(adj(t)\) mean the adjacent words of word \(s\) or \(t\).

Similarly, for corpus \({\cal C_{T}}\) we can obtain:

\[Obj({\cal C_{T}}|{\cal C_{T}})=\sum_{t\in{\cal C_{T}}}\sum_{w\in adj(t)}\log p (w|t), \tag{3}\]

and

\[Obj({\cal C_{S}}|{\cal C_{T}})=\sum_{t\in{\cal C_{T}}}\sum_{w\in adj(s),t \leftrightarrow s}\log p(w|t) \tag{4}\]

Combining equations 1, 2, 3 and 4, we get the objective for obtaining bilingual word embeddings from parallel corpus:

\[Obj({\cal C})=\alpha_{1} Obj({\cal C_{S}}|{\cal C_{S}})+\alpha_{2}Obj({\cal C_{T}}|{\cal C_{ S}})\] \[+\alpha_{3} Obj({\cal C_{T}}|{\cal C_{T}})+\alpha_{4}Obj({\cal C_{S}}|{ \cal C_{T}})\]

where \(\alpha_{1},\alpha_{2},\alpha_{3}\) and \(\alpha_{4}\) are scalar parameters.

We still have to incorporate the sentiment information into the bilingual word embeddings. Similar to previous studies Zhou et al. (2015), we make use of the sentiment polarity of texts as supervision in the learning process. Given a labeled semitental corpus \({\cal C_{L}}^{3}\), we use \(S^{*}\) to represent a sentence in \({\cal C_{L}}\) and \(w\) as a word in \(S^{*}\). And \(x^{T}\) is a sum of word embeddings in \(S^{*}\). We simply adopt the logistic regression classifier to enforce the sentiment constraint, and thus make the bilingual word embeddings absorb the corresponding sentiment information. The objective function is:

\[L({\cal C_{L}})=\sum_{S^{*}\in{\cal C_{L}}}y\log\sigma(Wx^{T}+b)\] \[+(1-y)\log\sigma(1-(Wx^{T}+b)) \tag{5}\]

where \(y\) is the label of the sentence \(S^{*}\), \(W\) is a weight vector and \(b\) is a bias.

The overall objective function for inducing bilingual sentiment-aware word embeddings is to maximize:

\[Obj({\cal C})+L({\cal C_{L}})\] 



### Pivot-Driven Bilingual Model

For some resource-poor target language \(T\), it is quite expensive to get direct parallel corpus between \(T\) and the source language \(S\). Without such parallel corpus, it is not possible to apply the above bilingual model to learn bilingual sentiment-aware word embeddings. In order to address this problem we propose our Vivot-Driven Bilingual Model **(PDBM)** by using a pivot language to bridge \(T\) and \(S\). The model is inspired by Wu2007, in which pivot languages are used for phrase-based SMT. A pivot language \(P\) is chosen if the parallel corpus between \(P\) and \(S\), and the parallel corpus between \(P\) and \(T\) are easy to obtain. Given two parallel corpora: \(S\)-\(P\) and \(P\)-\(T\), our PDBM model tries to get the trilingual word embeddings by putting constrains on the two corpora. Under the well-designed constraint, the pivot language \(P\) can pass the sentiment information from the source language \(S\) to the target language \(T\). Similarly, we further assume the pivot language \(P\) with \(|P|\) words, and use \({\cal C}_{\mathcal{P}}\) to denote the corpus in language \(P\).

We design constraints on the two parallel corpora \(S\)-\(P\) and \(P\)-\(T\), instead of direct constraints on \(S\) and \(T\). Derived from the BM model, we can get three monolingual constraints \({\cal C}_{\mathcal{S}}\)\(\rightarrow\)\({\cal C}_{\mathcal{S}}\), \({\cal C}_{\mathcal{T}}\)\(\rightarrow\)\({\cal C}_{\mathcal{T}}\), \({\cal C}_{\mathcal{P}}\)\(\rightarrow\)\({\cal C}_{\mathcal{P}}\) and four bilingual constraints \({\cal C}_{\mathcal{S}}\)\(\rightarrow\)\({\cal C}_{\mathcal{P}}\), \({\cal C}_{\mathcal{T}}\)\(\rightarrow\)\({\cal C}_{\mathcal{P}}\), \({\cal C}_{\mathcal{P}}\)\(\rightarrow\)\({\cal C}_{\mathcal{S}}\) and \({\cal C}_{\mathcal{P}}\)\(\rightarrow\)\({\cal C}_{\mathcal{T}}\). The final objective function for learning the trilingual word embeddings can be summarized as:

\[Obj_{p}({\cal C})=\beta_{1} Obj({\cal C}_{\mathcal{S}}|{\cal C}_{\mathcal{S}})+\beta_{2} Obj({\cal C}_{\mathcal{S}}|{\cal C}_{\mathcal{P}})\] \[+\beta_{3}Obj({\cal C}_{\mathcal{T}}|{\cal C}_{\mathcal{T}})+ \beta_{4}Obj({\cal C}_{\mathcal{T}}|{\cal C}_{\mathcal{P}})\] \[+\beta_{5}Obj({\cal C}_{\mathcal{P}}|{\cal C}_{\mathcal{S}})+ \beta_{6}Obj({\cal C}_{\mathcal{P}}|{\cal C}_{\mathcal{T}})\] \[+\beta_{7}Obj({\cal C}_{\mathcal{P}}|{\cal C}_{\mathcal{P}})\]

where \(\beta_{1},\beta_{2},\beta_{3},\beta_{4},\beta_{5},\beta_{6},\beta_{7}\) are scalar parameters. Similarly, the objective for enforcing the sentiment constraint is the same as equation 5, so we combine them together to get the overall objective function:

\[Obj_{p}({\cal C})+L({\cal C}_{\mathcal{L}})\]

Through the pivot language, the sentiment information can be passed from a source language to a target language by maximizing the above objective function.

### Universal Multilingual Model

The bilingual model and the pivot-driven bilingual model lay the foundations of build a universal multilingual model for sentiment classification in many languages. Given a source language \(S\) and a few other languages \(\{T_{1},T_{2},...,T_{N}\}\). If there exist parallel data between a language \(T_{i}\) and \(S\), then the bilingual sentiment-aware word embeddings can be learned by the bilingual model. If the parallel data between languages \(T_{i}\) and \(S\) are not available, a pivot language can be selected and the pivot-driven model can be applied to learn the trilingual sentiment-aware word embeddings. Even when a single pivot language cannot be found for languages \(T_{i}\) and \(S\), we still can find two or more pivot languages \(\{P_{1},P_{2},...,P_{M}\}\) to form a pivot chain and the sentiment information in the source language can be passed through the pivot chain (\(S-P_{1}-...-P_{M}-T_{i}\)) to the target language.

Therefore, in this model, we will make use of all parallel corpora between any pair of languages (including parallel corpora between the source language and any other language, and parallel corpora between other languages) and learn the sentiment-aware word embeddings in all the languages simultaneously. The monolingual objective in each language and the cross-lingual objective for any available parallel corpus are defined in the same way as in the above models, and we sum all the objectives and denote it as \(Obj_{universal}(C)\), and this objective is then combined with the sentiment constraint as follows:

\[Obj_{universal}({\cal C})+L({\cal C}_{\mathcal{L}})\]

By maximizing the above objective function, the sentiment-aware word embeddings in all the languages will be learned.

## 3 Evaluations

### Dataset

Without loss of generality, we evaluate our models in five languages (including three western languages and two Asian languages): English (en), German (de), French (fr), Japanese (jp) and Chinese (en/zh). Among these languages, the English language is the source language with labeled training data, and we do no use any labeled data in the other languages

Particularly, we use the multilingual multi-domain Amazon review dataset 4 provided by Prettenhofer and Stein, 2010) and the NLPC-C2013 dataset 5. The review dataset provided by Prettenhofer and Stein, 2010) contains labeled data in four languages: English, German, French and Japanese, and the NLPCC2013 dataset further provides labeled data in Chinese. The reviews in each language are divided into three domains: _dvd_, _music_ and _books_. Each domain of product reviews contains a balanced training set and test set, each of which consists of 1000 positive and 1000 negative reviews for each language except for Chinese. While for Chinese language, the test set consists of 2000 positive and 2000 negative reviews. We only use English training data as the labeled data in the experiments.

Footnote 4: [https://www.uni-weimar.de/medien/webis/corpora/cporus-weighs-cls-l0/](https://www.uni-weimar.de/medien/webis/corpora/cporus-weighs-cls-l0/)

We further obtain unlabeled parallel data from Europarl v7 6(Koehn, 2004) (_Eu v7_) and The United Nations Parallel Corpus v1.0 7 (Ziemski et al., 2016) (_UN v1.0_). The Europarl corpus contains bilingual parallel corpus between English and other 20 Europe languages. The United Nations Parallel Corpus is composed of official records and other parliamentary documents of the United Nations that are in the public domain. These documents are mostly available in the six official languages of the United Nations. Besides, we use the _cldc-2009-004_8 Chinese-English (_CNEN_) news parallel corpus and _Japanese-English Bilingual Corpus of Wikipedia's Kyoto Articles Version 2.01_9 (_JP-EN_), which is created manually by translating Japanese Wikipedia articles (related to Kyoto) into English. In addition, _CYWikiCorpus_ (_CN-JP_) is a Chinese-Japanese Parallel Corpus Constructed from Wikipedia 10

Footnote 5: [http://tocai.ccf.org/cnference/2013/pages/page04_evares.html](http://tocai.ccf.org/cnference/2013/pages/page04_evares.html)

Footnote 6: [http://www.statmt.org/europarlv7/](http://www.statmt.org/europarlv7/)

Footnote 7: [https://conferences.unite.un.org/UNCorpus/](https://conferences.unite.un.org/UNCorpus/)

For the the BM model, we use _en-de_ (\(\in\)_Eu v7_) and _en-fr_ (\(\in\)_Eu v7_), _enh-zh_ (\(\in\)_CNEN_), and _en-jp_ (\(\in\)_JP-EN_).

For the PDBM model, we use _en-fr_ (\(\in\)_UN v1.0_) with _fr-de_ (\(\in\)_Eu v7_) to get the case _en-fr-de_ (_fr_ acts as a pivot), _en-zh_ (\(\in\)_CNEN_) with _zh-jp_ (\(\in\)_CNP_) to build the case _enh-zh-jp_ (\(zh\) acts as a pivot), _en-zh_ (\(\in\)_CNEN_) with _zh-fr_ (\(\in\)_UN v1.0_) to build _en-zh-fr_ (\(zh\) acts as a pivot), and _en-fr_ (\(\in\)_Eut v7_) with _zh-fr_ (\(\in\)_UN v1.0_) to build with _zh-fr_ (\(\in\)_UN v1.0_) to build _enh-zh-fr_ (\(\in\)_UU v1.0_) to build with _zh-fr_ (\(\in\)_UU v1.0_) to build _enh-zh-fr_ (\(\in\)_UU v1.0_) to build _enh-zh-fr_ (\(\in\)_EU v7_) with _zh-fr_ (\(\in\)_UN v1.0_) to build _enh-zh-fr_ (\(\in\)_UU v1.0_) to build _enh-zh-fr_ (\(\in\)_NU v1.0_) to build _enh-zh-fr_ (\(\in\)_TU v1.0_) to build _enh-zh-fr_ (\(\in\)_TU v1.

 

sion and SVM to learn different classifiers based on the translated Chinese training data. Bag of words features are used for classification.

**Bi-PV**: The same as that described above.

**BSWE**: It uses the bilingual sentiment word embedding algorithm based on denoising autoencoders [23] to learns word representations. Each document is then represented by the sentiment words and the corresponding negation words.

### Settings and Preprocessing

We utilize _cdec_[40] as an alignment tool to get word-level alignment, and we also use it to lowercase the characters in western languages. We use the _stanford-segmenter_11 to segment Chinese words, and use _Mecab_12 to segment Japanese words. The _SnowNLP_13 is used to convert traditional words to simplified ones. Besides, we remove all the irregular characters (e.g., \raisebox{-0.9pt}{$\circ$}\), \(\mathcal{L}\), \(\bigtriangledown\)) in the texts.

Footnote 11: [http://nlp.stanford.edu/software/segmenter.shtml](http://nlp.stanford.edu/software/segmenter.shtml)

Footnote 12: [http://faku910.github.io/mecab/](http://faku910.github.io/mecab/)

Footnote 13: [https://github.com/isnowfly/snownlp](https://github.com/isnowfly/snownlp)

For all the three models, we use stochastic gradient descent (SGD) for learning, with a default learning rate of 0.025, negative sampling with 30 samples, skip-gram with context window of size 5, and a subsampling rate of value 1e-4. The embedding size is set to 400. The training epochs are all set to 10. All the parameters of \(\alpha\) and \(\beta\) used in the three models are simply set to 1. The word embeddings in a document are averaged to get the document representation, and then the logistic regression classier is adopted for sentiment classification.

### Results

The sentiment classification results of our three models and the CL classification methods in the three domains and in the German, French and Japanese languages are presented in Table 2. The results in the Chinese language are presented in Table 3. Note that the results of the CL methods are not reported on English test sets, and we only compare our three models on English test sets in Table 3.

First and most importantly, we compare our three models. The BM model relies on the direct parallel data between the source and target languages, and it generally works slightly better than the other models, including the PMDB model and the UMM model. The reason is that direct parallel data can be used for transferring the sentiment information from the source language to the target language directly. However, the performance achieved by the PDBM model is very close to the BM model in most test cases. In some cases (DE-DVD, JP-book and EN-music), the PDBM model can even outperform the BM model. Note that the PDBM model does not leverage the direct parallel data between the source and target languages, but uses a pivot language as a bridge. The results demonstrate that the pivot-driven model is very effective for learning bilingual / trilingual sentiment-aware word embeddings. The results also verify the feasibility of using pivot languages to address the problem of sentiment classification in resource-poor languages, which lays a good foundation for building a universal sentiment classifier in multiple languages. When comparing the UMM model with BM and PDBM, the results of UMM are very close to that of BM and PDBM in most cases, Note that the UMM model does not use the direct parallel corpora of _en-de

\begin{table}
\begin{tabular}{l|l|l} \hline \hline Model & Parallel corpora with size & Test case \\ \hline  & _en-de_ (\(\in\)_Eu v7_, 1.92M) & _en-de_ \\ BM & _en-fr_ (\(\in\)_Eu v7_, 2.0M) & _en-fr_ \\  & _en-zh_ (\(\in\)_CN-EN_, 1.0M) & _en-zh_ \\  & _en-jp_ (\(\in\)_JP-EN_, 0.5M) & _en-jp_ \\ \hline  & _en-fr_ (\(\in\)_UN v1.0_, 2.0M) + _fr-de_ (\(\in\)_Eu v7_, 1.5M) & _en-fr-de_ \\ PDBM & _en-zh_ (\(\in\)_CN-EN_, 1.0M) + _zh-jp_ (\(\in\)_CN-JP_, 0.12M) & _en-zh-jp_ \\  & _en-zh_ (\(\in\)_CN-EN_, 1.0M) + _zh-fr_ (\(\in\)_UN v1.0_, 2.0M) & _en-zh-fr_ \\  & _en-fr_ (\(\in\)_Eu v7_, 2.0M) + _zh-fr_ (\(\in\)_UN v1.0_, 2.0M) & _en-fr-zh_ \\ \hline UMM & _all the corpora used in PDBM_ & _en,de,fr;zh,jp_ \\ \hline \hline \end{tabular}
\end{table}
Table 1: Parallel corpora used in our models.

 

and _en-jp_, but relies on pivot-based methods for bridging language gaps. We also find that the d-afferent parallel corpora used by the UMM model are of different quality and genres, and if they are used at the same time, they may have some negative influence on each other and thus the learned word embeddings are not always better than the BM and PDBM models using only one or two parallel corpora. What's more, the available parallel data in different language pairs are of various sizes (0.12M \(\sim\) 2.0M). Considering all these issues, the results of UMM are promising because the learned single sentiment classifier can work generally well in multiple languages. We believe that if more high-quality and balanced parallel data are used, the performance of the universal sentiment classifier will be improved.

Second, we compare our models with typical CL classification methods. In Table 2, we can see our models can outperform MT-BOW, CL-SCL, and CR-RL in most test cases, and outperform BSE in the German language. Our models can achieve very close results with the other sophisticated CL methods, including Bi-PV. In Table 3, we can see our models can generally outperform MT-LR and MT-SVM, and achieve very competitive results with other strong CL methods, including Bi-PV and BSWE. Most CL classification methods rely on commercial machine translation systems (e.g. Google Translate) for translating the reviews (including the training reviews, the test reviews and additional unlabeled reviews) to get parallel data. Compared with the large amount of parallel data used by commercial machine translation systems, the parallel data used by our models are of a very small size. Though our models are simply based on word embeddings, and the parallel data used by our models are in a small scale, the performance achieved by our models are very competitive.

In Figure 1, we show the visualization of word embeddings learned by the UMM model for some example words. We can see that similar sentiment words in different languages appear nearby with each other. The figure demonstrate that the UMM model are successful in learning sentiment-aware word embeddings in multiple languages.

## 4 Related Work

The most closely related work is cross-lingual sentiment classification, which aims to leverage the labeled sentiment data from a language with rich sentiment resources (e.g., English) to perform sentiment classification in a target language lacking sentiment resources (e.g., Japanese). Some studies tried to transfer labeled data from the source

\begin{table}
\begin{tabular}{l|l||c|c|c|c|c|c|c|c} \hline \hline
* & 78.25 & 84.25 \\ FR & DVD & 81.86 & 81.68 & 80.27 & 78.83 & 78.80 &
- & 74.83 & 79.60 \\  & music & 81.51 & 80.03 & 79.41 & 75.78 & 77.92 &
- & 78.71 & 80.09 \\ \hline  & book & 70.93 & 71.59 & 71.23 & 70.22 & 73.09 & 70.75 & 71.11 & 71.75 \\ JP & DVD & 74.62 & 72.82 & 72.55 & 71.30 & 71.07 & 74.96 & 73.12 & 75.40 \\  & music & 76.48 & 76.26 & 75.38 & 72.02 & 75.11 & 77.06 & 74.38 & 75.45 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison results (accuracy) on DE (German), FR (French) and JP (Japanese).

\begin{table}
\begin{tabular}{l|l||c|c|c||c|c|c|c} \hline \hline

- \\ EN & DVD & 81.7 & 80.8 & 79.5 &
- \\  & music & 76.8 & 78.8 & 77.9 &
- \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison results (accuracy) on CN (Chinese) and EN (English).
 

language to the target language (Banea et al., 2008; Wan, 2009; Gao et al., 2015; Chen et al., 2015), and some other studies tried to build a unified feature/semantic space in both two languages(Prettenhofer and Stein, 2010; Xiao and Guo, 2013; Zhou et al., 2015, 2016, 2016, 2017). In the latter case, the sentiment classifier learned in the source language can be used for sentiment classification in both languages. Particularly, Wan (2009) used machine translation to translate the source language to the target language to bridge the gap and applied the co-training approach. Prettenhofer and Stein (2010) provided a CL-SCL model based on structural correspondence learning (SCL) for sentiment classification. Lu et al. (2011) explored to increase the labeled data in both the source and target languages by applying an extra unlabeled parallel data. Xiao and Guo (2013) expected to get cross-lingual discriminative word embeddings to perform the multiple document classification tasks. Their intuitive thought is based on a delicate log-losses function, which aims to increase the probability of the documents with their labels. Like Lu et al. (2011), Meng et al. (2012) also proposed their cross-lingual mixture model to leverage an unlabeled parallel dataset. They intended to learn the previously unseen sentimental words from the big parallel corpus. Some studies have attempted to address multi-lingual sentiment classification (Deriu et al., 2017), but different from our study, they directly leverage training data in multiple languages, by assuming the training data can be obtained directly or in a distant supervision way in each language, and they did not consider the resource or data transfer problem at all.

Word embeddings have shown its great practicable usability in plenty of natural language processing tasks, such as information retrieval (Diaz et al., 2016; Zuccon et al., 2015), machine translation (Shi et al., 2016; Zhang et al., 2014), sentiment analysis (Ren et al., 2016; Xu et al., 2015; Tang et al., 2014) and so on. Bilingual word embeddings have been induced for cross-lingual NLP tasks (Vulic and Moens, 2015; Guo et al., 2014; Zou et al., 2013; Tang et al., 2014; Luong et al., 2015; Zhou et al., 2015). In particular, Luong et al. (2015) proposed the BiSkip model to induce bilingual word embeddings, which is extended from the monolingual skip-gram model in _word2vec_ to a bilingual model. They added constraint mutually on both the source language and the target language, while the monolingual model only has constraint on a single language. Zhou et al. (2015) proposed an approach to learning bilingual sentiment word embeddings by using sentiment information of text as supervision, based on labeled corpora and their translations. Ferreira et al. (2016) used a single optimization problem by combining a co-regularizer for the bilingual embeddings with a task-specific loss. However, these methods for inducing bilingual word embeddings usually rely on directly parallel corpus.

## 5 Conclusion and Future Work

In this paper, we proposed an approach to build a universal sentiment classifier in multiple languages. Particularly we proposed a pivot-based model to transfer the sentiment information from the source language to any resource-poor language via pivot languages. Evaluation results show that the pivot-based model can learn bilingual sentiment-aware word embeddings as well as the bilingual model using direct parallel data. Moreover, the universal sentiment classifier built in the five languages can achieve promising result-s.

In future work, we will investigate using more advanced document embedding techniques (e.g., CNN, RNN) to directly model document-level sentiment information. We will also extend our model to other languages.

Figure 1: Visualization of word embeddings in UMM (Chinese, Japanese, English, French, German). The similar words are marked in the same color.

 



## Acknowledgments

This work was supported by NSFC (61331011), 863 Program of China (2015AA015403) and Key Laboratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We thank the anonymous reviewers for helpful comments. Xiaojun Wan is the corresponding author.

## References

* Agarwal et al. (2011) Apoorv Agarwal, Boyi Xie, Ilia Vovsha, Owen Rambow, and Rebecca Passonneau. 2011. Sentiment analysis of twitter data. In _Proceedings of the workshop on languages in social media_, pages 30-38. Association for Computational Linguistics.
* Banea et al. (2008) Carmen Banea, Rada Mihalcea, Janywe Wiebe, and Samer Hassan. 2008. Multilingual subjectivity analysis using machine translation. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_, pages 127-135. Association for Computational Linguistics.
* Blitzer et al. (2007) John Blitzer, Mark Dredze, Fernando Pereira, et al. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In _Association for Computational Linguistics_, volume 7, pages 440-447.
* Bollegala et al. (2016) Danushka Bollegala, Tingting Mu, and J Y Goulermas. 2016. Cross-domain sentiment classification using sentiment sensitive embeddings. _IEEE Transactions on Knowledge and Data Engineering_, 28(2):398-410.
* Chen et al. (2015) Qiang Chen, Wenjie Li, Yu Lei, Xule Liu, and Yanxiang He. 2015. Learning to adapt credible knowledge in cross-lingual sentiment analysis. In _Association for Computational Linguistics_, pages 419-429.
* Deriu et al. (2017) Jan Deriu, Aurelien Lucchi, Valeria De Luca, Aliaksei Severyn, Simon Muller, Mark Cieliebak, Thomas Hofmann, and Martin Jaggi. 2017. Leveraging large amounts of weakly supervised data for multilanguage sentiment classification. In _Proceedings of the 26th International Conference on World Wide Web_, pages 1045-1052. International World Wide Web Conferences Steering Committee.
* Diaz et al. (2016) Fernando Diaz, Bhaskar Mitra, and Nick Craswell. 2016. Query expansion with locally-trained word embeddings. _Association for Computational Linguistics_, pages 367-377.
* Dyer et al. (2010) Chris Dyer, Jonathan Weese, Hendra Setiawan, Adam Lopez, Ferhan Ture, Vladimir Eidelman, Jurri Ganitkevitch, Phil Blunsom, and Philip Resnik. 2010. cdec: a decoder, alignment, and learning framework for finite-state and context-free translation models. In _ACL 2010, Proceedings of the Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden, System Demonstrations_, pages 7-12.
* Ferreira et al. (2016) Daniel C. Ferreira, Andre F. T. Martins, and Mariana S. C. Almeida. 2016. Jointly learning to embed and predict with multiple languages. In _Meeting of the Association for Computational Linguistics_, pages 2019-2028.
* Gao et al. (2015) Dehong Gao, Furu Wei, Wenjie Li, Xiaohua Liu, and Ming Zhou. 2015. Cross-lingual sentiment lexicon learning with bilingual word graph label propagation. _Computational Linguistics_.
* Guo et al. (2014) Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Li. 2014. Learning sense-specific word embeddings by exploiting bilingual resources. In _International Conference on Computational LinguisticsCOLING_, pages 497-507.
* Koehn (2004) Philipp Koehn. 2004. A parallel corpus for statistical machine translation. _Proceedings of the Third Workshop on Statistical Machine Translation_, (1):3-4.
* Li et al. (2017) Nana Li, Shuangfei Zhai, Zhongfei Zhang, and Boying Liu. 2017. Structural correspondence learning for cross-lingual sentiment classification with one-to-many mappings. pages 3490-3496.
* Liu (2012) Bing Liu. 2012. Sentiment analysis and opinion mining. _Synthesis lectures on human language technologies_, 5(1):1-167.
* Lu et al. (2011) Bin Lu, Chenhao Tan, Claire Cardie, and Benjamin K Tsuou. 2011. Joint bilingual sentiment classification with unlabeled parallel corpora. In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1_, pages 320-330. Association for Computational Linguistics.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In _The Workshop on Vector Space Modeling for Natural Language Processing_, pages 151-159.
* Meng et al. (2012) Xinfan Meng, Furu Wei, Xiaohua Liu, Ming Zhou, Ge Xu, and Houfeng Wang. 2012. Cross-lingual mixture model for sentiment classification. In _Meeting of the Association for Computational Linguistics: Long Papers_, pages 572-581.
* Pang et al. (2002) Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In _Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10_, pages 79-86. Association for Computational Linguistics.
* Pham et al. (2015) Hieu Pham, Thang Luong, and Christopher Manning. 2015. Learning distributed representations for multilingual text sequences. In _The Workshop on Vector Space Modeling for Natural Language Processing_, pages 88-94

Peter Prettenhofer and Benno Stein. 2010. Cross-language text classification using structural correspondence learning. In _ACL 2010, Proceedings of the Meeting of the Association for Computational Linguistics, July 11-16, 2010, Uppsala, Sweden_, pages 1118-1127.
* Ren et al. (2016) Yafeng Ren, Ruimin Wang, and Donghong Ji. 2016. A topic-enhanced word embedding for twitter sentiment classification. _Information Sciences_, 369:188-198.
* Shi et al. (2016) Chen Shi, Shujie Liu, Shuo Ren, Shi Feng, Mu Li, Ming Zhou, Xu Sun, and Houfeng Wang. 2016. Knowledge-based semantic embedding for machine translation. pages 2245-2254.
* Tang et al. (2014) Duyu Tang, Furu Wei, Nan Yang, Ming Zhou, Ting Li, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. pages 1555-1565.
* Tang and Wan (2014) Xuewei Tang and Xiaojun Wan. 2014. Learning bilingual embedding model for cross-language sentiment classification. In _Proceedings of the 2014 IEEE/WIC/ACM International Joint Conferences on Web Intelligence (WI) and Intelligent Agent Technologies (IAT)-Volume 02_, pages 134-141. IEEE Computer Society.
* Vulic and Moens (2015) Ivan Vulic and Marie-Francine Moens. 2015. Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 363-372. ACM.
* Wan (2009) Xiaojun Wan. 2009. Co-training for cross-lingual sentiment classification. In _ACL 2009, Proceedings of the Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing of the Afnlp, 2-7 August 2009, Singapore_, pages 235-243.
* Wu and Wang (2007) Hua Wu and Haifeng Wang. 2007. Pivot language approach for phrase-based statistical machine translation. _Machine Translation_, 21(3):165-181.
* Xiao and Guo (2013) Min Xiao and Yuhong Guo. 2013. Semi-supervised representation learning for cross-lingual text classification. In _Conference on Empirical Methods in Natural Language Processing_, pages 1465-1475.
* Xu et al. (2015) Ruizen Xu, Tao Chen, Yunqing Xia, Qin Lu, Bin Liu, and Xuan Wang. 2015. Word embedding composition for data imbalances in sentiment and emotion classification. _Cognitive Computation_, 7(2):226-240.
* Zhang et al. (2014) Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, Chengqing Zong, et al. 2014. Biltingually-constrained phrase embeddings for machine translation. pages 111-121.
* Zhou et al. (2015) Huiwei Zhou, Long Chen, Fulin Shi, and Degen Huang. 2015. Learning bilingual sentiment word embeddings for cross-language sentiment classification. In _Association for Computational Linguistics_, pages 430-440.
* Zhou et al. (2016a) Xinjie Zhou, Xiaojun Wan, and Jiangguo Xiao. 2016a. Attention-based lstm network for cross-lingual sentiment classification.
* Zhou et al. (2016b) Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016b. Cross-lingual sentiment classification with bilingual document representation learning. In _Meeting of the Association for Computational Linguistics_, pages 1403-1412.
* Ziemski et al. (2016) Micha Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In _Lrec_.
* Zou et al. (2013) Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. pages 1393-1398.
* Zuccon et al. (2015) Guido Zuccon, Bevan Koopman, Peter Bruza, and Leif Azizopardi. 2015. Integrating and evaluating neural word embeddings in information retrieval. page 12