Venue: NAACL 2016

Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification

 Aditya Mogadala

Institute AIFB

Karlsruhe Institute of Technology

Karlsruhe, Germany

aditya.mogadala@kit.edu &Achim Rettinger

Institute AIFB

Karlsruhe Institute of Technology

Karlsruhe, Germany

rettinger@kit.edu

###### Abstract

In many languages, sparse availability of resources causes numerous challenges for textual analysis tasks. Text classification is one of such standard tasks that is hindered due to limited availability of label information in low-resource languages. Transferring knowledge (i.e. label information) from high-resource to low-resource languages might improve text classification as compared to the other approaches like machine translation. We introduce BRAVE (_Bilingual paRAgraph VEctors_), a model to learn bilingual distributed representations (i.e. embeddings) of words without word alignments either from sentence-aligned parallel or label-aligned non-parallel document corpora to support cross-language text classification. Empirical analysis shows that classification models trained with our bilingual embeddings outperforms other state-of-the-art systems on three different cross-language text classification tasks.

## 1 Introduction

The availability of language-specific annotated resources is crucial for the efficiency of natural language processing tasks. Still, many languages lack rich annotated resources that support various tasks such as part-of-speech tagging, dependency parsing and text classification. While the growth of multilingual information on the web has provided an opportunity to build these missing annotated resources, but still lots of manual effort is required to achieve high quality resources for every language separately.

Another possibility is to utilize the unlabeled data present in those languages or transfer knowledge from annotation-rich languages. For the first alternative, recent advancements made in learning monolingual distributed representations of words Mikolov et al. (2013); Pennington et al. (2014); Levy and Goldberg (2014) (i.e. monolingual word embeddings) capturing syntactic and semantic information in an unsupervised manner was useful in numerous NLP tasks Collobert et al. (2011). However, this may not be sufficient for several other tasks such as cross-language information retrieval Grefenstette (2012), cross-language word semantic similarity Vulic and Moens (2014), cross-language text classification CLTC, henceforth Klementiev et al. (2012); Xiao and Guo (2013); Prettenhofer and Stein (2010); Tang and Wan (2014) and machine translation Zhao et al. (2015) due to irregularities across languages. In these kind of scenarios, transfer of knowledge can be useful.

Several approaches Hermann and Blunsom (2014); Sarath Chandar et al. (2014); Gouws et al. (2015); Coulmance et al. (2015) tried to induce monolingual distributed representations into a language independent space (i.e. bilingual or multilingual word embeddings) by jointly training on pair of languages. Although the overall goal of these approaches is to capture linguistic regularities in words that share same semantic and syntactic space across languages, they differ in their implementation. One set of methods either perform offline alignment of trained monolingual embeddings or jointly-train both monolingual and cross-lingual objectives, while the other set uses only cross-lingual objective. Jointly-trained or offline alignment methods can be further divided based on the type of par 

allel corpus (e.g. word-aligned, sentence-aligned) they use for learning the cross-lingual objective. Table 1 summarizes different setups to learn bilingual or multilingual embeddings for the various tasks.

Methods in the Table 1 that use word-aligned parallel corpus as offline alignment Mikolov et al. [2013] assume single correspondence between the words across languages and ignore polysemy. While, the jointly-train methods Klementiev et al. [2012] that use word-alignment parallel corpus and consider polysemy perform computationally expensive operation of considering all possible interactions between the pairs of words in vocabulary of two different languages. Methods Hermann and Blunsom [2014]; Sarath Chandar et al. [2014] that overcame the complexity issues of word-aligned models by using sentence-aligned parallel corpora limits themselves to only cross-lingual objective, thus making these approaches unable to explore monolingual corpora. Jointly-trained models Gouws et al. [2015] overcame the issues of both word-aligned and purely cross-lingual objective models by using monolingual and sentence-aligned parallel corpora. Nonetheless, these approaches still have certain drawbacks such as usage of only bag-of-words from the parallel sentences ignoring order of words. Thus, they are missing to capture the non-compositional meaning of entire sentence. Also, learned bilingual embeddings were heavily biased towards the sampled sentence-aligned parallel corpora. It is also sometimes hard to acquire sentence-level parallel corpora for every language pair. To subdue this concern, few approaches Rajendran et al. [2015] used pivot languages like English or comparable document-aligned corpora Vulic and Moens [2015] to learn bilingual embeddings specific to only one task.

This major downside can be observed in other aforementioned methods also, which are inflexible to handle different types of parallel corpora and have a tight-binding between cross-lingual objectives and the parallel corpora. For example, a method using sentence-level parallel corpora cannot be altered to leverage document-level parallel corpora (if available) that might have better performance for some tasks. Also, none of the approaches do leverage widely available label/class-aligned non-parallel documents (e.g. sentiment labels, multi-class datasets) across languages which share special semantics such as sentiment or correlation between concepts as opposed to parallel texts.

In this paper, we introduce BRAVE a jointly-trained flexible model that learns bilingual embeddings based on the availability of the type of corpora (e.g. sentence-aligned parallel or label/class-aligned non-parallel document) by just altering the cross-lingual objective. BRAVE leverages paragraph vector embeddings Le and Mikolov [2014] of the monolingual corpora to effectively conceal semantics of the text sequences across languages and build a cross-lingual objective. Method closely related to our approach is by Pham et al. [2015] who uses shared context sentence vector across lan

\begin{table}
\begin{tabular}{l|p{86.7pt}|p{99.6pt}|p{99.6pt}} \hline  & \multicolumn{3}{c}{Cross-Language Setups} \\ \hline Objective & Method & Tasks & Parallel Corpus \\ \hline \multirow{5}{*}{Monolingual+ Cross-lingual} & Klementiev et al. [2012] & CLDC & Word-Aligned \\  & Zou et al. [2013] & MT,NER & Word-Aligned \\  & Mikolov et al. [2013b] & MT & Word-Aligned \\  & Faruqui and Dyer [2014] & Word Similarity & Word-Aligned \\  & Lu et al. [2015] & Word Similarity & Word-Aligned \\  & Gouws and Sogaard [2015] & POS,SuS & Word-Aligned \\ \cline{2-4}  & Gouws et al. [2015] & CLDC,MT & Sentence-Aligned \\  & Coulmance et al. [2015] & CLDC,MT & Sentence-Aligned \\ \hline Cross-lingual & Hermann and Blunsom [2014] & CLDC & Sentence-Aligned \\  & Sarath Chandar et al. [2014] & CLDC & Sentence-Aligned \\  & Luong et al. [2015] & Word Similarity, CLDC & Sentence-Aligned \\  & Pham et al. [2015] & CLDC & Sentence-Aligned \\ \hline \end{tabular}
\end{table}
Table 1: Summary of bilingual or multilingual embedding methods that support Cross-language Document Classification (CLDC), Machine Translation (MT), Named Entity Recognition (NER), Part-of-Speech Tagging (POS), Super Sense Tagging (SuS).

 

guages to learn multilingual text sequences.

The main contributions of this paper are:

* We jointly train monolingual part of parallel corpora with the improved cross-lingual alignment function that extends beyond bag-of-word models.
* Introduced a novel approach to leverage nonparallel data sets such as label or class aligned documents in different languages for learning bilingual cues.
* Experimental evaluation on three different CLTC tasks, namely cross-language document classification, multi-label classification and cross-language sentiment classification using learned bilingual word embeddings.

## 2 Related Work

Most of the related work can be associated to the approaches that aim to learn latent topics across languages or distributed representations of the words and larger pieces of text for supporting various cross-lingual tasks.

### Cross-Language Latent Topics

Various approaches have been proposed to identify latent topics in monolingual Blei et al. (2013); Rus et al. (2013) and multilingual Mimno et al. (2009); Fukumasu et al. (2012) scenarios for cross-language semantic word similarity and document comparison. Extraction of cross-language latent topics or concepts use context-insensitive Zhang et al. (2010) and context-sensitive methods Vulic and Moens (2014) to build word co-occurrence statistics for document representations.

### Distributed Representations

Continuous word representations Bengio et al. (2003); Mikolov et al. (2013); Pennington et al. (2014) was further extended to multilingual Hermann and Blunsom (2014); Kocisky et al. (2014); Coulmance et al. (2015), bilingual Gouws et al. (2015); Vulic and Moens (2015); Luong et al. (2015) and polylingual Al-Rfou et al. (2013) settings by projecting multiple or pair of languages into the shared semantic space. Also, word representations were extended to meet larger textual units like phrases, sentences and documents either monolingual Socher et al. (2012); Le and Mikolov (2014) or bilingual Pham et al. (2015). Some approaches fine tuned the embeddings for specific tasks such as cross-lingual sentiment analysis Zhou et al. (2015), cross-language POS tagging Gouws and Sogaard (2015), machine translation Cho et al. (2014) etc.

## 3 Brave Model

In this section, we present the BRAVE model along with its variations whose aim is to learn bilingual embeddings that can generalize across different languages.

### Bilingual Paragraph Vectors (BRAVE)

Most of the NLP tasks require fixed-length vectors. Tasks like CLTC also require fixed-length vectors to incorporate inherent semantics of sentences or documents. Distributed representation of sentences and documents i.e. paragraph vectors Le and Mikolov (2014) are designed to out-perform certain text classification tasks by overcoming constraints posed by the bag-of-words models.

Here, we leverage paragraph vectors distributed memory model (PV-DM) as the monolingual objective \({\cal M}(\cdot)\) and jointly optimize with bilingual regularization function \(\varphi(\cdot)\) for learning bilingual embeddings similar to the earlier approaches Gouws et al. (2015); Coulmance et al. (2015). Equation 1 shows the formulation of the overall objective function that is minimized.

\[{\cal L}=\min_{\theta^{1_{1}},\theta^{l_{2}}}\sum_{t\in\{l_{1},l_{2}\}}\sum_{C ^{l}}{\cal M}^{l}(w_{t},h;\theta^{l})+\frac{\lambda\varphi(\theta^{l_{1}}, \theta^{l_{2}})}{2} \tag{1}\]

Here, \(C^{l}\) represent the corpus of individual languages (i.e. \(l_{1}\) or \(l_{2}\) ). Given any sequence of words (\(w_{1}^{l},w_{2}^{l}...w_{T}^{l}\)) in \(C^{l}\), \(w_{t}\) is the predicted word in a context \(h\) constrained on paragraph \(p\) (i.e. sentence or document) and sequence of words.

Formally, the first term (i.e. \({\cal M}(\cdot)\)) in the Equation 1 maximizes the average log probability based on word vector matrix \(W^{l}\) and a unique paragraph vector matrix \(P^{l}\). Equation 2 represents the average log probability.

\[{\cal M}^{l}(w_{t},h;\theta^{l})=\frac{\Sigma_{t=k}^{T-k}y_{w_{t}}^{l}-log(\sum _{i}e^{y_{i}^{l}})}{T}\] (2) 

where each \(y_{i}^{l}\) is log-probability of predicted word \(i\) and is given by Equation 3.

\[y^{l}=b+Uh(w_{t-k}^{l}....w_{t+k}^{l};W^{l},P^{l}) \tag{3}\]

To optimize for efficiency, hierarchical softmax [146] is used in training with \(U\) and \(b\) as parameters. Binary Huffmann tree is utilized to represent hierarchial softmax [258]. Analogous to Pham et al., [109], we also derive \(h\) by concatenating paragraph vector from \(P^{l}\) with the average of word vectors in \(W^{l}\). This helps to fine tune both word and paragraph vectors independently.

Now, to capture the bilingual cues, the regularization function (\(\varphi(\cdot)\)) is learned in two different ways. In the first approach a sentence-aligned parallel corpora is used, while in the second approach a label-aligned document corpora.

### BRAVE with Sentence-Aligned Parallel corpora (BRAVE-S)

To compute the bilingual regularization function \(\varphi(\cdot)\), we slightly deviate from earlier approaches [7]. Instead of simply performing L\({}_{2}\)-loss between the mean of word vectors in each sentence pair (\(s_{j}^{l_{1}}\),\(s_{j}^{l_{2}}\)) of the sentence-aligned parallel corpus (PC) at each training step. We use the concept of elastic net regularization [146] and employ linear combination of L\({}_{2}\)-loss between _sentence paragraph vectors_\(SP_{j}^{l_{1}}\) and \(SP_{j}^{l_{2}}\in R^{d}\) precomputed from the monolingual term \(\mathcal{M}(\cdot)\) with L\({}_{2}\)-loss between the mean of word vectors observed in sentences. This induces a constraint on the usage of monolingual part of parallel training data to learn \(\mathcal{M}(\cdot)\). At the same time, it has an advantage of using combination of paragraph and word vectors which combines compositional and non-compositional meanings of sentences.

Also, it eliminates the need for word-alignment and makes an assumption that each word observed in the sentence of language \(l_{1}\) can potentially find its alignment in the sentence of language \(l_{2}\). Theoretically, low value of \(\varphi(\cdot)\) ensures that words across languages which are similar are embedded closer to each other. Equation 4 shows the regularization term.

\[\alpha||SP_{j}^{l_{1}}-SP_{j}^{l_{2}}||^{2}+(1-\alpha)||\frac{1}{m}\sum_{w_{ i}es_{j}^{l_{1}}}^{m}W_{i}^{l_{1}}-\frac{1}{n}\sum_{w_{k}es_{j}^{l_{2}}}^{n} W_{k}^{l_{2}}||^{2} \tag{4}\]

Where \(W_{i}^{l_{1}}\) and \(W_{k}^{l_{2}}\) represent word embeddings obtained for the words \(w_{i}\) and \(w_{k}\) in each sentence (\(s_{j}\)) of length \(m\) and \(n\) in languages \(l_{1}\) and \(l_{2}\) respectively.

### BRAVE with Non-Parallel Document Corpora (BRAVE-D)

Sometimes it is hard to acquire sentence-aligned parallel corpora for many languages. Availability of non-parallel corpora such as topic-aligned (e.g. Wikipedia) or label/class-aligned document corpora (e.g. sentiment analysis and multi-class classification data sets) in different languages can be leveraged to learn bilingual embeddings for performing CLTC. Earlier approaches like CL-LSI [146] and CL-KCCA [146] were used to learn bilingual document spaces for the tasks comparable to CLTC. Although these approaches provide decent results, they face serious scalability issues and are mostly limited to Wikipedia. Cross-lingual latent topic extraction models [147] showed promising results for the tasks like word-level or phrase-level translations, but have certain drawbacks for CLTC tasks.

Here, we propose a two step approach to build bilingual embeddings with label/class-aligned document corpora.

* In the first step, we perform manifold alignment using Procrustes analysis [146] between sets of documents belonging to same class/label in different languages. This will help to identify the closest alignment of a document in language \(l_{1}\) with a document in another language \(l_{2}\).
* In the second step, we use the pair of partially aligned documents belonging to same class or label in different languages to extract bilingual cues similar to the approach mentioned in SS 3.2. Only difference being paragraph vector is learned for the entire document 



**Step-1:**

Let \(S^{l_{1}}\) and \(S^{l_{2}}\) be the sets containing languages \(l_{1}\) and \(l_{2}\) training documents associated to label or a class. Below, we provide the three step procedure to attain partial alignment between the documents present in these sets.

* Learning low-dimensional embeddings of the sets \((S^{l_{1}},S^{l_{2}})\) is key for alignment. We use **document paragraph vectors**Le and Mikolov (2014) to learn low-dimensional embeddings of the documents in each language. Let \(X^{l_{1}}\) and \(X^{l_{2}}\) be the low-dimensional embeddings of \(S^{l_{1}}\) and \(S^{l_{2}}\) respectively.
* To find the optimal values of transformation, Procrustes superimposition is done by translating, rotating and scaling the objects (i.e. rows of \(X^{l_{2}}\) is transformed to make it similar to the rows of \(X^{l_{1}}\)). Transformation is achieved by
* Translation:Taking mean of all the members of set to make centroids (\(\sum_{i=1}^{|S^{l_{1}}|}\frac{X^{l_{1}}}{|S^{l_{1}}|},\sum_{i=1}^{|S^{l_{2}}|} \frac{X^{l_{2}}}{|S^{l_{2}}|}\)) lie at origin.
* Scaling and Rotation:The rotation and scaling that maximizes the alignment is given by orthogonal matrix (\(Q\)) and scaling factor (\(k\)). They are obtained by minimizing orthogonal Procrustes problem Schonemann (1966) and is provided by Equation 5. \[\arg\min_{k,Q}||X^{l_{1}}-X^{l_{2}}_{*}||_{F}\] (5) where \(X^{l_{2}}_{*}\) a matrix of transformed \(X^{l_{2}}\) values given by \(kX^{l_{2}}Q\) and \(||.||_{F}\) is the Frobenius norm constrained over \(Q^{T}Q=I\).
* If \(S^{l_{2}}_{*}\) represents the new document set obtained after identifying the close alignment among documents in \(S^{l_{1}}\) and \(S^{l_{2}}\) with cosine similarity between \(X^{l_{1}}\) and \(X^{l_{2}}_{*}\), then the partially aligned corpora \(\{S^{l_{1}},S^{l_{2}}_{*}\}\) contains one-to-one correspondence between the two languages documents that are used to learn bilingual cues in the second step.

From perturbation theory of spectral spaces Kostrykin et al. (2003) it can be understood that the difference between low-dimensional embedding subspaces (i.e. \(X^{l_{1}}\) and \(X^{l_{2}}_{*}\)) is always bounded, thus the new alignment obtained between document sets \(\{S^{l_{1}},S^{l_{2}}_{*}\}\) is insensitive to perturbations. Which also means that Procrustes analysis has provided best possible document alignments.

**Step-2:**

Now, document pairs (\(d^{l_{1}}_{j}\),\(d^{l_{2}}_{j}\)) of the partially-aligned corpus (PAC) is used to compute bilingual regularization function \(\varphi(\cdot)\). At each training step, L\({}_{2}\)-loss of precomputed _document paragraph vectors_\(DP^{l1}_{j}\) and \(DP^{l2}_{j}\in R^{d}\) obtained from the monolingual term \(\mathcal{M}(\cdot)\) is combined with the L\({}_{2}\)-loss between vector of words weighted by the probability of their occurrence in a particular label/class of entire **PAC**. Consideration of word probabilities will help to induce label/class specific information. Equation 6 provides the regularization term.

\[\begin{split}&\alpha||DP^{l_{1}}_{j}-DP^{l_{2}}_{j}||^{2}\\ &+(1-\alpha)||\sum_{w_{i}ed^{l_{1}}_{j}}^{m}\frac{p_{w_{i}}W^{l_{ 1}}_{i}}{\sum_{m}p_{w_{i}}}-\sum_{w_{k}ed^{l_{2}}_{j}}^{n}\frac{q_{w_{k}}W^{l_ {2}}_{k}}{\sum_{n}q_{w_{k}}}||^{2}\end{split} \tag{6}\]

Where \(w_{i}\),\(w_{k}\) are words and their embeddings \(W^{l_{1}}_{i}\),\(W^{l_{2}}_{k}\) observed in each document (\(d_{j}\)) of length \(m\) and \(m\) in languages \(l_{1}\) and \(l_{2}\) respectively. While, \(p_{w_{i}}\) and \(q_{w_{k}}\) represents probability of occurrence of words \(w_{i}\) and \(w_{k}\) in a specific label/class of entire **PAC**. Figure- 1 shows overall goal of both the approaches.

## 4 Experiments

In this section, we report results on three different CLTC tasks to comprehend whether our learned bilingual embeddings are semantically useful across languages. First, cross-language document classification (CLDC) task proposed by Klementiev et al. (2012) using the subset of Reuters RCV1/RCV2 corpora Lewis et al. (2004). Second, a multi-label CLDC task with more languages using TED corpus1 of Hermann et al. (2014) . Subsequently, a cross-language sentiment classification (CLSC) proposed by Prettenhofer et al. (2010) on a multi-domain sentiment dataset.





### Parallel and Non-Parallel Corpora

For sentence-aligned parallel corpora, Europarl-v7 2(EP) is used as both monolingual and parallel training data. While for label-aligned non-parallel document corpora, only training and testing collections of the cross-language multi-domain Amazon product reviews(CL-APR) [35] corpus with sentiment labels is used.

Footnote 2: [http://www.statmt.org/euoraparl/](http://www.statmt.org/euoraparl/)

### Implementation

Our implementation launches monolingual paragraph vector [10] threads for each language along with bilingual regularization thread. Word and paragraph embeddings matrices are initialized with normal distribution (\(\mu=0\) and \(\sigma^{2}=0.1\)) for each language and all threads access them asynchronously. Following Pham et al. (2015) suggested combination (P=5*W) of paragraph and word embeddings, we chose paragraph embeddings with dimensionality of 200 and 640 when word embeddings are of 40 and 128 dimensions respectively. Asynchronous stochastic gradient descent (ASGD) is used to update parameters (i.e. \(P^{l}\),\(W^{l}\),\(U\) and \(b\)) and train the model.

For each training pair in parallel or non-parallel corpora, initially monolingual threads sample context \(h\) with window size of 8 from a random paragraph (i.e. sentence or document) in each language. Then the bilingual regularization thread along with monolingual threads make update to parameters asynchronously. Learning rate is set to 0.001 which decrease with the increase of epochs, while \(\alpha\) is chosen to be 0.6 (can be fine tuned based on empirical analysis) to give more weight to paragraph vectors. All models are trained for 50 epochs.

### Document Representation

Documents are represented with tf-idf weighted sum of embedding vectors of the words that are present in them.

### Results

The experimental results for each of the CLTC tasks are presented separately.

#### 4.4.1 Cross-Ianguage Document Classification (CLDC) - Rcv1/rcv2

Goal of this task is to classify target language documents with the labeled examples from the source language. To achieve it, we used the subset of Reuters RCV1/RCV2 corpora as the training and evaluation sets and replicated the experimental setting of Klementiev et al. (2012). From the English, German, French and Spanish collection of the dataset, only those documents are selected which was labeled with a single topic (i.e. CCAT, ECAT, GCAT and MCAT). For the classification experiments, 1000 labeled documents from source language are selected to train a multi-class classifier using averaged perceptron [39; 12] and 5000 documents were used as the testing data.

English-German, English-French and English-Spanish portion of **EP** corpora (i.e. each with around 1.9M sentence-pairs) is used both as monolingual and parallel training data with **BRAVE-S** approach to build vocabulary of around 85k English, 144k German, 119k French and 118k Spanish. While training and testing collections belonging to all domains in English-German, English-French languages of **CL-APR** ((i.e. around 12,000 document-pairs)) was used both as monolingual and partially aligned data with **BRAVE-D** approach to build vocabulary of around 21k English, 22k German and

Figure 1: Bilingual word embeddings learned using sentence or document paragraph vectors (SP/DP) along with word vectors.



18k French. Further, documents in the training and testing data of RCV1/RCV2 corpora are represented as described in SS 4.3 with the vocabulary built. Table 2 shows the comparison of our approaches with the existing systems.

#### 4.4.2 Multi-label CLDC - TED Corpus

To understand the applicability of our approaches to wider range of languages3 and class labels, we perform experiments with the subset of TED corpus Hermann and Blunsom (2014). Aim of this task is same as SS 4.4.1, but experiments were conducted with larger variety of languages and class labels. TED Corpus contains English transcriptions and their sentence-aligned translations for 12 languages from the TED conference. Entire corpus is further classified into 15 topics (i.e. class labels) based on the most frequent keywords appearing in them.

Footnote 3: Our goal is not to evaluate shared multilingual semantic representation.

To conduct our experiments, we follow the _single_ mode setting of Hermann et al. (2014) (i.e. embeddings are learned only from a single language pair). Entire language pair (i.e. en\(\rightarrow\)L2) training data of the TED corpus is used both as monolingual and parallel training data to learn bilingual word embeddings with dimensionality of 128 using **BRAVE-S** approach. Bilingual word embeddings of 128 dimensions learned with **EP** and **CL-APR** are also used for comparison. Documents in the training and testing data of TED corpus are represented as described in SS 4.3 using each of these embeddings. A multi-class classifier using averaged perceptron is built using training documents in source language to be applied on target language testing data for predicting the class labels. Table 3 shows the cumulative F1-scores.

#### 4.4.3 Cross-language Sentiment Classification (CLSC)

The objective of the third CLTC task is to identify sentiment polarity (e.g. positive or negative) of the data in target language by exploiting the labeled data in source language. We chose subset of publicly available Amazon product reviews (CL-APR) Prettenhofer and Stein (2010) dataset mainly English(E), German(G) and French(F) languages belonging to three different product categories (books(B), dvds(D) and music(M)) to conduct our experiments. For each language-category pair, corpus consists of training, testing sets comprising 1000 positive and 1000 negative reviews each with an additional unlabeled reviews varying from 9,000 to 170,000.

We constructed 12 different CLSC tasks using different languages (i.e. E,G and F) for three categories (i.e. B,D and M). For example, AFM refers English music reviews as source language and French music reviews as target language. Bilingual word embeddings with dimensionality of 128 learned with

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c} Model & Dim & en → de \(\to\) en & en → fr & fr→ en & en → es & es • en \\ \hline Majority class & 40 & 46.8 & 46.8 & 22.5 & 25.0 & 15.3 & 22.2 \\ MT & 40 & 68.1 & 67.4 & 76.3 & 71.1 & 52.0 & 58.4 \\ \hline I-Matrix Klementiev et al. (2012) & 40 & 77.6 & 71.1 & 74.5 & 61.9 & 31.3 & 63.0 \\ RAE-cr Sarath Chandar et al. (2014) & 40 & **91.8** & 74.2 & **84.6** & 74.2 & 49.0 & 64.4 \\ CVM-Add Herrmann and Blunsom (2014) & 40 & 86.4 & 74.7 & - & - & - & - \\ DWA Kocisky et al. (2014) & 40 & 83.1 & 75.4 & - & - & - & - \\ BilBOWA Gouws et al. (2015) & 40 & 86.5 & 75 & - & - & - & - \\ UnsupAlign Luong et al. (2015) & 40 & 87.6 & 77.8 & - & - & - & - \\ Trans-gram Coulmance et al. (2015) & 40 & 87.8 & 78.7 & - & - & - & - \\ BRAVE-S(EP) & 40 & 88.1 & **78.9** & 79.2 & **77.8** & **56.9** & **67.6** \\ BRAVE-D\({}_{\text{(CL-APR)}}\) & 40 & 69.4 & 67.9 & 64.1 & 56.5 & - & - \\ \hline CVM-BI Hermann and Blunsom (2014) & 128 & 86.1 & 79.0 & - & - & - & - \\ UnsupAlign Luong et al. (2015) & 128 & 88.9 & 77.4 & - & - & - & - \\ BRAVE-S(EP) & 128 & 89.7 & **80.1** & 82.5 & **79.5** & **60.2** & **70.4** \\ BRAVE-D\({}_{\text{(CL-APR)}}\) & 128 & 70.4 & 70.6 & 66.2 & 57.6 & - & - \\ \end{tabular}
\end{table}
Table 2: CLDC Accuracy with 100 labeled examples on RCV1/RCV2 Corpus. en/de, en/fr and en/es results of Majority class, MT, I-Matrix and BLEU-cr are adopted from Sarath Chandar et al. (2014) 



**BRAVE-S** and **BRAVE-D** are used to represent each review as described in SS 4.3. To have fair comparison with earlier approaches, sentiment classification model is then trained with libsvm4 default parameter settings using source language training reviews5 to classify target language test reviews. Table 4 shows the accuracy and standard deviation results after we randomly chose subset of target language testing documents and repeated the experiment for 10 times for all CLSC tasks.

Footnote 4: [https://www.csie.ntu.edu.tw/](https://www.csie.ntu.edu.tw/) cjlin/libsvm/

Footnote 5: We do not use 100 labeled target language reviews in model training, as it was shown by earlier approaches that 100 labeled target language reviews does not have much impact.

## 5 Discussion

First CLTC task (i.e. CLDC) results presented in Table 2 shows that BRAVE-S was able to outperform most of the existing systems. Success of BRAVE-S can be attributed to its ability to incorporate both non-compositional and compositional meaning observed in entire sentence and the individual words respectively. Thus making it different from other models which use only bag-of-words Gouws et al. (2015) or bi-grams Hermann and Blunsom (2014).

Similarly, second CLTC task (i.e. multi-label

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c} \hline  & \multicolumn{6}{c}{Cross-Language Sentiment Classification (en\(\rightarrow\)L2 and Vice versa)} \\ \hline Task & CL-SCI & CL-SSMC & CL-SLF & CL-DCI\({}_{100}\) & BSE & BRAVE-S & BRAVE-D \\  & & & & & (EP) & (CL-APR) \\ \hline EFB & 79.86\(\pm\)0.22 & 83.05\(\pm\)0.26 & 82.61\(\pm\)0.25 & 82.30 & - & 72.24\(\pm\)0.31 & 82.57\(\pm\)0.33 \\ EFD & 78.80\(\pm\)0.25 & 82.70\(\pm\)0.20 & 82.70\(\pm\)0.45 & 82.40 & - & 74.95\(\pm\)0.25 & **82.90\(\pm\)0.35** \\ EFM & 75.95\(\pm\)0.31 & 80.46\(\pm\)0.20 & 80.19\(\pm\)0.40 & 81.05 & - & 72.80\(\pm\)0.20 & **80.70\(\pm\)0.45** \\ FEB & 77.26\(\pm\)0.22 & 80.05\(\pm\)0.26 & 80.48\(\pm\)0.33 & - & - & 75.45\(\pm\)0.38 & 80.28\(\pm\)0.21 \\ FED & 76.57\(\pm\)0.20 & 79.40\(\pm\)0.28 & 78.76\(\pm\)0.38 & - & - & 73.75\(\pm\)0.26 & **79.80\(\pm\)0.15** \\ FEM & 76.76\(\pm\)0.25 & 78.82\(\pm\)0.17 & 79.18\(\pm\)0.33 & - & - & 73.66\(\pm\)0.17 & 78.56\(\pm\)0.33 \\ \hline GEB & 77.77\(\pm\)0.28 & 81.88\(\pm\)0.42 & 79.91\(\pm\)0.47 & 81.40 & 80.27\(\pm\)0.50 & 75.95\(\pm\)0.16 & **81.75\(\pm\)0.45** \\ EgMD & 79.93\(\pm\)0.23 & 82.25\(\pm\)0.20 & 81.86\(\pm\)0.31 & 79.95 & 77.16\(\pm\)0.30 & 78.30\(\pm\)0.42 & 81.56\(\pm\)0.26 \\ EGM & 73.95\(\pm\)0.30 & 81.30\(\pm\)0.20 & 79.59\(\pm\)0.42 & 83.30 & 77.98\(\pm\)0.51 & 75.95\(\pm\)0.33 & 81.20\(\pm\)0.17 \\ GEB & 77.85\(\pm\)0.27 & 79.06\(\pm\)0.23 & 78.61\(\pm\)0.34 & - & - & 72.25\(\pm\)0.20 & **80.23\(\pm\)0.17** \\ QED & 77.83\(\pm\)0.33 & 80.89\(\pm\)0.16 & 80.27\(\pm\)0.35 & - & - & 73.28\(\pm\)0.23 & **80.78\(\pm\)0.20** \\ GEM & 77.37\(\pm\)0.34 & 79.85\(\pm\)0.17 & 79.80\(\pm\)0.26 & - & - & 74.41\(\pm\)0.22 & **79.77\(\pm\)0.36** \\ \hline \end{tabular}
\end{table}
Table 4: Average classification accuracies and standard deviations for 12 CLSC tasks. Results of other baselines are adopted from CL-SCI (Prettenhofer and Stein, 2010), CL-SSMC (Xiao and Guo, 2002), CL-SLF (Zhou et al., 2015a), CL-DCI\({}_{100}\)(Esuli and Fernandez, 2015) and BSE (Tang and Wan, 2014)

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c|c|c|c} \hline Method & de & es & fr & it & nl & pt & po & ro & ru & tr \\ \hline en \(\rightarrow\) L2 & & & & & & & & & \\ \hline MT-_Baseline_ & 0.465 & **0.518** & **0.526** & **0.514** & **0.505** & 0.470 & 0.445 & **0.493** & 0.432 & 0.409 \\ DOC/ADD & 0.424 & 0.383 & 0.476 & 0.485 & 0.264 & 0.354 & 0.402 & 0.418 & 0.448 & 0.452 \\ DOC/BI & 0.428 & 0.416 & 0.445 & 0.473 & 0.219 & 0.400 & 0.403 & 0.467 & 0.421 & 0.457 \\ \hline BRAVE-S\({}_{\text{(ETD)}}\) & **0.484** & 0.436 & 0.456 & 0.507 & 0.328 & **0.506** & **0.453** & 0.488 & **0.456** & **0.491** \\ BRAVE-S\({}_{\text{(EP)}}\) & 0.418 & 0.365 & 0.387 & 0.418 & 0.284 & 0.454 & 0.412 & 0.424 & - & - \\ BRAVE-D\({}_{\text{(CL-APR)}}\) & 0.385 & - & 0.212 & - & - & - & - & - & - \\ \hline L2 \(\rightarrow\) en & & & & & & & & & \\ \hline MT-_Baseline_ & 0.469 & 0.486 & 0.358 & **0.481** & **0.463** & 0.374 & **0.460** & **0.486** & 0.404 & 0.441 \\ DOC/ADD & 0.476 & 0.422 & 0.464 & 0.461 & 0.251 & 0.338 & 0.400 & 0.407 & 0.471 & 0.435 \\ DOC/BI & 0.442 & 0.365 & **0.479** & 0.460 & 0.235 & 0.380 & 0.393 & 0.426 & **0.467** & 0.477 \\ \hline BRAVE-S\({}_{\text{(ITED)}}\) & **0.492** & **0.495** & 0.465 & 0.475 & 0.384 & **0.388** & 0.442 & 0.464 & 0.457 & **0.484** \\ BRAVE-S\({}_{\text{(EP)}}\) & 0.458 & 0.404 & 0.437 & 0.443 & 0.338 & 0.312 & 0.374 & 0.418 & - & - \\ BRAVE-D\({}_{\text{(CL-APR)}}\) & 0.366 & - & 0.278 & - & - & - & - & - & - \\ \hline \end{tabular}
\end{table}
Table 3: Cumulative F1-scores on TED Corpus using training data in English language and evaluation on other languages (i.e. German (de), Spanish (es), French (fr), Italian (it), Dutch (nl), Portuguese (pt), Polish (po), Romanian (ro), Russian (ru) and Turkish (tr)) and vice versa. MTS_Baseline_, DOC/ADD, DOC/BI represents single language pair of Hermann et al., (2014) as document features. Underline shows the best results amongst embedding models.



CLDC) results presented in Table 3 shows that BRAVE-S learned with the training data of TED corpus outperformed _single mode_ DOC/* embedding models Hermann and Blunsom (2014), BRAVE-S learned with **EP** and BRAVE-D. The BRAVE-S(TED) was able to capture better linguistic regularities across languages that is more specific to the corpus, than the general purpose bilingual embeddings learned with **EP**. Though in some cases, all our embedding models could not outperform machine translation baseline. This can be due to the asymmetry between languages induced by the language specific words which could not find its equivalents in English.

Also, it can be apprehended from the Table 2 and Table 3 that BRAVE-D results are not as expected. Though being a general approach like BRAVE-S which can capture both non-compositional and compositional meaning from larger pieces of texts, minimal overlap of vocabulary learned with BRAVE-D using cross-language sentiment label-aligned corpora with other domains (i.e. Reuters and TED) produce unfavorable results. Thus, we understand that the choice of label/class-aligned corpora is crucial.

Final CLTC task (i.e. CLSC) results presented in Table 4 shows that BRAVE-D outperforms other baseline approaches in most of the cases. As BRAVE-D learns bilingual word embeddings using **CL-APR**, it was able to inherently encompass sentiment label information effectively like earlier approaches Tang and Wan (2014); Zhou et al. (2015b) than the general purpose embeddings learned using BRAVE-S with **EP** and similar approaches Meng et al. (2012). Thus making it more suitable for sentiment classification task. Also unlike CL-SSMC Xiao and Guo (2002) and CL-SLF Zhou et al. (2015a), BRAVE-D is not highly parameter dependent where the results of the former approaches show big variance based on the parameter settings. To visualize the difference in embeddings learned with BRAVE-S and BRAVE-D, we selected sentiment words and identified cross-language nearest neighbors in Table 5. It can be observed that BRAVE-D was able to identify better sentiment (either positive or negative) word neighbors than BRAVE-S.

## 6 Conclusion and Future Work

In this paper, we presented an approach that leverages paragraph vectors to learn bilingual word embeddings with sentence-aligned parallel and label-aligned non-parallel corpora. Empirical analysis exhibited that embeddings learned from both of these types of corpora have shown good impact on CLTC tasks. In future, we aim to extend the approach to learn multilingual semantic spaces with more labels/classes.

## Acknowledgments

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no. 611346.

\begin{table}
\begin{tabular}{l|p{85.4pt}|p{99.6pt}|p{99.6pt}} \hline \multicolumn{4}{c}{Top-3 Nearest Neighbors (Euclidean Distance)} \\ \hline English Words & Models & German & French \\ \hline great & \multirow{3}{*}{BRAVE-S} & wachstum super spielen m’efforcerai \\ \cline{3-4}  & & schärfe fusion morgen traffvollé & firstressante attachment \\ \hline bored & \multirow{3}{*}{BRAVE-S} & boykottiert eunuyê precédera \\ \cline{3-4}  & & leere ênce ausgehen & comprometent \\ \cline{2-4}  & \multirow{3}{*}{BRAVE-D} & ableben lichtblikt traurigen & réserve intensité constrènén \\ \cline{3-4}  & & täwrien génrénch. & constère \\ \hline \end{tabular}
\end{table}
Table 5: Nearest Neighbors for English Words in German and French.

 



## References

* [Al-Rfou et al.2013] R. Al-Rfou, B. Perozzi, and S. Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. In _CoNLL_.
* [Bengio et al.2003] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. 2003. A neural probabilistic language model. _The Journal of Machine Learning Research._, 3:1137-1155.
* Blei (2012) D. M. Blei. 2012. Probabilistic topic models. _Communications of the ACM._, 55(4):77-84.
* Cho et al. (2014) K. Cho, B. van Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, and Y. Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In _EMNLP_.
* Collins (2002) M. Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In _ACL-EMNLP_., pages 1-8.
* Collobert et al. (2011) R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. _The Journal of Machine Learning Research._, 12:2493-2537.
* Coulmance et al. (2015) J. Coulmance, J. M. Marty, G. Wenzek, and A. Benhalloum. 2015. Trans-gram, fast cross-lingual word-embeddings reyes- mannde= reginait- femmerfr. In _EMNLP_.
* Dumais et al. (1997) S. T. Dumais, Todd A. Letsche, Michael L. Littman, and Thomas K. Landauer. 1997. Automatic cross-language retrieval using latent semantic indexing. In _AAAI spring symposium on cross-language text and speech retrieval_.
* Esuli and Fernandez (2015) A. Esuli and A. M. Fernandez. 2015. Distributional correspondence indexing for cross-language text categorization. In _Advances in Information Retrieval._, pages 104-109.
* Faruqui and Dyer (2014) M. Faruqui and C. Dyer. 2014. Improving vector space word representations using multilingual correlation. In _ACL_.
* Freund and Schapire (1999) Y. Freund and R. E. Schapire. 1999. Large margin classification using the perceptron algorithm. _The Journal of Machine Learning Research._, 37(3):277-296.
* Fukumasu et al. (2012) K. Fukumasu, Koji Eguchi, and Eric P. Xing. 2012. Symmetric correspondence topic models for multilingual text analysis. In _NIPS_, pages 1295-1303.
* Gouws and Sogaard (2015) S. Gouws and A. Sogaard. 2015. Simple task-specific bilingual word embeddings. In _NAACL-HLT_., pages 1386-1390.
* Gouws et al. (2015) S. Gouws, Y. Bengio, and G. Corrado. 2015. Bilbowa: Fast bilingual distributed representations without word alignments. In _ICML_.
* Grefenstette (2012) G. Grefenstette. 2012. Cross-language information retrieval. _Springer Science and Business Media_, 2.
* Hermann and Blunsom (2014) K. M. Hermann and P. Blunsom. 2014. Multilingual models for compositional distributed semantics. In _ACL_.
* Klementiev et al. (2012) A. Klementiev, I. Titov, and B. Bhattarai. 2012. Inducing crosslingual distributed representations of words. In _COLING_.
* Kocisky et al. (2014) T. Kocisky, K. M. Hermann, and P. Blunsom. 2014. Learning bilingual word representations by marginalizing alignments. In _ACL_, pages 224-229.
* Kostrykin et al. (2003) V. Kostrykin, K. Makarov, and A. Motovilov. 2003. On a subspace perturbation problem. _American Mathematical Society._, pages 3469-3476.
* Le and Mikolov (2014) Q. Le and T. Mikolov. 2014. Distributed representations of sentences and documents. In _ICML_, pages 1188-1196.
* Levy and Goldberg (2014) O. Levy and Y. Goldberg. 2014. Dependency based word embeddings. In _ACL_., pages 302-308.
* Lewis et al. (2004) D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. 2004. Rcv1: A new benchmark collection for text categorization research. _The Journal of Machine Learning Research._, 5:361-397.
* Lu et al. (2015) A. Lu, W. Wang, M. Bansal, K. Gimpel, and K. Livescu. 2015. Deep multilingual correlation for improved word embeddings. In _NAACL-HLT_.
* Luong et al. (2015) T. Luong, H. Pham, and C. D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In _First Workshop on Vector Space Modeling for Natural Language Processing._, pages 151-159.
* Meng et al. (2012) X. Meng, F. Wei, X. Liu, M. Zhou, G. Xu, and H. Wang. 2012. Cross-lingual mixture model for sentiment classification. In _ACL_., pages 572-581.
* Mikolov et al. (2013a) T. Mikolov, K. Chen, G. Corrado, and J. Dean. 2013a. Efficient estimation of word representations in vector space. In _arXiv preprint arXiv:1301.3781_.
* Mikolov et al. (2013b) T. Mikolov, Q. V. Le, and I. Sutskever. 2013b. Exploiting similarities among languages for machine translation. In _arXiv preprint arXiv:1309.4168_.
* Mimno et al. (2009) D. Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In _EMNLP_, pages 880-889.
* Mnih and Hinton (2009) A. Mnih and G. E. Hinton. 2009. A scalable hierarchical distributed language model. In _NIPS._, pages 1081-1088.
* Pennington et al. (2014) J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In _EMNLP_.
* Pham et al. (2015) H. Pham, M. T. Luong, and C. D. Manning. 2015. Learning distributed representations for multilingual text sequences. In _NAACL-HLT_, pages 88-94.
* Prettenhofer and Stein (2010) P. Prettenhofer and B. Stein. 2010. Cross-language text classification using structural correspondence learning. In _ACL_., pages 1118-1127.
* Rajendran et al. (2015) J. Rajendran, M. M. Khapra, S. Chandar, and B. Ravindran. 2015. Bridge correlational neural networks for multilingual multimodal representation learning. In _arXiv preprint arXiv:1510.03519_

V. Rus, M. C. Lintean, R. Banjade, N. B. Niraula, and D. Stefanescu (2013)Semilar: The semantic similarity toolkit. In ACL(Conference System Demonstrations), pp. 163-168. Cited by: SS4.
* A. P. Sarath Chandar, S. Lauly, H. Larochelle, M. Khapra, B. Ravindran, V. C. Raykar, and A. Saha (2014)An autoencoder approach to learning bilingual word representations. In NIPS., pp. 1853-1861. Cited by: SS4.
* P. H. Schonemann (1966)A generalized solution of the orthogonal procrustes problem. The Journal of Psychometrika..31 (1), pp. 1-10. Cited by: SS4.
* R. Socher, B. Huval, C. D. Manning, and A. Y. Ng (2012)Semantic compositionality through recursive matrix-vector spaces. In EMNLP, pp. 1201-1211. Cited by: SS4.
* X. Tang and X. Wan (2014)Learning bilingual embedding model for cross-language sentiment classification. In Web Intelligence (WI) and Intelligent Agent Technologies (IAT), 2014 IEEE/WIC/ACM International Joint Conferences., Vol. 2, pp. 134-141. Cited by: SS4.
* A. Vinokourov, J. Shawe-Taylor, and N. Cristianini (2003)Inferring a semantic representation of text via cross-language correlation analysis. In NIPS., pp. 1497-1504. Cited by: SS4.
* I. Vulic and M. F. Moens (2014)Probabilistic models of cross-lingual semantic similarity in context based on latent cross-lingual concepts induced from comparable data. In EMNLP. Cited by: SS4.
* I. Vulic and M. F. Moens (2015)Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In ACL, Cited by: SS4.
* C. Wang and S. Mahadevan (2008)Manifold alignment using procrustes analysis. In ICML., pp. 1120-1127. Cited by: SS4.
* M. Xiao and Y. Guo (2002)Semi-supervised matrix completion for cross-lingual text classification. In AAAI, Cited by: SS4.
* M. Xiao and Y. Guo (2013)Semi-supervised representation learning for cross-lingual text classification. In EMNLP, pp. 1465-1475. Cited by: SS4.
* D. Zhang, Qiaozhu Mei, and ChengXiang Zhai (2010)Cross-lingual latent topic extraction. In ACL, pp. 1128-1137. Cited by: SS4.
* K. Zhao, H. Hassan, and M. Auli (2015)Learning translation models from monolingual continuous representations. In NAACL-HLT, Cited by: SS4.
* G. Zhou, T. He, J. Zhao, and W. Wu (2015a)A subspace learning framework for cross-lingual sentiment classification with partial parallel data. In IJCAI, Cited by: SS4.
* H. Zhou, L. Chen, F. Shi, and D. Huang (2015b)Learning bilingual sentiment word embeddings for cross-language sentiment classification. In ACL, pp. 430-440. Cited by: SS4.
* H. Zou and T. Hastie (2005)Regularization and variable selection via the elastic net. The Journal of the Royal Statistical Society: Series B (Statistical Methodology).67 (2), pp. 301-320. Cited by: SS4.
* W. Y. Zou, R. Socher, D. M. Cer, and C. D. Manning (2013)Bilingual word embeddings for phrase-based machine translation. In EMNLP., pp. 1393-1398.