Robust Cross-lingual Embeddings from Parallel Sentences 
Authors: Ali Sabet
Prakhar Gupta
Jean-Baptiste Cordonnier
Robert West
Martin Jaggi
Venue:  (2020)
Keywords: Cross-lingual embeddings
sent2vec
word2vec
bilingual
word translation
sentence retrieval
text
NLP
word vectors
sentence vectors

Abstract: Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pretrained word embeddings from different languages into a shared space through a linear transformation. However, these approaches assume word embedding spaces are isomorphic between different languages, which has been shown not to  hold in practice (Søgaard et al., 2018), and fundamentally limits their performance. This motivates investigating joint learning methods which can overcome this impediment, by simultaneously learning embeddings across languages via a cross-lingual term in the training objective. Given the abundance of parallel data available (Tiedemann, 2012), we propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word and sentence representations. Our approach significantly improves cross-lingual sentence retrieval performance over all other approaches, as well as convincingly outscores mapping methods while maintaining parity with jointly trained methods on word-translation. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task, requiring far fewer computational resources for training and inference. As an additional advantage, our bilingual method also improves the quality of monolingual word vectors despite training on much smaller datasets.  We make our code and models publicly available.


Under review as a conference paper at ICLR 2020
ROBUST CROSS-LINGUAL EMBEDDINGS FROM PARALLEL SENTENCES
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances in cross-lingual word embeddings have primarily relied on mapping-based methods, which project pre-trained word embeddings from dif- ferent languages into shared space through linear transformation. However, these approaches assume word embedding spaces are isomorphic between dif- ferent languages, which has been shown to hold in practice (Søgaard al.) (018)) and fundamentally limits their performance. This motivates investigating joint learning methods which can overcome this impediment, by simultancously learning embeddings across languages via a cross- cross-lingual term in the training ob- jective. Given the abundance of parallel data available (Tiedemann] 2012). we propose a bilingual extension of the CBOW method which leverages sentence- aligned corpora to obtain robust cross-lingus word and sentence representations. Our approach significantly improves cross-lingual sentence retrieval performance over all other approaches, as well as convincingly outscores mapping methods while maintaining parity with jointly trained methods word-translation. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task, requiring far fewer computational resources for training and inference. As an additional advantage, our bilingual method also improves the quality of monolingual word vectors despite training much smaller datasets. We make our code and models publicly available.
INTRODUCTION
Cross-lingual representations--See feature space-have become a key technique multilingual natural language processing. They of- fer strong promise towards the goal of joint understanding of concepts across languages, well as for enabling the transfer of knowledge and machine learning models between different languages. Therefore, cross-lingual embeddings can icon induction, cross- lingual information retrieval, machine translation and many applications of zero-shot transfer learning, which is particularly impactful from resource-rich low-resource lan- guages.
embeddings of words and phrases into single comparable
variety of downstream tasks such as bilingual lex-
Existing methods be broadly classified into groups (Ruder al. 2017): mapping meth- ods leverage existing monolingual embeddings which are treated as independent, and apply post- process step to map the embeddings of each language into a shared space, through linear transfor- mation (Mikolov et al. 2013b] Conneau et al. 2017 Joulin et al., 2018). On the other hand, joint methods learn representations concurrently for multiple languages, by combining monolingual and cross-lingual training tasks (Luong et al.) 2015 Coulmance et al. 2015 Gouws et al. 2015 Vulic Moens 2015:
Chandar et al. 2014 Hermann & Blunsom 2013).
While recent work on word embeddings has focused almost exclusively on mapping methods, which require little to no cross-lingual supervision, (Søgaard et al. 2018)) establish that their performance hindered by linguistic and domain divergences in general, and for distant language pairs in particular. Principally, their analysis shows that cross-lingus hubness, where a few words (hubs) in the source language are nearest cross-lingual neighbours of many words in the target language, and structural non-isometry between embeddings do impose a fundamental barrier to the performance of linear mapping methods.
(Ormazabal et al. 2019) propose using joint learning as a means of mitigating these issues. Given parallel data, such sentences, a joint model learns to predict either the word context in both

---

Under review as a conference paper at ICLR 2020
source and target languages. As we will demonstrate with results from our algorithm, joint methods yield compatible embeddings which are closer to isomorphic, less sensitive to hubness, and perform better cross- loogull benchmarks.
Contributions. We propose the BI-SENT2VEC algorithm, which extends the SENT2VEC algorithm (Pagliardini et al. 2018, Gupta al. 2019) to the cross-lingual setting. We also revisit TRANS- GRAM Coulmance et al. (2015), another joint learning method, to assess the effectiveness of joint learning over mapping-based methods. Our contributions are
On cross-lingual sentence-retrieval and monolingual word representation quality evalua- tions, BI-SENT2VEC significantly outperforms competing methods, both jointly trained as well mapping- basedd ones while preserving state-of-the-ar performance cross- lons-llngual word retrieval tasks. For dis-similar language pairs, BI-SENT2VEC outperform their com- petitors by an even larger margin on all the tasks hinting towards the robustness of our method.
BI-SENT2VEC performs on par with a multilingual RNN based sentence encoder, LASER (Artetxe & Schwenk 2018), on MLDoc (Schwenk & Li 2018)) zero-shot cross- lingual transfer task on documents in multiple languages. Compared to LASER, our method improves computational efficiency by an order of magnitude for both training and infer- ence, making it suitable for resource or applications.
on-device cross-lingual NLP
We verify that joint learning methods consistently dominate state-of-the-ee mapping meth- ods on standard benchmarks, i.c., cross-lingual word and sentence retrieval.
Training on parallel data additionally enriches monolingual representation quality, evident by the superior performance of BI-SENT2VEC over FASTTEXT embeddings trained on 100x larger corpus.
We make our models and code publicly available.
RELATED WORK
The literature on cross-lingual representation learning is extensive.
Most recent advances in the
field pursue unsupervised (Artetxe et al.) 2017 Conneau et al. 2017 Chen & Cardie 2018 8 Hoshen & Wolf, 2018: Grave al. (018b) or supervised (Joulin et al. [2018] Conneau et al. |017 map-
ping or alignment-based algorithms. All these methods use existing monolingual word embeddings, followed by cross-lingual alignment procedure as a post-processing step- that to learn a sim- ple (typically linear) mapping from the source language embedding space to the target language embedding space.
Supervised learning of a linear map from source embedding space to another target embedding space Miikolov et al. 2013b) based on a bilingual dictionary was one of the first approaches towards cross-lingual word embeddings. Additionally enforcing orthogonality constraints results in rotations, and can be formulated as an orthogonal Procrustes problem (Smith et al.) (017)) However, the authors found the translated embeddings to suffer from hubness, which they mitigate by introducing the inverted softmax as a corrective search metric at inference time. (Artetxe et al.) 2017) align embedding spaces starting from parallel seed lexicon such as digits and iteratively build larger bilingual dictionary during training.
the linear map
In their seminal work, (Conneau al. (017) propose adversarial training method to learn linear orthogonal map, avoiding bilingual supervision altogether. They further refine the learnt mapping by applying the Procrustes procedure iteratively with synthetic dictionary generated through adver- sarial training. They also introduce the "Cross-Domaie Similarity Local Scaling' (CSLS) retrieval criterion for translating between spaces, which further improves on the word translation accuracy over nearest-neighhour and inverted softmax metrics. They refer to their work as Multilingual Un- supervised and Supervised Embeddings (MUSE). this paper, we will use MUSE to denote the unsupervised embeddings introduced by them, and "Procrustes + refine" to denote the supervised embeddings obtained by them. (Chen & Cardie 2018) similarly use "multilingual adversarial train- obtain unsupervised multilingual word embed-
ing" followed by sreudd--upervisee refinement" dings (UMWE), as opposed to bilingual word embeddings by (Conneau et al., 2017). Hoshen & Wolf 2018) describe an unsupervised approach where they align the second moment of the two
2

---

Under review as a conference paper at ICLR 2020
word embedding distributions followed by further refinement. Building the success of CSLS in reducing retrieval sensitivity to hubness, (Joulin et al. 2018)) directly optimize a convex relaxation of the CSLS function (RCSLS) to align existing mono-lingua embeddings using bilingual dictionary.
While none of the methods described above require parallel corpora, all assume structural isomor- phism between existing embeddings for each language (Mikolov al.. 2013b), i.e. there exists simple (typically linear) mapping function which aligns all existing embeddings. However, this is not always realistic assumption (Søgaard al., 2018) in small toy-examples it clear that many geometric configurations of points can not be linearly mapped to their targets.
Joint learning algorithms such TRANSGRAM (Coulmance et al. 2015) and Cr5 (Josifoski al., 2019), circumvent this restriction by simultancously learning embeddings as well as their alignment. TRANSGRAM, for example, extends the Skipgram (Mikolov et al., 2013a) method to jointly train bilingual embeddings in the same space, on a corpus composed of parallel sentences. In addition to the monolingual Skipgram loss for both languages, they introduce similar cross-lingual loss where word from sentence in one language is trained to predict the word-contents of the sentence in the other. Cr5, on the other hand, uses document-aligned corpora to achieve state-of-the-art results for cross-lingual document retrieval while staying competitive at cross-lingual sentence and word retrieval. TRANSGRAM embeddings have been absent from discussion in most of the recent work. However, the growing abundance of sentence-aligned parallel data (Tiedemann) 2012) merits a reappraisal of their performance.
OOmazabal et al. 2019) use BIVEC (Luong et al.) 2013), another bilingual extension of Skipgram, which uses a bilingual dictionary in addition to parallel sentences to obtain word-alignments and compare it with the unsupervised version of VECMAP (Artetxe et al. 2018b), another mapping- based method. Our experiments show this extra level of supervision in the case of BIVEC is redun- dant in obtaining state-of-th saate-of-the-ar of-tth-arrt performance.
3 MODEL
Proposed Model. Our BI-SENT2VEC model is cross-lingual extension of SENT2VEC proposed by (Pagliardini et al. 2018). which in turn is an extension of the C-BOW embedding method (Mikolov al.) al.] 2013a). SENT2VEC is trained on sentence contexts, with the word and higher-order word n-gram embeddings specifically optimized toward obtaining robust sentence embeddings using additive composition. Formally, SENT2VEC obtains representation v. of sentence S by averaging the word-ngram embeddings (including unigrams) as v. == R(S) Σ WER(S) Te where R(S) is the set of word n-grams in the sentence S.
The SENT2VEC training objective aims to predict a masked word token Wt in the sentence using the rest of the sentence representation J- To formulate the training objective, we use logistic loss l z + log (1 + e E²) in conjunction with negative sampling. More precisely, for raw text corpus C, the monolingual training objective for SENT2VEC is given by
min U.V
ΣΣ SEC WE
(( US) + Σ ((-uu W'NN
(I)
where ue is the target word and, V and U are the source n-gram and target word embedding matrices respectively. Here, the set of negative words Nw,, is sampled from a multinomial distribution where the probability of picking word is directly proportional to the square root of its frequency in the corpus. Each target word W; is sampled with probability min(1, Vt/fu, tt/f fw. where fw, is the frequency of the word in the corpus.
We adapt the SENT2VEC model to bilingual corpora by introducing cross- cross-lingual loss in addition the monolingual loss in equation ( Given sentence pair S = (St, St,) where St, and St, are translations of each other in languages I and 12, the cross-lingual loss for a target word w, in 1 is given by
8 [u US
+ E((-u.y u.
(2)
W'EN..,
Thus, we use the sentence St, to predict the constituent words of Si, and vice-versa in similar fashion to the monolingual SENT2VEC, shown in Figure [ embeddings of both languages lie in the same space.
This ensures that the word and n n-gram
3

---

Under review as a conference paper at ICLR 2020
Bilingual Dataset
Monotingual Prediction
Crosstingual Prediction
antiing
antivo
The
cer
Antiving
[SPLLIT]
politure
artiaw
Prodiet
Figure I: An illustration of the BI-SENT2VEC training process. A word from a sentence pair is chosen target and the algorithm learns to predict using the rest of the sentence(monoliggual training component) and the translation of the sentence(cross-linge lingual component).
Assuming C' to be a sentence aligned bilingual corpus and combining equations and 2 our BI-SENT2VEC model objective function is formulated as
min U.V
Σ
Σ E.
[(u (S/((w))) + Σ
uwy US Mort) + el elu usp) + {(-
Σ EeENN
E(- ue: USp))
us;
L''E( (s) 1/f1"
monolingual
cross-lngual loss
Implementation Details. We build our C++ implementation the top of the FASTTEXT li- brary (Bojanowski et al. 2016 Joulin et al. (016)) Model parameters are updated by asynchronous SGD with a linearly decaying learning rate.
Our model is trained on the ParaCrawl (Esplà-Gomis 2019) Anglish-German, English-French, English-Spanish, Monglish-Hungarte and English-Finnist lan- guage pairs. For the English- Russian language pair, we concatenate the OpenSubtitle corpus Lison & Tiedemann 2016) and the Tanzil projec (Quran translations) corpus. The number paral- lel sentence pairs in the corpora except for those of English-Finnish and English-Hungarria used by us range from 17-32 Million. Number of parallel sentence pairs for the dis-similar language pairs(English-HH)) Hungarran and English-Finnishh is upproximately 2 million. Evaluation results for these two language pairs can be found in Subsection 4.4 Exact statistics regarding the different corpora be found in the Table 7 in the Appendix. All the sentences tokenized using Spacy tokenizer for their respective languages.
v4.0 datasets for the English-Itae Italian,
For each dataset, trained two different models: with unigram embeddings only, and the other additionally augmented with bigrams. The earlier TRANSGRAM models (Coulmance et al. 2015) were trained on small amount of data (Europarl Corpus (Koehn) 2005 To facilitate fair com- parison, we train new TRANSGRAM embeddings on the same data used for BI-SENT VVCC. Given that TRANSGRAM and BI-SENT2VEC are cross-lingual extension of Skipgram and SENT2VEC re- spectively, we use the same parameters as (Bojanowski et al., (2016) and (Gupta et al. 2019), except increasing the number of epochs for TRANSGRAM 8, and decreasing the same for BI-SENT2VEC to 5. Additionally, a preliminary hyperparameter search (except changing the number of epochs) on BI-SENT2VEC and TRANSGRAM did not improve the results. All parameters for training the TRANSGRAM and BI-SENT2VEC models can be found in the Table6in the Appendix.
In order to make the comparison more extensive, we also train VECMAP (mapping- (Artetxe et al. 2018b a) and BIVEC (iiiitt-rriiing((LLuoo et al. 2015) methods on the same corpora using the exact pipeline as (Ormazabal al. 2019).
EVALUATION
To assess the quality of the word and sentence embeddings obtained as well their cross-lingual alignment quality, we compare our results using the following four benchmarks
Cross-lingual word retrieval Monolingual word representation quality
httt::/www..oeessbtittteess https://spacy.io.
http://tanzil
4

---

Under review as a conference paper at ICLR 2020
Cross-lingual sentence retrieval
Zero-shot cross-lingual transfer of document classifiers
where benchmarks are presented in order of increasing linguistic granularity, i.e. word, sentence, and document level. We also analyze the effect training data by studying the relationship between representation quality and corpus size.
We use the code available in the MUSE library (Conneau et al. 2017) for all evaluations except the zero-shot classifier transfer, which tested the MLDoc task Schwenk & Li. 2018
4.1 WORD TRANSLATION
The task involves retrieving correct translation(s) of a word source language from a target lan- guage. To evaluate translation accuracy, we use the bilingual dictionaries constructed by (Conneau et al. 2017) We consider 1500 source-test queries and 200k target words for each language pair and report P@ scores for the supervised and unsupervised baselines as well as our models in Table
Method
en-es
en-fr
en-de +-
en-ru
en-it
MUSE (Coneaau et al. 2017
81.7 83.3
82.3
82.1
74.0
72.2
44.0 59.1
78.6
77.9
73.5
UMWE (Chen & C Cardie 2013
82.5 83.1
82.5 82.1
74.6 72.5
61.7 49.5
78.3 77.0
74.4
Procrustes refine (Conneau al. 2017
82.4 83.9
82.3 83.2
75.3
73.2
50.1
63.5
77.5
77.6
74.9
RCSLS (Joulin af. 2018
83.7 87.1
84.1 84.7
79.2
77.5
70.2 60.9
81.1 82.7
79.1
TRANSGRAM (Coulmance al. 2015
91.6 88.6
89.1 90.1
87.2 87.5
65.6 73.7
88.6 89.5
85.2
VECMAP (unsupervised) (Artetxe et al. 201
87.4 87.8
88.3
88.5
84.3
87.2
48.6 50.5
87.4 86.5
79.6
VECMAP (supervised) (Artetx al. 20120 Sa
87.2 90.2
87.6
90.4
86.8 87.3
49.7 65.6
87.2
89.2
82.1
BIVEC NN (Luong et al. 2015
87.4 88.6
86.8 89.1
87.5
64.0
59.1
86.8 84.0
81.7
BIVEC CSLS ( Luong
2015
87.6 89.1
88.8 90.3
86.4 87.2
66.1 70.6
87.6 87.8
84.3
BI-SENT2VEC uni. NN
86.9 91.6
86.9 91.0
86.0 88.7
58.0 72.8
88.3 92.4
84.3
BI-SENTZVEC uni. + bi. NN
89.4 92.9
89.3 92.8
86.7 89.3
59.0
70.2
89.5 91.8
85.1
BI-SENTZVEC uni. CSLS
86.0 91.7
86.4 91.4
84.6 88.8
60.5
73.0
88.2 91.8
84.2
BI-SENT2VEC uni. + bi. CSLS
89.0 92.1
88.9 92.4
86.5 89.0
61.0 73.5
89.6 91.4
85.3
Table 1: Word translation retrieval P@1 1 for various language pairs of MUSE evaluation die- tionary (Conneau al. 2017). NN: nearest neighbours. CSLS:: Cross-Domain Similarity Local Scaling. ("en' is English, "fr' is French, 'de' is German, 'ru' is Russian, 'it' is Italian) ('uni.' and 'bi.' denote unigrams and bigrams respectively) (- denotes translation from the first language to the second and < the other way around.)
4.2 MONOLINGUAL WORD REPRESENTATION QUALITY
We assess the monolingual quality improvement of our proposed cross-lingual training by evaluating performance on monolingual word similarity tasks. To disentangle the specific contribution of the cross-lingual loss, we train the monolingual counterpart of BI-SENT2VEC, SENT2VEC on the same corpora as our method.
Performance monolingual word-similarity tasks is evaluated using the English SimL 99 (Hill ct al, (014)) and its Italian and German translations, English WS-353 (Finkelstein al., 2001) and its translation of the RG-65 (Joubarne & Inkpen| 201T) dataset. Pearson scores are used to measure the correlation between -anaan--d word similarities and predicted cosine similarities. We also include FASTTEXT monolingual vectors trained on CommonCraw/ data (Grave et al., 2018a) which is comprised of 600 billion, 68 billion, 66 billion, 72 billion and 36 billion words of English, French, German, Spanish and Italian respectively and is at least 100x larger than the corpora on which we trained BI-SENT2VEC, We report Pearson
Genman, Italian and Spanish translations. For French,
correlation scores on different word-similarity datasets for En-It pair in Table 2 Evaluation results
on other language pairs are similar and can be found in the appendix in Tables
9 and 10
5

---

Under review as a conference paper at ICLR 2020
SimLex- 9999
WS-353
Method Dataset
en
it
en
it
MUSE
0.38
0.30
0.74
0.64
RCSLS
0.38
0.30
0.74
0.64
FASTTEXT- Common Crawl
0.49
0.32
0.75
0.57
BIVEC
0.40
0.36
0.70
0.60
TRANSGRAM
0.43
0.37
0.73
0.63
SENT2VEC uni.
0.49
0.38
0.73
0.60
BI-SENT2VEC uni.
0.57
0.47
0.79
0.65
BI-SENT2VEC uni. + bi.
0.58
0.50
0.80
0.69
Table Monolingual word similarity task performance of our methods when trained on en-it ParaCrawl data. We report Pearson correlation scores.
4.3 CROSS-LINGUAL SENTENCE RETRIEVAL
The primary contribution of our work is to deliver improved cross-lingual sentence representations. We test sentence embeddings for each method obtained by bag-of- oo-words composition for sentence retrieval across different languages the Europarl corpus. In particular, the tf-idf weighted average used to construct sentence embeddings from word embeddings. We consider 2000 sentences the source language dataset and retrieve their translation among 200K sentences in the target language dataset. The other 300K sentences in the Europarl corpus are used to calculate tf-idf weights. Results for P@ of unsupervised and supervised benchmarks
models included in Table 3
en-fr
en-de
en-it
Method
avg.
MUSE
72.7
71.5
69.2
68.8
53.3
53.4
66.1
64.3
64.9
RCSLS
26.9
26.7
19.3
21.2
8.8
11.3
15.1
17.6
18.4
TRANSGRAM
83.5
81.4
80.4
81.6
64.8
69.9
77.2
77.9
77.1
VECMAP (unsupervised)
81.7
82.1
79.8
80.4
62.8
64.6
69.0
71.1
74.0
VECMAP (supervised)
81.3
81
80.4
80.7
62.6
64.3
67.8
71
73.6
BIVEC NN
69.8
77.1
54.7
75.5
56.1
44.1
58.2
45.1
60.1
BIVEC CSLS
81.6
83.4
78.1
81.6
71.6
68.1
74.2
72.4
76.4
BI-SENT2VEC uni. NN
87.8
86.4
85.2
83.4
82.3
80.2
85.9
85.8
84.6
BI-SENT2VEC uni. + bi. NN
87.9
87.8
86.1
83.9
79.5
79.7
85.1
85.3
84.4
BI-SENT2VEC uni. CSLS
89.5
88.5
87.1
86.4
84.4
83.0
88.2
87.5
86.8
BI-SENT2VEC uni. bi. CSL.S
89.7
89.6
87.8
87.4
84.2
84.0
87.9
87.6
87.3
Reduction in error
37.5%
37.3%
37.8%
31.5%
44.4%
46.8%
46.9%
43.9%
Table Cross-lingual Sentence retrieval. We report P@ scores 2000 source queries searching over 200 000 target sentences. Reduction in error is calculated with respect to BI-SENT2VEC uni. + bi. CSLS and the best non-Bn- BI-SENT2VEC method.
4.4 PERFORMANCE ON DIS-SIMILAR LANGUAGE PAIRS
We report a substantial improvement on the performance of previous models on cross-lingual word and sentence retrieval tasks for the dis-similar language Hairs(Engiish-inniin and English- Hungarian). We use the same evaluation scheme as Subsections and 4.3 Results for these pairs included in Table
4.5 ZERO-SHOT CROSS-LINGUAL TRANSFER OF DOCUMENT CLASSIFIERS
The MLDoc multilingual document classification task (Schwenk & Li 2018 consists of news documents given in 8 different languages, which need to be classified into different categories. To demonstrate the ability transfer trained classifiers in robust fashion between languages,

---

Under review as a conference paper at ICLR 2020
word retrieval
sentence retrieval
Method
en-fi
en-hu
en-fi
en-hu
MUSE
48.1
59.5
53.9
64.9
21.7
29.5
39.1
46.7
RCSLS
61.8
69.9
67.0
73.0
3.2
4.8
3.6
5.1
VECMAP (unsupervised)
62.5
66.8
61.6
68.7
13.2
14.7
20.5
19.3
VECMAP (supervised)
62.6 78.3
63.7
76.6
15.0
16.9
20.9
21.7
BIVEC NN
62.1
55.3
62.1
53.7
14.2
9.7
26.2
13.7
BIVEC CSLS
69.6 78.0
72.4
78.4
33.3
32.0
46.7
41.3
TRANSGRAM
69.7
81.1
73.1
80.8
35,4
40.5
52.1
55
BI-SENT2VEC uni. NN
71.2
85.4
75.6
83.9
63.5
64.2
75.2
76.2
BI-SENT2VEC uni. + bi. NN
68.5
81.7
71.4
79.4
57.5
55.9
65.8
65.2
B1-SENT2VEC uni. CSLS
72.0
86.5
76.3
85.1
70.2
69.0
81.4
80.8
B1-SENT2VEC uni. + bi. CSLS
70.1
84.4
73.7
81.7
66
64.1
73.8
74.5
Table 4: Cross-lingual Word and Sentence retrieval for dis-similar language pairs (P@1 scores). 'en' is English, 'fi' is Finnish, 'hu' is Hungarian
use zero-shot setting, i.e., we train classifier on embeddings in the source language, and report the accuracy of the same classifier applied to the target language. As the classifier, we use a simple beed-forward neural network with two hidden layers size 10 and 8 respectively, optimized using the Adam optimizer. Each document is represented using the sum of its sentence embeddings.
Method
en-es
en-fr
en-de
en-it
avg.
LASER
79.3 69.6
78.0 80.1
86.3
80.8
70.2
74.2
77.3
BI-SENT2VEC
74.0 71.5
81.6
82.2
86.5 79.2
75.0
72.6
77.8
Table 5: MLDoc Benchmark results (Schwenk Li, 2018). A document classifier was trained on one language and tested on another without additional training/fine-tuneg We report % accuracy.
We compare the performance of B1-S BI-SENT2VEC with the LASER sentence embeddings (Artetxe & Schwenk 2018) in Table LASER sentence embedding model is multi-lingual sentence embed- ding model which is composed of a biL.TM encoder and an LSTM decoder. It uses a shared byte pair encoding based vocabulary of 50k words. The LASER model which we compare to was trained on 223M sentences for 93 languages and requires 5 days to train on 16 V100 GPUs compared to our model which takes 1-2.5 hours for each language pair on 30 CPU threads.
4.6 EFFECT OF CORPUS SIZE ON REPRESENTATION QUALITY
We conduct an ablation study on how B1- BI-SENT2VEC embeddings' performance depends on the size of the training corpus. We uniformly sample smaller subsets of the En-Fr ParaCrawl dataset and train BI-SENT2VEC model on them. We test word/sentence translation performance with the CSLS retrieval criterion, and monolingual embedding quality for En-Fr with increasing ParaCrawl corpus size. The results are illustrated in Figures and
5 DISCUSSION
In the following section, we discuss the results monolingual and cross-lingual benchmarks, pre- sented in Tables and a data ablation study for how the model behaves with increasing parallel corpus size in Figure The most impressive outcome of our experiments is improved cross- lingual sentence retrieval performance, which we elaborate on along with word translation in the next subsection.

---

Under review as a conference paper at ICLR 2020
Corpus vs, word-translation quality
Corpus cross-lingual quality
100
Accuracy (mn %0
Accuracy (m
en-fr word translation fr-en word translation
en-fr sentence retrieval
fr-en sentence retrieval
10a Fraction original corpus used
Fraction the original corpus used
Figure Effect of corpus size on cross-lingual word/sentence retrieval performance.
Corpus size vs. monolingual embedding quality
0.7
Pearson Correlation
06-
dataset- EN SN-SimLex-999
dataset FR-RG-88 65
10"
Fraction of the original corpus used
Figure 3: Effect of corpus size on monolingual word quality. We use SimLex-999, WS-353, and FR-RG datasets for measuring monolingual word embedding quality.
Cross-lingual evaluations For cross-lingual tasks, we observe in Table [ that jointly trained embeddings produce much better results on cross-lingual word and sentence retrieval tasks. B1- SENT2vEC's performance
word-retrieval tasks is uniformly superior to mapping methods, achieving up to 11,5% more in P@ than RCSLS for the English to German language pair, consistent with the results from (Ormazabal et al. 2019). It is also on-par with, better than competing joint methods except on translation from Russian to English, where TRANSGRAM receives a significantly discernible difference between CSLS/NN criteria
better score. For word retrieval tasks, there for BI-SENT2VEC, suggesting the relative absence of the hubness phenomenon which significantly hinders the performance of cross-lingual word embedding methods.
Our principal contribution is in improving cross-lingual sentence retrieval. Table 3 shows Bi- SENT2VEC decisively outperforms all other methods by a wide margin, reducing the relative P@ еπтоr anywhere from 31.5% to 13.1% Our model displays considerably less variance than others in quality across language pairs, with at most a 5% deficit between best and worst, and nearly symmetric accuracy within a language pair.
TRANSGRAM also outperforms the mapping- based methods, but still falls significantly short of Bi- SENT2VEC's. These results can be attributed to the fact that BI-SENT2VEC directly optimizes for obtaining robust sentence embeddings using additive composition of its word embeddings. Since BI-SENT2VEC's learning objective is closest to sentence retrieval task amongst current state-of- the-art methods, it can surpass them without sacrificing performance on other tasks.
Cross-lingual evaluations on dis-similar language pairs Unlike other language pairs in the pairs are composed of languages from two dif-
evaluation, English-Finnnn Finnish and ferent language families(English being an Indo-European language and the other language being Finno-UPe Ugric language). Table we see that the performance boost achieved by BI-SENT2VEC competing methods methods compared to paris of languages close to each other. This observation affirms the suitaibility Bi- SENT2vEC for learning joint representations languages from different families.
pronounced in the case of dis-similar language pairs as
8

---

Under review as a conference paper at ICLR 2020
Monolingual word quality For the monolingual word similarity tasks, observe large gains over existing methods. SENT2VEC is trained on the same corpora as us, and FASTTEXT vectors are trained on the CommonCrawl corpora which are more than 100 times larger than ParaCrawl v4.0. In Table 2 we see that B1-SENT2VEC outperforms them by a significant margin on SimLee-999 and WS-353, two important monolingual word quality benchmarks. This observation is in accordance with the fact (Faruqui & Dyer 2014) that bilingual contexts can be surprisingly effective for learning monolingual word representations. However, amongst the joint-training methods, BI-SENT2 also outperforms TRANSGRAM and BIVEC trained on the same corpora by significant margin, again hinting at the superiority of the sentence level loss function over fixed context window loss.
Effect of n-grams Guupa et al. 2019) report improved results on monolingual word representa- tion evaluation tasks for SENT2VEC and FASTTEXT word vectors by training them alongside word n-grams. Our method incorporates their results based on the observation that unigram vectors trained alongside with bigrams significantly outperform unigrams alone on the majority of the evaluation tasks. We can see from Tables I 3 that this holds for the bilingual case well. However, in case of dis-similar language pairs(Table T. we observe that using n-grams degrades the cross-lingual performance of the embeddings. This observation suggests that use of higher order n n-grams may not be helpful for language pairs where the grammatical structures are contrasting.
Effect of corpus size Considering the cross-lingual performance curve exhibited by B1- SENT2VEC in Figure increasing corpus size for the English- Freschh datasets up to 1-3.1M lines appears to saturate the performance of the model on cross-lingual word/sentence retrieval, after which it either plateaus or degrades slightly. This is an encouraging result, indicating that joint methods can use significantly less data to obtain promising performance. This implies that joint methods may not necessarily be constrained to high-resource language pairs as previously assumed, though further experimentation needed verify this claim.
It should be noted from Figure 3 that the monolingual quality does keep improving with an increase in the size of the corpus. potential way to overcome this issue of plateauing cross- poss-linggul perfor- mance is to give different weights to the monolingual and cross-lingual component of the loss with the weights possibly being dependent on other factors such as training progress.
Comparison with cross-lingual sentence embedding model and performance on document level task On the ML.Dc classifier transfer task (Schwenk & Li| 2018) where we evaluate a clas- sifier learned on documents in one language on documents in another, Table [ shows we achieve parity with the performance of the LASER model for language pairs involving English, where B1- SENT2VEC's 's average accuracy of 77.8% is slightly higher than LASER s 77.3% While the compari- son is not completely justified as LASER is multilingual in nature and is trained on a different dataset, one must emphasize that BI-SENT2VEC is bag-of-words method compared to LASER which uses a multi-layere biL.TTM sentence encoder. Our method only requires to average a set of vectors encode sentences reducing its computational footprint significantly. This makes BI-SENT2VEG an ideal candidate for on-device computationally efficient cross-lingual NLP, unlike LASER which has a huge computational overhead and specialized hardware requirement for encoding sentences.
CONCLUSION AND FUTURE WORK
We introduce cross-lingual extension of an existing monolingual word and sentence embedding method. The proposed model is tested at three levels of linguistic granularity: words, sentences and documents. The model outperforms all other methods by wide margin on the cross-lingual sentence retrieval task while maintaining parity with the best-performing methods on word transla- tion tasks. Our method achieves parity with LASER on zero-shot document classification, despite being much simpler model. We also demonstrate that training on parallel data yiclds significant improvement in the monolingual word representation quality.
The success of our model on the bilingual level calls for its extension to the multilingual level espe- cially for pairs which have little or no parallel corpora. While the amount of bilingual/multtiingual parallel data has grown in abundance, the amount of monolingual data available is practically limit- less. Consequently, we would like to explore training cross- lingull embeddings with a large amount of raw text combined with smaller amount of parallel data.
9

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2018.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Learning bilingual word embeddings with (almost) no bilingual data. In ACL, 2017.
Mikel Artetxe, Gorka Labaka, and Eneko Agire. Generalizing and improving bilingual word em- bedding mappings with a multi-step framework of linear transformations. In Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence, pp. 5012-5019, 2018a.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. A robust self-learning method for fully unsuper-
vised cross-lingual mappings of word embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 789-798, 2018b.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Compulational Linguistics, 5:135-146, 2016.
A. Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh M. Khapra, Balaraman Ravindran, Vikas C. Raykar, and Amrita Saha. An autoencoder approach to learning bilingual word repre- sentations. In NIPS, 2014.
Xilun Chen and Claire Cardie. Unsupervised multilingual word embeddings. In EMNLP, 2018.
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou.
Word translation without parallel data. ArXiv, abs/1710.044087 2017.
Jocelyn Coulmance, Jean-Mare Marty, Guillaume Wenzek, and Amine Benhalloum. Trans-gram, Fast Cross-lingual Word-embeddings. In EMNLP Proceedings of the 2015 Conference on Em- pirical Methods in Natural Language Processing, pp. 1109-1113, 2015.
M. Esplå-Gomis, Gomis, ParaCrawl: Web-scale parallel corpora for the languages of the EU. 2019.
Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In EACL, 2014.
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: the concept revisited. In WwW, 2001.
Stephan Gouws, Yoshua Bengio, and Greg Corrado. Bilbowa: Fast bilingual distributed representa- tions without word alignments. 2015.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. ArXiv, abs/ 1es/1882.0689,, 2018a.
Edouard Grave, Armand Joulin, and Quentin Berthet. Unsupervised alignment of embeddings with wasserstein procrustes. In AISTATS, 2018b.
Prakhar Gupta, Mattco Pagliardini, and Martin Jaggi. Better word embeddings by disentangling contextual n-gram information. In NAACL-HLT, 2019.
Karl Moritz Hermann and Phil Blunsom. Multilingual distributed representations without word alignment. In ICLR 2014, 2013.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 41:665-695, 2014.
Yedid Hoshen and Lior Wolf. Non-adversarial unsupervised word translation. In EMNLP, 2018.
Martin Josifoski, Ivan S Paskov, Hristo S Paskov, Martin Jaggi, and Robert West. Crosslingual document embedding as reduced-rant ridge regression. In Proceedings of the Twelfth ACM In- fernational Conference on Web Search and Data Mining, pp. ACM, 2019.
10

---

Under review as a conference paper at ICLR 2020
Colette Joubarne and Diana Inkpen. Comparison of semantic similarity for different languages using the google n- n-gram corpus and second-order co-occurrence measures. In Canadian Conference on AI, 2011.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. In EACL, 2016.
Armand Joulin, Piotr Bojanowski. Tomas Mikolov, Hervé Jégou, and Edouard Grave. Loss in trans- lation: Learning bilingual word mapping with retrieval criterion. In EMNLP, 2018.
Philipp Kochn. Europarl: A parallel corpus for statistical machine translation. 2005.
Pierre Lison and Jörg Tiedemann. Opensubritles2016: Extracting large parallel corpora from movie and subtitles. In LREC, 2016.
Thang Luong, Hieu Pham, and Christopher D Manning. Bilingual word representations with mono- lingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 151-159, 2015.
Tomas Mikolov, Kai Chen, Gregory Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781, 2013a.
Tomas Mikolov, Quoe V. Le, and Ilya Sutskever. Exploiting similarities among languages for ma- chine translation. ArXiv, abs/ 309/ 4466,, 2013b.
Aitor Ormazabal, Mikel Artetxe, Gorka Labaka, Aitor Soroa, and Eneko Agirre. Analyzing the limitations of cross-lingual word embedding mappings. ACI, 2019.
Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. Unsupervised learning of sentence embed- dings using compositional n-gram features. In NAACL-HLT, 2018.
Sebastian Ruder, Ivan Vulié, and Anders Søgaard. A survey of cross-lingual word embedding mod- els. arXiv preprint arXiv: 04902, 2017.
Holger Schwenk and Xian Li. A corpus for multilingual document classification in eight languages. arXiv preprint arXiv: 000809 2018.
Samuel Smith, David H. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual 2017.
word vectors, orthogonal transformations and the inverted softmax. ArXiv,
Anders Søgaard, Sebastian Ruder, and Ivan Vulie. On the limitations of unsupervised bilingual dictionary induction. In ACI., 2018.
Jörg Tiedemann. Parallel data, tools and interfaces in opus. In Proceedings of the Eighth Interna- tional Conference on Language Resources and Evaluation (LREC-2012), pp. 2214-2218, 2012.
Ivan Vulic and Marie-Francine Moens. Bilingual word embeddings from non non-parallel document- aligned data applied to bilingual lexicon induction. In ACL, 2015.

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.I DATASET STATISTICS
Number of tokens
Dataset
Number of sentences
(English tokens if bilingual)
En-De ParaCrawl v4.0
17 Million
308 Million
En-Es ParaCrawl v4.0
22 Million
477 Million
En-Fi ParaCrawl v4.0
2.16 Million
42 Million
En-Fr ParaCrawl v4.0
32 Million
665 Million
En-Hu ParaCrawl v4.0
1.91 Million
31 Million
En-lt ParaCrawl v4.0
13 Million
261 Million
En-Ru OpenSubtitles + Tanzil
27 Million
363 Million
Wikipedia- En
70 Million
1792 Million
Wikipedia De
1384 Million
Wikipedia Fr
1108 Million
Wikipedia
Es
797 Million
Wikipedia It
702 Million
Wikipedia Ru
824 Million
Common Crawl En
600 Billion
Common Crawl De
66 Billion
Common Crawl Fr
68 Billion
Common Crawl
36 Billion
Common Crawl Es
72 Billion
Table 6: Dataset Sizes. "En',Dee,, 'Fe', 'r',, 'Hu', "Is' and 'Ru' stand for English, German, Finnish, French, Hungarian, Italian, Spanish and Russian respectively.
We used ParaCrawl v4.0 corpora for training B1-SENT2VEC, SENT2VEC, BEEDVEE.BBVVEE.. BICE VECMAP and TRANSGRAM embeddings except for En-Ru pair for which we used OpenSubtitles and Tanzil cor- pora combined. MUSE and RCSLS vectors were trained from FASTTEXT vectors obtained from Wikipedia dumps(Grave et al. 2018a).
12

---

Under review as a conference paper at ICLR 2020
A.2
TRAINING PARAMETERS FOR TRAINED MODELS
BI-SENT2VEC
BI-
SENT2VEC
Model
TRANSGRAM
uni.
uni. + bi.
uni.
Embedding dimension
300
300
300
300
Max vocabulary size
750k
750k
750k
750k
Minimum word count
8
5
Initial Learning Rate
0.2
0.2
0.2
0.025
Epochs
5
5
5
8
Subsampling hyper- parameter
1.1000
5
10 6
10
10
Word-Ngrams Bucket Size
2M
Word-Ngrams dropped per context
4
Window size
5
Number of negatives sampled
10
10
10
5
Table 7: Hyperparameters for the trained models
13

---

Under review as a conference paper at ICLR 2020
A.3
ADDITIONAL MONOLINGUAL QUALITY TABLES
SimLex-999
WS-353
Method Dataset
en
en
es
MUSE RCSLS
0.38
0.74
0.61
0.38
0.74
0.62
FASTTEXT- Common Crawl
0.49
0.75
0.54
BIVEC
0.40
0.72
0.57
TRANSGRAM
0.42
0.74
0.59
SENT2VEC uni.
0.49
0.58
0.51
BI-SENT2VEC uni.
0.57
0.78
0.60
BI-SENT2VEC uni. + bi.
0.60
0.82
0.66
Table Monolingual word similarity task performance of our methods when trained on en-es ParaCrawl data. We report Pearson correlation scores.
SimLex-999 en
WS-353
RG-65
Method Dataset
en
MUSE
0.38
0.74
0.72
RCSLS
0.38
0.74
0.70
FASTTEXT- Common Crawl
0.49
0.75
0.76
BIVEC
0.40
0.70
0.74
TRANSGRAM
0.39
0.72
0.74
SENT2VEC uni.
0.46
0.75
0.71
BI-SENT2VEC uni.
0.55
0.78
0.74
BI-SENT2VEC uni. bi.
0.59
0.79
0.78
Table Monolingual word similarity task performance our methods when trained on en-fr ParaCrawl data. We report Pearson correlation scores.
SimLex-999 en
WS-353
Method Dataset
de
en
de
MUSE
0.38
0.41
0.74
0.68
RCSLS
0.38
0.43
0.74
0.70
FASTTEXT- Common Crawl
0.49
0.39
0.75
0.64
BIVEC
0.40
0.41
0.71
0.62
TRANSGRAM
0.42
0.42
0.74
0.66
SENT2VEC uni.
0.48
0.38
0.70
0.63
BI-SENT2VEC uni.
0.56
0.47
0.76
0.68
BI-SENT2VEC uni. + bi.
0.59
0.53
0.75
0.70
Table 10: Monolingual word similarity task performance our methods when trained en- de ParaCrawl data. We report Pearson correlation scores.
14


Reviews:
[{"review_id": "r1xkQQ6iYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n***Update***\n\nI thank the reviewers for answering my questions, and I have read the reviews from the other reviewers.  I am borderline on this paper, but still learn towards rejection. I feel that it is a bit incremental and still a little misleading/over-stated. For instance, xnli isn't mentioned but mldoc is, it isn't clear in the table which methods are mappings and which aren't (big data difference), in the mldoc experiment was more parallel data used for these specific languages than was used for LASER? the experiment I asked for wasn't done on the same corpora head-to-head with LASER and their approach. I think any claims about outperforming LASER need to be evaluated on the same corpus as much as possible (Europarl) and weaknesses of the approach should also be mentioned (it is not multilingual in these evaluations). Otherwise you conflate data/multilingualness/model which makes it hard to draw conclusions from the experiments. Also CSLS  was not applied to the ACL paper and this makes a huge difference and should be accounted for (and also idf possibly). Also this paper still isn't at least mentioned which is very related in my opinion.\n\nThis paper proposes a method to learn bilingual word embeddings by modifying the Sent2Vec (which is based on word2vec) approach, and applying it to bitext. They evaluate on monolingual and bilingual word similarity, bitext mining, and a zero-shot document classification task where a classifier is trained on data in one language but evaluated on another (using their embeddings as features in both cases).\n\nI have the following concerns about this paper:\n\n1) For the sentence mining tasks - how come Ormazabal 2019 is not included? The baselines are weak and the task is not standard. One of the baselines used here is MUSE which is unsupervised (The refined version of MUSE was also not included in these experiments). There are fixed datasets that people have experimented with (like BUCC) that would help tie in your results here with the literature a little better. These could include LASER (which is already compared to for document classification) and \"Simple and Effective Paraphrastic Similarity from Parallel Translations\" ACL 2019, which like this paper, proposes a pooled token embedding approach and outperforms more complicated architectures. Neither of these approaches uses idf either (how much does idf help - is it really needed?).\n\n2) For the zero-shot document classification task, it should also be pointed out that LASER can handle many languages all at once. Can this approach also work well if all languages were trained jointly?. Also did you evaluate on XNLI for zero-shot as well? LASER does very well here. I do realize comparing to LASER is not really fair since it is trained on so much data, however the model is similar to previous versions of LASER that were trained on Europarl, which could also be used as the data for training your models for a more apples-to-apples comparison.\n\n3) Note that also the improvements on monolingual similarity using parallel data are well known. For instance \"Embedding Word Similarity with Neural Machine Translation\" (arXiv 2014).. Also even the base for the current state of the art models on SimLex-999, Paragram (TACL 2015), used paraphrases created by pivoting on parallel data.\n\nI think the main contributions of this paper are modifying Sent2vec so it can be used on bilingual data and using it to learn nice representations for words and sentences (and documents). I think that to be published, it should clearly outperform and/or have advantages over all previous works - and this is not clear from this paper in its current form. It is okay if it doesn't do the best on everything, but it is hard to tell how this work currently fits into the literature especially in terms of the sentence-level tasks which are a focus.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BygJyY52FS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper does not bring anything novel to the field of cross-lingual representation learning: it just revisits some older ideas (from the period of 2013-2015), now revamped, given the fact that more sophisticated and more effective methods are used to model exactly the same intuitions. I see this work as largely incremental, and it just further supports what has been known before, and it further supports recent findings (which are all quite straightforward) from the work of Ormazabal et al. (ACL 2019). The actual model implementation is a straightforward extension of the Sent2Vec model to cross-lingual scenarios, inspired by previous work (e.g., the work on TransGram and BiVec), so the paper is also very incremental from the methodological perspective.\n\nI am puzzled why MUSE is selected as the unsupervised baseline given that fact that: 1) previous work showed its non-robustness for many language pairs, 2) the VecMap model of Artetxe et al. has been proven as the most robust unsupervised cross-lingual word embedding model in several recent empirical analyses - see e.g., Glavas et al. (ACL 2019), Heyman et al. (NAACL 2019), or the original VecMap work. Also, I am puzzled why the paper overstates the rekindled interest towards TransGram, given that TransGram and especially BiVec are well-known models that learn from parallel data.\n\nAnother note related to evaluation: to really establish how different cross-lingual embeddings compare to each other, a wider set of experiments and downstream evaluation is definitely required, see the work of e.g., Glavas et al. (ACL 2019). \n\nMost importantly, the paper evaluates only on very similar language pairs. The main reason why much recent work has focused on alignment-based/projection-based methods was quite pragmatic: we need such weakly supervised methods where we cannot assume the abundance of parallel data to enable cross-lingual transfer in resource-poor settings. If parallel data exists, it is quite intuitive and obvious (and also empirically validated before) that joint modeling is a better choice than a weakly supervised method that just uses 1K or 5K translation pairs. In fact, I am not even sure that it is fair to compare models that rely only on 1K translation pairs with models that draw their strength from 1M or 2M parallel sentences. This paper just shows that, if we have parallel data (which we do for many resource-richer language pairs), it is better to do joint modeling instead of learning simple alignments, but that is a pretty trivial finding imho.\n\nAre the results on MLDoc really state-of-the-art? The results are actually quite mixed, and the advantage of Bi-Sent2Vec is its quicker training. However, what about more recent methods such as XLM which rely on exactly the same resources as Bi-Sent2Vec to do the zero-shot classification task?\n\nTable 2: it is a well-known fact that multilingual training can improve performance in monolingual supervision: see e.g., the work of Faruqui and Dyer (EACL 2014, not cited). Alignment-based approaches that apply the Orthogonal Procrustes mapping cannot improve on monolingual word similarity simply because the orthogonality constraint preserves the topology of the original space. Therefore, evaluating different embedding methods on the intrinsic word similarity task is not a sound evaluation protocol imho - it would be much more informative to plug the embeddings as features in a classification or a parsing task (or something else).\n\nFigure 2: corpus size. Based on the results presented, it seems that the performance saturates by adding more parallel data, but the authors fail to fully understand their evaluation data in the first place. For instance, there are multiple problems with the MUSE datasets, as discussed in the recent work of Kementchedjhieva et al. (EMNLP 2019) - it evaluates mostly high-frequent word (actually - noun) translation, and of course that this saturates more quickly. It doesn't by any means imply that joint training therefore requires less data to reach peak performance: this is true only with the MUSE dataset, and is not a general truth.\n\nMinor:\n- The work of Artetxe et al. (ACL 2017) should be cited when talking about bootstrapping alignment-based methods from limited bilingual supervision (instead of the work of Artetxe and Schwenk which concerns learning multilingual sentence embeddings).\n- Many very relevant and historically important papers are omitted from the related work section: e.g., Hermann and Blunsom's work, Chandar et al., Soyer et al., Vulic and Moens, Gouws et al., Levy et al., to name only a few.\n- I am not sure that the statement that BiVec is not needed in the presence of TransGram is true in general: it mostly suggests that there are some deficiencies with the evaluation protocol.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeROFsS5H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Recommendation\nWhile the contributions in this work are not staggeringly innovative, they are well grounded in existing work and well supported by experiments. Therefore I think the paper should be accepted.\n\n* Summarize paper’s major contributions\n\nThe authors aimed to improve on the task of cross-lingual sentence retrieval, by introducing a model with a modified objective function, which utilizes a cross-lingual loss. They demonstrated that this objective function led to large improvements on word-level representation tasks and cross-lingual sentence retrieval, and achieves competitive performance on a document-level task while being more computationally efficient. The authors performed an in-depth ablation study, to support their claims that the proposed model addresses some of the key problems with other existing approaches to cross-lingual representation learning (e.g. hubness). \n\n* Comments on the paper\n\nThis paper is exceptionally well written, organized, and clear. In addition to a solid introduction and related works sections, which frame the problem nicely, the conducted experiments thoroughly demonstrate the performance of the proposed model, as they evaluate on several granularities (word, sentence, document), as well as a robust ablation study and analysis. By the end of the paper, I am sufficiently convinced by the work and its contributions. \n\nMeanwhile, I believe the paper could benefit from more discussion or analysis of cases where the proposed model did not lead to improvements, in particular with Russian in the word-translation retrieval experiment, where TRANSGRAM outperforms the proposed model. Although the authors briefly note this in the Discussion section, there is unfortunately no conversation about why this may be. The proposed model becomes less convincing when I consider that it might only work for other agglutinative, English-like languages, and I wonder how this approach would fair with other morphologically rich languages similar to Russian, and non-agglutinative languages in general. \n\n* Minor corrections:\n\n- In Figure-1’s caption, at the very end, there is a space missing between “sentence_(cross-lingual compotent).”\n\n- Some missing colon’s (:) throughout the paper when breaking from a paragraph to introduce a list (e.g. “Contributions” in the Introductions)\n\n- The x-axis of Figure 2 and Figure 3 are unclear to me, and a bit difficult to read. How do I interpret “10^-2” as a corpus size? In other words, what are “10^-2 amounts of data.” Fix this. \n\nOverall, great work. I also appreciate the details about training both in the paper and appendix, that will be useful for those wishing to reproduce this work. \n\n* Questions for authors\n\n- Why do you think that TRANSGRAM outperformed your system on the word-translation retrieval experiment for Russian? Do you have any reasons to believe that the proposed model cannot extend well to other morphologically rich languages, or languages very dissimilar to English? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a new approach to learning cross-lingual embeddings from parallel data. For an overview of this literature, see [0]. Reviews are mixed, and some objections seem unresolved. The authors also ignore a new line of research in which pretrained language models are used to align vocabularies across languages, e.g., [1-2] The paper would also benefit from a discussion of massively parallel resources such as JW300 and WikiMatrix. Finally, it feels odd not to compare to distilled representations from NMT architectures, e.g., [3]. 

[0] http://www.morganclaypoolpublishers.com/catalog_Orig/product_info.php?products_id=1419
[1] https://www.aclweb.org/anthology/N19-1162.pdf
[2] https://www.aclweb.org/anthology/K19-1004.pdf
[3] https://arxiv.org/abs/1901.07291
Decision: Reject