Title: Word translation without parallel data
Authors: Guillaume Lample
Alexis Conneau
Marc'Aurelio Ranzato
Ludovic Denoyer
Hervé Jégou
Venue: ICLR 2018
Keywords: unsupervised learning
machine translation
multilingual embeddings
parallel dictionary induction
adversarial training

Abstract: State-of-the-art methods for learning cross-lingual word embeddings have relied on bilingual dictionaries or parallel corpora. Recent studies showed that the need for parallel data supervision can be alleviated with character-level information. While these methods showed encouraging results, they are not on par with their supervised counterparts and are limited to pairs of languages sharing a common alphabet. In this work, we show that we can build a bilingual dictionary between two languages without using any parallel corpora, by aligning monolingual word embedding spaces in an unsupervised way. Without using any character information, our model even outperforms existing supervised methods on cross-lingual tasks for some language pairs. Our experiments demonstrate that our method works very well also for distant language pairs, like English-Russian or English-Chinese. We finally describe experiments on the English-Esperanto low-resource language pair, on which there only exists a limited amount of parallel data, to show the potential impact of our method in fully unsupervised machine translation. Our code, embeddings and dictionaries are publicly available.

TL;DR: Aligning languages without the Rosetta Stone: with no parallel data, we construct bilingual dictionaries using adversarial training, cross-domain local scaling, and an accurate proxy criterion for cross-validation.

Review discussion:
This paper presents a new method for obtaining a bilingual dictionary, without requiring any parallel data between the source and target languages. The method consists of an adversarial approach for aligning two monolingual word embedding spaces, followed by a refinement step using frequent aligned words (according to the adversarial mapping). The approach is evaluated on single word translation, cross-lingual word similarity, and sentence translation retrieval tasks.
The paper presents an interesting approach which achieves good performance. The work is presented clearly, the approach is well-motivated and related to previous studies, and a thorough evaluation is performed.
My one concern is that the supervised approach that the paper compares to is limited: it is trained on a small fixed number of anchor points, while the unsupervised method uses significantly more words. I think the paper's comparisons are valid, but the abstract and introduction make very strong claims about outperforming "state-of-the-art supervised approaches". I think either a stronger supervised baseline should be included (trained on comparable data as the unsupervised approach), or the language/claims in the paper should be softened. The same holds for statements like "... our method is a first step ...", which is very hard to justify. I also do not think it is necessary to over-sell, given the solid work in the paper.
Further comments, questions and suggestions:
- It might be useful to add more details of your actual approach in the Abstract, not just what it achieves.
- Given you use trained word embeddings, it is not a given that the monolingual word embedding spaces would be alignable in a linear way. The actual word embedding method, therefore, has a big influence on performance (as you show). Could you comment on how crucial it would be to train monolingual embedding spaces on similar domains/data with similar co-occurrence statistics, in order for your method to be appropriate?
- Would it be possible to add weights to the terms in eq. (6), or is this done implicitly?
- How were the 5k source words for Procrustes supervised baseline selected?
- Have you considered non-linear mappings, or jointly training the monolingual word embeddings while attempting the linear mapping between embedding spaces?
- Do you think your approach would benefit from having a few parallel training points?
Some minor grammatical mistakes/typos (nitpicking):
- "gives a good performance" -> "gives good performance"
- "Recent works", "several works", "most works", etc. -> "recent studies", "several studies", etc.
- "i.e, the improvements" -> "i.e., the improvements"
The paper is well-written, relevant and interesting. I therefore recommend that the paper be accepted.
We thank the reviewer for the feedback and comments.
It is true that the supervised approach is limited in the sense that it only considers 5000 pairs of words.
However, previous works have shown that using more than 5000 pairs of words does not improve the performance (Artetxe et al. (2017)), and can even be detrimental (see Dinu et al. (2015)). This is why we decided to consider 5000 pairs only, to be consistent with previous works.
Also, note that we made our supervised baseline (Procrustes + CSLS) as strong as possible, and it is actually state-of-the-art.
Regarding the claim "this is a first step towards fully unsupervised machine translation", what we meant is that the method proposed in the paper could potentially be used in a more complex framework for unsupervised MT at the sentence level. We rephrased this sentence in the updated version of the paper.
We now address the comments / suggestions of the reviewer:
- The abstract could indeed benefit from details about the model. We will add some.
- The co-occurrence statistics have indeed an impact on the overall performance of the model. This impact is consistent for both supervised and unsupervised approaches.
Indeed, our unsupervised method obtains 66.2% accuracy on the English-Italian pair on the Wikipedia corpora (Table 2), and 45.1% accuracy on the UKWAC / ITWAC non-comparable corpora. This result was not in the paper (we thought it was redundant with Table 1), but we added it in Table 2 in the updated version. Figure 3 in the appendix also gives insights about the impact of the similarity of the two domains, by comparing the quality of English-English alignment using embeddings trained on different English corpora.
- It would indeed possible to add weights in Equation (6). We tried to weight the r_S and r_T terms, but we did not observe a significant improvement compared to the current equation.
- In the supervised approach, we generated translations for all words from the source language to the target language, and vice-versa (a translation being a pair (x, y) associated with the probability for y of being the correct translation of x).
Then, we considered all pairs of words (x, y) such that y has a high probability of being a translation of x, but also that x has a high probability of being a translation of y.
Then, we sorted all generated translation pairs by frequency of the source word, and took the 5000 first resulting pairs.
- We tried to use non-linear mappings (namely a feedforward network with 1 or 2 hidden layers), but in these experiments, the adversarial training was quite unstable, and like in Mikolov et al. (2013), we did not observe better results compared to the linear mapping. Actually, the linear mapping was working significantly better, and since the Procrustes algorithm in the refinement step requires the mapping to be linear, we decided to focus on this type of mapping.
Moreover, the linear mapping is convenient because we can impose the orthogonality constraint, which guarantees that the quality of the source monolingual embeddings is preserved after mapping.
- We did not try to jointly learn the embeddings as well as the mapping, but this is a nice idea and definitely something that needs to be investigated. We think that the joint learning could improve the cross-lingual embeddings, but especially, it could significantly improve the quality of monolingual embeddings on low-resource languages.
- Our approach would definitely benefit from having a few parallel training points. These points could be used to pretrain the linear mapping for the adversarial training, or even as a validation dataset. This will be the focus of future work.
Thank you for the very detailed response, also to the other reviewers' comments: all the questions and concerns were addressed very well.
The paper proposes a method to learn bilingual dictionaries without parallel data using an adversarial technique. The task is interesting and relevant, especially for in low-resource language pair settings.
The paper, however, misses comparison against important work from the literature that is very relevant to their task — decipherment (Ravi, 2013; Nuhn et al., 2012; Ravi & Knight, 2011) and other approaches like CCA.
The former set of works, while focused on machine translation also learns a translation table in the process.
Besides, the authors also claim that their approach is particularly suited for low-resource MT and list this as one of their contributions. Previous works have used non-parallel and comparable corpora to learn MT models and for bilingual lexicon induction. The authors seem aware of corpora used in previous works (Tiedemann, 2012) yet provide no comparison against any of these methods.
While some of the bilingual lexicon extraction works are cited (Haghighi et al., 2008; Artetxe et al., 2017), they do not demonstrate how their approach performs against these baseline methods. Such a comparison, even on language pairs which share some similarities (e.g., orthography), is warranted to determine the effectiveness of the proposed approach.
The proposed methodology is not novel, it rehashes existing adversarial techniques instead of other probabilistic models used in earlier works.
For the translation task, it would be useful to see performance of a supervised MT baseline (many tools available in open-source) that was trained on similar amount of parallel training data (60k pairs) and see the gap in performance with the proposed approach.
The paper mentions that the approach is “unsupervised”.
However, it relies on bootstrapping from word embeddings learned on Wikipedia corpus, which is a comparable corpus even though individual sentences are not aligned across languages. How does the quality degrade if word embeddings had to be learned from scratch or initialized from a different source?
We thank the reviewer for the feedback and comments.
The main concern of the review is about the lack of comparisons with existing works.
- The reviewer reproaches the lack of comparison against CCA, while the comparison against CCA is provided in Table 2. The reviewer also points out the lack of comparison against Artetxe et al. (2017). This comparison is also provided in the paper.
- We agree that our method could be compared to decipherment techniques, and would have been happy to try the method of Ravi & Knight but there is no open-source version of their code available online (like for Faruqui & Dyer, Dinu et al, Artexte et al, Smith et al).
Therefore, considering the large body of literature in that domain, we focused on comparing our approach with the most recent state-of-the-art and supervised approaches, which in our opinion is a fair way to evaluate against reproducible baselines.
The second reviewer concern is about the performance of the model on non-comparable corpora. We considered that this was redundant with the results on Wikipedia provided in Table 1 and Table 2. As explained in one previous comment, our strategy was to first show that our supervised method (Procrustes-CSLS) is state-of-the-art, and then to compare our unsupervised approach against this new baseline. We added the result of our unsupervised approach (Adv - Refine - CSLS) on non-comparable WaCky corpora in Table 2.
In particular, our unsupervised model on the non-comparable WaCky datasets is also state of the art with 45.1% accuracy.
The reviewer criticises the lack of novelty. To the best of our knowledge, the fact that an adversarial approach obtains state-of-the-art cross-lingual embeddings is new. Most importantly, the contributions of our paper are not limited to the adversarial approach. The CSLS method introduced to mitigate the hubness problem is new, and improves the state-of-the-art by up to 24% on the sentence retrieval task, as well as it improves the supervised baseline. We also introduced an unsupervised criterion that is highly correlated with the cross-lingual embeddings quality, which is also novel as far as we know, and a key element for training.
Al last, please consider that we made our code publicly available and provided high-quality dictionaries for 110 oriented language pairs to help the community, as this type of resources are very difficult to find online.
An unsupervised approach is proposed to build bilingual dictionaries without parallel corpora, by aligning the monolingual word embeddings spaces, i.a. via adversarial learning.
The paper is very well-written and makes for a rather pleasant read, save for some need for down-toning the claims to novelty as voiced in the comment re: Ravi & Knight (2011) or simply in general: it's a very nice paper, I enjoy reading it *in spite*, and not *because* of the text sales-pitching itself at times.
There are some gaps in the awareness of the related work in the sub-field of bilingual lexicon induction, e.g. the work by Vulic & Moens (2016).
The evaluation is for the most part intrinsic, and it would be nice to see the approach applied downstream beyond the simplistic task of English-Esperanto translation: plenty of outlets out there for applying multilingual word embeddings. Would be nice to see at least some instead of the plethora of intrinsic evaluations of limited general interest.
In my view, to conclude, this is still a very nice paper, so I vote clear accept, in hope to see these minor flaws filtered out in the revision.
We thank the reviewer for the feedback and comments.
As mentioned in the comments, we added to the paper citations to the work of Ravi & Knight (2011) and some subsequent works on decipherment, and down-toned some claims in the paper.
Thank you for pointing the paper of Vulic & Moens, we were not aware of this paper and we added a citation in the updated version of the paper. Note however that the work of Vulic & Moens relies on document-aligned corpora while our method does not require any form of alignment.
We evaluated the cross-lingual embeddings on 4 different tasks: cross-lingual word similarity, word translation, sentence retrieval, and sentence translation.
It is true that the quality of these embeddings on other downstream tasks would be interesting and will be investigated in future work.
We thank all the reviewers for the feedback and comments. We replied to each of them individually, and uploaded a revised version of the paper.
In particular, we:
- Rephrased one of the claims made in the abstract about unsupervised machine translation
- Added the requested 45.1% result of our unsupervised approach on the WaCky datasets
- Fixed some typos
- Added missing citations

Meta-review:
role: PC
data: comment: There is significant discussion on this paper and high variance between reviewers:  one reviewer gave the paper a low score.  However the committee feels that this paper should be accepted at the conference since it provides a better framework for reproducibility, performs more large scale experiments than prior work.  One small issue the lack of comparison in terms of empirical results between this work and Zhang et al's work, but the responses provided to both the reviewers and anonymous commenters seem to be satisfactory.

Decision: Accept (Poster)