Title: Offline bilingual word vectors, orthogonal transformations and the inverted softmax
Authors: Samuel L. Smith
David H. P. Turban
Steven Hamblin
Nils Y. Hammerla
Venue: ICLR 2017
Keywords: Natural language processing
Transfer Learning
Applications

Abstract: Usually bilingual word vectors are trained "online''. Mikolov et al. showed they can also be found "offline"; whereby two pre-trained embeddings are aligned with a linear transformation, using dictionaries compiled from expert knowledge. In this work, we prove that the linear transformation between two spaces should be orthogonal. This transformation can be obtained using the singular value decomposition. We introduce a novel "inverted softmax" for identifying translation pairs, with which we improve the precision @1 of Mikolov's original mapping from 34% to 43%, when translating a test set composed of both common and rare English words into Italian. Orthogonal transformations are more robust to noise, enabling us to learn the transformation without expert bilingual signal by constructing a "pseudo-dictionary" from the identical character strings which appear in both languages, achieving 40% precision on the same test set. Finally, we extend our method to retrieve the true translations of English sentences from a corpus of 200k Italian sentences with a precision @1 of 68%.

TL;DR: We show that a linear transformation between word vector spaces should be orthogonal and can be obtained analytically using the SVD,  and introduce the inverted softmax for information retrieval.

Review discussion:
1. Gouws et al., 2015 were not the first to show that aligned sentences can be used alongside monolingual source to learn online bilingual vectors. It was Chandar et al., 2014 [1] who first showed that you can use both monolingual data and sentence aligned bilingual data to learn bilingual representations. Please correct this.
2. Table 4,5,6 are incomplete without comparison with Dinu and Mikolov. Note that you can also use pseudo-dictionary and phrase dictionary in Dinu or Milovo's models. I would like to see those numbers before accepting the message from these tables.
3. Faruqui and Dyer 2014 already did offiline bilingual word vectors using CCA. I agree that their motivation was different. I also agree that authors have some good theory to back it up.
However it is extremely important to highlight the fact that this is not a new model and has been already proposed before. Paper would benefit if authors can list all the major contributions of this paper in the beginning: theory, pseudo-dictionary, inverted softmax, ..
References:
[1] Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. An autoencoder approach to learning bilingual word representations. In Advances in
Neural Information Processing Systems, pp. 1853–1861, 2014
Thank you for your helpful comments!
We will upload a new version soon to incorporate your suggestions and the extra experiments you request, but to answer your specific questions:
1. We apologise for this and will ensure the work you mention is cited properly.
2. We have run the additional experiments you requested, and will include them in the updated version. CCA + inverted softmax outperforms Mikolov and Dinu’s methods in all three cases. The advantage is particularly pronounced when using the pseudo-dictionary, for which the precision @1 when translating from English to Italian was 0.01 for Mikolov’s method and 0.06 for Dinu’s method, compared to 0.40 when using CCA with the inverted softmax. We believe that this difference arises because orthogonal transformations are more robust to the high level of noise present in the pseudo-dictionary. We think these results strengthen the paper, and we are grateful to the referee for requesting them.
3. We certainly do not claim to be the first authors to obtain bilingual word vectors using CCA. Our goal is to show the close connection between the works of Mikolov, Dinu, Faruqui and Xing, and enhance the translation performance using the inverted softmax.
Finally, we demonstrate that high-quality offline word vectors can be obtained without dictionaries from either cognate words or aligned sentences, and that bilingual word vectors can be used to retrieve sentence translations. We will ensure this is clearer in the new version.
we've uploaded a new version, incorporating the comments above.
This paper extends preceding works to create a mapping between the word embedding space of two languages. The word embeddings had been independently trained on monolingual data only, and various forms of bilingual information is used to learn the mapping. This mapping is then used to measure the precision of translations.
In this paper, the authors propose two changes: "CCA" and "inverted softmax".  Looking at Table 1, CCA is only better than Dina et al in 1 out of 6 cases (It/En @1).  Most of the improvements are in fact obtained by the introduction of the inverted softmax normalization.
Overall, I wonder which aspect of this paper is really new. You mention:
- Faruqui & Dyer 2014 already used CCA and dimensionality reduction
- Xing et al 2015 argued already that Mikolov's linear matrix should be orthogonal
Could you make clear in what aspect your work is different from Faruqui & Dyer 2014 (other the fact that you applied the method to measure translation precision) ?
Using cognates instead of a bilingual directory is a nice trick. Please explain how you obtained this list of cognates ? Obviously, this only works for languages with the same alphabet (for instance Greek and Russian are excluded)
Also, it seems to me that in linguistics the term "cognate" refers to words which have a common etymological origin - they don't necessarily have the same written form (e.g. night, nuit, noche, Nacht). Maybe, you should use a different term ? Those words are probably proper names in news texts.
Thank you for your review.
We’d also like to thank you for correcting our use of the term “cognates”, which we have removed from the new version of the text. The pseudo-dictionary is formed from every character string (e.g. “Ciao”) in the English vocabulary which also appears in the Italian vocabulary. This dictionary is obtained by searching for perfect character string matches between the two vocabularies.
Alongside theoretical clarification of the existing literature, our work makes a number of novel contributions, including:
1.	The inverted softmax
2.	The identical strings pseudo-dictionary
3.	Offline bilingual vectors from a phrase dictionary
4.	Sentence translation retrieval using bilingual vectors
Previously offline bilingual word vectors have only been obtained using expert word dictionaries, and they have not been used to retrieve sentence translations.
The paper focuses on bilingual word representation learning with the following setting:
1. Bilingual representation is learnt in an offline manner i.e., we already have monolingual representations for the source and target language and we are learning a common mapping for these two representations.
2. There is no direct word to word alignments available between the source and target language.
This is a practically useful setting to consider and authors have done a good job of unifying the existing solutions for this problem by providing theoretical justifications.
Even though the authors do not propose a new method for offline bilingual representation learning, the paper is significant for the following contributions:
1. Theory for offline bilingual representation learning.
2. Inverted softmax.
3. Using cognate words for languages that share similar scripts.
4. Showing that this method also works at sentence level (to some extent).
Authors have addressed all my pre-review questions and I am ok with their response. I have few more comments:
1. Header for table 3 which says “word frequency” is misleading. “word frequency” could mean that rare words occur in row-1 while I guess authors meant to say that rare words occur in row-5.
2. I see that authors have removed precision @5 and @10 from table-6. Is it because of the space constraints or the results have different trend? I would like to see these results in the appendix.
3. In table-6 what is the difference between row-3 and row-4? Is the only difference NN vs. inverted softmax? Or there are other differences? Please elaborate.
4.
Another suggestion is to try running an additional experiment where one can use both expert dictionary and cognate dictionary. Comparing all 3 methods in this setting should give more valuable insights about the usefulness of cognate dictionary.
We would like to thank you for your positive assessment of our work.
In response to your remaining comments:
1.	We have replaced “Word frequency” with “Word ranking by frequency”
2.	Yes, these results were removed due to the space constraints, but we have now added them in an appendix.
3.	Yes, the only difference here is NN vs. inverted softmax. Both columns use exactly the same orthogonal transformation. We have changed the text to make this clearer.
4.	We did try naively combining the expert dictionary with the identical-strings pseudo-dictionary, but this performed worse than the expert dictionary alone. In hindsight this is not surprising, since the pseudo-dictionary is significantly larger but lower quality. It might be possible to get improved performance by introducing a weighting factor between the two dictionaries, but we have not tried this.
This paper discusses aligning word vectors across language when those embeddings have been learned independently in monolingual settings. There are reasonable scenarios in which such a strategy could come in helpful, so I feel this paper addresses an interesting problem. The paper is mostly well executed but somewhat lacks in evaluation. It would have been nice if a stronger downstream task had been attempted.
The inverted Softmax idea is very nice.
A few minor issues that ought to be addressed in a published version of this paper:
1) There is no mention of Haghighi et al (2008) "Learning Bilingual Lexicons from Monolingual Corpora.", which strikes me as a key piece of prior work regarding the use of CCA in learning bilingual alignment. This paper and links to the work here ought to be discussed.
2) Likewise, Hermann & Blunsom (2013) "Multilingual distributed representations without word alignment." is probably the correct paper to cite for learning multilingual word embeddings from multilingual aligned data.
3) It would have been nicer if experiments had been performed with more divergent language pairs rather than just European/Romance languages
4) A lot of the argumentation around the orthogonality requirements feels related to the idea of using a Mahalanobis distance / covar matrix to learn such mappings. This might be worth including in the discussion
5) I don't have a better suggestion, but is there an alternative to using the term "translation (performance/etc.)" when discussing word alignment across languages? Translation implies something more complex than this in my mind.
6) The Mikolov citation in the abstract is messed up
We would like to thank you for your positive assessment of our work, and of the inverted softmax in particular.
We have realised that our procedure, while very similar to CCA, is not identical. We apologise for this mistake, which we have corrected in the new version. We believe this realisation strengthens the manuscript. We have included additional experiments, and a discussion of the very close relationship between the methods. The two methods have very similar performance, but our approach is numerically cheaper.
In response to your specific comments,
1.	We have now cited this work, and briefly discuss it in the text.
2.
Similarly, this paper is now cited. We still consider Chandar et al. the first to obtain bilingual vectors from monolingual corpora and paired sentences, since Hermann and Blunsom cite an early version of Chandar’s work in their manuscript.
3.	We hope to explore additional language pairs in future.
However, we believe that the range of tasks we consider in this manuscript, including a number of new tasks not considered in previous work, does provide a rigorous experimental evaluation. These new tasks include the pseudo dictionary of identical strings, the phrase dictionary, and sentence retrieval between languages. One of the goals of this work was to show that offline bilingual vectors can be used in a number of ways not previously considered in the literature.
4.	This is a very interesting point. The Mahalanobis distance is an alternative to the cosine similarity.
Although we do not discuss it in the manuscript, one could devise an alternative alignment procedure, which minimises the Mahalanobis distance of the dictionary rather than maximising the cosine similarity. I suspect that this relates to the slight difference between our procedure and CCA;
but we have been unable to find a reference which discusses this.
5.	We do recognise your concern, but I’m afraid we couldn’t come up with an appropriate alternative to “translation”.
6.	We fixed this.
Dear reviewers and readers,
We’d like to thank you all for your positive comments about our manuscript. We were particularly pleased that all three reviewers recommended our work be accepted, and by the interest reviewers expressed in the “inverted softmax”. We have uploaded an updated version. There are three main changes we would like to draw readers’ attention to:
Our use of the term “cognates” was misleading and we have removed it from the new version. To be completely clear, we extract the pseudo-dictionary by finding the identical character strings like “DNA” and “Ciao” which appear in both the English and Italian vocabularies. These identical strings can be found trivially without any expert knowledge.
We also realised that our procedure, while very similar to CCA, is not identical. We apologise for this mistake, which we have corrected in the new version. We believe this realisation strengthens the manuscript. We provide additional experiments, and a discussion of the very close relationship between the methods. The two methods have very similar performance, but our approach is numerically cheaper.
Shortly after our manuscript was submitted to ICLR, another paper was published [1], which presents a similar theoretical analysis of offline bilingual word vectors. We would like to thank the anonymous reader for bringing this work to our attention, now properly cited. This paper also discusses the need for an orthogonal transformation, and proposes the same novel SVD procedure we propose here to obtain this transformation.
However, our work contains a number of contributions not present in their work, including:
1.	The use of dimensionality reduction after the SVD
2.	The inverted softmax
3.	The identical strings pseudo-dictionary
4.	Offline vector alignment using a phrase dictionary
5.	Sentence translation retrieval using bilingual vectors
We will respond to the specific comments of each reviewer underneath their reviews.
Best wishes,
Sam
[1] Artetxe, M., Labaka, G., & Agirre, E. (2016). Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP-16), 2289–2294.
We have uploaded the final version. The text is unchanged, but we have modified the title to emphasise the aspects of the paper which have been of most interest to readers (particularly the inverted softmax).
We'd like to thank the PC for accepting our manuscript,
Sam

Meta-review:
role: PC
data: comment: This is a nice contribution and that present some novel and interesting ideas. At the same time, the empirical evaluation is somewhat thin and could be improved. Nevertheless, the PCs believe this will make a good contribution to the Conference Track.

Decision: Accept (Poster)