Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper focuses on bilingual word representation learning with the following setting:

1. Bilingual representation is learnt in an offline manner i.e., we already have monolingual representations for the source and target language and we are learning a common mapping for these two representations.
2. There is no direct word to word alignments available between the source and target language.

This is a practically useful setting to consider and authors have done a good job of unifying the existing solutions for this problem by providing theoretical justifications. Even though the authors do not propose a new method for offline bilingual representation learning, the paper is significant for the following contributions:

1. Theory for offline bilingual representation learning.
2. Inverted softmax.
3. Using cognate words for languages that share similar scripts.
4. Showing that this method also works at sentence level (to some extent).

Authors have addressed all my pre-review questions and I am ok with their response. I have few more comments:

1. Header for table 3 which says “word frequency” is misleading. “word frequency” could mean that rare words occur in row-1 while I guess authors meant to say that rare words occur in row-5.
2. I see that authors have removed precision @5 and @10 from table-6. Is it because of the space constraints or the results have different trend? I would like to see these results in the appendix.
3. In table-6 what is the difference between row-3 and row-4? Is the only difference NN vs. inverted softmax? Or there are other differences? Please elaborate.
4. Another suggestion is to try running an additional experiment where one can use both expert dictionary and cognate dictionary. Comparing all 3 methods in this setting should give more valuable insights about the usefulness of cognate dictionary.

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends preceding works to create a mapping between the word embedding space of two languages. The word embeddings had been independently trained on monolingual data only, and various forms of bilingual information is used to learn the mapping. This mapping is then used to measure the precision of translations.

In this paper, the authors propose two changes: "CCA" and "inverted softmax".  Looking at Table 1, CCA is only better than Dina et al in 1 out of 6 cases (It/En @1).  Most of the improvements are in fact obtained by the introduction of the inverted softmax normalization.

Overall, I wonder which aspect of this paper is really new. You mention:
 - Faruqui & Dyer 2014 already used CCA and dimensionality reduction
 - Xing et al 2015 argued already that Mikolov's linear matrix should be orthogonal

Could you make clear in what aspect your work is different from Faruqui & Dyer 2014 (other the fact that you applied the method to measure translation precision) ?

Using cognates instead of a bilingual directory is a nice trick. Please explain how you obtained this list of cognates ? Obviously, this only works for languages with the same alphabet (for instance Greek and Russian are excluded)

Also, it seems to me that in linguistics the term "cognate" refers to words which have a common etymological origin - they don't necessarily have the same written form (e.g. night, nuit, noche, Nacht). Maybe, you should use a different term ? Those words are probably proper names in news texts.

review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper discusses aligning word vectors across language when those embeddings have been learned independently in monolingual settings. There are reasonable scenarios in which such a strategy could come in helpful, so I feel this paper addresses an interesting problem. The paper is mostly well executed but somewhat lacks in evaluation. It would have been nice if a stronger downstream task had been attempted.

The inverted Softmax idea is very nice.

A few minor issues that ought to be addressed in a published version of this paper:

1) There is no mention of Haghighi et al (2008) "Learning Bilingual Lexicons from Monolingual Corpora.", which strikes me as a key piece of prior work regarding the use of CCA in learning bilingual alignment. This paper and links to the work here ought to be discussed.
2) Likewise, Hermann & Blunsom (2013) "Multilingual distributed representations without word alignment." is probably the correct paper to cite for learning multilingual word embeddings from multilingual aligned data.
3) It would have been nicer if experiments had been performed with more divergent language pairs rather than just European/Romance languages
4) A lot of the argumentation around the orthogonality requirements feels related to the idea of using a Mahalanobis distance / covar matrix to learn such mappings. This might be worth including in the discussion
5) I don't have a better suggestion, but is there an alternative to using the term "translation (performance/etc.)" when discussing word alignment across languages? Translation implies something more complex than this in my mind.
6) The Mikolov citation in the abstract is messed up
review_title: Inverted Softmax is nice

final_score: rating: null
review_content: 1. Gouws et al., 2015 were not the first to show that aligned sentences can be used alongside monolingual source to learn online bilingual vectors. It was Chandar et al., 2014 [1] who first showed that you can use both monolingual data and sentence aligned bilingual data to learn bilingual representations. Please correct this.

2. Table 4,5,6 are incomplete without comparison with Dinu and Mikolov. Note that you can also use pseudo-dictionary and phrase dictionary in Dinu or Milovo's models. I would like to see those numbers before accepting the message from these tables.

3. Faruqui and Dyer 2014 already did offiline bilingual word vectors using CCA. I agree that their motivation was different. I also agree that authors have some good theory to back it up. However it is extremely important to highlight the fact that this is not a new model and has been already proposed before. Paper would benefit if authors can list all the major contributions of this paper in the beginning: theory, pseudo-dictionary, inverted softmax, ..

References:

[1] Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. An autoencoder approach to learning bilingual word representations. In Advances in
Neural Information Processing Systems, pp. 1853–1861, 2014
review_title: Few questions and comments

Meta-review: This is a nice contribution and that present some novel and interesting ideas. At the same time, the empirical evaluation is somewhat thin and could be improved. Nevertheless, the PCs believe this will make a good contribution to the Conference Track.
Decision: Accept (Poster)