Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: * Recommendation
While the contributions in this work are not staggeringly innovative, they are well grounded in existing work and well supported by experiments. Therefore I think the paper should be accepted.

* Summarize paper’s major contributions

The authors aimed to improve on the task of cross-lingual sentence retrieval, by introducing a model with a modified objective function, which utilizes a cross-lingual loss. They demonstrated that this objective function led to large improvements on word-level representation tasks and cross-lingual sentence retrieval, and achieves competitive performance on a document-level task while being more computationally efficient. The authors performed an in-depth ablation study, to support their claims that the proposed model addresses some of the key problems with other existing approaches to cross-lingual representation learning (e.g. hubness). 

* Comments on the paper

This paper is exceptionally well written, organized, and clear. In addition to a solid introduction and related works sections, which frame the problem nicely, the conducted experiments thoroughly demonstrate the performance of the proposed model, as they evaluate on several granularities (word, sentence, document), as well as a robust ablation study and analysis. By the end of the paper, I am sufficiently convinced by the work and its contributions. 

Meanwhile, I believe the paper could benefit from more discussion or analysis of cases where the proposed model did not lead to improvements, in particular with Russian in the word-translation retrieval experiment, where TRANSGRAM outperforms the proposed model. Although the authors briefly note this in the Discussion section, there is unfortunately no conversation about why this may be. The proposed model becomes less convincing when I consider that it might only work for other agglutinative, English-like languages, and I wonder how this approach would fair with other morphologically rich languages similar to Russian, and non-agglutinative languages in general. 

* Minor corrections:

- In Figure-1’s caption, at the very end, there is a space missing between “sentence_(cross-lingual compotent).”

- Some missing colon’s (:) throughout the paper when breaking from a paragraph to introduce a list (e.g. “Contributions” in the Introductions)

- The x-axis of Figure 2 and Figure 3 are unclear to me, and a bit difficult to read. How do I interpret “10^-2” as a corpus size? In other words, what are “10^-2 amounts of data.” Fix this. 

Overall, great work. I also appreciate the details about training both in the paper and appendix, that will be useful for those wishing to reproduce this work. 

* Questions for authors

- Why do you think that TRANSGRAM outperformed your system on the word-translation retrieval experiment for Russian? Do you have any reasons to believe that the proposed model cannot extend well to other morphologically rich languages, or languages very dissimilar to English? 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper does not bring anything novel to the field of cross-lingual representation learning: it just revisits some older ideas (from the period of 2013-2015), now revamped, given the fact that more sophisticated and more effective methods are used to model exactly the same intuitions. I see this work as largely incremental, and it just further supports what has been known before, and it further supports recent findings (which are all quite straightforward) from the work of Ormazabal et al. (ACL 2019). The actual model implementation is a straightforward extension of the Sent2Vec model to cross-lingual scenarios, inspired by previous work (e.g., the work on TransGram and BiVec), so the paper is also very incremental from the methodological perspective.

I am puzzled why MUSE is selected as the unsupervised baseline given that fact that: 1) previous work showed its non-robustness for many language pairs, 2) the VecMap model of Artetxe et al. has been proven as the most robust unsupervised cross-lingual word embedding model in several recent empirical analyses - see e.g., Glavas et al. (ACL 2019), Heyman et al. (NAACL 2019), or the original VecMap work. Also, I am puzzled why the paper overstates the rekindled interest towards TransGram, given that TransGram and especially BiVec are well-known models that learn from parallel data.

Another note related to evaluation: to really establish how different cross-lingual embeddings compare to each other, a wider set of experiments and downstream evaluation is definitely required, see the work of e.g., Glavas et al. (ACL 2019). 

Most importantly, the paper evaluates only on very similar language pairs. The main reason why much recent work has focused on alignment-based/projection-based methods was quite pragmatic: we need such weakly supervised methods where we cannot assume the abundance of parallel data to enable cross-lingual transfer in resource-poor settings. If parallel data exists, it is quite intuitive and obvious (and also empirically validated before) that joint modeling is a better choice than a weakly supervised method that just uses 1K or 5K translation pairs. In fact, I am not even sure that it is fair to compare models that rely only on 1K translation pairs with models that draw their strength from 1M or 2M parallel sentences. This paper just shows that, if we have parallel data (which we do for many resource-richer language pairs), it is better to do joint modeling instead of learning simple alignments, but that is a pretty trivial finding imho.

Are the results on MLDoc really state-of-the-art? The results are actually quite mixed, and the advantage of Bi-Sent2Vec is its quicker training. However, what about more recent methods such as XLM which rely on exactly the same resources as Bi-Sent2Vec to do the zero-shot classification task?

Table 2: it is a well-known fact that multilingual training can improve performance in monolingual supervision: see e.g., the work of Faruqui and Dyer (EACL 2014, not cited). Alignment-based approaches that apply the Orthogonal Procrustes mapping cannot improve on monolingual word similarity simply because the orthogonality constraint preserves the topology of the original space. Therefore, evaluating different embedding methods on the intrinsic word similarity task is not a sound evaluation protocol imho - it would be much more informative to plug the embeddings as features in a classification or a parsing task (or something else).

Figure 2: corpus size. Based on the results presented, it seems that the performance saturates by adding more parallel data, but the authors fail to fully understand their evaluation data in the first place. For instance, there are multiple problems with the MUSE datasets, as discussed in the recent work of Kementchedjhieva et al. (EMNLP 2019) - it evaluates mostly high-frequent word (actually - noun) translation, and of course that this saturates more quickly. It doesn't by any means imply that joint training therefore requires less data to reach peak performance: this is true only with the MUSE dataset, and is not a general truth.

Minor:
- The work of Artetxe et al. (ACL 2017) should be cited when talking about bootstrapping alignment-based methods from limited bilingual supervision (instead of the work of Artetxe and Schwenk which concerns learning multilingual sentence embeddings).
- Many very relevant and historically important papers are omitted from the related work section: e.g., Hermann and Blunsom's work, Chandar et al., Soyer et al., Vulic and Moens, Gouws et al., Levy et al., to name only a few.
- I am not sure that the statement that BiVec is not needed in the presence of TransGram is true in general: it mostly suggests that there are some deficiencies with the evaluation protocol.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
***Update***

I thank the reviewers for answering my questions, and I have read the reviews from the other reviewers.  I am borderline on this paper, but still learn towards rejection. I feel that it is a bit incremental and still a little misleading/over-stated. For instance, xnli isn't mentioned but mldoc is, it isn't clear in the table which methods are mappings and which aren't (big data difference), in the mldoc experiment was more parallel data used for these specific languages than was used for LASER? the experiment I asked for wasn't done on the same corpora head-to-head with LASER and their approach. I think any claims about outperforming LASER need to be evaluated on the same corpus as much as possible (Europarl) and weaknesses of the approach should also be mentioned (it is not multilingual in these evaluations). Otherwise you conflate data/multilingualness/model which makes it hard to draw conclusions from the experiments. Also CSLS  was not applied to the ACL paper and this makes a huge difference and should be accounted for (and also idf possibly). Also this paper still isn't at least mentioned which is very related in my opinion.

This paper proposes a method to learn bilingual word embeddings by modifying the Sent2Vec (which is based on word2vec) approach, and applying it to bitext. They evaluate on monolingual and bilingual word similarity, bitext mining, and a zero-shot document classification task where a classifier is trained on data in one language but evaluated on another (using their embeddings as features in both cases).

I have the following concerns about this paper:

1) For the sentence mining tasks - how come Ormazabal 2019 is not included? The baselines are weak and the task is not standard. One of the baselines used here is MUSE which is unsupervised (The refined version of MUSE was also not included in these experiments). There are fixed datasets that people have experimented with (like BUCC) that would help tie in your results here with the literature a little better. These could include LASER (which is already compared to for document classification) and "Simple and Effective Paraphrastic Similarity from Parallel Translations" ACL 2019, which like this paper, proposes a pooled token embedding approach and outperforms more complicated architectures. Neither of these approaches uses idf either (how much does idf help - is it really needed?).

2) For the zero-shot document classification task, it should also be pointed out that LASER can handle many languages all at once. Can this approach also work well if all languages were trained jointly?. Also did you evaluate on XNLI for zero-shot as well? LASER does very well here. I do realize comparing to LASER is not really fair since it is trained on so much data, however the model is similar to previous versions of LASER that were trained on Europarl, which could also be used as the data for training your models for a more apples-to-apples comparison.

3) Note that also the improvements on monolingual similarity using parallel data are well known. For instance "Embedding Word Similarity with Neural Machine Translation" (arXiv 2014).. Also even the base for the current state of the art models on SimLex-999, Paragram (TACL 2015), used paraphrases created by pivoting on parallel data.

I think the main contributions of this paper are modifying Sent2vec so it can be used on bilingual data and using it to learn nice representations for words and sentences (and documents). I think that to be published, it should clearly outperform and/or have advantages over all previous works - and this is not clear from this paper in its current form. It is okay if it doesn't do the best on everything, but it is hard to tell how this work currently fits into the literature especially in terms of the sentence-level tasks which are a focus.
review_title: Official Blind Review #2

Meta-review: The authors propose a new approach to learning cross-lingual embeddings from parallel data. For an overview of this literature, see [0]. Reviews are mixed, and some objections seem unresolved. The authors also ignore a new line of research in which pretrained language models are used to align vocabularies across languages, e.g., [1-2] The paper would also benefit from a discussion of massively parallel resources such as JW300 and WikiMatrix. Finally, it feels odd not to compare to distilled representations from NMT architectures, e.g., [3]. 

[0] http://www.morganclaypoolpublishers.com/catalog_Orig/product_info.php?products_id=1419
[1] https://www.aclweb.org/anthology/N19-1162.pdf
[2] https://www.aclweb.org/anthology/K19-1004.pdf
[3] https://arxiv.org/abs/1901.07291
Decision: Reject