Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes a new and simple framework for learning cross-lingual embeddings that, based on well-supported insights, unifies two standard frameworks: joint learning and alignment-based approaches. While I like and acknowledge the fact that the paper examines these frameworks critically and has also some didactic value, I still have some concerns regarding the current experiments, and would like to see some additional analyses in the paper. Honestly, this type of work would work better as a short ACL/EMNLP paper, as the core methodological contribution is a very simple combination of the existing techniques.

With joint methods, it is true that shared words can be used as implicit bilingual learning signal which gets propagated further in the model. Even in the case of alignment-based methods, it was shown that this signal can assist in learning shared cross-lingual spaces, but: 1) such spaces are of lower-quality than the spaces learned by relying on seed dictionaries (see e.g., the work of Vulic and Korhonen (ACL 2016)), 2) this is useful only for similar language pairs which use the same script. The paper fails to provide an insightful discussion on how the scores differ when we move towards more distant languages. For instance, English-Chinese is included as a more distant language pair, and the combined method seems to work fine, but the reader is left wondering why that happens. The same is true for English-Russian. The paper should definitely provide more information and insights here. 

One experiment that would be quite interesting imho is to take seed dictionaries to initialise the joint training method. It will not be unsupervised any more, but it would be very useful to report the differences in results between fully unsupervised joint initialisation (which, as mentioned should work only with more similar languages) and such weakly supervised init: e.g., the method could take all one-to-one translation pairs from the seed dictionary as 'shared words'. It would also be interesting to combine such 'shared words' and words that are really shared (identically spelled words) as initialisation and measure how it affects the results, also in light of differences in language distance. Adding one such experiment would make the paper more interesting.

Some recent strong baselines are missing: e.g., I wonder how the model that does self-learning on top of seed dictionaries (similar to the work of Vulic et al., EMNLP 2019) compares to the proposed method. Also, can we use the same self-learning technique with the combined method here? Would that lead to improved results? Another work I would like to see comparisons to is the work of Zhang et al. (ACL 2019) and despite the fact that the authors explicitly stated that they decided not to compare to the work of Artetxe et al. (ACL 2019) as they see it as concurrent work, I would still like to see that comparison as the model of Artetxe et al. seems very relevant to the presented work.

I do not see the extension to contextualized representations (described in Section 3.2) as a real contribution: this is a very straightforward method to apply an alignment-based method on multilingual BERT representations, and very similar techniques have been probed in previous work (e.g., by Aldarmaki & Diab), only the bilingual signal/dictionary came from parallel data instead.

Finally, as mentioned before, one of the must-have experiments is including more (distant and diverse) language pairs and analysing the results across such language pairs, with an aim to further understand how the levels of sharing, over-sharing, and non-isomorphism affect performance. Also, while the authors state that reduced isomorphism is the key problem of alignment-based methods, I still fail to see how exactly the alignment refinement step does not suffer from that problem? More discussion is needed here.

Other comments:
- It would be useful to also point to the survey paper of Ruder et al. (JAIR 2019) where the difference between alignment-based and joint models is described in more detail and could inform an interested reader beyond the confines of the related work section in this paper. Similarly, differences between joint models and alignment-based models have also been analysed by Upadhyay et al. (ACL 2016); Vulic and Korhonen (ACL 2016).

- I like how Figure 1 clearly visualizes the main intuitions behind the proposed framework, and how mitigating the oversharing problem leads to better alignments. This was very nice.

- In light of the problems with silver standard MUSE datasets (see also the recent work of Kementchedjhieva, EMNLP 2019), I would suggest to run BLI experiments on more language pairs: e.g., there are BLI datasets of Glavas et al available for 28 language pairs.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper compares two approaches to learn cross-lingual word embeddings: cross-lingual alignment (where separately trained embeddings in different languages are mapped into a shared space) and joint training (which combines the training data in all languages and jointly learns a cross-lingual space). The authors argue that each approach has its own advantages, and propose a "unified framework" that essentially applies them sequentially (first train jointly, and then further align them after the necessary vocabulary reallocation).

I have generally positive feelings about the paper. To be honest, I do not like the way the authors frame their work (e.g. the way the method is motivated in Section 2.3 or calling it a "unified framework"), but the actual method they propose does make sense, the experimental evaluation is solid, and the results are generally convincing.


Strengths:

- Good coverage of related work, including recent publications.

- Thorough evaluation in 3 different tasks, much better than what is common in the area (usually limited to BLI).

- The authors experiment with contextual embeddings in addition to conventional word embeddings.

- Generally convincing results with relevant ablations and reasonable baselines.


Weaknesses:

- I feel that framing this as a "unified framework" for cross-lingual alignment and joint training is going a bit too far. The proposed method is as simple as first doing the joint training and then the cross-lingual alignment (with a special treatment for vocabulary reallocation). This is just a sequential application of two class of methods, and not what I would understand as a "unified framework" (which suggests some form of generalization or at least a closer interaction to me). At the same time, the only "joint training" that the authors explore is training regular embeddings over concatenated monolingual corpora, which as far as I know has only been explored by Lample et al. (2018), and I would not consider as a representative example of this family of methods.

- It is not clear how the different vocabulary spaces are handled in BLI. My understanding is that if the query is in the source vocabulary subset the retrieval is done over the target subset, and if it is in the shared subset it is the same query word that is returned, but this is not clear at all.

- I think that the issue of "oversharing" is magnified in the paper. I understand that this is directly connected to the reallocation step in the proposed method, and it of course makes sense to also map words that predominantly appear in a single language. However, in relation to the downstream tasks themselves, oversharing only seems relevant for BLI, where one needs to delimit the retrieval space and avoid returning the query word (which is of course its own nearest neighbor) unless it actually exists in the target language. This is connected to my previous point, and I think should be discussed in isolation.


Other minor things to improve:

- Please make Figure 2c an actual table instead of an image.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper compares to approaches to bilingual lexicon induction, one which relies on joint training and the other which relies on projecting two languages' representations into a shared space. It also shows which method performs better on which of three tasks (lexicon induction, NER and MT). The paper includes an impressive number of comparisons and tasks which illuminate these two popular methods, making this a useful contribution. It also compares contextualized to non-contextualized embeddings.

I have some questions:
-can you goldilocks the amount of sharing in an alignment method? Put a different way, how much is performance affected if you perform alignment w/variously sized sub-partitions of the seed dictionary? Can you find a core lexicon (perhaps most common words shared cross-lingually) that will provide a good-enough alignment to joint-align with? If you were very ambitious, you could try artificially vary the amount of lexicon disjointness across languages (for camera-ready) and report how much your results are affected by incomplete overlap in translation variants.
-for Table 1, do you have any ideas about why certain languages do better on eng--> them and others do better on them-->eng?
-do you have any analysis exploring what is shared? wrt this sentence "It also suggests detecting what to share is crucial to achieve better cross-lingual transfer."

Please address:
- I would like more intuitive motivation for (6).

Small notes: 
-Fig. 1 font is too small, you could also remove excess axes. There's also overlap between the labeled arrow and the y-axis label btw (a) and (b).
-"oversharing" should be typographically delimited as a definition in "We refer to this problem as oversharing"
-typos: bottom of p4: "Secition 2.3"
-"e.g." or "i.e." should be followed by a ","
-list which languages you use on bottom of p5
-Table 3 caption looks crazy, what happened to your spacing there?
review_title: Official Blind Review #3

Meta-review: Reviewer worries include: whether the approach scales to distant language pairs, overselling of the paper as a "framework", a few citations and comparisons missing. I agree and encourage the authors not to use the word "framework" here. I would also encourage the authors to evaluate on more interesting language pairs, and analyze what vocabularies are relocated, as well as what their method is better at compared to previous work. 
Decision: Accept (Poster)