Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposed a pre-training method for strengthening the contextual embeddings alignment. Given parallel sentences from a different language, the authors proposed to enforce corresponding words that have a similar representation by minimizing the squared error loss. The authors also proposed to use the an regulation that prevents the learned embedding from drift too far. The authors evaluated the proposed pre-training on the contextual alignment metric and show the BERT has variable accuracy depends on the language. The proposed method improved significantly on zero-shot XNLI compares to the base model. 

The paper is well written, and the proposed aligned loss makes sense and should augment the multi-lingual pre-training from a high level. The authors did a good job of analyzing the bert for multi-lingual. There some details may help the reader understand the paper better

1: Why use L2 distance as the metric function, what is the performance of using the inner product as a metric function? and what is the difference here? 

2: The authors mentioned the word pairs are extracted from the existing method which may be noisy. I wonder is there any ablations study with respect to how the word pairs affect the pretraining? 

3: When finetuning on zero-shot transfer, what is the finetune setting? Is there any strategy to avoid the lower layer embedding from drifting away? 

4: In table 3, the Fully supervised Base Bert on English is close to the zero-shot setting and the base BERT model is better than Alignment bert, I wonder can the authors explain more on this? 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper presents a new method to further align multilingual BERT by learning a transformation to minimize distances in a parallel corpus.

I think that this is overall a solid work. Although simple, the proposed method is well-motivated, and the reported results are generally convincing. However, I think that the paper lacks an appropriate comparison with similar methods in the literature, and the separation between the real evaluation in a downstream task (XNLI) and the analysis on a rather artificial contextual word retrieval task (which favors the proposed system) is not clear enough.

More concretely, these are the aspects that I think the paper could (and should) improve:

- You are not comparing to any baseline using parallel data with contextual embeddings. You should at least compare your method to Schuster et al. (2019) and/or Aldarmaki & Diab (2019), who further align multilingual BERT in a supervised manner as you do, as well as Lample and Conneau (2019), who propose an alternative method to leverage parallel data during the training of multilingual BERT. In fact, while you do improve over multilingual BERT, your results in XNLI are far from the current state-of-the-art, and this is not even mentioned in the paper.

- The "contextual word retrieval" task you propose is rather artificial and lacks any practical interest. It is not surprising that your proposed method is strong at it, as this is essentially how you train it (you are even using different subsets of the exact same corpus for train/test). The task is still interesting for analysis -which is in fact one of the main strengths of the paper- but it should be presented as such. Please consider restructuring your paper and moving all these results to the analysis section, where they really belong.

- I do not see the point of the "non-contextual word retrieval" task, when you are in fact using the context (the fact that there is only one occurrence per word type doesn't change that). This task is even more artificial than the "contextual word retrieval" one. Again, it can have some interest as part of the analysis (showing that the gap between aligned fasttext and aligned BERT goes down from table 1 to table 2), but presenting it as a separate task as if it had some value on its own looks wrong. From my point of view, the real "non-contextual word retrieval" task would be bilingual lexicon induction (i.e. dictionary induction), which is more interesting as a task (as the induced dictionaries can have practical applications) and has been widely studied in the literature.

- I really dislike the statement that contextual methods are "unequivocally better than non-contextual methods for multilingual tasks" on the basis of the non-contextual word retrieval results. If you want to make such a strong statement, you should at least show that your method is better than non-contextual ones in a task where the latter are known to be strong (i.e. bilingual lexicon induction, see above). However, your comparison is limited to a new task you introduce that clearly favors your own method, and in fact requires using the non-contextual methods in a non-standard way (concatenating the word embeddings with the avg/max/min sentence embeddings). Please either remove this statement or run a fair comparison in bilingual lexicon induction (and preferably do both).

- BERT works at the subword level but, from what I understand, your parallel corpus (both for train/test) is aligned at the word level. It is not clear at all how this mismatch in the tokenization is handled.


Minor details that did not influence my score:

- Calling "fully-supervised" to the "translate-train" system is misleading. Please simply call it "translate-train".

- I assume you want to refer to Figure 3 instead of Figure 2 in Section 5.2
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper conducts a series of experiments on the multilingual BERT model of Devlin et al., aiming to inject stronger bilingual knowledge into the model for improved 'Aligned BERT'. The knowledge originating from parallel (Europarl) data improves the model significantly as shown on tasks such as contextual and non-contextual word retrieval as well as in zero-shot XNLI task. The paper continues the line of work on cross-lingual contextualised word embeddings, and it brings several minor contributions, but overall I do not see it as a very inspiring piece of work, and it leaves open several very important questions, in particular its relationship to prior work and some potentially stronger baselines than the ones reported in the paper, plus more experiments with more distant language pairs.

I am not exactly sure that the comparison between 'Aligned BERT' and the main baseline 'Aligned fastText + sentence' is completely fair. 'Aligned BERT' uses more than 2M Europarl sentences to learn the alignment, while the standard alignment methods for learning cross-lingual word embeddings (see e.g. Ruder et al.'s survey) typically rely only on 5k translation pairs or even less pairs. There is a huge difference in the strength of the bilingual signal between 2M parallel sentences and, say, 2k, word translation pairs. 

The main goal of the paper is to improve alignment of the starting multilingual BERT model, but I wonder why the authors have not compared to a more suitable XLM baseline of Lample and Conneau (NeurIPS 2019; the paper has been on arXiv since January 2019) - the XLM model uses exactly the same resources as 'Aligned BERT': parallel sentences from Europarl, while the main baseline here uses only seed dictionaries to learn the mapping. Regarding the baselines, it is also not clear to me why the authors have not compared to previous work of Schuster et al. (2019) and Aldarmaki and Diab (2019) at least in tasks where the models can be directly compared (XNLI or non-contextual word retrieval). Also, another non-contextual model which is worth trying is a joint model which relies on parallel sentences (similar to Ormazabal et al., ACL-19).

For the 'Aligned fastText + sentence' baseline, it would be interesting to report numbers with another (hybrid) baseline model that combines aligned fastText vectors with sentence encodings produced by multilingual BERT or some other multilingual sentence encoder (such as LASER, see Schwenk et al., 2019). Simply taking min, max, and avg vectors over all the sentence words might not be the best way to encode the sentence, and I would like to see more experiments here.

The paper makes some claims on novelty which 1) partially overlap with prior work, or 2) it does not cite related work while it leans on its findings. For instance on Page 4, the authors claim that their "(...) alignment method departs from prior work, in which each non-English language is rotated to match the English embedding space through individual learned matrices." However, there is at least one previous paper (Heyman et al., NAACL 2019) which did the same thing as the authors and showed that departing from learning projections only to English leads to more robust multilingual embeddings. Further, also on Page 4, the authors discuss that the assumption on learning good rotation matrices relies on the assumption of rough/approximate isomorphism without citing a body of related work that actually investigated this assumption such as the work of Sogaard et al. (ACL 2018). Also, the paper should do a better job in Section 2 and cover "word vector alignment" in more detail (e.g., a good starting point might be Ruder et al.'s survey paper on cross-lingual word embeddings).

The assumption of rough/approximate isomorphism is problematic also for non-contextual cross-lingual embeddings in settings with more distant language pairs. The authors mention that it may not hold for 'contextual pre-trained models given their increased complexity'. This is very imprecise writing taking place imho: 1) it is not clear why it should not hold in the case of contextual pre-trained models (at least for similar languages). Are there any properties of the contextual models that invalidate that assumption? It is also not exactly shown why contextual pre-trained models have increased complexity compared to e.g. fastText. How does one measure that 'model complexity' in objective terms? In fact, the paper would contribute immensely from more precise writing: e.g., on Page 3 contextual alignment of the model f is defined as accuracy in contextual word retrieval. This reads as defining a critical concept or a task as an evaluation measure (that measures the success of that task). In Introduction, the paper aims to "better understand BERT’s multilingualism", but I do not see how it contributes to our better understanding of BERT's multilingualism besides a pretty straightforward claim that it shows less multilingual potential when doing experiments with Greek and Bulgarian that use different scripts. Figure 2 and Figure 3 also do not bring anything new - the paper seems to just state known facts without proposing new solutions on how to e.g. learn better alignments for Greek or Bulgarian.

One important analysis aspect is missing from the paper: there are no experiments with more distant language pairs (the most distant language pair is English-Greek). I would like to see more experiments in this space. Another experiment which would contribute to the paper is the analysis of the importance of parallel corpora size. How much does the model lose in its performance by shrinking the parallel corpus? We cannot expect having 2M sentences for so many language pairs, and, even if we do have the data, the paper does not convince me that I should use 'Aligned BERT' instead of e.g. the XLM model of Lample and Conneau.

Minor remarks:
As a variant of the contextual word retrieval, have the authors tested if a correct target language sentence can be retrieved only looking at the context of the source language word? This would provide some insight on the importance of modeling context via BERT versus via simple context averaging.

Regarding the analysis between closed-class and open-class words performance, the difference in performance can be due to mere frequency: closed-class word types are very scarce, but their corpus frequency is quite high which also leads to learning better representations in the first place, as well as better alignments later on.
review_title: Official Blind Review #3

Meta-review: This paper proposes a method to improve alignments of a multilingual contextual embedding model (e.g., multilingual BERT) using parallel corpora as an anchor. The authors show the benefit of their approach in a zero-shot XNLI experiment and present a word retrieval analysis to better understand multilingual BERT.

All reviewers agree that this is an interesting paper with valuable contributions. The authors and reviewers have been engaged in a thorough discussion during the rebuttal period and the revised paper has addressed most of the reviewers concerns.

I think this paper would be a good addition to ICLR so I recommend accepting this paper.
Decision: Accept (Poster)