Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In this work, the authors propose a way to transfer a pre-trained English BERT model to a new language within a short amount of time. The key insight is to map English embeddings to the foreign language and have separate embeddings for both English and the foreign language. The resulting bilingual LM is evaluated for zero-shot transfer learning on two tasks: XNLI and dependency parsing.

Pros:
- The authors provide good details into their hyperparameter settings and about how the obtain the foreign language word embeddings.
- By leveraging existing pre-trained models, they’re able to do pre-training for their bilingual LM within 2 days.

Cons:
I find that a key comparison point in this paper is missing, which is Bilingual BERT trained on just the two languages that are being considered for their RAMEN system. This is not a fair comparison while mBERT which is trained on 100+ languages is not.
All comparisons are not fair since a simple baseline of just training mBERT on two languages with monolingual data and with a shared WPM is not evaluated here.
The proposed system has an unfair advantage over mBERT since it’s initialized from BERT/RoBERTA and fine-tuned only on two languages. Hence most of the parameters are used for just the two languages while mBERT uses the parameters for 104 languages.
Given this unfair comparison, I’m not sure if we can draw a meaningful conclusion from all the experiments. 

Rating justification:
Given the lack a fair comparison between the bilingual and multilingual BERT models, I don't think the conclusions are insightful.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper presents a method to efficiently transfer pre-trained english language model to bilingual language model. The obtained representations are evaluated on downstream NLP task (natural language inference and dependency parsing) with state-of-the-art performances.


Pros:

- Experiments clearly show that, using the proposed method, stronger pre-trained English embedding leads to stronger bilingual language model and thus to better performances for downstream foreign tasks.

Cons: 

While it is generally  intelligible, some structural modifications could be done to  improved the clarity of the paper. For instance, the method used to align foreign word vectors with English word vectors, when no aligned corpus is available, should appear sooner. It is described in 3.1 but should probably appear in 2.1 subsection Learning from Monolingual Corpus.

Minor issues:

- in section 3: RoBERA -> RoBERTa
- in section 5.1: the third sentence is syntactically incorrect
- in Conclusion: our approach produces better than -> our approach performs better than

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method to adapt a pretrained BERT model from English to another languages with a limited time/GPU budget. Evaluation on 6 target languages shows good performance for natural language inference and dependency parsing. 

Concretely, the proposed approach consists of, starting from a pretrained English language model, first training language-specific embeddings and then fine-tuning the entire pretrained model on English *and* the target language, using those embeddings. The language-specific embeddings are initialized based on the English embeddings (the authors propose two different ways for doing that).

I like about the paper that the approach is simple and fast. The experiments seem reasonable, too. The only minor negative point is that the approach is not particularly exciting.
review_title: Official Blind Review #2

Meta-review: This paper proposes a method to transfer a pretrained language model in one language (English) to a new language. The method first learns word embeddings for the new language while keeping the the body of the English model fixed, and further refines it in a fine-tuning procedure as a bilingual model. Experiments on XNLI and dependency parsing demonstrate the benefit of the proposed approach.

R3 pointed out that the paper is missing an important baseline, which is a bilingual BERT model. The authors acknowledged this in their rebuttal and ran a preliminary experiment to obtain a first set of results. However, since the main claim of the paper depends on this new experiment, which was not finished by the end of the rebuttal period, it is difficult to accept the paper in its current state. In an internal discussion, R1 also agreed that this baseline is critical to support the paper.

As a result, I recommend to reject this paper for ICLR. I encourage the authors to update their paper with the new experiment for submission to future conferences (given consistent results).
Decision: Reject