Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: What is the task?
Comprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability.

What has been done before?
(Wu & Dredze, 2019) and (Pires et al., 2019) identified the cross-lingual success of the model and tried to understand it. However, both works treated M-BERT as a black box and compared M-BERT’s performance on different languages. This work, on the other hand, examines how B-BERT performs cross-lingually by probing its components, along multiple aspects. Some of the architectural conclusions have been observed earlier, if not investigated, in other contexts.

Authors claim that “Pires et al. (2019) hypothesizes that the cross-lingual ability of M-BERT arises because of the shared word-pieces between source and target languages.” is not entirely correct. Pires et al. (2019) did show M-BERT’s ability to transfer between languages that are written in different scripts, and thus have effectively zero lexical overlap. There were results (e.g. Figure 1) showing that while performance using EN-BERT depends directly on word piece overlap, M-BERT’s performance is largely independent of overlap, indicating that it learns multilingual representations deeper than simple vocabulary memorization.

Pires et al. (2019) also showed that structural similarity is crucial for cross-lingual transfer

What are the main contributions of the paper?
First comprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability. Novel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT
Methodology that facilitates the analysis of similarities between languages and their impact on cross-lingual models by mapping English to a Fake-English language, that is identical in all aspects to English but shares no word-pieces with any target language.

What are the key dimensions studied/analyzed?
Different components/aspects of Multilingual BERT investigated:
(i) Linguistics properties and similarities of target and source languages  (has been studied in prior work)
(ii) Network Architecture (novel)
(iii) Input and Learning Objective (moderately novel)

What are the main results? Are they significant?
Lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an important part of it.  

Strengths 
Novel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT
Weaknesses
Pires et al. (2019) work has been misrepresented. (see above for more details)
Pires et al. (2019) did  study linguistics properties and similarities of target and source languages for Multilingual BERT and had similar findings as this work.


Questions
What kind of difference in the numbers is considered significant by the authors ? For example, according to them, an increase of more than 2 points in accuracy in Table 3 (e.g. 1 head vs. 6 heads) is considered insignificant. But a decrease of a point in accuracy in Table 5  (e.g. enfake-ru NSP vs. No-NSP) is significant.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper evaluates the cross-lingual effectiveness of Multilingual BERT along three dimensions:
- Linguistics
- Architecture
- Input and learning objective

In each of these three dimensions, the authors run experiments to test why BERT is effective at cross-lingual transfer.
For simplicity, they run their experiments on B-BERT which is trained on two languages. The fine-tuning is done on language, and the zero-shot performance is tested on the other one.

Pros:
I found the en-fake experiments enlightening. The authors find that wordpiece overlap is not as important for cross-lingual transfer as was suggested by previous papers (Wu & Pires).
The idea of creating a unicode shifted version of English and use it for testing is a first of its kind and quite interesting.
Most experiments were well motivated and the authors draw good conclusions about the need for more depth, that only a few attention heads are sufficient.
They end with an experiment that shows that the cross-lingual effectiveness drops significantly when the premise and hypothesis are in different languages. This is a good motivating experiment to end the paper on.

Cons:
- The architecture experiments were not that insightful and they authors did not reach concrete suggestions on a minimum number of parameters or a minimum depth.
- While the two language setting is easier to experiment with, I wonder how these conclusions will change if they were repeated with the 100+ language version.
- For structural similarity, it would have been more concrete if the authors were able to visualize and show that the newly created en-fake still aligned with corresponding similar words in the other languages. This would have proven that despite no wordpiece overlap, similar words still align.


Minor comments:
- typo: "training also training also"
- bad grammar - last para in page 1
- I found the introduction was filled with grammar and bad English. Please fix.
- Why are the numbers in Table 4 in a different format?
- 3.4.2 It's not clear if this is a clear trend. The authors claim that lang-id helps.
- Please explicitly state the sentencepiece or wordpiece setup in a central piece. I found the detail hidden in section 3.1.2
- With the word vs char vs wordpiece experiments, I think more care should be taken to make sure that the number of parameters remains the same across all three setups. e.g. with only a few chars as vocab, the model has far fewer parameters. This should be compensated for.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
CONTRIBUTIONS:

C1. Cross-linguistic token overlap. Fake-English: English, but with the Unicode codes of English characters all shifted by a large constant so that there is no overlap between Fake-English characters and those of actual languages, but the language-internal structure remains that of English. 

C2. A bilingually-trained BERT, pretrained on languages L and L’, is then trained on a downstream task in L then tested on that task in L’. The task is Cross-Lingual NLI (XNLI) or Cross-Lingual NER. L’ is Spanish, Hindi, or Russian. L is English or Fake-English. Comparing the success at test when L = English vs. when L = Fake-English, it is shown that eliminating all token overlap between L and L’ has a small effect (less than 1.5% on XNLI, less than about 3.5% on NER). (Table 1)

C3. Several architectural parameters of BERT are varied holding the others roughly constant (same tasks as C2, with L = Fake-English). This shows that depth (Table 2) and level of tokenization matter (Table 7), while little effect results from varying the number of attention heads (Table 3), number of parameters (Table 4), whether the next sentence prediction task is used for training (Table 5), or whether the language of an input is explicitly given (Table 6).

C4. Testing cross-language entailment on XNLI by B-BERT shows that there is a large reduction in performance when the hypothesis and premises sentences are from different languages. 

RATING: Weak reject

REASONS FOR RATING (SUMMARY). Aside from the invention of Fake-English, which as far as I know is original and a clever approach to assessing the importance of token overlap in cross-language transfer, the other contributions are reporting results of mechanical changes. The paper’s contributions are useful, but do not reach a level of generality, originality, or depth justifying presentation at ICLR.

Although it did not factor into my rating, I would like to point out that saying ‘structural similarity is important’ and saying ‘word-piece overlap is not important’ is saying exactly the same thing twice, since the gain not attributable to word-piece overlap, by their definition, equals the gain due to ‘structural similarity’, which is a concept otherwise undefined and unmeasurable. 
review_title: Official Blind Review #2

Meta-review: This paper introduces a set of new analysis methods to try to better understand the reasons that multilingual BERT succeeds. The findings substantially bolster the hypothesis behind the original multilingual BERT work: that this kind of model discovers and uses substantial structural and semantic correspondences between languages in a fully unsupervised setting. This is a remarkable result with serious implications for representation learning work more broadly.

All three reviewers saw ways in which the paper could be expanded or improved, and one reviewer argued that the novelty and scope of the paper are below the standard for ICLR. However, I am inclined to side with the two more confident reviewers and argue for acceptance. I don't see any substantive reasons to reject the paper, the methods are novel and appropriate (even in light of the prior work that exists on this question), and the results are surprising and relevant a high-profile ongoing discussion in the literature on representation learning for language.
Decision: Accept (Poster)