Cross-Lingual Ability of Multilingual BERT: An Empirical Study
Authors: Karthikeyan K
Zihan Wang
Stephen Mayhew
Dan Roth
Venue:  (2020)
Keywords: Cross-Lingual Learning
Multilingual BERT

Abstract: Recent work has exhibited the surprising cross-lingual abilities of multilingual BERT (M-BERT) -- surprising since it is trained without any cross-lingual objective and with no aligned data. In this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross-lingual ability. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectives. The experimental study is done in the context of three typologically different languages -- Spanish, Hindi, and Russian -- and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project page: http://cogcomp.org/page/publication_view/900.

Published as a conference paper at ICLR 2020
CROSS-LINGUAL ABILITY OF MULTILINGUAL BERT: AN EMPIRICAL STUDY
Karthikeyan K
Zihan Wang'
Department of Computer Science and Engineering Indian Institute of Technology Kanpur Kanpur, Uttar Pradesh 208016, India kkarthi@cse iitk. ac.in
Department of Computer Science
University of Illinois Urbana-Champaige Urbana, IL 61801, USA zihanw2 www.eiiiiiiiiii.....miiimiiiiiee edu
Stephen Mayhew Duolingo Pittsuurh,, PA, 15206, USA
Dan Roth
Department of Computer and Information Science University of Pennsylvania
stephendduolingon com
Philadelphia, PA 19104, USA
danrothSseas upenn edu
ABSTRACT
Recent work has exhibited the surprising cross- cross-lingual abilities of multilingual BERT (M-BERT) surprising since it is trained without any cross-lingual objec- tive and with no aligned data. this work, we provide a comprehensive study of the contribution of different components in M-BERT to its cross- cross-lingual abil- ity. We study the impact of linguistic properties of the languages, the architecture of the model, and the learning objectíves. The experimental study is done in the context of three typologically different languages Spanish, Hindi, and Russian and using two conceptually different NLP tasks, textual entailment and named entity recognition. Among our key conclusions is the fact that the lexical overlap between languages plays a negligible role in the cross-lingun success, while the depth of the network is an integral part of it. All our models and implementations can be found on our project pago
INTRODUCTION
Embeddings of natural language text via unsupervised learning, coupled with sufficient supervised training data, have been ubiquitous in NLP in recent years and have shown success in a wide range of monolingual NLP tasks, mostly in English. Training models for other languages has been shown more difficult, and recent approaches relied on bilingual embeddings that allowed the transfer of supervision in high resource languages like English to models in lower resource languages; however, inducing these bilingual embeddings required some level of supervision (Upadhyay et al. 2016)).
Multilingual BERT (M-BERT),, transformer-based (Vaswani et al. 2017) language model trained on raw Wikipedia text of 104 languages suggests an entirely different approach. Not only is the model contextual, but its training also requires no supervision
alignment between the languages is done. Nevertheless, and despite being trained with no explicit cross-lingual objective, M-BERT produces representation that seems to generalize well across languages for variety of downstream tasks Wu & Dredze, 2019).
In this work, we attempt to develop an understanding of the success of M-BERT. We study range of aspects on two different NLP tasks, in order to identify the key components in the success of the model. Our study done in the context of only two languages, source (typically English) and target (multiple, quite different languages). By involving only pair of languages, we can study
Equual Contribution; most of this work was done while the author interned at the University of Pennsylvania. This work done while the author student the University of Pennsylvania.
cat on inationnvie///00 http:///ahiiooooooc/////aaaaaaaa
http: http:://ooccmp.or///aaeeee
qual....

---

Published as a conference paper at ICLR 2020
the performance on given target language, ensuring that it is influenced only by the cross-lingud transfer from the source language, without having to worry about a third language interfering.
We analyze the two-languages version of M-BERT (B- (B-BERT, for bilingual BERT, from now on) in three orthogonal dimensions: (i) linguistic properties and similarities of target and source languages; (ii) network architecture, and (iii) input and learning objective.
One hypothesis that has been discussed as a way to explain the success of M-BERT has to do with some level of language similarity. This could be lexical similarity (shared words or word- parts) or structural similarities (word-ordering or word-frequency), or both. Therefore, we begin by investigating the contribution of word-piece overlap the extent which the same word-pieces appear in both source and target languages and distinguish it from other similarities, which we group into a structural similarity between the source and target languages. Surprisingly, as we show, B-BERT is cross- conssllnguall even when there is absolutely no word-piece overlap. That is, other aspects of language similarity must be contributing to the cross-lingual capabilities of the model. This is contrary to the Pires et al. (2019) hypothesis that M-BERT gains its power from shared word-pieces. Furthermore, we show that the amount of word-piece overlap in B-BERT's 's training data contributes little to performance improvements.
Our study of the model architecture addresses the importance of (i) the network depth, (ii) the number of attention heads, and (iii) the total number of model parameters in B-BERT. Our results suggest that depth and the total number of parameters of B-BERT are crucial for both monolingual and cross-lingual performance, whereas multi-head attention is not significant factor. A single attention head B-BERT can already give satisfactory results.
To understand the role of the learning objective and the input representation, we study the effect of (i) the next sentence prediction objective, (ii) the language identifier in the training data, and (iii) the level of tokenization in the input representation (character, word-piece, word tokenization). Our results indicate that the next sentence prediction objective actually hurts the performance of the model while identifying the language in the input does not affect B-BERT's performance cross- lingually. Our experiments also show that character-level and word-level tokenization of the input results in significantly worse performance than word-piece level tokenization.
Overall, we provide an extensive set of experiments on three targetttareet language pairs, English- Spanish, English-Russian, Russian, and English- Hindii selected for variance of script and typological fea- tures. We evaluate the performance of B-BERT on two very different downstream tasks: cross- lingual Named Entity Recognition a sequence prediction task that requires only local context and cross-lingual Textual Entailment Dagan et al. (2013) that requires more global representation of the text.
Ours is not the first study of M-BERT. Wu & Dredze (2019) and Pires et al. (2019) identified the cross-lingual success of the model and tried to understand it. The former by considering M-BERT layer-wise, relating cross- lonssll performance with the amount of shared word-pieces and the latter by considering the model's ability to transfer between languages as a function of word order similar- ity in languages. However, both works treated M-BERT as black box, comparing performance on different languages. This work, on the other hand, examines how B-BERT performs cross-lingually by probing its components along multiple aspects.
We also note that some of the architectural conclusions have been observed earlier, if not inves- tigated, in other contexts. Liu al. (2019) and Yang al. (2019) argued that the next sentence prediction objective of BERT (the monolingual model) is not very useful; we show that this is the case in the roossllnngual setting. Voita et al. (2019) prunes attention heads for transformer based machine translation model and argues that most attention heads are not important; in this work, we show that the number of attention heads is not important in the cross-lingual setting.
Our contributions are threefold: (i) we provide the first extensive study of the aspects of M-BERT that give rise to its cross-lingual ability; (ii) we develop a methodology that facilitates the analysis of similarities between languages and their impact cross-lingual models; we do this by mapping En- glish to Fake-English language, that identical in all aspects to English but shares no word-pieces with any target language; finally, (iii) we develop a set of insights into B-BERT, along linguistic, architectural, and learning dimensions, that would contribute to further understanding and to the development of more advanced cross- lingual neural models.
2

---

Published as a conference paper at ICLR 2020
2 BACKGROUND
2.1 BERT
BERT (Devlin et 2019 is a Transformer-baser (Vaswani et al. 2017) pre-trained language representation model that has already seen wide use. In contrast to traditional language model objectives that predict the next word given context, BERT learns to predict the value of masked words (so called, Masked Language Modelling, or MLM (Taylor) (953)), as well as decide if two sentences are contiguous, known as next sentence prediction (NSP). Input to BERT A A and B, such that half of the time B comes after A in the original text and the rest of the time B is randomly sampled sentence. Some tokens from the input are randomly masked, and the MLM objective is to predict the masked tokens. Devlin al. 2019) argues that MLL enables a deep representation from both directions, and NSP helps understand the relationship between two sentences and can be beneficial representations.
pair of
sentences
After BERT is pre-trained on a massive amount of unlabeled text, the representations can then be used in downstream tasks. Typically, new task-specific layer added to the top of BERT, and all parameters are fine-tuned on the target task.
2.2 MULTILINGUAL BERT
Multilingual BERT is pre-trained in the same way as monolingual BERT except using Wikipedia text from the top 104 languages. To account for the differences in the size of Wikipedia, some languages are sub-sampled, and some are super-sampled using exponential smoothing (Devlin et al., 2018). It's worth mentioning that there are cross-lingual objectíves specifically designed nor any cross- cross-lingual data, e.g. parallel corpus, used.
3 WHY MULTILINGUAL BERT WORKS
We present analysis for the cross lonss-lingual ability of multilingual BERT (in our case B-BERT) in three dimensions: (i) linguistic properties and similarities of target and source languages; (ii) network architecture, and (iii) input and learning objective.
3.1 DATASETS AND EXPERIMENTAL SETUP
In this work, we conduct all our experiments on two conceptually different downstream tasks cross- lingual Textual Entailment (TE) and cross-lingual Named Entity Recognition (NER). TE measures natural language understanding (NLU) at sentence and sentence pair level, whereas NER measures NLU at token level. We use the Cross-lingual Natural Language Inference (XNLI) (Conneau al. 2018 dataset to evaluate cross- lons-l TE performance and LORELEI dataset (Stassel & Tracey
2016 for Cross-Lingus NER.
3.1.1
CROSS-LINGUAL NATURAL LANGUAGE INFERENCE (XNLI)
XNLI is standard cross-lingual textual entailment dataset that extends MultiNLI (Williams al., 2018 dataset by creating a new dev and test set and manually translating into 14 different languages. Each input consists of a premise and hypothesis pair, and the task is classify the relationship between premise and hypothesis into one of the three labels: entailment, contradiction, and neutral. While training, both premise and hypothesis are English, and while testing, both are in the target language. XNLI uses the same set of premises and hypotheses for all the language, making the comparison across languages possible.
3.1.2 CROSS-LINGUAL NAMED ENTITY RECOGNITION (NER)
Named Entity Recognition is the task of identifying and labeling text spans as named entities, such as people's names and locations. The NER dataset (Strassel & Tracey 2016) we use consists of news and social media text labeled by native speakers following same guideline in several languages, including English, Hindi, Spanish, and Russian. We subsample 80%. 10%. 10% of English NER
In the implementation, the input pair of segments, which contain multiple sentences
3

---

Published as a conference paper at ICLR 2020
data as training, development, and testing. We use the whole dataset of Hindi, Spanish, and Russian for testing purposes. The vocabulary size is fixed at 60000 and is estimated through the unigram language model in the SentencePiece library (Kudo 2018
3.1.3 NOTATION AND EXPERIMENTAL SETUP
We denote B-BERT trained language A and B as A-B, c.g., B-BERT trained on English (en) and Hindi (hi) as en-hi, similarly for Spanish (es) and Russian (ru). For pre-training, we subsample en, es, and ru Wikipedia to IGB and the entire Wikipedia for Hindi. Unless otherwise specified, for B-BERT training, we use a batch size of 32, a learning rate of 0.0001, and 2M training steps.
the same fine-tuning approach BERT uses in English and report accuracy. For NER, we extract BERT representations as features and fine-tune a Bi-LSTM CRF model (without character-wise emheddings) and report F1 score averaged from 5 runs and its standard deviation.
For XNLI,
3.2 LINGUISTIC PROPERTIES
Pires et al. (2019) hypothesizes that the cross-lingual ability of M-BERT arises because of the shared word-pieces between source and target languages. However, our experiments show that B-BERT is cross-lingual even when there is no word-piece overlap. Similarly, WW & Dredze (019) hypothe- sizes that source language should be selected such that it shares more word-pieces with the target language while our experiment suggests that structural similarity (e.g. word-ordering) is much more important. Motivated by the above hypotheses, in this section, we study the contribution of word- piece overlap and structural similarity for the cross-lingual ability of B-BERT.
3.2.1 WORD-PIECE OVERLAP
M-BERT is trained using Wikipedia text from 104 languages, and the texts from different languages share some common word-piece vocabulary (like numbers, links, etc.. including actual words, they have the same script), we refer to this as word-piece overlap. Previous work (Pires et al. 2019) hypothesizes that M-BERT generalizes across languages because these shared word-pieces force the other pord-pieces be mapped to the same shared space.
In this section, we perform experiments to compare cross- tross-lingual performance with and without word-piece overlap. We construct new corpus Fake- Bake--Enllish (enfake), by shifting the Unicode of each character in English Wikipedia text by large constant so that there is strictly no character overlap with any other Wikipedia text [1 We may consider Fake-English as different language than English, but having the exact same properties except word surface forms.
We measure the contribution of word-piece overlap as the drop in performance when the word- word-piece overlap is removed. From Table we can see B-BERT is cross-lingual even when there is word- piece overlap. We can also see that the contribution of word-piece overlap is very small, which is quite surprising and contradictory to prior hypotheses (Pires et al. 2019 Wu & Dredze 2019).
3.2.2
WORD-ORDERING SIMILARITY
Words are ordered differently between languages. For example, English has Subject-Verrs Verb-Object order, while Hindi has a Subject-Objedt Object--erb Verb order. We analyze whether similarity in how words are ordered affects learning cross-lingual transferability. We study the effect of word-ordering similarity by destroying the word-ordering structure through randomly permuting some percentage of words in sentences during pre- pre-training. We permute both source (Fake-Engliny permuting any one of them would also be sufficient). This way, the similarity of word-ordering is hidden from B-BERT. We quantify the amount of permutation by sampling 25%, 50%, 100% of the (H) pairs of word-pieces in a sentence with L becomes .p..... wp.). This way of shuffling by no means generates a uniform random distribution on permutations, but provides roughly good way to control the randomness. Note that for each
and target language (although
and swap each pair (e.g. WPi,..., wp;
Before feeding the text to BERT's transformer model, there is an additional vocabulary embedding step. so that each word-piece gets mapped to a certain integer (unrelated to its unicode). Therefore, BERT will not have access to the exact values of the unicode, and will not be able to learn directly from the linear shift of unicode we introduced.
4

---

Published as a conference paper at ICLR 2020
XNLI
NER
B-BERT
Train
Test
Accuracy
Word-pies Contribution
Fi-Score
enfake-es
en-es
enfake es
72.3
70.9
1.4
62.6 (+: 1.6)
61.9 (#:0.8)
en-hi enfake-hi
60.1
61.6 (+0.77)
0.5
enfake
59.6
62.9 (+0.7)
enfake-ru
et-rru enfake
ru
66.4
65.7
0.7
57.1" (10.9)
54.2 (+0.7)
en-enfake
enfake
enfake
78.0
(10.7)
0.5
en-enfake
enfake
77.5
76.6(=0.8)
The Effect of Word-piece Overlap and of Structural Similarity For different pairs of B-BERT languages, and for two tasks (XNLI and NER), we show the contribution of word-pieces the success of the model. In every two consecutive rows, we show results for pair (e.g.. English English-Spanish) and then for the corresponding pair after mapping English to disjoint set of word word-pieces. The between the performance in each group of two rows indicates the loss due to completely eliminating the word-piece contribution. We add asterisk the number for NER when the results statistically significant the 0.05 level.
Table 1:
word-piece, the other word- word-pieces that appear in its context (other word-pieces in the sentence) are not changed, although the order is changed. We also do not permute during fine-tuning, as we only want to control cross-lingual ability gained during pre-traine pre-training.
B-BERT
Permutation amount
XNLI (acc)
NER (Fi (Soore)
0.0
70.9
62.6 (+1.6) 42.3 (# 0.6)
Pre-train: enfake-es
0.25
68.9
Fine-tune:
0.5
65.5
41.3 (43.8)
Evaluate:
1.0
62.5
39.4 (d:1.5)
0,0
59,6
62.9 (+
Pre-train: enfake-hi
0.25
51,4
24.8 (42.0)
Fine-tune:
0.5
48.3
8.7 (+1.9)
Evaluate:
1.0
43.1
2.9 (+1.9)
0.0
65.7
54.2 (+0.7)
Pre-train:
enfake-ru
63.6
23.6 (+2.3)
Fine-tune:
Evaluate:
0.5
59.7
16.3 (+0.4)
1.0
53.6
16.3 (1009)
Table 2: Contribution of Word-Ordering similarity: We study the importance of word-order similarity by analysing the performance of XNLI and NER when some percent of word-order similarity is reduced. The percent p controls the amount of similarity (how random each sentence is permuted). We order similarity quite important, however there must be other components structural similarity that could
that word- contribute for the cross- moos-llnngull ability, as the performance of almost random still passable.
From Table 2 we can see that the performance drops significantly when we curtail the word-order similarity between the two languages. However, the cross-lingual performance is still significantly better than random, which indicates that there are other components of structural similarity, which could contribute to cross-lingual ability of B-BERT.
3.2.3 WORD-FREQUENCY SIMILARITY
We also study whether only knowing unigram word frequency allows for good cross-lingual repre- sentations. Indeed, Zipf's law indicates that words appear with different frequency, and one may suggest similar meaning words appear with relatively similar frequency in pair of languages, help- ing B-BERT learn cross-lingually. We collect the frequency of words in the target language and generate a new monolingual corpus by sampling words based on the frequency, i.c., each sentence is set of random words sampled from the original unigram frequency. The only information B-BERT learss from the languages is their unigram frequency, and perhaps sub-wor word level information and distribution of lengths of sentences. We train B-BERT using Fake-Englist and this newly generated target corpus. From Table 3 we can see that the performance very poor. Therefore, unigram frequencies alone don't contain enough information for cross-lingual learning.
5

---

Published as a conference paper at ICLR 2020
B-BERT
Frequency Based
XNLI (acc)
NER (Fi-Score)
No
70.9
62.6 (2:1.))))))
enfake-es
35.4
15.0(4: 1.1)
No
59.6
62.9 (#0.7)
enfake-hi
36.3
1.2 (4:0.9
No
65.7
(#0.7)
enfake-ru
34.4
5.7 (+0.5)
Table 3: Cross-lingual ability from only unigram frequency: We study if only unigram frequency useful for cross-lingual transfer. enfake-es indicates B-BERT trained with Fake-English and the new created corpus, where each sentence is a set of random words sampled from the same unigram distribution as cs. The results show that only unigram frequency is not enough for reasonable cross-lingual performance.
3.2.4 STRUCTURAL SIMILARITY
We define the structure of a language as every property of an individual language that is invariant to the script of the language (c.g.. morphology. word-ordering, word frequency are all parts of structure of a language). From Table [ we can see that BERT transfers very well from Fake-English to English. Also note that, despite not sharing any vocabulary, Fake-English transfers to Spanish, Hindi, Russian almost as well as English. On XNLI, where the scores between languages can be compared, the cross- lonsull transferability from Fake-English to Spanish is much better than from Fake-English to Hindi/Russian. Since they do not share any word-pieces, this better transferability comes from the structure similarity being closer between Spanish and Eige-sEnlish.
we know that between two languages, even when there overlap, nor any particular order of words in a sentences, B-BERT still learns some cross-lingun features. Section 3.2.3 shows from the other side that if B-BERT is given only little information (unigram frequency), it learns almost no cross-lingual features. These results suggest that we should shed more light studying the structural similarity between languages, for example, higher order of k-gram or c--oocccureence frequencies (notice that the word ordering experiment can be seen as giving B-BERT k- co-occurrence word frequencies where 1 is the length of the longest sentence in the corpus). In this study, we don't further dissect the structure of language. Despite its amorphous definition, our experiment clearly shows that structural similarity is crucial for cross-lingual transfer.
Through Section 3.2.2
word-piece
3.3 MODEL ARCHITECTURE
we observe that B-BERT recognizes language structure effectively. In this see- tion, we hypothesize this ability is an emergent property of the model architecture. We study the contribution of different components of B-BERT architecture namely (i) depth, (ii) head at- tention and (iii) the total number of parameters. The motivation is to understand which components are crucial for its cross- cross-lingual ability.
From Section 3.2
We perform all our cross-lingual experiments the XNLI dataset with Fake-English as the source and Russian as the target language; we measure cross- linsull ability by the difference between the performance of Fake- EakelSEglish and Russian (lesser the difference better the cross- rross-linuaal ability).
3.3. DEPTH
We presume the ability of B-BERT extract good semantic and structural features is a crucial reason for its cross-lingual effectiveness, and the deepness of B-BERT helps it extract good language features. In this section, we study the effect of depth on both the monolingual and cross-lingut performance of B-BERT. We fix the number of attention heads and change the size of hidden units and intermediate units such that the total number of parameters are almost the same (size of intermediate units is always 4x size of hidden units).
From Table 4 we can see that deeper models not only perform better on English but are also better cross-lingually. We can also see strong correlation between performance English and cross- lingual ability (A), which further supports our assumption that the ability to extract good semantic and structural features is crucial reason for its cross-lingual effectiveness.

---

Published as a conference paper at ICLR 2020
XNLI
Parameters (in Millions)
Depth
Multi-head Attention
Fake-Engliss
Russian
A
138.69
1
12
66.6
45.0
21.6
136.32
2
12
73.7
55.7
18.0
136.20
4
76.9
59.0
17.9
138.86
6
12
78.3
63.1
15.2
136.10
18
12
79.1
66.0
13.1
139.33
24
12
78.9
67.6
11.3
132.78
12
12
79.0
63.7
13.3
Table 4: The Effect of Depth of B-BERT Architecture: We use Fake-English and Russian B-BERT and study the effect of depth of B-BERT towards XNLI. We vary the depth and fix both the number of attention heads and the number of parameters the size of hidden and intermediate units changed that the total number parameters remains almost the same. We train only on Fake- Bikke-h and on both Fake- Bake-EEnlish and Russian and report their test accuracy. The difference between the performance on Fake EakglSh and Russian(A) is our measure of cross-lingual ability (lesser the difference, hetter the cross-lingual ability).
3.3.2 MULTI-HEAD ATTENTION
In this section, study the effect of multi-head attention the cross-lingual ability of B-BERT. We fix the depth and the total number of parameters - which is a function of depth and size of hidden and intermediate layers and study the performance for the different number of attention heads. From Table we can see that the number of attention heads doesn't have a significant effect on cross- lingual ability (A). B-BERT satisfactorily cross-lingual even with single attention head, which is in agreement with the recent study on monolingual BERT (Voita et al. 2019 Clark et al. 2019).
XNLI
Paramefers (in Millions)
Depth
Multi-head Attention
Fake-English
Russian
A
132.78
12
77.4
63.2
14.2
132.78
12
78.3
62.8
15.5
132.78
12
79.5
65.3
14.2
132.78
12
78.9
66.7
12.2
132.78
12
16
77.9
64.9
13.0
132.78
12
77.9
63.9
14.0
132.78
12
12
79.0
65.7
13.3
Table 5: The Effect of Multi-head Attention: We study the effect of the number of attention heads of B-BERT the performance of Fake-English and Russian language XNLI data. We fix both the number of depth and number parameters B-BERT vary the number attention heads. The difference between the performance on Fake-English and Russian (A) is our measure of cross-lingual ability.
3.3.3 TOTAL NUMBER OF PARAMETERS
Similar to depth, we also anticipate that large number of parameters could potentially help B-BERT extract good semantic and structural features. We study the effect of the total number of parameters cross-lingual performance by fixing the number of attention heads and depth; change the number of parameters by changing the size of hidden and intermediate units (size of in- termediate units is always 4x size of hidden units). From Table 6 of parameters is not as significant as depth: however, below a threshold, the number of parameters seems significant, which suggests that B-BERT requires certain minimum number parameters to extract good semantic and structural features.
that the total number
3.3.4 GENERALIZATION BERT WITH MORE LANGUAGES
Here we show that the results for model structure hold also for a more multilingual case; to further illustrate this, experiment four language BERT (en, hi, ru). From Table 7
that the performance on XNLI is comparable even with just 15% of parameters, and just or attention heads when the depth is good enough, which is in agreement with our observations in the main text.

---

Published as a conference paper at ICLR 2020
3.4 INPUT AND LEARNING OBJECTIVE
In this section, study the effect of input representation and learning objectives on cross-lingual ability of B-BERT. Recall that BERT is trained with Masked Language Modeling (MLM) and Next Sentence Prediction (NSP) objectives. We study the effect of NSP recent works (Conneau & 2019 Joshi et al. 2019 Liu et al. 2019) show that the NSP objective hurts the performance on several monolingual tasks. To verify whether B-BERT can learn language agnostic settings, we also study the use of language identity markers. We are also interested in studying the effect of tokenization and language representation, using characters and words vocabulary instead word- pieces.
2019
3.4.1 NEXT SENTENCE PREDICTION (NSP)
The input BERT is pair of sentences separated by special token such that half the time the second sentence is the next and rest half the time, it is a random sentence. The NSP objective of BERT (B-BERT) to predict whether the second sentence comes after the first one the original text. We study the effect of NSP objective by comparing the performance of B-BERT pre-trained with and without this objective. From Table X we can see that the NSP objective hurts the cross- lingual performance even more than monolingual performance.
XNLI
Parameters (in Millions)
Depth
Multi-head Attention
Fake- Bakk-EEglish
Russian
A
7.87
3
68.5
43.2
25.3
12.19
3
70.1
44.1
26.0
16.78
3
70.8
50.4
20.4
8.40
6
70.2
49.7
20.5
13.37
6
72.4
56.2
16.2
18.87
6
73.3
54.4
18.9
4.23
12
12
65.5
46.6
18.9
11.83
12
71.3
56.3
15.0
29.65
12
76.6
61.4
15.2
132.78
12
12
79.0
65.7
13.3
283.11
12
12
79.6
65.4
14.2
Table 6: The Efffet of Total Number of Parameters: We study the effect of the total number of parameters of B-BERT the performance of Fagge-Englihh and Russian language XNLI data. We fix both the number of depth and number of attention heads B-BERT and vary the total number of parameters by changing the size of hidden and intermediate units. The difference between the performance Fake- Lakee-Englihh and Russian (A) our measure of cross-ling cross-lingual ability.
XNLI
Parameters (in Millions)
Depth
Multi-head Attention
hi
132.78 (100%)
12
12
79.0
70.0
50.7
65.2
20.05 (15.1%)
16
1
74.1
63.2
49.6
58.9
20.05 (15.1%)
16
3
75.0
65.3
51.1
60.3
24.20 (18.23%%
24
1
75.0
64.5
48.3
58.6
24.20 (18.23%)
24
3
75.7
65.5
48.9
60.2
30.43 (22.92%)
36
1
75.2
66.5
47.1
60.3
30.43 (22.92%)
36
3
76.0
64.8
47.8
59.8
Table 7: Similar results on M-BERT with four languages: We show that the insights derived from bilingual BERT is also valid in the of multilingual BERT (4 language BERT). Further, also show that with enough depth, we only need fewer number of parameters and attention heads get comparable results.
8

---

Published as a conference paper at ICLR 2020
XNLI
NER
B-BERT
Train
Test
NSP
No-NSP
NSP
No-NSP
enfake
78.5
78.7
80.3 (1:0.6)
80.7 (1:1.4)
enfake-es
enfake
cs
70.9
72.7
62.6 1=1.
64.6 (#1,4)
enfake
79.3
80.1
81.4 (110.9)
80.0 (111.1)
enfake-hi
enfake
hi 59.6
60.7
62.9 (1.0.7)
62.4 (11,.4)
enfake
79.0
79.0
80.2 (1.0.7)
80.3 (1:0.8)
enfake-ru
enfake
65.7
66.7
54.2 (#:0.7)
55.7 (4.0.3
Table 8: Rffeet of Next Sentence Prediction Objective: We study the effect of NSP ohjective on XNLI and NER. Column NSP and No-NSP show the performance (accuracy for XNLI and average (stdev) Fi-score for NER) when B-BERT is trained with and without NSP objective respectively. The difference between the NSP No-NSP shows that objective hurts performance.
3.4.2 LANGUAGE IDENTITY MARKER
argue that B-BERT is cross-lingual because of its ability to recognize language structure similarities, and hence we presume adding a language identity marker doesn't affect its cross-lingual ability. Even if we don't add language identity marker, BERT learns language iden- tity (Wu Dredze 2019). To incorporate language identity in the input, we add different end of string tokens ([SEP) for different languages (i.e.. our input format is [CLS] SENTI [SEP-A] SENT2 [SEP-B], where A and B are languages corresponding to SENTI and SENT2 respectively). From that adding language identity marker doesn't It affect cross-lingual performance.
In Section 3.2
Table 9
XNLI
NER
B-BERT
Train
Test
No Lang-id
With Lang-id
No Lang-id
With Lang-id
enfake 78.5
78.4
80.3 (#0.6)
81.7 (#1.1)
enfake-es enfake cs
70.9
72.2
62.6(+1.6)
62.2 (+0.4)
enfake-hi enfake enfake enfake 79.0
79.3
79.0
81.4 (1:0.9)
80.7 (#116)
hi 59.6
59.6
62.9 (100.7
61.0 (100.7)
78.4
80,2 (1.0.7)
79.1 (1.1.)
enfake-ru
enfake
65.7
65.3
54.2 (110.7)
55.7 (1.0.6)
Table 9: Effect of Language Identity Marker in the Input: We study the effect of adding a language identifier in the input data. We use different end of string ([SEP]) tokens for different languages serving as language identity marker. Column "With Lang-id" and "No Lang-id" show the performance when B-BERT trained with and without language identity marker the imput.
3.4.3 CHARACTER VS. WORD-PIECE VS. WORD
We compare the performance of B-BERT with character, word-piece, and word tokenized input. For character B-BERT, we use all the characters as vocabulary, and for word B-BERT, we use the most frequent 100000 words as vocabulary. From Table 10 we can see that the cross-lingual performance (difference between source and target language performance) of B-BERT with word- piece tokenized similar that of word tokenized, while both are better than character tokenized. We believe that this is because word-pieces and words carry much more information than characters, allowing B-BERT to learn similarities between the two languages easier.
DISCUSSION AND FUTURE WORK
This paper provides a systematic empirical study addressing the cross-lingual ability B-BERT. The analysis covers three dimensions: (i) linguistic properties and similarities of target and source languages; (ii) network architecture, and (iii) input and learning objective.
In order to gauge the language similarity aspect needed make B-BERT successful, created new language Pake-Englishh and this allows us to study the effect of word-picce overlap while maintaining all other properties of the source language. Our experiments reveal some interesting and

---

Published as a conference paper at ICLR 2020
XNLI
NER
B-BERT
Train
Test
Char
WordPiece
Word
Char
WordPiece
Word
enfake-es enfake enfake
73.7
80.0
80.3
78.8 (1:1.3)
80.3 (=)) 64.8 (=0.9)
74,9 (1: 2.2) 57.5 (+0.4)
es
66.6
74.9
74.4
62.0 (=0.8)
enfake hi 53.8
73.9
80.3
80.0
79.6 (110.9)
79.7 (=1.1)
75.0 (1: 1.9)
enfake-hi enfake
61.7
60.3
53.1 (1.0.4)
58.8 (=1.2)
56.6 (1.0.8)
enfake enfake
74.2
80.7
79.2
77.2 (1:1.))
80.8 (= 1.3)
73.8 (#: 0.9)
enfake-ru
61.4
68.1
65.0
52.1 (40.5)
56.5 (.3.3)
46.4 (4:1.3
Table 10: Effect of Character Word-Piece vs Word tokenization. We compare the performance of B-BERT different tokenized input on XNLI NER data. Column Char Word-Piece, Word reports performance of B-BERT with character, word-piece and work tokenized input respectively. We use 2k batch size and 500k epochs.
surprising results. The most notable finding is that word-piece overlap the one hand, and multi- head attention on the other, are both not significant, whereas structural similarity and the depth of B-BERT are crucial for its cross-lingual ability.
While order to better control interference among languages, we studied the cross-lingual ability of B-BERT instead of those of M-BERT, it would be interesting now to extend this study, allowing for more interactions among languages. We leave it to future work to study these interactions. In particular, one important question
understand the extent which adding M-BERT languages that are related to the target language, helps the model's cross cross-lingus lingual ability.
We introduced the term Structural Similarity, despite its obscure definition, and show its significance in cross cross-lingual ability. Another interesting future work could be to develop a better definition and, consequently, finer set of experiments, better understand the Structural similarity and study its individual components.
Finally, we note an interesting observation made Table We observe a drastic drop in the en- tailment performance of B-BERT when the premise and hypothesis are in different languages. (This data was created using XNLI where in the original form the languages contain the same premise and hypothesis pair). One of the possible explanations could be that BERT is learning to make textual entailment decisions by matching words or phrases in the premise to those the hypothesis. This question, too, is left as future direction.
Premise Language Hypothesis Language (XNLI)
B-BERT Target
enfake-target
target-enfake
enfake-enfake
target-target
enfake-es cs
57.9
61.1
78.5
70.9
enfake-hi hi
45.7
55.6
79.3
59.6
enfake-ru
ru
51.1
57.9
79.0
65.7
Table 11: Premise and Hypothesis in different language: Using XNLI test set, construct textual tailment data with premise and hypothesis different languages. The column A-B (e.g. enfake-target) refers test data with premise language (enfake) and hypothesis language B (target). always train on Fake-English and report test accuracy.
ACKNOWLEDGMENTS
This work was supported by Contracts W911NF-15-1-046 HR0011-15-C-0113. and HR001 18- 2-0052 from the US Defense Advanced Research Projects Agency (DARPA), by Google Cloud, and by Cloud TPUs from Googles TensorFlow Research Cloud (TFRC). The views expressed are those of the authors and do not reflect the official policy position of the Department of Defense the U.S. Government or Google.
REFERENCES
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's attention. CoRR,
2019. URL/http: htl.ihtt::/aaxiv.orrr
10

---

Published as a conference paper at ICLR 2020
abs/1906. 04341
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Ro- man Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Confer- ence Neural Information Processing Systems 2019, NewrIPS 2019, 8-14 December 2019, n cc cc/paper/
Vancouver, BC, Canada, pp. 7057-7067, 2019. http:ht::///att mode pretraining
6928-cross- Lww_--innnaaagggaaaaa
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 10.18653/v 1/118- 1269. httptsss: /doi org 1ttp:1111816555 1ew.31861vvddll 1269
Association for Computational Linguistics, 2018. doi:
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzoto. Recognizing Textual Entail- ment: Models and Applications. Morgan and Claypool, 2013.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Multilingual bert
I, 2018. URL https:
multilingus mc
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 1.116633/11n19-1422 URL 1tt.////1165/////11111122
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. CoRR, Abss/197..10599 2019. URL
10529
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Com- putational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Pa- doi: 10.18653/v1/P18- 1007. URL https: www. aclweb orq/
pers. pp. 66-75, 2018
2athelogy/P18-10077
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.1 1692, 2019. URL htttttt///////l...ot org/ abs 907 11692
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? In Anna Korhonen, David R. Traum, and Lluis Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 4996-5001, Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19.11 1493. UtLhttts:://dlt org
wwwooorg//o..........ooo /00 1tt.:11..11666.1
Stephanie M. Strassel and Jennifer Tracey. LORELEI language packs: Data, tools, and resources for technology development in low resource languages. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asunción Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Tenth In- ternational Conference Language Resources and Evaluation LREC 2016, Portorož. Slove- nia, May 23-28, 2016. European Language Resources Association (ELRA), 2016. URL http: www. wrw.-rre-ie http://wwww.....ooeeeooeee///eee wwwp://nnnn.eeeeeeereeeee http://rrc2211//umrrreeeeee summaries /1138 html
Wilson Taylor. cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30
(4):415-433, 1953.
11

---

Published as a conference paper at ICLR 2020
Shyam Upadhyay, Manaal Faruqui, Chris Dyer, and Dan Roth. Cross-lingure models of word em- beddings: An empirical comparison. In Proc. of the Annual Meeting of the Association for Com- putational Linguistics (ACL), 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, UI- rike von Luxburg, Samy Bengio, Hanna M. Wallach. Rob Fergus. S. V. N. Vishwanathan, and Roman Gamntt (eds.), Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998-6008. 2017. URL httt:://pappeee n hipp..o. CC ww/pppppp.....ccc.pppcc.
71s1-attention is-issslaa all- you need
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest be pruned. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1; Long Papers, pp. 5797-5808, Association for Computational Linguistics, 2019. doi: 10.18653/ v1/p19-1580. URL http:///ooooooo///./oo/oo 10 11111.1166//11/p.... 1
Adina Williams, Nikita Nangia, and Samuel R. Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Marilyn A. Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the As- sociation for Computational Linguisties: Human Language Technologies, NAACL-ILT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume (Long Papers), pp. 1112-1122. As- doi: 10.18653/v1nn18- 1101. URL :
sociation for Computational Linguisties, 2018.
10..0.1653/ww//n/ 1110
Shijie Wu and Mark Dredze. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNIP-IJCNILP 2019, Hong Kong, China, Association for Computational Linguistics, 2019. doi: 10.
November 3-7, 2019, pp. 18653/v1/D19-11 1077. URL httpps//:://///ttt
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 5754-5764, 2019.
12


Reviews:
[{"review_id": "rJlUTFWI9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the task?\nComprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability.\n\nWhat has been done before?\n(Wu & Dredze, 2019) and (Pires et al., 2019) identified the cross-lingual success of the model and tried to understand it. However, both works treated M-BERT as a black box and compared M-BERT’s performance on different languages. This work, on the other hand, examines how B-BERT performs cross-lingually by probing its components, along multiple aspects. Some of the architectural conclusions have been observed earlier, if not investigated, in other contexts.\n\nAuthors claim that “Pires et al. (2019) hypothesizes that the cross-lingual ability of M-BERT arises because of the shared word-pieces between source and target languages.” is not entirely correct. Pires et al. (2019) did show M-BERT’s ability to transfer between languages that are written in different scripts, and thus have effectively zero lexical overlap. There were results (e.g. Figure 1) showing that while performance using EN-BERT depends directly on word piece overlap, M-BERT’s performance is largely independent of overlap, indicating that it learns multilingual representations deeper than simple vocabulary memorization.\n\nPires et al. (2019) also showed that structural similarity is crucial for cross-lingual transfer\n\nWhat are the main contributions of the paper?\nFirst comprehensive study of the contribution of different components in Multilingual BERT to its cross-lingual ability. Novel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT\nMethodology that facilitates the analysis of similarities between languages and their impact on cross-lingual models by mapping English to a Fake-English language, that is identical in all aspects to English but shares no word-pieces with any target language.\n\nWhat are the key dimensions studied/analyzed?\nDifferent components/aspects of Multilingual BERT investigated:\n(i) Linguistics properties and similarities of target and source languages  (has been studied in prior work)\n(ii) Network Architecture (novel)\n(iii) Input and Learning Objective (moderately novel)\n\nWhat are the main results? Are they significant?\nLexical overlap between languages plays a negligible role in the cross-lingual success, while the depth of the network is an important part of it.  \n\nStrengths \nNovel findings about the effect of network architecture, input representation and learning objective on cross lingual ability of M-BERT\nWeaknesses\nPires et al. (2019) work has been misrepresented. (see above for more details)\nPires et al. (2019) did  study linguistics properties and similarities of target and source languages for Multilingual BERT and had similar findings as this work.\n\n\nQuestions\nWhat kind of difference in the numbers is considered significant by the authors ? For example, according to them, an increase of more than 2 points in accuracy in Table 3 (e.g. 1 head vs. 6 heads) is considered insignificant. But a decrease of a point in accuracy in Table 5  (e.g. enfake-ru NSP vs. No-NSP) is significant.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Sklz-Z8JcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper evaluates the cross-lingual effectiveness of Multilingual BERT along three dimensions:\n- Linguistics\n- Architecture\n- Input and learning objective\n\nIn each of these three dimensions, the authors run experiments to test why BERT is effective at cross-lingual transfer.\nFor simplicity, they run their experiments on B-BERT which is trained on two languages. The fine-tuning is done on language, and the zero-shot performance is tested on the other one.\n\nPros:\nI found the en-fake experiments enlightening. The authors find that wordpiece overlap is not as important for cross-lingual transfer as was suggested by previous papers (Wu & Pires).\nThe idea of creating a unicode shifted version of English and use it for testing is a first of its kind and quite interesting.\nMost experiments were well motivated and the authors draw good conclusions about the need for more depth, that only a few attention heads are sufficient.\nThey end with an experiment that shows that the cross-lingual effectiveness drops significantly when the premise and hypothesis are in different languages. This is a good motivating experiment to end the paper on.\n\nCons:\n- The architecture experiments were not that insightful and they authors did not reach concrete suggestions on a minimum number of parameters or a minimum depth.\n- While the two language setting is easier to experiment with, I wonder how these conclusions will change if they were repeated with the 100+ language version.\n- For structural similarity, it would have been more concrete if the authors were able to visualize and show that the newly created en-fake still aligned with corresponding similar words in the other languages. This would have proven that despite no wordpiece overlap, similar words still align.\n\n\nMinor comments:\n- typo: \"training also training also\"\n- bad grammar - last para in page 1\n- I found the introduction was filled with grammar and bad English. Please fix.\n- Why are the numbers in Table 4 in a different format?\n- 3.4.2 It's not clear if this is a clear trend. The authors claim that lang-id helps.\n- Please explicitly state the sentencepiece or wordpiece setup in a central piece. I found the detail hidden in section 3.1.2\n- With the word vs char vs wordpiece experiments, I think more care should be taken to make sure that the number of parameters remains the same across all three setups. e.g. with only a few chars as vocab, the model has far fewer parameters. This should be compensated for.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xvDADRYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nCONTRIBUTIONS:\n\nC1. Cross-linguistic token overlap. Fake-English: English, but with the Unicode codes of English characters all shifted by a large constant so that there is no overlap between Fake-English characters and those of actual languages, but the language-internal structure remains that of English. \n\nC2. A bilingually-trained BERT, pretrained on languages L and L’, is then trained on a downstream task in L then tested on that task in L’. The task is Cross-Lingual NLI (XNLI) or Cross-Lingual NER. L’ is Spanish, Hindi, or Russian. L is English or Fake-English. Comparing the success at test when L = English vs. when L = Fake-English, it is shown that eliminating all token overlap between L and L’ has a small effect (less than 1.5% on XNLI, less than about 3.5% on NER). (Table 1)\n\nC3. Several architectural parameters of BERT are varied holding the others roughly constant (same tasks as C2, with L = Fake-English). This shows that depth (Table 2) and level of tokenization matter (Table 7), while little effect results from varying the number of attention heads (Table 3), number of parameters (Table 4), whether the next sentence prediction task is used for training (Table 5), or whether the language of an input is explicitly given (Table 6).\n\nC4. Testing cross-language entailment on XNLI by B-BERT shows that there is a large reduction in performance when the hypothesis and premises sentences are from different languages. \n\nRATING: Weak reject\n\nREASONS FOR RATING (SUMMARY). Aside from the invention of Fake-English, which as far as I know is original and a clever approach to assessing the importance of token overlap in cross-language transfer, the other contributions are reporting results of mechanical changes. The paper’s contributions are useful, but do not reach a level of generality, originality, or depth justifying presentation at ICLR.\n\nAlthough it did not factor into my rating, I would like to point out that saying ‘structural similarity is important’ and saying ‘word-piece overlap is not important’ is saying exactly the same thing twice, since the gain not attributable to word-piece overlap, by their definition, equals the gain due to ‘structural similarity’, which is a concept otherwise undefined and unmeasurable. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a set of new analysis methods to try to better understand the reasons that multilingual BERT succeeds. The findings substantially bolster the hypothesis behind the original multilingual BERT work: that this kind of model discovers and uses substantial structural and semantic correspondences between languages in a fully unsupervised setting. This is a remarkable result with serious implications for representation learning work more broadly.

All three reviewers saw ways in which the paper could be expanded or improved, and one reviewer argued that the novelty and scope of the paper are below the standard for ICLR. However, I am inclined to side with the two more confident reviewers and argue for acceptance. I don't see any substantive reasons to reject the paper, the methods are novel and appropriate (even in light of the prior work that exists on this question), and the results are surprising and relevant a high-profile ongoing discussion in the literature on representation learning for language.
Decision: Accept (Poster)