Language-independent Cross-lingual Contextual Representations
Authors: Xiao Zhang
Song Wang
Dejing Dou
Xien Liu
Thien Huu Nguyen
Ji Wu
Venue:  (2020)
Keywords: contextual representation
cross-lingual
transfer learning

Abstract: Contextual representation models like BERT have achieved state-of-the-art performance on a diverse range of NLP tasks. We propose a cross-lingual contextual representation model that generates language-independent contextual representations. This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models, on top of contextual representation models like BERT. We provide a formulation of language-independent cross-lingual contextual representation based on mono-lingual representations. Our formulation takes three steps to align sequences of vectors: transform, extract, and reorder. We present a detailed discussion about the process of learning cross-lingual contextual representations, also about the performance in cross-lingual transfer learning and its implications.

Under review as a conference paper at ICLR 2020
LANGUAGE-INDEPENDENT CROSS-LINGUAL CONTEX- TUAL REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Contextual representation models like BERT have achieved state-of-the-art per- formance on a diverse range NLP tasks. We propose cross-lingual contextual representation model that generates language-independent contextual representa- tions. This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models, top of contextual representation models like BERT. We provide for- mulation of Iinguag--indeennde mono-lingual representations. Our formulation takes three steps align se- quences of vectors: transform, extract, and reorder. We present a detailed discus- sion about the process of learning cross-lingual contextual representations, also about the performance in cross-lingual transfer learning and its implications.
cross-lingual contextual representation based
INTRODUCTION
cross-lingual text representation can generate ingguage-indpeeneen representations: for pieces of text that are semantically similar, the produced representations are also similar, regardless of the language of the text. This provides a basis for dealing with multiple languages NL.. Training model cross- cooss-lnngull representations is a common approach for cross-lingual transfer learning
(Haag et al. 2013 Artetxe & Schwenk 2019).
To achieve desirable transfer learning performance, the cross-lingual representation model should be effective two aspects: 1) the quality of text representation; and 2) the quality of cross-lingual mapping.
The current state-of-the-are text representation models are a family of pre-trained language model representations such BERT (Devlin et al. 2019) and XLNet (Yang al. 2019). These models produce contextual representations of text, Le.,, a sequence of vectors corresponding to the sequence of text tokens. Each vector represents "a token in context," e.g., the semantics of word in its local context and in the global context of the sentence.
Research cross-lingual representation has far been largely concentrated word embeddings (Mikolov et al. 2013 Gouws et al., 2015 Ruder et al. 2017) and sentence embeddings (Schwenk) & Douze 2017 Artetxe Schwenk 2019). Word embeddings are static representations and less expressive than contextual representations. Sentence embeddings perform respectably in some tasks but inferior in others, because there is information loss when too much information crammed into a single vector (Conneau et al. (018a). However, word embeddings and sentence embeddings are relatively easy to generalize to cross-lingual representations because they do not have dynamic inter-dependency among vectors.
We regard as an interesting topic to investigate cross-lingual contextual representations. First, an interesting problem in its own to match two (or more) sequences of vectors, where each vector sequence has its order and locality relationships. Second, we hope to further improve cross-lingual transfer learning by leveraging the state-of-the-an text representation models.
In this study, we try to clarify the definition and characteristics of cross- coss-lingual contextual represen- tations, and propose formulation to lear lingulin contextual representations model based on existing mono- lingull pre-trained language models. We perform experiments to analyze the effect of the ingredients in formulation and the final performance in transfer learning.

---

Under review as a conference paper at ICLR 2020
We also hope investigation help understand the nature of text representations in different lan- guages: what are the commonalities between representations, how to cover the gaps, and ultimately what is transferable across languages with the help of state-of-the-an representation models.
Related work Multilingual BERT learns a mulli--lnguua representation by training on combined corpus of multiple languages. For dissimilar languages, the transfer performance is not satisfactory (Pires et al. 2019). XLM {Lample & Conneau, 2019) uses parallel corpus and translational ob- jective, and has achieved good transfer performance for many languages. However, XLM does not produce the same representation for different languages, there is guarantee of the performance in transfer learning. In multi-way neural machine translation (Firat ct al. 2016) the situation is simi- lar: multiple encoders produce the representation for each language, but because the representations are not Language-indepente
the decoder still needs to be trained on multiple language inputs.
A recent work aligning contextual representations is (Schuster et al.) 2019), which treats vectors in contextual representations as individual embeddings and learns a matrix to transform the embed- dings space of different languages. The idea resembles the "transform" step in our formulation.
CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS
First we define
cross-lingual contextual representations (CLCR)" and estab- lish mathematical notations that we use in the following discussion.
2.1 DEFINITION
A cross-lingual contextual representation (CLCR) of text is a parameterized representation model f()) (ao: ..., az) + (ro...,,TN) where ag 7r is the sequence of text tokens in sentence and ro...., PN is the sequence of vectors produced by f as the representation of s. A language- independent f satisfies the following conditions:
1. (cross-lingual correspondence) if sentence s in i-th language and sentence s in j- th language are translations of each other, then by some distance metric D we have < 6, where ó a threshold.
((f(s';)) ((); (($;;
2. (isometric property) if for some semantic metric S, sentence st in i-th language and sen- tence ş in j-th language satisfy S(s', s)) < % i.., sf and s are close in meaning, then by some distance metric we have D(f (s' (s):)), f(s); 0)) 8(7). where 8 is a monotonic increasing function of
3. (contextual representation) 3f1, fe, f3 s.t. r, = fa(fa(a)). (v)).. The vector can roughly be interpreted as representation of token a; the context of the whole sentence.
Condition 1 and 2 ensure that CLCR produces similar representations for sentences with similar meaning, independent of language. Condition 3 is a characteristic of contextual representation mod- els such as pre-trained language models like BERT.
2.2 EXISTENCE OF CLCR
To show that such cross- cross-lingual contextual representation exists, provide formulation of CLCR based on mono-lingual contextual representations. We propose that one can generate CLCR with a three-step process. Starting from mono-lingual representations, each of the three steps corresponds to one problem that needs to be solved, in order to arrive at a cross-lingual contextual representation.
Alignment: the vectors of mono-lingual representation of different languages reside in different spaces. First align the vector spaces R with transformations T, r' + r to transform into a common space RC.
2. Granularity: different languages have different token granularities. Some language use several tokens to represent word. Token aligmment between translations s; and s, can be represented by set A {((a),..... ..., di),, (a/ az))}, where tokens (ai...., af) and (at....,o) at)) are aligned "phrases."
2

---

Under review as a conference paper at ICLR 2020
Definition of Finest Common Granularity: find the token alignment A that not be broken into a finer alignment. The finest common granularity for s' w.t.. language and j is defined as grouping of the token sequence of s in A FC)((') = {(ag...., nno. (#ng+1 "n,), (amm+1:- at)}. where each element (oi,+1) )
In nutshell, we want to group a series of tokens (1 ox into a "phrase token" (key-point token), so that the key-point tokens have one-to-one correspondence between two languages. With key-point tokens derived from FCG, translations 8; and 8j have the same number of key-point tokens, and can then be represented by the same number of vectors.
Order: many different word orders exist in different languages, example the SVO (subject-verb-object and the SOV (subject-object-verh) order. We define an order map- ping to map the orders token from language to language j: Ri-+ : m|s' + w, where (( tin..... "m (a +1: a # )) E A. The order mapping Ri-j maps the position of a key-point token in the source sentence to the position of the corresponding key-point token in the target language.
To generate cross- looss-lingual representations, we perform alignment transformation on the vectors of two mono-lingut lono-lingual representations, find their common granularity, and then reorder the vectors, to make the two representations similar.
2.3 TRANSFER LEARNING WITH CLCR
Lunguage-independdee cross-lingual representations enables zero-shot transfer learning across lan- guages. CLCR allows the most versatile models to be trained on cross-lingual representations. For model M taking contextual representations (Po, Px) as input, M be trained dataset {(s', {)} in language i, by first using CLCR f to encode inputs: M(f(s") + I. In testing, model can perform inference on input in any language within f's input languages.
LEARNING CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS
In this section, we detail our approach to learn CLCR based on the formulation in Section 2.2 We use two mono-lingual contextual representation models on language and learn lross-lingual contextual representation model.
3.1 TRANSFORM
In the first step, we learn a transformation T parameterized by a multi-layer network:
Trr) Linear (r) + Linear (Reiear(Reeu(Liiii Linear(r))
The network is trained to minimize the discrepancy between the transformed representation in lan-
guage and the representation language D(T(r) ..., rhe l (rō ... rr)). )). Because we do not have alignment between the two sequences of vectors yet, we define D with attention matching between the two sequences:
D(T(r)...,, 1 x2)) Σ Σ IIT(Aeentioo(() I
- r1ll
where Atlention is defined using r² query perform weighted-sum of (rt, ..., r:).
To train transformation T, we minimize the above loss D on a parallel corpus {(* { s2).. We observed that straightforward gradient descent does not converge. We therefore used attention nealing during training to help our model converge to an optimal matching:
aa/TT
so sa.. ..., aN) E; e
where = 1/(1 + B step). As training progresses softmaz will gradually be replaced by mar, which let the model find one-to-one matching between the tokens. This is exactly required by the next step.
3

---

Under review as a conference paper at ICLR 2020
3.2 EXTRACT
In this step, we want to extract "key-points" from a sentence. These key-points are summarizations of local information (which may consist of multiple tokens), and the granularity of "key-points" is ideally consistent across different languages.
We extract key-points based on finest common granularity (FCG) between two languages: for group of tokens (i .... a ) E FCG(s*), we use a single vector Pno to represent them. The number of vectors in the CLCR of s' is thus equal to |FCG(s*))-
Next, we train a network
predict key-points in FCG(s"). Although we can generate FCG(s') by aligning parallel documents, in general CLCR should not rely on parallel documents and should predict key-points in single sentence independently:
rk is the representation for key-point token (ann +1. ann)
(4)
3! d E {a d J, P(af)
(5)
The prediction network predicts one the tokens in key-point token be the "pivot token." And the vector for that pivot token is regarded as the representation for the whole key-point token.
Finally, we need to generate ground-truth labels for training key-point prediction. In the definition above, the question "which word is the pivot token in a phrase?" do not have definite answers. Therefore, we train our representation model and let the model determine the pivot token in a key- point token. This is achieved by forced alignment via annealing in Section 3.1 After training the transformations T, we calculate the pairwise distance between v. and r. D(T(r))) + ((T()))ll?)) and use threshold to find aligned pivot tokens.
Aligned pivot tokens (which is at the same time aligned key-point tokens) generated in this fashion closely resembles key-point tokens derived from FCG. As we do not have ground-trus word align- ment of parallel corpus to generate real FCGs, we use our aligned key-points as neeudo-groundtrung alignments and use them as labels to train predictors in this and the next section.
One caveat of using parallel corpus is that translations are not unique. There can be multiple valid translations of sentence. Thus labels derived from parallel corpus are intrinsically noisy and non- unique. We generate more reliable labels to train our prediction network, with a technique similar to (Liu & Tao 2015).
3.3
REORDER
The aim of reordering is to make the order of contextual vectors (ro, rk) in language compat- ible with the order in language 2. This can be achieved by learning order prediction model, to predict the order mapping Ri R1- We propose two methods for order prediction:
Absolute order prediction: learn prediction model to directly predict the position of token a in language 2:
P(at) R1 R1-12(i)
(6)
Inference:
minimize m.E2"
Σ cost((((s)) = m..) s.l. my # my Vi,j
(7)
Relative order prediction: learn a model to predict the relative order two tokens af and af in language 2:
P(a a²) sgn( - Ri Ri-)2(i))
(8)
Inference:
minimize m
ΣΣ cost (ost(P(ai,ou a,) - ssg(mm------- mm)) s.t. my + my Vi 12
(9)
The learned order prediction is R1-12 ils m, The vectors in language can be reordered according to RI-22, so that the reordered vectors have the order of language 2. The reordered vector sequence is (r1 ( m. rl (N))) where R2-,1 is the reverse mapping of R1-22-
4

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
In this section, we perform experiments to learn langunge-inddeenddet cross-lingual contextual rep- resentation based on the approach in Section 3 We also perform transfer learning and analysis experiments to examine the characteristics of CLCR.
4.1 LEARNING
We sequentially examine the details and the effectiveness of each of the three steps in our formula- tion generate CLCR: transform, extract, and reorder.
The basic experiment setup is: we use mono-lingual pre-trained language models from Google BERT (Devlin et al. ) for English and Chinesq The 12-layer BERT-base model is used. We use parallel corpus from The United Nations Parallel Corpus v1.0 (Ziemski et al. (016)). Experi- ments are performed with Pytorch (Paszke et al. 2017).
Transform As shown in Table ! training transformation T to minimize distance D with atten- tion annealing managed to converge to a low distance between the two representations. Lower distance indicates that better alignment found (because unaligned and misaligned tokens always contribute to high distance). And good alignment is also conducive to learning better transfor- mation.
Table I: Distance D achieved by transformation T
Model
D(T(s)))(s²)
D(T(s²)
Linear Regression
0.055
0.078
Attention (w/o annealing 1)
0.138
0.207
Attention (with annealing 0.001)
0.028
0.048
Attention (with annealing 8 = 0.01)
0.025
0.037
During training, we alternately optimize the representation model for language and 2, so that they do not learn trivial solution producing the same constant representation. The learning curve shown in Figure I
The produced alignment is used as labels in the next two steps.
0.6
Phase
Phase
Phose
D(
Distance
0.4
0.2
10k Steps
Figure Learning curve of distance D. Training phase (left): fix both model, learn T only. Training phase 2 (middle): finetune language representation model with annealing. Training phase (right); finetune language representation model with annealing.
Extract A binary classifier is trained to predict whether token is pivot token of key-point. An example prediction is illustrated in Figure It can be observed that only tokens possess significant meaning is predicted as pivot tokens. For languages (c.g.. Chinese) that use multiple tokens to represent a word, only one token per word is predicted as pivot tokens. Considering the labels are
5

---

Under review as a conference paper at ICLR 2020
grouud--gountuuthh produced from only translation per sentence, the overall prediction accuracy is respectable (see Table 2)
Using confident labels slightly improves prediction accuracy, and makes the prediction significantly less ambiguous. This very important for producing consistent predictions of key pointss such that transfer learning models on top of CLCR can be trained consistent representation.
Table Key-point prediction accuracy
Accuracy zh
Precision
Recall
Model
zh
z/
Finetune
0.859
0.823
0.886
0.854
0.936
0.847
Confident labels
0.864
0,826
0.875
0.838
0.959
0.872
And
that
where
what
my
and
what
team
X
36
that
what or
where
and
our what
team
Figure An example of predicted key-points, with (below) and without (above) confident labels
Reorder absolute order prediction, we use an N-way classifier to predict the absolute position of token. In relative order prediction, binary classifier is used to predict the relative order of pair tokens. During inference, beam search is used to find the best order assignment tokens, so that the total cost is minimized. We simply the probability p of the classifier output as the cost of particular assignment.
Table rreports the reordering performance of two methods with different beam size We calculate 2-gram to 4-gram BLEU with respect to the groundtruth order. Relative order prediction performs much better than absolute order prediction, getting 2-gram predictions correct over half of the time and 4-gram predictions correct nearly one-third of the time.
Table 3: Reorder performance
BLEU (2-gram) zh
BLEU (3 (3-gram)
BLEU (4-gram)
Model
en
en
zh
en
zh
Absolute order prediction (n 1)
0.309
0.386
0.166
0.227
0.101
0.1.2
Absolute order prediction (n 10)
0.331
0.393
0.189
0.239
0.121
0.165
Relative order prediction (n 1)
0.535
0.566
0.384
0.410
0.290
0.308
Relative order prediction (n 10)
0.582
0.624
0.419
0.468
0.316
0.368
The example in Figure 3 illustrates that absolute order prediction suffers from ambiguity except near the beginming of the sentence, because the exact position depends on the specific translation and therefore does not have unique correct answer. Relative order prediction is much more robust because it is invariant to the location of the pair of tokens in the sentence, the classifier only makes predictions based on the relative order relation between them.
4.2 CROSS-LINGUAL TRANSFER
Natural Language inference XNLI (Conneau et al. 2018b) a cross-lingual natural language
inference corpus consisting of test examples in 15 languages. We the English and Chinese test set in our evaluation. We use MultiNLI (Willams et al. 2018). an English NLI dataset as the training set to train the task model.

---

Under review as a conference paper at ICLR 2020
Figure 3: A example order prediction. From left to right: (1) ground-truth absolute order. (2)
predicted absolute order. (3) ground-truth relative order. (4) predicted relative order.
To evaluate the performance of our cross- -oossll contextual representation model, we train the ESIM
model (Chen et al., 2017) with representation produced by CLCR as input. An advantage of CLCR multi-lingual pre-trained language models is that complex task models can be trained top of the representation. To examine the effect of each of the three steps in our formulation of CLCR, we also experimented with different combinations of steps.
Our full CLCR model (T+E+R) achieved a transfer accuracy of 71.7%, much better than multilingual English. The transfer gap is on par with state-of-
BERT and even better than translating the-art models (Table
It clear that CLCR works best when all three steps are present. Without extraction and reordering, model on top of CLCR does not perform as well. This is because the ESIM model makes use of order information of the input sequence, the number and the order of vectors should match for successful zero-shot transfer to another language.
is also observed that extraction and reordering lower performance on English by roughly 1%. The performance loss is expected, and not beyond acceptable range. During the training of CLCR, the English representation model is finetuned accommodate another language, and during imperfect extraction and reordering some information is inevitably lost.
We also set an upper bound for the transfer performance CLCR by using alignments (T+pGA). Eseudo-groundtrg and indicates the performance of the best possible T+E+R model. The remaining transfer gap (6.4%) thus intrinsic to the task (variation of test set difficulty) and the mono-lingual representation mod- els (different performance of BERT-bbase-nt and BERT--ase-zh).
alignments give the "correct" extraction and reordering,
Table 4: Performance of zero-shot cross- coossllnggull transfer. E, R stands for Transform, Extract, and Reorder, respectively. pGA stands for pseudo-groundtruth alignment. Gap stands for the transfer gap from English to Chinese (the drop of accuracy in percentage)
XNLI test set en Acc.
Model
zh Acc.
Gap
CLCR
T+pGA
81.2
74.8
6.4
81.2
68.0
13.2
T+E
80.6
69.3
11.3
T+E+R
80.2
71.7
8.5
Mon-CLCC models
Multilingual BERT
81.4
63.8
17.6
Translate train
81.4
74.2
7.2
Translate test
81.4
70.1
11.3
XLM
85.0
76.5
8.5

---

Under review as a conference paper at ICLR 2020
Although the performance of CLCR Chinese XNLI is inferior XLM, the transfer gap is roughly the same. XLM uses much more data and training time than our approach, and thus starts with better performance than BERT-base English. Also XLM is not bunguae-indeeenders which means it is not feasible to use XLM as representations to learn models for cross-lingual transfer. One only fine-tune XLM itself on language and hope that other languages also benefit from fine- tuning.
4.3 ANALYSIS
Edge probing To investigate the effect of transforma- tion. extraction, and reordering, measure what kind of information is retained in CLCR by comparing the tech-
Table 5: Edge probing statistics of CLCR
Fl score
representation before and after training. We nique called Edge Probing (Tenney et al.) (019b). In
BERT
T
T+E
edge probing, a series of tasks are used as "probes" to measure the syntactic and semantic information con- tained in representations. The jiant toolkit (Wang et al., 2019) is used to perform the experiments.
POS
97.6
92.4
82.2
Consts.
86.8
80.8
75.4
Deps.
95.4
92.8
84.5
Entities
96.1
94.8
90.9
Comparing CLCR with the original BERT-base English model in Table [5] the transformation step (T) generally lowers scores most probes by small fraction. This indicates while most information remains in the repre- sentation after training, there a certain portion of in- compatible syntactic and semantic representations across languages. That information is lost and explains the performance loss on XLI on the source language (English). Extraction (E) further low- scores, but mainly due to failing to predict labels non-pivot tokens. Because the probing model used in (Tenney et al. (209bb) is an attention pooling-based order-less model, reordering (R) does not alter the performance on these probing tasks.
SRI
89.3
86.0
83.1
Coref.
95.8
92.0
91.5
SPR
83.4
83.1
82.6
Relations
77.7
75.9
75.9
Training cost and beyond A total of 250,000 pairs of parallel sentences used in this study. This is less than 2% the data used to train XLM. Increasing the size of parallel corpus does not lead to further improvement. Also the total training time of CLCR, including the three steps is less than hours, on a single GPU. This indicates that with good mono-lingual contextual representations like BERT, learning cross-lingual alignment of representations could be relatively easy task, without the need for extensive supervision and training. Under our formulation, the performance of CLCR built upon and also limited by the quality of mono-lingual representations. But this more of an opportunity than problem because parallel corpus is always the more limited resource.
5 CONCLUSION
In this paper we investigated language-independent cross-lingual contextual representations. We proposed a three-step process to learn cross- cross-lingual contextual representations from mono-lingual representations. In each of the three steps, an intuitive method is employed with the aim of letting the model itself learn to ultimately align two representations. Experiments confirmed that state- of-the-art pre-trained language models are powerful enough learn to align two languages with moderate supervision. This could indicate that with models like BERT now it is easier than ever to cover the gap in cross- lonssll transfer learning.
Our formulation provides new approach to combine contextual representations with arbitrary task models for zero-shot cross-ling -ingual transfer learning. We experimented with the NLI task and the ESIM model, but the method could in principle be applied wide range of tasks and models.
With the advancement of pre-trained language models and their availability, we hope to extend the approach to more languages and more powerful models. We also hope investigation could inspire further insight into the nature of bagguage-indeeendee representations and cross- cross-lingual transfer learning.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. Enhanced LSTM for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties ( (Volume I: Long Pupers), pp. 1657-1668, July 2017.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2126-2136, July 2018a.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2475-2485, Association for Computational Linguistics, 2018b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Pupers), pp. 4171-4186, 2019.
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T Yarman Vural, and Kyunghyun Cho. Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 268-277, 2016.
Stephan Gouws, Yoshua Bengio, and Greg Corrado. Bilbowa: Fast bilingual distributed repre- sentations without word alignments. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning Volume 37, ICML' 15, pp. 748-756. JMLR.org. 2015.
J. Huang. J. Li, D. Yu, L. Deng, and Y. Gong. Cross-language knowledge transfer using multilin- gual deep neural network with shared hidden layers. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7304-7308. May 2013.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291 2019.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. JEEE Transactions pattern analysis and machine intelligence, 38(3)4447-461, 2015.
Tomas Mikolov, Quoe V Le, and Ilya Sutskever. Exploiting similarities among languages for ma- chine translation. arXiv preprint
2013.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? arXiv preprint 1oXiv:1906.01 01...11500 502, 2019.
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. survey of cross-lingual embedding models. arXiv preprint arXiv: /701v:/706.04902, 2017.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1599-1613, June 2019.
Holger Schwenk and Matthijs Douze. Learning joint multilingual sentence representations with
neural machine translation. arXiv preprint arXiv: 1704.04154 041,,, 2017.

---

Under review as a conference paper at ICLR 2020
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. Ernie: Enhanced representation through knowledge integration. arXiv preprint
2019.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp- 4593-4601, July 2019a.
lan Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? probing for sentence structure contextualized word representations. In International Conference on Learning Representations, 2019b.
Alex Wang, lan F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pap- pagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang. Edouard Grave, Najoung Kim, Phu Mon Htut, Thibault Fevry, Berlin Chen, Nikita Nangia, Haokun Liu, Anhad Mohananey, Shikha Bordia. Nicolas Patry, Ellie Pavlick, and Samuel R. Bowman. jiant 1.1: A software toolkit for research on general-purpose text understanding models. http://jiant, info/ 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers). pp. 1112-1122, 2018.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint a7Xiv:1906.08237. 2019.
Michal Ziemski, Marcin Manczys-Dowmin and Bruno Pouliquen. The united nations parallel cor- pus v1. Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pp. 3530-3534, 2016
10

---

Under review as a conference paper at ICLR 2020
A METHOD OVERVIEW
provide an overview diagram to show the whole process of generating language-inddeeeddnt cross-lingual contextual representations, using example in Figure
There was no atternative but to develop new international agreement on climate change to replace the existing Kyoto Protocol
(BERT-en)
(E)
(F)
(E)
(T)
(BEAT-2h)
1
RX
Figure 4: An example illustrating CLCR in inference. For pair of language (in this example, En- glish and Chinese), one can decide to perform transformation and reordering on one side, and per- form extraction both sides. After training of the transformation (T), extraction (E) and reordering (R) model, for input text in one of the two languages, a sequence of operation is performed: Embed- T-E-R or Embed-E depending (in the box with dashed line) is the same for text in both languages, assuming perfect T, E, and R models.
the language. The generated sequence-of-vector representation
B ZERO-SHOT CROSS-LINGUAL TRANSFER
Language-independee cross- lins-lln contextual representations enable zero-shot tross-lingual trans- fer for variety NLP models. For example, with learned CLCR between English and Chinese, can train task model with Embed-E on English, then perform inference with Embed-T--E-R Chinese (Figure(5).
Model (Training)
Model (Inference)
CLCR
(RI
(BERT-en)
(BERT-2))
A
(1 T. 96 F.
Figure 5: An example of zero-shot liosslinggull transfer with CLCR
11

---

Under review as a conference paper at ICLR 2020
C MODEL DETAILS AND MORE ANALYSIS
In our experiments, we use the ninth layer output of the 12-layer BERT-base model as mono-lingual contextual representations. Based on the observation in (Tenney et al. 2019a), middle-high layers of transformer language models typically exhibit good semantic properties. The English and Chinese BERT-base models we used are identical except the language of the corpus used trained the model. We also experimented with using different pre-trained language models, for example ERNIE (Sun et al., 2019) for Chinese (BERT-base base is still used for English), but the cross-lingual transfer performance is inferior. This shows that for best aligning of representations in CL.RR, the two mono-lingual representation models should share the same structure and objective in pre-training.
Table 6: Edge probing statistics
BERT-base
T+E (Ó
0.2)
T+E (Ó 0.5)
Fl
Pre.
Rec.
F1
Pre.
Rec.
F1
Pre.
Rec.
F1
Pre.
Rec.
POS
97.6
97.3
97.8
92.4
91.7
93.2
82.2
91.5
74.6
77.6
92.8
66.7
Consts.
86.8
82.4
79.2
75.4
80.2
71.1
71.5
80.1
64.6
Deps.
95.4
96.2
94.7
92.8
94.1
91.5
84.5
90.4
79.2
78.8
89.4
70.5
Entities
96.1
96.6
95.6
94.8
95.5
94.2
90.9
95.3
86.8
87.8
95.3
81,4
SRL
89.3
91.5
87.1
86.0
89.0
83.1
83.1
88.8
78.0
80.8
88.6
74.2
Coref.
95.8
95.8
95.8
92.0
90.8
93.2
91.5
91.5
91.5
89.4
89.4
89.4
SPR
83.4
84.2
82.6
83.1
83.3
82.9
82.6
84.5
80.8
82.5
82.3
82.6
Relations
77.7
85.5
71.2
75.9
85.4
68.2
75.9
84.1
69.2
74.8
84.3
67.3
In Table 6 we list more detailed statistics in probing the cross-lingual representations. The drop of score in the extraction (E) step is closely linked the threshold ő used in key-point prediction. Threshold 8 controls the number of key-points predicted in sentence. A higher 8 means smaller number tokens are predicted as key-points. This results in better accuracy in matching the gran- ularity of two languages by only predicting more confident key-points, but fewer key-points means more tokens are discarded and thus more information is potentially lost. This becomes obvious by comparing the recall of T and T+E models in probing tasks. We empirically chose a trade-off value of 0.2 in all our experiments.
We need to note that in E step the reduction of recall in probing tasks does not necessarily mean reduced performance of NLP models on top of CLCR. Significant decrease of recall is mainly ob- served part-of-speech, constituent and dependency labeling tasks. These tasks have dense labels, and labels on non-key-poit tokens are ignored by CLCR after step E. However, in training to align sentences the representation model is forced to summarize local information in the representation of key-points, so that information from non-key- point tokens could still be available for task model on top of CLCR.
12


Reviews:
[{"review_id": "ryexDZ32KB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new method to obtain cross-lingual contextual embeddings by aligning monolingual ones through 3 steps: transform each token individually, merge them as needed to obtain a uniform granularity across languages, and reorder them.\n\nWhile I think that the proposed method has some interest, and the extensive ablation experiments are useful to better understand its behavior, I do not think that the it makes enough merits to be accepted in the conference. I feel that the paper tends to overly complicate things, and it is often difficult to extract any clear idea from it. The proposed method is also much more complicated than previous approaches, yet it does not perform better than them (XLM has better absolute results and exactly the same cross-lingual transfer gap, while being substantially simpler). More concretely:\n\n- The paper tends to overly complicate things. For instance, Section 2.1 and 2.2 try to mathematically formalize very basic intuitions. Unless the formalization is important for clarity (which is not, as these are obvious ideas) or necessary later in the paper (which is not either), there is no point in doing that. This only makes the paper more difficult to follow than what it should.\n\n- The only extrinsic evaluation is in XNLI, where the authors evaluate the zero-shot cross-lingual transfer performance from English into Chinese. However, the proposed method does not bring any improvement over the current state-of-the-art in this setup. The proposed method gets 80.2% and 71.7% accuracy in English and Chinese, respectively, leaving an absolute transfer gap of 8.5%, while the XLM model from Lample and Conneau (2019) obtains substantially better results (85.0% and 76.5%) with the exact same transfer gap of 8.5%. This could still be good enough if the proposed method had some other advantage over the previous SOTA, but I cannot find any and, in contrast, I do find some disadvantages (see below).\n\n- In addition to being more complicated than previous approaches, the proposed method also introduces new hyperparameters and seems more difficult to train. For instance, the authors need to incorporate annealing to train the transformation module, and the model seems quite sensitive to the corresponding hyperparameter (Table 1).\n\n- While both multilingual BERT and XLM simply fine-tune a pre-trained BERT model to perform some downstream task, the proposed method is used as a feature extractor, and the authors train an ESIM (LSTM) model on top. The reported experiments do not control this factor (i.e. what would happen if one learns an ESIM model on top of XLM)?\n\n- The authors highlight that their system can be trained in \"less than 5 hours, on a single GPU\", while \"XLM uses much more data and training time\", but this is quite deceptive. Your approach also requires training a monolingual BERT model for each language, which is even more expensive than training a single joint model as XLM does. It is true that one could potentially use publicly available monolingual models, but most pre-trained models in languages other than English are already multilingual, anyway, so I do not see a strong practical justification for this.\n\n- The proposed model and the ones it is compared to do not use the same training data. This can have some justification (it might be computationally prohibitive for the authors to pre-train their own models, which might be the reason why they use public models trained on different data) but they should be more upfront about this. In relation to this, it is unfair to remark that the proposed method uses less parallel data than XLM, while not even mentioning that it uses more monolingual data (if I am not wrong, XLM was only trained in Wikipedia, while BERT also used a book corpus at least for English). To make things worse, the authors claim that \"XLM uses much more data and training time than our approach\", which seems wrong.\n\n- The authors criticize XLM because it \"does not produce the same representation for different languages, so there is no guarantee of the performance in transfer learning\". This might be true, but is it anyhow different for your proposed method? Your method is not better empirically, and it does not have any theoretical guarantee either.\n\n- As the authors themselves acknowledge, the proposed method is similar in spirit to Schuster et al. (NAACL'2019) -which is also much simpler- but they do not compare to it in their experiments.\n\nAll in all, I think that the paper tends to overly complicate things, and ultimately fails to answer a simple central question: why should one prefer your approach over previous methods like XLM, or what is it that it makes it otherwise interesting or relevant?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyxGx3e6YH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method to learn language-independent cross-lingual contextual representations by mapping the representations of a monolingual model in one language to the representations of a monolingual model in another language. The proposed approach consists of three steps: 1. A transformation is learned that minimizes the distance between the contextual word representations of the two models of a sentence and its translation in a parallel corpus. 2. Each sentence is summarized as a sequence of key-point tokens based on phrase alignment. 3. The contextual word vectors are reordered based on an order prediction model.\nThe authors perform experiments on intrinsic tasks and on XNLI, mapping between an English and a Chinese BERT model. They outperform multilingual BERT (mBERT) on the latter.\n\nOverall, the approach is novel, but the steps seem overly complicated. The extrinsic evaluation is the weakest point of the paper. Because of this, I tend to a Weak Reject. I would be willing to increase my score if additional languages are added to the evaluation and if the steps are better motivated and compared to simpler alternatives.\n\nThe high-level steps of the approach (transform, align, reorder) make sense. They seem to be inspired by classical phrase-based MT pipelines, but this connection is not made clear. In particular, some of the steps seem unnecessarily complicated and I am wondering whether the authors tried or compared to simpler alternatives. As a parallel corpus is used in the first step, word alignment could be automatically obtained by FastAlign without the use of attention matching or using a phrase table that is learned in an unsupervised way as in recent work in NMT (https://arxiv.org/pdf/1804.07755.pdf). For the second step, embeddings of aligned phrases could be averaged or the head of a phrase could be used instead of predicting key-points with a separate network.\n\nThe data requirements between the steps seem somewhat inconsistent. A parallel corpus is used in the first step, but explicitly not used in the second step. While I agree that language-independent cross-lingual representations should not use a parallel corpus, it would have been good if the first step could have also been performed without one (or with a smaller size).\n\nThe extrinsic evaluation of the paper could be improved. The accuracy on each intrinsic step is evaluated. Without any reference or comparison, I found it hard to tell how good these numbers are so this evaluation did not add much for me. The main extrinsic evaluation of the paper is on XNLI but only employs an English and a Chinese model. When training cross-lingual models, the aim is to train approaches that not only work for one or two languages but for many. In light of this, I find it hard to tell from the results on a single language pair how well the approach will generalize to other languages, particularly as mBERT's performance on zh XNLI is comparatively weak (https://arxiv.org/pdf/1901.07291.pdf). There are other publicly available BERT models such as for German (https://deepset.ai/german-bert) on which the approach could be tried. If compute is a concern, then the approach could be applied to ELMo representations. This would also enable a comparison to Schuster et al. (2019; https://arxiv.org/abs/1902.09492).\n\nFinally, the data requirement of 250k parallel sentences is prohibitive for many language pairs where this approach would be valuable. It would be good to see a chart on how model performance develops with the number of parallel sentences to see if a smaller number of parallel sentences would be viable in practice.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BklxL4DDYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approach to learning contextualised cross-lingual representations based on a three-step procedure called: transform-extract-reorder. The proposed model is tested on zero-shot cross-lingual transfer for one language pair: English-Chinese, where the gains are reported over the initial multilingual BERT model which does not rely on any bilingual signal. I like the main modeling idea, and the actual model is described quite nicely and in an intuitive way. However, I believe that much more work is needed in terms of proper experiments including: a) additional strong and insightful baselines; b) experiments with more language pairs; c) experiments across other transfer tasks. The paper also does not include nor compares to all relevant previous work. My main comments and remarks are as follows:\n\n*Related work and Novelty* The topic of learning contextualised cross-lingual representations has become quite popular since the rise of ELMo and BERT. However, the authors cite only one paper that proposed such contextualised xling representations (the work of Schuster of al.). However, they never compare to that model, which is quite weird given the fact that the main goal of all xling representation models is to enable cross-lingual transfer in the first place. A comparison to the cited work of Schuster et al., plus additional comparisons to other (non-cited) models are required to clearly understand the main empirical benefits of the proposed three-step framework. Other relevant papers which are not cited nor compared against: Mulcaire et al. (NAACL 2019); Aldarmaki and Diab (NAACL 2019).\n\nThe authors should really stress the key novelty of their approach in comparison to the existing literature (which is mostly based on simpler projection-based methods).\n\n- Note that there are more papers on the subject getting published soon such as Mulcaire et al. (CoNLL 2019) and Liu et al. (CoNLL 2019); while making comparisons to models introduced in those papers is not required, it would be nice to also briefly mention that latest work in the related work section once it gets published.\n\n*Comparisons to XLM* Based on the limited set of results reported in the paper, it seems that the proposed model cannot match the performance of the XLM model (Lample and Conneau, NeurIPS 2019). The authors try to justify the usage of their method over XLM by emphasising the fact that their method produces language-independent xling representations, which XLM cannot do. However, there are multiple issues here: 1) it is not clear why exactly this requirement of language-independence is needed for a task such as zero-shot XNLI; 2) the paper does not demonstrate the usefulness of that language-independence property empirically at all. To me, it just seems like an unsuccessful attempt to make a conceptual distinction between the proposed model and XLM given the fact that XLM seems to significantly outperform the proposed model.\n\nPage 8 (\"Training cost and beyond\"). The authors claim that 250k pairs of parallel sentences is enough to learn a near-optimal cross-lingual model and that the model saturates with more parallel sentences. I don't see it as a positive aspect of the model as claimed by the authors. This means that the model cannot optimally encode knowledge about variety of cross-lingual contexts during its training, i.e., it stops learning earlier than expected. Why does that happen? A plot showing transfer learning results of XLM versus the proposed model versus some static word embedding model that also exploits parallel data would be very useful. The authors also do not even speculate why their model saturates already with 250k sentence pairs. Given the limited set of results (only one task and only language pair) it also remains unknown how general this phenomenon is and whether the same point is hit with some other language pairs and with other tasks. This definitely requires further and more thorough investigation.\n\n*Other Comparisons and Experiments*\nAs mentioned, the paper is very limited when it comes to proper thorough evaluation. Most experiments focus on intrinsic/internal evaluations of different steps of the proposed framework, or some probing tests, which can also be seen simply as diagnostic experiments. The actual downstream experiments are conducted on only one language pair and for only one (zero-shot) transfer task. This is definitely not sufficient to draw any generalisable claims, and it prevents us to dig deeper into other interesting aspects of the model. I would suggest the authors to maybe run the model across the same range of tasks as done in the XLM work of Lample and Conneau, and definitely for more languages (ideally diverse target languages).\n\nAlso, besides XLM and translation-based baselines reported by multilingual BERT's github page, there is actually no other baselines, which really makes it hard to put this work in context, and understand its usefulness. For instance, static cross-lingual word embeddings could be added to ESIM and used to evaluate on XNLI as well (see, e.g., the work of Glavas et al. (ACL 2019)). It would be interesting to report the benefits of replacing such static vectors with truly contextualised representations. Also, as mentioned before, comparisons to other models that learn contextualised cross-lingual representations are definitely something that should be included in the paper.\n\n*Other Comments*\n- Based on the result from Table 4, it seems that, in order to enable fully language-independent representations, one must sacrifice some of the monolingual performance, as the numbers drop from T over T+E to T+E+R variant. Is the same pattern visible for other languages monolingually? Why does this happen? \n- As mentioned before, the whole emphasis is of language-independence is somewhat oversold in the whole paper without providing sufficient empirical evidence that this is crucial for transfer performance.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to learn cross-lingual representations by aligning monolingual models with the help of a parallel corpus using a three-step process: transform, extract, and reorder. Experiments on XNLI show that the proposed method is able to perform zero-shot cross-lingual transfer, although its overall performance is still below state-of-the-art jointly trained method XLM.

All three reviewers suggested that the proposed method needs to be evaluated more thoroughly (more datasets and languages). R2 and R4 raise some concerns around the complexity of the proposed method (possibly could be simplified further). R3 suggests a more thorough investigation on why the model saturates at 250,000 parallel sentences, among others.

The authors acknowledged reviewers' concerns in their response and will incorporate them in future work.

I recommend rejecting this paper for ICLR.
Decision: Reject