Title: LANGUAGE-INDEPENDENT CROSS-LINGUAL CONTEX-TUAL REPRESENTATIONS

Abstract: Contextual representation models like BERT have achieved state-of-the-art performance on a diverse range of NLP tasks. We propose a cross-lingual contextual representation model that generates language-independent contextual representations. This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models, on top of contextual representation models like BERT. We provide a formulation of language-independent cross-lingual contextual representation based on mono-lingual representations. Our formulation takes three steps to align sequences of vectors: transform, extract, and reorder. We present a detailed discussion about the process of learning cross-lingual contextual representations, also about the performance in cross-lingual transfer learning and its implications.

1. INTRODUCTION:

A cross-lingual text representation can generate language-independent representations: for pieces of text that are semantically similar, the produced representations are also similar, regardless of the language of the text. This provides a basis for dealing with multiple languages in NLP. Training a model on cross-lingual representations is a common approach for cross-lingual transfer learning (Huang et al., 2013; Artetxe & Schwenk, 2019) .

To achieve desirable transfer learning performance, the cross-lingual representation model should be effective in two aspects: 1) the quality of text representation; and 2) the quality of cross-lingual mapping.

The current state-of-the-art text representation models are a family of pre-trained language model representations such as BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019) . These models produce contextual representations of text, i.e., a sequence of vectors corresponding to the sequence of text tokens. Each vector represents "a token in context," e.g., the semantics of a word in its local context and in the global context of the sentence.

Research on cross-lingual representation has so far been largely concentrated on word embeddings (Mikolov et al., 2013; Gouws et al., 2015; Ruder et al., 2017) and sentence embeddings (Schwenk & Douze, 2017; Artetxe & Schwenk, 2019) . Word embeddings are static representations and are less expressive than contextual representations. Sentence embeddings perform respectably in some tasks but inferior in others, because there is information loss when too much information is crammed into a single vector (Conneau et al., 2018a) . However, word embeddings and sentence embeddings are relatively easy to generalize to cross-lingual representations because they do not have dynamic inter-dependency among vectors.

We regard it as an interesting topic to investigate cross-lingual contextual representations. First, it is an interesting problem in its own to match two (or more) sequences of vectors, where each vector sequence has its order and locality relationships. Second, we hope to further improve cross-lingual transfer learning by leveraging the state-of-the-art text representation models.

In this study, we try to clarify the definition and characteristics of cross-lingual contextual representations, and propose a formulation to learn a cross-lingual contextual representations model based on existing mono-lingual pre-trained language models. We perform experiments to analyze the effect of the ingredients in our formulation and the final performance in transfer learning.

We also hope our investigation help to understand the nature of text representations in different languages: what are the commonalities between representations, how to cover the gaps, and ultimately what is transferable across languages with the help of state-of-the-art representation models.

Related work:

Multilingual BERT learns a multi-lingual representation by training on a combined corpus of multiple languages. For dissimilar languages, the transfer performance is not satisfactory (Pires et al., 2019) . XLM (Lample & Conneau, 2019) uses parallel corpus and a translational objective, and has achieved good transfer performance for many languages. However, XLM does not produce the same representation for different languages, so there is no guarantee of the performance in transfer learning. In multi-way neural machine translation (Firat et al., 2016) the situation is similar: multiple encoders produce the representation for each language, but because the representations are not language-independent, the decoder still needs to be trained on multiple language inputs.

A recent work on aligning contextual representations is (Schuster et al., 2019) , which treats vectors in contextual representations as individual embeddings and learns a matrix to transform the embeddings space of different languages. The idea resembles the "transform" step in our formulation.

2. CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS:

First we define "language-independent cross-lingual contextual representations (CLCR)" and establish mathematical notations that we use in the following discussion.

2.1. DEFINITION:

A cross-lingual contextual representation (CLCR) of text is a parameterized representation model f (θ) : (a 0 , ..., a L ) → (r 0 , ..., r N ), where a 0 , ..., a L is the sequence of text tokens in a sentence s, and r 0 , ..., r N is the sequence of vectors produced by f as the representation of s. A languageindependent f satisfies the following conditions:

1. (cross-lingual correspondence) if sentence s i in i-th language and sentence s j in jth language are translations of each other, then by some distance metric D we have D(f (s i ; θ), f (s j ; θ)) < δ, where δ is a threshold.

2.. (isometric property:

) if for some semantic metric S, sentence s i in i-th language and sentence s j in j-th language satisfy S(s i , s j ) < γ, i.e., s i and s j are close in meaning, then by some distance metric D we have D(f (s i ; θ), f (s j ; θ)) < δ(γ), where δ is a monotonic increasing function of γ.

3.. (contextual representation:

) ∃f 1 , f 2 , f 3 s.t. r i = f 3 (f 2 (a i ), f 1 (a 0...L\i ))

. The vector r i can roughly be interpreted as a representation of token a i in the context of the whole sentence.

Condition 1 and 2 ensure that CLCR produces similar representations for sentences with similar meaning, independent of language. Condition 3 is a characteristic of contextual representation models such as pre-trained language models like BERT.

2.2. EXISTENCE OF CLCR:

To show that such cross-lingual contextual representation exists, we provide a formulation of CLCR based on mono-lingual contextual representations. We propose that one can generate CLCR with a three-step process. Starting from mono-lingual representations, each of the three steps corresponds to one problem that needs to be solved, in order to arrive at a cross-lingual contextual representation.

1. Alignment: the vectors of mono-lingual representation of different languages reside in different spaces. First we align the vector spaces R i with transformations T i : r i → r C , to transform into a common space R C .

2. Granularity: different languages have different token granularities. Some language use several tokens to represent a word. Token alignment between translations s i and s j can be represented by a set A = {((a i m , ..., a i n ), (a j p , ..., a j q ))}, where tokens (a i m , ..., a i n ) and (a j p , ..., a j q ) are aligned "phrases."

Definition of Finest Common Granularity: find the token alignment A * that cannot be broken into a finer alignment. The finest common granularity for s i w.r.t. language i and j is defined as a grouping of the token sequence of s i in A * : F CG(s i ) = {(a i 0 , ..., a i n0 ), (a i n0+1 , ..., a i n1 ), ..., (a i nm+1 , ..., a i L )}, where each element (a i ni+1 , ..., a i ni+1 ) ∈ A * . In a nutshell, we want to group a series of tokens (a i n0+1 , ..., a i n1 ) into a "phrase token" (key-point token), so that the key-point tokens have one-to-one correspondence between two languages. With key-point tokens derived from FCG, translations s i and s j have the same number of key-point tokens, and can then be represented by the same number of vectors. 3. Order: many different word orders exist in different languages, for example the SVO (subject-verb-object) and the SOV (subject-object-verb) order. We define an order mapping to map the orders of token from language i to language j: R i→j : m|s i → w, where ((a i nm+1 , ..., a i nm+1 ), (a j nw+1 , ..., a j nw+1 )) ∈ A. The order mapping R i→j maps the position of a key-point token in the source sentence to the position of the corresponding key-point token in the target language.

To generate cross-lingual representations, we perform alignment transformation on the vectors of two mono-lingual representations, find their common granularity, and then reorder the vectors, to make the two representations similar.

2.3. TRANSFER LEARNING WITH CLCR:

Language-independent cross-lingual representations enables zero-shot transfer learning across languages. CLCR allows the most versatile models to be trained on cross-lingual representations. For a model M taking contextual representations (r 0 , ..., r N ) as input, M can be trained on dataset {(s i , l)} in language i, by first using CLCR f to encode inputs: M (f (s i )) → l. In testing, model can perform inference on input in any language within f 's input languages.

3. LEARNING CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS:

In this section, we detail our approach to learn CLCR based on the formulation in Section 2.2. We use two mono-lingual contextual representation models on language 1 and 2 to learn a cross-lingual contextual representation model.

3.1. TRANSFORM:

In the first step, we learn a transformation T parameterized by a multi-layer network:

T (r) = Linear(r) + Linear(Relu(Linear(r)))

(1) The network is trained to minimize the discrepancy between the transformed representation in language 1 and the representation in language 2: D(T (r 1 0 , ..., r 1 M )||(r 2 0 , ..., r 2 N )). Because we do not have alignment between the two sequences of vectors yet, we define D with attention matching between the two sequences:

D(T (r 1 0 , ..., r 1 M )||(r 2 0 , ..., r 2 N )) = i 1 N ||T (Attention(r 2 i , (r 1 0 , ..., r 1 M )) -r 2 i || (2)

where Attention is defined as using r 2 i as query to perform a weighted-sum of (r 1 0 , ..., r 1 N ). To train transformation T , we minimize the above loss D on a parallel corpus {(s 1 , s 2 )}. We observed that straightforward gradient descent does not converge. We therefore used attention annealing during training to help our model converge to an optimal matching: sof tmax(a 0 , ..., a N ) = e a0/T i e ai/T

(3)

where

T = 1/(1 + β • step).

As training progresses sof tmax will gradually be replaced by max, which let the model find a one-to-one matching between the tokens. This is exactly required by the next step.

3.2. EXTRACT:

In this step, we want to extract "key-points" from a sentence. These key-points are summarizations of local information (which may consist of multiple tokens), and the granularity of "key-points" is ideally consistent across different languages.

We extract key-points based on finest common granularity (FCG) between two languages: for a group of tokens (a i n0+1 , ..., a i n1 ) ∈ F CG(s i ), we use a single vector r n0 to represent them. The number of vectors in the CLCR of s i is thus equal to |F CG(s i )|.

Next, we train a network P to predict key-points in F CG(s i ). Although we can generate F CG(s i ) by aligning parallel documents, in general CLCR should not rely on parallel documents and should predict key-points in a single sentence independently:

r i k is the representation for key-point token (a i n0+1 , ..., a i n1 ) (4) ⇐⇒ ∃! a i k ∈ {a i n0+1 , ..., a i n1 }, P (a i k ) = 1 (5)

The prediction network predicts one of the tokens in a key-point token to be the "pivot token." And the vector for that pivot token is regarded as the representation for the whole key-point token.

Finally, we need to generate ground-truth labels for training key-point prediction. In the definition above, the question "which word is the pivot token in a phrase?" do not have definite answers. Therefore, we train our representation model and let the model determine the pivot token in a keypoint token. This is achieved by forced alignment via annealing in Section 3.1. After training the transformations T , we calculate the pairwise distance between r 1 i and r 2 j :

D(T (r 1 i )||r 2 j ) + D(T (r 2 j )||r 1 i )

, and use a threshold to find aligned pivot tokens. Aligned pivot tokens (which is at the same time aligned key-point tokens) generated in this fashion closely resembles key-point tokens derived from FCG. As we do not have ground-truth word alignment of parallel corpus to generate real FCGs, we use our aligned key-points as pseudo-groundtruth alignments and use them as labels to train predictors in this and the next section.

One caveat of using parallel corpus is that translations are not unique. There can be multiple valid translations of a sentence. Thus labels derived from parallel corpus are intrinsically noisy and nonunique. We generate more reliable labels to train our prediction network, with a technique similar to (Liu & Tao, 2015) .

3.3. REORDER:

The aim of reordering is to make the order of contextual vectors (r 1 0 , ..., r 1 N ) in language 1 compatible with the order in language 2. This can be achieved by learning an order prediction model, to predict the order mapping R 1→2 . We propose two methods for order prediction:

• Absolute order prediction: learn a prediction model to directly predict the position of token a 1 i in language 2:

EQUATION

Inference:

minimize

EQUATION

• Relative order prediction: learn a model to predict the relative order of two tokens a 1 i and a 1 j in language 2:

EQUATION

Inference:

minimize

EQUATION

The learned order prediction is R 1→2 : i|s 1 → m i . The vectors in language 1 can be reordered according to R 1→2 , so that the reordered vectors have the order of language 2. The reordered vector sequence is (r 1 R2→1(0) , ..., r 1 R2→1(N ) ), where R 2→1 is the reverse mapping of R 1→2 .

4. EXPERIMENTS:

In this section, we perform experiments to learn language-independent cross-lingual contextual representation based on the approach in Section 3. We also perform transfer learning and analysis experiments to examine the characteristics of CLCR.

4.1. LEARNING:

We sequentially examine the details and the effectiveness of each of the three steps in our formulation to generate CLCR: transform, extract, and reorder.

The basic experiment setup is: we use mono-lingual pre-trained language models from Google BERT (Devlin et al., 2019) , for English and Chinese1 . The 12-layer BERT-base model is used. We use parallel corpus from The United Nations Parallel Corpus v1.0 (Ziemski et al., 2016) . Experiments are performed with Pytorch (Paszke et al., 2017) .

Transform As shown in Table 1 , training transformation T to minimize distance D with attention annealing managed to converge to a low distance between the two representations. Lower distance indicates that a better alignment is found (because unaligned and misaligned tokens always contribute to high distance). And a good alignment is also conducive to learning a better transformation. During training, we alternately optimize the representation model for language 1 and 2, so that they do not learn a trivial solution of producing the same constant representation. The learning curve is shown in Figure 1 .

The produced alignment is used as labels in the next two steps. 

< l a t e x i t s h a 1 _ b a s e 6 4 = " G z B y n A F d j f 9 T H C w 4 R z A A f e s s e O A = " >:

A A A B 9 X i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m I C F 9 K i i R 6 J e v C I C R 8 m U M h 2 2 c K G 7 b b Z 3 W p I 4 X 9 4 8 a A x X v 0 v 3 v w 3 L t C D g i + Z 5 O W 9 m c z M 8 y L O l L b t b y u z t r 6 x u Z X d z u 3 s 7 u 0 f 5 A + P m i q M J a E N E v J Q P n h Y U c 4 E b W i m O X 2 I J M W B x 2 n L G 9 3 M / N Y j l Y q F o q 7 H E X U D P B D M Z w R r I 3 V v i / W i 6 j q l y U R 1 K 6 V e v m C X 7 T n Q K n F S U o A U t V 7 + q 9 M P S R x Q o Q n H S r U d O 9 J u g q V m h N N p r h M r G m E y w g P a N l T g g C o 3 m V 8 9 R W d G 6 S M / l K a E R n P 1 9 0 S C A 6 X G g W c 6 A 6 y H a t m b i f 9 5 7 V j 7 V 2 7 C R B R r K s h i k R 9 z p E M 0 i w D 1 m a R E 8 7 E h m E h m b k V k i C U m 2 g S V M y E 4 y y + v k m a l 7 J y X K / c X h e p 1 G k c W T u A U i u D A J V T h D m r Q A A I S n u E V 3 q w n 6 8 V 6 t z 4 W r R k r n T m G P 7 A + f w C Q F p F B < / l a t e x i t > D(T (s 2 )||s 1 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " t z B m L R 1 K x 6 l T Z J W I B + S 5 E S c L z f o = " > A A A B 9 X i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m I C F 9 K i i R 6 J e v C I C R 8 m U M h 2 2 c K G 7 b b Z 3 W p I 4 X 9 4 8 a A x X v 0 v 3 v w 3 L t C D g i + Z 5 O W 9 m c z M 8 y L O l L b t b y u z t r 6 x u Z X d z u 3 s 7 u 0 f 5 A + P m i q M J a E N E v J Q P n h Y U c 4 E b W i m O X 2 I J M W B x 2 n L G 9 3 M / N Y j l Y q F o q 7 H E X U D P B D M Z w R r I 3 V v i / W i 6 l Z K k 4 n q O q V e v m C X 7 T n Q K n F S U o A U t V 7 + q 9 M P S R x Q o Q n H S r U d O 9 J u g q V m h N N p r h M r G m E y w g P a N l T g g C o 3 m V 8 9 R W d G 6 S M / l K a E R n P 1 9 0 S C A 6 X G g W c 6 A 6 y H a t m b i f 9 5 7 V j 7 V 2 7 C R B R r K s h i k R 9 z p E M 0 i w D 1 m a R E 8 7 E h m E h m b k V k i C U m 2 g S V M y E 4 y y + v k m a l 7 J y X K / c X h e p 1 G k c W T u A U i u D A J V T h D m r Q A A I S n u E V 3 q w n 6 8 V 6 t z 4 W r R k r n T m G P 7 A + f w C Q H J F B < / l a t e x i t > Phase 1

Phase 2 Phase 3

Distance Extract A binary classifier is trained to predict whether a token is a pivot token of a key-point. An example prediction is illustrated in Figure 2 . It can be observed that only tokens possess significant meaning is predicted as pivot tokens. For languages (e.g., Chinese) that use multiple tokens to represent a word, only one token per word is predicted as pivot tokens. Considering the labels are pseudo-groundtruth produced from only one translation per sentence, the overall prediction accuracy is respectable (see Table 2 ).

Using confident labels slightly improves prediction accuracy, and makes the prediction significantly less ambiguous. This is very important for producing consistent predictions of key-points, such that transfer learning models on top of CLCR can be trained on a consistent representation. Reorder In absolute order prediction, we use an N-way classifier to predict the absolute position of a token. In relative order prediction, a binary classifier is used to predict the relative order of a pair of tokens. During inference, beam search is used to find the best order assignment of tokens, so that the total cost is minimized. We simply use the probability p of the classifier output as the cost of a particular assignment.

Table 3 reports the reordering performance of two methods with different beam size n. We calculate 2-gram to 4-gram BLEU with respect to the groundtruth order. Relative order prediction performs much better than absolute order prediction, getting 2-gram predictions correct over half of the time and 4-gram predictions correct nearly one-third of the time. The example in Figure 3 illustrates that absolute order prediction suffers from ambiguity except near the beginning of the sentence, because the exact position depends on the specific translation and therefore does not have a unique correct answer. Relative order prediction is much more robust because it is invariant to the location of the pair of tokens in the sentence, the classifier only makes predictions based on the relative order relation between them.

4.2. CROSS-LINGUAL TRANSFER:

Natural Language inference XNLI (Conneau et al., 2018b ) is a cross-lingual natural language inference corpus consisting of test examples in 15 languages. We use the English and Chinese test set in our evaluation. We use MultiNLI (Williams et al., 2018) , an English NLI dataset as the training set to train the task model. 1994

年 ， 我 曾 任 惠 特 尼 博 物 馆 馆 长 时 举 办 了 一 个 叫 做 " 黑 人 男 性 " 的 展 览

In To evaluate the performance of our cross-lingual contextual representation model, we train the ESIM model (Chen et al., 2017) with representation produced by CLCR as input. An advantage of CLCR over multi-lingual pre-trained language models is that complex task models can be trained on top of the representation. To examine the effect of each of the three steps in our formulation of CLCR, we also experimented with different combinations of steps.

Our full CLCR model (T+E+R) achieved a transfer accuracy of 71.7%, much better than multilingual BERT and even better than translating test set to English. The transfer gap is on par with state-ofthe-art models (Table 4 ).

It is clear that CLCR works best when all three steps are present. Without extraction and reordering, model on top of CLCR does not perform as well. This is because the ESIM model makes use of order information of the input sequence, the number and the order of vectors should match for successful zero-shot transfer to another language.

It is also observed that extraction and reordering lower performance on English by roughly 1%. The performance loss is expected, and not beyond acceptable range. During the training of CLCR, the English representation model is finetuned to accommodate another language, and during imperfect extraction and reordering some information is inevitably lost.

We also set an upper bound for the transfer performance of CLCR by using pseudo-groundtruth alignments (T+pGA). Pseudo-groundtruth alignments give the "correct" extraction and reordering, and indicates the performance of the best possible T+E+R model. The remaining transfer gap (6.4%) is thus intrinsic to the task (variation of test set difficulty) and the mono-lingual representation models (different performance of BERT-base-en and BERT-base-zh). Although the performance of our CLCR on Chinese XNLI is inferior to XLM, the transfer gap is roughly the same. XLM uses much more data and training time than our approach, and thus starts with better performance than BERT-base on English. Also XLM is not language-independent, which means it is not feasible to use XLM as representations to learn models for cross-lingual transfer. One can only fine-tune XLM itself on a language and hope that other languages also benefit from finetuning. (Tenney et al., 2019b) . In edge probing, a series of tasks are used as "probes" to measure the syntactic and semantic information contained in representations. The jiant toolkit (Wang et al., 2019) is used to perform the experiments.

4.3. ANALYSIS:

Comparing CLCR with the original BERT-base English model in Table 5 , the transformation step (T) generally lowers scores of most probes by a small fraction. This indicates while most information remains in the representation after training, there is a certain portion of incompatible syntactic and semantic representations across languages. That information is lost and explains the performance loss on XLI on the source language (English). Extraction (E) further lowers scores, but mainly due to failing to predict labels on non-pivot tokens. Because the probing model used in (Tenney et al., 2019b ) is an attention pooling-based order-less model, reordering (R) does not alter the performance on these probing tasks.

Training cost and beyond A total of 250,000 pairs of parallel sentences are used in this study. This is less than 2% of the data used to train XLM. Increasing the size of parallel corpus does not lead to further improvement. Also the total training time of CLCR, including the three steps is less than 5 hours, on a single GPU. This indicates that with good mono-lingual contextual representations like BERT, learning cross-lingual alignment of representations could be a relatively easy task, without the need for extensive supervision and training. Under our formulation, the performance of CLCR is built upon and also limited by the quality of mono-lingual representations. But this more of an opportunity than a problem because parallel corpus is always the more limited resource.

5. CONCLUSION:

In this paper we investigated language-independent cross-lingual contextual representations. We proposed a three-step process to learn cross-lingual contextual representations from mono-lingual representations. In each of the three steps, an intuitive method is employed with the aim of letting the model itself learn to ultimately align two representations. Experiments confirmed that stateof-the-art pre-trained language models are powerful enough to learn to align two languages with moderate supervision. This could indicate that with models like BERT now it is easier than ever to cover the gap in cross-lingual transfer learning.

Our formulation provides a new approach to combine contextual representations with arbitrary task models for zero-shot cross-lingual transfer learning. We experimented with the NLI task and the ESIM model, but the method could in principle be applied to a wide range of tasks and models.

With the advancement of pre-trained language models and their availability, we hope to extend the approach to more languages and more powerful models. We also hope our investigation could inspire further insight into the nature of language-independent representations and cross-lingual transfer learning.

A METHOD OVERVIEW:

We provide an overview diagram to show the whole process of generating language-independent cross-lingual contextual representations, using an example in Figure 4 .

There was no alternative but to develop a new international agreement on climate change to replace the existing Kyoto Protocol . B ZERO-SHOT CROSS-LINGUAL TRANSFER Language-independent cross-lingual contextual representations enable zero-shot cross-lingual transfer for a variety of NLP models. For example, with learned CLCR between English and Chinese, one can train a task model with Embed-E on English, then perform inference with Embed-T-E-R on Chinese (Figure 5 ). 

:

https://github.com/google-research/bert

Figures and Tables:

(s 1 )||s 2 )

Figure 1: Learning curve of distance D. Training phase 1 (left): fix both model, learn T only. Training phase 2 (middle): finetune language 1 representation model with annealing. Training phase 3 (right): finetune language 2 representation model with annealing.

Figure 2: An example of predicted key-points, with (below) and without (above) confident labels

Figure 3: A example of order prediction. From left to right: (1) ground-truth absolute order. (2) predicted absolute order. (3) ground-truth relative order. (4) predicted relative order.

Figure 4: An example illustrating CLCR in inference. For a pair of language (in this example, English and Chinese), one can decide to perform transformation and reordering on one side, and perform extraction on both sides. After training of the transformation (T), extraction (E) and reordering (R) model, for input text in one of the two languages, a sequence of operation is performed: Embed-T-E-R or Embed-E depending on the language. The generated sequence-of-vector representation (in the box with dashed line) is the same for text in both languages, assuming perfect T, E, and R models.

Figure 5: An example of zero-shot cross-lingual transfer with CLCR



Table content: <table><tr><td>Linear Regression</td><td>0.055</td><td>0.078</td></tr><tr><td>Attention (w/o annealing T = 1)</td><td>0.138</td><td>0.207</td></tr><tr><td>Attention (with annealing β = 0.001)</td><td>0.028</td><td>0.048</td></tr><tr><td>Attention (with annealing β = 0.01)</td><td>0.025</td><td>0.037</td></tr></table>

Key-point prediction accuracy

Table content: <table><tr><td/><td/><td/><td/><td colspan="3">Model</td><td/><td/><td>Accuracy en zh</td><td/><td/><td/><td>Precision en zh</td><td>Recall en</td><td>zh</td></tr><tr><td/><td/><td/><td colspan="4">Finetune</td><td/><td colspan="7">0.859 0.823 0.886 0.854 0.936 0.847</td></tr><tr><td/><td/><td/><td colspan="12">Confident labels 0.864 0.826 0.875 0.838 0.959 0.872</td></tr><tr><td>A n d t h a t '</td><td>s</td><td>,</td><td>k in d o f</td><td>,</td><td>w h e r e I</td><td>s e e w h a t m y</td><td>j o b</td><td>is</td><td>a n d w h a t o u r t e a m '</td><td>s</td><td>j o b</td><td>is</td><td>.</td></tr><tr><td>A n d t h a t '</td><td>s</td><td>,</td><td>k in d o f</td><td>,</td><td>w h e r e I</td><td>s e e w h a t m y</td><td>j o b</td><td>is</td><td>a n d w h a t o u r t e m a '</td><td>s</td><td>j o b</td><td>is</td><td>.</td></tr></table>

Reorder performance

Table content: <table><tr><td>Model</td><td colspan="6">BLEU (2-gram) BLEU (3-gram) BLEU (4-gram) en zh en zh en zh</td></tr><tr><td>Absolute order prediction (n = 1)</td><td>0.309</td><td>0.386</td><td>0.166</td><td>0.227</td><td>0.101</td><td>0.152</td></tr><tr><td colspan="2">Absolute order prediction (n = 10) 0.331</td><td>0.393</td><td>0.189</td><td>0.239</td><td>0.121</td><td>0.165</td></tr><tr><td>Relative order prediction (n = 1)</td><td>0.535</td><td>0.566</td><td>0.384</td><td>0.410</td><td>0.290</td><td>0.308</td></tr><tr><td>Relative order prediction (n = 10)</td><td>0.582</td><td>0.624</td><td>0.419</td><td>0.468</td><td>0.316</td><td>0.368</td></tr></table>

Performance of zero-shot cross-lingual transfer. T, E, R stands for Transform, Extract, and Reorder, respectively. pGA stands for pseudo-groundtruth alignment. Gap stands for the transfer gap from English to Chinese (the drop of accuracy in percentage)

Table content: <table><tr><td>Model</td><td colspan="3">XNLI test set en Acc. zh Acc. Gap</td></tr><tr><td>CLCR</td><td/><td/><td/></tr><tr><td>T+pGA</td><td>81.2</td><td>74.8</td><td>6.4</td></tr><tr><td>T</td><td>81.2</td><td>68.0</td><td>13.2</td></tr><tr><td>T+E</td><td>80.6</td><td>69.3</td><td>11.3</td></tr><tr><td>T+E+R</td><td>80.2</td><td>71.7</td><td>8.5</td></tr><tr><td>Non-CLCR models</td><td/><td/><td/></tr><tr><td>Multilingual BERT</td><td>81.4</td><td>63.8</td><td>17.6</td></tr><tr><td>Translate train</td><td>81.4</td><td>74.2</td><td>7.2</td></tr><tr><td>Translate test</td><td>81.4</td><td>70.1</td><td>11.3</td></tr><tr><td>XLM</td><td>85.0</td><td>76.5</td><td>8.5</td></tr></table>



Table content: <table><tr><td colspan="4">: Edge probing statistics of CLCR</td></tr><tr><td/><td/><td>F1 score</td></tr><tr><td/><td>BERT</td><td>T</td><td>T+E</td></tr><tr><td>POS</td><td>97.6</td><td colspan="2">92.4 82.2</td></tr><tr><td>Consts.</td><td>86.8</td><td colspan="2">80.8 75.4</td></tr><tr><td>Deps.</td><td>95.4</td><td colspan="2">92.8 84.5</td></tr><tr><td>Entities</td><td>96.1</td><td colspan="2">94.8 90.9</td></tr><tr><td>SRL</td><td>89.3</td><td colspan="2">86.0 83.1</td></tr><tr><td>Coref.</td><td>95.8</td><td colspan="2">92.0 91.5</td></tr><tr><td>SPR</td><td>83.4</td><td colspan="2">83.1 82.6</td></tr><tr><td>Relations</td><td>77.7</td><td colspan="2">75.9 75.9</td></tr></table>

Peer review (iclr2020):
This paper proposes a new approach to learning contextualised cross-lingual representations based on a three-step procedure called: transform-extract-reorder. The proposed model is tested on zero-shot cross-lingual transfer for one language pair: English-Chinese, where the gains are reported over the initial multilingual BERT model which does not rely on any bilingual signal. I like the main modeling idea, and the actual model is described quite nicely and in an intuitive way. However, I believe that much more work is needed in terms of proper experiments including: a) additional strong and insightful baselines; b) experiments with more language pairs; c) experiments across other transfer tasks. The paper also does not include nor compares to all relevant previous work. My main comments and remarks are as follows: *Related work and Novelty* The topic of learning contextualised cross-lingual representations has become quite popular since the rise of ELMo and BERT. However, the authors cite only one paper that proposed such contextualised xling representations (the work of Schuster of al.). However, they never compare to that model, which is quite weird given the fact that the main goal of all xling representation models is to enable cross-lingual transfer in the first place. A comparison to the cited work of Schuster et al., plus additional comparisons to other (non-cited) models are required to clearly understand the main empirical benefits of the proposed three-step framework. Other relevant papers which are not cited nor compared against: Mulcaire et al. (NAACL 2019); Aldarmaki and Diab (NAACL 2019). The authors should really stress the key novelty of their approach in comparison to the existing literature (which is mostly based on simpler projection-based methods). - Note that there are more papers on the subject getting published soon such as Mulcaire et al. (CoNLL 2019) and Liu et al. (CoNLL 2019); while making comparisons to models introduced in those papers is not required, it would be nice to also briefly mention that latest work in the related work section once it gets published. *Comparisons to XLM* Based on the limited set of results reported in the paper, it seems that the proposed model cannot match the performance of the XLM model (Lample and Conneau, NeurIPS 2019). The authors try to justify the usage of their method over XLM by emphasising the fact that their method produces language-independent xling representations, which XLM cannot do. However, there are multiple issues here: 1) it is not clear why exactly this requirement of language-independence is needed for a task such as zero-shot XNLI; 2) the paper does not demonstrate the usefulness of that language-independence property empirically at all. To me, it just seems like an unsuccessful attempt to make a conceptual distinction between the proposed model and XLM given the fact that XLM seems to significantly outperform the proposed model. Page 8 (*Training cost and beyond*). The authors claim that 250k pairs of parallel sentences is enough to learn a near-optimal cross-lingual model and that the model saturates with more parallel sentences. I don*t see it as a positive aspect of the model as claimed by the authors. This means that the model cannot optimally encode knowledge about variety of cross-lingual contexts during its training, i.e., it stops learning earlier than expected. Why does that happen? A plot showing transfer learning results of XLM versus the proposed model versus some static word embedding model that also exploits parallel data would be very useful. The authors also do not even speculate why their model saturates already with 250k sentence pairs. Given the limited set of results (only one task and only language pair) it also remains unknown how general this phenomenon is and whether the same point is hit with some other language pairs and with other tasks. This definitely requires further and more thorough investigation. *Other Comparisons and Experiments* As mentioned, the paper is very limited when it comes to proper thorough evaluation. Most experiments focus on intrinsic/internal evaluations of different steps of the proposed framework, or some probing tests, which can also be seen simply as diagnostic experiments. The actual downstream experiments are conducted on only one language pair and for only one (zero-shot) transfer task. This is definitely not sufficient to draw any generalisable claims, and it prevents us to dig deeper into other interesting aspects of the model. I would suggest the authors to maybe run the model across the same range of tasks as done in the XLM work of Lample and Conneau, and definitely for more languages (ideally diverse target languages). Also, besides XLM and translation-based baselines reported by multilingual BERT*s github page, there is actually no other baselines, which really makes it hard to put this work in context, and understand its usefulness. For instance, static cross-lingual word embeddings could be added to ESIM and used to evaluate on XNLI as well (see, e.g., the work of Glavas et al. (ACL 2019)). It would be interesting to report the benefits of replacing such static vectors with truly contextualised representations. Also, as mentioned before, comparisons to other models that learn contextualised cross-lingual representations are definitely something that should be included in the paper. *Other Comments* - Based on the result from Table 4, it seems that, in order to enable fully language-independent representations, one must sacrifice some of the monolingual performance, as the numbers drop from T over T+E to T+E+R variant. Is the same pattern visible for other languages monolingually? Why does this happen? - As mentioned before, the whole emphasis is of language-independence is somewhat oversold in the whole paper without providing sufficient empirical evidence that this is crucial for transfer performance.