Title: LANGUAGE-INDEPENDENT CROSS-LINGUAL CONTEX-TUAL REPRESENTATIONS

Abstract: Contextual representation models like BERT have achieved state-of-the-art performance on a diverse range of NLP tasks. We propose a cross-lingual contextual representation model that generates language-independent contextual representations. This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models, on top of contextual representation models like BERT. We provide a formulation of language-independent cross-lingual contextual representation based on mono-lingual representations. Our formulation takes three steps to align sequences of vectors: transform, extract, and reorder. We present a detailed discussion about the process of learning cross-lingual contextual representations, also about the performance in cross-lingual transfer learning and its implications.

1. INTRODUCTION:

A cross-lingual text representation can generate language-independent representations: for pieces of text that are semantically similar, the produced representations are also similar, regardless of the language of the text. This provides a basis for dealing with multiple languages in NLP. Training a model on cross-lingual representations is a common approach for cross-lingual transfer learning (Huang et al., 2013; Artetxe & Schwenk, 2019) .

To achieve desirable transfer learning performance, the cross-lingual representation model should be effective in two aspects: 1) the quality of text representation; and 2) the quality of cross-lingual mapping.

The current state-of-the-art text representation models are a family of pre-trained language model representations such as BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019) . These models produce contextual representations of text, i.e., a sequence of vectors corresponding to the sequence of text tokens. Each vector represents "a token in context," e.g., the semantics of a word in its local context and in the global context of the sentence.

Research on cross-lingual representation has so far been largely concentrated on word embeddings (Mikolov et al., 2013; Gouws et al., 2015; Ruder et al., 2017) and sentence embeddings (Schwenk & Douze, 2017; Artetxe & Schwenk, 2019) . Word embeddings are static representations and are less expressive than contextual representations. Sentence embeddings perform respectably in some tasks but inferior in others, because there is information loss when too much information is crammed into a single vector (Conneau et al., 2018a) . However, word embeddings and sentence embeddings are relatively easy to generalize to cross-lingual representations because they do not have dynamic inter-dependency among vectors.

We regard it as an interesting topic to investigate cross-lingual contextual representations. First, it is an interesting problem in its own to match two (or more) sequences of vectors, where each vector sequence has its order and locality relationships. Second, we hope to further improve cross-lingual transfer learning by leveraging the state-of-the-art text representation models.

In this study, we try to clarify the definition and characteristics of cross-lingual contextual representations, and propose a formulation to learn a cross-lingual contextual representations model based on existing mono-lingual pre-trained language models. We perform experiments to analyze the effect of the ingredients in our formulation and the final performance in transfer learning.

We also hope our investigation help to understand the nature of text representations in different languages: what are the commonalities between representations, how to cover the gaps, and ultimately what is transferable across languages with the help of state-of-the-art representation models.

Related work:

Multilingual BERT learns a multi-lingual representation by training on a combined corpus of multiple languages. For dissimilar languages, the transfer performance is not satisfactory (Pires et al., 2019) . XLM (Lample & Conneau, 2019) uses parallel corpus and a translational objective, and has achieved good transfer performance for many languages. However, XLM does not produce the same representation for different languages, so there is no guarantee of the performance in transfer learning. In multi-way neural machine translation (Firat et al., 2016) the situation is similar: multiple encoders produce the representation for each language, but because the representations are not language-independent, the decoder still needs to be trained on multiple language inputs.

A recent work on aligning contextual representations is (Schuster et al., 2019) , which treats vectors in contextual representations as individual embeddings and learns a matrix to transform the embeddings space of different languages. The idea resembles the "transform" step in our formulation.

2. CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS:

First we define "language-independent cross-lingual contextual representations (CLCR)" and establish mathematical notations that we use in the following discussion.

2.1. DEFINITION:

A cross-lingual contextual representation (CLCR) of text is a parameterized representation model f (θ) : (a 0 , ..., a L ) → (r 0 , ..., r N ), where a 0 , ..., a L is the sequence of text tokens in a sentence s, and r 0 , ..., r N is the sequence of vectors produced by f as the representation of s. A languageindependent f satisfies the following conditions:

1. (cross-lingual correspondence) if sentence s i in i-th language and sentence s j in jth language are translations of each other, then by some distance metric D we have D(f (s i ; θ), f (s j ; θ)) < δ, where δ is a threshold.

2.. (isometric property:

) if for some semantic metric S, sentence s i in i-th language and sentence s j in j-th language satisfy S(s i , s j ) < γ, i.e., s i and s j are close in meaning, then by some distance metric D we have D(f (s i ; θ), f (s j ; θ)) < δ(γ), where δ is a monotonic increasing function of γ.

3.. (contextual representation:

) ∃f 1 , f 2 , f 3 s.t. r i = f 3 (f 2 (a i ), f 1 (a 0...L\i ))

. The vector r i can roughly be interpreted as a representation of token a i in the context of the whole sentence.

Condition 1 and 2 ensure that CLCR produces similar representations for sentences with similar meaning, independent of language. Condition 3 is a characteristic of contextual representation models such as pre-trained language models like BERT.

2.2. EXISTENCE OF CLCR:

To show that such cross-lingual contextual representation exists, we provide a formulation of CLCR based on mono-lingual contextual representations. We propose that one can generate CLCR with a three-step process. Starting from mono-lingual representations, each of the three steps corresponds to one problem that needs to be solved, in order to arrive at a cross-lingual contextual representation.

1. Alignment: the vectors of mono-lingual representation of different languages reside in different spaces. First we align the vector spaces R i with transformations T i : r i → r C , to transform into a common space R C .

2. Granularity: different languages have different token granularities. Some language use several tokens to represent a word. Token alignment between translations s i and s j can be represented by a set A = {((a i m , ..., a i n ), (a j p , ..., a j q ))}, where tokens (a i m , ..., a i n ) and (a j p , ..., a j q ) are aligned "phrases."

Definition of Finest Common Granularity: find the token alignment A * that cannot be broken into a finer alignment. The finest common granularity for s i w.r.t. language i and j is defined as a grouping of the token sequence of s i in A * : F CG(s i ) = {(a i 0 , ..., a i n0 ), (a i n0+1 , ..., a i n1 ), ..., (a i nm+1 , ..., a i L )}, where each element (a i ni+1 , ..., a i ni+1 ) ∈ A * . In a nutshell, we want to group a series of tokens (a i n0+1 , ..., a i n1 ) into a "phrase token" (key-point token), so that the key-point tokens have one-to-one correspondence between two languages. With key-point tokens derived from FCG, translations s i and s j have the same number of key-point tokens, and can then be represented by the same number of vectors. 3. Order: many different word orders exist in different languages, for example the SVO (subject-verb-object) and the SOV (subject-object-verb) order. We define an order mapping to map the orders of token from language i to language j: R i→j : m|s i → w, where ((a i nm+1 , ..., a i nm+1 ), (a j nw+1 , ..., a j nw+1 )) ∈ A. The order mapping R i→j maps the position of a key-point token in the source sentence to the position of the corresponding key-point token in the target language.

To generate cross-lingual representations, we perform alignment transformation on the vectors of two mono-lingual representations, find their common granularity, and then reorder the vectors, to make the two representations similar.

2.3. TRANSFER LEARNING WITH CLCR:

Language-independent cross-lingual representations enables zero-shot transfer learning across languages. CLCR allows the most versatile models to be trained on cross-lingual representations. For a model M taking contextual representations (r 0 , ..., r N ) as input, M can be trained on dataset {(s i , l)} in language i, by first using CLCR f to encode inputs: M (f (s i )) → l. In testing, model can perform inference on input in any language within f 's input languages.

3. LEARNING CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS:

In this section, we detail our approach to learn CLCR based on the formulation in Section 2.2. We use two mono-lingual contextual representation models on language 1 and 2 to learn a cross-lingual contextual representation model.

3.1. TRANSFORM:

In the first step, we learn a transformation T parameterized by a multi-layer network:

T (r) = Linear(r) + Linear(Relu(Linear(r)))

(1) The network is trained to minimize the discrepancy between the transformed representation in language 1 and the representation in language 2: D(T (r 1 0 , ..., r 1 M )||(r 2 0 , ..., r 2 N )). Because we do not have alignment between the two sequences of vectors yet, we define D with attention matching between the two sequences:

D(T (r 1 0 , ..., r 1 M )||(r 2 0 , ..., r 2 N )) = i 1 N ||T (Attention(r 2 i , (r 1 0 , ..., r 1 M )) -r 2 i || (2)

where Attention is defined as using r 2 i as query to perform a weighted-sum of (r 1 0 , ..., r 1 N ). To train transformation T , we minimize the above loss D on a parallel corpus {(s 1 , s 2 )}. We observed that straightforward gradient descent does not converge. We therefore used attention annealing during training to help our model converge to an optimal matching: sof tmax(a 0 , ..., a N ) = e a0/T i e ai/T

(3)

where

T = 1/(1 + β • step).

As training progresses sof tmax will gradually be replaced by max, which let the model find a one-to-one matching between the tokens. This is exactly required by the next step.

3.2. EXTRACT:

In this step, we want to extract "key-points" from a sentence. These key-points are summarizations of local information (which may consist of multiple tokens), and the granularity of "key-points" is ideally consistent across different languages.

We extract key-points based on finest common granularity (FCG) between two languages: for a group of tokens (a i n0+1 , ..., a i n1 ) ∈ F CG(s i ), we use a single vector r n0 to represent them. The number of vectors in the CLCR of s i is thus equal to |F CG(s i )|.

Next, we train a network P to predict key-points in F CG(s i ). Although we can generate F CG(s i ) by aligning parallel documents, in general CLCR should not rely on parallel documents and should predict key-points in a single sentence independently:

r i k is the representation for key-point token (a i n0+1 , ..., a i n1 ) (4) ⇐⇒ ∃! a i k ∈ {a i n0+1 , ..., a i n1 }, P (a i k ) = 1 (5)

The prediction network predicts one of the tokens in a key-point token to be the "pivot token." And the vector for that pivot token is regarded as the representation for the whole key-point token.

Finally, we need to generate ground-truth labels for training key-point prediction. In the definition above, the question "which word is the pivot token in a phrase?" do not have definite answers. Therefore, we train our representation model and let the model determine the pivot token in a keypoint token. This is achieved by forced alignment via annealing in Section 3.1. After training the transformations T , we calculate the pairwise distance between r 1 i and r 2 j :

D(T (r 1 i )||r 2 j ) + D(T (r 2 j )||r 1 i )

, and use a threshold to find aligned pivot tokens. Aligned pivot tokens (which is at the same time aligned key-point tokens) generated in this fashion closely resembles key-point tokens derived from FCG. As we do not have ground-truth word alignment of parallel corpus to generate real FCGs, we use our aligned key-points as pseudo-groundtruth alignments and use them as labels to train predictors in this and the next section.

One caveat of using parallel corpus is that translations are not unique. There can be multiple valid translations of a sentence. Thus labels derived from parallel corpus are intrinsically noisy and nonunique. We generate more reliable labels to train our prediction network, with a technique similar to (Liu & Tao, 2015) .

3.3. REORDER:

The aim of reordering is to make the order of contextual vectors (r 1 0 , ..., r 1 N ) in language 1 compatible with the order in language 2. This can be achieved by learning an order prediction model, to predict the order mapping R 1→2 . We propose two methods for order prediction:

• Absolute order prediction: learn a prediction model to directly predict the position of token a 1 i in language 2:

EQUATION

Inference:

minimize

EQUATION

• Relative order prediction: learn a model to predict the relative order of two tokens a 1 i and a 1 j in language 2:

EQUATION

Inference:

minimize

EQUATION

The learned order prediction is R 1→2 : i|s 1 → m i . The vectors in language 1 can be reordered according to R 1→2 , so that the reordered vectors have the order of language 2. The reordered vector sequence is (r 1 R2→1(0) , ..., r 1 R2→1(N ) ), where R 2→1 is the reverse mapping of R 1→2 .

4. EXPERIMENTS:

In this section, we perform experiments to learn language-independent cross-lingual contextual representation based on the approach in Section 3. We also perform transfer learning and analysis experiments to examine the characteristics of CLCR.

4.1. LEARNING:

We sequentially examine the details and the effectiveness of each of the three steps in our formulation to generate CLCR: transform, extract, and reorder.

The basic experiment setup is: we use mono-lingual pre-trained language models from Google BERT (Devlin et al., 2019) , for English and Chinese1 . The 12-layer BERT-base model is used. We use parallel corpus from The United Nations Parallel Corpus v1.0 (Ziemski et al., 2016) . Experiments are performed with Pytorch (Paszke et al., 2017) .

Transform As shown in Table 1 , training transformation T to minimize distance D with attention annealing managed to converge to a low distance between the two representations. Lower distance indicates that a better alignment is found (because unaligned and misaligned tokens always contribute to high distance). And a good alignment is also conducive to learning a better transformation. During training, we alternately optimize the representation model for language 1 and 2, so that they do not learn a trivial solution of producing the same constant representation. The learning curve is shown in Figure 1 .

The produced alignment is used as labels in the next two steps. 

< l a t e x i t s h a 1 _ b a s e 6 4 = " G z B y n A F d j f 9 T H C w 4 R z A A f e s s e O A = " >:

A A A B 9 X i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m I C F 9 K i i R 6 J e v C I C R 8 m U M h 2 2 c K G 7 b b Z 3 W p I 4 X 9 4 8 a A x X v 0 v 3 v w 3 L t C D g i + Z 5 O W 9 m c z M 8 y L O l L b t b y u z t r 6 x u Z X d z u 3 s 7 u 0 f 5 A + P m i q M J a E N E v J Q P n h Y U c 4 E b W i m O X 2 I J M W B x 2 n L G 9 3 M / N Y j l Y q F o q 7 H E X U D P B D M Z w R r I 3 V v i / W i 6 j q l y U R 1 K 6 V e v m C X 7 T n Q K n F S U o A U t V 7 + q 9 M P S R x Q o Q n H S r U d O 9 J u g q V m h N N p r h M r G m E y w g P a N l T g g C o 3 m V 8 9 R W d G 6 S M / l K a E R n P 1 9 0 S C A 6 X G g W c 6 A 6 y H a t m b i f 9 5 7 V j 7 V 2 7 C R B R r K s h i k R 9 z p E M 0 i w D 1 m a R E 8 7 E h m E h m b k V k i C U m 2 g S V M y E 4 y y + v k m a l 7 J y X K / c X h e p 1 G k c W T u A U i u D A J V T h D m r Q A A I S n u E V 3 q w n 6 8 V 6 t z 4 W r R k r n T m G P 7 A + f w C Q F p F B < / l a t e x i t > D(T (s 2 )||s 1 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " t z B m L R 1 K x 6 l T Z J W I B + S 5 E S c L z f o = " > A A A B 9 X i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m I C F 9 K i i R 6 J e v C I C R 8 m U M h 2 2 c K G 7 b b Z 3 W p I 4 X 9 4 8 a A x X v 0 v 3 v w 3 L t C D g i + Z 5 O W 9 m c z M 8 y L O l L b t b y u z t r 6 x u Z X d z u 3 s 7 u 0 f 5 A + P m i q M J a E N E v J Q P n h Y U c 4 E b W i m O X 2 I J M W B x 2 n L G 9 3 M / N Y j l Y q F o q 7 H E X U D P B D M Z w R r I 3 V v i / W i 6 l Z K k 4 n q O q V e v m C X 7 T n Q K n F S U o A U t V 7 + q 9 M P S R x Q o Q n H S r U d O 9 J u g q V m h N N p r h M r G m E y w g P a N l T g g C o 3 m V 8 9 R W d G 6 S M / l K a E R n P 1 9 0 S C A 6 X G g W c 6 A 6 y H a t m b i f 9 5 7 V j 7 V 2 7 C R B R r K s h i k R 9 z p E M 0 i w D 1 m a R E 8 7 E h m E h m b k V k i C U m 2 g S V M y E 4 y y + v k m a l 7 J y X K / c X h e p 1 G k c W T u A U i u D A J V T h D m r Q A A I S n u E V 3 q w n 6 8 V 6 t z 4 W r R k r n T m G P 7 A + f w C Q H J F B < / l a t e x i t > Phase 1

Phase 2 Phase 3

Distance Extract A binary classifier is trained to predict whether a token is a pivot token of a key-point. An example prediction is illustrated in Figure 2 . It can be observed that only tokens possess significant meaning is predicted as pivot tokens. For languages (e.g., Chinese) that use multiple tokens to represent a word, only one token per word is predicted as pivot tokens. Considering the labels are pseudo-groundtruth produced from only one translation per sentence, the overall prediction accuracy is respectable (see Table 2 ).

Using confident labels slightly improves prediction accuracy, and makes the prediction significantly less ambiguous. This is very important for producing consistent predictions of key-points, such that transfer learning models on top of CLCR can be trained on a consistent representation. Reorder In absolute order prediction, we use an N-way classifier to predict the absolute position of a token. In relative order prediction, a binary classifier is used to predict the relative order of a pair of tokens. During inference, beam search is used to find the best order assignment of tokens, so that the total cost is minimized. We simply use the probability p of the classifier output as the cost of a particular assignment.

Table 3 reports the reordering performance of two methods with different beam size n. We calculate 2-gram to 4-gram BLEU with respect to the groundtruth order. Relative order prediction performs much better than absolute order prediction, getting 2-gram predictions correct over half of the time and 4-gram predictions correct nearly one-third of the time. The example in Figure 3 illustrates that absolute order prediction suffers from ambiguity except near the beginning of the sentence, because the exact position depends on the specific translation and therefore does not have a unique correct answer. Relative order prediction is much more robust because it is invariant to the location of the pair of tokens in the sentence, the classifier only makes predictions based on the relative order relation between them.

4.2. CROSS-LINGUAL TRANSFER:

Natural Language inference XNLI (Conneau et al., 2018b ) is a cross-lingual natural language inference corpus consisting of test examples in 15 languages. We use the English and Chinese test set in our evaluation. We use MultiNLI (Williams et al., 2018) , an English NLI dataset as the training set to train the task model. 1994

年 ， 我 曾 任 惠 特 尼 博 物 馆 馆 长 时 举 办 了 一 个 叫 做 " 黑 人 男 性 " 的 展 览

In To evaluate the performance of our cross-lingual contextual representation model, we train the ESIM model (Chen et al., 2017) with representation produced by CLCR as input. An advantage of CLCR over multi-lingual pre-trained language models is that complex task models can be trained on top of the representation. To examine the effect of each of the three steps in our formulation of CLCR, we also experimented with different combinations of steps.

Our full CLCR model (T+E+R) achieved a transfer accuracy of 71.7%, much better than multilingual BERT and even better than translating test set to English. The transfer gap is on par with state-ofthe-art models (Table 4 ).

It is clear that CLCR works best when all three steps are present. Without extraction and reordering, model on top of CLCR does not perform as well. This is because the ESIM model makes use of order information of the input sequence, the number and the order of vectors should match for successful zero-shot transfer to another language.

It is also observed that extraction and reordering lower performance on English by roughly 1%. The performance loss is expected, and not beyond acceptable range. During the training of CLCR, the English representation model is finetuned to accommodate another language, and during imperfect extraction and reordering some information is inevitably lost.

We also set an upper bound for the transfer performance of CLCR by using pseudo-groundtruth alignments (T+pGA). Pseudo-groundtruth alignments give the "correct" extraction and reordering, and indicates the performance of the best possible T+E+R model. The remaining transfer gap (6.4%) is thus intrinsic to the task (variation of test set difficulty) and the mono-lingual representation models (different performance of BERT-base-en and BERT-base-zh). Although the performance of our CLCR on Chinese XNLI is inferior to XLM, the transfer gap is roughly the same. XLM uses much more data and training time than our approach, and thus starts with better performance than BERT-base on English. Also XLM is not language-independent, which means it is not feasible to use XLM as representations to learn models for cross-lingual transfer. One can only fine-tune XLM itself on a language and hope that other languages also benefit from finetuning. (Tenney et al., 2019b) . In edge probing, a series of tasks are used as "probes" to measure the syntactic and semantic information contained in representations. The jiant toolkit (Wang et al., 2019) is used to perform the experiments.

4.3. ANALYSIS:

Comparing CLCR with the original BERT-base English model in Table 5 , the transformation step (T) generally lowers scores of most probes by a small fraction. This indicates while most information remains in the representation after training, there is a certain portion of incompatible syntactic and semantic representations across languages. That information is lost and explains the performance loss on XLI on the source language (English). Extraction (E) further lowers scores, but mainly due to failing to predict labels on non-pivot tokens. Because the probing model used in (Tenney et al., 2019b ) is an attention pooling-based order-less model, reordering (R) does not alter the performance on these probing tasks.

Training cost and beyond A total of 250,000 pairs of parallel sentences are used in this study. This is less than 2% of the data used to train XLM. Increasing the size of parallel corpus does not lead to further improvement. Also the total training time of CLCR, including the three steps is less than 5 hours, on a single GPU. This indicates that with good mono-lingual contextual representations like BERT, learning cross-lingual alignment of representations could be a relatively easy task, without the need for extensive supervision and training. Under our formulation, the performance of CLCR is built upon and also limited by the quality of mono-lingual representations. But this more of an opportunity than a problem because parallel corpus is always the more limited resource.

5. CONCLUSION:

In this paper we investigated language-independent cross-lingual contextual representations. We proposed a three-step process to learn cross-lingual contextual representations from mono-lingual representations. In each of the three steps, an intuitive method is employed with the aim of letting the model itself learn to ultimately align two representations. Experiments confirmed that stateof-the-art pre-trained language models are powerful enough to learn to align two languages with moderate supervision. This could indicate that with models like BERT now it is easier than ever to cover the gap in cross-lingual transfer learning.

Our formulation provides a new approach to combine contextual representations with arbitrary task models for zero-shot cross-lingual transfer learning. We experimented with the NLI task and the ESIM model, but the method could in principle be applied to a wide range of tasks and models.

With the advancement of pre-trained language models and their availability, we hope to extend the approach to more languages and more powerful models. We also hope our investigation could inspire further insight into the nature of language-independent representations and cross-lingual transfer learning.

A METHOD OVERVIEW:

We provide an overview diagram to show the whole process of generating language-independent cross-lingual contextual representations, using an example in Figure 4 .

There was no alternative but to develop a new international agreement on climate change to replace the existing Kyoto Protocol . B ZERO-SHOT CROSS-LINGUAL TRANSFER Language-independent cross-lingual contextual representations enable zero-shot cross-lingual transfer for a variety of NLP models. For example, with learned CLCR between English and Chinese, one can train a task model with Embed-E on English, then perform inference with Embed-T-E-R on Chinese (Figure 5 ). 

:

https://github.com/google-research/bert

Figures and Tables:

(s 1 )||s 2 )

Figure 1: Learning curve of distance D. Training phase 1 (left): fix both model, learn T only. Training phase 2 (middle): finetune language 1 representation model with annealing. Training phase 3 (right): finetune language 2 representation model with annealing.

Figure 2: An example of predicted key-points, with (below) and without (above) confident labels

Figure 3: A example of order prediction. From left to right: (1) ground-truth absolute order. (2) predicted absolute order. (3) ground-truth relative order. (4) predicted relative order.

Figure 4: An example illustrating CLCR in inference. For a pair of language (in this example, English and Chinese), one can decide to perform transformation and reordering on one side, and perform extraction on both sides. After training of the transformation (T), extraction (E) and reordering (R) model, for input text in one of the two languages, a sequence of operation is performed: Embed-T-E-R or Embed-E depending on the language. The generated sequence-of-vector representation (in the box with dashed line) is the same for text in both languages, assuming perfect T, E, and R models.

Figure 5: An example of zero-shot cross-lingual transfer with CLCR



Table content: <table><tr><td>Linear Regression</td><td>0.055</td><td>0.078</td></tr><tr><td>Attention (w/o annealing T = 1)</td><td>0.138</td><td>0.207</td></tr><tr><td>Attention (with annealing β = 0.001)</td><td>0.028</td><td>0.048</td></tr><tr><td>Attention (with annealing β = 0.01)</td><td>0.025</td><td>0.037</td></tr></table>

Key-point prediction accuracy

Table content: <table><tr><td/><td/><td/><td/><td colspan="3">Model</td><td/><td/><td>Accuracy en zh</td><td/><td/><td/><td>Precision en zh</td><td>Recall en</td><td>zh</td></tr><tr><td/><td/><td/><td colspan="4">Finetune</td><td/><td colspan="7">0.859 0.823 0.886 0.854 0.936 0.847</td></tr><tr><td/><td/><td/><td colspan="12">Confident labels 0.864 0.826 0.875 0.838 0.959 0.872</td></tr><tr><td>A n d t h a t '</td><td>s</td><td>,</td><td>k in d o f</td><td>,</td><td>w h e r e I</td><td>s e e w h a t m y</td><td>j o b</td><td>is</td><td>a n d w h a t o u r t e a m '</td><td>s</td><td>j o b</td><td>is</td><td>.</td></tr><tr><td>A n d t h a t '</td><td>s</td><td>,</td><td>k in d o f</td><td>,</td><td>w h e r e I</td><td>s e e w h a t m y</td><td>j o b</td><td>is</td><td>a n d w h a t o u r t e m a '</td><td>s</td><td>j o b</td><td>is</td><td>.</td></tr></table>

Reorder performance

Table content: <table><tr><td>Model</td><td colspan="6">BLEU (2-gram) BLEU (3-gram) BLEU (4-gram) en zh en zh en zh</td></tr><tr><td>Absolute order prediction (n = 1)</td><td>0.309</td><td>0.386</td><td>0.166</td><td>0.227</td><td>0.101</td><td>0.152</td></tr><tr><td colspan="2">Absolute order prediction (n = 10) 0.331</td><td>0.393</td><td>0.189</td><td>0.239</td><td>0.121</td><td>0.165</td></tr><tr><td>Relative order prediction (n = 1)</td><td>0.535</td><td>0.566</td><td>0.384</td><td>0.410</td><td>0.290</td><td>0.308</td></tr><tr><td>Relative order prediction (n = 10)</td><td>0.582</td><td>0.624</td><td>0.419</td><td>0.468</td><td>0.316</td><td>0.368</td></tr></table>

Performance of zero-shot cross-lingual transfer. T, E, R stands for Transform, Extract, and Reorder, respectively. pGA stands for pseudo-groundtruth alignment. Gap stands for the transfer gap from English to Chinese (the drop of accuracy in percentage)

Table content: <table><tr><td>Model</td><td colspan="3">XNLI test set en Acc. zh Acc. Gap</td></tr><tr><td>CLCR</td><td/><td/><td/></tr><tr><td>T+pGA</td><td>81.2</td><td>74.8</td><td>6.4</td></tr><tr><td>T</td><td>81.2</td><td>68.0</td><td>13.2</td></tr><tr><td>T+E</td><td>80.6</td><td>69.3</td><td>11.3</td></tr><tr><td>T+E+R</td><td>80.2</td><td>71.7</td><td>8.5</td></tr><tr><td>Non-CLCR models</td><td/><td/><td/></tr><tr><td>Multilingual BERT</td><td>81.4</td><td>63.8</td><td>17.6</td></tr><tr><td>Translate train</td><td>81.4</td><td>74.2</td><td>7.2</td></tr><tr><td>Translate test</td><td>81.4</td><td>70.1</td><td>11.3</td></tr><tr><td>XLM</td><td>85.0</td><td>76.5</td><td>8.5</td></tr></table>



Table content: <table><tr><td colspan="4">: Edge probing statistics of CLCR</td></tr><tr><td/><td/><td>F1 score</td></tr><tr><td/><td>BERT</td><td>T</td><td>T+E</td></tr><tr><td>POS</td><td>97.6</td><td colspan="2">92.4 82.2</td></tr><tr><td>Consts.</td><td>86.8</td><td colspan="2">80.8 75.4</td></tr><tr><td>Deps.</td><td>95.4</td><td colspan="2">92.8 84.5</td></tr><tr><td>Entities</td><td>96.1</td><td colspan="2">94.8 90.9</td></tr><tr><td>SRL</td><td>89.3</td><td colspan="2">86.0 83.1</td></tr><tr><td>Coref.</td><td>95.8</td><td colspan="2">92.0 91.5</td></tr><tr><td>SPR</td><td>83.4</td><td colspan="2">83.1 82.6</td></tr><tr><td>Relations</td><td>77.7</td><td colspan="2">75.9 75.9</td></tr></table>

Peer review (iclr2020):
This paper proposes a method to learn language-independent cross-lingual contextual representations by mapping the representations of a monolingual model in one language to the representations of a monolingual model in another language. The proposed approach consists of three steps: 1. A transformation is learned that minimizes the distance between the contextual word representations of the two models of a sentence and its translation in a parallel corpus. 2. Each sentence is summarized as a sequence of key-point tokens based on phrase alignment. 3. The contextual word vectors are reordered based on an order prediction model. The authors perform experiments on intrinsic tasks and on XNLI, mapping between an English and a Chinese BERT model. They outperform multilingual BERT (mBERT) on the latter. Overall, the approach is novel, but the steps seem overly complicated. The extrinsic evaluation is the weakest point of the paper. Because of this, I tend to a Weak Reject. I would be willing to increase my score if additional languages are added to the evaluation and if the steps are better motivated and compared to simpler alternatives. The high-level steps of the approach (transform, align, reorder) make sense. They seem to be inspired by classical phrase-based MT pipelines, but this connection is not made clear. In particular, some of the steps seem unnecessarily complicated and I am wondering whether the authors tried or compared to simpler alternatives. As a parallel corpus is used in the first step, word alignment could be automatically obtained by FastAlign without the use of attention matching or using a phrase table that is learned in an unsupervised way as in recent work in NMT (https://arxiv.org/pdf/1804.07755.pdf). For the second step, embeddings of aligned phrases could be averaged or the head of a phrase could be used instead of predicting key-points with a separate network. The data requirements between the steps seem somewhat inconsistent. A parallel corpus is used in the first step, but explicitly not used in the second step. While I agree that language-independent cross-lingual representations should not use a parallel corpus, it would have been good if the first step could have also been performed without one (or with a smaller size). The extrinsic evaluation of the paper could be improved. The accuracy on each intrinsic step is evaluated. Without any reference or comparison, I found it hard to tell how good these numbers are so this evaluation did not add much for me. The main extrinsic evaluation of the paper is on XNLI but only employs an English and a Chinese model. When training cross-lingual models, the aim is to train approaches that not only work for one or two languages but for many. In light of this, I find it hard to tell from the results on a single language pair how well the approach will generalize to other languages, particularly as mBERT*s performance on zh XNLI is comparatively weak (https://arxiv.org/pdf/1901.07291.pdf). There are other publicly available BERT models such as for German (https://deepset.ai/german-bert) on which the approach could be tried. If compute is a concern, then the approach could be applied to ELMo representations. This would also enable a comparison to Schuster et al. (2019; https://arxiv.org/abs/1902.09492). Finally, the data requirement of 250k parallel sentences is prohibitive for many language pairs where this approach would be valuable. It would be good to see a chart on how model performance develops with the number of parallel sentences to see if a smaller number of parallel sentences would be viable in practice.