Title: LANGUAGE-INDEPENDENT CROSS-LINGUAL CONTEX-TUAL REPRESENTATIONS

Abstract: Contextual representation models like BERT have achieved state-of-the-art performance on a diverse range of NLP tasks. We propose a cross-lingual contextual representation model that generates language-independent contextual representations. This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models, on top of contextual representation models like BERT. We provide a formulation of language-independent cross-lingual contextual representation based on mono-lingual representations. Our formulation takes three steps to align sequences of vectors: transform, extract, and reorder. We present a detailed discussion about the process of learning cross-lingual contextual representations, also about the performance in cross-lingual transfer learning and its implications.

1. INTRODUCTION:

A cross-lingual text representation can generate language-independent representations: for pieces of text that are semantically similar, the produced representations are also similar, regardless of the language of the text. This provides a basis for dealing with multiple languages in NLP. Training a model on cross-lingual representations is a common approach for cross-lingual transfer learning (Huang et al., 2013; Artetxe & Schwenk, 2019) .

To achieve desirable transfer learning performance, the cross-lingual representation model should be effective in two aspects: 1) the quality of text representation; and 2) the quality of cross-lingual mapping.

The current state-of-the-art text representation models are a family of pre-trained language model representations such as BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019) . These models produce contextual representations of text, i.e., a sequence of vectors corresponding to the sequence of text tokens. Each vector represents "a token in context," e.g., the semantics of a word in its local context and in the global context of the sentence.

Research on cross-lingual representation has so far been largely concentrated on word embeddings (Mikolov et al., 2013; Gouws et al., 2015; Ruder et al., 2017) and sentence embeddings (Schwenk & Douze, 2017; Artetxe & Schwenk, 2019) . Word embeddings are static representations and are less expressive than contextual representations. Sentence embeddings perform respectably in some tasks but inferior in others, because there is information loss when too much information is crammed into a single vector (Conneau et al., 2018a) . However, word embeddings and sentence embeddings are relatively easy to generalize to cross-lingual representations because they do not have dynamic inter-dependency among vectors.

We regard it as an interesting topic to investigate cross-lingual contextual representations. First, it is an interesting problem in its own to match two (or more) sequences of vectors, where each vector sequence has its order and locality relationships. Second, we hope to further improve cross-lingual transfer learning by leveraging the state-of-the-art text representation models.

In this study, we try to clarify the definition and characteristics of cross-lingual contextual representations, and propose a formulation to learn a cross-lingual contextual representations model based on existing mono-lingual pre-trained language models. We perform experiments to analyze the effect of the ingredients in our formulation and the final performance in transfer learning.

We also hope our investigation help to understand the nature of text representations in different languages: what are the commonalities between representations, how to cover the gaps, and ultimately what is transferable across languages with the help of state-of-the-art representation models.

Related work:

Multilingual BERT learns a multi-lingual representation by training on a combined corpus of multiple languages. For dissimilar languages, the transfer performance is not satisfactory (Pires et al., 2019) . XLM (Lample & Conneau, 2019) uses parallel corpus and a translational objective, and has achieved good transfer performance for many languages. However, XLM does not produce the same representation for different languages, so there is no guarantee of the performance in transfer learning. In multi-way neural machine translation (Firat et al., 2016) the situation is similar: multiple encoders produce the representation for each language, but because the representations are not language-independent, the decoder still needs to be trained on multiple language inputs.

A recent work on aligning contextual representations is (Schuster et al., 2019) , which treats vectors in contextual representations as individual embeddings and learns a matrix to transform the embeddings space of different languages. The idea resembles the "transform" step in our formulation.

2. CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS:

First we define "language-independent cross-lingual contextual representations (CLCR)" and establish mathematical notations that we use in the following discussion.

2.1. DEFINITION:

A cross-lingual contextual representation (CLCR) of text is a parameterized representation model f (θ) : (a 0 , ..., a L ) → (r 0 , ..., r N ), where a 0 , ..., a L is the sequence of text tokens in a sentence s, and r 0 , ..., r N is the sequence of vectors produced by f as the representation of s. A languageindependent f satisfies the following conditions:

1. (cross-lingual correspondence) if sentence s i in i-th language and sentence s j in jth language are translations of each other, then by some distance metric D we have D(f (s i ; θ), f (s j ; θ)) < δ, where δ is a threshold.

2.. (isometric property:

) if for some semantic metric S, sentence s i in i-th language and sentence s j in j-th language satisfy S(s i , s j ) < γ, i.e., s i and s j are close in meaning, then by some distance metric D we have D(f (s i ; θ), f (s j ; θ)) < δ(γ), where δ is a monotonic increasing function of γ.

3.. (contextual representation:

) ∃f 1 , f 2 , f 3 s.t. r i = f 3 (f 2 (a i ), f 1 (a 0...L\i ))

. The vector r i can roughly be interpreted as a representation of token a i in the context of the whole sentence.

Condition 1 and 2 ensure that CLCR produces similar representations for sentences with similar meaning, independent of language. Condition 3 is a characteristic of contextual representation models such as pre-trained language models like BERT.

2.2. EXISTENCE OF CLCR:

To show that such cross-lingual contextual representation exists, we provide a formulation of CLCR based on mono-lingual contextual representations. We propose that one can generate CLCR with a three-step process. Starting from mono-lingual representations, each of the three steps corresponds to one problem that needs to be solved, in order to arrive at a cross-lingual contextual representation.

1. Alignment: the vectors of mono-lingual representation of different languages reside in different spaces. First we align the vector spaces R i with transformations T i : r i → r C , to transform into a common space R C .

2. Granularity: different languages have different token granularities. Some language use several tokens to represent a word. Token alignment between translations s i and s j can be represented by a set A = {((a i m , ..., a i n ), (a j p , ..., a j q ))}, where tokens (a i m , ..., a i n ) and (a j p , ..., a j q ) are aligned "phrases."

Definition of Finest Common Granularity: find the token alignment A * that cannot be broken into a finer alignment. The finest common granularity for s i w.r.t. language i and j is defined as a grouping of the token sequence of s i in A * : F CG(s i ) = {(a i 0 , ..., a i n0 ), (a i n0+1 , ..., a i n1 ), ..., (a i nm+1 , ..., a i L )}, where each element (a i ni+1 , ..., a i ni+1 ) ∈ A * . In a nutshell, we want to group a series of tokens (a i n0+1 , ..., a i n1 ) into a "phrase token" (key-point token), so that the key-point tokens have one-to-one correspondence between two languages. With key-point tokens derived from FCG, translations s i and s j have the same number of key-point tokens, and can then be represented by the same number of vectors. 3. Order: many different word orders exist in different languages, for example the SVO (subject-verb-object) and the SOV (subject-object-verb) order. We define an order mapping to map the orders of token from language i to language j: R i→j : m|s i → w, where ((a i nm+1 , ..., a i nm+1 ), (a j nw+1 , ..., a j nw+1 )) ∈ A. The order mapping R i→j maps the position of a key-point token in the source sentence to the position of the corresponding key-point token in the target language.

To generate cross-lingual representations, we perform alignment transformation on the vectors of two mono-lingual representations, find their common granularity, and then reorder the vectors, to make the two representations similar.

2.3. TRANSFER LEARNING WITH CLCR:

Language-independent cross-lingual representations enables zero-shot transfer learning across languages. CLCR allows the most versatile models to be trained on cross-lingual representations. For a model M taking contextual representations (r 0 , ..., r N ) as input, M can be trained on dataset {(s i , l)} in language i, by first using CLCR f to encode inputs: M (f (s i )) → l. In testing, model can perform inference on input in any language within f 's input languages.

3. LEARNING CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS:

In this section, we detail our approach to learn CLCR based on the formulation in Section 2.2. We use two mono-lingual contextual representation models on language 1 and 2 to learn a cross-lingual contextual representation model.

3.1. TRANSFORM:

In the first step, we learn a transformation T parameterized by a multi-layer network:

T (r) = Linear(r) + Linear(Relu(Linear(r)))

(1) The network is trained to minimize the discrepancy between the transformed representation in language 1 and the representation in language 2: D(T (r 1 0 , ..., r 1 M )||(r 2 0 , ..., r 2 N )). Because we do not have alignment between the two sequences of vectors yet, we define D with attention matching between the two sequences:

D(T (r 1 0 , ..., r 1 M )||(r 2 0 , ..., r 2 N )) = i 1 N ||T (Attention(r 2 i , (r 1 0 , ..., r 1 M )) -r 2 i || (2)

where Attention is defined as using r 2 i as query to perform a weighted-sum of (r 1 0 , ..., r 1 N ). To train transformation T , we minimize the above loss D on a parallel corpus {(s 1 , s 2 )}. We observed that straightforward gradient descent does not converge. We therefore used attention annealing during training to help our model converge to an optimal matching: sof tmax(a 0 , ..., a N ) = e a0/T i e ai/T

(3)

where

T = 1/(1 + β • step).

As training progresses sof tmax will gradually be replaced by max, which let the model find a one-to-one matching between the tokens. This is exactly required by the next step.

3.2. EXTRACT:

In this step, we want to extract "key-points" from a sentence. These key-points are summarizations of local information (which may consist of multiple tokens), and the granularity of "key-points" is ideally consistent across different languages.

We extract key-points based on finest common granularity (FCG) between two languages: for a group of tokens (a i n0+1 , ..., a i n1 ) ∈ F CG(s i ), we use a single vector r n0 to represent them. The number of vectors in the CLCR of s i is thus equal to |F CG(s i )|.

Next, we train a network P to predict key-points in F CG(s i ). Although we can generate F CG(s i ) by aligning parallel documents, in general CLCR should not rely on parallel documents and should predict key-points in a single sentence independently:

r i k is the representation for key-point token (a i n0+1 , ..., a i n1 ) (4) ⇐⇒ ∃! a i k ∈ {a i n0+1 , ..., a i n1 }, P (a i k ) = 1 (5)

The prediction network predicts one of the tokens in a key-point token to be the "pivot token." And the vector for that pivot token is regarded as the representation for the whole key-point token.

Finally, we need to generate ground-truth labels for training key-point prediction. In the definition above, the question "which word is the pivot token in a phrase?" do not have definite answers. Therefore, we train our representation model and let the model determine the pivot token in a keypoint token. This is achieved by forced alignment via annealing in Section 3.1. After training the transformations T , we calculate the pairwise distance between r 1 i and r 2 j :

D(T (r 1 i )||r 2 j ) + D(T (r 2 j )||r 1 i )

, and use a threshold to find aligned pivot tokens. Aligned pivot tokens (which is at the same time aligned key-point tokens) generated in this fashion closely resembles key-point tokens derived from FCG. As we do not have ground-truth word alignment of parallel corpus to generate real FCGs, we use our aligned key-points as pseudo-groundtruth alignments and use them as labels to train predictors in this and the next section.

One caveat of using parallel corpus is that translations are not unique. There can be multiple valid translations of a sentence. Thus labels derived from parallel corpus are intrinsically noisy and nonunique. We generate more reliable labels to train our prediction network, with a technique similar to (Liu & Tao, 2015) .

3.3. REORDER:

The aim of reordering is to make the order of contextual vectors (r 1 0 , ..., r 1 N ) in language 1 compatible with the order in language 2. This can be achieved by learning an order prediction model, to predict the order mapping R 1→2 . We propose two methods for order prediction:

• Absolute order prediction: learn a prediction model to directly predict the position of token a 1 i in language 2:

EQUATION

Inference:

minimize

EQUATION

• Relative order prediction: learn a model to predict the relative order of two tokens a 1 i and a 1 j in language 2:

EQUATION

Inference:

minimize

EQUATION

The learned order prediction is R 1→2 : i|s 1 → m i . The vectors in language 1 can be reordered according to R 1→2 , so that the reordered vectors have the order of language 2. The reordered vector sequence is (r 1 R2→1(0) , ..., r 1 R2→1(N ) ), where R 2→1 is the reverse mapping of R 1→2 .

4. EXPERIMENTS:

In this section, we perform experiments to learn language-independent cross-lingual contextual representation based on the approach in Section 3. We also perform transfer learning and analysis experiments to examine the characteristics of CLCR.

4.1. LEARNING:

We sequentially examine the details and the effectiveness of each of the three steps in our formulation to generate CLCR: transform, extract, and reorder.

The basic experiment setup is: we use mono-lingual pre-trained language models from Google BERT (Devlin et al., 2019) , for English and Chinese1 . The 12-layer BERT-base model is used. We use parallel corpus from The United Nations Parallel Corpus v1.0 (Ziemski et al., 2016) . Experiments are performed with Pytorch (Paszke et al., 2017) .

Transform As shown in Table 1 , training transformation T to minimize distance D with attention annealing managed to converge to a low distance between the two representations. Lower distance indicates that a better alignment is found (because unaligned and misaligned tokens always contribute to high distance). And a good alignment is also conducive to learning a better transformation. During training, we alternately optimize the representation model for language 1 and 2, so that they do not learn a trivial solution of producing the same constant representation. The learning curve is shown in Figure 1 .

The produced alignment is used as labels in the next two steps. 

< l a t e x i t s h a 1 _ b a s e 6 4 = " G z B y n A F d j f 9 T H C w 4 R z A A f e s s e O A = " >:

A A A B 9 X i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m I C F 9 K i i R 6 J e v C I C R 8 m U M h 2 2 c K G 7 b b Z 3 W p I 4 X 9 4 8 a A x X v 0 v 3 v w 3 L t C D g i + Z 5 O W 9 m c z M 8 y L O l L b t b y u z t r 6 x u Z X d z u 3 s 7 u 0 f 5 A + P m i q M J a E N E v J Q P n h Y U c 4 E b W i m O X 2 I J M W B x 2 n L G 9 3 M / N Y j l Y q F o q 7 H E X U D P B D M Z w R r I 3 V v i / W i 6 j q l y U R 1 K 6 V e v m C X 7 T n Q K n F S U o A U t V 7 + q 9 M P S R x Q o Q n H S r U d O 9 J u g q V m h N N p r h M r G m E y w g P a N l T g g C o 3 m V 8 9 R W d G 6 S M / l K a E R n P 1 9 0 S C A 6 X G g W c 6 A 6 y H a t m b i f 9 5 7 V j 7 V 2 7 C R B R r K s h i k R 9 z p E M 0 i w D 1 m a R E 8 7 E h m E h m b k V k i C U m 2 g S V M y E 4 y y + v k m a l 7 J y X K / c X h e p 1 G k c W T u A U i u D A J V T h D m r Q A A I S n u E V 3 q w n 6 8 V 6 t z 4 W r R k r n T m G P 7 A + f w C Q F p F B < / l a t e x i t > D(T (s 2 )||s 1 ) < l a t e x i t s h a 1 _ b a s e 6 4 = " t z B m L R 1 K x 6 l T Z J W I B + S 5 E S c L z f o = " > A A A B 9 X i c b V B N T 8 J A E J 3 i F + I X 6 t H L R m I C F 9 K i i R 6 J e v C I C R 8 m U M h 2 2 c K G 7 b b Z 3 W p I 4 X 9 4 8 a A x X v 0 v 3 v w 3 L t C D g i + Z 5 O W 9 m c z M 8 y L O l L b t b y u z t r 6 x u Z X d z u 3 s 7 u 0 f 5 A + P m i q M J a E N E v J Q P n h Y U c 4 E b W i m O X 2 I J M W B x 2 n L G 9 3 M / N Y j l Y q F o q 7 H E X U D P B D M Z w R r I 3 V v i / W i 6 l Z K k 4 n q O q V e v m C X 7 T n Q K n F S U o A U t V 7 + q 9 M P S R x Q o Q n H S r U d O 9 J u g q V m h N N p r h M r G m E y w g P a N l T g g C o 3 m V 8 9 R W d G 6 S M / l K a E R n P 1 9 0 S C A 6 X G g W c 6 A 6 y H a t m b i f 9 5 7 V j 7 V 2 7 C R B R r K s h i k R 9 z p E M 0 i w D 1 m a R E 8 7 E h m E h m b k V k i C U m 2 g S V M y E 4 y y + v k m a l 7 J y X K / c X h e p 1 G k c W T u A U i u D A J V T h D m r Q A A I S n u E V 3 q w n 6 8 V 6 t z 4 W r R k r n T m G P 7 A + f w C Q H J F B < / l a t e x i t > Phase 1

Phase 2 Phase 3

Distance Extract A binary classifier is trained to predict whether a token is a pivot token of a key-point. An example prediction is illustrated in Figure 2 . It can be observed that only tokens possess significant meaning is predicted as pivot tokens. For languages (e.g., Chinese) that use multiple tokens to represent a word, only one token per word is predicted as pivot tokens. Considering the labels are pseudo-groundtruth produced from only one translation per sentence, the overall prediction accuracy is respectable (see Table 2 ).

Using confident labels slightly improves prediction accuracy, and makes the prediction significantly less ambiguous. This is very important for producing consistent predictions of key-points, such that transfer learning models on top of CLCR can be trained on a consistent representation. Reorder In absolute order prediction, we use an N-way classifier to predict the absolute position of a token. In relative order prediction, a binary classifier is used to predict the relative order of a pair of tokens. During inference, beam search is used to find the best order assignment of tokens, so that the total cost is minimized. We simply use the probability p of the classifier output as the cost of a particular assignment.

Table 3 reports the reordering performance of two methods with different beam size n. We calculate 2-gram to 4-gram BLEU with respect to the groundtruth order. Relative order prediction performs much better than absolute order prediction, getting 2-gram predictions correct over half of the time and 4-gram predictions correct nearly one-third of the time. The example in Figure 3 illustrates that absolute order prediction suffers from ambiguity except near the beginning of the sentence, because the exact position depends on the specific translation and therefore does not have a unique correct answer. Relative order prediction is much more robust because it is invariant to the location of the pair of tokens in the sentence, the classifier only makes predictions based on the relative order relation between them.

4.2. CROSS-LINGUAL TRANSFER:

Natural Language inference XNLI (Conneau et al., 2018b ) is a cross-lingual natural language inference corpus consisting of test examples in 15 languages. We use the English and Chinese test set in our evaluation. We use MultiNLI (Williams et al., 2018) , an English NLI dataset as the training set to train the task model. 1994

年 ， 我 曾 任 惠 特 尼 博 物 馆 馆 长 时 举 办 了 一 个 叫 做 " 黑 人 男 性 " 的 展 览

In To evaluate the performance of our cross-lingual contextual representation model, we train the ESIM model (Chen et al., 2017) with representation produced by CLCR as input. An advantage of CLCR over multi-lingual pre-trained language models is that complex task models can be trained on top of the representation. To examine the effect of each of the three steps in our formulation of CLCR, we also experimented with different combinations of steps.

Our full CLCR model (T+E+R) achieved a transfer accuracy of 71.7%, much better than multilingual BERT and even better than translating test set to English. The transfer gap is on par with state-ofthe-art models (Table 4 ).

It is clear that CLCR works best when all three steps are present. Without extraction and reordering, model on top of CLCR does not perform as well. This is because the ESIM model makes use of order information of the input sequence, the number and the order of vectors should match for successful zero-shot transfer to another language.

It is also observed that extraction and reordering lower performance on English by roughly 1%. The performance loss is expected, and not beyond acceptable range. During the training of CLCR, the English representation model is finetuned to accommodate another language, and during imperfect extraction and reordering some information is inevitably lost.

We also set an upper bound for the transfer performance of CLCR by using pseudo-groundtruth alignments (T+pGA). Pseudo-groundtruth alignments give the "correct" extraction and reordering, and indicates the performance of the best possible T+E+R model. The remaining transfer gap (6.4%) is thus intrinsic to the task (variation of test set difficulty) and the mono-lingual representation models (different performance of BERT-base-en and BERT-base-zh). Although the performance of our CLCR on Chinese XNLI is inferior to XLM, the transfer gap is roughly the same. XLM uses much more data and training time than our approach, and thus starts with better performance than BERT-base on English. Also XLM is not language-independent, which means it is not feasible to use XLM as representations to learn models for cross-lingual transfer. One can only fine-tune XLM itself on a language and hope that other languages also benefit from finetuning. (Tenney et al., 2019b) . In edge probing, a series of tasks are used as "probes" to measure the syntactic and semantic information contained in representations. The jiant toolkit (Wang et al., 2019) is used to perform the experiments.

4.3. ANALYSIS:

Comparing CLCR with the original BERT-base English model in Table 5 , the transformation step (T) generally lowers scores of most probes by a small fraction. This indicates while most information remains in the representation after training, there is a certain portion of incompatible syntactic and semantic representations across languages. That information is lost and explains the performance loss on XLI on the source language (English). Extraction (E) further lowers scores, but mainly due to failing to predict labels on non-pivot tokens. Because the probing model used in (Tenney et al., 2019b ) is an attention pooling-based order-less model, reordering (R) does not alter the performance on these probing tasks.

Training cost and beyond A total of 250,000 pairs of parallel sentences are used in this study. This is less than 2% of the data used to train XLM. Increasing the size of parallel corpus does not lead to further improvement. Also the total training time of CLCR, including the three steps is less than 5 hours, on a single GPU. This indicates that with good mono-lingual contextual representations like BERT, learning cross-lingual alignment of representations could be a relatively easy task, without the need for extensive supervision and training. Under our formulation, the performance of CLCR is built upon and also limited by the quality of mono-lingual representations. But this more of an opportunity than a problem because parallel corpus is always the more limited resource.

5. CONCLUSION:

In this paper we investigated language-independent cross-lingual contextual representations. We proposed a three-step process to learn cross-lingual contextual representations from mono-lingual representations. In each of the three steps, an intuitive method is employed with the aim of letting the model itself learn to ultimately align two representations. Experiments confirmed that stateof-the-art pre-trained language models are powerful enough to learn to align two languages with moderate supervision. This could indicate that with models like BERT now it is easier than ever to cover the gap in cross-lingual transfer learning.

Our formulation provides a new approach to combine contextual representations with arbitrary task models for zero-shot cross-lingual transfer learning. We experimented with the NLI task and the ESIM model, but the method could in principle be applied to a wide range of tasks and models.

With the advancement of pre-trained language models and their availability, we hope to extend the approach to more languages and more powerful models. We also hope our investigation could inspire further insight into the nature of language-independent representations and cross-lingual transfer learning.

A METHOD OVERVIEW:

We provide an overview diagram to show the whole process of generating language-independent cross-lingual contextual representations, using an example in Figure 4 .

There was no alternative but to develop a new international agreement on climate change to replace the existing Kyoto Protocol . B ZERO-SHOT CROSS-LINGUAL TRANSFER Language-independent cross-lingual contextual representations enable zero-shot cross-lingual transfer for a variety of NLP models. For example, with learned CLCR between English and Chinese, one can train a task model with Embed-E on English, then perform inference with Embed-T-E-R on Chinese (Figure 5 ). 

:

https://github.com/google-research/bert

Figures and Tables:

(s 1 )||s 2 )

Figure 1: Learning curve of distance D. Training phase 1 (left): fix both model, learn T only. Training phase 2 (middle): finetune language 1 representation model with annealing. Training phase 3 (right): finetune language 2 representation model with annealing.

Figure 2: An example of predicted key-points, with (below) and without (above) confident labels

Figure 3: A example of order prediction. From left to right: (1) ground-truth absolute order. (2) predicted absolute order. (3) ground-truth relative order. (4) predicted relative order.

Figure 4: An example illustrating CLCR in inference. For a pair of language (in this example, English and Chinese), one can decide to perform transformation and reordering on one side, and perform extraction on both sides. After training of the transformation (T), extraction (E) and reordering (R) model, for input text in one of the two languages, a sequence of operation is performed: Embed-T-E-R or Embed-E depending on the language. The generated sequence-of-vector representation (in the box with dashed line) is the same for text in both languages, assuming perfect T, E, and R models.

Figure 5: An example of zero-shot cross-lingual transfer with CLCR



Table content: <table><tr><td>Linear Regression</td><td>0.055</td><td>0.078</td></tr><tr><td>Attention (w/o annealing T = 1)</td><td>0.138</td><td>0.207</td></tr><tr><td>Attention (with annealing β = 0.001)</td><td>0.028</td><td>0.048</td></tr><tr><td>Attention (with annealing β = 0.01)</td><td>0.025</td><td>0.037</td></tr></table>

Key-point prediction accuracy

Table content: <table><tr><td/><td/><td/><td/><td colspan="3">Model</td><td/><td/><td>Accuracy en zh</td><td/><td/><td/><td>Precision en zh</td><td>Recall en</td><td>zh</td></tr><tr><td/><td/><td/><td colspan="4">Finetune</td><td/><td colspan="7">0.859 0.823 0.886 0.854 0.936 0.847</td></tr><tr><td/><td/><td/><td colspan="12">Confident labels 0.864 0.826 0.875 0.838 0.959 0.872</td></tr><tr><td>A n d t h a t '</td><td>s</td><td>,</td><td>k in d o f</td><td>,</td><td>w h e r e I</td><td>s e e w h a t m y</td><td>j o b</td><td>is</td><td>a n d w h a t o u r t e a m '</td><td>s</td><td>j o b</td><td>is</td><td>.</td></tr><tr><td>A n d t h a t '</td><td>s</td><td>,</td><td>k in d o f</td><td>,</td><td>w h e r e I</td><td>s e e w h a t m y</td><td>j o b</td><td>is</td><td>a n d w h a t o u r t e m a '</td><td>s</td><td>j o b</td><td>is</td><td>.</td></tr></table>

Reorder performance

Table content: <table><tr><td>Model</td><td colspan="6">BLEU (2-gram) BLEU (3-gram) BLEU (4-gram) en zh en zh en zh</td></tr><tr><td>Absolute order prediction (n = 1)</td><td>0.309</td><td>0.386</td><td>0.166</td><td>0.227</td><td>0.101</td><td>0.152</td></tr><tr><td colspan="2">Absolute order prediction (n = 10) 0.331</td><td>0.393</td><td>0.189</td><td>0.239</td><td>0.121</td><td>0.165</td></tr><tr><td>Relative order prediction (n = 1)</td><td>0.535</td><td>0.566</td><td>0.384</td><td>0.410</td><td>0.290</td><td>0.308</td></tr><tr><td>Relative order prediction (n = 10)</td><td>0.582</td><td>0.624</td><td>0.419</td><td>0.468</td><td>0.316</td><td>0.368</td></tr></table>

Performance of zero-shot cross-lingual transfer. T, E, R stands for Transform, Extract, and Reorder, respectively. pGA stands for pseudo-groundtruth alignment. Gap stands for the transfer gap from English to Chinese (the drop of accuracy in percentage)

Table content: <table><tr><td>Model</td><td colspan="3">XNLI test set en Acc. zh Acc. Gap</td></tr><tr><td>CLCR</td><td/><td/><td/></tr><tr><td>T+pGA</td><td>81.2</td><td>74.8</td><td>6.4</td></tr><tr><td>T</td><td>81.2</td><td>68.0</td><td>13.2</td></tr><tr><td>T+E</td><td>80.6</td><td>69.3</td><td>11.3</td></tr><tr><td>T+E+R</td><td>80.2</td><td>71.7</td><td>8.5</td></tr><tr><td>Non-CLCR models</td><td/><td/><td/></tr><tr><td>Multilingual BERT</td><td>81.4</td><td>63.8</td><td>17.6</td></tr><tr><td>Translate train</td><td>81.4</td><td>74.2</td><td>7.2</td></tr><tr><td>Translate test</td><td>81.4</td><td>70.1</td><td>11.3</td></tr><tr><td>XLM</td><td>85.0</td><td>76.5</td><td>8.5</td></tr></table>



Table content: <table><tr><td colspan="4">: Edge probing statistics of CLCR</td></tr><tr><td/><td/><td>F1 score</td></tr><tr><td/><td>BERT</td><td>T</td><td>T+E</td></tr><tr><td>POS</td><td>97.6</td><td colspan="2">92.4 82.2</td></tr><tr><td>Consts.</td><td>86.8</td><td colspan="2">80.8 75.4</td></tr><tr><td>Deps.</td><td>95.4</td><td colspan="2">92.8 84.5</td></tr><tr><td>Entities</td><td>96.1</td><td colspan="2">94.8 90.9</td></tr><tr><td>SRL</td><td>89.3</td><td colspan="2">86.0 83.1</td></tr><tr><td>Coref.</td><td>95.8</td><td colspan="2">92.0 91.5</td></tr><tr><td>SPR</td><td>83.4</td><td colspan="2">83.1 82.6</td></tr><tr><td>Relations</td><td>77.7</td><td colspan="2">75.9 75.9</td></tr></table>

Peer review (iclr2020):
This paper presents a new method to obtain cross-lingual contextual embeddings by aligning monolingual ones through 3 steps: transform each token individually, merge them as needed to obtain a uniform granularity across languages, and reorder them. While I think that the proposed method has some interest, and the extensive ablation experiments are useful to better understand its behavior, I do not think that the it makes enough merits to be accepted in the conference. I feel that the paper tends to overly complicate things, and it is often difficult to extract any clear idea from it. The proposed method is also much more complicated than previous approaches, yet it does not perform better than them (XLM has better absolute results and exactly the same cross-lingual transfer gap, while being substantially simpler). More concretely: - The paper tends to overly complicate things. For instance, Section 2.1 and 2.2 try to mathematically formalize very basic intuitions. Unless the formalization is important for clarity (which is not, as these are obvious ideas) or necessary later in the paper (which is not either), there is no point in doing that. This only makes the paper more difficult to follow than what it should. - The only extrinsic evaluation is in XNLI, where the authors evaluate the zero-shot cross-lingual transfer performance from English into Chinese. However, the proposed method does not bring any improvement over the current state-of-the-art in this setup. The proposed method gets 80.2% and 71.7% accuracy in English and Chinese, respectively, leaving an absolute transfer gap of 8.5%, while the XLM model from Lample and Conneau (2019) obtains substantially better results (85.0% and 76.5%) with the exact same transfer gap of 8.5%. This could still be good enough if the proposed method had some other advantage over the previous SOTA, but I cannot find any and, in contrast, I do find some disadvantages (see below). - In addition to being more complicated than previous approaches, the proposed method also introduces new hyperparameters and seems more difficult to train. For instance, the authors need to incorporate annealing to train the transformation module, and the model seems quite sensitive to the corresponding hyperparameter (Table 1). - While both multilingual BERT and XLM simply fine-tune a pre-trained BERT model to perform some downstream task, the proposed method is used as a feature extractor, and the authors train an ESIM (LSTM) model on top. The reported experiments do not control this factor (i.e. what would happen if one learns an ESIM model on top of XLM)? - The authors highlight that their system can be trained in *less than 5 hours, on a single GPU*, while *XLM uses much more data and training time*, but this is quite deceptive. Your approach also requires training a monolingual BERT model for each language, which is even more expensive than training a single joint model as XLM does. It is true that one could potentially use publicly available monolingual models, but most pre-trained models in languages other than English are already multilingual, anyway, so I do not see a strong practical justification for this. - The proposed model and the ones it is compared to do not use the same training data. This can have some justification (it might be computationally prohibitive for the authors to pre-train their own models, which might be the reason why they use public models trained on different data) but they should be more upfront about this. In relation to this, it is unfair to remark that the proposed method uses less parallel data than XLM, while not even mentioning that it uses more monolingual data (if I am not wrong, XLM was only trained in Wikipedia, while BERT also used a book corpus at least for English). To make things worse, the authors claim that *XLM uses much more data and training time than our approach*, which seems wrong. - The authors criticize XLM because it *does not produce the same representation for different languages, so there is no guarantee of the performance in transfer learning*. This might be true, but is it anyhow different for your proposed method? Your method is not better empirically, and it does not have any theoretical guarantee either. - As the authors themselves acknowledge, the proposed method is similar in spirit to Schuster et al. (NAACL*2019) -which is also much simpler- but they do not compare to it in their experiments. All in all, I think that the paper tends to overly complicate things, and ultimately fails to answer a simple central question: why should one prefer your approach over previous methods like XLM, or what is it that it makes it otherwise interesting or relevant?