Language-independent Cross-lingual Contextual Representations

Contextual representation models like BERT have achieved state-of-the-art performance on a diverse range of NLP tasks . We propose a cross-lingual contextual representation model that generates language-independent contextual representations . This helps to enable zero-shot cross-lingual transfer of a wide range of NLP models , on top of contextual representation models like BERT . We provide a formulation of language-independent cross-lingual contextual representation based on mono-lingual representations . Our formulation takes three steps to align sequences of vectors : transform , extract , and reorder . We present a detailed discussion about the process of learning cross-lingual contextual representations , also about the performance in cross-lingual transfer learning and its implications . 1 INTRODUCTION . A cross-lingual text representation can generate language-independent representations : for pieces of text that are semantically similar , the produced representations are also similar , regardless of the language of the text . This provides a basis for dealing with multiple languages in NLP . Training a model on cross-lingual representations is a common approach for cross-lingual transfer learning ( Huang et al. , 2013 ; Artetxe & Schwenk , 2019 ) . To achieve desirable transfer learning performance , the cross-lingual representation model should be effective in two aspects : 1 ) the quality of text representation ; and 2 ) the quality of cross-lingual mapping . The current state-of-the-art text representation models are a family of pre-trained language model representations such as BERT ( Devlin et al. , 2019 ) and XLNet ( Yang et al. , 2019 ) . These models produce contextual representations of text , i.e. , a sequence of vectors corresponding to the sequence of text tokens . Each vector represents “ a token in context , ” e.g. , the semantics of a word in its local context and in the global context of the sentence . Research on cross-lingual representation has so far been largely concentrated on word embeddings ( Mikolov et al. , 2013 ; Gouws et al. , 2015 ; Ruder et al. , 2017 ) and sentence embeddings ( Schwenk & Douze , 2017 ; Artetxe & Schwenk , 2019 ) . Word embeddings are static representations and are less expressive than contextual representations . Sentence embeddings perform respectably in some tasks but inferior in others , because there is information loss when too much information is crammed into a single vector ( Conneau et al. , 2018a ) . However , word embeddings and sentence embeddings are relatively easy to generalize to cross-lingual representations because they do not have dynamic inter-dependency among vectors . We regard it as an interesting topic to investigate cross-lingual contextual representations . First , it is an interesting problem in its own to match two ( or more ) sequences of vectors , where each vector sequence has its order and locality relationships . Second , we hope to further improve cross-lingual transfer learning by leveraging the state-of-the-art text representation models . In this study , we try to clarify the definition and characteristics of cross-lingual contextual representations , and propose a formulation to learn a cross-lingual contextual representations model based on existing mono-lingual pre-trained language models . We perform experiments to analyze the effect of the ingredients in our formulation and the final performance in transfer learning . We also hope our investigation help to understand the nature of text representations in different languages : what are the commonalities between representations , how to cover the gaps , and ultimately what is transferable across languages with the help of state-of-the-art representation models . Related work Multilingual BERT learns a multi-lingual representation by training on a combined corpus of multiple languages . For dissimilar languages , the transfer performance is not satisfactory ( Pires et al. , 2019 ) . XLM ( Lample & Conneau , 2019 ) uses parallel corpus and a translational objective , and has achieved good transfer performance for many languages . However , XLM does not produce the same representation for different languages , so there is no guarantee of the performance in transfer learning . In multi-way neural machine translation ( Firat et al. , 2016 ) the situation is similar : multiple encoders produce the representation for each language , but because the representations are not language-independent , the decoder still needs to be trained on multiple language inputs . A recent work on aligning contextual representations is ( Schuster et al. , 2019 ) , which treats vectors in contextual representations as individual embeddings and learns a matrix to transform the embeddings space of different languages . The idea resembles the “ transform ” step in our formulation . 2 CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS . First we define “ language-independent cross-lingual contextual representations ( CLCR ) ” and establish mathematical notations that we use in the following discussion . 2.1 DEFINITION . A cross-lingual contextual representation ( CLCR ) of text is a parameterized representation model f ( θ ) : ( a0 , ... , aL ) → ( r0 , ... , rN ) , where a0 , ... , aL is the sequence of text tokens in a sentence s , and r0 , ... , rN is the sequence of vectors produced by f as the representation of s. A languageindependent f satisfies the following conditions : 1 . ( cross-lingual correspondence ) if sentence si in i-th language and sentence sj in jth language are translations of each other , then by some distance metric D we have D ( f ( si ; θ ) , f ( sj ; θ ) ) < δ , where δ is a threshold . 2 . ( isometric property ) if for some semantic metric S , sentence si in i-th language and sentence sj in j-th language satisfy S ( si , sj ) < γ , i.e. , si and sj are close in meaning , then by some distance metric D we have D ( f ( si ; θ ) , f ( sj ; θ ) ) < δ ( γ ) , where δ is a monotonic increasing function of γ . 3 . ( contextual representation ) ∃f1 , f2 , f3 s.t . ri = f3 ( f2 ( ai ) , f1 ( a0 ... L\i ) ) . The vector ri can roughly be interpreted as a representation of token ai in the context of the whole sentence . Condition 1 and 2 ensure that CLCR produces similar representations for sentences with similar meaning , independent of language . Condition 3 is a characteristic of contextual representation models such as pre-trained language models like BERT . 2.2 EXISTENCE OF CLCR . To show that such cross-lingual contextual representation exists , we provide a formulation of CLCR based on mono-lingual contextual representations . We propose that one can generate CLCR with a three-step process . Starting from mono-lingual representations , each of the three steps corresponds to one problem that needs to be solved , in order to arrive at a cross-lingual contextual representation . 1 . Alignment : the vectors of mono-lingual representation of different languages reside in different spaces . First we align the vector spaces Ri with transformations Ti : ri → rC , to transform into a common space RC . 2 . Granularity : different languages have different token granularities . Some language use several tokens to represent a word . Token alignment between translations si and sj can be represented by a set A = { ( ( aim , ... , ain ) , ( ajp , ... , ajq ) ) } , where tokens ( aim , ... , ain ) and ( ajp , ... , a j q ) are aligned “ phrases. ” Definition of Finest Common Granularity : find the token alignment A∗ that can not be broken into a finer alignment . The finest common granularity for si w.r.t . language i and j is defined as a grouping of the token sequence of si in A∗ : FCG ( si ) = { ( ai0 , ... , ain0 ) , ( ain0+1 , ... , ain1 ) , ... , ( ainm+1 , ... , aiL ) } , where each element ( aini+1 , ... , a i ni+1 ) ∈ A∗ . In a nutshell , we want to group a series of tokens ( ain0+1 , ... , a i n1 ) into a “ phrase token ” ( key-point token ) , so that the key-point tokens have one-to-one correspondence between two languages . With key-point tokens derived from FCG , translations si and sj have the same number of key-point tokens , and can then be represented by the same number of vectors . 3 . Order : many different word orders exist in different languages , for example the SVO ( subject-verb-object ) and the SOV ( subject-object-verb ) order . We define an order mapping to map the orders of token from language i to language j : Ri→j : m|si → w , where ( ( ainm+1 , ... , a i nm+1 ) , ( a j nw+1 , ... , ajnw+1 ) ) ∈ A . The order mapping Ri→j maps the position of a key-point token in the source sentence to the position of the corresponding key-point token in the target language . To generate cross-lingual representations , we perform alignment transformation on the vectors of two mono-lingual representations , find their common granularity , and then reorder the vectors , to make the two representations similar . 2.3 TRANSFER LEARNING WITH CLCR . Language-independent cross-lingual representations enables zero-shot transfer learning across languages . CLCR allows the most versatile models to be trained on cross-lingual representations . For a model M taking contextual representations ( r0 , ... , rN ) as input , M can be trained on dataset { ( si , l ) } in language i , by first using CLCR f to encode inputs : M ( f ( si ) ) → l. In testing , model can perform inference on input in any language within f ’ s input languages . 3 LEARNING CROSS-LINGUAL CONTEXTUAL REPRESENTATIONS . In this section , we detail our approach to learn CLCR based on the formulation in Section 2.2 . We use two mono-lingual contextual representation models on language 1 and 2 to learn a cross-lingual contextual representation model . 3.1 TRANSFORM . In the first step , we learn a transformation T parameterized by a multi-layer network : T ( r ) = Linear ( r ) + Linear ( Relu ( Linear ( r ) ) ) ( 1 ) The network is trained to minimize the discrepancy between the transformed representation in language 1 and the representation in language 2 : D ( T ( r10 , ... , r 1 M ) || ( r20 , ... , r2N ) ) . Because we do not have alignment between the two sequences of vectors yet , we define D with attention matching between the two sequences : D ( T ( r10 , ... , r 1 M ) || ( r20 , ... , r2N ) ) = ∑ i 1 N ||T ( Attention ( r2i , ( r10 , ... , r1M ) ) − r2i || ( 2 ) where Attention is defined as using r2i as query to perform a weighted-sum of ( r 1 0 , ... , r 1 N ) . To train transformation T , we minimize the above loss D on a parallel corpus { ( s1 , s2 ) } . We observed that straightforward gradient descent does not converge . We therefore used attention annealing during training to help our model converge to an optimal matching : softmax ( a0 , ... , aN ) = ea0/T∑ i e ai/T ( 3 ) where T = 1/ ( 1 + β · step ) . As training progresses softmax will gradually be replaced by max , which let the model find a one-to-one matching between the tokens . This is exactly required by the next step .

Summary: This paper proposes a method to learn language-independent cross-lingual contextual representations by mapping the representations of a monolingual model in one language to the representations of a monolingual model in another language. The proposed approach consists of three steps: 1. A transformation is learned that minimizes the distance between the contextual word representations of the two models of a sentence and its translation in a parallel corpus. 2. Each sentence is summarized as a sequence of key-point tokens based on phrase alignment. 3. The contextual word vectors are reordered based on an order prediction model.