On Learning Universal Representations Across Languages

1 INTRODUCTION . Pre-trained models ( PTMs ) like ELMo ( Peters et al. , 2018 ) , GPT ( Radford et al. , 2018 ) and BERT ( Devlin et al. , 2019 ) have shown remarkable success of effectively transferring knowledge learned from large-scale unlabeled data to downstream NLP tasks , such as text classification ( Socher et al. , 2013 ) and natural language inference ( Bowman et al. , 2015 ; Williams et al. , 2018 ) , with limited or no training data . To extend such pretraining-finetuning paradigm to multiple languages , some endeavors such as multilingual BERT ( Devlin et al. , 2019 ) and XLM ( Conneau & Lample , 2019 ) have been made for learning cross-lingual representation . More recently , Conneau et al . ( 2020 ) present XLM-R to study the effects of training unsupervised cross-lingual representations at a huge scale and demonstrate promising progress on cross-lingual tasks . However , all of these studies only perform a masked language model ( MLM ) with token-level ( i.e. , subword ) cross entropy , which limits PTMs to capture the co-occurrence among tokens and consequently fail to understand the whole sentence . It leads to two major shortcomings for current cross-lingual PTMs , i.e. , the acquisition of sentence-level representations and semantic alignments among parallel sentences in different languages . Considering the former , Devlin et al . ( 2019 ) introduced the next sentence prediction ( NSP ) task to distinguish whether two input sentences are continuous segments from the training corpus . However , this simple binary classification task is not enough to model sentence-level representations ( Joshi et al. , 2020 ; Yang et al. , 2019 ; Liu et al. , 2019 ; Lan et al. , 2020 ; Conneau et al. , 2020 ) . For the latter , ( Huang et al. , 2019 ) defined the cross-lingual paraphrase classification task , which concatenates two sentences from different languages as input ∗Work done at Alibaba Group . Yue Hu and Heng Yu are the co-corresponding authors . We also made an official submission to XTREME ( https : //sites.research.google/xtreme ) , with several improved techniques used in ( Fang et al. , 2020 ; Luo et al. , 2020 ) . and classifies whether they are with the same meaning . This task learns patterns of sentence-pairs well but fails to distinguish the exact meaning of each sentence . In response to these problems , we propose to strengthen PTMs through learning universal representations among semantically-equivalent sentences distributed in different languages . We introduce a novel Hierarchical Contrastive Learning ( HICTL ) framework to learn language invariant sentence representations via self-supervised non-parametric instance discrimination . Specifically , we use a BERT-style model to encode two sentences separately , and the representation of the first token ( e.g. , [ CLS ] in BERT ) will be treated as the sentence representation . Then , we conduct instance-wise comparison at both sentence-level and word-level , which are complementary to each other . At the sentence level , we maximize the similarity between two parallel sentences while minimizing which among non-parallel ones . At the word-level , we maintain a bag-of-words for each sentence-pair , each word in which is considered as a positive sample while the rest words in vocabulary are negative ones . To reduce the space of negative samples , we conduct negative sampling for word-level contrastive learning . With the HICTL framework , the PTMs are encouraged to learn language-agnostic representation , thereby bridging the semantic discrepancy among cross-lingual sentences . The HICTL is conducted on the basis of XLM-R ( Conneau et al. , 2020 ) and experiments are performed on several challenging cross-lingual tasks : language understanding tasks ( e.g. , XNLI , XQuAD , and MLQA ) in the XTREME ( Hu et al. , 2020 ) benchmark , and machine translation in the IWSLT and WMT benchmarks . Extensive empirical evidence demonstrates that our approach can achieve consistent improvements over baselines on various tasks of both cross-lingual language understanding and generation . In more detail , our HICTL obtains absolute gains of 4.2 % ( up to 6.0 % on zero-shot sentence retrieval tasks , e.g . BUCC and Tatoeba ) accuracy on XTREME over XLM-R. For machine translation , our HICTL achieves substantial improvements over baselines on both low-resource ( IWSLT English→X ) and high-resource ( WMT English→X ) translation tasks . 2 RELATED WORK . Pre-trained Language Models . Recently , substantial work has shown that pre-trained models ( PTMs ) ( Peters et al. , 2018 ; Radford et al. , 2018 ; Devlin et al. , 2019 ) on the large corpus are beneficial for downstream NLP tasks . The application scheme is to fine-tune the pre-trained model using the limited labeled data of specific target tasks . For cross-lingual pre-training , both Devlin et al . ( 2019 ) and Conneau & Lample ( 2019 ) trained a transformer-based model on multilingual Wikipedia which covers various languages , while XLM-R ( Conneau et al. , 2020 ) studied the effects of training unsupervised cross-lingual representations on a very large scale . For sequence-to-sequence pre-training , UniLM ( Dong et al. , 2019 ) fine-tuned BERT with an ensemble of masks , which employs a shared Transformer network and utilizing specific self-attention mask to control what context the prediction conditions on . Song et al . ( 2019 ) extended BERT-style models by jointly training the encoder-decoder framework . XLNet ( Yang et al. , 2019 ) trained by predicting masked tokens auto-regressively in a permuted order , which allows predictions to condition on both left and right context . Raffel et al . ( 2019 ) unified every NLP problem as a text-to-text problem and pre-trained a denoising sequence-to-sequence model at scale . Concurrently , BART ( Lewis et al. , 2020 ) pre-trained a denoising sequence-to-sequence model , in which spans are masked from the input but the complete output is auto-regressively predicted . Previous works have explored using pre-trained models to improve text generation , such as pretraining both the encoder and decoder on several languages ( Song et al. , 2019 ; Conneau & Lample , 2019 ; Raffel et al. , 2019 ) or using pre-trained models to initialize encoders ( Edunov et al. , 2019 ; Zhang et al. , 2019a ; Guo et al. , 2020 ) . Zhu et al . ( 2020 ) and Weng et al . ( 2020 ) proposed a BERTfused NMT model , in which the representations from BERT are treated as context and fed into all layers of both the encoder and decoder . Zhong et al . ( 2020 ) formulated the extractive summarization task as a semantic text matching problem and proposed a Siamese-BERT architecture to compute the similarity between the source document and the candidate summary , which leverages the pre-trained BERT in a Siamese network structure . Our approach also belongs to the contextual pre-training so it could be applied to various downstream NLU and NLG tasks . Contrastive Learning . Contrastive learning ( CTL ) ( Saunshi et al. , 2019 ) aims at maximizing the similarity between the encoded query q and its matched key k+ while keeping randomly sampled keys { k−0 , k − 1 , k − 2 , ... } faraway from it . With similarity measured by a score function s ( q , k ) , a form of a contrastive loss function , called InfoNCE ( Oord et al. , 2018 ) , is considered in this paper : Lctl = − log exp ( s ( q , k+ ) ) exp ( s ( q , k+ ) ) + ∑ i exp ( s ( q , k − i ) ) , ( 1 ) where the score function s ( q , k ) is essentially implemented as the cosine similarity q T k ‖q‖·‖k‖ . q and k are often encoded by a learnable neural encoder , such as BERT ( Devlin et al. , 2019 ) or ResNet ( He et al. , 2016 ) . k+ and k− are typically called positive and negative samples . In addition to the form illustrated in Eq . ( 1 ) , contrastive losses can also be based on other forms , such as margin-based loses ( Hadsell et al. , 2006 ) and variants of NCE losses ( Mnih & Kavukcuoglu , 2013 ) . Contrastive learning is at the core of several recent work on unsupervised or self-supervised learning from computer vision ( Wu et al. , 2018 ; Oord et al. , 2018 ; Ye et al. , 2019 ; He et al. , 2019 ; Chen et al. , 2020 ; Tian et al. , 2020 ) to natural language processing ( Mikolov et al. , 2013 ; Mnih & Kavukcuoglu , 2013 ; Devlin et al. , 2019 ; Clark et al. , 2020b ; Feng et al. , 2020 ; Chi et al. , 2020 ) . Kong et al . ( 2020 ) improved language representation learning by maximizing the mutual information between a masked sentence representation and local n-gram spans . Clark et al . ( 2020b ) utilized a discriminator to predict whether a token is replaced by a generator given its surrounding context . Iter et al . ( 2020 ) proposed to pre-train language models with contrastive sentence objectives that predict the surrounding sentences given an anchor sentence . In this paper , we propose HICTL to encourage parallel cross-lingual sentences to have the identical semantic representation and distinguish whether a word is contained in them as well , which can naturally improve the capability of cross-lingual understanding and generation for PTMs . 3 METHODOLOGY . 3.1 HIERARCHICAL CONTRASTIVE LEARNING . We propose hierarchical contrastive learning ( HICTL ) , a novel comparison learning framework that unifies cross-lingual sentences as well as related words . HICTL can learn from both non-parallel and parallel multilingual data , and the overall architecture of HICTL is illustrated in Figure 1 . We represent a training batch of the original sentences as x = { x1 , x2 , ... , xn } and its aligned counterpart is denoted as y = { y1 , y2 , ... , yn } , where n is the batch size . For each pair 〈xi , yi〉 , yi is either the translation in the other language of xi when using parallel data or the perturbation through reordering tokens in xi when only monolingual data is available.x\i is denoted as a modified version of x where the i-th instance is removed . Sentence-Level CTL . As illustrated in Figure 1a , we apply the XLM-R as the encoder to represent sentences into hidden representations . The first token of every sequence is always a special token ( e.g. , [ CLS ] ) , and the final hidden state corresponding to this token is used as the aggregate sentence representation for pre-training , that is , rx = f ◦ g ( M ( x ) ) where g ( · ) is the aggregate function and f ( · ) is a linear projection , ◦ denotes the composition of operations . To obtain universal representation among semantically-equivalent sentences , we encourage rxi ( the query , denoted as q ) to be as similar as possible to ryi ( the positive sample , denoted as k + ) but dissimilar to all other instances ( i.e. , y\i ∪ x\i , considered as a series of negative samples , denoted as { k−1 , k − 2 , ... , k − 2n−2 } ) in a training batch . Formally , the sentence-level contrastive loss for xi is defined as Lsctl ( xi ) = − log exp ◦s ( q , k+ ) exp ◦s ( q , k+ ) + ∑|y\i∪x\i| j=1 exp ◦s ( q , k − j ) . ( 2 ) Symmetrically , we also expect ryi ( the query , denoted as q̃ ) to be as similar as possible to rxi ( the positive sample , denoted as k̃+ ) but dissimilar to all other instances in the same training batch , thus , Lsctl ( yi ) = − log exp ◦s ( q̃ , k̃+ ) exp ◦s ( q̃ , k̃+ ) + ∑|y\i∪x\i| j=1 exp ◦s ( q̃ , k̃ − j ) . ( 3 ) The sentence-level contrastive loss over the training batch can be formulated as LS = 1 2n n∑ i=1 { Lsctl ( xi ) + Lsctl ( yi ) } . ( 4 ) For sentence-level contrastive learning , we treat other instances contained in the training batch as negative samples for the current instance . However , such randomly selected negative samples are often uninformative , which poses a challenge of distinguishing very similar but nonequivalent samples . To address this issue , we employ smoothed linear interpolation ( Bowman et al. , 2016 ; Zheng et al. , 2019 ) between sentences in the embedding space to alleviate the lack of informative samples for pre-training , as shown in Figure 2 . Given a training batch { 〈xi , yi〉 } ni=1 , where n is the batch size . In this context , having obtained the embeddings of a triplet , an anchor q and a positive k+ as well as a negative k− ( supposing q , k+ and k− are representations of sentences xi , yi and y−i ∈ x\i ∪ y\i , respectively ) , we construct a harder negative sample k̂− to replace k − j : k̂− = { q + λ ( k− − q ) , λ ∈ ( d + d− , 1 ] if d − > d+ ; k− if d− ≤ d+ . ( 5 ) where d+ =‖ k+− q ‖2 and d− =‖ k−− q ‖2 . For the first condition , the hardness of k̂− increases when λ becomes smaller . To this end , we intuitively set λ as λ = ( d+ d− ) ζ·p+avg , ζ ∈ ( 0 , 1 ) ( 6 ) where p+avg = 1 100 ∑ ∈ [ −100 , −1 ] e −L ( ) S is the average log-probability over the last 100 training batches and LS formulated in Eq . ( 4 ) is the sentence-level contrastive loss of one training batch . During pre-training , when the model tends to distinguish positive samples easily , which means negative samples are not informative already . At this time , p+avg ↑ and d + d− ↓ , which leads λ ↓ and harder negative samples are adaptively synthesized in the following training steps , vice versa . As hard negative samples usually result in significant changes of the model parameters , we introduce the slack coefficient ζ to prevent the model from being trained in the wrong direction , when it accidentally switch from random negative samples to very hard ones . In practice , we empirically set ζ = 0.9 . Word-Level CTL . Intuitively , predicting the related words in other languages for each sentence can bridge the representations of words in different languages . As shown in Figure 1b , we concatenate the sentence pair 〈xi , yi〉 as xi ◦ yi : [ CLS ] xi [ SEP ] yi [ SEP ] and the bag-of-words of which is denoted as B . For word-level contrastive learning , the final state of the first token is treated as the query ( q̄ ) , each word wt ∈ B is considered as the positive sample and all the other words ( V\B , i.e. , the words in V that are not in B where V indicates the overall vocabulary of all languages ) are negative samples . As the vocabulary usually with large space , we propose to only use a subset S ⊂ V\B sampled according to the normalized similarities between q̄ and the embeddings of the words . As a result , the subset S naturally contains the hard negative samples which are beneficial for learning high-quality representations ( Ye et al. , 2019 ) . Specifically , the word-level contrastive loss for 〈xi , yi〉 is defined as Lwctl ( xi , yi ) = − 1 |B| |B|∑ t=1 log exp ◦s ( q̄ , e ( wt ) ) exp ◦s ( q̄ , e ( wt ) ) + ∑ wj∈S exp ◦s ( q̄ , e ( wj ) ) . ( 7 ) where e ( · ) is the embedding lookup function and |B| is the number of unique words in the concatenated sequence xi ◦ yi . The overall word-level contrastive loss can be formulated as : LW = 1 n n∑ i=1 Lwctl ( xi , yi ) . ( 8 ) Multi-Task Pre-training . Both MLM and translation language model ( TLM ) are combined with HICTL by default , as the prior work ( Conneau & Lample , 2019 ) has verified the effectiveness of them in XLM . In summary , the model can be optimized by minimizing the entire training loss : L = LLM + LS + LW , ( 9 ) where LLM is implemented as either the TLM when using parallel data or the MLM when only monolingual data is available to recover the original words of masked positions given the contexts .

Summary: The paper proposes a pre-trained language model variant which extends XLM-R (multilingual masked model) with two new objectives. The main difference to most other models is that the new losses are contrastive losses (however, as pointed out by the authors, other contrastive losses had been used before in e.g. ELECTRA). The first additional loss is a sentence-level one - where a [CLS] token is trained to be close to the positive sample, the paired sentence, with other sentences as negative samples. The same is done at word level, where the bag of words constructed from two sentences becomes the set of positive samples and other vocabulary words are negative samples.