XD: Cross-lingual Knowledge Distillation for Polyglot Sentence Embeddings

Current state-of-the-art results in multilingual natural language inference ( NLI ) are based on tuning XLM ( a pre-trained polyglot language model ) separately for each language involved , resulting in multiple models . We reach significantly higher NLI results with a single model for all languages via multilingual tuning . Furthermore , we introduce cross-lingual knowledge distillation ( XD ) , where the same polyglot model is used both as teacher and student across languages to improve its sentence representations without using the end-task labels . When used alone , XD beats multilingual tuning for some languages and the combination of them both results in a new state-of-the-art of 79.2 % on the XNLI dataset , surpassing the previous result by absolute 2.5 % . The models and code for reproducing our experiments will be made publicly available after de-anonymization . 1 INTRODUCTION . Many sentence level tasks in natural language processing have seen efficient solutions based on sentence vector representations ( embeddings ) and supervised tuning , however labelled data is scarce for all but a handful of resource-rich languages like English . This motivates the development of cross-lingual methods that can perform knowledge transfer from sentence representations in one language to labels assigned to sentences in another language . We focus on one such task : natural language inference ( NLI ) , where the aim is to detect , whether the meaning of one sentence can be inferred from another one , contradicts it , or neither . For instance , the sentence You can leave can be inferred from the sentence You don ’ t have to stay there . The example is taken from the XNLI dataset ( Conneau et al. , 2018 ) , which was created for testing crosslingual NLI and includes labelled English sentence pairs , translated into 15 languages ; manually for the development and test sets , automatically for the training set . The best results on XNLI so far have been achieved by using XLM ( Lample & Conneau , 2019 ) , a contextualized multilingual language model that is pre-trained on unlabelled text and then tuned in a supervised manner separately for each language in the XNLI dataset . While this yields competitive results , it wastefully employs a separate NLI model for each language ( Lample & Conneau , 2019 ) . Our contributions are two-fold : first we describe a multilingual tuning scenario , in which we achieve a significantly higher average XNLI accuracy with a single model for all 15 languages . Furthermore , we introduce XD , a cross-lingual knowledge distillation approach that uses one and the same XLM model to serve both as teacher ( for English sentences ) and student ( for their translations into other languages ) . The approach does not require end-task labels and can be applied in an unsupervised setting . We describe the relevant background and our methods in Section 2 . In the experimental part of this paper ( Section 3 ) we compare the performance of XD and multilingual tuning for multiple language combinations , in order to cover low and high resource settings as well as related and distant language pairs . Our results show that XD reaches the same or better results than multilingual tuning alone , depending on language . A combination of both methods brings an even better XNLI average result , which outperforms both of them alone and surpasses the previous state-of-the-art as well as results of concurrent work . 2 METHODOLOGY . 2.1 BACKGROUND . We build both our methods ( described in the following subsections ) based on a large-scale pretrained cross-lingual language model ( XLM ) introduced by Lample & Conneau ( 2019 ) . XLM is trained either with the Masked Language Modeling ( MLM ) objective alone or in combination with the Translation Language Modeling ( TLM ) objective . In MLM ( Devlin et al. , 2018 ) we mask words in the input sequence and teach the model to fill in the gaps . In XLMMLM we train a model on sentences from different languages by employing joint multilingual wordpiece vocabulary ( Sennrich et al. , 2015 ) . Shared cross-lingual wordpieces lead to the sharing of cross-lingual wordpiece embeddings . In XLMMLM+TLM setting we train the model on pairs of parallel sentences while masking words similarly to XLMMLM . However in XLMMLM+TLM model can attend not only to surrounding context to predict the missing word , but also to its translation from parallel sentence which results in stronger cross-lingual signal . Both methods belong to the pretraining time cross-lingual alignment category . After the language model is trained , we substitute the word prediction head with classification layer ( 3-class softmax in our case ) . We place this head on top of the contextual embedding of special ” CLS ” token prepended to the sentence . Next , we can tune result model on source language training data and directly apply it to target languages obtaining zero-shot classification results . By the source language we mean that language we have labeled training data and by target languages we mean the languages that we try to transfer knowledge to ( languages without out-of-the-box supervised data ) . On the other hand , absolute state-of-the-art result is obtained by translating English portion of the data into target languages with MT system ( s ) , duplicating pretrained XLMMLM+TLM as many times as there are target languages , and tuning on each language individually ( Lample & Conneau , 2019 ) . Another concept important for understanding our method is knowledge distillation ( Hinton et al. , 2015 ) . In the knowledge distillation framework we train so-called student model to mimic behaviour of the ( usually larger and stronger ) teacher model . In case of classification , one way to perform KD is to make teacher model predict on unlabeled examples producing vector of real-valued logits ( unnormalized probabilities over discrete label space ) . Student model then takes the same example as its input and uses teacher ’ s logits as its target . 2.2 MULTILINGUAL TUNING . In multi-language fine-tuning , similarly to current state-of-the-art approach , we use machine translation system to translate data to other languages . Then we however use all the obtained data at the same time to tune a single XLMMLM or XLMMLM+TLM . Since the pseudo corpus is fully aligned , model does not see a single new example . However , the network might improve by discovering task-oriented rules and regularities learnt between languages . 2.3 XD : CROSS-LINGUAL KNOWLEDGE DISTILLATION . We propose to employ knowledge distillation method to perform cross-lingual transfer in a single cross-lingual language model . We duplicate the network twice and mark original one as a student and its copy as teacher model . Then for each target language we use parallel source-target text pairs as following : 1. we pass source text to the teacher model to get continous logits vector representation ; 2. we pass target text to the student model to get continous logits vector representation ; 3. we compute L2 loss between teacher logits and student logits 4. we backpropagate through student while keeping teacher frozen Above-mentioned knowledge distillation procedure is based on the idea that the sentence should get same latent space representation no matter in what language it was proxied to the model ( Conneau et al. , 2018 ; Aldarmaki & Diab , 2019a ) . Since our polyglot language model was tuned on labeled downstream task data for English language we align all other language representations to the English latent space . Using the same polyglot model as teacher and student brings us following benefits : • we do not need to train a separate student model for each target language ; instead we just align cross-lingual representations from the same model ; • we do not need to train a separate teacher model for source language ; instead we just operate on the source language logits latent space of our polyglot language model ; • we get a decent student initialization where representations for source and target text are already close to each other to some degree which simplifies and speeds up convergence Also knowledge distillation procedure abolishes the need of using target language labels for crosslingual alignment . Multilingual tuning can also be viewed from the knowledge distillation perspective . Teacher logits for a given sentence can be viewed as continuous ( approximate ) representation for this sentence . True sentence ’ s label can be viewed as discrete representation for this sentence . One important conceptual difference is that in the former case , different sentences belonging to the same class will get the same discrete representation ( same label ) . In the latter case , different sentences belonging to the same class will get different continuous space ( logits ) representations . Continuous targets provide richer but at the same time more restrictive signal . Depending on the strength of teacher model this might lead or not lead to the better training comparing to learning from discrete vectors . The quality of parallel corpora as well as languages relatedness are another important factors that emerge in the context of XD . From the transfer learning perspective XD method belongs to the sequential learning paradigm ( Ruder , 2019 ) because we first pretrain language model , then fine-tune it repeatedly . It also can be classified as cross-lingual learning since the last fine-tuning step is done for multiple languages in terms of the same polyglot model . Next we describe the experiments in which we validate both our introduced approaches . 3 EXPERIMENTS AND RESULTS . 3.1 EXPERIMENTAL SETTINGS . We choose the XNLI dataset ( Conneau et al. , 2018 ) as our test bed for evaluating models from all following experiments . XNLI provides human translated validation and test data for 15 languages that belong to different language families . Baselines are tuned on the English MNLI corpus ( Williams et al. , 2018 ) that contains more than 400k training examples . For multilingual tuning we use machine translated versions of MNLI from ( Conneau et al. , 2018 ) for a fair comparison with current state-of-the-art tuning approach ( Lample & Conneau , 2019 ) . For XD unlabeled parallel data is needed , so we use the same parallel sentences but with labels removed . The implications of this choice are two-fold . On one hand using the translated training set for XD is ideal for the teacher network to show its knowledge during distillation . On the other hand , it allows for a fair comparison with multilingual tuning because we do not show the model any new examples ( but only translations of what it had already seen ) . Also , the parallel data are not required to be synthetic for XD and better results can likely be achieved with actual parallel data . For all the XD experiments we use English as our source language ( since there is ground-truth training data available for it ) and for case studies we choose French , Swahili and Urdu as our target languages . We base our choice on quality of synthetic parallel corpus and relatedness of languages to English . The quality of translations defines how far our setting falls from what could be achieved with real parallel data and implicitly represents the resource richness of the target language . Relatedness of the languages defines the amount of shared wordpieces ( Sennrich et al. , 2015 ) which also impacts XD . French ( fr ) is similar to English while also being a high-resource language with a high translation BLEU score ( 49.3 ) ; Urdu ( ur ) is an unrelated language with a low BLEU score of 24.1 ; Swahili ( sw ) is loosely between French and Urdu in terms of relatedness to English but with a low BLEU score of 24.6 . For multilingual tuning zero-shot results we choose German , Swahili , and Urdu for the same reasons . Refer to the Table 1 for other languages ’ BLEU scores . We choose XLMMLM/MLM+TLM as our polyglot language model because it gave state-of-the-art cross-lingual results at the moment of writing1 . For all XD and multilingual tuning experiments we use hyperparameters optimized for our baselines . We use the Adam optimizer with a learning 1Concurrently with our work Huang et al . ( 2019 ) presented an even stronger polyglot language model rate of 5×10−6 , batch size of 8 sentences from the same language ( truncating sentences to be up to 256 words per example ) , and small epochs of 20 , 000 examples each ( following Lample & Conneau ( 2019 ) ) . We use L2 as our loss function for XD . Together with the codebase we also publish training configuration files for all the experiments we ran where the setup can be seen in exact details . 2 . We use pretrained XLM from pytroch-transformers3 repository and allennlp4 as our NLP framework .

Summary: This paper proposes two improved strategies for fine-tuning XLM (a multilingual variant of BERT) for cross-lingual NLI. First of all, it shows that fine-tuning a single model on the combination of all languages (the original English data from MultiNLI and their MT translation into the rest of languages) performs better than fine-tuning a separate model for each language. Furthermore, they show that minimizing the L2 distance between the English training sentences and their MT translation into the rest of languages, which does not explicitly use any labels in the foreign languages and is presented as a way of performing cross-lingual knowledge distillation, also performs better than zero-shot transferring a regular model fine-tuned in English.