Learning Invariant Representations on Multilingual Language Models for Unsupervised Cross-Lingual Transfer

Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts , evidenced by their decent zero-shot transfer performance . While studies have attributed this success to cross-lingually shared representations , quantitative analyses are sparse . Towards a better understanding of the role of multilingual representations , in this work , we first make the following observations through empirical analysis : ( 1 ) invariance of the feature representations strongly correlates with transfer performance , and ( 2 ) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work . Based on our findings , we propose an unsupervised cross-lingual learning method , called importance-weighted domain alignment ( IWDA ) , that performs representation alignment , prior shift estimation , and correction . Experiment results demonstrate its superiority under large prior shifts . In addition , our method delivers further performance gains when combined with existing semi-supervised learning techniques . 1 INTRODUCTION . Many recent state-of-the-art results in natural language processing ( NLP ) have been achieved on Transformer-based deep neural language models ( LMs ) using the “ pre-train then fine-tune ” paradigm ( Devlin et al. , 2019 ; Conneau & Lample , 2019 ) . Multilingual versions of these LMs are pre-trained on unannotated and unparallel texts from more than one language , such as multilingual BERT ( mBERT ) and XLM-R ( Conneau et al. , 2020a ) , and it is found that the learned representations are shared across languages . In particular , the cross-lingual knowledge acquired by these models has enabled zero-shot cross-lingual transfer , where after fine-tuned on data in a source language , not only do they work well when evaluated on the source , but also quite decently for the same task on almost all languages they have seen during pre-training ( Wu & Dredze , 2019 ) . Their success on zero-shot cross-lingual transfer has prompted numerous studies on their multilingual abilities . Among the many explanations , a recent hypothesis suggests that the deep representation combined with parameter sharing induced intermediate feature representations that are shared across languages ( K et al. , 2019 ; Conneau et al. , 2020b ; Muller et al. , 2021 ) . Following this , we begin by studying the following question for mutilingual neural LMs : What is the role of shared representations in unsupervised cross-lingual learning ? In an effort to answer this question in a quantitative manner , through empirical analysis on mBERT , we find that ( 1 ) its cross-lingual transfer performance strongly correlates with the alignment of feature representations for the NLP task at hand between the source and target languages ( Section 2.1 ) . In addition , ( 2 ) distributional shift in class priors between data in the source and target languages negatively affects transfer performance ( sections 2.2 and 4.2 ) . Despite being a common issue with real-world data under unsupervised settings , it has been largely overlooked in prior work . Based on these findings on the cross-lingual transfer of mutilingual neural LMs , towards a principled approach for the unsupervised setting where access to unlabeled data in the target language for the downstream task at hand is available in addition to labeled ones in the source ( hereafter referred to as UCL1 ) , we propose importance-weighted domain alignment ( IWDA ) . This method performs representation alignment , class prior shift estimation , and correction ( Section 3 ) . Experiment results show that IWDA outperforms the more popular semi-supervised learning methods under large prior shifts , and can be additionally combined with them for further performance gains ( Section 4 ) . 1.1 RELATED WORK . Our proposed method is heavily inspired by Tachet des Combes et al . ( 2020 ) , which studied the issues with alignment-based domain adaptation ( DA ) under the presence of class prior shifts , and proposed the inclusion of prior shifts estimation and correction procedures in DA . Our method also shares similar components with the DA-based UCL method by Keung et al . ( 2019 ) , while highlighting its limitations discovered in the present work ( Section 2.2 ) . Analyzing Multilingual Neural LMs . Most studies analyze mBERT ’ s cross-lingual ability via probing experiments , and identified influencing factors including but not limited to linguistic similarity , size of pretraining corpus , domain similarity , parameter sharing and model depth ( Pires et al. , 2019 ; K et al. , 2019 ; Lauscher et al. , 2020 ; Conneau et al. , 2020b ; Dufter & Schütze , 2020 ) . Another line of research analyze intermediate representations and observed that mBERT learns crosslingually shared representations ( Wu & Dredze , 2019 ; Conneau et al. , 2020b ; Muller et al. , 2021 ) . Cross-Lingual Representation Alignment . The idea of learning invariant representations has powered a long history of approaches for cross-lingual transfer on neural models . Under zero-shot settings , it is achieved by aligning word embeddings on bilingual dictionaries that could be induced via adversarial training ( Mikolov et al. , 2013 ; Smith et al. , 2017 ; Zhang et al. , 2017 ; Artetxe & Schwenk , 2019 ; Cao et al. , 2019 ) , a method that has powered many machine translation techniques ( Artetxe et al. , 2018 ; Lample et al. , 2018 ; Arivazhagan et al. , 2019 ; Pham et al. , 2019 ; Hu et al. , 2021 ) It is found on static embedding models that the goodness of alignment affects transfer performance ( Patra et al. , 2019 ; Dubossarsky et al. , 2020 ) . On deep models such as mBERT , it could also take the form of shared feature representations in the Transformer body ( Artetxe et al. , 2020b ; Pfeiffer et al. , 2020 ) . Under UCL and supervised settings , techniques based on representation alignment were proposed for models ranging from distributed word embeddings ( Joty et al. , 2017 ; Chen et al. , 2018 ) , RNN ( Kim et al. , 2017 ; Huang et al. , 2019 ) , to mBERT ( Keung et al. , 2019 ; Zheng & Lapata , 2019 ; Xia et al. , 2021 ) . For a survey , see Ruder et al . ( 2019 ) and Ruder ( 2019 ) . Connection to Domain Adaptation . Domain adaptation is a popular framework for handling domain shifts between training and test data , e.g . learning a POS tagger on news articles but to be deployed on biomedical papers with a significantly different word distribution ( Ben-David et al. , 2007 ) . The concept of domain is extended naturally to cross-lingual learning by treating the distributions of sentence and document pertaining to the same task in each language as separate domains2 ( Ruder , 2019 ) . Thereby , adaptation techniques proposed for domain shift can be applied to UCL . Besides methods aiming at extracting shared representations ( Li et al. , 2020 ; Vernikos et al. , 2020 ) , another family of successful approaches is semi-supervised learning , which include self-training ( Dong & de Melo , 2019 ) , tri-training ( Ruder & Plank , 2018 ) , knowledge distillation ( Wu et al. , 2020 ) , and data augmentation ( Wang et al. , 2018 ; Maharana & Bansal , 2020 ; Bari et al. , 2021 ) . 2 FACTORS AFFECTING CROSS-LINGUAL TRANSFER . To develop and provide evidence to the hypothesis that attributes the cross-lingual transfer success of multilingual neural LMs to shared representations , as well as to understand its implications on the design of unsupervised transfer algorithms , we attempt to quantify the relation between representation invariance and transfer performance in this section . We briefly summarize the “ pre-train then fine-tune ” training paradigm and introduce notation along the way . Let g : X → Z denote the feature mapping provided by the multilingual LM , and z = 1With an abuse of notation . In this work , the acronym UCL refers to the learning setting considered , and is meant to distinguish from the umbrella term “ unsupervised cross-lingual learning ” , which also includes the study of the zero-shot setting where no target language task data is given . 2Hence the terms domain and language are used interchangeably in this work . g ( x ) ∈ Z ⊆ Rd the feature vector computed on an input sentence or token x ∈ X . In practice as well as in our studies and experiments , the feature is the last-layer wordpiece embeddings for token classification tasks , and the embedding of a special start-of-sentence marker ( symbolized as [ CLS ] ) for sequence classification tasks . Hence , on BERT-Base models , d = 768 and Z ≈ Sd−1 sphere due to LayerNorm . Let h : Z → Y denote the output layer ( s ) added on top of g with an appropriate activation , usually a single linear layer with Softmax for classification tasks in practice . Hence , an end-to-end model is obtained , which computes predictions via ŷ = h ( g ( x ) ) . By fine-tuning both g , h on labeled task data ( x , y ) sampled from the source language domain pS only , we get a zeroshot transfer model for the downstream task at hand , where the term zero-shot points to the fact that the model does not use any task data from the target language ( x′ , y′ ) ∼ pT during training . Based on the following decomposition of the joint feature-label distribution into the product of class-conditional feature distribution and marginal prior distribution , p ( z , y ) = p ( z|y ) p ( y ) : = p ( { x ∈ X : g ( x ) = z } |y ) p ( y ) , we examine how cross-lingual transfer performance is influenced by ( 1 ) invariance of the feature representation of data between the source and target languages , with perfect invariance being pS ( z|Y = j ) = pT ( z|Y = j ) for all j ∈ Y , and ( 2 ) the distributional shift in the class priors , with no shift being pS ( Y = j ) = pT ( Y = j ) for all j ∈ Y . On classification tasks , Y = { 1 , 2 , · · · , k } is discrete . For legibility in later sections , we use the shorthands to denote the class-conditional feature distribution and the marginal prior distribution : pZ|Y and pY . Setup . Our empirical analysis is performed on the representations learned by mBERT zero-shot fine-tuned on English for two multilingual tasks : sentiment analysis on the Multilingual Amazon Reviews Corpus ( MARC ) , and named-entity recognition ( NER ) on the WikiANN dataset ( see descriptions in Section 4 ) . To simulate the effects of varying amount of representation misalignment and class prior difference , evaluation is performed on 500 smaller datasets randomly subsampled from the original MARC according to different class priors , and roughly 700 datasets from WikiANN . MARC covers five high-resource languages ( excluding English ) , and ( our selected subset of ) WikiANN covers 39 languages of distinct linguistic properties and varying resourcefulness . 2.1 INVARIANCE OF FEATURE REPRESENTATIONS . For studying the effects of representation invariance , we compare model performance to the discrepancy between the class-conditional feature distributions on the source and target domains , pZ|YS and pZ|YT . We take the average to aggregate the discrepancies computed when conditioned on each class , 1 k ∑k j=1D ( p Z|Y =j S , p Z|Y =j T ) , and refer to this quantity by conditional feature shift . The discrepancy measure we use computes the ` 1-distance between the mean feature vectors3 , D ( p , q ) : = ∥∥∥Ex∼p [ x ] − Ex′∼q [ x′ ] ∥∥∥ 1 = d∑ i=1 ∣∣∣Ex∼p [ xi ] − Ex′∼q [ x′i ] ∣∣∣ . Measuring the distance between the means of representations for the purpose of analyzing mBERT has appeared in prior work . A similar quantity is referred to as language centroids by Libovický et al . ( 2020 ) . The important distinction here is that we explicitly remove the effects of class prior shift by conditioning on the class labels , which we argue is an artifact of the dataset since arbitrary amounts of unconditioned feature shift can be generated by altering the class priors . First , we consider zero-shot transfer . In Fig . 1a , we plot theF1 score4 attained by the zero-shot model against conditional feature shift between English features and those of the target language , and observe that better representation alignment is indicative of better transfer performance . Findings in prior work on the cross-lingual ability of mBERT are reflected . For instance , linguistic similarity , a factor commonly argued to be important for cross-lingual transfer , could be related to the alignment 3Comparisons to more sophisticated metrics such as RBF-kernel MMD and linear CKA were made in our preliminary studies ( Gretton et al. , 2012 ; Kornblith et al. , 2019 ) , and we found that they correlate tightly with our first-moment discrepancy measure for analyzing conditional feature shift . 4To study the effects of class prior shift , we use macro-averaged F1 as the performance metric in this section , because classification accuracy could be inflated on datasets with skewed priors ( Azizzadenesheli et al. , 2018 ) . of features : Spanish , French and German are more linguistically similar to English than Japanese and Chinese , whose features are less aligned with those of English . These results also reflects the successes of zero-shot learning approaches on mBERT that aim to achieve representation invariance . While the above finding shows that representation alignment is desirable for zero-shot transfer , is it also suitable for UCL settings where task data from the target language are available ? We start by recognizing that the objective function of most unsupervised transfer algorithms consists of two parts : loss on labeled source data , and terms involving unlabeled data that in theory are linked to the loss on target domain . Based on this , we compared against conditional feature shift the performance of the model fine-tuned supervisedly on labeled data from source and target languages , as it would be the ideal objective for unsupervised algorithms . The results are plotted in Fig . 1b . Instead of observing increases in conditional feature shifts , which would have implied that language-specific features are preferred by over shared ones for good performance on each language , we observe much-improved representation alignments between English and each of the target languages . This means that achieving representation invariance is also appropriate under UCL settings . To demonstrate that the correlation between last-layer class-conditional alignment and performance is not an artifact of the linear classification head , we show that model performance is also correlated with ( 1 ) conditional feature shift in intermediate layers , plotted on the bottom panels of Fig . 1 , and ( 2 ) class-balanced feature shift , plotted in Fig . 5 in the appendix , where the removal of prior shift effects is achieved by reweighting the class priors to be uniform ( eq . 1 ) . Furthermore , we include results on the XNLI dataset , a higher-level semantic task ( Fig . 7 ) , and results on MARC with the more powerful XLM-R Large model ( Fig . 8 ) .

Summary: The paper first demonstrates the importance of feature invariance (language invariant representations) and class-prior invariance across languages on zero-shot cross-lingual performance. By analyzing the zero-shot performance on different languages on the MARC reviews and WikiANN NER tasks, and comparing against the class conditional distance between the source language (English) and target language feature representations, the authors illustrate the how high feature invariance results in better zero-shot performance. By also synthetically modifying the class prior for the target language, the authors demonstrate how increasing differences in class priors result in decreasing zero-shot cross-lingual transfer performance.  Building on their observations, the authors propose an approach that: (i) Introduces an adversarial loss term to penalize distortion in average class conditional feature representations between the source and target languages. (ii) Adds an importance weighting term to ensure the approach doesn't fail under class prior shifts.  Empirical studies demonstrate that the proposed approach improves significantly over the vanilla zero-shot model on both MARC sentiment analysis and NER tasks, also improving over self-training on sentiment analysis. Comparisons on synthetic datasets (sub-sampled from NER and MARC datasets) that enhance the class prior shift highlight the robustness of the approach under large class prior shifts where previous approaches fail.