XLA: A Robust Unsupervised Data Augmentation Framework for Cross-Lingual NLP

1 INTRODUCTION . Self-supervised learning in the form of pretrained language models ( LM ) has been the driving force in developing state-of-the-art natural language processing ( NLP ) systems in recent years . These methods typically follow two basic steps , where a supervised task-specific fine-tuning follows a large-scale LM pretraining ( Devlin et al. , 2019 ; Radford et al. , 2019 ) . However , getting annotated data for every target task in every target language is difficult , especially for low-resource languages . Recently , the pretrain-finetune paradigm has also been extended to multi-lingual setups to train effective multi-lingual models that can be used for zero-shot cross-lingual transfer . Jointly trained deep contextualized multi-lingual LMs like mBERT ( Devlin et al. , 2019 ) and XLM-R ( Conneau et al. , 2020 ) coupled with supervised fine-tuning in the source language have been quite successful in transferring linguistic and task knowledge from one language to another without using any task label in the target language . The joint pretraining with multiple languages allows these models to generalize across languages . Despite their effectiveness , recent studies ( Pires et al. , 2019 ; K et al. , 2020 ) have also highlighted one crucial limiting factor for successful cross-lingual transfer . They all agree that the cross-lingual generalization ability of the model is limited by the ( lack of ) structural similarity between the source and target languages . For example , for transferring mBERT from English , K et al . ( 2020 ) report about 23.6 % accuracy drop in Hindi ( structurally dissimilar ) compared to 9 % drop in Spanish ( structurally similar ) in cross-lingual natural language inference ( XNLI ) . The difficulty level of transfer is further exacerbated if the ( dissimilar ) target language is low-resourced , as the joint pretraining step may not have seen many instances from this language in the first place . In our experiments ( §4.2 ) , in cross-lingual NER ( XNER ) , we report F1 reductions of 28.3 % in Urdu and 30.4 % in Burmese for XLM-R , which is trained on a much larger multi-lingual dataset than mBERT . One attractive way to improve cross-lingual generalization is to perform data augmentation ( Simard et al. , 1998 ) , and train the model ( e.g. , XLM-R ) on examples that are similar but different from the labeled data in the source language . Formalized by the Vicinal Risk Minimization ( VRM ) principle ( Chapelle et al. , 2001 ) , such data augmentation methods have shown impressive results recently in computer vision ( Zhang et al. , 2018 ; Berthelot et al. , 2019 ; Li et al. , 2020a ) . These methods enlarge the support of the training distribution by generating new data points from a vicinity distribution around each training example . For images , the vicinity of a training image can be defined by a set of operations like rotation and scaling , or by linear mixtures of features and labels ( Zhang et al. , 2018 ) . However , when it comes to text , such methods have rarely been successful . The main reason is that unlike images , linguistic units ( e.g. , words , phrases ) are discrete and a smooth change in their embeddings may not result in a plausible linguistic unit that has similar meanings . In NLP , the most successful data augmentation method has so far been back-translation ( Sennrich et al. , 2016 ) which generates paraphrases of an input sentence through round-trip translations . However , it requires parallel data to train effective machine translation systems , acquiring which can be more expensive for low-resource languages than annotating the target language data with task labels . Furthermore , back-translation is only applicable in a supervised setup and to tasks where it is possible to find the alignments between the original labeled entities and the back-translated entities , such as in question answering ( Yu et al. , 2018 ; Dong et al. , 2017 ) . In this work , we propose XLA , a robust unsupervised cross-lingual augmentation framework for improving cross-lingual generalization of multilingual LMs . XLA augments data from the unlabeled training examples in the target language as well as from the virtual input samples ( sentences ) generated from the vicinity distribution of the source and target language sentences . With the augmented data , it performs simultaneous self-learning with an effective distillation strategy to learn a strongly adapted cross-lingual model from noisy ( pseudo ) labels for the target language task . We propose novel ways to generate virtual sentences using a multilingual masked LM ( Conneau et al. , 2020 ) , and get reliable task labels by simultaneous multilingual co-training . This co-training employs a two-stage co-distillation process to ensure robust transfer to dissimilar and/or low-resource languages . We validate the effectiveness and robustness of XLA by performing extensive experiments on three different zero-resource cross-lingual transfer tasks – XNER , XNLI , and PAWS-X , which posit different sets of challenges . We have experimented with many different language pairs ( 14 in total ) comprising languages that are similar/dissimilar/low-resourced . XLA yields impressive results on XNER , setting SoTA in all tested languages outperforming the baselines by a good margin . In particular , the relative gains for XLA are higher for structurally dissimilar and/or low-resource languages , where the base model is weaker : 28.54 % , 16.05 % , and 9.25 % absolute improvements for Urdu , Burmese , and Arabic , respectively . For XNLI , with only 5 % labeled data in the source , it gets comparable results to the baseline that uses all the labeled data , and surpasses the standard baseline by 2.55 % on average when it uses all the labeled data in the source . We also have similar findings in PAWS-X . We provide a comprehensive analysis of the factors that contribute to XLA ’ s performance . 2 BACKGROUND . Contextual representation and cross-lingual transfer In recent years , significant progress has been made in learning contextual word representations and pretrained models . Notably , BERT ( Devlin et al. , 2019 ) pretrains a Transformer ( Vaswani et al. , 2017 ) encoder with a masked language model ( MLM ) objective , and uses the same model architecture to adapt to a new task . It also comes with a multilingual version mBERT , which is trained jointly on 102 languages . RoBERTa ( Liu et al. , 2019 ) extends BERT with improved training , while XLM ( Lample and Conneau , 2019 ) extends mBERT with a conditional LM and a translation LM ( using parallel data ) objectives . Conneau et al . ( 2020 ) train the largest multilingual language model XLM-R with RoBERTa framework . Despite any explicit cross-lingual supervision , mBERT and its variants have been shown to learn cross-lingual representations that generalize well across languages . Wu and Dredze ( 2019 ) and Pires et al . ( 2019 ) evaluate the zero-shot cross-lingual transferability of mBERT on several tasks and attribute its generalization capability to shared subword units . Pires et al . ( 2019 ) also found structural similarity ( e.g. , word order ) to be another important factor for successful cross-lingual transfer . K et al . ( 2020 ) , however , show that the shared subword has a minimal contribution ; instead , the structural similarity between languages is more crucial for effective transfer ( more in Appendix D ) . Vicinal risk minimization ( VRM ) Data augmentation supported by the VRM principle ( Chapelle et al. , 2001 ) can be an effective choice for achieving better out-of-distribution adaptation . In VRM , we minimize the empirical vicinal risk defined as : Lv ( θ ) = 1N ∑N n=1 l ( fθ ( x̃n ) , ỹn ) , where fθ denotes the model parameterized by θ , and D̃ = { ( x̃n , ỹn ) } Nn=1 is an augmented dataset constructed by sampling the vicinal distribution ϑ ( x̃i , ỹi|xi , yi ) around the original training sample ( xi , yi ) . Defining vicinity is however quite challenging as it requires the extraction of samples from a distribution without hurting their labels . Earlier methods apply simple rules like rotation and scaling of images ( Simard et al. , 1998 ) . Recent work ( Zhang et al. , 2018 ; Berthelot et al. , 2019 ) show impressive results in image classification with simple linear interpolation of data . However , to our knowledge , none of these methods have so far been successful in NLP due to the discrete nature of texts.1 LM-based supervised augmentation Recently , a number of data-augmentation methods have been proposed using contextualized LMs like BERT , e.g. , Contextual Augmentation ( Kobayashi , 2018 ) , Conditional BERT ( Wu et al. , 2018 ) , and AUG-BERT ( Wu et al. , 2018 ) . These approaches use a constrained augmentation method which alters a pretrained LM to a label-conditional LM for a specific task . This means these methods update the parameters of the pretrained LM using the labels . 3 XLA FRAMEWORK . While recent cross-lingual transfer learning efforts have relied almost exclusively on multi-lingual pretraining and zero-shot transfer of a fine-tuned source model , there is a great potential for more elaborate methods that can leverage the unlabeled data better . Motivated by this , we present XLA - our unsupervised data augmentation framework for zero-resource cross-lingual task adaptation . Figure 1 gives an overview of XLA . Let Ds = ( Xs , Ys ) and Dt = ( Xt ) denote the training data for a source language s and a target language t , respectively . XLA augments data from various origins at different stages of training . In the initial stage ( epoch 1 ) , it uses the augmented training samples from the target language ( D′t ) along with the original source ( Ds ) . In later stages ( epoch 2-3 ) , it uses virtual ( vicinal ) sentences generated from the vicinity distribution of source and target examples : ϑ ( x̃sn|xsn ) and ϑ ( x̃tn|xtn ) , where xsn ∼ Xs and xtn ∼ Xt . It performs self-training on the augmented data to acquire the corresponding pseudo labels . To avoid confirmation bias with self-training where the model accumulates its own errors , it simultaneously trains three task models to generate virtual training data through data augmentation and filtering of potential label noises via multi-epoch co-teaching ( Zhou and Li , 2005 ) . In each epoch , the co-teaching process first performs co-distillation , where two peer task models are used to select “ reliable ” training examples to train the third model . The selected samples with pseudo labels are then added to the target task model ’ s training data by taking the agreement from the other two models , a process we refer to as co-guessing . The co-distillation and co-guessing mechanism ensure robustness of XLA to out-of-domain distributions that can occur in a multilingual setup , e.g. , due to a structurally dissimilar and/or low-resource target language . Algorithm 1 gives a pseudocode of the overall training method . Each of the task models in XLA is an instance of XLM-R fine-tuned on the source language task ( e.g. , English NER ) , whereas the pretrained masked LM parameterized by θmlm ( i.e. , before fine-tuning ) is used to define the vicinity distribution ϑ ( x̃n|xn , θmlm ) around each selected example xn . 1Considering papers that have been published ( or accepted ) through peer review . There has been some concurrent work that uses pretrained LMs like BERT to craft adversarial examples ( Li et al. , 2020b ) . Although relevant , these methods have a different objective than ours , and none of them is cross- or multi-lingual . Algorithm 1 XLA : a robust unsupervised data augmentation framework for cross-lingual NLP Input : source ( s ) and target ( t ) language datasets : Ds = ( Xs , Ys ) , Dt = ( Xt ) ; task models : θ ( 1 ) , θ ( 2 ) , θ ( 3 ) , pre-trained masked LM θmlm , mask ratio P , diversification factor δ , sampling factor α , and distillation factor η Output : models trained on augmented data 1 : θ ( 1 ) , θ ( 2 ) , θ ( 3 ) = WARMUP ( Ds , θ ( 1 ) , θ ( 2 ) , θ ( 3 ) ) . warm up with conf . penalty . 2 : for e ∈ [ 1 : 3 ] do . e denotes epoch . 3 : for k ∈ { 1 , 2 , 3 } do 4 : X ( k ) t , Y ( k ) t = DISTIL ( Xt , ηe , θ ( k ) ) . infer and select tgt training data for augmentation . 5 : for j ∈ { 1 , 2 , 3 } do 6 : if k == j then Continue 7 : / * source language data augmentation * / 8 : X̃s = GEN-LM ( Xs , θmlm , P , δ ) . vicinal example generation . 9 : X ( k ) s , Y ( k ) s = DISTIL ( X̃s , ηe , θ ( k ) ) ; X ( j ) s , Y ( j ) s = DISTIL ( X̃s , ηe , θ ( j ) ) 10 : D̃s = AGREEMENT ( D ( k ) s = ( X ( k ) s , Y ( k ) s ) , D ( j ) s = ( X ( j ) s , Y ( j ) s ) ) 11 : / * target language data augmentation ( no vicinity ) * / 12 : X ( j ) t , Y ( j ) t = DISTIL ( Xt , ηe , θ ( j ) ) 13 : D′t = AGREEMENT ( D ( k ) t = ( X ( k ) t , Y ( k ) t ) , D ( j ) t = ( X ( j ) t , Y ( j ) t ) ) . see line 4 14 : / * target language data augmentation * / 15 : X̃t = GEN-LM ( Xt , θmlm , P , δ ) . vicinal example generation . 16 : X ( k ) t , Y ( k ) t = DISTIL ( X̃t , ηe , θ ( k ) ) ; X ( j ) t , Y ( j ) t = DISTIL ( X̃t , ηe , θ ( j ) ) 17 : D̃t = AGREEMENT ( D ( k ) t = ( X ( k ) t , Y ( k ) t ) , D ( j ) t = ( X ( j ) t , Y ( j ) t ) ) 18 : / * train new models on augmented data * / 19 : for l ∈ { 1 , 2 , 3 } do 20 : if l 6= j and l 6= k then 21 : with sampling factor α , train θ ( l ) on D , . train progressively 22 : where D = { Ds1 ( e ∈ { 1 , 3 } ) ∪ D′t1 ( e ∈ { 1 , 3 } ) ∪ D̃s1 ( e = 3 ) ∪ D̃t1 ( e ∈ { 2 , 3 } ) } 23 : Return { θ ( 1 ) , θ ( 2 ) , θ ( 3 ) } Although the data augmentation proposed in Contextual Augmentation ( Kobayashi , 2018 ) , Conditional BERT ( Wu et al. , 2018 ) , AUG-BERT ( Wu et al. , 2018 ) also use a pretrained masked LM , there are some fundamental differences between our method and these approaches . Unlike these approaches our vicinity-based LM augmentation is purely unsupervised and we do not perform any fine-tuning of the pretrained vicinity model ( θlm ) . The vicinity model in XLA is a disjoint pretrained entity whose weights are not trained on any task objective . This disjoint characteristic gives our framework the flexibility to replace θlm even with a better monolingual LM for a specific target language , which in turn makes XLA extendable to utilize stronger LMs that may come in the future . In the following , we describe the steps in Algorithm 1 .

Summary: The paper  presents a data augmentation framework for zero-shot cross-lingual transfer learning. The framework uses different types of data (labeled source data, unlabeled source data, automatically generated augmented data) for training a model for the target language. Experiments are conducted on three different tasks: Named Entity Recognition (NER), Natural Language Inference (NLI) and paraphrase identification (PAWS). The approach combines multiple components together namely self-training, augmented sentence generation and confidence penalty.