XLA: A Robust Unsupervised Data Augmentation Framework for Cross-Lingual NLP

1 INTRODUCTION . Self-supervised learning in the form of pretrained language models ( LM ) has been the driving force in developing state-of-the-art natural language processing ( NLP ) systems in recent years . These methods typically follow two basic steps , where a supervised task-specific fine-tuning follows a large-scale LM pretraining ( Devlin et al. , 2019 ; Radford et al. , 2019 ) . However , getting annotated data for every target task in every target language is difficult , especially for low-resource languages . Recently , the pretrain-finetune paradigm has also been extended to multi-lingual setups to train effective multi-lingual models that can be used for zero-shot cross-lingual transfer . Jointly trained deep contextualized multi-lingual LMs like mBERT ( Devlin et al. , 2019 ) and XLM-R ( Conneau et al. , 2020 ) coupled with supervised fine-tuning in the source language have been quite successful in transferring linguistic and task knowledge from one language to another without using any task label in the target language . The joint pretraining with multiple languages allows these models to generalize across languages . Despite their effectiveness , recent studies ( Pires et al. , 2019 ; K et al. , 2020 ) have also highlighted one crucial limiting factor for successful cross-lingual transfer . They all agree that the cross-lingual generalization ability of the model is limited by the ( lack of ) structural similarity between the source and target languages . For example , for transferring mBERT from English , K et al . ( 2020 ) report about 23.6 % accuracy drop in Hindi ( structurally dissimilar ) compared to 9 % drop in Spanish ( structurally similar ) in cross-lingual natural language inference ( XNLI ) . The difficulty level of transfer is further exacerbated if the ( dissimilar ) target language is low-resourced , as the joint pretraining step may not have seen many instances from this language in the first place . In our experiments ( §4.2 ) , in cross-lingual NER ( XNER ) , we report F1 reductions of 28.3 % in Urdu and 30.4 % in Burmese for XLM-R , which is trained on a much larger multi-lingual dataset than mBERT . One attractive way to improve cross-lingual generalization is to perform data augmentation ( Simard et al. , 1998 ) , and train the model ( e.g. , XLM-R ) on examples that are similar but different from the labeled data in the source language . Formalized by the Vicinal Risk Minimization ( VRM ) principle ( Chapelle et al. , 2001 ) , such data augmentation methods have shown impressive results recently in computer vision ( Zhang et al. , 2018 ; Berthelot et al. , 2019 ; Li et al. , 2020a ) . These methods enlarge the support of the training distribution by generating new data points from a vicinity distribution around each training example . For images , the vicinity of a training image can be defined by a set of operations like rotation and scaling , or by linear mixtures of features and labels ( Zhang et al. , 2018 ) . However , when it comes to text , such methods have rarely been successful . The main reason is that unlike images , linguistic units ( e.g. , words , phrases ) are discrete and a smooth change in their embeddings may not result in a plausible linguistic unit that has similar meanings . In NLP , the most successful data augmentation method has so far been back-translation ( Sennrich et al. , 2016 ) which generates paraphrases of an input sentence through round-trip translations . However , it requires parallel data to train effective machine translation systems , acquiring which can be more expensive for low-resource languages than annotating the target language data with task labels . Furthermore , back-translation is only applicable in a supervised setup and to tasks where it is possible to find the alignments between the original labeled entities and the back-translated entities , such as in question answering ( Yu et al. , 2018 ; Dong et al. , 2017 ) . In this work , we propose XLA , a robust unsupervised cross-lingual augmentation framework for improving cross-lingual generalization of multilingual LMs . XLA augments data from the unlabeled training examples in the target language as well as from the virtual input samples ( sentences ) generated from the vicinity distribution of the source and target language sentences . With the augmented data , it performs simultaneous self-learning with an effective distillation strategy to learn a strongly adapted cross-lingual model from noisy ( pseudo ) labels for the target language task . We propose novel ways to generate virtual sentences using a multilingual masked LM ( Conneau et al. , 2020 ) , and get reliable task labels by simultaneous multilingual co-training . This co-training employs a two-stage co-distillation process to ensure robust transfer to dissimilar and/or low-resource languages . We validate the effectiveness and robustness of XLA by performing extensive experiments on three different zero-resource cross-lingual transfer tasks – XNER , XNLI , and PAWS-X , which posit different sets of challenges . We have experimented with many different language pairs ( 14 in total ) comprising languages that are similar/dissimilar/low-resourced . XLA yields impressive results on XNER , setting SoTA in all tested languages outperforming the baselines by a good margin . In particular , the relative gains for XLA are higher for structurally dissimilar and/or low-resource languages , where the base model is weaker : 28.54 % , 16.05 % , and 9.25 % absolute improvements for Urdu , Burmese , and Arabic , respectively . For XNLI , with only 5 % labeled data in the source , it gets comparable results to the baseline that uses all the labeled data , and surpasses the standard baseline by 2.55 % on average when it uses all the labeled data in the source . We also have similar findings in PAWS-X . We provide a comprehensive analysis of the factors that contribute to XLA ’ s performance . 2 BACKGROUND . Contextual representation and cross-lingual transfer In recent years , significant progress has been made in learning contextual word representations and pretrained models . Notably , BERT ( Devlin et al. , 2019 ) pretrains a Transformer ( Vaswani et al. , 2017 ) encoder with a masked language model ( MLM ) objective , and uses the same model architecture to adapt to a new task . It also comes with a multilingual version mBERT , which is trained jointly on 102 languages . RoBERTa ( Liu et al. , 2019 ) extends BERT with improved training , while XLM ( Lample and Conneau , 2019 ) extends mBERT with a conditional LM and a translation LM ( using parallel data ) objectives . Conneau et al . ( 2020 ) train the largest multilingual language model XLM-R with RoBERTa framework . Despite any explicit cross-lingual supervision , mBERT and its variants have been shown to learn cross-lingual representations that generalize well across languages . Wu and Dredze ( 2019 ) and Pires et al . ( 2019 ) evaluate the zero-shot cross-lingual transferability of mBERT on several tasks and attribute its generalization capability to shared subword units . Pires et al . ( 2019 ) also found structural similarity ( e.g. , word order ) to be another important factor for successful cross-lingual transfer . K et al . ( 2020 ) , however , show that the shared subword has a minimal contribution ; instead , the structural similarity between languages is more crucial for effective transfer ( more in Appendix D ) . Vicinal risk minimization ( VRM ) Data augmentation supported by the VRM principle ( Chapelle et al. , 2001 ) can be an effective choice for achieving better out-of-distribution adaptation . In VRM , we minimize the empirical vicinal risk defined as : Lv ( θ ) = 1N ∑N n=1 l ( fθ ( x̃n ) , ỹn ) , where fθ denotes the model parameterized by θ , and D̃ = { ( x̃n , ỹn ) } Nn=1 is an augmented dataset constructed by sampling the vicinal distribution ϑ ( x̃i , ỹi|xi , yi ) around the original training sample ( xi , yi ) . Defining vicinity is however quite challenging as it requires the extraction of samples from a distribution without hurting their labels . Earlier methods apply simple rules like rotation and scaling of images ( Simard et al. , 1998 ) . Recent work ( Zhang et al. , 2018 ; Berthelot et al. , 2019 ) show impressive results in image classification with simple linear interpolation of data . However , to our knowledge , none of these methods have so far been successful in NLP due to the discrete nature of texts.1 LM-based supervised augmentation Recently , a number of data-augmentation methods have been proposed using contextualized LMs like BERT , e.g. , Contextual Augmentation ( Kobayashi , 2018 ) , Conditional BERT ( Wu et al. , 2018 ) , and AUG-BERT ( Wu et al. , 2018 ) . These approaches use a constrained augmentation method which alters a pretrained LM to a label-conditional LM for a specific task . This means these methods update the parameters of the pretrained LM using the labels . 3 XLA FRAMEWORK . While recent cross-lingual transfer learning efforts have relied almost exclusively on multi-lingual pretraining and zero-shot transfer of a fine-tuned source model , there is a great potential for more elaborate methods that can leverage the unlabeled data better . Motivated by this , we present XLA - our unsupervised data augmentation framework for zero-resource cross-lingual task adaptation . Figure 1 gives an overview of XLA . Let Ds = ( Xs , Ys ) and Dt = ( Xt ) denote the training data for a source language s and a target language t , respectively . XLA augments data from various origins at different stages of training . In the initial stage ( epoch 1 ) , it uses the augmented training samples from the target language ( D′t ) along with the original source ( Ds ) . In later stages ( epoch 2-3 ) , it uses virtual ( vicinal ) sentences generated from the vicinity distribution of source and target examples : ϑ ( x̃sn|xsn ) and ϑ ( x̃tn|xtn ) , where xsn ∼ Xs and xtn ∼ Xt . It performs self-training on the augmented data to acquire the corresponding pseudo labels . To avoid confirmation bias with self-training where the model accumulates its own errors , it simultaneously trains three task models to generate virtual training data through data augmentation and filtering of potential label noises via multi-epoch co-teaching ( Zhou and Li , 2005 ) . In each epoch , the co-teaching process first performs co-distillation , where two peer task models are used to select “ reliable ” training examples to train the third model . The selected samples with pseudo labels are then added to the target task model ’ s training data by taking the agreement from the other two models , a process we refer to as co-guessing . The co-distillation and co-guessing mechanism ensure robustness of XLA to out-of-domain distributions that can occur in a multilingual setup , e.g. , due to a structurally dissimilar and/or low-resource target language . Algorithm 1 gives a pseudocode of the overall training method . Each of the task models in XLA is an instance of XLM-R fine-tuned on the source language task ( e.g. , English NER ) , whereas the pretrained masked LM parameterized by θmlm ( i.e. , before fine-tuning ) is used to define the vicinity distribution ϑ ( x̃n|xn , θmlm ) around each selected example xn . 1Considering papers that have been published ( or accepted ) through peer review . There has been some concurrent work that uses pretrained LMs like BERT to craft adversarial examples ( Li et al. , 2020b ) . Although relevant , these methods have a different objective than ours , and none of them is cross- or multi-lingual . Algorithm 1 XLA : a robust unsupervised data augmentation framework for cross-lingual NLP Input : source ( s ) and target ( t ) language datasets : Ds = ( Xs , Ys ) , Dt = ( Xt ) ; task models : θ ( 1 ) , θ ( 2 ) , θ ( 3 ) , pre-trained masked LM θmlm , mask ratio P , diversification factor δ , sampling factor α , and distillation factor η Output : models trained on augmented data 1 : θ ( 1 ) , θ ( 2 ) , θ ( 3 ) = WARMUP ( Ds , θ ( 1 ) , θ ( 2 ) , θ ( 3 ) ) . warm up with conf . penalty . 2 : for e ∈ [ 1 : 3 ] do . e denotes epoch . 3 : for k ∈ { 1 , 2 , 3 } do 4 : X ( k ) t , Y ( k ) t = DISTIL ( Xt , ηe , θ ( k ) ) . infer and select tgt training data for augmentation . 5 : for j ∈ { 1 , 2 , 3 } do 6 : if k == j then Continue 7 : / * source language data augmentation * / 8 : X̃s = GEN-LM ( Xs , θmlm , P , δ ) . vicinal example generation . 9 : X ( k ) s , Y ( k ) s = DISTIL ( X̃s , ηe , θ ( k ) ) ; X ( j ) s , Y ( j ) s = DISTIL ( X̃s , ηe , θ ( j ) ) 10 : D̃s = AGREEMENT ( D ( k ) s = ( X ( k ) s , Y ( k ) s ) , D ( j ) s = ( X ( j ) s , Y ( j ) s ) ) 11 : / * target language data augmentation ( no vicinity ) * / 12 : X ( j ) t , Y ( j ) t = DISTIL ( Xt , ηe , θ ( j ) ) 13 : D′t = AGREEMENT ( D ( k ) t = ( X ( k ) t , Y ( k ) t ) , D ( j ) t = ( X ( j ) t , Y ( j ) t ) ) . see line 4 14 : / * target language data augmentation * / 15 : X̃t = GEN-LM ( Xt , θmlm , P , δ ) . vicinal example generation . 16 : X ( k ) t , Y ( k ) t = DISTIL ( X̃t , ηe , θ ( k ) ) ; X ( j ) t , Y ( j ) t = DISTIL ( X̃t , ηe , θ ( j ) ) 17 : D̃t = AGREEMENT ( D ( k ) t = ( X ( k ) t , Y ( k ) t ) , D ( j ) t = ( X ( j ) t , Y ( j ) t ) ) 18 : / * train new models on augmented data * / 19 : for l ∈ { 1 , 2 , 3 } do 20 : if l 6= j and l 6= k then 21 : with sampling factor α , train θ ( l ) on D , . train progressively 22 : where D = { Ds1 ( e ∈ { 1 , 3 } ) ∪ D′t1 ( e ∈ { 1 , 3 } ) ∪ D̃s1 ( e = 3 ) ∪ D̃t1 ( e ∈ { 2 , 3 } ) } 23 : Return { θ ( 1 ) , θ ( 2 ) , θ ( 3 ) } Although the data augmentation proposed in Contextual Augmentation ( Kobayashi , 2018 ) , Conditional BERT ( Wu et al. , 2018 ) , AUG-BERT ( Wu et al. , 2018 ) also use a pretrained masked LM , there are some fundamental differences between our method and these approaches . Unlike these approaches our vicinity-based LM augmentation is purely unsupervised and we do not perform any fine-tuning of the pretrained vicinity model ( θlm ) . The vicinity model in XLA is a disjoint pretrained entity whose weights are not trained on any task objective . This disjoint characteristic gives our framework the flexibility to replace θlm even with a better monolingual LM for a specific target language , which in turn makes XLA extendable to utilize stronger LMs that may come in the future . In the following , we describe the steps in Algorithm 1 .

Summary: The authors present an unsupervised data augmentation framework for cross-lingual NLP. Their method, called XLA, combines self-learning with co-learning and filtering. They generate additional synthetic examples by replacing words with predictions from pretrained multilingual masked LM (taken from XLM Conneau 2020). The key contribution in this paper is how they get reliable labels by simultaneously co-training three student models and using them to filter examples for training each other to avoid confirmation bias.