XLA: A Robust Unsupervised Data Augmentation Framework for Cross-Lingual NLP
Authors: M Saiful Bari
Tasnim Mohiuddin
Shafiq Joty
Venue:  (2021)
Keywords: multi-lingual
cross-lingual
data-augmentation
nlp

Abstract: Transfer learning has yielded state-of-the-art (SoTA) results in many supervised NLP tasks. However,  annotated data for every target task in every target language is rare, especially for low-resource languages. We propose XLA, a novel data augmentation framework for self-supervised learning in zero-resource transfer learning scenarios. In particular, XLA aims to solve cross-lingual adaptation problems from a source language task distribution to an unknown target language task distribution, assuming no training label in the target language task. At its core, XLA performs simultaneous self-training with data augmentation and unsupervised sample selection. To show its effectiveness, we conduct extensive experiments on zero-resource cross-lingual transfer tasks for Named Entity Recognition (NER), Natural Language Inference (NLI) and paraphrase identification on Paraphrase Adversaries from Word Scrambling (PAWS). XLA achieves SoTA results in all the tasks, outperforming the baselines by a good margin. With an in-depth framework dissection, we demonstrate the cumulative contributions of different components to XLA's success. 

Under review as a conference paper at ICLR 2021
XLA: A ROBUST UNSUPERVISED DATA AUGMENTA- TION FRAMEWORK FOR -OOS-LLN NLP
Anonymous authors Paper under double-blind review
Abstract
Transfer learning has yielded state-of-the-ar (SoTA) results in many supervised NLP tasks. However, annotated data for every target task in every target language is rare, especially for low-resource languages. We propose XLA, a novel data augmentation framework for self-supervised learning in zero-resource transfer learning scenarios. In particular, XLA aims solve Pross-lingual adaptation prob- lems from a source language task distribution to an unknown target language task distribution, assuming no training label in the target language task. At its core, XI.A performs simultaneous self-training with data augmentation and unsupervised sample selection. To show its effectiveness, we conduct extensive experiments on zero-resource cross-lingual transfer tasks for Named Entity Recognition (NER), Natural Language Inference (NLI) and paraphrase identification on Paraphrase Adversaries from Word Scrambling (PAWS). XLA achieves SoTA results in all the tasks, outperforming the baselines by a good margin. With an in-depth framework dissection, demonstrate the cumulative contributions of different components to XLA's success.
INTRODUCTION
Self-supervised learning in the form of pretrained language models (LM) has been the driving force in developing state-of-the-art natural language processing (NLP) systems in recent years. These methods typically follow two basic steps, where a supervised task-specific fine-tuning follows a large-scale scale LM pretraining (Devlin al. 2019 Radford al.) (019). However, getting annotated data for every target task in every target language is difficult, especially for low-resource languages.
Recently, the pretrain-finetune paradigm has also been extended to multi-lingual setups to train effective multi-lingual models that can be used for zero-shot cross- transfer. Jointly trained deep contextualized multi munti-linguul LMs like mBERT (Devlin al.) 2019) and XLM-R (Conneau et al. 2020) coupled with supervised fine-tuning in the source language have been quite successful in transferring linguistic and task knowledge from one language to another without using any task label in the target language. The joint pretraining with multiple languages allows these models generalize across languages. Despite their effectiveness, recent studies (Pires et al.) 2019 K et al., 2020) have also highlighted one crucial limiting factor for successful gross-lingual transfer. They all agree that the cross-lingual generalization ability of the model is limited by the (lack of) structural similarity between the source and target languages. For example, for transferring mBERT from English, K et al.) (2020) report about 23.6% accuracy drop in Hindi (structurally dissimilar) compared to 9% drop in Spanish (structurally similar) in cross-lingual natural language inference (XNLI). The difficulty level of transfer is further exacerbated if the (dissimilar) target language is low-resourced, as the joint pretraining step may not have seen many instances from this language the first place. In our experiments ($4.2), in cross-lingual NER (XNER), we report F1 reductions of 28.3% in Urdu and 30.4% in Burmese for XLM-R, which is trained on a much larger multi-lingual dataset than mBERT.
One attractive way to improve cross-lingual generalization is to perform data augmentation (Simard ct al. 1998), and train the model (e.g., XLM-R) on examples that are similar but different from the labeled data in the source language. Formalized by the Vicinal Risk Minimization (VRM) principle (Chapelle et al., 2001), such data augmentation methods have shown impressive results recently in computer vision (Zhang al. 2018 Berthelot et al., 2219 Li et al. 2020a). These methods enlarge the support of the training distribution by generating new data points from a vicinity distribution around each training example. For images, the vicinity of training image can be defined by set

---

Under review as a conference paper at ICLR 2021
of operations like rotation and scaling, or by linear mixtures of features and labels (Zhang et al.) 2018). However, when it comes to text, such methods have rarely been successful. The main reason that unlike images, linguistic units (e.g., words, phrases) are discrete and smooth change in their embeddings may not result in a plausible linguistic unit that has similar meanings.
In NLP, the most successful data augmentation method has so far been back trekstrrantttoon (Sennrich et al.) 2016) which generates paraphrases of an input sentence through round-trip translations. However, it requires parallel data to train effective machine translation systems, acquiring which can be more expensive for low-resource languages than annotating the target language data with task labels. Furthermore, back-translation only applicable in a supervised setup and to tasks where is possible to find the alignments between the original labeled entities and the back-tram backstranslated entities, such as in question answering (Yu et al.
2018 Dong et al., 2017).
In this work, we propose XLA, robust unsupervised cross-lingua augmentation framework for improving cross-lingual generalization of multilingual LMs. XLA augments data from the unlabeled training examples in the target language as well as from the virtual input samples (sentences) generated from the vicinity distribution of the source and target language sentences. With the augmented data, performs simultaneous self-learning with an effective distillation strulegy to learn a strongly adapted cross-lingual model from noisy (pseudo) labels for the target language task. We propose novel ways to generate virtual sentences using a multilingual masked LM (Conneau et al. 2020), and get reliable task labels by simultaneous multilingual co-training. This co-training employs a two-stage co-distillation process to ensure robust transfer to dissimilar and/or low-resource languages.
We validate the effectiveness and robustness of XLA by performing extensive experiments on three different zero-resource cross-lingual transfer tasks- XNER, XNLI, and PAWS-X, which posit different sets of challenges. We have experimented with many different language pairs (14 in total) comprising languages that are httplaaddiismiila//oowwrsssuitss XNER, setting SoTA in all tested languages outperforming the baselines by a good margin. In particular, the relative gains for XLA are higher for structurally dissimilar and/or low-resource languages, where the base model is weaker: 28.544 38.54, 16.05%. and 9.25% absolute improvements for Urdu, Burmese, and Arabic, respectively. For XNLI, with only 5% labeled data in the source, gets comparable results to the baseline that uses all the labeled data, and surpasses the standard baseline by 2.55% on average when it uses all the labeled data in the source. We also have similar findings in PAWS-X. We provide a comprehensive analysis of the factors that contribute to XLA's performance.
XLA yields impressive results on
BACKGROUND
Contextual representation and cross-lingual transfer In recent years, significant progress has been made in learning contextual word representations and pretrained models. Notably, BERT (Devlin et al. 2019) pretrains Transformer (Vaswani et al.) 2017) encoder with masked language model (MLM) objective, and uses the same model architecture to adapt to a new task. It also comes with a multilingual version mBERT, which is trained jointly on 102 languages. ROBERTa (Liu al. 2019) extends BERT with improved training, while XLM (Lample and Conneaul 2019) extends m B ERT with a conditional LM and a translation LM (using parallel data) objectives. Conneau et al. (2020) train the largest multilingual language model XLM-R with RoBERTa framework.
Despite any explicit cross-lingua supervision, mBERT and its variants have been shown learn lingsll representations that generalize well across languages. Wu and Dredze (2019) and PPres jet al.) (2019) evaluate the zero-shot cross-lingual transferability of mBERT on several tasks and attribute its generalization capability to shared subword units. |Pres al. (2019) also found structural similarity (e.g., word order) to be another important factor for successful cross-lingual transfer. KK et al. (2020), however, show that the shared subword has a minimal contribution; instead, the structural similarity between languages is more crucial for effective transfer (more in Appendix D).
Vicinal risk minimization (VRM) Data augmentation supported by the VRM principle (Chapelle et al. 2001) can be an effective choice for achieving better out-of-distribution adaptation. In VRM, we minimize the empirical vicinal risk defined as: L.) 1/ Σ.> ((fo(En)) ).)) where fo denotes the
model parameterized by 8, and D- = (@ )n)) N=1 is an augmented dataset constructed by sampling the vicinal distribution (f.,, ys 22,, yt) around the original training sample (I;, y.). Defining vicinity
is however quite challenging as it requires the extraction of samples from a distribution without
hurting their labels. Earlier methods apply simple rules like rotation and scaling of images (Simard
2

---

Under review as a conference paper at ICLR 2021
(
g(2)
Wazetip
(1)
D
Paeudo-label
Labeled
Source
(,K,,
Samples
fit
Co-distilation
Train 0 1+SLiHi
Samples
Gan-Iat (2t, Pajmm)
((())
O
co-diotiiation
25, (3,, 3
Samples
2,
Figure 1: Training flow diagram of XLA. After training the base task models ((1), 0(2), and ((3) source labeled data D, (WarmUp), we use two of them (0U), ((k) to pseudo-label and co-distill the unlabeled target language data (D)). A pretrained LM (Gen-LM) is used to generate new (vicinal) training samples for both source and target languages, which are also then labule-l and co- distilled using the two task models (065, ((*)) to generate D, and D.. The third model (0) is then progressively trained on these datasets: {D,,, D;} in epoch Dr in epoch 2, and all in epoch 3.
et al. 1998). Recent work (Zhang al.) 2018 Berthelot al.) 2019) show impressive results in image classification with simple linear interpolation of data. However, to our knowledge, none of these methods have so far been successful in NLP due to the discrete nature of texts
LM-based supervised augmentation Recently, a number of data-augmentation methods have been proposed using contextualized LMs like BERT, e.g., Contextual Augmentation (Kobayashi 2018). Conditional BERT (Wu al.) 2018), and AUG-BERT (Wu et al., 2018). These approaches use constrained augmentation method which alters pretrained LM to a label-conditional LM for specific task. This means these methods update the parameters of the pretrained LM using the labels.
3 XLA FRAMEWORK
While recent cross-lingual transfer learning efforts have relied almost exclusively on multi-lingual pretraining and zero-shot transfer of fine-tuned source model, there is a great potential for more elaborate methods that can leverage the unlabeled data better. Motivated by this, present XLA our unsupervised data augmentation framework for zero-resource pooss-lnngull task adaptation.
Figure D gives overview of XLA. Let D, (1)) and D) (xx) denote the training data for a source language s and a target language t. respectively. XLA augments data from various origins at different stages of training. In the initial stage (epoch 1), it uses the augmented training samples from the target language (D)) along with the original source (D.). In later stages (epoch 2-3), it uses virtual (vicinal) sentences generated from the vieinity distribution of source and target examples: (Fz (= and (( where x ~ X. and x X. It performs self-training on the augmented data acquire the corresponding pseudo labels. To avoid confirmation bias with self-training where the model accumulates its own errors. it simultancously trains three task models to generate virtual training data through data augmentation and filtering of potential label noises via multi-epoch co-teaching (Zhou and Li 2005). In each epoch, the co-teaching process first performs co-distillation, where two peer task models are used to select "reliable" training examples to train the third model. The selected samples with pseudo labels are then added to the target task model's training data by taking the agreement from the other two models, a process we refer to as co-guessing. The co-distillation and co-guessing mechanism ensure robustness of XLA to out-of-domain distributions that can occur in a multilingual setup, e.g., due to a structurally dissimilar and/or low low-resource target language. Algorithm I gives pseudocode of the overall training method.
Each of the task models in XLA is an instance of XLM-R fine-tuned on the source language task (c.g.. English NER), whereas the pretrained masked LM parameterized by 6mim (i.e., before fine-tuning) is used to define the vicinity distribution 0 En, fmlm) around each selected example C..-
'Considering papers that have been published (or accepted) through peer review. There has been some concurrent work that uses pretrained Mss like BERT to craft adversarial examples (Li et al. (0200). Although relevant, these methods have different objective than ours, and none of them is cross- or multi-ingual.
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 XLA: a robust unsupervised data augmentation framework for cross-lingual NLP
Input: source (8) and targer (t) language datasets: D. ()), D. = (Xt); task models; ,()) g(3), g63), pre pre-trained masked LM Omim. mask ratio P. diversification factor 6, sampling factor a, and distillation factor Output: models trained augmented data
1: 000) 0(22) ((3) WARMUP(D., (()) 000) ((3)) (00))
warm up with conf. penalty. b e denotes epoch.
2: for e [1 3] do
3: for & E {1. 2, do
4:
x, ))(A)
= (A, 0(h)
D infer and select igt training data for augmentation.
5:
for j (1, do
6:
if = then Continue
7:
/+ source language data augmentation
8:
R, GEN-IMM(&, mlm, P)
vicinal example generation.
xxx) y²(² DISTIL Xr, ne, g0h);; X19) 20t00) DISTIL(E ne, (6))
10:
D. AGREEMENT D
(2 ) DJ (x.))
11:
/ target language data augmentation (no vicinity) *//
12:
X(O), yV.i DISTIL Me, 061)
13:
D' (DR (2( , D(u) = (2(1),))0))
see line 4
14:
target language data augmentation
15:
X. = GEN-LM(X,, 0mlm, P,
& vicinal example generation.
(K)) 2/(() ( ne, )) X1 x,1),,(0)) DISTIL ne, 8(5)
16:
D. = (GREEMENT(D." = (2(1) (( D." = ,)()))))))))
17:
18:
/- train new models on augmented data ./
19:
for I E 21,2,3} do
20:
i j and l & & then with sampling factor train 809 on D,
21:
train progressively where (D.1(e e (1,3)) UD e (1,3)) DDA((e UD.a(e e (2,3)))
22:
23:
Return ( g(2) gg)))
Although the data augmentation proposed in Contextual Augmentation (Kobayashi 2018), Condi- tional BERT (Wu et al.] 2018), AUG-BERT (Wu et al. 2018) also use a pretrained masked LM, there are some fundamental differences between our method and these approaches. Unlike these approaches our vicinity-based LMM augmentation is purely unsupervised and we do not perform any fine-tuning of the pretrained vicinity model (6tm). The vicinity model in XLA is a disjoint pretrained entity whose weights are not trained any task objective. This disjoint characteristic gives framework the flexibility to replace 9im even with a better monolingual LM for a specific target language, which in turn makes XLA extendable to utilize stronger LMs that may come in the future.
In the following, we describe the steps in Algorithm
3.1 WARM-UP STEP: TRAINING TASK MODELS WITH CONFIDENCE PENALTY
We first train three instances of the XLM-R model (0(1), ((2) ((3) with an additional task-specific linear layer on the source language (English) labeled data. Each model has the same architecture (XLM-R large) but is initialized with different random seeds. For token-level prediction tasks (e.g., NER), the token-level representations are fed into the classification layer, whereas for sentence-level tasks (e.g.. XNLI), the [CLS] representation is used as input to the classification layer.
Training with confidence penalty Our goal is to train the task models so that they can be used reliably for self- self-training on target language that is potentially dissimilar and low-resourced. such situations, an overly confident (overfitted) model may produce more noisy pseudo labels, and the noise will then accumulate as the training progresses. Overly confident predictions may also impose difficulties on our distillation methods ($3.3) in isolating good samples from noisy ones. However, maximum likelihood training with the standard cross-entropy (CE) loss may result overfitted models that produce overly confident predictions (low entropy), especially when the class distribution is not balanced. We address this by adding a negative entropy term - H to the CE loss.
L())
Σl
nd log (x) + + log (x))
(1)
CE
4

---

Under review as a conference paper at ICLR 2021
where x is the representation that goes to the output layer, and yr and P (x,) are respectively the ground truth label and model predictions with respect to class c. Such regularizer of output distribution has been shown be an effective generalization method for training large models (Pereyra et al. 2017). In our experiments ) we report significant gains with confidence penalty for tross-lingual transfer. Appendix shows visualizations on why confidence penalty is helpful for distillation.
3.2 VICINITY DISTRIBUTION AND SENTENCE AUGMENTATION
Our augmentated sentences comes from two different sources: the original target language samples X2, and the virtual samples generated from the vicinity distribution of the source and target samples: of ((7,) | mmlm) and ( [ Omim), where x, X, and x,, X, It has been shown that contextual LMs pretrained on large-scale datasets capture useful linguistic features and can be used to generate fluent grammatical texts (Hewitt and Manning, (019). We use the XLM-R masked LM (Conneau ct al. 2020) as our vicinity model miim- which is trained on massive multilingual corpora (2.5 TB of Common- Crawl data in 100 languages). Note that the vicinity model is disjoint pretrained entity whose parameters are not trained on any task objective.
In order to generate samples around each selected example, we first randomly choose P% of the input tokens. Then we successively (i.e., one at a time) mask one of the chosen tokens and ask f predict a token in that masked position, i.e., we compute 0 (Tmm (ar, 0mlm) with m being the index of the masked token. For a specific mask, we sample S candidate words from the output distribution. We then generate novel sentences by following one of the two alternative approaches.
Successive max In this approach, we take the most probable output token (S = 1) at each prediction step, Ôm = arg max, ((mm oo mmlm). A new sentence is then constructed by P% newly generated tokens. We generate 5 virtual samples for each original example x, by randomly masking P% tokens each time. Here, 5 is the diversification factor.
Successive cross In this approach, we divide each original sample into two parts and use successive max to create two sets of augmented samples of size 61 and &2 respectively. We then take the cross of these two sets to generate 61 X 82 augmented samples.
Augmentation of sentences through successive max or successive cross is carried out within the GEN-LM (generate via LM) module in Algorithm [ For tasks involving a single sequence (e.g., XNER), we directly use successive max. Pairwise tasks like XNLI and PAWS-X have pairwise dependencies: dependencies between a premise and a hypothesis in XNLI or dependencies between a sentence and its possible paraphrase in PAWS-X. To model such dependencies, we use successive cross, which uses cross prosucp of two successive max applied independently to each component.
3.3 CO-LABELING OF AUGMENTED SENTENCES THROUGH CO- DO-DISTILLATION
Traditional VRM based data augmentation methods assume that the samples generated by the vicinity model share the same class so that the same class labels can be used for the newly generated data (Cappelle et al. 2001). This approach does not consider the vicinity relation across examples of different classes Recent methods relax this assumption and generate new images and their labels as simple linear interpolations (Berthelot et al., 2019). However, due to the discrete nature of texts, such linear interpolation methods have not been successful so far in NLP. The mcaning of sentence (e.g.. sentiment, word meanings) can change entirely even with minor variations in the original sentence. For example, consider the following example generated by our vicinity model (more in aapenndx G).
Original text:
EU rejects German call to boycott british lamb. <mask> rejects ge german call boycott british lamb.
Masked text:
MLM prediction: Trump rejects german call to boycott british lamb.
Here, EU is an Organization whereas the newly predicted word Trump a Person (different name type). Therefore, we need to relabel the augmented sentences no matter whether the original sentence has labels (source) or not (target). However, the relabeling process can induce noise, especially for dissimilar/low-reson languages, since the base task model may not be adapted fully in the early training stages. We propose a two-stage sample distillation process to filter out noisy augmented data.
Sample distillation by single-model. The first stage of distillation involves predictions from a single peer model for which we propose two alternatives:
(i) Distillation by model confidence: In this approach, we select samples based on the model's s prediction confidence. This method is similar in spirit to the selection method proposed by Ruder and
5

---

Under review as a conference paper at ICLR 2021
Plank (0118) For sentence-level tasks (e.g., XNLI). the model produces a single class distribution for each training example. In this case, the model's confidence is computed by p = max.c((1..0) (x)... For token-level sequence labeling tasks (e.g., NER), the model's confidence is computed by:
I = I (I..c Pg p((x)))==r where T is the length of the sequence. The distillation is then done by selecting the top 17% samples with the highest confidence scores.
1
(it) Sample distillation by clustering: We propose this method based the finding that large neural models tend to learn good samples faster than noisy ones, leading to a lower loss for good samples and higher loss for noisy ones (Han al., 2018 Arazo al. 2019). We use a Id two-component Gaussian Mixture Model (GMM) to model per-sample loss distribution and cluster the samples based on their goodness. GMMs provide flexibility in modeling the sharpness of distribution and can be easily fit using Expectation-Marxe Maximization (EM) (Appendix B). The loss is computed based on the pseudo labels predicted by the model. For each sample x, its goodness probability is the posterior probability p(= g|x, fамм), where 9 is the component with smaller mean loss. Here, distillation hyperparameter 1) is the posterior probability threshold based on which samples are selected.
Distillation by model agreement. In the second stage of distillation, we select samples by taking the agreement (co-guess) of two different peer models ((1) and ((k) to train the third (()), Formally,
AGREEMENT(D!A. = (D())
y(h) = yO0 s.t. k j
3.4 DATA SAMPLES MANIPULATION
XLA uses multi-epoch co-teaching. It uses D, and Dj, in the first epoch. In epoch 2, it uses De (target virtual), and finally it uses all the four datasets D,, Di, De, and D. (line 22 in Alg. 1 The datasets used at different stages can be of different sizes. For example, the number of augmented samples in D. and Dr grow polynomially with the successive cross masking method. Also, the co-distillation produces sample sets of variable sizes. To ensure that our model does not overfit on one particular dataset, we employ balanced sampling strategy. For N number of datasets {D, (D;)N N with probabilities, [p:). we define the following multinomial distribution to sample from:
f
74
Pi
where Si
ΣS=
where a is the sampling factor and n4 is the total number of samples in the 2th dataset. By tweaking o, we can control how many samples a dataset can provide in the mix.
4 EXPERIMENTS
We consider three tasks in the zero-resource cross-lingual transfer setting. We assume labeled training data only in English, and transfer the trained model to target language. For all experiments, we report the mean score of the three models that use different seeds (variance shown in Appendix F)
4.1 TASKS & SETTINGS
XNER: As a sequence labeling task, XNER evaluates the model's capability to learn task-specific contextual representations that depend on language structure. We use the standard CoNLL datasets (Sang) 2002] Sang and Meulder (2033) for English (en). German (de), Spanish (es) and Dutch (nl). We also evaluate on Finnish (fi) and Arabic (ar) datasets collected from Bari et al. (2020). Note that Arabic is structurally different from English, and Finnish is from a different language family. To show how the models perform extremely low-resource languages, we experiment with three structurally different languages from WikiANN (Pan et al., (017) of different (unlabeled) training data sizes: Urdu (ur-20k training samples), Bengali (bn- 10k samples), and Burmese (my-100 samples).
XNLI XNLI judges the model's ss ability to extract a reasonable meaning representation of sentences across different languages. We use the standard dataset (Conneau et al.) 2018). For a given pair of sentences, the task is to predict the entailment relationship between the two sentences, i.e., whether the second sentence (hypothesis) is Entailment, Contradiction, Neutral with respect to the first one (premise). We experiment with Spanish, German, Arabic, Swahili (sw), Hindi (hi) and Urdu.
PAWS-X The Paraphrase Adversaries from Word Scrambling Cross-lingual task (Yang et al.) 9a) requires the models to determine whether two sentences are paraphrases. We evaluate on all the six (typologically distinct) languages: fr, es, de, Chinese (zh), Japanese (ja), and Korean (ko).

---

Under review as a conference paper at ICLR 2021
Settings Our goal is to adapt task model from a source (language) distribution to an unknown target (language) distribution assuming no labeled data in the target. In this scenario, there might be two different distributional gaps: (f) the generalization gap for the source distribution, and (ti) the gap between the source and target language distribution. We wish to investigate our method in tasks that exhibit such properties. We use the standard task setting for XNER, where take 100% samples from the datasets as they come from various domains and sizes without any specific bias.
However, both XNLI and PAWS- X training data come with machine-translated texts in tar- get languages. Thus, the data is parallel and lacks enough diver- sity (source and target come from the same domain). Sross-lingual models trained in this setup may pick up distributional bias (in the label space) from the source. Artetxe et al. 2020) also argue that the translation process can induce subtle artifacts that may have a notable impact on models. Therefore, for XNLI and PAWS- X, we experiment with two dif- ferent setups. First, to ensure dis- tributional differences and non- parallelism, we use 5% of the training data from the source language and augment a different (nonparallel) 5% dataset for the target language. We used a different seed each time retrieve the 5 5% target language data. Second, to compare with previous methods, we also evaluate on the standard 100% setup. However, the evaluation done on the entire test set in both setups. We will refer to these two settings as 5 % and 100% %. Details about model settings and hyperparameters are in Appendix E
Table I: F1 scores in XNER on datasets from CoNLL and Bari et al. (020). represents no results were reported for the setup.
Model
de
fi
Supervised Results (TRANSLATE
CAATE---TRI--LLL
LSTM-CRE (Bari (Bariarred al. 2020
89.77
84.71
85.16
78.14
75.49
84.21
XLM-R (Connca el
92.92
89.72
92.53
85.81
X (our imp.)
92,9
89.2
92.9
86.2
86.8
92.4
Jero-Resouree Baseline
nBERTTwd (our imp.)
91.13
74.76
79.58
70.99
45.48
65.95
XLM R (our imp.) XLM-R (ensemble)
92.23
79.29
80.87
73.40
49.04
75.57
92.76
80.62
81.46
75.40
52.30
76.85
Our Method
mBERT caed #con-penalty
90.81
75.06
79.26
72.31
47.03
66.72
HLLM-R+con-pealll
92.49
80.45
81.07
73.76
49.94
76.05
XLA XLA (ensemble)
83.05
85.21
80.33
57.35
79.75
83.24
85,32
80.99
58.29
79.87
4.2 RESULTS
XNER Table reports the XNER results on the datasets from CoNLL and Bari et al. 2020), where we also evaluate an ensemble by averaging the probabilities from the three models. We observe that after performing warm-up with conf-penalty, XLM-R performs better than mBERT on average by ~3.8% for all the languages. On average, XIAA gives a sizable improvement of ~5.5% on five different languages. Specifically,
get absolute improvement of 3.76%, 4.34%, 6.94%, 8.31% %,, and 4.18% for es, nl, de, ar, and fi, respectively. Interestingly, XLA surpasses supervised LSTM-CRF for n/ and de without using any target labeled data. It also produces comparable results for es.
In Table 2. report the results on the three low-resource langauges from WikiANN. From these results and the results of ar and fi in Table 1| we see that XLA is very effective for languages that structurally dissimilar and/or low-resourced, especially when the base model is weak: 28.54%, 16.05%, and 9.25% absolute improvements for ur, my and ar, respectively.
XNLI-5% From Table 3, we see that the performance of XLM-R trained on 5% data surprisingly good com- pared to the model trained on full data (XLM-R (our imp.)), lagging by only 5.6% on average. In our single GPU imple- mentation of XNLI, we could not reproduce the reported results of Conneau et al. 2020). However, our results re- semble the reported X results of XTREME (Hu et al.] 2020). We consider XTREME as our standard baseline for XNLI-100%
Table 2: XNER results on WikiANN
Model
ur
bn
my
Supervised Results
XLM-R (our-impl)
97.1
97.8
76.8
Zero-Resouree Results
XLM-R (XTREMB)
56.4
78.8
54,3
XLM-R (our imp.)
$6.45
78.17
54.56
XLA
84,99
82.68
70.61
We observe that with only 5% labeled data in the source, XLA gets comparable results to the XTREME baseline that uses 100% labeled data (lagging behind by only ~0.7% on average); even for and sw, we get 0.22% and 1.11% & improvements, respectively. It surpasses the standard 5% baseline by 4.2% on average. Specifically, XLA gets absolute improvements of 3.05%, 3.34%. 5.38%, 5.01% %, 4.29%, and 4. 4.12% for es, de. sw, hi, and ur, respectively. Again, the gains are relatively higher for low-resource and/or dissimilar languages despite the base model being weak in such cases.
7

---

Under review as a conference paper at ICLR 2021
XNLI-100% Now, considering XLA's performance on the full (100%) labeled source data in Table 3 we see that it achieves state-of-the-art results for all of the languages with an absolute improvement of 2.55% on average from the XTREME baseline. Specifically, XLA gets absolute improvements of 1.95%. 1.68%, 4.30%, 3.50%, 3.24%, and 1.65% for de, sw, hi, and ur, respectively.
Table 3: Results in accuracy for XNI.I.
Model
en
de
ar
SW
hi
ur
Supervised Results (T (TRANSLATE: MEELLLATE--R----EA (RALN-LLL)
XLM-R
89.1
85.7
83.1
78.0
81.6
78.1
Zere- Resource Baseline for Full (100%)
English labeled training sct
XLM-R (XTREME)
88.7
83.7
82.5
77.2
71.2
75.6
71.7
XLM-R (our imp.)
88.87
84.34
82,78
78.44
72.08
76.40
72.
XLM-B R. (ensemble)
89.24
84.73
83.27
79.06
73.17
77.23
73.07
XI J.MM---co-penultyy
88.3
84.30
82.86
78.
71.85
76.24
71
XLA
85.65
84.15
80.50
74.
78.74
23.35
XLA (ensemble)
86.12
84.61
80.89
74,89
78.98
73.45
Zero Resource Buseline for 5%
English labeled training
PAWS-X Similar to XNLI, we ob- serve sizable improvements for XLA over the baselines on PAWS-X for both 5% and 100% settings Table 4). Specif- ically, in 5% setting. XLA gets absolute gains of 5.33%, 5.94%, 5.04%, 6.85%, 7.00%, and 5.45% for de, es, fr. ja. ko, and zh, respectively, while in 100% setting, it gets 2.21%. 2.36%, 2.00%, 3.99%, 4.53%, and 4.41% improvements re- spectively. In general, we get an average improvements of 5.94% and 3.25% in PAWS-X-5% and PAWS-X-100PP settings respectively. Moreover, our 5% setting outperforms 100% XLM-R baselines accuracies compared to supervised finetuning of XLM-R on all target language training dataset.
XLM-R (our imp.)
83.08
78:48
77.54
72.04
67.3
70.41
66.72
XLM-B R. (ensemble)
84.65
79.56
78.38
72.22
66.93
71.00
66.79
XI.M-R wa--Rccnn-peutty
84.24
79.23
79.47
72.43
67.72
71.08
67.63
XLA XLA (ensemble)
81.53
80.88
77,42
72.31
74.70
70.84
82.35
81.93
78.56
73.53
75.20
71.15
for es, ja, and zh. Interestingly, in the 100% setup, our XLA (ensemble) achieves almost similar
5 ANALYSIS
In this section, we further analyze XLA by dissecting it and measuring the con- tribution of its different components. For this, we use the XNER task and analyze the model based on the results in T
Table 4: Results in accuracy for PAWS-X.
Model
de
Fr
zh
Supervised Results (TRANSI TREITRAIN.LL..
XLM-R R four impl.)
95.8
92.5
92.8
93.5
85.5
86.6
87.6
5.1 ANALYSIS
ZemoRRsounce Raseline for Full (100%)
English labeled rraining
OF DISTILLATION METHOOS
XLM-R (XTREME)
94.7
89.7
90.1
90.4
78.7
79.0
82.3
XLM-R R (our imp.)
95.46
90.06
89.92
90.85
79.89
79.74
82.49
Model confidence vs. clustering We first analyze the performance of our single-model distillation methods ($3.3) to see which of the two alternatives works better. From Table [] we see that both perform similarly, with model confidence being slightly better. In our main experi- ments (Tables 4 and subsequent analy- sis, we use model confidence for distilla- tion. However, we should not rule out the clustering method it gives more general solution to consider other distillation features (e.g., sequence length, language) than model prediction scores, which we did not explore in this paper.
XLM-R (ensemble)
90.75
80:55
80.70
83.45
XLM Recon penally
95.38
90.75
90.72
91.71
81.77
82.07
84.25
XLA
92.27
92.28
92.85
83.88
84.27
86.90
XLA (ensemble)
92.55
92.35
93.35
84.30
Zero Resource Baseline for 5%
English labeled training set
XLM-R (our imp.)
91.15
83,72
84.32
85. 08
73.65
72.60
77.22
XLM-R (emsemble)
$4.05
85.75
74.30
77.50
XI M-M Ricon-penalty
91.85
86.15
86.38
85.98
76.03
75.44
79.15
XLA
89.05
90.27
90.12
80,50
79.60
82.65
XLA (emsemble)
89.25
90.25
80.15
82.90
Distillation factor 7) We next show the results for different distillation factor (r)) in Table [ Here 100% refers to the case when single-model distillation is done based on model confidence. We notice that the best results for each of the languages are obtained for values other than 100%, which indicates that distillation is indeed an effective step in XLA. See See/Appendix C.2 for more on n7
Two-stage distillation We now validate whether the second-stage distillation (distillation by model agreement) is needed. In Table 5] we also compare the results with the model agreement (shown n) to the results without using any agreement (shown as Φ). We observe better performance with model agreement in all the cases on top of the single-model distillation, which validates its utility.
5.2 DIFFERENT TYPES OF AUGMENTATION IN DIFFERENT STAGES
Figure 2 presents the effect of different types of augmented data used by different epochs in our multi-epoch co-teaching framework. We observe that in every epoch, there is significant boost in
8

---

Under review as a conference paper at ICLR 2021
Table Analysis of distillation on XNER. Results after epoch-1 training that uses (D,, D
Agreement
ml
de
6
Distallation by clustering
82.28
83.25
78.86
52.64
78.47
0.5
0
82.35
83.11
78.16
54,20
78.28
Distillation model confidence
82.52
82.46
75.95
52.00
77.51
6
81.66
82.26
77.19
52.97
77.77
82.33
83.53
78.50
54.48
78.43
80%
0
81.61
83.03
77.08
53,31
78.34
Wermip
[9, 0/)
(,,,0,0
90%
0
82.80
79.03
52.41
81.21
82.77
77.28
52,20
77.93
Figure 2: Validation F1 results in XNER for multi-epoch co-teaching training of XLA.
0
82.50
82.35
77.06
52.58
77.51
100%
81.89
82,15
76.97
52.68
78.01
Fl scores for each of the languages. Arabic, being structural dissimilar to English, has a lower base score, but the relative improvements brought by XLA higher for Arabic, especially in epoch when it gets exposed to the target language virtual data (D,) generated by the vicinity distribution.
5.3 ROBUSTNESS OF XLA
Table 6: F1 scores on XNER for all target languages. Each column (e.g., es) under XLA Model represents results in all target languages for XLA model trained with the augmented data in a specific language (e.g., es). The Zero shot+con-penalty column represents the zero-shot results for the model after WarmUp.
Tshlo the robustness of the finetuned XLA model on XNER task. After finetuning in specific target language, the Fl scores in English remain almost similar. For some lan- guages, XLA adaptation on different lan- guage also improves the performance. For example, Arabic gets improvements for all XLA-adapted models (compare 50.88 with others). This indicates that augmentation of XLA does not overfit on a target language.
Tgt lang
Zero shot + con-penalty
XLA Model
de
ar
92.88
92.80
92.68
81 4222
83.24
82.01
77.71
80.29
81 97
81.27
81.22
85,32
80.54
82.36
84.20
75.20
73.63
75.03
80.03
76.97
73.77
50.88
$2.66
53.08
52.52
58.29
53.80
5.4 EFFECT
76.97
77.02
77.06
76.69
77.13
80.11
OF CONFIDENCE PENALTY & ENSEMBLE
For all the three tasks, we get reasonable improvements over the baselines by training with confidence penalty regularizer ($3.1) Specifically, we get 0.56%, 0.74%, 1.89%, and 1.18% improvements in XNER, XNLI-5%, PAWS-X-5% and NAWS-X-1000 respectively (Table 4 The improvements XNLI-100% are marginal and inconsistent, which we suspect due to the balanced class distribution.
From the results of ensemble models, we see that the ensemble boosts the baseline XLM-R. However, our regular XLA still outperforms the ensemble baselines by sizeable margin. Moreover, ensembling the trained models from XLA further improves the performance. These comparisons ensure that the capability of XLA through co-teaching and co-distillation is beyond the ensemble effect.
CONCLUSION
We propose novel data augmentation framework, XLA, for zero-resource cross-lingual task adap- tation. XLA performs simultaneous self-training with data augmentation and unsupervised sample selection. With extensive experiments on three different cross-lingual tasks spanning many language pairs, we have demonstrated the effectiveness of XLA. For the zero-resource XNER task, XLA sets a new SoTA for all the tested languages. For both XNLI and PAWS-X X tasks, with only 5% labeled data in the source, XLA gets comparable results to the baseline that uses 100% labeled data. Through an in-depth analysis, we show the cumulative contributions of different components of XLA.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E O' CoCnnor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International Conference on Machine Learning (ICML). June 2019.
Devansh Arpit, Stanislaw Jastrzundefineddski,, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and et al. A closer look memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 233-243 JMLR.org. 2017.
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot lingull transfer and beyond. CoRR, abs/1812.10044 2018. URL http::ttttti....////
00ss1812.104653
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolin- gual representations, 2019.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre learning, 2020.
Translation artifacts in cross-lingual transfer
M Saiful Bari, Shafiq Joty, and Prathyusha Jwalapuram. Zero-Resource Cross-Lingual Named Entity Recognition. In Proceedings of the 34th AAAI Conference on Artifical Intelligence, AAAI '20, pages XX-xx, New York, USA, 2020. AAAI.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: holistic approach to semi-supervised learning. H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-But E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5050-5060. Curran Associates, Inc., 2019.
Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. In T. K. Leen, G. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems 13, pages 416 422. MIT Press, 2001. URL 1876 wtt.--iiisiiiiiiiiiiiiiiiiio
Alexis Conneau, Guillaume Lample, Marc' Aurelio Ranzato. Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. CoRR, abs/1710.040S7, 2017. URL
erg/abs/ 1710.04087
Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. CoRR, abs/1809.050ss 809500555,, 2018 (ttt::h:////xx
org/ abs 809. 05053
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online, July 2020. Association for Computational Linguisties. doi: 10. 1/2120653/12222..ac main. URL https: www.waclwwe..
main. 747.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers). pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19. 1423. URL |RLhttps: awweeb...b........ w/w..ac..wb..oo///aatttta.t..... htwpr///ogggaaatooaaaaa--------.
Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. Learning to paraphrase for question answering. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Pro- cessing, pages
Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v 1/917- URL hctps: www. ac lweb. org/ anthology/
D1 D17-1091
10

---

Under review as a conference paper at ICLR 2021
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018), 2018
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Bengio, H. Wallach, Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 8527-8537. Curran Associates, Inc., 2018.
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4129 4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1419. URL https: www wwwwwb.www........ wttp://wwwwwtttttooo////////// 1111111111 1419
Jeremy Howard and Sehastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Mclbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P11 1--111 M https: www. ac wwwewwwbb.ooo org/ anthology/ 113-
Junjie Hu,, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR, abs/2003. 1080, 2020.
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing SEMMNLP-IJCNLL pages 2495,2494, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-11 12522. UtLhhttss:/ /wwp///wwwww//// c web wwwp:oorgggatttoggo.ooooooooo 9- 1252
Karthikeyan K, Zihan Wang, Stephen Mayhew, and Dan Roth. Cross-lingual ability of multilingual [bert): An empirical study. In International Conference on Learning Representations, 2020. URL
https:
net/ www.:eeee/o.tiie..
Phillip Keung, yichao lu, and Vikas Bhardwaj. Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and ner. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNILP), 2019. doi: 10. 00.18653/v1/d19. 138. URL
http://dx.don org/10. 18653/v1/d13-1115
Sosuke Kobayashi. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 11118653/1//N18-0000 URL htwph:ttt///i/. /wwww///wwwwww// a c web. http.//rggggatooggg//i//.oo/ -272
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix:
Learning with noisy labels semi- supervised learning- In International Conference on Learning Representations, 2020a. URL.
httpss:/cpenreviiee net/ wot.:/oommmmmduuxaeaaaaae
Linyang Li, Ruotian Ma. Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert, 2020b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 10ts/997..11622. 2019. URL.http: httl//hcpp///aarr..ooo/////////s

---

Under review as a conference paper at ICLR 2021
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6297-6308, 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111-3119, Curran Associates, Inc., 2013.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier. and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946-1958, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 18.18653/v 1/t1///1//11117 URL
https: www. www.wwlwwwb.......... orq/ wwt.aalogooy//--11111
Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In EMNLP'I4, 14, pages 1533-1543, Doha, Qatar, October 2014. |
/
1162
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regularizing neural networks by penalizing confident output distributions. CoRR, 0e//1701005445 2017. URL
atttp://ariii org abt/aabl..110..
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
Telmo Pires, Eva Schlinger, and Dan Garrette, How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1996-5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. (0.18863/v1/ 14919 URL https: www. a www.bwewbbb...... http./////ttoooooo//.s--------
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under-
standing by generative pre-training. 2018. Utt.:ttts: semanticscholar
[dd1 Ccd18/800a00 200000febbs (aabb6681cccccc 915::5550033
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Sebastian Ruder and Barbara Plank. Strong baselines for neural semi-supervised learning un- der domain shift. In Proceedings of the 56th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1044-1054, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.18653/v1/P18- 1/P11-1098...... URL https? 7/ 1/www. www.acllweb... org/ http://www...gaaaologogg////////
Erik Tjong Kim Sang. Introduction to the conll-2002 shared task: Language-independen named entity recognition. CoRR, es. CL/0000000 2002.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. In CoNLI,, 2003.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation mod- els with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany, Au- gust 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-110 1009. URL httpp:: http///wwaccc www.ww.w..bb.................... www.or//aaaaa w wwwpanhhoooooooolloagggg ogy/p16-1009
Patrice Y. Simard, Yann A. LeCun, John S. Denker, and Bernard Victorii. Transformation In-
variance in Pattern Recognition Tangent Distance and Tangent Propagation, pages 239 274. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998. ISBN 378-3-540-49430-0 10.100773-540-444300 13.
doi:
/ooi www.or/g////. 10070..100//00 5
12

---

Under review as a conference paper at ICLR 2021
Leslie N. Smith. No more pesky learning rate guessing games. CoRR, abs/1506.01 186, 2015. URL http::/ /ttp:////axvvv.o// wwtp://gg///......//////// 01188
Christian Szegedy, Wojciech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference Learning Representations, 2014.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne, 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998 6008. Curran Associates, Inc., 2017. URL -atentionniisie -oo----lllo------- need. pdd
http::/papere http:///ppperssniss.oo httppsscccpppppppppeeep
Zirui Wang, Jiateng Xie, Ruochen Xu, Yiming Yang. Graham Neubig, and Jaime Carbonell. Cross- lingual alignment vs joint training: A comparative study and a simple unified framework, 2019.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume (Long Papers). pages 1112-1122. Association for Computational Linguistics, 2018. URL http:: web.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' R'emi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of-the thete-rrtt natural language processing. ArXiv, abs/1910.03771, 2019.
Shijie Wu and Mark Dredze. Beto, bentz, becas:
The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833-844, Hong Kong, China, November 2019. Association for Computational Linguisics.. doi: 10. 1ttp:///////1900000
Cttpttts: www. awwwwb.www..c. wttp://wwwwwwo//tttttooooeeeeeee http://oooooonntoooooo///--11110
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional BERT contextual augmentation. CoRR, abs/1811.06777 2018. /utp:ttte//aarii..orr//a////111 06705
Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A. Smith, and Jaime G. Carbonell. Neural cross- 2018. URL
lingual named entity recognition with minimal resources. CoRR,
http:://axiis wrw.
09861
Yinfei Yang. Yuan Zhang, Chris Tar, and Jason Baldridge. PAWS-X: A cross-lingual adversar- ial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China, Novem- ber 2019a. Association for Computational Linguisties. doi: 10.18653/1//11-11388 URL
https: / www. ac www.we.......ee..ooe......oo.e.e wtt.o//////t.oolog1ol11111111111
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ Salakhutdinov, and Quoe V Le. Xinet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32, pages 5753-5763, Curran Associates, Inc., 2019b.
Adams Wei Yu, David Dohan, Quoe Le, Thang Luong, Rui Zhao, and Kai Chen. Fast and accurate reading comprehension by combining self-attention and convolution. In International Confer- ence on Learning Representations, 2018. URL
B14T1G RW
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018. URL https
//openrevis ew net./ fonte/forum?ee wtt.:e//ooorrmmmoorooooo Rb
13

---

Under review as a conference paper at ICLR 2021
Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 1298-1308, 1308, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1//3//11N19111110 1131. URthttpp:://ww/ ac web. www.borrg//ttooooo.....oo 9N-11111
Zhi-Hua Zhou and Ming Li. Tri-training: exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering, 17:1529- 1541, 2005.
14

---

Under review as a conference paper at ICLR 2021
APPENDIX
Here we provide the additional contents regarding the XLA framework. appendix A we present the justification for design choices of XLA framework. In appendix [ discuss the mathematical details of EM training for two components GMM clustering algorithm. In appendix C we visualize various effects of confidence penalty. In appendix D we present the setup details of our experiments. we elaborate on the related work and results with standard deviation as well as comparing with prior research work, respectively. Finally, in appendix we present the examples of augmented samples generated by our vicinity model for XNER, XNLI, and PAWS-X.
In appendix [D] and |
A JUSTIFICATIONS FOR DESIGN METHODOLOGY OF XLA FRAMEWORK
Here are our justifications for various design principles the XLA framework.
Is using three models with different initialization necessary? Yes, different initialization sures different convergence paths, which results in diversity during inference. Co-labeling (Section 3.3) utilizes this property. There could be some other ways achieve the same thing. For example, our initial attempt with three different heads (sharing a backbone net) didn't 't work well.
Is using three epochs necessary? We utilize different types of datasets in different epochs. While pseudo-labeling may induce noise, the model's predictions for in-domain cross poossllnngual samples are usually better. Because of this, for smooth transition, we apply the vicinal samples in the second epoch. Finally, inspired by the joint training of the cross-lingual language model, in the third epoch we use all four datasets. We also include the labeled source data which ensures that our model does not overfit on target distribution as well persists the generalization capability of the source distribution.
Need for the combination co-teaching, co-distillation and co-guessing? The combination of these helps to distill out the noisy samples better.
Efficiency of the method and expensive extra costs for large-scale pretrained models It is a common practice in model selection train 3-5 disjoint LM-based task models (e.g., XLM-R NER) with different random seeds and report the ensemble score or score of the best (validation set) model. In contrast, XLA uses different models and jointly trains them where the models assist each other through distillation and co-labeling. In that sense, the extra cost comes from distillation and co-labeling, which not significant and is compensated by the significant improvements that XLA offers.
B DETAILS ON DISTILLATION BY CLUSTERING
One limitation of the confidence-based (single-model) distillation is that it does not consider task- specific information. Apart from classifier confidence, there could be other important features that can distinguish a good sample from a noisy one. For example, for sequence labeling, sequence length can be an important feature as the models tend to make more mistakes (hence noisy) for longer sequences Bari et al. (020)) One might also want to consider other features like fluency, which can be estimated by pre-trained conditional LM like GPT Radford al. (2019). In the following, we introduce a clustering-base based method that can consider these additional features to separate good samples from bad ones.
Here our goal is to cluster the samples based on their goodness. It has been shown in computer vision that deep models tend learn good samples faster than noisy ones, leading to a lower loss for good samples and higher loss for noisy ones Han et al. (2018); Arpit et al. (2017). We propose to model per-sample loss distribution (along with other task-specific features) with a mixture model, which we fit using an
(EM) algorithm. However, contrary to those approaches which use actual (supervised) labels, we use the model predicted pseudo labels to compute the loss for the samples.
We use a two-component Gaussian Mixture Model (GMM) due to its flexibility in modeling the sharpness of distribution Li et al. (2020a). In the following, we describe the EM training of the
15

---

Under review as a conference paper at ICLR 2021
GMM for one feature, i.e, per-sample loss, but it is trivial extend it to consider other indicative task-specific features like sequence length or fluency score (see any textbook on machine learning).
EM training for two-component GMM Let IR denote the loss for sample x; and {0, denote its cluster id. We can write the 1d GMM model as:
0(
Nax (il ok)
(3)
k=0
where 0g {pt, ) are the parameters of the k-th mixture component and TT p(2; k) the probability (weight) of the k-th component with the condition 0 Tx S 1 and Σx 1.
In EM, we optimize the expected complete data log likelihood @ 01-1) defined as:
Q(0,0--1)
E
loglp(>; 2,(0)))
(4)
E
Σ I(z: log|o(2;(0m))Tk)
(5)
Σ
Σ B[I(2, log[p(x;;(0)
(6)
Σ E p(zz = k/ 0- log[p(x;; (0x)
ΣΣna (11100-))oggp(::1111 + r log To
(8)
where (0"=)) is the responsibility that cluster takes for sample x,, which is computed in the E-step so that we can optimize Q00,0* 10 (Eq, 8) GMM can be written as:
the M-step. The E-step and M-step for ld
E-step: Compute re, (0t-1)
N(n;; )m) En Nm:10) )
M-step: Optimize (1-) w.r.t and
Sirck Fh E. Pe Pilk
E rit
E. Pi.e E.
E - of=
E, PLA
Inference For sample x, its goodness probability is the posterior probability p(= g|x, 0), where g {0,1} } is the component with smaller mean loss. Here, distillation hyperparameter 1j is the posterior probability threshold based on which samples are selected.
Relation with distillation by model confidence Astute readers might have already noticed that per-sample loss has direct deterministic relation with the model confidence. Even though they are different, these two distillation methods consider the same source of information. However, as mentioned, the clustering- basser method allows to incorporate other indicative features like length, fluency, etc. For fair comparison between the two methods, we use only the per-sample loss in our primary (single-model) distillation methods.
C VISUALIZING THE EFFECT OF CONFIDENCE PENALTY
C. EFFECT OF CONFIDENCE PENALTY IN CLASSIFICATION
we present the effect of the confidence penalty (Eq. 1 in the main paper) in the target language (Spanish) classification on the XNER dev. data (i.e., after training English NER). We show the class distribution from the final logits (on the target language) using t-SNE plots van der Maaten and Hinton (2008).
In Figure 3
16

---

Under review as a conference paper at ICLR 2021
From the figure, it is evident that the use of confidence penalty in the warm-up step makes the model more robust to unseen cut-of-distributtee target language data yielding better predictions, which in turn also provides a better prior for self- ceff-training with pseudo labels.
(a) Without confidence penalty.
(b) With confidence penalty.
Figure 3: Effect of training with confidence penalty in the warm-up step on target (Spanish) lan- guage XNER classification using t-SNE plots. From the visualization, can be seen that the model trained with confidence penalty shows better inter-class separation which exhibits robustness of the multilingual model.
C.2 EFFECT OF CONFIDENCE PENALTY IN LOSS DISTRIBUTION
Figures aa) and 4 () present the per-sample loss (i.e., mean loss per sentence w.r.l. the pseudo labels) distribution in histogram without and with confidence penalty, respectively. Here, accurate-2 refers the sentences which have at most two wrong NER labels, and sentences containing more than two errors are referred to as noisy samples. It shows that without confidence penalty, there are many noisy samples with a small loss which confidence penalty helps to separate the clean samples from the noisy ones either by clustering or by model confidence.
desired. In addition to that, the figures also suggest that the
1o11 229
1000
1055
A12
130
020
loss
(a) Without confidence penalty.
(b) With confidence penalty.
Figure 4: Histogram of loss distribution target (Spanish) language XNER classification.
17

---

Under review as a conference paper at ICLR 2021
Figures 5 a) and () present the loss distribution in scatter plot by sorting the sentences based on their length in the x-axis; y-axis represents the loss. As we can see, the losses are indeed more scattered when train the model with confidence penalty, which indicates higher per-sample entropy. as expected. Also, we can see that as the sentence length increases, there are more wrong predictions. Our distillation method should be able to distill out these noisy pseudo samples.
Sertance conmirs accumss libe: teel (1667 Senzance contale ecarect (44)
coParetioonec abel tral (291
038
57-
1055
200
006
0U8
322
12:00
1000
37.0
2000
sentences sorted by lengths
sentences sorted by lengths
(a) Without confidence penalty.
(b) With confidence penalty.
Figure 5: Scatter plot of loss distribution on target (Spanish) language XNER classification.
Finally, Figures and b()) show the length distribution of all vs. the selected sentences (by Distillation by model confidence) without and with confidence penalty. Bari al. (2020) shows that cross-lingual NER inference is heavily dependent on the length distribution of the samples. In general, the performance of the lower length samples more accurate. However, we only select lower length samples we will easily overfit. From these plots, we observe that the confidence penalty also helps to perform a better distillation as more sentences are selected (by the distillation procedure) from the lower length distribution, while still covering the entire lengths. This shows that using the confidence penalty in training, model becomes more robust.
conclude that training without confidence penalty can make the model more prone to over over-fitting, resulting in more noisy pseudo labels. Training with confidence penalty not only improves pseudo labeling accuracy but also helps the distillation methods to perform better noise filtering.
In summary, comparing the Figures 4 6
18

---

Under review as a conference paper at ICLR 2021
frequency
length
342
(a) Without confidence penalty.
frequepcy
length
(b) With confidence penalty.
Figure 6: Distribution of selected sentence lengths target (Spanish) language XNER classification.
19

---

Under review as a conference paper at ICLR 2021
D EXTENDED RELATED WORK
Contextual representation and cross-lingual transfer. In earlier approaches, word representa- tions are learned from simple variants of the skip-gram model Mikolov et al. (2013), where each word has single representation regardless of its context Grave et al. (2018), [ et al. (2014). Recent approaches learn word representations that change based on the context that the word appears in McCann et al. (2017);) Peters et al. (2018); Howard and Ruder (2018) Devlin et al. (2019); Yang
ct al.
(2019b): Radford et al. (2019).
Peters ct al. (2018) propose ELMo bidirectional LSTM-based LM pre-training method for learning contextualized word representations. ELMo uses a linear combination of all of its layers' representations for predicting on target task. However, because of sequential encoding, LSTM- based LM pre-training is hard to train at scale. Vaswani et al. (017) propose the Transformer architecture based on multi-headed self-attention and positional encoding. The Transformer encoder can capture long-range sequential information and allows constant time encoding of a sequence through parallelization. RRaffodd et al. (2018) propose GPT-1, which pre-trains Transformer decoder with conditional language model objective and then fine-tune it on the task with minimal changes to the model architecture. In the same spirit, Devlin et al. (2019) propose BERT, which pre-trains a Transformer encoder with a masked language model (MLM) objective, and uses the same model architecture to adapt to a new task. The advantage of MLM objective is that it allows bidirectional encoding, whereas the standard (conditional) LM is unidirectional (i.e., uses either left context or right context).
BERT also comes with multilingual version called mBERT, which has 12 layers, 12 heads and 768 hidden dimensions, and it is trained jointly on 102 languages with a shared vocabulary of 110K subword tokens Despite any explicit moss-lingual supervision, mBERT has been shown to learn cross-lingual representations that generalise well across languages. Wu and Dredze (2019): Pires et al. 2019) evaluate the zero-shot cross- cross-llnguaal transferability of mBERT several NLP tasks and attribute its generalization capability to shared subword units. |ires et al. (2019) additionally found structural similarity (e.g.. word order) to be another important factor for successful cross-lingual transfer. K et al. (2020). however, show that the shared subword has minimal contribution, rather the structural similarity between languages is more crucial for effective transfer. Artetxe et al. (2019) further show that joint training may not be necessary and propose an alternative method to transfer a monolingual model to a bi-lingual model through learning only the word embeddings in the target language. They also identify the vocabulary size per language as an important factor.
Lample and Conneau (2019) extend mBERT with a conditional LM and a translation LM (using parallel data) objectives and a language embedding layer. They train a larger model with more monolingual data. Huang et al. (2019) propose to use auxiliary tasks such as cross-lingual word recovery and paraphrase detection for pre-training. Recently, Conneau et al. (2020) train the largest multilingual language model with 24-layer transformer encoder, 1024 hidden dimensions and 550M parameters. Keung et al. (2019) use adversarial fine-tuning of mBERT to achieve better language invariant contextual representation for cross- pross-inggall NER and MLDoc document classification.
Vicinal risk minimization. One of the fundamental challenges in deep learning is to train models that generalize well to examples outside the training distribution. The widely used Empirical Risk Minimization (ERM) principle where models are trained to minimize the average training error has been shown to be insufficient to achieve generalization on distributions that differ slightly from the training data Szegedy et al. (2014); Zhang et al. (2018). Data augmentation supported by the Vicinal Risk Minimization (VRM) principle Chapelle et al. (2001) can be an effective choice for achieving better out-of of-train generalization.
In VRM, we minimize the empirical vicinal risk defined as:
N ([fo(En). Fn)
1 N
c,
where fo denotes the model parameterized by 0. and D ((ān. ) is augmented dataset constructed by sampling the vicinal distribution (i, u|z,, y1) around the original training sample
20

---

Under review as a conference paper at ICLR 2021
(z,, Ve). Defining vicinity is however challenging as it requires to extract samples from distribution without hurting the labels. Earlier methods apply simple rules like rotation and scaling of images Simard al. (1998). Recently, Zhang et al. (2018); Berttelot et al. (2019) and Li et al. (2200a) show impressive results in image classification with simple linear interpolation of data. However, to our texts.
knowledge, none of these methods has far been successful in NLP due the discrete nature of
E SETUP DETAILS
E. ZERO-SHOT VS. REROURE TRANSFER
Previous work on cross- cross-lingua lingual transfer has followed different training-validationn standards, Xie et al. (2018) perform cross-lingual transfer of NER from a source language to a target language, where they train their model on translations of the source language training data and validate it (for model selection) with target language development data. They call this as an unsupervised setup as they use an unsupervised word translation model (onneau et al. (2017). Several other studies Conneau (ct al. (2018); Lample and Conneau (2019); Wang et al. (2019) also apply the same setting and select their model based on target language development set performance. On the other hand, Artetxe and Schwenk 2018), Wu and Dredze (2019) validate their models using source language development data. Bari et al. (2020) show significant performance differences between validation with source vs. target language development data for NER. Later, Conneau et al. (2020) provide a comprehensive analysis of different training-validation setups and encourage validating with the source language development data. Therefore, is clear that there is no unanimous agreement regarding the proper setup. Following the previous work and landscape of the problem, we think that different settings should be considered under different circumstances.
In pure zero-shot cross-lingual transfer, no target language data should be used either for training or for model selection. The goal here is to evaluate the generalizability and transferability of a model trained on a known source language distribution an unknown target language distribution. this sense, zero-shot setting is suitable to measure the tross-lingual transferability of a pre-trained model.
Our goal in this work not to propose new pre-training approach, rather to propose novel cross- lingual adaptation methods and evaluate their capability on downstream tasks. Our proposed XIA framework performs simultaneous self-training with data augmentation and unsupervised sample selection. As our objective is to evaluate cross-lingual adaptation performance and not cross-lingual representation, we train our model with the original source and augmented source and target language data, while validating it with target development data for model selection. We refer this zero- resource setup, which is still a minimal supervision setting for task adaptation because no true target labels used for training the model. This setup also gives way to compare how far we from the supervised adaptation setting (train and validate on target language data).
F.2
USE OF MBERT VS. XLM-R
From Table we see that mBERT LIBERDDevi et al. (2019) trains the smallest multi-lingual language model (LM) in terms of training data size and model parameters, while XLM-R is the largest one.
Table 4: Training data size and number of model parameters of Cross-lingual Language Models.
Model Name Tokenization
Language
#Head
#Layer
#Representation
#ocab.
#Params.
Dataset.
Data size.
MBERT MBERT
cased
104
12
12
767
110k
172M
wiki
100 GB
uncased
102
12
767
172M
wiki
100
XL
uncased
15
8
12
1024
95K
250M
wiki
100 GB
XLM-17
cased
17
16
1024
570M
wiki
100
XLM-100
cased
100
16
1280
200k
570M
wiki
100 GB
XLM-R
cused
100
16
1280
200k
570
CC-100
2.5 TB
At its heart, XLA uses the generation capability of a pre-trained LM for data augmentation, which could be bottleneck for XLA's performance. In our initial experiments, found that the generation quality of mBERT not as good as that of XLM-R. Using mBERT as the vicinity model can thus generate noisy samples that can propagate the task models and may thwart from getting the
21

---

Under review as a conference paper at ICLR 2021
maximum benefits from the XLA framework. Thus to ensure the generation of better vicinity samples, we choose to use XLM-R the best performing multi- -ingual LM to date, as the vicinity model fmm in our framework.
For the task model ((0), in principle we can use any multilingual model (e.g. mBERT, XLM-R) while using XLM-R the vicinity model. However, if we weaker model (e.g., mBERT) compared to the vicinity model, the performance gain may not be easily distinguishable, i.e., the gain may come from the increased generalization capability of the stronger vicinity model. This, in turn, can make us unable to evaluate the XLA framework properly in terms of its adaptation capability. In addition, from Table and Table 2 (in the main paper), we observe that the zero-shoo shot XLM-R outperforms mBERT in the warm-up step by ~ 3.8% in NER and ~ 13.46% in XNLI. Therefore, we choose to use XLMMRR for both the task model ((0) and vieinity model ftm- Using this setup, an improvement over the baseline in XLA strictly indicates the superior performance of the framework.
It is also both attractive and challenging to use a single LM (XLM-R) as the vicinity model Oim over different languages. Note that the vicinity model in our framework is disjoint pre-trained entity whose weights are not trained on any task objective. This disjoint characteristic gives our framework the flexibility to replace fim with a better monolingual LM for specific target language, which in turn makes our model extendable to utilize stronger and new LMs that may come in future.
E.3 DATASETS (EXTENDED VERSION)
For XNER, we transfer from English (en) to Spanish (es), German (de), Dutch (nl), Arabic (ar), and Finnish (fi). For English and German, we consider the dataset from CoNLL-2003 shared task (Sang and Meulder (203)) while for Spanish and Dutch, we use the dataset from CoNLL-2002 shared task (Sag (2002) We collected Arabic and Finnish NER datasets from Bari al. (020). The NER tags are converted from IOB1 to IOB2 for standardization and all the tokens of each of the six (6) datasets are classified into five (5) categories: Person, Organization, Location, Misc., and Other. Pre-trained LMs like XLM-R generally operate at the subword level. As a result, when the labels are at the word level, a word is broken into multiple subwords, we mask the prediction of non-first subwords. Table 9 presents the detail statistics of the XNER datasets. We see that the datasets for different languages vary in size. Also the class distss-ittiibtiion is not balanced in these datasets. Therefore, use the micro F1 score the evaluation metric for XNER.
XNER.
Table 5: Statistics of training, development and test datasets in different languages for XNER.
Lang
Train
Dev.
Test
XLMR data
% of en
English
14041
3250
3453
300.8 GB
100
Spanish
8323
1915
1517
53.3 GB
117.0
Dutch
15519
2821
5076
29.3 GB
~9.74
German
12152
2867
3005
66.6 GB
222.1
Arabic
2166
267
254
28.0 GB
~9.30
Finnish 13497
986
3512
54.3 GB
188.0
XNLI. We use the standard XNLI dataset Conneau et al. 2018) which extends the MultiNL dataset Williams al. (2018) to 15 languages. For a given pair of sentences, the task is to predict the entailment relationship between the two sentences, i.e., whether the second sentence (hypothesis) an Entailment, Contradiction, or Neutral with respect the first one (premise). For XNLI, we experiment with transferring from English to Spanish (es), German (de), Arabic (ar), Swahili (sw), Hindi (hi), and Urdu (ur). Unlike NER, from Ta we see that the dataset sizes are same for all languages. Also the class-distributien is balanced in all the languages. Thus, we use accuracy the evaluation metric for XNLI.
PAWS-X. The task of PAWS (Paraphrase Adversaries from Word Scrambling) (hhng al. 2019) to predict whether each pair is a paraphrase or not. PAWS-X contains six typologically distinct languages: French, Spanish, German, Chinese, Japanese, and Korean. For this task, we experiment
22

---

% of en Under review as a conference paper at ICLR 2021
Table 6: Statistics of training, development and test datasets different languages for XNLI.
Lang
Train
Dev.
Test
XLMR data
% of en
English
392702
2490
5010
300.8 GB
100
Spanish
392702
2490
5010
53.3 GB
~17.0
German
392702
2490
5010
66.6 GB
~22.14
Arabic
392702
2490
5010
28.0 GB
~9.30
Swahili
392702
2490
5010
1.5 GB
00.5
Hindi
392702
2490
5010
20.2 GB
6.77
Urdu
392702
2490
5010
5.7 GB
~1.8
with transferring from English to all of these six languages. Table presents the detail statistics of the PAWS-X datasets. Similar to XNLI, we use accuracy as the evaluation metric for this task.
Table Statistics training, development and test datasets different languages for PAWS-X
Lang
Train
Dev.
Test
XLMR data
English
49401
8000
8000
300.8 GB
100
French
49401
2490
5010
56.8 GB
18..8
Spanish
49401
1962
1999
53.3 GB
17..0
German
49401
1932
1967
66.6 GB
~22.14
Chinese
49401
1984
1975
63.5 GB
~1111
Japanese
49401
1980
1946
69.3 GB
233.4
Korean
49401
1965
1972
54.2 GB
~18.02
E.4 SETTINGS (EXTENDED VERSION)
We present the hyperparameter settings for XNER and XNLI tasks for the XLA framework in Table 8 In the warm-up step, we train and validate the task models with English data. However, for cross- Tingual adaptation, we validate (for model selection) our model with the target language development We train our model with respect the number of steps instead of the number of epochs. In the case of a given number of epochs, we convert it to total number of steps.
For both tasks, observe that learning rate a crucial hyperparameter. In table 8 Ir-warm-up-steps refer to the warmup-step from triangular learning rate scheduling Smmth (2015). This hyperparameter be confused with Warm-up step of the XLA framework. In our experiments, batch-size is another crucial hyperparameter that can be obtained by multiplying per GPU training batch size with total number of gradient accumulation steps. We fix the maximum sequence length to 280 for XNER and 128 tokens for XNLL.
For each the experiments, we report the average score three task models, (11) g(2), ((3), which initialized with different seeds. We perform each of the experiments in single GPU setup with float32 precision.
23

---

Under review as a conference paper at ICLR 2021
Table 8: Hyperparameter settings for XNER, XNLI, and PAWS-X - task.
Hyperparameter
PAWS-X
Warm-up step
X-lingual adaptation
Warm-up step
X-lingual adaptation
Warnsup step
X-lingnal adaptation
Training- by perparameters
modzi-type
maim-up-cckp
xln-z
varr-up- pacc--pp-ckpp -.pp..
xlars
wain- up-ckktt
sampling factor
0.7
0.7
0.7
drop-oui
0.1
0.1
0.1
0.1
0,1
max-seg-length
280
280
128
pes-apu-train-bee
goufacemmttiioneepe
logging-seep
50
50
50
learning-rate cate (n
3-
56
Ie-d
Ic
Ie-t
le-0
warm up steps
200
10% of train
10% of train
10% train
10%
10% train
weight-deeay
adam aaai-opsilon
1c
1e-6
1c
max- grad pram-
1.0
1,0
1.0
L.D
1.0
1.0
num of srain epochs
XLA-cpochss
3
6
10
steps main--luaagerccataage
3000
100
conf-penalty
True
Palas
True
Fallw
Palae
Distilation--p perparmeeers
Emisture--oomponent
postesior thresbold
0.5
covariance-
distilation- factor
80,
distillation type
concidenne
confidence
conEidance
Angmontation--oyuen hyperprrameerrrs
da wow-owwerccom case
False
Faise eaccesoive-nax
False
False
Falez successive-croEE
aag-type
#ucceseive-croos
percentage
30
30
40
diversifiattio factor
2x2
2x2
24

---

Under review as a conference paper at ICLR 2021
F RESULTS (EXTENDED VERSION)
We include detailed results for CONLL-XNER, XNLI, and PAWS-X datasets to compare with previous literatures. We also provide standard deviations over three different random seeds here.
Table 9: Results in Fl score for Cross-lingual Named Entity Recognition (XNER). "x" represents model fails to converge and represents no results were reported for the setup.
Model
en
65
ml
de
ar
Supervised Result
(Chur-faxrTex) bi-LSTM-CRI Bari
2020
89.77 0.19
84.71 0.06
85.16 0:21
78.14 0.42
75.49 0.53
84.21 0.13
XLM-R Connenu 2020
92.92
89.72
92.53
85.81
mmp
0.32
86.8 0.53
Zero- Baseline
fasrText-bi-I STM-CKF RF Bari
2020
88.98 0.25
CCaar-aasTootibLLSSTTT Ban al 2020
89.92 0.15
26.76 1.45
20.94
0.74
1.43
BERT SERRT-hare-coo cased
91.21 0.18
52.88 1,33
29.16
3.30
44.41 2.36
30.18 1.93
Keune
Winp Ce 4019
75.77
79.03
70.54
Wir and Dieu72
2019
74.9%
77.57
69:56
2020
78.64
80,80
71.40
BiT
2020
75.93 :: 0.81
74.61 + 1.24
65. 65.24 = 0.56
36.91
2.74
53.77
1.54
implementationn
XLMMR (Our implementation)
92.23 0.19
79.29 0.43
80.87
0.90
73.40 = 0.96
49.04
1.19 52.3
75.57
0.94
XLM-R (ensemble)
92.76
80.6
81,46
75.4
76.85
Our Method
mBERT-cased conf-poualty
90.81 0.17
75.06 0.63
79.26 = 0.65
72.31 = 0.52
47.03 = 1.65
66.72
0.44
XLM-R conf-penalty
92.49 0.09
80.45 0.42
81.07
0.12
73.76
1.01
49.94
0.43
76.05 0.25
XLA
83.05 0.38
85.21 0.23
80.33 = 0.07
0.56 57.35
79.75 0.34
XLA (ensemble)
83.24
85.32
80.99
$8.29
79.87
Table 10: Results in Accuracy for Cross lingull Natural Language Inference (XNLI) task.
Model
en
es
de
ar
SW
ur
Supervised Result TRRANSLATE-RRAINNCOE
Huung et (WIL+MM (Home et al. 2019
85.6 85.4
82.3
$0.9
78.2
73.8
73.4
69.6
XLM-R (Rese Coemenu 20.00
82.2
80.3
77.3
73.1
76.1
73.0
XLM-R Conneou
$9.1
86.6
85.7
83.1
78.0
81.6
78.1
Zero Resoure Baseline for Full (100%) English laheled tmining set
ample Wott
76.3
XLM IK traper Connent di 2000
$9.1
85.1
83,9
79.8
73.9
76.9
73.8
XLM-R R (XTREbI, ai 4020
$8.7
83.7
82.5
77.2
71.2
75.6
71.7
XLM-R (Our implenkamaton)
88.87 = 0.31
$4.34 = 0.37
82.78 0.56
78.44
0.50
72.08 105
76.40 4: 0.87
72.10 :: 1.22
XLM M-RRRR (ensemble)
89.24
84.73
83.27
79.06
73.17
77.23
73,07
Mathod
XLM-R conf pona-penalty
XLA (ensembbie)
86.12
84.61
$0.89
74,89
78.98
73.45
Baseline
English labeled urining
XLM-R (Our implementation) XLM (ensemble)
83.08 # 1.04
78.48 0.76
77.54 0.60
72.04 0.99
67.3 a 0.66
70.41 + 0.09
66.72 0.29
$4.65
79.56
78.38
72.22
66,93
71.00
66.79
Method
XLM-R conf- conf-penalty
74.70
XLA (ensembbie)
$2.35
81.93
78.56
73.53
75.20
71.15
25

---

Under review as a conference paper at ICLR 2021
Table 11: Results in Accuracy for PAWS-X X task.
Model
de
ko
zh
Supervised (TRANSI ATE ALAIN-ALI
XLM-R (our impl.)
95.8 J. 0.23
92.5
0.29
92.8
0.15
93.5
0.12
85.5
0.32
86.6
0.48
87.6 J. 0.1
Zero Resource Baseline for Full (100% English labeled training set
XLM-R (XTREME)
94.7
89.7
90.1
90.4 90.85
78.7
79.0
82.3
XLM-R R. (our imp.)
95.46
0.36
90.06 -: 0.59
89,92
0.54
0.71
79.89 1 1.17
79.74 1.47
82.49 1. 0.82
XLMMR (ensemble)
96.10
90.75
90.55
91.80
80.55
80.70
83.45
XLM R+con penalty
95.38
0.15
90.75 J. 0.29
90.72
0,56
91.71 92.85
+: D.31
81.77
1 0.63
82.07
0.54
84.25 1: 0.36
92.27
0.75
92.28
84,27
86.90
XLA (ensemble)
92.55
92.35
93.35
84.30
84.35
86.95
Zero-Resonee Resource Baseline 5% English labeled training
XLM imp.)
83.72 1.64
84.32
1.76
85.08
1.29
73.65
72.60
XI. M-R (ensemble)
92.05
84.05
84.65
85.75
74.30
71.95
77.50
Recon-penally
91.85 0.70
86.15
1.37
86.38
1.02
85,98
0.44
76.03 + 1.51
75.43 1.32
79.15 + 1.14
XLA
89.05
0.85
90.27
0.38
90. + 0.28
80.50 + 0.73
79.60
0.43
82.65 + 0.44
XLA (ensemble)
89.25
90.85
90.25
81.15
80.15
82.90
26

---

Under review as a conference paper at ICLR 2021
G EXAMPLES OF AUGMENTED DATA
We present examples of augmented samples generated by our vicinity model for XNER, XNLI, and PAWS-X datasets in Tables 12 13 and 1 respectiivel
Table 12: Examples of augmented data from XNER dataset.
English
Original: Motor-bik registration rose 32.7 percent in the period.
Augmented: Motor- bike sales rose 32.7 percent in the US
Original: He will be replaced by Eliahu Ben-Elissar former Israeli envoy to Egypt and right-wing Likud party politician
Augmented: Hc will be led by Eliahu Cohen, a former UN Secretary to Egypt and fight-wing opposition party leader.
Original:
Isracli Isracli-Syrian peace talks have been deadlocked over the Golan since 1991 despite the previous govermment' willingness to make Golan concessions.
Augmented: The peace talks have been deadlocked over the Golan since 201 despite the Saudi govermment 's willingness to make Golan concessions.
Spanish
Original: esto comida abunda demasiado patriotería.
Augmented: En medio de la guerra abunda demasiado violencia.
Original: Pero debe cómo no estar abierta incorporaciones foráneas.
Augmented: También dehe, cómo estar ahierta personas diferentes
Original: Deutsche Telekom calificó esta compra . cuyo precio especificó, como otro paso hacia internacionalizacónn mediante adquisiciones mayoritarias destinadas a tener el control de la dirección de esas empresas.
Augmented: Deutsche Bank calificó esta operación cuyo importe no especificó como otro paso hacia expansión mediante acciones mayoritarias destinadas a tener el control la dirección de las empresas...
Dutch
Original: Onvoldoende
straf uit te spreken, luidt het
Augmented: Onvoldoende om een zware waarheid uit leggen is het.
Original: Dit hof verbindt geen straf aan de schuld die ze vaststelt.
Augmented: Dit hof geeft de schuld aan de schuld die vaststelt.
Original: Wat jaren meeging als een omstreden CVP-dossier krijgt nu door de rechterlijke uitspraak het cachet van een oude koe in de gracht.
Augmented: Wat jaren begon als omstreden CVP-dossier krijgt door de rechterlijke macht het cachet van de heilige koe in de gracht.
German
Original: Gleichwohl bleiht diese wissenschafttich abgeleitete Klassifizierung von Erzähltypen nur äußerlich
Augmented: Gleichwohl bleiht die daraus abgeleitete Klassiftzierung von Erzähltypen hegrenzt.
Original:
Dies führt vielmehr zu anderen grundlegenden Mißverstäändnissen, die zur Verwischung entschei- dender Unterschiede beitragen.
Augmented: Dies führt vielmehr zu sehr großen Mißverständnissen. die zur Verwischung entscheidender Informationen führen.
Original: Die eine Geschichte zerfällt dabei in viele Erzählungen, die wiederum wissenschaftlich genau nach unterschiedlichen Genres klassifiziert werden können Augmented: Die ganze Geschichte zerfällt dabei in viele Erzählungen die nicht ganz genau verschiedene Genres gestellt werden können.
27

---

Under review as a conference paper at ICLR 2021
Table 13: Examples of augmented data from XNLI dataset.
English
Original:
text One of our number will carry out your instructions minutely.
text_b: A member of my team will execute your orders with immense precision.
Augmented:
text_a: One of our number will carry out your order immediately
text_b. A member of my team will execute your orders with immense care.
Original:
text a: my walkman broke so i 'm upset now i just have to turn the stereo up real loud text_h, I 'mm upset that my walkman broke and now have to turn the stereo up really loud.
Augmented:
text_a: my stereo broke so 'm stuck. just have to turn the stereo up super loud
be I 'm upset because my phone broke and now I have turn the music up really loud.
Spanish
Original:
text a: Bueno, porque lo caliente que quiero decir como en el más frio que se pone en invierno ahí abajo. cuánto es?? ?
text_ b: Hace calor todo el tiempo donde vivo, incluido el invierno..
Augmented:
text_a _a: Bueno, pero lo primero que quiero decir como en el caso calor que se pone en invierno ahí arriba, cuánto es? ?
text_b: Tengo calor todo el tiempo que puedo. incluido el invierno..
Original:
text a: Si, es como en louisiana donde ese tipo que es como un miembro del ku klux klan algo fue elegido poco aterrador cuando piensas eso
text_b: Un miembro del ku klux klan ha sido elegido louisiana.
Augmented:
text_a: Sí, estuvieron louisiana y ese tipo que aparece como un miembro del ku klux klan algo sido poco aterrador cuando piensas en eso
text_h: Un miembro del kumite klan ha sido detenido en louisiana.
Arabic
Original:
text a: الإ السوق في الأففل اللات من السدد مناك بططية wil text_b. افهل اله كجديي ان
ابع اाل
منه
Augmented:
text_a:iلl السوق في الأففل الللات الكتر منال i مميح wsl
text_b. انهلل का تتجللل طريقة أبع مند
Original:
text_at dلل جكنن لي الجريية الرحلات رسفن التفاق be النطال في سااجية سفن جي توجد ل
سفن طلوور زلل التعرقق وقد
Augmented:
text_a. زال عدد لي البرية المرب وسف
السباات الخااق
طبور زلله
سبي وفد
be. النطقة في أحري أننة توجد ل
28

---

Under review as a conference paper at ICLR 2021
Table 14: Examples of augmented data from PAWS-X dataset.
English
Original:
text At this time Philips Media was taken over by Infogrames. , who became the publisher of the game.
text_b: At the time, Infogrames was taken over by Philips Media, who became the publisher of the game.
Augmented:
text_a: At this time Philips Games was taken over by Ubisoft, which became developer of the game.
text text_b: At the end, was taken over by Philips Software. who became the publisher of the magazine..
Original:
text November 2010, she was rated as the fifth highest ranked under-20 female player in the world text bext_h: November 2010, , she was rated as the fifth highest player the world.
Augmented:
text_a: In March, 2010, she was rated as the second highest ranking professional female player in the world text_b:In March 2015, she was listed as the second highest player in the world.
French
Original:
text Vers 1685, il s s'installe à Neu - France et réside quelque temps au Québec. text_b: II s s'installe Québec vers 1685 réside Nouvelle- Francee depuis un certain temps.
Augmented:
text_a En 1950 s' s'inntaalle à Neu York et passe quelque temps au Québec. text_h: Il arrive à Paris vers 1960 et vit en Nouvelle-Frann depuis un certain temps.
Original:
text_a Roger Kirk est East London. a été élevé et éduqué Norfolk. text_h, hh: Roger Kirk est à East London. a fait ses études et grandi Norfolk.
Augmented:
text a: William Shakespeare est né East London. Il a été élevé et educat à Norfolk. re Robert Kirk est né à East. Il a fait études et grandi Norfolk.
Korean
Original:
text a: 9 a лd >|.. 18
text rext_b:T99)
2 Aol स,, 2/q,
24
WE
Augmented:
text_a:o 2 Hol, 1 Zt ب ge Sh
EE N 7
2 gO 64C
text text_b:o
J 7o.
de
Original:
text text_at#7712 (Bulhar)=
/ E. 1 944
text 7 =
- #
न.. 44.
Augmented:
text_a: = (Buhhar)
1 7 84
text b: 7| (Bulhar) x
A 3 2 7
H ACKNOWLEDGEMENT
We would like to thank the authors and contributors of the Transformers library (Wolf et al. 2019). We use their implementation for XLM-R language model. We would also like to thank the authors and contributors of the fairseq library (Ott et al.) 2019). We use their implementation of XLM-R language model from torch We also thank Alexis Conneau for his replies in github repositories.
29


Reviews:
[{"review_id": "uineFvBjwqD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper  presents a data augmentation framework for zero-shot cross-lingual transfer learning. The framework uses different types of data (labeled source data, unlabeled source data, automatically generated augmented data) for training a model for the target language. Experiments are conducted on three different tasks: Named Entity Recognition (NER), Natural Language Inference (NLI) and paraphrase identification (PAWS). The approach combines multiple components together namely self-training, augmented sentence generation and confidence penalty.\n\nData Augmentation:\nThe paper states that data augmentation has been successfully used in images but not so much in  text (excluding back-translation). In fact, replacing creating augmenting text by masking and replacing word has been used in NLP both before and after pre-trained LMs such as BERT, etc.\n\n- Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations. Kobayshi. NAACL 2018\n- Conditional BERT Contextual Augmentation. Wu et al. 2018.\n- AUG-BERT: An Efficient Data Augmentation Algorithm for Text Classification. Shi et al. CSPS 2019\n- A lexical and frame-semantic embedding based data augmentation approach to automatic categorization .... Wang and Yang. EMNLP 2015.\n\nThe paper argues that generating new samples for data augmentation using the vicinity distribution of the source and target samples is better than back-translation since you cannot transfer the labels in sequence tagging tasks with back translation. However, similar problems would occur with vicinity distributions based augmentation since even changing a single word may result in changing the meaning of the sentence and hence the label (as the paper argues in Section 3.3.) . Given that, it would be useful to see more discussion/experiments comparing the two types of augmentation strategies especially that the self-training step can leverage augmented data without labels\n\nExperiments:\nThe paper provide a lot of interesting ablation and analysis. However one of the key questions that I couldn't get an answer to is what is the value of each source of data and could they be used in any different way. For example, what happens if we only do data augmentation for the source only, or the target only or by using translation or back-translation, etc. \n\nOn a related note, it looks like the specific order in which the datasets are used is important as shown in the experiments but it is not very clear what is the intuition behind that choice and whether other choices were considered or tried.\n\nOther questions:\n\nWhat is the benefit of Successive cross (vs. successive max)?\n\nWikiann is much bigger and has covers 40 languages. any reason why only 3 languages were considered from Wikiann and other languages from CoNLL?\n\n -----\nEdited after authors responses:\nI would like to thank the authors for the detailed response and the changes they have made to the paper. \n \n- Regarding contextual data augmentation (Kobayshi et al., Wu et al., etc.): Thanks for pointing out that these method use the labeled data to finetune the LM to make sure that words are replaced with other \"label-compatible\" words. Note that the comparison is not intended to necessarily show that the proposed method outperforms these baselines. Rather it is intended to guide the reader into making a decision about which method is more appropriate for which problem. If the findings are that the performance is comparable but one method will eliminate the additional finetuning step, that would be a useful finding to share. Also , it is not clear that  these methods would require labeled data in the target language for the finetuning or not.  For example, can the source labeled data be used for the finetuning step?\n\n- Regarding translation: Cross-Lingual transfer via Machine Translation does suffer from the label transfer problem for sequence tagging tasks (transferring labels for sentence-level tasks is straightforward). However, there has been several methods to address this in the literature by using unsupervised word alignment  (e.g., Yarowsky et al., 2001; Ni et al., 2017) or attention weights from NMT models (Schuster et al., 2019), heuristic approaches (e.g., Ehrmann et al., 2011) or co-learning alignment and tagging (Xu et al., 2020).\n\nA detailed comparison and discussion of the trade-off between the performance of each method and the resources they require would make the paper much stronger ", "extra_scores": {}, "extra_text": {}}, {"review_id": "peqfm5lbo4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper improves 1-to-1 cross-lingual NLP by proposing a new data augmentation method. Overall, the paper is clearly written. The proposed method is intuitive to understand and is novel in crafting augmentation texts using masked language model of BERT and relabeling these vicinal examples.  \n\nExperiments on cross-lingual NER, NLI and paraphrase detection tasks demonstrates the effectiveness of XLA, outperforming  XLM-R methods by a large margin and setting up new baselines for future cross-lingual models.\n\nHowever, as the paper focuses on data augmentation in NLP, it should discuss about data augmentation methods in mono-lingual (e.g. English) and cross-lingual NLP (if there exists), pointing out their advantages/disadvantages under the paper's setting.\n\nAs the paper crafts examples with and without vicinity separately for the target language, which part contributes more to the improvement of XLA over XLM-R? \n\nPros:\n- The augmentation algorithm is clear and results are promising.\n\nCons:\n- Lacking comparisons and discussions with other data augmentation methods in NLP.\n- Hard to infer the contribution of $D_t^\\prime$, $\\tilde{D_t}$ and $\\tilde{D_s}$ separately from Fig.1.", "extra_scores": {}, "extra_text": {}}, {"review_id": "laQgAlNGwuE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new data augmentation framework for low-resourse (and zero resource) cross-lingual task adaptation by combining several methods (entropy regluarized training, self-training). The authors conducted extensive experiments on three cross-lingual tasks, demonstrating the effectiveness of XLA. In addition, the authors compared different choices in the XLA distilation stage and claimed that the gain from XLA is beyond the model ensemble effect.\n\nAccording to the experiment results, the XLA framework has a remarkable gain over previous methods. However, it's not clear which component actually contributed to this gain. An thorough ablation study on different model component will help clarify this concern.\n\nThe writing style is a bit redundant and confusing. For instance, the author use \"model distillation\" to define the label selection procedure, yet, this term is another technique in machine learning literature.\n\nLastly from a methodology perspective, I think the algorithm seems like a combination of existing methods and its novelty is incremental to me. I hope the authors can edit the paper more carefully with a clarification on their novelty.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3kggt2iMQQM", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present an unsupervised data augmentation framework for cross-lingual NLP. Their method, called XLA, combines self-learning with co-learning and filtering. They generate additional synthetic examples by replacing words with predictions from pretrained multilingual masked LM (taken from XLM Conneau 2020). The key contribution in this paper is how they get reliable labels by simultaneously co-training three student models and using them to filter examples for training each other to avoid confirmation bias.\n\nStrengths: \nThe authors present three representative cross-lingual transfer tasks: XNER, XNLI, and PAWS-X with strong results for all three. For XNER XLA gets SoTA in all languages, and they show higher scores for structurally dissimilar and low-resource languages.\n\nWeaknesses:\nThe method is quite elaborate requiring three epochs, with different training sets at each step, three different models for labelling, and then a combination of co-teaching, co-distillation and \"co-guessing\". I think that the justification for all these design decisions is not entirely convincing and it feels slightly over-engineered. How robust is this framework? How generalisable? \n\nI also think that the explanation of the architecture is not as clear as it could be. The text in the intro to Section 3 is augmented with an Algorithm and a diagram in the appendix. Each of these explanations are presented separately, and neither the algorithm not the figure are fully explained. I think the algorithm should go in the appendix and be fully explained. I think the figure should go in the main article and be referred to in the text when describing the model at a high level.\n\nEven with these limitations, I think this paper makes enough of a contribution to warrant being accepted. Hopefully some of my concerns can be addressed before the camera ready paper. \n\nTable 1: Ensemble of what three models? \n\nWhy not report the supervised results ie. Like the Conneau et al 2020 paper? I know the main point is the zero-shot performance, but it is an interesting data point anyway especially as you have already included the baseline results for this.\n\nFigure 3: Needs to be described - how did you plot this graph? And how is the reader supposed to see that the right hand plot is more robust to OOD data??\n\nThe vicinal risk minimization principle takes up a lot of space and adds quite a bit of complexity to the description of the model. I think you could leave it out and improve the paper - this kind of data augmentation is being done already: \n\t- Sosuke Kobayashi. 2018. Contextual augmentation:Data augmentation by words with paradigmatic re-lations. InProceedings of the 2018 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, Volume 2 (Short Papers), pages 452–457,New Orleans, Louisiana. Association for Computa-tional Linguistics\n\nThe distillation by clustering method is not fully leveraged as extra information is not present in the experiments. It would be a nice extra contribution to the paper if features such as sentence length were indeed included. As it is you might as well leave clustering out of the paper and use the space for something else. \n\nYour appendix is extremely large. Please consider what is truly important and remove the rest. \n\n------\n\nResponse to author's replies:\nI am impressed by the detailed response and the changes they have made to their paper and I am happy for this paper to be accepted. I still feel like the XLA framework is quite involved and it would have been good to understand which components of this framework are crucial to its success which is why I do not increase my score. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper clearly has merits, presenting a reasonable approach to zero-shot cross-lingual learning with good results, but with limited novelty, perhaps. I am sympathetic to the departure from XTREME on NER, agreeing with the authors that using CoNLL data is more interesting than WikiANN. 

The post-rebuttal discussion centered on novelty and baselining - and specifically, whether other approaches to unsupervised data augmentation exist that should be used to baseline the proposed work. The authors argued that most of the approaches mentioned by the reviewers were in some way supervised. I personally think the confusion is a result of the paper being somewhat poorly framed:

Reviewer 2, for example, suggests a bunch of baselines. Some of these require gold labels for supervised fine-tuning to condition the MLM, but this seems like a trivial difference, which is orthogonal to using the augmentation strategies as baselines? Also, other papers have been presented that do not require gold labels, e.g. https://www.aclweb.org/anthology/D18-1100.pdf

Also, on the discussion of Täckström et al. (2012): Older approaches relying on distributional clusters *are* in fact data augmentation methods. Training on augmented data with words replaced is, in the limit, equivalent to training with clusters, when replacement words are sampled from clusters. Others have in the past proposed to use FSAs or clusters induced from static embeddings.

What the authors suggest is a form of co-training procedure, so similarly, semi-supervised algorithms - e.g., tri-training - could have been used as baselines.  

In sum, I think the sentiment shared across the reviewers is that the results are largely unsurprising, and could likely be obtained in different ways, including jointly training with a target language modeling objective, tri-training, etc. Finally, I agree with Reviewer 2 that a “detailed comparison and discussion of the trade-off” between the different approaches to data augmentation, even beyond what’s apples-to-apples, would benefit the paper. Maybe there's other advantages to the proposed approach over other baselines (effectiveness, robustness)? 

Decision: Reject