Cross-Lingual Transfer with Class-Weighted Language-Invariant Representations
Authors: Ruicheng Xian
Heng Ji
Han Zhao
Venue: ICLR 2022 Poster (2022)
Keywords: cross-lingual transfer
unsupervised cross-lingual learning
multilingual neural language model
domain adaptation

Abstract: Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from non-parallel texts and enabling zero-shot downstream transfer. While their success is often attributed to shared representations, quantitative analyses are limited. Towards a better understanding, through empirical analyses, we show that the invariance of feature representations across languages—an effect of shared representations—strongly correlates with transfer performance. We also observe that distributional shifts in class priors between source and target language task data negatively affect performance, a largely overlooked issue that could cause negative transfer with existing unsupervised approaches. Based on these findings, we propose and evaluate a method for unsupervised transfer, called importance-weighted domain alignment (IWDA), that performs representation alignment with prior shift estimation and correction using unlabeled target language task data. Experiments demonstrate its superiority under large prior shifts, and show further performance gains when combined with existing semi-supervised learning techniques.

Published as a conference paper at ICLR 2022
CROSS-LINGUAL TRANSFER WITH LANGUAGE-INVARIAN REPRESENTATIONS
Ruicheng Xian, Heng Ji & Han Zhao
Department of Computer Science
University of Illinois Urbana-Champaign
Urbana, IL 61801, USA {rxian2, hengji, i, hanzhaofeillinois..
ABSTRACT
Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from non-parallel texts and enabling zero-shot downstream transfer. While their success is often attributed to shared representations, quantitative analyses are limited. Towards better un- derstanding, through empirical analyses, show that the invariance of feature representations across languages- an effect of shared representations -stongly correlates with transfer performance. We also observe that distributional shifts in class priors between source and target language task data negatively affect perfor- mance, largely overlooked issue that could cause negative transfer with existing unsupervised approaches. Based on these findings, we propose and evaluate a method for unsupervised transfer, called aeppotaance-eightee domain alignment (IWDA), that performs representation aligmment with prior shift estimation and correction using unlabeled target language task data. Experiments demonstrate its superiority under large prior shifts, and show further performance gains when combined with existing semi-supervised learning techniques.
INTRODUCTION
Many recent state-of-the-art results on natural language processing (NLP) tasks are achieved on transformer-bot -based deep neural language models (LMs) under the "pre-train then fine-tune" paradigm (Devlin et al., 2019 Conneau Lample] 2019). Multilingual versions of these LMs are pre- trained unannotated and non-parallel texts in more than one language, such as multilingual BERT (mBERT) and XLM-R (Conneau al., 2020a), and their transformer encoder shared across languages. The cross-lingual knowledge these models acquired has enabled zero-shot trans- fer, meaning that after fine-tuned for downstream task in one source language, not only would they work well when evaluated in the source, but also quite decently in almost all languages seen during pre-training WW & Dredze 2019).
Their success zero-shot cross-lingual transfer has prompted numerous studies their multi- lingual abilities. One recurring hypothesis is that the deep architecture these models combined with parameter sharing induced intermediate feature representations that are shared across languages Karthikeyan et al. 22020] Conneau et al., 2020b; Muller et al. 2021), but they fall short of providing insights the question how shared representations contribute to cross- cross-lingual learning?
A potential effect of shared representations is representation invariance, where the model outputs similar feature representations for semantically similar inputs across all languages (Zhao et al.) 2020). Having language-invarian features means that any predictor trained in one language can be immediately transferred to other languages, giving the same predictions on inputs with similar semantics. For classification tasks, the weaker condition of class-conditional invariance is sufficient for unsupervised cross- longul transfer, where the distributions of the features conditioned on each class label are aligned across languages (Zhao et al. 2019 Ben-David et al. 2007).
Our code is available at
htt. anniinnlai innniiamann-llgnnmentt gnment

---

Published as a conference paper at ICLR 2022
We hypothesize that deep wansformer-basee multilingual LMs achieve cross- cross-lingual transfer with class- cansiconditional language-invarinn invariant representations acquired during pre-training and fine-tuning. To provide quantitative evidence, through empirical analyses mBERT and XLM-R, show that (1) transfer performance strongly correlates with the class-conditional alignment of feature repre- sentations for the downstream task (Section 2.1). During studies, we also observed that (2) per- formance is negatively affected by distributional shifts in the class priors between source and target task data (Section 2.2 Despite being a common issue with real-world data, prior shifts have been largely overlooked, and existing unsupervised gross-lingus transfer methods could cause negative transfer under its presence (Section 4.2)
Based on the above finding, for cross-lingual learning under the unsupervised setting where access to target language unlabeled task data is available along with source language labeled data, we propose importance-weighen domain alignment (IWDA). This method aims to learn language-invarian representations with feature alignment while accounting for the class prior shift with an importance weight estimation and correction procedure (Section 3 Our experiments and evaluations on multi- lingual sentiment analysis, named-entity recognition, and textual entailment show its effectiveness for unsupervised transfer and superiority over the more common semi-supervised learning (SSL) methods under large prior shifts. We also show that IWDA and SSL are compatible, where they can be combined to achieve further performance gains. The results are detailed in Section 4 along with analyses and discussions.
TWO FACTORS AFFECTING CROSS-LINGUAL TRANSFER
This section describes the experiment setup for the empirical analyses on the effects of represen- tation invariance and class prior shift on cross-lingual transfer performance, and discusses their implications on the design of unsupervised transfer methods.
Background. We introduce notation by briefly summarizing the "pre-train then fine-tune" paradigm below. Let X' + Z denote the feature mapping provided by the pre-trained LM (Z CRRC 765 in BERT-Base), and g(x) the feature computed on an input token sentence æ. For token classification tasks, the transformer's last-layer contextualized token embeddings are taken as features. For sequence classification, we follow the practice of using the embedding of the start-of- sentence marker (symbolized as [CLS]); alternatives include mean-pooling the token embeddings. To compute class label predictions ò from the features, a single linear layer with softmax activation h: 2 + y 21,2,... }} is added top of so that ÿ h(g(z)).
This end-to-end model is transferred and adapted to the downstream task by simultancously fine- tuning g and training i on task data. In zero-shot settings, fine-tuning is performed on source language labeled data only, (I, y) ~ ps, without access to any target language task data pr-
Setup. Our analysis begins with the following decomposition of the joint feature-label distribution into a product class-conditional feature distribution and marginal prior distribution,
p(2, y) p(zlu)p()) = p( E X g(x) = 2)|Y = v)p(Y y)-
Achieving representation invariance between the source and target language means that ps Pr (2ly) for all 2 E Z, y J/, and the presence of class prior shift means that ps(v) * pv( for some y. For simplicity, we use the shorthands 7 and p to denote class-conditions feature distribution and marginal class prior distribution, respectively.
To study their effects on transfer performance empirically, we compare model performance of BBERT (cased) and XLM-R Large against the alignment of their class-conditioned features and prior shift of the dataset on three multilingual downstream classification tasks: sentiment analysis on the Multilingual Amazon Reviews Corpus (MARC) which covers six high-resource languages, named-entity recognition the WikiANN dataset which covers 39 languages of varying linguistic properties and resources, and textual entailment on the XNLI dataset which covers 15 languages. Unless otherwise noted, models are only fine-tuned on English data. Due space constraints, present results from mBERT on MARC in this section while deferring the remaining results to Ap- pendix A.I and B
2

---

Published as a conference paper at ICLR 2022
MARC {de, fr, ja, zh) zero-shot cross-lluul transfer
Score
130
shift
45
30
110
F1 3.ccore
100
110
40
Itodaai feature shift
100
Conditional
40
80
0.0
0.2
Class prior shift
Class prior shift
Figure 1: sero-shot transfer performance mBERT on MARC vs. conditional feature shift (upper- left; with averages marked by language), class prior shift (lower-left), and both (right). Each scatter represents a result evaluated on one subsampled target dataset.
On these datasets and language data generally, the (mis)alignment of model representations across languages could be influenced by corpus-level semantic differences besides linguistic variations. For instance, Chinese reviews in MARC are disproportionally about books. To simulate these conditions and study the effects of class prior shifts, perform our evaluations on 500 smaller datasets sub- sampled from MARC with various class priors (each contains 2,500 test examples), and 700 from WikiANN.
2.1 INVARIANCE OF FEATURE REPRESENTATIONS
To measure representation invariance, or the alignment of clads-cond features between two languages Ps
2 Z|Y
PT we compute the following quantity, referred to as conditional feattrr shift:
ZY=j - .Pr
D(Ps
and we use the disittaeee between the feature means as the discrepancy measure
D(p, 4) = Remp(ar) By'ngla
Eam [ma]] - Ex [1
(2)
Similar first-moment measures have appeared in prior work Libbovický al. 2020), but the distinc- tion here is that the effects class prior shift are explicitly removed via conditioning, as arbitrary amounts of unconditioned feature shift could be generated from adjusting the class priors.
In the upper-left panel of Fig. we plot the zero-shot transfer performance on MARC of a fine- tuned mBERT against the conditional feature shift between English and the target language. We ob- serve that mBERT generally transfers better to languages whose features align well with the source, and the degree of alignment between languages agrees with prior findings cross-lingual transfer. For instance, the commonly analyzed factor of linguistic similarity is reflected by the alignment on this dataset (Karthikeyan al. 2020): Spanish, French, and German are considered more linguisti- cally similar to English and have better feature alignment with English than Japanese and Chinese.
The above findings suggest that achieving representation invariance is desirable for improving zero- shot cross-lingual transfer, with evidence of empirical success including the work by [Co et al. 2020) that improves mBERT performance via aligning contextualized word embeddings. But is
We compared first-moment measure
sophisticated metrics such RBF-kemel MMD Gretton
2012) and linear CKA (Korbbiihh al. 2019) preliminary studies, but found tight correlations hetween the results in our use case.
"Masuured in macro-averaged F. score instead of accuracy because accuracy could be inflated on datasets with skewed priors (Azizzadenesheli 2019).

---

Published as a conference paper at ICLR 2022
MARC (de, es, fr, zh)
MARC en (de, es, fr, ja, 2h)
features
Fi score
[CLS] features
F score
200
Conditional shift
45
Conditional shift
65
emb
10 11 12tanh
40
emb
10 12tanh
Mean-paoled
Mean-pooled
35
150
150
30
100
50
emb
5
12
emb
10
Layer
Layer
(a) Zero-shot
(b) Bilingually supervised
Figure 2: Transfer performance of mBERT MARC under two learning settings vs. conditional
shift of intermediate representations. mBERT is fine-tuned with the classification head added on top of [CLS] embedding, but plot the shifts of both [CLS] (upper panels) and mean-pooled emheddings (lower panels). This is because [CLS] embeddings are not informative until they are dataset.
contextualized in upper layers. Each curve represents a result evaluated on one subsampled target
invariance suitable when access to target language (unlabeled) task data is available under unsuper- vised transfer settings considered prior work (Keung et al. 2019 Wu et al. 2020 We perform the same empirical evaluation above, but instead of testing every unsupervised approach, we fine-tune the model with supervision on both source and target languages. This because most unsupervised objectives consist of loss on labeled source data, and terms involving unlabeled data that are in theory linked to the true loss on target data, so their ideal objective the supervised loss.
In Fig. 2b plot the conditional feature shift of mBERT trained under bilingual supervision. because last-layer conditional alignment could be an artifact of having a lincar classi- fication head top and using cross-entropy loss (Papyan et al. 2020), the shift all intermediate layers of mBERT is shown. Compared to the zero-shot model in Fig. 2a we see that the supervised model's 's better performance is accompanied by improved representation alignment across all layers. This suggests that language-invariant features are preferred over potentially language-specific ones for cross-lingual transfer, and serves the basis for algorithm in Section 3
Unlike Fig. |
2.2 DISTRIBUTIONAL SHIFTS IN CLASS PRIORS
To measure class prior shift between source and target task data, we compute the total variation between 2 and pf. In the lower-left panel of Fig. we plot the zero-shot transfer performance of mBERT against class prior shifts, and observe that performance generally degrades as prior shift in- creases regardless of how well the features are aligned. Furthermore, we observe in Appendix A. that the degradation is aggravated when the source prior distribution is skewed, i.c. highly concen- trated on few classes.
Although prior shift is typical with real-world data, most benchmarks for cross- transfer eval- uation are constructed with uniform class priors so as not shift the focus "'trick'' for how to best handle the class imbalance" (Schwenk & L.j 2018). However, unsupervised methods that leverage unlabeled target data could be affected by prior shifts, and cases, as we show with ex- isting approaches in Section 4 result in worse performance than zero-shot transfer. This was not detected in prior work because evaluations were performed on class-balanced data only. Our pro- posed method includes estimation and correction procedure, and our evaluation data with prior shifts demonstrates its consistent performance even under large shifts (Section 4.2).
To conclude analysis, plot conditional feature shift jointly with prior shift against mBERT zero-shot transfer performance in the right panel of Fig. [ and observe strong correlations between performance and either of the two factors when controlling the other.
4

---

Published as a conference paper at ICLR 2022
3 IMPORTANCE-WEIGHTED DOMAIN ALIGNMENT
For unsupervised cross- lingual learning with access to target language unlabeled task data along with labeled ones in source, the findings the previous section suggest that good transfer performance could be achieved with class-conditional feature alignment and prior shift correction. Based on these principles, we propose class-imoortance. weighhed domain alignment (IWDA), which closely follows the work by Tacher des Combes et al.] (2020). Here, we provide a high-level description of the algorithm and highlight fixes for addressing previously undiscovered optimization instabilities. Implementation details are deferred to Appendix
Assume for now that class priors are known, and denote the true importance weights (IWs) by wj := PY = /Y = j).. In Section 2.1 we demonstrated that caasicconditional feature alignment, 2 2 is desired for unsupervised transfer, but achieving this goal requires knowing the true label for every target example. Instead, we aim for a weaker goal but a necessary condition (Tachet
Ps Pr
des Combes et al. 2020), called class-weighted feature alignment, Ps - ,2 p4. where
P5 (x, Y j) = >s)ss\=))pss(>>>>>>
j)ps ( jim = ps( = j)pr = j)-
If the true w" is unknown, can be estimated when conditional feature alignment is satisfied (also known the label shift assumption) by solving of equations obtained from the following first- order relation between the target prediction distribution Y h(g(X)), source confusion matrix, and wn (Saerens et al. 2002 Lipton et al. 2018):
pr(V = = E pr = i|y jV = 8) = - ps(P = 1 10 = - ps = =i, f)ug. j=1
However, if none of the assumptions are satisfied, then there is no guarantee for achieving either of the goals above without other assumptions. The hope here is that because zero-shot models are shown to be largely aligned, by alternating the optimization towards both goals, small incremental improvements to feature alignment could make prior shift estimates more accurate, and vice versa.
Optimization.
We define the following two objectives for and w.
min (( ((M()(()))))) + 2(pg W. .17)). and min 9.
pr( Σ ps (Ū. v))))
The alignment objective minimizes source loss and the distributional discrepancy between w- weighted source features and target features. The IW estimation objective solves eq. with MSE loss using current statistics from g, h, with the constraints w 2 0 and w p. 1.
We reformulate the alignment objective into a minimax problem optimized with adversarial training, replacing D with shallow ReLU network Wasserstein-1 critic and zero-centered gradient penalty (Arjovsky et al.] 2017] Thanh-Tung et al.] (201)). We found that Wasserstein- 1 adversarial loss responds to importance weighting better than cross-entropy loss (Ganin et al. 2016).
Improving Stability of IW Estimates. In our preliminary experiments with the IW estimation procedure by Tachet des Combes al. (2020), which solves eq. 3 using quadratic programming once every epoch, we found that the estimates always tend to in long training episodes (i.e. no shift is estimated) although prior shifts are present, and even if they were accurate in previous iter- ations. This issue was previously documented, and we speculate the cause to be the noise and fluctuations from the optimization procedure for feature alignment. Another behavior we found is the tendency of overestimating the majority class of the target data.
To address these instabilities, we include three fixes in our implementation while leaving further investigations to future work. We (1) update w at every training step with projected gradient descent for always keeping it close to optimality, (2) use a decaying learning rate schedule for w, and (3) add fz-regularization to Azizzadennshelli al. 2019
5

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
We apply importance-eiihhedd domain alignment (IWDA) to fine-tune mBERT (cased) and XLM- Large for unsupervised cross-lingual learning, and compare it to baseline methods based on two semi-supervised learning (SSL) techniques. The NLP tasks are sentiment analysis, named-entity recognition (NER), and textual entailment. As with most work on cross-lingual transfer, English is used the source language in experiments.
In addition to standard multilingual benchmarks where the priors are (largely) the same across lan- guages (Section T. we also use datasets subsampled with varying priors as in Section to evaluate performance under prior shifts and investigate the limitations of existing approaches (Section 4.2)
Datasets. For NER, two multilingual benchmarks are used: CoNLL-2002 & 2003 Toong
Kim Sang, 2002, Tjong Kim Sang & De Meulder (003) and WikiANN (Pan al., 2017 Rahimi
et al. 2019). The former includes English, German, Spanish and Dutch, and the latter covers over 200 languages, from which a subset of 40 in the XTREME benchmark is selected (Hu et al. 2020). Although prior shifts (relative to English) are present in these datasets, they are mostly mild except for some languages in WikiANN, For sentiment analysis, the Multilingual Amazon Reviews Cor- pus (MARC) is used (Keung et al.) 2020). which consists of product ratings and reviews in English, German, Spanish, French, Japanese, and Chinese. For textual entailment, the XNIII dataset is used (Conneau et al. 2018). where each example is a sentence pair and the task is to determine whether the first sentence entails or contradicts the second, or neither. The test and validation examples are human-translated from English into 14 languages, and the training examples are machine-translate Conneau & Lample 2019).
Baselines. Knowledge distillation (KD) transfers knowledge from the source to the target domain by first training teacher model on labeled source data then training a student model to mimic the teacher outputs on unlabeled target data (Hinton et al., 2013). A variant of KD was evaluated on NER by Wu al. ) whose modifications include freezing the bottom three layers of the teacher and student mBRT (Wu& & Dredze 2019). Self-training (ST) trains the model on source labeled data first, then iteratively assigns pseudo-labels to unlabeled target data and trains on the most confi- dent predictions (Nigam & Ghani 2000). Our implementation is based on Dong & de Melo (2019), which was evaluated on sequence classification tasks with mBERT. While these baselines are not exhaustive, they are representative in that most methods for unsupervised cross-lingual transfer in- volve procedures that share the same fundamental ideas. For instance, |Bari et al. al.] (202T) combined data augmentation with an extension of self-training, called tri-training (Zhou & 2005).
For IWDA, a two-stage procedure is applied, where we first fine-tune the model on labeled source data only (zero-shot learning), then continue with the IWDA objectives on the source and unlabeled target task data, both for four epochs. We also evaluate the effectiveness of class-weighted feature alignment by assuming knowledge of target class priors, and label the results IWDA (oracle).
The hyperparameter settings are included in Appendix C.2
Due to space constraints, we present results from mBERT sentiment analysis and NER in this section while deferring results XNLI and from XLM-R in Appendix [A.2 and Unless otherwise noted, results are from our implemen- tation.
4.1 RESULTS ON STANDARD BENCHMARKS
Results with mBERT for sentiment analysis on MARC are presented in Table and those for NER on CoNLL datasets and three low-resource languages on WikiANN are presented in Tables and respectively. The remaining WikiANN results are included Table 4 in the appendix. All IWDA results show improvements upon the zero-shot baselines, and accompanied by decreases in conditional feature shift. The average decreases (measured with eqs. [ and [ with 100% being perfect alignment) are 51.53% and 64.77% (oracle) MARC, 21.60% and 22.33% (oracle) on CoNLL, and 32.91% % and 36.34% (oracle) on WikiANN.
3
IWDA, however, sometimes fall short of SSL baselines, which is likely due to the noise from the optimization and adversarial training as well as some amounts of incorrect alignment that cannot generally be prevented, discussed in Section 3 One way reduce noise is to improve optimality

---

Published as a conference paper at ICLR 2022
Table 1: mBERT on MARC (in accuracy).
Supervised and sero-shot buselines
Supervised (in language)
58.50
61.19
57.76
57. 05
$7.97
53.74
Zero-shot
58.50
44.80
46.49
46.02
37.37
38.48
Unsupervised readlt
Wa ct allal222000 2020
47.43
49.56
49.06
35.77
38.54
ST
58.25
50.74
48.80
47.96
42.10
41.40
79.78
KD
58.55
54,02
51.74
51.67
45.14
45
ST
53.11
51.00
IWDA (orucle)
55.96
51.95
50.83
50.01
44.91
45.96
Table 2: mBERT on CoNLL (in Fi).
Table 3: mBERT on WikiANN (in Fj).
do
bn
Supervised and zero shot buselines
Supervised and zero shot buselines
Supervised (in-language)
91.97
82.82
87.35 38
90.94
Supervised (in-langunge)
83.73
92.69 69
69
93.29
7ern shat
90.57
69.77
74.14
78.28
7em Zem-shot
83.73
68.55
50.16
35.78
Unsupervised results
Unsupervised results
Keung et al. 2019
71.9
74.3
77.6
Wa et al. 2020
71.78
46.32
44.52
2020
73.22
IWDA
71.05
55
75.79
IWDA
90.77
72.56
76.11
78.63
KD
83.07
73.15
56.05
77.58
90.89
73.71
77.14
79.72
IWDA (oracle)
83.93
72.19
60.22
74.98
TWDA (onacle)
90.75
72.58
76.48
79.17
of the adversarial critic throughout training with more updates and passes over the data, and with which we observe improved performance on CoNLL (Table (5). Another way reduce noise is to follow IWDA with a final stage SSL fine-tuning because of their label smoothing effects (Yuan et al.) 2020). Alternatively, we can view IWDA as a pre-processing step for SSL, since they theoretically rely on the assumption that source and target data identically distributed, which IWDA helps satisfy with class-weighted feature alignment. These experiments are labeled by IWDA + KD and IWDA + ST, and mostly achieve the best results among unsupervised methods (in bold).
Finally, we note that the performance from applying IWDA alone depends on (1) whether pre- trained representations already partially aligned, which is shown for mBERT in Section 2 and (2) whether the benefits enhancing the feature alignment outweigh the noise from IWDA and the potential small amounts of misalignment. The latter may not always hold, especially larger models with better zero-shot transferability. For instance, zero-shot learning with XLM-R Large on MARC achieves better performance and feature alignment than mBERT, and although applying IWDA reduces conditional feature shift, results are worse than zero-shot baselines (Table 77). In these cases, we expect the utility IWDA to be realized when combined with additional resources, but leave investigations to future work. For instance, Bari et al. (2021) achieved good performance on XLM-R with data augmentation that leverages the mulillinuual mask- filling ability of XLM-R.
4.2 RESULTS UNDER CLASS PRIOR SHIFTS
In this section, we evaluate IWDA and SSL methods on mBERT under class prior shift between source and target task data by performing unsupervised transfer from English to Japanese on MARC and from English to German on CoNLL,, using the same datasets subsampled with varying priors in Section 2 The results relative to zero-shot baselines plotted in Fig. 3
For IWDA, we observe consistent improvements across the range prior shifts. SSL approaches are also performant under mild to no shifts, but quickly deteriorate as the shift increases, eventually resulting in negative transfer. This failure mode not due to prior shift alone, but also the lack of conditional alignment between source and target data that SSL assumes. As demonstrated with IWDA + ST and KD results, when feature alignment is improved, subsequent SSL training can deliver better performance and avoid negative transfer under large shifts.

---

Published as a conference paper at ICLR 2022
MARC
CONLL en
IMDA ST IWDA toracle)
IWDA + KD
% improvement
30
Improvement
(oracle) Wu et al.. 2020
ST
20
0-
-14.6
39.1
0.0
0.4
0.0
0.10
0:15
0.20
0:25
0.30
Class prior shift
Class prior shift
(a) MARC English Japanese
(b) CoNLL English to German
Figure 3: Percent improvement in mBERT transfer performance (macro-averee -avaro-d F1) over zero-shot baselines (0%) when applying unsupervised methods under presence of class prior shifts, evaluated on datasets subsampled from MARC and CoNLL with varying priors.
We note that the importance weighting (IW) procedure in IWDA is integral to its performance under the presence of any amount of prior shift, and any alignment based methods that do not account for it will result in negative transfer. This is because under prior shift, unweighted feature alignment means that some target features from one class must be aligned with source features from a differ- ent class, hence any classifier trained the source will make mistakes those misaligned target features (Zhao et al. 2019). We demonstrate this on CoNLL English to German transfer in Fig. 7 in the appendix, where IWDA without IW sees decreasing performance as training progresses even though the shift is mild.
Interestingly, Keung al. (2019) apply alignment-bass method CONLL NER without IW that does not cause negative transfer (Table(2). This is because their method does not align the token embeddings that are used features by the linear classification head, but the [CLS] embeddings. Since the [CLS] J embeddings contain context information computed from the (intermediate) token embeddings, their alignment can be viewed weak alignment of the features. Although avoids the negative transfer from prior shifts, it also misses opportunities for greater performance gains, as comparisons with IWDA results show.
Finally, we report the accuracy of the IW estimates. The average percent prior shift corrections, computed as the decrease in total variation (TV) between true target priors and final estimated priors normalized by the amount of shift (with 100% being perfectly accurate), are 87.33% on CONLII and 28.08% on MARC for shifts over 0.8 in TV. For shifts under 0.8, the estimates are often worse than simply (although incorrectly) assuming uniform prior, with -335,8% in correction. This is because MARC reviews have noisy labels (as one could waver between giving a 3- or 4-star rating), and datasets with more class balance are harder to train of the IW estimator could be affected by bad source confusion matrices (Lipton et al. 2018).
with supervision, the performance
ADDITIONAL RELATED WORK
Cross-Lingual Ability of Multilingual LMs. Most work investigating the cross- lingull ability of
mBERT perform analysis via probing experiments, through which they identify factors that affect
transfer performance that include linguistic similarity, size of pre-training corpus, domain simi-
larity, parameter sharing, and model depth (Pires et al.) 2019 Karthikeyan et al.) 2020) Lauscher
et al., 2020, Conneau et al., 2020b; Dufter & Schütze 2020). Another line of research, includ- ing the present work, examines the intermediate representations and observes that mBERT learns cross-linge looss-liggualy shared representations that contribute to transfer performance (Wu & Dredze 2019
Conneau et al. 2020b; Muller et al. 2021).
Language-Invariane Representations. The idea of learning language-invariant invariant representations is not new and has served as the basis for a long history of approaches for cross-lingus transfer on neural LMs.
8

---

Published as a conference paper at ICLR 2022
For static word embedding models, language- invariance is achieved by aligning the embeddings of bilingual dictionary pairs (Mikolov et al.) 2013; Smith et al. 2017). With adversarial training, the dictionary need not be provided
could be automatically induced (Zhang et al. 2017 Artetxx & 2019) Dubossarsky et al. 2020), and this approach has powered many unsupervised machine trans ation approaches (Artetxe al. 2018 Lample al. | Arivazhagan
Schwenk 2019; Patra et al.
ctt al. 2019 Pham et al.
2019 Hu et al.] (2021). For deep models such as mBERT, language- invariance achieved with cross-lingual feature representation induced by sharing the transformer
body (Artetxe et al. 2020b; Pfeiffer et al., 2020).
Under unsupervised and supervised settings, prior work has proposed cross-lingual learning tech- niques based on representation alignment for models ranging from static word embeddings (Joty
al.
2017:
Chen al.) 2018), RNN (Kim et al., 2017 Huang et al. 2019), to mBERT (Keung Zheng & Lapata 2019;
et al.
2019
Xia et al. 2021). For a survey, see Ruder (201)).
Domain Adaptation. Our method, based the work of Tachet des Combes et al. 2020), is initially motivated by the problem of (unsupervised) domain adaptation, a framework for handling domain shifts between training and test data (Ben-David al. (007). For example, training a POS tagger on news articles with the goal of deploying it on biomedical papers, whose vocabulary is significantly different.
For unsupervised cross-lingual learning, we may apply adaptation techniques proposed for domain shift for cross-lingual transfer by treating task data in each language distinct domain. Besides alignment-based -ased methods (Li et al. 2020 Vernikos et al. 2020), including ours, another family of
approaches is supervssere learning, such as self-training (Dong de Melo 2019), tri-training (Ruder & Plank] 2018)) knowledge distillation (Wu et al. 2020), and data augmentation (Wang et al.
2018 Maharana Bansal 2020).
CONCLUSION AND FUTURE WORK
Motivated by the question of how shared representations contribute to cross-lingual learning on multilingual neural LMs, we perform empirical analyses that showed that downstream cross-lingual transfer performance is strongly correlated with the invariance of feature representations and nega- tively affected by the class prior shift between source and target task data. Based on these findings, propose and evaluate importance- -mportance-wiighted weighted domain alignment (IWDA) for unsupervised cross- lingual transfer, and show its effectiveness for unsupervised transfer and superiority over semi- supervised learning (SSL) methods under large prior shifts. Furthermore, by combining IWDA and SSL, further performance gains are achieved.
While present implementation of IWDA is largely effective, could be improved with future work on unsupervised domain adaptation, and techniques for prior shift estimation and distribu- tion alignment, including or beyond adversarial training. Following the results from combining IWDA and SSL, further experiments could include combining it with more elaborate approaches for unsupervised cross-lingual transfer. Finally, the success of existing zero-shot and unsupervised methods is contingent upon the cross-lingual representations acquired by the pre-trained LMs with- out supervision. Fundamental improvements to their transfer require continued investigation on the emergence of their cross-lingual ability.
ACKNOWLEDGMENTE
We thank Keyulu Xu and Mozhi Zhang for helpful discussions, and the anonymous reviewers for valuable comments. HZ would like to thank the support from a Facebook research award.
REFERENCES
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum Likelihood with Bias- Corrected Calibration is Hard-To Haad-To-Beer To-Beat at Label Shift Adaptation. In International Conference Machine Learning, pp. 222-232, PMLR, November 2020.

---

Published as a conference paper at ICLR 2022
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Roee Aharoni, Melvin Johnson, and Wolfgang Macherey. The Missing Ingredient in Zero-Shot Neural Machine Translation. 1rXXvv1903.000 070001 [cs], March 2019.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein Generative Adversarial Net- works. In International Conference on Machine Learning, pp. 214-223. PMLR, July 2017.
Mikel Artetxe and Holger Schwenk. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the Association for Computational Linguis- tics, 7:597-610, September 2019. doi: (0.1162/tacL...00255
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised Neural Machine Translation. In International Conference Learning Representations, 2018.
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. Translation Artifacts in Cross-lingual Transfer Learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7674-7684, Online, November 2020a. Association for Computational Linguistics. doi: 10. 101118861122220 10065///1122000mmmmmmmmammm...
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the Cross-lingual Transferability of Mono- lingual Representations. In Proceedings of the 58th Annual Meeting of the Association for Com- Online, July 2020b. Association for Computational Lin- putational Linguistics. pp. acl-aan..cc-
guistics. doi: 10.18653/w1/2022.acl/
Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. A Call for More Rigor in Unsupervised Cross-lingual Learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. -375-7388, Online, July 2020c. Association for Computational Linguisties. doi: 10.18653/v1/2020.0 aac.0aaalmmaaii main.658.
Kamyar Azizzadenesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized Learn- ing for Domain Adaptation under Label Shifts. In International Conference on Learning Repre- sentations, 2019.
M Saiful Bari, Tasnim Mohiuddin, and Shafiq Joty. UXLA: A Robust Unsupervised Data Aug- mentation Framework for Zero-Resource Cross-Lingual NLP. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pp. 1978-1992, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2021 acl--lnn.......... 1544
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19, MIT Press, 2007.
Steven Cao, Nikita Kitaev, and Dan Klein. Multilingual Alignment of Contextual Word Represen- tations. In International Conference on Learning Representations, 2020.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. Adversarial Deep Averaging Networks for Cross-Lingual Sentiment Classification. Transactions of the Association for Computational Linguistics, 6:557-570, 2018. doi: 10. 10.1162/tacla.0000 Saattacll__000000
Alexis Conneau and Guillaume Lample. Cross-lingual Language Model Pretraining. In Advances in Neural Information Processing Systems, volume 32, 2019.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating Cross-lingual Sentence Representations. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 447552448, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653/v/D11--12222
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised Cross-lingust Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-845) 8451, Online, July 2020a. Association for Computational Linguistics. doi: 10. 1/265///1/222.aalmmmmmo
10

---

Published as a conference paper at ICLR 2022
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging Cross- lingual Structure in Pretrained Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6022-6034, Online, July 2020b. Association for Computational Linguistics. doi: 10.18653/v1/200 1203://1/000..ccmaama..a. acl-main.536.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-418888 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/11/N19.
Xin Dong and Gerard de Melo. A Robust Self-Learning Framework for Cross-Lingual Text Classifi- cation. In Q. pp. 6306 6310, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 11551//11//11--655
Haim Dubossarsky, Ivan Vulié, Roi Reichart, and Anna Korhonen. The Secret is in the Spectra: Predicting Cross-lingual Task Performance with Spectral Similarity Measures. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2377-2390, 2390, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ v1/2020. wwt.220.mmllppmaai
Philipp Dufter and Hinrich Schütze. Identifying Elements Essential for BERT's Multilinguality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 4423-4437, Online, November 2020, Association for Computational Linguistics.
doi: 10.18653/v 2t105//1122200.mmmmmaaiaaa
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Vietor Lempitsky. Domain-Adversarial Training of Neural Net- works. Journal of Machine Learning Research, 17(59): 2016. ISSN 1533-7928.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. A Unified View of Label Shift Estimation. In Advances in Neural Information Processing Systems, volume 33, pp. 3290- 3300, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola.
A Kernel Two-Sample Test. Journal of Machine Learning Research, 13(25):723-773, 2012.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On Calibration of Modern Neural Networks. In International Conference on Machine Learning, pp. 1321-1330. 103011RR..M.RR July 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the Knowledge in Neural Network. arXiv:1503.0253/ [cs, stat), March 2015.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Gener- alisation. In International Conference on Machine Learning, pp. 4411-4421. PMLR, November 2020.
Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. Explicit Alignment Objectives for Multilingual Bidirectional Encoders. In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, pp. (op.63333643 Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v
naacl-main. 284.
Lifu Huang, Heng Ji, and Jonathan May. Cross-lingual Multi-Level Adversarial Transfer to Enhance Low- wow-Resource Name Tagging. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Vol- ume (Long and Short Papers), pp. 5823-3833, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1886/11//11913333
11

---

Published as a conference paper at ICLR 2022
Shafiq Joty, Preslav Nakov, Lluís Märquez, and Israa Jaradat. Cross-language Learning with Ad- versarial Neural Networks. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pp. for Computational Linguistics. doi: 10. 1111886/11/K1111000
Vancouver, Canada, August 2017. Association
K Karthikeyan, Zihan Wang. Stephen Mayhew, and Dan Roth. Cross-Lingual Ability of Multilingual BERT: An Empirical Study. In International Conference on Learning Representations, 2020.
Phillip Keung, Yichao Lu, and Vikas Bhardwaj. Adversarial Learning with Contextual Embeddings for 7ero-resource Cross-lingual Classification and NER. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP) pp. 1355-1360, 1360, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 111118666//vD// 1/1/3///1/I1--11////// 1138.
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. The Multilingual Amazon Reviews Corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4563-4565 Online, November 2020. Association for Computational Linguisties. doi: 10.18653/v http://1/2222mmmmmm mwtp//0200mmmmmmmmin3336.6
Joo-Kyung Kim, Young-Bune Bum Kim, Ruhi Sarikaya, and Eric Fosler-Lussies Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources. In Proceedings of the 2017 Confer- ence on Empirical Methods in Natural Language Processing, pp. 2832-28833 2238, Copenhagen, Den- mark, September 2017. Association for Computational Linguistics. doi: 10. 10.18653/1/DI1--1300
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of Neural Network Representations Revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, May 2019.
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4483-4499, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/
1222000.emmlp-main..0
Juntao Li, Ruidan He, Hai Ye, Hwee Tou Ng, Lidong Bing, and Rui Yan. Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model. In Twenty-Ninth International Joint Conference on Artificial Intelligence, volume 4, pp. 3672-3678, July 2020. doi: 10.24963/iccai. 2020/508.
Jindrich Libovicky, Rudolf Rosa, and Alexander Fraser. On the Language Neutrality of Pre-trained Multilingual Representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1663-1674, Online, November 2020. Association for Computational Linguis- tics. doi: 10. 16588661202200 100.:////2200...nngggeemmml///m findings-emmll.15000
Zachary Lipton. Yu-Xiang Wang, and Alexander Smola. Detecting and Correcting for Label Shift with Black Box Predictors. In International Conference on Machine Learning, pp. 3122-3130. PMLR, July 2018
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael Jordan. Conditional adversarial domain adaptation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates. Inc., 2018.
Adyasha Maharana and Mohit Bansal. Adversarial Augmentation Policy Search for Domain and Cross Lingual Generalization in Reading Comprehension. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3723-3738, Online, November 2020. Association for Computational Linguistics. doi: 10.18633/v12000 1000://///0000..mnnnnememm////// www.dinge--nlel..33.......3
12

---

Published as a conference paper at ICLR 2022
Julian McAuley and Jure Leskovec. Hidden factors and hidden topies: Understanding rating dimen- sions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys 13, pp. 165-172. New York, NY. USA, October 2013. Association for Computing Ma- chinery. ISBN 978-1 100-----503-2409-44 doi: 10. /41114/577157.2557711
Tomas Mikolov, Quoc V. Le, and Ily Sutskever. Exploiting Similarities among Languages for Machine Translation. arXiv:1309, 4168 [cs], September 2013.
Benjamin Muller, Yanai Elazar, Benoit Sagot, and Djamé Seddah. First Align, then Predict: Under- standing the Cross-Lingual Ability of Multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp- 2214 2231, Online, April 2021. Association for Computational Linguistics.
Kamal Nigam and Rayid Ghani. Analyzing the effectiveness and applicability of co-training. In Proceedings of the Ninth International Conference Information and Knowledge Management, CIKM '00, pp. 86 93, New York, NY, USA, November 2000. Association for Computing Ma- chinery. ISBN 978-1 778-1-58113-322-2 doi: 10.1 (0.1145/5447563354005
Xiaoman Pan, Boliang Zhang, Jonathan May. Joel Nothman, Kevin Knight, and Heng Ji. Cross- lingual Name Tagging and Linking for 282 Languages. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 1946-1958, Vancouver. Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/w1/P11-11177
Vardan Papyan, X. Han, and David I.. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117(40): 24652-24663, October 2020. ISSN 0027-8424, 1091-6490. doi: 10. 20003///...01555009
Barun Patra, Joel Ruben Antony Moniz, Sarthak Garg, Matthew R. Gormley, and Graham Neu- big. Bilingual Lexicon Induction with Semi-supervision Non-Isometric Embedding Spaces. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- tics, pp. 184-193, 193, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/11--008.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7654 7673, Online, Novem- ber 2020. Association for Computational Linguisties. doi: 10.18653/w1/202ocoo emnlp- monlp-main.66 main.617. 617.
Ngoc-Que Quan Pham, Jan Niehues, Thanh-Le Ha, and Alexander Waibel. Improving Zero-shot Trans- lation with aanguage-Independer Constraints. In Proceedings of the Fourth Conference on Ma- chine Translation (Volume 1: Research Papers). pp. 13-23, Florence, Italy, August 2019, Asso- ciation for Computational Linguistics. doi: N0.18653/11/119933
Telmo Pires, Eva Schlinger, and Dan Garrette. How Multilingual is Multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1996-5001 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/119-143
Afshin Rahimi, Yuan Li, and Trevor Cohn. Massively Multilingual Transfer for NER. In Pro- ceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 151- 164, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 18.58313/v1/ 101-
Sebastian Ruder. Neural Transfer Learning for Natural Language Processing. PhD thesis, NUI Galway, 2019.
Sebastian Ruder and Barbara Plank. Strong Baselines for Neural SuperSuper Learning under Domain Shift. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers), pp. 1044-1040 1054, Melbourne, Australia, July 2018. Associa- tion for Computational Linguistics. doi: 10. 10.1866 1/555/111P18-10000
13

---

Published as a conference paper at ICLR 2022
Marco Saerens, Patrice Latinne, and Christine Decaestecker. Adjusting the Outputs of Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation, 14(1):21-41, January 2002. ISSN 0899-7667. doi: 10. 10.1162200 1011666629976666224444
Holger Schwenk and Xian Li. A Corpus for Multilingual Document Classification in Eight Lan- guages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Associa- tion (ELRA).
Samuel L.. Smith, David H. P. Turban, Steven Hamblin, and Nils Y. Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. International Conference on Learning Representations, 2017.
Remi Tachet des Combes, Han Zhao, Yu-Xiang Wang, and Geoffrey Gordon. Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift. In Advances in Neural Information Processing Systems, volume 33, pp. 19276-19289, 2020.
Hoang Thanh-Tung, Truyen Tran, and Svetha Venkatesh. Improving Generalization and Stability of Generative Adversarial Networks. In International Conference on Learning Representations, 2019.
Erik F. Tjong Kim Sang. Introduction to the CoNLL-2002 Shared Task: Language-Independee Named Entity Recognition. COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002). 2002. 2002.
Erik Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 Shared Task: Named Entity Recognition. In Proceedings of the Seventh Conference Natural Language Learning at HL.T-NAACI. 2003, pp. 142-147, 2003.
Giorgos Vernikos, Katerina Margatina, Alexandra Chronopoulou, and Ion Androutsopoulos. Do- main Adversarial Fine- Tine-Tunigg as an Effective Regularizer. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 3103-3112, Online, November 2020. Association for Computational Linguistics. doi: 10.18653.4 10.18863/1/2220 /20.:////00....nndnngmm.mmmmm fnndings-emnlp.278.
Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. SwitchOut: An Efficient Data Aug- mentation Algorithm for Neural Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 856-861, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi:
Qianhui Wu, Zijia Lin, Börje Karlsson, Jian MaanGuang Lou, and Biqing Huang. Single-/Multi-Soure Cross-Lingual NER via Teacher-Student Learning Unlabeled Data in Target Language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3505-6514 Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl- 58i
Shijie Wu and Mark Dredze. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNIP- IJCNLP), pp. 833-844, Hong Kong, China, November 2019. Association for Computational Lin- guisties. doi: 10.18653/v 1/D19//07//
Shijie Wu and Mark Dredze. Are All Languages Created Equal in Multilingual BERT? In Proceed- ings of the 5th Workshop on Representation Learning for NLP, pp. 120-130, Online, July 2020. Association for Computational Linguistics. doi: 10. (0.188633/112220..ep4pppp
Mengzhou Xia, Guoqing Zheng, Subhabrata Mukherjee, Milad Shokouhi, Graham Neubig, and Ahmed Hassan Awadallah. MetaXL: Meta Representation Transformation for Low-resource Cross- lingual Learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 49.499-55 499-511, On- line, June 2021. Association for Computational Linguistics. doi: 10. 1018665331//22/ naacl- main. 42.
14

---

Published as a conference paper at ICLR 2022
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting Knowledge Distillation via Label Smoothing Regularization. 2020 JEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3902-3910, June 2020. doi: 10.1 10.1109/CVPR2200.220..000
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. Adversarial Training for Unsupervised Bilingual Lexicon Induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1959-1970, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.186633v1/P11-1117
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. On Learning Invariant Representations for Domain Adaptation. In International Conference on Machine Learning, pp- 7523 75322. PMLR, May 2019.
Han Zhao, Junjie Hu, and Andrej Risteski. On Learning Language-Invaie Invariant Representations for Universal Machine Translation. In International Conference on Machine Learning, pp. 11352- 11364, PMLR, November 2020.
Hao Zheng and Mirella Lapata. Sentence Centrality Revisited for Unsupervised Summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6236-6247, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 18653/v1/P19- 1628
Zhi-Hua Zhou and Ming Li. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on Knowledge and Data Engineering, 17(11):1529-1555 1541, November 2005. ISSN 1558-2191. doi: 10.1 20000/
WikiANN en - 38 langs zero-shat crass:
crass-lingua linss-lingallll transfer
score
sccoore
my
80
Conditional Feature shift
F1
70
bn
60
Conditional feature shift
Conditional shift
50
400
40
200
30
Layer
Class prior shift
Figure 4: Zero-shot transfer performance of mBERT on WikiANN vs. conditional shift of intermedi- ate representations (lower-left), final-layer features (upper-left; with averages marked by language), and jointly with class prior shift (right).
XNLI en 14 langs
Accuracy
60
0g
37
M
th SW
50
125
225
Conditional feature shift
Conditional shift
Acc.
200
70
60
emb
10
11
12 tanh
Layer
Figure 5: Zero-shot transfer performance of mBERT on XNLI vs. conditional shift of intermediate representations (lower-left), and final-layer features (upper-left; with averages marked by language). The [CL5] embeddings are examined.
15

---

Published as a conference paper at ICLR 2022
MARC en (skewed) - 5 langs
score
WIkIANN en (skewed) 38 langs
Conditonnal feature shift
Conditional feature shift
70
300
60
250
35
0.40
200
40
150
10
Class prior shift
Class prior shift
(a) MARC
(b) WikiANN
Figure 6: Zero-shot transfer performance of mBERT on two datasets vs. conditional feature shift jointly with class prior shift. Models are trained on source data with skewed prior distributions.
A
ADDITIONAL MBERT RESULTS
A.1 ADDITIONAL MBERT ANALYSES
In Section 2
we demonstrate that transfer performance of mBERT strongly correlated with feature alignment and (negatively) class prior shift between source and target task data. Here, we show that these observations also hold when transferring to NLP tasks besides sentiment analysis.
Named-Entee Recognition (NER). We use the WikiANN dataset, and analysis is performed on 700 smaller datasets subsampled from the original WikiANN with varying priors. As the 39 lan- guages this datasets have distinct linguistic properties and different resourcefulness (a summary of the languages
be found in Table 5 of Hu al. (2020), and the sizes their pre-training data in Table 1 of Wu & Dredze (0200), results based on it shall be less influenced by confounding effects, if any.
The results are provided in Fig. We observe the same correlations, but they are weaker between the prior shift and the performance. This is likely because NER structured prediction problem where mBERT could leverage dependencies between words for its predictions, making it robust to prior shifts. However, the effects of prior shift will become pronounced if the source data have skewed prior distribution, which we show below in Fig. 6b and discuss in Section A.I.I
Textual Entailment. For this high-level semantic task, we use the XNLI dataset, where we train the model on the English portion and evaluate on saman-translated validation data in 14 target languages. We do not subsample XNLI because of the small data size, hence prior shift results are included. The results are presented in Fig. 5
A. SOURCE DATA WITH SKEWED PRIORS
So far in our analyses, we have only evaluated models that trained on relatively class-balanced source datasets. We show that if the class prior distribution of the source data is skewed, then the negative effects of prior shift on transfer performance are more prominent.
For this study, we use MARC. We generate a skewed English dataset via subsampling according to the prior distribution of 14.86%, 6.43%, 8.03%. 19.28%, and 51.41% corresponding to the proportion of 5-star reviews. We then train mBERT on this dataset and evaluate its performance on subsampled target datasets. The results are presented in Fig. [aa] where the correlation between performance and prior shift much stronger than when the source is class-balanced (Fig. D.
We also evaluate on WikiANN. Although its original English portion is already considerably skewed (with class making up 50.75% of the labels), influence is observed to mild likely because
TThis the actual distribution of product ratings on Amazon.com McAuley & Leskavec 2013).
16

---

Published as a conference paper at ICLR 2022
Table 4: mBERT on WikiANN (in FI).
bg
Supervised
83,73
91.36
87
92.64
92.69
89.47
90.55
42
91.55 58
91.72
Zero-shot
71.19
76.73
IWDA
83,93
77.50
37.59
78.57
71,05
78.86
72.19
83.37
79.16
67.11
IWDA (oracle)
83.93
77.18
37.29
79.65
72.19
79.07
72.
83.72
80.23
72.69
fr
he
id
it
Supervised
92.11
91.20
90.15
85.19
86. 10
92.67
91.88
91.44
73.18
73.71
Zero-shot
40.27
77.87
80.47
56.55
60.67
80.67
IWDA
28.32
78.43
79.99
56.57
64.37
76.
51.35
80.46
38.05
61.48
(oracle)
57.62
72.00
80.99
ka
kk
ko
ml
nl
Supervised
85.06
87.71
82.68
85.42
93.27
69.
91.5 31.37
91.45
88.28
Zero-shot
66.41
46.87
60.52
53.33
57.97
67.63
50.16
82.16
80.07
65.61
TWDA
67.14
48.15
61.18
53.75
61.80
69.78
55.86
81.03
82.40
66.00
IWDA (oracle)
67.39
49.11
63,29
59.47
61.55
69.01
60.22
81.05
83.66
72.00
Ia
Ie
11
V1
zh
Supervised
78.53
76.84
93.29
90.83
82.93
Zero-shot
69.35
50,22
50.63
1.31
69.57
73.78
35.78
71.19
49.29
43.29
IWDA
67.56
53.93
$2.09
1.60
75.16
74.36
75.79
80.20
52.07
49.90
IWDA (oracle)
67.40
55.46
53.84
8.39
76.50
78.64
74.98
78.85
54.60
56.26
CoNLL en de
Table 5: mBERT on CoNLL NER (in Fi) when the adversary enjoys better optimization. Re- sults in bold are best among unsupervised meth- ods including those in Table 2]
Score
0.675
0.650
0.625
DA tho IW)
1000
2000
3000
4000
5000
6000
7000
Optimization steps
IWDA
91.19
72.91
76.93
78.62
73,83
77.87
79.96
IWDA (oracle)
91.32
72.47
71.24
79.31
Figure mBERT transfer performance on CoNLL from English to German vs. update steps. Comparing IWDA to without IW.
NER is structured prediction problem as discussed above (Fig. We skew its prior distribution by keeping only sequences containing least label, increasing its proportion 80.92%. The results of mBERT trained on this dataset are presented in Fig. 6b where we now see pronounced effects from prior shifts on transfer performance.
A.2
ADDITIONAL MBERT EXPERIMENTS
In Table 4 present IWDA results on WikiANN in the 36 target languages not included in Table
In Table 5 we present improved IWDA results on CoNLL when adversary enjoys better op- timization with more updates (we alternate between four adversary steps and one representation alignment step instead of performing simultancous updates) and passes over the data (training took 40 passes over the data). We decrease lambda iw from 2 to 1.
In Fig. we show that IWDA without IW would cause negative transfer under prior shifts as dis- cussed in Section 4.2 even under the mild shift between English and German CoNLL.
Textual Entailment. In addition to the NLP tasks sentiment analysis and NER on which IWDA is evaluated in Section 4 we also perform evaluations on the high-level semantic task of textual entailment using the XNLI dataset.
'The had zero-shot and unsupervised performance on Thai is due differences in the tokenization schemes used by mBERT and WikiANN dataset creators, who split diacritical characters into multiple tokens.
17

---

Published as a conference paper at ICLR 2022
Table mBERT on non-benchmark XNLI (in accuracy).
hi
Supervited tero-shor bazzlines
Supervised (bilingual)
73.60
65.32
68.58
68.91
66.52
71.18
69.28
63.02
67.66
57.32
63.36
63.02
48.60
68.21
69.62
Zoro-shol
73,60
60.64
60.60
67.12
63.14
67.58
67.06
57.30
64.30
47.28
51,40
57.36
43.16
62,98
63.70
Unspervvied learning
ST
73.04
60.20
63.40
66.30
62.00
67.92
67.16
56.62
64.32
47.76
53.40
57
46.04
62.88
63.90
IWDA
73.03
61.16
64.16
67.30
62.68
68.28
67.34
$7.92
65.06
47.78
$4,80
58.00
44.24
63.60
65.08
ST
73.10
60.22
66.66
48.26
53.78
(ornele)
72.94
60.74
MARC
(de, fr,
zero-shot cross-lingual transfer
score
s.cccre
2th
Conditional feature shift
100
80
55
40
60
80
100
Conditional feature shift
Conditional shift
60
50
45
100
40
emb
10
14
20
22
1anh
0.0
0.2
0.4
0.6
0.8
1.0
Layer
Class prior shift
Figure 8: Zero-shot transfer performance of XLM-R on MARC vs. conditional shift of intermediate representations (lower-left), final-layer features (upper-left; with averages marked by language), and jointly with class prior shift (right). Each curve and scatter represents result on one subsampled dataset, and the [CLS] embeddings are examined.
from English, but the training data are only avail- able in machine translations (Conneau et al. (020a).. Since IWDA training requires unlabeled target task data, avoid confounding effects due data parallelism and test-train domain mismatch from machine-translation artifacts (Artetxe al. 2020a), we partition machine-transllted data into equal portions size 20,000 (5% of the original size) and randomly select disjoint partitions to use as source and target data, as done in (Bari et al. 2021). The results, which are not benchmark due to the setup, are included Table 6
Non- Non-English XNLI test data
B XLM-R RESULTS
Besides mBERT, we also include some analysis and experiment results with XLM-R Large, a much larger model pre-trained more data than mBERT and achieving state-of-the-arr cross-lingual learning performance.
In Fig. on the next page, by evaluating MARC, show that the correlations between feature alignment, prior shift and performance also hold on XLM-R.
In Table we present IWDA and SSL results when applied to XLMMR MARC. A discussion on these results is included in Section 4.1, In Tables 9 and TO we present results on CoNLL,
WikiANN (low-resource languages), and XNLI, respectively.
In Fig. 9 we present IWDA and SSL results on MARC English to Japanese transfer and CoNLL English German under prior shifts. We that IWDA also outperforms SSL with XLM-R under large prior shifts, where the latter often causes negative transfer.
18

---

Published as a conference paper at ICLR 2022
Table 7: XLM-R on MARC (in accuracy).
Supervised and sero-shot baselines
Supervised (in language)
62.05
64.66
60.70
59.40
60.40
56.24
Zero-shot
62.05
62.52
59.04
56.43
54.32
51.42
Unsupervised readlt
Wa 2020
62.96
57.50
56.50
53.38
ST
61.00
62.32
58.09
56.62
55.27
$2.10
56.27
KD
60.26
62.42
58.60
57.24
56.94
52.20
ST
56.05
IWDA (orucle)
59.99
61.33
57.65
$6.20
56.27
$2.39
Table 8: XLM-R on CoNLL (in Fi).
Table XLM-R on WikiANN (in Fi).
my
Supervised and sero-shot busellnees
Supervised and Jero-shot baselines
Supervised (in (errlanguage)
92.92
85.81
89.72
92.53
Supervised (uu et al. 2020
84.7
97.8
76.8
97.1
Zero-sbot
92.49
72.37
77.06
82.03
Zero-shot
83.55
76.24
58.12
62.66
Unsupervised results
Unmpervised renalt
2020
Bari M1. 2021
80.99
83.24
85.32
5an et aL 2027
82.68
70.61
84.99
92.55
+ KD
92.26
76.66
79.43
82.05
KD
81.34
79.40
55.87
78.03
IWDA (oracle)
92.42
75.32
(oracle)
83.49
77.22
Table 10: XLM-R non-benchmark XNLI (in accuracy).
hi
Supervited tero-thor bazelines
Supervised (bilingual)
NA. 36.36
77.48
77 08
M0.42
80.26
81.18
16
70.82
78.24
68
77.24
74412
49.68
77:20
77.08
Zero-shot
84.36
74.72
77.42
78.58
77.68
80.26
79.14
68.12
76,84
64. 44.30
73,82
72.30
45.82
74.92
75.20
Unzupervised learning
ST
84.35
74.78
78.14
78.10
77.60
79.48
79.26
67.20
76.90
61.60
73.91
72.00
47.26
73.82
74.54
IWDA
$4.61
74.31
77.5
78.36
77.22
80.14
79.54
67.92
76.94
64 20
74,12
71 64
45.20
14.44
74.94
ST
84.51
75.62
77.36
78.82
77.46
79.94
20
67.94
76.84
65.70
74.14
72.30
47.12
74.14
75.86
(ornele)
84.69
74.68
76.00
74.14
72.84
MARC en - ja
CoNLL en de
7.5
30
IDDA
5.0
IWDA
% improvement
improvement
20
IWDA (oracle)
Wu al., 2020
1.
10-
-5.0
IWDA IWDA ST IWDA foracle)
-7.5
-10.0
-5
-18.6
0.2
0.4
0.6
0.8
1.0
0.10
0.15
0.20
0.25
0.30
Class priar shift
Class prior shift
(a) MARC English to Japanese
(b) CoNLL English to German
Figure 9; Percent improvement in XLM-R transfer performance (macro-averaged Fi) over zero-shot baselines (0%) when applying unsupervised methods under presence of class prior shifts, evaluated on datasets subsampled from MARC and CoNLL with varying priors.
19

---

Published as a conference paper at ICLR 2022
C IMPLEMENTATION DETAILS
C.I ADDITIONAL IWDA DETAILS
This section complements the high-level description of IWDA given in Section with more details on the implementation.
Recall the alignment and importance weight (IW) estimation objectives from Section below:
reproduced
2 ps (v. y)
min g.h
+ D(ps 2 .f4)). and min
pr(O)
Σ y=
Aligument Objective. The alignment objective contains a discrepancy term between importance- weighted source and target feature distributions. The source task loss added prevent g from degenerating to trivial representation, e.g. constant maps x c.
We reformulate with a critic function (also called the adversary). in generative adversarial networks (GAN) literature (Ganin et al. 2016 Goodfellow al. 2014). The role of the adversary to distinguish features == g(x) computed
minimax problem by replacing the explicit evaluation of the discrepancy D
examples z sampled from the source domain from those target examples, by outputting scores for predicting the label of the originating domain (w.l.o.g. source inputs are labeled 0). The adversary would not distinguish source and target features when they well-aligned.
The above turns the alignment objective into
min ((aa))pe(((((((x))) + Amxx 9.h
E Rappy (Lll)(()))(( 0)] Bown ((a()((())))
fEF
where lat is a specific choice of adversarial loss accompanied by an adversary function class F 3 f- The inputs to this loss function are the scores computed by the adversary and the domain labels of the examples.
In our implementation, we use Wasserstein-11 I loss for fad, whose accompanying adversary function class F is the set of 1-Lipschitz functions (Arjovsky et al., 2017). It is shown that when f attains its maximum, the term (A) exactly computes the Wasserstein-1 distance between p and pz. We parameterize F as a ReLU network with one hidden layer of width 2048. The Lipschitz condition enforced using soft constraints in the form of zero-centered gradient penalty, which penalizes the gradient norm of f evaluated at randomly sampled points ž E Z interpolating between pairs of source and target features (Thanh- Tung et al. 2019).
For added stability, we condition the alignment of features on their pseudo-labels using the method called CDAN proposed by Long et al. (2018). This method replaces the original input the adver- sary, == g(x)) E Z, by the outer product between z and the classifier output distribution, namely g(x) @ M((x)) E Rdck
Putting everything together-minimax penalty, and conditional alignment- the empirical alignment loss computed on mini-batch of source examples (xi, 1 and target examples (z/))²
minimax reformulation, Wasserstein-1 loss, zero-centered gradient
Σ ((Mo(()))) + f(o(a)) @ M(((x))) 1 m Mood) (
where ń min(n, m) and & is random point the line connecting ((x;) and g(z(). Here, two hyperparameters are introduced: (1ambda da) controls the strength of alignment, and (1 (lambda gp) controls the strength of enforcing the Lipschitz condition.
20

---

Published as a conference paper at ICLR 2022
IW Estimation Objective. Our proposed modifications for improving its stability in Section
include adding fg-reuularization to w (Azizzadenesheli et al. 2019). thereby the loss becomes
min
(T
Σ Ps(
+ $ ( 2 2 10)
where (0()) is an initial IW estimate. One hyperparameter is introduced, & (lambda_iw) _i)), which controls the strength of the regularization. When fz-regularization is implemented with weight decay, the effective value of & multiplied by 2 times the learning rate of (lr_iw).
In our case, because IWDA is preceded by zero-shot fine-tuning whose decent performance means that the features are partially aligned, we initialize (() with the solution w' to the unregularized IW loss statistics obtained from the zero-shot model, computed using quadratic programming (Tachet des Combes et al. 2020). Since the initial estimate may be unreliable, we set (o) with hyperparameter p (lambda.iw.img)
p)w' pl
Lastly, to minimize potential effects due the miscalibration of neural models (Guo et al. 2017 Alexandari et al. (020), hard confusion matrix and hard target output distribution are used in the estimation of IWs (Lipton et al. 2018 Garg et al. 2020).
C.2 HYPERPARAMETER SETTINGS
We include the hyperparameter settings used in experiments mBERT, and the only difference in the settings on XLM-R is that the model learning rate is halved. The Adam optimizer used in all our experiments, and weight decay of 0.01 is always applied. For discussion the methodology for hyperparameter tuning for unsupervised cross-lingual learning, see Artetxe et al. 2020cc.
Zero-Shot Fine-Tuning: Learning rate is le-5 with 10% warmup and linear schedule. Batch size is 8.
IWDA. Model learning rate is le-5 with 10% warmup and linear schedule. Adversary learning rate is 5e-4 with weight decay of 0.01, lambda-gp is 10, and lambda_da is 5e-3 with 10% warmup. ir_iw is 5e-4, lambda_iw (weight decay) 2, and Lambda iw.init is 0.25. Batch size is 8 per domain (totals to 16 per step).
Knowledge Distillation (KD). Our implementation follows Hinton et al. (2015). Given teacher LM fine-tuned either zero-shot or using IWDA, we distill its knowledge on a pre-trained student LM by minimizing the mean squared error (MSE) between student softmax outputs and teacher outputs, on source and target data without labels. If source labeled data available, we also train the student to minimize source task loss.
We set the softmax temperature to 3, and the weight of the source task loss to 0.1. Learning rate is chosen from (5e-5, le-4, 2e-4), and batch size is 32. As in (Wu et al. 2020 we freeze the bottom three layers the student model during distillation.
Self- Training (ST). Our implementation follows Dong de Melo (011)). Given an LM fine- tuned source only, ST continues fine-tuning the model by iteratively assigning pseudo-labels to target unlabeled data and training on the ones with the highest confidence. Source loss is also minimized during self-training
The hyperparameters of ST involve the number of pseudo-labeled examples to select and train on in each iteration. We set this {0.05%. 2% of the total number of unlabeled target examples. Learning rate 1e-5, and batch size is 128.
Number of Training Steps. On sequence classification tasks, the number of training steps is equal to 4 epochs source data. However, token classification tasks (including NER), do not set the number of steps according to the number of sequences in the dataset. This is be- cause noticed that the average length of the sequences differs between languages, which means
21

---

Published as a conference paper at ICLR 2022
that the total amount of gradient updates received by each token may vary due to the averaging in the loss function: it is smaller on domains with longer sequence lengths. Therefore, we scale the number of steps target language data using English data the reference, that the total amount of gradient updates received by each token is the same on average. The scaling is given by (average num. of tokens per sequence in (averaae num. of tokens per sequence in English).
For example, on CoNLL, the scaling is 1.17 for German, 2.18 for Spanish, and 0.88 for Dutch, and fine-tuning for the equivalent of 4 epochs over the English portion with the batch size of 8 (7021 steps) means 7135 steps on German, 9086 on Spanish, and 6954 on Dutch.
22


Reviews:
[{"review_id": "iU9C-206udk", "reviewer": "Reviewer_nGDq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides extensive analyses on cross-lingual transfer in the commonly used approaches that follow the pretrain-fine-tune strategy. In this strategy, the multilingual model finetuned on English task data is known to have zero-shot capability in the other languages, however, it is not well studied that, in unsupervised cross-lingual learning such as multilingual language model, what the role of shared representations is. Considering this, the paper gives a good start with substantial analyses that are helpful to understand what are the key factors to successful cross-lingual transfer learning.\nSince XTREME benchmark provides a variety of multilingual NLP tasks, the experiment section could be extended with more results. The targeted languages in the current experiment are considered as a high-resource language. It would be better if the authors could move the remaining results into the main 0 pages and give more discussion on them. ", "summary_of_the_paper": "In this paper, the authors provide substantial analyses on the cross-lingual transfer performance in the multilingual neural language models and reported that the performance is strongly correlated with representation invariance and negatively affected by distributional shift in class priors between data in the src/tgt languages. Based on these findings, the authors propose an unsupervised cross-lingual learning method, called importance-weighted domain adaptation (IWDA), where it performs feature alignment, prior shift estimation, and correction. The authors experimented on two different NLP tasks such as multilingual NER and multilingual sentiment analysis tasks, and experimentally showed the effectiveness. Besides that, they demonstrated that the proposed approach improves performance further, when combined with existing semi-supervised learning approaches.", "summary_of_the_review": "The paper is mostly well organized, and provides extensive analyses and experimental results. They will be helpful to the studies in crosslingual transfer learning. However, some important descriptions on the model settings or experimental results are shown in Appendix, which makes the paper difficult to read. I would suggest to revisit and reorganize the sections for better readability. Regarding the experiments, since mBERT that the authors used as a pretrained model serves diverse language representations, they could provide deeper discussion, by moving Table 4 to the main 9 pages.\n####\nI read the responses from authors, and appreciate their response and showing more results. I am okay with accepting the paper, but keep the score 6. Because one concern might be that those results/analyses are mostly described in Appendix. The authors would need to reconstruct the manuscript by moving them into the main pages. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GfM8ABq-fWz", "reviewer": "Reviewer_aVzf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper first presents insightful analysis that highlights the role of feature invariance and class prior shifts on the extent of zero-shot cross-lingual transfer. The insights from the analysis are then adapted to develop Importance-weighted Domain Adaptation for zero-shot crosslingual learning, resulting in improved performance on MARC sentiment analysis and WikiANN NER; with significantly improved robustness under class-prior shifts. \n\nStrengths:\n1. Well written and easy to understand.\n2. Insightful analysis that grounds the presented approach. \n3. Results from analysis with synthetically modified class prior distributions support the hypothesis that IWDA is improving robustness to class-prior shifts.\n\nWeaknesses:\n1. The paper evaluates on a limited set of tasks; having additional results on a wider range of tasks (for eg. additional tasks from the Xtreme benchmark) could significantly strengthen the results.\n\nSuggestions / comments / questions:\n1. Incorporating feature-invariance for domain adaptation has been studied for several NLP applications, including multilingual pre-training and zero-shot Neural Machine Translation. Discussion on several references is missing in the paper. [1, 2, 3]\n2. Last paragraph on page 3: pratrained -> pre-trained\n3. While the analysis in Figure 1 suggests that F1-score is directly correlated with conditional feature shift, is it possible there are other confounds in this analysis; for eg. the amount of pre-training data used for each of these languages?\n4. Could the relatively weaker results on NER also be caused by challenges with aligning representations at a sub-word level across languages (with different levels of tokenization granularity across different languages)?\n\nReferences:\n[1] Explicit Alignment Objectives for Multilingual Bidirectional Encoders, Hu et al.\n[2] The Missing Ingredient in Zero-Shot Neural Machine Translation, Arivazhagan et al.\n[3] Improving Zero-shot Translation with Language-Independent Constraints, Pham et al.", "summary_of_the_paper": "The paper first demonstrates the importance of feature invariance (language invariant representations) and class-prior invariance across languages on zero-shot cross-lingual performance. By analyzing the zero-shot performance on different languages on the MARC reviews and WikiANN NER tasks, and comparing against the class conditional distance between the source language (English) and target language feature representations, the authors illustrate the how high feature invariance results in better zero-shot performance. By also synthetically modifying the class prior for the target language, the authors demonstrate how increasing differences in class priors result in decreasing zero-shot cross-lingual transfer performance.\n\nBuilding on their observations, the authors propose an approach that:\n(i) Introduces an adversarial loss term to penalize distortion in average class conditional feature representations between the source and target languages.\n(ii) Adds an importance weighting term to ensure the approach doesn't fail under class prior shifts.\n\nEmpirical studies demonstrate that the proposed approach improves significantly over the vanilla zero-shot model on both MARC sentiment analysis and NER tasks, also improving over self-training on sentiment analysis. Comparisons on synthetic datasets (sub-sampled from NER and MARC datasets) that enhance the class prior shift highlight the robustness of the approach under large class prior shifts where previous approaches fail.", "summary_of_the_review": "Overall, the paper first presents insightful analysis that highlights the role of feature invariance and class prior shifts on the extent of zero-shot cross-lingual transfer. The insights from the analysis are then adapted to develop Importance-weighted Domain Adaptation for zero-shot crosslingual learning, resulting in improved performance on MARC sentiment analysis and WikiANN NER; with significantly improved robustness under class-prior shifts. \n\nThe paper produces some valuable insights and develops a well-grounded approach that improves the robustness of zero-shot crosslingual learning. However, the empirical results are limited to just two (relatively) small scale tasks. Having additional results on a wider range of tasks (for eg. additional tasks from the Xtreme benchmark) could significantly strengthen the results.\n\nGiven the thorough analysis, but limited range of tasks, I am leaning towards acceptance. If authors include additional empirical results I would be willing to update my recommendation to strong accept.\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8B0wAt60sYg", "reviewer": "Reviewer_gtqu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "===== UPDATE AFTER THE RESPONSE =====\nI would like to thank the authors for the new set of experiments and their very thorough author response. The new experiments indeed strengthen the paper and better outline the key contributions of the work. Some of my main concerns still do remain though: I am not fully certain that, given its mixed performance in the transfer tasks, the proposed framework will be very useful with more powerful transfer methods. This preliminary evidence has not fully convinced me on its future impact, and I am still not fully convinced by the choice of baselines, and whether to understand this preliminary empirical evidence as fertile ground for future enhancements in UCL. Some other clarifications were provided in the response, so I have increased my score, but as said - I still have reservations when it comes to the paper's empirical contributions and consequent impact.\n=====\n\n\nOverall, the paper provides some nice insights, especially treating the problem of cross-lingual transfer as largely a domain transfer problem, which allows it to use some domain adaptation (DA) machinery from prior work and apply it to the UCL problem. The empirical gains, although not huge, do demonstrate the alignment between the research hypothesis and empirical scores. The main strength of the paper imo is Section 2 which delves deeper into analysing factors that affect cross-lingual transfer. \n\nThe ideas of having more invariant representations to improve cross-lingual transfer are not new, and they date back to the work on cross-lingual word embeddings (e.g., see the work of Dubossarsky et al. EMNLP 2020; or Patra et al., ICLR 2019, or Zheng et al., ACL 2019). Also the idea of treating cross-lingual transfer as a DA problem is also not novel - e.g., PhD thesis of Ruder discusses the similarity of the two problems in a nice and detailed way.\n\nOne of the major concerns I have with the current paper and its current presentation is that it is very difficult to discern between novelty of this work and what was done in prior research and simply reapplied to UCL. For instance, it seems a bit that this work is a (largely incremental) application of the prior (more fundamental) work of Tachet des Combes et al. (NeurIPS 2020) to a new (but highly similar) problem. For instance, the authors should clearly note in Section 3 if they bring any methodological contribution here or they just describe the previous method of Tachet des Combes et al.\n\nIn a similar vein, the paper can also be seen as an extension of the work from Keung et al., adding this mitigation of prior class shift into the mix, which yields slight improvements. \n\nI am also not convinced by the results and the entire evaluation protocol, detecting several potential problems and weaknesses here:\n- Evaluation is conducted only on high-resource languages (from the NLP perspective), while the main promise of UCL is to improve on NLP tasks for lower-resource languages as done in plenty of contemporary NLP research. I wonder how IWDA would behave for such languages and whether it would bring any benefits. Doing zero-shot transfer from EN to DE really does not make much sense imo...\n- Evaluation is conducted on two reasonably simple tasks: NER and MARC sentiment analysis (which is a classification task with a small number of classes) - a more detailed empirical analysis on other higher-level semantic tasks (e.g., XNLI, PAWS-X) is warranted for a clearer picture of the benefits of the proposed approach.\n- The gains are typically not very high, and are close to none (when compared to the simple alignment method of Wu et al.) in the NER task. Moreover, performance on MARC sentiment is measured only against the vanilla zero-shot transfer and an older self-training (ST) baseline. What precludes the authors to again compare against Wu et al. or some other more recent techniques that go beyond the simplest zero-shot transfer protocol?\n- The paper does not really put results into the perspective of plenty of related work that has been conducted in this area recently: e.g., check the leaderboards of XTREME and XGLUE for more sophisticated baselines: a true gain would be showing that applying IWDA along with these stronger models can yield further benefits. With the current set of experiments, I believe that the paper will have very limited impact.\n- Along the same line, why is only mBERT evaluated? All concurrent work also provides evaluations with a stronger XLM-R Base model, and I also wonder whether these gains would remain with a larger and an even stronger XLM-R Large model. \n\nThe paper also misses some very relevant related work, e.g., it seems very close to this work in its optics and design: https://arxiv.org/pdf/2011.11499.pdf \n- I would like to see a discussion regarding their (dis)similarity.\nSome other papers that should have been briefly discussed and cited in this paper:\n- https://arxiv.org/pdf/2104.07908.pdf\n- https://arxiv.org/pdf/2005.00396.pdf (analysis of mBERT's multilinguality)", "summary_of_the_paper": "This paper looks at the problem of unsupervised cross-lingual transfer (termed UCL) in the paper through the optics of domain adaptation. After empirically analysing and validating that distributional shifts in class priors might cause a huge problem for UCL (which wasn't tackled in previous research), the authors proceed with an introduction of a new method that aims to mitigate that problem. The idea is to get rid of that shift through a approach called importance-weighted domain adaptation (IWDA), which is largely the adaptation of the work from Tachet des Combes et al. (NeurIPS 2020) to the UCL problem.\n\nThe results on two tasks in the UCL setup (NER and MARC classification) show slight gains over the standard zero-shot transfer when IWDA is applied, with more prominent gains reported when a stronger domain shift is observed - however, such a setup has been created mostly artificially, to further demonstrate the benefits of modelling the shift in the model.", "summary_of_the_review": "The paper presents a DA-inspired view on (unsupervised) cross-lingual transfer, offering some insightful analyses, but it seems as an eclectic (mostly incremental) work, with insufficient and lacking empirical validations, incomplete baselines, and inadequate positioning against previous work in this area, which would negatively affect its potential impact and its overall contributions - more work and a stronger empirical foundation are needed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7krmCoqfbDU", "reviewer": "Reviewer_WfZj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe proposed method is well motivated based the empirical analysis. \n\nExperiments show effectiveness of the proposed method.\n\n\nWeaknesses:\n\nThe experimental results are not convincing enough. 1) Only two of the tasks are considered. 2) only mBERT is tested. Would the same observations and conclusions hold when other widely-used pretrained models such as XLM are used. 3) According to Table 1, the proposed method is comparable with the existing work (Wu et al., 2020) on average. 4) There’s no baselines in Table 2. 5) Seemingly the proposed method is much more time consuming during training than prior works. \n", "summary_of_the_paper": "This paper investigates a method for improving the cross-lingual transfer of pretrained multilingual models. The paper first empirically analyzed the influence of representation invariance and distributional class shift. Then, the paper proposed a method to improve the representation invariance and correcting the class shift. Experiments showed its superiority under large prior shifts.", "summary_of_the_review": "The proposed method is well motivated given that the empirical analysis reveal the influence of representation invariance and class shift. Experiments show the effectiveness of the proposed method under large class shift. However, the experimental results are not convincing enough and the paper can be improved by conducting more experiments and analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a domain adaptation approach based on the importance weighting for unsupervised cross-lingual learning. The paper first analyzes factors that affect cross-lingual transfer and finds that the cross-lingual transfer performance is strongly correlated with feature representation alignments as well as the distributional shift in class priors between the source and the target. Then the paper designs an approach based on the observations.

Pros: 
+ The paper is well written and the proposed approach is well motivated. 
+ The analysis about which factors affect cross-lingual transfer is interesting and provides some great insight. 

Cons: 
- As the reviewer pointed out, the experiments for verifying the proposed approach are relatively weak.

Overall, the paper presents nice insights to connect cross-lingual transfer with domain adaptation. All reviewers lean to accept the paper and I also found the paper is in general interesting.
Decision: Accept (Poster)