Year: 2021

Abstract:
Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, evidenced by their decent zero-shot transfer performance. While studies have attributed this success to cross-lingually shared representations, quantitative analyses are sparse. Towards a better understanding of the role of multilingual representations, in this work, we first make the following observations through empirical analysis: (1) invariance of the feature representations strongly correlates with transfer performance, and (2) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work. Based on our findings, we propose an unsupervised cross-lingual learning method, called importance-weighted domain alignment (IWDA), that performs representation alignment, prior shift estimation, and correction. Experiment results demonstrate its superiority under large prior shifts. In addition, our method delivers further performance gains when combined with existing semi-supervised learning techniques.

text: Recent advances in neural modeling have produced deep multilingual language models capable of extracting cross-lingual knowledge from unparallel texts, evidenced by their decent zero-shot transfer performance. While studies have attributed this success to cross-lingually shared representations, quantitative analyses are sparse. Towards a better understanding of the role of multilingual representations, in this work, we first make the following observations through empirical analysis: (1) invariance of the feature representations strongly correlates with transfer performance, and (2) distributional shift in class priors between data in the source and target languages negatively affects performance—an issue that is largely overlooked in prior work. Based on our findings, we propose an unsupervised cross-lingual learning method, called importance-weighted domain alignment (IWDA), that performs representation alignment, prior shift estimation, and correction. Experiment results demonstrate its superiority under large prior shifts. In addition, our method delivers further performance gains when combined with existing semi-supervised learning techniques.

heading: 1 INTRODUCTION
text: Many recent state-of-the-art results in natural language processing (NLP) have been achieved on Transformer-based deep neural language models (LMs) using the “pre-train then fine-tune” paradigm (Devlin et al., 2019; Conneau & Lample, 2019). Multilingual versions of these LMs are pre-trained on unannotated and unparallel texts from more than one language, such as multilingual BERT (mBERT) and XLM-R (Conneau et al., 2020a), and it is found that the learned representations are shared across languages. In particular, the cross-lingual knowledge acquired by these models has enabled zero-shot cross-lingual transfer, where after fine-tuned on data in a source language, not only do they work well when evaluated on the source, but also quite decently for the same task on almost all languages they have seen during pre-training (Wu & Dredze, 2019).
Their success on zero-shot cross-lingual transfer has prompted numerous studies on their multilingual abilities. Among the many explanations, a recent hypothesis suggests that the deep representation combined with parameter sharing induced intermediate feature representations that are shared across languages (K et al., 2019; Conneau et al., 2020b; Muller et al., 2021). Following this, we begin by studying the following question for mutilingual neural LMs: What is the role of shared representations in unsupervised cross-lingual learning?
In an effort to answer this question in a quantitative manner, through empirical analysis on mBERT, we find that (1) its cross-lingual transfer performance strongly correlates with the alignment of feature representations for the NLP task at hand between the source and target languages (Section 2.1). In addition, (2) distributional shift in class priors between data in the source and target languages negatively affects transfer performance (sections 2.2 and 4.2). Despite being a common issue with real-world data under unsupervised settings, it has been largely overlooked in prior work.
Based on these findings on the cross-lingual transfer of mutilingual neural LMs, towards a principled approach for the unsupervised setting where access to unlabeled data in the target language for the downstream task at hand is available in addition to labeled ones in the source (hereafter referred
to as UCL1), we propose importance-weighted domain alignment (IWDA). This method performs representation alignment, class prior shift estimation, and correction (Section 3). Experiment results show that IWDA outperforms the more popular semi-supervised learning methods under large prior shifts, and can be additionally combined with them for further performance gains (Section 4).

heading: 1.1 RELATED WORK
text: Our proposed method is heavily inspired by Tachet des Combes et al. (2020), which studied the issues with alignment-based domain adaptation (DA) under the presence of class prior shifts, and proposed the inclusion of prior shifts estimation and correction procedures in DA. Our method also shares similar components with the DA-based UCL method by Keung et al. (2019), while highlighting its limitations discovered in the present work (Section 2.2).
Analyzing Multilingual Neural LMs. Most studies analyze mBERT’s cross-lingual ability via probing experiments, and identified influencing factors including but not limited to linguistic similarity, size of pretraining corpus, domain similarity, parameter sharing and model depth (Pires et al., 2019; K et al., 2019; Lauscher et al., 2020; Conneau et al., 2020b; Dufter & Schütze, 2020). Another line of research analyze intermediate representations and observed that mBERT learns crosslingually shared representations (Wu & Dredze, 2019; Conneau et al., 2020b; Muller et al., 2021).
Cross-Lingual Representation Alignment. The idea of learning invariant representations has powered a long history of approaches for cross-lingual transfer on neural models. Under zero-shot settings, it is achieved by aligning word embeddings on bilingual dictionaries that could be induced via adversarial training (Mikolov et al., 2013; Smith et al., 2017; Zhang et al., 2017; Artetxe & Schwenk, 2019; Cao et al., 2019), a method that has powered many machine translation techniques (Artetxe et al., 2018; Lample et al., 2018; Arivazhagan et al., 2019; Pham et al., 2019; Hu et al., 2021) It is found on static embedding models that the goodness of alignment affects transfer performance (Patra et al., 2019; Dubossarsky et al., 2020). On deep models such as mBERT, it could also take the form of shared feature representations in the Transformer body (Artetxe et al., 2020b; Pfeiffer et al., 2020). Under UCL and supervised settings, techniques based on representation alignment were proposed for models ranging from distributed word embeddings (Joty et al., 2017; Chen et al., 2018), RNN (Kim et al., 2017; Huang et al., 2019), to mBERT (Keung et al., 2019; Zheng & Lapata, 2019; Xia et al., 2021). For a survey, see Ruder et al. (2019) and Ruder (2019).
Connection to Domain Adaptation. Domain adaptation is a popular framework for handling domain shifts between training and test data, e.g. learning a POS tagger on news articles but to be deployed on biomedical papers with a significantly different word distribution (Ben-David et al., 2007). The concept of domain is extended naturally to cross-lingual learning by treating the distributions of sentence and document pertaining to the same task in each language as separate domains2 (Ruder, 2019). Thereby, adaptation techniques proposed for domain shift can be applied to UCL. Besides methods aiming at extracting shared representations (Li et al., 2020; Vernikos et al., 2020), another family of successful approaches is semi-supervised learning, which include self-training (Dong & de Melo, 2019), tri-training (Ruder & Plank, 2018), knowledge distillation (Wu et al., 2020), and data augmentation (Wang et al., 2018; Maharana & Bansal, 2020; Bari et al., 2021).

heading: 2 FACTORS AFFECTING CROSS-LINGUAL TRANSFER
text: To develop and provide evidence to the hypothesis that attributes the cross-lingual transfer success of multilingual neural LMs to shared representations, as well as to understand its implications on the design of unsupervised transfer algorithms, we attempt to quantify the relation between representation invariance and transfer performance in this section.
We briefly summarize the “pre-train then fine-tune” training paradigm and introduce notation along the way. Let g : X → Z denote the feature mapping provided by the multilingual LM, and z =
1With an abuse of notation. In this work, the acronym UCL refers to the learning setting considered, and is meant to distinguish from the umbrella term “unsupervised cross-lingual learning”, which also includes the study of the zero-shot setting where no target language task data is given.
2Hence the terms domain and language are used interchangeably in this work.
g(x) ∈ Z ⊆ Rd the feature vector computed on an input sentence or token x ∈ X . In practice as well as in our studies and experiments, the feature is the last-layer wordpiece embeddings for token classification tasks, and the embedding of a special start-of-sentence marker (symbolized as [CLS]) for sequence classification tasks. Hence, on BERT-Base models, d = 768 and Z ≈ Sd−1 sphere due to LayerNorm. Let h : Z → Y denote the output layer(s) added on top of g with an appropriate activation, usually a single linear layer with Softmax for classification tasks in practice. Hence, an end-to-end model is obtained, which computes predictions via ŷ = h(g(x)). By fine-tuning both g, h on labeled task data (x, y) sampled from the source language domain pS only, we get a zeroshot transfer model for the downstream task at hand, where the term zero-shot points to the fact that the model does not use any task data from the target language (x′, y′) ∼ pT during training. Based on the following decomposition of the joint feature-label distribution into the product of class-conditional feature distribution and marginal prior distribution,
p(z, y) = p(z|y)p(y) := p({x ∈ X : g(x) = z}|y)p(y),
we examine how cross-lingual transfer performance is influenced by (1) invariance of the feature representation of data between the source and target languages, with perfect invariance being pS(z|Y = j) = pT (z|Y = j) for all j ∈ Y , and (2) the distributional shift in the class priors, with no shift being pS(Y = j) = pT (Y = j) for all j ∈ Y . On classification tasks, Y = {1, 2, · · · , k} is discrete. For legibility in later sections, we use the shorthands to denote the class-conditional feature distribution and the marginal prior distribution: pZ|Y and pY .
Setup. Our empirical analysis is performed on the representations learned by mBERT zero-shot fine-tuned on English for two multilingual tasks: sentiment analysis on the Multilingual Amazon Reviews Corpus (MARC), and named-entity recognition (NER) on the WikiANN dataset (see descriptions in Section 4). To simulate the effects of varying amount of representation misalignment and class prior difference, evaluation is performed on 500 smaller datasets randomly subsampled from the original MARC according to different class priors, and roughly 700 datasets from WikiANN. MARC covers five high-resource languages (excluding English), and (our selected subset of) WikiANN covers 39 languages of distinct linguistic properties and varying resourcefulness.

heading: 2.1 INVARIANCE OF FEATURE REPRESENTATIONS
text: For studying the effects of representation invariance, we compare model performance to the discrepancy between the class-conditional feature distributions on the source and target domains, pZ|YS and pZ|YT . We take the average to aggregate the discrepancies computed when conditioned on each class, 1 k ∑k j=1D(p Z|Y =j S , p Z|Y =j T ), and refer to this quantity by conditional feature shift.
The discrepancy measure we use computes the `1-distance between the mean feature vectors3,
D(p, q) := ∥∥∥Ex∼p[x]− Ex′∼q[x′]∥∥∥
1 = d∑ i=1 ∣∣∣Ex∼p[xi]− Ex′∼q[x′i]∣∣∣. Measuring the distance between the means of representations for the purpose of analyzing mBERT has appeared in prior work. A similar quantity is referred to as language centroids by Libovický et al. (2020). The important distinction here is that we explicitly remove the effects of class prior shift by conditioning on the class labels, which we argue is an artifact of the dataset since arbitrary amounts of unconditioned feature shift can be generated by altering the class priors.
First, we consider zero-shot transfer. In Fig. 1a, we plot theF1 score4 attained by the zero-shot model against conditional feature shift between English features and those of the target language, and observe that better representation alignment is indicative of better transfer performance. Findings in prior work on the cross-lingual ability of mBERT are reflected. For instance, linguistic similarity, a factor commonly argued to be important for cross-lingual transfer, could be related to the alignment
3Comparisons to more sophisticated metrics such as RBF-kernel MMD and linear CKA were made in our preliminary studies (Gretton et al., 2012; Kornblith et al., 2019), and we found that they correlate tightly with our first-moment discrepancy measure for analyzing conditional feature shift.
4To study the effects of class prior shift, we use macro-averaged F1 as the performance metric in this section, because classification accuracy could be inflated on datasets with skewed priors (Azizzadenesheli et al., 2018).
of features: Spanish, French and German are more linguistically similar to English than Japanese and Chinese, whose features are less aligned with those of English. These results also reflects the successes of zero-shot learning approaches on mBERT that aim to achieve representation invariance.
While the above finding shows that representation alignment is desirable for zero-shot transfer, is it also suitable for UCL settings where task data from the target language are available? We start by recognizing that the objective function of most unsupervised transfer algorithms consists of two parts: loss on labeled source data, and terms involving unlabeled data that in theory are linked to the loss on target domain. Based on this, we compared against conditional feature shift the performance of the model fine-tuned supervisedly on labeled data from source and target languages, as it would be the ideal objective for unsupervised algorithms. The results are plotted in Fig. 1b. Instead of observing increases in conditional feature shifts, which would have implied that language-specific features are preferred by over shared ones for good performance on each language, we observe much-improved representation alignments between English and each of the target languages. This means that achieving representation invariance is also appropriate under UCL settings.
To demonstrate that the correlation between last-layer class-conditional alignment and performance is not an artifact of the linear classification head, we show that model performance is also correlated with (1) conditional feature shift in intermediate layers, plotted on the bottom panels of Fig. 1, and (2) class-balanced feature shift, plotted in Fig. 5 in the appendix, where the removal of prior shift effects is achieved by reweighting the class priors to be uniform (eq. 1). Furthermore, we include results on the XNLI dataset, a higher-level semantic task (Fig. 7), and results on MARC with the more powerful XLM-R Large model (Fig. 8).

heading: 2.2 DISTRIBUTIONAL SHIFT IN CLASS PRIORS
text: Most benchmarks for evaluating cross-lingual transfer are constructed so that the class prior distributions between data in the source and target languages are the same. This is desired for assessing zero-shot learning approaches, as the focus should not be on “‘tricks’ for how to best handle the class imbalance” (Schwenk & Li, 2018). However, for UCL settings with unlabeled target data, we argue that making the best use of the additional data involves the detection and correction of prior shift. Otherwise, as we show in this and later sections, negative transfer could occur.
To examine the effects of class prior shift on cross-lingual transfer, we compare zero-shot transfer performance to the difference in prior distributions, D(pYS , p Y T ), equivalent to total variation (TV). The results are presented in Fig. 2. We also plot the effects of representation alignment and prior shift jointly in Fig. 3, where stronger correlations are revealed: (1) transfer performance generally takes a hit when prior shifts are large, regardless of how well the features are aligned, and (2) on target domains of the same amount of prior shift, performance decreases when features are less aligned. The findings suggest that in addition to achieving representation invariance, good transfer performance also requires the detection and handling of potential prior shifts. While this is not possible under zero-shot settings without any data from the target domain, the unlabeled target task data provided under UCL settings could be leveraged toward this goal.
We examined existing approaches proposed for multilingual neural LMs for the UCL settings, but did not find procedures for prior shift correction, and moreover, there is a lack of attention to the potential issues and effects of class prior shift to the algorithms. Specifically, for alignment-based methods, Zhao et al. (2019) and Tachet des Combes et al. (2020) showed that adaptation could lead to negative transfer when prior shifts are not accounted for due to erroneous alignment. For methods based on semi-supervised learning, negative transfer is observed empirically under large shifts in our studies (Section 4.2). These cases of negative transfer call for more considerations to the effects of class prior shift in future work on cross-lingual transfer that leverage target data.
Interestingly, although the alignment-based method proposed by Keung et al. (2019) did not account for prior shift, it avoided negative transfer on the CoNLL NER task (which exhibits shifts) by performing feature alignment on the [CLS] sentence embeddings instead of the word embeddings (recall that predictions are made on word embeddings on token classification tasks). Since sentence embeddings on Transformer-based models could be loosely related to a weighted sum of the word embeddings, the procedure of Keung et al. (2019) is performing coarse-grained representation alignment, which avoids negative transfer while providing decent performance gain. However, as we compare in Table 1, this misses out opportunities for larger gains from word embedding alignment. If alignment were performed on word embeddings without prior shift correction, deteriorating performance would have been observed (Fig. 11).

heading: 3 IMPORTANCE-WEIGHTED DOMAIN ALIGNMENT
text: For unsupervised cross-lingual learning on multilingual neural LMs with access to unlabeled target language data (which we refer to as the UCL setting), the findings in Section 2 suggest that good performance can be achieved via aligning representations and correcting class prior shifts. Based on these principles, we propose class-importance-weighted domain alignment (IWDA) for UCL, following closely the work by Tachet des Combes et al. (2020). In this section, we provide a highlevel description of the algorithm and its connection to the findings above, and refer interested readers to the aforementioned work and Appendix C for further implementation details.
We have seen that class-conditional feature alignment is desired for cross-lingual transfer, however, this is not possible to achieve under the UCL setting since the target labels are unknown. Assume for now that the target class prior distribution is known, then we can compute the true ratio of the class priors, w∗j := pT (Y = j)/pS(Y = j), also known as the importance weights (IWs), and aim for a weaker importance-weighted feature alignment, pw∗,ZS = p Z T , where
pw ∗ S (x, Y = j) = pS(x|Y = j)pS(Y = j)w∗j = pS(x|Y = j)pT (Y = j). (1) Besides as shown in Fig. 5 that importance-weighted feature alignment also correlates with transfer performance, it is also a necessary condition for the ideal goal of class-conditional alignment (Tachet des Combes et al., 2020).
Since the true w∗ are unknown under UCL, they are estimated by matching first-moment statistics using the confusion matrix (Saerens et al., 2002; Lipton et al., 2018). Assume now that the features are conditionally aligned, pZ|Y =jS = p Z|Y =j T for all j ∈ Y (known as the label shift assumption), and the label space is discrete, Y = {1, · · · , k}. Then for each class i ∈ [k], we have the following relation between predicted target prior distribution, the source confusion matrix, and w∗: pT (Ŷ = i) = k∑
j=1
pT (Ŷ = i|Y = j)pT (Y = j) = k∑
j=1
pS(Ŷ = i|Y = j)pT (Y = j) = k∑
j=1
pS(Ŷ = i, Y = j)w ∗ j ,
(2)
where Ŷ := h(g(X)) are label predictions. Since we have the same number of equations as unknowns, w∗ could be recovered under mild assumptions (Lipton et al., 2018).
Careful readers may have noticed that achieving either goal above requires having the other as an assumption. Indeed, attempting to estimate IW without conditional alignment could fail, as well as alignment without accurate IWs. The hope is that the optimization will be predominantly guided by the implicit bias of the pre-trained multilingual LM for selecting largely invariant features (hence conditionally aligned), so that small improvements to feature alignment and updates to prior shift estimates could bring incremental improvements to both components.
Optimization. The two goals above define an optimization problem over g, h and w,
min g,h
( E(x,y)∼pS [`(h(g(x)), y)] + λD(p w,Z S , p Z T ) ) , min
w ∑ ŷ∈[k] ( pT (ŷ)− ∑ y pS(ŷ, y)wy )2 ,
where alignment objective minimizes source task loss and the discrepancy of w-importanceweighted feature distributions (the metric should be more fine-grained than the first-moment metric used in our analysis), and the IW estimation objective solves eq. 2 using current statistics based on h, g with the constraints w ≥ 0 and w>pYS = 1. In our implementation, we use the minimax formulation of the alignment objective optimized with adversarial training, where D is replaced by a shallow ReLU network Wasserstein-1 critic and zero-centered gradient penalty (Arjovsky et al., 2017; Thanh-Tung et al., 2019). We found the Wasserstein-1 adversarial loss to be more responsive to importance weighting than cross-entropy loss (Ganin et al., 2016).
Improving Stability of IW Estimates. In Tachet des Combes et al. (2020), the IWs estimation objective is solved by quadratic programming (QP), and updated once every epoch of training in their implementation. Since accurate IWs estimates are integral to successful adaptation, we (1) update w using projected gradient descent at every training step for maintaining their optimality. This, however, led to the discovery of previously unexplored practical issues with the IW estimator. In our preliminary experiments, we observed that w tends overestimate the majority class on the target domain, and diverges during longer training episodes.
These issues are unique to unsupervised adaptation settings, and are due to inefficiencies of the critic and noises introduced by the alignment objective. They could also be reproduced with the previous QP procedure. We address these issues by (2) `2-regularizing w (Azizzadenesheli et al., 2018), and (3) using a decaying learning rate schedule for w. These modifications are necessary for achieving best performance, as long training episodes are required for fine-tuning on many NLP tasks, and also needed to maintain the critic near optimality for guiding properly the alignment.

heading: 4 EXPERIMENTS
text: In our experiments, we compare importance-weighted domain alignment (IWDA) to two baseline methods for cross-lingual transfer on three NLP tasks, under the unsupervised setting where access to unlabeled task data from the target language is provided (UCL setting). Besides supporting our findings and our motivated algorithm, these experiments are aimed at investigating the limitations of existing approaches under distributional shifts in prior distributions between source and target. We also illustrate the potential benefits of IWDA in light of stronger cross-lingual learning methods.
Datasets. The tasks considered are named-entity recognition (NER), sentiment analysis, and textual entailment. NER: Two benchmark multilingual datasets are used, CoNLL (Tjong Kim Sang, 2002; Tjong Kim Sang & De Meulder, 2003) and WikiANN (Pan et al., 2017; Rahimi et al., 2019). The former includes English, German, Spanish and Dutch, and the latter covers over 200 languages, from which we select a subset of 40 according to the XTREME benchmark (Hu et al., 2020). Sentiment analysis: Multilingual Amazon Reviews Corpus (MARC) is used (Keung et al., 2020), a balanced dataset with product reviews in English, German, Spanish, French, Japanese and Chinese. Textual entailment: The XNLI dataset is used (Conneau et al., 2018), but due to space constraints, its setup and results are relegated to Appendix B. As in most work on cross-lingual transfer, English is used as the source language for transfer in our evaluations.
Baselines. We perform extensive comparative experiments with two semi-supervised learning (SSL) techniques. Both require having a model fine-tuned on source data only. Knowledge distillation (KD): It distills and transfers knowledge of the source teacher model to the target domain by training a newly initialized student network to mimic teacher outputs on target data (Hinton et al., 2015). A variant of KD was proposed for NER by Wu et al. (2020), which also freezes the bottom three layers of mBERT during training (Wu & Dredze, 2019). Self-training (ST): It iteratively assigns pseudo-labels to unlabeled target data and train on the most confident ones. Our implementation follows closely the work by Dong & de Melo (2019), which considered sequence classification.5
For IWDA experiments, the model is first fine-tuned on source only, then continue fine-tuned with the IWDA objective on both source and unlabeled target data. We also include results for when the true IWs are known and hard-coded as an upper bound on IWDA performance (IWDA (oracle)).
Experiments are performed on BERT-Base Multilingual Cased and XLM-R Large models. More details on IWDA and our baselines are included in Appendices C and D along with hyperparameter settings. Unless otherwise noted, results are from our implementation.
5These baselines are chosen because they require no more resources than the unlabeled target data provided, are not tailored to any particular model, and can be easily implemented on any existing models, which are qualities shared by IWDA. As many elaborate methods are also based on the same fundamental ideas as these baselines, we hope that they are representative of approaches based on SSL, if not exhaustive.

heading: 4.1 RESULTS ON STANDARD BENCHMARKS
text: To begin, through evaluations on standard benchmarks, we demonstrate that IWDA brings performance gains via improving representations to be better-aligned across languages. Results on CoNLL and low-resource WikiANN NER are presented in Tables 1 and 2, and those on MARC in Table 3. Remaining results on WikiANN with mBERT are included in Table 4 in the appendix.
In all mBERT IWDA experiments, improvements upon zero-shot baseline are observed, and are accompanied by improved representation alignment. The average decrease in conditional feature shift is 21.60% and 22.33% on CoNLL with IWDA and IWDA (oracle) respectively (with 100% indicating perfect alignment), 51.53% and 64.77% on MARC, and 32.91% and 36.34% on WikiANN.
However, improvements sometimes fall short of SSL baselines, which we suspect is due to the noise introduced during the alignment process. Potential sources include fluctuations in IW estimates and insufficiently trained critics. By training IWDA with more passes on the data and improving the optimality of the critic, further improvements on CoNLL NER are observed (Table 6).
Another way to reduce noise is to perform a final-stage SSL training on the IWDA-aligned model. These experiments are labeled IWDA + KD and IWDA + ST, on which we observe some best results with mBERT on both tasks. This is an appealing strategy, because SSL algorithms theoretically depend on the assumption that the source and target domains are the same, IWDA training can be viewed as a preprocessing step on the feature representations to satisfy this assumption. We will revisit this point in the next section where we consider cross-lingual transfer under prior shifts.
On XLM-R models, IWDA also brings improvements on NER. However on MARC, gains are only observed for languages distant from English. Here, we remark that the performance gain from IWDA (if any) will come from enhancements to the alignment of last-layer features in their current configurations, but not from learning entirely different representations with fundamentally better class-conditional alignment. This cannot be achieved absent target labels, additional supervision or resources. If the representations learned by the powerful XLM-R model during zero-shot training are already efficient, then the potentials of IWDA will be limited. Opportunities for further improvement in this case could come from additional resources, e.g. Bari et al. (2021) perform unsupervised data augmentation by leveraging the multilingual mask-filling ability of XLM-R.

heading: 4.2 RESULTS UNDER CLASS PRIOR SHIFTS
text: In this section, we evaluate on datasets with large prior shifts, a likely property of real-world data that is insufficiently investigated for UCL, and contrast the brittleness of SSL baselines with the robustness of IWDA. Because of the lack of multilingual benchmarks exhibiting prior shifts, similar to Section 2.2 , we evaluate on smaller datasets subsampled from the German portion of CoNLL and the Japanese portion of MARC according to varying amounts of shifts.
Results plotting percent improvements over zero-shot baselines against the amount of prior shift are presented in Fig. 4. Although SSL baselines are generally better than IWDA under mild to no shifts,
their performances quickly deteriorate as prior shift increases, until eventually resulting in negative transfer. IWDA on the other hand does not cause negative transfer from prior shifts owing to its estimation and correction capabilities. The relative success of IWDA demonstrates that it is possible to estimate prior shift from unlabeled target data. We include XLM-R MARC results in Fig. 10.
In mBERT experiments, the average percent correction of prior shift, computed from the decrease in TV between true target domain priors and end-of-training estimates, normalized by the amounts of shift, is 87.33% on CoNLL (with 100% indicating perfect IW estimation), and 28.08% on MARC under shifts larger than 0.8 in TV. On MARC with smaller shifts, we observe estimates that are worse than assuming a uniform prior distribution (-335.80% in correction). This is because these are harder problems with noisy labels (as it is naturally difficult to distinguish a 2-star review from a 3-star one), the suboptimality of the source model from which IWDA is trained means that the initial representations may be conditionally misaligned, thereby affecting the accuracy of the initial as well as subsequent IW estimates.
Since IWDA brings improvements to class-conditional alignment, as already pointed out in the previous section, it is appealing to complement SSL and potentially more elaborated methods with IWDA to improve robustness against prior shift. As demonstrated in our IWDA + KD and ST experiments, SSL no longer suffer from negative transfer when preceded by IWDA, and altogether provide further performance gains.

heading: 5 CONCLUSION AND FUTURE WORK
text: From the observations on multilingual neural LMs that cross-lingual transfer performance is strongly correlated with representation invariance and negatively affected by class prior shift, we proposed importance-weighted domain alignment (IWDA) for unsupervised cross-lingual learning. Experiment results support our findings, and show that IWDA is effective at the estimation and correction of class prior shift from unlabeled data, and robust to large shifts, under which in contrast existing UCL approaches based on SSL could result in negative transfer. As prior shift is a prevalent issue under unsupervised settings, more care to it is necessary in the design of future UCL algorithms for them to achieve good performance not only on curated benchmarks, but also on real-world data.
While our present implementation of IW estimation with the moment-matching approach is largely effective, future work on the search and study of potentially more stable estimation methods would enable IWDA to achieve better and more consistent transfer performance. Since the strength of IWDA lies in its ability to detect prior shifts and improve on class-conditional representation alignment, with our experiments on combining IWDA and SSL baselines as a preliminary demonstration of its potential usefulness as a feature preprocessing step and in improving robustness to prior shifts, it would be of interest to investigate its benefits on stronger models and methods towards better cross-lingual transfer.

heading: A ADDITIONAL ANALYSES AND FINDINGS FROM SECTION 2
text: In this section, we provide additional results to support our findings by considering more languages (WikiANN NER), a higher-level task (XNLI), and a more powerful model (XLM-R Large).
First and foremost, to demonstrate that the correlation between class-conditional representation alignment and performance is not confounded by the clustering structure induced by the linear classifier trained on last-layer features, we plot and observe similar correlation between transfer performance and class-balanced representation alignment in Fig. 5.
WikiANN. As an attempt to minimize the effects from potential confounders on the empirical findings in the main sections, we perform the same experiments for studying the relation between transfer performance, representation alignment and prior shift by zero-shot transferring mBERT from English to 39 target languages included in the WikiANN NER dataset. The results are provided in Fig. 6. These languages have distinct linguistic properties and varying degrees of resourcefulness. A summary of them can be found in Table 5 of Hu et al. (2020), and sizes of their pretraining corpora in Table 1 of Wu & Dredze (2020).
It is observed that the same correlation between representation alignment and performance still holds, but the effects of prior shift are mostly mild. We suspect that this is because the task we evaluated on, named-entity recognition, is a structured prediction problem, hence mBERT’s ability to leverage dependencies between words made it robust to prior shifts on these tasks. We later observed that if the source dataset on which the model is trained on has a skewed prior distribution, then the effects of prior shift will be pronounced. The mBERT WikiANN results under this setting are shown in Fig. 9b, and more discussions are coming up in Section A.1.
XNLI. So far, we have seen empirical evidence on sentiment analysis and NER tasks, and one may wonder if the same conclusions still hold when considering cross-lingual transfer to high-level semantic tasks. For thoroughness, we include a plot in Fig. 7 comparing zero-shot transfer performance to representation alignment when mBERT is trained on a textual entailment task, from English to 14 languages included in the XNLI dataset. Here, the same observations are made, demonstrating that the findings hold across NLP tasks.
XLM-R. Another variable in our empirical analyses is the model used for evaluation. While the lightness of mBERT afforded us to perform extensive experiments, it would be of practical interest to see whether our findings also hold on larger and more powerful models. In Fig. 8, we compared performance to alignment and prior shift on XLM-R Large model on MARC. Again, the observations support our findings, and show that they also hold across models.
A.1 EFFECTS FROM SKEWED SOURCE PRIOR DISTRIBUTIONS
As briefly mentioned above, we have only considered the cross-lingual transfer from relatively balanced source datasets in previous sections. Here, we show that when the model is fine-tuned on source datasets with skewed prior distributions, the effects of class prior shift are more pronounced.
We first experiment on MARC. We generate an skewed English source dataset via subsampling according to the prior distribution of 14.86%, 6.43%, 8.03%, 19.28% and 51.41%6, corresponding to the proportion of 1- through 5-star reviews. Then, we repeat the previous empirical analysis by evaluating an mBERT fine-tuned on this skewed English dataset, and the results are shown in Fig. 9a. Compared to when the source is class-balanced (Fig. 3), here the correlation between performance and class prior shift is much stronger.
On WikiANN, although the original English portion of the dataset is already considerably skewed (with class O makes up 50.75% of the labels), the correlation between prior shift and model performance is weak (Fig. 6b), likely due to NER being a structured prediction problem where the dependencies between words in the same sentence helped the model to be more robust. Repeating the same experiment above with the skewed English source dataset generated by keeping only sentences containing tokens with an O label (so that now 80.92% of the tokens belong to class O), we again observe pronounced effects of prior shift on performance (results in Fig. 9b).
6This is the actual distribution of product ratings on Amazon.com.

heading: B ADDITIONAL EXPERIMENT RESULTS FROM SECTION 4
text: By comparing the transfer performance of IWDA to SSL on standard benchmarks as well as datasets exhibiting prior shifts in the main sections, we demonstrated that better performance can be achieved with better representation alignment under the UCL setting, as well as the limitations with existing approaches under prior shifts. While the prior shifts experiments in Section 4.2 are performed on mBERT, in Fig. 10, we provide XLM-R Large results on MARC Japanese prior shift experiments.
Furthermore, we provide full mBERT WikiANN results of applying IWDA (Table 4), and results on the XNLI dataset, a high-level semantic task of textual entailment.
XNLI. Because XNLI does not provide human-generated training data for languages other than English, to obtain unlabeled target language task data for our studies on the UCL setting, we use the machine-translation of the English training set provided by Conneau et al. (2020a). To remove potential effects due to parallelism, similar to the setup considered in Bari et al. (2021), we partition the dataset into equal portions of size 20,000 (5% of the original size) and randomly select disjoint partitions to use as source and target data. In addition, since the original XNLI test set is humantranslated but our target data is machine-translated, to avoid the effects from this discrepancy (or domain shift) between train and test, evaluations are performed on the machine-translated portion of the dataset in our experiments (we refer interested readers to Artetxe et al. (2020a) for a discussion on XNLI translation artifacts).
Due to our setup, our results are non-benchmark and cannot be compared to models evaluated on the original XNLI test set. However, the goal of this experiment is to study whether our principle of achieving domain alignment is beneficial on high-level tasks.
The results are included in Table 5. It is observed that IWDA generally improves transfer performance upon the zero-shot baseline, although the gains are not significant. As discussed in Section 4.1, the potentials of IWDA will be limited if the zero-shot representations are already largely aligned, which is reflected by the small performance gap between zero-shot and supervised results in our setup.
IWDA vs. DA. Besides demonstrating that SSL suffers from negative transfer under large prior shifts in our experiments in Section 4.2, we also discussed in Section 2.2 that domain alignment without prior shift correction leads to negative transfer.
In Fig. 11, we compare the performance on benchmark CoNLL German, which exhibits a prior shift from CoNLL English, from applying IWDA and domain alignment without prior shift correction (DA). CDAN refers to the conditioning technique by Long et al. (2018), which we use in our IWDA implementation by default. It is observed that without prior shift correction, as training progresses,
transfer performance of vanilla DA deteriorates as the model makes more and more erroneous feature alignments caused by the shift.
Better Training of Adversary. As briefly mentioned in Section 4, while the IWDA models for the experiments in the main sections are trained for four epochs with simultaneous adversary and model updates for fair comparisons with SSL techniques, better performance could be obtained from IWDA if the optimality of the critic (adversary) can be better maintained throughout training with more passes over the data and alternating updates.
In Table 6, we observe that IWDA delivers much-improved performance on CoNLL when we perform four adversary updates per model update and allow 40 passes over the dataset (we also set lambda iw to 1). Although this setting would lengthen the training time, this could be of practical interest for getting the best IWDA performance.
C IMPLEMENTATION DETAILS OF IWDA
A high-level description of IWDA was given in Section 3 with discussions that connect it to the motivating findings in Section 2. This section provides more details on the implementation of the algorithm along with the hyperparameter settings7 used in our experiments (Table 7).
First, recall the joint objective, reproduced below, which consists of an alignment objective and an importance weight (IW) estimation objective,
min g,h
( E(x,y)∼pS [`(h(g(x)), y)] + λD(p w,Z S , p Z T ) ) , min
w ∑ ŷ∈[k] ( pT (ŷ)− ∑ y pS(ŷ, y)wy )2 . (3)
Note that the alignment objective contains the discrepancy between (important-weighted) source and target feature distributions and source task loss, added to prevent g from degenerating to a trivial representation with perfect source and target alignment, e.g. constant maps x 7→ c.
Alignment Objective. We reformulate the alignment objective in eq. 3 as a minimax objective, and replace explicit evaluation of the discrepancy D with a critic function f (also known as discriminator or adversary; Ganin et al. 2016; Goodfellow et al. 2014). This turns the alignment objective into
min g,h
( E(x,y)∼pS [`(h(g(x)), y)] + λmax
f∈F ( Ex∼pw,XS [`ad(f(g(x)), 0)]− Ex∼pXT [`ad(f(g(x)), 1)]︸ ︷︷ ︸
(4)
)) ,
where `ad is a specific choice of adversarial loss accompanied by an adversary function class F , that takes as input the scores computed by the adversary and the domain labels (here, source domain is labeled 0). The role of the adversary is to distinguish features computed from inputs sampled on the source domain from those on the target domain, and output a score on each feature that reflects its domain identity. When the features are aligned, the adversary would not be able to distinguish source and target features.
In our implementation, we use Wasserstein-1 loss for `ad, whose corresponding adversary function classF is 1-Lipschitz functions (Arjovsky et al., 2017). It is shown that when f attains its maximum, the term (4) exactly computes the Wasserstein-1 distance between the importance-weighted feature
7We use the same settings on all IWDA experiments. For methodological discussions on hyperparameter tuning for unsupervised cross-lingual learning, see Artetxe et al. (2020c).
distributions, pw,ZS and p Z T . We parameterize F as a shallow ReLU network of width 2048, and enforce the Lipschitz condition using soft constraints in the form of zero-centered gradient penalty, which penalizes the gradient of f between pairs of source and target features (Thanh-Tung et al., 2019).
Last but not least, for added stability, we condition the alignment of features on their (pseudo-)classes as proposed by Long et al. (2018) (CDAN). This is implemented as follows. Instead of feeding the adversary directly with features output by g, it receives the outer product between features and classifier output distributions, g(x)⊗h(g(x)). This means that the input dimension of f is increased to the dimension of features multiplied by the number of output classes k. When k is large, Long et al. (2018) used random projections to reduce dimensionality, which we did not explore.
Putting everything together—minimax reformulation, Wasserstein-1 loss, zero-centered gradient penalty, and conditional alignment—upon receiving a mini-batch of source labeled examples, (xi, yi) n i=1, and target unlabeled examples, (x ′ j) m j=1, the empirical alignment loss is
1
n n∑ i=1 `(h(g(xi)), yi) + λ
( 1
n n∑ i=1 f(g(xi)⊗ h(g(xi)))− 1 m m∑ j=1 f(g(x′j)⊗ h(g(x′i))) ) − µ ñ ñ∑ i=1 ‖(∇f)(z̃i)‖2,
where ñ = min(n,m) and z̃i is a random point on the line connecting g(xi) and g(x′i). Here, we have two hyperparameters: λ (lambda da) controls the strength of alignment, and µ (lambda gp) controls the strength on enforcing the Lipschitz condition. The learning rate of the adversary (lr ad) is set to be larger than that of the model (lr; Heusel et al. 2017). AdamW optimizer is used throughout our experiments.
On this loss, we back-propagate gradients to f on maximizing the second and third terms, g on minimizing the first two terms, and h on minimizing the first term only. In our experiments in the main sections, all three models are optimized with simultaneous updates, but we note that alternating updates (standard in GAN literature) could allow IWDA to deliver further performance gains with longer training time. In Table 6, we observe improved mBERT CoNLL results when the training alternates between four updates to f and one update to g for 40 epochs.
IW Estimation Objective. We proposed several modifications to the IW estimation objective in Section 3 to improve its stability, and one of them is adding `2-regularization to w (Azizzadenesheli et al., 2018).
Given first moment statistics in the form of the source confusion matrix, Cij := pS(Ŷ = i, Y = j), and target output distribution, νi := pT (Ŷ = i), the regularized IWs objective minimizes
k∑ i=1 ( νi − k∑ j=1 Cijwj )2 + ξ k∑ i=1 (wi − w(0))2, (4)
where w(0) is an initial IW estimate (set to all-ones vector by default, i.e. no prior shift). Here, one hyperparameter that controls the strength of the regularization is introduced, ξ (lambda iw). For simplicity, the `2-regularization is implemented with and documented as weight decay. Hence, ξ equals to lambda iw multiplied by 2 times the learning rate of w.
Instead of initializing w(0) = 1, because IWDA fine-tuning is preceded by zero-shot fine-tuning in our experiments and its decent performance reflects partial class-conditional feature alignment, we initialize it to a crude IW estimate w′ obtained from the zero-shot model using the QP by Tachet des Combes et al. (2020). Since the initial alignment will not be perfect and w′ not entirely reliable, we set w(0) = (1− ρ)w′ + ρ1, where ρ (lambda iw init) is a hyperparameter. The statistics C, ν evolve during training as the feature representation computed by g changes, so it needs to be updated throughout the learning process. Re-evaluating them at each optimization step
on a held-out set would make training prohibitively slow, hence we update them with the statistics Ĉ, ν̂ computed on training mini-batches: C(t) = (1 − η)C(t−1) + ηĈ, and similarly for ν(t). This update rate η (lr iw stats) is not the same as IW learning rate (lr iw).
Lastly, to minimize potential effects from the miscalibration of neural models (Guo et al., 2017; Alexandari et al., 2020), hard confusion matrix and target output distribution are used in the estimation of IWs (Lipton et al., 2018; Garg et al., 2020).

heading: D DESCRIPTION OF SSL BASELINES
text: Knowledge Distillation (KD). Our implementation follows the original paper by Hinton et al. (2015) closely (from which the Wu et al. (2020) variant is moderately different). Given a teacher LM fine-tuned either under zero-shot or using IWDA, we distill its knowledge on a newly initialized pre-trained student LM by minimizing the mean-squared-error difference of the teacher and student Softmax outputs on source and target data without labels. If labeled data from the source is available, we also train the student to minimize source task loss.
We set the temperature to 3, and the weight of the source task loss to 0.1. Learning rate is 1e-4, batch size is 32, and we train for 4 epochs.
In addition, Wu et al. (2020) observed that when distilling to pretrained LMs, better performance can be achieved if the bottom three layers of the student is frozen (motivated by findings from Wu & Dredze (2019)). We also apply this technique in our KD experiments.
Self-Training (ST). Our implementation follows Dong & de Melo (2019) closely. Given an LM fine-tuned on source domain only, ST continues fine-tuning the model by first assigning pseudolabels to target data, collecting the ones with the highest confidence, and then train on them with the pseudo-labels. Source task loss is also minimized during self-training.
The hyperparameters of ST involve selecting how any pseudo-labeled examples to train on. On MARC with 200,000 training examples, at each epoch, we select top 4,000 most confident pseudolabeled examples from the target domain conditioned on each (pseudo-)class, so that 20,000 examples are selected each round (MARC has five classes). These numbers are scales proportionally on datasets of other sizes. We perform self-training for 4 epochs, and set learning rate to 1e-5 and batch size to 128.
On IWDA + ST prior shift experiments, we select much less examples for self-training to avoid training on low-confidence examples from minority classes. This adjustment of hyperparameters relies on knowledge about the prior distribution on the target domain, which is approximated by the IW estimates from IWDA.

Review summaries:
This paper investigates a method for improving the cross-lingual transfer of pretrained multilingual models. The paper first empirically analyzed the influence of representation invariance and distributional class shift. Then, the paper proposed a method to improve the representation invariance and correcting the class shift. Experiments showed its superiority under large prior shifts.

In this paper, the authors provide substantial analyses on the cross-lingual transfer performance in the multilingual neural language models and reported that the performance is strongly correlated with representation invariance and negatively affected by distributional shift in class priors between data in the src/tgt languages. Based on these findings, the authors propose an unsupervised cross-lingual learning method, called importance-weighted domain adaptation (IWDA), where it performs feature alignment, prior shift estimation, and correction. The authors experimented on two different NLP tasks such as multilingual NER and multilingual sentiment analysis tasks, and experimentally showed the effectiveness. Besides that, they demonstrated that the proposed approach improves performance further, when combined with existing semi-supervised learning approaches.

This paper looks at the problem of unsupervised cross-lingual transfer (termed UCL) in the paper through the optics of domain adaptation. After empirically analysing and validating that distributional shifts in class priors might cause a huge problem for UCL (which wasn't tackled in previous research), the authors proceed with an introduction of a new method that aims to mitigate that problem. The idea is to get rid of that shift through a approach called importance-weighted domain adaptation (IWDA), which is largely the adaptation of the work from Tachet des Combes et al. (NeurIPS 2020) to the UCL problem.  The results on two tasks in the UCL setup (NER and MARC classification) show slight gains over the standard zero-shot transfer when IWDA is applied, with more prominent gains reported when a stronger domain shift is observed - however, such a setup has been created mostly artificially, to further demonstrate the benefits of modelling the shift in the model.

The paper first demonstrates the importance of feature invariance (language invariant representations) and class-prior invariance across languages on zero-shot cross-lingual performance. By analyzing the zero-shot performance on different languages on the MARC reviews and WikiANN NER tasks, and comparing against the class conditional distance between the source language (English) and target language feature representations, the authors illustrate the how high feature invariance results in better zero-shot performance. By also synthetically modifying the class prior for the target language, the authors demonstrate how increasing differences in class priors result in decreasing zero-shot cross-lingual transfer performance.  Building on their observations, the authors propose an approach that: (i) Introduces an adversarial loss term to penalize distortion in average class conditional feature representations between the source and target languages. (ii) Adds an importance weighting term to ensure the approach doesn't fail under class prior shifts.  Empirical studies demonstrate that the proposed approach improves significantly over the vanilla zero-shot model on both MARC sentiment analysis and NER tasks, also improving over self-training on sentiment analysis. Comparisons on synthetic datasets (sub-sampled from NER and MARC datasets) that enhance the class prior shift highlight the robustness of the approach under large class prior shifts where previous approaches fail.