Enhancing Cross-lingual Transfer by Manifold Mixup
Authors: Huiyun Yang
Huadong Chen
Hao Zhou
Lei Li
Venue: ICLR 2022 Poster (2022)
Keywords: cross-lingual transfer
cross-lingual understanding
manifold mixup

Abstract: Based on large-scale pre-trained multilingual representations, recent cross-lingual transfer methods have achieved impressive transfer performances. However, the performance of target languages still lags far behind the source language. In this paper, our analyses indicate such a performance gap is strongly associated with the cross-lingual representation discrepancy. To achieve better cross-lingual transfer performance, we propose the cross-lingual manifold mixup (X-Mixup) method, which adaptively calibrates the representation discrepancy and gives a compromised representation for target languages. Experiments on the XTREME benchmark show X-Mixup achieves 1.8% performance gains on multiple text understanding tasks, compared with strong baselines, and significantly reduces the cross-lingual representation discrepancy.

Published as a conference paper at ICLR 2022
ENHANCING CROSS-LINGUAL TRANSFER BY MANI- FOLD MIXUP
Huiyun Yang', Huadong Chen', Hao Zhou"l Lei Li
'ByteDance AI Lab
University of California, Santa Barbara (yanghuiyun. 11, chenhuadong howard zhouhao. nip]@bytedance. com. leilif cs.
wwl.liss.usb......... ucsu. edu
ABSTRACT
Based on large-scale pre-trained multilingual representations, recent cross-lingual transfer methods have achieved impressive transfer performances. However, the performance of target languages still lags far behind the source language. In this paper, our analyses indicate such a performance gap is strongly associated with the cross-lingual representation discrepancy. To achieve better cross-lingud transfer performance, we propose the cross -inguul manifold mixup (X-MiXXU M-IXUP) method, which adaptively calibrates the representation discrepancy and gives compromised representations for target languages. Experiments on the XTREME benchmark show X-MIXUP achieves 1.8% performance gains on multiple understanding tasks, compared with strong baselines, and reduces the cross-lingual representation discrepancy significantly]
INTRODUCTION
Many natural language processing tasks have shown exciting progress utilizing deep neural models. However, these deep models often heavily rely on sufficient annotation data, which is not the case in the multilingual setting. The fact is that most of the annotation data are collected for popular languages like English and Spanish (Ponti et al.) 2019 Joshi et al.) 2020), while many long-tail languages could hardly obtain enough annotations for supervised training. As result, cross-lingual transfer learning (Prettenhofer Stein 201T; Wan et al. 201 Ruder et al.] 2019) is crucial, transferring knowledge from the annotation-nie source language to low-resource zero-resource target languages. In this paper, we focus on the zero-resource setting, where labeled data are only available in the source language.
Recently, multilingual pre-trained models (Conneau & Lample, 2019
Conneau et al, 2020a Xue
al. 2020) offer effective way for cross-lingual transfer, which yield a universal embedding space across various languages. Such universal representations make it possible to transfer knowledge from the source language to target languages through the embedding space, significantly improving
the transfer learning performance (Chen et al. 2019 Zhou et al. 2019 Keung et al. 2019 Fang al. 2020). Besides, Conneau al. (2018) proposes translate- train, a simple yet effective cross- lingual data augmentation method, which constructs pseudo-training data for each target language via machine translation. Although these works have achieved impressive improvements
cross- lingull transfer (Hu et al.] 2020; Ruder et al.) (02T). significant performance gaps between the source language and target languages still remain (see Table []. Hu et al. (2020) refers to the gap as the cross-lingual transfer gap, a difference between the performance on the source and target languages.
To investigate how the cross-lingual transfer gap emerges, we perform relevant analyses, demonstrat- ing that transfer performance correlates well with the cross-lingual representation discrepancy (see Section for details). Here the cross-lingual representation discrepancy means the degree of differ- ence between the source and target language representations in the universal embedding space. As shown in Figure ((a) in translate-train the representation distribution of Spanish almost overlaps
"Correspondigg author.
Code is available at https://///g// wwt.lca..littpae////hhh....oooyp

---

Published as a conference paper at ICLR 2022
15
Lang
Lang
10.
5-
SW
-10
-10-
-15
-15-
-20
-10
100
20
-20
-10
10
20
(a) Translate-train
(b) X-MIXUP
Figure 1: Representation visualization of four languages: English (en), Spanish (es), Arabic (ar) and Swahili (sw) based on XLM-R. We plot the sentence representation of the XNLI test set, which is parallel across 15 languages. We average hidden states of the last layer to get sentence representations and implement the dimensionality reduction by PCA. Obviously, the cross-lingual representation discrepancies large translate-train, but X-MIXUP reduces the discrepancy significantly.
with English, while Arabic shows a certain representation discrepancy compared with English and Swahili performs larger discrepancy, where translate-train achieves 88. accuracy on English, 85.7 7 on Spanish, 82.2 on Arabic and 77.0 on Swahili. Intuitively, a larger representation discrepancy could lead to worse cross-lingual transfer performance.
In this paper, we propose the Cross-Lingudl Manifold Mixup (X-MIXUP) approach to fill the cross- lingual transfer gap. Based on our analyses, reducing the cross-lingual representation discrepancy is promising way to narrow the transfer gap. Given the cross-lingual representation discrepancy is hard to remove, X-MIXUP directly faces the issue and explicitly accommodates the representation discrep- ancy the neural networks, by mixing the representation of the source and target languages during training and inference. With X-MIXUP, the model itself can Iearn how to escape the discrepancy, which adaptively calibrates the representation discrepancy and gives compromised representations for target languages to achieve better cross-lingual transfer performance. X-MIXUP is motivated by robust deep learning (Vincent et al. 2008), while X-MIXUP adopts the mixup (Zhang et al. 2018 idea to handle the cross-lingual discrepancy.
Specifically, X- MIXUP designed upon the translate-train approach, faced with the exposure bias (Ranzato al. 2016) problem and data noise problem. During training, the source sequence is real sentence and the target sequence is translated one, while situations are opposite during inference. Besides, the translated text often introduces some noises due to imperfect machine translation systems. To address them, we further impose the Scheduled Sampling (Bengio et al. 2015) and Mixup Ratio in X-MIXUP handle the distribution shift problem and data noise problem, respectively.
We verify X-MIXUP on the cross-ling linsull understanding benchmark XTREME (Hu et al. (020). which includes several understanding tasks and covers languages from diverse language families. Experimental results show X-MIXUP achieves 1.8% performance gains across different tasks and lan- guages, comparing with strong baselines. also reduces the cross-lingual representation discrepancy significantly, as Figure 1(b) shows.
2 RELATED WORK
Multilingual Representation Learning Recent studies have demonstrated the superiority of large-
scale pre-trained multilingual representations on downstream tasks. Multilingual BERT (mBERT; Devlin et al. 2019) is the first work to extend the monolingual pre-training to the multilingual setting.
Then, several extensions achieve better gross-lingual performances by introducing more monolingual
parallel data and new pre-training tasks, such Unicoder (Huang al.) 2019), ), XLM-R (Conneau
et al. 2020a). ALM Yangg et al. 2020), MMTE (Siddhant et al. 2020), InfoXLM (Chi et al. (020).
HICTL (Wei al. 2020), ERNIE-M (Ouyang et al. 2020). (Xue et al. 2020), nm'5 (Kale
2

---

Published as a conference paper at ICLR 2022
Table 1: Cross-lingual transfer performances of POS and NER tasks languages with different data resources or different language families, where there are only labeled training data in English. The data resource refers the resource of each language utilized in the pre- pre-training process. For the language family, English belongs to the Germanic languages, so we divide languages into two types: Germanic one and others. Results show high-resource languages outperform low-resource ones significantly and languages dissimilar to the source language tend to perform worse
Language Resource
Language Family
Language Type
Source
High-resource
Low-resource
Germanic
Other
Language
en
es
1I
Kk
m1r
a1
de
nl
ar
hi
ja
MBERT
POS
95.5
86.9
88.4
86.2
60.7
70.3
69.4
86.6
85.2
88.6
36.2
67.2
49.2
NER
85.2
77.4
81.5
80.8
66.3
45.8
58.2
77.4
78.0
81.8
41.1
65.0
29.0
POS
96.1
88.3
89.4
87.6
72.5
78.1
80.8
89.8
88.5
89.5
67.5
76.4
15.9
XIMM-R
NER
84.7
79.6
81.3
81.9
60.9
56.2
68.1
78.9
78.8
84.0
53.0
73.0
23.2
al. 2021p. AMBER et al. 2021) and VECO (Luo al. 2021).
They have been the standard
bones of current cross-lingual transfer methods.
Cross-lingual Transfer Learning Cross-lingual transfer learning (Prettenhofer & Stein 2011
Wan et al. 2011 Ruder et al.] 2019) aims to transfer knowledge learned from source languages to the type of transfer learning (Pan & Yang 2010), previous cross- lingual transfer methods can be divided into three categories: instance transfer, parameter transfer, and feature transfer. The cross-lingus monsu-lingual transferability improves a lot when engaged with the instance transfer by translation (i.e. translate- train, translate-test) or other cross-lingans data augmentation
target languages. According
methods (Singh et al.] 2019 Bornea et al., 2020; Qin et al.] 2020: Zheng et al. 2021). Chen et al. (2099) and /Zhou et al. (2199) focus on the parameter transfer to learn a share-private model architecture. Besides, other works implement the feature transfer to learn the language-invariant features by adversarial networks (Keung et al., 2019: Chen et al. 2019) re-alignment (Libovicky ctt al. 2020) Zhao et al.) (020). X-MIXUP utilizes both the instance transfer and feature transfer, which is based the translate-train data augmentation approach and implements the feature transfer by cross-lingual manifold mixup.
Mixup and Its Variants Mixup (Zhang et al. 2018) proposes to train models on the linear interpo- lation at both the input level and label level, which is effective to improve the model robustness and generalization. Generally, the interpolated pair is selected randomly. Manifold mixup (Verma et al. 2019) performs the interpolation in the latent space by conducting the linear combinations of hidden states. Previous mixup methods (Chen al. 2020 Jindal et al., 2020) focus on the monolingual setting. However, X-Mixup focuses on the cross- tross-lingual setting and faces many new challenges (see Section for details). Besides, in contrast to previous mixup methods, X-MIXUP mixes the parallel pairs, which share the same semantics across different languages. As a result, the choice of parallel pairs for interpolation can build a smart connection between the source and target languages.
ANALYSES OF THE CROSS-LINGUAL TRANSFER PERFORMANCE
In this sectior we concentrate on the cross-lingual transfer performance and find strongly associated with the cross-lingual representation discrepancy. Firstly, observe the cross-lingual transfer performance on different target languages and propose an assumption. Then we conduct qualitative and quantitative analyses to verify it.
Although previous studies (Hu et al. 2020 Ruder et al. 2021 have shown impressive improvements
lingull transfer, the cross-lingua transfer gap is still pretty large, more than 16 points in Hu et al. (2020). Furthermore, results in Table [ show the performance of low-resource languages and dissimilar languages fall far behind other languages in cross-lingual transfer tasks.
Compared with English, the representations of other languages, especially low-resource languages, not well-trained (Lauscher et al.) 2020 Wu & Dredze, 2020),, because high-resource languages dominate the representation learning process, which results in the cross-lingual representation
In our analyses, we take English as the source language, and the dissimilar language is the language which dissimilar Figlish.
3

---

Published as a conference paper at ICLR 2022
Table 2: Spearman's is rank correlation p between the CKA score and cross-lingual transfer performance on two XTREME tasks, where i denotes training on the source language, and denotes the translate- train approach. denotes the p-value is lower than 0.05. Results indicate the correlation solid.
Task
XNLI
XNLI
PAWS-Xi
PAWS-X
P
0.76
0.69
0.90
0.93*
discrepancy. Besides, dissimilar languages often show differences in language characteristics (like vocabulary, word order), which also leads to the representation diserepancy. As a result, we assume that the cross- lossslinguual transfer performance is closely related to the representation discrepancy between the source language and target languages.
Following Conneau et al. (2020b), we utilize the linear centered kernel alignment (CKA; Kornblith al. 2019) score indicate the cross-lingual representation discrepancy
| X
CKA(X,Y) = ||X X Y F
(1)
where and Y are parallel sequences from the source and target languages, respectively. A higher CKA score denotes a smaller cross-lingual representation discrepancy.
To verify our assumption, we perform qualitative and quantitative analyses on the relationship between the CKA score and cross-lingual transfer performance. Figure 3 Appendix inndiiattes higher CKA score tends to induce better cross-lingual transfer performance. We also calculate the Spearman's 's rank correlation between the CKA score and the transfer performance in Table 2 which shows a strong correlation between them. Both the trend and correlation score confirm the cross- lingual transfer performance is highly related to the cross-lingual representation discrepancy.
METHODOLOGY: X-MIXUP
Based on the aforementioned analyses, we believe that reducing the cross-lingual representation discrepancy is the key to filling the cross-lingual transfer gap. In this section, we propose X-MIXUP explicitly reduce the representation discrepancy by implementing the manifold mixup between the source language and target language. With X-MIXUP, the model can adaptively calibrate the representation discrepancy and give compromised representations for target languages. This section will first introduce the overall architecture of X-MIXUP and its details. After that, the training objectives and inference process will be shown.
4.1 OVERALL ARCHITECTURE
Figure 2 illustrates the overall architecture of X-Mixup. Sequences from the source and target languages are first encoded separately. Then within the encoder, X-MIXUP implements the manifold mixup between the paired sequences (original sequence and its translation) within specific layer, where Mixup Ratio controls the degree of mixup and Scheduled Sampling schedules the data sampling process during training.
Notations We use S' to denote the source language and T to denote the target language. h' denotes the hidden states of a sequence in layer L. DD denotes the real text data collection and D denotes the translation data collection. For downstream understanding tasks, there are annotation data in the source language Dimin (XTTrain VTrin)) and raw test data in the target language D (ZTE)). Through translate-train, we can get pscudo-training data in the target language DiFam (Arirain (iHtain) Similarly, through translate-test. we can get pseudo-test data in the source language DIST (R During training, the Scheduled Sampling process uses translation data DD (ATriin) from the source language. Note that we use translation data (LTTiin and X and translate-
train labels () from the official XTREME repository, which is in the same setting as basclines.
"hheee data acquired by forward translation (from S T) then backward translation (from T to S).
4

---

Published as a conference paper at ICLR 2022
task
Output layer
Add LN
Feed Forward
hg
X- X-Mixup
hrjs
lt
Lask
Target language
Output layer
Add & LN
consist
Multi-head Attention
Food Forward
n
ktv
Source language
Add Multi-head
hr
Share parameters
Add
Attention
Multi-head
k 0
Attention
K
Train: Xs E DDgtatn Uirraan
21prain
Inference: Xs E DeSse
D.fst
Figure 2: The model architecture X-MIXUP, where the cross-lingual manifold mixup process is in the green block. Note that the manifold mixup process is implemented only in certain layer (the same layer both sides), and in other layers the process is omitted.
Basic Model We use mBERT (Devlin et al. 2019) or XLM-R (Conneau et al. 2020a) as the backbone model. Within each layer, there feed-forward layer followed by the residual connection and layer norm. We use the same multi-head attention layer (see details Appendix [A.I] as BERT (Devlin et al.) 2019). where inputs are query, key, and value respectively. In layer I + 1. the hidden states of the source sequence Øs and target sequence or are acquired by the head attention
sub-layers: the multi-head attention layer and the
h+1 MultiHeaddhh', h/s h'),, h MultiHead h4y hly.
Manifold Mixup To reduce the cross-lingual representation discrepancy, a straightforward idea is to find compromised representations between the source and target languages. It's difficult to find such representations because of varying degrees of differences across languages, like vocabulary and word order. However, manifold mixup provides elegant way to get intermediate representations by conducting linear interpolation on hidden states.
To extract target-related information from the source hidden states, the target hidden states are used as the query, and the source hidden states are used as the key and value. This cross-attention process computed as
h MultiHead h h1s 1111)
which shares parameters with the multi-head attention. The manifold mixup process mixes the target hidden states h and the source-award target hidden states h based on the mixup ratio X
h4 LN(A
T1S + +1 --))4)))...
where À is an instance-level parameter, ranging from 0 to 1, and indicates the degree of manifold mixup. LN denotes the layer norm operation.
Mixup Ratio The machine translation process may change the original semantics and introduce data noises in varying degrees (Castilho al. 2017 Fomicheva et al. 2020). Thus we introduce the
In this section, the feed-forward layer is omitted for simplification.

---

Published as a conference paper at ICLR 2022
translation quality modeling in the mixup process to handle this problem. Following Fomicheva et al. (2020). we use the entropy of attention weights to measure the translation quality
hT, hig
H(A)
H(A) - Σ E A,,oogAjo where A. softmax
(5)
vn
I is the number of target tokens and J is the number of source tokens. Lower entropy implies better cross-lingual alignment and higher translation quality.
To introduce the translation quality modeling into the manifold mixup process, compute the mixup ))) + +bJ, where a is the sigmoid function, and W. b are trainable parameters. A is the max value of the mixup ratio, which is set to 0.5 in this paper. We consider two-way alignment in the translation quality modeling, i.c. H(A) and H(AT).
ratio as À = Ao 1h-of [o((( + (H(ATT)) ))
Scheduled Sampling The source sequences utilized in training and inference are drawn from different distributions. During training, the source sequence is a real text from Dirann while during inference, the source sequence is translation from Diest This discrepancy, commonly called the exposure bias, leads to a gap between training and inference.
Motivated by the scheduled sampling approach (Bengio et al. 2015) in NMT, sample the source sequence dynamically during training. Specifically, the source sequence fed into the manifold mixup either a real text from Irain or translation from Dirain with a certain probability p
<>=p", x, EDDS Train
p p> p", 2, D. Train
where p" decreasing during training to match the situation of inference. We utilize the inverse sigmoid decay (Bengio et al. 2015), which decreases p* as a function of the index of mini-batch.
4.2 FINAL TRAINING OBJECTIVE
The training loss is composed of two parts: the task loss and the consistency loss
L
Lask + MSE(rs, (L. Pr)
task loss
consistency
where MSE(-) is Mean Squared Error and (.) is Kullback- divergence, т, is the sequence representation and p. is the predicted probability distribution of downstream tasks.
The task loss Lussk the sum of source language task loss C and target language one L weighted by the hyper-parameter a, which is utilized to balance the training process
Cusk = a + (1 - a
(8)
For classification, structured prediction, and span extraction tasks, the task loss is the cross-entropy loss (see details in Appendix A.2). For structured prediction tasks, is non-trivial to implement the token-level label mapping across different languages. Thus use the label probability distribution, predicted by the source language task model, as the pseudo-label for training, where tokens and labels corresponding.
The consistency loss is composed of two parts: the representation consistency loss and the prediction consistency loss. The first loss is regularization term and provides way to align representations across different languages (Ruder et al. 2019). The second loss is to make better use of the supervision of downstream tasks. It only exists in the classification task, as the translation process does not change the label of this task, while in other tasks, it does.
4.3 INFERENCE
During inference, the manifold mixup process is the same as training, except for the Scheduled Sampling process. Concretely, for the source language, only translation data are available in the
WWe utilize the mean pooling of the last layer's hidden states as the sequence representation, which is independent of the sequence length.

---

Published as a conference paper at ICLR 2022
Table 3: Main results on the XTREME benchmark. denotes using other data augmentation strategy in addition machine translation. denotess results from Ruder et (2021), which is an updated version of Hu et al. (2020).
Pair Sentence
Structured Prediction
Question Answering
Model
Avg.
XNLI
PAWS-X
POS
NER
XQUAD
MLQA
TyDIQA
Metries
Acc
Acc
F1
FI/EM
FI/MM
FI/MM
Based XLM-R- large
XLM-R (Hu al. 2020
79.2
86.4
73.8
65.4
76.6/60.8
71.6/53.2
65.1/45.0
70.1
Trans-train (Wi ot al. 2020
82.9
90.1
74.6
66.8
80.4/65.6
72.4/54.7
66.2/48.2
72.6
Filter (Fang cl 2020
83.9
91.4
76.2
67.7
82.4/68.0
76.2/57.7
68.3/50.9
74.4
XTUNE (Zhong al. 2021
84.8
79.3
82.5/69.01 of
75.0/57.1
75.4/60.8
76.5
X-MIXUF
85.3
91.8
78.4
69.0
82.6/69.3
76.5/58.1
69.0/52.8
75.5
Based mBERT
mBERT (Hu al 2020
65.4
81.9
71,5
62.2
64,5/49,4
61.4/44.2
59.7/43.9
63.2
Joint-Align Zhao et al. 2020
72.3
Trans-train Hi al. 2020
75.1
88.9
72.4/58.3
67. 67.6/49.8
59.5/45.8
X-MIXUP
78.8
76.5
65.0
73.3/58.9
69.0/50.9
60.8/46.5
70.0
inference stage, without real data, we use a, E Di For classification tasks, we synthesize the predictions of both the source and target sequences by taking the mean the predicted probability distributions as the final prediction. For structured prediction and QA tasks, we only consider the prediction of the target sequence.
5 EXPERIMENTS
This section first introduces the cross-lingual understanding benchmark, XTREME. Then briefly introduces the configurations of downstream tasks and baselines. Finally, shows the main results of baselines and X-MIXUP on XTREME.
5.1 TASKS AND SETTINGS
Tasks In our experiments, we focus on three types of tasks in XTREME: (1) sentence pair classifi-
cation task: XNLI (Conneau et al. 2018) and PAWS-2 X (Yang et al. 2019; (2) structured prediction
task: POS (Nivre al. 2018) and NER (Pan et al. 2017); (3) question answering task: XQuAD (Artetxe et 2020). MLQA Lewis et al. 2020) and TyDiQA CClark et al., 2020). The details of these datasets refer to Hu et al. (222).. We utilize the translate-train and translate-test data from
the XTREME repo which also provide the pseudo-label of translate-train data for classification tasks and question answering tasks. The rest translation data are from Google Translate
Models Experiments are based on two multilingual pre-trained models: mBERT and XLM-R. We the pre-trained models of Huggingface Transformers? as the backbone model.
Hyper-parameters We select XNLI, POS, and MI.QA as representative tasks to search for the best hyper-parameters. The final model is selected based the averaged performance of all languages on the dev set. We perform grid search over the balance training parameter and learning rate from [0.2, 0.4, 0.6, 0.8] and [3e-6, 5e-6, 2e-5, 3e-5]. We also search for the best manifold mixup layer from [1, 4, 8, 12, 16, 20, 24]. In final results, we implement mixup in the first layer for classification tasks, 4th layer for structured prediction tasks. For QA tasks, we implement mixup in the 16th layer for large model, 8th layer for base model. Concrete details of experiments are presented in Appendix C.
5.2 BASELINES
We conduct experiments on two strong multilingual pre-trained models to verify the generality of methods: (1) mBERT Multilingual BERT is 12-layer transformer model pre-trained on the
hattsss//tttbbbb.cmmm//ole-- com/
We use bert-base--ulllee tiningual sed for mBERT and x1 immroberra--argg for XLM-R.
7

---

Published as a conference paper at ICLR 2022
Table 4: Comparisons between X-MIXUP and xTUNE under the same setting: XL-M-Rbbsse model and machine translation data augmentation. Results of XTUNE are from Zheng et al. 2021) Table 4.
Model
XNLI
POS
MLQA
XTUNER1
Zheng et al.
2021
79.7
XTUNER2
Zheng et al.
2021
78.9
76.6
68.7/51.1
X-MIXUP
80.4
77.8
71.2/53.1
Wikipedias of 104 languages. (2) XLM-R X-M-R-large is a 24-layer transformer model pre- trained on 2.5T data extracted from Common Crawl covering 100 languages. Based them, these are some strong baselines: (1) Trans-train Abbreviation for Translate-train. The training set of the source language is machine-translated to each target language and then the model is trained on the concatenation of all training sets. (2) Joint-Align Zhao et al. (2020) aligns the monolingual sub-spaces of the source and target language by minimizing the distances of embeddings for matched word pairs. (3) Filter Fang et al.] (2020) splices the representation of the target sequence and its
(4) XTUNE Zheng al. (2021) uses two types of consistency regularization based on four types of data augmentation.
translation in intermediate layers to extract multilingual knowledge.
5.3
MAIN RESULTS
Results on the XTREME benchmark are shown in Table Concrete results for each task are presented in Appendix C.2 Compared with strong baselines, X-MIXUP shows its superiority across different backbones and tasks, which indicates its generality. X-MIXUP outperforms Trans-train by 2.2% based on mBERT, and X-MIXUP outperforms Filter by 1.5% based on XLM-R. The superiority of X-MIXUP over Filter is that X- X-MIXUP gives a calibrated representation for target languages, not just the concatenation of two representations. Besides, X-MIXUP considers the noise of translation data and limits the noise propagation by introducing mixup ratio.
XTUNE achieves the best results on structured prediction tasks and the low-resource QA task TyDiQA (only 3.7k training data in English), but XTUNE uses three other data augmentation approaches in addition to machine translation. To make fairer comparison, we conduct experiments under the same setting in Table 4 which indicates X-MIXUP outperforms XTUNE on three types of tasks with only machine translation data augmentation. Besides, X-MIXUP only needs one-stage training, while XTUNE implements a two-stage training algorithm. However, X-MIXUP and XTUNE are complementary, where the former focuses on finding better representations for target languages while the latter concentrates on the lross-lingual data augmentation and consistency regularization.
6 ANALYSIS AND DISCUSSION
To better understand X-MIXUP and explore how X-MIXUP influences the cross-lingual transfer performance, we conduct analyses on several questions. Results show X-MIXUP achieves perfor- mance improvements across different languages and it also reduces the cross- coos-lingual representation discrepancy obviously. Table in Appendix B verifies the effectiveness of X X-Mixup on both seen and unseen languages. Besides, ablation results show the cross-lingual manifold mixup training contributes lot to cross-lingual transfer.
(Q1) How X-MIXUP influences the cross-lingual representation discrepancy? Language centroid (Rosenberg & Hirschberg 2007) is the mean of the representations within each language. We plot the language centroid of different methods (see Figure 4 in Appendix B which indicates X-MIXUP brings closer language centroids significantly. We also caleulate the CKA scores of the XNLI dataset (see Table 7 in Appendix [B). Results show X-MIXUP reduces the cross- mooss-lingual representation discrepancy evenly across different target languages, improving the CKA score by 10.4% on average. In conclusion, both the language centroids visualization and the CKA score improvement indicate X-MIXUP reduces the cross-lingual representation discrepancy effectively.
'In this section, we utilize the --rrge- model the backhone model.
8

---

Published as a conference paper at ICLR 2022
Table 5: Ablation results on X-MIXUP, where w/o mixup denotes remove the cross-lingual manifold mixup during training and inference and À = Ao denotes a constant mixup ratio.
Model
XNLI
POS
MLQA
X-MIXUP
85.3
78.4
76.5/58.1
w/o mixup
82.9
75.7
72.7/54.8
w/o mixup inference
84.0
77.6
75.6/57.3
w/o scheduled sampling
84.6
78.0
76.3/57.9
w/o consistency loss
84.2
78.0
76,5/58.0
X Ào
84.1
77.8
75.8/57.5
(Q2) How X-MIXUP influences the cross-lingual transfer gap? We compare the cross-lingual transfer gap in Appendix B T Compared with Trans-train, X-MIXUP reduces the averaged gap by 39.8% and shows its superiority across three types of tasks. Compared with state-of-the-oe methods, X-MIXUP achieves the smallest cross- coss-lingual transfer gap on four out of seven datasets, which suggests the effectiveness of X-MIXUP on classification and QA tasks.
(Q3) What is the essential component of X-MIXUP? There are five major components of X- MIXUP: gross-lingual manifold mixup training, mixup inference, Mixup Ratio, Scheduled Sampling. and consistency loss. To better understand X- K-MIXUP, Comparisons between X-MIXUP and w/o mixup show the effectiveness of cross-lingual manifold mixup across different tasks, and even without mixup inference (translate-test data), the mixup training can also achieve 2.6% improvements on average. Besides, comparisons between X-MIXUP and À λo show the effectiveness of introducing the translation quality modeling in the mixup process. Scheduled sampling achieves more performance improvements on the classification task, as the task shares labels across languages, and scheduled sampling can prevent the model from solely relying on the gold source sequence to make predictions. In addition, the consistency loss is also more effective on the classification task, because there is additional prediction consistency loss which can transfer the task capability from the source language target languages. Detailed ablation results on the consistency loss are shown in Appendix B Table 9 which shows the KL consistency loss contributes more than the MSE consistency loss the classification task.
implement ablation studies in Table S
(Q4) Which layer is the best to implement the manifold mixup? We implement the cross- cross-lingual manifold mixup in different layers (see Figure 5 in Appendix for details) and find different tasks prefer different mixup layers. Although different tasks have their own preferences, no matter which layer we mix, the cross-lingual transfer performance can be improved, except for mixing within a higher layer on classification tasks. The drop in classification task is mainly because the source and target sequences share the same task label. Performing mixup the model rely on the source sequence and ignore the target sequence. The structured prediction task is not sensitive the mixup layer, mainly because this task relies on both the short and long dependence. For QA tasks, the cross- lingual transfer performance shows trend from rise to decline as the mixup layer increases. The QA task needs higher-level understanding, but higher layers are more language-specific, where sequences from different languages have different gold answers.
higher layer may make
7 CONCLUSION
This paper focuses on enhancing the cross-lingual transfer performance on understanding tasks. Considering the large cross-lingual transfer gap in recent works, this paper first analyses related factors and finds this gap is strongly associated with the cross-lingual representation discrepancy. Then X-MIXUP is proposed to alleviate the discrepancy, which gives compromised representations for target languages by implementing the manifold mixup between the source and target languages. Empirical evaluations on XTREME verify the effectiveness X- X-MIXUP across different tasks and languages. Besides, both the visualization and quantitative analyses show X-MIXUP reduces the cross-lingual representation discrepancy effectively. Furthermore, X-Mixup can also be applied to the multilingual mre-training process by implementing the cross- lons-llngual manifold mixup parallel data. Findings on the relationship between the cross- -ingull transfer performance and representation discrepancy shed light promising way boost cross-lingual transfer for future research.

---

Published as a conference paper at ICLR 2022
REFERENCES
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolin- gual representations. In Proceedings of the 58th Annual Meeting of the Association for Compu- national Linguistics, pp. 4623-4637, Online, July 2020. Association for Computational Linguis- tics. doi: 10.18653/v1/200 w00.663/////22..... acl-:ma44..mmmmm222222 URL //w.tps://ww... awwebb. org/ anthology/ 2020. acl maimann iia.in... 421
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Conference on Neural Informa- tion Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1171-1179, 2015, URL
Mihaela Bornea, Lin Pan, Sara Rosenthal, Radu Florian, and Avirup Sil. Multilingual transfer learning for QA using translation as data augmentation. CoRR, abs/2012.05958, 2020. URL https://arxiv/ /ttp//xxwi..rg/////222.22200.
Sheila Castilho, Joss Moorkens, Federico Gaspari, lacer Calixto, John Tinsley, and Andy Way. Is neural machine translation the new state of the art? Prague Bull. Math.
Linguistics, 108:109-120, 2017. URL lattp:/ufaal .ff.faf..ff....m mff. cuni az/pbm1/10s
art castilho moorkens gaspari tinsley way. pdf
Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: Linguisticall-iiforned interpolation of hid-
den space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2147-2157, Online, July 2020. Asso- 194. URL https
ciation for Computational Linguistics. doi: 10.
//acl ogy /200 acl - in 194
Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, and Claire Cardie. Multi-source cross-lingual model transfer: Learning what share. In Proceedings of the 57th Annual Meeting Florence, Italy, July 2019.
of the Association for Computationnll Linguistics, pp. Association for Computational Linguistics. doi: 10. 10.18653/v//P1.. 1299. URLhttps: //ww
aclweb. ww.. htw.g//nthooooooooo wwt.orrgg/tnolgyyoo.ooo
/P19- 1299
Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang. Xia Song, Xian-Ling information-theorrticc Mao, Heyan Huang, and Ming Zhou. Infoxlm: An information theoretic framework for cross- lingual language model pre-training. CoRR, ahs/2007.078334 2020. htt.h:ttss:/// /aapss///axxtte
arg/abs/// 2007. 07834
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: A benchmark for information-seekiin question answering in typologically diverse languages. Transactions of the Association for Computational Lin- guistics, 8:454-470, 2020. doi: 1622/accla.000111 URL http://wwwwwlllllc. org 1.30
Enthelogy/2020,cco
Alexis Conneau and Guillaume Lample. Cross-lingual language model pretraining. Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett (cds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 7057-7067, 2019, URL http:stttt///rccccinnggg neurips. cC/ /appp/r///// 2019 2019/han httpa///co..l22222220o553c722223
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating tross-lingual sentence representations. In Proceedings of the 2018 Conference Brussels, Belgium, October-Nor Novbmbee 2018. Association for Computational Linguistics. doi: 10. 10.18653/v//D1--1 1269. URL _ www. a a www..rrg//tthhl.oggooo//o///
Empirical Methods in Natural Language Processing, pp. 2475-2485,
10

---

Published as a conference paper at ICLR 2022
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross- posss-lingual representation learning at scale. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguisties, pp. 0440-8451, Online, July 2020a. Association for Computational Linguistics. doi: 1tt/://////1122001lmmmmaaaaaaaaa 747. URL
https:/ wwwp://wwwwwk www.waccleee...
wttts::/www.a..wbb.oo wwt.oor/g/ntooog////o0/o.
main. 747.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. Emerging cross- lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6022-6034, Online, July 2020b. Association for Computational Linguistics. doi: 10.18653/w1/2022.acc. acc.aa.n..mmmmmm55366
URL https:/ //wp::::/www
ac web. http:/anhhoooyyyyy//
ma mww.a.nn......m. 536
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volune I (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18665//1//N191422, URL https www. www.waclwebb. wttpgg/nthholcooooo
1423
Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. FILTER: an enhanced fusion method for cross-lingual language understanding. CoRR, has/2009.00116., 2020. (tthtt
05166
Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia. Unsupervised quality estimation for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:539-555, 2020. https
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.
CoRR, abs/2003. 11080, 2020.
Junjie Hu, Melvin Johnson, Orhan Firat, Aditya Siddhant, and Graham Neubig. Explicit alignment objectives for multilingual bidirectional encoders. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani- Iz Beltagy. Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human I Language Technologies, NAACL-HLTT 2021, Online, June 6-11, 2021, pp. 3633-3643, Association for Computational Linguistics, 2021. URL https: /wi. org/ 100 18653866331
naacl mww.aaiinn.... .244
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong. Linjun Shou, Daxin Jiang, and Ming Zhou. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pp. 2485-2494, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19 1252. (tLLhttps:/ www. & C web. org/ hee//nthoor
Amit Jindal, Anijit Ghosh Chowdhury. Aniket Didolkar, Di Jin, Ramit Sawhney, and Rajiv Ratn Shah. Augmenting NLP models using latent feature interpolations. In Donia Scott, Núria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online). December 8- 8-13, 2020, pp. 6931 -936. International Committee on Computational Linguisticss, 2020. doi: 10. 1011186661222000 ht122/00..ooin-mmiinn col.cooinnngg main 611
611. URL /tttps://doi.rret wtt../.rggl..oo wtt..//////1...13/11lll...
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282 6293, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2002 acl- main. URL wgt.olloyyyy22200/m ma 560.
https: // www.wwww..mmmmm a web.

---

Published as a conference paper at ICLR 2022
Mihir Kale, Aditya Siddhant, Rami Al-Rfou, Linting Xue, Noah Constant, and Melvin Johnson. nmt5 is parallel data still relevant for pre- -re-triinngg massively multilingual language models? In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1thh International Joint Conference on Natural Language Processing, ACI/IJCNIP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pp. 683-691, Association for Computational Linguistics, 2021. doi: 10. 11853//v/20/// acl-short.87. URL https: /twi. org/ www.111663///2222 acl short
Phillip Keung, Yichao Lu, and Vikas Bhardwaj. Adversarial learning with contextual embeddings for zero-resource cross-lingual classification and NER. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1355-1360, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: N..18653//1/D199 138. URL https
/ www. , www.wb.wwbb......... org/ http://aatthhooo/yy-113333
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural net- work representations revisited. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceed- ings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3519-3529. PMLR, 2019. URL http::t/rrrcceeinngg mir wrw.spessssse..pe. / v woooooobblihht... 9athhh99aaa html
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. From zero hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 4483- 4499, Online. November 2020. Association for Computational Linguistics. doi: 1..18853v1/20222
emnlp- main.363. URL https: www acwweb wtt./aanoooogg///22222000
emnlp main
363
Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 7315-7330, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022.aad aww.:0000..mmmmmmm55...... URL https: www
cc..ebb.orr//atttooor///2222222 aat.0000.ccmaaaaa 653
Jindich Libovicky, Rudolf Rosa, and Alexander Fraser. On the language neutrality of pre-trained multilingual representations. In Findings of the Association for Computational Linguisties: EMNLP 2020, pp. 1663-1674, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. htt.:////////2...ooinnnnsoo/se wwwddding-eemll.... 2020. f -innnniennl...p...... 150
URL hattps://wwwt awwweb.clwbbbb.. org/ anthology
Fuli Luo, Wei Wang. Jiahao Liu, Yijia Liu, Bin Bi, Songfang Huang, Fei Huang, and Luo Si. VECO: variable and flexible cross-lingual pre-training for language understanding and generation. In Chengging Zong, Fei Xia, Wenjie Li. and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNLP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 3980-3994, Association for Computational Linguistics. wttp:///1111111311111111
2021. doi: 10. 1/253////2222.aallooooo 308. URL
acl long. . 308
Joakim Nivre, Mitchell Abrams, Zeljko Agic, Lars Ahrenberg, Lene Antonsen, and et al. Universal De- pendencies 2.2 2018 URL [ http/aaaa..hhaaaaaaaaaaaaeaeaaaa archi www.lllrrchieeeeeeeeeeeee www.lvve---vveeeeeeeeeeeeeeeeeee ft/hal-01930733 LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Xuan Ouyang, Shuohuan Wang. Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE- M: enhanced multilingual representation by aligning cross-lingual semantics with monolingual corpora. CoRR, abs/2012.
2020. BtLhhttss: http:://axxxvvvvvo.. www./o/g//////ooo////oooeeeeeseo abs////s///22 15674
Sinno Jialin Pan and Qiang Yang. A survey transfer learning. IEEE Trans. Knowl. Data Eng., 22 1345-194,, 2010. doi: 10. 00.1009/KKDE.2009.999 URL
TKDE 2009. 191
12

---

Published as a conference paper at ICLR 2022
Xiaoman Pan, Boliang Zhang. Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1958. Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 18653/v1/P17-1178 URL https: www. ac web. http://nthooooyyotooooooo
Edoardo Maria Ponti, Helen O'Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. Modeling language variation and universals: A survey typological linguistics for natural language processing. Computational Linguistics, 45(3):
559-601, September 2019, doi: 10.1162/coli a 00357. BRLLhttps://ww..cllee anthology/ J19 3005
Peter Prettenhofer and Benno Stein. Cross-lingual adaptation using structural strutuural coreessoennee learse learning. ACM Trans. Intell. Syst. Technol., 3()):13:1 2011. doi: 10.11452203666442036677, URL
https: /oi...i...//////// org 10. 1ew...1155229000 http://..111452222222222222
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. Cosda-ml: Multi-lingual code-switching
data augmentation for zero-shot cross-lingual NLP. In Christian Bessiere (ed.), Proceedings of the
Twenty-Ninth International Joins Conference on Artificial Intelligence, ICCI 2020, pp. 3853-3860. ijcai.org, 2020. doi: 10.24996/ijcai..2205533. URL https: /ooi org 10 2020/533
Marc' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level training with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 20/6, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. WR..tttpp//axii..rrrt
111 06732
Andrew Rosenberg and Julia Hirschberg. V-measure: A conditional entropy- based external cluster evaluation measure. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP C.NLLL,,, pp. 420, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: wwww aac.aalwww. wtt.://w......g..eoeoeeeeoeeeeee http:///wwww..co......o..oaaaoor
Sebastian Ruder, Ivan Vulic, and Anders Søgaard. A survey of cross-lingual word embedding models. J. Artif. Intell. Res., 65:569-631, 2019. doi: 1.1613/jair.1.1166.. URL https
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant. Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Graham Neubig, and Melvin Johnson. XTREME-R: towards more challenging and nuanced multilingual evaluation. CoRR, abs/2104.07412, 2021. URL http:/tts///axi....t.
abs/ abs/2104 07412
Aditya Siddhant, Melvin Johnson, Henry Tsai, Naveen Ari, Jason Riesa, Ankur Bapna, Orhan Firat, and Karthik Raman. Evaluating the cross-lingual effectiveness of massively multilingual neural machine translation. The Thirty- Fourth AAAI Conference on Artificial Intelligence,
AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI
2020, New York, NY, USA, February 7-12, 2020, pp. 1854-88611 AAAI Press, 2020. URL
https: aaai. org. ojs/ htt///ndxxxxp/// php/ hwt.:ppppppptaaieeeeeecce
6414
Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. XLDA: cross-lingual data augmentation for natural language inference and question answering. CoRR, abs/1905.1 1471, 2019. URL WtRLhhtp://aax... http:::://axxll.oo/////s/oe/// wbw./o/gg//bb.bbo.ooooooooo ab90/ 11471
Vīkas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019. Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1433-64477 PMLR, 2019. URL http: fttp:://prceedie ngs ml www.sesss/eee v7 ve rma 1 9a html
13

---

Published as a conference paper at ICLR 2022
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre- Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008
C. Wan, Rong Pan, and Jiefei Li. Bi-weighting domain adaptation for cross-language text classifica- tion. In IJCAI, 2011.
Xiangpeng Wei, Yue Hu, Rongxiang Weng, Luxi Xing, Heng Yu, and Weihua Luo. On learning universal representations across languages. CoRR, abs/2007. 15960. 2020. URL https://
Brxiv.org/abs//2007 15960
Shijie Wu and Mark Dredze. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 120-130, Online, July 2020. Association for Computational Linguisties. doi: 10.18663//112022.eepllmm 1.16. httpss
/ h/www.acl www.:ew.....gggtoo htw.:/borrr/aattooooo///2/220000
www.empppppppppp 1.16
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: massively multilingual pre text-to-text transformer. CoRR, 01s/2010.11134, 2020. URL
www.
Jian Yang, Shuming Ma, Dongdong Zhang, Shuangzhi Wu, Zhoujun Li, and Ming Zhou. Alternating language modeling for pross-lingual pre-training. In The Thirty-Fourt AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelli- gence Conference, IAA/ 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 9386-9393, AAAI Press, 2020. URL /attps::/aaaaee /tg/oo/g///ooooooooo index.ndxxxeei phpp httpp////.aaaattte//eeeeeee////e
Yinfei Yang, Yuan Zhang. Chris Tar, and Jason Baldridge. PAWS-X: A cross-linguat adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 3687-3692, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10. 111111/1//D11113331 URL httpp:://///mmm www. aewweb
org/ hog//atthloogy/D19-111
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, net forum?id=r1Dopl
2018. URL https:tts///peeeereee
Wei Zhao, Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. Inducing language multilingual representations. CoRR, abs/2008.09) 091112, 2020.
2008. 09112
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. Consistency regularization for cross-ling -onssll fine-tuning In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ithh International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3403-3417, Online, August 2021, Association for Computational Linguistics. doi: 10.1865.. 264. URL wattpp::/. http::::///aaaanoooooooooooooooe
org /201 2ttp//211.....1oooa acl- 264
Joey Tianyi Zhou, Hao Zhang, Di Jin, Hongyuan Zhu, Meng Fang, Rick Siow Mong Goh, and Kenneth Kwok. Dual adversarial neural transfer for low-resource named entity recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 3461 -3471, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.588/3/v 100/5/////P19/ 19- 1336. 1336
URL httpt:tt///// / www. a wwwwwb. org/ hwt.://www....o///ttttoooooo////
14

---

Published as a conference paper at ICLR 2022
A METHOD DETAILS
A.1 MULTI-HEAD ATTENTION
In the multi-head attention layer, multiple attention heads are concatenated
MultiHead( Q, K, Concat(head,,.... W
(9)
and each head the scaled dot-product attention
head, Attention(QW? KW VW?)),
(10)
QKT
Attention (Q, K, V) = softmax
vd V
(11)
where W W W and W are trainable parameters.
A.2 TRAINING OBJECTIVE
For classification tasks (e.g. NLI), the task loss is the cross-entropy loss
Lunsk Σ y loggo,
(12)
where C' is the size of the label set.
For structured prediction tasks (e.g. POS) and span extraction tasks (e.g. QA), the task loss is also the cross-entropy loss
Lunk
Σ Σ Viy ViylogPij-
(13)
where n is the sequence length.
15

---

Published as a conference paper at ICLR 2022
B ANALYSIS RESULTS
95
85
Accuracy
Accuracy
80-
XLM-R
XLM-R
Trans-train
75
Trans-train
en
e5
fr
de ko
zh
ja
enesde vi fr br tr hi elswru ur zh
Language
Language
(a) PAWS-X
XNLI
Figure 3: Performances PAWS-X and XNLI test set, where languages sorted by decreasing CKA scores. The trend indicates the performance gets worse along with the CKA score decreasing.
mr
Zh
fr
11
12-
zh
4.-
UtD
P1
04
2
MIXUP
(b) X-MIXUP
Figure 4: Language centroids visualization of the POS test set, which indicates X-Mixup brings closer these centroids obviously.
84
POS
MLQA
78
74-
24
Figure 5: Performances on implementing X-MIXUP (solid line) in different layers and Trans-tres train (dashed line) on three downstream tasks.
16

---

Published as a conference paper at ICLR 2022
Table 6: Performances on XNLI test where Trans-train and X-MIXUP trained 8 languages and tested on both these seen languages and 7 unseen languages. A is the performance difference between X- X-MIXUP and Trans-train Results show X-MIXUP performs better than Trans- train by a large margin on both seen and unseen languages.
Model
Seen Languages
Unseen Languages
Avg.
bg
el
fr
th
Trans train
87.6
84.7
84.2
84.6
82.9
80.1
76.5
83.0
82.6
84,5
85.0
80.6
77.8
82.7
82.7
82.6
M-MIXUP
86.3
84.8
+1.9
+2.4
+1,9
42.2
+1.8
+2.6
+2.5
42.0
+2.7
+1.8
+1.9
+1.7
+2.5
+1.8
+1.8
+1.8
Table 7: CKA scores and performances on XNLI test set, where is the score or performance difference between X-MIXUP and Trans-train. Results show X-MIXUP improves the CKA scores evenly across different target languages and the performance improvements are diverse. There no obvious correlation between the CKA score improvement and performance improvement.
Model
en
C5
de
vi
fr
bg
tr
el
ru
ar
hi
SW
ur
th
zh
Avg.
CKA score
Trans-train
1.00
0.81
0.80
0.78
0.78
0.78
0.76
0.75
0.75
0.74
0.74
0.73
0.72
0.72
0.72
0.77
MIXUP
1.00
0.88
0.87
0.86
0.86
0.86
0.85
0.83
0.82
0.84
0.83
0.82
0.81
0.80
0.79
0.85
0.00
0.07
0.07
0.08
0.08
0.08
0.09
0.08
0.07
0.10
0.09
0.09
0.09
0.08
0.07
0.08
Performance
82.6
85.2
82.1
84.5
81.8
80.8
80.2
82.7
82.6
X. MIXUP
89.9
87.7
86.9
85.4
87.1
87.3
84.9
86.8
85.1
85.2
83.5
81.2
79.6
83.2
85.2
85.3
+2.0
+2.4
+2.8
+2.9
+2.1
+2.8
+2.7
+4.2
+1.9
+3.0
+2.5
+2.7
Table 8: The cross-lingual transfer gap (lower is better) of different methods on the XTREME benchmark. For QA tasks, we only show EM scores. denotes results from Wei et al. (2020). Overall, X-MIXUP achieves the smallest cross-lingual transfer gap four out of seven datasets.
Model
XNLI
PAWS-X
POS
NER
XQuAD
MLQA
TyDiQA
Avg.
mBERT (uu et al. 2020
16.5
14.1
25.5
23.6
25.0
27.5
22.2
22.1
XLM-R (Hlu et a.. 2020
10.2
12.4
24.3
19.8
16.3
19.1
13.1
16.5
Trans-train (Hu
2020)
7.3
9.0
22.41
20.5
17.6
22.2
24.2
17.6
Filter (Fang et al. 2020
6.0
5.2
19.7
7.3
9.2
11.3
XTUNE (Zheng et al. 2021
5.5
5.2
17.3
14.8
10.1
18.5
0.9
10.3
X-MIXUP
4.9
5.2
18.1
15.9
6.7
13.9
9.6
10.6
Table 9: Ablation results the consistency loss, which show the KL consistency loss contributes more than the MSE consistency loss on the classification task.
Model
XNLI
POS
MLQA
X MIXUP
85.3
78.4
76.5/58.
w/o MSE consistency loss
84.6
78.0
76.5/58.0
w/o KL consistency loss
84.3
w/o both
84.2
17

---

Published as a conference paper at ICLR 2022
C EXPERIMENTAL DETAILS
C.I PAPER-PARAMETERS
For all tasks, we fine-tune on Nvidia V100-32GB GPU cards with the batch size 64. For XQuAD and MLQA, we finetune 2 epochs. For other tasks, we finetune epochs. There is no dev set in XQuAD, so we use the dev set of MLQA for the model selection. Table shows parrmmtrrseeters used for X-MIXUP.
Table 10: eyper-parameters used for X-MIXUP, where used for balanced training in Eq 8 and Pk is the scheduled sampling decay rate.
Parameter
Classification
Structured Prediction
QA
0.4
0.8
0.2
Pk
1000
1000
2000
C.2 DETAILED RESULTS
Detailed results of each tasks and languages are shown below. Results of mBERT, XLM, MMTE and XI.M-R are from XTREME (Hu et al. 2020). Results of Filter is the best results of Fang et al. (222)).
Model
bg
de
el
es
fr
hi
SW
tr'
ur
vi
zh
Avg.
mBERT
81.9
73.8
77.6
77.6
75.9
79.1
77.8
70.7
75.4
70.5
70.0
74.3
67.4
77.0
77.6
75.1
XLM
72.7
70.4
75.5
74.3
62.5
70.7
70.2
MMTE
79.6
64.9
70.4
68.2
67.3
71.6
69.5
63.5
66.2
61.9
66.2
63.6
60.0
69.7
69.2
67.5
XLM-R
88.6
82.2
85.2
84.5
84.5
85.7
84.2
80.8
81.8
77.0
80.2
82.1
77.7
82.6
82.7
82.6
Filter
89.5
83.6
86.4
85.6
85.4
86.6
85.7
81.1
83.7
78.7
83.2
83,9
83.8
83.9
XTUNE
89.9
84.0
87.0
86.5
86.2
87.4
86.6
83.2
85.2
80.0
82.7
84.1
79.6
84,8
84.3
84,8
X-MIXUP
89.9
85.2
87.3
86.9
86.8
87.7
87.1
83.5
85.1
81.2
83.2
84.9
79.6
85.4
85.2
85.3
Table 11: XNLI accuracy scores for each language.
Model
de
fr
ja
ko
zh
Avg.
mBERT
94.0
85.7
87.4
87.0
73.0
69.6
77.0
81.9
XLM
94.0
85.9
88.3
87.4
69.3
64.8
76.5
80.9
MMTE
93.1
85.1
87.2
86.9
72.0
69.2
75.9
81.3
XLM-R
94.7
89.7
90.1
90.4
78.7
79.0
82.3
86.4
Filter
95.9
92.8
93.0
93.7
87.4
87.6
89.6
91.5
XTUNE
96.1
92.6
93.1
93.9
87.8
89.0
88.8
91.6
X-MIXUP
96.3
93.2
93.6
94.6
87.3
88.2
89.5
91.8
Table 12: PAWS-X accuracy scores for each language.
18

---

Published as a conference paper at ICLR 2022
Model
ar
bg
de
Ch
eu
fa
be
hi
huu
MBERT
86.6
85.0
85.2
86.9
66.7
78,9
67.2
78.3
88.4
XLM
88.5
63.1
85.0
85.8
84.3
95.4
85
78.
62.8
64.7
78.4
82.8
65.9
66.2
77.3
70.2
87.4
XLM-R
88.1
88.5
96.1
88.3
76.4
82.6
89.4
Fiter
88.7
66.1
88.5
89.2
88.3
96.0
89.1
86.3
78.0
70.8
86,
88.9
64.9
76.7
82.6
72.6
89.8
XTUNE
74.7
87.2
83.6
X X-MIXUP
89.4
70.1
88.8
88.7
86.7
96.0
89.0
88.3
76.2
72.5
87.0
88.2
82.4
78.0
83.8
72.4
90.3
Model
ja
kik
ko
mi
pt
ru
ta
le
th
ur
vi
yo
Avg.
MBERT
49.2
70.5
49.6
69.4
88.6
86.2
85.5
41.7
81.4
68.5
57.0
53.2
55.7
61.6
71.5
XLM
49.0
70.2
50.1
68.7
88.1
84.9
86.5
59.8
76.8
55.2
76.3
66.4
61.2
52.4
20.5
65.4
71.3
XLM-R
15.9
78.1
53.9
80.8
89.5
87.6
89.5
65.2
47.2
92.2
76.3
70.3
56.8
24.6
25.7
73.8
Filer
40.44
80.4
53.3
86.4
89.4
88.3
90.5
65.3
87.3
57.2
94,1
77.0
70.9
58.0
43.1
53.1
76.9
XTUNE
65.3
79.8
56.0
85.5
89.7
89.3
90.8
85.5
61.4
93.8
78.3
74.0
57.5
27.9
68.8
79.3
X-MIXUP
62.7
79.0
55.3
84,8
89.6
88.8
90.1
63.6
87.4
59.9
93,1
77.1
72.4
59.4
27.3
68.3
78.4
Table 13: POS results (F1) for each language.
Model
af
ar
bg
bn
de
el
et
eu
fa
Fr
hre
hi
hu
id
ja
Jv
MBERT
85.2
77.4
41.1
77.0
70.0
78.0
72.5
77.4
75.4
66.3
46.2
77.2
79.6
56.6
65.0
76.4
53.5
81.5
29.0
66.4
XLM
74.9
44.8
76.7
70.0
75.5
74.8
74.8
62.3
78.5
57.7
66.1
63.0
XLM-R
78.8
79.5
79.6
80.5
56.8
73.0
79.8
Filer
83.5
80.4
60.7
83.5
78.4
S0.4
80.7
74.0
81.0
66.9
71.3
80.2
79.9
57.4
74.3
82.2
$4.0
81.9
24.3
63.5
XTUNE
85.0
30.4
59.1
84.8
79.1
80.5
82.0
78.1
81.5
64.5
65.9
82.2
81.9
62.0
75.0
82.8
55.8
83.1
30.5
65.9
X-MIXUP
84.5
79.0
58.4
84.0
81.4
$0.6
81.4
73.8
81.5
65.7
61.6
80.4
80.3
64.4
74.7
82.0
53.4
K2.2
38.8
63.5
Model
my
p4
th
y0
zh
12.7
XLM
67.7
57.2
26.3
59.4
62.4
69.6
47.6
81.2
77.9
63.5
68.4
49.6
0.3
78.6
71.0
70.1
26.5
32.4
MMTE
60.9
43.9
58.2
44.8
58.5
68.3
42.9
74.8
72.9
58.2
66.3
48.1
46.9
3.9
64.1
61.9
37.2
68.1
32.1
28.9
XLMR
71.6
56.2
60.0
67.8
68.
57.1
$4.3
81.0
81.9
69.1
70.5
59.5
55.8
1.3
73.2
76.1
56.4
79.4
33.6
33.1
Filler
71.0
51.1
63.8
70.2
69.8
69.3
59.0
84.6
82.1
71.1
70.6
64.3
58.7
2.4
74.4
83.0
73.4
75.8
42.9
35.4
XTUNE
67.1
72.6
1.1
77.5
44.9
36.5
X-MIXUP
4.4
75.0
47.8
Table 14: NER results (F1) for each language.
Midel
31
+1
bi
th
vh
Avg:
IBBETT
83.5/72.2
61.5/45.1
70.6
/4.0
62.6/443 24.9
75.5/56.9
59:2:4600
71.3/53.3
42.3 33.5
55.41
69.51 49.6
55.01 48.3
64.5/49.4
XLM
74.2/62.1
61.4/4.7
66.01 49.7
$7.5139.1 39.1
68.1149.8
36.6:40.3
65.3/ 48.2
24.5
41.2
63.8/47.6
49:7139.77
59.81 46.3
MNITE
30.1/68.1
63.2/46.2
68.8/50.3 50.3
63.3/3.9
72.41525
61.3:472
68.4145.2
48.4 35.9
58.11 40.9
70.9 50.1
(3.8153 36.4
64.47 46.2
XLM R
36.5/75.7
49.0
80.4/13.4
79.8661.7
82.01635
36.7:597
30.1/63.
78.2 62.
73.91 99.3
79.1 59.0
50.3
76.6/608
Eliter
36.4/7466
19.5/50.1
63./60.0
83.0/616
85.01679
85.1/666
428/6774
19.6/73.
50.41 64.4
618/64.7
75.9/7100
82.4/680
88.3/78.1
84.7/68.
8071649
82.2/666
76.1
80.3/760
82.5/69.0
51,3/633
84.366.6
33.//6.7
72.4/756
32.6/693
Table 15: XQuAD results (F1 EM) for each language.
Model
en
ar
de
es
Avg.
mBERT
/ 67.0
52.3 34.6
59.0 43.8
67.4/49.2
50.2/
61.2/ 40.7
59.6 38.6
61.4/ 44.2
XL.M
55.2
42.5/25.2
50.8/37.2
/4.7/37.99
34.4/21.1
48.3/30.2
40.5/21.9
48.5/32.6
MMTE
78.5
56.1/-
58.41-
64.9/-
46.2/-
59.4/-
58.3/-
60.3/41.4
XLM-R
70.5/70.6
66.6
70.1
76.6
70.6/53.1
74.0/
62.1/37.0
71.6/
Filter
84.0/ 70.8
72.1
74.8 /60.0
78.1 /601
57.0
78.1 /57.5
70.5
76.2/
XTUNE
85.3/72.9
/5.7
72.3/57.3
76.3/
74.0 56.0
55.5
70.8
75.0/
X-MIXUP
83.1
71.95111
54.5/59.4
/
77.3
58.0
73.7
76.57
Table 16: MLQA results (F1
EM) for each language.
Model
ba
id
ko
Avg.
62.2
49.3
59.7
65.8/4588
58.8 50.0
57.5
38.4
59.7/43.9
XLM
66.9/5399
39.4/41.2
27.2/ 15.0
58.2/41.4
62.5145.8
14.2/5.1
49.2/30.7
39.6/21.6
15.5/6.9
43.6/ 29.1
MMTE
62.9/493
63.1/39.
55.8/41.9
53.9/42.1
60.9/476
49.9/42.6
58.9/37.9
65.1/47.2
54.1/45.8
58.1/43.
71.5/56.8
67.6/404
77.4:619
67.0/2.11
66.1
Filter
72.4/59.1
728//088
70.5/56.6
73.3/57.2
76881598
33.1/12.3
68.94665
774/65.7
69.9/5044
68.3/50.9
XTUNE
73.8/616
77.8/60.
73.5/61.1
77.0/62.2
80.8/68.1
66.9/36.5
72.1/51.9
77.9.655
77.6/ 60.7
75.3 60.8
X-MIXUP
73.9/614
7388554.
67.4
75.4760.6
788/66.0
32.9/129
69.1/52.2
78.0766.9 66.9
72.0753.
69.07
Table 17: TyDiQA-GolP results (F1 EM) for each language.
19


Reviews:
[{"review_id": "sR8r9Pi9-8R", "reviewer": "Reviewer_DF3R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like this idea that considers the source languages and the target languages at the same time to encourage the model to learn more similar representations for different languages. Here are some suggestions and questions:\n- Figure 4 only contains a few languages and they are mostly high-resource languages. It would be more convincing if the authors can show results of more languages, especially low-resource languages.\n- What if there are multiple source languages? Do we randomly sample one of them during training and testing?\n- It seems like we need to know what target languages we will care about in advance. What if there are some new target languages? Some analysis on this can be quite interesting. For example, only consider part of target languages during training and see if those languages help the model learn good representations for unseen languages as well.\n- Why the consistency loss consists of two terms: MSE and KL. Any ablations on this?\n- Does more CKA score improvement lead to more task performance improvement? Any relations between them?\n", "summary_of_the_paper": "This paper proposes X-Mixup, a model that considers the source languages and target languages together for cross-lingual transfer. The designed model takes a pair of sentences (or the translated sentences) in a source language and a target language as the input and computes the cross-attention between them. The author also adds the consistency loss to encourage similar predictions for the source and the target. They also propose scheduled sampling to handle the different distributions during training and testing. The experimental results and the ablation studies show promising performances for cross-lingual transfer.", "summary_of_the_review": "The proposed idea is interesting. They show good and promising experimental results and ablation studies. Some questions need to be clarified and some more analysis is suggested.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qgxzcob69E", "reviewer": "Reviewer_9Shh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a technique called *cross-lingual manifold mixup* or *X-mixup*, which has been inspired by *mixup* (Zhang et al., 2018) and *Filter* (Fang et al., 2020). The general idea is to combine the hidden representations corresponding to the source-language input with the hidden representations corresponding to the target-language input (in a smarter way than, e.g., simply concatenating them). The authors combine this with 1) a specific mixup ratio based on translation quality and 2) scheduled sampling. They perform multiple experiments using two pretrained models (mBERT and XLM-R) and show that their proposed approach improves over multiple strong baselines.\n\nStrengths: \n- The approach works: the authors show improved results over multiple strong baselines.\n- The paper is well written and the authors perform a nice analysis/investigation of smaller research questions in Section 6.\n\nWeaknesses:\n- I am unsure if framing *X-mixup* as a cross-lingual transfer version of *mixup* is the best framing. I only read the *mixup* paper superficially, but it seems to me that that paper's goal is to generate additional training examples by averaging (independent) pairs for both *x* and *y* to make the model more robust. In contrast, what the authors do here is more closely related to simply concatenating source and target sentence in the input. \n- The experiment section isn't very clean. For example, it is unclear why different methods are used on top of mBERT and XLM-R, respectively (see Table 3).", "summary_of_the_paper": "This paper proposes a technique which the authors call *cross-lingual manifold mixup* or *X-mixup*. The approach has been inspired by *mixup* (Zhang et al., 2018) and *Filter* (Fang et al., 2020). The general idea is to combine the hidden representations corresponding to the source-language input with the hidden representations corresponding to the target-language input (in a smarter way than, e.g., simply concatenating them). The authors perform multiple experiments using two pretrained models (mBERT and XLM-R) and show that their proposed approach improves over multiple strong baselines.", "summary_of_the_review": "The paper presents *X-mixup*, a method for cross-lingual transfer based on translate-train, which improves performance over multiple strong baselines. However, I have doubts regarding the framing of the paper as well as regarding the conducted experiments. Thus, I think the paper should be revised before being published.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "m2KVxQaKm4K", "reviewer": "Reviewer_pZJi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. The paper is well written and the motivation is clear. The paper first conducts some priori experiments to strength the motivation, and transforms the performance gap to the representation discrepancy through the visiualization analysis.\n2. The manifold mixup seems to be valuable for bridging the cross-lingual representation discrepancy across languages.\n3. The improvements on XTREME over XLM-R is pretty strong. \n\nCons:\n\n1. What does $\\tilde{\\mathcal{D}}_{S}^{Train}$ means? I didn't find its definition in the \"Notations\" part.\n2. I would like to see ablations for the mixup ratio $\\lambda$ where you use randomly sampled value for $\\lambda$ or compute a gate using $h_{T|S}$ and $h_{T}$, rather depend on $A$. \n3. Why you need the consistency loss in the training objective? What does $P_S$ and $P_T$ exactly mean? How to deal with the KL collapse issue?\n4. What does $\\tilde{D}_S$ (appeared in the Scheduled Sampling and Inference parts) means? \n5. The scheduled sampling strategy has a great impact on training efficiency. Can you provide the training time of the X-Mixup and translate-train respectively?", "summary_of_the_paper": "The work applies and adjusts manifold mixup in the subject area of zero-shot cross-lingual transfer. The work first identifies the challenges with the current landscape of cross-lingual transfer methods, in which the performance of the target languages still lags for far behind the source language. The paper make the gap associated with cross-lingual representation discrepancy. To take care this, the authors propose a X-Mixup method that interpolates the representations between the source and target languages, and give this compromised representation for target language prediction. Compared to strong baselines, the work obtains significant gains on the XTREME benchmark.", "summary_of_the_review": "I score this paper a 6. I think the manifold mixup is very interesting for this use-case and made my reading and evaluation of this paper more interesting. I look forward to the responses from the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ih9rF9U0i9", "reviewer": "Reviewer_VD9r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n- x-mixup is a clear idea and seems straightforward to include on top of existing models like mBERT or XLM-R.\n- The improvements over strong baselines on multiple XTREME tasks are convincing.\n\nWeaknesses:\n- One big downside I see is that the authors use mixup at different layers for different tasks. What happens if a single setting is picked and used for all tasks? How does the comparison look?\n- The regularization term to account for MT quality should be further tested by controlling the quality of MT and seeing if it's robust to it. Otherwise it's not clear how valuable it is.\n- Another downside is that this method relies on translation data (both translate-train and translate-test) during training. I would like to see an ablation where only translate-train data is used or only translate-test data is used. This would be fair comparison to the XTREME submissions which only use translate-train and not translate-test.\n\n\nDetailed comments:\n- The related work section is a bit weak.\n- Some missing citations:\n     - there's a whole body of work on using translation data for cross-lingual learning: please cite\n                 VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation\n                 Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation\n                 Explicit Alignment Objectives for Multilingual Bidirectional Encoders\n                 nmT5 -- Is parallel data still relevant for pre-training massively multilingual language models?\n- Footnote #9 is a very important detail. It should be moved to the main paper.\n- Some minor grammar mistakes but overall clear paper.\n- Please have a discussion on how this be applied to the retrieval tasks in XTREME. Can this approach be moved to the pre-training set up by leveraging some parallel data?\n- Please also discuss the impact on training time by adding x-mixup compared to translate-train. \n\n", "summary_of_the_paper": "The paper proposes cross-lingual mixup, a technique which performs manifold mixup of source and target sequences. This technique also includes mixup ratio which factors in MT quality and scheduled sampling which deals with exposure bias. Experiments on 3 task types from the XTREME benchmark show that x-mixup leads to significant gains over translate-train baselines and other strong submissions to the leaderboard. The authors also present some analyses with a few ablations.", "summary_of_the_review": "The authors propose x-mixup a technique which leverages parallel data and forces cross-lingual representations to be aligned. The proposed method leverages translation data (translate-train and translate-test) during training. This leads to improvements in several XTREME downstream tasks over strong baselines. The authors perform some ablations which showcase the importance of the different components in the proposed method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes X-Mixup, a model that considers the source languages and target languages together for cross-lingual transfer. The designed model takes a pair of sentences (or the translated sentences) in a source language and a target language as the input and computes the cross-attention between them.  

The empirical results are convincing. Reviewers think this paper is well-written and the idea is interesting.
Decision: Accept (Poster)