Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models
Authors: Yixin Ji
Jikai Wang
Juntao Li
Hai Ye
Min Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: isotropic representation
cross-lingual
multilingual

Abstract: With the development of multilingual pre-trained language models (mPLMs), zero-shot cross-lingual transfer shows great potential.
To further improve the performance of cross-lingual transfer, many studies have explored representation misalignment caused by morphological differences but neglected the misalignment caused by the anisotropic distribution of contextual representations.
In this work, we propose enhanced isotropy and constrained code-switching for zero-shot cross-lingual transfer to alleviate the problem of misalignment caused by the anisotropic representations and maintain syntactic structural knowledge.
Extensive experiments on three zero-shot cross-lingual transfer tasks demonstrate that our method gains significant improvements over strong mPLM backbones and further improves the state-of-the-art methods.\footnote{Our code will be available at \url{https://github.com/Dereck0602/IsoZCL}.}

Isotropic Representation Can Improve Zero-Shot Cross-Lingual Transfer on Multilingual Language Models
Yixin Ji', Jikai Wang', Juntao Li"; Hai Ye², Min Zhang'
Institute of Computer Science and Technology, Soochow University, China Department of Computer Science, National University of Singapore {jiyixin169, risus254Jégmail.. com;
yehaiêcomp.. nus edu.sg sg;
{ljt, minzhang (esehhang@suda..e.. edu.. cn
Abstract
With the development of multilingual pre- trained language models (mPLMs), zero-shot cross-lingual transfer shows great potential. To further improve performance of cross- lingual transfer, many studies have explored representation misalignment caused by mor- phological differences but neglected mis- alignment caused by the anisotropic distribu- tion of contextual representations. In this work, we propose enhanced isotropy and con- strained code-switching for zero-shot cross- lingual transfer to alleviate the problem mis- alignment caused by the anisotropic represen- tations and maintain syntactic structural knowl- edge. Extensive experiments on three zero- shot cross- mossulinguua transfer tasks demonstrate that our method gains significant improvements over strong mPLM backbones and further im- proves the date-of-the-an methods."
C25
(a) anisotropic representa- tions
(b) isotropic representations
Figure 1: An illustration of anisotropic and ideally isotropic multilingual representations.
The core of cross-lingual transfer to align rep- resentations among different languages (Lample et al., 201 8b; Cao et al., 2020; Pan et al., 2021: Dou and Neubig, 2021). Existing mPLMs can align the representation well for myriads of the cross-lingual transfer scenarios but fail to handle these language pairs differed significantly under the zero-shot set- ting, especially for languages with distinct morpho- logical features (Ahmad et al., 2019 20.9a. Therefore, many works have explored the key factors affecting the alignment of language representations (Pires et al., 2019; Karthikeyan et al., 2020; Libovický et al., 2019; de Vries et al., 2022) and proposed solutions accordingly (Cao et al., 2020; Chi et al., 2021: Pan al., 2021; Zhao et al., 2021; Huang et al., 2021b). Existing methods can be roughly divided into three categories: 1) using parallel cor- pus (Chi et al., 2021; Wei et al., 2021; ; Feng et al., 2022) bilingual dictionary (Cao et al., 2020; Qin et al., 2021) to better align contextualized word embedding spaces; 2) utilizing morphological syntactic features (Ahmad et al., 2021; Yu et al., 2021; Zhao al., 2021) eliminate misalignment; 3) leveraging robust training methods (Huang al., 2021b) to tolerate misaligned representations.
1 Introduction
Cross-lingual transfer aims utilize the rich se- mantics and syntactic knowledge in high-resource source languages improve the performance of low-resource target languages. Benefiting from the "scaling effeet", language models pre-trained on hundreds of languages have dominated cross- lingual transfer for years owing to their superior performance and generalization capability, which can learn unified representation space through self-supervised learning (Devlin et al., 2019; Con- neau and Lample, 2019; Conneau al., 2020a; Xue al., 2020; Chi et al., 2021, 2022; Scao et al., 2022a). These powerful multilingual pre-trained language models (mPLMs) not merely improve task performance with full supervised learning but even the zero-shot tross-lingual transfer (Wu and Dredze, 2019 9a; Hsu et al., 2019; Li et al., 2021; Sherborne and Lapata, 2022).
"Corresponding author. 'Our cnde is available at hetps://githhn com/ Dereck6602/Iso2CL.
However, additional parallel corpora are diffi- cult to obtain for many extremely low-resource

---

languages, while annotating morphological or syn- tactic features requires considerable human effort, Though the robust training methods are free from additional supervision signals, they ignore the mis- alignment caused by the distributional properties of representation. As found by Rajace and Pilchvar (2022), the representation distribution of mPLMs is highly anisotropic, where most words are squeezed into narrow region in the representation space. As shown in Figure la, even semantically irrele- vant representations can appear in the neighbor- hood. Thus, there are reasons to believe that highly anisotropic representations hurt cross mooss-lingual align- ment in the representation space.
its the expressiveness of contextual representations (Gao et al., 2019; Wang al., 2020; Li et al., 2020; Su et al., 2021; Rajaee and Pilehvar, 2021a). Next, we will introduce two different metrics for measur- ing isotropy quantitatively.
Cosine Similarity. Since the word vectors are squeezed together, the external manifestation of anisotropic representations is that for any two words, the cosine similarity large. If represen- tations are isotropic, cosine similarities of random representations are close zero (Gao et al., 2019; Ethayarajh, 2019). The metric can be formulated as follows:
N
To alleviate the aforementioned obstacles, propose isotropy enhancement strategy that can improve the representation alignment at the seman- tic level while maintaining the essential syntac- tic knowledge, which is indispensable for cross- lingual transfer (Ahmad et al., 2021). As an ex- cessively isotropic representation space is risky to pull apart representations that should be aligned, we also introduce constrained code-switching method to better utilize the readily available bilin- gual dictionaries. To verify the effectiveness of our proposed method, we launch experiments on three zero-shot cross-lingual tasks, i.e., paraphrase identi- fication, natural language inference, and sentiment classification. Experimental results demonstrate that our proposed method significantly improves the performance of zero-shot cross-lingual trans- fer upon strong mPLMs. Further analytical explo- ration confirms that our method alleviate the anisotropy problem pre-trained representations while preserving syntactic knowledge implicit in the representations as much as possible.
(W))
1
Σ L.
Cos (x),
(1)
N
where Tú, 2j are randomly sampled representations.
N is the number of sampled representation pairs. Icss (W) closer to 0 indicates that the representa- tions are more isotropic.
Principal Components. Following Mu and partition function
Viswanath (2019), (Arora al., 2016) to measure the isotropy:
F(u)
Σ exp (u ww:) 1=1
(2)
where ws W is contextual word embedding. N is the number of embeddings in the represen- tation space, E U the eigenvector of the em- bedding matrix W According to Arora et al. (2016), if representations are isotropic, F(u) could be approximated using a constant. Thus, Mu and Viswanath (2019) propose metric based prin- cipal components:
IPc(W) ~ mingeu F(u)
2 Preliminary
(3)
max F(u)
In this section, we briefly introduce the anisotropic problem representation learning and the risk of undermining knowledge of syntactic structures in existing methods of mitigating anisotropy.
IPC(W) closer to indicates representations are more isotropic.
2.2 Syntactic Knowledge Probing of Existing Isotropy Enhancement Methods
2.1 Anisotropic Problem of Contextual Representations
Currently, research on enhancing the isotropy of contextual representations focuses on feature- based learning or training from scratch. For fine-tuning the pre-trained language model, although the fine- tuned model still has severe anisotropy, directly applying existing methods to enhance isotropy can- not effectively improve the performance and may even degrade it (Rajaee and Pilehvar, 202 Zhang
Anisotropy is geometrical property of contextual representations. As defined by Li al. (2020), anisotropic representations occupy narrow cone in the vector space. Conversely, isotropic repre- sentations are uniformly dispersed in the vector space. is widely believed that anisotropy lim-

---

unbbe Sherecktter
p(y
p(yxx)
50
Classiffer
Classifier
Coslk,t
10
O000
OOOO
86 H1
Ar
56
(a) LAS
(b) UAS
Figure 2: Depprobe results on the vanilla fine-tuned model and three isotropic enhancement methods.
mPLM
et al., 2022). Because the fine-tuning has changed the distribution of linguistic and task-specific repre- sentations in the pre-trained language model, exist- ing methods may destroy task-essential knowledge (Rajace and Pilehvar, 2021b).
code-switching
! Isotropy Enhancement
Constrained Code-switching
Figure 3: Overview of our proposed method which includes isotropy enhanced fine-tune and constrained code-switching.
We analyze that most methods are applied to semantic-level tasks, such as sentence representa- tion and semantic textual similarity. Thus they ne- glect to consider the structural syntactic knowledge implicit in the representation. However, structured syntactic knowledge implicit in representations is essential for many natural language understanding tasks, especially for cross- messslliguul transfer (Ahmad et 2021). To verify our suspicions, we use the re- cently proposed Depprobe 2022), linear probe that can extract labeled and directed dependency parse trees from contextual representations, to measure the structural knowl- edge of fine-tuned representations.
and structural levels.
Method In this section, we introduce our method for zero-shot cross-lingual transfer, which comprises of isotropy enhancement and constrained code- switching. Figure 3 shows the overview of our proposed method. Before elaborating our method in detail, we first present some necessary notations. Given the training dataset D {(r,, M)) the loss of fine-tuning can be written by:
et al.,
In experiments, we train the Depprobe on the En- glish treebank and evaluate on six target languages. We report representative metrics in dependency parsing, labeled attachment scores (LAS) and unla- beled attachment scores (UAS), which measure the accuracy of predicted dependency graphs. All these datasets are from Universal Dependencies v2.82. The mPLM used has been fine-tuned on the XNLI dataset. We compare the impact of three repre- sentative isotropic enhancement methods (whiten- ing transformation (Su et al., 2021), cluster-based methods (Rajaee and Pilehvar, 2021a), and CosReg (Gao et al., 2019)) on structural knowledge. Fig- ure shows the LAS and UAS of six languages on XLM-R. We can observe that all these methods lead to a sharp drop in the results of the syntactic probe experiments. Therefore, it is necessary to de- vise an isotropic enhanced method for pre-trained models while preserving knowledge at semantic
ct = Σ (ff(=00).9)
(4)
where a;; is token sequence (e.g., a sentence), and Vi is its label. f(;0) 0) is the model to be optimized, and I(:; .) is the loss function to learn task-specific model on the source language. For classification tasks, I(::-) -) is usually the cross-entropy loss. We denote the contextual representation of a; as h; Rixd where L is the length of the sample and d is the dimension of the mPLM.
3.1 Isotropy Enhancement
To enhance the isotropy of the token representa- tion space of the multilingual pre-trained language model and meanwhile maintain semantic and syn- tactic features as much as possible, we introduce an isotropy-aware loss a regular term force the representation distribution H to be close an isotropic distribution. For fcasibility, we suppose obeys a normal distribution M(u, E), which is
Phtp::/hdl httt:://hdl.hnddlcnne//// handle.net/ 1ttie/neeee/1///1-3363333 1234/1-3687

---

reasonable assumption and has also acquiesced in other methods such as whitening transformation. As we know, the zero-mean isotropic normal distribution N o", is a perfect isotropic distri- bution, thus we can use the Wasserstein distance between the two normal distributions to measure the isotropy degree of H (Salmona et al., 2021):
representations (Conneau et al., 2020b; Yang al., 2020; Qin et al., 2021). We randomly select words in the source language text, query bilingual dic- tionaries, and replace them with target language words to obtain code-switching data. More details can be found in $4.2.
Consistency Constraints. We denote the origi- nal and corresponding code-switching data {wi, W2, "., and x' {wi. w2...., w.. where uy means the replaced source language token by target languages. For a sample xj E D and its code-switching data n' E D', the mPLM will pro- duce two different representations h;, h C Lxd To further promote the alignment among represen- tations with similar semantic in different languages, utilize the following consistency constraints be- tween the original and its code-switching data:
W = 1
(5)
When Wi is used regular term, ensure the sta- bility of the optimization process, the value of W should be on the same order magnitude as 2/ by setting the a to a small value. Therefore, or is a very sensitive hyperparameter. To avoid tedious hy- perparameter search, we calculate the Wasserstein distance between the normalized representation dis- tribution and N (0, I), which has been proved to be equivalent to (5) by Fang al. (2023):
00 A2 Σ Cos(h h4')
(10)
2 Vd
2,ED
W = |l +1+ Tr(
T> (6)
ZED'
where Cos(:,)) is the cosine similarity between two representations and A2 a hyper-parament
For given batch of samples, the mean and covari- ance matrix of representations are as follows:
3.3 Training
N
h
(7) As mentioned above, the whole training process is divided into two stages. Our method first im- proves the spatial distribution of multilingual rep- resentations via isotropic enhancement, and then promotes representation alignment via constrained switchnnt For the first stage, the loss func- tion takes the form of:
N
N
hi
h;
E
14
-
(8)
N
12:112
[h,112
where || ll2 denotes L2 norm, N is the number of token representations in the given batch. The proposed isotropy-aware loss can be formulated as:
L1 = Ch + Liso
(11)
C iso = WM
where L is to learn a task-specific model on the source language. Liso is to enhance the isotropy of contextual representations during fine-tuning by narrowing the difference between the representa- tion space and the standard normal distribution. In the second stage, the loss function is:
where A is hyperparameter which controls the degree of isotropy.
3.2 Constrained Code-switching
Isotropy enhancement provides suitable represen- tation space property for multilingual alignment, making unrelated representations not misaligned due to representation space degradation. How- ever, an excessively isotropic representation space is risky to pull apart representations that should be aligned. Thus, we align representations with similar semantics by constraining code-switch data, Code-switching means that more than one language alternates a sentence, which is widely considered to provide anchor points for aligning multilingual
1 L2 = (cc + +b/ reg
(12)
where c.f c.fi are the fine-tuning loss on orig- inal data and code-switching, respectively, Creg is for aligning constraints between the representa- tions of code-switching samples and the original data. We split training into two stages for two rea- sons. First, these losses may interfere with each other because isotropy enhancement widens the

---

distance between representations so that representa- tions are distributed more evenly, while constrained code-switching makes multilingual representations with the same semantics close. When optimizing, achieving a delicate balance is challenging. Also, simultancous optimizations add more cost (Eq.5 has more computational cost than 10.
4.3 Baselines
We compare our methods with the following isotropy enhancement methods and strong zero- shot cross-lingual transfer baselines:
Fine-tune. We fine-tune all the parameters of the mPLM with the English training set and then eval- uate on test sets of all target languages.
BN. Zhao et al. (2021) propose a vector space normalization method. They apply batch normal- ization to the last layer representations of mPLM to induce -ggnust representations and increase the discriminativeness of embeddings. IsoBN. Zhou et al. (2021) explore the isotropy of the pre-trained [CLS] embeddings and propose isotropic batch normalization (IsoBN), They as- sume that the absolute correlation matrix of embed- dings is block-diagonal.
4 Experiments
4.1 Datasets
We conduct our experiments on three different cross-lingual tasks, including paraphrase identifi- cation (PAWS-X: Yang et al., 2019), natural lan- guage inference (XNLI; Conneau et al., 2018) and sentiment classification (MARC); Keung et al., 2020). For MARC, following Keung et a2.(20,)), we splice the "review title" and "product category" after the review. Due to the limitations of com- puting resources, we sampled 25% of the original training set as training set. The data characteristics are shown in Appendix A.
CosReg. The high cosine similarity between word representations is an extrinsic indication of repre- sentation degeneracy. Thus, Gao et al. (2019) add CosReg loss to mínimize the cosine similarities between any two contextual token embeddings. Noisy Wu et al. (2022) inject noise into the parameters of the pre-trained model when fine- tuning, preventing the pre-trained model from over- fitting on the source language.
4.2 Experimental Setup
Our experiments are based on two mPLMs, mBERT (Devlin et al., 2019) and XLM-R-large (Conneau al., 2020a). On each task, we fine- tune the mPLM for five epochs with batch size 32 on the English training set, select the best model on the English development set, and then evalu- ate the cross-lingual performance on test sets of all target languages. We run 2 epochs in the first stage and epochs in the second stage for PAWS- X and MARC. For XNLI, the epochs of the two stages is and 4. For PAWS-X and XNLI, we tune the learning rate in ( 11e-6, 2e-6, 5e-6, le-5, 2e-5); for MARC, learning rate in {8e-7, 1e-6, 2e-6, le- 5, 2e-5). We tune coefficient A and X in (0.5,
DA. Huang al. (2021b) improve zero-shot cross- lingual transfer through robust training based data augmentation. They use a predefined synonym generate augmentation examples. For fair comparison with our method, we set the number of augmented examples to for all datasets.
4.4 Main Results
Table 1 shows zero-shot cross-lingund results on three datasets using two mPLMs. Following He et al. (2021), to better compare cross-lingual trans- fer languages of different language families, we show the average results for three cases, including all languages (All), target languages other than En- glish (Target), and non-Indo- Eurooenn languages (Distant). We find that existing methods for en- hancing isotropy in the fine-tuning stage do not significantly improve cross- minssll transfer perfor- mance. BN, IsoBN, and CosReg even perform worse than the vanilla fine-tuning some datasets. In comparison, our method has consistent and sig- nificant performance gains on all these datasets and mPLMs. Especially for non-Indo-Furopean lan- guages, our method has better transfer ability. Then, our method achieves comparable or even better per-
1.0J. We report details of all Syyper-parameter in Appendix B. When constructing code-switching data, we set the probability that each token in sample is replaced with a target language token to 0.5. The bilingual dictionaries we used are from MUSE (Lample et al., 201 8a)4. We report the av- erage score on the test set of 5 runs with different seeds. We conduct the experiments on one NVIDIA GTX3090 GPU.

---

PAWS-X
XNLI
MARC
Models
Avg.
All
Target
Distant
All
Target
Distant
All
Target
Distant
mBERT
83.51
81.77
76.22
66.34
65.20
61.43
45,80
42.68
38.18
65.22
mBERT+BN
82.99
81.14
75.40
66.39
65.23
61.40
44,43
40.98
36.39
64.60
mBERT+1soBN
83.34
81.57
75.96
66.60
65.47
61.88
45,38
42,13
37.56
65.11
mBERT+ConReg
83.18
81.38
75.42
66.28
65.11
61.55
46.72
43.51
38.63
65.39
mBERT=NoisyTune
83.86
82.22
76.83
66.44
65.27
61.54
46.34
43.32
38.86
65,55
mBERT+DA
84.86
83.40
78.79
66.32
65.23
61.52
46.98
44.04
38.93
66.05
mBERT+ours
85.29
83.81
78.76
67.27
66.
62.96
47.90
44.83
39.52
66.79
mBERT+DA+ours
85.50
84.19
79.67
67.47
66.51
63.19
48.21
45.34
39.73
67.06
XLM-R
87.01
85.63
81.09
79.35
78.71
76.94
58.88
57.52
53,99
75.08
XLM-R+BN
87.45
86.11
81.49
79.46
78.81
77.10
59.29
57.97
54.57
75.40
XLM-R+IsoBN
87.60
86.30
81.84
79.66
79.03
77.36
59.21
57.90
54.59
75.49
87.73
86.42
81.66
79.27
78.63
76.90
59.46
58.11
54.55
75.49
-LMM-R-NNisyTune
87.53
86.22
81.56
79.73
79.10
77.43
58.99
57.71
54.31
75,42
XLM-R+DA
88.86
87.74
83.89
81.10
80.60
79.10
59.48
58.05
54.28
76.48
3LM-R+ourss
89.03
87.93
83.86
80.90
80.31
78.71
59.71
58.34
54.54
76.50
XLM-R+DA+our
89.48
88.37
84.51
81.40
80.90
79.21
59.89
58.56
54.92
76.92
Overall comparison zero-shot cross-lingual performance between our proposed model and baseline models. All is the average result of all languages. Target the average result of target languages other than English. Distant is the average result of non-Indo- non-Indo-European languages. Avg. is the average result of three datasets.
Table I:
Models
En
De
Es
Fr
Ja
Ko
Avg.
XLM-R
95.57
89.77
90.19
90.57
80.30
79.80
83.17
87.01
XLM-R+ours
95.64
91.37
91.98
92.61
83.19
83.15
85.25
89.03
only isotropy
95.86
91.25
91.64
92.23
82.42
81.52
84.31
88.46
only constrained
95.43
90.90
91.05
91.25
82.38
82.50
84.13
88.23
one stage
95.78
91.00
91.46
92.29
82.06
81.50
85.41
88.50
Table Ablation results on PAWS-X based on XLM-R. only isotropy means only using isotropy enhancement, only constrained means only using constrained code-swtching, and stage means using isotropy enhancement and constrained code-switching together.
formance than the state-of-ard model. Combining our method with DA can lead new state-of-art model. In conclusion, all the results confirm the effectiveness of our method for zero-shot cross- language transfer tasks. The results for each target language are detailed in the Appendix D.
tations hurt both high-resource and low-resource languages. However, using only constrained code- switching sacrifices a little source language perfor- mance. We think this because code- switching inevitably introduces some noisy samples. After removing the constrained code-switching, the per- formance of Japanese, Korean and Chinese, which belongs to a different language family than the source language, is severely degraded. It indicates that for target languages that different from the source language, constrained code-switching can play a great role. We also investigate the necessity of two-stage training, and the experimental results provide a positive affirmation. We can observe that although the one-stage training has a signif- icant performance improvement compared to the baseline. it still lags behind the two-stage training.
5 Analysis and Discussion
5.1 Ablation Study
We conduct ablation experiments to analyze the contributions of isotropy enhancement, constrained code-switching and two-stage training. From the results in Table 2, enhancement and constrained code-switching sig- nificantly impact the cross-lingual transfer perfor- mance. Using only isotropy enhancement consis- tently increases the performance of all languages in the test set, suggesting that anisotropic represen-
see that both isotropy

---

1.0
Xitte
Ximme D1rr
2.7
0.9
0.8
032
043
0.7
0.6
AM66
0.5
En-Fr
Ev:3i
Es-Ar
En
Figure 4: CKA scores the XNLI test set.
5.2 Cross-lingual Representation Discrepancy
Cross- Cross-lingual representation discrepancy quantita- tively measures the degree of divergence between source and target language representations in the same embedding space. Yang et al. (2022) demon- strate that the cross-lingual transfer performance is highly related to the cross- linsull representation discrepancy. The smaller gross-lingual representa- tion discrepancy correlates to better pross-lingual transfer performance. Following Conneau et al. (2020b), utilize the linear centered kernel align- ment (CKA) (Kornblith al., 2019) score indi- cate the cross- posss-lingual representation discrepancy:
Figure 5: The isotropy metric on XNLL
tuning and basclines. We note that although Cos- Reg can obtain a near-perfect Icos by constraining the cosine similarity between word representations, it exhibits a high degree of anisotropy under the IPC. Rajace and Pilchvar (2021b) find cosine sim- ilarity might fail in high-dimensional space. Al- though the cosine similarity is close zero, it's only isotropic in some dimensions and remains highly anisotropic from a global perspective. There- fore, we consider that using the CosReg method during fine-tuning does not inherently correct the anisotropic distribution of the representation.
Y
CKA(X,Y) |X X TY
(13)
5.4 Visualization
where X and Y are features of parallel sequences from the source and target languages. A higher CKA score means smaller cross-lingual represen- tation discrepancy. We use the cross coossllnggual repre- sentation discrepancy as a quantitative measure of sentence-level representation alignment. We con- duct vanilla fine-tuning and our method on XNLI and evaluate CKA scores on test sets. Figure shows CKA scores for six language pairs. We can observe that our method achieves higher CKA score. Thus, claim that our method helps induce better aligned multilingual representations.
To demonstrate the effectiveness of our method on the representation distribution, we utilize PCA to reduce the contextual representations of XLM- R PAWS-X to two dimensions Specifically, randomly sample 10,000 token representations from the source language. As presented in Figure 6, each data point the plots represents a contextual representation. We can observe from the left plot that after vanilla fine-tuning, most representations are concentrated in two regions of space, which reflects the highly anisotropic distribution of these representations. In contrast, the representations on the right plot more evenly distributed through- out the space. The PCA visualization demonstrates that our isotropy enhancement method can produce more uniformly distributed representation.
5.3 Isotropy Measuring
To verify whether our proposed isotropy enhance- ment method can induce an isotropic representation space, measure the isotropy of contextual rep- resentations on the test sets of XNLI using two metrics introduced in $2.1. As shown in Figure whether it is Ioos or TPC. our proposed method can obtain more isotropic contextual representations on mBERT and XLM-R model than the vanilla fine-
5.5 Syntax Probing
Through the discussion in $5.2, our method nar- rows the representation discrepancy between multi-

---

representations with high anisotropy negatively affect downstream tasks. Furthermore, Rajace and Pilehvar (2022) find multilingual pre-trained lan- guage models have a higher degree of anisotropy than the corresponding monolingual ones.
Currently, a number of methods have been pro- posed to mitigate the degradation of contextual representations. Gao et al. (2019), Zhang et al. (2020) and Wang al. (2020) tackle this problem by adding additional loss function constraints in the pre pre-training phase. In addition, there has been some work to enhance isotropy through post-processing methods. Li et al. (2020) utilize flow-based gen- erative model to map contextual representations into isotropy standard normal distribution. Su al. (2021) and Huang et al. (2021a) achieve the same effect by the whitening transformation. Rajacc and Pilehvar (2021a) propose a local cluster-based method. However, Rajace and Pilehvar (2021b) find that though fine-tuning pre-trained language models can achieve considerable performance boost, the representation space of fine-tuned mod- els is still highly anisotropic. Unfortunately, many existing methods for adjusting the fine-tuned rep- resentation space for isotropy will hurt its perfor- mance. Our study finds that existing methods de- stroy the syntactic knowledge implied by original representations when applied to fine-tuning, con- trast, our proposed isotropy enhancement method can preserve as much important syntactic knowl- edge as possible in the fine-tuning stage.
(a) vanilia fize oune
(b) our isctrpy ethancenent metbod
Figure 6: PCA visualization of representations vanilla fine-tuned XML-R Ra and fine-tuned by our method.
Model
Ar
En
Hi
Ru
Tr
Zh
LAS
XLM-R
17.4
48.5
21.0
32.5
24.4
4.7
+BN
4.5
19.2
5.7
7.4
9.4
4.1
4soBN
4.8
18.1
6.4
7.7
10.7
3.8
+ours
23.6
54.6
21.7
34.6
26.9
5.8
UAS
XLM-R
32.4
55.7
37.5
44.3
41.2
17.7
15.8
32.1
19.3
19.0
24.3
14.3
+lsoBN
15.7
30.5
19.5
18.9
25.5
12.9
+ours
37.2
61.0
39.5
47.0
43.8
18.8
Table 3:
Depprohe results on vanilla fine-tuned models and our proposed isotropy enhencement methods.
plc languages at the semantic level. As we saw in 82.2, many existing isotropic transformation methods do not preserve knowledge at the struc- tural level, and we also conduct dependency syntax probe experiments on our proposed method with the same setting. Table 3 shows the LAS and UAS on XLM-R. Our proposed method can exceed the performance of the vanilla fine-tuning method on both metrics in languages. In summary, the ex- perimental results show that our proposed isotropy enhancement method enhances the isotropy of the representation distribution while maintaining the structural knowledge of the original representation. This may be because the distribution transforma- tion by optimizing the Wasserstein distance can preserve some geometric characteristies of the orig- inal distribution (Panaretos and Zemel, 2019).
Zero-shot Cross-lingual Transfer: Owing to the significant progress multilingual pre-trained lan- guage models (mPLMs) (Devlin al., 2019; Con- neau and Lample, 2019; Conneau et al., 2020a; Xue et al., 2021; Chi et al., 2021, 2022; Scao et al., 2022b), zero-shot cross-lingual transfer achieves surprising performance in various NL.P tasks (Hsu et al.. 2019; Li et al., 2021; Sherborne and Lapata, 2022; Zheng al., 2021))
However, to date, researchers have not been able to figure out what factors influence the ability zero-shot cross-lingual transfer. Wu and Dredze (2019b) observe the sub-word overlap between the source and target languages has positive effects the zero-shot performance. In contrast, some researchers (Pires et al., 2019; Karthikeyan et al., 2020) show no direct relationship between lexical overlap and cross-lingual transfer effects. de Vries et al. (2022) confirm the impact of some typolog- ical features, such as lexical- phonnttcc distances,
6 Related Work
Anisotropy of Contextual Representation. Nowadays, contextual representation (Peters et al., 2018; Devlin et al., 2019) often performs better than static representation in many NLP tasks. However, several studies (Gao et al., 2019; Ethayarajh, 2019; Cai et al., 2021) find contextual representations exhibit severe anisotropy in geometric properties. Their experiments show that

---

word order differences, and writing systems. Mean- while, Chai et al. (2022) further find that word composition plays a more important role in cross- lingual transfer than other language properties. Ad- ditionally, some studies have focused on the rep- resentation level and proposed many methods to align representations and induce language-agnostic representations (Cao et al., 2020; Libovický et al., 2019; Zhao et al., 2021; Tanti et al., 2021). Huang al. (2021b) considered the difference repre- sentation between the source and target languages as noise in the contextual embedding and utilized robust training methods to tolerate noise.
especially since calculating Wasserstein distance requires more computation. We will explore more efficient isotropy enhancement methods for cross-lingual transfer in the future.
A Acknowledgements
This work is supported by the National Science Foundation of China (NSFC No. 62206194), the Natural Science Foundation of Jiangsu Province, China (Grant No. BK20220488).
References
7 Conclusion
Wasi Ahmad, Haoran Li, Kai-Wei Chang, and Yashar Mehdad. 2021. Syntax-augmented multilingual BERT for cross-lingual transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 1ith International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages Association for Computational Linguisties.
In this paper, we propose a simple but effec- tive method to improve the performance of zero- shot cross-lingual transfer. By introducing the isotropy enhanced fine-tuning and constrained code-switching, our proposed method can induce moderate isotropic representation and align mul- tilingual representation. Experimental results on three zero-shot cross-lingual transfer tasks demon- strate the performance superiority of our method over existing methods. Extensive analytical ex- periments further confirm the effectiveness of our method for enhancing isotropic representations and reducing cross-ling pooss-lingual representation discrepancy.
Online.
Wasi Ahmad, Zhisong Zhang, Xuezhe Ma, Eduard Hovy, Kai-Wei Chang, and Nanyun Peng. 2019a. On difficulties of cross-lingual transfer with order differences: A case study on dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2440-2452, Minncapolis. Minnesota. Association for Computa- tional Linguistics.
Limitations
Wasi Uddin Ahmad, Zhisong Zhang, Xuezhe Ma, Kai- Wei Chang, and Nanyun Peng. 2019b. Cross-lingual dependency parsing with unlabeled auxiliary lan- guages. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 372-382, Hong Kong, China. Association for Computational Linguistics.
Even though our work improves cross-lingual per- formance effectively, some limitations are still listed below:
Our method is based on a widely accepted assumption: multilingual pre-trained language models can map the semantics of different lan- guages to the same representation space, and rep- resentation alignment significantly affects cross- lingual generalization ability. However, pre- trained model cannot cover all languages world- wide. For languages not seen the pre-training stage, they may not be in the same representation space as the source language, and our method may have little effect.
Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016. A latent variable model approach to pmi- word embeddings. Transac- tions of the Association for Computational Linguis- rics, 4:385-399.
Xingyu Cai, Jiaji Huang, Yuchen Bian, and Kenneth Church, 2021. Isotropy in the contextual embedding space: Clusters and manifolds. In International Con- ference on Learning Representations.
Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multi- lingual alignment of contextual word representations. In International Conference Learning Representa- tions.
We conduct experiments on two strong masked language models. However, we have not success- fully applied our method to the most promising generative pre-training models, such as BLOOM, and we will continue to explore in the future. Compared with the original fine-tuning, our method increases the training phase's ' time cost,
Yuan Chai, Yaobo Liang, and Nan Duan. 2022. Cross- lingual ability of multilingual masked language mod- els: study of language structure. Proceedings of the 60th Annual Meeting of the Association for

---

Computetiinnal Linguistics (Volume 1: Long Papers), pages Dublin, Ireland. Association for Computational Linguistics.
deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information- theorrettccccc framework for cross-lingual language model pre-training. In Proceedings of the 2021 I Conference of the North American Chapter of the Association for Computational Linguistics: Hu- mun Language Technologies. pages 3576-3588, On- line. Association for Computational Linguistics.
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume. pages 2112-2128. Online. Association for Computational Linguistics.
Zewen Chi, Shaoban Huang, Li Dong, Shuming Ma. Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song. Xian-Ling Mao, Heyan Huang, and Furu Wei. 2022. XLM-E: Cross-lingual language model pre-training via ELECTRA. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6170-6182, Dublin, Ireland. Association for Computational Lin- guistics.
Kawin Ethayarajh. 2019. How contextual are contextu- alized word representations? Comparing the geom- etry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNTPP) pages 55-65, Hong Kong, China. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer. and Veselin Stoyanov. 2020a. Unsupervised cross-lingal representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 8440- 8451, Online. Association for Computational Lin- guistics.
Xianghong Fang, Jian Li, Xiangchu Feng, and Benyou Wang. 2023. Rethinking uniformity in self- supervised representation learning.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022. sanguage-agnostie BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 878-891, Dublin, Ireland. Association for Computa- tional Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in neural information processing systems, 32.
Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2019. Representation degeneration prob- lem in training natural language generation models. In International Conference Learning Representa- tions.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams. Samuel Bowman. Holger Schwenk, and Veselin Stoyanov. 2018 XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021. On the effectiveness of adapter- based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 2208- 2222, Online. Association for Computational Lin- guistics.
Alexis Conneau, Shijie Wo, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language mod- els. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics. pages 6022-6034, Online. Association for Computational Linguistics.
Tsung-Yur Yuan Hsu, Chi-Liang Liu, and Hung-yi Lee. 2019. Zero-shot reading comprehension by cross- lingual transfer learning with multi-lingual lan- guage representation model. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP--JCNIIP) pages 5933-5940, Hong Kong, China. Association for Computational Linguistics.
Wietse de Vries. Martijn Wieling, and Malvina Nissim. 2022. Make the best of cross- lingull transfer: Ev- idence from POS tagging with over 100 languages. In Proceedings of the 60th Annual Meeting of the A Association for Compurational Linguistics ( (Volume
1: Long Papers), pages (676-7685,1 Dublin, Ireland. Association for Computational Linguistics.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan

---

Duan. 2021a. WhiteningBERT: An easy unsuper- vised sentence embedding approach. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 238-244, Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Max Müller-Ebersterr Rob van der Goot, and Barbara Plank. 2022. Probing for labeled dependency trees. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Voolue
1: Long Papers), pages 7711 1-722,, Dublin, Ireland. Association for Computational Linguisties.
Kuan-Hao Huang, Wasi Ahmad, Nanyun Peng, and Kai-Wei Chang. 2021b. Improving zero-shot cross- lingual transfer learning via robust training. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 1684 1697, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Lin Pan, Chung-Wa Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, and Mo Yu. 2021. Multilingual BERT post-pretraining alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computaaional Linguistics: Hu- man Language Technologies. pages 210-219, Online. Association for Computational Linguistics.
K Karthikeyan, Zihan Wang. Stephen Mayhew, and Dan Roth. 2020, Cross-lingual ability of multilingual bert: An empirical study. In International Conference on Learning Representations.
Victor M Panaretos and Yoav Zemel. 2019. Statistical aspects of wasserstein distances. Annual review of statistics and its application. 6:405 431.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt
Phillip Keung, Yichao Lu, György Szarvas, and Noah A. Smith. 2020. The multilingual Amazon reviews cor-
Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Pupers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguisties.
pus. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4563 4568, Online. Association for Computational Linguistics.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519-3529. PMLR.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019, How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 1996-5001 Flo- rence, Italy. Association for Computational Linguis- tics.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representa- tions.
Libo Qin. Minheng Ni, Yue Zhang, and Wanxiang Che. 2021. Cosda-ml: multi- lingual sode-switching data augmentation for zero-shot cross-lingual nip. In Pro- ceedings of the Twenty-Ninth International Confer- ence International Joint Conferences on Artificial Intelligence, pages 3853-3860.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In Interna- rional Conference on Learning Representations.
Sara Rajace and Mohammad Taher Pilchvar. 202 la. A cluster-basee approach for improving isotropy in con- textual embedding space. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisticy and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 575-584, Online. Association for Computational Linguistics.
Bohan Li. Hao Zhou, Junxian He, Mingxuan Wang. Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 91 19-9130, Online. Association for Computa- tional Linguistics.
Juntao Li, Ruidan He, Hai Ye, Hwee Tou Ng. Lidong Bing, and Rui Yan. 2021. Unsupervised domain adaptation of a pretrained cross-lingual language model. In Proceedings of the Twenty-Ninte Interna- tional Conference on International Joint Conferences on Artificial Intelligence, pages 3672-3678.
Sara Rajaee and Mohammad Taher Pilehvar. 2021b. How does fine-tuning affect the geometry of em- bedding space: A case study on isotropy. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 3042-3049 Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jindřich I. Libovicky, Rudolf Rosa, and Alexander Fraser. 2019. How language-neutral is multilingual bert? arXiv preprint arXiv: 1.000310.
Sara Rajace and Mohammad Taher Pilehvar. 2022. An isotropy analysis in the multilingual BERT embed- ding space. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 1309-1316, Dublin, Ireland. Association for Computational Lin- guistics.
Jiaqi Mu and Pramod Viswanath. 2019. All-but-the-top: Simple and effective postprocessing for word repre- sentations. International Conference on Learning Representations.

---

Antoine Salmona, Julie Delon, and Agnès Desolneux. 2021. wraserewasserstein distances between gaus- sian distributions. arXiv preprint arXiv: 104.07970.
Shijie Wu and Mark Dredze. 2019b. Beto, bentz, becas: The surprising cross- coos--lnggual effectiveness of BERT. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natural Language Processing ( (EMNLP-IJCNLP)) pages 833-844 Hong Kong, China, Association for Computational Linguis- tics.
Teven Le Scao, Angela Fan, Christopher Akiki, El- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni. François Yvon, Matthias Gallé, et al. 2022a. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100
I. Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. In North Amer- ican Chapter of the Association for Computational Linguistics.
Teven Le Scao, Thomas Wang. Daniel Hesslow, Lu- cile Saulnier, Stas Bekman, M Saiful Bari, Stella Bideman, Hady Elsahar. Niklas Muennighoff, Jason Phang, et al. 2022b. What language model to train if you have one million gpu hours? arXiv preprint arXiv:2210. 15424.
Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies. pages 483 -488.
Tom Sherborne and Mirella Lapata. 2022. Zero-shot cross-lingual semantic parsing. In Proceedings of the 60th Annual Meeting of the Association for Compu- national Linguistics (Volume 1: Long Papers), pages 4134 4153, Dublin, Ireland. Association for Compu- tational Linguistics.
Huiyun Yang, Huadong Chen, Hao Zhou, and Lei Li. 2022. Enhancing cross-lingual transfer by manifold mixup. In International Conference on Learning Representations
Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021. Whitening sentence representations for bet- ter semantics and faster retrieval. arXiv preprint 1rXiv:2103.15316.
Jian Yang. Shuming Ma, Dongdong Zhang. Shuangzhi Wu, Zhoujun Li, and Ming Zhou. 2020. Alternating language modeling for cross-lingual pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 9386-9393.
Mare Tanti, Lonneke van der Plas, Claudia Borg, and
Albert Gatt. 2021. On the language-specificee of multilingual BERT and the impact of fine-tuning, In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP,, pages 214-227, Punta Cana, Dominican Repub- lic. Association for Computational Linguistics.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: cross-lingual adversar- ial dataset for paraphrase identification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing TEMNLP-IJCNLA ).,, pages 3687-3692, Hong Kong. China, Association for Computational Linguis- tics.
Lingxiao Wang. Jing Huang. Kevin Huang. Ziniu Hu, Guangtao Wang. and Quanquan Gu. 2020. Improv- ing neural language generation with spectrum control. In International Conference on Learning Representa- tions.
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021. On learning univer- sal representations across languages. In International Conference on Learning Representations.
Sicheng Yu, Hao Zhang, Yulei Niu, Qianru Sun, and Jing Jiang. 2021. COSY: COunterfactual SYntax for lingull understanding.
Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 577-589, Online. Asso- ciation for Computational Linguistics.
Chuhan Wu, Fangzhao Wu, Tao Qi. and Yongfeng Huang. 2022. NoisyTune: little noise can help you finetune pretrained language models better. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume Short Papers), pages 680-685, Dublin, Ireland. As- sociation for Computational Linguistics.
Haode Zhang, Haowen Liang, Yuwei Zhang, Li-Ming Zhan, Xiao-Ming Wu, Xiaolei Lu, and Albert Lam. 2022. Fine-tuning pre-trained language models for few-shot intent detection: Supervised pre- traininin and isotropization. In Proceedings of the 2022 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 532-542, Seattle, United States. Association for Computational Linguistics.
Shijie Wu and Mark Dredze. 2019a. Beto, bentz, becas: The surprising cross- lins-linguual effectiveness of bert. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 833-844.

---

Zhong Zhang, Chongming Gao, Cong Xu, Rui Miao. Qinli Yang, and Junming Shao. 2020. Revisiting rep- resentation degeneration problem in language mod- cling. In Findings of the Association for Computa- lional Linguistics: EMNLP 2020, pages 518-527, Online. Association for Computational Linguistics.
C Confidence interval
Table shows the 95% confidence interval of the vanilla fine-tuning and our method.
PAWS-X
XNLI
MARC
MBERTT
83.51=0.07
66.34:10.48
45.80++2.00
Wei Zhao, Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. 2021. Inducing language-agnostic mul- tilingual representations. In Proceedings of *SEM 2021: The Tenth Join! Conference on Lexical and Computational Semanties, pages 229-240, Online. Association for Computational Linguistics.
mBERT+oWrS
85.29=:0.29
67.20:10.15
47.90+10,78
mBBORTDA++Orr
85,50=00.16
67.47=0.13
48.21
XLM-R
87.01
79.35=+0.19
58.88+0. 18
89.03
80,90
59.71 10.13
L.M-R+DA+uuRS
89.48 1.028
81,40
59.19
Table 6: The 95% confidence interval of results.
Bo Zheng, Li Dong. Shaohan Huang, Wenhui Wang. 7ewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, and Furu Wei. 2021. Consistency reg- ularization for gross-lingual fine-tuning. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joins Conference on Natural Language Processing (Volume 1: Long Papers), pages 1403-3411
D Results for each task and language
Table 7-9 show detailed results for each language in PAWS-X, X, XNLI and MARC.
Wenxuan Zhou, Bill Yuchen Lin, and Xiang Ren. 2021. Isobn: fine-tuning bert with isotropic batch normal- ization. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14621- 14629.
A
The characteristics of datasets
Table shows the detailed characteristics of datasets.
Dataset
PAWS-X
XNLI
MARC
Task
Paraphrase
NLI
Sentiment
Class
3
ILangl
7
15
6
Metric
Acc.
Acc.
Acc.
iTrainl
49,401
392,702
50,000
IDevl
2,000
2,490
5,000
iTest
2,000
5,010
5,000
Table 4: Characteristics of datasets
B Hyper-parameters
Table 5 shows the detail of hyper-parameters.
PAWS-X
XNLI
MARC
IBERRT
XLM-R
mBERT
XLM-R
mBERTT
XLM-R
Ir
2e-5
5e-6
le.5
5c-6
2e-5
2e-6
A
0.5
0.5
0.5
1.0
0.5
0.5
Aa
1.0
1.0
1.0
1.0
1.0
1.0
Table 5: Details of flpee-paarameters

---

Models
dc
frr
ja
ko
zh
avg
mBERT
93.97
85,92
88.16
87.85
75.16
74.34
79.17
83.51
mBERT+BN
94.07
85,90
87.69
87.07
75.09
72.77
78.34
82.99
mBERT+1soBN
93.98
85.76
88.12
87.64
74.97
73.77
79.14
83.34
mBERT+CosReg
93.96
85.75
88.60
87.70
74.28
73.30
78.67
83.18
&Noiy
93.69
86.59
88,22
88.02
76.42
74.23
79.85
83,86
mBERT+DA
93.47
88.3
88.27
88.33
77.90
77.04
80.71
84.86
mBERT+ours
94.16
87.53
89.70
89.35
78.06
77.30
80.91
85,29
mBERT+DA+ours
93.40
87.91
88,90
89.29
79.35
77.87
81.79
85,50
XLM-R
95.57
89.77
90.19
90.57
80.30
79.80
83.17
87.01
XLM-R+BN
95.46
90.38
90.82
91.02
80.63
80.25
83.59
87.45
XLM-R-
95.42
90.64
90.86
90.75
81.51
80.66
83.36
87.60
XI M--R-CosReg
95.59
90.79
90.99
91.75
80.95
80.22
83.82
87.73
XLM-R+ Looo---o
95.39
90.58
90.90
91.17
80.80
80.61
83.26
87.53
XI MM-R-DA
95.56
91.21
91.51
92.06
82.71
83.70
85.27
88.86
XLM-R+ours
95.64
91.37
91.98
92.61
83.19
83.15
85,25
89.03
XL. +O-R-
96.09
92.00
92.13
92.59
83.66
84,26
85.60
89.48
Table 7: Detailed results in different languages PAWS-X.
Models
en
bg
de
el
es
MBERT
82.36
65.21
69.03
71.54
67.30
74.84
74.00
60.09
mBERT+BN
82.61
65.04
68.89
71.53
67.27
74.99
74.44
60.07
MBERT+IsoBB.
82.38
64.98
69.25
71.46
67.37
74.55
74.06
60.56
mBERT+CosReg
82.68
64.89
68.36
71.55
67.17
74.48
73.99
59.91
mBERT+NoisyTum
82.73
65.03
68.89
71.72
67.05
74.57
74.07
60.74
MBERT+DA
81,63
64.73
68.48
71.22
66.39
74.77
73.71
60.08
mBERT+ours
82.73
65.47
68.74
72.34
67.67
75.67
61.62
mBERT+DA+OUrS
80.97
66.04
68.88
72.24
67.90
75.13
74.40
63.27
XLM-R
88.34
77.71
82.43
82.87
81.41
83.81
82.60
75.74
XLM--R M-MN--BBN
88,60
77.99
82.42
82.90
81.53
83,92
82.61
75.82
MLMM---soBBN
88.52
78.30
82.63
82.90
81.79
84.08
82.75
75.95
XLM-R RAMM-R-CoRReg
88.26
77.62
82.21
82.78
81.43
83.58
82.75
75.62
XLM-R+NoosyTTun
88.61
78.11
82.97
82.96
81.80
83,96
82.75
76.20
KLM-R-DA
88.13
80.35
84.61
83,28
83.04
84.80
83.82
77.85
XLM-R+ourS
89.13
80.33
84.06
83.55
82.84
85.22
83.79
77.90
XLM-R-DA+OUR
88.42
81.07
84.83
83.44
83.78
85.10
84.49
79.05
Models
th
tr
vi
zh
mBERT
69.28
49.42
53.56
61.06
58.10
70.07
69.24
66.34
mBERT+BN
69.50
49.61
52,91
61.51
58.15
70.24
69.09
66.39
mBERT+IsoBN
69.46
49.71
54.25
61.82
58.64
70.82
69.69
66.60
mBERT
68.92
50.19
53.36
61.41
57.83
70.27
69.19
66.28
mBERT NoossyTouu
69.45
49.04
54.02
61.00
58.14
70.57
69.58
66.44
mBERT+DA
69.60
48.66
53.89
60.70
59.78
70.58
70.58
66.32
mBERT+ours
69.15
51.35
55.03
63.57
58.68
71.80
70.55
67.27
mBERT+DA+OurS
69.26
50,90
54.89
62.96
60.92
72,28
72.06
67.47
XLM-R
79.94
71.38
76.41
78.39
71.49
79.22
78.51
79.35
XLM-R+BN
79.90
71.59
76.71
78.26
71.60
79.35
78.70
79.46
XLM-R- M-R-IsoBN
80.20
71.87
76.92
78.67
71.90
79.65
78.77
79.66
XLM-R- M-MM----RReg
79.65
71.34
76.45
78.38
71.36
79.28
78.34
79.27
XLM-R- M----NoiyyTuns
80.18
72.17
77.20
78.89
71.93
79.34
78.88
79.73
XLM-R+DA
81.70
73.25
79.06
79.91
74.64
81.49
80.57
81.10
XLM-R+ours
81.10
72.92
78.24
79.98
73.70
81.08
79.72
80.90
LLLM-RDDA+OURE
81.88
72.40
79.42
79.45
75.02
81.77
81.11
81.40
Table 8: Detailed results in different languages XNLI.

---

Models
de
fr
zh
mBERT
61.37
45.39
44.81
46.86
39.15
37.22
45.80
mBERT+BN
61.69
44.52
42.88
44.71
38.65 38.63
34.15
44.43
mBERT+1soBN
61.65
44.85
44.38
46.27
38.11
37.02
45.38
mBERT+CosReg
62.75
47.51
45.91
46.88
39.79
37.48
46.72
mBERT=NoisyTune
61.44
46.60
45.23
47.06
39.57
38.15
46.34
mBERT+DA
61.69
48.40
46.70
47.22
39.26
38.61
46.98
mBERT+ours
63.25
48.99
47.9
48.18
39.68
39.36
47.90
mBERT+DA+oUrS
62.60
50.58
48.14
48.51
39.84
39.63
48.21
XLM-R
65.64
63.62
57.68
58.34
55.13
52.85
58.88
XLM-R+BN
65.90
64.35
57.82
58.54
56.03
53.10
59.29
"LM-R+IsoBA
65.78
63.98
$7.58
58.74
55.88
$3.30
59.21
XLM R+CosReg
66.23
64.29
58.22
58.93
55.66
53.43
59.46
XLM-R+Nooss N+NosyTuue
65.41
63.90
57.72
58.28
56.06
$2.57
58.99
XLM-R+DA
66.62
64.37
58.54
58.80
54.68
53.87
59.48
XLM-R+ours
66.53
64.54
58.90
59.22
56.28
$2.81
59.71
XLM-R+DA
66.54
65.15
58.84
58.97
56.77
53.08
59.89
Table 9: Detailed results in different languages on MARC.


Reviews:
[{"review_id": "wP5GlVbrh5", "reviewer": "Reviewer_u81d", "summary": "", "questions": "1 - Although it is part of state-of-the-art baselines in the context of zero-shot cross-lingual transfer, it seems to me a bit confusing that you compare to the DA baseline. Your method seems perfectly compatible with theirs, and motivated by different purposes. As you are neither claiming or seeking for the state-of-the-art results *for zero-shot cross-lingual transfer in general* in this study, why did you include a comparison with this orthogonal method?\n\n2 - What are the final hyperparameters that performed the best in your search? In the paper, I could only find ranges, which in my opinion could harm reproducibility.\n\n3 - Did the choice of $\\lambda_1$ and $\\lambda_2$ matter? Would it make sense to try more extreme (<<1 or >1) values?\n\n4 - Several times along the paper, it is argued that isotropy regularization methods can degrade syntactical/downstream performance (e.g. L534-537). I may have missed it, but I could not find a reference for this in your paper. Are you referring to the Stable Anisotropic Regularization (https://arxiv.org/pdf/2305.19358.pdf) paper?\n\n5 - I feel that the point made in L537-539 (\"existing methods destroy the syntactic knowledge implied by original representations when applied to fine-tuning[, and ours didn't]\") is not convincingly addressed in the paper. Did you run experiments for Table 3 on the ablation models? If using the isotropy loss only degraded the performance, do you think it would consolidate this claim?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review comments and constructive suggestions.\nWe hope the information attached below can mitigate your concern and the revision plan can make this paper more soundful.\n\n- **Q1**: The necessity of two-stage training is unclear.\n\n  - **A1**: We have tried adding both losses simultaneously, but it was unsatisfactory. We split training into two stages for two reasons. First, these losses may interfere with each other because isotropy enhancement widens the distance between representations so that representations are distributed more evenly, while constrained code-switching makes multilingual representations with the same semantics close. When optimizing, achieving a delicate balance is challenging. Also, simultaneous optimizations add more cost (equation 6 has more computational cost than equation 10). In order to dispel your concerns, we do ablation experiments on PAWSX, as shown in the table below. We can observe that although the one-stage training has a significant performance improvement compared to the baseline, it still lags behind the two-stage training.\n|| En | De | Es | Fr | Ja | Ko | Zh | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| xlm-r | 95.97 | 89.77|90.19|90.57|80.30|79.80|83.17|87.01|\n|xlm-r+ours|95.64|91.37|91.98|92.61|83.19|83.15|85.25|**89.03**|\n|one-stage|95.78|91.00|91.46|92.29|82.06|81.50|85.41|88.50|\n\n- **Q2**: The empirical proof of the syntactic is insufficient.\n\n  - **A2**: Your constructive suggestion is very helpful to improve the soundness of our paper. We have reported the syntactic performance of three isotropy-enhancing methods in Section 2.2 (Figure 2). We complement the performance of BN and IsoBN as follows:\n|Model| Ar | En | Hi | Ru | Tr | Zh |\n| --- | --- | --- | --- | --- | --- | --- |\n|LAS|\n| XLM-R | 17.4|48.5|21.0|32.5|24.4|4.7|\n|+BN|4.5|19.2|5.7|7.4|9.4|4.1|\n|+IsoBN|4.8|18.1|6.4|7.7|10.7|3.8|\n|+ours|**23.6**|**54.6**|**21.7**|**34.6**|**26.9**|**5.8**|\n|UAS|\n| XLM-R | 32.4|55.7|37.5|44.3|41.2|17.7|\n|+BN|15.8|32.1|19.3|19.0|24.3|14.3|\n|+IsoBN|15.7|30.5|19.5|18.9|25.5|12.9\n|+ours|**37.2**|**61.0**|**39.5**|**47.0**|**43.8**|**18.8**|\n\n- **Q3**: Why you compare to the DA baseline?\n\n  - **A3**: We hope that our approach has the practicality to further promote the cross-lingual transfer capability of mPLMs, and DA is one of the strongest baselines currently available, so we compare with it and find that our method can be compatible with it to achieve better cross-lingual transfer performance.\n\n- **Q4**: What are the final hyperparameters that performed the best in your search?\n  - **A4**: To ensure the reproducibility of our method, we report the final hyperparameters in the table below:\n| | PAWSX |  | XNLI |  | MARC |  |\n| --- | --- | --- | --- | --- | --- | --- |\n||mBERT|XLM-R|mBERT|XLM-R|mBERT|XLM-R|\n|lr | 2e-5 | 5e-6|1e-5|5e-6|2e-5|2e-6|\n|$\\lambda_1$|0.5|0.5|0.5|1.0|0.5|0.5|\n|$\\lambda_2$|1.0|1.0|1.0|1.0|1.0|1.0|\n\n- **Q5**: Does the choice of $\\lambda_1$ and$\\lambda_2$ matter?\n  - **A5**: We report a more detailed study about the choice of$\\lambda_1$ and$\\lambda_2$. We can observe that our method is relatively robust to$\\lambda_1$ and$\\lambda_2$, but extreme (<<1 or >1) values will negatively affect performance.\n| $\\lambda_1$ & $\\lambda_2$| En | De | Es | Fr | Ja | Ko | Zh | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| 0.1 & 0.1 | 95.90|91.25|91.80|92.35|82.20|82.65|84.85|88.71|\n|0.5 & 0.5|95.76|91.13|91.75|92.54|82.30|83.06|85.31|88.84|\n|0.5 & 1.0|95.64|91.37|91.98|92.61|83.19|83.15|85.25|**89.03**|\n|1.0 & 0.5|95.76|91.39|91.88|92.51|82.63|82.85|85.18|88.88|\n|1.0 & 1.0|95.78|91.16|91.91|92.64|82.51|83.05|85.33|88.91|\n|5.0 & 5.0|95.65|90.44|91.08|91.68|81.09|80.86|84.54|87.90|\n\n- **Q6**: What are the references on isotropy regularization methods can degrade syntactical/downstream performance?\n  - **A6**: We mention the references in Section 2.2 (Line 152-162), the corresponding references are Rajaee and Pilehvar, 2021b and Zhang, 2022. Regarding the missing reference you mentioned, we think it is highly relevant to our topic and we will cite it in the revised version.\n\nReferences:\n\n[1] Sara Rajaee and Mohammad Taher Pilehvar. How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy. findings of EMNLP, 2021\n\n[2] Zhang H, Liang H, Zhang Y, et al. Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization. NAACL, 2022: 532-542.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well motivated and addresses a clear research problem. The method is novel and sensible. The experiments are thorough and the results are very promising. The authors successfully make the representation space more isotropic according to two metrics by using the Wasserstein distance, which may pave the way for further investigation of this technique in other works.\nThis paper is also relevant in the current discussion about whether isotropy is actually a desirable property of language models (https://arxiv.org/pdf/2305.19358.pdf), as it proves the point that CosReg can be outperformed both in isotropy enhancement and downstream performance.", "reasons_to_reject": "A weakness of the paper is the lack of convincing explanation of the reason why these two losses work well together. Considering that isotropic regularization alone already outperforms most of the presented baselines, it is not clear why a second loss is really required to prove the main point of the paper. The authors argue that their study shows how \"existing methods destroy the syntactic knowledge implied by original representations when applied to fine-tuning\", although they present no empirical proof of this fact. For instance, it would have been very convincing if this comment was supported by an evaluation of the syntactic capabilities of all isotropy-enhancing methods in Table 3.\n\nThe DA baseline is not really relevant in my opinion as it is somehow orthogonal to the submitted article, and can be combined with the method as shown in the tables. It is also not clear if the two-phase approach is required, as it is not proven in the ablation study.\n\nA weakness of the method itself, which is discussed in the limitations section, could be the overhead induced by the computation of the auxiliary losses. Depending on the batch size used and the embedding dimensions, the Wasserstein distance may be computationally costly, and it would be relevant to see the extent of this drawback measured in this paper.", "questions_to_authors": "1 - Although it is part of state-of-the-art baselines in the context of zero-shot cross-lingual transfer, it seems to me a bit confusing that you compare to the DA baseline. Your method seems perfectly compatible with theirs, and motivated by different purposes. As you are neither claiming or seeking for the state-of-the-art results *for zero-shot cross-lingual transfer in general* in this study, why did you include a comparison with this orthogonal method?\n\n2 - What are the final hyperparameters that performed the best in your search? In the paper, I could only find ranges, which in my opinion could harm reproducibility.\n\n3 - Did the choice of $\\lambda_1$ and $\\lambda_2$ matter? Would it make sense to try more extreme (<<1 or >1) values?\n\n4 - Several times along the paper, it is argued that isotropy regularization methods can degrade syntactical/downstream performance (e.g. L534-537). I may have missed it, but I could not find a reference for this in your paper. Are you referring to the Stable Anisotropic Regularization (https://arxiv.org/pdf/2305.19358.pdf) paper?\n\n5 - I feel that the point made in L537-539 (\"existing methods destroy the syntactic knowledge implied by original representations when applied to fine-tuning[, and ours didn't]\") is not convincingly addressed in the paper. Did you run experiments for Table 3 on the ablation models? If using the isotropy loss only degraded the performance, do you think it would consolidate this claim?", "paper_topic_and_main_contributions": "This paper introduces a novel fine-tuning method to improve downstream performance of multilingual models. The approach is based on two auxiliary losses. The first loss pushes isotropy by penalizing the Wasserstein distance between the in-batch output distribution and a normal Gaussian. The second loss enforces a high cosine-similarity between original and code-switched versions of input texts. The authors run fine-tuning experiments with two steps in which the classification loss, the isotropy loss and the code-switched loss are weighted differently.\nExtensive downstream results are presented for several datasets across various languages. The method seems to outperform related work by a substantial margin. The authors also run an ablation study and experiments to measure the syntactical performance of their models, and argue that the combination of their two losses allows isotropic distributions that don't discard syntactic information.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity, Hammerl et al. 2023, https://aclanthology.org/2023.findings-acl.439.pdf\n\nThis paper is very recent but deals with similar topics and has relatively similar conclusions. I wanted to make sure that you were aware of it.", "Typos_Grammar_Style_and_Presentation_Improvements": "### Typos\nL113: \"even uniformly\" => \"evenly\" or \"uniformly\"?\nL251: which control**s**\nL558: researcher**s**\n\n### Style\nL77: \"impedes\" is a bit strong, as the anisotropic XLM-R still has cross-lingual capacity. Maybe \"damages\" or \"hurts\" would be better?\n"}}, {"review_id": "ROJUBmMQ82", "reviewer": "Reviewer_7ZXD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review comments and acknowledgement.\n\nDue to limited computational resources and the time constraints during the rebuttal phase, we are unable to complete the experiments on the entire MARC dataset, and if the final paper is accepted, we will report the relevant experiments in the appendix of the camera ready version.\n\nThank you again for your recognition and suggestions for our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is well written and easy to follow\n* The method seems intuitive, code will be released (attached as supplementary) for easy reproducibility \n* Ablations are provided (combination of consistency and isotropic losses)\n* Additional analysis are provided, ie isotropy enhances syntactic properties of the model (table 3)\n", "reasons_to_reject": "* The results for MARC are reported using 25% of the training data and might not carry to the full data (the authors state they did not have enough compute resources), though for other datasets the authors use the full data so the issue is minor", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper explores isotropic representations for zero-shot cross lingual transfer. Anisotropic representations are defined as occupying a narrow cone in the vector space while isotropic ones are uniformly distributed. It is hypothesized that isotropic representations are helpful for multilingual models as they ensure better distribution of the word representations. The authors introduce a loss enforcing the isotropic property with a Wasserstein distance to a perfect isotropic distribution (normal distribution), this is combined with a \"consistency\" loss that enforces (sentence s, code switched s) pairs to be close by cosine distance (code switched s is synthesized with a bilingual dictionary). Results on 3 classification tasks (PAWS-X, XNLI and MARC) show improved performance over previous work. ", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kd3cviPUHQ", "reviewer": "Reviewer_AcYD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your valuable review comments. We hope the following response can address your concerns:\n\n- **Q1**: Different sets of methods is used for different experiments.\n\n  - **A1**: This suggestion is very helpful to improve the soundness of our paper, and due to rebuttal time constraints, for Fig. 3, we add the results of batch and IsoBN on mBERT, and the results on XLM-R, which we promise to give in the revised version.\n| $I_{cos}$ |de|en|fr|hi|ru|ar|sw|tr|vi|zh|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|mbert | 0.27|0.25|0.27|0.31|0.28|0.30|0.32|0.30|0.29|0.28|\n|+cosreg|0.00|0.00|0.00|0.02|0.01|0.01|0.01|0.02|0.01|0.01|\n|+batch|0.27|0.26|0.28|0.30|0.28|0.30|0.29|0.29|0.29|0.28|\n|+isobn|0.24|0.21|0.24|0.27|0.25|0.26|0.25|0.25|0.25|0.25|\n|+ours|0.05|0.01|0.05|0.07|0.04|0.07|0.09|0.07|0.06|0.06|\n|log$I_{pc}$ |**de**|**en**|**fr**|**hi**|**ru**|**ar**|**sw**|**tr**|**vi**|**zh**|\n|mbert|-6.3|-5.0|-5.5|-4.5|-5.0|-4.7|-6.2|-6.2|-5.2|-6.2|\n|+cosreg|-5.5|-6.3|-6.1|-5.4|-6.2|-6.3|-6.4|-5.2|-5.7|-5.5|\n|+batch|-6.3|-5.1|-4.8|-2.6|-4.8|-2.5|-4.8|-4.8|-5.1|-4.7|\n|+isobn|-4.9|-4.8|-4.9|-5.0|-5.3|-5.1|-4.7|-4.9|-4.9|-4.9|\n|+ours|-1.5|-1.8|-2.4|-1.8|-1.3|-2.6|-2.7|-1.8|-1.9|-1.9|\n\n- **Q2**: The hyperparameter tuning budget is more than baseline methods.\n\n  - **A2**: Undeniably, our method introduces some hyperparameters compared with the vanilla method, but we didn't spend much energy on the hyperparameter search. $\\lambda_1$ and $\\lambda_2$ are only carried out in {0.5, 1}. Meanwhile, we find that the experimental results of our method are more robust to these two hyperparameters within a reasonable range of values. Therefore, we think that in practical applications, our method does not need to spend too much time on the fine search of hyperparameters.\n|$\\lambda_1$ & $\\lambda_2$|En|De|Es|Fr|Ja|Ko|Zh|Avg|\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|0.1 & 0.1|95.90|91.25|91.80|92.35|82.20|82.65|84.85|88.71|\n|0.5 & 0.5|95.76|91.13|91.75|92.54|82.30|83.06|85.31|88.84|\n|0.5 & 1.0|95.64|91.37|91.98|92.61|83.19|83.15|85.25|89.03|\n|1.0 & 0.5|95.76|91.39|91.88|92.51|82.63|82.85|85.18|88.88|\n|1.0 & 1.0|95.78|91.16|91.91|92.64|82.51|83.05|85.33|88.91|\n|5.0 & 5.0|95.65|90.44|91.08|91.68|81.09|80.86|84.54|87.90|\n\n- **Q3**: Confidence intervals for the results are not reported.\n\n  - **A3**: Thank you for highlighting the importance of reporting confidence intervals for our results. We apologize for not including this crucial information in our initial submission. To rectify this oversight, we report the 95% confidence interval of our method below and we will report confidence intervals for all experiments in Table 1 in our revised version.\n||PAWSX|XNLI|MARC|\n| --- | --- | --- | --- |\n|mbert+ours|85.29$\\pm$0.29|67.20$\\pm$0.15|47.90$\\pm$0.78|\n|mbert+da+ours|85.50$\\pm$0.16|67.47$\\pm$0.13|48.21$\\pm$0.93|\n|xlmr+ours|89.03$\\pm$0.13|80.90$\\pm$0.56|59.71$\\pm$0.13|\n|xlmr+da+ours|89.48$\\pm$0.28|81.40$\\pm$0.08|59.89$\\pm$0.12|\n\n- Typos and presentation improvements:\n\n1. Regarding the concept of word embedding you mentioned, we are sorry for the misunderstanding we caused. Like Timkey and Schijndel (2021), we consider that word embedding includes contextual and non-contextual word representation. In order to avoid misunderstanding, we will replace ''word embedding'' with the precise ''contextual word/token representation''.\n\n2. The XLM-R we use is the xlm-roberta-large, and we will make this clear in the revised paper.\n\n3. The MUSE dictionaries is released by Conneau et.al, 2017\n\nWe surely will address other presentation issues and revise our paper accordingly. We are looking forward to seeing more inspiring feedback!\n\nReferences:\n\n[1] Timkey W, van Schijndel M. All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality. EMNLP 2021: 4527-4546.\n\n[2] Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. J´egou. Word translation without parallel data. arXiv:1710.04087, 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method seems to work and has beneficiary effects on both semantic and syntactic properties of the model.\n\n- The proposed method is compared to other SOTA methods in Section 4.4 and is able to beat them all.\n\n- The analysis section is quite strong and provide a lot of additional evidence for the proposed method.", "reasons_to_reject": "- Overall, different sets of methods is used for different experiments, e.g., compare methods used in Section 2.2, 4.3 and 5.3. It is hard to make a conclusion about the other methods when we are not presented with a comprehensive overview of their behavior in different circumstances. The gap is especially noticeable for the isotropy measures in Section 5.3, because the only baseline used is CosReg and the results for I_cos are discarded.\n\n- There is a =discrepancy in hyperparameter tuning budget for the baseline methods vs the proposed methods. If I understand it correctly, it has at least 4x more training runs as the baselines due to lambda_1 and lambda_2 tuning.\n\n- Confidence intervals for the results are not reported. This is important because the results are calculated from N=5 runs and they are within 1-2% of each other.", "questions_to_authors": "", "paper_topic_and_main_contributions": "I was reviewing this paper for the ACL 2023 as well, so I will reuse relevant points here.\n\nThe paper proposes a multilingual language model fine-tuning methodology that aims to improve the anisotropy of the token representations. The method is based on a loss function that explicitly increases the isotropy and a code-switching methodology that improve the consistency of the representations. Authors fine-tune two popular multilingual LMs (mBERT, XLM-R) and observe how the crosslingual performance and representations differ for up to 7 different training methodologies, including the proposed one. They conclude that the proposed methodology is the best, yielding the best crosslingual performance, while managing to preserve syntactic properties of the model (as showcased by dependency parsing probing). Other techniques (visualization, CKA, etc) were also used to strengthen the claims.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2 and 5 - This types of visualization with the points connected is better suited for time series.\n\nIn some cases, authors talk about \"word embeddings\". However, this term is usually used for non-contextual word representations (e.g. word2vec). In other cases they are talking about \"token representations\". Are these the same concepts?\n\nThe version of XLM-R is not reported (base vs large).\n\nL321 - I don't think the models were released by HuggingFace\n\nL140 W^T W is not an embedding matrix\n\nL337 - What exactly are MUSE dictionaries. The paper references the following paper. Is that it? Conneau, G. Lample, M. Ranzato, L. Denoyer, and H. J´egou. Word translation without parallel data. arXiv:1710.04087, 2017.\n\nL833 - Wrong accent in the name Libovický.\n\nOne appendix does not have a text at all, the other has a placeholder text only"}}]

Author rebuttal:

Meta-review: We thank the authors for their submission and engagement in the rebuttal. 

The paper presents a two-step fine-tuning method for better zero-shot cross-lingual transfer. 
The first step optimizes an isotropy-aware objective. The second step consists of code-switched data representation alignment based on cosine similarity. 

Reasons to Accept: 
- The introduced method improves cross-lingual transfer on three tasks and for two models. 
- Analysis of the internal representation pre- and post-isotropy enhancement supports the method's effectiveness. 

Reasons to Reject: 
- The boost in performance for some languages is minor and would benefit confidence interval reporting.
Decision: Accept-Findings