Analyzing and Reducing the Performance Gap in Cross-Lingual Transfer with Fine-tuning Slow and Fast
Year: 2023

Abstract:
Existing research has shown that a multilingual pre-trained language model fine-tuned with one (source) language also performs well on downstream tasks for non-source languages, even though no fine-tuning is done on these languages. However, there is a clear gap between the performance of the source language and that of the non-source languages. This paper analyzes the fine-tuning process, discovers when the performance gap changes and identifies which network weights affect the overall performance most. Additionally, the paper seeks to answer to what extent the gap can be reduced by reducing forgetting. Based on the analysis results, a method named Fine-tuning slow and fast with four training policies is proposed to address these issues. Experimental results show the proposed method outperforms baselines by a clear margin.

heading: 1 Introduction
text: Multilingual pre-trained language models (LMs), such as mBERT (Devlin et al., 2018) and XLMR (Conneau et al., 2019) have shown strong ZeroShot Cross-Lingual transfer capabilities. Such a model F is usually pre-trained with unlabeled corpora D in multiple languages S to enable the model to learn cross-lingual knowledge Hprecross. To adapt to a downstream task, the pre-trained LM F is typically fine-tuned with a supervised dataset Dŝ of the downstream task T in one source language ŝ ∈ S due to data scarcity in other languages. When the fine-tuned model F is applied to the test set of the same task in the source language ŝ, it achieves strong performance Pŝ. Interestingly, when F is applied to non-source languages, it can also achieve good performance (Conneau et al., 2019). We denote the average performance on the test sets of other languages than ŝ as PS/ŝ. However, the
*Work done during internship at Microsoft Research Asia †Corresponding author
gap Pŝ − PS/ŝ is quite large (e.g., 13 percent for the XNLI dataset in Figure 1). One potential reason is that: during the fine-tuning of the model, the performance of non-source languages firstly increases with the performance of source language, then the arising of the performance of non-source languages becomes slower than that of the performance of source language as the forgetting of crosslingual knowledge, resulting in a larger gap. Inspired by the study of catastrophic forgetting (CF) phenomenon in continual learning (CL), we introduce a classical concept in CL here to help solve our problem: the dilemma of plasticity vs. stability.
Plasticity vs Stability. In CL (Kirkpatrick et al., 2017), the learner needs to learn a sequence of different tasks incrementally. Plasticity means learning and performing well on the new task and stability means maintaining the learned knowledge of the previous tasks.The learner needs to find a balance between plasticity and stability because too much plasticity (e.g, changing the entire model drastically) causes serious CF of the learned knowledge, and too much stability (e.g. freezing the whole
4002
model) makes the model can not learn new things. Fine-tuning a multi-lingual LM F using only the corpus of one source language also meets this balance dilemma. Thus, Fine-tuning LM F needs to protect the cross-lingual knowledge Hprecross (stability) and also learn the new task knowledge Hnewtask via fine-tuning to adapt to the specific downstream task (plasticity). However, further analysis of the performance gap and the dilemma of plasticity and stability in cross-lingual fine-tuning is needed.
This paper further investigates three research questions: 1) When does the performance gap arise during fine-tuning using a labeled source language corpus? 2) Where is the most important part of the pre-trained model for achieving strong zero-shot cross-lingual performances? 3) To what extent can we reduce the performance gap by reducing the forgetting of Hprecross? Based on the experiments on three datasets of different downstream tasks, our analysis found that the performance gap arises significantly in the initial fine-tuning phase and increases slowly in the later phase (see Figure 2). Feed-forward weights in the bottom four layers are the key weights for the cross-lingual knowledge (See Figure 3 and Table 1) and should be updated slowly to avoid forgetting Hprecross. Attention weights in the top two layers have the pre-training task (e.g., Masked-Language Modeling) knowledge Hpretask and H pre task is useless for the downstream task. So these weights should be updated fast to encourage forgetting Hpretask. We also find that protecting the cross-lingual knowledge by freezing the weights related to it can reduce the performance gap (enough stability) but cannot eliminate the gap completely (See Figure 4 ). That means only reducing the forgetting of Hprecross is not enough for solving the performance gap.
Un-forgetting vs forgetting. Based on the above analysis, we propose a method called Finetuning slow and Fast algorithm to mitigate the forgetting of cross-lingual knowledge (stability) and also to selectively forget the knowledge related to the pre-training task (plasticity) to adapt F to the downstream task in fine-tuning F . Note that traditional techniques for solving the forgetting problem in continual learning are not applicable to our setting directly (see the reasons in Sec 5).
The proposed method consists of four learning rate policies. Policies I and II (stability policies) are respectively designed to avoid forgetting of Hprecross in the first fine-tuning stage and to avoid
the forgetting of Hprecross based on the tendency of the learning curve in the second fine-tuning stage. Policies III and IV (plasticity policies) are respectively designed to selectively forget the pre-training task knowledge in Hpretask in the initial fine-tuning stage where the loss drops drastically and to further encourage forgetting of the pre-training task knowledge Hpretask and the learning of H new task in the second fine-tuning stage. This paper’s main contributions are as follows: (1) We analyze the performance gap in crosslingual fine-tuning and answer to what extent we can reduce the performance gap by avoiding forgetting cross-lingual knowledge.
(2) We propose a method consisting of four learning rate policies to reduce forgetting of crosslingual knowledge (stability) and to encourage forgetting of pre-training task-related knowledge (plasticity).
(3) We test our method in multiple datasets under zero and few-shot settings. Compared to the baseline, our method reduces the performance gap (Figure 1(XNLI) and Figure 5 in Appendix A (MLQA and NER)) and achieves better overall performance (Table 2) by protecting the cross-lingual knowledge and learning better task representation.

heading: 2 Analysis of Performance Gap and Forgetting of Cross-Lingual Knowledge in Fine-Tuning
text: This section studies three research questions, i.e., when Pŝ − PS/ŝ happens and where the weights influence the overall performance mostly? It also answers to what extent we can reduce the performance gap Pŝ −PS/ŝ by reducing the forgetting of cross-language knowledge in Hprecross.

heading: 2.1 Overall Setup
text: We directly use the multilingual pre-trained model XLM-R (Conneau et al., 2019) as the base LM due to its strong zero-shot cross-lingual transfer performance. We consider the Cross-lingual Natural Language Inference (XNLI) dataset (Conneau et al., 2018) which is a cross-lingual textual entailment dataset (classification task). Multilingual Question Answering (MLQA) dataset (Lewis et al., 2019) which is a multilingual machine reading comprehension task and NER dataset (named entity recognition task) in XTREME benchmark (Hu et al., 2020b). The metric for MLQA and NER is the F1 score and the metric for XNLI is accuracy. All
results are the average of 4 random seeds. We use the zero-shot cross-lingual transfer setting with English as the source language for all experiments. More training details are in Appendix B.

heading: 2.2 When does the Performance Gap Arise?
text: We record the loss and calculate the performance gap on the validation set every hundred updates. Figure 2 shows that the occurrence of the performance gap can be divided into two phases: (1) In the first phase P1 (the first 20% of iterations), the performance gap occurs early and increases dramatically as the loss drops quickly in the initial training stage. (2) In the second phase P2 (the last 80% iterations), the gap increases but is obviously slower than in the first phase and the loss drops slowly.

heading: 2.3 Where is the Knowledge that Helps Cross-Lingual Transfer?
text: We use freezing and re-initializing functions to investigate the influence of the weights of each layer on overall performance. Note that we only choose one layer to do re-initializing/freezing operations in each experiment. Figure 3 shows that the cross-lingual knowledge Hprecross widely exists in the first 10 layers as re-initializing the weights in any of the ten layers causes performance drop and is mainly located in the first four layers as re-initializing the weights in one of the first four layers makes the performance drops obviously and freezing them boosts the performance. Also interestingly, the pre-trained knowledge in the last two layers has little influence on performance. Sometimes re-initializing one layer in the two layers even makes the performance better than the baseline performance (e.g., for the MLQA dataset). That is because the task of pre-training (e.g. Masked Language Model task) is different from the downstream task and mainly located in the last two layers. We call this kind of knowledge learned from
the pre-training task the pre-training task knowledge Hpretask, which can have a negative transfer to the downstream task.
2.4 How much can We Reduce the Performance Gap by Reducing Forgetting?
To study this question, we fine-tune only the last one/two/three layers to provide strong stability for Hprecross. Figures 2 and 4 show that fine-tuning only the last few layers delays the first appearance of the performance gap and clearly decreases the performance gap. Also, the fewer layers are fine-tuned, the smaller the gap is. However, (1) a great gap still exists even if we only fine-tune the last layer (e.g., 9% difference on the XNLI dataset). That means avoiding the forgetting of the pre-trained Hprecross can reduce the gap to some extent, but cannot solve the problem entirely. (2) Fine-tuning fewer layers makes the overall performance drops significantly as the model does not have enough space to learn Hnewtask (see Table 1). That means a smaller performance gap is not equal to better overall performance and we need to consider the plasticity too.

heading: 3 Method
text: This section proposes a method to avoid the forgetting of cross-lingual knowledge Hprecross (stability) and to encourage the forgetting of task knowledge for the pre-training task Hpretask and learn new task’s knowledge Hnewtask (plasticity). The core is to set different learning rate policies for the weights of the model based on both the layer’s location and the training phase.

heading: 3.1 Reducing Forgetting of Cross-Lingual Knowledge with Fine-tuning slow
text: We consider the protection of Hprecross first. The key challenge here is to strike a balance between
maintaining Hprecross and learning Hnewtask . Based on the above analysis, we propose a fine-tuning slow algorithm consisting of the following two training policies and apply them to different sets.
Policy I: Avoiding drastic update of weights related to cross-lingual knowledge in the first fine-tuning phase P1. The performance gap increases quickly in P1 and H pre cross in weights are forgetting quickly. That is because the loss in this phase drops drastically and gives a big gradient update for each weight to update, and the stability for Hprecross is not enough. So our goal here is to reduce the update of weights related to Hprecross in this stage by multiplying their learning rate with a
learning rate multiplier K = c1 (c1 < 1).
Policy II: Adjusting the learning rate of the key weights for cross-lingual transfer dynamically in the second fine-tuning phase P2. After the first phase, the gap increases slowly and our goal is to make the weights adapt to the downstream task and to avoid forgetting cross-lingual knowledge. So we set K = 1 for weights related to Hprecross to provide more plasticity and dynamically adjust the learning rate of the key weights related to Hprecross additionally to avoid the forgetting. Our idea for the key weights is to provide more plasticity for them when the loss drops quickly to learn a new task and to provide more stability (avoid-
ing unnecessary forgetting) when the loss drops slowly. We propose to set a dynamic multiplier K based on the learning curve for the key weights of Hprecross. Assume that Lt is the training loss at the tth iteration and ϕ(Lt) ∈ [0, 1] is a function reflecting the tendency of the learning curve. The bigger the ϕ(Lt) is, the faster the loss drops. Then we have K = R(ϕ(Lt)), where R is a monotonic function. In this way, when the loss of the model drops quickly to adapt to the new task, K is also bigger to provide more plasticity. Note that policy II in P2 has no conflict with policy I as the drastic loss drop in P1 is not desirable for the weights related to Hprecross to adapt to the task.
Layers to apply policies I and II. If reinitializing the weights in one layer obviously drops the performance across three datasets, we denote the weights in this layer belong to SIθ . If freezing the weights in one layer improves the performance, we denote the weights in this layer belong to SIIθ (S II θ ∈ SIθ ). The latter is usually more important for cross-lingual transfer. Based on the re-initializing/freezing experiment (see Figure 3), we know that weights in the first 10 layers belong to SIθ and weights in the first 4 layers belong to SIIθ . For policy I, we apply it to S I θ as we do not want to forget cross-lingual knowledge Hprecross due to the big gradient updates.
Attention vs Feed-forward To further investigate the best choice of protecting weights in the first four layers, we conduct experiments to freeze all weights/all weights of the multi-head layer/all weights of the feed-forward layer in the first four layers. The results in the second part of Table 1 show that freezing all weights of the feed-forward layer in the first four layers achieves the best performance over three datasets. With the additional reinitialization experiments (see Table 6 in Appendix C), we find that is because the weights of the feedforward layer are the most important weights for cross-lingual transfer in the first four layers. So we apply policy II to the weights of the feed-forward layer in SIIθ as we want to protect H pre cross and to provide more plasticity to learn Htasknew .

heading: 3.2 Encouraging Forgetting of Pre-training Task Knowledge with Learning Fast
text: As shown earlier, the pre-training task knowledge Htaskpre is usually useless or even harmful to the downstream task. Here we design an algorithm to utilize big gradient updates (naturally happen in the
first phase or are created by enlarging the learning rate) to encourage the model to forget Htaskpre and to learn better downstream task’s knowledge Htasknew . We refer to this as the fine-tuning fast algorithm consisting of two training policies and apply them to different sets:
Policy III: Do not slow down the update of the weights related to Htaskpre in the first fine-tuning phase P1. In P1, the model is actively looking for a point that can reduce the loss drastically and has enough energy to break the limitation of the pretrained knowledge Htaskpre . So we allow the model to update the weights related to Htaskpre in this phase without lessening their learning rate.
Policy IV: Increasing the learning rate of the key weights related to Htaskpre in the second finetuning phase P2. In P2, the loss drops gradually and the model finally needs to converge to local minima. But the model may not stop learning the new task’s knowledge Htasknew . To verify this, we use the representation similarity metric CKA (Kornblith et al., 2019) to measure the similarity of the representation of the current training data batch to the pre-trained model and to the current training model. Figure 6 in Appendix D shows that the similarity of the hidden representation from the last two layers is still dropping in the second phase (except the NER dataset) and the model is striving to learn a better task representation that is different from the pre-trained one. But the loss becomes small and drops slowly in P2 and the model doesn’t have enough energy (Pezeshki et al., 2021) to forget the pre-training task knowledge and to learn Htasknew . So if the representation similarity of the last two layers is still dropping in P2, we encourage the model to update the key weights relevant to the task knowledge by multiplying their learning rate with a learning rate multiplier K = c2 (c2 > 1).
Layers to apply policies III and IV. Based on the re-initializing experiment (Figure 2), we know that re-initializing the weights in the last two layers improves the performance or drops the performance slightly. That means that the weights in the two layers have little cross-lingual knowledge and have Htaskpre which has a negative effect on the learning of the downstream task. We denote the set of weights that has this property as V Iθ and apply policy III to it.
Attention vs Feed-forward In the second phase, the model is trying to learn a better task representation and needs to converge to a stable point. So
enlarging the learning rate of all weights in V Iθ may not be the best choice (e.g., disturbing the convergence). To investigate the best choice of weights in the last two layers, we conduct experiments with an increased learning rate of different weight sets. Based on the results of the third part of Table 1, we find that increasing the learning rate of all weights in the attention layer of the last two layers achieves the best performance. That implies the weights of the attention layer are the key weight in the learning of the downstream task and that not changing the learning rate of other weights in the last two layers provides much stability. So we denote the weights of the attention layer in V Iθ as V II θ and apply policy IV to it.

heading: 3.3 Fine-tuning slow and Fast Algorithm
text: Formally, a typical multi-lingual pre-trained model F comprises a stack of L transformer layers with each layer containing an attention head layer laθ followed by a feed-forward network lfθ . At the tth training iteration of the fine-tuning process, the updating rule of the weight θt of model F based on our fine-tuning slow and fast algorithm is:
θt =    θt−1 −K · r∇θt−1 if t ∈ P1 ∧ θ ∈ SIθ || tinP2 ∧ θ ∈ V IIθ ⋃ SIIθ
θt−1 − r∇θt−1 otherwise (1)
where r is the learning rate and ∇θt−1 is the weight modification calculated by the backpropagation algorithm. SIθ , S II θ , V I θ and V II θ are the weight sets for the application of policy I, II, III, and IV respectively. We use t ∈ P1 to identify if the t-th iteration belongs to the first phase P1. The learning rate multiplier K is determined by:
K =   
c1 if t ∈ P1 ∧ θ ∈ SIθ c2 if t /∈ P1 ∧ θ ∈ V IIθ R(ϕ(Lt)) if t /∈ P1 ∧ θ ∈ SIIθ
(2)
where c1 and c2 are constant and R(ϕ(Lt)) is a monotonic function based on the function ϕ(Lt) that can reflect the tendency of the learning curve. Our method maintains the stability for cross-lingual knowledge and increases the plasticity to adapt to the new task. We verify it in the following section.

heading: 4 Experiment
text: We now use three downstream tasks: Named Entity Recognition (NER), Question Answering (QA), and Natural Language Inference (NLI), to experimentally evaluate the performance of our proposed
Fine-tuning slow and fast algorithm under the zero-shot and few-shot settings.

heading: 4.1 Experiment Setup
text: Datasets: We adopt the NER (Hu et al., 2020b), MLQA (Lewis et al., 2019), and XNLI (Conneau et al., 2018) datasets from the XTREME benchmark (Hu et al., 2020b) for NER, QA, and NLI respectively. The details of the datasets and training details are listed in Section 2.
Zero-shot and Few-shot settings. We define the zero-shot setting as fine-tuning a pre-trained model for a downstream task using its labeled data in one source language (e.g. English). Then we apply the fine-tuned model to all target languages. We define the few-shot setting as fine-tuning a pre-trained model for a downstream task using its labeled data in one source language (e.g., English) and a few labeled data from other languages. All labeled data are mixed to form a training dataset and then we use it to fine-tune the pre-trained model. For the source of the few-shot data, we split the original validation set into the few-shot data group and the new validation set. Note that the number of data points in the validation set is usually larger than 5000. So extracting the few-shot data from the validation set does not influence its validation function.
Baselines: (1) Directly Fine-tuning (DF) the model with the English training corpus; (2) NoisyTune (Wu et al., 2022), which prevents LMs from overfitting the data in pre-training and reducing the gap between pre-training and downstream tasks by adding a small amount of noise to perturb the LM parameters before fine-tuning. (3) Fine-tuning slow algorithm (FS), which fine-tunes the model with the fine-tuning slow algorithm. (4) Learning Fast algorithm (FF), which fine-tunes the model with the fine-tuning fast algorithm.
Choice of adaptive multiplier R(ϕ(Lt)) and t ∈ P1, and hyper-parameters: For R(ϕ(Lt)) in Eq. 2, we first calculate the average value of the losses in the recent 100 iterations as Lt−100:t100 and the losses in the 100 iterations prior to the recent 100 iterations as Lt−200:t−100100 . Then we define ϕ(Lt) = Lt−100:tLt−200:t−100 . When the loss drops quickly (Lt−200:t−100 ≫ Lt−100:t), ϕ(Lt) is close to 0. And when the loss drops slowly, ϕ(Lt) is close to 1. We do not use LtLt−1 to represent ϕ(Lt) as the losses in adjacent iterations usually do not have a big difference and so it cannot accurately describe the tendency. Then R(ϕ(Lt)) is defined
as: R(ϕ(Lt)) = max(1− ϕ(Lt)r, 0) (3)
where r is a hyper-parameter. When the loss drops quickly, R(ϕ(Lt)) is close to 1 and gives the parameters more plasticity to adapt to the new task and vice versa. We choose Lt−200:t−100−Lt−100:t100 > 0.1 to represent t ∈ P1 in Eq. 1 as the loss drops very quickly in this case and the model needs policies I and III to protect Hprecross and learn Hnewtask . We set r as 3. We set c1 and c2 (Eq. 2) as 0.01 and 10 respectively. The ablation study is in Section 4.6.

heading: 4.2 Results of Zero-Shot Fine-Tuning
text: To evaluate the zero-shot performance of our method and baselines, we record the average F1 score performance (mean and standard deviation) of all target languages for the MLQA and NER datasets and the average accuracy for the XNLI dataset. The results are reported in Table 2, which shows that our method achieves highly superior results to the baseline methods.

heading: 4.3 Results of Few-Shot Fine-Tuning
text: The results of few-shot fine-tuning performance are reported in Table 2, which shows that:(1) with
the increasing number of few-shot data per language, the performance improves as the model can learn better cross-lingual task representation from the multi-lingual training corpus. (2) Our method still outperforms baselines obviously as it protects the pre-trained cross-lingual knowledge Hprecross and forgets the pre-training task’s knowledge Hpretask.

heading: 4.4 Large Training Corpus Fine-Tuning
text: We collect and construct a QAM task dataset (Liang et al., 2020) with 12 million English training data points by a business search engine. QAM classification task aims to predict whether a <question, passage> pair is a QA pair. Zero-shot fine-tuning model on a corpus like this is more challenging as the model is easier to forget Hprecross. From Table 3, we observe that our method outperforms the baseline obviously, which shows that our method also works well with a large dataset. From Figure 7 in Appendix E, we find that (1) the performance of non-source languages firstly increases and then drops (due to forgetting) (2) the performance of source language increases during the whole training process and the gap becomes larger in the later phase. (3) our method reduces the gap by protecting cross-lingual knowledge and improves the performance of non-source languages. More analyses and training details are in Appendix E.

heading: 4.5 Analysis of the Influence on Source Language and Non-Source Language
text: Our method improves both source and non-source languages’ performance in Table 4. Additionally, it helps non-source languages more, as it protects the cross-lingual knowledge in Hprecross, leading to a significant improvement in non-source languages’ performance. We report each language’s performance on the XNLI dataset in Appendix F.

heading: 4.6 Analysis of the Influence of Learning Rate
text: Multiplier Hyper-Parameters c1, c2 and r
We ablate on the learning rate multiplier’s hyperparameters c1, c2 (Eq. 2) and r(Eq. 3). We analyze their influence by setting different values for them and recording the final overall performance. From Table 5, we observe that reducing the value of c1 from 0.5 to 0 improves performance initially (by protecting Hprecross) but then decreases performance (due to lack of enough plasticity). We set c1 as 0.01 as it strikes a balance between stability and plasticity. Increasing the value of c2 from 5 to 100 improves performance initially (by learning better Hnewtask) but then decreases performance (due to lack of stability to converge). So we set c2 to 10. Increasing r from 1 to 4 improves performance initially (as the adaptive multiplier is closer to 1 and provides more stability) but then decreases performance (due to lack of enough plasticity). So we set r as 3. Our choice is consistent across the three datasets, showing our method’s robustness.

heading: 4.7 Analysis of the Influence of the Usage
text: Order of Weight Sets SIθ , S II θ ,V I θ and V II θ .
To further verify the effectiveness of our method in applying different weight sets for each policy, we conduct experiments that apply both Policies I and II to only SIθ /S II θ respectively, as well as experiments that apply Policy I to SIIθ and Policy II to SIθ . We also perform similar experiments for sets V Iθ and V II θ . From Table 8 in Appendix G, we find that our method achieves the best performance by making a good balance between plasticity and stability. Further analyses are in Appendix G.

heading: 5 Related Work
text: Fine-tuning multilingual language models (MLLMs). Recent MLLM systems (e.g., mBERT (Devlin et al., 2018), mT5(Xue et al., 2020), and XLM-R (Conneau et al., 2019)) have shown strong zero-shot transfer ability to non-source languages when fine-tuning with only a source language
corpus. However, the performance gap between the source language and non-source languages is still large. Most previous works focus on learning robust task representation(Fang et al., 2021; Zheng et al., 2021; Jiang et al., 2022) and strong pre-trained cross-lingual representation (Chi et al., 2020; Wang et al., 2020; Ouyang et al., 2020; Hu et al., 2020a). But they haven’t analyzed the performance gap in fine-tuning and the relation between forgetting and the gap. We fill this research gap.
Forgetting in continual learning. Continual learning aims to design algorithms to learn tasks incrementally. Its main challenge is forgetting. Many methods have been proposed to reduce forgetting. (1) Regularization methods(Kirkpatrick et al., 2017; Chen et al., 2020; Li et al., 2022; Lee et al., 2021) penalize the changes on important weights for previous tasks, (2) Replay methods(Buzzega et al., 2020; Rolnick et al., 2019; Wang et al., 2022) re-train a few previous samples with new task’s data, and (3) Parameter-fixed methods(Vidoni et al., 2020; He et al., 2021; Xu et al., 2021) protect parameters learned for previous tasks. The regularization method needs to estimate the important weight during pre-training and the replay method needs to store data from the pre-training corpus. But we usually don’t have those during the fine-tuning phase. Moreover, all of those methods focus on avoiding forgetting but we find that pre-training task knowledge is not suitable for the adaptation of downstream tasks and those methods cannot get rid of it. We propose a novel method that controls the forgetting effect to avoid the forgetting of cross-lingual knowledge and encourage the forgetting of pre-training task knowledge to learn better new task knowledge. Our method is orthogonal to previous works in cross-lingual finetuning.

heading: 6 Conclusion
text: This paper first analyzed when the performance gap arises and where the important cross-lingual knowl-
edge is and reduced the lower bound of the performance gap by avoiding forgetting cross-lingual knowledge. Based on our analysis, a novel method is proposed to control the forgetting effect in finetuning a multi-lingual pre-trained model. We verify its effectiveness over multiple datasets and settings.

heading: 7 Limitations
text: Although we believe that controlling the forgetting in the fine-tuning phase to avoid forgetting cross-lingual/general knowledge and to reduce the negative interference from misaligned pre-training tasks and downstream tasks can benefit other finetuning settings (e.g. Multi-task setting), we have not yet investigated these settings. In the future, we will try to propose a more general method for fine-tuning a large pre-trained model across various settings.

heading: A Performance gap of the MLQA dataset
text: and NER dataset
From Figure 5, we observe that (1) our method delays the first appearance of the performance gap. (2) our method has a lower performance gap compared to the baseline.

heading: B Details of the datasets and model.
text: XNLI is a cross-lingual textual entailment dataset. In this dataset, we use the MultiNLI (Williams et al., 2017) training data (English) to fine-tune the pre-trained model and then test the fine-tuned model with all 15 languages.
MLQA is a multilingual machine reading comprehension task for question answering. The test performance gap is the gap between the F1 score of the source (English) and the average F1 score of the other six target languages (Arabic, German, Spanish, Hindi, Vietnamese and Chinese).
NER is a named entity recognition task and we use the Wikiann (Pan et al., 2017) dataset. The metric is the F1 score. We use the balanced train, dev, and test splits in (Rahimi et al., 2019).
Following (Hu et al., 2020b), the fine-tuning batch size is 32. We use the Adam optimizer with warm-up and learning rate 5e-6. For XNLI, we fine-tune the model with the English corpus for 10 epochs and evaluate it on the English dev set every 3k steps to select the best model. For NER, we fine-tune 20 epochs. For MLQA, we follow BERT (Devlin et al., 2018) for SQuAD (Rajpurkar et al., 2016) and set the learning rate to 3e-5, batch size to 12, and we train the model for 2 epochs.
We select the model with the best of the average result on the dev sets of all languages.
XLM-R has 550 million parameters and we run the experiments with GPU A100.

heading: C Re-initialization experiments for the weights in the first four layers
text: From Table 6, we find that (1) re-initializing weights in the first four layers reduces the overall performance as the cross-lingual knowledge is lost. (2) Re-initializing the feed-forward weights in the first four layers has a worse effect on the overall performance than re-initializing the attention weights in the first four layers. That means the feed-forward weights in the first four layers are more important than the attention weights in the first four layers for cross-lingual transfer.

heading: D Figure for the CKA representation similarity
text: Figure 6 describes the similarity of the hidden representations from the pre-trained model and the working model. We observe that the bottom layers usually have a higher similarity than the top layers, indicating that the top layers need larger adjustments to adapt to the downstream task. Also, we find that the similarity of the last two layers continues to decrease in the second phase over the MLQA and XNLI datasets, indicating that the model is still trying to learn a better task representation by modifying the weights in the last two layers during the second phase.

heading: E The performance analysis of the QAM dataset
text: We use the Adam optimizer with warm-up and learning rate 5e-6 to fine-tune the model with the English corpus for 1 epoch as the training corpus is big enough for the model to achieve the best performance by running one epoch. The batch size is 32. We select the model with the best of the average result on the dev sets of all languages (every 3k updates).
We record the performance gap and each language’s performance on the dev set every thousand updates and report the curve in the first 120k iterations (Figure 7) as the best overall performance model is selected in the first 120k iterations. In the later iterations, the gap rises and the overall performance drops. As shown in Figure 7, the tendency of the gap curve is monotonically increasing during the whole training process, and the main reason for this is the decline in the performance of the non-source languages. Our method improves overall performance by reducing the forgetting of cross-lingual knowledge.

heading: F Analysis of the influence of our method for each language
text: From Table 7, we find that the fine-tuning slow algorithm improves almost all languages’ performance as it protects the cross-lingual knowledge. The fine-tuning fast algorithm (learning the new task knowledge Hnewtask) also improves the performance of the source language and some non-source languages as it provides a better task representation. Our method achieves the best performance over all languages, especially for some low-resource languages (e.g., sw and ur).

heading: G Analysis of the influence of SIθ ,SIIθ ,V Iθ , and V IIθ
text: From Table 8, we observe that the ’Only SIθ /S II θ ’ experiments achieve worse performance than our method as they lack enough plasticity/stability for the cross-lingual knowledge, respectively. The ’SIIθ to S I θ ’ experiment achieves the worst performance as it lacks enough stability in the first phase and lacks enough plasticity in the second phase. The ’Only V Iθ /V II θ ’ experiments do not provide enough space for learning new tasks in the first phase/stability to converge in the second phase, respectively, so their performance is worse. The ’V IIθ to V Iθ ’ experiment has both disadvantages of the ’Only V Iθ /V II θ ’ experiments, so its performance is
also poor. We did not apply policy IV on the NER dataset, so we did not record the result of the ’V IIθ to V Iθ ’ experiment on the NER dataset.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
A4. Have you used AI writing assistants when working on this paper? Not applicable. Left blank.

heading: B Did you use or create scientific artifacts?
text: Not applicable. Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix 2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix 2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will try to propose a more general method for fine-tuning a large pre-trained model across various settings