Venue: NAACL 2024

# FUN with Fisher: Improving Generalization of Adapter-Based

Cross-lingual Transfer with Scheduled Unfreezing

Chen Cecilia Liu\({}^{1}\), Jonas Pfeiffer\({}^{2}\), Ivan Vulic\({}^{3}\), Iryna Gurevych\({}^{1}\)

###### Abstract

Standard fine-tuning of language models typically performs well on _in-distribution data_, but suffers with generalization to _distribution shifts_. In this work, we aim to improve the generalization of adapter-based cross-lingual task transfer where such cross-language distribution shifts are imminent. We investigate scheduled unfreezing algorithms--originally proposed to mitigate catastrophic forgetting in transfer learning--for fine-tuning task adapters. Our experiments show that scheduled unfreezing methods close the gap to full fine-tuning and achieve stronger cross-lingual transfer performance, suggesting that these methods can go beyond just mitigating catastrophic forgetting. Next, aiming to understand these empirical findings, we investigate the learning dynamics of scheduled unfreezing using Fisher Information. Our experiments reveal that scheduled unfreezing induces different learning dynamics compared to standard fine-tuning, and provide evidence that the dynamics of Fisher Information during training correlate with cross-lingual generalization performance. We additionally propose a general scheduled unfreezing algorithm that achieves an average of 2 points improvement over four datasets compared to standard fine-tuning and provides empirical evidence for a theory-based justification of the heuristic unfreezing schedule for task adapter training. 1

Footnote 1: [https://github.com/UKPLab/nacl2024-fun](https://github.com/UKPLab/nacl2024-fun)

\({}^{1}\)Ubiquitous Knowledge Processing Lab,

Department of Computer Science and Hessian Center for AI (hessian.AI),

Technical University of Darmstadt

\({}^{2}\)Google DeepMind

\({}^{3}\)Language Technology Lab, University of Cambridge

www.ukp.tu-darmstadt.de

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 1998–2015

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

In the standard cross-lingual task transfer setup, a typical and often valid assumption is that only English data is available for fine-tuning and validation of a pretrained multilingual model, due to resource constraints in many languages Hu et al. (2020). However, models trained in this setup also need to generalize well to text inputs provided in other languages: a requirement that can be seen as an extreme but natural case of _distribution shifts_ generalization Ramponi and Plank (2020).

Parameter-efficient fine-tuning methods such as adapters Houlsby et al. (2019); Stickland and Murray (2019); Bapna and Firat (2019) with separate language and task components are often used to achieve effective cross-lingual transfer, especially to low-resource languages Pfeiffer et al. (2020); Ansell et al. (2021); Parovic et al. (2022). These adapters insert a small number of trainable parameters into a frozen pretrained multilingual language model (e.g., mBERT, Devlin et al., 2019; XLM-R, Conneau et al., 2020) to achieve positive transfer while avoiding catastrophic forgetting CF McCloskey and Cohen (1989) of previously learnt knowledge after adapting to new tasks.2 In other words, adapters enable _catastrophic forgetting free_ learning, referred to as _CF-free_ in this paper. While efficient, adapter methods often incur a cross-lingual performance gap when compared to full model fine-tuning.

Footnote 2: Adapters bypass this issue since the pretrained model is kept frozen, with its original weights unchanged.

_Gradual unfreezing_ (GU) is a technique which unfreezes layers of deep neural network models from top to bottom during training Howard and Ruder (2018). GU was previously proposed for general transfer learning of in-distribution (ID) data in monolingual contexts in NLP, and has been predominantly applied to full fine-tuning. More recently, 'Linear-Probing-then-Fine-Tuning' LPFT, Kumar et al. (2022) was proposed for transfer learning of both ID and distribution-shifted evaluation data using full fine-tuning in computer vision. LPFT first trains the classification layer only, and then the full model. The main notion connecting these methods is training different layers of a neural network by unfreezing layers at different times (i.e., with a _schedule_). Designed to mitigate CF, these _'scheduled unfreezing'_ methods have shown promising transfer learning results. However, it is

unclear whether scheduled unfreezing can do more than just mitigate CF, and benefit CF-free methods and cross-lingual transfer (which is a different type of distribution shift than previously studied).

In this work, we begin by asking the following question: Do scheduled unfreezing methods improve cross-lingual transfer and close the gap to full fine-tuning in the CF-free setting? We use scheduled unfreezing to train task adapters following the standard adapter-based cross-lingual transfer setup of Pfeiffer et al. (2020). We find that scheduled unfreezing enhances generalization, bridging the gap to full fine-tuning, confirming our hypothesis that, since cross-lingual transfer can be seen as a form of distribution shifts, methods such as GU are effective, even in the CF-free setting. Our results suggest that there indeed is more to the original scheduled unfreezing training than just mitigating catastrophic forgetting Howard and Ruder (2018).

We further analyze the learning dynamics during training, with a particular focus on GU, using the trace of the Fisher Information Matrix Kleinman et al. (2023), denoted by \({\rm tr}(F)\) henceforth. Our experiments reveal **1)** that scheduled unfreezing changes the dynamics of \({\rm tr}(F)\) during training, **2)** that \({\rm tr}(F)\) is a potential proxy for studying cross-lingual generalization.

Based on our analysis, we then propose an automatic scheduled unfreezing algorithm based on maximizing the \({\rm tr}(F)\) (termed **F**isher **U**nfreezing or FUN), to generalize from previous heuristic-based methods. FUN achieves comparable results to heuristic-based methods and provides empirical evidence that GU may implicitly maximize \({\rm tr}(F)\) during training in our experimental setting.

Contributions.In sum, our contributions are as follows. **1)** To the best of our knowledge, we are the first to demonstrate that scheduled unfreezing in adapter training for cross-lingual transfer closes the performance gap to full model fine-tuning. **2)** We present a generalized scheduled unfreezing framework that encompasses several existing methods, allowing easy extensions to new algorithms. **3)** We demonstrate that Fisher Information is an effective tool for studying generalization in cross-lingual transfer and find that the dynamics of Fisher Information correlate with cross-lingual transfer results. **4)** We propose a \({\rm tr}(F)\)-based scheduled unfreezing method (FUN), which achieves comparable performance to heuristic methods and indicates that GU achieves good generalization in adapter training.

## 2 Related Work

Scheduled UnfreezingHoward and Ruder (2018) propose Gradual Unfreezing (GU) to mitigate catastrophic forgetting when transferring a pretrained model for monolingual downstream tasks in non-Transformer architectures. Raffel et al. (2022) study GU for transferring full fine-tuning Transformers to in-distribution tasks, and empirically conclude that GU may not be effective. Recently, Kumar et al. (2022) (LPFT, Linear Probing Fine-Tuning) show promising results for distribution-shifted transfer in the full fine-tuning setting for computer vision tasks. Yang et al. (2022) extend LPFT for full fine-tuning to NLP. Lee et al. (2023) show that fine-tuning different parts of a network helps with generalization under different types of distribution shifts in computer vision. Different from our work, the prior work has not studied scheduled unfreezing methods for cross-lingual transfer in a CF-free setting.

Fisher Information and Generalization.Fisher Information Fisher (1925) is an established concept in optimization theory and practice Amari (1998), e.g., to measure parameter importance Kirkpatrick et al. (2017) or for pruning Singh and Alistarh (2020). Achille et al. (2019) study learning dynamics3 in neural network training with Fisher Information. Golatkar et al. (2019) show that Fisher Information correlates with generalization in computer vision and non-Transformer architectures. Jastrzebski et al. (2021) propose to regularize Fisher Information during the training of a neural network for better in-distribution generalization in computer vision. Xu et al. (2021); Sung et al. (2021) create sparse masks using Fisher Information for better parameter-efficient tuning. Concurrently, Lodha et al. (2023) uses Fisher Information selecting layers for fine-tuning transformers. We study Fisher Information in adapters and for cross-lingual transfer, which has the potential to guide the understanding of new methods in this area.

Footnote 3: Different from training dynamics Swayamdipta et al. (2020) which focus on the data, learning dynamics focus on the model and the optimization process.

## 3 Methodology

### Adapter-Based Cross-lingual Transfer

Adapters are components that are inserted into a multilingual pretrained Transformer model (termed 

the _base model_) to efficiently adapt the large base model to a specific language Pfeiffer et al. (2020) or task Houlsby et al. (2019).

The prior adapter-based fine-tuning process typically spans two stages for cross-lingual _task_ transfer. First, different language adapters are trained (often separately, per language) with the base model frozen, using the masked language modelling (MLM) objective in a target language. Second, task adapters and a task-specific output head are randomly initialized and inserted into the base model along with now-trained source language (i.e., typically _English_) adapters. In this stage, only the task adapters and a task-specific output head are trainable, while all other parameters are kept fixed. At inference time, the source language adapters are replaced by the language adapter of the target language, while the task adapter is retained, to achieve zero-shot cross-lingual transfer. In this work, we base our main experiments on the state-of-the-art cross-lingual adapter framework: MAD-X Pfeiffer et al. (2020). See Figure 6 in Appendix A for architecture details.

### Gradual Unfreezing and General Scheduled Unfreezing

First proposed by Howard and Ruder (2018), gradual unfreezing (GU) tunes a subset of parameters of a pretrained model starting from the top layer (see Figure 1b). Given a model with \(L\) layers, assuming that the index \(L-1\) refers to the top layer, and interval \(k\), GU unfreezes each layer starting from \(L-1\) to \(0\) in order, every \(k\) steps. Once a layer is unfrozen, it remains unfrozen. Hence, the number of trainable parameters increases every \(k\) steps under the GU regime.

Let SELECT\((*)\) be a layer-selection function. Let FORWARD\((*)\) be the standard forward pass through all layers, and for a subset \({\cal S}\) of layers of the network, let GRADIENT_UPDATE\(({\cal S})\) denote calculating gradients and performing updates on the parameters in \({\cal S}\). We define a _generalized scheduled unfreezing_ algorithm that encompasses GU, LPFT, and even the recent Surgical fine-tuning method Lee et al. (2023), as well as the other variants we propose later in this work, in Algorithm 1.4

Footnote 4: For LPFT, SELECT\((*)\) returns \(\emptyset\) for the first \(k\) steps and \({\cal J}=\{\theta_{j}\}\) for all layer indices \(j\) after the first \(k\) steps. In this work, we restrict ourselves to uniform time-intervals and leave the exploration of non-uniform intervals to future work.

```
0: An \(L\)-layer model with hyper \(j\in\{0,\dots,L-1\}\) parameterized by \(\theta_{j}\). An additional task-specific classification head \(C\). Total training budget (steps) N. Training interval \(k\). Typically \(kL\ll N\) for convergence. Initialize \(C\), \(\theta_{j}\) for all \(j\) \(S\leftarrow\{C\}\)for\(i=0\dots N\)do Sample a data batch \(b\sim D\) if\(i\mod k==0\)and\(i\leq KL\)then \(J\gets SELECT(*)\)\(\triangleright\) set of layer (task adapter) indices to unfreeze. \(S\gets S\cup\{\theta^{*}_{j};j\in J\}\) endif FORWARD\((*)\)GRADIENT_UPDATE\(({\cal S})\)endfor
```

**Algorithm 1** Generalized Scheduled Unfreezing

### Fisher Information

We use the _Fisher Information Matrix_ (\(F\)) to investigate changes in learning dynamics. Recent studies have shown that the Fisher Information Matrix correlates well with the generalization capabilities of neural network models Golatkar et al. (2019); Jastrzebski et al. (2021). Conveniently, as a 2nd-order metric based on gradients, \(F\) also provides insights into the optimization process.

Figure 1: **a) Standard, b) Gradual unfreezing versus c) \(\operatorname{tr}(F)\)-based scheduled unfreezing for training task adapters in adapter-based cross-lingual transfer. The classifier is not shown and is always trainable. All other components excluding task adapters, such as the original parameters of the base model and language adapters, are always frozen.**



In particular, we take the trace of \(F\) (i.e., \({\rm tr}(F)\)), since the full \(F\) is computationally expensive to obtain, and previous work has shown the \({\rm tr}(F)\) correlates well with \(F\) and shows similar general trends as the full \(F\)Achille et al. (2019). Let \(x\) be data input and consider a network parameterized by weights \(w\) that encodes the approximate posterior distribution \(p_{w}(y|x)\). The \({\rm tr}(F)\) is computed using the empirical data distribution \(\hat{Q}(x)\) as follows:

\[{\rm tr}(F)=\mathbb{E}_{x\sim\hat{Q}(x)}\,\mathbb{E}_{\tilde{y}\sim p_{w}( \tilde{y}|x)}\left||\nabla_{w}\log p_{w}(\hat{y}|x)\right||^{2} \tag{1}\]

Note that Eqn. (1) is not the "empirical Fisher" Kunstner et al. (2019), empirical Fisher uses the true data label \(y\)). Hence, one does not need the labels of input data \(y\) to calculate the true \(F\). They are sampled from the label distribution of the task (i.e., \(\hat{y}\sim p_{w}(\hat{y}|x)\), see pseudo-code in Appendix D).

One interpretation of \(F\) is that given \(w\) and a perturbed version of \(w^{\prime}\) (after applying one step of gradient descent, for example), the KL divergence between \(p_{w}(y|x)\) and \(p_{w^{\prime}}(y|x)\) is given by \(\delta w\cdot F\delta w+O(\delta w^{3})\) (up to 2nd-order approximation, where \(\delta w\) is the small perturbation in weights) Martens (2020).

\(F\) can be considered as a measure of how much a change in weights can affect the network output (i.e., how much information resides in the weights). Intuitively, this means a set of weights with near-zero entries in \(F\) likely means they do not significantly affect the network output (and thus task performance). Moreover, \(F\) is also a 2nd-order approximation of the Hessian of the loss function Shun-ichi and Hiroshi (2000); Martens (2020) and provides information on the curvature of the loss landscape near the current weights, that is, how fast the gradients change during the optimization.

## 4 Experiments

### Models

**Base Models.** The main experiments are conducted with two established pretrained multilingual models: mBERT (base-cased, Devlin et al., 2019) and XLM-R (base, Conneau et al., 2020).

**Adapters.** (\({}^{Ada}\)) We follow the adapter configurations from MAD-X Pfeiffer et al. (2020) for cross-lingual transfer, see SS3.1. We use pretrained language adapters available in the AdapterHub.5

Footnote 5: There are missing language adapters from the AdapterHub for mBERT; we have thus trained our own language adapters following the AdapterHub recommendations for hyperparameter values. Please see Appendix B for details.

**Scheduled Unfreezing Methods.** We apply and analyze two scheduled unfreezing methods from research in other areas of NLP and computer vision to the task of adapter-based cross-lingual transfer.

**LPFT**Kumar et al. (2022, **+LPFT**) first trains the classifier head (linear probing, LP) with the base model frozen, then unfreezes the entire model for fine-tuning (FT). In our setup, we first fine-tune the classifier, then followed by a step of unfreezing all the adapters for fine-tuning.

**Gradual Unfreezing**Howard and Ruder (2018, **+GU**) performs top-down unfreezing during training or fine-tuning. We fine-tune with the classifier and the top-most adapter unfrozen, and for every \(k\) steps we unfreeze the next adapter and continue.

### Datasets and Hyperparameters

We conduct experiments on a diverse set of tasks and target languages. We use MLQA Lewis et al. (2020) and XQuAD Artetxe et al. (2020) for question answering SQuAD, Rajpurkar et al. (2016) for training). We use XNLI Conneau et al. (2018) for natural language inference (training on MNLI, Williams et al., 2018), and we use XCOPA Ponti et al. (2020) for evaluating causal commonsense reasoning (training on COPA, Roemmele et al., 2011). The data statistics and language codes are summarized in Appendix C. We experiment in the zero-shot setting with English-only task data for training and validation.

**Hyperparameters.** We perform a hyperparameter search with the learning rates of \([\text{te-4}\), \(2\text{e-4}\), \(5\text{e-4}\), \(8\text{e-4}\)] for our main experiments on all datasets except COPA. For COPA, we found a smaller learning rate (\(1\text{e-5}\)) is better for scheduled unfreezing methods. See Appendix B for detailed hyperparameters per task per model. For scheduled unfreezing experiments, we search for the hyperparameter \(k\) in the following range [25, 50, 100, 800, 1000]. The reported results are averaged across 4 runs on A100 or V100 GPUs.

## 5 Results and Analysis

### Scheduled Unfreezing Improves Cross-Lingual Generalization

Figure 2 shows the relative performance of (a) task adapters fine-tuned in the standard way (\({}^{Ada}\)), (b) GU- (\({}^{Ada}\)+GU) and LPFT-tuned adapters (\({}^{Ada}\)+LPFT) compared to full fine-tuning with



mBERT and XLM-R on all datasets.6 We report the results averaged across all target languages in the respective datasets. Our experiments show that both LPFT and GU are effective in closing the gap to full fine-tuning across all tasks and models. Moreover, GU-trained task adapters perform better, even exceeding the performance of full fine-tuning in some cases.7

Footnote 5: The average accuracy is also slightly worse than the standard one for a single dataset.

Footnote 6: Baseline full parameter fine-tuning results used for Figure 2 are in Table 7 of the Appendix.

Our results suggest that scheduled unfreezing can do more than just mitigate catastrophic forgetting. Even in a CF-free setting like ours, they achieve better generalization for cross-lingual transfer. We focus on GU as the scheduled unfreezing method for further analyses, since it produced better empirical results than LPFT.

Since the training data for both XQuAD and XNLI are well over 50k instances; this amount of annotated task data might be unrealistic for many tasks in practice, even in English. We simulated low-data training settings (details and results are in Appendix G, Table 11), and found that even with a smaller amount of training data, we still observe the advantages of GU over standard task adapter fine-tuning.

### Scheduled Unfreezing Beyond Mitigating Catastrophic Forgetting

To understand why scheduled unfreezing helps even in the CF-free setting, we examine the learning dynamics during the training of task adapters.

Due to the unfreezing of task adapters at different times, the model has access to a different number of trainable parameters, which affects the optimization and information encoding for adapters differently under scheduled unfreezing than in standard fine-tuning. Hence, we draw our attention to higher-order metrics, captured in \({\rm tr}(F)\).

**GU Changes Learning Dynamics.** We plot \({\rm tr}(F)\) (moving average, normalized by the number of trainable adapters at the given step) during optimization. Figure 3 shows \({\rm tr}(F)\) during training on three datasets.8 The plots indicate that GU significantly changes the learning dynamics of adapters compared to their standard fine-tuning. With GU, due to the model having fewer parameters to encode the same amount of data initially, the \({\rm tr}(F)\) curve is higher than in standard fine-tuning. The training process induces a \({\rm tr}(F)\) curve that has a distinctive "hill" shape (i.e., a "learning period", with fast changes of gradients, and hence weights).

Footnote 8: Although we arrived at a different conclusion from Raffel et al. (2022), we emphasize that Raffel et al. (2022) compared full fine-tuning + GU with standard full fine-tuning, which is different from our work. We evaluate the cross-lingual transfer setup, which inherently comes with distribution shifts.

**Effects of the Unfreezing Schedule on \({\rm tr}(F)\) and Generalization.** While we have empirically validated that scheduled unfreezing changes \({\rm tr}(F)\) during learning, it is unclear which factors are the main drivers that impact \({\rm tr}(F)\), and potentially improve generalization. Previous work has studied factors such as learning rate, weight decay, optimizer and loss functions Golatkar et al. (2019); Jastrzebski et al. (2021). However, based on this work, another novel relevant factor is the _unfreezing schedule_ (i.e., the number of parameters to update at each optimization step). Here, we aim to further understand how unfreezing schedules change the learning dynamics and generalization.

We found that the sensitivity of the learning dynamics to schedules depends on the dataset and the base model. Hence, we focus on XLM-R for MLQA/XQuAD and mBERT for XNLI. These two settings are the most sensitive to schedules and best illustrate the effects, but they are different in terms of models and tasks to examine general patterns.

We randomly sampled 9 schedules (which are effectively permutations of layer indices, where we also treat the standard top-down order as the 10th schedule) that start unfreezing at either layer 10 or 9. The remaining experimental conditions are unchanged.9 We then aggregate runs with similar cross-lingual transfer results.10

Footnote 9: We consider these top layers, which likely carry similar information, in contrast to lower layers such as 0 or 1. The 11th layer is always unfrozen at the beginning, and we reject permutations of layer indices that do not start with 10 or 9. An accepted sampled schedule could look like [10, 1, 0, 3, 9, 6, 8, 2, 5, 7, 4].

We plot the \({\rm tr}(F)\) along with the validation metrics in Figure 4. We observe that decreases in \({\rm tr}(F)\) from the peak value are associated with rapid generalization (cf., a drastic increase of the validation metrics) of the network. Previous work points out that the peak of \({\rm tr}(F)\) - with contradicting claims from different works Golatkar et al. (2019); Achille et al. (2019); Jastrzebski et al. (2021) - correlates or anti-correlates with gen 

the underlining relationship may be more complex than just the peak value of the \({\rm tr}(F)\) curve.

Indeed, Figure 4 shows that a _wider_\({\rm tr}(F)\) curve (a longer learning period) often accompanies a large peak \({\rm tr}(F)\) value during the early phase of learning, and leads to better cross-lingual generalization during the test time. This could potentially lead to an additional new avenue of manipulating optimization with a regularization loss for cross-lingual transfer, which we leave for future work.

To the best of our knowledge, this is the first evidence indicating that \({\rm tr}(F)\) correlates with cross-lingual transfer performance in parameter-efficient fine-tuning and text-based Transformers. These results suggest that early-phase learning dynamics affect the generalization cross-lingually later on, and can be a potential measurement to study cross-lingual generalization. From the above results, we conjecture that _inducing large \({\rm tr}(F)\) and a longer learning period would improve generalization in the CF-free setting_,11 which motivates our experiments in the next section.

Footnote 11: Although we empirically observe similar patterns as Jastrzebski et al. (2021), we interpret the results differently. The prior work argues that sub-optimal small learning rate induces a higher peak \({\rm tr}(F)\) during learning (termed catastrophic Fisher explosion), and regularizing \({\rm tr}(F)\) to a smaller value helps with learning and generalization. In this work, we study cross-lingual generalization, which is a **shifted distribution** at test time instead of in-distribution generalization (the focus of the prior work). Furthermore, we also use a different optimizer, large learning rates and work in a CF-free setting. More discussion on combining with a regularizer is given in Appendix E.

### Auto-Scheduling by \({\rm tr}(F)\)

We have observed that scheduled unfreezing effectively changes the learning dynamics of the task adapters. A natural follow-up question is: _If freezing certain parameters changes the learning dynamics, can we systematically and automatically select which task adapter to unfreeze next?_

To answer this question, we propose to select the next layer to unfreeze based on ranked \({\rm tr}(F)\)

Figure 3: Average \({\rm tr}(F)\) per adapter during standard training versus using gradual unfreezing. Every point on the horizontal axis is 100 training steps for all datasets (except for XCOPA which is 50 steps).

Figure 2: The relative performance of adapters fine-tuned with scheduled unfreezing (i.e., GU-based and LPFT-based task adapters) and standard fine-tuned task adapters with full fine-tuning of mBERT and XLM-R.

 

during training (i.e., Figure 0(c)). According to our hypothesis, an induced large and wide \({\rm tr}(F)\) during learning leads to better generalization (discussed in Section 5.2).12

Footnote 12: \({\rm tr}(F)\) is calculated every \(k\) steps for \(L\) times, on 40 batches (maximum) of training data. We consider the additional computational cost to be negligible compared to GU.

\({\rm tr}(F)\)**-based Scheduled Unfreezing (Fun) Recovers GU.** Surprisingly, the \({\rm tr}(F)\)-based schedules recover the transfer performance as well as the top-down heuristic schedule (i.e., GU) in many cases. To illustrate this, we plot the unfreezing schedules generated by FUN along with the top-down schedule of GU (diagonal line) for all our experiments in Figure 5. From the figure, we can see that FUN nearly perfectly recovers the top-down schedule of GU for mBERT. We conjecture that GU is implicitly maximizing \({\rm tr}(F)\) at every unfreezing step. The XLM-R-based experiments largely follow the top-down order with more variance, which is likely due to noise in \({\rm tr}(F)\) estimation (for efficiency, we don't use the entire training data for estimation).

We show the results across all datasets in Table 1 and include an additional baseline (+_Rand_) that randomly selects the next layer to unfreeze at every time interval \(k\) (see Appendix F for de

Figure 4: Average \({\rm tr}(F)\) per adapter (normalized between 0-1 to plot together with the validation curve) and validation F1 or accuracy using a randomly sampled schedule. The average results indicated in the legend are the _averaged cross-lingual_ transfer results. a) averaged F1 of MLQA and XQuAD, b) XNLI.

 

tailed results). Table 1 shows that the FUN scheduler achieves near-identical results as GU with the mBERT model, which matches the observations in Figure 5. We also achieve comparable results as GU with XLM-R. The results are well above the random unfreezing baselines and standard training (e.g., improving mBERT from standard training on XNLI for 3.58 points, or the average of 2.03 points across all 4 datasets etc.). Although the source English performance is not the focus of our work, we also find that FUN achieves better English results in most of the cases (e.g., XLM-R with FUN is better than GU in English for 6 out of 8 cases, which shows the potential of FUN beyond the context of cross-lingual transfer explored here).

In addition, we highlight that scheduled unfreezing improved transfer results for the lowest-performing language across the board. For example, FUN improved the lowest-performing language under standard adapter training in all 4 cases for the mBERT model, and in 3 out of 4 cases for XLM-R. Some gains are quite substantial, e.g., Thai with mBERT increased from 34.53 to 42.55 (see Table 1, the _Lowest_ column for XQuAD).

**Beyond MAD-X.** Although less studied in the cross-lingual transfer setting, to show the generality of our findings, we perform additional experiments with LoRA adapters Hu et al. (2022), see the original paper for details). From Table 2, we observe that GU and FUN have comparable performance on average (FUN is better than GU for MLQA and GU is better than FUN for XQuAD with mBERT, and comparable for XLM-R) for cross-lingual transfer. More importantly, both unfreezing methods consistently outperform the standard tuning of LoRA. Additional results for other tasks with LoRA can be found in Table 14 in Appendix G.4, which also demonstrates comparable results for GU and FUN. Our results indicate that scheduled unfreezing is a general method applicable to different adapter types and improves generalization.

**Future Perspectives of FUN.** While GU remains effective, FUN offers the opportunity to potentially break away from the strict top-down unfreezing schedule and experiment with networks that have parallel structures (e.g., dual-network structures) in future work. Nonetheless, as the main finding in this work, FUN (i) provides evidence for a theory-based justification of heuristic unfreezing schedules from prior work, and (ii) it leads us to extend our understanding of learning dynamics during fine-tuning with scheduled unfreezing.

## 6 Conclusion

In this work, we first investigated whether scheduled unfreezing algorithms can help with generalization in the zero-shot cross-lingual transfer setting, and close the gap between parameter-efficient task-adapter training and full fine-tuning. Our experiments showed that scheduled unfreezing was indeed successful in closing this gap. Next, we investigated the training dynamics of scheduled unfreezing using the trace of the Fisher Information

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline  & \multicolumn{4}{c}{_XQ(aD) (/E1 / E2M)_} \\ \hline
**Method** & En & Lower (Tb) & Average \\ mBERT\({}^{LoRA}\) & 83.68(71.50 & 37.5692.44 & 61.53+0.37 / 45.01 \(\pm\) 0.44 \\ mBERT\({}^{LoRA}\)+GU & 84.24(72.75 & 40.7331.41 & **63.12\(\pm\)**0.207/45.96\(\pm\)**0.38 \\ mBERT\({}^{LoRA}\)+FUN & 83.58(71.13 & 38.9828.59 & **63.29\(\pm\)**0.46/ 45.80\(\pm\)**0.42 \\ \hline XLM\({}^{LoRA}\) & 83.35(72.25 & 80.0941.20 & 68.92 \(\pm\) 1.61 / 55.13 \\ XLM\({}^{LoRA}\)+GU & 84.70(731.62 & 66.7639.50 & **72.27\(\pm\)**0.12/ **75.46/ 70.34 \\ MLR\({}^{LoRA}\)+FNU & 84.27(72.29 & 65.69(\pm\)**0.482.6** & 72.13\(\pm\)**0.18/ 54.53\(\pm\)**0.14 \\ \hline \multicolumn{4}{c}{_MLQA (FI / EMJ)_} \\ \hline
**Method** & En & Lowest (Av) & Average \\ mBERT\({}^{LoRA}\) & 79.32(65.99 & 46.1729.67 & 55.55\(\pm\) 0.60 / 37.54\(\pm\) 0.63 \\ mBERT\({}^{LoRA}\)+GU & 78.63(65.03 & 47.7032.01 & 56.52\(\pm\) 0.78 / 37.22\(\pm\)0.79 \\ mBERT\({}^{LoRA}\)+FRUN & 78.86(71.47 & 77.129.98 & **56.66\(\pm\)**0.78/ **38.02\(\pm\)**0.97 \\ \hline XLM\({}^{LoRA}\) & 80.04(67.27 & 46.1928.92 & 59.40\(\pm\) 0.61 / 40.36\(\pm\)**0.38 \\ XLM\({}^{LoRA}\)+GU & 80.27(66.68 & 52.35734.36 & **63.11\(\pm\)**0.356/ **43.43\(\pm\)**0.13 \\ XLM\({}^{LoRA}\)+FNU & 80.51(67.18 & 52.9303.85 & **62.62\(\pm\)**0.50/ 43.21\(\pm\)**0.45 \\ \hline \end{tabular}
\end{table}
Table 2: Zero-shot transfer results across MLQA and XQuAD, with the LoRA adapter. We bold the highest and underline the second-highest average value.

Figure 5: Averaged unfreezing schedules for GU and FUN with different base models.

 

Matrix (\({\rm tr}(F)\)). Our experiments revealed a link between scheduled unfreezing, \({\rm tr}(F)\) and generalization performance. Finally, we proposed a general scheduled unfreezing algorithm (\({\rm tr}(F)\)-based scheduled unfreezing, FUN) that achieves performance comparable to existing heuristic variants, across multiple models and datasets. We hope to look into utilizing scheduled unfreezing to improve the cross-lingual generalization capabilities of large language models in the future.

## 7 Limitations

In this paper, we work with the trace of the Fisher Information Matrix as the metric for studying learning dynamics. While we believe our experiments and conclusions are widely applicable, there may be other complex factors and theoretical metrics (such as the eigenvalue spectrum of \(F\) or other matrix norms, etc.) that we could potentially investigate. Furthermore, our use of \({\rm tr}(F)\) is connected to prior work on the "critical learning period" during the early stages of training neural networks Achille et al. (2019); Kleinman et al. (2023), which could help us gain deeper theoretical insights into parameter-efficient training methods. We also speculate GU may not degrade the performance of full parameter fine-tuning Raffel et al. (2022) if it is done early in the training (i.e., \(kL\ll N\)) rather than evenly unfrozen throughout the entire training process (i.e. \(k=N/L\)). However, such investigations are outside of the scope of this work, and we leave it for future work.

This work was funded by the German Federal Ministry of Education and Research (BMBF) under the promotional reference 13N15897 (MISRIK), and the LOEWE initiative Hesse (Germany) within the emergenCITY center. The work was also supported in part by a personal Royal Society University Research Fellowship (no 221137; 2022-) awarded to Ivan Vulic.

We thank Sebastian Ruder, Ji-Ung Lee, Nico Daehim, and Tim Baumgartner for their valuable feedback and suggestions on a draft of this paper. We thank Kexin Wang and Alan Ansell for discussions about training adapters.

## References

* Achille et al. (2019) Alessandro Achille, Matteo Rovere, and Stefano Soatto. 2019. Critical learning periods in deep networks. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_.
* Amari (1998) Shun-ichi Amari. 1998. Natural gradient works efficiently in learning. _Neural Computation_, 10(2):251-276.
* Ansel et al. (2021) Alan Ansel, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavas, Ivan Vulic, and Anna Korhonen. 2021. MAD-G: Multilingual adapter generation for efficient cross-lingual transfer. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 4762-4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Artetxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4623-4637, Online. Association for Computational Linguistics.
* Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019. Simple, scalable adaptation for neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1538-1548, Hong Kong, China. Association for Computational Linguistics.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrava Chaudhry, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XnLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.

 

Rory A. Fisher. 1925. Theory of statistical estimation. Mathematical Proceedings of the Cambridge Philosophical Society, pp. 700 - 725. Cited by: SS3.
* A. Golatkar, A. Achille, and S. Soatto (2019)Time matters in regularizing deep networks: weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. Socher, K. Cho, and K. Graas (Eds.), Vol. , American Statistical Association, Chamon, New York, NY, USA, pp. 10677-10687. External Links: ISBN 978-1-4503-387X-0 Cited by: SS1.
* P. He, X. Liu, J. Gao, and W. Chen (2021)DeBERTa: decoding-enhanced bert with disentangled attention. In 9th International Conference on Learning Representations, ICLR 2021, Online, Austria, May 3-7, 2021, Cited by: SS1.
* N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019)Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, Vol. 97, pp. 2790-2799. External Links: Link Cited by: SS1.
* J. Howard and E. Ruder (2018)Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, pp. 328-339. External Links: Link Cited by: SS1.
* F. Lewis, B. Oguz, R. Rinott, S. Riedel, and H. Schwenk (2020)MLQA: evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7315-7330. External Links: Link Cited by: SS1.
* J. Hu, C. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022)LoRA: low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, External Links: Link Cited by: SS1.
* J. Hu, S. Ruder, A. Siddhant, G. Neubig, O. Firat, and M. Johnson (2020)XTREME: a massively multilingual multitask benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pp. 4411-4421. External Links: Link Cited by: SS1.
* S. Jastrzebski, D. Arpit, O. Astrand, G. B. Kerg, H. Wang, C. Xiong, R. Socher, K. Cho, and K. Graas (2021)Catastrophic fisher explosion: early phase fisher matrix impacts generalization. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139 of Proceedings of Machine Learning Research, pp. 4772-4784. External Links: Link Cited by: SS1.
* J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell (2017)Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences114 (13), pp. 3521-3526. External Links: Document, ISSN 0037-6905, Link Cited by: SS1.
* M. Kleinman, A. Achille, and S. Soatto (2023)Critical learning periods for multisensory integration in deep networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 24296-24305. External Links: Link, 2302.1190 Cited by: SS1.
* A. Kumar, A. Raghunathan, R. Matthew Jones, T. Ma, and P. Liang (2022)Fine-tuning can distort pretrained features and underperform out-of-distribution. In 10th International Conference on Learning Representations, ICLR 2019, Online, Apr 25-29, 2022, Cited by: SS1.
* F. Kunstner, P. Hennig, and L. Balles (2019)Limitations of the empirical fisher approximation for natural gradient descent. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, H. Lin, Y. Bengio, and C. Szegedy (Eds.), Lecture Notes in Artificial Intelligence, Vol. , MIT Press. Cited by: SS1.
* Y. Lee, A. S. Chen, F. Tajwar, A. Kumar, H. Yao, P. Liang, and C. Finn (2023)Surgical fine-tuning improves adaptation to distribution shifts. In The Eleventh International Conference on Learning Representations, ICLR 2023, Rigali, Rwanda, May 1-5, 2023, External Links: Link Cited by: SS1.
* P. Lewis, B. Oguz, R. Rinott, S. Riedel, and H. Schwenk (2020)MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7315-7330. External Links: Link Cited by: SS1.
* A. Lodha, G. Belapurkar, S. Chalkapurkar, Y. Tao, R. Ghosh, S. Basu, D. Petrov, and S. Srinivasan (2023)On surgical fine-tuning for language encoders. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 3105-3113. External Links: Link, Document Cited by: SS1.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, Cited by: SS1.
* J. Martens (2020)New insights and perspectives on the natural gradient method. Journal of Machine Learning Research21 (146), pp. 1-76. External Links: Link Cited by: SS1.
* M. McCloskey and N. J. Cohen (1989)Catastrophic interference in connectionist networks: the sequential learning problem. volume 24 of Psychology of Learning and Motivation, Academic Press.

 

Marinela Parovic, Goran Glavas, Ivan Vulic, and Anna Korhonen. 2022. BAD-X: Bilingual adapters improve zero-shot cross-lingual transfer. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1791-1799, Seattle, United States. Association for Computational Linguistics.
* Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7654-7673, Online. Association for Computational Linguistics.
* Pfeiffer et al. (2021) Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2021. UNKs everywhere: Adapting multilingual language models to new scripts. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 10186-10203, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavas, Olga Majewska, Qianchu Liu, Ivan Vulic, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal commonsense reasoning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 2362-2376, Online. Association for Computational Linguistics.
* Raffel et al. (2022) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(1).
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Ramponi and Plank (2020) Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in NLP--A survey. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 6838-6855, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Roemmele et al. (2011) Melissa Roemmele, Cosmin A. Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In _AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning_, Stanford University.
* Shun-ichi and Hiroshi (2000) Amari Shun-ichi and Nagaoka Hiroshi. 2000. Methods of information geometry. volume 191 of _Translations of Mathematical Monographs_.
* Singh and Alistarh (2020) Sidak Pal Singh and Dan Alistarh. 2020. Woodfisher: Efficient second-order approximation for neural network compression. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, Online_. Curran Associates, Inc.
* Stickland and Murray (2019) Asa Cooper Stickland and Iain Murray. 2019. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 5986-5995. PMLR.
* Sung et al. (2021) Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Training neural networks with fixed sparse masks. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, Online_, pages 24193-24205. Curran Associates, Inc.
* Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9275-9293, Online. Association for Computational Linguistics.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Xu et al. (2021) Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang, Songfang Huang, and Fei Huang. 2021. Raise a child in large language model: Towards effective and generalizable fine-tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9514-9528, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Yang et al. (2022) Zhuoyi Yang, Ming Ding, Yanhui Guo, Qingsong Lv, and Jie Tang. 2022. Parameter-efficient tuning makes a good classification head. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, Abu Dahbi, United Arab Emirates. Association for Computational Linguistics 



## Appendix A Adapter Architecture

Figure 6 shows the adapter architecture for cross-lingual transfer based on MAD-X. Each adapter consists of a down-projection followed by a ReLU activation and an up-projection, inserted after the feed-forward layer in every Transformer layer.

## Appendix B Hyperparameters

We include the detailed hyperparameters used in our experiments in Table 3 and 4. We use the AdamW optimizer (Loshchilov and Hutter, 2019) for all experiments, without warmups and a maximum gradient norm of 1 in all models.

For COPA we train the model with longer epochs for scheduled unfreezing. since the training data for COPA only contains 400 instances, the training time is still very small (<1 hour). We found standard training for task adapters for COPA does not benefit from longer training time and a smaller learning rate.

In addition, some of the language adapters are missing from the AdapterHub (just for mBERT or both for mBERT and XLM-R). We follow the same adapter configuration from Pfeiffer et al. (2020), and train those language adapters that are missing for mBERT (Italian, Tamil and Thai) with the Wikipedia data, a learning rate of 1e-4 , batch size of 64, sequence length of 512, and a maximum 100 epochs/training budget or 24 hours (whichever is reached first).

The task adapters have a reduction factor of 16 as indicated in MAD-X.

## Appendix C Dataset Statistics

We include the dataset statistics in Table 5. The training data for XQuAD and MLQA are SQuAD Rajpurkar et al. (2016). The training data for XCOPA is COPA Roemmele et al. (2011) and the training data for XNLI is MNLI Williams et al. (2018). All datasets used are available on HuggingFace. The language names and codes in our experiments are in Table 6.

## Appendix D Pseudo Code for \({\rm tr}(F)\) calculation

We provide the pseudo code for \({\rm tr}(F)\) calculation in Algorithm 2. Alternatively, please see our code at URL. Let FORWARD\((*)\) be the standard forward pass operation, SAMPLE\((*)\) be a function sampling labels from the label distribution of

\begin{table}
\begin{tabular}{l|c c c c c c} \hline \hline Model & Epochs & lr & batchsize & k-1k & k-5k & k-10k \\ \hline  & \multicolumn{7}{c}{_SQuAD_} \\ \hline mBERT & 20 & 5e-4 & 32 & 10 & 50 & 50 \\ XLM-R & 20 & 5e-4 & 32 & 10 & 50 & 50 \\ \hline  & \multicolumn{7}{c}{_MNLI_} \\ \hline mBERT & 50 & 5e-4 & 128 & 1 & 25 & 25 \\ XLM-R & 50 & 5e-4 & 128 & 1 & 25 & 10 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Hyperparameters used in the experiments with reduced task training data.

\begin{table}
\begin{tabular}{l|r r r r r r} \hline \hline Model & Epochs & lr & batchsize & k-DFT & k-GU & k-FUN \\ \hline \multicolumn{7}{c}{_SQuAD_ - LR Schedule: linear} \\ \hline mBERT & 5 & 5e-4 & 32 & 800 & 800 & 800 \\ XLM-R & 15 & 2e-4/5e-4 & 32 & 800 & 800 & 100 \\ \hline \multicolumn{7}{c}{_CORA_- LR Schedule: constant} \\ \hline mBERT & 500/5000 & 1e-4/1e-5 & 64 & 800 & 50 & 50 \\ XLM-R & 500/5000 & 1e-4/1e-5 & 64 & 800 & 1000 & 1000 \\ \hline \multicolumn{7}{c}{_MNNLI_- LR Schedule: linear} \\ \hline mBERT & 15 & 5e-4 & 128 & 25 & 800 & 50 \\ XLM-R & 15 & 5e-4 & 128 & 800 & 800 & 800 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Hyperparameters used in the main experiments. \(x/y\) denotes: \(x\) for standard adapter training and \(y\) for all scheduled unfreezing experiments.

Figure 6: Adapter Architecture.



data, and AGGAVG\((*)\) the function that aggregates \({\rm tr}(F)\) by task adapter blocks then taking the average over the number of trainable layers.

```
0: Number of batches \(N\) to sample from training data \(D\) for computing \({\rm tr}(F)\), \(\mathcal{P}\) trainable parameters.
1:Copy the model. \(\triangleright\) Avoid interference to the standard optimization.
2:for\(i=1\ldots N\)do
3: Sample a data batch \(b\sim D\)
4: outputs \(=\textsc{FORWARD}(*)\)
5: labels \(=\textsc{SAMPLE}(*)\)
6: prob = LogSoftmax(outputs)
7: loss \(=\textsc{NLL}(\text{prob};\text{labels})\)
8: loss.backward()
9:for\(p_{j}=1\ldots|\mathcal{P}|\)do
10: \({\rm tr}(F)_{j}=p_{j}.grad^{2}\left/\left|b\right|\right.\)
11:endfor
12:\({\rm tr}(F)=\textsc{AGGAVG}(*)\)
13:endfor
```

**Algorithm 2**\({\rm tr}(F)\) Calculation

## Appendix E Additional Discussion on Combining Scheduled Unfreezing with a Regularizer

Prior work Jastrzebski et al. (2021) proposed a FIM regularizer that relies on an artificially low learning rate (i.e., 10% of the optimal learning rate for standard training). Adapter training requires a higher learning rate in general (e.g., learning rate values in Table 3), hence it is not straightforward to use an FIM regularizer directly. In addition, the generalization experiments in Jastrzebski et al. (2021) are considered in-distribution (ID), whereas we focus on distribution shifts (i.e., cross-lingual and OOD).

Recent work such as LPFT Kumar et al. (2022) shows that ID and OOD performances do not correlate well, which we also observe in cross-lingual evaluation, especially when only English data is available for training and validation in zero-shot cross-lingual transfer (trends in English monolingual results can be inconsistent with trends in cross-lingual results).

Hence, although it is possible to combine scheduled unfreezing with a regularization method, it is not obvious how to do so in the best way. A more extensive investigation is outside of the scope of this paper and we reserve this topic for future work.

## Appendix F Detailed Results for Experiments

We show the detailed per-language experimental results in Tables 8 to 10.

The baselines (full parameter fine-tuning) results for plotting Figure 2 are in Table 7.

## Appendix G Additional Experiments

### Simulated Low-data Scenario

In order to simulate setups with fewer annotated data for training and analyze the impact of scheduled unfreezing in those setups, we sample 1k, 5k, and 10k training examples from SQuAD and MNLI. We evaluate GU against the standard adapter training baseline (Table 11). With a smaller amount of training data, we still observe the advantages of GU over standard task adapter fine-tuning.

### Reverse Gradual Unfreezing

We briefly experimented (2 runs) with the reverse order (bottom-up) of gradual unfreezing on MLQA, XQuAD and XNLI. We include the results for bottom-up GU (\(rev\)) in Figure 12 and included the numbers from standard GU for reference. The cross-lingual transfer results are significantly lower than the standard GU; however, the English results are similar.

### Experiments on Smaller Task Training Data with \({\rm tr}(F)\)-based Scheduling

We additionally performed the same experiments with smaller training data as described in our main paper, but now with \({\rm tr}(F)\)-based scheduling (+\(FUN\)). The results are in Table 13. Our results show that FUN is also comparable to GU when there are fewer training instances available. The

\begin{table}
\begin{tabular}{l c|c l l|l l} \hline \hline Language & Code & Language & Code & Language & Code \\ \hline Arabic & Ar & German & De & Greek & El \\ Spanish & Es & Hindi & Hi & Russian & Ru \\ Thai & Th & Turkish & Tr & Vietnamese & Vi \\ Estonian & Et & Haitian & Ht & Italian & It \\ Indonesian & Id & Quechua & Qu & Swahili & Sw \\ Chinese & Zh & Tamil & Ta & & \\ \hline \hline \end{tabular}
\end{table}
Table 6: Language code.

\begin{table}
\begin{tabular}{l|c r} \hline \hline Model & MLQA (Fi) & XQuAD (Fl) \\ \hline mBERT & 56.85 & 63.33 \\ XLM-R & 62.59 & 71.98 \\ \hline Model & XCOPA (Acc.) & XNLI (Acc.) \\ \hline mBERT & 53.39 & 63.60 \\ XLM-R & 54.99 & 73.43 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Baselines (full fine-tuning) results for cross-lingual transfer.

 

used in our experiment for FUN are (for 1k/5k/10k correspondingly): mBERT-XQuAD = [10,50,50], XLM-R-XQuAD = [10,50,25], mBERT-XNLI = [10,50,25], XLM-R-XNLI = [1,25,10]. The remaining hyperparameters are the same as in all other experiments.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline MLQA & En (F1 / EM) & Ar & De & El & Es & Hi & Ru & Th & Tr & Vi & Zh & **Avg. F1 / EM** \\ \hline mBERT\({}^{Ada}\) & 78.99/65.85 & 45.76 & 59.47 & - & 64.60 & 46.91 & - & - & - & 57.01 & 58.64 & 55.40\(\pm\)0.94 / 37.07\(\pm\)0.72 \\ mBERT\({}^{Ada}\)\({}_{a}\)Rand & 79.22/65.94 & 46.65 & 57.92 & - & 67.91 & 46.03 & - & - & 57.75 & 59.34 & 55.93\(\pm\)0.21 / 37.54\(\pm\)0.31 \\ mBERT\({}^{Ada}\)\({}_{a}\)GU & 78.04/64.20 & 47.96 & 57.95 & - & 68.64 & 51.75 & - & - & 58.95 & 58.95 & 57.37\(\pm\)0.32 / 38.27\(\pm\)0.27 \\ mBERT\({}^{Ada}\)\({}_{a}\)FPAN & 78.82/65.29 & 48.20 & 58.96 & - & 67.19 & 51.51 & - & - & - & 59.47 & 58.64 & 57.33\(\pm\)0.51 / 38.29\(\pm\)0.63 \\ \hline   KLM-R\({}^{Ada}\) & 79.52/65.99 & 51.74 & 59.64 & - & 68.33 & 61.77 & - & - & - & 64.85 & 61.88 & 61.31\(\pm\)0.46 / 42.10\(\pm\)0.42 \\ XLM-R\({}^{Ada}\)+Rand & 80.32/67.01 & 50.33 & 61.72 & - & 69.98 & 60.08 & - & - & - & 63.81 & 62.22 & 61.36\(\pm\)1.69 / 41.59\(\pm\)1.96 \\ XLM-R\({}^{Ada}\)+GU & 80.37/66.77 & 55.16 & 61.30 & - & 70.36 & 63.75 & - & - & - & 66.45 & 63.79 & 63.47\(\pm\)0.12 / 43.55\(\pm\)0.11 \\ XLM-R\({}^{Ada}\)+FUN & 80.92/66.70 & 53.17 & 62.38 & - & 70.04 & 63.77 & - & - & - & 65.38 & 63.86 & 63.10\(\pm\)0.79 / 43.37\(\pm\)0.51 \\ \hline   XQuAD & En (F1 / EM) & Ar & De & El & Es & Hi & Ru & Th & Tr & Vi & Zh & **Avg. F1 / EM** \\ \hline mBERT\({}^{Ada}\) & 83.58/71.74 & 57.95 & 71.20 & 57.60 & 73.11 & 53.75 & 70.05 & 34.53 & 50.15 & 68.38 & 69.56 & 60.63\(\pm\)1.04 / 43.90\(\pm\)0.85 \\ mBERT\({}^{Ada}\)+Rand & 83.86/72.31 & 59.09 & 71.58 & 62.06 & 74.84 & 52.61 & 70.00 & 38.91 & 48.49 & 69.29 & 69.92 & 61.68\(\pm\)0.33 / 47.42\(\pm\)0.55 \\ mBERT\({}^{Ada}\)+GU & 83.21/71.55 & 62.90 & 72.39 & 62.38 & 74.43 & 56.28 & 69.46 & 44.08 & 53.39 & 70.10 & 69.37 & 63.48\(\pm\)0.22 / 46.76\(\pm\)0.44 \\ mBERT\({}^{Ada}\)+FUN & 83.71/71.83 & 62.24 & 72.74 & 62.19 & 74.05 & 56.92 & 69.74 & 42.55 & 53.40 & 69.56 & 69.13 & 63.25\(\pm\)0.26 / 49.09\(\pm\)0.48 \\ \hline   XLM-R\({}^{Ada}\) & 83.48/72.69 & 65.47 & 72.74 & 71.92 & 74.88 & 67.70 & 73.58 & 66.53 & 66.36 & 72.38 & 69.36 & 70.09\(\pm\)0.60 / 53.77\(\pm\)0.40 \\ XLM-R\({}^{Ada}\)+Rand & 84.76/73.74 & 63.69 & 74.40 & 71.25 & 76.28 & 65.09 & 73.77 & 64.93 & 64.85 & 72.06 & 73.54 & 69.99\(\pm\)1.47 / 52.06\(\pm\)2.04 \\ XLM-R\({}^{Ada}\)+GU & 84.49/73.57 & 67.83 & 75.55 & 74.26 & 77.42 & 70.46 & 75.52 & 69.52 & 68.53 & 75.88 & 75.39 & 73.04\(\pm\)0.22 / 55.93\(\pm\)0.15 \\ XLM-R\({}^{Ada}\)+FNU & 84.91/73.80 & 66.69 & 75.94 & 74.07 & 76.58 & 69.59 & 75.48 & 67.59 & 68.19 & 74.52 & 74.77 & 72.34\(\pm\)0.40 / 55.21\(\pm\)0.63 \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 8: Zero-shot transfer results (F1) for MLQA and XQuAD. Average is the cross-lingual average without English.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c} \hline XCOPA & En & Et & Ht & It & Id & Qu & Sw & Zh & Ta & Th & Tr & Vi & **Avg. Acc.** \\ \hline mBERT\({}^{Ada}\) & 63.80 & 54.20 & 53.04 & 50.16 & 53.84 & 53.12 & 54.16 & 59.08 & 52.56 & 51.68 & 54.52 & 57.56 & 53.99\(\pm\)0.49 \\ mBERT\({}^{Ada}\)+Rand & 65.00 & 53.36 & 52.32 & 50.96 & 53.60 & 54.00 & 53.44 & 58.64 & 50.92 & 51.96 & 55.04 & 57.96 & 53.84\(\pm\)0.71 \\ mBERT\({}^{Ada}\)+GU & 66.60 & 54.44 & 52.60 & 50.00 & 54.88 & 53.52 & 53.52 & 59.76 & 52.12 & 52.36 & 55.44 & 58.60 & 54.29\(\pm\)0.60 \\ mBERT\({}^{Ada}\)+FUN & 66.40 & 53.44 & 52.92 & 50.68 & 54.76 & 53.48 & 54.40 & 59.00 & 51.36 & 51.08 & 54.32 & 58.36 & 53.98\(\pm\)0.64 \\ \hline   XLM-R\({}^{Ada}\) & 65.20 & 56.16 & 51.28 & 56.72 



\begin{table}
\begin{tabular}{l|c c} \hline \hline X0qAD (F1) & IK & SK & 10K \\ \hline mBERT^Ada +GU & 45.27\(\pm\)0.59 & 52.58\(\pm\)0.81 & 55.89\(\pm\)1.08 \\ mBERT^Ada +GU & 45.93\(\pm\)0.50 & 53.10\(\pm\)0.35 & 56.47\(\pm\)0.74 \\ mBERT^Ada +PUN & 46.30\(\pm\)0.71 & 53.68\(\pm\)0.38 & 56.50\(\pm\)0.97 \\ \hline XLM-R*^Ada & 44.11\(\pm\)1.43 & 57.20\(\pm\)0.36 & 61.75\(\pm\)0.68 \\ XLM-R*^Ada +QU & 48.42\(\pm\)1.20 & 59.88\(\pm\)1.51 & 65.28\(\pm\)0.76 \\ XLM-R*^Acd a+RUN & 47.67\(\pm\)1.73 & 60.72\(\pm\)1.07 & 65.16\(\pm\)1.16 \\ \hline XNLI (Accuracy) & IK & SK & 10K \\ \hline mBERT^Ada & 43.86\(\pm\)1.43 & 49.68\(\pm\)0.73 & 52.34\(\pm\)0.40 \\ mBERT^Ada +GU & 44.69\(\pm\)0.61 & 51.67\(\pm\)0.43 & 53.95\(\pm\)1.47 \\ mBERT^Ada +PUN & 44.86\(\pm\)0.49 & 51.61\(\pm\)0.58 & 53.40\(\pm\)0.93 \\ \hline XLM-R*^Ago & 52.75\(\pm\)2.03 & 64.22\(\pm\)1.04 & 65.80\(\pm\)0.61 \\ XLM-R*^Ago +QU & 52.86\(\pm\)1.38 & 64.15\(\pm\)0.35 & 65.91\(\pm\)0.56 \\ XLM-R*^Ago +RUN & 52.29\(\pm\)1.85 & 64.10\(\pm\)0.92 & 66.26\(\pm\)0.87 \\ \hline \end{tabular}
\end{table}
Table 13: Cross-lingual transfer performance with subsampled English task data for task fine-tuning.

\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c c} \hline \hline XNLI & En & Ar & De & El & Es & Hi & Ru & Sw & Th & Tr & Vi & Zh & **Avg. Acc.** \\ \hline mBERT^AAa_ & 82.05 & 42.09 & 65.81 & 62.16 & 70.84 & 57.92 & 63.76 & 37.45 & 40.89 & 61.53 & 68.01 & 65.08 & 57.78\(\pm\)1.68 \\ mBERT^AAa_+Rand & 81.64 & 53.98 & 66.32 & 62.85 & 70.33 & 58.15 & 65.08 & 45.95 & 41.80 & 61.25 & 67.73 & 65.12 & 59.87\(\pm\)0.96 \\ mBERT^AAda_+GU & 81.79 & 62.78 & 66.25 & 63.51 & 70.28 & 58.89 & 65.74 & 54.06 & 38.97 & 62.25 & 68.52 & 67.17 & 61.67\(\pm\)1.04 \\ mBERT^AAa_+FUN & 81.70 & 58.48 & 66.32 & 63.87 & 70.98 & 59.08 & 65.45 & 53.73 & 41.13 & 61.89 & 68.25 & 65.75 & 61.36\(\pm\)0.51 \\ \hline XLM-R*^Ada_ & 84.31 & 70.42 & 76.16 & 75.80 & 78.85 & 70.16 & 75.14 & 68.16 & 71.14 & 72.33 & 75.06 & 73.24 & 73.31\(\pm\)0.44 \\ XLM-R*^ada_+Rand & 84.52 & 69.91 & 75.91 & 75.05 & 78.04 & 69.56 & 74.51 & 67.29 & 70.40 & 72.05 & 74.25 & 72.48 & 72.68\(\pm\)0.56 \\ XLM-R*^adia_+GU & 84.24 & 70.22 & 75.92 & 75.7 & 78.32 & 70.61 & 75.70 & 68.24 & 71.99 & 71.97 & 75.36 & 73.82 & 73.44\(\pm\)0.24 \\ XLM-R*^adia_+FUN & 84.72 & 70.58 & 76.17 & 75.68 & 78.29 & 69.75 & 75.42 & 67.48 & 71.44 & 71.71 & 74.75 & 73.20 & 73.13\(\pm\)0.53 \\ \hline \hline \end{tabular}
\end{table}
Table 10: Zero-shot transfer results (Accuracy) for XNLI. Average is the cross-lingual average without English.





### Additional Results with LoRA Adapters

We provide additional results with LoRA adapters in Table 14. For simplicity and to draw comparisons to our experiments with MAD-X, we couple the unfreezing for query and value LoRAs (applied to 'q' and 'v' attentions) and use the default LoRA configuration [lora_r=8, lora_alpha=8] from AdapterHub.

From the results, we can see that GU and FUN are consistently comparable. GU is slightly worse than the standard training likely due to the extremely small training data size of COPA. Overall, scheduled unfreezing algorithms can be easily applied to different adapter architectures to provide a performance boost.

The hyperparameters used in this experiment are in Table 15, where we kept the number of epochs for training and batch size the same as in our main experiments (Table 3).

### Preliminary Results with mDeBERTa

We include additional experiments on another, more recent base model, mDeBERTa He et al. (2021). mDeBERTa is the multilingual version of the recently proposed DeBERTa He et al. (2021) model with disentangled attention to its word content and position representations. We used the (mdeberta-v3-base) model for all our experiments.

Note that we trained language adapters using MLM loss for the mDeBERTa model according to the setup described in Pfeiffer et al. (2020). However, we see very large discrepancies in terms of transfer results for both XCOPA and XNLI when compared to the standard fine-tuning (the gaps are also much larger than the gaps for mBERT or XLMR). We hypothesize that the discrepancies may be because mDeBERTa uses different attentions and the adapters we studied here are not designed for mDeBERTa (both in their architecture and in their training method). However, as tuning adapter architectures is beyond the scope of our study, we include the results here for completeness only



\begin{table}
\begin{tabular}{l|c c c|c c c} \hline \hline  & \multicolumn{4}{c}{_XXUQA (F1 / EM)_} \\ \hline
**Method** & En & Lowest (Th) & **Average** & En & Lowest (Ar) & **Average** \\ mBERT\({}^{LoRA}\) & 79.32/65.99 & 46.17/29.67 & 55.55\(\pm\)0.60 / 37.54\(\pm\)0.63 & 83.68/71.50 & 37.56/29.44 & 61.53\(\pm\)0.37 / 45.01\(\pm\)0.44 \\ mBERT\({}^{LoRA}\)+GU & 78.63/65.03 & 47.70/30.21 & 56.52\(\pm\)0.78 / 37.72\(\pm\)0.79 & 84.24/72.75 & 40.73/31.41 & **63.12\(\pm\)**0.20 / **45.96\(\pm\)**0.38 \\ mBERT\({}^{LoRA}\)+FNN & 78.80/65.40 & 47.12/29.98 & **56.65\(\pm\)**0.79 / **38.02\(\pm\)**0.97 & 83.58/71.83 & 39.88/28.59 & 62.92\(\pm\)0.46 / 45.80\(\pm\)0.42 \\ \hline
**Method** & En & Lowest (Ar) & **Average** & En & Lowest (Ar) & **Average** \\ XLM-R\({}^{LoRA}\) & 80.04/67.27 & 46.19/28.92 & 59.40\(\pm\)0.61 / 40.36\(\pm\)0.38 & 83.35/72.27 & 58.09/42.10 & 68.92\(\pm\)1.16 / 51.50\(\pm\)1.53 \\ XLM-R\({}^{LoRA}\)+GU & 80.27/66.68 & 52.35/34.36 & **63.11**\(\pm\)**0.35 / **43.43**\(\pm\)**0.13 & 84.70/73.12 & 66.72/49.50 & **72.27\(\pm\)**0.12 / **54.67\(\pm\)**0.34 \\ XLM-R\({}^{LoRA}\)+FNN & 80.51/67.18 & 52.39/33.85 & 62.62\(\pm\)0.50 / 43.21\(\pm\)0.45 & 84.22/72.29 & 65.69/48.62 & 72.13\(\pm\)0.18 / 54.53\(\pm\)0.14 \\ \hline \hline \end{tabular} 
\begin{tabular}{l|c c c|c c c} \hline \hline  & \multicolumn{4}{c}{_XXUQA (Accuracy)_} \\ \hline
**Method** & En & Lowest & **Average** & En & Lowest (Sw) & **Average** \\ mBERT\({}^{LoRA}\) & 68.00 & 50.76 (E) & 53.43\(\pm\)1.05 & 81.86 & 49.96 & 65.27\(\pm\)0.02 \\ mBERT\({}^{LoRA}\)+GU & 66.00 & 52.04 (Ta) & 54.67\(\pm\)0.45 & 81.27 & 49.93 & **65.33\(\pm\)**0.15 \\ mBERT\({}^{LoRA}\)+FUN & 66.80 & 51.84 (Sw) & **54.83**\(\pm\)0.26 & 81.05 & 50.23 & 65.32\(\pm\)0.15 \\ \hline
**Method** & En & Lowest & **Average** & En & Lowest (Sw) & **Average** \\ XLM-R\({}^{LoRA}\) & 63.60 & 50.35 (H) & 55.46\(\pm\)0.88 & 84.26 & 64.79 & 72.98\(\pm\)0.33 \\ XLM-R\({}^{LoRA}\)+GU & 64.00 & 51.10 (H) & 55.07\(\pm\)1.37 & 84.04 & 65.29 & 73.43\(\pm\)0.20 \\ XLM-R\({}^{LoRA}\)+UN & 64.40 & 49.04 (Qu) & **55.62**\(\pm\)0.09 & 84.10 & 65.88 & **73.65\(\pm\)**0.57 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Zero-shot transfer results with LoRA adapters. Average is the cross-lingual average without English. We bold the highest and underline the second-highest average value. _Lowest_ denotes the task performance for the lowest-performing target language per each evaluation dataset and base model.

\begin{table}
\begin{tabular}{l|c c c c} \hline \hline Model & Epochs & lr & batchsize k-GU k-FUNC \\ \hline \multicolumn{4}{c}{_SQuAD - LR Schedule: linear_} \\ \hline mBERT & 5 & 5e-4/8e-4 & 32 & 800 & 800 \\ XLM-R & 15 & 5e-4 & 32 & 800 & 800 \\ \hline \multicolumn{4}{c}{_COPA - LR Schedule: constant_} \\ \hline mBERT & 500/5000 & 1e-4 & 64 & 50 & 100 \\ XLM-R & 500/5000 & 1e-4 & 64 & 100 & 200 \\ \hline \multicolumn{4}{c}{_MNLI - LR Schedule: linear_} \\ \hline mBERT & 15 & 5e-4 & 128 & 50 & 50 \\ XLM-R & 15 & 5e-4 & 128 & 800 & 800 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Hyperparameters used in the experiments with LoRA adapters. \(x/y\) denotes: \(x\) for standard adapter training and \(y\) for all scheduled unfreezing experiments.





\begin{table}
\begin{tabular}{l|c|c c c c c c c c c c c c c} \hline \hline MLQA & En (F1 / EM) & Ar & De & El & Es & Hi & Ru & Th & Tr & Vi & Zh & **Avg. F1 / EM** \\ \hline mDeBERTa\({}^{Ada}\) & 86.94 & 72.09 & 78.57 & 76.03 & 80.92 & 69.41 & 76.98 & 68.45 & 68.73 & 75.93 & 74.62 & 73.30 & 74.09\(\pm\)0.77 \\ mDeBERta\({}^{Ada}\)+GU & 86.48 & 71.46 & 77.90 & 75.50 & 79.23 & 67.30 & 75.84 & 68.64 & 69.68 & 74.51 & 74.05 & 74.85 & 73.54\(\pm\)0.58 \\ \hline \hline \end{tabular}
\end{table}
Table 18: mDeBERTa: Zero-shot transfer results (Accuracy) XNLI. Average is the cross-lingual average without English.

\begin{table}
\begin{tabular}{l|c|c c c c c c c c c c c c c} \hline \hline XCOPA & En & Et & Ht & It & Id & Qu & Sw & Zh & Ta & Th & Tr & Vi & **Avg. Acc.** \\ \hline mDeBERTa\({}^{Ada}\) & 65.75 & 55.32 & 55.68 & 58.64 & 61.00 & 51.40 & 56.48 & 62.76 & 57.16 & 57.28 & 55.76 & 59.04 & 57.32\(\pm\)4.46 \\ mDeBERTa\({}^{Ada}\)+GU & 65.80 & 57.96 & 56.60 & 59.36 & 60.12 & 52.36 & 55.96 & 63.28 & 57.88 & 57.48 & 58.12 & 58.76 & 57.99\(\pm\)3.57 \\ \hline \hline \end{tabular}
\end{table}
Table 17: mDeBERTa: Zero-shot transfer results (Accuracy) XCOPA. Average is the cross-lingual average without English.

\begin{table}
\begin{tabular}{l|c|c c c c c c c c c c c c} \hline \hline XLI & En & Ar & De & El & Es & Hi & Ru & Sw & Th & Tr & Vi & Zh & **Avg. Acc.** \\ \hline mDeBERTa\({}^{Adap}\) & 86.94 & 72.09 & 78.57 & 76.03 & 80.92 & 69.41 & 76.98 & 68.45 & 68.73 & 75.93 & 74.62 & 73.30 & 74.09\(\pm\)0.77 \\ mDeBERTa\({}^{Ada}\)+GU & 86.48 & 71.46 & 77.90 & 75.50 & 79.23 & 67.30 & 75.84 & 68.64 & 69.68 & 74.51 & 74.05 & 74.85 & 73.54\(\pm\)0.58 \\ \hline \hline \end{tabular}
\end{table}
Table 18: mDeBERTa: Zero-shot transfer results (Accuracy) XNLI. Average is the cross-lingual average without English.