Free Lunch: Robust Cross-Lingual Transfer via Model Checkpoint Averaging
Year: 2023

Abstract:
Massively multilingual language models have displayed strong performance in zero-shot (ZSXLT) and few-shot (FS-XLT) cross-lingual transfer setups, where models fine-tuned on task data in a source language are transferred without any or with only a few annotated instances to the target language(s). However, current work typically overestimates model performance as fine-tuned models are frequently evaluated at model checkpoints that generalize best to validation instances in the target languages. This effectively violates the main assumptions of ‘true’ ZS-XLT and FS-XLT. Such XLT setups require robust methods that do not depend on labeled target language data for validation and model selection. In this work, aiming to improve the robustness of ‘true’ ZS-XLT and FSXLT, we propose a simple and effective method that averages different checkpoints (i.e., model snapshots) during task fine-tuning. We conduct exhaustive ZS-XLT and FS-XLT experiments across higher-level semantic tasks (NLI, extractive QA) and lower-level token classification tasks (NER, POS). The results indicate that averaging model checkpoints yields systematic and consistent performance gains across diverse target languages in all tasks. Importantly, it simultaneously substantially desensitizes XLT to varying hyperparameter choices in the absence of target language validation. We also show that checkpoint averaging benefits performance when further combined with run averaging (i.e., averaging the parameters of models fine-tuned over independent runs).

heading: 1 Introduction and Motivation
text: Massively multilingual transformers (MMT) such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have become the main driver of multilingual NLP research. When fine-tuned on sizable task data in a high-resource source language, typically English, MMTs demonstrate crosslingual transfer capabilities (Pires et al., 2019) in
zero-shot (ZS-XLT; without any task-annotated instances in the target language) and few-shot (FSXLT; only a few task-annotated instances/shots available in the target language) transfer setups (Hu et al., 2020; Lauscher et al., 2020). However, recent work has shown that both cross-lingual transfer (XLT) paradigms are subject to large variation in XLT performance, especially if the target language is typologically distant to the source (Keung et al., 2020; Zhao et al., 2021; Schmidt et al., 2022).
The protocols for model selection in previous XLT work vary broadly, which exacerbates the comparison of reported XLT results. Some studies (i) do not sufficiently discuss their protocol (Conneau et al., 2020; Xu et al., 2022), while others (ii) tune hyperparameters on the English development splits (Hu et al., 2020; Wu and Dredze, 2020b), or even (iii) perform model selection on the targetlanguage validation sets (Luo et al., 2021; Fang et al., 2021; Zhao et al., 2021). Assuming the availability of sufficiently large target-language validation sets for hyperparameter-tuning and model selection is unrealistic and violates the assumption of a true ZS-XLT and FS-XLT setup (Perez et al., 2021; Schmidt et al., 2022). On the other hand, model selection on English validation data often does not correlate well with target-language performance (Keung et al., 2020).
Furthermore, benchmarking new and emerging XLT approaches with existing methods is even more challenging when the code or models from prior work are not publicly available (e.g., Wei et al., 2021; Xu et al., 2022).1 We therefore seek methods that reliably improve ZS-XLT and FS-XLT irrespective of the underlying model and the transfer paradigm, are easy to implement, inexpensive to evaluate, robust to varying hyperparameters, and applicable to true XLT setups where the existence
1Even when they are available, conducting comparative evaluations incurs an overhead of navigating an unfamiliar code base and potentially higher runtime.
5712
of any target-language validation data cannot be assumed nor guaranteed.
In this work, we propose a simple and effective method of checkpoint averaging (CA) that satisfies all the desiderata above. The principal idea is to save model snapshots at periodic intervals during fine-tuning and then average the weights of the multiple single-run snapshots (i.e., checkpoints) prior to XLT evaluation. A similar procedure has been successfully adopted, for instance, in computer vision (Huang et al., 2017), other NLP domains such as machine translation (Vaswani et al., 2017; Gao et al., 2022, inter alia), and speech processing (Dong et al., 2018; Karita et al., 2019, inter alia); however, it has not investigated nor adequately leveraged in XLT, notorious for its sensitivity to different choices of shots and hyperparameters.
Averaging model weights can be extended to merging last or multiple model snapshots from multiple model runs in a straightforward manner. As we show later, within-run snapshot averaging performs comparable, or even better in individual experiments, than the computationally more expensive ensembling of last snapshots of multiple models (i.e., from different training runs).
Contributions. (1) To the best of our knowledge, we are the first to extensively benchmark and analyze CA for both ZS-XLT and FS-XLT; we do this on a range of higher-level semantic (NLI, extractive QA) and lower-level token classification tasks (NER, POS). CA yields two benefits in true XLT setups, coming for ‘free’ (i.e., at no additional computation cost): the transfer performance (i) improves consistently, and (ii) it becomes much less sensitive to varying hyperparameters. (2) We shed more light on averaging models across runs (i.e., ensembling). We first confirm that standard plain ensembling (i.e., averaging the models across multiple runs) does not improve over single runs for natural language understanding tasks (Wortsman et al., 2022). We then illustrate that sizable gains from run averaging (RA) are unlocked only once models are constrained a priori to converge to more structurally similar sets of parameters. We also show that averaging the averaged checkpoints as opposed to averaging only the final models further benefits performance. Further, (3) for multilingual FS-XLT, we benchmark CA against the established gradient surgery method (GS), which aims to better align gradients between languages in a batch during training for improved FS-XLT (Xu and Mur-
ray, 2022). We demonstrate that the intricate and hyperparameter-conditioned GS performs subpar to the simple CA. Finally, (4) we validate that benefits of CA, RA, and their combinations extend to a variety of experimental settings for XLT, across a large number of different languages.

heading: 2 Background and Related Work
text: Zero-Shot and Few-Shot XLT. Modern multilingual and cross-lingual NLP is underpinned by the MMTs like mBERT (Devlin et al., 2019), XLM(-R) (Lample and Conneau, 2019; Conneau et al., 2020), or mT5 (Xue et al., 2021), pretrained via language modeling (LM) objectives on web-scale corpora for 100+ languages. The MMTs support XLT by semantically aligning representation spaces across multiple languages. (Hu et al., 2020; Cao et al., 2020). However, some languages ‘are more equal than others’ in the MMTs’ representation spaces (Wu and Dredze, 2020a), and the expected quality of XLT is highly dependent on (i) the pretraining data size for the target languages, as well as on (ii) the degree of linguistic and typological (dis)similarity between the source and the target (Lauscher et al., 2020; Ruder et al., 2021).
Prior work on ZS-XLT thus typically aims at better aligning the language-specific subspaces for XLT. For instance, modular approaches such as adapters (Pfeiffer et al., 2020; Ansell et al., 2021) and sparse subnetworks (Ansell et al., 2022; Foroutan et al., 2022) extend MMT to new languages by assigning a small number of languagespecific parameters (i.e., modules) that can be combined with the base MMT. Another strand of work utilizes signals from word translations or parallel data aiming to tie cross-lingual representations of languages of interest closer together (Wang et al., 2019b; Wu and Dredze, 2020b; Hu et al., 2021).
Research on FS-XLT empirically validated that using even a handful of labeled instances in the target language along with source-language instances can considerably improve XLT beyond ZSXLT (Lauscher et al., 2020; Zhao et al., 2021; Xu and Murray, 2022; Schmidt et al., 2022). FS-XLT can be stabilized and improved with (i) joint training on source- and target-language data (Schmidt et al., 2022) or (ii) the so-called gradient surgery approach (GS) which ‘de-conflicts’ gradients between instances belonging to different languages within a training batch (Xu and Murray, 2022).
In general, the methods that aim to boost XLT
suffer from issues such as incurring large computational costs (Xu and Murray, 2022; Schmidt et al., 2022), require additional task-annotated data (Lauscher et al., 2020), and other external data (e.g., parallel data), which limits their wider portability to a multitude of possible tasks, domains, and languages (Ponti et al., 2019).
Averaging Model Weights. As a method that is simultaneously easy to implement and inexpensive to evaluate, averaging model weights has found successful application in areas such as computer vision (Huang et al., 2017; Izmailov et al., 2018; Wortsman et al., 2022), machine translation (Vaswani et al., 2017; Gao et al., 2022), and speech processing (Dong et al., 2018; Karita et al., 2019). The approaches can be clustered over two core axes: (i) what checkpoints to select to average model snapshots, (ii) and how to aggregate the selected model snapshots.
Stochastic weight averaging (SWA) leverages intraining CA to guide gradient descent towards a better generalization (Izmailov et al., 2018).2 CA has been proven to benefit machine translation (Vaswani et al., 2017; Gao et al., 2022). Popel and Bojar (2018) recommend taking a large number of model snapshots at broad intervals. ‘Model souping’ (SOUP) refers to averaging distinct runs with varying hyperparameters to further improve performance in computer vision tasks (Wortsman et al., 2022). In monolingual NLP contexts, Wang et al. (2022) simultaneously train multiple adapters with consistency constraints, allocating 2-10× more time to their total training than what would be allocated to training only a single task adapter for GLUE tasks (Wang et al., 2019a). In contrast, we do not expand training time or computational resources in our work. Wang et al. (2022) also show that subsequent adapter averaging outperforms conventional logit ensembling.
Checkpoint selection and weighting schemes are typically devised based on validation sets (Wortsman et al., 2022; Matena and Raffel, 2022). One strategy is to select the k checkpoints that perform best on the validation set (Wortsman et al., 2022), where k is a tunable hyperparameter. Matena and Raffel (2022) show that the Fisher information matrix can be exploited to compute a weighted average
2However, SWA is incompatible with adaptive optimizers and does not improve text classification over AdamW (Loshchilov and Hutter, 2019). See https://github. com/timgaripov/swa/issues/6 and https://discuss. huggingface.co/t/improvements-with-swa/858.
of models to boost transfer across tasks. In this work, we show that even (arguably) naive
hyperparameter-free strategies to average model snapshots improve both ZS-XLT and FS-XLT, and make transfer much more robust. They operate without any target-language validation data, do not increase computational demands, and even often exceed the performance of the best individual model selected using target-language validation.

heading: 3 Methodology
text: Motivated by the success of weight averaging discussed in §2, we hypothesize that the approach might also prove effective for XLT: weight averaging should ‘denoisify’ idiosyncratic variation in weights of different model snapshots, which should in turn stabilize training and improve transfer.
In particular, we propose checkpoint averaging (CA) and run averaging (RA) of model snapshots for ZS-XLT and FS-XLT. For CA, we first initialize the model with the parameters of the pretrained MMT: we refer to this set of parameters as θ0. We then fine-tune the MMT for T steps on the task data. We store the model weights k times at a regular interval of Tk training steps. Before inference, we then re-initialize the model with the averaged weights 1k ∑k j=1 θj = θ̄, and then use the averaged parameter set θ̄ for inference. Run averaging (RA) denotes the straightforward
extension of CA to average model snapshots taken at checkpoints across R independent training runs. For RA, we put forth and evaluate two different variants. First, we can average only the model snapshots taken at the last checkpoint of each individual run. The parameters at inference for this variant, termed RA-LAST are then computed as 1 R ∑R i=1 θ i k. Here, θ i k denotes the final (i.e., k-th) model snapshot at the end of run i, i = 1, . . . , R. The second variant, termed RA-CA, combines CA with RA: we average all k model snapshots per run over all R independent runs. Effectively, we average over all k · R different model snapshots. The final set of model parameters used for inference is then computed as 1R ∑R j=1 θ̄ i.
Checkpoint Selection. We only evaluate straightforward CA and RA strategies and dispose of more involved weighting schemes. Such schemes would require (i) either target-language validation data violating the true XLT setup or (ii) rely on the validation data of the source language, which often yields subpar XLT performance (Keung et al., 2020).
Ensuring Alignment for Run Averaging. Prior work hinted that ‘plain’ off-the-shelf RA does not improve over individual models (carefully selected on validation data) on monolingual sequence classification tasks (Wortsman et al., 2022).3 We suspect that the different random-uniform initialized classifiers from different runs draw models into unrelated training trajectories, which might also have a detrimental effect on ZS-XLT.4 Pairs of random highdimensional vectors, i.e., classifiers, are orthogonal and do not systemically align across self-contained individual runs. We have verified this hypothesis empirically in our preliminary experiments.
Put simply, independent models converge to output representations that are orthogonal. This in turn neutralizes potential benefits of RA, since the sets of checkpoints across runs are mutually ‘too distant’ to complement each other. We address this shortcoming in two steps. We first fine-tune the model on the task in a standard fashion, yielding the first single run. We then re-train the model R times, but now we freeze all the classifiers of the R models to the parameters to which the initial run converged. This boosts alignment of the parameters of the models’ respective Transformer ‘bodies’. Importantly, this procedure is not required in FS-XLT, as we initialize all models with the same monolingually (source language) fine-tuned weights θk, which ensures comparability across FS-XLT runs.5

heading: 4 Experimental Setup
text: Tasks and Languages. We follow prior work (Hu et al., 2020; Lauscher et al., 2020; Xu and Murray, 2022; Schmidt et al., 2022) and evaluate ZS-XLT and FS-XLT on benchmarks that require nuanced syntactic and semantic understanding for effective cross-lingual transfer, outlined in what follows.6
We always use English as the source language.
Natural Language Inference (NLI). We evaluate ZS-XLT on a broad range of typologically and geographically diverse NLI datasets spanning a total 37 languages: XNLI (Conneau et al.,
3See Table J.1 in (Wortsman et al., 2022). 4PyTorch defaults to random-uniform initialization for linear layers (He et al., 2015). 5For FS-XLT, in our preliminary experiments we did not find variation in performance if we freeze the original classifiers stemming from monolingual English training. We observe that classifiers hardly change, as measured by the cosine similarity of classifier weights between the monolingual and multilingual checkpoints (≥ 0.98).
6Please refer to Appendix A.1 for detailed descriptions and references of datasets by task.
2018), IndicXNLI (Aggarwal et al., 2022), JampatoisNLI (Armstrong et al., 2022), and AmericasNLI (AmNLI) (Ebrahimi et al., 2021). For FS-XLT experiments, we rely on 7 languages from AmericasNLI which come with sizable validation and test sets: Aymara (AYM), Bribri (BZD), Guarani (GN), Quechua (QUY), Raramuri (TAR), ShipiboKonibo (SHP), Wixarika (HCH). We feed the output [CLS] token of the embedded hypothesis-premise pair into the classifier.
Extractive QA (TyDiQA-GoldP). TyDiQA-GoldP consists of questions that can always be extracted from the provided gold passage (Clark et al., 2020). Our FS-XLT experiments enclose all languages: Arabic (AR), Bengali (BN), Finnish (FI), Indonesian (ID), Korean (KO), Russian (RU), Swahili (SW), and Telegu (TE). The embeddings of a question-passage pair are fed into a span classifier that predicts the start and the end of the answer.
Named Entity Recognition (NER). We evaluate XLT on a broad set of 24 languages from WikiANN (Pan et al., 2017) and 10 African languages from MasakhaNER (Adelani et al., 2021). We choose a subset of 9 heterogeneous languages for FS-XLT: Arabic (AR), Finnish (FI), Hungarian (HU), Swahili (SW), Tamil (TA), Turkish (TR), Urdu (UR), Vietnamese (VI), and Chinese (ZH). The token representations of a sequence are fed into the classifier.
POS Tagging (POS). We use the UD treebanks (Zeman et al., 2020) and evaluate ZS-XLT on 32 languages from the XTREME benchmark (Hu et al., 2020).7 FS-XLT experiments include the following typologically diverse language sample: Arabic (AR), Basque (EU), Chinese (ZH), Finnish (FI), German (DE), Indonesian (ID), Japanese (JA), Turkish (TR), and Urdu (UR). The model architecture exactly matches the one used for NER.
Training Setup. XLM-Rbase is the main MMT in our XLT experiments (Wolf et al., 2020; Conneau et al., 2020).8,9 We train models for 10 epochs with AdamW (Loshchilov and Hutter, 2019), weight decay of 0.05, the learning rate set to 2e−5 with a
7We omit Kazakh, Thai, Yoruba, and Tagalog from ZS-XLT results, since these languages do not comprise validation data to measure TRG-DEV.
8We empirically validated that our ZS-XLT & FS-XLT scores match those from other XLT work with similar hyperparameters (Wu and Dredze, 2020b; Hu et al., 2021; Schmidt et al., 2022; Xu and Murray, 2022).
9We preliminarily evaluated ZS-XLT experiments with XLM-Vbase and XLM-Rlarge, for which the results closely mimic the trends of our main results presented in Table 1.
linear schedule of 10% warm-up and decay, and mixed precision, unless stated otherwise.10 We simply take model snapshots at the end of each epoch.11 The maximum input sequence length is 256 subwords for NLI, 384 with a stride of 128 for TyDiQA, and 512 for NER and POS. We fine-tune models for ZS-XLT in batches of 32 instances. In FS-XLT experiments, we train with 4 examples per language in one batch.
FS-XLT Setup. We follow Schmidt et al. (2022) and compute a loss for examples of one language and subsequently average language-specific losses with equal weighting into a single loss. We furthermore compare against the gradient surgery (GS), the state-of-the-art approach for boosting multilingual FS-XLT (Xu and Murray, 2022). For GS, we randomly exclude one language in a batch from training. We then apply GS for the remaining languages with respect to the held-out language.12
Data Sampling and Shots. For FS-XLT experiments, we train models with s ∈ {5, 10, 50, 100, 250} target-language shots. The training and validation splits for TyDiQA-GoldP and AmNLI are sampled from the original training and validation sets, respectively. NER and POS datasets offer sizable training portions from which we sample the ‘few’ training shots.
Random Seeds. For ZS-XLT, we initially execute 5 single runs with distinct random seeds. We then run 5 more runs per each classifier we keep frozen from the initial runs. For FS-XLT, we sample 5 diverse sets of s shots, for each of which we conduct 5 differently seeded runs for RA.
Evaluation Metrics. We report average scores computed with the following metrics: accuracy for NLI, span-F1 score for TyDiQA-GoldP and tokenlevel F1 for NER and POS. In order to analyze robustness and sensitivity of results across different tasks and model variants, we also track and report
10We follow Schmidt et al. (2022) and keep hyperparameters fixed, except during ablations focusing directly on hyperparameter variation, where we analyse the impact of the number of epochs, checkpoints sampling frequency, learning rates, and scheduler.
11The TyDiQA-GoldP English training portion only comprises 3,696 instances which is why we train ZS-XLT models for 20 epochs. Given the size of English MNLI, we train models in FS-XLT for 1 epoch. We save snapshots at 10% of steps in an epoch.
12We exclude the hyperparameter α denotes the share of batches that actually apply GS from our replication of GS, since ‘the values of α are selected empirically’ (Xu and Murray, 2022), which again violates the ‘true’ FS-XLT setup.
the standard deviation over runs.
Model Variants in Evaluation. Beyond the proposed averaging strategies CA, RA-CA, and RALAST (see §3), we also evaluate other transfer variants outlined in what follows. LAST simply evaluates the model snapshot at the final checkpoint of a single run. SRC-DEV selects the checkpoint with the corresponding model snapshot that maximizes the source-language validation metric (Hu et al., 2020). TRG-DEV violates the assumption of true XLT and assumes that the best checkpoint for XLT can be selected using a validation set in the target language (Keung et al., 2020). This ‘upper-bound’ single-run variant is not directly comparable to the other variants and is used for analysis purposes.13
For ZS-XLT, run-averaging is additionally evaluated with the ‘model soups’ approach (Wortsman et al., 2022) (termed SOUP). It comprises 5 runs spanned by varying the learning rates {1, 2, 3}e−5 paired with a binary switch of using or not using a learning scheduler with 10% warm-up.14

heading: 5 Results and Discussion
text: The full results for each task, dataset, and language are available in Appendix A.2. In what follows, we analyse results top-down, by type of transfer, between single runs and ensembling, along metrics, and finally datasets.
ZS-XLT. Table 1 summarizes the main of ZS-XLT results. We verify that our results align with relevant work for respective tasks and datasets (Hu et al., 2021; Wu and Dredze, 2020b).
Single Run. Model snapshot selection based on the development set of the source language (SRCDEV) slightly but consistently improves over the last model snapshot (LAST), albeit with higher variance. CA steadily outperforms both LAST and SRCDEV, and often with significantly lower variance across runs. On higher-level tasks (NLI), CA even performs on a par with snapshot selection based on target language validation data (TRG-DEV), a setup
13Note that, for all considered tasks and languages, the number of validation instances would always yield much more pronounced gains if used for training rather than for model selection (Schmidt et al., 2022). Unlike other variants in our comparisons, TRG-DEV also requires maintaining up to k models as the selected models might vary across different target languages.
14We exclude the configuration which uses the learning rate of 3e−5 without a scheduler as it may diverge due to a large learning rate; this leaves the total of 6-1=5 configurations for the SOUP averaging. Corresponding single-run ZS-XLT results for these configurations are in Table 5.
that violates true ZS-XLT. The TRG-DEV strategy performs best by sizable margin on POS & NER because those test sets include a much larger number of target languages. In such a setup, TRG-DEV selects – for each of the many target languages – a snapshot tailored to a concrete language. The fact that all fair snapshot selection strategies (i.e., all except TRG-DEV) yield similar performance on POS suggests performance saturation when transferring from English with a single model.
Ensembling. On tasks other than POS, ensembling (i.e., run averaging) substantially boosts ZS-XLT, but only if applied with our proposed training curriculum (see “Ensuring Alignment for Run Averaging” in §3). The results indicate that withinrun CA is generally beneficial for ensembling too, with {RA, SOUP}-CA, in which average checkpointaverages of individual runs, often brings gains over {RA, SOUP}-LAST, in which we average only the last model snapshots of each run. NER in particular seems to benefit from CA prior to either runaveraging (RA) or souping (i.e., averaging of runs with different hyperparameters).
Overall, our results indicate that CA eliminates the need for model selection in ZS-XLT. For a single run (i.e., fixed random seed) CA clearly outperforms SRC-DEV– from the ZS-XLT perspective, this means that there is no need for a development set in the source language. In ensembling, RA-CA performs on a par with SOUP-CA and SOUP-LAST, and better than any single run with optimal hyperparameters (cf. Table 5), suggesting that it removes the need for hyperparameter optimization. CA could likely be further improved by weeding out poorly performing checkpoints. This primarily facilitates ZS-XLT for tasks with small training datasets, such as TyDiQA. If target-language shots are available (cf. FS-XLT), i.e. TRG-DEV, models are best trained on all shots for XLT (Schmidt et al., 2022).
FS-XLT. Few-shot transfer results are shown in
Table 2. We ensure that the results can, wherever possible, be directly compared to prior work (Xu and Murray, 2022; Schmidt et al., 2022).
Single Run. Unlike in ZS-XLT, LAST and SRCDEV result in almost identical FS-XLT performance, since they now most often select the same checkpoint. We confirm the findings of Schmidt et al. (2022) in two regards: (1) LAST gets closer to or even exceeds the oracle TRG-DEV as we increase the number of target-language shots; (2) using available target-language shots for training is better than leveraging them for model selection (compare, e.g., TRG-DEV with 50 shots against LAST with 100 shots). Unlike in ZS-XLT, in FS-XLT CA most often surpasses the oracle TRG-DEV, since all target languages (with few shots) are now part of training. The gains over TRG-DEV are particularly pronounced for TyDiQA and NER and generally larger for the smaller number of shots. CA’s gains over legitimate selection strategies (LAST and SRCDEV) are even more pronounced.
Replication of Gradient Surgery (GS). We do not find that GS-LAST (Xu and Murray, 2022) improves FS-XLT, if training batches are balanced across all target languages (Schmidt et al., 2022).15 We believe the gains that Xu and Murray (2022) report originate from the fact that, due to their small batch size (2-4), individual batches only couple English examples with those from only 1-3 target languages by accumulating the gradients across batches to update the model only when 32 examples are seen.16
They effectively apply GS on many ‘oracle’ languages instead of only one before a parameter update (cf. Algorithm 1 of Xu and Murray, 2022). We thus believe that GS mostly offsets the withinbatch imbalance between languages in the original experiments. Our replication further illustrates how
15GS-LAST and GS-SRC-DEV yield virtually same results. 16Code available at: https://github.com/fe1ixxu/
Mixed-Gradient-Few-Shot
challenging it is to reproduce the XLT results from prior work. Besides differing implementations, hidden effects – such as within-batch per-language imbalance in GS training, or other opaque hyperparameters – hinder replication.
Ensembling. RA-CA and RA-LAST average 5 runs with different random seeds for each of five different shot setups ({5, ..., 250}). Ensembling again brings gains, especially in configurations with smaller numbers of shots. The gains even extend to POS, a simple and saturated task on which it is otherwise difficult to improve performance. CA is beneficial in FS-XLT ensembling too, with RA-CA at least matching, and often notably outperforming RA-LAST. Overall, the FS-XLT results corroborate the effectiveness of CA that we noted in ZS-XLT.

heading: 5.1 Further Analyses and Discussion
text: To test the robustness of CA, we run additional ablations: we compare ZS-XLT results for models trained (1) with different learning rates; and (2) under different computational budgets.
Hyperparameters for ZS-XLT. We repeat ZS-XLT experiments with LRs of {1, 2, 3}e−5, with and without a scheduler of 10% warm-up and subsequent decay (5 runs for each combination). Figure 1 summarizes the findings for SRC-DEV and CA on NLI and NER (complete results are in Table 5 in the Appendix). In comparison with SRC-DEV, CA reduces the variance in results between runs
with different learning rates as well within different runs with the same learning rate for both tasks. This yields further benefits. CA, unlike SRC-DEV, allows for ZS-XLT performance to depend much less on the selection of learning rates, rendering hyperparameter tuning less important for the final performance. This also in part explains why RA-CA further improves over RA-LAST: it averages more robust models from individual runs (cf. ‘SOUPs‘ in Table 1). This ablation contributes to the explanation of why ZS-XLT results greatly differ in the literature (Keung et al., 2020). For example, with learning rate scheduling, LAST deteriorates much more severely than SRC-DEV (especially at higher learning rates). This again stresses the need for strategies such as CA that stabilize XLT performance across runs and hyperparameters.
Training Duration for XLT. Table 3 presents experiments for ZS-XLT and FS-XLT with {10, 250} shots, in which we halve and double the number of training steps.17 In ZS-XLT, the takeaways align with the original experiments of Table 1. For FSXLT, CA gains further ground relative to LAST and SRC-DEV in prolonged training. This particularly proves true when only 10 shots per target language are available. Performance may be further improved by distributing the added compute budget more diversely. Rather than doubling the steps along a single trajectory that well converges in the original compute budget (i.e., 1 B), averaging two runs likely mitigates unfavorable variation within the snapshots of each run. Our RA-variants in the main FS-XLT results in Table 2 hint at that this likely proves true in FS-XLT as averaging across runs consistently yielded sizable improvements. We however leave such experiments to future work.
ZS-XLT for Multilingual Models. We additionally test the behaviour of multilingual models – trained on large source-language dataset and a multilingual dataset consisting of few-shots of target languages (included in FS-XLT training) – in ZS-XLT to few
17For ZS-XLT in TyDiQA-GoldP, we increase the number of epochs from 20 to 30.
remaining unseen languages: (1) for NLI – 3 languages from AmNLI (Ebrahimi et al., 2021), all languages from JampatoisNLI (Armstrong et al., 2022) and IndicXNLI (Aggarwal et al., 2022); (2) for NER, all languages from MasakhaNER (Adelani et al., 2021). Table 4 summarizes the results of this experiment. We again observe similar trends. Within a single run, CA yields large gains, now even more pronounced with more multilingual shots. RA-CA continues to generally outperform RA-LAST in the ensembling setup. Interestingly, for NER, single-run CA even outperforms the RA-LAST ensemble. Results of this realistic transfer of a multilingually trained model to a new (unseen) language confirms the utility of model averaging in XLT.

heading: 6 Conclusion
text: It is hard to meaningfully compare prior work on XLT: experimental setups are opaque and models are (often unreportedly) selected based on performance on English development data or even targetlanguage instances. On the one hand, selecting models based on target-language performance violates the ‘zero-shot’ assumption of ZS-XLT and overestimates performance in both ZS-XLT and FSXLT. Model selection on source-language data, on the other hand, has been proven unreliable (Keung et al., 2020). Further, reproducing existing work on XLT is unwieldy: even if code and models are available, replication incurs a significant overhead in terms of integration efforts and computing resources. In this work, we propose to average checkpoints (CA) stored periodically in training as a simple, computationally cheap, and effective baseline for XLT that remedies for all of the above. We show that (1) CA consistently improves both ZS-XLT and FS-XLT over model selection based on source-language data XLT baselines and (2) brings stability in performance across different runs. Further, we propose a curriculum training that involves
freezing of classifier’s parameters, allowing CA benefits to propagate to ensembling, i.e., averaging of models from independent runs. We hope that future works adopts CA as a competitive and robust baseline. This would lead to more transparency and fairness in XLT evaluation, leading to more trustworthy results.

heading: Limitations
text: The primary weakness of ‘fairly’ averaging model weights for XLT is that sensible checkpoints need to be averaged. This manifests, for instance, in hyperparameter ablation for ZS-XLT on TyDiQAGoldP. TyDiQA-GoldP is a complex task with merely 3,696 training instances that observes unusual training dynamics. On such a dataset, the early checkpoints often underperform models that (nearly) have converged, especially if training utilizes low learning rates with schedulers. Here, SRCDEV could be used to weed out underperforming checkpoints, such that CA then always exceeds the baseline that performs model selection on sourcelanguage validation data. Whenever the English training portion is sizable – like in our other tasks – checkpoint averaging is consistently beneficial. Our experiments also demonstrate that XLT behaves differently by task. Averaging checkpoints consequently might affect other tasks differently like, for instance, document classification that reason about long contexts or retrieval tasks like Tatoeba that jointly require sequence- and word-level semantics. Another dimension we did not explore further due to a limited compute budget is how to ensure best that monolingual models are aligned for run averaging. For instance, it may not be required or even desirable to keep classifiers frozen throughout the second step of our proposed training curriculum (§3), as we would ideally also want to average out idiosyncratic noise of the original classifier.

heading: Acknowledgments
text: We thank the state of Baden-Württemberg for its support through access to the bwHPC. Ivan Vulić is supported by a personal Royal Society University Research Fellowship ‘Inclusive and Sustainable Language Technology for a Truly Multilingual World’ (no 221137; 2022–).

heading: A Appendix

heading: A.1 Reproduction Details
text: Code. Our code is available at: https://github. com/fdschmidt93/free-lunch-xlt
Model architectures. All models rely on the

heading: AutoModelFor{SequenceClassification, TokenClassification, QuestionAnswering}
text: of xlm-roberta-base implementations fitting the corresponding task of the transformers library (Wolf et al., 2020).
Compute Requirements. All the experiments were run on a single V100 with 32GB VRAM. The total required GPU time (training & evaluation) per run for ZS-XLT is c.2.75 hours and FS-XLT 5 hours on average. We repeated each set of experiments at least 5 (and up to 25) times to reliably measure mean and standard deviation of performance. For ZS-XLT, we trained, per task, 5 initial models, 25 × 2 additional models to evaluate RA and SOUPs (i.e. 5 varying classification heads, cf §3), and 20 further models per configuration for each hyperparameter ablation. We trained 25 models per s shots in FS-XLT (i.e. 5 sets of different s shots with 5 runs each). We roughly estimate that total
GPU time accumulates to 6,400 hours across all experiments.
Further Dataset Details. All datasets are accessed via the datasets library (Lhoest et al., 2021). We sub-sample shots for datasets that do not comprise a training split for FS-XLT experiments as follows. We first randomly shuffle the validation split with one of seed s ∈ {42, . . . , 46} with the built-in datasets shuffle method and then gather the initial {5, 10, 50, 100, 250} instances as training shots for our XLT experiments. We then validate our models on the the |ND| − 500 remaining instances to measure TRG-DEV performance.
Natural Language Inference (NLI). As is custom, we use the sizable training split of MNLI (Williams et al., 2018) as our high-resource training dataset with 393K training instances for English. The source-language validation split is the development portion of XNLI (Conneau et al., 2018). We furthermore evaluate on IndicXNLI (Aggarwal et al., 2022), JampatoisNLI (Armstrong et al., 2022), and AmericasNLI (AmNLI) (Ebrahimi et al., 2021).
Extractive QA (TyDiQA-GoldP). For TyDiQAGoldP, we sub-sample training and validation instances as per the procedure noted above from all the training sets and use the official validation splits for testing (Clark et al., 2020). We compute SRCDEV on the bases of the 440 ‘test’ set instances of English, as the training split merely comprises 3,696 instances. This favors SRC-DEV compared to other selection strategies based on the source language, as another 10% of the training data are used for early stopping.
Named Entity Recognition (NER). As with other tasks, we access both WikiANN and MasakhaNER via the Huggingface datasets library (Lhoest et al., 2021). We train monolingual models for ZS-XLT on the English training portion of Wikiann.
POS Tagging (POS). We use the UD treebanks (Zeman et al., 2020) and evaluate ZS-XLT on 32 languages from the XTREME benchmark (Hu et al., 2020). We omit Kazakh, Thai, Yoruba, and Tagalog from ZS-XLT results, since these languages do not comprise validation data to measure TRG-DEV.
Sample Implementation. The below exemplary code is a simple implementation to average the state_dict of identical PyTorch models. The resulting averaged parameter can the been used to reinitialize the model with model.load_state_dict(state_dict).
import torch
def average_weights( state_dicts: list[dict[str, torch.Tensor]] ) -> dict[str, torch.Tensor]: """Avg. state_dicts of models
with same architecture.""" avg_state_dict = {} K = len(state_dicts) for (
name, params,
) in avg_state_dict.items(): if params.is_floating_point(): avg_state_dict[name] = params / K for state_dict in state_dicts[1:]:
for ( name, params, ) in avg_state_dict.items(): if params.is_floating_point():
avg_state_dict[name] += ( state_dict[name] / K
) return avg_state_dict

heading: Languages AYM BZD GN HCH QUY SHP TAR CNI NAH OTO TOTAL

heading: A.2.1 ZS-XLT Results

heading: A.2 Full Results

heading: Languages HEBREW HINDI HUNGARIAN INDONESIAN ITALIAN JAPANESE KAZAKH KOREAN MARATHI PERSIAN PORTUGUESE

heading: Languages AM AR AY BG DE EL ES FI FR HE HI HU IG

heading: Languages AMH HAU IBO KIN LUG LUO PCM SWA WOL YOR TOTAL

heading: Languages Shots AR BN FI ID KO RU SW TE TOTAL

heading: A.2.2 FS-XLT Results

heading: Languages SHOTS ARABIC BASQUE CHINESE FINNISH GERMAN INDONESIAN JAPANESE TURKISH URDU TOTAL

heading: Languages SHOTS AR FI HU SW TA TR UR VI ZH TOTAL
text: ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? A
B1. Did you cite the creators of artifacts you used? No response.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? All open-source packages (e.g. ransformers, pytorch-lightning, wandb, hydra) use highly permissive licenses allowing for the free use for research.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4,A
C 3 Did you run computational experiments? 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4,A
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4,A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope that future works adopts CA as a competitive and robust baseline. This would lead to more transparency and fairness in XLT evaluation, leading to more trustworthy results