One For All $\&$ All For One: Bypassing Hyperparameter Tuning with Model Averaging for Cross-Lingual Transfer
Authors: Fabian David Schmidt
Ivan Vulić
Goran Glavaš
Venue: EMNLP 2023 Findings (2023)
Keywords: zero-shot cross-lingual transfer
multilingual representation learning
TL;DR: Naive model averaging fundamentally improves zero-shot cross-lingual transfer over model selection on source-language validation data

Abstract: Multilingual language models enable zero-shot cross-lingual transfer (ZS-XLT): fine-tuned on sizable source-language task data, they perform the task in target languages without labeled instances. The effectiveness of ZS-XLT hinges on the linguistic proximity between languages and the amount of pretraining data for a language. Because of this, model selection based on source-language validation is unreliable: it picks model snapshots with suboptimal target-language performance. As a remedy, some work optimizes ZS-XLT by extensively tuning hyperparameters: the follow-up work then routinely struggles to replicate the original results. Other work searches over narrower hyperparameter grids, reporting substantially lower performance. In this work, we therefore propose an unsupervised evaluation protocol for ZS-XLT that decouples performance maximization from hyperparameter tuning. As a robust and more transparent alternative to extensive hyperparameter tuning, we propose to accumulatively average snapshots from different runs into a single model. We run broad ZS-XLT experiments on both higher-level semantic tasks (NLI, extractive QA) and a lower-level token classification task (NER) and find that conventional model selection based on source-language validation quickly plateaus to suboptimal ZS-XLT performance. On the other hand, our accumulative run-by-run averaging of models trained with different hyperparameters boosts ZS-XLT performance and closely correlates with ``oracle'' ZS-XLT, i.e., model selection based on target-language validation performance.

One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer
Fabian David Schmidt', Ivan Vulić?, Goran Glavaš!
Ceente For Artificial Intelligence and Data Science, University of Würzburg, Germany
2 Language Technology Lab, University of Cambridge, UK de
(fabian. schmidt, goran. goran.glavas] Junnn-eunnweeee iv2500cam. ac. uk
Abstract
lingually zero-shot, i.e. without further annota- tions, to target languages (Hu al., 2020; Lauscher et al., 2020). However, pretraining corpora size and linguistic distance between the source and tar- get language dictate the quality of XLT (Lauscher et al., 2020). This is why model selection based on source-language validation data unreliably corre- lates with ZS-XLT and selects checkpoints that yield suboptimal target-language performance (Keung et al., 2020). Worse yet, there is no "best practice" for replicating XL- results of prior work. Some works, as our results suggest (cf. $4). (1) exhaust extraordinarily large hyperparameter grids and (2) monitor target-language performance for the best transfer (i.e., violating "true" 7.- to outper- form baselines (Conncau et al., 2020; Wei et al., 2021). Other works rerun baselines with little no hyperparameter tuning (Hu et al., 2020; Wu and Dredze, 2020): the re-evaluation then often trails original results by non-negligible margins." As a remedy, Keung et al. (2020) propose to evaluate zs- XLT on the snapshot that generalizes best to valida- tion data in the target language ("oracle" ZS-XLT): as such, oracle ZS-XLT stabilizes evaluation and denotes ideal transfer performance. Nonetheless, oracle 7S-XI.T overstates the performance of true ZS-XLI for which no target-language instances are available (Schmidt et al., 2023). If they are, target- language annotations are always better levered for training than for validation (Schmidt et al., 2022).
Multilingual language models enable zero-shot cross-lingual transfer (ZS-XLTT): fine-tuned on sizable source-kanguage task data, they perform the task target languages without labeled in- stances. The effectiveness of ZS-XLT hinges the linguistic proximity between languages and the amount of pretraining data for a lan- guage. Because of this, model selection based on nource-languant validation is unreliable: it picks model snapshots with suboptimal target- language performance. As remedy, some work optimizes ZS-XLT by extensively tuning hyperparameters: the follow-up work then rou- tinely struggles to replicate the original results. Other work searches over narrower hyperpa- rameter grids, reporting substantially lower per- formance. In this work, we therefore propose an unsupervised evaluation protocol for ZS-XLT XLT that decouples performance maximization from hyperparameter tuning. As a robust and more transparent alternative to extensive hyperparam- eter tuning, propose to accumulatively aver- snapshots from different runs into single model. We run broad zs XLT experiments on both higher-level semantic tasks (NLI, extrac- tive QA) and a lower-level token classification task (NER) and find that conventional model selection based on lource-language validation quickly plateaus to suboptimal 7S-XI.T perfor- mance. On the other hand, our accumulative run-by-run averaging of models trained with different hyperparameters boosts ZS-XLT per- formance and closely correlates with "orracee ZS-XLT, i.e. model selection based on target- language validation performance.
This calls for an evaluation protocol that (1) max- imizes "true" XS-XLT results and (2) makes them easily reproducible, regardless the extent of hy- perparameter tuning. this work, we find that model averaging fulfills both criteria. Weights av- eraging has proven effective in, e.g., MT (Vaswani et al., 2017) and recently NLU (Wang et al., 2022; Schmidt al., 2023). Schmidt et al. (2023) enable
1 Introduction and Motivation
Massively multilingual transformers (MMTs) like XLM-(R. (Conneau et al., 2020: Liang al., 2023) or mT5 (Xue et al., 2021) pretrained via language modeling on vast corpora encompass- ing 100+ languages. MMT fine-tuned on labeled task data in a source language can transfer cross-
For instance, when evaluating XLM-V Liang et al. (2023) have been unable to reproduce the original results of XLMMR Ress on the XNLI benehmark (Conneau et al., 2020).

---

model averaging for sizable gains in XLT. They first fine-tune an MMT on labeled source-language data and then re-train models (i.e., more runs) by copying and freezing the task head of the initially fine-tuned model: this aligns snapshots and enables weight averaging across runs."
language validation against accumulative model averaging as follows. We iteratively sample mod- els (i.e., runs) ([01.....,,, 0,) < r < 10) with different hyperparameters (i.e., pairs of learn ing rates and batch sizes) from the pool of runs containing N runs per hyperparameler configu- ration (cf. Appendix $A.2). We repeat this pro- cedure 10 times and report mean ZS-XLT perfor- mance. Standard model selection picks the model {arg max, Val( S í S r) at run r that max- imizes source- (target-) language validation, cap- turing the "truc" ("oracle") ZS-XLT performance. " "Accumulative averaging", in contrast, naively av- erages (i.e. without any supervision) all models of runs to single model Ej= 0j = -,
Contributions, In this work, we propose evalu- ation protocol that decouples maximizing ZS-XLT performance from hyperparameter tuning. The key idea is to accumulatively average snapshots of runs with different hyperparameters: this improves per- formance over model selection based on source- language validation performance. We run exhaus- tive experiments on higher-level (NLI, extractive QA) and lower-level (NER) NLU tasks on broad grid of hyperparameters and show, examining the cross-section of all runs, that model selection based on source-language validation almost exclusively picks snapshots suboptimal for ZS-XLT. We also confirm that conventional hyperparameter tuning on source-lanee language validation prematurely settles for models that maximize lource-language perfor- mance at the expense of ZS-XLT. Crucially, we show that accumulative model averaging performs on par or better than the best snapshot picked by source- languael validation already from the sec- ond (i.c. first averaged-in in) run and then consistently improves 2S-XLT with more runs. We addition- ally show that this accumulative model averaging closely correlates with oracle 7I-XI.T without re- quiring any source- or target-language validation data to maximize transfer performance.
3 Experimental Setup
Tasks and Languages. We select for our evalua- tion two higher-level semantic tasks (NLI and and extractive QA) and one lower-level structured pre- diction task (NER). For each task, we fine-tune the MMT on the provided English training splits.
Natural Language Inference (NLI). We evaluate NLI XNLI (Conncau et al., 2018) and In- dicXNLI (Aggarwal et al., 2022) which together cover 25 typologically diverse languages.
Extractive QA (TyDiQA-GoldP), TyDiQA-GoldP comprises questions that are answered by a span of text the provided gold passage and covers 9 diverse languages (Clark et al., 2020).
Named Entity Recognition (NER). We evaluate NER on 25 languages from WikiANN (Pan et al., 2017), 10 languages from MasakbaNER (Adelani et al., 2021), and 9 languages from MasakhaNER 2.0 (Adelani et al., 2022).
2
Accumulative Run Averaging
Prior work conducts model selection for z.s- XLT by extensive hyperparameter tuning using either source- langet-language validation data. Whereas the latter violates true ZS-XLT (Schmidt al., 2022), the former overfits to source-language performance (Keung et al., 2020). The recent suc- cess of snapshot averaging in XLT (Schmidt et al., 2023) motivates our research question: can (accu- mulative) averaging of models trained during hyper- parameter search outperform with fewer overall training runs the ZS-XLT performance of the "op- timal" model selected based lource-language validation performance?
Training Details.
We train XLM-Rtagge (Conneau et al.. 2020) for 10 epochs with AdamW W (Loshchilov and Hutter, 2019), weight decay of 0.01, gradient norm clipping to 1.0, and a LR schedule of 10% linear warm-up and decay. We save 10 snapshots per model, one at every 10% of total training steps. The maximum sequence length is 128 tokens for NLI and NER and 384 with a stride of 128 for TyDiQA-GoldP.
Hy perparameete Grids. We simulate conven- tional hyperparameter grid search over a broad set
We benchmark model selection based on source-
"Train portion of MNLI (Williams et al.. 2018), the en- closed 3,696 English training instances of TyDiQA-GoldP for QA, and the English training portion WikiANN for NER. "The training data of TTTDiQAA--ola consists of merely 3,696 instances; we thus fine-tune longer, for 40 cpachs.
'Fine-tuning models with different randomly initialized task heads otherwise yields sets of incompatible weights, hin- dering meaningful model averaging.

---

of 21 configurations, pairing seven learning rates e {0.1, 0.5, 1.5, 2, 2.5, 33e-5 with three batch sizes b {16, 32, 61). The grid is deliberately kept wide and same for all tasks to not reflect any prior knowledge on task-specific "good values". We retrain MMT for each pair (1, b) three times with different random seeds to account for variances over individual runs.
XLT performance run-for-run of all variants for model selection based on source-language valida- tion (Max. SRC-DEV) against our accumulative av- eraging of randomly sampled runs with different hyperparameters (cf. $2). On NLI, picking a sin- gle model on source- source-language validation only im- proves ZS-XLT when moving from having one to having two models (i.e., between first two rows of Table 1) and stagnates when having more models to choose from. With more runs, source-language validation may even prefer models that are worse at ZS-XLT on TyDiQA and NER. Conventional model selection thus maximizes source- lauuee- performance at the expense of ZS-XLT. Across the board, accumulative averaging already matches or surpasses Max. SRC-DEV (with any number of models) using merely two or three runs. Moreover, accumulative averaging consistently outperforms the overall best single-run model chosen from 3+ runs (highlighted in green), irrespective the task. On all tasks, accumulative averaging stabilizes zs- XI.T and reduces performance variance vis-a-vis Max. SRC-DEV counterparts.
Model Variants. We evaluate four model vari- ants: V [LAST, SRC-DEV, CA, TRG-DEV]. LAST is simply the final snapshot of a training run, SRC-DEV is the snapshot that maximizes source- language validation performance (Hu et al., 2020). CA averages all snapshots of a run to single model and, according to Schmidt et al. (2023), outper- forms LAST and SRC-DLV. TRG-DEV breaches "true" zs XLT and picks the snapshot that performs best on the target-language validation data (Keung et al., 2020): as such, it generally represents an upper-bound of single-run ZS-XLT performance."
4 Results and Discussion
Single-Run Performance. The full 7S-XI.T re- sults by hyperparameters are presented in Appendix $A.2 (cf. Table 3). We observe that optimal ZS-XLT of single runs depends on all axes of analysis: task, hyperparameters, and model variant. While LAST and SRC-DEV generally perform well, their ZS-XLT performance fluctuates substantially across hyper- parameter configurations, in line with (Keung et al., 2020; Schmidt et al., 2023). CA is strong and ro- bust baseline that often outperforms LAST and SRC- DEV by notable margins on TyDiQA and NER. In the context of single run, CA performs especially well with suboptimal hyperparameters, even some- times outperforming TRG-DEV. We also confirm that CA remedies variation in ZS-XLT both within and across hyperparameters (Schmidt et al., 2023). Table 3 (cf. Appendix $A.2) further highlights the notable gap in -IT performance between the best-performing hyperparameter configurations and those selected based source- langua-e vali- dation. Only the "oracle" model selection based on langet-language validation reliably correlates with the actual best (test) ZS-XLT performance.
Accumulatively averaging within-run snapshots (CA) outperforms LAST and SRC-DEV slightly on NLI and materially on NER. For NER, ZS-XLT from WikiANN to MasakhaNER (2.0) also repre- sents domain transfer (from Wiki to news), in which CA yields tremendous gains. In-domain (i.c., test on WikiANN), CA generally performs on par with LAST and SRC-DEV. The same is not true for QA, where CA performs slightly worse: we as- cribe this to averaging of "unconverged" snapshots, owing to the small TyDiQA training set (merely 3,696 instances), especially from runs with smaller learning rates and larger batches (cf. Table 3).
Further Analyses. Table 2 extends the run- by-run analysis to TRG-DEV and "model soups" (SOUP) to illustrate why accumulative model aver- aging outperforms model selection based on source- language validation. Rather than selecting a single snapshot, SOUP averages the five snapshots (among all available runs) with best lource-language vali- dation performance (Wortsman et al., 2022).
Compared to (oracle) TRG-DEV, accumulatively averaging runs performs on par on NLI, slightly bet- ter on TyDiQA and somewhat worse on NER. TRG- DEV selects language-specinn snapshots, thereby tailoring 7.T to each target language and rem- edying for the varying performance of Max. SRC- DEV in ZS XLT to many target languages. Such
Run-by-Run Analysis. Table compares Z.S-
*Our full results Table indicate that for each task we obtain maximal (oracle) zs -LTT performance with a different, task-specific hyperparumeter configurations.
"Irrespective of tasks and language, labeled instances in the langea-l bring larger gains if used for training rather than for model selection (Schmidt et al,, 2022).

---

NLI
TyDiQA-GoldP
NER
Max. SRC-DRV
Accumulative Averaging
Max. SRC-DEV
Accumulative Averaging
Max. SRC-DEV
Accumulative Averaging
LAST
S-DRV
LAST
S-DRV CA
LAST
S-DEY
S-DEY
LAST S-DRY CA
TLST
S-DEV
70.0u 76-bas 77.3,4170.60e
16.6a
77.30.
71.964
71.967 13.6 71.90. 71.9ur
73.6.9
40. 40.8 11.141.1.. 14.4444444p
44.62,
41.1s0
44.62.1
17. a 77.5o. 77.60. (77.600
47.86.
78.00
71.966
71.00 13.32 4. 73.31.1
72.9a
39.3o.1
39.3o.
43.51. (33.2s 13.2
45.61.
77.2aa 77.5a.4
(7..0a
77.964
78.1na
12.
71.8oa
14.1 74. 74.3az
73.8.
39.31.2
39.51.
44.0
(4500.
45.
47.9..
27.2ua 7T.5u,4
77..7o
17.904
78. luss
72.5ux
72.00. T3.9pb 74..5... 74.
74.1o.
40.22.0
40.822
45.05.
44.51.
45.0..
45.3,a
47.2
77.30 a7.6... 77..0.
18.00a
78. lo.1
72. 6o
72.00. T3.8 74.
74.
71.26a
10.30.0
41 22.
13.91.0
15.3)
15.517
17.51.
77.36. 77.6a. (77.6.
78.06a
78.
72.66.
72.00.s 74.2o.s 774.7or
74.40.s
74.26.
40.3s.n
41.2a.a 43.9h
45.7.
45.91
47.9h.2
77.3ua 7T.6u. 77.5u. (77.9oz
18.1o4
78.2
T2.3ou
71.70.7
74.3o 74. 74.3u.
74.2
40.02.
40.02a
44.1za
46.0..
46.11a
48.11.1
77. a 77.60i 778
278.
2o
78 3o.
12.10.
71.70.
T4.20.3 771
6o.7
7..30.5
74 3
40.02.
10.62.
11.6
16.00
(6.0..
16.11
18.21.0
77. 77.4a 77.6n. 77.60.
(88.0s.
78.1
78.3aa
72.0
71.767
74.2s.s 74.6a.s
74.4a.:
74.20.4
39.6a.a
39.9a.
44.3
46.00
16.
48.36.7
77.3o. 7T.6u, 27.00. 78.2
78.30a
72.0,.
71.70.7 74.2o.s 74.6um. 74.40.4
74.2oua
(3..22,
39 92.4 44.4 40 20.e 48.4u.
Table 1: [[01..... 0,) 1 10) models sampled for variants (LAST, SRC-DEV, CA from Table (ef. $3). "Max. SRC-DEV" picks the run (arg SreVal(0)) i S "Accumulative averaging" simply averages all runs - By. Metries: accuracy for NLI, span-Fi for TyDiQA and token-level F; for NER. Subscripts denote
std. deviation. Colored averaging outperforms +0.2 more
performs +0.1 of the best Max. SRC-DEV model.
NLI
TyDiQA-GoMP
NER
XLT without target-language validation. On the other hand, accumulative averaging improves zs- XLT threefold: (1) Unlike model selection, it does not plateau in Zs- -L.T on suboptimal single runs that maximize souree-language performance; (2) TRG-DEV showcases that accumulative averaging ingests further runs with snapshots that perform well on ZS-XLT; (3) Model averaging irons out idiosyncratic noise of individual runs, leading to better performance. This renders accumulative av- eraging a robust (i.e., replicable results) and fair (i.e. true zero-shot) evaluation protocol for ZS-XLT.
Max, DEV
Acc. Avg.
Max. Dev
Acc. Avg.
Max. DEV
Acc. Avg.
sRC TRG;
SRC TRG
SEC TRG
DEVI
SOUP
SOUP
DEY
77.0177.3
76.8
73.7
12.3
77.778811
77.6
73.5
30.5
77.6 77.1
77.6
72.0
73.4
74.3
41.2
49.7
77.6 78227722
77.8
71.7
73.7
73.7
73.9
40.6
49.9
48-1 4288
10 77.6 78.4
17.7
71.7 79.9. 74.2
73.8
39.9 49.9 48.4 42.8
selects the run < I r). where T is the set of target languages. SOUP averages the five checkpoints (from all available runs) that "Max. SRC-DEV", For other details, see Table 1.
Table 2: "Max. TRG-DEV"
farg max; [T] (0,)
a variation has been shown to be particularly pro- nounced in ZS-XLT on token-level tasks like NER POS (Schmidt et al., 2023). On TyDiQA, believe that accumulative averaging (slightly) bet- ter stabilizes the transfer from a small training set (3.7K instances). SOUPS however perform notably worse than both TRG-DEV and accumulative aver- aging on II. and NER. SOUPs lack the beneficial diversity of different runs, as the best snapshots often come from the same "good" run.? Anecdotal evidence further exemplifies why source-language validation is inapt for 7S-XITT. One of 63 SRC-DEV models replicates XNLI results of Conneau al. (2020), vastly exceeding all other runs (c.A+1.0), This "miraculous" run though merely ranks 3rd according to source- lonrua-language validation perfor- mance.
5 Conclusion
Inconsistent hyperparameter tuning and model se- lection protocols exacerbate replicating previous results on ZS-XLT. In this focused study, we de- vise a ZS-XLT evaluation protocol that addresses previous shortcomings and feeds two birds with one scone. We show that accumulatively averaging snapshots rather than selecting models based on language validation performance both im- proves and stabilizes zs ZS-XLT. Conventional model selection strategies prematurely settle for models that maximize source-language validation perfor- mance and discard runs that generalize better in ZS-XLT. Accumulative model averaging both in- corporates snapshots that transfer well and irons out models that perform badly. We find that model averaging correlates closely with "oracle" zs- which assumes models selection on larget-language validation instances. We hope future work adopts model averaging to promote fair and reproducible Zs XLT that puts models on equal footing.
The above suggests that even the more so- phisticated hyperparameter tuning strategies (e.g.,
Bayesian optimization) are unlikely to improve zs-
'Fxtending SOUP to average the top-10 best snapshots does not improve performance.

---

Limitations
Nakatumba-Nabende Neo Lerato Mokono, Ig- natius Ezcani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Nat- urul Language Processing, pages 4488-4508, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Additional factors must be taken into consideration, even though we aspire to evaluate ZS-XLT all levels of transparency (i.e., variants and strategies) across a varied set of downstream tasks on broad hyperparameter grids. Neither model selection on source-language validation data nor accumulative averaging may benefit 7.. on certain tasks, as Schmidt et al. (2023), e.g., do not find that any variant other than TRG-DEV yields gain over LAST on part-of-speech tagging. The underlying cause remains unclear. For instance, the gains on 7S-XLT stemming from model selection or accumulative averaging likely depend on the type of distribu- tional shift from the souree-language training data and the target-language instances to transfer to (cf. $4; e.g. dynamics of variants in ZS-XLT for NER). accumulative averaging nevertheless remains a ro- bust evaluation protocol, as ZS-XLT performance is not expected to deteriorate via-à-vis other "fair" strategies (e.g., max. SRC-DEV). In addition, there may exist a subset of pairs of learning rates and batch sizes that jointly maximize source- and target- language performance. However, as our results sug- gest ($4), runs on such hyperparameters likely are indistinguishable from those that exclusively per- form just as well on the source- source-language validation set.
David Ifeoluwa Adelani, Jade Abbott, Graham Neu- big, Daniel D' souza, Julia Kreutzer, Constantine Lig- nos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, Stephen Mayhew, Is- rael Abebe Azime, Shamsuddeen H. Muhammad, Chris Chinenye Emezue, Joyce Nakaauumba-Nahende Perez Ogayo, Aremu Anuoluwapo, Catherine Gitau, Derguene Mbaye, Jesujoba Alabi, Seid Muhie Yi- mam, Tajuddeen Rabiu Gwadabe, Ignatius Ezeani, Rubungo Andre Niyongabo, Jonathan Mukiibi, Ver- rah Otiende, Iroro Orife, Davis David, Samba Ngom, Tosin Adewumi, Paul Rayson, Mofetoluwa Adeyemi, Gerald Muriuki, Emmanuel Anebi, Chiamaka Chuk- wuneke, Nkiruka Odu, Eric Peter Wairagala, Samuel Oyerinde, Clemencia Siro, Tobius Saul Bateesa, Temilola Oloyede, Yvonne Wambui, Victor Akin- odc, Deborah Nabagereka, Maurice Katusiime, Ayo- dele Awokoya, Mouhamadane MBOUP, Dibora Ge- breyohannes, Henok Tilaye, Kelechi Nwaike, De- gaga Wolde, Abdoulaye Faye, Blessing Sibanda, Ore- vaoghene Ahia, Bonaventure F. P. Dossou, Kelechi Ogueji, Thierno Ibrahima DIOP, Abdoulaye Diallo, Adewale Akinfaderin, Tendai Marengereke, and Sa- lomey Osei. 2021. MasakhaNER: Named entity recognition for African languages. Transactions of the Association for Computational Linguistics, 9:1116-1131.
Acknowledgments
We thank the state of support through access to the bwHPC. Ivan Vulié is supported by a personal Royal Society Univer- sity Research Fellowship 'Incuuive and Sustain- able Language Technology for a Truly Multilingual World' (no 2022--);;
for its
Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022. Indicxnli: Evaluating multi- lingual inference for indian languages.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette. Tom Kwiatkowski. Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seocee question answering in typo- logically diverse languages. Transactions of the As- sociation for Computational Linguistics, 8:454-470.
References
David Adelani. Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya. Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukibbi, Godson Kalipe, Derguene Mbaye, Amelia Taylor, Fatoumata Ka- bore. Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo. Catherine Gitau, Edwin Munkoh- Buabeng, Victoire Memdjokam Koagne, Allah- sera Auguste Tapo, Tebogo Macuewa, Vukosi Mari- vate, Mboning Tehiaze Elvis, Tajuddeen Gwad- abe, Tosin Adewumi, Orevaoghene Ahia, Joyce
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
lexxs Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross- lingual sentence representations. In Proceedings of

---

the 2018 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computa- tional Linguisties.
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee. Xiaodong Liu, Jing Gao, Ahmed Hassan Awadal- lah, and Jianfeng Gao. 2022. AdaMix: Mixture- of-adaptations for parameter-effiiint model tuning. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 5744-5760, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: massively multilingual multi-task benchmark for evaluating cross- pross-lingual generalisa- tion. In International Conference Machine Learn- ing, pages 4411-4421, PMLR.
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021. On learning univer- sal representations across languages. International Conference on Learning Representations.
Phillip Keung, Yichao Lu, Julian Salazar, and Vikas Bhardwaj. 2020. Don't use English dev: On the zero-shot cross-lingual evaluation of contextual em- beddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 549-554, Online. Association for Computational Linguistics.
Adina Williams. Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Pupers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Anne Lauscher, Vinit Ravishankar, Ivan Vulié, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4483 4499, On- line. Association for Computational Linguistics.
Thomas Wolf, Lysandre Dehut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 38-45, Online, Association for Computational Linguistics.
Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Na- man Goyal, Marjan Ghazvininejad, Luke Zettle- moyer, and Madian Khabsa. 2023. XLM-V: Over- coming the Vocabulary Bottleneck in Multilingual Masked Language Models. arXiv e-prints, page arXiv:2301.1441
Ilya Loshchilov and Frank Hutter. 2019. Decou- pled weight decay regularization. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Mitchell Wortsman, Gabriel Iharcoo, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopss Ari S Mor- cos, Hongseok Namkoong, Ali Farhadi. Yair Car- mon, Simon Kornblith, and Ludwig Schmidt. 2022. Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing inference time. In Proceedings of the 39th Interna- tional Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23965-23992 PMLR
Xiaoman Pan, Boliang Zhang, Jonathan May. Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Cross- -ingual Pro-
name tagging and linking for 282 languages. ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pupers), pages 1946-1958, Vancouver, Canada. As- sociation for Computational Linguistics.
Fabian David Schmidt, Ivan Vulić, and Goran Glavaš. 2022. Don't stop fine-tuning: On training regimes for few-shot cross-lingual transfer with multilingual language models. In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 10725-10742, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Shijie Wu and Mark Dredze. 2020. Do explicit align- ments robustly improve multilingual encoders? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages
Online. Association for Computa- tional Linguistics.
Fabian David Schmidt, Ivan Vulić, and Goran Glavaš. 2023. Free lunch: Robust lingull transfer via model checkpoint averaging.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-498, On- line. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Ilion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30, Curran Associates, Inc.

---

A
Appendix
A.1 Reproduction Details
Code. Our code is available at: https://github.
comm/Fdschmidtt3/ofa-aoa
Model architectures. All models the Crsooboeelloorssseeeeecalasiiiae TokenClassification, Questionknswering) xlm--rbbert--larg for the corresponding task from the transformers library (Wolf et al., 2020).
Compute Requirements. We execute all experi- ments on single V100 with 32GB VRAM. We es- timate that we require total compute time of 1,000 hours over all fine-tuning iterations and evaluations. We arrive this budget follows. We age train models on NLI for about 1 11.5 hours, TyDiQA-Goldie for roughly 1.5 hours, and NER for an estimated hours. We therefore execute training runs (21 1 hyperparameter configurations ran on for seeds, cf. $3) for 16 hours for a total of c.IK GPU hours. We loosely estimate that accumu- lative averaging adds another 50 hours of runtime for evaluation.
Model Averaging. We follow Schmidt et al. (2023) enabling accumulative averaging of checkpoints for NLI and TyDiQA-GoldP. For these tasks, initially fine-tune XLM-RRarge with a batch size of 32 and a learning rate of 2e-5, For NER, we find that merely randomly initializing the tasks heads across all runs with the same head slightly improves performance ( A + 1.0) of all variants in single-run and accumulative averaging. We suspect that the original language modelling weights better align with NER as a token-level classification task and do diverge incompatible sets of parameters in fine-tuning (cf. $3 of Schmidt et al. (2023)).
A.2 Full Results By Hyperparameter Configuration

---

78-ALT Performance
Hyperparameters
TADIQA-GOMP
Learning Roie
Buich
Sine
LAST
SRC-DRY
CA
TRG-DRY:
EAST
SRC-BEV
CA
TRG-DEV
FAST
SRC-DRY
YRG-OKY
16
17.20.1
7r.00.2
77.96.5
71.30.3
71.30.1
08 4..9
7I.4g.3
45.86
45.90 a
46. 46.3g.3
37.80.1
4o.1
71.5g
77.70.2
16.
20.2
31.30.4
20.3
71.6g
45.3g.2
45.3g
44.70.3
45.80.3
64
17. 1
77.0n.3 3
77.40.1
77.80.1
70.96.4
70.90.4
49.80.2 2
70.96.1
45.3n 2
45.3n 2
42.9p.3
45.5
16
70.
76.96
77.50.3
77.75.1
71.60.2
71.90.7
72.20.1
72.00
44.00.
44.90.9
47.40.3
49.80.3
80
77.6g
77-66..1
78.30.5
71.60.1
71.50.1
71.60.1
72.30
42.8o.3
42.90.9 9
45.8o.6
47.71.0
64
77. 0
78.10.6
77.80.2
77..9..
71.96.0
70.90.0
09.0g.7
71.70.1
43.81.6
43.81 6
46.21.3
49.40.4
16
76. 60.2
76.60.2
77.50.2
17.
73.00.4
72.50.4
T3 53.5
73.70.6
40.60.1
10.71.
43.91.4
48. 48.02.5
1e GG
32
76. 8o.2
77.10.4
77.60.1
77.20.2
72.10.4
72. 60.5
T3 00.3
73 20.3
1..
1..11.8 8
42.81.5
46. 0s.1
64
3
77.50.2
77-70.2
72.0p.6
71.70.8
T1.T0.2
73.0g
4..1
43.4g 3
46.61.2
70.76.7
16
75.0g
76.50.4
73.70.5
73.3p.3
44.40.6
74.30.4
41.02
46.33.0
G0.01.1
1.5e -
32
76. 6
76.5g.1
77.40.2
77.00.1
73.0p.7
73.10.9
D4.
73.76.6
39.9h
40.61 3
43.3p.2
26.21.9
64
76. 80.8
77.10.5
77.60.3
77.70.6
72.90.5
72.60.7
13.30.4
T3.50.3
00. 00.02.1
41.12.0
42.81.5
06.01.1
16
74.10.3
74.1p.a
76.10.2
74.80.2
72.96.5
72.7o.3
74.10.2
38.61.2
2
43.81.4
25.2
2e
32
75. 90.3
76.10.4
77.10.1
76.40.5
73.10.1
72.30.9
14.1
73.3g 0g.3g.
39.30.7
30.01
42.31.5
44.32.5
64
76.60.5
7
77.40.1
77.70.2
71.90..
71.71.1
T3.30.5
72.80
39.61.3
39..1
42.31.1
26.2
1b
71.3g.4
75.0p.2
13.1o.1
73.20.4
72.40.9
14.10.1
73.4g.5
30.31.0
30.31 D
44.31.7
47.10.8
2.5e
32
74. 80.2
74.80-2
76.3p.1
76.1
72.3p.2
71.75.0
T4.80.3
73 45.4
39. 39.41.5
39.41.5
42.5p.6
44.50.7
64
76. 70.70.2
70.70.2
77.50..
77.30.5
72.60.6
72.30.7
74.20.1
73.31.
39.91.0
39.91
43.51.8
47. 42.8
16
97.90.6
01.70.2
73.9p.3
70.50.9
72.30.2
T1.70.4
74.40.4
72.40.5
37.41.5
37.31 1
43.0p.s
40.3g.5
3e
32
73.30.1
73.3g.4
75.80.2
74.40..
71.70.4
71.60.7
70.10.1
T3.6g.
37 37.91
38.0
43.71.6
47.2g.7
64
76.80.3
76.30.7
72.00-2
71.8o.4
74.10.5
39.41
24.01.2
0.0
0.6
0.3
0.0
2.0
2.0
1.0
0.0
5.2
0.2
Table 3: ZS-XLT averaged over all target languages by task, model variant, and hyperparameters (cf. $3). For each column, best ZS-XLT emphasized in bold and max. validation performance (cf. Table 4) shaded in green A is
the difference of best ZS-XLT and ZS-XLT on models that maximize validation performance. Metries: accuracy for NLL, span-Fi for TyDiQA and token-level Fj for NER. Subscripts denote std. deviation.
Validation Performance
Hyperparameters
NLI
TyDIQA-GoMP
NER
Learning Rate
Batch Size
LAST
SEC-DEV
CA
TRG-DEV
LAST
SEC-DEV
CA
TRG-DEV
LAST
SRC-DEV
CA
TRG-DEV
16
20.2
90.50.2
10.
19.
76.40.2
To.5
15.1
67.00.2 2
81.
81.90.1
79.30.2
49. 49.90.1
le
32
90. 20.2
90.3p.1
90.00.0
79.
77. 0
77.90.8
73.50.4
68 9
80. 1
80. 4g.1
76.50.5
48. 38.3
64
80 10.1
90.20
89.60.1
78.90.0
76.00.6
76.30.4
64.40.2
66.50.2
78.00.
78.00
71.40.4
47.70.2
29.
59.7g.2
1p.2
16.96.6
73.8p.9
16.0
15.9
68
53. 1g.3
65.30 4
61.5p.1
12.1
Se
32
69. 5
69 69.9g
90. 1p.1
79. 4
71.5p.5
75.80.7
76 20.9
68 2g
84.
84.7g.1
84.00.2
50.01.2
64
1
90.2n.3
90.00.1
79.40.3
75.10.7
76.20.5
75.
67.80.2
84.20.
84.20.2
83.00.3
51.20.1
16
98.90.3
89.1g.2
89.66.1
18.20.1
74.50.4
To.
75.80.8
40.4
55.5g.1
86.70.1 1
86.80.2
50.32.2
1e-8
32
89 19.10.1
80.3g.2
89.60..
78.60.3
74.40.6
75.90.8
To.T1.0
69 2g.
85.40.1
85.0g.2
85.36.2
48, 48.4
19.10.3
74.81.2
16.10.0
76.7o.4
85.0g.
84.50.1
62.00.8
16
40.3
88 5
89. 1o.2
17.50.2
74.80.7
76.30.3
76 50.
69 7o.
86. 0o.
80.
86.20.1
52.01.3
1.5e
32
89. 0g.5
89
89, 8
78. 78.20.2
75.
76 76.50.4
76
69.
85,
65.50
85 85.80.1
48. 48.31.0
$9.0g.5
80.4g.3
89.5p.2
7o.5
76.50.9
76.20.5
76.30.8
69.33
85-10.2
85.2g
85.20.5
48.9..1
16
97. 76.6
$7.9g 3
88.96.4
75.80.2
74.60.7
70. 70.50.8
70.5u.7
09.30 3g.
85.9g.
55.90.2
86.0p.2
48.25.1
2c 26-6
32
98.
88.8g.1
77.60.3
76.51
85. 85.4g.2
85.4g.3
86.70.a
46.92.5
3o
59.4g.1
89.8p.1
To.
73.9p.7
16.30.3
76.90.5
2g
83.2g.
65.4g a
85.4p.3
40.5
16
87.60.9
67.70.1
88.60.3
74.00.3
75.8p.9
76.40.2
77.20.5
68 69.3g
85.6o.
85.8g.2
86.00.3
49.50.5
2.
6
60.1
89.00.2
76.90.4
74.70.6
76.00.3
77.00.6
68.10.1
85.To.2
85.70.2
86-00.1
00.0
64
48.8 88.80.3
88.9g 9
89.4p.2
78.40.4
76.11.
76.10.7
76.90.9
69.2p 4
85.40.0
85.4g o
85.70.a
40.42.3
16
96. 86.7
58.8p.2
87.70.1
71.70.7
74.1p.6
75.61.5
76.10.8
68.50.2
$3.50.1
85.6g.1
86.20.0
48.52.4
32
8
88 0
89.0p.s
75.30.3
73.81.3
76.25.0
70.5a.s
09.1g.2
45.
85.4g.1
86.0p.1
49.45.0
64
88.
88
5
69.40.2
17.40.7
74.80.6
76.30.2
77.70.4
69.1
85 45.4
85.4
85.80.1
46.21.5
Table 4: Validation performance by task, model variant, and hyperparameters (ef. $3). LAST, SRC-DEV, and CA validate louree-language validation splits; TRG-DEV denotes performance averaged over individual snapshots of a run that perform best by larget-language validation set. For each column, best validation performance in bold. Metries: accuracy for NLI, span-Fi F, for TyDiQA and token-level F, for NER. Subscripts denote std. deviation.


Reviews:
[{"review_id": "scjHRSe52X", "reviewer": "Reviewer_H7uL", "summary": "", "questions": "•\tHow exactly is your sampling method different from Schmidt et al. (2023)? Particularly the RA and RA-CA variants of that paper?\n•\tWhat are the differences in scores between the methods in Scmidt et al. (2023) and your method? I can see tables in both for similar tasks, but very different numbers. How am I supposed to compare them? Ideally, running experiments with the same setup (aka pre-processing, tokenization, etc.) would be better than just reporting numbers from another paper. Since the methods are so similar, this should not be a large ask. This should be shown in a table in the main part of the paper (not an appendix).\n•\tWhat is the difference in runtimes of the methods? It is unclear to me how computationally expensive this method is compared to Schmidt et al. (2023). In particular, their definition of CA seems like it is only one run, so it would be much less computationally complex.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your comprehensive review and for asking specific clarification questions! We believe that our replies will clarify most of the points and lead to more favorable recommendation scores.\n\n> As an initial point, the CA acronym appears to be a bit overloaded and that this work, which also uses CA, appears to be a bit closer to their RA. It would help a lot to clear this up in the writing, or to use a different acronym.\n\nWe would first like to clarify this issue. Please note that CA and cumulative averaging are distinct terms.\n\n- `CA` (checkpoint averaging): a model \"variant\" which averages checkpoints of a single training run (cf. $\\S$3 Model Variants); the term is always abbreviated.\n- Cumulative averaging: average variants (i.e., `LAST`,\n$...$, `CA`) of {$1, ..., r$} training runs with various sampled hyperparameters run-by-run without sub-selecting checkpoints on source-language validation instances (cf. $\\S$2 Cumulative Run Averaging); cumulative averaging is never abbreviated.\n\nWe do acknowledge the unfortunate fact that cumulative averaging could also be abbreviated as CA, overloading the abbreviation. We will change cumulative averaging to \"accumulative averaging\" to separate the two intertwined terms more and avoid confusion. \n\n> How exactly is your sampling method different from Schmidt et al. (2023)? Particularly the RA and RA-CA variants of that paper? What are the differences in scores between the methods in Schmidt et al. (2023) and your method?\n\nSchmidt et al. (2023) report performance of `LAST`, `SRC-DEV`, and `CA` on a *single hyperparameter configuration*, averaged over 5 random seeds (these are the different runs, which they average). This benefits `RA` (i.e., run averaging over 5 runs) over `LAST` (last checkpoint), `SRC-DEV` (optimal checkpoint according to source language dev set performance), and `CA` (average of all checkpoints of the run). There is no model/checkpoint selection over all 5 runs together.\n\nIn contrast, we more realistically benchmark the single-run \\{`LAST`, ..., `CA`\\} model variants of $1, ..., r$ runs with *different hyperparameter configurations* that perform best on source-language validation data -- i.e., selecting the best possible checkpoint from all checkpoints of all $r$ runs (this is our primary baseline) -- against our simple, (ac)cumulative averaging in true zero-shot cross-lingual transfer. \n\nThe results show that simple, (ac)cumulative averaging is a crucial for model averaging in zero-shot cross-lingual transfer:\n\n- Model selection on source-language validation data (\\`max. src-dev') gets stuck in sub-optimal minima for both single-run variants and model averaging (`SOUP`)\n- (Ac)cumulative averaging closely aligns with ideal transfer performance (`TRG-DEV`)\n\nAs the reviewer correctly asserts, we closely mimic the experimental setup of Schmidt et al., 23 (i.e., model variants). The difference in scores stems from running experiments with a different multilingual Transformer as a backbone: we use XLM-R-Large rather than XLM-R-Base (Schmidt et al.) given the recent prevalence of larger LMs.\n\n> I can see tables in both for similar tasks, but very different numbers. How am I supposed to compare them? \n\n **To preface: where applicable, our results are in line with Schmidt et al. (2023).**\n\nAs model sizes differ (theirs: XLM-R-Base; ours: XLM-R-Large), we can only (roughly) compare the relative margins in performance task-by-task between different model variants (i.e., {`LAST`, $...$, `CA`}) between their results and our results.\n\nOne **loose comparison** would be to compare $r=1$ of Table 1 in our submission to Table 1 of Schmidt et al. (2023). $r=1$ reflects the average performance of 10 single runs with varying hyperparameters. Table 1 of Schmidt et al. (2023) reports averages of 5 runs with a single set of hyperparameters. `RA` or `SOUP` of Schmidt et al. could (loosely) be compared against our (ac)cumulative averaging at $r=5$.\n\n> Ideally, running experiments with the same setup (aka pre-processing, tokenization, etc.) would be better than just reporting numbers from another paper. Since the methods are so similar, this should not be a large ask. \n\nBarring evaluating on XLM-R-large rather than XLM-R-base, our experimental setup is aligned with Schmidt et al. (2023). The key difference is how performance of models by averaging checkpoints stored in various runs (theirs: `RA`, ours: (ac)cumulative averaging) is presented. As mentioned above, they re-run training on the same set of hyperparameters. We simulate re-training models with highly diverse learning rates and batch sizes and cumulatively average \\{`LAST`, $...$, `CA`\\} of these runs.\n\n> What is the difference in runtimes of the methods? It is unclear to me how computationally expensive this method is compared to Schmidt et al. (2023). In particular, their definition of CA seems like it is only one run, so it would be much less computationally complex.\n\nCrucially, cumulative averaging does not incur additional compute costs to standard hyperparameter tuning. The only cost comes from re-training models, which, if hyperparameters are tuned, is an expense that is incurred anyways. With respect to a single run (cf. $r=1$), we confirm that `CA`, in expectation, outperforms `LAST` and `SRC-DEV` in **true** zero-shot cross-lingual transfer. While (ac)cumulative averaging over multiple runs is clearly more computationally demanding than CA over a single run (r = 1 in Table 1) it brings substantial performance gains already for r = 2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a useful contribution to the field that demonstrates a simpler, and more effective way, of doing zero-shot cross-lingual transfer.", "reasons_to_reject": "The model is not compared with (or is unclear if it is) with the baselines of Schmidt et al. (2023)", "questions_to_authors": "•\tHow exactly is your sampling method different from Schmidt et al. (2023)? Particularly the RA and RA-CA variants of that paper?\n•\tWhat are the differences in scores between the methods in Scmidt et al. (2023) and your method? I can see tables in both for similar tasks, but very different numbers. How am I supposed to compare them? Ideally, running experiments with the same setup (aka pre-processing, tokenization, etc.) would be better than just reporting numbers from another paper. Since the methods are so similar, this should not be a large ask. This should be shown in a table in the main part of the paper (not an appendix).\n•\tWhat is the difference in runtimes of the methods? It is unclear to me how computationally expensive this method is compared to Schmidt et al. (2023). In particular, their definition of CA seems like it is only one run, so it would be much less computationally complex.", "paper_topic_and_main_contributions": "Increasing score after rebuttals ... assuming the authors will add the clarifications into the paper\n\n``````````````````````````````````````````````````````\n\n\nThis paper proposes a solution to the problem of over-fitting on source languages in zero-shot cross-lingual transfer by averaging models. In general, this is one of these papers whose benefits lie in its simplicity. By simply showing that model averaging, common in other NLP tasks, such as MT which they identify, can outperform more complicated methods that rely on more data and are tough to reproduce, the paper is a useful contribution to the field. The authors do a good job motivating the paper with why zero-shot doesn’t work – including highlighting issues with using validation examples in the target languages. However, my biggest concern with the paper is baseline comparisons and situating the work in relation to other work – particularly Schmidt et al. (2023). \n\nIn Schmidt et al. (2023), which the authors cite extensively, there is Checkpoint Averaging (CA) and Run Averaging  (RA). As an initial point, the CA acronym appears to be a bit overloaded and that this work, which also uses CA, appears to be a bit closer to their RA. It would help a lot to clear this up in the writing, or to use a different acronym.\n\nSecondly, since the authors cite Schmidt et al. (2023) so extensively, and mention that it motivates some of their research questions, I would have liked to have seen a more clear comparison their work and to use their work as a baseline. Normally, I would not ask a submission to reproduce work that has been on arxiv for only a few weeks before this conference’s deadline, but the fact that the paper is cited 10 times in this submission, makes me want to see an empirical comparison.\n\nThe bulk of my review basically boils down to a few questions that I would like to see answered by the authors in the rebuttal period:\n•\tHow exactly is your sampling method different from Schmidt et al. (2023)? Particularly the RA and RA-CA variants of that paper?\n•\tWhat are the differences in scores between the methods in Schmidt et al. (2023) and your method? I can see tables in both for similar tasks, but very different numbers. How am I supposed to compare them? Ideally, running experiments with the same setup (aka pre-processing, tokenization, etc.) would be better than just reporting numbers from another paper. Since the methods are so similar, this should not be a large ask. This should be shown in a table in the main part of the paper (not an appendix).\n•\tWhat is the difference in runtimes of the methods? It is unclear to me how computationally expensive this method is compared to Schmidt et al. (2023). In particular, their definition of CA seems like it is only one run, so it would be much less computationally complex.\n\nOverall, I liked the paper and would be happy to increase my score after author rebuttal if the above questions are answered. The paper has a lot of potential – but until the differences with a paper cited 10 times are reconciled, it is hard to give a good score.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Section 2 could define variables a bit more clearly.\n\nSection 3. Training Details. “We train XLM-R….” I think this should be “We fine-tune XLM-R…”"}, "extra_text": {}}, {"review_id": "5NYtW0Dh7R", "reviewer": "Reviewer_Znmg", "summary": "", "questions": "* Q1: While I really like the paper as a whole, ~I’m pretty skeptical about the proposed CA. How is it different from other strategies like model ensembling or checkpoint averaging? IIRC those strategies can be used without looking at target-language instances. In addition, I would like to see more intuitive explanations of how CA works.~\n* Q2: What’s really wrong with existing LAST and TRG-DEV? My current take is that we can use TRG-DEV if there is a target dev and LAST if there isn’t. ~CA~ accumulatively averaging seems to have a trade-off of slightly improved performance (only 0.1-ish) with the compensation of lower reproducibility over LAST due to its complexity. I would love to see more pushes for switching from LAST to ~CA~ accumulatively averaging.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and general appreciation of our work. We believe that there some important misunderstandings, which we attempt to clarify below. We hope that resolving this confusion will result in more favorable view of the merits of our work. \n\n> Q1: While I really like the paper as a whole, I’m pretty skeptical about the proposed CA. How is it different from other strategies like model ensembling or checkpoint averaging? IIRC those strategies can be used without looking at target-language instances. In addition, I would like to see more intuitive explanations of how CA works.\n\nWe first would like to clarify that `CA` and cumulative averaging denote different concepts: \n\n- `CA` always refers to averaging checkpoints of a single training run (cf. $\\S$3 Model Variants); the term is always abbreviated.\n- Cumulative averaging: average variants (i.e., `LAST`,   $...$, `CA`) of \\{$1, ..., r$\\} training runs with various sampled hyperparameters run-by-run without sub-selecting checkpoints on source-language validation instances; cumulative averaging is never abbreviated.\n\nWe do acknowledge the unfortunate fact that cumulative averaging could also be abbreviated as CA, overloading the abbreviation. We will change cumulative averaging to \"accumulative averaging\" to separate the two intertwined terms more and avoid confusion. \n\nIn other words, we do **not** introduce a new approach to model averaging. Our key contribution is as follows. If \\{$1, ..., r$\\} models are trained to maximize transfer performance (i.e., model selection over $r$ runs), it is better to cumulatively average all the $r$ `LAST`, `CA`, or `SRC-DEV` models rather than selecting the best individual `LAST`, `CA`, or `SRC-DEV` model on source-language validation instances among $r$ runs. We show that simple, (ac)cumulative averaging is crucial for model averaging in **true** zero-shot cross-lingual transfer:\n\n- Model selection on source-language validation data (\\`max. src-dev') gets stuck in sub-optimal minima for both single-run variants and model averaging (i.e., see `SOUP` in Table 2)\n- (Ac)cumulative averaging closely aligns with ideal transfer performance (`TRG-DEV`)\n\nUnlike our submission, Schmidt et al. (2023) did not fairly compare single-run variants against averaging models from different runs. For a more detailed comparison, please refer to our answer of the first question of Reviewer 1.\n\n> Q2: What’s really wrong with existing LAST and TRG-DEV? My current take is that we can use TRG-DEV if there is a target dev and LAST if there isn’t. CA seems to have a trade-off of slightly improved performance (only 0.1-ish) with the compensation of lower reproducibility over LAST due to its complexity. I would love to see more pushes for switching from LAST to CA.\n\nPlease note again that `CA` (i.e., checkpoint averaging over a single run) is not the contribution of this work. (Ac)cumulative averaging of checkpoints over multiple runs (for $r > 1$) is the central contribution of our work. (Ac)cumulative averaging is the strategy that can be executed over any of the single-run checkpoint selection strategies (i.e., `CA` or LAST or SRC-DEV), as visible from Table 1, where cumulative averaging has three subcolumns corresponding to the three within-run model selection strategies (LAST, S-DEV, and `CA`).   \n\nThat said, in the context of a single run ($r=1$), `CA` does outperform LAST by non-negligible margins. The margins are generally slimmer in higher-level tasks like NLI and generally larger in token-level tasks like NER. With cumulative averaging, `CA` frequently also frequently performs better than `LAST`.\n\nIf annotated target-language instances are available (i.e., `TRG-DEV`), they should not be used for model selection but for training! The gains of model selection on such sizable target-language validation instances (often 100s of annotations) would be easily surpassed tremendously by training on target-language instances (without further validation) instead.\n\n> The proposed new protocol CA seems to have some unwritten downsides due to its complexity, so the strong claim of advantages over existing protocols is a bit doubtful (see my questions).\n\nPlease see our clarification of the differences of `CA` and cumulative averaging above. We do not propose `CA` (CA was used in many works before in various tasks, from MT and in computer vision to Schmidt et al., 2023 who leverage it in the context of cross-lingual transfer). We introduce *cumulative averaging* (which we will rename to accumulative averaging to avoid confusion), a strategy that cumulatively selects and averages in models variants (individual or CA-ed checkpoints) from different runs. \n\nOur protocol is indeed somewhat more expensive than evaluating model variants ({`LAST`, ..., `CA`}) in a single-run context. However, previous work frequently performs model selection on source- or even target-language validation instances for zero-shot cross-lingual transfer over various runs. The core argument of our approach is that, if we are already retraining the model several times for model selection, we achieve better (or even ideal `TRG-DEV`-like) transfer performance without target-language instances by cumulatively averaging the resulting runs, rather than identifying the best model within those runs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* A comprehensive study of the evaluation protocol for cross-lingual transfer is presented, which is usually overlooked in practice, but it is a very important topic with respect to reproducibility.\n* In particular, the literature review part stands as a good contribution to the community.", "reasons_to_reject": "* The proposed new protocol ~CA~ accumulatively averaging seems to have some unwritten downsides due to its complexity, so the strong claim of advantages over existing protocols is a bit doubtful (see my questions).", "questions_to_authors": "* Q1: While I really like the paper as a whole, ~I’m pretty skeptical about the proposed CA. How is it different from other strategies like model ensembling or checkpoint averaging? IIRC those strategies can be used without looking at target-language instances. In addition, I would like to see more intuitive explanations of how CA works.~\n* Q2: What’s really wrong with existing LAST and TRG-DEV? My current take is that we can use TRG-DEV if there is a target dev and LAST if there isn’t. ~CA~ accumulatively averaging seems to have a trade-off of slightly improved performance (only 0.1-ish) with the compensation of lower reproducibility over LAST due to its complexity. I would love to see more pushes for switching from LAST to ~CA~ accumulatively averaging.", "paper_topic_and_main_contributions": "This paper presents a comprehensive study of the evaluation protocol for cross-lingual transfer. Within this cross-lingual transfer setting where training (train), development (dev), and test data may have been written in different languages, prior studies showed that an usual procedure using the dev written in the same language with train (SRC-DEV) for model snapshot selection and hyperparameter tuning is not a good fit with respect to test time performance. Therefore, this study explored alternative strategies including the use of the last checkpoint (LAST), the dev written in the same language with test (TRG-DEV; regarded as a potential upper bound), and cumulatively averaging within-run snapshots (CA), as well as a new meta-selection proposal called accumulatively averaging.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* C1: Standard deviation in tables is very nice information, but I have the following two doubts:\n  * C1-1: Does it actually mean “standard deviation” of our model parameters? AFAIK after changing random seed used for training, outputs would no longer fall into the same standard deviation. So I personally prefer showing raw values with ranges (+-) as compensation.\n  * C1-2: In addition, considering such a difficulty, is it safe to say “CA … often outperforms LAST and SRC-DEV by notable margins” (L202-204) ?"}}, {"review_id": "E7Ey9982nP", "reviewer": "Reviewer_c3Ph", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We kindly thank you for your thoughtful review. Below we provide replies to individual concerns and methodological proposals.\n\n> This work could benefit from the inclusion of a simple prediction ensemble baseline. Instead of averaging 10 snapshots, an alternative baseline could involve averaging the predictions from individual snapshots (for instance, employing majority votes or averaging the logits).\n\nThis arguably is a viable alternative. The reasoning behind not analyzing logits/prediction ensembling is two-fold:\n\nFirst, model averaging does not increase model complexity, whereas prediction ensembling does. Crucially, prediction ensembling exacerbates computational complexity (both space and time complexity): inference requires storing \"N\" models and making \"N\" forward passes (one with each model) for the same input instance. A fair evaluation should then compare an ensemble of \"N\" models with a single model that is \"N\" times larger. LM scaling laws empirically demonstrate this works better in NLU tasks than prediction ensembles. \n\nSecond, [Wang et al.](https://aclanthology.org/2022.emnlp-main.388.pdf) show that averaging adapters in loosely comparable manner beats logit-based ensembling on comparable NLU tasks. \n\nWe omitted this discussion due to space constraints (after all, this is a short paper submission), but will gladly include it in the camera ready version (provided the extra page).  \n\n> The experiments in this study focus solely on English-to-X cross-lingual transfer, disregarding standard baselines like \"translate-train\". Incorporating this stronger baseline as a model selection technique might result in higher performance. I am particularly intrigued to observe how this inclusion could impact the gap between S-DEV and CA.\n\nWe agree that \\`translate-train' would be quite interesting to consider. We respectfully would like to argue that the absence of \\`translate-train' should not be considered a \"reason to reject\" our work. Cross-lingual transfer by means of translation (translate-train or translate-test) is indeed an established paradigm, one that works particularly well for transfer between high-resource languages between which MT is reliable. For low(er)-resource languages, the conventional transfer by means of multilingual LMs (the paradigm which we address in this work) is better. Besides low-resource languages, translation-based XLT is challenging for token-level tasks like NER or question-answering tasks like TyDiQA-GoldP, for which label spans must be aligned in order to do label projection. \n\nOur averaging protocol is designed for the conventional XLT transfer with multilingual LMs, which is an established and prominent XLT technique and we show that our cumulative averaging improves this established XLT approach. We would also like to point out that our submission is a short paper, the expectation of which is a focused contribution. We believe that cumulative averaging, which improves one established XLT paradigm, is exactly such a focused contribution.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper proposes cumulative averaging as a means to report the performance of zero-shot cross-lingual transfer, instead of relying on the last checkpoint or selecting one based on the source language dev set.\n\n2. The paper conducts experiments on NER, NLI, and QA, encompassing a total of (25+9+10, 25, 9) languages, with 21 configurations of hyperparameters and 3 random seeds to support the claims made in the introduction.", "reasons_to_reject": "1. This work could benefit from the inclusion of a simple prediction ensemble baseline. Instead of averaging 10 snapshots, an alternative baseline could involve averaging the predictions from individual snapshots (for instance, employing majority votes or averaging the logits).\n\n2. The experiments in this study focus solely on English-to-X cross-lingual transfer, disregarding standard baselines like \"translate-train\". Incorporating this stronger baseline as a model selection technique might result in higher performance. I am particularly intrigued to observe how this inclusion could impact the gap between S-DEV and CA. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "In the existing literature, the zero-shot cross-lingual transfer experiment setting often involves selecting model checkpoints based on the source language development data. However, this practice has been found to result in suboptimal performance for the target language (Keung et al., 2020; Chen and Ritter, 2021). Prior studies (Hu et al., 2020) have reported average results of model runs without any hyperparameter tuning.\n\nTo address this limitation, this paper proposes the adoption of the \"model averaging\" approach, wherein model checkpoints from different runs (during training) are cumulatively averaged into a single model. The experiments, which include tasks like NLI, NER, QA, demonstrate the following key findings:\n\n1. Cumulative averaging leads to more robust cross-lingual transfer checkpoints on NLI, NER, and QA, when compared to the MAX SRC-DEV strategy.\n\n2. The LAST/S-DEV/CA approach, coupled with cumulative averaging, yields improved performance compared to counterparts using MAX SRC-DEV.\n\n3. Cumulative averaging with CA shows a positive correlation with the combination of Max Dev and TRG DEV (\"oracle\").\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Model Selection for Cross-Lingual Transfer, Chen and Ritter 2021, EMNLP (https://arxiv.org/abs/2010.06127)\nZero-shot cross-lingual transfer language selection using linguistic similarity, Eronen et al., (https://www.sciencedirect.com/science/article/pii/S030645732200351X)", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is indeed well-written; however, it could be further improved by providing a clearer explanation and distinction between \"CA\" (Line 186) and \"Cumulative Averaging\" (Lines 140-142) in Table 1. I was confused a little bit to distinguish between a weight-averaging technique and a model selection method.\n"}}]

Author rebuttal:

Meta-review: This paper proposes a cumulative averaging of model snapshots from different runts. This aims to avoid the need for extensive hyper-parameter search for zero-shot tasks. Overall the idea is innovative and might open the doors for future research. However, the reviewers also found a number of issues: lack of proper testing on standard methods like ``English-to-X cross-lingual transfer, disregarding standard baselines like "translate-train"''(Rev c3Ph); an unlcear writing. In general, having a stronger comparison with baselines could help to improve the paper. That said, the authors  clarified some not clear aspects of the paper.
Decision: Accept-Findings