Unifying Cross-Lingual Transfer across Scenarios of Resource Scarcity
Authors: Alan Ansell
Marinela Parović
Ivan Vulić
Anna Korhonen
Edoardo Ponti
Venue: EMNLP 2023 Main (2023)
Keywords: cross-lingual transfer
low-resource scenarios
parameter-efficient fine-tuning
TL;DR: We investigate how to integrate cross-lingual transfer techniques to address a range of scenarios of resource scarcity.

Abstract: The scarcity of data in many of the world's languages necessitates the transfer of knowledge from other, resource-rich languages. However, the level of scarcity varies significantly across multiple dimensions, including: i) the amount of task-specific data available in the source and target languages; ii) the amount of monolingual and parallel data available for both languages; and iii) the extent to which they are supported by pretrained multilingual and translation models. Prior work has largely treated these dimensions and the various techniques for dealing with them separately; in this paper, we offer a more integrated view by exploring how to deploy the arsenal of cross-lingual transfer tools across a range of scenarios, especially the most challenging, low-resource ones. To this end, we run experiments on the AmericasNLI and NusaX benchmarks over 20 languages, simulating a range of few-shot settings.
The best configuration in our experiments employed parameter-efficient language and task adaptation of massively multilingual Transformers, trained simultaneously on source language data and both machine-translated and natural data for multiple target languages. In addition, we show that pre-trained translation models can be easily adapted to unseen languages, thus extending the range of our hybrid technique and translation-based transfer more broadly. Beyond new insights into the mechanisms of cross-lingual transfer, we hope our work will provide practitioners with a toolbox to integrate multiple techniques for different real-world scenarios. Our code is available at https://github.com/parovicm/unified-xlt.

Unifying Cross-Lingual Transfer across Scenarios of Resource Scarcity
Alan A
Marinela P Ivan Vulić
Anna Korhonen' Edoardo Maria Pontf Language Technology Lab, University of Cambridge
University of Edinburgh
(aja63, mm939]0cam ac. uk
Abstract
category (Joshi et al.. 2020: Blasi al., 2022). The lack of labelled data makes difficult or im- possible to directly train effective NLP systems for these languages. For this reason, researchers have looked for ways to harness data from one or more high-resource "source" languages to compen- sate for a shortage of data in low-resource "target" languages in process known "cross-lingual transfer" (XLT). Different techniques have been developed deal with the various dimensions of resource scarcity, which encompass not just data availability, but also the degree of support by pre- trained models. These research threads have gener- ally been investigated somewhat independently. In this paper, we attempt to unify several of the most prominent threads of XLT research into single framework. Specifically, we synthesise findings from zero-shot (ZS) XLT with massively multi- lingual transformers (MMTs), few-shot (FS) XLTT, XLT for low-resource languages and XLT through machine translation (MT) to formulate practical, general-purpose approach to cross-lingual transfer. with a focus on low-resource scenarios.
The scarcity of data in many of the world's lan- guages necessitates the transfer of knowledge from other, resource-rich languages. However, the level of scarcity varies significantly across multiple dimensions, including: i) the amount task-specific data available in the source and target languages; ii) the amount of monolin- gual and parallel data available for both lan- guages; and iii) the extent which they are supported by pretrained multilingual and trans- lation models. Prior work has largely treated these dimensions and the various techniques for dealing with them separately; in this paper, we offer a more integrated view by exploring how to deploy the arsenal of cross-lingual trans- fer tools across a range of scenarios, especially the most challenging, low-resource ones. To this end, NLI and NusaX benchmarks over 20 languages, simulating a range of few-shot settings. The best configuration in our experiments employed parameter-efficient language and task adapta- tion of massively multilingual Transformers, trained simultaneously on source language data both muchine-translatdd and natural data for multiple target languages. In addition, show that pre-trained translation models can easily adapted to unseen languages, thus extending the range of our hybrid technique and translation- based transfer more broadly. Beyond new insights into the mechanisms of cross-lingual transfer, we hope work will provide practitioners with a toolbox to integrate multiple techniques for different real-world scenarios. Our code is available at https: //gi tig. httphhuu...oo////ee wwwccooo/oooooeeeeo comm/parovicum cwtroviicmuuuue cm/uni fie-xxlt.
experiments the Americas-
Massively multilingual Transformers (MMTs), Transformer-based architectures (Vaswani et al.. 2017) pretrained with an unsupervised objective such masked language modelling (MLM) on text from large number of languages, are per- haps the most fundamental tool for contempo- rary XLT. Prominent examples of MMTs include mBERT (Devlin et al., 2019), XLM-R (Conneau et al., 2020a) and mDeBERTa (He et al., 2023). In addition to providing broad language cover- age, MMTs have been shown to learn represen- tations which have a degree of cross-lingual align- ment, even though they do not receive any explicit cross-lingual signal during training (Conneau al., 2020b; Muller et al., 2021). This allows MMT fine-tuned for a specific task in given "source" language to perform the same task another "tar- get" language with a level of performance generally
1 Introduction
While training data available for wide range of NLP tasks in a handful of high-resource lan- guages, the vast majority of the world's languages with their billions of speakers do not fit into this
"qqual contribution.

---

techniques and the different data sources that they exploit are complementary, (ii) what is the most effective way of combining different data sources in order to maximise the performance, and (iii) While MMTs typically cover around 100 lan- guages, this is still only a small fraction of the world's estimated 7,000 languages. Pfeiffer et al. (2020) and Ansell et al. (2022) have shown that an effective strategy for ZS-XLT for target languages not covered by the MMT is to learn parameter- efficient fine-tuning (PEFT) specialise the MMT to that "unseen" language. The resulting language module can be composed with a task module, typ- ically yielding much better performance than the MMT could achieve without language adaptation.
much better than random chance (Pires al., 2019; Wu and Dredze, 2019), despite never having seen a single example of the task in the target language; this is known as "zero-shot" cross-lingual transfer.
how much each of the available sources of data contributes to the overall performance. We aim to equip practitioners with a recipe for how to use the available data resources in the most effective way.
We experiment on the AmericasNLI natural lan- guage inference (NLI) dataset for American lan- guages (Ebrahimi et al., 2022), and the NusaX sen- timent analysis (SA) dataset for Indonesian lan- guages (Winata et al., 2023). We find that com- bining language adaptation, few-shot learning and translation can be highly effective, yielding aver- age performance gains of 14-24 points over the zero-shot baseline without language adaptation.
In contrast to ZS-XLI, often more realistic is the few-shot case (FS-XLT) where a small number of gold-standard target language examples are avail- able during training. Though it may be expensive annotate target language data, especially for low- resource languages where native speakers are hard to access, prior work has shown that using even a small amount during training can yield significant gains in performance (Zhao et al., 2021). While early approaches to FS-XLT involved fine-tuning first on the source language data, then separately on the few target language shots (Lauscher et al., 2020), recent work has shown that it is more effec- tive to jointly train on both at once (Xu and Murray, 2022: Schmidt et al., 2022).
2 Methodology
2.1
Background
Because MMTs divide their capacity among many languages, they may often perform sub-optimally with respect to a single source or target language. Furthermore, we are sometimes interested in a tar- get language not covered by the MMT. A naive solution to these problems is to prepare the MMT with continued pretraining on the target language before proceeding to task fine-tuming. While this can improve performance, Pfeiffer et al. (2020) and Ansell et al. (2022) show that more effective ap- proach is to apply form of composable, parameter- efficient fine-tuning during continued pretraining: Pfeiffer et al. (2020) employ adapters (Rebuffi et al., 2017: Houlsby et al., 2019), while Ansell et al. (2022) propose sparse fine-tunings (SFTs) learned through an algorithm they call "Lottery Ticket Sparse Fine- Tuning" (LT-SFT). The result- ing langunge-specific module ("language module") can be composed with similar module trained for the task of interest ("task module") to perform zero-shot transfer. This approach is not only more efficient than sequential full fine-tuning in terms of model size and training time; keeping the MMT weights frozen while training the language modules helps prevent the model from forgetting important knowledge learned during pretraining.
Another tool often employed for cross-lingual transfer is machine translation (MT). MT ap- proaches can generally be categorised as translate- train or translate-test (Hu et al., 2020). To con- fine the scope of our work, we consider only the translate-train approach, which has so far been pre- dominant, although Artetxe et al. (2023) have made a strong case for considering translate-test further in future work. We consider two translate-train vari- ants: TTRAIN-SINGLE where a model is trained for each target language using only its own translated data; and TTRAIN-AL.E. where one model covering all target languages is trained on their translated data and the source language data simultaneously.
In this work, we consider how best to employ the above techniques in response to cross- lons-linguua trans- fer scenarios with varying levels of data scarcity. We thus explore several promising directions of integrating the zero-shot, few-shot and translate- train techniques across a range of resource levels in order to delve deeper into: (i) to what extent these
While SFT composition generally exhibits some- what better zero-shot cross-lingual transfer perfor- mance across a range of tasks than adapter compo- sition (Ansell et al., 2022; Alabi et al., 2022), and avoids the overhead incurred during inference by

---

adapters, adapters are more efficient to train and leave the base MMT fully unmodified. In this work, we consider both methods. However, we note that other modular, parameter-efficient methods exist and could be used with our tools in future work (Pfeiffer et al., 2023).
support. We therefore adapt NLLB unseen tar- get languages by initialising a new language token and embedding for the target language and then performing continued pretraining with parallel data for the relevant language pair.'
Experimental Setup
Multi-Source Training. Multi-source training is an extension to parameter-cffiiiet language adap- tation, where a task adapter is trained using data from several source languages simultancously, of- ten yielding large gains in cross- cross-lingual lingual transfer performance as result of the task adapter learn- ing more manguugg--gnosstee representations (Ansell et al., 2021, 2022). Multi-source training requires that each training batch consists of examples from the same language so that the module for the rele- vant language can be applied during each step.
3.1 Evaluation Tasks and Languages
We evaluate our models on two classification tasks: natural language inference (NLI) and sentiment analysis (SA). For NLI, we use the AmericasNLI dataset (Ebrahimi et al., 2022), which covers 10 low-resource languages from the Americas. For SA, we opt for the NusaX dataset (Winata et al., 2023), spanning 10 low-resource Indonesian lan- guages. In the NLI task, the source language is English, while for SA it is Indonesian. We pro- vide the list of all datasets and languages used in Table These tasks are particularly amenable to the translate-train approach since the labels are preserved even after data has been translated into another language. The complete overview of the languages and their codes is given in Appendix A.
2.2 Recipe
We propose a recipe for cross tross-lingual transfer which is flexible and effective across scenarios of resource scarcity. It can be summarised as: 1. Select a base MMT and train language modules for the source language and all target languages for which monolingual data is available.
3.2 Models and Training Details
Using a multilingual NMT model, translate the task data into every target language it supports; if there are target languages the MT model does not support but for which parallel data is available, adapt using this parallel data.
MMT. In this work, we use the base version of XLM-R (Conneau et al., 2020a), an MMT with
270M parameters pretrained on 100 languages.
NMT model. As primary MT model for ob- taining translated data, we choose the NLLB model with 3.3B parameters (NLLB Team et al., 2022). trained to translate between any pair of 200+ lan- guages, including many low-resource languages. We also experiment with two additional NLLB vari- ants: distilled models with 600M and 1.3B parame- ters, enabling us understand the effect of model size on the "quality" of the obtained data. Despite the broad language coverage, half of our target lan- guages are unsupported by the NLLB models (7
3. Learn a task module through joint multi-source training on all available data (i.e. source data, translated data and any gold-standard target lan- guage ("few-shor") data available).
We propose two methodological novelties to en- hance this recipe.
Few-shot upsampling. When gold-standard target language data is available during training, it is gen- erally present in much smaller quantity than the source language data (in our case, the difference is in orders of magnitude, but it can vary). Further- more, is typically higher in quality than machine- translated target language data. For this reason, we suggest upsampling this few-shot data relative to the source and machine-dranslated data during multi-source training. We show in our experiments that this can improve downstream performance.
'In fact, it is not necessary for the source language in the parallel corpus match the intended source language for cinss-lingual transfer, since multilingual NMT models can, theory, support unseen transfer directions provided the source and target languages have been seen as part other pairs during training: this the in NMT adaptation for AmericasNL.L J, where the parallel data Spanish-to- X but the transfer direction is English-to-Xx
"While more powerful MMTs are available, such as XLM- R-large or mDeBERTa (He et al., 2023), our primary purpose is not the maximisation of raw performance, nor a comparison of different MMTss so we opt for a smaller model to stretch our computational hudget over a broad range of scenarios and languages.
NMT model adaptation. While recent multilin- gual NMT models such as NLLBB (NLLB Team et al., 2022) provide impressive language cover- age, there are still many languages they do not

---

Task
Target Dataset(s)
Source Dataset(s)
MMT
Target Languages
Aymara", Asháninka"
Bribri
Natural Lan-
AmericasNLI (Ebrahimi
MultiNLI (tr: 393K / dev: 10K) (Williams el al., 2018)
XLM-R
Guarani", Náhuatl
Otomi
guage Infer- ence (NLD)
et al., 2022) (sh: 743/ ist: 750)
Base
Qucchua", Rarámuri",,
Shipibo-
Konibo Wixarika"
SMSA 1IK, dev: 1.3K) (Purwarianti and XLM-R Crisdayanti, 2019; Wilie Base al., 2020)
Acchnese", Balinese", Banjarese", Buginese", Javanese. Madurese"
Sentiment Analysis (SA)
NusaX (Winata et al., 2023) (sh: 600, tst: 400)
Minangkabau Ngaju" Sundanese.
Toba Batak
Table I: Details of tasks, datasets, MMTs and languages involved in our experiments. sh = # of few-shot examples available per target language; tst of test set examples; = of train examples; dev of development set examples; denotes languages unseen during MMT pretraining; denotes languages not supported by the NLLB MT model; 1 denotes languages for which no satisfactory monolingual corpus was available and hence no language module was trained. Further details of all the language and data sources used are provided in Appendix A. Note that since the NusaX dataset is created through human translation of subset of the SMSA dataset, we carefully remove every example from SMSA which appears in its original or modified form in the NusaX test set to avoid a data leak.
languages from the AmericasNLI and languages from NusaX dataset).
more than one language, the training examples are batched such that each batch consists of examples from single language, and the batches ordered randomly. For the configurations which employ language adaptation, the language module for the relevant language activated at the beginning of the training step and deactivated at the end of the step, following Ansell et al. (2021).
We adapt the 3.3B parameter NLLB model to unseen languages through continued pretraining on the parallel corpora listed in Appendix A. We perform full fine-tuning for 5 epochs with a batch size of 8 and an initial learning rate of 2 10-5 which is linearly decreased to zero during training.
AmericasNLI task modules are trained for 5 epochs with a batch size of 32 and an initial Iearn- ing rate of 10-5 Evaluation is carried out every 625 steps and the checkpoint with the best eval- uation accuracy is selected at the end of training. NusaX task modules are trained for 10 epuchs (or 3 during the full fine-tuning phase of LT-SFT), with a batch size of 16 and an initial learning rate of 2 10 5. They are evaluated after every 250 steps and the final module is the one with the best eval- uation Fl score. For both tasks, the learning rate is linearly decreased zero over the course of training.
Language Modules. In general, we use the same algorithms and hyperparameters as the original papers (Pfeiffer et al., 2020; Ansell et al., 2022) when training language modules. However, we use the variant of MAD-X proposed by Pfeiffer et al. (2021), where the last adapter layers are dropped for an increase in cross-lingual transfer performance. We provide list of resources for the monolingual corpora in Appendix A. Language modules are trained for minimum of 100 epochs and 100,000 steps with a batch size of 8, a learning rate of 3 10 5 and a maximum sequence length of 256. We evaluate the language modules every 1,000 steps with low-resource languages, and every 5,000 steps with high-resource languages. Finally, we choose the module that has obtained the lowest perplexity on the validation set, which is created by taking 5% of the unlabelled data for low-resource languages or 1% for high-resource languages.
3.3 Configurations and Ablations
ZS-XLT. We include zero-shot transfer results with language adaptation, equivalent MAD-X (Pfeif- fer al., 2020) in the case of adapters. We also have a variant where language adaptation is not employed, thus only the task module used for training and inference. These variants are denoted by ZS and Zs - LA, respectively.
Task Modules. We again follow Pfeiffer et al. (2020) and Ansell et al. (2022) except where stated otherwise. We train task adapters with reduction factor of 16 (i.e. the ratio between the dimension of the MMT hidden state and the dimension of the adapter hidden state 16) and task SFTs with 8% density. When jointly training on data from
FS-XLT. In our default FS-XLT setup. "FS- SINGLE" we add K 100 target shots to the source language task data, training separate task module for each target language. We also consider "FS-ALL", where a single task module is trained

---

on the source language data plus K 100 shots from each target language. We investigate the ef- fect of different numbers of shots by also carrying out FS-SINGLE experiments with K E 500). We employ language adaptation in all these setups, but as an ablation, we also test K = 100 without language adaptation (denoted as FS LA).
with ablations shown in Table 3. We find that the various cross- -ooss-lingual transfer techniques we consider can be combined very effectively to im- prove performance. For instance, the average SA performance can be improved from the most basic zS - LA setting by 14-17 points (depending on the PEFT method) through the use of language adapta- tion, translate-train and few-shot techniques with K 100 shots (FS + TTRAIN-ALLL). In the case of the NLI task, the gains under the same condi- tions are 19-24 points. Each of these components individually adds several points of performance. and although the gain from using FS and TTRAIN together is much smaller than the sum of their indi- vidual gains, it is still 1-2 points better than using either technique on its own Although we did not consider FS + TTRAIN-ALL with K 500 shots, the strength of I'S-SINGLE with K 500 as shown in Figure suggests that this gap would be larger with larger K. The finding that high-quality ma- chine translation of the entire source dataset cannot eliminate the utility of human-crafted examples contains a potentially useful lesson we would encourage designers of datasets for lrngsalingual transfer to provide at least two splits for target lan- guages, even if the training/validation split con- tains only 100 examples. The relative value of few-shot and machine-translane data appears to be dask-dependent. Whereas for AmericasNLI we see TTRAIN outperforming FS by 4-6 points, neither approach has clear advantage on the NusaX task.
In all FS experiments, the model is jointly trained on source and target data, per Xu and Murray (2022). We upsample the data in the target language(s) by a factor of 10 to increase its pres- ence during training, since K is still rather small compared to the number of examples available in the source language. During training, we only eval- uate on the source language data following Xu and Murray (2022), who point out that the presence of large evaluation sets in truly low-resource lan- guages is unrealistic*, and show that while evalu- ating on the target language is still beneficial for the joint training procedure, the gap becomes much smaller. They stress that such data would be better used for training, in line with Kann et al. (2019).
Translate- Train, In our main translate-train vari- ant, named TTRAIN-ALA ALL, we create single task module covering all the target languages, which is trained and evaluated on the translated data of all target languages together with the source language data. We also consider TTRAIN-SINGLE, where a separate task module is trained on the data of each target language alone.
FS-XLT meets Translate-Tren In final set of experiments, we investigate to what extent the ben- efits gained from the few-shot and translate-train methods add up when they are combined. To test this, we introduce the FS + TTRAIN-ALL, configu- ration, where we train a single task module on the union of the source language data and translated and few-shot data (with K = 100) for every target language. This module is evaluated on the source language data and the translated data in all target languages.
MT Model Size. In Table 4, we see the effect of translation model quality on ITRAIN-ALI perfor- mance, with gains of 0.5-2 points from upscaling the NLLB model from 600 million to 3.3 billion pa- rameters. This upscaling comes at relatively small cost in the translate-trant -tain setup, since the training data only needs to be translated once for each tar- get language. Translate-tes! setups, on the other hand, incur the cost of translating each example encountered at inference time, which is potentially much more costly for large-scale deployments.
4 Results and Discussion
MT adaptation. The adaptation of the NMT model to new languages appears to be highly effec- tive, with these languages generally enjoying large gains from the use of TTRAIN despite the small size of the parallel corpora available: the AmericasNL.I
Main Results. The results of our primary config- urations on NLI and SA are presented in Table 2.
FFor the ALe target languages NAH OTO we actually use 223 and 377 shots respectively under the 500 setting since this is the maximum available.
'The requirement for evaluation data in the target language originated from two-step methods, where such data is needed to prevent the model from overfitting to the small number of examples in the target language used during the second stage.
'For Nusa-X, our main results exclude Nu for which no monolingual data is available and thus no language module was trained; results without language adaptation are available in Appendix B.

---

Method
QUY
avg
ZS
53.0
42.8
44.8
59.6
40.3
50.8
41.2
55.2
50.0
40.0
47.77
SET ADAPTER
FS-SINGLE
55.2
47.1
47.7
58.3
40.7
55.8
45.9
59.7
52.9
47.5
51.08
FS-ALL
57.1
47.9
49.3
59.1
44.1
54.5
48.4
59.6
52.3
47.5
51.98
STRAIN-SINGLA
58.7
56.5
53.9
64.4
48.7
56.4
40.1
61.5
57.5
51.5
54.92
TTRAIN-ALL
63.3
60.5
56.1
66.1
52.1
60.4
42.4
64.9
63.3
55.3
58.44
FS + ITRAIN-ALL
63.6
59.1
$7.7
67.3
52.1
59.6
48.3
64.5
64.4
56.7
59.33
ZS
58.4
44.7
47.6
62.2
44.4
50.8
46.4
60.4
49.5
43.1
50.75
-S-SINGLE
59.2
58.7
53.2
63.9
45.9
54.6
49.1
61.2
53.1
51.2
55.01
FS-ALL
60.8
58.1
52.3
63.3
47.3
56.5
53.3
61.3
54.9
51.3
55.91
STRAIN-SINGLA
64.5
58.7
54.9
69.2
53.2
61.5
43.4
65.1
59.5
56.1
58.61
TTRAIN-ALL
65.5
61.9
56.7
70.5
55,5
62.2
42.6
68.5
66.0
58.7
60.81
FS + ITRAIN-ALL
65.1
62.1
58.5
70.0
54.8
61.8
51.2
68.8
67.6
60.3
62.02
(a) Americas accuracy
Method
ACE
BAN
BBC
BIN
BUG
JAY
MAD
MIN
SUN
avg
ZS
74.9
78.0
72.3
77.6
57.6
82.9
68.5
79.9
80.5
74.69
FS-SINGLE
79.1
80.5
77.2
86.0
72.0
85.0
77.5
84.8
83.7
80.64
FS-ALL
79.5
80.0
75.3
86,2
70.4
86.2
76.8
84,4
82.4
80.13
ITRAIN-SINGLE
74.0
77.7
73.8
82.0
66.6
83.6
70.8
78.0
79.7
76.24
TTRAIN-ALL
79.6
82.3
81.0
85.0
68.1
86.2
78.8
83.3
85.7
81.11
FS + ITRAIN-ALL
82.3
82.9
82.8
84.9
72.2
85.3
80.5
85.7
85.0
82.40
ZS
80.0
81.3
65.8
82.0
63.8
84.3
86.6
84.4
77.97
FS-SINGLE
82.2
84.1
80.6
88.3
77.7
88.0
78.6
89.2
85.1
83.76
SFT
FS-ALL
83.7
87.2
79.6
87.9
75.9
87.3
77.2
86.7
84.0
83.30
ITRAIN-SINGLE
SINGLE
82.4
82.0
83.5
85.4
68.1
85.6
80.9
87.0
83.0
81.93
ITRAIN-ALL
83.4
82.4
82.7
84.6
76.8
86.4
81.6
87.4
85.2
83.39
FS TTRAIN-ALL
85.0
85.8
82.7
87.4
78.7
87.4
81.0
88.8
86.5
84.81
(b) NusaX: F1
Table 2: Results of ZS, FS and TTRAIN methods AmericasNLI and NusaX with adapters and SFTs. For the FS methods, K 100, and for the TTRAIN methods, the MT model NLLB with 3.3B parameters. The last column is the average score over all languages. Bold: the best approach within Adapter/SFT. Underline: overall best score. "; NLLB MT model adaptation required to support this language.
ADAPTER
those used during cross-lingual transfer: the source language in the parallel corpora is Spanish, so the English-to-X direction required during translation of the MultiNLI dataset is completely unseen. The success of the TTRAIN configurations on this task is thus testament to the strength and flexibility of multilingual NMT.
Method
NLI
SA
NI.I
SA
ZS
47.77
74.69
50.75
77.97
LA
40.57
65.38
38.21
70.43
FS
51.08
80.64
55.01
83.76
FS LA
44.82
74.03
49.51
78.32
FS UPSAMPLE
48.27
76.78
52.60
83.18
TTRAIN-ALL
58.44
81.11
60.81
83.39
TTRAIN-ALE -LL-- ALL-BI LA
55.50
77.83
57.88
79.39
Number of Shots. We observe a rather large im- pact on performance from increasing the amount of few-shot data. While even 20 shots are enough to bring about 3-5 point average gain the NusaX task, we do not see plateau in performance either task even with the increase from 100 to 500 shots. Upsampling the few shots seems beneficial, yielding 0.5-4 point gain in performance when K = 100. A finer-grained and wider exploration of this finding is warranted in future work.
Table 3: Ablation experiments: LA denotes the cor- responding method without language adaptation (i.e. only task module is present); UPSAMPL indicates no upsampling of the gold-standard target shots is per- formed. FS refers to -SINGLE variant. The scores averages over all target languages.
languages have parallel corpora containing 5,000- 17,000 sentences, while the NusaX parallel corpora have fewer than 1,000 sentences. It is interesting to note that for AmericasNLI, the language pairs used in MT model adaptation are different from
Language Resourcefulness. As suggested by prior work (Pfeiffer et al., 2020; Ansell et al., 2021) and by Table 3, language adaptation has very large impact on all configurations. In the case of the

---

NL.I
SA
SFT ADAPTER
Model
AYM
GN
QUY
ACE
BAN
BIN
BUG
JAV
MIN
SUN
avg
NLLB 600M
62.9
65.1
63.2
63.73
76.6
79.0
83.7
66.0
80.9
81.5
81.1
78.40
NLLB 1.3B
62.3
67.9
63.2
64.47
73.4
80.2
85.3
64.8
83.4
80.6
83.0
78.67
NLLB 3.3B
62.4
67.6
65.2
65.07
79.6
82.4
84.6
66.4
84.8
81.9
84.6
80.61
NI.LB 600M
67.3
69.7
68.7
68.57
83.8
81.2
84.6
72.8
85.7
84.2
82.1
82.06
NLLB 1.3B
66.4
72.0
69.2
69.20
81.3
81.9
84.2
68.5
84.5
84,2
83.1
81.10
NLLB 3.3B
65.7
70.9
70.5
69.03
83.2
83.7
85.1
76.8
88.1
87.6
83.7
84.03
Table 4: Results of TTRAIN-ALL method on the target languages supported by the NLLB model(s) on AmericasNLI (aceuracy) and NusaX (F1 score) with adapters and SFTs. The models are labelled by the number of parameters.
Average with different number of shots
Monolingual Corpora Size vs Performance
90
0.8
SA-Adapter
Scones
SA-SFT
Relaiiv
0.7
TTRAIN-ALL
NLI-Adapter NLE-FFT
NLLB-seen MMT-scch
NLLD seen MMT-mnseen
Ni.B--unsen MMT-unaeen
50
40
10'
10'
10"
0
20 Number of shots K
100
500
Size monolingual corpora MB
Figure 3: The scores of zs LA and FS TTRAIN- ALL methods with SFTs against the monolingual cor- pus sizes. All target languages (from NLI and SA) are shown and they grouped based on the coverage by the NLLB and MMT models. Relative scores are dis- played fraction of the source language performance).
Figure 1: Performance when the number of gold- standard target shots varies, taking values 0, 20, 100, and 500. We show the average across all target lan- guages; the shaded area is the standard deviation (full results available Appendix C).
Monolingual Corpora Size Language Adaptation Effect
vary between 6-13 points. Figure 2 further illus- trates this effect all three methods with adapters, showing the gains from language adaptation on NLI languages against the size of their monolin- gual corpora: languages with larger corpora gen- erally exhibit larger gains. The effect visibly less pronounced for the TTRAIN-ALL where a rela- tively large amount of the target data (albeit trans- lated) lessens the significance of language modules and monolingual corpora size. Finally, Figure 3 illustrates that the performance of our strongest configuration (FS TTRAIN-ALL) with SFTs is the highest for languages that are seen by either the NLLB or MMT model - these are (inciden- tally) the languages with the largest corpora size too. Conversely, this pattern is absent with the zs LA configuration.
15
FS
TTRAIN-ALL
Accuracy Difftrence
10
of monoliaguull corpora
Figure 2: Gains from language adaptation against the size of monolingual corpora for NLI target languages. We show the difference in the accuracy of zs (i.e. zs I.A), and TTRAIN-AL. with adapters. Larger monolingual corpus size leads to larger gains when the language adaptation is present; the effect more pro- nounced with zs and ES than with ITRAIN-ALL.
Language families for our target languages are given in Appendix A. While they could have an impact on the performance, is difficult to disen- tangle the effect of the language family from the
TTRAIN-AL ALL configuration, language adaptation incurs gains of 3-4 points, while for zs and FS they

---

amount of data available without having a much larger of evaluation languages.
specific models, which becomes particularly attrac- tive when dealing with a large number of target languages. They also introduce stochastic gradi- ent surgery to circumvent the issue of conflicting gradients among languages.
PEFT Method. The relative performances of the various configurations are very similar regardless of PEFT method, and in accordance with previous work (Ansell et al., 2022; Alabi et al., 2022), SFTs consistently outperform adapters by 2-3 points. However, we estimate that training adapters is gen- erally around times faster than training SFTs.
Jundi and Lapesa (2022) compare few-shot and translation- approaches, trying to gain an in- sight into which approach is better and under which circumstances. We consider these approaches in combination rather than in competition and find that the use of few-shot data can enhance perfor- mance even when a machine translation of the full source language dataset is available. However, their work complements ours by proposing a way to iden- tify the examples which may be most profitable for humans to translate into target language "shots."
5 Related Work
Carameter-Efficient Fine-Tuning Methods. Parameter-efficient fine-tuning methods have emerged from necessity to reduce compute and memory requirements of fine-tuning when dealing with large models. These methods can generally be grouped into those that modify the subset of parameters of the pretrained LLM (Ben Zaken et al., 2022; Guo et al., 2021: Sung et al., 2021) and those that introduce a completely fresh set of parameters to be updated (Li and Liang, 2021; Lester al., 2021; Houlsby et al., 2019; Hu et al., 2022) allowing for different interactions with the pretrained model. They have been adopted for cross-lingual transfer as they are preferable when dozens of different fine-tunings for different languages and tasks need to be learned, stored and combined (Pfciffer et al., 2020; Ansell et al., 2022: Parović et al., 2022). For comprehensive overview of parameter-efficient fine-tuning, we refer the reader to Pfeiffer et al. (2023).
Winata et al. (2022) study few-shot cross-lingual transfer on languages unseen by MMTs using the NusaX dataset. They analyse the effectiveness of several few-shot strategies focusing on selecting languages for transfer and different learning dy- namies exhibited by different types of MMTs.
Machine Translation for Cross-Lingual Trans- fer. The translate-train and translate-test ap- proaches are common baselines for cross-lingual transfer (Conneau et al., 2020a; Hu et al., 2020). A number of enhancements have been proposed. Artetxe et al. (2020) showed that translate-test per- formance could be improved by training back- translated rather than the original source language data better model translation artefacts encoun- tered at inference time. Ponti et al. (2021) note that translation- based approaches suffer from an error accumulation over the phases of the pipeline. They re-interpret this pipeline as a single model with an intermediate "latent translation" between the target text and its classification label, permitting the translation model to be fine-tuned according to a feedback signal from the task loss. Oh et al. (2022) show that the translate-train and translate- test approaches can be combined synergistically. Artetxe et al. (2023) show that translate-tess more favourable relative to translate-train than previously thought when better translation and monolingual models are used, and when measures are taken to correct the MT-induced mismatch between the data encountered train and inference time. While we only consider applying translation-based cross- lingual transfer to classification tasks, prior work has considered its application to sequence labelling tasks as well (Jain et al., 2019; Fei et al., 2020;
Few-Shot Cross-Lingual Transfer. Lauscher et al. (2020) and later Zhao et al. (2021) demonstrate the effectiveness of few-shot over the zero-shot cross-lingual transfer, showing that continued train- ing of source-trained model on small number of labelled examples in the target language sig- nificantly increases performance (target-adapting) Schmidt al. (2022) trade-off efficiency for per- formance by replacing the sequential fine-tuning procedure of Lauscher et al. (2020) with joint train- ing on source and target language, showing it also improves training stability and robustness. They additionally show that first fine-tuning on multiple target languages provides extra performance gains. We adopt their joint training procedure, combin- ing it further with parameter-efffiient fine-tuning methods and language adaptation. Xu and Murray (2022) also exploit a joint source-and-targe target train- ing procedure, further extending it to all target lan- guages simultaneously instead of having language-

---

Carcia-Ferrert al., 2022: Fercia-Ferrero al., 2022). For simplicity, we employ only continued pretraining on the standard MT task when adapting NLLB to unseen languages. Ko et al. (2021) ) en- hance NMT model adaptation with additional tasks: denoising autoencoding, which exploits monolin- gual target language data; back-translation; and adversarial training which encourages the encoder output language-agnosicc features.
families could be done as part of future work. During the adaptation of the MT model to un- supported languages, we only consider continued training with parallel data. While further perfor- mance increases could be achieved with the usage of monolingual data sources and backtranslation following Ko et al. (2021), we opt for simplic- ity, exploiting the monolingual data only with the language modules. Furthermore, this aligns with our goal which is not to maximize the raw perfor- mance but rather to study the effects of different data sources and their mutual interactions.
6 Conclusions and Future Work
We have investigated how combine several cross- lingual transfer techniques which are applicable across several dimensions of resource scarcity into a single framework. We find that parameter- efficient language adaptation, few-shot learning and translate-trail are complementary when em- ployed in multi-source training setup with few- shot upsampling. However, training setup sup- ports the use of any subset of these techniques depending on the availability of the necessary data and models. We remark on the significance of the finding that gold-stane standar few-shot target data can improve performance even when the entirety of the training data is translated into the target lan- guage by a high-quality NMT model. We also observe that languages not natively supported by NMT model can benefit from translate-traii through a simple adaptation procedure even with a small amount of parallel data.
Due to large number of experiments across many methods and ablations, we report all our re- sults based single run. However, the large number of target languages we average over and the replication of the core findings across the two PEFT methods adds confidence that they are cor- rect.
Finally, training language modules is typically computationally expensive. However, the modular design of cross-lingual transfer methods that we consider, enables us to train language modules only once and reuse them across all of our experiments.
Acknowledgments
We would like to thank Mikel Artetxe for dis- cussing this work with us and providing helpful advice. We also thank the anonymous reviewers for their comments.
Alan wishes to thank David and Claudia Hard- ing for their generous support via the Harding Dis- tinguished Postgraduate Scholarship Programme and Churchill College, Cambridge for travel assis- tance. Marinela Parović supported by Trinity College External Research Studentship. Ivan Vulić acknowledges the support of personal Royal So- ciety University Research Fellowship 'Inclusive and Sustainable Language Technology for Truly Multilingual World' (no 221137; 2022))
Limitations
Our experiments are based on two parameter- efficient fine-tuning methods: adapters and SFTs. This choice facilitates comparisons with the prior work in the area of cross-lingus transfer since these two methods have been studied extensively. How- ever, we note that other modular and parameter- efficient fine-tuning methods are available and could be used in combination with our framework (Pfeiffer et al., 2023).
Our evaluation relies solely on classification tasks, as the data labels in these tasks are preserved upon translation into another language. This is not the case with sequence- labelliin tasks, where an additional challenge lies in projecting the labels after obtaining the translation. Restricting our ex- periments the classification tasks enables us to have a more controlled environment for studying only the effects of different data sources which is the main focus of this work. Studying other task
References
Željko Agić and Ivan Vulić. 2019. JW300: A wide- coverage parallel corpus for low-resource languages. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 3204- 3210. Florence, Italy. Association for Computational Linguistics.
Jesujoba O. Alabi, David Ifeoluwa Adelami, Marius
Mosbach, and Dietrich Klakow. 2022. Adapting pre- trained language models to African languages via

---

multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336-4349, Gyeongju, Republic of Korea, International Committce on Computational Linguistics.
Cahya Wirawan, Frederikus Hudi, Ivan Halim Parmo- nangan, Ika Alfina, Muhammad Satrio Wicaksono, 11- ham Firdausi Putra, Samsul Rahmadani, Yulianti Oe- nang, Ali Akbar Septiandri, James Jaya, Kaustubh D. Dhole, Arie Ardiyanti Suryani, Rifki Atina Putri, Dan Su, Keith Stevens, Made Nindyatama Nitassaa, Muhammad Farid Adilazuarda, Ryan Ignatius, Ryan- dito Diandaru, Tiezheng Yu, Vito Ghifari, Wenliang Dai, Yan Xu. Dyah Damapuspita, Cuk Tho. Ichwanul Muslim Karo Karo, Tirana Noor Fatyanosa, Ziwei Ji. Pascale Fung. Graham Neubig. Timothy Baldwin, Sehastian Ruder, Herry Sujaini, Sakriani Sakti, and Ayu Purwarianti. 2022. NusaCrowd: Open Source Initiative for Indonesian NLP Resources.
Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross- lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1796. Dublin, Ireland. Association for Computational Lin- guistics.
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Se- bastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021. MAD-G: Multilingual adapter gen- eration for efficient cross-lingual transfer. In Find- ings of the Association for Computational Linguis- rics: EMNLP 2021, pages 4762-4781, 1, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Luis Chiruzzo, Pedro Amarilla, Adolfo Ríos, and Gustavo Giménez Lugo. 2020. Development of Guarani Spanish parallel corpus. In Proceedings of the Twelfth Language Resources and Evaluation Conference, paaeesseeepeeesseeeeeee 2629-2633, Marseille, France. Eu- ropean Language Resources Association.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020a. Unsupervised cross- Moossllingual representation learning at scale. In Pro- ceedings of the 58th Annuul Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Mikel Artetxe, Vedanuj Goswami, Shruti Bhosale, An- gela Fan, and Luke Zettlemoyer. 2023. Revisiting machine translation for cross-lingual classification. ArXiv, abs/2305.14240
Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020. Translation artifacts in cross-lingual transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7674-7684, Online. Association for Computa- tional Linguisties.
Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettle- moyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language mod- els. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 6022 6034, Online. Association for Computational Linguisties.
Elad Ben 7aken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-effiiien fine-tuning for tramsformer-basee
masked language-models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Associa- tion for Computational Linguistics.
Rubén Cushimariano Romano and Richer C. Se- bastián Q 2008 Naantsipeta asháninkaki bi- rakochaki. diccionario asháninka-casttllaan ver- sión preliminar. httt://www. lengamer. pualicacioons/diciioarioos.
Damian Blasi, Antonios Anastasopoulos, and Gra- ham Neubig, 2022. Systematic inequalities in lan- guage technology performance across the world's languages. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486-5505, Dublin, Ireland. Association for Computational Linguistics.
Jacob Devlin, Ming-Wai Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
David Brambila. 1976.
Diccionario Raramuri-
Castellano: Tarahumar.
Gina Bustamante, Arturo Oncevay, and Roberto Zariquiey. 2020. No data craw1? monolingual corpus creation from PDF files truly low-resource languages in Peru. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 2914-2923, Marseille, France. European Language Resources Association.
Abteen Ebrahimi, Manuel Mager, Arturo Oncevay. Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo. Elisabeth Mager. Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022. AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. In Proceedingy of the
Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Indra Winata, Bryan Wilie, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vin- centio, Fajri Koto, Jennifer Santoso, David Moeljadi,

---

60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 6279-6299, Dublin, Ireland. Association for Compu- tational Linguistics.
Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-effiiennt transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Hao Fei, Meishan Zhang, and Donghong Ji. 2020. Cross-lingual semantic role labeling with high- quality translated training corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7014-7026, On- line. Association for Computational Linguistics.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Isaac Feldman and Rolando Coto-Solano. 2020. Neu- ral machine translation models with back-translation for the extremely low- wwwresource indigenous language Bribri. Proceedings of the 28th International Con- ference on Computational Linguistics, pages 3965- 3976, Barcelona, Spain (Online). International Com- mittee on Computational Linguisties.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411- 1111-4421. PMLR.
Ana-Paula Galarreta, Andrés Melgar, and Arturo On- cevay. 2017. Corpus creation and initial SMT ex- periments between Spanish and Shipibo-konioe konibo. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP Varna, Bulgaria. INCOMA
Alankar Jain, Bhargavi Paranjape, and Zachary C. Lip- ton. 2019. Entity projection via machine transla- tion for gross-lingual NER. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE) pages 1083-1092, Hong Kong. China. Association for Computational Linguistics.
2017. pages Ltd.
Iker Garefa-Ferrero, Rodrigo Agerri, and German Rigau. 2022. Model and data transfer for cross-lingual se- quence labelling in zero-resource settings. In Find- ings the the Association for Computational Linguisties: EMNLP 2022, pages 6400-6411, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Pratik Joshi. Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguistics.
Iker Garcfa-Ferreno Rodrigo Agerri, and German Rigau. 2022. T-projection: High quality annotation projec- tion for sequence labeling tasks.
Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 4884-4896 Online. Association for Computational Linguistics.
Iman Jundi and Gabriella Lapesa. 2022. How to trans- late your samples and choose your shots? analyz- ing translate-train few-shot cross-lingual transfer. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 129-150, Seattle, United States. Association for Computational Lin- guistics.
Katharina Kann, Kyunghyun Cho, and Samuel R. Bow- man. 2019, Towards realistic practices in low- resource natural language processing: The develop- ment set. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP)) pages 3342-3349, Hong Kong, China. Association for Com- putational Linguistics.
Ximena Gutierrez-Vasques, Gerardo Sierra, and Isaac Hermandez. Pompa. 2016. Axolotl: web accessible parallel corpus for Spanish-Nahuatl. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pages 4210-4214, Portorož, Slovenia. European Lan- guage Resources Association (ELRA).
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023, DeBERTav3: Improving deBERTa using ELECTRA- style pre-training with gradient- -iseeentdd embed- ding sharing. In The Eleventh International Confer- ence on Learning Representations.
Wei-Jen Ko, Ahmed El-Kishky, Adithya Renduchin- tala, Vishrav Chaudhary, Naman Goyal, Francisco Guzmán, Pascale Fung. Philipp Koehn, and Mona Diah. 2021. Adapting high-resource NMT models to translate low-resource related languages without par- allel data. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea

---

and the 11th International Joint Conference on Natu- val Language Processing (Volume 1: Long Papers), pages 802-812, Online. Association for Computa- tional Linguisties.
Gao, Vedanuj Goswami. Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers. Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human- centered machine translation.
Fajri Koto and Ikhwan Koto. 2020. Towards computa- tional linguisties in Minangkabau language: Studies on sentiment analysis and machine translation. In Proceedings of the 34th Pacific Asia Conference on Language, Information and Computation, pages 138- 148, Hanoi, Vietnam. Association for Computational Linguisties.
Jachoon Oh, Jongwoo Ko, and Se- Young Yun. 2022. Synergy with translation artifacts for training and inference in multilingual tasks. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 6747-6754, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 4483-4499, On- line. Association for Computational Linguistics.
John E Ortega, Richard Alexander Maatto,Mamani, and
Jaime Rafael Montoya Samame. 2020. Overcoming resistance: The normalization of an Amazonian tribal language. In Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages, pages 1-13, Suzhou, China, Association for Compu- tational Linguistics.
Brian Lester, Rami AI-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Marinela Parović. Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2022. BAD-X: Bilingual adapters im- prove zero-shot cross-lingual transfer. In Proceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1791-1799 Seattle, United States. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Jonas Pfeiffer, Sebastian Ruder, Ivan Vulić, and Edoardo Maria Ponti. 2023. Modular deep learning.
Jonas Pfeiffer, Ivan Vulić, Tryna Gurevych, and Se- bastian Ruder. 2020. MAD-X: An Adapter-Baa -assd Framework for Multi-Tash Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7654 7673, Online, Association for Computa- tional Linguistics.
Manuel Mager, Diónico Carrillo, and Ivan Meza. 2018. Probabilistic finite-state morphological segmenter for wixarika (huichol) language. Journal of Intelligent & Fuzzy Systems, 34(5):3081-300
Jonas Pfeiffer, Ivan Vulié, Tryna Gurevych, and Sebas- tian Ruder. 2021. UNKs everywhere: Adapting mul- tilingual language models to new scripts. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10186-10203, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguisties.
Elena Mihas. 2011. Añaani katonkosatzi parenini, El id- ioma del alto Perené. Milwaukee, WI: Clarks Graph- ics.
Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Un- derstanding the lingualingual ability of multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, pages 22 2214-2 Online. Association for Computational Linguistics.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 4996-5001, Flo- rence, Italy. Association for Computational Linguis- tics.
NL.BB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef- fernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood. Bapi Akula, Loic Bar- rault, Gabriel Eegiia-Gozzlezz Prangthip Hansanti. John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia
Edoardo Maria Ponti, Julia Kreutzer, Ivan Vulić, and Siva Reddy. 2021. Modelling latent translations for cross-lingue transfer.
Ayu Purwarianti and Ida Ayu Putu Ari Crisdayanti. 2019. Improving bi-lstm performance for indonesian senti- ment analysis using paragraph vector. In 2019 Inter- national Conference of Advanced Informatics: Con- cepts. Theory and Applications (ICAICTA), pages 1-5.

---

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. 2017. Learning multiple visual domains with residual adapters. In Advances in Neural In- formation Processing Systems, volume 30. Curran Associates, Inc.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-cowe coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guisties: Human Language Technologies, Volume (Long Papers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Sakriani Sakti and Satoshi Nakamura. 2013. Towards language preservation: Design and collection of graphemically balanced and parallel speech corpora of Indonesian ethnic languages. In 2013 Interna- tional Conference Oriental COCOSDA held jointly with 2013 Conference on Asian Spoken Language Research and Evaluation (O-COCOSDAVCASLEE)) pages 1-5.
Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preotiuc-Pietro, 2022. Cross- lingual few-shot learning unseen languages. In Proceedings of the 2nd Conference of the Sici-Pee Chapter of the Association for Computational Lin- guistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 777-791, Online only. Association for Computational Linguistics.
Fabian David Schmidt, Ivan Vulié, and Goran Glavaš. 2022. Don't stop fine-tuning: On training regimes for few-shot cross-lingual transfer with multilingual language models. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 10725-10743 10742, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Genta Indra Winata, Alham Fikri Aji, Samuel Cahyawi- jaya, Rahmad Mahendra, Fajri Koto, Ade Romad- hony, Kemal Kurniawan, David Moeljadi, Radi- tyo Eko Prasojo, Pascale Fung, Timothy Baldwin, Jey Han Lau, Rico Sennrich, and Sebastian Ruder. 2023. NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 815-834, Dubrovnik, Croatia. Association for Com- putational Linguistics.
Herry Sujaini. 2020. Improving the role of language model in statistical machine translation (Indonesian- Javanese). International Journal of Electrical and Computer Engineering. 10:2102-2109.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. 2021. Training neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems, volume 34, pages 24193-24205. Curran Associates, Inc.
Cahya Wirawan. 2022. https://hugginngaacc co/datasets/ patneesiaannnpllibriii -inoooiiiinnii.....
Jörg Tiedemann. 2012. Parallel data, tools and inter- faces in OPUS. In Proceedings of the Eighth In- lernational Conference on Language Resources and Evaluation (IRRE'112) 12), pages 22 2214-2218, Istanbul, Turkey, European Language Resources Association (ELRA).
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas:
The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing ( (EMNLP-IJCNIP)) pages 833-844, Hong Kong. China. Association for Computational Linguis- tics.
Bayu Distiawan Trisedya and Dyah Inastra. 2014. Cre- ating Indanesian-Javanese Parallel Corpora Using Wikipedia Articles. In 2014 International Confer- ence on Advanced Computer Science and Informa- tion System, pages 239-245.
Haoran Xu and Kenton Murray. 2022. Por qué não utiliser alla språk? mixed training with gradient opti- mization in few-shot cross-lingual transfer. In Find- ings of the Association for Computational Linguis- ties: NAACL 2022, pages 2043- 32599 Seattle, United States. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, F. ukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020. IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding. In Proceedings of the 1st Conference of the Asia-Pacifie Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Lan- guage Processing, pages 343-857, Suzhou, China. Association for Computational Linguistics.
Mengjie Zhao, Yi Zhu, Ehsan Shareghi, Ivan Vulić, Roi Reichart, Anna Korhonen, and Hinrich Schütze. 2021. A closer look at few-shot crosslingual transfer: The choice of shots matters. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1:
Long Papers), pages 5751-5767, Online, Association for Computational Linguistics.

---

A Languages
The complete overview of languages, their codes and families, together with the monolingual data sizes and resources is provided in Table 5. The sizes and resources for the parallel corpora used in the MT model adaptation are given in Table 6,
B Ablation Experiments
We present per language results of our ablation experiments in Table 7. The summarised results are given in Table 3.
C Full Results with Different Number of Shots K
We give full results with the different number of gold-standard target shots K. where K E (0, 20, 100, 500). The setting K = 0 resembles the ZS approach, while the rest of the values fall within FS. The results are shown in Table 8, with their summary given in Figure

---

Task
Langunge
Code
Family
Corpus dize (MB)
Corpus sourceis!
English
Endo-uane Gemanic
Source
Indonesian
10
Austronesian, Milayo-Sunhhwas
148,300
Wikipedia
Aymumi
sym
Aymaran
2.3
(2012): Wikipollis
Ashitinka
Arsoskin
Oniga (2030); Cishimurian Romono Q. (2008);
Milus (2010) Bostamane
(2020)
Bribi
bod
Chileban, Talamenca
0.3
Peldiaa and Coto Solaze (2020)
NLI
Giarni
Vihaal
Uho-Attsom, Artezm
Topsan Tupi Giarani
8,1
6.9
Gutieneze Sunienez-Vospes Ulihhuu Online Corpus
Chirazoo et al. (200)) Wikrpedia
(2016); Wikipollis
Ohomm
Oto Oon-Mnnueun, Otemian
0,4
Quachuc
qity
Quachan
Agié Volkč
Wikipedia
Ulo-Attzaan,
Brambila
Kaipile-Kmibu
<tp
Pamom
Gabarnaa et al. (2017);; Bosttmmntt et (2020)
Wisarila
bac
UleAAtrea,, Conchel
0.5
Mager et al. (2018)
Acchiess
Austrenesian, Milayo-Sumhawar
2023); (Cahyawjjys al., 2022,;
al, 2022), Wikipelia
INDspeech, NEWS. BitnncS (Saki and Nakamura, 2013), KoLI-NLLB
Balinese
ban
Austronesian, Sulayo-Sunhhoa
(Cabyuwijayy wijawwajjaaaa..... et al. 2022) LibiYex Indonsdeeesi (Wirawas, 2022); NLLB- Seed (NLLB Team al., 2022); Wikipedia
SA
Bangimoa
Ausinmesian, Malago-Sumbsoe
28
KoPL-NLLB (Cabyewsjana
NI R-Sunl LNI.B Torm al. 2002y, Wikipolla
2022), Korpss Nuaantan (Sujaini, 2020))
Buginese
bug
Austronesian, South Salawesi
Nustntara
20200; Libr Vox Indotesia (Wirawan, 2002):
Wikipedia
Javanese
Austronesian, Javanese
Wikipedia
Madumeee
mod
Austrmesian, Malayo-Sumbwon
088
Korpos Nussniara (Sujanni 2020); Wikipodla
Indo Wit. Partllel Corpora (Tiisedya and leastra, 2014;; KoPl NILBB
Minangkabu
min
Austroessian Malaye-Smmbrsee
93
(Cahyaw jays 2002), Korpus Nasantarr (Sujanni 20200; Libr Var- Indonesia (W (Wirawan, 2022); MitangNLP MT (Koto and Koto, 2020); Wiligolia
Mgaju Sundunsse
Austronesian,
Molyys-Sunhasan
Wikipedia
Rabak
Austonesiaan Narthwest Sumatra-
Barrier Ndlnds
Korpus Nuemniama (Sujaini 20200
Table 5: Details of the languages and monolingual data used for training and evaluation of SFTs and adapters. The corpora of Bustamante et al. (2020) are available at https: / com/ httpt//.../iiiauuuu///iiiiiiiuga -dataaaaaalaaaeeererr other NIJ corpora mentioned are available ht http://a.w......eeeec////////aaa SA corpora (Cahyawijaya et al., 2022) are available through https: http:://iidolllggttttti..o//ssse
all the
Target Language
Source Language
Corpus size (Hsent)
Corpus source(s)
Bribri
Spanish
8,502
Reldman and Coto-Solano (2020)
Nähuad
Spanish
16,733
Vasques al. (2016)
Otomi
Spanish
5,488
Hüähhü Online Corpus
Rardimuri
Spanish
15,714
Brmhila (1976)
Shipibu-Komibu Komibo
Spunish
15.586
Galuretaa et al. (2017); Bustamanie al. (2020)
Wixarika
Spanish
9,960
Mager (2018)
Madurese
Indoncsian
Ngaju
Indonesian
629
Winata et al. (2023)
Toba Batak
Indonesium
629
Table 6: Details of the parallel corpora used for NLLB MT model adaptation. #sent number of sentences in train dev set.

---

Method
AYM
BZD
CNI
GN
НCH
NAH
OTO
QUY
SHP
TAR
avg
ZS
53.0
42.8
44.8
59.6
40.3
50.8
41.2
55.2
50.0
40.0
47.77
7S LA
40.1
36.9
42.0
42.9
39.6
42.7
41.8
40.8
43.2
35.7
40.57
ADAPTER
FS
55.2
47.1
47.7
58.3
40.7
55.8
45.9
59.7
52.9
47.5
51.08
FS
LA
43.3
45.7
42.3
46.3
43.7
46.4
47.2
42.0
44.82
FS UPSAMPLE
50.3
45.5
46.8
58.3
40.3
48.9
46.4
56.3
46.0
43.9
48.27
TTRAIN-ALL
63.3
60.5
56.1
66.1
52.1
60.4
42.4
64.9
63.3
55.3
58.44
ITRAIN-ALL LA
60.1
58.3
50.3
60.7
50.7
55.6
44.7
60.8
57.1
56.7
55.50
ZS ZS-LA
58.4
44.4
50.8
46.4
60.4
43.1
50.75
35.7
38.7
37.3
39.1
38.3
41.3
37.7
36.3
40.9
36.8
38.21
FS
59.2
58.7
53.2
63.9
45.9
54.6
49.1
61.2
53.1
51.2
55.01
SFT
ES LA FS UPSAMPLE
53.3
54.9
47.6
48.9
42.9
53.8
45.5
53.3
46.5
48.4
49.51
59.1
51.2
50.3
63.9
42.9
52.7
48.1
62.8
50.1
44.9
$2.60
ITRAIN-ALL
65.5
61.9
56.7
70.5
62.2
42.6
68.5
66.0
58.7
60.81
ITRAIN-ALL-
63.6
48.0
65.6
54.7
57.3
46.9
63.6
62.7
55.7
57.88
Americas accuracy
Method
ACE
BAN
BBC
BJN
BUG
JAV
MAD
MIN
NIJ
SUN
avg
ZS 7.S LAA
74.9
78.0
72.3
77.6
57.6
82.9
68.5
79.9
80.5
74.69
68.0
70.8
37.6
78.3
31.9
80.9
63.5
77.9
66.9
78.0
65.38
ADAPTER
FS
79.1
80.5
77.2
86.0
72.0
85.0
77.5
84.8
83.7
80.64
FS LA
74.7
73.5
64.3
79.6
61.7
82.3
74.0
81.5
70.9
77.8
74.03
FS
UPSAMPLE
76.5
78.4
71.9
80.6
64.5
74.6
80.1
81.9
76.78
TTRAIN-ALL
79.6
82.3
81.0
85.0
68.1
78.8
83.3
85.7
81.11
TTRAIN-ALL LA
78.5
76.5
73.1
83.2
67.6
82.7
77.8
82.0
77.4
79.5
77.83
ZS ZS
80.0
81.3
65.8
82.0
63.8
84.3
73.5
86.6
84.4
77.97
72.8
72.3
79.3
47.7
82.2
71.9
81.4
70.3
79.1
70.43
FS
82.2
84.1
80.6
88.3
77.7
88.0
78.6
89.2
85.1
83.76
SFT
ES LA
78.9
76.2
68.6
81.1
71.0
86.3
78.2
84.0
76.0
82.9
78.32
FS UPSAMPLE
82.7
84.5
79.5
87.9
74.7
86.5
78.0
87.8
87.0
83.18
TTRAIN- ALL
83.4
82.4
82.7
84.6
76.8
86.4
81.6
87.4
85.2
83.39
TTRAIN-ALL. LL
78.9
81.0
75.
83.0
67.1
86.2
79.7
82.8
77.6
82.5
79.39
(b) NusaX:
Table 7: Per-language results of ablation experiments with ZS, FS, and TTRAIN-ALL methods on NLI and SA
with adapters and SFTs: LA denotes the absence of language adaptation (i.e. only the task module is present), while UPSAMPLE indicates there is no upsampling of the gold atandaddd target shots.
Method
AYM
BZD
CNI
GN
HCH
NAH
OTO
QUY
SHP
TAR
ADAPTER
7S (K 0)
53.0
42.8
44.8
59.6
40.3
50.8
41.2
55.2
50.0
40.0
47.77
FS (K 20)
50.4
43.3
46.0
56.9
39.9
50.4
43.2
58.3
50.5
44,0
48.29
FS (K 100)
55.2
47.1
47.7
58.3
40.7
55.8
45.9
59.7
52.9
47.5
51.08
FS (K = 500)
57.2
54.0
53.7
61.6
45.1
55.1
48.0
59.1
54.7
50.5
53.90
ZS K 0)
58.4
44.7
47.6
62.2
44.4
50.8
46.4
60.4
49.5
43.1
50.75
SFT
FS (K- 20)
57.9
48.7
46.4
59.9
44.3
51.5
48.7
61.2
51.1
43.7
$1.34
ES (K- 100)
59.2
58.7
53.2
63.9
45.9
34.6
49.1
61.2
53.1
$1.2
$5.01
FS (K 500)
61.1
59.5
55.7
65.6
48.7
58.9
52.5
64.4
61.5
54.1
58.20
(a) AmericasNLE accuracy
Method
ACE
BUG
JAV
avg
ADAPTER
ZS (K = 0)
74.9
78.0
72.3
77.6
57.6
82.9
68.5
79.9
80.5
74.69
FS (K 20)
74.6
78.9
73.6
81.0
66.1
84.3
81.3
82.4
77.83
FS (K-
79.1
80.5
77.2
86.0
72.0
85.0
77.5
84.8
83.7
80.64
FS (K 500)
81.1
84.2
78.5
88.0
74.8
87.9
79.4
88.0
88.1
83.33
ZS (K 0)
80.0
81.3
65.8
82.0
63.8
84.3
73.5
86.6
84.4
77.97
SFT
FS (K
81.9
83.1
79.0
87.9
73.4
86.3
87.0
87.2
82.42
FS (K 100)
82.2
84.1
80.6
88.3
77.7
88.0
78.6
89.2
85.1
83.76
FS (K 500)
86.5
89.3
83.1
87.9
79.7
91.3
80.6
90.5
87.5
86.27
(b) NusaX: F1
Table Por-language results on NLI and SA with the different number gold-standard target shots K. We consider ZS (K 0) and FS with K {20, 100, 5003 with adapters and SFTs.


Reviews:
[{"review_id": "YE01Md8qcr", "reviewer": "Reviewer_TbEt", "summary": "", "questions": "It would be beneficial for the authors to provide analyses on:\n- The effect of varying corpus sizes;\n- The impact related to different language families;\n- The influence of distinct language types (e.g., SVO, SOV...) within the context of the transfer-learning approach.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very encouraging review, we really appreciate it! \n\nIn answer to your questions on additional analyses:\n* We provide a study on the effect of varying corpus sizes in the paragraph Language Resourcefulness (L450-472) and Figures 2 and 3, showing that: (i) languages with larger corpora exhibit larger gains from language adaptation; (ii) languages seen by MMT and/or MT models which are also languages with the largest corpora obtain the highest overall performance.\n* We will use the extra space to include a paragraph describing the impact of different language families. \n* Previous work has shown that the word order of source and target languages has an impact on the cross-lingual transfer (K et al. Cross-Lingual Ability of Multilingual BERT: An Emprical Study [ICLR 2020]; Pires et al. How multilingual is Multilingual BERT? [ACL 2019]), so there is a reason to believe different language types could affect the transfer gap in our experiments too. However, we see such analysis as beyond the scope of this work whose aim is to integrate various cross-lingual transfer tools and evaluate their performances in synergy.\n\nRegarding reproducibility, we will release the code which will make our findings easily reproducible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is an essential contribution to the field as it innovatively illustrates the unification of various cross-lingual transfer techniques across different dimensions of resource scarcity, all within a single cohesive framework. What sets this work apart is the discovery that parameter-efficient language adaptation, few-shot learning, and translate-train approaches are not just parallel strategies but are in fact complementary. When these are harmoniously integrated within a multi-source training setup coupled with few-shot upsampling, the synergy leads to a more robust and efficient model. The paper's results bear significant implications for the future of language processing, and the techniques proposed have the potential to reshape the way we approach multilingual challenges in resource-constrained environments.", "reasons_to_reject": "No.", "questions_to_authors": "It would be beneficial for the authors to provide analyses on:\n- The effect of varying corpus sizes;\n- The impact related to different language families;\n- The influence of distinct language types (e.g., SVO, SOV...) within the context of the transfer-learning approach.", "paper_topic_and_main_contributions": "The paucity of data across numerous global languages mandates the transference of knowledge from those languages endowed with abundant resources. Historically, the dimensions and techniques employed to address this issue have been handled in isolation. In contrast, this paper presents a more synthesized perspective, investigating the application of an extensive suite of cross-lingual transfer tools across various scenarios, with particular emphasis on the most exigent, low-resource situations.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zN8lEav1Ht", "reviewer": "Reviewer_jn1b", "summary": "", "questions": "- Why have you chosen to evaluate on those datasets and languages?\n- Where are studies of different languages with respect to different degrees of coverage in the pre-trained model?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and encouraging feedback!\n\nClarifications and responses to the reasons to reject:\n* We focus on parameter-efficient cross-lingual transfer methods as they have been shown to be the most effective ways of transfer to low-resource and unseen languages which is where all our languages land (18/20 are unseen by the MMT) - see the original MAD-X and SFT papers. There is not much bias due to the resourcefulness of target languages during MMT pre-training because most of the languages we consider are unseen by the MMT. However, we perform some analysis of this type in Figure 3, where we can see from the red group of symbols that the languages seen during pre-training are among the best-performing initially, but they still benefit from LA/FS/TTRAIN.\n* We give the outline of our experiments in section 3.3 Configurations and Ablations. However, we will use the extra space to make this clearer, and we appreciate your presentation suggestions which could also help with this.\n* We are a bit perplexed by your suggestion that the study should be more holistic, given that it unifies a number of approaches that have not been considered together before across a range of cross-lingual transfer scenarios, and one of the main goals of this study has been indeed to provide a more holistic approach to cross-lingual transfer that integrates different components researched and applied previously in isolation. We are also unclear on what you mean by \"unbiased\" in this context.\n\nClarifications and responses to the questions:\n* We justify our choice of the datasets in L252-256. We focus on low-resource, classification benchmarks that have sufficiently large train/validation splits for few-shot learning to simulate different resource levels. We wanted the datasets to have languages with a range of resource levels within the low-resource umbrella, i.e. seen or unseen by the MMT/translation model and various corpus sizes.\n* This analysis is provided in the paragraph Language Resourcefulness (L450-472) and Figure 3.\n\nIn response to your presentation improvements:\n* We provide a detailed overview of the related work in Section 5, with the subheadings denoting the class of cross-lingual transfer approach considered - generally, there is no mixing of different approaches within these papers. XTREME and XTREME-R are examples of treating these approaches separately, without utilizing any monolingual corpora for further improvements.\n* We restrict the analysis of varying corpora size across different languages rather than varying it within a single language - while this would be a valid and interesting experiment, we prioritized coverage over more languages to improve the generality of our findings. On the other hand, we do provide analysis on varying the number of gold-standard target shots.\n* We will take into account your suggestions on mentioning Table 3 later in the text, presenting the results with the bar charts, and increasing the font on the plots. Thank you very much for the suggestions - it should be possible to further smooth the presentation details given one extra page of content!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- It is nice to see a study that provides a comprehensive analysis spanning different dimensions of resourcefulness in transfer learning which a lot of studies have focused on separately. \n- There are a lot of generalizable insights and recommendations which can be used in using multiple transfer learning techniques\n- It is useful to understand the complementarity of different techniques of resourcefulness\n", "reasons_to_reject": "- There are a lot of techniques not explored and the study is more focused on parameter-efficient language models only. Analyzing also the biases in the resourcefulness of those language models which are believed to have an equally if not higher impact on the outcome of the transfer is missing. \n- It is a bit hard to get a hand on different experiments performed and get a full picture. It would help to outline those experiments in a clearer manner in the experimental setup (see presentation improvements below).\n- The study sounds a bit incremental. Instead of adopting different techniques why not just have a more holistic unbiased analysis. ", "questions_to_authors": "- Why have you chosen to evaluate on those datasets and languages?\n- Where are studies of different languages with respect to different degrees of coverage in the pre-trained model?", "paper_topic_and_main_contributions": "This paper conducts a comprehensive study combining different cross-lingual techniques and applying them across several dimensions of resource scarcity. This study provides insights into the complementarity of different data used to bridge the gap between resource-rich and resource-scarce languages. More specifically, their analysis investigates the effectiveness of the resourcefulness of various data sources employed at different stages of the transfer: pre-training and fine-tuning either zero-shot or few-shot manner or using data augmentation in the form of machine-translated data. All this provides a valuable toolbox to integrate different transfer learning strategies and adapt them to different scenarios. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Abstract:\n\n- Line 011-013: it would be helpful to list those prior work and compare them with your work by highlighting which specific dimensions have they lacked in their evaluation and maybe summarizing that in some table or figure that you can show in the first page. I see some overalp with large-scale performance evaluation works like XTREME and XTREME-R but you don't discuss those differences in details in the introduction.\n\n- Section 3: I think this could benefit from some diagrams to explain the strategies to evaluate the complementarity of different techniques planned. \n- Line 382: Table 3 should be either talked about right here or shown later on in language resourcefulness paragraph\n- Table 2: doesn't make sense to compare zero-shot, few-shot, translation etc. they all use different levels of resources. it would be more interesting to have correlation analysis where you assess the improvements across different techniques but while fixing the amount of resourcefulness and define some measure to do that.\n- Hard to read Table 2 with two datasets with so many numbers. I think it would be better if bar plots are used or the differences in the performances are highlighted for more visual easiness.\n- Figure 2: small x and y axis labels\n\n\n\n\n"}}, {"review_id": "EWX1yOwdeg", "reviewer": "Reviewer_8j5G", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging review!\n\nWhile we agree that the contributions are primarily empirical rather than methodological, we would argue that the way we have unified the various “off-the-shelf” methods is not trivial and does not appear to have been attempted before in the literature. It is not obvious how complementary different tools and data sources will be, nor how much improvements they could give when combined. For instance, the finding that large amounts of machine-translated data cannot eliminate the utility of human-crafted examples is a useful, and in our opinion non-obvious, finding. In general, we hope that the empirical contributions of our work will provide a set of useful reference points for future research on the integration of different cross-lingual transfer techniques.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Presents empirical evidence that nearly all the low-resource strategies for cross-lingual transfer learning scattered across the literature are viable (at least in the downstream NLI/classification task) and complementary so any available combination will provide additive performance benefits. Similarly, a small amount of gold \"real\" data remains valuable even when a large amount of synthetic data is available.", "reasons_to_reject": "The individual methods used in the paper are mostly off-the-shelf (although the paper makes some minor methodological adjustments noted above). Additive performance benefits are not very surprising, since there is no clear way in which one method would undo another.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper combines a number of techniques designed to improve cross-lingual transfer for low-resource languages into a single experimental framework.\n\n1) Few shot learning as an improvement to zero-short. Noting how effective a small amount of gold data is, the authors further elaborate on the methodology by upsampling the few-shot data compared to other generated training data.\n\n2) Parameter-efficient finetuning  (e.g., adaptor networks) to \"adapt\" base multilingual LLMs to a new language.\n\n3) Translate-train methods to produce synthetic training data in the low-resource language via NMT translation from a higher-resource language. NMT systems can themselves be adapted by additional training on even a small amount of parallel data in the low-resource language.\n\nAll of the methods add some performance to a downstream NLI task, but the experiments in this paper show that the benefits are complementary/additive - each additional method used adds more performance, so as many as possible should be used as resources allow.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper conducts a comprehensive study on how several approaches work together for very low-resource languages including few-shot learning, parameter-efficient tuning, and translate-train methods. The paper finds that these methods could be complementary and offer recommendations on how to combine them together. The experiments are comprehensive as well.
Decision: Accept-Main