Analysis of Multi-Source Language Training in Cross-Lingual Transfer (2024)

Abstract: The successful adaptation of multilingual language models (LMs) to a specific languagetask pair critically depends on the availability of data tailored for that condition. While cross-lingual transfer (XLT) methods have contributed to addressing this data scarcity problem, there still exists ongoing debate about the mechanisms behind their effectiveness. In this work, we focus on one of the promising assumptions about the inner workings of XLT, that it encourages multilingual LMs to place greater emphasis on language-agnostic or taskspecific features. We test this hypothesis by examining how the patterns of XLT change with a varying number of source languages involved in the process. Our experimental findings show that the use of multiple source languages in XLT—a technique we term Multi-Source Language Training (MSLT)—leads to increased mingling of embedding spaces for different languages, supporting the claim that XLT benefits from making use of language-independent information. On the other hand, we discover that using an arbitrary combination of source languages does not always guarantee better performance. We suggest simple heuristics for identifying effective language combinations for MSLT and empirically prove its effectiveness.

heading: 1 Introduction
text: There has been recent interest in multilingual language models (LMs), where multiple languages are used in training an LM, due to its capability of accepting multilingual inputs and achieving similar or even better performance compared to monolingual LMs at natural language processing (NLP) tasks. However, despite all these advantages, fine-tuning multilingual LMs using a specific language and an NLP task hinges on acquiring labeled data tailored to a specific purpose, which is not always possible.
*Corresponding author.
To address this challenge amid linguistic resource disparities, cross-lingual transfer (XLT) approaches have been proposed. In the context of finetuning multilingual LMs, XLT leverages resourcerich source languages in fine-tuning to boost performance on low-resource target languages.
Although there is still active debate regarding the inner workings of XLT, a significant body of previous research indicates that multilingual LMs are capable of separating language-specific and language-agnostic information from text (Muller et al., 2021), and that XLT favors the enhancement of language-agnostic features while reducing the emphasis on language-specific ones (Qi et al., 2022; Tu et al., 2022; Wang et al., 2022; inter alia).
While the standard practice in XLT is to employ only one language as a source for transfer, there exist attempts that have investigated the use of multiple source languages (Singh et al., 2019; Roy et al., 2020; Kew et al., 2023; Chai et al., 2024; Shaham et al., 2024). These studies reveal that employing multiple source languages in XLT—the concept we refer to as Multi-Source Language
1 712
Training (MSLT)—leads to performance improvement. However, despite it is evident that MSLT has clear advantages, several research questions remain unexplored, including: (1) the specific changes that it induces in the internal states of LMs, (2) the conditions under which it achieves its optimal performance, and (3) the breadth of applications where it can be effectively applied, among others.
In this work, we introduce a series of comprehensive analyses exploring the inner workings of MSLT, along with offering guidance for its effective utilization. Specifically, we first attempt to illustrate the utility of MSLT with a couple of intuitive visualizations, to clarify the underlying reasons for its effectiveness. In addition, our experimental results discover various intriguing findings. Regarding the number of languages involved in MSLT, we observe its positive correlation with performance in general, although the trend tends to plateau as the number of languages exceeds a certain threshold.
Our empirical discovery also suggests that the enhanced diversity of source languages in MSLT does not invariably lead to improved performance. This underscores the importance of carefully selecting an effective combination of source languages (see Figure 1 for an intuitive illustration). Therefore, we examine various heuristics that can identify a suitable selection of source languages from an exponentially large pool of possible combinations. Based on our tests, we propose several competent strategies, some of which are derived from the statistical properties of languages in terms of pretraining, while others utilize linguistic characteristics of languages. Lastly, we discuss intriguing patterns evident in the group of languages selected as sources for transfer, providing insights into the interplay of languages in MSLT.

heading: 2 Related Work
text: Multilingual LMs (Conneau et al., 2019; Scao et al., 2022; inter alia) are widely recognized for their ability to process inputs from diverse languages in an integrated manner. They have recently been the subject of extensive research for uncovering their working mechanisms, and this effort has converged on the robust hypothesis that the internal states of multilingual LMs can be categorized into language-sensitive and language-agnostic components (Choenni and Shutova, 2020; Chang et al., 2022; Zhao et al., 2020; Muller et al., 2021).
Meanwhile, cross-lingual transfer (XLT) aims to improve the performance of multilingual LMs on specific tasks in languages with limited resources (i.e., target languages), by utilizing the support from resource-rich languages (i.e., source languages). Previous research on XLT has explored diverse implementations of the concept. For instance, Yang et al. (2022) attempt to achieve XLT by blending representations from source and target languages, while Zheng et al. (2021) introduce data augmentation techniques such as subword sampling and code-switching. Wang et al. (2022) propose a contrastive learning framework aimed at reducing the differences in sentence embeddings between languages. Concurrently, Qi et al. (2022) present a prompt-based approach that facilitates information transfer from a source language to prompts composed in the target language.
Note that the majority of existing methods for XLT (Libovickỳ et al., 2020; Tiyajamorn et al., 2021; Yang et al., 2022), including the aforementioned studies, adhere to a common theme: they are crafted to facilitate XLT by prompting the enhancement of language-agnostic or universal features within multilingual LMs. In this work, we focus on a simple strategy that yields a similar effect, namely, introducing multiple source languages in XLT instead of just one as is customary, deliberately steering clear of complex algorithms.
In the literature, there is empirical acknowledgment that increasing the diversity of source languages in XLT can boost its performance. Singh et al. (2019) suggest that pairing sentences written in different languages (for example, one sentence in one language and another in a different language) is beneficial for improving performance on the crosslingual natural language inference (XNLI; Conneau et al., 2018) task. They additionally observe that the performance fluctuates based on the languages from which the sentences are sampled. Roy et al. (2020) demonstrate a similar trend in questionanswering tasks. Further, Kew et al. (2023) and Shaham et al. (2024) reveal that MSLT is also advantageous for instruction-tuned LMs.
Despite the recent surge in attention towards MSLT, it is important to highlight the lack of a comprehensive analysis or tailored strategies in this area so far. For instance, Kew et al. (2023) consider only pre-defined language sets in their experiments, ignoring the potential of discovering the best combination of source languages. On the other hand, Singh et al. (2019) and Shaham et al. (2024) utilize
2
all available languages without specifically seeking to determine the ideal number of languages for the optimal performance of MSLT. Our work tackles these limitations, by presenting a series of wellsupported illustrations that clarify how and why MSLT operates, along with providing guidelines for enhanced utilization of the MSLT framework.

heading: 3 Terminology
text: We first clarify the core concepts and their corresponding acronyms in this section.
Cross-Lingual Transfer (XLT) XLT is a technique applicable to multilingual LMs, particularly when obtaining a satisfactory amount of data for the language-task pair of interest is challenging. This is built on the assumption that fine-tuning a multilingual model for a task T in a resource-rich language A can lead to a synergistic improvement in performance for another language B, especially if B has limited data available for the task T .
In this work, we focus on the zero-shot XLT setting, under the assumption that no suitable data is available for the target language.
Single-Source Language Training (SSLT) SSLT represents an environment for XLT where multilingual LMs are trained on a single language. From now on, we adopt the notation ‘SSLT(L)’ to signify the setting where the language L is exclusively utilized as a source language in XLT.
Multi-Source Language Training (MSLT) MSLT refers to an environment in which two or more source languages are considered for XLT. We use the notation ‘MSLT(L1, L2, . . . , Ln)’ to denote the setting where MSLT is conducted with languages L1, L2, . . . , Ln.
MSLT, when feasible, enables the possibility of leveraging an exponentially higher number of data points by allowing the use of instances from multiple languages. However, as the purpose of this study is to analyze the impact of language diversity, we decide to keep consistent the total number of data points used in experiments. For example, if SSLT(en) utilizes 1,000 data points in English, the corresponding MSLT(en, es) incorporates 500 data points each from English and Spanish, respectively.

heading: 4 Verifying the Effectiveness of MSLT
text: We present a thorough analysis to confirm the effectiveness of MSLT, and additionally provide insights into its inner workings. We describe the
intuition behind how MSLT enhances XLT performance (§4.2) and present visualizations related to our intuition (§4.3). We also provide empirical findings with respect to the ideal number of source languages in MSLT (§4.4). Section 4.1 presents the configurations used throughout the section.

heading: 4.1 Experimental Settings
text: To investigate the inner workings of MSLT, we conduct experiments using XLM-RoBERTaBase (XLMRBase; Conneau et al., 2019), a compact multilingual encoder that covers around 100 languages. We select three tasks from the XTREME benchmark (Hu et al., 2020) as our testbed for analysis: named entity recognition (WikiANN; Pan et al., 2017), cross-lingual natural language inference (XNLI; Conneau et al., 2018), and cross-lingual paraphrase identification (PAWS-X; Yang et al., 2019). More details on the datasets and training processes are specified in Appendix A and B, respectively.

heading: 4.2 Advantages of MSLT over SSLT
text: Rooted in the widely accepted notion within the literature that multilingual LMs can categorize language-specific and language-agnostic concepts (Muller et al., 2021), there is a prevalent belief that XLT can be more effectively activated by encouraging the influence of language-agnostic features while diminishing the role of language-specific ones. We draw on this wisdom to intuitively explain why MSLT generally outperforms SSLT.
Figure 2 illustrates the potential advantages of
3
MSLT over SSLT. When data from a single language is used in training for XLT, as in the case of SSLT, it may be challenging to guide a model to establish a decision boundary for classification (indicated by blue in Figure 2) that can robustly function across different languages. Conversely, with MSLT, the model is exposed to signals from a diverse range of languages, encouraging more frequent exploitation of language-agnostic features. This leads to representations that are comparatively languageindependent, enabling the identification of a more robust decision boundary applicable across different languages (red in Figure 2). In summary, we speculate that MSLT provides a more advantageous environment for XLT compared to SSLT, by underscoring the utility of language-agnostic features.

heading: 4.3 Visualization of Embeddings after MSLT
text: To better understand the impact of MSLT and empirically validate the hypothesis presented in §4.2, we conduct a comparative study by visualizing embeddings before and after applying SSLT and MSLT. To be specific, we first train the XLM-RBase model with data instances from the XNLI dataset, resulting in three variants: (1) the original XLM-R, (2) XLM-R with SSLT(en), and (3) XLM-R with MSLT(en,es,de).2 We then visualize the embeddings computed by each model variant,3 using a set of sentences from the Parallel Universal Dependencies (PUD) treebanks (De Marneffe et al., 2021). This decision stems from our goal to evaluate the model’s general capabilities, not just its
2en: English, es: Spanish, de: German. 3Embeddings are derived by mean-pooling the hidden
states from the last layer of the language model.
performance on fine-tuning tasks, but also its effectiveness with out-of-domain data instances. We make use of t-SNE (van der Maaten and Hinton, 2008) for dimension reduction.
In Figure 3, we observe that the XLM-R model with MSLT (the rightmost) demonstrates significantly better language-agnostic alignment for unseen languages (ar and id), compared to both (1) the original XLM-R and (2) XLM-R after SSLT. This implies that the diversification of source languages prevents a model from learning language-specific features and encourages it to be more language-independent, which is beneficial for achieving better performance in XLT.

heading: 4.4 On the Optimal Number of Languages
text: In the previous part, we qualitatively demonstrated the effectiveness of MSLT through graphical illustrations. Here, we present a supplementary quantitative analysis on the impact of the number of source languages on XLT, addressing two research questions: (1) Is MSLT consistently superior to SSLT, at least in the evaluated environments? (2) What is the optimal number of languages for MSLT? We select English, Spanish, German, Chinese, and French as candidates for source languages and evaluate the performance of different language groups on cross-lingual datasets, calibrating their size from 1 (SSLT) to 5 (MSLT with five languages).4
Figure 4 reveals that regardless of the task type or data quantity, task performance remarkably improves as the number of source languages increases from 1 to 3, demonstrating the effectiveness of
4The scores of groups with the same size are averaged.
4
MSLT. However, when the size of the group exceeds 3, the performance exhibits minimal improvement or even declines in some cases.5 Taking into account the outcomes of our experiments and the expenses involved in creating datasets across various languages, we propose opting for three source languages as a pragmatic and sensible selection for MSLT. Consequently, we make use of threelanguage groups in the forthcoming experiments.

heading: 5 Language Set Composition in MSLT
text: In the previous section, we demonstrated that MSLT exhibits markedly superior performance on XLT compared to SSLT. Assuming the selection of k source languages from a total of n languages, there exist (nk) possible combinations. Given that n can be significantly larger than k, fine-tuning with every possible combination of source languages to identify the optimal mix is not time- and resourceefficient. Therefore, it is necessary to select an appropriate combination more efficiently. In this section, we first establish the importance of selecting an effective set of source languages and then proceed to examine various criteria for choosing an optimal source language combination.

heading: 5.1 Experimental Settings
text: We perform experiments utilizing the BLOOM-7B model (Scao et al., 2022), in conjunction with the previously utilized XLM-R. We fix the size of the source language set at 3, and its elements are chosen from a pool of 7 language candidates,6 which are commonly considered resource-rich languages.
5In the case of XLM-RLarge, performance shows a slight improvement even with more than three languages.
6Arabic (ar), German (de), English (en), Spanish (es), French (fr), Russian (ru), Chinese (zh).
When applying MSLT to XLM-R, we reuse the WikiAnn and XNLI datasets for fine-tuning and evaluation, as was done in the previous section.7 For experiments with BLOOM-7B, we leverage separate datasets for instruction-tuning and evaluation, respectively. By introducing this instructiontuning setting, we aim to evaluate whether MSLT is also effective in this new paradigm, which would further extend the applicability of MSLT. BLOOM7B is instruction-tuned using the Bactrian-X dataset (Li et al., 2023). For evaluation, we employ another suite of datasets, which includes XCOPA (Ponti et al., 2020), XWinograd (Muennighoff et al., 2023), and XStoryCloze (Lin et al., 2022). Also note that we fine-tune BLOOM-7B with QLoRA (Dettmers et al., 2023) for computational efficiency and to ascertain how well MSLT operates in a parameter-efficient fine-tuning setup. Details on the experimental settings can be again found in Appendix A and B.

heading: 5.2 The Harmony of Languages Matters
text: We compare the performance of MSLT across various source language combinations. According to our experiments, the magnitude of performance improvement varies greatly depending on the combination used, and in some cases, a poor choice may lead to performance deterioration. Figure 5 represents the upper and lower bounds of the performance of MSLT, along with that of SSLT as a baseline. A notable performance gap is observed between the optimal and worst combinations in most
7PAWS-X is omitted owing to its limited linguistic variety.
5
experiments. In certain tasks, e.g., XCOPA and XWinograd, MSLT with the least effective combination fails to outperform SSLT. These findings imply that the combination of source languages in MSLT has a substantial impact on performance, highlighting the need to establish criteria for selecting appropriate source languages.

heading: 5.3 Criteria for Source Language Selection
text: We further explore the criteria for selecting source language combinations by empirically testing several hypotheses related to them.

heading: 5.3.1 Size of Pretraining Data
text: We examine whether using a combination of the most frequent languages in the pretraining data leads to improved performance in MSLT. The underlying motivation is that the proportion of a language in the pretraining data might correlate with how much a model acquires the knowledge specific to that language during pretraining. Based on the volume of pretraining data for each language, as reported in the original papers (Conneau et al., 2019; Scao et al., 2022), we select three languages from the source language pool for fine-tuning: English, Russian, and German for XLM-R, and English, Simplified Chinese, and French for BLOOM-7B.

heading: 5.3.2 Vocabulary Coverage
text: Inspired by the findings of Pires et al. (2019), which suggest that XLT depends on vocabulary overlap between source and target languages, we test a heuristic that chooses source languages with extensive vocabulary coverage. Suppose there
are n source languages available, represented as L = {L1, · · · , Ln}, and let the vocabulary set for each language Li be denoted by VLi . The coverage of a candidate set LC ⊆ L is then defined by
Coverage(LC) =
∣∣∣ ⋃ Li∈LC VLi ∣∣∣
∣∣⋃ Li∈L VLi ∣∣ .
We anticipate that language combinations possessing extensive lexical breadth will encompass a broader spectrum of the target language’s vocabulary, ultimately leading to better XLT scores.

heading: 5.3.3 Linguistic Diversity
text: The last criterion we examine is the selection of source languages that exhibit distinct linguistic properties. This criterion shares a similar motivation with the previous ones, aiming to use languages that can generalize to other unseen languages. However, it uniquely posits that utilizing as diverse a range of source languages as possible will lead to a more stable and comprehensive modification of the model’s internal mechanism.
To achieve this, we first represent each language as language vectors and then identify a candidate set whose language vectors have the least similarity to each other. We introduce Lang2Vec (Littell et al., 2017), a framework offering a suite of vectors for each language that mirrors its linguistic properties. In particular, we test five variants of Lang2Vec, i.e., Syntax, Phonology, Inventory, Family, and Geometry vectors. Formally, given a set of languages L = {L1, . . . , Ln} and their language vectors {v1, . . . , vn}, the diversity score of a com-
6
bination LC ⊆ L is defined by the sum of pairwise cosine similarity of language vectors:
Diversity(LC) = ∑
{Li,Lj}⊆LC (1− sim(vi, vj)) .
Since our objective is to find a combination LC having the maximum diversity, i.e. the minimal uniformity, thus the desired combination is obtained by argmaxLCDiversity(LC).
On the other hand, we consider an extra baseline called just ‘Language Embedding from LMs’ or just ‘Embedding’. This process is similar to that of Lang2Vec, but the vectors here are derived directly from pretrained LMs, specifically XLM-R or BLOOM-7B, depending on the case. The language vector for each language is calculated by averaging the embeddings of sentences in that language.

heading: 5.4 Evaluation on Selection Criteria
text: Table 1 represents the results from experiments on source language selection criteria presented in §5.3. The scores presented in the table are the averages of all scores for each task across target languages. Target languages will have the same language set when the selection criterion applied is identical.
The findings suggest that selection criteria based on the statistical properties of language models, referenced in Sections §5.3.1 and §5.3.2, do not substantially improve the performance of XLT. In contrast, selecting source languages based on linguistic diversity demonstrates a significant performance gain. This implies that in MSLT, prioritizing diversity within the source language combination is more important than relying on the characteristics of corpora used in pretraining LMs.
Furthermore, although the ‘Embedding’ method, which measures diversity directly based on embed-
dings from LMs, led to a fairly reasonable performance gain, measuring diversity using Lang2Vec vectors demonstrated the best outcomes. This further reinforces the importance of properly considering the inherent properties of languages in NLP.

heading: 5.5 Language Selection Sticking to English
text: This section features a case-study assuming that one of the source languages is necessarily English, a common scenario in practical applications. Table 2 reports that even in cases where English is always included in combinations, selecting other source languages based on diversity (or least similarity) according to Lang2Vec still exhibits reasonable performance.

heading: 6 Discussion

heading: 6.1 Which Language is Suitable for MSLT?
text: To identify the necessary conditions of the languages comprising the optimal set, we manually examine top-ranked language combinations for each task in practice. Table 3 showcases the top five language combinations that exhibit the best performance for each task. The results reveal that, across all tasks, Chinese (zh), Arabic (ar), and German (de) are the most commonly used languages, with frequencies of 17, 15, and 15, respectively. Furthermore, we find that each of the top 5 combinations includes two or more distinct writing systems, such as Arabic scripts for Arabic, Latin scripts for European languages, Chinese characters for Chinese, and Cyrillic scripts for Russian. This suggests that
7
the diversity of writing systems can act as a key factor in constructing good language sets for MSLT.
To further investigate this perspective, we conduct an extra experiment whose outcomes are reported in Table 4. In this experiment, we categorize source language sets (in total, 35) into three types based on the extent of diversity in their writing systems, and then compare the performance of each group. The outcomes show that combinations of language with distinct writing systems always outperform other cases, implying the importance of diversity in writing systems among languages. We believe that Lang2Vec-based criteria can properly exploit similar linguistic information.

heading: 6.2 Is Increasing Diversity Really Helpful?
text: In §5.3.3, we follow the intuition that selecting language combinations with elements exhibiting distinct characteristics is beneficial. In fact, we
can also consider the opposite direction, grouping languages with relatively high similarities rather than low ones. To determine which direction is more helpful for MSLT, we conduct an experiment using the two contrastive approaches.
Experimental results in Figure 6 reveal that constructing combinations with maximum diversity (as in §5.3.3) yields better performance than combinations having minimum diversity (i.e., high similarity) across the majority of tasks.8 Notably, there is a significant performance gap between the two contrasting strategies for WikiAnn. We also observe similar patterns for other datasets, confirming the desirability of our decision to consider the diversity of source languages in §5.3.3.

heading: 7 Conclusion
text: This work demonstrates the positive impact of Multi-Source Language Training (MSLT) in crosslingual transfer (XLT). By leveraging multiple source languages, MSLT facilitates the learning of language-agnostic features in multilingual LMs. We provide qualitative and quantitative evidence suggesting that this approach promotes crosslingual semantic alignment, leading to better XLT performance compared to using a single source language. Furthermore, we identify the importance of selecting the optimal size and composition of source language sets for MSLT. To guide this selection efficiently, we propose diverse criteria based on the statistical and linguistic properties of candidate languages, such as Lang2Vec. Importantly,
8Full experimental results are listed in Appendix, Table 6.
8
MSLT demonstrates effectiveness across various configurations, including different tasks, architectures, and training paradigms. We hope this study serves as a springboard for further exploration of factors influencing the development of effective cross-lingual transfer methodologies.
Limitations
Limited source language pools Given the practical availability of language-specific datasets and the ease of experimentation, this paper designates the top 7 languages that appeared most frequently in various multilingual datasets as source language candidates. By analyzing the permutations of selecting three out of these 7 languages, we aim to provide insights based on a wide range of scenarios to establish relevance in a generalized setting. However, it is important to note that further experiments with a greater number of languages as candidates for source languages are left as future work.
Comparison with other source language selection methods Our study shares many similarities with other research on source language selection for cross-lingual transfer, aiming to find the optimal source languages. However, many selection methods tend to focus on the correlation between the source and target languages, making the selection process target-specific and often favoring singlelanguage choices, which makes it challenging to quantitatively compare them with our approach.
Ethics Statement
In this study, we utilize models and datasets publicly available on Huggingface, ensuring that no ethical issues are associated with their usage. Additionally, we endeavor to minimize content related to social biases. However, we acknowledge the possibility that some biases may have been infused during the pretraining process of language models, highlighting the need for sufficient consideration in this regard.

heading: Acknowledgements
text: This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2020-II201373, Artificial Intelligence Graduate School Program(Hanyang University)). This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) under the
artificial intelligence semiconductor support program to nurture the best talents (IITP-2024-RS2023-00253914) grant funded by the Korea government(MSIT). This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(*MSIT) (No.2018R1A5A7059549). *Ministry of Science and ICT.

heading: A Dataset
text: WikiAnn is a named entity recognition dataset sourced from Wikipedia, comprising 282 language data segments. In our experimental setup, we utilize 7 languages (Arabic (ar), German (de), English (en), Spanish (es), French (fr), Russian (ru), Chinese (zh)) as the source language and 8 languages (Indonesian (id), Greek (el), Hebrew (he), Finnish (fi), Thai (th), Turkish (tr), Japanese (ja), Korean (ko)) as the target language for crosslingual transfer.
PAWS-X comprises 23,659 human-translated evaluation pairs and 296,406 machine-translated training pairs across six typologically distinct languages: French (fr), Spanish (es), German (de), Chinese (zh), Japanese (ja), and Korean (ko). These translated pairs originate from examples within PAWSWiki. Two languages that are not included in the source language are used as target languages.
XNLI is an evaluation corpus for language transfer and cross-lingual sentence classification in 15 languages. XNLI cover 15 languages: Arabic (ar), Bulgarian (bg), Chinese (zh), English (en), French (fr), Greek (el), German (de), Hindi (hi), Russian (ru), Spanish (es), Swahili (sw), Thai (th), Turkish (tr), Urdu (ur), and Vietnamese (vi). Among the aforementioned 15 languages, all except the one employed as the source language in our experiment serve as target languages.
XCOPA, known as Cross-lingual Choice of Plausible Alternatives, is a benchmark designed to evaluate machine learning models’ capability to transfer commonsense reasoning across different languages. This dataset comprises translations and re-annotations of the English COPA, encompassing 11 languages (Estonian (et), Haitian (ht), Indonesian (id), Italian (it), Quechua (qu), Swahili (sw), Tamil (ta), Thai (th), Turkish (tr)) from diverse language families and regions worldwide. All languages in XCOPA are evaluated as target languages by instruction-tuned models.
XWinograd is a multilingual benchmark designed for assessing commonsense reasoning, consisting of Winograd Schema Challenge problems presented in six languages. The task entails selecting the most plausible sentence from slightly varied
options. We used Japanese (ja) and Portuguese (pt) as target languages for our experiments.
XStoryCloze comprises the English StoryCloze dataset professionally translated into 10 nonEnglish languages. Out of these, we chose six languages as target languages, ensuring they did not overlap with the source language set. The selected target languages are as follows: Basque(eu), Hindi (hi), Indonesian (id), Burmese (my), Swahili (sw), and Telugu (te).

heading: B Training Details
text: We report the hyperparameters used for training each model in Table 5. For XLM-RBase and XLMRLarge, we conducted a hyperparameter search to find an appropriate learning rate for training. As for BLOOM-7B, we attempted to follow the settings from the original LoRA-tuned model,9 but due to the lack of computation resources, we were unable to replicate the original settings such as batch size 128. Instead, we adjusted the training steps the same as the original LoRA-tuning setting. For the BLOOM model, prompt strategies employed for instruction-tuning and inference are listed in Table 7 and Table 8, respectively.
9https://huggingface.co/MBZUAI/ bactrian-x-bloom-7b1-lora
12
13
14

heading: Future Work
text: However, it is important to note that further experiments with a greater number of languages as candidates for source languages are left as future work. Comparison with other source language selection methods Our study shares many similarities with other research on source language selection for cross-lingual transfer, aiming to find the optimal source languages. However, many selection methods tend to focus on the correlation between the source and target languages, making the selection process target-specific and often favoring singlelanguage choices, which makes it challenging to quantitatively compare them with our approach