X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity
Authors: Taejun Yun
Jinhyeon Kim
Deokyeong Kang
Seonghoon Lim
Jihoon Kim
Taeuk Kim
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-lingual transfer
Multilingual Language Model

Abstract: Cross-lingual transfer (XLT) is an emergent ability of multilingual language models that preserves their performance on a task to a significant extent when evaluated in languages that were not included in the fine-tuning process.
While English, due to its widespread usage, is typically regarded as the primary language for model adaption in various tasks, recent studies have revealed that the efficacy of XLT can be amplified by selecting the most appropriate source languages based on specific conditions.
In this work, we propose the utilization of sub-network similarity between two languages as a proxy for predicting the compatibility of the languages in the context of XLT.
Our approach is model-oriented, better reflecting the inner workings of foundation models. 
In addition, it requires only a moderate amount of raw text from candidate languages, distinguishing it from the majority of previous methods that rely on external resources.
In experiments, we demonstrate that our method is more effective than baselines across diverse tasks.
Specifically, it shows proficiency in ranking candidates for zero-shot XLT, achieving an improvement of 4.6% on average in terms of NDCG@3.
We also provide extensive analyses that confirm the utility of sub-networks for XLT prediction.

X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity
Taejun Yuni, Jinhyeon Kim', Deokyeong Kang', Seong Hoon Lim', Jihoon Kim", Taeuk Kim -13
Dept. of Computer Science & *Dept. of AI Application, Hanyang University
HHyundai Motor Company
{tj1676, kimj kimjinnyyee inhyeon, rkdejr dmammf1, 1, (amtaeek)@haayamm ac. kr ac......ooon2555eennaa....o
Abstract
Cross-lingual transfer (XLT) an emergent ability of multilingual language models that preserves their performance on task to sig- nificant extent when evaluated in languages that not included in the fine-tuning process. While English, due to its widespread usage, is typically regarded as the primary language model adaption in various tasks, recent stud- ies have revealed that the efficacy of XI.T can be amplified by selecting the most appropriate source languages based on specific conditions. this work, we propose the utilization of sub-network similarity between two languages proxy for predicting the compatibility of the languages the context XLT. Our ap- proach is model-oriented, better reflecting the inner workings of foundation models. addi- tion, it requires only a moderate amount of raw text from candidate languages, distinguishing it from the majority of previous methods that rely on external resources. In experiments, we demonstrate that our method is more effective than baselines across diverse tasks. Specifically, it shows proficiency in ranking candidates for zero-shot XLT, achieving an improvement of 4.6% on average in terms NDCG@3. We also provide extensive analyses that confirm the utility of sub-networks for XLT prediction.
0.8
0.6
0.5
03
AR
ME
10
IT
JA
KD
2H
Target Language
Figure I: This figure shows that the performance of zero- shot XLT for named entity recognition (NER) varies greatly depending
the language used for task fine- tuning (i.e., the source language). Although English is mostly a default choice for XLT,, relying on often lags behind the best results. In contrast, source languages that are nearly optimal. The dotted lines represent average scores for each approach.
method suggests
agenda is partially becoming reality, although there still remain numerous issues be addressed. A prevalent obstacle to the broader use of mul- tilingual models the need for considerable amount of task-specific data in the target language to properly adapt the models for a particular pur- pose. This is especially true when following the "pre-training and fine-tuning" paradigm, general approach for leveraging models of reasonable sizes. That is, merely incorporating support for specific language in model insufficient address a downstream task in that language; addition, one needs to secure labeled data in the target language, which is often unfeasible in low-resource scenarios. Fortunately, the aforementioned problem can be partially mitigated by employing the cross-lingual transfer (XLT) method (Pires et al., 2019; Cao al., 2020) that exploits supervision from another lan- guage dubbed the source language. For instance, a multilingual model tuned on English reviews can be directly utilized for classifying the sentiment of
1 Introduction
One of the aspired objectives in the field of natural language processing (NLP) is to ensure equal treat- ment of text regardless of the language it originates from (Ruder et al., 2019). This goal is highly desir- able in that it promotes the democratization of NLP technologies, making them accessible everyone worldwide. With the emergence of multilingual language models (Devlin et al. (2019); Conneau et al. (2020); Lin et al. (2022), inter alia) that can process over a hundred languages, the ambitious
"Carresponding author.

---

In NLP, the term cross-lingual transfer represents a technique first tuning a model for a task of inter- est in one language (i.e., source language) and then applying the model for input from the same task but the other language (i.e., target language). With the introduction of multilingual language models, e.g., mBERT (Devlin et al., 2019) and RoB.Raa (Conneau et al., 2020), and the initial findings sug- gesting that this technique is viable based on such models (Pires et al., 2019: Wu and Dredze, 2019; Cao et al., 2020), it has emerged as an off-the-shelf solution when only a limited amount of data is available for a specific task in the target language. reviews in German with decent performance. As seen in the example, English is commonly selected as the source language due to its abundant and di- verse data resources available for various tasks.
of base model as it derives the similarity score from the model's internal values, i.c., gradients, instead of relying on the information pre-defined by external resources. In addition, in comparison to previous data-centric methods, our technique is efficient in terms of the quantity of data required. Moreover, we show in experiments that our method is widely applicable across many tasks, It exhibits superiority over competitive baselines in the major- ity of considered settings. When evaluated in terms of NDCG@3, 3, it achieves an improvement of 4.6% on average compared to its competitors.
However, it is not intuitive that English would always be the best option, particularly consider- ing its heterogeneity with non-Latin languages. In fact, recent studies (Lauscher et al., 2020; Ture et al., 2021; Pelloni et al., 2022) have substantiated the conjecture, verifying that certain East Asian languages such as Chinese, Japanese, and Korean exhibit mutual benefits in cross-lingual transfer, sur- passing English. In our preliminary study, we also discovered that the performance of zero-shot XLT for NER considerably depends on the choice of the source language (see Figure 1). These findings have naturally led to research into identifying opti- mal source languages for different configurations of cross-lingual transfer, as well as investigating the factors that contribute to their effectiveness.
The outline of this paper is as follows. We first present related work ($2) and explain the details of our approach ($3). After presenting experimental settings (64), we perform comprehensive compar- isons against baselines ($5) and conduct extensive analyses provide insights into using sub-network similarity for estimating the performance of XLT ($6). We finally conclude this paper in $7.
Popular strategies for uncovering the optimal source language cross-lingual transfer include those based on aequiistically-insiree features (Lit- tell et al., 2017; Xia et al., 2020), ones that count on statistical characteristics of corpus of cach can- didate language (Pelloni et al., 2022), and methods that aggregate various clues for conducting regres- sion (de Vries et al., 2022; Muller al., 2023). However, the existing techniques have several limi- tations: (1) they normally require a combination of a myriad of features, each of which costly to ob- tain, (2) their effectiveness has only been validated for a limited number of tasks, (3) they present a general recommendation rather than one tailored to individual settings, or (4) they do not reflect the inner workings of the utilized language model.
2 Related Work
2.1 Cross-lingual Transfer Prediction
To address these challenges, this work presents a novel and efficient method called X-SNS, which aims to predict the most appropriate source lan- guage for a given configuration cross-lingual transfer. We introduce the concept of sub-network similarity as a single key feature, which is com- puted as the ratio of overlap between two specific regions within a model that are activated during the processing of the source and target language. The main intuition behind the proposed method lies in the notion that the degree of resemblance in the structural changes of model activations, induced by each language, determines the efficacy XLT.
There is ongoing debate in the literature on what factors contribute to the performance of XLT. Karthikeyan et al. (2020) claim that the lexical overlap between languages does not significantly impact its effectiveness, whereas the depth of the employed network is crucial. In contrast, Lauscher et al. (2020) argue that two key aspects inffuence the level of success in cross-lingual transfer: (1) the linguistic similarity between the source and target languages, and (2) the amount of data used for pre-training in both languages. Similarly, de Vries et al. (2022) exhaustively explore all possible pairs with 65 source and 105 target languages for POS tagging and derive a conclusion that the presence of both source and target languages in pre-training
Our approach is bbaedd data-efficient and versatile. is closely tied to the actual workings

---

Mustinuual
Legguege
CLB
Top
(7%
(5)
Soormn Candkoate 2H
EN
(2)
Ovarlasping
EN
ZH
KD
Sproife
Flater
19.2975.3447 77734477..80
Intoration
Prodedin
E+
Z Subratoook
Top
73
Cross-ing ingual
Emnracting
100
27 11.000000
Temnder
Source Ex Fi HD
Muttlingull
Language Perr Samierty
Soor 04-42 55330 0881
Labele
Unbdbeled
Taak
Taak
Vel
Figure 2: Illustration of the proposed method (X-SNS). To extract Language-speciffg uub-network from multilin- gual language model, (1) it first computes the gradients of an objective function with respect to model parameters for each language. Subsequently. (2) it approximates the Fisher Information of the model's parameters using their gradients and (3) selects the top p% parameters based on the computed values, constructing (binary) sub-networks. Then, (4) the similarity of two languages is defined the Jaccard coefficient of their respective sub-networks. Finally, (5) the similarity score is used as proxy for predicting the effectiveness of XLT between two languages. The symbol denotes the optimal configuration for the proposed method (see the related experiment $5.1.)
is the most critical factor.
sis, which states that every neural network contains subset that performs on par with the original net- work. Similarly, research in the field of model pruning (Han al. (2015); Liu et al. (2019); inter alia) primarily concerned with iteratively remov- ing a portion the original network create its lightweight revision while keeping performance. Meanwhile, there is another line of research that focuses modifying only portion of neural network while freezing the remaining part. Specif- ically, Xu et al. (2021) show that better perfor- mance and domain generalization can be achieved by tuning only a small subset of given network. Ansell et al. (2021) propose the separate training of language- and task-specific sub-networks, which are subsequently combined to facilitate the execu- tion of a specific task in the target language.
One of the practical challenges in applying cross- lingual transfer arises when multiple source lan- guages are available for the task of interest. In such scenarios, it is necessary to test various variations of a language model by fine-tuning it with data from each source language, which can be cumber- some. To alleviate this burden, several previous studies have proposed methods aimed at estimat- ing the performance of XLT for given pair of languages and target task, even without direct fine-tuning of foundation models. Lin et al. (2019) propose an approach to ranking candidate source languages according to scoring function based on different features including phylogenetic similarity, typological properties, and lexical overlap. Simi- larly, Muller et al. (2023) estimate numeric transfer scores for diverse tasks by conducting linear regres- sion on linguistic and data-driven features. On the other hand, Pelloni et al. (2022) propose single feature known as Subword Evenness (SuE), which displays a robust correlation with transfer scores in masked language modeling. The authors claim that the optimal source language for the task is the one in which tokenization exhibits unevenness. In this study, also present an intuitive method for XLT prediction that is more effective in most cases.
In this work, we follow the practice established by previous studies regarding sub-networks, but do not engage in network pruning or directly training a portion of a network. Instead, we identify language-sensitive areas within the network using gradient information and predict the level cross- lingual transfer based on the overlap of these areas.
3 Proposed Method: X-SNS
We present the technical details of our approach, dubbed X-SNS (Cross(X)-Linguall Transfer Predic- tion through Lub-Network Similarity). Its primary objective is to suggest proper source language for XLT, eliminating the need for developing separate copies of language models fine-tuned by task data in each language. To this end, we propose utilizing the similarity between a pair of -aguuag--specffic
2.2 Sub-Network
The notion of sub-networks is broadly embraced in the machine learning community means of extracting or refining crucial components of the original network. For instance, Frankle and Carbin (2019) propose the Lottery Ticket Hypothe-

---

sub-networks as an indicator to estimate the com- patibility of two languages in XLT. The overall procedure of the method illustrated in Figure 2. The core mechanism of X-SNS lies in the deriva- tion of sub-networks customized for individual lan- guages and the computation of similarity between a pair of sub- neb-neeworks. As crucial component in the construction of our targeted sub-network, we intro- duce the Fisher information (Fisher, 1922), which provides a means of quantifying the amount of in- formation contained in parameters within a neural network (Tu et al., 2016; Achille et al., 2019). Con- cretely, we derive the (empirical) Fisher informa- tion of a language model's parameters as follows. First, given a data point (x, v) from the true data distribution- input x and output y-we define the Fisher Information Matrix F for parameters 0 as:
words, we identify a group of parameters within the network whose importance scores are positioned in the top ndh percentile. Note that the proposed method only considers the relative significance of each parameter in terms of its Fisher information, ignoring its actual values. By doing so, X-SNS not only avoids the need for extra training of base mod- els but also ensures its efficiency. This also sets it apart from previous work that employs similar framework for model pruning or task fine-tuning. There remain two unspecified factors that play vital role in acquiring language sub-networks us- ing X-SNS. The first is the choice of the corpus D used for obtaining f. We utilize a corpus from language of our interest as the basis for deriving a sub-networt specific to that language. Besides, we test two types of corpora for each language: a dask-oriented corpus (T) and a general domain corpus (W), i.e., Wikipedia. As shown in $5, a task- oriented corpus guarantees superior performance, while general one enables broader application of the approach with satisfactory results. In exper- iments, also confirm that our method is data- efficient, being nearly optimal when |D] < 1000. The second factor is the output distribution of the model p[y|| ; 0). In previous studies (Xu et al., 2021; Foroutan et al., 2022), p(y) 0) is usually defined as task-specific layer (T), e.g., clas- sifier, implemented on the base model, requiring additional training of the top module. However, as we prioritize the efficiency of the algorithm, the ex- tra fine-tuning is not considered in our case, even in the environment where a labeled, task lask-oriented cor- pus is available. Instead, we imitate such an avenue with a random classifier combined with the cross- entropy loss for the target task. Mcanwhile, another realistic direction of defining p(y) x; 0) is rely on the pre-training objective of the base model, i.e., (masked) language modeling (C; Ansell et al. (2021)). This not only eliminates the need for task supervision but also allows one to recycle the lan- guage modeling head built during the pre-training phase. We thus regard L as the primary option, Refer to $5.1 where we conduct related analysis. Lastly, given a pair of sub-networks S, and st for the source (s) and target (t) languages respec- tively, the similarity of the languages is calculated as the Jaccard similarity coefficient between the two vectors: |s O sel / U Spl. In practice, set
i log p(u) 0) θ log plylp 0) T]
F(0) E
00
00
Second, since it is practically intractable to derive the exact F, we assume this is a diagonal matrix, aligning with the approaches employed in the re- lated literature (Sung et al., 2021; Xu et al., 2021). 2 As result, we obtain the Fisher information vector f == diag( (F)).. Finally, approximate f with f, which is the average of Monte Carlo estimates of f based on a given corpus D {(x), }5)} Formally, the (approximated) Fisher information of the 2h parameter in 0 is formulated as:
1
ID|
O log p(uy) Xx; 2
(0)
Σ
D
00
We utilize the final vector F, which has the same dimensionality 0, as a source of knowledge to assess the importance of the corresponding model parameters. Each element of this vector can also be interpreted as the square of the gradient with re- spect to a model parameter, positioning our method within the realm of model-oriented approaches.
As the next step, we transform into a binary vector that represents the structure of the sub- network we intend to extract. Specifically, each component of the sub-network vector s given F, say s(0), is specified as 1 if (0) is in the top p% of all element values in F, 0 otherwise. In other
'As depicted in Figure the model parameters 0 do not include the embedding and aask-specific layers in our setting. For more technical details on the approximation process, we refer readers to Kirkpatrick et al. (2017).
FFor masked language modeling, we set the mask pertur- hation ratio as 0.15, making it identical as in pre-training.

---

of resource-rich languages, as well as the target language for which we aim to perform XLT, are provided. We then proceed by computing the sim- ilarity of the target language and each candidate language in the set, and sort the candidates based on the similarity scores in descending order.
Tusk
p(u|x;
Pearson
Spearman
Top 1
NDCG@3
T
70.67
52.94
33:33
77.42
NER
7
C
W
4
78.80
58.20
39.22
78.12
T
65. 76
29.44
POS
T
C
72.55
65.52
33.33
83.73
74.20
62.65
32.78
84.43
T
16.97
20.96
9.63
NLA
T
24.47
31.52
11.11
68.73
4 Experimental Settings
9.12
3.12
3.70
58.30
T.
18.41
16.16
19.05
73.02
PI
T
73.59
64.47
52.38
89.82
4.1 Tasks and Datasets
W
41.14
45,01
25.40
83.49
To assess the overall effectiveness of the proposed method across configurations, we employ five tasks from the XTREME benchmark (Hu et al., 2020):
T. 72.58
23. 23.86
33.33
75,80
QA
T
4
68.15
50.00
87.95
49.07
45,49
9.72
81.51
Table I: According to the selection for the corpus D and the output distribution p(y]x; 0), the proposed method suggests three variations of sub-networks. For D. there exist two options- T task data and W: text from Wikipedia. Meanwhile, p(y 0) also have two choices-T: output from task-oriented layers, which requires labeled data, and L: the distribution from (masked) language modeling. The combination D=T and p(y[x: =f= generally produces the best outcomes.
named entity recognition (NER), part-of-speech (POS) tagging, natural language inference (NLI), paraphrase identification (PI), and question answer- ing (QA). We explain the details of each task and the corresponding dataset in Appendix A.
4.2 Baselines
We consider three distinct categories of methods as baselines: linguistic (L.2V), statistical (LEX, SuE), and model-based (EMB). We refer readers to Ap- pendix B for more details on each approach.
4.3 Configurations and Metrics
We employ XLM-ROBERTa (Conneau al., 2020) as our base language model. As the proposed method operates on a small set of data instances, we randomly sample a subset from an available corpus. In order to mitigate potential bias arising from data selection, we execute the method three times with different random seeds and consider its averaged outcomes as its final result. Meanwhile, it is also necessary to compute gold-standard XLT scores, which serve the standard for assessing the power of prediction methods. To this end, for a given pair of source and target languages and the specific task at hand, we fine-tune the model three times using the source language data. We then evaluate the fine-tuned ones on the target language and calculate the average value, thereby alleviating the instability inherent in neural network training. Note that focus on the zero-shot XLTT setting.
Lang2Vec (L2V; Littell et al. (2017)) pro- vides diverse range of language representations, which are determined based on external linguistic databases.* 4 We utilize the "typological" vectors within the suite, encompassing syntax, phonology, and inventory features. The cosine similarity of two language vectors is regarded as an indicator that predicts the efficacy of XLT between the respective languages. On the other hand, Lexical Divergence (LEX) characterizes the lexical similarity between two languages. Inspired by Muller al. (2023), we specify this as the Jensen-Shannon Divergence between the subword uni-gram frequency distribu- tions of the source and target languages. Mean- while, Pelloni et al. (2022) propose the Subword Evenness (SuE) score that evaluates the evenness of word tokenization into subwords. The authors claim that the language with the lowest SuE score (i.e. one showing the most unevenness) is the best candidate for XLT. We compare their suggestion with that of our method. Finally, Muller et al. (2023) propose employing the cosine similarity be- tween two language vectors. In contrast to L2V, these vectors are obtained by taking the average of sentence embeddings computed by language mod- cls. We call this Embedding Similarity (EMB).
Four metrics are introduced for evaluation. The Pearson and Spearman correlation coefficients are used to gauge the relationship between the gold- standard scores and the predicted scores suggested by each method. We also utilize the Top I accuracy to evaluate the ability of each method in identi- fying the best source language for a given target language. Lastly, Normalized Discounted Cumula- tive Gain (NDCG; Järvelin and Kekäläänen (2002)) is leveraged to test the utility of the ranked list of
'c.g. The WALS datahase

---

Task (Dataset)
Method
Pearson
Spearman
Top 1 NDCG@3
Lang2Vec (L2V; Littell al. (2017))
14.58
17.73
13.73
62.35
Lexical Divergence (LEX; Muller et al. (2023))
67.55
53.92
23.53
76.20
NER (WikiANN)
Subword Evenness (SuE: Pellomi et al. (2022))
30.33
10.29
3.92
47.46
Embeeding Similarity (EMB; Muller al. (2023))
78.18
49.52
31.37
76.06
Ours (X-SNS)
78.80
58.20
39.22
78.12
Ven (1.2V; Littell al. (2017))
67.62
56.97
23.33
78.06
Lexical Divergence (LEX; Muller et al. (2023))
48.29
39.78
28.33
77.44
POS (UD)
20 Subword Evenness (SuR; Pelloni et al. (2022))
42.83
48.99
0.00
33.26
Embedding Similarity (EMB; Muller al. (2023))
70.12
51,81
16.67
74.65
Ours (X-SNS)
72.55
65.52
33.33
83.73
Veng2Vec (L2V; Littell et al. (2017))
10.24
12.80
13.33
59.77
Lexical Divergence (LEX: Muller et al. (2023))
37.73
9.73
6,67
60.19
NLI (XNLI)
15 Subword Evenness (Sul: Pelloni et al. (2022))
0.04
2.75
8.89
58.45
Embedding Similarity (EMB; Muller et al. (2023))
23.09
27.63
8.89
63.15
Ours (X-SNS)
24.47
31,52
11.11
68.73
Leng2Vec (L2V; Littell et al. (2017))
68.55
62.12
47.62
86.81
Lexical Divergence (LFX; Muller et al. (2023))
34.41
28.16 16
38. 10
77.11
PI (PAWS-X)
7 Subword Evenness (SuE; Pelloni et al. (2022))
20.41
28.26
14.29
60.37
Embedding Similarity (EMB; Muller et al. (2023))
47.12
48.92
23.81
83.51
Ours (X-SNS)
73.59
64.47
52.38
89.82
Veng2Vec (L2V; Littell et al. (2017))
66.27
54,76
62.50
84,52
Lexical Divergence (LEX; Muller et al. (2023))
18.64
29.46
20.83
72.14
QA (TyDiQA)
8 Subword Evenness (SuE: Pelloni et al. (2022))
39.77
36.46
0.00
73.21
Embedding Similarity (EMB; Muller et al. (2023))
64.80
65.18
20.83
86.00
Ours (X-SNS)
72.58
68.15
50.00
87.95
Table 2: Comparison with baselines that rank candidate languages hased on their (estimated) suitability the source. The # column indicates the size of the language pools, which consist of elements used as both the source and target. All the reported scores represent the average performance of each method across all the target languages considered. X-SNS consistently outperforms the baselines across various metrics and tasks, confirming its effectiveness.
languages presented by the evaluated method for selecting the proper source language. Since all the metrics are computed regarding one target language and a set of its candidate languages for XLT, we calculate the scores for every target language and average them across all tested languages, showing the overall effectiveness of each method. Scores are displayed in percentage for better visualization.
suspect this is partially due the fact that we de- cide to use random layers rather than trained ones for classification heads. Furthermore, considering that this direction necessitates annotated data for its implementation, we can infer that it is not advis- able to employ it for predicting the optimal source language. On the other hand, we achieve the best result when creating sub-networks through the ap- plication of (masked) language modeling based on raw text extracted from task-specific datasets, i.e., D=T and p(y) x; O)=L. This is encouraging in that it is more feasible to collect in-domain raw text than labeled data. 6 Given its cheap requirements and exceptional performance, we regard this option our default choice in the following sections. Lastly, conduct tests deriving sub-networks by ex- ploiting a general corpus, in effort expand the potential application scope of the proposed ap- proach (D=W and p(y) x; =)=C). Table reports that in terms of D, applying W mostly performs
5
Main Results
5.1 Performance by Sub-Network Types
We initially explore the efficacy of our method by evaluating its performance regarding the class of sub-networks utilizes. Results from all available options are listed in Table 1. Note that for W, we adopt the WikiANN (Pan et al., 2017) version of Wikipedia, making T=W for D on NER.
The construction of sub-networks relying on spec-ic labeled data (D=T and p(y|x; 0)=7) yields somewhat unsatisfactory performance. We
"In XLT setup, acquiring plain text in the target domain is readily feusible. Text the source language can be obtained from the training set, while text in the target language can he gathered directly from the evaluated data instances themselves.
WWe employ WikiANN, a processed version of Wikipedia, to reduce costs for preprocessing. We plan to apply our method to other corpora sets in future work.

---

Tusk
NER
POS
NLI
P1
QA
Approach/ Metrie
RMSB
Top
NDCGes
RMSE
Top
NDOGes
RMSE
TTpp
NDOGes
RMSE
Top NDDGGS
RMSE
Top NDOGe3
X-POS (muhi)
11.97
27.45
74.09
9.40
33.33
88.32
4.18
8.89
58.08
3,80
33.33
75.15
9,04
29.17
X-SNS (sing.) + OLS
8,88
39.22
78.12
9.58
33.33
83.73
3,95
11.11
68,73
4,98
$2.38
89.82
8,93
50,00
87.95
X-POS (multi) MER
7.18
27.45
77.10
4.71
33.33
88.21
1.69
6.67
61.00 00
0.35
53.97
90.98
7.40
37 50
77.77
X-SNS (sing.) MER
5.12
39.22
78.12
5.68
33.33
83.73
1.67
11.11
68.73
1.01
52.
89.82
5.80
50.00
87.95
Table 3: Evaluation in the framework of XLT regression. There are two regression techniques ordinary least square (OLS) and mixed effect regression (MER)- three metrics-RMSE (L). Top (1), and NDCG@1 @ (t). The results show that the single feature computed by X-SNS can substitute multiple linguistic features, verifying the integrity and impact of method.
worse than T, highlighting the importance of in- domain data, We leave investigation on the better use of general domain data as future work.
While X-SNS is originally designed as a ranking method, its outcomes can also be utilized as a fea- ture in the regression of XLT scores. We thus delve into the potential of X-SNS in the realm of XLT regression. As baselines, we consider using of the multiple features employed in X-POS (de Vries et al., 2022) that include information on language family, script, writing system type, subject-object-verb word order, etc. Following the previous work, we present the results of applying two regression techniques ordinary least squares (OLS) and mixed-effects regression (MER)- on the features. In our case, conduct regression relying solely on sub-network similarity. As evalu- ation indices, the root mean squared error (RMSE) is introduced in addition to ranking metrics (Top and NDCG@ 3). The points representing the XLT results of all possible pairs of source and target languages are regarded as inputs for the regression process. Table 3 show that X-SNS + OLS surpasses X-POS OLS in 4 tasks, with the exception of POS tagging for which the X-POS features are specifi- cally devised in prior research. This implies that the single feature provided by X-SNS is sufficient to replace the multiple features proposed in the previ- ous work, emphasizing the richness of information encapsulated in sub-network similarity. X-SNS + OLS also exhibits comparable performance with X-POS + MER, despite its algorithmic simplicity. Furthermore, the effectiveness of our method in score approximation, evidenced by the RMSE, can be enhanced by incorporating MER (X-SNS MER).
5.2 Comparison with Ranking Methods
We here compare our method against baselines that rank candidate languages based on their (estimated) suitability the source. The results are in Table 2. Overall, the proposed method demonstrates its superiority across diverse metrics and tasks, thereby confirming its effectiveness for reliably pre- dicting the success of XLT. For instance, it achieves an improvement of 4.6% on average in terms of NDCG@3, 3. On the other hand, L2V generally ex- hibits inferior performance, although it attains the highest Top 1 accuracy on NLI and QA. We sus- pect the high scores achieved in those cases partially attributed to the limited size of the lan- guage pools used for the tasks, making the tasks vulnerable to randomness. SuE also shows unsatis- factory performance. This is mainly because SuE lacks mechanism to reffect task-specific cues, re- sulting in over- gene-generalized suggestions which are presumably suboptimal. EMB closely resembles our method in terms of both its mechanism and per- formance. However, X-SNS provides more accu- rate predictions, and this gap becomes substantial when we assess the practical utility of the predic- tions, as confirmed in $6.4. In addition, our method opens up avenues for analyzing the structural ac- tivation patterns of language models, implying its potential utility for probing. Finally, note that all methods including ours perform poorly in the NLI task. We analyze this phenomenon in Appendix C.
6 Analysis
5.3 Comparison with Regression Methods
Previously, we focused on evaluating ranking meth- ods. Meanwhile, there another line of research that seeks to estimate precise XLTT scores, going beyond the scope of identifying suitable source lan- guages (de Vries et al., 2022; Muller et al., 2023).
We present extensive analyses helpful for deeper understanding of X-SNS. We explore some factors that can have an impact on its performance, such as the sub-network ratio p ($6.1), the base language model ($6.2), and the size of the corpus D (Ap-

---

16.
76.
NER
962
15.
X-5K2
816
PO5
12V
242
1EX
72.0
24.0
76.0
72.1
90.3
82.0
24.0
85.0
3.55 33
033
02
9.55
00
030
6A.
045
0.5
145
1.55
335 0.2
Averaag
Score
66372
Figure 4: Performance on NER and POS tasks in DMT settings, according to the selection of source languages following suggestions from each prediction (or ranking) method. The evaluation metric is the Fl score averaged over all target languages. X-SNS outperforms baselines and achieves exclusive improvement in the POS task.
Figure 3: Fluctuations in performance (in NDCG@3) when varying the value of p. For token-level tasks (NER, POS). moderate level of p is optimal, whereas the lowest value for p is the best for other semantic tasks.
Pearson
Spearman
Top
NDCG@3
FMB X-SNS
EMB
X-SNS
EMB
X-SNS
EMB
X-SNS
on the NER task, comparing the outcomes with those of EMB, model-based method similar X-SNS. The experimental results are illustrated Table 4, which show that X-SNS outperforms EMB when applied on mBERT. Meanwhile, X-SNS and EMB demonstrate comparable performance when evaluated on mT5. While experiments with mTS5 illuminate the potential of extending our approach to various types of pre-trained models, reserve deeper exploration in this direction for future work
mBERT 81.21
86.83
40.18
58.67
23.53
41.18
73.81
80.15
XLM-R 78.
78.00
52
58.20
31.37
39.22
76.06
78.12
mTS 88.77
68.06
56.14
43.65
25.49
29.41
76.41
76.72
Table 4: Comparison of the performance of X-SNS
and EMB when they are applied on various types of language models.
pendix D). Moreover, we assess the efficacy of the approach in low-resource scenarios ($6.3) and in XLT with multiple source languages ($6.4).
6.3 Evaluation on Low-Resuurce Languages
6.1 Grid Search on Hyperparameters
The advantage of XLT is particularly magnified in scenarios where there are constraints on data col- lection for the target task and language. So far, we have utilized the same language pool for both the source and target languages to facilitate evaluation. However, targets for XLT often in practice encom- pass languages that were not the pre-training phase of base models. To validate our method in such scenarios, we perform an exper- iment on NER with low-resource languages, with the number data instances for evaluation ranging from 1000 to 5000. The selection consists of 15 languages, three of which are not present in XLM- R pre-training. We observe that in 11 out of 15 cases, X-SNS outperforms using English, achiev- ing average F1 score improvement of 1.8 points. More details this experiment are in Appendix E,
In this study, we intentionally fix the value of p as 0.15, following the mask perturbation ratio lan- guage modeling, to minimize an effort in hyperpa- rameter search. We examine how much robust this decision is, checking fluctuations in performance while varying the value of p. Figure reveals that the prediction of X-SNS is optimized when utiliz- ing low values of for high-level tasks (PI, NLI, QA), while moderate values of p are more effective for low-level tasks. However, the gap between the best and worst performance of the method is not substantial, except for the case of QA.
participated in
6.2 Impact of Base Language Models
While main experiments were concentrated on test- ing methods with XIMMRRoEETT,, our approach can be applicd on other multilingual models such al., 2019) and IXT5 al., 2021). To validate the robustness of our ap- proach, we replicate our experiments conducted on XLM-R, replacing the language model with mBERT and mT5. We evaluate these variations
6.4 XLT with Multiple Source Languages
(Devii
A prior study (Singh et al., 2019) reported that in XLT, incorporating multiple source languages during training can enhance performance, refer- ring to this setup as "disjoint multilingual train-

---

ing (DMT)" settings. In Figure 4, our findings also reveal that BEST-3 (XLT based on mixture of the top 3 source languages) surpasses BEST-1 (trained on the single best source language), indi- cating that the number of source languages used for training in XLT can fairly impact its performance. However, the vast number of exponentially pos- sible combinations in selecting source languages poses challenge in searching the optimal one for DMT settings. In this condition, source language prediction (or ranking) methods including X-SNS can come as a rescue to the problem, providing plausible options to be selected. We therefore con- duct an experiment on testing the ability of ranking methods in DMT settings. Concretely, we employ the combination of the top 3 source languages as suggested by each method to refine base model through fine-tuning. In Figure 4, we confirm that X-SNS yields the best performance. Note that the amalgamation of the source languages proposed by X-SNS exhibits exclusive enhancements in the POS task when compared to BEST-1 (the blue ver- tical line), implying that it is the only option that proves to be truly beneficial in that configuration.
serves as notable example of implementing this framework. Consequently, we anticipate that re- searchers within the community will collaborate to enhance the framework in the near future by devel- oping more effective methodologies for extracting sub-networks from a base model.
Limited availability of multilingual evaluation datasets Despite our diligent efforts to evaluate our approach across multiple configurations, in- cluding five evaluation tasks, it is crucial to ac- knowledge the scarcity of multilingual datasets that can effectively test the practical efficacy of cross-lingual transfer across the vast arra of over hundred languages. As result of this limitation, were constrained to focus our analysis only few selected tasks, such as NER and POS tagging. We advocate for the development of additional mul- tilingual datasets that encompass a broad spectrum of languages. Such comprehensive datasets would greatly facilitate the evaluation of methods for XLT, including our own approach.
Relatively heavy computational cost Regarding computational resources, our method necessitates relatively higher storage capacity to accommodate gradient information alongside model parameters. Nonetheless, this issue can be largely alleviated by reducing the amount of data required for the algorithm's execution., e.g.. |D| 256 1024. Furthermore, our algorithm eliminates the need for directly tuning a variety of models for testing the success of XLT, making it eco-friendly.
7 Conclusion
We present X-SNS, method that efficiently pre- diets suitable source languages for XLT with min- imal requirements. We identify that XLT perfor- mance between two languages has a strong correla- tion with the similarity of sub-networks for those languages, and propose a way of exploiting this fact in forecasting the success of XLT. X-SNS demon- strates exceptional versatility and robustness, ver- ifying impressive performance across various en- vironments and tasks. We believe that our sugges- tions and findings have the potential to contribute the exploration of the inner workings of multi- lingual language models, leaving as future work.
Ethics Statement
In this work, we perform experiments using a suite of widely recognized datasets from the existing litcrature. As far as our knowledge extends, it is highly unlikely for ethical concerns to arise in rela- tion to these datasets. However, since the proposed method has the potential to be applied to address new datasets in low-resource languages, it is crucial to approach data collection with careful considera- tion from an ethical standpoint in those cases.
Limitations
Need for exploration on sub-network construc- tion strategies While our proposal focuses on leveraging Fisher Information as a source of infor- mation for constructing sub-networks, it is impor- tant to note that there exist other alternatives that can be utilized to achieve the same objective. We argue our main contribution lies in the fact that we introduce the framework of using sub-networks for predicting cross lingull transfer (XLT) perfor- mance and that the utilization of Fisher Information
Acknovledgements
This work was supported by Hyundai Motor Company and Kia. This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea goverMmen((MSIT) (No.2020- 0-01373, Artificial Intelligence Graduate School

---

Program (Hanyang University)). This work was supported by the National Research Foundation of Korea(NRFF grant funded by the Korea govern- ment(MSIT) (No.2022RIFI.) FFA107777))
Negar Foroutan, Mohammadreza Banaei, Rémi Lebret. Antoine Bosselut, and Karl Aberer. 2022. Discov- ering language-neutral sub-networks in multilingual language models. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing.
References
Jonathan Frankle and Michael Carbin. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.
Alessandro Achille, Giovanni Paolini, and Stefano Soatto. 2019. Where is the information in a deep neural network? arXiv preprint arXiv:1905.12211
Song Han, Jeff Pool, John Tran, and William Dally. 2015. Learning both weights and connections for efficient neural network. Advances in neural infor- mation processing systems.
Alan Ansell, E. Ponti, Anna Korhonen, and Ivan Vulic. 2021. Composable sparse fine-tuning for cross- lingual transfer. In Annual Meeting of the Associ- ation for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task bench- mark for evaluating cross-lingual generalisation. In International Conference on Machine Learning.
Steven Cao. Nikita Kitaev, and Dan Klein. 2020. Multi- lingual alignment of contextual word representations. In International Conference on Learning Representa- tions.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A bench- mark for coomaation-seeiing question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics.
Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumu- lated sain-based evaluation of ir techniques. ACM Trans. Inf. Syst.
Kaliyaperumal Karthikeyan, Zihan Wang. Stephen May- hew, and Dan Roth. 2020. Cross-lingual ability of multilingual bert: An empirical study. In Interna- tional Conference on Learning Representations.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale, Pro- ceedings of the 58th Annual Meeting of the Associa- rion for Computational Linguistics.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz. Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Ctrabska-Barwinge ing catastrophic forgetting in neural networks. Pro- ceedings of the national academy of sciences.
et al. 2017. Overcom-
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams. Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference Empirical Methods in Natu- ral Language Processing.
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with mul- tilingual Transformers. In Proceedings of the 2020 Conference Empirical Methods in Natural Lan- guage Processing.
Wietse de Vries, Martijn Wieling, and Malvina Nissim. 2022. Make the best of cross-lingual transfer: Ev- idence from POS tagging with over 100 languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics.
Xi Victoria Lin, Todor Mihaylov. Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' O'Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techmologies.
Yu-Hsiang Lin, Chian- Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junx- ian He, Zhisong Zhang, Xuezhe Ma, Antonios Anas- tasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learn- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Ronald A Fisher. 1922. On the mathematical founda- tions of theoretical statistics. Philosophical transac- tions of the Royal Society of London. Series A, con- taining papers of a mahematical or physical charac- ter.

---

Patrick Littell, David R. Mortensen, Ke Lin. Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceed- ings of the 15th Conference of the European Chapter of the Association for Compatational Linguisties.
International Joint Conference on Natural Language Processing).
Mengzhou Xia, Antonios Anastasopouloss, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020. Predicting performance for natural language processing tasks, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. 2019. Rethinking the value of network pruning. In International Conference on Learning Representations.
Runxin Xu, Fuli Luo, Zhiyuan Zhang, Chuanqi Tan, Baobao Chang. Songfang Huang. and Fei Huang. 2021. Raise child in large language model: To- wards effective and generalizable fine-tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Benjamin Muller, Deepanshu Gupta, Jean-Phl Philippe Fau- connier. Siddharth Patwardhan, David Vandyke, and Sachin Agarwal. 2023. Languages you know influ- ence those you learn: Impact of language character- istics on multi-lingual text-to-text transfer. In Pro- ceedings of The 1st Transfer Learning for Natural Language Processing Workshop.
Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021, m55: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Knight, and Heng 2017. Cross-lingual name tagging and linking for 282 languages. Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: cross-lingual adversar- dataset for paraphrase identification. Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing.
Olga Pelloni, Anastassia Shaitarova, and Tanja Samardzic. 2022. Subword evenness (SuE) as a pre- dictor of cross-lingual transfer to low-resource lan- guages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics.
Daniel Zeman, Joakim Nivre, et al. 2021. Universal dependencies 1.
Sebastian Ruder, Anders Søgaard, and Ivan Vulić. 2019. Unsupervised cross- lonsulln representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Jasdeep Singh, Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2019. Xlda: Cross-lingual data augmentation for natural language inference and question answering. arXiv preprint arXiv: 11471.
Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Train- ing neural networks with fixed sparse masks. In Advances in Newral Information Processing Systems.
Ming Tu, Visar Berisha, Yu Cao, and Jae-Sun Seo. 2016. Reducing the model order of deep neural networks using information theory. 2016 JEEE Computer Society Annual Symposium on VLSI.
Julia Turc, Kenton Lec, Jacob Eisenstein, Ming-Wei Wei Chang, and Kristina Toutanova. 2021. Revisiting the primacy of english zero-shot Pross-lnngual transfer. CoRR.
Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of BERT. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th

---

Appendix
mine whether a given pair of sentences is a para- phrase of one another. We use the PAWS-X dataset (Yang et al., 2019), spanning 7 languages: Chinese (ZH), English (EN), French (FR), German (DE), Japanese (JA), Korean (KO), and Spanish (ES). it contains 50K muuchine-translatee instances in the training set, as well as 2K humanntranslated ones in the test set. Accuracy is used for evaluation.
A Details on Tasks and Datasets
Named Entity Recognition (NER) is task that in- volves the classification of pre-defined entity types for each name mention within sentence. We em- ploy WikiANN (Pan et al., 2017) for evaluation on named entity recognition. This dataset covers 282 languages existing in Wikipedia, but we curtail the number of languages to be considered as a source to 17 based on two criteria: (1) the selected lan- guage should contain over 20K data instances, and (2) it was evaluated in the previous study (Lauscher al., 2020). The final list of the candidate lan- guages as follows: Arabic (AR), Chinese (ZH), English (EN), Finnish (FD), French (FR), Greek (EL), Hebrew (HE), Indonesian (ID), Italian (IT), Japanese (JA), Korean (KO), Russian (RU), Span- ish (ES), Swedish (SV), Thai (TH), Turkish (TR). and Vietnamese (VI). The micro F1 score is used as metric.
Question Answering (QA), more specifically, reading comprehension aims to retrieve a continu- ous span of characters in the context that answers the given question. We utilize TyDiQA (Clark et al., 2020) which provides a non-translated dataset that consists of typologically diverse 9 languages: Ara- bic (AR), Bengali (BG), English (EN), Finnish (FI), Indonesian (ID), Korean (KO), Russian (RU), Swahili (SW), Telugu (TE). To train the base model, we split the original training set into 80% and 20% for training and validation respectively. For stable evaluation, we exclude Korean which has the small- est number of training data. The Fl score is used as a metric.
Part-of-Speech Tagging (POS) is a sequence- labeling task that assigns the most appropriate part- of-speech tag to each word in a sentence. For this task, we adopt a variation of Universal Dependen- cics 2.8 (Zeman et al., 2021) processed by de Vries al. (2022). We consider 20 languages candi- dates for transfer. whose number of training data is over 4K: Bulgarian (BG), Chinese (ZH), Dutch (NL), English (EN), Finnish (FI), French (FR), Hindi (HI), Indonesian (ID), Italian (IT), Japanese (JA), Korean (KO), Norwegian (NO), Polish (PL),
B Detailed Specification on Baselines
Lang2Vec (L2V) We incorporate all features be- longing to the "typological" category in the L2V utility: syntactic, phonological, and inventory vec- tors with the KNN predicted version to fill in miss- ing elements in the vectors. We concatenate all these vectors to express the typological character- isties of language as single vector. We then calculate a cosine similarity between two represen- tative vectors of each source and target language. and use as predictor for the efficacy of XLT.
Portuguese (PT), Russian (RU), Slovakia (SK),
Spanish (ES), Swedish (SV), Turkish (TR), and Ukrainian (UK). We use micro F1 for evaluation.
Lexical Divergence (LEX) Following Muller et al. (2023), we define the lexical divergence be- tween source and target languages as the Jensen- Shannon Divergence (JSD) of the sub-word uni- gram distributions of each language:
Natural Language Inference (NLI) aims to clas- sify the entailment relationship between two pieces of text. We choose the XNLI (Conneau et al., 2018) dataset for use, covering 15 languages: Ara- bic (AR), Bulgarian (BG), Chinese (ZH), English (EN), French (FR), Greek (EL), German (DE), Hindi (HT), Russian (RU), Spanish (ES), Swahili (SW), Thai (TH), Turkish (TR), Urdu (UR), and Vietnamese (VI). It comprises 390K instances of mransae-translated training data for each language, along with 5K instances of human-translated test data per language. For evaluation, accuracy is used,
LEX (s.t) (3,t) ISD(D,, Di),
where Dj corresponds to the sub-wort uni-grams distribution of the language I.
Subword Evenness (SuE) As another instance of statistical approaches, we consider the Suli score proposed by Pelloni et al. (2022). We first construct a source language corpus for each task utilizing its training data. And then we derive the SuE score
Paraphrase Identification (PI) intends to deter-

---

per language (s) and task follows:
SuE, = 180 arctan 1| aarttan ki), where arctan 1| and arctan k| are the lower an- gles of the triangle that describes the distribution of points that correspond to each word in corpus, whose x-axis is about the length of the word and y-axis represents unevenness scores. For more detailed explanation this algorithm, recom- mend readers to refer Pelloni et (2022).
-10
20
-40
550
-00
NER
POS
NU
QA
Embedding Similarity (EMB) Following Muller et al. (2023), we calculate the cosine similarity between embedding vectors of the source and target language as follows:
Figure 5: Visualization of the distributions of perfor- mance differences (P, P.) caused by the selection of source languages. Ps corresponds to XLT perfor- mance when fine-tuned on the source language a while Po is the task performance shown when the language model is task-specifically trained on the target language itself. Therefore, strate the expected amounts of performance drop when we substitute the target language with each candidate source language. Note that there exist even positive gaps (/', P. > 0) in the NLI task, implying that the XNLI dataset that represents the NLI task might be not suitable for reliable evaluation of performance in XLT settings.
EMB (8,1)
ee ||l e,|l
distributions P. P. demon-
Note that Css Ct are embedding vectors represent- ing the source and target languages respectively. To derive e, we compute a mean-pooled vector by averaging token embeddings from the last layer of language model. We then compute by tak- ing an average of all those mean vectors derived from 1024 data examples. We choose 1024 as the number of examples for EMB to guarantee fair comparison with our method.
NER
POS
P1
C Performance Gap in XLT across Tasks
0.90
0.88
This section explains the reasons why experimental results on ILI were noisy for all considered meth- ods. First, it turns out that predicting appropriate source languages for both NLI and Pl tasks is tricky because the variations of XLT scores across differ- ent source languages those tasks are minimal, as shown in Figure 5. Secondly, in the context of NLI, encounter instances where the performance of XLT surpasses that of conventional fine-tuning on the target language itself, raising questions about the suitability of the XNLI dataset as a reliable benchmark for evaluating the effectiveness of XLT.
0.86
0.84
0.82
0.60
128
256
10000
Number of Data Instance for X-SNS
Figure The NDCG@3 scores plotted based on the number of data instances (ID) used to construct sub- network in the condition of D=T and p(y|x; =f= The vertical orange line indicates the number data (1024) applied for our main experiments in this paper.
Impact of [ on Performance
this part, we examine the fluctuations in perfor-
mance observed within our method as we modify the value of D]_ We sample the size of data exam- ples in the set [64, 128, 256, 512, 1024, 10000). As depicted in Figure 6, the NDCG@3 scores con- verges when utilizing more than 1024 examples, justifying our decision of using this number in our main experiments. Note also that the data size of nearly 256 can be a reasonable choice for efficiency without compromising performance.
E Evaluation Low-Resouree Languages
In experiment, utilize set of 17 resource- rich languages from the WikiANN dataset source languages. For target languages, con- sider 15 low-resource languages: Afrikaans (AF), Breton (BR), Kurdish (CKB), Western Frisian (FY), Irish (GA), Hindi (HI), Icelandic (IS), Kazakh

---

WORST
0.9
Zero-shat, XLT Scores for NER [F1)
0.8
0.7
0.6
0.5-
0.4
0.3
AF BR CKB PY
GA
15
KK
L3
50 SW
TE
TT
Uz
Low-resource Target Language
Figure 7: This figure illustrates that zero-shot XLT performance for low-resource target languages on NER can vary depending on choice of source languages. Our method outperforms using English in 11 out of 15 cases. The horizontal lines are the average score for each case.
(KK), Luxembourgish (LB), Mauritania (MR), Al- banian (SQ), Swahili (SW), Telugu (TE), Tatar (TT), Uzbek (UZ). These languages selected from the same dataset whose number of data (for evaluation) ranges from 1000 5000. We choose the pseudo-optimal source language by identifying the one that exhibits the highest sub-network sim- ilarity for each target language. From the results in Figure 7, we find out that we can obtain better results when conducting XLT with the source lan- guages recommended by our method, compared to the case of using English the source language, resulting in average improvement of 1.8 points.


Reviews:
[{"review_id": "hGczFHBcpf", "reviewer": "Reviewer_39c7", "summary": "", "questions": "1. Values missed in Table 1?\n", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for taking your valuable time to give constructive and fruitful opinions on our work.\n\nWe appreciate the opportunity to address your questions, clarify any misconceptions, and alleviate your concerns. \n\nBy doing so, we hope to foster a constructive discussion on the suggestions you've provided to enhance our paper.\n\n---\n\n**1. Missing MER results for X-SNS in Table 3**\n\n---\nThanks for pointing this out. \n\nIn the draft, we reported only the performance of the combination of X-SNS and OLS (Ordinary Least Square), not considering X-SNS + MER (Mixed Effect Regression).\n\nOur decision was grounded on two facts: (1) X-SNS + OLS already achieved performance comparable to baselines; and (2) we thought the cost of applying MER (it is much more expensive than that of OLS) outweighs the benefits, hurting the efficiency of our algorithm whose major goal is to predict transfer scores in an efficient manner.\n\nNonetheless, we admit that it is also desired to evaluate the upper bound of X-SNS’s performance by combining it with MER. \n\nTo this end, we have conducted an additional experiment to verify the effectiveness of X-SNS in the suggested setting, whose results (metric: RMSE; lower is better) are listed in the table below.\n\nOur results confirm that X-SNS + MER significantly outperforms X-SNS + OLS. \n\nFurthermore, X-SNS + MER surpasses X-POS + MER in 3 out of 5 tasks. \n\nWe will include this outcome in the next version of our draft to emphasize the competence of our approach.\n\nWe thank again Reviewer 1 for proposing this additional experiment.\n\n| Method (Metric: RMSE)  | NER   | POS  | NLI  | CLS  | QA   |\n|----------|-------|------|------|------|------|\n| XPOS+OLS | 11.97 | 9.40 | 4.18 | 3.80 | 9.04 |\n| XPOS+MER | 7.18  | 4.71 | 1.69 | 0.85 | 7.40 |\n| XSNS+OLS | 8.88  | 9.58 | 3.95 | 4.98 | 8.93 |\n| XSNS+MER (newly added) | 5.12  | 5.68 | 1.67 | 1.01 | 5.80 |\n\n---\n  \n**2. The performance of X-SNS (for NER and POS) seems changing greatly as the p-value change. Is there any report about X-SNS’s performance with diverse p-values? An efficient selection method for p-value might be needed because searching for p-value can become a major cost of applying X-SNS.**\n\n---\nAs we mentioned in the paper, we intentionally fixed the value of $p$ as 0.15, following the mask perturbation ratio in language modeling, to minimize an effort in hyperparameter search.\n\nWe first would like to highlight the fact that although we did not take much care about hyperparameter optimization, X-SNS outperformed competitive baselines.\n\nIn fact, as illustrated in Figure 3, our choice of 0.15 for $p$ was even the worst for the NER task.\n\nNevertheless, X-SNS won over all the baselines considered in our work for the task, which we believe is a promising result.\n\nMoreover, in Section 6.1, we allocated a dedicated section for examining the impact of hyperparameters.\n\nIn the subsection, we reported that the performance of X-SNS (metric: NDCG@3) remains largely robust concerning the choice of the $p$-value.\n\nFor more explanation, let us introduce another table below. \n\nThis table represents the gaps between the upper and lower bounds of NDCG@3 (in the first row), in addition to the gaps between the cross-lingual transfer scores of the source languages (in the second row), while varying the $p$-value.\n\nFrom the table, we have discovered that our method is generally robust to the selection of $p$, with exceptions in the NLI and QA tasks in terms of NDCG@3 (in the first row). \n\nHowever, we’d like to argue that these exceptions can be further mitigated when the performance is evaluated with respect to the transfer scores of the chosen source languages, as we can see in the second row of the table.\n\nAs the main objective of our approach is to select the most appropriate **single** source language (which is expected to record the highest transfer score) rather than choosing 3 candidates (as measured in NDCG@3), we believe that the setting for the second row is more practical and that the differences shown in the second row are acceptable enough.\n\nWe will report this extra investigation in the revision, along with a discussion on the positive and negative impacts of hyperparameter search for our method.\n\n| Max-Min | NER  | POS  | NLI   | CLS  | QA    |\n|---------|------|------|-------|------|-------|\n| NDCG@3  | 3.62 | 3.09 | 10.20 | 1.88 | 13.89 |\n| Transfer Score | 2.27 | 0.47 | 1.44  | 0.72 | 3.81  |\n\n---\n  \n**3. Lack of details about applying wiki data as a corpus.**\n\n---\n\nOn Line 408-410 (in Section 5.1), we already mentioned that we utilized the WikiANN version of Wikipedia. \n\nHowever, let us specify this fact more clearly on Section 4.3 (Configurations and Metrics) in the next revision.\n\nOn the other hand, the reason why we adopted WikiANN as a source of raw text is that it is used for constructing one of our tasks, i.e., NER.\n\nTherefore, by adopting WikiANN (only raw text, without labels for language modeling), we attempted to circumvent the cost needed for preprocessing a large amount of Wikipedia data.\n\nIn the revision, we will conduct additional experiments to evaluate the effect of such a corpus.\n\n---\n  \n**4. X-SNS seems to work poorly in mT5 in Person and Spearman metrics, this requires explanation because if X-SNS works poorly in generation-based models, its application will become much shallower. And I believe more experiments for generation-based backbones may be better.**\n\n---\n\nFirst of all, we would like to highlight that we mainly focused on evaluating our method with pre-trained encoder models, following the wisdom of the literature that cross-lingual transfer is generally more feasible for the tasks of natural language understanding, rather than generation.\n\nIn this sense, relying on encoder models was a reasonable choice, considering that most of the tasks we evaluated can be classified as NLU tasks.\n\nAn extension of the paradigm of cross-lingual transfer to generation tasks, with the aid of encoder-decoder and decoder models, is actually out of the scope of this work, and in fact, we are preparing to initiate research on this topic as a following study of this submission.\n\nNonetheless, in the draft, we attempted to test our approach with encoder-decoder models, such as mT5, as a preliminary step of such an extension.\n\nWe admit that it is probable that mT5 is not the best option for the problem setting discussed in this work.\n\nTherefore, we have a plan to conduct extra work that contributes to an extended application of our approach to decoder models (maybe for generation), as a separate future work.\n\n---\n\n**5. Values missed in Table 1?**\n\n---\n\nAs answered in Q3, we used the raw text part of WikiANN as a Wikipedia dataset. \n\nAs it is also a dataset for NER at the same time (when its labels are used together), the datasets for the two different settings (task-specific vs. language modeling) become identical, and that’s why we reported only one set of values to remove repetition. \n\nWe mentioned this fact on **Line 410** in the draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The motivation for using sub-network similarity is reasonable. The evaluations are comprehensive. The results are decent.", "reasons_to_reject": "1. missing results MER in X-SNS and testing its effectiveness?\n2. The p-value in NER and POS change greatly as the p-value change. So maybe present some results of X-SNS with diverse p-values in a table for analysis? An efficient selection method for p-value is needed because searching for p-value mat becomes a major cost of applying X-SNS.\n3. Lack of details about applying wiki data as a corpus.\n4. X-SNS seems to work poorly in mT5 in Person and Spearman metrics, this requires explanation because if X-SNS works poorly in generation-based models, its application will become much shallower. And I believe more experiments for generation-based backbones may be better.", "questions_to_authors": "1. Values missed in Table 1?\n", "paper_topic_and_main_contributions": "This work proposes a method to utilize sub-network similarity between two languages as a proxy for predicting the compatibility of the languages in cross-lingual transfer.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7MMPTqjKKY", "reviewer": "Reviewer_TZ8W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your effort in writing a careful review for our paper, which we believe would be a cornerstone for improving the quality of our submission.\n\nYou have pointed out some positive points of our work: (1) the paper is well-written and easy to follow, (2) proposed an interesting idea that is relevant to the NLP community, (3) conducted experiments on five datasets, which are extensive enough to demonstrate the utility of the proposed method.\n\nOn top of that, we would like to relieve your remaining concerns about our work.\n\n---\n\n**1. It must be clarified if the proposed method holds across a wide range of tasks.**\n\n---\n\nOn the journey of doing this research, we tried our best to evaluate our method on as many tasks as possible.\n\nWe believe that this is one of the reasons why you mentioned that our experimental settings are extensive.\n\nIn addition, we would like to inform that unfortunately, there exist a limited number of datasets suitable for testing the ability of cross-lingual transfer.\n\nProbably due to this reason, previous studies mostly considered only a subset of the tasks we adopted for evaluation. For instance, Pelloni et al. (2022)[1] tested their method only on a single masked language modeling task, de Vries et al. (2022)[2] handled only one POS task, and Muller et al. (2023)[3] conducted experiments with three tasks (NER, NLI, and QA). \n\nGiven this background, we can argue with confidence that our work definitely covered a wide range of tasks, validating the extensive effectiveness of our approach.\n\n>**Reference** \n>\n> [1] [Subword Evenness (SuE) as a Predictor of Cross-lingual Transfer to Low-resource Languages](https://aclanthology.org/2022.emnlp-main.503) (Pelloni et al., EMNLP 2022)\n>\n> [2] [Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages](https://aclanthology.org/2022.acl-long.529) (de Vries et al., ACL 2022)\n>\n> [3] [Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer](https://proceedings.mlr.press/v203/muller23a.html) (Muller et al., PMLR 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Strengths:\n\nThe paper is well-written and easy to follow.\n\nAn interesting idea that is relevant to the NLP community.\n\nConducted experiments on five datasets.\n\nExtensive experiments show the utility of the proposed method.\n", "reasons_to_reject": "\nIt must be clarified if the proposed method holds across a wide range of tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors evaluated the impact of sub-network similarity on cross-lingual transfer prediction.\nThe authors present their results for five tasks (NER, POS, NLI, PI, and QA) from the XTREME benchmark.\nThis paper has a clear contribution to cross-lingual prediction and will undoubtedly impact future research.  \nThis paper conducts a range of experiments studying the proposed objectives. \n\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "References are adequate\n"}, "extra_text": {}}, {"review_id": "QNPbAUzTs4", "reviewer": "Reviewer_uSgC", "summary": "", "questions": "\nA.Line 18, what do ‘external resources’ refer to?\n\nB.In Figure 2, Is Step 3 (top p%) necessary? Is it possible to obtain the similarity based on the outputs of Step 2?\n\nC.Line 275-279, does previous paper for choosing source languages require ‘the need for extra training of base models’, Please give the citation of previous work. Does ‘doing so’ refer to ‘considers the relative significance’?\n\nD.Line 287, why is the general domain set as Wikipedia instead of CC-100? CC-100 is the corpus used for training XLM-R.\n\nE.Line 304, what is random classifier?\n\nF.Line 340-345, lang2vec provides six metrics for measuring linguistics similarity across languages. Do this paper use the ‘feature’ one? If yes, please also compare with other 5 separate metrics. Lin et al. 2017 show that ‘feature’ is not best one for predicting performance of cross-lingual transfer.\n\nG.Table 2, 3, why are different baselines used?\n\nH.Line 544-548, can X-SNS deal with the case that combining a new language might worse the results?\n\nI.Line 550, while considering 3 source languages, are there the same amount of data used for fine-tuning as considering single source language?\n\nJ.Are the conclusions the same if X-SNS is applied in some other multilingual language models, such as large language models, e.g., Bloom, and speech-based models, e.g., XLS-R?\n\nK.Analysis on fail cases can show the shortages of X-SNS and provide insights for reaching the upperbound (best one).\n\nL.Is the fine-tuning on the source language conducted in the whole multilingual model or the shared sub-network? Analysis on how the sub-network change during fine-tuning can better explain the success/fail cases of X-SNS.\n\nM.Please provide the time cost of X-SNS.\n\nN.How is the source language set defined? \n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the dedication and effort you've invested in grasping the nuances of our work. \n\nIn the following, let us address your questions. \n\nThrough this process, we anticipate further improving the quality of our work.\n\n---\n\n**A. Line 18, what do ‘external resources’ refer to?**\n\n---\n\nThe term ‘external resources’ refers to linguistic databases employed by previous work, such as the ASJP database in de Vries et al. (2022)[1] and the WALS database in Muller et al. (2023)[2].\n\nIn our revision, we will elucidate this point, emphasizing that our work depends on plain text instead of pre-defined features, which may not always be available for every language.\n\n---\n\n**B. In Figure 2, Is Step 3 (top p%) necessary? Is it possible to obtain the similarity based on the outputs of Step 2?**\n\n---\n\nYour suggestion is possible in principle and also reasonable.\n\nTherefore, in the next revision, let us consider conducting extra experiments on the proposed variant (removing Step 3).\n\nHowever, we would like to highlight that Step 3 is vital for keeping our language vectors as simple as possible (i.e., binary masks or subtrees).\n\nThis is desired because the language vectors would be frequently used for computing the similarity between every possible language pair, causing computational burdens. \n\nIf we manage to use binary features, we can readily rely on bitwise AND and OR operations, which are more efficient than doing the dot products of real-valued vectors.\n\nThat’s the main reason why we decided to introduce Step 3.\n\nOn the other hand, in terms of controlling the $p$-value, we presented an analysis on this topic in Section 6.1. \n\nPlease take a look at the section.\n\n---\n\n**C. Line 275-279, does previous paper for choosing source languages require ‘the need for extra training of base models’, Please give the citation of previous work. Does ‘doing so’ refer to ‘considers the relative significance’?**\n\n---\n\nThe phrase \"doing so\" refers to the entire process for calculating sub-networks.\n\nLet us revise the writing in that part if our intention was not clearly delivered.\n\nIn the paragraph, we aimed to emphasize that in the process of constructing sub-networks,  we only compute gradients and do not actually update the weights of the model, unlike previous research (Xu et al. 2021)[3].\n\nWe will append appropriate citations to the proper location.\n\n---\n\n**D. Line 287, why is the general domain set as Wikipedia instead of CC-100? CC-100 is the corpus used for training XLM-R.**\n\n---\n\nWikipedia is frequently used as a base for pre-training numerous language models.\n\nFurthermore, it stands as a pivotal resource in building datasets for a wide range of tasks.\n\nThis underscores our rationale for choosing Wikipedia to represent the general domain.\n\nIndeed, we recognize the merit of your suggestion. \n\nIn our subsequent revision, we will contemplate evaluating our approach using CC-100 in lieu of Wikipedia.\n\nThanks again for your comment.\n\n---\n\n**E. Line 304, what is random classifier?**\n\n---\n\nAs we mentioned in Section 3, we need a classifier to calculate a task-specific gradient in the case of $p(y|x;\\theta) = T$.\n\nFor this purpose, we add a randomly initialized classifier on top of the pre-trained model and compute the gradient. \n\nHowever, when $p(y|x;\\theta)=W$, the task that contributes to computing the gradient is an MLM task, so a pre-trained LM head is used instead of the random classifier.\n\n---\n\n**F. Line 340-345, lang2vec provides six metrics for measuring linguistics similarity across languages. Do this paper use the ‘feature’ one? If yes, please also compare with other 5 separate metrics. Lin et al. 2017 show that ‘feature’ is not best one for predicting performance of cross-lingual transfer.**\n\n---\n\nTo the best of our knowledge, lang2vec offers six ‘metrics’ (including what you referred to as the 'feature'), and all of these are distance-based.\n\nIn our preliminary experiments, we discovered that instead of directly leveraging the distance-based ‘metrics’, it is more competitive for the baseline to manually compute cosine similarities based on the ‘feature vectors’ also provided by lang2vec.\n\nTherefore, we decided to adopt the latter approach (manual computation of similarity), as it guarantees the improved performance of the baseline (lang2vec).\n\n---\n\n**G. Table 2, 3, why are different baselines used?**\n\n---\n\nFor the comparative analysis of our research, we categorized baselines into two groups following the literature.\n\nSpecifically, the first group concentrates on recommending languages anticipated to transfer effectively, while the second group directly predicts the precise transfer scores calculated when the transfer is actually executed.\n\nTable 2 showcases the performance of approaches from the first group. The primary objective of these approaches is to provide features that predict the compatibility between the source and target languages.\n\nConversely, Table 3 lists the performance of methods from the second group. These methods utilize a regression approach to predict the actual transfer score for the source language.\n\nIn summary, our aim was to compare baselines within their respective groups, which is why we presented the results for each group in distinct tables.\n\n---\n\n**H. Line 544-548, can X-SNS deal with the case that combining a new language might worsen the results?**\n\n---\n\nAs you highlighted, increasing the number of source languages doesn't necessarily yield positive outcomes.\n\nStill, we assert that our method has the capability to discerningly select languages that reduce potential negative effects, highlighting its advantage over other baselines.\n\nTo substantiate this claim with tangible results, we outline below the frequency of instances where the 3-language-tuning setting, as chosen by each method, leads to a decline in performance relative to that of Best-1.\n\n|         | X-SNS | EMB | L2V | SuE | LEX |\n|---------|:-------:|:-----:|:-----:|:-----:|:-----:|\n| NER(17) | 2     | 3   | 4   | 11  | 16  |\n| POS(20) | 5     | 9   | 9   | 18  | 18  |\n\n`Table` The number in parentheses next to the task means the number of target languages, and the elements in the table correspond to the number of cases causing negative effects.\n\nWe will incorporate the findings from this additional experiment to further bolster our claim that our method remains effective, even under extreme conditions.\n\nThank you for suggesting we consider this dimension.\n\n---\n\n**I. Line 550, while considering 3 source languages, are there the same amount of data used for fine-tuning as considering single source language?**\n\n---\n\nYes, to clarify:\n\nWhen using multiple source languages (for instance, 3), we allocate an equal amount of data ($|D|$) for each language. \n\nThis results in a total data size of $3 \\times |D|$ for the combined training.\n\nLet's assume the number of epochs used for the multi-language training is $e$.\n\nNow, when calculating the Best-1 score using a single language, we train with data of size $|D|$ but over $3 \\times e$ epochs. \n\nThis ensures that the total number of training steps remains consistent across both scenarios.\n\n---\n\n**J. Are the conclusions the same if X-SNS is applied in some other multilingual language models, such as large language models, e.g., Bloom, and speech-based models, e.g., XLS-R?**\n\n---\n\nWe believe that your suggestion reveals a future direction we should pursue, i.e., attempting to extend the application of our approach to the case of pre-trained decoders.\n\nNevertheless, we assert that our experimental setup is comprehensive, particularly given that cross-lingual transfer has predominantly been explored within the context of encoder-based models.\n\nThank you again for your constructive suggestion for future avenues.\n\n---\n\n**K. Analysis on fail cases can show the shortages of X-SNS and provide insights for reaching the upperbound (best one).**\n\n---\n\nThanks for the suggestion.\n\nWe also believe that it is important to show both the strengths and weaknesses of our method.\n\nWe tested with various target languages, and most of the languages produced successful predictions, but there were some failed cases. \n\nTo be specific, we found that in three low-resource languages, i.e., Afrikaans, Uzbek, and íslenska, the transfer performance of our method was worse than using English. \n\nFor more details, please refer to Figure 1 and Figure 7 in addition to Appendix F.\n\n---\n\n**L. Is the fine-tuning on the source language conducted in the whole multilingual model or the shared sub-network? Analysis on how the sub-network change during fine-tuning can better explain the success/fail cases of X-SNS.**\n\n---\n\nFor your better interpretation, we should first make it clear that our method does not involve fine-tuning. \n\nWe only calculate the gradient for each language as a source of information. \n\nHowever, we do not undertake any optimization steps (like gradient descent) based on this gradient.\n\nThe rationale behind this approach is that our objective is to identify a source language optimal for transfer without actively tuning the model. \n\nIf we were to update the model, it would introduce a contradiction to our intended method.\n\nIndeed, your recommendation to examine the dynamics of sub-network alterations is intriguing.\n\nWe believe it can be integrated into the source tuning phase, either to analyze patterns by language or to enhance the elucidation of our methodology.\n\n---\n\n**M. Please provide the time cost of X-SNS.**\n\n---\n\nExtracting a feature for one language takes 37.14 seconds, while comparing features requires 3.81 seconds. \n\nWhen computing the language feature with 1024-shot data, the gradient calculation consumes approximately 20.74 seconds. \n\nHowever, this duration can be shortened to 6.32 seconds if we reduce the dataset size for feature calculation to 256.\n\nNotably, this approach has a time-efficiency advantage. \n\nInstead of undergoing the actual source training process, which takes 2 minutes and 31 seconds per epoch, we can identify the source language in a fraction of that time, yielding significant savings in terms of time cost.\n\n---\n\n**N. How is the source language set defined?**\n\n---\n\nFor alignment with existing studies, we curated our language set to encompass languages that prior research has employed for transfer language prediction. \n\nA comprehensive breakdown of the language set composition can be found in Appendix A.\n\n---\n\nOnce again, we deeply appreciate your diligent efforts and the insightful questions you've brought forward.\n\nWe hope that our responses address your questions satisfactorily.\n\n\n>**Reference** \n>\n> [1] [Make the Best of Cross-lingual Transfer: Evidence from POS Tagging with over 100 Languages](https://aclanthology.org/2022.acl-long.529) (de Vries et al., ACL 2022)\n>\n> [2] [Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer](https://proceedings.mlr.press/v203/muller23a.html) (Muller et al., PMLR 2023)\n>\n> [3] [Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning](https://aclanthology.org/2021.emnlp-main.749) (Xu et al., EMNLP 2021)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents a novel language similarity measure for zero-shot improving cross-lingual transfer.\n- The experimental results show that the proposed method outperforms previous methods.", "reasons_to_reject": "- Although the paper is well organized, some details are not clear and some experiments are missing (see questions in details)", "questions_to_authors": "\nA.Line 18, what do ‘external resources’ refer to?\n\nB.In Figure 2, Is Step 3 (top p%) necessary? Is it possible to obtain the similarity based on the outputs of Step 2?\n\nC.Line 275-279, does previous paper for choosing source languages require ‘the need for extra training of base models’, Please give the citation of previous work. Does ‘doing so’ refer to ‘considers the relative significance’?\n\nD.Line 287, why is the general domain set as Wikipedia instead of CC-100? CC-100 is the corpus used for training XLM-R.\n\nE.Line 304, what is random classifier?\n\nF.Line 340-345, lang2vec provides six metrics for measuring linguistics similarity across languages. Do this paper use the ‘feature’ one? If yes, please also compare with other 5 separate metrics. Lin et al. 2017 show that ‘feature’ is not best one for predicting performance of cross-lingual transfer.\n\nG.Table 2, 3, why are different baselines used?\n\nH.Line 544-548, can X-SNS deal with the case that combining a new language might worse the results?\n\nI.Line 550, while considering 3 source languages, are there the same amount of data used for fine-tuning as considering single source language?\n\nJ.Are the conclusions the same if X-SNS is applied in some other multilingual language models, such as large language models, e.g., Bloom, and speech-based models, e.g., XLS-R?\n\nK.Analysis on fail cases can show the shortages of X-SNS and provide insights for reaching the upperbound (best one).\n\nL.Is the fine-tuning on the source language conducted in the whole multilingual model or the shared sub-network? Analysis on how the sub-network change during fine-tuning can better explain the success/fail cases of X-SNS.\n\nM.Please provide the time cost of X-SNS.\n\nN.How is the source language set defined? \n", "paper_topic_and_main_contributions": "This paper focuses on the zero-shot cross-lingual transfer method, which fine-tunes multilingual language models on the source language, and then evaluates on the target language.\n\nInstead of using English as the source language or using features which are costly to obtain for choosing source language, this paper proposes sub-network similarity to choose source language. Specifically, if a source language gets a higher sub-network similarity to the target language, it would be regarded as a better source language while conducting zero-shot cross-lingual transfer for the target language.\n\nThe experimental results show that sub-network similarity can choose better source languages compared to previous methods on a diverse set of downstream tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes and studies a method to utilize sub-network similarity between two languages for predicting the source language in cross-lingual transfer. Overall, the reviewers agree that the sub-network similarity is novel for the selection of languages for cross-lingual transfer. Some concerns regarding the clarity and motivation for the proposed framework (which is depicted in Figure 2) and the proposed approach do not work well in some settings such as when using mT5. In summary, the AC believes that the usage of sub-network similarity in choosing the optimal source language is interesting but also recommends addressing the clarity in details and limitations pointed out by reviewers in the next version of the paper.
Decision: Accept-Findings