GradSim: Gradient-Based Language Grouping for Effective Multilingual Training
Authors: Mingyang Wang
Heike Adel
Lukas Lange
Jannik Strötgen
Hinrich Schuetze
Venue: EMNLP 2023 Main (2023)
Keywords: Effective multilingual learning
language grouping
gradient-based similarity
TL;DR: The paper proposes a gradient-based language grouping approach for multilingual training, and shows its superior performance compared to other language similarity measures in three diverse multilingual datasets.

Abstract: Most languages of the world pose low-resource challenges to natural language processing models. With multilingual training, knowledge can be shared among languages. However, not all languages positively influence each other and it is an open research question how to select the most suitable set of languages for multilingual training and avoid negative interference among languages whose characteristics or data distributions are not compatible. In this paper, we propose GradSim, a language grouping method based on gradient similarity. Our experiments on three diverse multilingual benchmark datasets show that it leads to the largest performance gains compared to other similarity measures and it is better correlated with cross-lingual model performance. As a result, we set the new state of the art on AfriSenti, a benchmark dataset for sentiment analysis on low-resource African languages. In our extensive analysis, we further reveal that besides linguistic features, the topics of the datasets play an important role for language grouping and that lower layers of transformer models encode language-specific features while higher layers capture task-specific information.

GradSim:
Gradient-Based Language Grouping for Effective Multilingual Training
Mingyang Wangh Heike Adel Lukas Lange'
Jannik Strötgen* Hinrich Schüizee
Bosch Center for Artificial Intelligence, Renningen, Germany
LMU Mumich, Germany HHocschuule der Medien, Stuttgart, Germany Karlsruhe University of Applied Sciences, Germany
Sereyynng.wang2200 wwwnge@de.bosch..oom com
Abstract
X
am
Most languages of the world pose low-resource challenges to natural language processing mod- els. With multilingual training, knowledge can be shared among languages. However, not all languages positively influence each other and it is open research question how to select the most suitable set of languages for multilin- gual training and avoid negative interference among languages whose characteristies data distributions are not compatible. In this pa- per, we propose GradSim, language grouping method based on gradient similarity. Our ex- periments on three diverse multilingual bench- mark datasets show that leads the largest performance gains compared to other similar- ity measures and it is better correlated with cross-lingual model performance. As result, we set the new state of the art on AfriSenti, a benchmark dataset for sentiment analysis on low-resource African languages. In our exten- sive analysis, we further reveal that besides lin- guistic features, the topies of the datasets play an important role for language grouping and that lower layers of transformer models encode language-specifie features while higher layers capture task-specific information.
Niger-Conga
English-Creule
pom
X
Indo-Eurapean
dama
Afro-sslatic
target language
Exemplary transfer learning setup with African languages: The motivation for this work is that neither language family (indicated by node col- ors) typological distance (indicated by distance of languages in the plot) are consistent predictors of good performance when choosing source language for cross- lingual transfer. Red cross: source language affecting performance negatively. Green tick: source language affecting performance positively.
Figure 1:
As a solution, previous work investigates dif- ferent measures for language similarity and selects only languages similar to the target language for the multilingual training set (i.a. Tan et al., 2019; Lin et al., 2019; Pires et al., 2019; Oncevay et al., 2020; Shaffer, 2021: Snæbjarnarson et al., 2023). How- ever, it is an open research question whether lan- guage similarity translates into performance gains of multilingual models. For multilingual training, other characteristics might play rolc, such as top- ical shifts of the training data. As a result, it is still unclear how to select the set of languages that leads to the most effective multilingual training setup.
1 Introduction
Most natural language processing (NLP) rescarch today still focuses on small number of languages. Extending NLP models to further languages poses different challenges, i.a., little (annotated) data (Hedderich et al., 2021). Multilingual training can help in those cases by sharing knowledge across languages. However, adding new languages to the multilingual training set may not necessarily lead to performance gains. In fact, certain languages might actually hurt the performance on downstream tasks in specific target language (Adelani et al., 2022: Snæebjarnarson et al., 2023), for instance, due to unrelatedness to the target language.
In this paper, we study multilingual fine-tuning of language models with a diverse set of training languages.' In particular, we show that linguistics- based language similaritics are only wcakly corre- lated with cross-lingual transfer performance. Fig- ure 1 illustrates a sample case in which neither
NNte that our proposed method is generic and could also he applied for multilingual pre-training.

---

language family information (indicated by node colors) nor similarity of language embeddings (in- dicated by proximity in the vector space) is helpful for finding languages that have a positive cross- lingual transfer score with the target language. Thus, prior information about the languages, such their language families or typological features, alone is not enough for effective multilingual training. Instead, similarity measures that capture additional information about the data and task be- yond linguistics similarity may achieve better per- formance. Wang et al. (2020), for instance, show that gradient similarity across languages measured along the optimization trajectory correlates with language proximity and cross-lingual performance.
2 Related Work
Multilingual and multi-task training. grow- ing number of research projects investigates mul- tilingual training to cover a variety of languages, including low-resource languages (Hu et al., 2020; Lange et al., 2020; Hedderich et al., 2021; FitzGer- ald et al., 2022). In the context of low-resource sentiment analysis, Wang et al. (2023) recently use the cross-lingual transfer score between pairs lan- guages to select source languages for multilingual training. Our approach differs from these works in that we investigate language interactions from a global optimization perspective.
Considering each language as a separate task, multilingual training can be treated as a multi-task learning (MTL) problem (Ruder, 2017). A line of existing work utilizes gradient-based techniques to improve multi-task learning (Chen et al., 2018 88; Sener and Koltun, 2018; Yu et al., 2020; Wang et al., 2020). They show that negative cosine sim- ilarity between gradients leads to negative inter- ference for MTL optimization, and projecting out the conflicting gradients can improve the optimiza- tion dynamics. Our work follows this insightful observation. However, in contrast to their work, we propose to leverage multilingual gradients for lan- guage grouping to ensure that gradients are aligned in each language group.
We draw inspiration from this observation. How- ever, instead of projecting conflicting gradients throughout the training process, we propose to leverage the gradient similarity to group languages with branch-and- -ouddd--ond-ikeee algorithm that opti- mizes the overall similarity score of all languages. This approach has the following ad advantages: (i) It be applied without any prior knowledge of the languages or topics of the given datasets, (ii) it is well correlated with downstream task performance of the multilingual model, (iii) finds the best language groups from a global perspective, i.c., in- stead of selecting source languages independently of each other (which may create groups of mutually interfering languages), we form each group based a criterion that evaluates the group whole. In our experiments, we show the superior perfor- mance of our grouping method compared to various baseline approaches on three multilingual datasets with different tasks and set the new state of the art on AfriSenti, sentiment analysis dataset in 12 low-resource African languages.
Language similarity measures. In order to group languages for multilingual training or trans- fer learning, related work has proposed different ways estimate the similarity between languages, e.g., leveraging the language family taxonomy (Tan et al., 2019; Shaffer, 2021; Chronopoulou et al., 2023; Snæbjarnarson et al., 2023) or represent- ing languages as information-rich vectors based on their typological or conceptual features (Littell et al., 2017; Lin et al., 2019; Oncevay et al., 2020; Liu et al., 2023).
Furthermore, we extensively analyze our models with a topic analysis, a correlation-based analy- sis and an ablation study, revealing important in- sights, for instance that the topic distribution of the training data heavily affects multilingual train- ing and that lower layers of transformer models encode language-specific features while higher lay- ers capture task-specific information. This con- firms results from prior work (i.a., Raganato and Tiedemann, 2018; Jawahar et al., 2019; Tenney et al., 2019; Kovaleva et al., 2019) from another (correlation-base) perspective.
Another line of works measures language simi- larity based on embeddings from multilingual pre- trained language models (mPLMs) (Raganato and Tiedemann, 2018; Lange et al., 2021b; Chang et al., 2022; Lin et al., 2023). Tan et al. (2019) and Shaf- fer (2021), for instance, perform language grouping for multilingual named entity recognition and neu- ral machine translation based on embeddings. In contrast to these studies, we propose to use the gra- dient cosine similarity between languages as the similarity measure for language grouping. This
The code base for GradSim is available online,

---

model-based similarity measure reflects how each language interacts in the optimization process, with no need of any prior knowledge of the languages.
branch-and--oundd-le algorithm in Standley et al. (2020) and Fifty et al. (2021).² The algorithm evaluates different combinations of K language groups under the constraint that each language is included in at least one, but potentially multiple groups. We fimally select the language grouping that leads to the highest overall similarity score.
3 Method
In this section, we describe our proposed language grouping approach and the general multilingual training in which we apply its results. Note that the gradient-based similarity estimation is purely model-based, thus, can be applied other settings, e.g., multi-domait or multi-task problems, as well.
Given r = {71.... 7K) } as a potential grouping result, we define the overall similarity score for Σi Sim(4/ where Sim(4/[) is the collective similarity score of language I in its language group Y E r. The collective similarity score of 4 E Yj is defined as the average of all pair-wise similarities between I; and the other languages in Yj.
3.1 Step I: Gradient Similarities
Due the high discrepancy among languages, multilingual optimization often suffers from the conflicting gradient issue (Wang al., 2020; Xu and Murray, 2022), i... gradients of different lan- guages point into different directions. Previous works show that gradient similarity is correlated with model performance (Chen et al., 2018 8; Sener and Koltun, 2018; Wang et al., 2020; Yu et al., 2020). Inspired by this observation, we propose to use gradient similarity for grouping languages.
3.3 Steps III: Training and Inference
Given the language groups I from Step II, we train one multilingual model per group "i C r, using the training data from the respective languages. For inference, we select the appropriate multilingual model for each target language and apply it to the test data. If target language 4 appears in more than one group, we select the group with the highest collective similarity score of 14 for inference.
Given a set of languages = (2,,2......
we study the gradient similarities across languages by training a multilingual model jointly on all languages and measure the language gradients G= (01.92. ......... .IN) along the optimization pro- cess. To reduce computational costs, average language gradients first at the epoch level and cal- culate the per-epoch gradient cosine similarity be- tween languages. Then we average the gradient similarity over all epochs. Finally, we get gradient similarity matrix
Experiments
In this section, we describe our experimental set- tings as well as our results for three tasks.
4.1 Tasks and Datasets
We experiment with the following three datasets covering different languages as well text classifi- cation and sequence tagging tasks. (Dataset statis- tics are given in Table and in Appendix A.)
N across N languages, with $i.j = cos(g, 9j) = 9::9;
AfriSenti (Muhammad et al., 2023a,b): This shared task dataset provides challenging testbed for sentiment analysis: Both the languages (12 African languages) and the text genre (Twitter) pose challenges to NLP models. To investigate multilingual training results, we focus on the mul- tilingual subtask of the shared task (Subtask B), and report macro-weighted Fl scores following Muhammad et al. (2023b).
Since it is very expensive to calculate the gra- dient similarity based on the gradients w.r.t. all parameters, we choose to only use the gradients based on the classification layer of the model. An analysis of gradients of different layers and abla- tions studies can be found in Sections 5.2 and 5.3.
3.2 Step II: Language Grouping
Based on the pairwise similarity matrix S from Step I, we next determine the best grouping into a pre-defined number of K groups.
WikiAnn (Pan et al., 2017): This dataset of- fers automatically extracted labels for named entity recognition (NER). Following Shaffer (2021), we select 15 languages for our experiments and use micro-averaged Fl as the evaluation metric.
In particular, our goal is find the K language groups which (i) cover all languages of the given language set T., and (ii) maximize the overall simi- larity score all languages, which is a reduction from the Set-Cover problem. We solve it using the
AAternatively, the binary integer program (BIP) solver could he used as in Zamir et al. (2018).

---

1- Measure Gradient Similarities
III Train LMs for Each
II Language Grouping
across Languages
Language Group
I Language Groups
Mottilingual Model
Similarity Matrix
Group
Group
Figure 2: Overview of our proposed method GradSim for language grouping. (Step 1) We train all languages in one multilingual model to measure the gradient similarities across languages. (Step II) We determine the best K language groups based on the similarity measure from the first step. (Step III) We train one language model on each language group and deploy for inference.
Universal Dependency (UD) treebank v1.2
language families are covered by the 27 languages we study.
(Nivre et al., 2016): We experiment with part-of- speech (POS) tagging using the 17 Universal POS labels. Following prior work (Yasunaga et al., 2018; Lange et al., 2a) dataset with 21 high-resource and 6 low-resource languages and report accuracy for evaluation.
On all three datasets, we use the A opti- mizer (Loshchilov and Hutter, 2017). The training was performed Nvidia A100 GPUs. All re- ported results are averaged over random seeds.
27 languages from the
4.3 Baselines
4.2 Training Details
Besides a monolingual model (trained only on the target language) and a purely multilingual model (trained on all available languages), we consider the following baselines for language grouping that have been presented by prior work:
For our experiments AfriSenti, trained AfroXLM-R large transformer (Alabi et al., 2022), an XLM-R model adapted to African lan- guages, as our base model. To measure language gradients in Step I, we use 25% of the training data in AfriSenti and set the batch size to for compu- tational efficiency. For multilingual training and inference (Step III) ), we use all training data and batch size of 32. In both stages, we finetune the model with learning rate of le-5 and set the maxi- mum sequence length 128. We set the number of language groups to K which equals the num- ber of language families in the dataset. We further provide comparison of different K Section 5.3. For the NER task, we follow the training setup used in Shaffer (2021) for fair comparison. Specifically, we use XLM-R as our base model and finetune it for epochs. We the batch size to 20 with a learning rate of 2e-5 and a max se- quence length of 300. Following Shaffer (2021), we set the number of language groups to K = 4.
the pre-
Language family. We group languages based on their language family information and train one multilingual model per language family. Language family-base grouping is also studied by, i.a., Tan et al. (2019); Shaffer (2021); Chronopoulou et al. (2023); Snæebaarnaroon et al. (2023).
Typological similarity. Languages can be rep- resented by typological features, e.g., the syn- tax, phonology or inventory features. Using the lang2vec tool and the URIEL knowledge base (Lit- tell et al., 2017), we retrieve language vectors and use the pairwise distances among them as the simi- larity measure of our algorithm. This is similar to Lin et al. (2019) and Oncevay et al. (2020). Embedding distance. Multilingual pretrained language models (mPLMs) also encode language- specific information (Raganato and Tiedemann, 2018: Chang al., 2022). Tan al. (2019) and Shaffer (2021) use mPL.M-based language embed- dings to determinc language similarities for lan- guage grouping. Following this idea, we compute
For the POS tagging task, we use the XLM-R
model as well and set the training epoch to 20.
We use a batch size of 8, learning rate of 2e-5 and a maximum sequence length of 128. Here, we specify K = 6 for language grouping, as
'All experiments ran on a carbon-neutral GPU cluster.

---

Method
avg"
am'
dz
ha
Ig'
kr'
ma'
pt
sw
twi
yo"
Oracie upper bound
71.46
69.87
69.45
80.15
78.48
70.72
52.41
68.67
71.85
61.59
55.42
64.79
75.52
Multilingual
59.97
$1.99
56.62 62
66.10
61.03
43.07
55,70
67.37
58.18
42.86
62.87
Monolingual
68.29
49.00
57.16
80.36
79.55
70.72
48.73
68.24
66.12
62.50
44.66
54.56
75.09
Language family
66.
67.84
68.54
78.88
66.16
$1.10
67.32
64.88
58.38
47.04
$4.60
64.36
Typological similarity
68,93
66,10
69.27
79.38
78.00
70.09
49.17
63.81
66.
63.23
46,79
61.25
73.95
Einheddingg dis. (PLM)
68.81
72.09
67.20
71.57
74.49
71.89
51.49
69.09
67.32
62.50
52.01
60.88
75.09
Embedding dis. (FT)
69.62
59.69
67.36
79.78
78.71
69.99
50.01
66.43
62.62
49,20
64.01
75.07
GradSim (ours)
71.34
66.11
67.90
79.97
79.55
72.12
53.68
68,40
72.30
63.05
50,36
63.46
75.09
Table 1: Results on AfriSenti, a benchmark dataset for sentiment analysis on low-resource African languages. Bold shows best results, underline bighlights second-best results per language and on average. * indicates the settings with statistically significant improvements (p-value < 0.05) using GradSim compared Embedding dis. (FT), the overall second-best system.
Method
Average Fl
pem, or typological similarity for sw), GradSim performs best or second best for most languages and, as a result, best on average. Its average result comes very close to the oracle upper bound, which, in contrast to our approach, requires prior knowl- edge about cross- cross-lingur lingual transfer performance.
SOTA Single medel GradSin+TAPT Single model (ours)
75.29
SOTA Ensemble
75.06
GradSin+TAPT Ensemble (ours)
75.34
Table Results AfriSenti in comparison the state of the art (Wang et al., 2023). We apply task-adaptive pretraining (TAPT) and ensemble methods on top of GradSim for fair comparison to the state of the art.
We also compare GradSim with the state-of-the- art method on AfriSenti (Wang et al., 2023), which uses AfroXLM-R with task-adaptive pretraining (TAPT) (Gururangan et al., 2020) and performs transfer learning after selecting the best source lan- guages based on their cross-lingual transfer score. For a direct comparison, we also apply TAPT and use GradSim to group languages for multilingual training. As shown in Table 2, GradSim sets the new state of the art on AfriSenti. It superior to the previous approach of Wang al. (2023) that only considers the pairwise transfer scores, neglecting possible interactions of different source languages. Instead, GradSim maximizes the overall gradient similarities from a global perspective.
sentence embeddings using the pretrained encoder from our base model and average sentence embed- dings of the same language. Then, we use the em- bedding distance across languages as the similarity measure in Step (denoted by Embedding distance (PLM)). As an alternative, we also consider embed- dings from the language model fine-tuned on the task (denoted by Embedding distance (FT)).
Oracle upper bound. As an upper bound, we group languages based on the post-hoc cross- lingual transfer performance. The cross-lingual transfer performance is often used for source lan- guage selection as in Adelani et al. (2022) and Wang et al. (2023). We consider this an oracle upper bound as it is direct indication of how knowledge learned from one language affects the performance on another language. Note that this ap- proach is computationally expensive as it requires N x N transfer experiments for N' languages, while our gradient-based approach only needs single training run for collecting gradient information.
Sequence tagging. Table 3 provides our results for multilingual named entity recognition. We re- port the state-of-the-art results from Shaffer (2021) as baseline results. Our approach GradSim out- performs the prior state of the art on most high- resource languages and all low-resource languages, again leading the best overall results.
Our results for POS tagging are provided in Table 4. GradSim outperforms multilingual and monolingual training without language grouping as well as language grouping based on other metrics. It performs best on average over the low-resource languages as well as average over all languages. Given the results on sequence tagging tasks, we find that low-resource languages benefit more from language grouping. For high-resource languages, additional training sources from other languages
4.4 Results
Text classification. Table shows our experimen- tal results on the AfriSenti dataset (per language and on average). While for a few languages, a grouping based on our baseline approaches per- forms best (e.g., embedding distance for am and

---

Embed. (prior)
GradSim
Embed.
GradSim
NER Multi.
Mono.
Family
Multi
Mono.
Family
(ours)
(FT)
(ours)
high-resource
high-resource
ar' 86.65
85,25
84,92
85.25
88.02
bg'
99.42
99.34
99.21
99.38
99,40
he
84.21
84.51
82.47
84.83
84.06
CS
99.00
99.00
98.91
98.99
99.01
da"
90.00
87.57
89.64
90.49
91.65
da"
22
98.66
98.12
98.55
de
84.42
82.42
84,18
85.73
87.27
de
94.47
94.72
94.43
94.59
94,43
en
81.97
77.91
81.28
83.37
83.31
en 97.06
97 27.34
96.88
97
97.18
es"
82.01
89,90
90.85
97.35
97.
97.18
97.21
88.22
82.83
$7.78
89.79
89.54
95.95
96.01
96.01
96.04
96.09
hi
87.30
84.04
85,51
87.17
88.25
fa 97.30
97.31
97.20
97.35
97.41
86.03
90.52
90.72
97.
97.
97.70
ru'
88.32
88,18
$7.77
88.55
88.70
fr' 96.58
96.41
96.21
96.40
96.67
ko
85,97
86.54
84.66
86.91
85,92
he 97.39
97.25
97.25
97.31
97.43
ja
66.83
71.40
75.56
97.65
97.55
zh
79.36
73,66
73,66
79.12
75.49
hr 97.66
97.67
97.55
97. 67.65
97.57
id 91.16
91.78
91.78
91.56
91.20
avg" 85.34
82.14
83.55
85.62
86.10
98.68
low-resource
nl" 93.76
93.94
93.66
93.80
93.88
pl 98.88
no"
99.03
98.91
98.95
99.01
xw" 88.22
63.30
55.11
90.13
90.22
yo" 77.24
7.74
21.81
85.33
86.22
pl 98.49
98.51
98.43
98,44
98.54
avg"
35.52
38.46
87.73
88..2
al" 98.94
99.03
98. 98.90
98.96
99.02
98.77
avg (all)"
84.99
75.92
77.54
85,90
86.39
avg
97.27
97.36
97.21
97.28
97.32
Table 3: Results on WikiAnn, NER benchmark, in micro FI. The numbers of the four baseline previous state-or of ahe methods are taken from Shaffer (2021) and micro-averaged over the different classes. indi- cates the settings with statistically significant improve- ments using GradSim compared to Embed. (prior), the second-bes system. Further baseline results are given in Table 10 in Appendix B for space reasons.
low-resource
el'
98.56
98. 41
98 20
98. 38.57
98.59
er"
95.34
94.76
95.46
95.27
95.78
ga"
93.34
92.49
93.32
93.10
93.52
hu
96.82
96.87
97.01
97.14
96.94
ro
95,74
94.65
95.32
95.74
95.14
85.63
84.55
84.55
87.16
88.32
avg 96.60
93.62
93.98
94.50
94.72
avg (all))
96.53
96.49
96.66
96.74
Table 4: Results on the UD POS tagging dataset ac- curacy. * indicates the settings with statistically signifi- cant improvements using Gr GradSim i compared to Embed. (FT), the second- system. Further baseline results provided in Table 11 in Appendix B.
have less prominent impact when enough in- language training data is available. It highlights the value of multilingual learning with well-suited lan- guages to enhance the performance of low-resource languages, providing key strategy for advancing future low-resource NLP research.
derstand differrences coming from data peculiarities (Section 5.1), a correlation analysis to explain why some grouping methods work better than others (Section 5.2), and an ablation study to investigate the impact of our design choices (Section 5.3).
Significance tests. We rum permutation- based significance tests following Dror et (2018) with a significance level of 0.05 between GradSim and the respective second-best system on all three datasets. In Tables 1, 3 and 4, settings with statistically significant improvements when using GradSim marked with * The results show that GradSim m is significantly better than the second-best system in 32 out of 37 single language settings where GradSim outperforms the second-best sys- tem across three datasets. In addition, its average performance across all languages is significantly better than the other systems on all three datasets.
5.1 Topic Analysis
Although data analysis is valuable for research progress, it is challenging for foreign languages. Therefore, we choose a semi-automatic approach involving machine translation and manual inspec- tion for better understanding the input data of our models: For each language, we first extract the 50 most relevant keywords via a term frequency- inverse document frequency (TF-IDF) method. Then, we use the pygoogletranslate APP to trans- late the keywords into English and remove dupli- cate words and stop words. Table 6 provides exem-
5 Analysis
To analyze the behavior of the model, we perform the following analyses on the AfriSenti dataset: qualitative analysis of the data order to better un-
htttps: thub. www.Saravanansllll.... py- gogoottaaaaainooo

---

Pearson correlation coefficient
Grouping methods
+ transfer score
+ typological similarity
+ topic similarity
Baseline
Cross-Transfer Score (oracle)
0.353
0.5079
Language Family
0.0869
0,5278
0,2680
Typological similarity
0.3530
0,0353
Embedding distance (PLM)
0,4029
0.7696
-0,2383
Embedding distance (FT)
0.4667
0.7240
-0.0252
Gradient similarity wrt. different layers (from deep to shallow)
Classification layer
0.6963
0.3944
0.4749
Encuder luyer 23
0.6485
0.6377
0.1486
Encoder layer 21
0,5526
0.7811
0.1134
Encoder layer 18
0,4462
0,8181
-0.0601
Eacoder layer 15
0,4602
0.8329
0,1083
Encoder layer 12
0.4532
0.8566
-0.0731
Encoder layer 6
0.4542
0.8586
-0.0342
Encoder layer O
0,4526
0.8526
0,0721
Table Results our correlation analysis on the AfriSenti dataset.
Language (family)
Keywords
language family-based grouping would cluster and is into the same group.
Swahili (Niger-Congo)
god, thank you, major, national, minister. better, package, service, continue, dr, education, citizens, news, world, construction, people, region, police, state, president, fa- ther, army
5.2 Correlation Analysis
Table provides the results our correlation anal- ysis that we perform on the AfriSenti dataset. particular, we compute the Pearson correlation co- efficient between the different grouping methods (similarity measures) that we study and different characteristics, such istics (measured by cross-lingual transfer score), lang Linguage-specifgg characteristics (measured by ty- pological similarity) and topic-specific characteris- tics (measured by keyword embedding distance).
Amharic
flower, city, season, a matter. government, discussion, decem- ber, press release, district, informa- tion, administration, public, gov- ermment, the racist, man, poison
(Afro-Asiatic)
character-
Xitsonga
mozumbique, listen, wake up. awake, live, conform, sugar, home, lake, leave, speed, connect, come
(Niger-Congo)
Table 6: Exemplary keywords from AfriSenti tweets of different languages (reduced to subset for space reasons, full set provided in Appendix (Table 15)).
From the results, we can draw number of inter- esting conclusions, namely:
(i) The transfer score is not correlated with lan- guage family information and only weakly corre- lated with embedding baasdd similarity measures often used in related work. For gradient similar- ity, we see considerably higher correlation values, supporting our proposal of using this similarity measure for language grouping.
plary results for three languages of the AfriSenti dataset. The complete of keywords for all lan- guages is provided in Appendix D (see Table 15).
While the keywords extracted for Swahili (sw) and Amharic (am) are mainly centered around po- litical and administrative topics, e.g., national, min- ister, education, government etc, the keywords for Xitsonge (ts) are more related to every-day life as- pects. The multilingual model performance reveals that indeed Swahili and Amharic can effectively be trained together while Swahili and Xitsonga rather harm each other, even though Swahili and Xitsonga belong the same language family and Swahili and Amharic do not. When looking at the language grouping results, GradSim indeed groups sw and am together (see Table 12 in Appendix D), thus, is able to capture their topical similarity, while a
(ii) There a relatively weak correlation be- tween the cross-lingual transfer score and the ty- pological similarity, while language family and embedding-based similarity measures show high correlation with typological language similariy. This indicates that these similarity measures cap- ture the linguistics-based language information well, which, however, does not translate into better transfer performance. Similar to the oracle mea- sure (transfer score), gradient similarity based on the classifier parameters only weakly correlated

---

with typological language similarity.
vestigate the performance for K {1...8} in Figure Choosing K can already improve the performance compared to purely multilingual training (K = 1). Until K - 4, the performance further improves and then converges for larger K.
(iii) Based on the keywords extracted for our analysis in Section 5.1, we retrieve keyword em- beddings from the pretrained model encoder and average them for each language. We then compare the similarities of the keyword-based language em- beddings with our different similarity measures using Pearson correlation. We find that they only weakly correlated with language family infor- mation and even weakly negatively correlated with embedding distances. However, the correlation with the cross-transfer score and our proposed gra- dient similarity is larger, indicating that the gradient similarity can indeed pick up the topic information of the data.
Gradient from layer
Task performance
Classification layer
71.34
Encoder layer 23
69.95
Encoder layer 21
69.43
Encoder layer
69.21
Encoder layer
68.91
Encoder layer 12
69.19
Encoder layer 6
69.19
Encnddr layer O
68.22
Table 7: Ablation study of gradients from different lay- the AfriSenti dataset.
(iv) While higher layers are higher correlated with task performance, lower layers show higher correlation with typological distance. This in- dicates that lower layers encode rather general language-specific information while higher layers capture task-related information.
Discussion
In this section, we summarize our main findings.
Language similarity is not enough to determine transfer suitability. When sharing knowledge across languages, the information about linguistics- based language similarity (e.g., whether the lan- guages come from the same language family how languages are typologically similar each other) is not enough for optimal performance. This observation in line with the findings by Tan et al. (2019), Shaffer (2021) and Malkin et al. (2022) that languages from the same family may still ex- hibit distinct linguistic features and, thus, language- family based grouping can enhance model perfor- mance only to certain extent. In addition, find that there are other aspects that will affect multilin- gual model performance and, therefore, need to be taken into account, such as the topical distribution of the data.
5.3 bllttoon Study
Gradients from different layers. Table shows an ablation study of our model. The main design choice of our approach is the position in the model where to take the gradients. In our analysis, we compare model performance when using gradients from different layers. We see clear trend that higher layers are better suited than lower layers. In particular, taking the gradients directly from the classification layers leads to the best results.
70
F1 score
Gradient-based method does not require any prior knowledge. Our proposed gradient- approach for grouping languages is a pure model- based approach, thus, does not require any prior knowledge about the language, task or data. As a result, it can be successfully applied, even when the data distribution (e.g., topical distribution) is unknown (c.g., because are dealing with foreign languages). While our current work only presents results for language grouping for multilingual mod- els, the method itself is more general and can be applied to other settings as well, such as multi-tast learning or multi-domani setups.
Number of language groups
Figure 3: Ablation study of different number of groups AfriSenti: average Fl w.r.t. number of groups
Number of language groups. For our experi- ments, choose the number of language groups K to be the same the number of language fam- ilies covered in the datasets. 6 However, K is hyperparameter of our method. Therefore, we in-
Except for WikiAnn, where we set K to the same number as prior work (Shaffer, 2021) for a fair comparison.

---

Lower layers capture language, upper layers task information. Adding to previous work on analyzing transformer-based pretrained language models (Raganato and Tiedemann, 2018; Jawahar et al., 2019; Tenney et al., 2019), our correlation analysis shows that gradient similarity between lan- guages from lower layers are more correlated to language-specite distances, i..., low layers seem to encode language-specifis information, while gradi- ent similarity from upper layers are more correlated to tasc-specific performance, i.e., upper layers tend to capture task-specific information.
However, we chose the datasets for our experi- ments with the aim of covering a broad variety of languages, including African languages which are typically under-explored in NLP research.
Ethics Statement
Our work focuses on multilingual and low low-resource settings. For instance, we investigate our models on African languages which are typically under- represented and under-explored in NLP research. Including them into NLP research is important from ethical point of view.
7 Conclusion
In this paper, we addressed the challenging prob- lem of grouping languages for effective multi- lingual training. We proposed gradient- based grouping approach and showed in our experiments that it is better correlated to cross-lingual trans- fer performance than language family language embedding- grouping. In our analysis, we identified topical distribution differences as one potential challenge that can be addressed effec- tively by our approach. Furthermore, our corre- lation analysis confirmed results from prior work that lower layers of transformer-burt pretrained models seem to encode language-specific features, while upper layers capture task-specific informa- tion. Our method shows superior performance com- pared a variety of baseline methods for language grouping on three diverse datasets and, in particu- lar, sets the new state of the art on a multilingual sentiment analysis benchmark dataset consisting of low-resource African languages.
References
David Adelani, Graham Neubig, Sebastian Ruder. Shruti Rijhwani, Michael Beukman, Chester Palen- Michel, Constantine Lignos, Jesujoba Alabi, Sham- suddeen Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Rooweither Mabuya, Bonaventure F. P. Dossou, Blessing Sihanda, Happy Buzaaba, Jonathan Mukiibi. Godson Kalipe. Derguene Mbaye, Amelia Taylor, Fatoumata Ka- bore, Chris Chinenye Emezue, Anuoluwapo Aremu, Perez Ogayo, Catherine Gitau, Edwin Munkoh- Buabeng, Victoire Memdjokam Koagne, Allah- sera Auguste Tapo, Tehogo Macuewa, Vukosi Mari- vate, Mboning Tchiaze Elvis, Tajuddeen Gwad- abe, Tosin Adewumi, Orevaoghene Ahia, Joyce Nakatumba--abendee Neo Lerato Mokono, Ig- natius Ezcani, Chiamaka Chukwuneke, Mofetoluwa Oluwaseun Adeyemi, Gilles Quentin Hacheme, Idris Abdulmumin, Odunayo Ogundepo, Oreen Yousuf, Tatiana Moteu, and Dietrich Klakow. 2022. MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 4488 4508, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Limitations
Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Moshach, and Dietrich Klakow, 2022. Adapting pre- trained language models to African languages via multilingual adaptive tueinunnng. In Proceedings of the 29th International Conference on Computational Linguisties, pages 4336-4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
One limitation of our work is the scope of evalu- ation. While we performed experiments on three diverse text classification and sequence tagging tasks, GradSim is generally applicable to a wide range of tasks and could thus be evaluated on even further tasks.
Besides, our experiments currently focus on mul- tilingual settings and datasets. Experiments for multi-domain and multi-task settings are outside the scope of this work, however, an interesting di- rection for future work.
Tyler Chang, Zhuowen Tu, and Benjamin Bergen, 2022. The geometry of multilingual language model repre- sentations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 119-1166 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Finally, compared the large number of lan- guages in the world, the set of languages our work still limited and, thus, our results might not be representative for all languages of the world.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. 2018 Gradnorm: Gradient normalization for adaptive loss balancing in deep

---

multitask networks. In International conference on machine learning, pages 794-803, PMLR.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joinl Conference on Natu- ral Language Processing TEMNLP-IJCNTE pages 4365-4374, Hong Kong, China. Association for Com- putational Linguistics.
Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. 2023. Language-famin family adapters for low-resource multilingual neural machine trans- lation. In Proceedings of the The Sixth Workshop on Technologies for Machine Translation of Low- Resource Languages (LoResMT 2023), pages 59-72, Dubrovnik, Croatia. Association for Computational Linguistics.
Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021a. FAME: Feature-based adversarial meta- eeeta-emeddiings for robust input representations. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 3382-8395, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Rotem Dror. Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker's s guide to testing sta- tistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the A Association for Computational Linguistics ( (Volume I: Long Papers), pages 1383-1392, Melbourne, Aus- tralia. Association for Computational Linguistics.
Lukas Lange, Anastasiia Iurshina, Heike Adel, and Jan- nik Strötgen. 2020. Adversarial alignment of multi- lingual models for extracting temporal expressions from text. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 103-109, Online. Association for Computational Linguistics.
Chris Fifty, Ehsan Amid, 7he Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503- 27516.
Lukas Lange, Jannik Strötgen, Heike Adel, and Diet- rich Klakow. 2021b. To share or not to share: Pre- dicting sets of sources for model transfer learning. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 8744-8753, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Jack FizzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash. Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natara- jan. 2022. Massive: A Im-example multilin- gual natural language understanding dataset with 51 sypologically-dverss languages.
Peiqin Lin, Chengzhi Hu, Zheyu Zhang, André F. T. Martins, and Hinrich Schütze. 2023. mplm-sim: Un- veiling better cross- lengull similarity and transfer in multilingual pretrained language models.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipla, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-838 8360, Online. Association for Computational Linguistics.
Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang. Mengzhou Xia, Shruti Rijhwani, Junx- ian He, Zhisong Zhang, Xuezhe Ma, Antonios Anas- tasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross- cosss-lingul learn- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125-3135 Florence, Italy. Association for Compu- tational Linguistics.
Michael A. Hedderich, Lukas Lange, Heike Adel, Jan- nik Strötgen, and Dietrich Klakow, 2021. A survey on recent approaches for natural language process- ing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2545-2568, Online. Association for Computational Linguistics.
Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages typological, geographical, and phylogenetic vectors. Proceed- ings of the 15th Conference of the European Chap- ler of the Association for Computational Linguistics: Volume 2, Short Papers. pages 8-14, Valencia, Spain. Association for Computational Linguistics.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi- task bench- mark for evaluating cross-lingual generalization.
Yihong Liu, Haotian Ye, Leonie Weissweiler, Philipp Wicke, Renhao Pei, Robert Zangenfeind. and Hin- rich Schütze. 2023. A crosslingual investigation of conceptualization in 1335 languages.
Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint 17X

---

Dan Malkin, Tomasz Limisiewicz, and Gabriel Stanovsky. 2022. A balanced data approach for eval- uating cross-lingual transfer: Mapping the linguistic blood bank. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx; Human Language Tech- nologies, pages 4903-4915, Seattle, United States. Association for Computational Linguistics.
Computational Linguisties, pages 4996-5001, Flo- rence, Italy, Association for Computational Linguis- tics.
A lessandro Raganato and Jörg Tiedemann. 2018. An amalysis of encoder representations in transformer- based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNI.P: Analyzing and Interpreting Neural Networks for NLP,, pages 287-297, Brussels, Belgium. Association for Com- putational Linguistics.
Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Abinew Ali Ayele, Nedjma Ousidhoum, David Ifc- oluwa Adelani, Seid Muhie Yimam, Ibrahim Sa'id Ahmad, Meriem Beloucif, Saif Mohammad, Se- bastian Ruder, Oumaima Hourrane, Pavel Brazdil, Felermino Dário Mário António Ali, Davis Davis, Salomey Osei, Bello Shehu Bello, Falalu Ibrahim, Tajuddeen Gwadabe, Samuel Rutunda, Tadesse Be- lay. Wendimu Baye Messelle. Hailu Beshada Balcha, Sisay Adugna Chala, Hagos Tesfahun Gebremichael, Bernard Opoku, and Steven Arthur. 2023a. AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languages.
Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098
Ozan Sener and Vladlen Koltun. 2018. Multi-task learn- ing as multi-objective optimization. Advances in neural information processing systems, 31.
Kyle Shaffer. 2021. Language clustering for multilin- gual named entity recognition. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 40-45, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Shamsuddeen Hassan Muhammad, Idris Abdulmu- min, Seid Muhie Yimam, David Ifeoluwa Ade- lani, Ibrahim Sa'id Ahmad, Nedjma Ousidhoum, Abinew Ali Ayele, Saif M. Mohammad, Meriem Beloucif, and Sebastian Ruder. 2023b. SemEval- 2023 Task 12: Sentiment Analysis for African Lan- guages (AfriSenti-SemEWve
Vésteinn Snæehjarnarson, Annika Simonsen, Goran Glavaš, and Ivan Vulić. 2023, Transfer to a low- resource language via close relatives: The case study on Faroese. In Proceedings of the 24th Nordic Con- ference on Computational Linguisties (NoDaLiDa), pages 728-737, Tórshavn, Faroe Islands. University of Tartu Library.
In Proceedings of the 17th International Workshop on Semantic Evalua- tion (SemEval-2023)) Association for Computational Linguistics.
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Yoav Goldberg, Jan Hajič, Christopher D. Man- ning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth In- ternational Conference on Language Resourees and Evaluation (LREC'16) pages (659-1666, Portorož, Slovenia. European Language Resources Association (ELRA).
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2020. Which tasks should be learned together in multi-task learning? In International Conference on Machine Learning, pages 9120-9112 9132. PMLR.
Xu Tan, Jiale Chen, Di He, Yingee Xia, Tao Qin, and Tie- Yan Liu. 2019. Multilingual neural machine translation with language clustering. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 963-973, Hong Kong, China. Association for Computational Lin- guistics.
Arturo Oncevay, Barry Haddow, and Alexandra Birch. 2020. Bridging linguistic typology and multilingual machine translation with multi-view language repre- sentations. In P Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2391-2406, Online. Association for Computational Linguistics.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy, Association for Computational Linguistics.
Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Noth- man, Kevin Knight, and Heng Ji. 2017. Kross-lingual name tagging and linking for 282 languages. In Pro- ceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers). pages 1946-1958, Vancouver, Canada. As- sociation for Computational Linguistics.
Mingyang Wang. Heike Adel, Lukas Lange, Jan- nik Strötgen, and Hinrich Schütze. 2023. NI- nde at semeval-2023 task 12: Adaptive pretrain- ing and source language selection for low-resource multilingual sentiment analysis. arXiv preprint arXiv:2305.00099.
Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceed- ings of the 57th Annual Meeting of the Association for

---

Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. 2020. Gradient vaccine: Investigating and improv- ing multi-task optimization in massively multilingual models. arXiv preprint 4rX1v:2010.05874.
Haoran Xu and Kenton Murray. 2022. Por qué não utiliser alla språk? mixed training with gradient opti- mization in few-shot cross-lingual transfer. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages States. Association for Computational Linguisties.
Seattle, United
Michihiro Yasunaga. Jungo Kasai, and Dragomir Radev, 2018. Robust multilingual ofrt-of-speech tagging via adversarial training. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 976-986, New Orleans. Louisiana. Association for Computational Linguistics.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning, Advances Neural Information Processing Systems, 33:5824- 5836.
Amir R Zamir, Alexander Sax. William Shen, Leonidas J Guibas. Jitendra Malik. and Silvio Savarese. 2018. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference computer vision and pattern recognition. pages 3712-372

---

A Dataset Information
WikiAnn NER dataset
Family
Language
Table 8 provides statistics of AfriSenti, the bench- mark dataset we chose for our text classification ex- periments. Table 9 gives language information for the WikiAnn and Universal Dependencies datasets. We selected 15 languages from WikiAnn for NER following prior work (Shaffer, 2021) and 27 lan- guages for POS tagging following Yasunaga al. (2018); Lange et al. (2021a).
Afro-Asiastic
Arabic (ur)
Hebrew (he)
Indo-Furopean
Danish (da)
German (de)
English (en)
Spanish (es)
French (fr)
Hindi (hi)
Italiun (it)
Russian (ru)
Niger-Congo
Swahili (sw)
Dataset
Yorubu (yo)
Family
Language
Train
Dev
Test
Koreanic
Korean (ko)
Afro-Asiatic
Amharic (am)
5,985
1,498
2,000
Japonic
Japanese (ja)
Algerian Arabic (dz)
1,652
415
959
Sino-Tihetar
Mandarin (zh)
Hausa (ha)
14.173
2.678
5,304
Moroccan Arabic (ma)
5,584
1,216
2,962
UD POS tagging
English-Creole
Nigerian Pidgin (pem)
5,122
1,282
4,155
Family
Languages
Mozambican
Indo-European
Bulgurian (bg)
Indo-European
3,064
768
3,663
Portuguese (pt)
Czech (cd)
Danish (da)
Niger-Congo
Igbo (ig)
10,193
1,842
3,683
German (de)
Kinyarwanda (kr)
3,303
828
1,027
Greek (el)
Swahili (sw)
1,198
454
749
English (en)
Xitsonga (ts)
805
204
255
Spanish (es)
Twi (twi)
3,482
389
950
Persian (fa)
Yoruba (yo)
8,523
2,091
4,516
French (fr)
Irish (ga)
Table 8: Language information and dataset statistics AfriSenti. 12 African languages from language families are included in our study.
Hindi (hi)
Croatian (hr)
Italian (it)
Dutch (nl)
Norwegian (no)
Polish (pl)
Portuguese (pt)
B Additional Baseline Experimental Results
Romanian (ro)
Slovenian (sl)
Swedish (sv)
Here we provide the experimental results the WikiAnn and UD POS tagging dataset with group- ing methods based on cross-lingual transfer score. typological language similarity and language em- beddings distance.
Basque
Basque (eu)
Uralic
Finnish (fi)
Estonian (ct)
Hungarian (hu)
Afro-Asiastic
Hebrew (he)
Austroncsian
Idonesian (id)
Comparing GradSim to other baseline methods in Table 10 and 11 we can draw similar conclu- sions as in Section 4.4: First, GradSim achieves the best average performance and performs especially well in low-resource languages, revealing the im- portance of multilingual learning with suitable sets of source languages for low-resource languages. Additionally, GradSim comes very close to the ora- cle upper bound without any prior knowledge about the cross Pross-lingual transfer performance.
Dravidian
Tamil (ta)
Table 9: Language information of WikiAnn and Uni- versal Dependencies datsets for NER POS tagging tasks, respectively.
D Keyword extraction results
In Table 15, we give the full set of keywords ex- tracted from the AfriSenti dataset of 12 African languages.
C Language Grouping Reults
In this section, we provide further details in Table 12, 13 and 14 of our language grouping results with the best language groups based on different similarity measures on the three datasets.

---

Oracle upper bound
Typological similarity
Embedding (PLM FT)
GradSim (ours)
high-resource
ar
88.43
87.40
88.38
88.02
he
85.22
84.12
84.88
84.06
da
92.36
92.22
92.17
91.65
de
88.24
88.01
87.89
87.27
en
83,98
83,75
83,42
83.31
es
91,58
91.92
91. 11
90.85
fr
89.72
90.13
89.83
89.54
hi
88.99
86.20
89.00
88.25
91.13
91.16
90.84
90.72
ru
88.71
88,
88.70
ko
87.03
84.61
86,99
85.92
67.74
67.80
67.33
75.56
2h
76.43
76.72
75.93
75.49
av8
86.12
85.60
85.88
86.10
low-resource
SW 90.37
78.79
89.70
90.22
yo
86.22
61.91
6.20
88.30
70.35
47.95
88.22
avg (all)
86.41
83.56
80.82
86.39
Table 10: Additional experimental results on the WikiAnn dataset. We compare the grouping performance based on different language simialrity measures. We have the same language grouping results for Embedding distance (PLM) and Embedding distance (FT) (See Table 13) and merge them in one column.
Oracle upper bound
Typological similarity
Embedding distance (PLM)
Embedding distance (FT)
GradSim (ours)
high-resource
bg
99.42
99,40
99.35
99.37
99.40
Cs
99.01
98.99 99.99
99.01
98.99
99.01
da
98.56
98.04
98.09
98.07
98.55
de
94.68
94.61
94.36
94.55
94.43
en
97.36
97.16
97.26
97.18
97.36
97.35
97.23
97.25
97.21
96.06
96.01
95.98
96.07
96.09
fa
97.34
97.24
97.35
97.33
97.41
di
97.66
97.50
97.63
97.53
97.70
fr
96.59
96.40
96.51
96.45
96.67
he
97.44
97.24
97.29
97.30
97.43
hi
97.60
97.60
97.61
97.63
97.55
97.68
97.40
97.74
97.65
97.57
91.36
91.75
91.36
91.59
91.20
98.60
98.56
98.58
98.51
98.58
n1
94.01
93.53
93.72
93.79
93.88
99.04
98.91
98.92
98.96
99.01
pl
98.59
98.54
98.49
98.49
98.52
pl
98.56
98,56
98.49
98.42
98,34
sl
99.03
98.87
99.02
98.96
99.02
sv
98.80
97.37
97.25
97.27
97.29
97.32
low-resource
cl
98.62
98.42
98,50
98.56
98.59
et
95.58
95.65
95.04
95.26
95.78
93.50
93.46
93.28
93.08
93.52
96.99
96.90
97.39
97.14
96.94
ro
95.20
95.92
94.76
95.79
95.14
88.33
86.16
88.04
87.02
88.32
avg
94.70
94.42
94.50
94.48
94.72
avg (all)
96.77
96.62
96.66
96.66
96.74
Table Additional experimental results on the UD POS tagging dataset.

---

Method
Language groups
group am, ha, kr, ma, pt, sw group
dz, pem
Oracle upper bound
group
group pem, twi, yo
group am. dz, ha, ma group ig.
yo
Language Family
group pem
group pt
group twi, dz, ha, ig, mu,
Typological dis,
group kr
group pcm,
Method
Language groups
group pt
group
bg, de, fa, pl, sl, sv, el es, da, en, es, fr,, hr,
group 0 am, dz, ma, pt. twi
group ha, ig, kr, pem, is, twi group sw yo
group
pt, ga, ro, fi, et, hu, he
Embedding dis. (PLM)
Orucle upper bound
group 3
group
group am, kr, pt, sw group dz, ma, pem
group
group
Embedding dis. (FT)
group ha, ig, ma,
group ta
group ma, 1s, twi
bg, cs, da, de, en, f'a, fir, hi, hr, it, nl, no, pl, group
group ha, ma, pt, dz, pem, twi
pt, sl, sv, el, ga, ro
group
GradSim (ours)
group ig
Language Family
group eu group
group yo
group he group id group ta
Table 12:
Language grouping results on AfriSenti dataset. Unbolded languages the source-only lan- guages in the group. i.e., they only participate in the multilingual training but are evaluated in another lan- guage group during inference (details see Section 3.2).
bg. da, de, en, fr,, hr, it, nl, no, pl. pt. sl, sv,el, et, ga, hu, ro
group group eu group group he group group hi, ta
Typological dis.
Method
Language groups
group
ar, sw, yo
group
bg.nl
group
da, de, hi, it, ko ja
es, da, de, en, it, no, pl,
Oracle upper bound
group 2
group
pt, sv, fi,
group 3
ru, zh
Eihhedinn dis, (PLM)
group
es, fa, hi, el
group 0
he
fr, he, ta group
group da, de, en, es, fr,, hi, it, ru
group
ga, eu, et group sl, ro, hu
group 2 sw, yo
Language Family
group 3
bg, cle, fa, nl, pl, sl, el
group 4 ja
group
group 5 zh
group es, da, hr; it, no, ro, hu
group ar, group da, en, fir,
Embedding dis, (FT)
group en, es, fir, pt,
group
Typological
group 2
group ga, fi,
group 3
hi, ko, ja. zh
group id, ta
group
fr. hi, it,
group bg. es, pt, ro, hu, he
group he, da, de, ru, ko ja. zh yo
group da, cle, fi. et
Embedding dis. (PLM)
group 2
group en, es, fr,, hi, it, id
GradSim (ours)
group fä, pl. el
group 3
group hr;
group 0
ar, en, es, fr, hi, it, sw
group no,
he, da, de, ru, ko ja, group yo
group 1
Embedding dis. (FT)
group 2
Table 14: Language grouping results the Universal
Dependency POS tagging dataset.
group 0
ar, en, es, sw, yo de, ko, ja fr, hi, it,
group 1
GradSim (ours)
group 2
group 3 zh
Table 13: Language grouping results on the WikiAnn dataset. The languages we study covers 6 language families, while we use K 4 numbers of groups for our experiment following prior work Shaffer (2021) for a fair comparison.

---

Languages
Keywords
am
flower, city, season, a matter, government, discussion, december, press release, district, information, administration, public, govemmeen, the racist, man, poison
dz
advantage, god, my mind, welcome, my lord, dramatically, from, i swear, the people. sugar, complain, we manage, overwhelming. was overwhelmed. wicked, the incident, need, cash, poor
ha
ameen, safe, bring, amen, lord, bless you, fight, money, now, ai, whether, month, nonsense
ig
love. good, bless, husband, god, happy. birthday, leader, glory. king, thank you, sustain, glory. jesus, money, people, crazy, power, mouth, people, problem, dug, picces, down, stupid, shut up. devil, fear, kill, call, poison, anger
kr
very, good, Rwanda, god, good, peace, day, best, thank you, lord, together, comfort, under, discus- sion, news, plan, president, problem, person', bad, child, now, woman
ma
god, justice, for you, thanks, upon you, amazing, regards, good luck, congrats, blessed, development, one thousand, cave, between, facing, good, awake, president, facing, good, awake, president, special, causes, minister, film, desert, causes, minister, film, desert
pem
you, love, fine, god, sweet, my, baby, fit, bless, thank, today, hustle, rush, happy, enjoy, like, even, life, no, say. like, person, people, pain, go, even
say, like, them, again, person, want, people. you all, use, pain, why, god, lord, eternal, gospel. sin, Christ, church, earth, repentunce, Jesus, name, message, June, worse, nothing, people, unfortunately, problem
sw
god, thank you, major, national, minister, better, package, service, continue, dr, education, citizens, news, world, construction, people, region, police, state, president, father, army
ts
mozambique, listen, wake up, awake, live, conform, sugar, home, lake, leave, speed, connect, come
mother, god, thousand, money kill, sleep, yes, little, even, why, who, stop, team, good, father, word, say, that
yo
all, god, give, good, day, lord, come, amen, how, that, know, put, son, word, may, want, who, her, become, go
Table 15: Keyword extraction results.


Reviews:
[{"review_id": "vlTec8O4yX", "reviewer": "Reviewer_J9J7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing our paper and the recognition of the work’s contribution. Regarding your comment:\n\n- **“The authors' experiments were conducted only on text categorization and sequence annotation tasks, lacking experimental comparisons with other e.g. reading comprehension, QA tasks.”**\n  \n  For this work, we chose to ensure depth and robustness on the three selected tasks. Conducting comprehensive experiments across multiple NLP tasks would require significantly more resources and time. Despite our experiments' specificity, the methodology we introduced can certainly be adapted or extended to other tasks like reading comprehension or QA. We believe our findings serve as a valuable foundation on which future research on other multilingual tasks can be built. \n\nThanks for bringing up this point, which is valuable to further improve and extend our work. Please let us know if you have any further comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They find that the topic distribution of the training data heavily influences multilingual training, and that the lower layers of the converter model encode language-specific features while the higher layers capture task-specific information. Their  method shows superior performance compared to a variety of baseline methods for language grouping on three diverse datasets and, in particular, sets the new state of the art on a multilingual sentiment analysis benchmark dataset consisting of low-resource African languages.", "reasons_to_reject": "The authors' experiments were conducted only on text categorization and sequence annotation tasks, lacking experimental comparisons with other e.g. reading comprehension, QA tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose GradSim, a language grouping method based on gradient similarity.They find that the topic distribution of the training data heavily influences multilingual training, and that the lower layers of the converter model encode language-specific features while the higher layers capture task-specific information. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bmeW3LjY1W", "reviewer": "Reviewer_VWWR", "summary": "", "questions": "- Table 5 looks at the AfriSenti dataset only. I understand the space constraints, but to what extent do these correlations hold across the other tasks that you looked at? \n- are all of the averaged F-1 scores micro- or macro-averaged across languages? ", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and valuable feedback! We are encouraged by your recognition of the value and contribution of our work. We provide answers to your comments and questions below.\n\n- **“improvements on most language pairs for the three tasks are relatively small”**\n\n  GradSim aims at finding the best language groups from a global perspective, i.e., it achieves the highest possible overall performance across all languages rather than on each individual language. \n\n  On Afrisenti, we have achieved an overall improvement of ~ 2 F1 points over the second best methods (embedding-based language grouping). For sequence tagging tasks (NER and POS tagging), the improvement is relatively small as the baseline methods already achieve high performance. \n\n  We have run significance tests and proved that the performance improvement from our approach is statistically significant on most languages (32 out of 37) across the three datasets, detailed results are given in the response to Reviewer 5L1B. We will add these more detailed results in the updated version of our paper.\n    \n  \n- **Q1: “Table 5 looks at the AfriSenti dataset only. I understand the space constraints, but to what extent do these correlations hold across the other tasks that you looked at?\"**\n\n  Thanks for bringing this up. We agree that doing this correlation analysis on other datasets as well would give some interesting information. To address your concerns, we will add the correlation analysis of the other two datasets in our subsequent paper version. We will make use of the additional page in the camera-ready version to do so.\n\n- **Q2: “are all of the averaged F-1 scores micro- or macro-averaged across languages?”**\n\n  For AfriSenti, We use the macro-averaged F1 across languages as the averaged performance, to be consistent with the dataset paper [1]. For the other two datasets (WikiAnn and UD POS tagging), we report the micro-averaged F1 following previous work [2, 3] for a fair comparison. We will clarify this in Section 4.1 in the updated paper.\n\n  [1] Muhammad, Shamsuddeen Hassan, et al. \"Afrisenti: A twitter sentiment analysis benchmark for african languages.\" arXiv preprint arXiv:2302.08956 (2023).\n\n  [2] Shaffer, Kyle. \"Language clustering for multilingual named entity recognition.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.\n\n  [3] Yasunaga, Michihiro, Jungo Kasai, and Dragomir Radev. \"Robust multilingual part-of-speech tagging via adversarial training.\" arXiv preprint arXiv:1711.04903 (2017).\n\n- **Typos**\n\n  Thanks for pointing this out, we will correct that in the updated version of our paper.\n\nWe are grateful for your comments and suggestions, which are indeed valuable to improving the work. Hope our explanation addresses your concerns. Please let us know if you have any further comments. \n\nWe appreciate the positive feedback you provided and would be grateful if you could share your insights and in-depth understanding of our paper's strengths during the reviewer rebuttal phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- analysis is comprehensive and convincing re: a) incorporation of topical information via their method and b) which layers of the model correspond to linguistic features and which ones correspond to task-specific features. \n- small but consistent improvements over corresponding baselines on the tasks evaluated", "reasons_to_reject": "- improvements on most language pairs for the three tasks are relatively small", "questions_to_authors": "- Table 5 looks at the AfriSenti dataset only. I understand the space constraints, but to what extent do these correlations hold across the other tasks that you looked at? \n- are all of the averaged F-1 scores micro- or macro-averaged across languages? ", "paper_topic_and_main_contributions": "This paper proposes a simple step in pretraining multilngual embeddings, namely to cluster languages based on their gradient similarity, and train separate multilingual models for each grouping. Evaluating this new pretraining setup on downstream tasks (sentiment analysis on African languages, NER, POS tagging) a) generally (but not always) results in improvements across different languages compared to the baselines (and on AfriSenti, gets very close to the oracle upper bound result), b) in combination with task-adaptive pretraining, gets better than SoTA on AFriSenti, c) also results in improvements on NER and POS tagging (but less so for high-resource languages on the latter). They look at the correlation between the gradients from different encoder layers and other grouping methods, make the argument that topic information in the corpora is important (more so than typological and linguistic information) and show that their method takes advantage of this more effectively. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 479: I think you are referring to Table 7 in the text?"}, "extra_text": {}}, {"review_id": "QCe4QZK5YQ", "reviewer": "Reviewer_5L1B", "summary": "", "questions": "A. Can your method also be applied for pre-training?\n\nB. Did you consider assigning some languages to multiple groups? For instance, when they have gradients similar to many languages in different groups?\n\nC. Regarding the finding iv) in 5.2. The difference between layers in correlations with task performance and typological distance may be the result of fine-tuning. When computing gradients, were the layers of the encoder frozen or updated in this experiment? I would suggest doing the former in an attempt to interpret the information encoded throughout the layers.\n\nD. Why the results for the embedding-based baseline are not shown in Table 4? It looks like there’s enough space to add them.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments and suggestions. To address your concerns and questions:\n\n- **“The authors show neither standard deviations nor the significance of differences between the analyzed methods”**\n\n  Thanks for pointing this out. To address your concerns, we have run permutation-based significance tests [1] and included the standard deviation results on all three dataset. \n\n  For the significance test, we did the test between GradSim and the second best system: on AfriSenti and UD POS tagging, the second best system is the embedding (finetune)-based model, on WikiAnn is the model of prior work [2]. \n\n  Results are shown below, **bold languages** represent languages on which GradSim outperforms the second best system (refer to Table 1, 3, 4 in the paper), and the ***mark** represents the language with significance $p\\ value<0.05$. On most languages where GradSim performs better, we found statistically significant differences (32 out of 37 languages), indicating that GradSim is significantly better than the second best.\n\n  | AfriSenti | | | | | | | |\n  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|---|\n  | **Language** | **overall*** | **am*** | **dz** | **ha** | **ig*** | **kr*** | **ma*** |\n  | **p-value** | 0.0001 | 0.0001 | 0.0549 | 0.0884 | 0.0304 | 0.0001 | 0.0001 |\n  | **Language** | pcm | **pt*** | **sw*** | **ts** | twi | **yo*** |  |\n  | **p-value** | 0.1925 | 0.0035 | 0.0005 | 0.2662 | 0.8890 | 0.0291 |  |\n\n  | WikiAnn | | | | | | | | | |\n  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n  | **Language** | **overall*** | **overall-high*** | **ar*** | he | **da*** | **de** | en | **es*** | fr |\n  | **p-value** | 0.0294 | 0.0472 | 0.0343 | 0.8359 | 0.0142 | 0.0050 | 0.7589 | 0.0080 | 0.9739 |\n  | **Language** | **hi*** | it | **ru*** | ko | **ja*** | zh | **overall-low*** | **sw*** | **yo*** |\n  | **p-value** | 0.0488 | 0.0501 | 0.0451 | 0.9999 | 0.0316 | 0.6555 | 0.0400 | 0.0451 | 0.0045 |\n\n  | UD POS | | | | | | | | | | |\n  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|---|\n  | **Language** | **overall*** | **overall-high** | **bg*** | **cs*** | **da*** | de | en | es | **eu** | **fa*** |\n  | **p-value** | 0.0084 | 0.1455 | 0.0017 | 0.0428 | 0.0172 | 0.8261 | 0.7643 | 0.6461 | 0.1289 | 0.0305 |\n  | **Language** | **fi*** | **fr*** | **he*** | hi | hr | id | **it*** | **nl*** | **no*** | **pl*** |\n  | **p-value** | 0.0376 | 0.0463 | 0.0451 | 0.6106 | 0.7896 | 0.5611 | 0.0268 | 0.0468 | 0.0218 | 0.0482 |\n  | **Language** | **pt*** | **sl*** | sv | **overall-low*** | **el*** | **et*** | **ga*** | hu | ro | **ta*** |\n  | **p-value** | 0.0469 | 0.0228 | 0.7268 | 0.0202 | 0.0428 | 0.0351 | 0.0478 | 0.8049 | 0.8228 | 0.0401 |\n\n  Additionally, we also calculate the standard deviation of GradSim and other baseline models. We give the results of GradSim and the best baseline model here, so as not to take up too much space. We will revise our paper to include standard deviations and significance tests between the analyzed methods.\n\n  | AfriSenti | | | | | | | |\n  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n  | **Method** | **overall** | **am** | **dz** | **ha** | **ig** | **kr** | **ma** |\n  | **GradSim** | **71.34±0.16** | **66.11±2.50** | **67.90±1.23** | **79.97±0.68** | **79.55±0.57** | **72.12±0.94** | **53.68±1.78** |\n  | **Embedding(FT)** | 69.62±0.81 | 59.09±5.45 | 67.36±1.28 | 79.78±0.37 | 78.71±0.44 | 69.79±1.57 | 50.01±1.55 |\n  | **Method** | **pcm** | **pt** | **sw** | **ts** | **twi** | **yo** |  |\n  | **GradSim** | 68.40±0.5 | **72.30±0.36** | **63.05±1.09** | **50.36±1.82** | 63.46±2.86 | **75.09±0.51** |  |\n  | **Embedding(FT)** | **68.46±0.22** | 66.43±2.77 | 62.62±0.40 | 49.20±1.80 | **64.01±2.28** | 75.07±0.16 |  |\n  \n  | WikiAnn | | | | | | | | |\n  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n  | **Method** | **overall** | **ar** | **he** | **da** | **de** | **en** | **es** | **fr** |\n  | **GradSim** | **86.39±0.28** | **88.02±0.29** | 84.06±0.19 | **91.65±0.18** | **87.27±0.15** | 83.31±0.23 | **90.85±0.09** | 89.54±0.12 | \n  | **Prior Work** | 85.90±(-) | 85.25±0.23 | **84.83±0.16** | 90.49±0.09 | 85.73±0.21 | **83.37±0.06** | 89.90±0.21 | **89.79±0.14** | \n  | **Language** | **hi** | **it** | **ru** | **ko** | **ja** | **zh** | **sw** | **yo** |\n  | **GradSim** | **88.25±0.32** | **90.72±0.11** | **88.70±0.18** | 85.92±0.40 | **75.56±4.12** | 75.49±0.08 | **90.22±0.20** | **86.22±0.33** |\n  | **Prior Work** | 87.17±0.42 | 90.52±0.16 | 88.55±0.19 | **86.91±0.18** | 71.40±0.38 | **79.12±0.24** | 90.13±0.43 | 85.33±0.79 | \n\n  (Results of prior work are taken from [2] where the overall standard deviation is not given.)\n\n  | UD POS tagging | | | | | | | | \n  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n  | **Method** | **overall** | **bg** | **cs** | **da** | **de** | **en** | **es** |\n  | **GradSim** | **96.74±0.06** | **99.40±0.03** | **99.01±0.01** | **98.55±0.14** | 94.43±0.07 | 97.18±0.09 | 97.21±0.11 |\n  | **Embedding(FT)** | 96.66±0.03 | 99.37±0.04 | 98.99±0.02 | 98.09±0.08 | **94.56±0.08** | **97.26±0.04** | **97.25±0.10** |\n  | **Method** | **eu** | **fa** | **fi** |  **fr** | **he** | **hi** | **hr** | \n  | **GradSim** | **96.09±0.09** | **97.41±0.11** | **97.70±0.07** | **96.67±0.11** | **97.43±0.04** | 97.55±0.05 | 97.57±0.08 |\n  | **Embedding(FT)** | 96.08±0.06 | 97.33±0.07 | 97.57±0.15 | 96.44±0.11| 97.31±0.04 | **97.62±0.06** | **97.67±0.06** |\n  | **Method** | **id** | **it** | **nl** | **no** | **pl** | **pt** |  **sl** | \n  | **GradSim** | 91.20±0.12 | **98.58±0.04** | **93.88±0.20** | **99.01±0.03** | **98.52±0.05** | **98.54±0.07** | **99.02±0.04** | \n  | **Embedding(FT)** | **91.60±0.13** | 98.52±0.04 | 93.78±0.06 | 98.95±0.03 | 98.49±0.04 | 98.44±0.05 | 98.97±0.03 |\n  | **Method** | **sv** | **el** | **et** | **ga** | **hu** | **ro** | **ta** |\n  | **GradSim** | 98.81±0.01 | **98.59±0.06** | **95.78±0.17** | **93.52±0.27** | 96.94±0.07 | 95.14±0.25 | **88.32±0.17** |\n  | **Embedding(FT)** | **98.84±0.04** | 98.56±0.03 | 95.28±0.34 | 93.02±0.19 | **97.17±0.18** | **95.67±0.29** | 86.81±0.76 |\n\n  [1] Dror, Rotem, et al. \"The hitchhiker’s guide to testing statistical significance in natural language processing.\" Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: Long papers). 2018.\n\n  [2] Shaffer, Kyle. \"Language clustering for multilingual named entity recognition.\" Findings of the Association for Computational Linguistics: EMNLP 2021. 2021.\n\n\n- **“I don’t agree with the author’s claim that the method is data agnostic (claimed in L520). Clearly, training data for downstream tasks are required to compute language-specific gradients.”**\n\n  Please note that **we didn’t claim ‘data agnosticism’ in the paper** (neither in L520 nor anywhere else).  In L520, we state that “Our proposed gradient-based approach for grouping languages is a pure model- based approach, thus, does not require any prior knowledge about the language, task or data.” Our statement is correct that GradSim is a pure model-based approach: **it can be used even if there is no prior knowledge available about the application setting (task, domain, data etc),** in contrast to grouping methods based on linguistics similarity which require prior knowledge about the training data, e.g., the language family information or typological features.\n\n  Our understanding of ‘data agnosticism’ is that: if a method is not limited to some specific types of data, the method is data-agnostic. **In that respect, GradSim is data-agnostic**, as it is not limited to multilingual data, but can also be applied to multi-task or multi-domain data. Extending our methodology to multi-task or multi-domain settings could be an interesting future work direction.\n\n- **“this baseline requires the same number of training steps (in N monolingual training instead of joint multilingual training). Admittedly, the “Oracle method” requires N x (N - 1) evaluation runs. Still, for the analyzed tasks, the evaluation runs are not computationally demanding.”**\n\n  From the number of evaluation runs, it’s a **$1$ VS $N \\times N$** comparison between GradSim and the cross-lingual transfer oracle. \n\n  To measure the gradient similarities across languages, GradSim only required **$1$** joint multilingual training rather than N individual monolingual trainings. We measure the gradients wrt. each language along the optimization process (As described in Section 3.1, L177-L188). \n\n  However, the oracle method always requires **$N \\times N$** runs (not $N \\times (N-1)$) for $N$ languages. Specifically, taking each language as a target language, we select each of all N languages as the source language (also including the target language itself, therefore it is $N \\times N$ rather than $N \\times (N-1))$ to form a (source, target) language pair and run a source-to-target cross-lingual transfer experiment. We then use cross-lingual transfer results as the similarity measure between languages for language grouping.\n\n   In practice, on the AfriSenti dataset with 12 languages,  the joint multilingual training of **GradSim takes ~1 hour**, while **the oracle method takes ~17 hours** with the same experimental setup (computational resource, batch size, training epoch etc.). The oracle method will become extremely computationally expensive given a larger number of languages, while GradSim is far more computationally efficient than the oracle method. \n\n- **“Until section 4, it was not clear that the method was applied for fine-tuning instead of pre-training. Using the term “training” is giving a false impression that it’s the latter.”**\n\n  Thanks for your comment, we will clarify this in the updated version of our paper to avoid confusion.\n\n- **Q1: “Can your method also be applied for pre-training?”**\n\n  **Yes, our method can be applied for pretraining.** Based on the same steps described in Section 3, we can first calculate gradient similarities across languages with joint multilingual pretraining (Step 1), then determine the best K language groups based on the similarity measure from the last step (Step 2). Finally, we can pretrain one language model on each language group (Step 3) which can be deployed for downstream fine-tuning tasks. This could be an interesting future work direction.\n\n- **Q2: “Did you consider assigning some languages to multiple groups? ”**\n\n  **Yes, our method is able to assign languages to multiple groups** whenever it's beneficial. For detailed description please refer to Section 3.2 (L206) and 3.3 (L224).\n\n- **Q3: “When computing gradients, were the layers of the encoder frozen or updated in this experiment? ”**\n\n  The encoder layers are updated while computing gradients in our experiments, **following the normal LM fine-tuning setup.** We agree that the difference between layers in correlations with task performance and typological distance may be the result of fine-tuning, which is not a problem from our point of view. In the end we want to study how the deployed model for inference (the encoder of which is fine-tuned) encodes the information. \n\n  We ran new experiments with the frozen encoder and found the overall tendency of correlation across layers holds, i.e., lower layers of transformer models encode language-specific features while higher layers capture task-specific information. Below are the results of the correlation analysis:\n\n  |  | Encoder frozen |  |  | Encoder finetuned |  |  |\n  |:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n  | | Transfer score | Typological distance | Topic similarity | Transfer score | Typological distance | Topic similarity |\n  | Classifier | 0.6817 | 0.3483 | 0.4163 | 0.6963 | 0.3944 | 0.4749 |\n  | Encoder layer-23 | 0.7207 | 0.3851 | 0.0683 | 0.6485 | 0.6377 | 0.1486 |\n  | Encoder layer-21 | 0.7183 | 0.4581 | 0.0916 | 0.5526 | 0.7811 | 0.1134 |\n  | Encoder layer-18 | 0.6810 | 0.6623 | 0.1691 | 0.4462 | 0.8181 | -0.0601 |\n  | Encoder layer-15 | 0.4458 | 0.8101 | 0.0723 | 0.4602 | 0.8329 | 0.1083 |\n  | Encoder layer-12 | 0.4507 | 0.8400 | 0.0362 | 0.4532 | 0.8566 | 0.0362 |\n  | Encoder layer-6 | 0.4541 | 0.8358 | -0.1893 | 0.4542 | 0.8586 | -0.0342 |\n  | Encoder layer-0 | 0.4560 | 0.8458 | -0.1741 | 0.4526 | 0.8526 | -0.0721 |\n\n- **Q4: “Why the results for the embedding-based baseline are not shown in Table 4?”**\n  \n  We would have to add two more columns in Table 4 (using the embeddings from the pretrained model and fine-tuned model). Due to space reasons, we only show the results in the appendix. We will think of a way to visualize everything in the updated version.\n\n- **Missing references & Typos**\n\n  Thanks for pointing this out. We will add the missing references and correct the typos in the updated paper.  \n\nAgain, thank you for your detailed reviews, which are valuable to improving the work. We appreciate that you recognized the motivation of our work, the effectiveness of the proposed method and the value of our correlation analysis which are precisely the main contributions of our work. \n\nWe hope our explanation addresses your concerns. Please let us know if you have any further comments. We would appreciate it if you updated your review accordingly and can share your thoughts during the reviewer discussion given our responses to your questions .", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Well-motivated research problem about finding the optimum grouping of languages for multilingual training. Especially aimed to benefit low-resource languages.\n\n- I appreciate the analysis exploring the details of knowledge transferability to explain the proposed method's effectiveness.", "reasons_to_reject": "- The authors show neither standard deviations nor the significance of differences between the analyzed methods. It’s suspicious, especially given that the experiments were run with 5 random seeds (claimed in L287)\n\n- I don’t agree with the author’s claim that the method is data agnostic (claimed in L520). Clearly, training data for downstream tasks are required to compute language-specific gradients.\n\n- The authors state that their method is more efficient than an “Oracle method” based on a cross-lingual transfer map. However, this baseline requires the same number of training steps (in N monolingual training instead of joint multilingual training). Admittedly, the “Oracle method” requires N x (N - 1) evaluation runs. Still, for the analyzed tasks, the evaluation runs are not computationally demanding. \n\n- The introduction of the methodology is opaque to me. Until section 4, it was not clear that the method was applied for fine-tuning instead of pre-training. Using the term “training” is giving a false impression that it’s the latter.", "questions_to_authors": "A. Can your method also be applied for pre-training?\n\nB. Did you consider assigning some languages to multiple groups? For instance, when they have gradients similar to many languages in different groups?\n\nC. Regarding the finding iv) in 5.2. The difference between layers in correlations with task performance and typological distance may be the result of fine-tuning. When computing gradients, were the layers of the encoder frozen or updated in this experiment? I would suggest doing the former in an attempt to interpret the information encoded throughout the layers.\n\nD. Why the results for the embedding-based baseline are not shown in Table 4? It looks like there’s enough space to add them.", "paper_topic_and_main_contributions": "The past results have shown that in the face of scarcity of in-language data for downstream tasks, it’s beneficial to fine-tune the model with the data in multiple languages.\n\nThis work introduces a method for selecting language groups for multilingual training. The first step of the approach is multilingual training on all available data. The main step is computing average gradients from the backpropagation of training examples in specific languages. Then similarity between languages is defined as cosine between their averaged gradients. Lastly, a predefined number of language groups is obtained by solving the set cover problem (aimed to minimize similarity between languages in each group).\n\nThe authors show that their method of fine-tuning is competitive for low-resource languages in different downstream tasks: AfriSenti, WikiAnn, and UD. The paper addresses the important issue of effectively utilizing cross-lingual transfer for low-resource languages. However, as I read it, the authors overstate the benefits of their solution. Based on the presented results, I suspect that the presented method may not be significantly better than the baselines.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L167 “Du to”  -> “Due to”\nL319 “data peculiarities” is not the best word choice IMO\nL411 “in the appendix” -> “in Appendix”"}, "extra_text": {"Missing_References": "L046 \nA few previous works analyzed the impact of typological features on cross-lingual transfer, e.g., Pires et al. 2019\n\nL515\nMalkin et al. 2022, explore other aspects affecting cross-lingual transfer: sharing writing-system and pre-training data size."}}]

Author rebuttal:

Meta-review: This work proposes a method to group languages in multilingual tasks, e.g., text classification and sequence labeling, by leveraging the gradient similarity. The languages are clustered according to the metric, and a model is trained for each cluster.

Strengths
* The proposed method is well-motivated with comprehensive analyses on the impact of topical distributions and representation in layers.
* Experiments were carried out on various tasks with consistent gains especially targeting low-resource language settings.

Weaknesses
* The gains are small, and they have not experimented on other tasks, such as natural language understanding.
* Some clarity issues in the current manuscript, which could be addressed in the revised one without changing the conclusion.
Decision: Accept-Main