# Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languages

Quang Phuoc Nguyen, David Anugraha, Felix Gaschi, Jun Bin Cheng, En-Shiun Annie Lee

► To cite this version:

Quang Phuoc Nguyen, David Anugraha, Felix Gaschi, Jun Bin Cheng, En-Shiun Annie Lee. Rethinking what Matters: Effective and Robust Multilingual Realignment for Low-Resource Languages. IJCNLP - AACL, Dec 2025, Mumbai (Bombay), India. pp.1877-1905. hal-05511569

HAL Id: hal-05511569

https://hal.science/hal-05511569v1

Submitted on 15 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Rethinking what Matters: Effective and Robust
Multilingual Realignment for Low-Resource Languages

Quang Phuoc Nguyen<fnref n="1" />*, David Anugraha²*, Felix Gaschi³*,
Jun Bin Cheng<fnref n="1" />, En-Shiun Annie Lee<fnref n="1" />,⁴

¹Ontario Tech University ²Stanford University ³SAS Posos ⁴University of Toronto
quangphuoc.nguyen@ontariotechu.net, david.anugraha@stanford.edu,
felix@posos.fr

**Abstract**

Realignment is a promising strategy to improve cross-lingual transfer in multilingual language models. However, empirical results are mixed and often unreliable, particularly for typologically distant or low-resource languages (LRLs) compared to English. Moreover, word realignment tools often rely on high-quality parallel data, which can be scarce or noisy for many LRLs. In this work, we conduct an extensive empirical study to investigate whether realignment truly benefits from using all available languages, or if strategically selected subsets can offer comparable or even improved cross-lingual transfer, and study the impact on LRLs. Our controlled experiments show that realignment can be particularly effective for LRLs and that using carefully selected, linguistically diverse subsets can match full multilingual alignment, and even outperform it for unseen LRLs. This indicates that effective realignment does not require exhaustive language coverage and can reduce data collection overhead, while remaining both efficient and robust when guided by informed language selection.<fnref n="1" />

# 1 Introduction

Multilingual pre-trained language models such as mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) enable cross-lingual transfer, where models fine-tuned to a certain task with an English dataset can be generalized to the same task in other languages (Pires et al., 2019; Wu and Dredze, 2019). However, their performance often degrades for typologically distant languages, such as low-resource languages (LRLs) (Pires et al., 2019). A promising strategy to address this issue is to perform realignment, which explicitly re-trains models to produce similar representations for translated sentence pairs using objectives inspired

by multilingual word embeddings (Conneau et al., 2017; Artetxe et al., 2018).

Despite a strong correlation between alignment and cross-lingual transfer (Gaschi et al., 2023), results from realignment methods remain mixed. While some studies report benefits of realignment (Cao et al., 2020; Zhao et al., 2020), others observe limited or even negative effects (Wu and Dredze, 2020; Efimov et al., 2023). These findings align with previous observations, where multilingual models exhibit good alignment for closely related languages, but remain more misaligned for distant or LRLs (Dou and Neubig, 2021).

In addition, realignment is not always feasible for all languages. It requires high-quality translation data, which may be unavailable for many LRLs (Gu et al., 2018; Liu et al., 2021; Anugraha et al., 2024). Even when resources exist, alignment quality can vary significantly across languages, potentially degrading downstream performance. This raises a central question: **Do we need to use all available languages for a better realignment, or could a carefully selected subset of languages offer similar or improved cross-lingual transfer performance?**

In this work, we conduct an extensive empirical study to investigate whether realignment truly benefits from using all available languages, or if strategically selected subsets can offer comparable or even improved cross-lingual transfer. In summary, our key contributions are:

1. We conduct the first large-scale, systematic evaluation of realignment across 65 languages, including 29 LRLs, 3 tasks, 4 seeds, and 2 models (with a strong focus on low-resource scenarios). By introducing a sentence-level averaging and contrastive objective that removes the need for word aligners, we show significant gains of up to 10 points in cross-lingual transfer, especially for

<footnote n="1">¹Equal contribution.</footnote>

<footnote n="1">¹Our code can be found at https://github.com/felixgaschi/multilingual-alignment-and-transfer.</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: Overall diagram of the realignment process. Our goal is to empirically investigate how language selection within the realignment dataset impacts overall downstream task performance.</figcaption></figure>

LRLs unseen during pre-training.

2. We systematically investigate language subset selection for efficiency, demonstrating that informed subsets chosen via heuristics (like URIEL featural diversity) can match or surpass full multilingual realignment. This shows that linguistic diversity matters more than the sheer number of languages.

3. We perform comprehensive ablation studies, including out-of-distribution robustness. We evaluate on unseen, out-of-distribution benchmarks (e.g., AmericasNLI) to show that diverse subset selection generalizes effectively. We also conduct ablations by scaling the number of languages and varying initial language pools to reflect realistic resource constraints, showcasing the importance of including LRLs in realignment.

To the best of our knowledge, we are the first to evaluate realignment massively on truly LRLs.

## 2 Methodology

Recall that we perform realignment to explicitly retrain multilingual encoders to produce similar representations for translated sentence pairs. In particular, we first perform realignment as a separate training phase, which is then followed by full-model fine-tuning on a downstream task, following previous work Wu and Dredze (2020); Gaschi et al. (2023); Bakos et al. (2025). For the realignment

phase, we adopt the method proposed by Wu and Dredze (2020), which modifies the encoder to produce similar representations for semantically equivalent words across languages. This is achieved using a contrastive loss applied to word-level alignment pairs extracted from parallel corpora.

Prior work has typically relied on extracting word pairs from parallel sentences using word aligners such as FastAlign (Dyer et al., 2013) or bilingual dictionaries Gaschi et al. (2023). However, these alignment resources are often unreliable or entirely unavailable, especially for LRLs, and their use typically requires substantial computational resources. Thus, we propose a simple alternative that removes the dependency on word aligners while requiring significantly less time and computational resources. Our method instead averages the representations of words in each sentence of a translation pair and directly minimizes the distance between these sentence-level representations.

Formally, let $B$ denote the batch size, and let $H = \{(h_i, \tilde{h}_i)\}_{i=1}^B$ represent a batch of $B$ aligned sentences, where $h_i$ is the averaged embedding of the words in a source (e.g., English) sentence and $\tilde{h}_i$ is the embedding of its aligned counterpart in the target language. The goal is to bring $h_i$ and $\tilde{h}_i$ closer together in the embedding space while pushing $h_i$ away from all other unaligned sentences in the batch. This is achieved via the following contrastive loss:
$$ \mathcal{L}(\theta) = \frac{1}{2B} \sum_{h \in H} \log \frac{\exp(\text{sim}(h, \text{aligned}(h))/T)}{\sum_{h' \in H, h' \neq h} \exp(\text{sim}(h, h')/T)} \quad (1) $$

where sim(h, h') denotes cosine similarity between two representations and T is a temperature hyperparameter, set to 0.1 in our experiments.

Note that the contrastive loss defined above implicitly depends on the translation data used, particularly the set of languages involved. Prior work typically performs realignment using all available languages for their parallel data (Wu and Dredze, 2020; Gaschi et al., 2023; Bakos et al., 2025). In contrast, we hypothesize that a carefully selected subset of languages may suffice to achieve comparable or even improved downstream cross-lingual generalization.

Formally, let $L = \{\ell_1, \ell_2, \dots, \ell_m\}$ be the full set of languages for which parallel corpora with English are available. Let $\mathcal{D}_S$ denote the parallel data involving English and the languages in subset $S \subseteq L$, and let $\mathcal{M}_S$ denote the model after realignment using this data. Let $\mathcal{T} = \{T_1, T_2, \dots, T_k\}$ be a set of downstream tasks, and for each task $T_i \in \mathcal{T}$, we fine-tune the realigned model $\mathcal{M}_S$ on task-specific supervision to obtain the fine-tuned model $\mathcal{M}_S^{T_i}$. We then compute the corresponding evaluation score $\text{Score}_{T_i}(\mathcal{M}_S^{T_i})$. Overall, our goal is to find the subset $S^* \subseteq L$ that maximizes the macro-average across all downstream tasks:<fnref n="2" />

$$ S^* = \arg \max_{S \subseteq L} \frac{1}{|\mathcal{T}|} \sum_{T_i \in \mathcal{T}} \text{Score}_{T_i}(\mathcal{M}_S^{T_i}) \quad (2) $$

Since the downstream evaluation metric is non-differentiable and trying all possible subsets of $L$ is expensive, we do not optimize this objective directly. Instead, we construct subsets using linguistic-motivated heuristics as will be described in Section 3.1.

# 3 Experimental Setup

## 3.1 Language Subsets

We construct and evaluate subsets of languages based on heuristics designed to capture different dimensions of cross-lingual diversity and coverage.

These heuristics consider factors such as linguistic feature diversity, language family affiliation, and script variation. To assess the effectiveness of these heuristics, we also compare their performance against randomly selected realignment languages, thereby evaluating the significance of each heuristic.

All subsets are drawn from the same pool of 65 languages, which we denote as $L_{65}$. This pool consists of 47 languages from XTREME-R (Ruder et al., 2021) together with 21 additional African languages, with some overlap between the two groups. The sets of 21, 47, and 65 languages serve as our baseline subsets. Details of the languages in $L_{65}$ are provided in Table 5. For each heuristic, we evaluate subsets of size $n \in \{5, 10, 20, 40\}$, corresponding to increasing coverage when available.

**Baselines.** We include two types of baselines to contextualize the performance of our subset selection strategies. The first baseline uses the fixed language sets of size 21, 47, and 65 as mentioned above. The second baseline consists of random subsets sampled uniformly from $L_{65}$ with $n \in \{5, 10, 20, 40\}$. These baselines help distinguish the effect of informed linguistic heuristics from arbitrary selection. All random subsets are generated using fixed random seeds for reproducibility.

**Language Featural Diversity.** This heuristic aims to compute diversity for languages based on their structural linguistic features. These features are obtained using the URIEL+ database (Khan et al., 2025), which is a language vector resource that encodes languages based on typological, geographic, phonological, syntactic, and phonetic inventory feature vectors. These representations allow for the computation of pairwise distances between languages, using angular distance over their vectorized representations. We compare two types of subsets: (1) subsets where we have the most diverse set of languages from set $L_{65}$, and (2) subsets where we have the least diverse set of languages from set $L_{65}$. To construct our most diverse subsets, we select languages that maximize their pairwise featural distance from English, with English included in the subset calculation but not considered during realignment. We also constructed the least diverse subsets to contrast with the diverse case by minimizing the total pairwise featural distance. The formal definition of the objective can be found in Section A.1.

<footnote n="2">²This performance evaluation setup follows prior work in multi-task multilingual learning, such as by XTREME-R (Ruder et al., 2021).</footnote>
**Language Family Diversity.** This heuristic in-
vestigates whether diversity in genetic lineage con-
tributes to effective realignment. We compare two
types of subsets: (1) subsets where each language
comes from a distinct language family other than
the Indo-European family, and (2) subsets that are
restricted to a single language family, specifically,
Indo-European languages, to contrast with the di-
verse case.

**Script Diversity.** This heuristic investigates whether diversity in language scripts contributes to effective realignment. We compare three types of subsets: (1) subsets where each language is drawn from a distinct script other than Latin, (2) diverse subsets (as defined in Language Featural Diversity) but restricted to languages in *L*<sub>65</sub> that use only the Latin script, and (3) least diverse subsets restricted to the Latin script, serving as a contrast to the diverse case.

All the language subsets and their languages are
listed in the Appendix.

**3.2 Models and Datasets**

**Realignment dataset** We use OPUS-100 (Zhang et al., 2020) and NLLB (Costa-Jussà et al., 2022), which contain parallel corpora with sentence pairs across 100 and 200 languages, respectively. Whenever a language is not covered by OPUS-100, we fall back to the NLLB dataset.

**Training and Downstream Task Datasets** To evaluate cross-lingual transfer, we fine-tune all models exclusively on the English subset and evaluate them directly on other languages without additional fine-tuning. Our evaluation is mostly focused on *in-distribution* datasets, where evaluation languages are part of the realignment language set. However, we also included an *out-of-distribution* (*OOD*) dataset scenarios, which contain languages not seen during pre-training or realignment. Detailed dataset statistics are provided in Table 7.

**In-Distribution Datasets.** We consider three downstream tasks: Part-of-Speech (PoS) tagging, Named Entity Recognition (NER), and Natural Language Inference (NLI), evaluated on datasets covering both the XTREME-R language set and their African counterparts. For each task, we evaluate on datasets covering both the XTREME-R language set and African counterparts:

* For PoS tagging, we fine-tune on the UDPOS dataset (De Marneffe et al., 2021) and evaluate

on both UDPOS and MasakhaPOS (Dione et al., 2023).

* For NER, we fine-tune on the English subset of WikiANN (Pan et al., 2017) and evaluate on WikiANN and MasakhaNER (Adelani et al., 2022).

* For NLI, we fine-tune on the English subset of XNLI (Conneau et al., 2018) and evaluate on four datasets: XNLI, IndoNLI (Mahendra et al., 2021), Myanmar-XNLI (Htet and Dras, 2025a), and AfriXNLI (Adelani et al., 2024). For AfriXNLI, we restrict evaluation to its African languages.

**Out-of-Distribution Datasets.** To study generalization beyond the languages present during pre-training or realignment, we evaluate NLI performance on AmericasNLI (Ebrahimi et al., 2021), which covers 10 typologically diverse languages absent from both the pre-training and realignment language sets. This dataset serves as a challenging out-of-distribution benchmark to assess zero-shot cross-lingual transfer.

**Models** We use mBERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) as our multilingual pre-trained language models. All experiments are run on 4 different seeds. More details about the hyperparameters can be found in Section A.6.

# 4 Results and Analysis

**4.1 Results Overview**

Figure 2 presents a comparison of the best average performance across different tasks for both XLM-R and mBERT, evaluated under different language subset heuristics. Detailed per-task results are reported in Tables 8 and 10.

First, Figure 2 shows that realignment provides significantly better overall results than the fine-tuning baseline. Except for two selection methods, realignment provides at minimum a one-point improvement, indicating the benefits of having to perform realignment.

However, performing realignment using all lan-
guages is not necessary to achieve comparable or
even better performance. Figure 2 shows that the re-
alignment strategy based on URIEL featural diver-
sity and URIEL featural diversity within languages
with Latin script consistently yields the best results,
achieving performance comparable to using the full
set of 65 realignment languages. This demonstrates
<figure><img src="image_3.png" /><figcaption>Figure 2: Average performance across PoS Tagging, NER, and NLI for XLM-R and mBERT. The baselines are compared against the best-performing configuration from each language subset heuristic.</figcaption></figure>

that carefully selecting a smaller but linguistically diverse subset of languages can be as effective as, or even better than, using all languages. These two heuristics also outperform other baselines, including random selection, XTREME-R only, and African-only subsets.

Our results further highlight, across all heuristics, the least diverse subsets consistently underperform compared to their more diverse counterparts. Language subsets selected to maximize diversity in featural space outperform those that minimize such diversity across both models. Likewise, selecting languages from distinct families offers clear benefits over limiting realignment to a single family. These results show that realignment benefits from the inclusion of languages that provide diverse linguistic signals, probably because such signals help to anchor multilingual representations more robustly.

Finally, diversity affects realignment performance in different ways depending on the dimension of diversity considered. For example, diversity based on genetic lineage does not yield strong results, while selecting languages with distinct scripts, excluding Latin, produces the worst performance. This suggests that script diversity can be beneficial, but the absence of Latin script hurts alignment performance, likely due to English being the pre-training language.

## 4.2 Results Based on Language Resource Level

To assess how different realignment methods impact different languages in terms of the level of resources, we categorize the evaluation languages into four groups: high-resource languages (HRLs), medium-resource languages (MRLs), low-resource

languages (LRLs) that are seen during pre-training, and LRLs that are unseen during pre-training<fnref n="3" />. Figure 3 provides the detailed breakdown of overall performance for XLM-R and mBERT across different subsets of evaluation languages.

Realignment yields substantial gains for LRLs, particularly for languages unseen during pre-training. For both models, the best realignment configuration improves LRL-unseen performance by up to 10 points over standard fine-tuning. These results demonstrate that representation alignment is especially effective when cross-lingual transfer is weakest.

For HRLs and MRLs, the trends differ. Fine-tuning alone remains competitive on these languages, and applying realignment leads to slight performance drops. This pattern is consistent with prior findings that realignment benefits do not always extend to higher-resource languages (Wu and Dredze, 2020; Gaschi et al., 2023), which have stronger initial cross-lingual representations.

We also compare different strategies for selecting realignment languages. While language choice has little influence on HRLs or MRLs, it noticeably affects LRL performance. The advantage of diversity-based over random selection observed in aggregate results primarily stems from improvements on LRLs.

Overall, these results highlight a key insight: even though realignment may offer limited gains for HRLs and MRLs, it provides consistent and substantial improvements for LRLs, especially those absent from pre-training. This makes realignment a promising direction for extending multilingual

<footnote n="3">³HRLs = Joshi class 5, MRLs = 3 and 4, LRLs = 0, 1, and 2 (Joshi et al., 2020)</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 3: Heatmaps showing overall performance (averaged across four seeds) for different language subsets - HRLs, MRLs, and LRLs - seen and unseen during pre-training of XLM-R and mBERT. The fine-tuning only baseline remains strong for HRLs and MRLs, while realignment significantly improves performance on LRLs. Diversity-based language selection further amplifies these gains for LRLs.</figcaption></figure>

encoders to truly underrepresented languages.

## 4.3 Results on Out-of-Distribution Languages

To complement our in-distribution analysis, we further evaluate different realignment approaches on AmericasNLI, which contains LRLs that were not used for realignment.

Figure 4 shows that the results on LRLs unseen during realignment do not differ much from languages used for realignment. Similarly to Figure 2, realignment significantly outperforms the fine-tuning only baseline. Diversity-based language selection outperforms the random baseline, and their homegenous counterparts, with the exception of maximizing URIEL diversity within Latinscripted languages, which suggests that diversity should be enforced in all aspects (featural, script, and family).

One key difference with in-distribution results is that diversity-based selection, namely when using URIEL features, outperforms realignment on the entire set of available languages. Thus, when it comes to improving results across the board, including languages unseen during realignment, diversity might become more important than the number of languages involved.

As shown in Figure 4, realignment again substantially outperforms the fine-tuning baseline, mirroring the in-distribution trends. Between language subsets used for realignment, diversity-based selection continues to outperform both random selection and homogeneous subsets, with one exception: maximizing URIEL diversity within Latin-script languages does not provide the same advantage, suggesting that meaningful diversity must span features, scripts, and families rather than being constrained to a single script group. Furthermore, URIEL-based selection outperforms realignment

on the full set of languages, demonstrating that when the goal is broad cross-lingual improvement, including languages never seen during realignment, the type of diversity in the realignment set matters more than the number of languages it contains.

# 5 Language-Scaling Behavior of Realignment Methods

Figure 5 shows how performance changes as we scale the number of languages used in each subset-selection strategy for realignment, while keeping the total computational budget fixed.

Across the board, every realignment strategy improves over simple fine-tuning, even with only five languages, indicating that cross-lingual realignment is beneficial even at very small scales. Among selection strategies, subsets based on distinct families or distinct scripts generally lag behind random sampling, whereas URIEL-diverse and diverse Latin-script language subsets provide stronger gains. Interestingly, the diverse Latin-script language subsets exhibit a non-monotonic trajectory, dipping from 10 to 20 languages before rising again at larger scales, suggesting that mid-scale expansions can occasionally introduce detrimental interactions before recovering.

For XLM-R, most strategies plateau around 20 languages, implying that the model absorbs most of the transferable signal once moderate coverage is reached. The diverse Latin-script language selection strategy is an exception, since the performance increases again at 40 languages and reversing its earlier dip. This suggests that additional gains still exist at large scales for strategies other than Latin-script diversity, but other optimal subsets may exist given a different language selection strategy.

For mBERT, the scaling behavior is more grad-
<figure><img src="image_5.png" /><figcaption>Figure 4: Averaged out-of-distribution performance of XLM-R and mBERT on the AmericasNLI dataset, comparing different language selection heuristics against three realignment baselines and a fine-tuning-only baseline. Realignment with diversity-based language subsets outperforms both the realignment and fine-tuning-only baselines.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Figure 5: Scaling of average cross-lingual transfer performance with the number of languages used for realignment for XLM-R (left) and mBERT (right).</figcaption></figure>

ual and nearly linear. Several strategies, such as URIEL-based and diverse Latin-script language selection, even surpass the full 65-language baseline at intermediate scales. This indicates that mBERT continues to benefit from expanded cross-lingual supervision over a wider range than XLM-R, and that its saturation point occurs later.

Overall, realignment is consistently beneficial, and that URIEL-based diversity and diverse Latin-script language selection are the most reliable and data-efficient approaches across different number of languages. We also find that different models follow distinct scaling dynamics: XLM-R saturates early, whereas mBERT accumulates gains more steadily across larger language sets.

# 6 Ablation study

In this ablation study, we focus specifically on analyzing the impact of including languages of differ-

ent resource levels in the realignment mix. Specifically, we consider the case where only limited resources are available to collect high-quality parallel data for realignment. Our goal is to determine whether, under such constraints, lower-resource or unseen languages remain necessary for improving cross-lingual transfer, or if higher-resource languages can serve as effective substitutes. For the sake of clarity, rather than reporting results for all heuristics, we include only the random selection heuristic, alongside two baselines: fine-tuning only, and realignment using the entire $L_{65}$ set. Random language selection helps isolate the effect of different language pools on realignment performance, which is the main objective of this ablation study. Results for other selection heuristics can be found in Tables 9 and 11 in the Appendix.

We compare performance when randomly sampling 10 languages from different pools for realign-
<table><caption>Table 1: Performance of different realignment strategies for XLM-R and mBERT under a 10-language constraint. Only the random language selection strategy is shown. Bold indicates the highest result per task and model (excluding baselines). Standard deviation and results for other selection strategies are shown in Tables 9 and 11 in Appendix.</caption><thead><tr><th>Language Pool</th><th>POS</th><th>NLI</th><th>NER</th><th>Avg.</th></tr></thead><tbody><tr><td><strong>XLM-R</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Joshi 4 and 5</td><td>67.5</td><td>58.6</td><td>54.4</td><td>60.2</td></tr><tr><td>Joshi 3, 4, and 5</td><td>67.2</td><td>58.8</td><td>53.8</td><td>59.9</td></tr><tr><td>Joshi 3</td><td>67.0</td><td>58.9</td><td>51.0</td><td>59.0</td></tr><tr><td>Joshi 2</td><td>68.8</td><td><strong>59.9</strong></td><td><strong>54.5</strong></td><td><strong>61.1</strong></td></tr><tr><td>Unseen Languages</td><td><strong>69.1</strong></td><td>59.8</td><td>53.5</td><td>60.8</td></tr><tr><td>Seen Languages</td><td>67.1</td><td>58.8</td><td>53.3</td><td>59.7</td></tr><tr><td><em>Fine-tuning only</em></td><td>66.0</td><td>58.6</td><td>51.1</td><td>58.6</td></tr><tr><td><em>65 langs baseline</em></td><td>69.1</td><td>59.4</td><td>57.1</td><td>61.9</td></tr><tr><td><strong>mBERT</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Joshi 4 and 5</td><td>63.7</td><td>53.3</td><td>50.7</td><td>55.9</td></tr><tr><td>Joshi 3, 4, and 5</td><td>63.5</td><td>53.3</td><td>51.5</td><td>56.1</td></tr><tr><td>Joshi 3</td><td>62.9</td><td>53.2</td><td>50.3</td><td>55.5</td></tr><tr><td>Joshi 2</td><td><strong>65.1</strong></td><td><strong>55.0</strong></td><td><strong>52.5</strong></td><td><strong>57.5</strong></td></tr><tr><td>Unseen Languages</td><td>64.5</td><td>54.4</td><td><strong>52.5</strong></td><td>57.1</td></tr><tr><td>Seen Languages</td><td>63.7</td><td>53.4</td><td>51.2</td><td>56.1</td></tr><tr><td><em>Fine-tuning only</em></td><td>62.2</td><td>53.1</td><td>52.2</td><td>55.8</td></tr><tr><td><em>65 langs baseline</em></td><td>66.9</td><td>55.4</td><td>54.8</td><td>59.0</td></tr></tbody></table>

ment: higher-resource languages only (Joshi 4 and 5), MRLs only (Joshi 3), a mixed set of MRLs and HRLs (Joshi 3, 4 and 5), and languages either seen or unseen during pre-training. Our results from Table 1 show that in most cases, performing realignment on only 10 languages leads to improved cross-lingual transfer performance across tasks compared to fine-tuning alone. While using a reduced set of 10 languages for realignment does result in a performance drop relative to the 65-language baseline, the decrease is modest, ranging from 0.8% to 1.5%. This is a reasonable trade-off given the over sixfold reduction in the number of languages involved. On the other hand, comparisons among the ablation experiments reveal that language pools composed of Joshi Class 2 languages and unseen pretraining languages tend to yield better performance than other configurations. This highlights the importance of including LRLs and unseen languages in the realignment process to improve transfer on these same categories.

Our ablation results also indicate that other language pools - such as mid- to high-resource languages from Joshi Classes 4–5 in the case of XLM-R, as well as seen pretraining languages - can serve as practical substitutes for LRLs when the latter are not applicable. Although there is a performance drop, it remains relatively minor (less than 1%), while the availability of high-quality parallel data from these higher-resource language pools is considerably more likely.

## 7 Related Work

**Realignment Strategies** Realignment typically involves two components: the *alignment tool*, which identifies word correspondences between languages, and the *training strategy*, which updates model parameters to enforce alignment (Hämmerl et al., 2024). Different alignment tools can be used, such as the statistical FastAlign (Dyer et al., 2013) and the neural AwesomeAlign (Dou and Neubig, 2021). For our training strategy, we adopt the method proposed by Wu and Dredze (2020), which performs realignment to all layers by using a contrastive loss to word-level alignment pairs extracted from parallel corpora. Alternative training strategies include contrastive frameworks with different loss formulations (Chen et al., 2020), or architectural choices such as selectively realigning specific model layers (Bakos et al., 2025).

**Data Selection Strategies** Multiple works across machine learning research have been able to show that strategic data selection, rather than using all available data, can lead to better generalization, efficiency, and robustness (Wang and Neubig, 2019; Albalak et al., 2024; Liu et al., 2024; Anugraha et al., 2025b). In the context of cross-lingual transfer, prior work has shown that linguistic similarity between source and target languages is a strong predictor of transfer success, often outperforming naive or pivot-based strategies (Duong et al., 2015; Eronen et al., 2023). However, the optimal set of source languages depends heavily on the task, due to divergences in features like morphology, syntax, and script (Philippy et al., 2023). In contrast to methods like LangRank (Lin et al., 2019), which identify the best single transfer language per target language and downstream task, our work seeks to identify combinations of languages that optimize the average downstream performance across multiple target languages.
## 8 Conclusion

In this paper, we investigate whether realigning multilingual models with carefully selected language subsets can match or even surpass alignment using the full language set using linguistic-motivated heuristics. Our large-scale experiments demonstrated that realignment with a smaller language subsets often match the full set across models and tasks, especially when chosen based on their linguistic diversity, and evaluated on out-of-distribution languages. Moreover, our analysis shows that realignment most benefits LRLs, suggesting that realignment is particularly effective for languages whose embeddings are not yet well aligned. Our ablation studies further reveal that when the number of languages is limited, having LRLs in the language subset yields the strongest improvements, although HRLs and MRLs can still help enhance cross-lingual transfer. These results demonstrate that strategic language selection not only reduces computational and data overhead but can also strengthen multilingual generalization, pointing toward more efficient and targeted approaches to cross-lingual realignment.

## Limitations

In this paper, we demonstrate the importance of linguistic diversity as a more inclusive and effective approach to improving cross-lingual generalization in multilingual language models. Rather than simply collecting all available data, our work shows that carefully selecting diverse subsets of languages can enhance cross-lingual transfer, indicating that our work is both more efficient in terms of data collection overhead and more effective overall.

A limitation of this study is the absence decoder-only models, which are increasingly prevalent in current research. While the Appendix Section A.3 presents our tentative realignment results on Llama 3.1 8B using LoRA adapters, these results indicate that the same realignment method does not straightforwardly transfer to decoder-only architectures, highlighting the need for careful adaptation in such settings. Moreover, encoder-only architectures remain highly relevant for cross-lingual classification due to their efficiency, stable representations, and strong transfer capabilities. Recent developments, such as ModernBERT (Warner et al., 2025) and LLM2Vec (BehnamGhader et al., 2024), which adapt decoder-only models into encoder-style architectures, further highlight their enduring importance.

As shown in our small-scale experiment in Section A.4, encoder-based models can even outperform much larger multilingual decoder-only models on several classification tasks. Future work could also explore applying our algorithm-agnostic heuristics to decoder-only model fine-tuning for multilingual tasks (Anugraha et al., 2025a) or to reinforcement learning setups in multilingual contexts (Dang et al., 2024).

Another is that although averaging does not explicitly target word-level alignment, we empirically find that it provides results that are slightly lower but still comparable to FastAlign (Appendix A.2). Therefore, we opt for the averaging approach for practical reasons, as FastAlign is significantly more resource- and time-intensive.

Our exploration is also limited to heuristic-based language selection, although we have shown that such subsets do exist. A promising direction for future work is to move beyond heuristics by developing predictive algorithms that estimate downstream performance and dynamically determine both the optimal languages and the appropriate subset size (Anugraha et al., 2024).

We acknowledge that our full realignment language set, $L_{65}$, does not cover the entire spectrum of global linguistic diversity, despite covering many LRLs. We hope that our approach encourages the creation of parallel corpora for underrepresented languages, enabling greater diversity in alignment sets and fostering more inclusive multilingual models. Ultimately, we aim for our work to contribute toward broader and fairer access to language technologies, especially in the context of cross-lingual NLP research and deployment.

## Acknowledgements

This research on "Multilingual multicultural NLP and LLMs" was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant (RGPIN-2024-06887) and Discovery Launch Supplement (DGECR-2024-00008). The authors also acknowledge the computational resources and support provided by the Digital Research Alliance of Canada (formerly Compute Canada) through grant RRG no. 5397.

This project was provided with HPC computing and storage resources by GENCI at IDRIS thanks to the grant 2025-AD010316268 on the supercomputer Jean Zay's H100 partition.
## References

David Ifeoluwa Adelani, Graham Neubig, Sebastian Ruder, Shruti Rijhwani, Michael Beukman, Chester Palen-Michel, Constantine Lignos, Jesujoba O. Alabi, Shamsuddeen H. Muhammad, Peter Nabende, Cheikh M. Bamba Dione, Andiswa Bukula, Roowei-ther Mabuya, Bonaventure F. P. Dossou, Blessing Sibanda, Happy Buzaaba, Jonathan Mukiibi, Godson Kalipe, Derguene Mbaye, and 26 others. 2022. *MasakhaNER 2.0: Africa-centric transfer learning for named entity recognition*. In *Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing*, pages 4488–4508, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, and 1 others. 2024. *Irokobench: A new benchmark for african languages in the age of large language models*. *arXiv preprint arXiv:2406.03368*.

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. *MEGA: Multilingual evaluation of generative AI*. In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 4232–4267, Singapore. Association for Computational Linguistics.

Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Hae-won Jeong, and 1 others. 2024. A survey on data selection for language models. *arXiv preprint arXiv:2402.16827*.

David Anugraha, Shou-Yi Hung, Zilu Tang, Annie En-Shiun Lee, Derry Tanti Wijaya, and Genta Indra Winata. 2025a. mr3: Multilingual rubric-agnostic reward reasoning models. *arXiv preprint arXiv:2510.01146*.

David Anugraha, Zilu Tang, Lester James V Miranda, Hanyang Zhao, Mohammad Rifqi Farhansyah, Garry Kuwanto, Derry Wijaya, and Genta Indra Winata. 2025b. R3: Robust rubric-agnostic reward models. *arXiv preprint arXiv:2505.13388*.

David Anugraha, Genta Indra Winata, Chenyue Li, Patrick Amadeus Irawan, and En-Shiun Annie Lee. 2024. Proxylm: Predicting language model performance on multilingual tasks via proxy models. *arXiv preprint arXiv:2406.09334*.

Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. *arXiv preprint arXiv:1805.06297*.

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the Cross-lingual Transferability of Monolingual Representations. In *Proceedings of ACL 2020*.

Mikel Artetxe and Holger Schwenk. 2019. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. *Transactions of the ACL 2019*.

Steve Bakos, David Guzmán, Riddhi More, Kelly Chutong Li, Félix Gaschi, and En-Shiun Annie Lee. 2025. *AlignFreeze: Navigating the impact of realignment on the layers of multilingual models across diverse languages*. In *Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)*, pages 562–586, Albuquerque, New Mexico. Association for Computational Linguistics.

Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. *Llm2vec: Large language models are secretly powerful text encoders*. *Preprint*, arXiv:2404.05961.

Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. *arXiv preprint arXiv:2002.03518*.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. *Preprint*, arXiv:2002.05709.

Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. In *Transactions of the Association of Computational Linguistics*.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8440–8451, Online. Association for Computational Linguistics.

Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. *arXiv preprint arXiv:1710.04087*.

Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. *arXiv preprint arXiv:1809.05053*.

Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe
Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, and 1 others. 2022. No language left behind: Scaling human-centered machine translation. *arXiv preprint arXiv:2207.04672*.

John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. 2024. Rlhf can speak many languages: Unlocking multilingual preference optimization for llms. *arXiv preprint arXiv:2407.02552*.

Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. *Computational linguistics*, 47(2):255–308.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers)*, pages 4171–4186.

Cheikh M. Bamba Dione, David Ifeoluwa Adelani, Peter Nabende, Jesujoba Alabi, Thapelo Sindane, Happy Buzaaba, Shamsuddeen Hassan Muhammad, Chris Chinenye Emezue, Perez Ogayo, Anuoluwapo Aremu, Catherine Gitau, Derguene Mbaye, Jonathan Mukiibi, Blessing Sibanda, Bonaventure F. P. Dossou, Andiswa Bukula, Rooweither Mabuya, Allahsera Auguste Tapo, Edwin Munkoh-Buabeng, and 25 others. 2023. MasakhaPOS: Part-of-speech tagging for typologically diverse African languages. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 10883–10900, Toronto, Canada. Association for Computational Linguistics.

Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. *arXiv preprint arXiv:2101.08231*.

Long Duong, Trevor Cohn, Steven Bird, and Paul Cook. 2015. Cross-lingual transfer for unsupervised dependency parsing without parallel data. In *Proceedings of the Nineteenth Conference on Computational Natural Language Learning*, pages 113–122, Beijing, China. Association for Computational Linguistics.

Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In *Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 644–648, Atlanta, Georgia. Association for Computational Linguistics.

Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishray Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Mezauruiz, and 1 others. 2021. Americasnli: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages. *arXiv preprint arXiv:2104.08726*.

Pavel Efimov, Leonid Boytsov, Elena Arslanova, and Pavel Braslavski. 2023. The impact of cross-lingual adjustment of contextual word representations on zero-shot transfer. In *European Conference on Information Retrieval*, pages 51–67. Springer.

Juuso Eronen, Michal Ptaszynski, and Fumito Masui. 2023. Zero-shot cross-lingual transfer language selection using linguistic similarity. *Information Processing & Management*, 60(3):103250.

Felix Gaschi, Patricio Cerda, Parisa Rastin, and Yannick Toussaint. 2023. Exploring the relationship between alignment and cross-lingual transfer in multilingual transformers. *arXiv preprint arXiv:2306.02790*.

Jiatao Gu, Hany Hassan, Jacob Devlin, and Victor OK Li. 2018. Universal neural machine translation for extremely low resource languages. *arXiv preprint arXiv:1802.05368*.

Katharina Hämmerl, Jindřich Libovický, and Alexander Fraser. 2024. Understanding cross-lingual Alignment—A survey. In *Findings of the Association for Computational Linguistics: ACL 2024*, pages 10922–10943, Bangkok, Thailand. Association for Computational Linguistics.

Aung Kyaw Htet and Mark Dras. 2025a. Myanmar xnli: Building a dataset and exploring low-resource approaches to natural language inference with myanmar. *arXiv preprint arXiv:2504.09645*.

Aung Kyaw Htet and Mark Dras. 2025b. Myanmar xnli: Building a dataset and exploring low-resource approaches to natural language inference with myanmar. *Preprint*, arXiv:2504.09645.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6282–6293, Online. Association for Computational Linguistics.

Aditya Khan, Mason Shipton, David Anugraha, Kaiyao Duan, Phuong H. Hoang, Eric Khiu, A. Seza Doğruöz, and En-Shiun Annie Lee. 2025. URIEL+: Enhancing linguistic inclusion and usability in a typological and multilingual knowledge base. In *Proceedings of the 31st International Conference on Computational Linguistics*, pages 6937–6952, Abu Dhabi, UAE. Association for Computational Linguistics.

Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating Cross-lingual Extractive Question Answering. In *Proceedings of ACL 2020*.

Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In *The 57th Annual Meeting of the Association for Computational Linguistics (ACL)*, Florence, Italy.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. 2024. Regmix: Data mixture as regression for language model pre-training. *arXiv preprint arXiv:2407.01492*.

Zihan Liu, Genta Indra Winata, and Pascale Fung. 2021. Continual mixed-language pre-training for extremely low-resource neural machine translation. *arXiv preprint arXiv:2105.03953*.

Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, and Clara Vania. 2021. IndoNLI: A natural language inference dataset for Indonesian. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 10511–10527, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. 2024. Understanding and mitigating language confusion in LLMs. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 6653–6677, Miami, Florida, USA. Association for Computational Linguistics.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.

Fred Philippy, Siwen Guo, and Shohreh Haddadan. 2023. Towards a common understanding of contributing factors for cross-lingual transfer in multilingual language models: A review. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 5877–5891, Toronto, Canada. Association for Computational Linguistics.

Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? *arXiv preprint arXiv:1906.01502*.

Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pages 10215–10245, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hefernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and

20 others. 2022. No language left behind: Scaling human-centered machine translation. *Preprint*, arXiv:2207.04672.

Jörg Tiedemann. 2009. *News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces*, volume V, pages 237–248. University of Helsinki.

Xinyi Wang and Graham Neubig. 2019. Target conditioned sampling: Optimizing data selection for multilingual neural machine translation. *arXiv preprint arXiv:1905.08212*.

Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference. In *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 2526–2547, Vienna, Austria. Association for Computational Linguistics.

Shijie Wu and Mark Dredze. 2019. Beto, bentz, becas: The surprising cross-lingual effectiveness of bert. *arXiv preprint arXiv:1904.09077*.

Shijie Wu and Mark Dredze. 2020. Do explicit alignments robustly improve multilingual encoders? *arXiv preprint arXiv:2010.02537*.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In *Proceedings of EMNLP 2019*, pages 3685–3690.

Biao Zhang, Philip Williams, Ivan Titov, and RicoSENrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 1628–1639.

Wei Zhao, Steffen Eger, Johannes Bjerva, and Isabelle Augenstein. 2020. Inducing language-agnostic multilingual representations. *arXiv preprint arXiv:2008.09112*.

Pierre Zweigenbaum, Serge Sharoff, and Reinhard Rapp. 2018. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora. In *Proceedings of 11th Workshop on Building and Using Comparable Corpora*, pages 39–42.
# A Detailed Methodology and Experimental Setup

## A.1 Language Featural Diversity Calculation

Let $L = \{\ell_1, \ell_2, \dots, \ell_m\}$ be the list of all available languages, $\text{vec}(\ell)$ denote the URIEL featural vector of language $\ell$, and let $d(u, v)$ denote the angular distance between vectors $u, v \in \mathbb{R}^d$. The angular distance is defined as:

$$d(u, v) = \frac{\arccos\left(\frac{u \cdot v}{\|u\| \|v\|}\right)}{\pi}$$

where:

* $u \cdot v$ is the dot product of $u$ and $v$,

* $\|u\|$ is the Euclidean norm of vector $u$,

* $d(u, v) \in [0, 1]$ (normalized angle between vectors).

In order to find the most diverse subset size $n$, we try to maximize the total pairwise angular distance:

$$S^* = \arg\max_{S \subseteq L, |S|=n} \sum_{\{\ell_i, \ell_j\} \in \binom{S}{2}} d(\text{vec}(\ell_i), \text{vec}(\ell_j))$$

Similarly, to find the least diverse subset size $n$, we try to minimize the total pairwise angular distance:

$$S^* = \arg\min_{S \subseteq L, |S|=n} \sum_{\{\ell_i, \ell_j\} \in \binom{S}{2}} d(\text{vec}(\ell_i), \text{vec}(\ell_j))$$

## A.2 Realignment using average words' representations vs FastAlign

<table><caption>Table 2: Comparison of Average and FastAlign performance across 5 random seeds. Avg. Time reflects the average time taken by realignment step only across all tasks (in hours); each seed reused the same realignment checkpoint.</caption><thead><tr><th>Model</th><th>Task</th><th>Avg. Tokens</th><th>FastAlign</th></tr></thead><tbody><tr><td rowspan="3">mBERT</td><td>NER</td><td>54.89 ± 0.93</td><td><strong>55.39 ± 0.52</strong></td></tr><tr><td>NLI</td><td>55.34 ± 0.28</td><td><strong>56.58 ± 0.25</strong></td></tr><tr><td>POS</td><td>66.85 ± 0.24</td><td><strong>70.31 ± 0.15</strong></td></tr><tr><td rowspan="3">XLM-R</td><td>NER</td><td><strong>57.19 ± 0.79</strong></td><td>56.72 ± 1.62</td></tr><tr><td>NLI</td><td>59.5 ± 0.27</td><td><strong>60.65 ± 0.33</strong></td></tr><tr><td>POS</td><td>69.1 ± 0.22</td><td><strong>71.29 ± 0.25</strong></td></tr><tr><td colspan="2"><strong>Avg. Time (h)</strong></td><td><strong>0.35 ± 0.003</strong></td><td>1.93 ± 0.039</td></tr></tbody></table>

Realignment methods introduced before this work take a batch of pairs of translated sentences, extract pairs of corresponding words across those pairs using alignment tools like FastAlign and AwesomeAlign, and compute an in-batch contrastive loss on those pairs of words. Our work introduces a simple “averaging trick”: instead of computing the contrastive loss on word pairs extracted with an aligner, we compute the average of all tokens in a sentence and align sentences instead of words. This change is not made to improve cross-lingual transfer but rather to alleviate the need for a word aligner, which considerably reduces the time necessary to perform realignment.

We compare the time efficiency and performance of the two realignment methods using a different GPU type from that used in our main experiments, with the results presented in Table 2. Overall, FastAlign performs slightly better than the token averaging method on most tasks. However, we adopt the averaging approach for practical reasons, as FastAlign is considerably more resource- and time-intensive - even without accounting for the additional preprocessing required to prepare datasets for FastAlign across 65 languages - making it unsuitable for large-scale experiments. It is important to emphasize that we employ the averaging method **not to enhance the baseline performance**, but rather **to reduce computational overhead**, thereby enabling large-scale comparisons across different language selection strategies.

## A.3 Tentative Decoder-only Results

<table><caption>Table 3: Comparison of Llama model performance with and without realignment on downstream tasks.</caption><thead><tr><th>Metric</th><th>Llama w/o realignment</th><th>Llama w/ realignment</th></tr></thead><tbody><tr><td>PoS target acc.</td><td>38.8</td><td><strong>41.9</strong></td></tr><tr><td>NER target F1</td><td><strong>31.6</strong></td><td>30.2</td></tr><tr><td>NLI accuracy</td><td>56.5</td><td><strong>57.8</strong></td></tr></tbody></table>

We experimented with Llama 3.1 8B using LoRA adapters to overcome computational limitations. The adapters were trained for realignment and then fine-tuned for the downstream task, following a similar procedure as for encoder-only models. The results are shown in Table 3.

While realignment improved PoS tagging and NLI, we observed that it negatively impacted NER performance for LLaMA, which suggests that **re-alignment does not transfer straightforwardly**
**to decoder-only architectures.** Our preliminary experiments are not conclusive on whether realignment could ultimately benefit decoder-only models. We plan to investigate this further in future work.

More broadly, applying realignment to decoder-only LLMs for generative tasks raises unique challenges. In particular, **making such models entirely language-agnostic could exacerbate issues such as language confusion** (Marchisio et al., 2024), where the model generates text in an unintended language. This highlights an important avenue for future work and motivates careful consideration of how realignment should be adapted for decoder-only settings.

## A.4 Encoder-only models vs Decoder-only models on cross-lingual transfer classification

<table><caption>Table 4: Performance comparison of XLM-R, Llama 3.1 8B, and Gemma 2 9B across various three downstream tasks: POS Tagging, NER and NLI under the same experimental settings. **XL** indicates cross-lingual.</caption><thead><tr><th>Task</th><th>XLM-R</th><th>Llama 3.1</th><th>Gemma 2</th></tr></thead><tbody><tr><td>POS en</td><td><strong>96.2</strong></td><td>90.9</td><td>93.5</td></tr><tr><td>POS XL</td><td><strong>62.0</strong></td><td>38.8</td><td>49.1</td></tr><tr><td>NER en</td><td><strong>82.0</strong></td><td>72.5</td><td>73.9</td></tr><tr><td>NER XL</td><td><strong>49.1</strong></td><td>31.6</td><td>35.9</td></tr><tr><td>NLI en</td><td>83.5</td><td>90.2</td><td><strong>91.3</strong></td></tr><tr><td>NLI XL</td><td>54.6</td><td><strong>56.5</strong></td><td>55.0</td></tr></tbody></table>

We fine-tuned LLaMA 3.1 8B on PoS, NER, and NLI under the same setup. Our results on Table 4 show that XLM-R (encoder-only) not only significantly outperforms Llama and Gemma on cross-lingual transfer for some tasks, but can even surpass them in English. Interestingly, this seems to be true for word-level tasks (POS and NER), but not sentence-level ones (NLI). This finding aligns with prior work suggesting that small fine-tuned encoder-only models often outperform prompted LLMs on classification tasks (Ahuja et al., 2023).

In conclusion, adapting realignment methods to encoder-only models and generative tasks is a modern and exciting direction for future research. Nevertheless, we focus on encoder-only models, which remain a relevant contribution for cross-lingual classification, especially for multilinguality.

## A.5 Languages

Table 5 contains the full list of the 65 languages used in Gaschi et al. (2023). Table 6 contains the list of languages used in each experiment.

Table 6 shows the languages chosen within each experiment. All experiments are run with seeds of 17, 23, 42, and 66, including the selection of languages in the Random Subsets (Seeded) setting.

Figures 6, 7, and 8 show the language trees of the 65 languages used.

## A.6 Hyperparameters and Resources

For both tasks, we follow the experimental setup used in Gaschi et al. (2023); Bakos et al. (2025). All experiments are conducted using NVIDIA H100 GPUs and run with 4 random seeds to account for variability. Realignment is performed for 24,544 steps. This is followed by task-specific fine-tuning: 5 epochs for PoS tagging and 2 epochs for NLI. We use a learning rate of 2e-5, a batch size of 32 for both training and evaluation, and a maximum input length of 200 tokens for source and target sequences. During the realignment stage, we use a reduced maximum sequence length of 96 and a smaller batch size of 16.

## A.7 Statistics about the datasets used

The size of the datasets used for training and evaluating are reported in Table 7.

## A.8 Licenses for artifacts used

Below is a list of the datasets under study:

* The XNLI corpus (Conneau et al., 2018) has the CC BY-NC 4.0 license.

* The AfriXNLI dataset (Adelani et al., 2024) has the Apache 2.0 license.

* The IndoNLI dataset (Mahendra et al., 2021) has the CC-BY-SA 4.0 license.

* The Myanmar-XNLI dataset (Htet and Dras, 2025b) has the Apache 2.0 license.

* The UDPOS dataset (De Marneffe et al., 2021) has the CC0-1.0 license.

* The MasakhaPOS dataset (Dione et al., 2023) has the MIT license.

* The WikiANN dataset (Pan et al., 2017) has the Apache 2.0 2.0 license.
* The MasakhaNER 2.0 dataset (Adelani et al., 2022) has the AFL 3.0 license.

* The OPUS-100 dataset (Zhang et al., 2020) has no explicit license; it is a filtered subset of OPUS (Tiedemann, 2009), which aggregates translation corpora that is generally considered redistributable.

* The NLLB dataset (Team et al., 2022) has the ODC-By license.

* The XTREME-R benchmark suite (Ruder et al., 2021) does not have a unified license; it aggregates multiple datasets, each with its own license or terms of use:

- The XNLI corpus (Conneau et al., 2018) has the CC BY-NC 4.0 license.

- The PAWS-X dataset (Yang et al., 2019) is free to use for any purpose.

- The UDPOS dataset (De Marneffe et al., 2021) has the CC0-1.0 license.

- The WikiANN dataset (Pan et al., 2017) has the Apache 2.0 2.0 license.

- The XQuAD dataset (Artetxe et al., 2020) has the CC BY-SA 4.0 license.

- The MLQA dataset (Lewis et al., 2020) has the CC BY-SA 3.0 license.

- The TyDiQA-GoldP dataset (Clark et al., 2020) has the Apache 2.0 license.

- The BUCC 2018 dataset for the shared task on bitext mining (Zweigenbaum et al., 2018) is available for academic research use only; redistribution may be restricted.

- The Tatoeba dataset (Artetxe and Schwenk, 2019) has the CC BY 2.0 license.

Below is a list of the other artifacts under study:

* The code for realignment comes from Gaschi et al. (2023) and has the MIT license.

* The URIEL+ knowledge base and distance calculation functions (Khan et al., 2025) have the CC BY-SA 4.0 license.

* The weights of XLM-R Base (Conneau et al., 2020) have the MIT license.

* The weights of mBERT Base (Devlin et al., 2019) have the Apache 2.0 license.

All artifacts were thus used in accordance with their open-source or non-commercial licenses.

## A.9 Use of AI

For the writing of this paper, AI was solely used to reformulate some text, and as an autocompletion tool for writing the code used in the experiments.

## B More Detailed Results

This section contains the full results of the experiments in this paper.
<table><thead><tr><th>Language</th><th>Language Code<sup>‡</sup></th><th>Script</th><th>Language Family</th><th>Joshi Class<sup>†</sup></th><th>Vitality</th></tr></thead><tbody><tr><td>Afrikaans</td><td>afr</td><td>Latin</td><td>Germanic</td><td>3</td><td>MRL</td></tr><tr><td>Akan</td><td>twi</td><td>Latin</td><td>Atlantic-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Amharic</td><td>amh</td><td>Amharic</td><td>Semitic</td><td>2</td><td>LRL</td></tr><tr><td>Arabic</td><td>ara</td><td>Arabic</td><td>Semitic</td><td>5</td><td>HRL</td></tr><tr><td>Azerbaijani</td><td>aze</td><td>Latin</td><td>Oghuz</td><td>1</td><td>LRL</td></tr><tr><td>Bambara</td><td>bam</td><td>Latin</td><td>Mande</td><td>1</td><td>LRL</td></tr><tr><td>Basque</td><td>eus</td><td>Latin</td><td>N/A</td><td>4</td><td>MRL</td></tr><tr><td>Bengali</td><td>ben</td><td>Bengali</td><td>Indo-Iranian</td><td>3</td><td>MRL</td></tr><tr><td>Bulgarian</td><td>bul</td><td>Cyrillic</td><td>Balto-Slavic</td><td>3</td><td>MRL</td></tr><tr><td>Burmese</td><td>mya</td><td>Burmese</td><td>Sino-Tibetan</td><td>1</td><td>LRL</td></tr><tr><td>Chinese</td><td>zho</td><td>Chinese</td><td>Sino-Tibetan</td><td>5</td><td>HRL</td></tr><tr><td>Dholuo</td><td>luo</td><td>Latin</td><td>Nilo-Saharan</td><td>0</td><td>LRL</td></tr><tr><td>Dutch</td><td>nld</td><td>Latin</td><td>Germanic</td><td>4</td><td>MRL</td></tr><tr><td>Eastern Punjabi</td><td>pan</td><td>Gurmukhi</td><td>Indo-Iranian</td><td>2</td><td>LRL</td></tr><tr><td>Estonian</td><td>est</td><td>Latin</td><td>Finnic</td><td>3</td><td>MRL</td></tr><tr><td>Ewe</td><td>ewe</td><td>Latin</td><td>Atlantic-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Finnish</td><td>fin</td><td>Latin</td><td>Finnic</td><td>4</td><td>MRL</td></tr><tr><td>Fon</td><td>fon</td><td>Latin</td><td>Atlantic-Congo</td><td>0</td><td>LRL</td></tr><tr><td>French</td><td>fra</td><td>Latin</td><td>Romance</td><td>5</td><td>HRL</td></tr><tr><td>Ganda</td><td>lug</td><td>Latin</td><td>Atlantic-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Georgian</td><td>kat</td><td>Georgian</td><td>Kartvelian</td><td>3</td><td>MRL</td></tr><tr><td>German</td><td>deu</td><td>Latin</td><td>Germanic</td><td>5</td><td>HRL</td></tr><tr><td>Greek</td><td>ell</td><td>Greek</td><td>Hellenic</td><td>3</td><td>MRL</td></tr><tr><td>Gujarati</td><td>guj</td><td>Gujarati</td><td>Indo-Iranian</td><td>1</td><td>LRL</td></tr><tr><td>Hausa</td><td>hau</td><td>Latin</td><td>Chadic</td><td>2</td><td>LRL</td></tr><tr><td>Hebrew</td><td>heb</td><td>Hebrew</td><td>Semitic</td><td>3</td><td>MRL</td></tr><tr><td>Hindi</td><td>hin</td><td>Devanagari</td><td>Indo-Iranian</td><td>4</td><td>MRL</td></tr><tr><td>Hungarian</td><td>hun</td><td>Latin</td><td>Ugric</td><td>4</td><td>MRL</td></tr><tr><td>Igbo</td><td>ibo</td><td>Latin</td><td>Atlantic-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Indonesian</td><td>ind</td><td>Latin</td><td>Malayic</td><td>3</td><td>MRL</td></tr><tr><td>Italian</td><td>ita</td><td>Latin</td><td>Romance</td><td>4</td><td>MRL</td></tr><tr><td>Japanese</td><td>jpn</td><td>Japanese</td><td>Japonic</td><td>5</td><td>HRL</td></tr><tr><td>Javanese</td><td>jav</td><td>Latin</td><td>Javanic</td><td>1</td><td>LRL</td></tr><tr><td>Kazakh</td><td>kaz</td><td>Cyrillic</td><td>Kipchak</td><td>3</td><td>MRL</td></tr><tr><td>Kinyarwanda</td><td>kin</td><td>Latin</td><td>Atlantic-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Korean</td><td>kor</td><td>Korean</td><td>Koreanic</td><td>4</td><td>MRL</td></tr><tr><td>Lingala</td><td>lin</td><td>Latin</td><td>Atlantic-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Lithuanian</td><td>lit</td><td>Latin</td><td>Balto-Slavic</td><td>3</td><td>MRL</td></tr><tr><td>Malay</td><td>msa</td><td>Latin</td><td>Malayic</td><td>3</td><td>MRL</td></tr><tr><td>Malayalam</td><td>mal</td><td>Malayalam</td><td>Southern Dravidian</td><td>1</td><td>LRL</td></tr><tr><td>Marathi</td><td>mar</td><td>Devanagari</td><td>Indo-Iranian</td><td>2</td><td>LRL</td></tr><tr><td>Mossi</td><td>mos</td><td>Latin</td><td>Gur</td><td>0</td><td>LRL</td></tr><tr><td>Nyanja</td><td>nya</td><td>Latin</td><td>Benue-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Oromo</td><td>orm</td><td>Latin</td><td>Cushitic</td><td>1</td><td>LRL</td></tr><tr><td>Persian</td><td>fas</td><td>Arabic</td><td>Indo-Iranian</td><td>4</td><td>MRL</td></tr><tr><td>Polish</td><td>pol</td><td>Latin</td><td>Balto-Slavic</td><td>4</td><td>MRL</td></tr><tr><td>Portuguese</td><td>por</td><td>Latin</td><td>Romance</td><td>4</td><td>MRL</td></tr><tr><td>Romanian</td><td>ron</td><td>Latin</td><td>Romance</td><td>3</td><td>MRL</td></tr><tr><td>Russian</td><td>rus</td><td>Cyrillic</td><td>Balto-Slavic</td><td>4</td><td>MRL</td></tr></tbody></table>
<table><caption>Table 5: The 65 languages used for the realignment phase with their vitality class mapping. The language codes follow ‡ISO639-3 coding. Languages are mapped to their †rarity taxonomy based on Joshi et al. (2020) vitality classes: Low Resource Language (LRL, 0-2), Medium Resource Language (MRL, 3-4), and High Resource Language (HRL, 5).</caption><tr><td>Shona</td><td>sna</td><td>Latin</td><td>Atlantic-Congo</td><td>1</td><td>LRL</td></tr><tr><td>Spanish</td><td>spa</td><td>Latin</td><td>Romance</td><td>5</td><td>HRL</td></tr><tr><td>Swahili</td><td>swa</td><td>Latin</td><td>Atlantic-Congo</td><td>2</td><td>LRL</td></tr><tr><td>Tagalog</td><td>tgl</td><td>Latin</td><td>Philippine</td><td>3</td><td>MRL</td></tr><tr><td>Tamil</td><td>tam</td><td>Tamil</td><td>Southern Dravidian</td><td>3</td><td>MRL</td></tr><tr><td>Telugu</td><td>tel</td><td>Telugu</td><td>South-Central Dravidian</td><td>1</td><td>LRL</td></tr><tr><td>Thai</td><td>tha</td><td>Thai</td><td>Kra-Dai</td><td>3</td><td>MRL</td></tr><tr><td>Tswana</td><td>tsn</td><td>Latin</td><td>Atlantic-Congo</td><td>2</td><td>LRL</td></tr><tr><td>Turkish</td><td>tur</td><td>Latin</td><td>Oghuz</td><td>4</td><td>MRL</td></tr><tr><td>Ukrainian</td><td>ukr</td><td>Cyrillic</td><td>Balto-Slavic</td><td>3</td><td>MRL</td></tr><tr><td>Urdu</td><td>urd</td><td>Arabic</td><td>Indo-Iranian</td><td>3</td><td>MRL</td></tr><tr><td>Vietnamese</td><td>vie</td><td>Latin</td><td>Austroasiatic</td><td>4</td><td>MRL</td></tr><tr><td>Wolof</td><td>wol</td><td>Latin</td><td>Atlantic-Congo</td><td>2</td><td>LRL</td></tr><tr><td>Xhosa</td><td>xho</td><td>Latin</td><td>Atlantic-Congo</td><td>2</td><td>LRL</td></tr><tr><td>Yoruba</td><td>yor</td><td>Latin</td><td>Atlantic-Congo</td><td>2</td><td>LRL</td></tr><tr><td>Zulu</td><td>zul</td><td>Latin</td><td>Atlantic-Congo</td><td>2</td><td>LRL</td></tr></table>
<table><thead><tr><th>Method</th><th>#</th><th>Languages<sup>‡</sup></th></tr></thead><tbody><tr><td colspan="3"><strong>Baseline</strong></td></tr><tr><td>All 65 languages</td><td>65</td><td>afr, amh, ara, aze, bul, ben, deu, ell, spa, est, eus, fas, fin, fra, guj, heb, hin, hun, ind, ita, jpn, kat, kaz, kor, lit, mal, mar, msa, mya, nld, pan, pol, por, ron, rus, tam, tha, tur, ukr, urd, vie, zho, bam, ewe, fon, hau, ibo, kin, lin, lug, luo, mos, nya, gaz, sna, swh, tsn, twi, wol, xho, yor, zul, jav, tgl, tel</td></tr><tr><td>Present in XTREME-R</td><td>47</td><td>afr, ara, aze, bul, ben, deu, ell, spa, est, eus, fas, fin, fra, guj, heb, hin, hun, ind, ita, jpn, jav, kat, kaz, kor, lit, mar, mal, msa, mya, nld, pan, pol, por, ron, rus, swh, tam, tel, tgl, tha, tur, ukr, urd, vie, wol, yor, zho</td></tr><tr><td>Present in Africa</td><td>21</td><td>amh, bam, ewe, fon, hau, ibo, kin, lin, lug, luo, mos, nya, gaz, sna, swh, tsn, twi, wol, xho, yor, zul</td></tr><tr><td colspan="3"><strong>Featural Diversity</strong></td></tr><tr><td>Most diverse from English</td><td>5<br>10<br>20<br>40</td><td>fon, kat, kaz, lin, gaz<br>afr, ara, fon, kat, jpn, kaz, lin, gaz, sna, vie<br>afr, ara, aze, eus, zho, fon, lug, kat, ell, hau, heb, ibo, jpn, kaz, kor, lin, luo, gaz, sna, twi, vie, yor<br>afr, ara, aze, eus, mya, zho, ewe, fon, fra, lug, kat, ell, hau, heb<br>ibo, jpn, kaz, kin, kor, lin, msa, mal, mar, nya, gaz, fas, rus, sna<br>spa, tgl, tam, tel, tha, tur, twi, urd, vie, xho, yor, zul</td></tr><tr><td>Least diverse from English</td><td>5<br>10<br>20<br>40</td><td>ita, por, ron, spa, ukr<br>bul, deu, ell, spa, fra, ita, nld, por, ron, ukr<br>bul, nld, est, fin, fra, deu, ell, guj, hin, hun, ita, lit, fas, pol<br>por, pan, ron, rus, spa, ukr<br>amh, ara, aze, bam, eus, ben, bul, nld, est, fin, fra, deu, ell, guj<br>hau, heb, hin, hun, ind, ita, jav, lit, luo, mal, mar, mos, fas, pol<br>por, pan, ron, rus, spa, tgl, tam, tel, tur, ukr, urd, wol</td></tr><tr><td colspan="3"><strong>Phylogenetic Diversity</strong></td></tr><tr><td>Most diverse families</td><td>5<br>10</td><td>kat, kaz, lin, gaz, vie<br>ara, zho, kat, jpn, kaz, lin, msa, gaz, tam, vie</td></tr></tbody></table>
<table><tr><td></td><td>20</td><td>ara, aze, eus, zho, fra, kat, ell, hau, jpn, kaz, kor, lin, luo, msa, mar, gaz, rus, tam, tha, vie</td></tr><tr><td></td><td>25</td><td>ara, aze, bam, eus, zho, fin, fra, kat, ell, hau, hun, jpn, kaz, kor</td></tr><tr><td></td><td></td><td>lin, luo, msa, mar, mos, gaz, rus, tam, tel, tha, vie</td></tr><tr><td rowspan="3">Most diverse families within<br/>Indo-European</td><td>5</td><td>afr, nld, deu, ita, por</td></tr><tr><td>10</td><td>afr, bul, nld, fra, deu, ita, por, ron, spa, ukr</td></tr><tr><td>20</td><td>afr, ben, bul, nld, fra, deu, ell, guj, hin, ita, lit, mar, pol, por</td></tr><tr><td></td><td></td><td>pan, ron, rus, spa, ukr, urd</td></tr></table>

## Script Diversity

<table><tr><td rowspan="3">Most diverse using distinct<br/>scripts</td><td>5</td><td>ara, kat, jpn, kaz, tha</td></tr><tr><td>10</td><td>ara, mya, zho, kat, ell, heb, jpn, kaz, tam, tha</td></tr><tr><td>18</td><td>amh, ara, ben, mya, zho, kat, ell, guj, heb, hin, jpn, kaz, kor, mal, pan, tam, tel, tha</td></tr><tr><td rowspan="4">Most diverse using Latin scripts</td><td>5</td><td>aze, fon, lin, gaz, tgl</td></tr><tr><td>10</td><td>afr, aze, eus, fon, lin, gaz, sna, tgl, twi, vie</td></tr><tr><td>20</td><td>bam, nld, est, fin, fra, deu, hau, hun, ind, ita, jav, lit, luo, msa, pol, por, ron, spa, tgl, wol</td></tr><tr><td>41</td><td>afr, aze, bam, eus, nld, est, ewe, fin, fon, fra, lug, deu, hau, hun</td></tr><tr><td></td><td></td><td>ibo, ind, ita, jav, kin, lin, lit, luo, msa, mos, nya, gaz, pol, por</td></tr><tr><td></td><td></td><td>ron, sna, spa, swh, tgl, tsn, tur, twi, vie, wol, xho, yor, zul</td></tr><tr><td rowspan="3">Least diverse using Latin scripts</td><td>5</td><td>nld, fra, deu, ita, por</td></tr><tr><td>10</td><td>nld, est, fin, fra, deu, hun, ita, por, ron, spa</td></tr><tr><td>20</td><td>afr, aze, eus, ewe, fon, fra, lug, lin, lit, msa, gaz, pol, sna, spa</td></tr><tr><td></td><td></td><td>tbl, tur, twi, vie, yor, zul</td></tr></table>

## Ablation Languages

<table><tr><td>Joshi Class = 2 (Random)</td><td>10</td><td>amh, aze, bam, ewe, fon, gaz, guj, hau, ibo, jav</td></tr><tr><td>Joshi Class = 2 (Most URIEL)</td><td>10</td><td>aze, mya, fon, kin, lin, mar, gaz, sna, tel, yor</td></tr><tr><td>Joshi Class = 2 (Most Family)</td><td>10</td><td>aze, mya, hau, jav, lin, luo, mar, mos, gaz, tel</td></tr><tr><td>Joshi Class = 2 (Most Script)</td><td>10</td><td>amh, mya, guj, lin, mal, mar, gaz, pan, tel, yor</td></tr><tr><td>Joshi Class = 3 (Random)</td><td>17</td><td>afr, bul, ben, ell, est, heb, ind, kat, kaz, lit, msa, ron, tam, tgl, tha, ukr, urd</td></tr><tr><td>Joshi Class = 3 (Most URIEL)</td><td>10</td><td>afr, kat, ell, heb, kaz, msa, tgl, tam, tha, urd</td></tr><tr><td>Joshi Class = 3 (Most Family)</td><td>10</td><td>est, kat, ell, heb, kaz, lit, msa, tam, tha, urd</td></tr><tr><td>Joshi Class = 3 (Most Script)</td><td>10</td><td>ben, bul, kat, ell, heb, kaz, tam, tha, ukr, urd</td></tr><tr><td>Joshi Class = 3,5 (Random)</td><td>23</td><td>afr, ara, bul, ben, deu, ell, spa, est, eus, fas, fin, fra, heb, hin, hun, ind, ita, jpn, kat, kaz, kor, lit, msa,</td></tr></table>
<table><caption>Table 6: List of languages used for each experiment and selection strategy. Language codes follow ‡ISO639-3 coding.</caption><tr><td></td><td>nld, pan, pol, por, ron, rus, tam, tgl, tha, tur, ukr, urd, vie, zho</td></tr><tr><td>Joshi Class = 3,5 (Most URIEL)</td><td>10 afr, ara, eus, zho, kat, jpn, kaz, msa, tam, vie</td></tr><tr><td>Joshi Class = 3,5 (Most Family)</td><td>10 ara, zho, kat, ell, jpn, kaz, msa, tam, tha, vie</td></tr><tr><td>Joshi Class = 3,5 (Most Script)</td><td>10 ara, zho, kat, ell, heb, jpn, kaz, kor, tam, tha</td></tr><tr><td>Joshi Class = 4,5 (Random)</td><td>20 ara, deu, spa, eus, fas, fin, fra, hin, hun, ita, jpn, kor, nld, pol, por, rus, tam, tur, vie, zho</td></tr><tr><td>Joshi Class = 4,5 (Most URIEL)</td><td>10 ara, eus, zho, fra, jpn, kor, fas, rus, tur, vie</td></tr><tr><td>Joshi Class = 4,5 (Most Script)</td><td>10 ara, eus, fra, hin, jpn, kor, fas, rus, tur, vie</td></tr><tr><td>Seen by mBERT (Random)</td><td>47 afr, ara, aze, bul, ben, deu, ell, spa, est, eus, fas, fin, fra, guj, heb, hin, hun, ind, ita, jpn, jav, kat, kaz, kor, lit, mar, mal, msa, mya, nld, pan, pol, por, ron, rus, swh, tam, tel, tgl, tha, tur, ukr, urd, vie, yor, zho</td></tr><tr><td>Seen by mBERT (Most URIEL)</td><td>10 afr, ara, zho, kat, jpn, kaz, swh, tam, vie, yor</td></tr><tr><td>Seen by mBERT (Most Family)</td><td>10 ara, aze, zho, kat, jpn, kaz, msa, tam, vie, yor</td></tr><tr><td>Seen by mBERT (Most Script)</td><td>10 ara, mya, zho, kat, ell, heb, jpn, kaz, tam, tha</td></tr><tr><td>Seen by XLM-R (Random)</td><td>51 afr, amh, ara, aze, bul, ben, deu, ell, spa, est, eus, fas, fin, fra, gaz, guj, hau, heb, hin, hun, ind, ita, jpn, jav, kat, kaz, kor, lit, mar, mal, msa, mya, nld, pan, pol, por, ron, rus, swh, tam, tel, tgl, tha, tur, ukr, urd, vie, xho, zho</td></tr><tr><td>Seen by XLM-R (Most URIEL)</td><td>10 afr, ara, zho, kat, jpn, kaz, msa, gaz, swh, vie</td></tr><tr><td>Seen by XLM-R (Most Family)</td><td>10 ara, zho, kat, jpn, kaz, msa, gaz, tam, vie, xho</td></tr><tr><td>Seen by XLM-R (Most Script)</td><td>10 ara, mya, zho, kat, ell, heb, jpn, kaz, tam, tha</td></tr><tr><td>Unseen by mBERT (Random)</td><td>34 amh, bam, ewe, fon, gaz, hau, ibo, kin, lin, lug, luo, mos, nya, sna, tsn, twi, wol, xho, yor, zul</td></tr><tr><td>Unseen by mBERT (Most URIEL)</td><td>10 amh, ewe, fon, hau, lin, luo, gaz, sna, twi, xho</td></tr><tr><td>Unseen by mBERT (Most Family)</td><td>10 amh, bam, fon, hau, lin, luo, mos, gaz, sna, twi</td></tr><tr><td>Unseen by XLM-R (Random)</td><td>30 bam, ewe, fon, ibo, kin, lin, lug, luo, mos, nya, sna, tsn, twi, wol, yor, zul</td></tr><tr><td>Unseen by XLM-R (Most URIEL)</td><td>10 ewe, fon, lin, luo, mos, sna, twi, wol, yor, zul</td></tr><tr><td>Unseen by XLM-R (Most Family)</td><td>10 bam, fon, lin, luo, mos, sna, twi, wol, yor, zul</td></tr></table>
<figure><img src="image_7.png" /><figcaption>Figure 6: Phylogenetic tree for Indo-European languages (including Basque as an isolate) used in realignment.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>Figure 7: Phylogenetic trees for Niger-Congo languages, and Dholuo (from the Nilo-Saharan family, due to its proximity) used in realignment.</figcaption></figure>
<figure><img src="image_9.png" /><figcaption>Figure 8: Residual phylogenetic trees for all other languages used in realignment (Non-Indo-European, Non-Niger-Congo, Non-Nilo-Saharan, Non-Basque).</figcaption></figure>
<table><thead><tr><th rowspan="2">Language</th><th rowspan="2">NLI</th><th rowspan="2">PoS-tagging</th><th rowspan="2">NER</th><th colspan="4">Seen by</th></tr><tr><th>XLM-R</th><th>mBERT</th><th>OPUS-100</th><th>NLLB</th></tr></thead><tbody><tr><td>English (training)</td><td>392,702</td><td>21,787</td><td>20,000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Afrikaans</td><td>-</td><td>425</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Arabic</td><td>5010</td><td>1680</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Azerbaijani</td><td>-</td><td>-</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Basque</td><td>-</td><td>1799</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Bengali</td><td>-</td><td>-</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Bulgarian</td><td>5010</td><td>1116</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Burmese</td><td>5010</td><td>-</td><td>100</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Chinese</td><td>5010</td><td>3455</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Czech</td><td>-</td><td>10159</td><td>-</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Dutch</td><td>-</td><td>1471</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>English (evaluation)</td><td>5010</td><td>5440</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Estonian</td><td>-</td><td>4127</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Finnish</td><td>-</td><td>6544</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>French</td><td>5010</td><td>7542</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Georgian</td><td>-</td><td>-</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>German</td><td>5010</td><td>22358</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Greek</td><td>5010</td><td>456</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Gujarati</td><td>-</td><td>-</td><td>100</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Hebrew</td><td>-</td><td>491</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Hindi</td><td>5010</td><td>2684</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Hungarian</td><td>-</td><td>449</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Indonesian</td><td>2984</td><td>1931</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Italian</td><td>-</td><td>3518</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Japanese</td><td>-</td><td>2365</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Javanese</td><td>-</td><td>-</td><td>100</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td></tr><tr><td>Kazakh</td><td>-</td><td>1047</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Korean</td><td>-</td><td>4276</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Lithuanian</td><td>-</td><td>739</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Malay</td><td>-</td><td>-</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Malayalam</td><td>-</td><td>-</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Marathi</td><td>-</td><td>47</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Persian</td><td>-</td><td>2055</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Polish</td><td>-</td><td>4942</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Portuguese</td><td>-</td><td>2680</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Eastern Punjabi</td><td>-</td><td>-</td><td>100</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Romanian</td><td>-</td><td>2272</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Russian</td><td>5010</td><td>8973</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Spanish</td><td>5010</td><td>3147</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Tagalog</td><td>-</td><td>-</td><td>1000</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td></tr><tr><td>Tamil</td><td>-</td><td>654</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Telugu</td><td>-</td><td>146</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Thai</td><td>5010</td><td>1000</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Turkish</td><td>5010</td><td>6647</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Ukrainian</td><td>-</td><td>892</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Urdu</td><td>5010</td><td>535</td><td>1000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Vietnamese</td><td>5010</td><td>800</td><td>10000</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Akan</td><td>-</td><td>628</td><td>1211</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr></tbody></table>
(continued)

<table><caption>Table 7: The size of the combined datasets. The table is split into 3 sections: 1) The original 44 languages used for realignment 2) African languages exclusive to AfriXNLI and MasakhaPOS, 3) South American languages exclusive to AmericasNLI.</caption><thead><tr><th>Language</th><th>NLI</th><th>PoS-tagging</th><th>NER</th><th>XLM-R</th><th>mBERT</th><th>OPUS-100</th><th>NLLB</th></tr></thead><tbody><tr><td>Amharic</td><td>600</td><td>-</td><td>-</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td></tr><tr><td>Bambara</td><td>-</td><td>619</td><td>1274</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Chichewa</td><td>-</td><td>-</td><td>1785</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Dholuo</td><td>-</td><td>606</td><td>1474</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Ewe</td><td>600</td><td>582</td><td>1001</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Fon</td><td>-</td><td>646</td><td>1228</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Ghomala</td><td>-</td><td>599</td><td>966</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Hausa</td><td>600</td><td>601</td><td>1633</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td></tr><tr><td>Igbo</td><td>600</td><td>642</td><td>2181</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td></tr><tr><td>Kinyarwanda</td><td>600</td><td>604</td><td>2235</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td></tr><tr><td>Lingala</td><td>600</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Ganda</td><td>600</td><td>586</td><td>1412</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Mossi</td><td>-</td><td>604</td><td>1294</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Naija</td><td>-</td><td>-</td><td>1613</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Oromo</td><td>600</td><td>-</td><td>-</td><td>✓</td><td>✗</td><td>✓</td><td>✗</td></tr><tr><td>Setswana</td><td>-</td><td>602</td><td>996</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Shona</td><td>600</td><td>596</td><td>1773</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Southern Sotho</td><td>600</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Swahili</td><td>600</td><td>553</td><td>1883</td><td>✓</td><td>✓</td><td>✗</td><td>✓</td></tr><tr><td>Wolof</td><td>600</td><td>625</td><td>1312</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Xhosa</td><td>600</td><td>601</td><td>1633</td><td>✓</td><td>✗</td><td>✓</td><td>✓</td></tr><tr><td>Yoruba</td><td>600</td><td>713</td><td>1964</td><td>✗</td><td>✓</td><td>✓</td><td>✓</td></tr><tr><td>Zulu</td><td>600</td><td>601</td><td>1670</td><td>✗</td><td>✗</td><td>✓</td><td>✓</td></tr><tr><td>Aymara</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Asháninka</td><td>-</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Bribri</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Guaraní</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Nahuatl</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Otomí</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Quechua</td><td>750</td><td>-</td><td>100</td><td>✗</td><td>✗</td><td>✗</td><td>✓</td></tr><tr><td>Rarámuri</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Shipibo-Konibo</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr><tr><td>Wixárika</td><td>750</td><td>-</td><td>-</td><td>✗</td><td>✗</td><td>✗</td><td>✗</td></tr></tbody></table>
<table><caption>Table 8: Accuracy of XLM-R on NLI, PoS-Tagging, NER tasks. Results are averaged across 4 seeds along with the standard deviation.</caption><thead><tr><th>Method</th><th>#Languages</th><th>NLI</th><th>PoS-Tagging</th><th>NER</th></tr></thead><tbody><tr><td><strong>Baseline</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>All 65 languages</td><td>65</td><td><strong>55.43</strong> ± 0.29</td><td><strong>66.87</strong> ± 0.27</td><td><strong>54.75</strong> ± 1.02</td></tr><tr><td>Present in XTREME-R</td><td>47</td><td>53.51 ± 0.27</td><td>65.06 ± 0.38</td><td>52.54 ± 1.41</td></tr><tr><td>Present in Africa</td><td>21</td><td>54.94 ± 0.38</td><td>65.79 ± 0.18</td><td>53.72 ± 0.86</td></tr><tr><td>Fine-tuning only</td><td>0</td><td>53.12 ± 0.25</td><td>62.20 ± 0.78</td><td>52.25 ± 1.05</td></tr><tr><td><strong>Featural Diversity</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Most diverse from English</td><td>5<br>10<br>20<br>40</td><td>53.86 ± 0.24<br>54.40 ± 0.31<br>54.94 ± 0.31<br><strong>55.94</strong> ± 0.24</td><td>64.08 ± 0.30<br>64.87 ± 0.27<br>65.66 ± 0.06<br><strong>66.24</strong> ± 0.07</td><td>50.73 ± 0.73<br>53.64 ± 0.59<br>54.38 ± 1.05<br><strong>54.82</strong> ± 0.45</td></tr><tr><td>Least diverse from English</td><td>5<br>10<br>20<br>40</td><td>53.22 ± 0.17<br>53.26 ± 0.52<br>53.26 ± 0.12<br>53.69 ± 0.28</td><td>62.57 ± 0.33<br>63.14 ± 0.60<br>63.33 ± 0.39<br>65.83 ± 0.08</td><td>51.43 ± 1.41<br>52.70 ± 1.14<br>51.58 ± 0.69<br>52.85 ± 0.49</td></tr><tr><td><strong>Phylogenetic Diversity</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Most diverse families</td><td>5<br>10<br>20<br>25</td><td>54.03 ± 0.30<br>53.86 ± 0.25<br>53.98 ± 0.34<br><strong>54.25</strong> ± 0.19</td><td>63.53 ± 0.06<br>63.85 ± 0.51<br>63.87 ± 0.18<br><strong>65.13</strong> ± 0.37</td><td>49.69 ± 1.06<br>51.92 ± 0.82<br>53.98 ± 0.84<br><strong>54.73</strong> ± 0.85</td></tr><tr><td>Most diverse families within<br>Indo-European</td><td>5<br>10<br>20</td><td>52.86 ± 0.35<br>53.18 ± 0.45<br>53.06 ± 0.29</td><td>61.89 ± 0.48<br>62.40 ± 0.49<br>63.59 ± 0.87</td><td>50.20 ± 0.68<br>52.52 ± 1.56<br>51.08 ± 0.80</td></tr><tr><td><strong>Script Diversity</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Most diverse scripts</td><td>5<br>10<br>18</td><td>52.80 ± 0.25<br>52.54 ± 0.34<br>52.69 ± 0.13</td><td>61.74 ± 0.66<br>62.11 ± 0.87<br>62.82 ± 0.74</td><td>51.19 ± 1.55<br>50.41 ± 1.14<br>50.59 ± 0.73</td></tr><tr><td>Most diverse using Latin script</td><td>5<br>10<br>20<br>41</td><td>53.94 ± 0.31<br>54.75 ± 0.13<br>53.89 ± 0.12<br><strong>55.96</strong> ± 0.15</td><td>63.92 ± 0.38<br>64.95 ± 0.40<br>65.25 ± 0.15<br><strong>67.23</strong> ± 0.20</td><td>51.38 ± 0.50<br>53.28 ± 0.73<br>53.10 ± 0.89<br><strong>54.00</strong> ± 0.48</td></tr><tr><td>Least diverse using Latin script</td><td>5<br>10<br>20</td><td>53.20 ± 0.56<br>53.14 ± 0.31<br>55.73 ± 0.19</td><td>61.36 ± 0.13<br>62.89 ± 0.51<br>66.05 ± 0.25</td><td>49.22 ± 0.47<br>51.04 ± 0.33<br>53.60 ± 0.91</td></tr><tr><td><strong>Random Selection</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Random Seeded</td><td>5<br>10<br>20<br>40</td><td>54.21 ± 0.86<br>54.24 ± 0.88<br>54.64 ± 0.76<br><strong>55.36</strong> ± 0.24</td><td>63.61 ± 0.55<br>64.78 ± 0.34<br>65.27 ± 0.42<br><strong>66.06</strong> ± 0.63</td><td>51.76 ± 1.90<br>51.45 ± 0.51<br>52.92 ± 1.14<br><strong>53.42</strong> ± 0.48</td></tr></tbody></table>
<table><caption>Table 9: Ablation studies: Accuracy of mBERT on NLI, POS-Tagging, NER tasks. Results are averaged across 4 seeds along with the standard deviation.</caption><thead><tr><th>Method</th><th>#Languages</th><th>NLI</th><th>POS-Tagging</th><th>NER</th></tr></thead><tbody><tr><td colspan="5"><strong>Most featural diversity from English</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td>54.88 ± 0.18</td><td><strong>64.72</strong> ± 0.26</td><td>51.42 ± 0.30</td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>53.06 ± 0.36</td><td>63.30 ± 0.64</td><td>50.17 ± 0.81</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>53.20 ± 0.47</td><td>63.35 ± 1.02</td><td>51.13 ± 1.74</td></tr><tr><td>Joshi Class = 4,5</td><td>20</td><td>53.24 ± 0.30</td><td>63.84 ± 0.52</td><td>50.66 ± 0.41</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>53.85 ± 0.32</td><td>64.50 ± 0.88</td><td><strong>52.96</strong> ± 1.14</td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>54.27 ± 0.28</td><td>64.39 ± 0.26</td><td>52.71 ± 0.87</td></tr><tr><td>Unseen by mBERT</td><td>34</td><td><strong>55.14</strong> ± 0.40</td><td>64.02 ± 0.23</td><td>52.63 ± 0.44</td></tr><tr><td>Unseen by XLM-R</td><td>30</td><td>54.88 ± 0.34</td><td>64.70 ± 0.58</td><td>52.80 ± 0.73</td></tr><tr><td colspan="5"><strong>Most Phylogenetic Diversity</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td>54.11 ± 0.60</td><td>63.96 ± 0.18</td><td>51.23 ± 0.68</td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>52.96 ± 0.39</td><td>63.60 ± 0.49</td><td>50.73 ± 1.91</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>53.10 ± 0.29</td><td>63.68 ± 0.23</td><td>52.53 ± 1.67</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>53.57 ± 0.25</td><td>64.06 ± 0.67</td><td>52.52 ± 1.12</td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>54.42 ± 0.58</td><td>64.26 ± 0.17</td><td>53.34 ± 0.80</td></tr><tr><td>Unseen by mBERT</td><td>34</td><td><strong>54.59</strong> ± 0.37</td><td>64.02 ± 0.33</td><td><strong>53.37</strong> ± 0.82</td></tr><tr><td>Unseen by XLM-R</td><td>30</td><td><strong>54.59</strong> ± 0.38</td><td><strong>65.04</strong> ± 0.47</td><td>52.54 ± 0.97</td></tr><tr><td colspan="5"><strong>Most Script Diversity</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td><strong>54.20</strong> ± 0.17</td><td><strong>63.92</strong> ± 0.28</td><td>50.50 ± 0.66</td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>52.67 ± 0.18</td><td>61.73 ± 1.00</td><td>49.46 ± 0.31</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>52.60 ± 0.19</td><td>62.46 ± 0.57</td><td><strong>52.24</strong> ± 1.45</td></tr><tr><td>Joshi Class = 4,5</td><td>20</td><td>53.16 ± 0.13</td><td>63.54 ± 0.35</td><td>51.61 ± 0.35</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>52.73 ± 0.16</td><td>61.95 ± 0.37</td><td>49.73 ± 0.79</td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>52.73 ± 0.16</td><td>61.95 ± 0.37</td><td>49.73 ± 0.79</td></tr><tr><td colspan="5"><strong>Random Seeded</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td><strong>55.03</strong> ± 0.65</td><td>65.06 ± 0.53</td><td>52.46 ± 0.97</td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>53.28 ± 0.25</td><td>62.94 ± 0.49</td><td>50.31 ± 2.05</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>53.36 ± 0.21</td><td>63.54 ± 0.40</td><td>51.51 ± 0.70</td></tr><tr><td>Joshi Class = 4,5</td><td>20</td><td>53.30 ± 0.15</td><td>63.69 ± 0.14</td><td>50.67 ± 1.50</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>53.43 ± 0.37</td><td>63.67 ± 0.81</td><td>51.25 ± 0.40</td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>53.86 ± 0.48</td><td>63.84 ± 0.62</td><td>51.08 ± 0.65</td></tr><tr><td>Unseen by mBERT</td><td>34</td><td>54.45 ± 0.87</td><td>64.55 ± 0.25</td><td>52.48 ± 1.19</td></tr><tr><td>Unseen by XLM-R</td><td>30</td><td>54.87 ± 0.20</td><td><strong>65.11</strong> ± 0.29</td><td><strong>52.62</strong> ± 0.53</td></tr></tbody></table>
<table><caption>Table 10: Accuracy of mBERT on NLI, PoS-Tagging, NER tasks. Results are averaged across 4 seeds along with the standard deviation.</caption><thead><tr><th>Method</th><th>#Languages</th><th>NLI</th><th>PoS-Tagging</th><th>NER</th></tr></thead><tbody><tr><td><strong>Baseline</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>All 65 languages</td><td>65</td><td>59.43 ± 0.17</td><td>69.14 ± 0.24</td><td>57.07 ± 0.86</td></tr><tr><td>Present in XTREME-R</td><td>47</td><td><strong>59.44</strong> ± 0.39</td><td>67.32 ± 0.59</td><td><strong>57.24</strong> ± 0.73</td></tr><tr><td>Present in Africa</td><td>21</td><td>59.90 ± 0.33</td><td><strong>69.92</strong> ± 0.15</td><td>54.10 ± 1.14</td></tr><tr><td>Fine-tuning only</td><td>0</td><td>58.61 ± 0.10</td><td>65.98 ± 0.73</td><td>51.09 ± 0.96</td></tr><tr><td><strong>Featural Diversity</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Most diverse from English</td><td>5<br>10<br>20<br>40</td><td>58.97 ± 0.28<br>58.87 ± 0.19<br>59.27 ± 0.07<br><strong>59.64</strong> ± 0.24</td><td>67.99 ± 0.37<br>68.48 ± 0.43<br>68.57 ± 0.40<br><strong>68.89</strong> ± 0.26</td><td>53.39 ± 1.17<br>56.11 ± 0.77<br>56.51 ± 1.16<br><strong>56.39</strong> ± 0.98</td></tr><tr><td>Least diverse from English</td><td>5<br>10<br>20<br>40</td><td>58.59 ± 0.17<br>58.53 ± 0.09<br>58.74 ± 0.11<br>58.75 ± 0.17</td><td>66.04 ± 0.79<br>65.99 ± 0.80<br>67.10 ± 0.25<br>68.10 ± 0.36</td><td>52.80 ± 0.58<br>52.87 ± 1.47<br>53.96 ± 1.34<br>56.21 ± 0.42</td></tr><tr><td><strong>Phylogenetic Diversity</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Most diverse families</td><td>5<br>10<br>20<br>25</td><td>58.81 ± 0.20<br>58.91 ± 0.29<br><strong>59.15</strong> ± 0.31<br>59.06 ± 0.12</td><td>67.20 ± 0.37<br>67.31 ± 0.22<br>66.64 ± 0.10<br><strong>67.87</strong> ± 0.26</td><td>51.63 ± 0.72<br>53.81 ± 1.04<br><strong>55.53</strong> ± 0.77<br>54.99 ± 0.66</td></tr><tr><td>Most diverse families within<br>Indo-European</td><td>5<br>10<br>20</td><td>58.74 ± 0.30<br>58.59 ± 0.24<br>58.92 ± 0.10</td><td>66.39 ± 0.94<br>67.14 ± 0.64<br>67.15 ± 0.33</td><td>51.90 ± 1.23<br>53.81 ± 2.65<br>52.35 ± 1.42</td></tr><tr><td><strong>Script Diversity</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Most diverse scripts</td><td>5<br>10<br>18</td><td>58.76 ± 0.03<br>58.70 ± 0.27<br>58.77 ± 0.20</td><td>67.26 ± 0.46<br>67.04 ± 0.73<br>67.04 ± 0.92</td><td>49.89 ± 1.18<br>51.40 ± 1.97<br>50.83 ± 1.50</td></tr><tr><td>Most diverse using Latin script</td><td>5<br>10<br>20<br>41</td><td>58.75 ± 0.35<br>59.15 ± 0.30<br>58.75 ± 0.23<br><strong>59.88</strong> ± 0.06</td><td>67.88 ± 0.26<br>68.11 ± 0.15<br>67.75 ± 0.09<br><strong>69.62</strong> ± 0.30</td><td>53.74 ± 0.80<br>56.27 ± 0.34<br>55.47 ± 1.45<br><strong>56.94</strong> ± 0.44</td></tr><tr><td>Least diverse using Latin script</td><td>5<br>10<br>20</td><td>58.86 ± 0.28<br>58.77 ± 0.22<br>59.75 ± 0.04</td><td>65.49 ± 1.16<br>65.82 ± 1.30<br>68.65 ± 0.29</td><td>51.00 ± 1.02<br>53.27 ± 1.10<br>56.44 ± 0.22</td></tr><tr><td><strong>Random Selection</strong></td><td></td><td></td><td></td><td></td></tr><tr><td>Random Seeded</td><td>5<br>10<br>20<br>40</td><td><strong>59.49</strong> ± 0.44<br>59.13 ± 0.50<br>59.27 ± 0.54<br><strong>59.49</strong> ± 0.20</td><td>67.46 ± 1.51<br>67.91 ± 0.78<br>68.26 ± 0.46<br><strong>68.69</strong> ± 0.32</td><td>54.18 ± 1.44<br>54.29 ± 1.59<br><strong>56.66</strong> ± 1.18<br>56.04 ± 0.73</td></tr></tbody></table>
<table><caption>Table 11: Ablation studies: Accuracy of XLM-R on NLI, PoS-Tagging, NER tasks. Results are averaged across 4 seeds along with the standard deviation.</caption><thead><tr><th>Method</th><th>#Languages</th><th>NLI</th><th>PoS-Tagging</th><th>NER</th></tr></thead><tbody><tr><td colspan="5"><strong>Most featural diversity from English</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td>59.49 ± 0.25</td><td><strong>68.83 ± 0.40</strong></td><td>52.75 ± 1.66</td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>59.00 ± 0.21</td><td>67.46 ± 0.36</td><td>51.90 ± 2.04</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>58.79 ± 0.15</td><td>67.13 ± 0.12</td><td>53.95 ± 1.32</td></tr><tr><td>Joshi Class = 4,5</td><td>20</td><td>58.75 ± 0.24</td><td>67.27 ± 0.37</td><td>54.26 ± 1.83</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>59.04 ± 0.38</td><td>68.04 ± 0.55</td><td><strong>55.39 ± 0.98</strong></td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>59.00 ± 0.38</td><td>67.58 ± 0.54</td><td>52.55 ± 1.75</td></tr><tr><td>Unseen by mBERT</td><td>34</td><td><strong>59.86 ± 0.22</strong></td><td>68.25 ± 0.30</td><td>54.70 ± 0.57</td></tr><tr><td>Unseen by XLM-R</td><td>30</td><td>59.68 ± 0.44</td><td>68.62 ± 0.22</td><td>54.78 ± 1.39</td></tr><tr><td colspan="5"><strong>Most Phylogenetic Diversity</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td>59.01 ± 0.12</td><td>67.59 ± 0.19</td><td>53.02 ± 1.29</td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>58.88 ± 0.21</td><td>67.62 ± 0.34</td><td>51.15 ± 0.83</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>58.63 ± 0.20</td><td>67.91 ± 0.43</td><td>52.44 ± 0.68</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>58.79 ± 0.20</td><td>68.38 ± 0.25</td><td><strong>56.23 ± 0.43</strong></td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>59.07 ± 0.17</td><td>67.99 ± 0.22</td><td>55.08 ± 1.52</td></tr><tr><td>Unseen by mBERT</td><td>34</td><td><strong>59.77 ± 0.10</strong></td><td>68.58 ± 0.27</td><td>54.10 ± 1.09</td></tr><tr><td>Unseen by XLM-R</td><td>30</td><td>59.34 ± 0.37</td><td><strong>68.73 ± 0.13</strong></td><td>53.34 ± 0.98</td></tr><tr><td colspan="5"><strong>Most Script Diversity</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td><strong>59.11 ± 0.23</strong></td><td>67.48 ± 0.15</td><td><strong>54.33 ± 0.30</strong></td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>58.47 ± 0.21</td><td>67.48 ± 0.35</td><td>50.89 ± 1.63</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>58.66 ± 0.22</td><td><strong>67.52 ± 0.28</strong></td><td>51.64 ± 0.85</td></tr><tr><td>Joshi Class = 4,5</td><td>20</td><td>58.65 ± 0.16</td><td>67.07 ± 0.29</td><td>53.26 ± 0.61</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>58.67 ± 0.20</td><td>66.91 ± 1.18</td><td>50.46 ± 0.65</td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>58.67 ± 0.20</td><td>66.91 ± 1.18</td><td>50.46 ± 0.65</td></tr><tr><td colspan="5"><strong>Random Seeded</strong></td></tr><tr><td>Joshi Class = 2</td><td>10</td><td><strong>59.93 ± 0.05</strong></td><td>68.84 ± 0.68</td><td><strong>54.54 ± 1.80</strong></td></tr><tr><td>Joshi Class = 3</td><td>17</td><td>58.95 ± 0.19</td><td>67.05 ± 0.79</td><td>51.00 ± 0.98</td></tr><tr><td>Joshi Class = 3,4,5</td><td>37</td><td>58.80 ± 0.24</td><td>67.16 ± 0.43</td><td>53.80 ± 1.23</td></tr><tr><td>Joshi Class = 4,5</td><td>20</td><td>58.69 ± 0.29</td><td>67.47 ± 0.70</td><td>54.38 ± 1.82</td></tr><tr><td>Seen by mBERT</td><td>47</td><td>58.74 ± 0.13</td><td>67.39 ± 0.90</td><td>52.39 ± 2.13</td></tr><tr><td>Seen by XLM-R</td><td>51</td><td>58.88 ± 0.21</td><td>67.10 ± 0.44</td><td>53.26 ± 2.65</td></tr><tr><td>Unseen by mBERT</td><td>34</td><td>59.78 ± 0.38</td><td>69.01 ± 0.23</td><td>53.26 ± 1.33</td></tr><tr><td>Unseen by XLM-R</td><td>30</td><td>59.82 ± 0.26</td><td><strong>69.09 ± 0.14</strong></td><td>53.54 ± 1.42</td></tr></tbody></table>