LangSAMP: Language-Script Aware Multilingual Pretraining
Authors: Yihong Liu, Haotian Ye, Chunlan Ma, Mingyang Wang, Hinrich Schuetze
Accepted at: acl25

LANGSAMP: Language-Script Aware Multilingual Pretraining

Yihong Liu1,2,*, Haotian Ye1,2,*, Chunlan Ma1,2, Mingyang Wang1,2,3, and Hinrich Schütze1,2

1Center for Information and Language Processing, LMU Munich
2Munich Center for Machine Learning (MCML)
3Bosch Center for Artificial Intelligence
{yihong, yehao, chunlan, mingyang}@cis.lmu.de

Abstract

Recent multilingual pretrained language models (mPLMs) often avoid using language embeddings – learnable vectors assigned to individual languages. However, this places a significant burden on token representations to encode all language-specific information, which may hinder language neutrality. To address this limitation, we propose Language-Script Aware Multilingual Pretraining (LANGSAMP), a method that incorporates both language and script embeddings to enhance representation learning. Specifically, we integrate these embeddings into the output of the Transformer blocks before passing the final representations to the language modeling head for prediction. We apply LANGSAMP to the continual pretraining of XLM-R (Conneau et al., 2020) on a highly multilingual corpus covering more than 500 languages. The resulting model consistently outperforms the baseline in zero-shot crosslingual transfer across diverse downstream tasks. Extensive analysis reveals that language and script embeddings capture language- and script-specific nuances, which benefits more language-neutral representations, proven by improved pairwise cosine similarity. In our case study, we also show that language and script embeddings can be used to select better source languages for crosslingual transfer. We make our code and models publicly available at https://github.com/cisnlp/LangSAMP.

1 Introduction

Encoder-only mPLMs are often regarded as universal text encoders (Cer et al., 2018; Huang et al., 2019; Yang et al., 2020), where the sentence-level or token-level representations are applied to various downstream tasks across different languages (Wei et al., 2021). One of the most attractive aspects of these representations is their utility in crosslingual transfer (Zoph et al., 2016; Wu and Dredze, 2019; Artetxe et al., 2020a). That is, representations from a single source language can be used to fine-tune a multilingual task-specific model (e.g., an mPLM + a task-specific classifier). The fine-tuned model can be applied directly to other languages, without further training. Such a pipeline is particularly useful for low-resource languages, where training data is often scarce (Artetxe et al., 2020b).

The effectiveness of this pipeline depends on the transferability of crosslingual representations. However, previous studies have shown that the representations from recent mPLMs encode a lot of language- and script-specific information (Datta et al., 2020; Chang et al., 2022; Wen-Yi and Mimno, 2023). This is generally not advantageous, as language neutrality, i.e., representations from different languages share a unified subspace, is important for effective crosslingual transfer (Libovický et al., 2020; Chang et al., 2022; Hua et al., 2024). While

![An illustration of LANGSAMP for a single batch. Each text may come from different languages and different scripts. Language and script embeddings are added to the transformer output before feeding into the language modeling head. This setup improves the language neutrality of the representations as the auxiliary embeddings share the burden by encoding some language- and script-specific information useful for decoding specific tokens in masked language modeling.](page_1012_370_420_312.png)

Figure 1: An illustration of LANGSAMP for a single batch. Each text may come from different languages and different scripts. Language and script embeddings are added to the transformer output before feeding into the language modeling head. This setup improves the language neutrality of the representations as the auxiliary embeddings share the burden by encoding some language- and script-specific information useful for decoding specific tokens in masked language modeling.
some approaches attempt to post-align these representations (Cao et al., 2020; Pan et al., 2021; Liu et al., 2024b; Xhelili et al., 2024), limited efforts have focused on enhancing language neutrality from the architectural perspective of mPLMs during pretraining.

Early mPLMs, such as XLM (Conneau and Lample, 2019), leverage language embeddings – learnable vectors assigned to different languages. These embeddings are added to the token embeddings before being fed into the transformer (Vaswani et al., 2017) blocks, aiming to alleviate the burden of encoding language-specific information within the token embeddings. Language embeddings can also guide generation toward the correct target language in machine translation (Conneau and Lample, 2019; Song et al., 2019; Liu et al., 2022). However, more recent mPLMs, such as XLM-R (Conneau et al., 2020) and mBERT (Devlin et al., 2019), have discarded these embeddings. The two primary reasons are that (1) mPLMs are expected to have a single, unified parameter set for all languages, and (2) they need to function seamlessly as universal text encoders without requiring language IDs as input. However, the removal inevitably reduces the language neutrality of token embeddings and representations (contextual token embeddings), which may negatively impact crosslingual transfer.

To address this limitation, this work proposes Language-Script Aware Multilingual Pretraining (LANGSAMP), a method that incorporates both language and script embeddings to facilitate better representation learning. Instead of adding these embeddings to the token embeddings before feeding them into the transformer blocks, we add them to the output of the transformer blocks (final contextual token embeddings) before feeding them into the language modeling head, as shown in Figure 1. In the pretraining phase, language and script IDs are required to obtain language and script embeddings, offloading the burden and helping decode specific tokens in masked language modeling. After pretraining, the backbone (token embeddings and transformer blocks) can function seamlessly as a universal text encoder, which can be fine-tuned together with a task-specific classifier for downstream tasks, without any language or script IDs as input, which are the same as most recent mPLMs.

To validate our approach, we continually pretrain XLM-R (Conneau et al., 2020) using LANGSAMP on Glot500-c (ImaniGooghari et al., 2023), a multilingual dataset containing over 500 languages. We evaluate the resulting model across a diverse set of downstream tasks, including sentence retrieval, text classification, and sequence labeling, consistently achieving superior performance compared to the baseline. We show that better language neutrality is achieved – LANGSAMP improves the pairwise cosine similarity across languages. Additionally, we observe that language and script embeddings encapsulate typological features, making their similarities a useful resource for selecting optimal source languages in crosslingual transfer.

Our main contributions are as follows: (i) We propose LANGSAMP, an effective multilingual pretraining method to improve the language neutrality of representations. (ii) We conduct extensive experiments across a spectrum of downstream tasks, demonstrating that our method consistently improves crosslingual transfer performance. (iii) Our case study shows that language embeddings, as a byproduct, can effectively assist in selecting the optimal source language for crosslingual transfer.

2 Related Work

2.1 Multilingual Pretrained Language Models

Multilingual pretrained language models (mPLMs) are models that are trained on many languages, with one or multiple self-supervised objectives, such as masked language modeling (MLM) (Devlin et al., 2019) or causal language modeling (Radford et al., 2019). These models can be generally classified as encoder-only (Devlin et al., 2019; Conneau et al., 2020; Liang et al., 2023), encoder-decoder (Liu et al., 2020; Fan et al., 2021; Xue et al., 2021), and decoder-only models (Lin et al., 2022; Shliahzko et al., 2022; Scao et al., 2022). Decoder-only models that have considerably many parameters and are pretrained on a lot of data are also referred to as large language models (LLMs) (Achiam et al., 2023; Touvron et al., 2023; Üstün et al., 2024), which are good at natural language generation tasks, typically for high- and medium-resource languages. In parallel, some recent encoder-only models attempt to scale horizontally, i.e., cover more languages, especially low-resource ones (Ogueji et al., 2021; Alabi et al., 2022; ImaniGooghari et al., 2023; Liu et al., 2024a). These highly multilingual encoder-only models are particularly good at understanding tasks in a zero-shot crosslingual fashion.
2.2 Language Embeddings

Language embeddings are vectors that explicitly or implicitly capture the linguistic characteristics of languages. Early works construct such embeddings using prior knowledge of the languages, resulting in vectors where each dimension encodes a specific linguistic feature (Östling, 2015; Ammar et al., 2016; Littell et al., 2017). However, such features have to be manually defined and may be unavailable for less-studied languages (Yu et al., 2021). Therefore, researchers also explore learning language embeddings directly from parallel corpora (Malaviya et al., 2017; Östling and Tiedemann, 2017; Bjerva and Augenstein, 2018; Tan et al., 2019; Liu et al., 2023; Chen et al., 2023) or monolingual corpora (Conneau and Lample, 2019; Yu et al., 2021). This is usually done by assigning an ID to each language, initializing a fixed-length learnable vector, and integrating the vector into the input from that language. The embeddings can capture linguistic features and help crosslingual tasks, e.g., guiding language-specific generation in machine translation in XLM (Conneau and Lample, 2019). This line of approaches requires language IDs as input for both pretraining and downstream fine-tuning. In contrast, language embeddings are only leveraged in our pretraining. The backbone can be used as a universal text encoder without language IDs for fine-tuning on downstream tasks.

3 Methodology

We present LangSAMP, an approach that incorporates both language and script embeddings to facilitate learning more language-neutral representations in multilingual pretraining. LangSAMP preserves the same architecture as the most recent multilingual encoder-only models, except for requiring auxiliary language and script IDs/embeddings in pretraining. In the fine-tuning stage, these auxiliary IDs and embeddings are not required. We introduce the key components in the following.

3.1 Language and Script Embeddings

Language and script embeddings are introduced to share the token representations’ burden of encoding language- and script-specific information. Let \( E^{Lang} \in \mathbb{R}^{L \times D} \) and \( E^{Script} \in \mathbb{R}^{S \times D} \) be the language and script embeddings, respectively, where \( L \) is the number of languages, \( S \) is the number of scripts, and \( D \) is the embedding dimension of the model. We use \( E^{Lang}_l \) (resp. \( E^{Script}_s \)) to denote the embedding of a specific language \( l \) (resp. script \( s \)). Similar to token embeddings (which represent relations between tokens in vector space), the language/script embeddings are also expected to capture structural and typological similarities of languages (\S 5.2) and be useful for selecting good source language for crosslingual transfer (\S 5.4).

3.2 Language-Script Aware Modeling

In the standard MLM pretraining, Transformer blocks generate the final representation at a masked position. Subsequently, this representation is fed to the language modeling head to reconstruct the original token. Since the original token is used by a specific language and written in a specific script, language- or script-specific information is particularly necessary to decode this token. From this perspective, the Transformer output used for decoding is not language-neutral by nature. Our intuition is that we can ease the decoding by giving hints (e.g., the token should be generated in a specific language or script) to the language modeling head. In this way, the output of the Transformer blocks does not need to encode much language- and script-specific information, and can thus be more language-neutral. Inspired by this, we add language and script embeddings to the output of Transformer blocks and feed the resulting representations to the language modeling head for decoding, as shown in Figure 2.

Formally, let a training instance (an input sentence) be \( X = [x_1, x_2, \cdots, x_n] \) that comes from language \( l \) and is written in script \( s \). We feed \( X \) into Transformer blocks and obtain the final contextualized embeddings from the last layer: \( H = [h_1, h_2, \cdots, h_n] \). We then add the language and script embedding to these outputs to form the final representations: \( o_i = h_i + E^{Lang}_l + E^{Script}_s \). The final representations at the masked positions are used to decode the original tokens in MLM:

\[
\mathcal{L}_{MLM} = - \sum_{i \in \mathcal{M}} \log P_{MLM}(x_i|o_i)
\]

where \( \mathcal{M} \) is the set of masked positions in \( X \) and \( P_{MLM}(x_i|o_i) \) is the probability of decoding the original token \( x_i \) given the final representation \( o_i \), which is computed by the language modeling head. Since \( E^{Lang}_l \) and \( E^{Script}_s \) provide language and script-specific information, we expect that \( h_i \) will be more language-neutral (\S 5.3), which is beneficial to zero-shot crosslingual transfer (\S 4.3).
Figure 2: Illustration of LANGSAMP applied to a German sentence (left) and a Ukrainian sentence (right), both meaning “I like the cute cat”. Language and script embeddings are added to the outputs from the transformer block. The resulting representation is used to predict the original tokens at the [mask] positions in MLM training.

3.3 Fine-tuning on Downstream tasks

Since we only leverage language and script embedding in the pretraining for MLM, the core architecture (token embeddings + Transformer blocks) remains the same as most mainstream mPLMs, such as XLM-R. In this way, we do not need any language or script IDs as input to obtain the Transformer output, i.e., the final contextualized embeddings \( \mathbf{H} \). This means our pretrained model can be fine-tuned in the standard way in the NLP pipeline. Specifically, for any downstream tasks that require a task-specific classifier (either token-level or sequence-level tasks), we can feed the final contextualized embeddings \( \mathbf{H} = [h_1, h_2, \cdots, h_n] \) to the classifier and update the model parameters according to the fine-tuning objective, where language or script embeddings are not participating at all. In addition, as \( \mathbf{H} \) is more language-neutral thanks to LANGSAMP, we expect the representations to boost zero-shot crosslingual transfer (\S4.3).

It is important to note that we do not increase the number of parameters used to compute token or sentence representations, as the auxiliary language/script embeddings are employed only during pretraining. This contrasts with prior work, which often introduces additional components – such as Adapters – during downstream fine-tuning (Pfeiffer et al., 2022; Balne et al., 2024). Consequently, any improvements in downstream task performance can only be attributed to enhanced representations learned by the Transformer itself, rather than to added model capacity because of more parameters.

4 Experiments

4.1 Setups

Training Corpora and Tokenizer We use Glot500-c (ImaniGooghari et al., 2023), a corpus that has monolingual data from more than 500 languages written in 30 different scripts. We treat each language-script as a separate entity and refer to those covered by XLM-R (Conneau et al., 2020) as head languages, whereas the remaining are tail languages (also low-resource languages). We use the tokenizer of Glot500-m (ImaniGooghari et al., 2023), which is a SentencePiece Unigram tokenizer (Kudo and Richardson, 2018; Kudo, 2018) whose vocabulary is merged from the subwords in XLM-R and new subwords learned from Glot500-c.

Continued pretraining We use the weights from XLM-R to initialize our LANGSAMP model for MLM pretraining. Language and script embeddings are randomly initialized with dimensions \( \mathbb{R}^{610 \times 768} \) and \( \mathbb{R}^{30 \times 768} \) respectively. We continually train our model on Glot500-c, where we sample data from a multinomial distribution with a temperature of 0.3, to increase the amount of training instances of low- and medium-resource languages. We use AdamW optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2019) with \( (\beta_1, \beta_2) = (0.9, 0.999) \) and \( \epsilon = 1e\text{-}6 \). The initial learning rate is set to 5e-5. The effective batch size is 1,024 in each training step, where the gradient accumulation is 8 and the per-GPU batch size is 32. We train the model on 4 NVIDIA RTX6000 GPUs. Each training instance in a batch contains
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="2">tail</th>
    <th colspan="2">head</th>
    <th colspan="2">Latin</th>
    <th colspan="2">non-Latin</th>
    <th colspan="2">all</th>
  </tr>
  <tr>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
  </tr>
  <tr>
    <td>SR-B</td>
    <td>36.9 (0.0)</td>
    <td><b>39.5</b> (0.0)</td>
    <td>60.6 (0.0)</td>
    <td><b>61.3</b> (0.0)</td>
    <td>40.7 (0.0)</td>
    <td><b>42.8</b> (0.0)</td>
    <td>51.2 (0.0)</td>
    <td><b>53.5</b> (0.0)</td>
    <td>42.9 (0.0)</td>
    <td><b>45.1</b> (0.0)</td>
  </tr>
  <tr>
    <td>SR-T</td>
    <td>56.9 (0.0)</td>
    <td><b>58.6</b> (0.0)</td>
    <td>74.8 (0.0)</td>
    <td><b>76.1</b> (0.0)</td>
    <td>67.5 (0.0)</td>
    <td><b>68.7</b> (0.0)</td>
    <td>73.7 (0.0)</td>
    <td><b>75.6</b> (0.0)</td>
    <td>69.7 (0.0)</td>
    <td><b>71.1</b> (0.0)</td>
  </tr>
  <tr>
    <td>Taxi1500</td>
    <td>47.1 (4.8)</td>
    <td><b>50.8</b> (2.4)</td>
    <td>59.9 (2.9)</td>
    <td><b>61.2</b> (1.2)</td>
    <td>48.2 (4.6)</td>
    <td><b>51.7</b> (2.1)</td>
    <td>58.8 (3.1)</td>
    <td><b>60.1</b> (1.7)</td>
    <td>50.3 (4.2)</td>
    <td><b>53.4</b> (2.0)</td>
  </tr>
  <tr>
    <td>SIB200</td>
    <td>69.0 (1.4)</td>
    <td><b>70.2</b> (1.9)</td>
    <td>82.2 (1.4)</td>
    <td><b>82.6</b> (1.2)</td>
    <td>72.1 (1.3)</td>
    <td><b>73.1</b> (1.8)</td>
    <td>81.1 (1.5)</td>
    <td><b>81.7</b> (1.2)</td>
    <td>75.0 (1.3)</td>
    <td><b>75.9</b> (1.6)</td>
  </tr>
  <tr>
    <td>NER</td>
    <td>60.1 (0.6)</td>
    <td><b>60.8</b> (0.8)</td>
    <td>64.0 (0.6)</td>
    <td><b>64.1</b> (0.6)</td>
    <td>67.0 (0.5)</td>
    <td><b>67.6</b> (0.6)</td>
    <td><b>53.9</b> (0.7)</td>
    <td>53.9 (0.5)</td>
    <td>62.2 (0.5)</td>
    <td><b>62.6</b> (0.6)</td>
  </tr>
  <tr>
    <td>POS</td>
    <td>61.3 (1.0)</td>
    <td><b>61.4</b> (0.9)</td>
    <td>76.0 (0.4)</td>
    <td><b>76.2</b> (0.4)</td>
    <td>74.6 (0.5)</td>
    <td>74.5 (0.4)</td>
    <td>66.2 (1.0)</td>
    <td><b>66.8</b> (0.8)</td>
    <td>71.5 (0.6)</td>
    <td><b>71.6</b> (0.5)</td>
  </tr>
</table>

Table 1: Performance of LANGSAMP and baseline on six downstream tasks across five random seeds. We report the performance by grouping languages according to two characteristics: (1) whether it is a head or a tail language, and (2) whether it is written in Latin script or non-Latin script. The average performance within each group and the standard deviation (in parentheses) are computed. LANGSAMP consistently achieves on-par performance or outperforms the baseline across all groups and downstream tasks. Bold: best result for each group in each task.

sentences from the same language-script which are concatenated to a chunk of 512 tokens. Each batch contains instances from different language-scripts. We store checkpoints every 5K steps and apply early stopping with the best average performance on downstream tasks. We set the maximum steps to 150K. The training takes about 4 weeks.

Baseline To validate LANGSAMP, we create a baseline where language and script embeddings are not used. This baseline can be regarded as a reproduction of Glot500-m (ImaniGooghari et al., 2023). For a fair comparison, the training hyperparameters and training data (100% data of Glot500-c) are the same as LANGSAMP. However, in our ablation study §5.1, due to a constrained computing budget, we cannot continually pretrain model variants on full Glot500-c for validating each component individually (with/without language or script embeddings). Instead, we create such variants and pretrain them using a small portion (5%) of Glot500-c. As a result, the baseline model in Table 1 is different from the vanilla model in Table 2.

4.2 Downstream Tasks

We consider the following three evaluation types, with two datasets for each type. The evaluation is done in an English-centric zero-shot crosslingual transfer style for evaluation types that require fine-tuning. That is, we first fine-tune the pretrained model on the English train set, then select the best checkpoint on the English development set, and finally evaluate the best checkpoint on the test sets of all other languages. For Sentence Retrieval, which does not involve any fine-tuning, we simply use English as the retrieval query language. For all tasks, only a subset of languages (head and tail languages) supported by Glot500-c are considered. We show the detailed information of the used dataset and hyperparameter settings in §A. We introduce the evaluation types and datasets in the following.

Sentence Retrieval. We use Bible (SR-B) and Tatoeba (Artetxe and Schwenk, 2019) (SR-T). The pairwise similarity for retrieving the target sentences is calculated using the mean pooling of contextualized word embeddings at the 8th layer.

Text Classification. We use Taxi1500 (Ma et al., 2023) and SIB200 (Adelani et al., 2024). The former is a dataset based on the Bible, whereas the latter is based on FLORES-200 (Costa-jussà et al., 2022) with more modern genres like technology.

Sequence Labeling. We use WikiANN for named entity recognition (NER) (Pan et al., 2017) and Universal Dependencies (de Marneffe et al., 2021) for Part-Of-Speech (POS) tagging.

4.3 Results and Discussion

We evaluate the LANGSAMP model and baseline to understand how the integration of language and script embeddings influences crosslingual transfer. We group the transfer target languages based on two characteristics: (1) whether it is a head or tail language, and (2) whether it is written in Latin or a non-Latin script. This grouping aims to directly identify the effectiveness of LANGSAMP on low-resource languages and languages written in a less common script. The results are shown in Table 1.

Both tail and head languages benefit. We observe consistent improvements in tail and head languages across tasks. The enhancement is more obvious in tail languages. For example, LANGSAMP improves the performance by 7% for tail languages vs 1% for head languages in SR-B. A similar phenomenon can also be seen for other tasks. This pattern indicates that LANGSAMP can be more
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">SR-B</th>
    <th colspan="3">SR-T</th>
    <th colspan="3">Taxi1500</th>
    <th colspan="3">SIB200</th>
    <th colspan="3">NER</th>
    <th colspan="3">POS</th>
  </tr>
  <tr>
    <th>tail</th><th>head</th><th>all</th>
    <th>tail</th><th>head</th><th>all</th>
    <th>tail</th><th>head</th><th>all</th>
    <th>tail</th><th>head</th><th>all</th>
    <th>tail</th><th>head</th><th>all</th>
    <th>tail</th><th>head</th><th>all</th>
  </tr>
  <tr>
    <td>vanilla model</td>
    <td>11.9</td><td>56.4</td><td>23.2</td>
    <td>46.0</td><td>77.7</td><td>68.6</td>
    <td>18.1</td><td>58.6</td><td>28.4</td>
    <td>56.1</td><td>83.0</td><td>68.3</td>
    <td>55.1</td><td>62.8</td><td>59.3</td>
    <td>49.9</td><td>75.7</td><td>67.8</td>
  </tr>
  <tr>
    <td>w/ \( E^{Lang} \)</td>
    <td>13.1</td><td>57.9</td><td>24.5</td>
    <td>49.1</td><td>79.0</td><td>70.5</td>
    <td>18.3</td><td>58.5</td><td>28.5</td>
    <td>57.2</td><td>82.7</td><td>68.8</td>
    <td>55.2</td><td>63.0</td><td>59.5</td>
    <td>49.9</td><td>75.8</td><td>67.8</td>
  </tr>
  <tr>
    <td>w/ \( E^{Script} \)</td>
    <td>12.5</td><td>57.4</td><td>23.9</td>
    <td>48.3</td><td>78.4</td><td>69.8</td>
    <td>18.5</td><td>57.0</td><td>28.2</td>
    <td>56.6</td><td>82.1</td><td>68.2</td>
    <td>55.1</td><td>62.4</td><td>59.0</td>
    <td>50.8</td><td>76.2</td><td>68.4</td>
  </tr>
  <tr>
    <td>w/ \( E^{Lang} \) and \( E^{Script} \)</td>
    <td>13.4</td><td>58.7</td><td>24.9</td>
    <td>49.1</td><td>79.5</td><td>70.8</td>
    <td>20.6</td><td>58.8</td><td>30.3</td>
    <td>57.9</td><td>83.0</td><td>69.3</td>
    <td>54.9</td><td>61.6</td><td>58.6</td>
    <td>49.7</td><td>75.6</td><td>67.6</td>
  </tr>
</table>

Table 2: Ablation study. We investigate the effectiveness of language and script embeddings on downstream performance. Note that the vanilla model and w/ \( E^{Lang} \) and \( E^{Script} \) are different from Baseline and LANGSAMP in Table 1 because of the smaller pretraining data size. By including both types of embeddings, the model achieves the overall best performance among all variants. Bold (underlined): best (second-best) result for each column.

helpful for those tail languages, for which the training data is scarce. With the help of language embeddings sharing the burden, the LANGSAMP model can have more language-neutral representations for these languages, resulting in better performance.

Both non-Latin and Latin languages benefit.
We observe similar consistent improvements when grouping languages into Latin or non-Latin languages. Different from the trend seen in tail/head groups, we see that no group shows an obvious larger enhancement compared to the other group. This can be explained by the fact that head and tail languages are distributed more equally in Latn and non-Latn groups. In addition, the improvements indicate the incorporation of script embeddings is helpful. By decoupling some script-specific information from the representations, the output generated by the backbone is more script-neutral, leading to better crosslingual transfer across scripts.

Improvements can vary slightly across tasks.
We observe more consistent large improvements for sequence-level tasks – retrieval and classification – where LANGSAMP outperforms the baseline in all groups. However, on sequence labeling tasks, LANGSAMP achieves very close performance to the baseline. For example, LANGSAMP scores are 0.1 less compared to the baseline on NER. This could be related to the difficulty of the tasks: both NER and POS are relatively easy tasks, and models can transfer well in prevalent classes, e.g., nouns, through shared vocabulary (ImaniGooghari et al., 2023; Liu et al., 2024a). Therefore, decoupling language- or script-specific information from the Transformer output can be less helpful for these tasks. Nevertheless, the overall improvements across tasks indicate the superiority of LANGSAMP compared with the baseline.

5 Analysis

5.1 Ablation Study
In the ablation study, we want to explore the effectiveness of language embeddings and script embeddings individually. However, due to a limited computation budget, we cannot run experiments on the full corpora for each variant. Therefore, we select 5% data for each language from Glot500-c and continually pretrain XLM-R using the same hyperparameters used in the main experiments described in §4.1. Specifically, we consider four variants: a) model without language/script embeddings; b) model with only language embeddings; c) model with only script embeddings; and d) model with both language and script embeddings. The performance of each variant is shown in Table 2.

Either language or script embeddings help.
The vanilla model achieves the overall worst performance among all model variants. As long as language or script embeddings are included, we generally observe a consistent improvement across all downstream tasks. This indicates that both language and script embeddings can share the burden of encoding too much language- and script-specific information in the token representations. As a result, the representations generated by the model variants with language or script embeddings are more language-neutral, benefiting the crosslingual transfer. The best overall performance is achieved when both language and script embeddings are used, suggesting that decoupling both language- and script-specific information would be the best option for improving crosslingual transfer.

Improvement varies across task types. Similar to the findings in §4.3, we observe that including the auxiliary embeddings is very helpful for sequence-level tasks, especially sentence retrieval, where we observe the highest enhancement, while
Figure 3: PCA visualizations of head language embeddings (left) and script embeddings (right). We see that some related languages and scripts are close to each other, indicating that they encode language- and script-specific information. Data imbalance may have caused some languages/scripts with limited data to appear as outliers.

less helpful for token-level tasks. It is also noticeable that including language embeddings is the most effective for sentence retrieval (either best or the second best per column). On the other hand, the sequence labeling task does not enjoy large improvements: most model variants achieve on-par performance with each other. The reason has been discussed in §4.3: NER and POS are relatively simple tasks since models can transfer easily in prevalent classes. Nevertheless, the overall results show the effectiveness of the auxiliary embeddings.

5.2 Qualitative Exploration: Visualization

We visualize language and script embeddings in Figure 3. Only head language embeddings are chosen for better readability. We observe that similar or related languages are located close to each other. For example, cmn and zho (simplified and traditional Chinese, lower left) are closest to each other, as are pes (Iranian Persian) and prs (Dari). The languages that are mutually influenced by Chinese to a large extent, jpn, kor, and vie, are also close to each other. Most European languages, as well as Indian languages that belong to the Indo-European family, form a rather dense cluster in the middle.

In the plot on the right, most scripts of the Indian subcontinent are found close to each other (Deva, Telu, Mlym, Taml, Knda, Sinh, Beng), despite some outliers (e.g., Gujr and Guru), probably due to the small amount of data that is written in these scripts. Hani and scripts of languages that are mutually influenced (Hang and Jpan) are not far from each other. The same is true for two very related scripts, Thai and Lao0. In summary, the learnable language and script embeddings can capture language- and script-specific information in the training, which can be helpful for the language-neutrality of the output of transformer blocks.

Figure 4: Similarity improvement (by percentage) from baseline to LANGSAMP in terms of the pairwise cosine similarity. Similarity is increased for each pair, indicating better language neutrality of the representations.

5.3 Quantitative Exploration: Similarity

We expect that LANGSAMP can generate more language-neutral representations, meaning that representations of semantically equivalent sentences from different languages are similar. To evaluate this, we selected 10 high-resource languages that differ typologically and use a diverse set of scripts: eng_Latn, rus_Cyril, zho_Hani, arb_Arab, hin_Deva, jpn_Jpan, tur_Latn, spa_Latn, ind_Latn, and swa_Latn. We calculated the pairwise cosine similarity of sentence representations using 100 randomly sampled parallel sentences from SR-B. Sentence representations are obtained by mean-pooling the token representations at the 8th layer, followed by subtracting the language centroid (the average of all 100 sentence representations for that language). We report the pairwise cosine similarity in Figure 5 in §B and show the improvement (by percentage) in Figure 4.

We can observe that the similarity between any
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="2">tail</th>
    <th colspan="2">head</th>
    <th colspan="2">Latin</th>
    <th colspan="2">non-Latin</th>
    <th colspan="2">all</th>
  </tr>
  <tr>
    <th>English</th><th>Donor</th>
    <th>English</th><th>Donor</th>
    <th>English</th><th>Donor</th>
    <th>English</th><th>Donor</th>
    <th>English</th><th>Donor</th>
  </tr>
  <tr>
    <td>Taxi1500</td><td>47.3</td><td>48.3</td><td>59.1</td><td>60.3</td><td>48.4</td><td>49.0</td><td>58.1</td><td>60.5</td><td>50.2</td><td>51.2</td>
  </tr>
  <tr>
    <td>SIB200</td><td>67.9</td><td>67.9</td><td>81.2</td><td>81.6</td><td>71.0</td><td>71.1</td><td>80.3</td><td>80.6</td><td>74.0</td><td>74.2</td>
  </tr>
  <tr>
    <td>NER</td><td>61.2</td><td>61.7</td><td>64.1</td><td>65.6</td><td>67.5</td><td>66.9</td><td>54.6</td><td>58.5</td><td>62.8</td><td>63.8</td>
  </tr>
  <tr>
    <td>POS</td><td>63.2</td><td>53.8</td><td>77.0</td><td>72.3</td><td>75.5</td><td>68.4</td><td>68.1</td><td>63.6</td><td>72.8</td><td>66.6</td>
  </tr>
</table>

Table 3: Performance of LANGSAMP, using English vs the closest donor language (based on cosine similarity induced from language embeddings) as the source language for zero-shot crosslingual transfer. Each number is the average over all target languages in a class. **Bold**: the result that is better for an English/Donor comparison.

<table>
  <tr>
    <th></th>
    <th colspan="2">Taxi1500</th>
    <th colspan="2">SIB200</th>
    <th colspan="2">NER</th>
    <th colspan="2">POS</th>
  </tr>
  <tr>
    <th></th>
    <th>eng</th><th>jpn</th>
    <th>eng</th><th>jpn</th>
    <th>eng</th><th>jpn</th>
    <th>eng</th><th>jpn</th>
  </tr>
  <tr>
    <td>tha</td><td>63.8</td><td>63.8</td><td>85.4</td><td>85.7</td><td>2.1</td><td>10.2</td><td>58.3</td><td>27.5</td>
  </tr>
  <tr>
    <td>yue</td><td>55.4</td><td>67.7</td><td>-</td><td>-</td><td>25.7</td><td>73.5</td><td>42.6</td><td>80.9</td>
  </tr>
  <tr>
    <td>san</td><td>-</td><td>-</td><td>72.9</td><td>76.6</td><td>38.4</td><td>53.4</td><td>25.5</td><td>32.7</td>
  </tr>
  <tr>
    <td>urd</td><td>-</td><td>-</td><td>79.1</td><td>80.6</td><td>65.1</td><td>76.8</td><td>69.7</td><td>89.7</td>
  </tr>
  <tr>
    <td>lin</td><td>47.1</td><td>54.7</td><td>68.2</td><td>73.3</td><td>47.6</td><td>55.9</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>run</td><td>48.0</td><td>55.2</td><td>65.2</td><td>72.7</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 4: Languages with large improvements when using the closest donor language. In each task, the first/second column indicates results using English/the donor language as the source language. “-” indicates the language is not covered by the task. **Bold**: best result for each language in each task.

two languages is improved in LANGSAMP. The enhancement is especially noticeable for typologically distinct languages using different scripts. For example, arb_Arab is in a different language family and written in a different script compared to the other 9 languages; the similarity involving arb_Arab is greatly improved: 4.7% for eng_Latn and 4.1% rus_Cyril. Importantly, since LANGSAMP does not incorporate additional parallel data, this improvement is solely attributed to the inclusion of language and script embeddings during pretraining. This indicates that LANGSAMP effectively generates more language-neutral representations by decoupling language- and script-specific features into auxiliary embeddings.

5.4 Case Study: Source Language Selection

Previous studies show language similarities have been useful for selecting good source languages for crosslingual transfer (Lin et al., 2019; Lauscher et al., 2020; Nie et al., 2023; Wang et al., 2023b,a; Lin et al., 2024). We expect this to also apply to the similarities induced by our language embeddings. Therefore, we conduct a case study and use the languages mentioned in §5.3 as the donor languages. When performing the downstream task for a specific target language, instead of always using English as the source language, we select the donor language that is the most cosine-similar to the target language. We evaluate the LANGSAMP model on Taxi1500, SIB200, NER, and POS in a zero-shot crosslingual transfer style. The aggregated results are reported in Table 3, and we select representative target languages that benefit from choosing a good donor language in Table 4.

Effects of donor vary across tasks. Our results suggest that the performance gain from using a donor language varies across tasks. The gain in the text classification task is more consistent than in the sequence labeling task. We assume the primary reason is that the training data for NER and POS are not parallel, and the size of the training data is highly variable across languages. For example, English has much more data than some of the other donor languages for these two tasks.

Non-Latin languages benefit more. For the text classification task, greater improvements can be observed in non-Latin script languages than in Latin script languages. This reflects previous findings that non-Latin script languages are less represented in mPLMs (Muller et al., 2021) and indicates the effectiveness of leveraging language embeddings in selecting better donor languages for them.

Donor is frequently from the same family. We find that language embeddings frequently identify a donor language of the same family as the target language, leading to a large performance improvement over English as the source. For example, as shown in Table 4, zho_Hani as a donor language for yue_Hani leads to large performance
gains on all three tasks. Similar gains are seen using hin_Deva for san_Deva. Positive effects can also be found across scripts, as in the case of using hin_Deva for urd_Arab, two very similar languages written in different scripts.

Interesting cases of unrelated donors. We also notice some interesting cases where the closest donor language is not or only partially related to the target language, but nevertheless aids transfer performance as shown in Table 4. For example, jpn_Jpan has a positive effect for tha_Thai. Similarly, for tuk_Latn, using rus_Cyril as the source achieves better transfer performance than English.

6 Conclusion

We propose LANGSAMP, a multilingual pretraining approach that leverages auxiliary language and script embeddings to facilitate more language-neutral representations by offloading the burden of encoding language- and script-specific information within the Transformer outputs. These embeddings are added to the output of the transformer blocks before being fed into the language modeling head for decoding. In this way, we keep the model structure simple, allowing it to function as a universal text encoder, without requiring language or script IDs as input, while easing the burden of the output encoding too much language- and script-specific information. Through extensive experiments, we show LANGSAMP consistently outperforms the baseline on various downstream tasks, especially in sequence-level tasks. Our ablation study confirms the effectiveness of both language and script embeddings. LANGSAMP exhibits improved language neutrality, as reflected by increased pairwise similarity across all donor languages. Furthermore, our case study demonstrates that the byproducts – auxiliary language/script embeddings – encode language- and script-specific information, which can facilitate the selection of optimal source languages for more effective crosslingual transfer.

7 Future Direction

Looking ahead, a promising research direction is to further explore and refine the use of auxiliary language/script embeddings to guide language-neutral representation learning, particularly in the middle layers of multilingual models. Prior studies have shown that intermediate layers of encoder-only or decoder-only models often exhibit higher language neutrality (Jalili Sabet et al., 2020; ImaniGooghari et al., 2023; Zhao et al., 2024; Li et al., 2025).

Future work could investigate ways to explicitly steer middle-layer representations toward greater neutrality, for example, by combining auxiliary embeddings with layer-specific objectives or contrastive learning alignment techniques.

Furthermore, language and script embeddings hold potential for enhancing controlled multilingual generation in decoder-only and encoder-decoder models, enabling more accurate and consistent generation in the target language based on instructions provided in the prompt.

Limitations

Due to the constraints of computing resources, we are not able to continue pretraining the model using the full Glot500-c data in our ablation study. However, as all variants are trained in a strictly controlled environment, their results can be compared in a fair way, and the consistent improvement suggests the effectiveness of the language embeddings.

In addition, we do not consider the possibility of introducing language and script embeddings before the Transformer blocks. Although this is also a possible architecture, it does not fulfill our aim and therefore is not relevant to us. Our primary prerequisite is that the resulting model can work as a universal text encoder without any language or script IDs as input, just like most highly multilingual models (e.g., XLM-R (Conneau et al., 2020) and mBERT (Devlin et al., 2019)). LANGSAMP only requires language or script IDs in the pretraining stage. After that, the backbone (token embeddings + the Transformer blocks) acts exactly as a universal text encoder. Investigating whether architectures that integrate language/script embeddings before the Transformer could improve language representations at scale is outside the scope of our work, but we consider it a promising direction for future research.

Another potential limitation is the coverage of languages and scripts. Our model uses 610 languages and 30 scripts from Glot500-c. For low-resource languages not supported by our model, we can still generate representations since language IDs are not required as input. However, without a corresponding language embedding, it becomes challenging to select the optimal donor language for crosslingual transfer. Nonetheless, when adapting to these languages, the language embeddings
can be expanded, similar to the approach commonly used for vocabulary extension.

Finally, while our results support a strong correlation between improved language neutrality and enhanced crosslingual transfer, we acknowledge that this relationship is not necessarily causal. However, prior studies have shown that improved language neutrality or alignment does not always yield better downstream outcomes (Gaschi et al., 2023; Hua et al., 2024; Liu et al., 2025), suggesting that language neutrality alone may not be a sufficient condition. We believe further research is needed to understand when and how language-neutral representations contribute effectively to transfer, which remains an open and important question for the multilingual NLP community.

Acknowledgments

This work was funded by Deutsche Forschungsgemeinschaft (project SCHU 2246/14-1) and The European Research Council (NonSequeToR, grant #740516).

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shaymal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

David Adelani, Hannah Liu, Xiaoyu Shen, Nikita Vassilyev, Jesujoba Alabi, Yanke Mao, Haonan Gao, and En-Shiu Lee. 2024. SIB-200: A simple, inclusive, and big evaluation dataset for topic classification in 200+ languages and dialects. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 226–245. St. Julian’s, Malta. Association for Computational Linguistics.

Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4336–4349, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2016. Many languages, one parser. Transactions of the Association for Computational Linguistics, 4:431–444.

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020a. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637, Online. Association for Computational Linguistics.

Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020b. A call for more rigor in unsupervised cross-lingual learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7375–7388, Online. Association for Computational Linguistics.

Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610.

Charith Chandra Sai Balne, Sreyoshi Bhaduri, Tamoghna Roy, Vinija Jain, and Aman Chadha. 2024. Parameter efficient fine tuning: A comprehensive analysis across applications. Preprint, arXiv:2404.13506.

Johannes Bjerva and Isabelle Augenstein. 2018. From phonology to syntax: Unsupervised linguistic typology at different levels with language embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 907–916, New Orleans, Louisiana. Association for Computational Linguistics.

Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.

Tyler Chang, Zhuowen Tu, and Benjamin Bergen. 2022. The geometry of multilingual language model representations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 119–136, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yiyi Chen, Russa Biswas, and Johannes Bjerva. 2023. CoLex2Lang: Language embeddings from semantic typology. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 673–684, Tórshavn, Faroe Islands. University of Tartu Library.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised
cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

Arindrima Datta, Bhuvana Ramabhadran, Jesse Emond, Anjuli Kannan, and Brian Roark. 2020. Language-agnostic multilingual modeling. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8239–8243.

Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Çelebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. Journal of Machine Learning Research, 22(107):1–48.

Felix Gaschi, Patricio Cerda, Parisa Rastin, and Yannick Toussaint. 2023. Exploring the relationship between alignment and cross-lingual transfer in multilingual transformers. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3020–3042, Toronto, Canada. Association for Computational Linguistics.

Tianze Hua, Tian Yun, and Ellie Pavlick. 2024. mOthello: When do cross-lingual representation alignment and cross-lingual transfer emerge in multilingual models? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1585–1598, Mexico City, Mexico. Association for Computational Linguistics.

Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2485–2494, Hong Kong, China. Association for Computational Linguistics.

Ayyoob ImaniGooghari, Peiqin Lin, Amir Hossein Kargaran, Silvia Severini, Masoud Jalili Sabet, Nora Kassner, Chunlan Ma, Helmut Schmid, André Martins, François Yvon, and Hinrich Schütze. 2023. Glot500: Scaling multilingual corpora and language models to 500 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1082–1117, Toronto, Canada. Association for Computational Linguistics.

Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. SimAlign: High quality word alignments without parallel training data using static and contextualized embeddings. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1627–1643, Online. Association for Computational Linguistics.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.

Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational Linguistics.

Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Association for Computational Linguistics.

Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. Association for Computational Linguistics.

Daoyang Li, Haiyan Zhao, Qingcheng Zeng, and Mengnan Du. 2025. Exploring multilingual probing in large language models: A cross-language analysis. Preprint, arXiv:2409.14459.

Davis Liang, Hila Gonen, Yuning Mao, Rui Hou, Marnan Goyal, Marjan Ghazvininejad, Luke Zettle-
moyer, and Madian Khabsa. 2023. XLM-V: Overcoming the vocabulary bottleneck in multilingual masked language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13142–13152, Singapore. Association for Computational Linguistics.

Jindřich Libovický, Rudolf Rosa, and Alexander Fraser. 2020. On the language neutrality of pre-trained multilingual representations. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1663–1674, Online. Association for Computational Linguistics.

Peiqin Lin, Chengzhi Hu, Zheyu Zhang, Andre Martins, and Hinrich Schutze. 2024. mPLM-sim: Better cross-lingual similarity and transfer in multilingual pretrained language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 276–310, St. Julian’s, Malta. Association for Computational Linguistics.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuhui Chen, Daniel Sinig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Yu-Hsiang Lin, Chian-Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. 2019. Choosing transfer languages for cross-lingual learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125–3135, Florence, Italy. Association for Computational Linguistics.

Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 8–14, Valencia, Spain. Association for Computational Linguistics.

Yihong Liu, Haris Jabbar, and Hinrich Schutze. 2022. Flow-adapter architecture for unsupervised machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1253–1266, Dublin, Ireland. Association for Computational Linguistics.

Yihong Liu, Peiqin Lin, Mingyang Wang, and Hinrich Schutze. 2024a. OFA: A framework of initializing unseen subword embeddings for efficient large-scale multilingual continued pretraining. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1067–1097, Mexico City, Mexico. Association for Computational Linguistics.

Yihong Liu, Chunlan Ma, Haotian Ye, and Hinrich Schutze. 2024b. TransliCo: A contrastive learning framework to address the script barrier in multilingual pretrained language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2476–2499, Bangkok, Thailand. Association for Computational Linguistics.

Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayooob ImaniGooghari, Orgest Xhelili, Haotian Ye, Chunlan Ma, François Yvon, and Hinrich Schütze. 2025. How transliterations improve crosslingual alignment. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2417–2433, Abu Dhabi, UAE. Association for Computational Linguistics.

Yihong Liu, Haotian Ye, Leonie Weissweiler, Philipp Wicke, Renhao Pei, Robert Zangenfeind, and Hinrich Schütze. 2023. A crosslingual investigation of conceptualization in 1335 languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12969–13000, Toronto, Canada. Association for Computational Linguistics.

Yinhai Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.

Chunlan Ma, Ayooob ImaniGooghari, Haotian Ye, Ehsaneddin Asgari, and Hinrich Schütze. 2023. Taxi1500: A multilingual dataset for text classification in 1500 languages. arXiv preprint arXiv:2305.08487.

Chaitanya Malaviya, Graham Neubig, and Patrick Littell. 2017. Learning language representations for typology prediction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2529–2535, Copenhagen, Denmark. Association for Computational Linguistics.

Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being unseen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462, Online. Association for Computational Linguistics.
Ercong Nie, Sheng Liang, Helmut Schmid, and Hinrich Schütze. 2023. Cross-lingual retrieval augmented prompt for low-resource languages. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8320–8340, Toronto, Canada. Association for Computational Linguistics.

Kelechi Ogueji, Yuxin Zhu, and Jimmy Lin. 2021. Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 116–126, Punta Cana, Dominican Republic. Association for Computational Linguistics.

Robert Östling. 2015. Word order typology through multilingual word alignment. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 205–211, Beijing, China. Association for Computational Linguistics.

Robert Östling and Jörg Tiedemann. 2017. Continuous multilinguality with language vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 644–649, Valencia, Spain. Association for Computational Linguistics.

Lin Pan, Chung-Wei Hang, Haode Qi, Abhishek Shah, Saloni Potdar, and Mo Yu. 2021. Multilingual BERT post-pretraining alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 210–219, Online. Association for Computational Linguistics.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada. Association for Computational Linguistics.

Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. 2022. Lifting the curse of multilinguality by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3479–3495, Seattle, United States. Association for Computational Linguistics.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.

Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. 2022. mgpt: Few-shot learners go multilingual. arXiv preprint arXiv:2204.07580.

Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 5926–5936. PMLR.

Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with language clustering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 963–973, Hong Kong, China. Association for Computational Linguistics.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008.

Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, and Hinrich Schütze. 2023a. GradSim: Gradient-based language grouping for effective multilingual training. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4631–4646, Singapore. Association for Computational Linguistics.

Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, and Hinrich Schütze. 2023b. NLNDE at SemEval-2023 task 12: Adaptive pretraining and source language selection for low-resource multilingual sentiment analysis. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 488–497, Toronto, Canada. Association for Computational Linguistics.
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021. On learning universal representations across languages. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Andrea W Wen-Yi and David Mimno. 2023. Hyperpolyglot LLMs: Cross-lingual interpretability in token embeddings. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1124–1131, Singapore. Association for Computational Linguistics.

Shijie Wu and Mark Dredze. 2019. Beto. bentz. becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 833–844, Hong Kong, China. Association for Computational Linguistics.

Orgest Xhelili, Yihong Liu, and Hinrich Schuetze. 2024. Breaking the script barrier in multilingual pre-trained language models with transliteration-based post-training alignment. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11283–11296, Miami, Florida, USA. Association for Computational Linguistics.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, Online. Association for Computational Linguistics.

Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 87–94, Online. Association for Computational Linguistics.

Dian Yu, Taiqi He, and Kenji Sagae. 2021. Language embeddings for typology and cross-lingual transfer learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7210–7225, Online. Association for Computational Linguistics.

Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1568–1575, Austin, Texas. Association for Computational Linguistics.

A Settings and Hyperparameters

We show the information of the evaluation datasets and used measures in Table 5 and introduce the detailed settings and hyperparameters as follows.

Sentence Retrieval  We use English-aligned sentences (up to 500 and 1000 for SR-B and SR-T, respectively) from languages covered by Glot500c (ImaniGooghari et al., 2023). No fine-tuning is needed for this evaluation type: we directly use each model as a text encoder and generate the sentence-level representation by averaging the contextual token embeddings at the 8th layer, similar to previous work (Jalili Sabet et al., 2020; ImaniGooghari et al., 2023; Liu et al., 2024a). We perform retrieval by sorting the pairwise similarities.

Text Classification  We add a 6-class or 7-class (for Taxi1500 and SIB200, respectively) sequence-level classification head onto the backbone model (no language or script IDs are required as input since the language modeling head is not needed in this sequence-level classification model). By default, we train the model on the English train set and store the best checkpoint on the English validation set. We train all models using AdamW optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2019) for a maximum of 40 epochs, with a learning rate of 1e-5 and an effective batch size of 16 (batch size of 8, gradient accumulation of 2). We use a single GTX 1080 Ti GPU for training. The evaluation is done in zero-shot transfer: we directly apply the best checkpoint to the test sets of all other languages.

Sequence Labeling  We add a 7-class or 18-class (for NER and POS, respectively) token-level classification head onto the backbone model (no language or script IDs are required as input since the language modeling head is not needed in this token-level classification model). Similarly, we train the model on the English train set and store the best checkpoint on the English validation set by default. We train all models using the AdamW optimizer (Kingma and Ba, 2015; Loshchilov and Hutter, 2019) for a maximum of 10 epochs. The learning rate is set to 2e-5, and the effective batch size is
<table>
  <tr>
    <th></th>
    <th>lhead</th>
    <th>ltail</th>
    <th>ILatnl</th>
    <th>lnon-Latnl</th>
    <th>#class measure (%)</th>
  </tr>
  <tr>
    <td>SR-B</td>
    <td>94</td>
    <td>275</td>
    <td>290</td>
    <td>79</td>
    <td>- top-10 Acc.</td>
  </tr>
  <tr>
    <td>SR-T</td>
    <td>70</td>
    <td>28</td>
    <td>64</td>
    <td>34</td>
    <td>- top-10 Acc.</td>
  </tr>
  <tr>
    <td>Taxi1500</td>
    <td>89</td>
    <td>262</td>
    <td>281</td>
    <td>70</td>
    <td>6 F1 score</td>
  </tr>
  <tr>
    <td>SIB200</td>
    <td>78</td>
    <td>94</td>
    <td>117</td>
    <td>55</td>
    <td>7 F1 score</td>
  </tr>
  <tr>
    <td>NER</td>
    <td>89</td>
    <td>75</td>
    <td>104</td>
    <td>60</td>
    <td>7 F1 score</td>
  </tr>
  <tr>
    <td>POS</td>
    <td>63</td>
    <td>28</td>
    <td>57</td>
    <td>34</td>
    <td>18 F1 score</td>
  </tr>
</table>

Table 5: Information of the evaluation datasets and used measures. lhead (resp. ltail): number of head (resp. tail) language-scripts. ILatnl (resp. lnon-Latnl): number of languages written in Latin script (resp. non-Latin scripts). #class: the number of the categories if it belongs to a text classification or sequence labeling task.

set to 32 (batch size of 8, gradient accumulation of 4). The training is done on a single GTX 1080 Ti GPU. The evaluation is done in zero-shot transfer: we directly apply the best checkpoint to the test sets of all other languages.

B Pairwise Cosine Similarity

As introduced in §5.3, we select 10 topologically different languages that are written in diverse scripts to assess the language neutrality: eng_Latn, rus_Cyril, zho_Hani, arb_Arab, hin_Deva, jpn_Jpan, tur_Latn, spa_Latn, ind_Latn, and swa_Latn. We report the pairwise cosine similarity for the baseline and LANGSAMP in Figure 5.

It can be observed that the similarity between any two languages in LANGSAMP is consistently higher than in the baseline. The absolute increase is small in general, due to the fact that (1) without the introduction of the auxiliary language and script embeddings, the baseline already assigns good similarity to translations and (2) LANGSAMP does not introduce any additional parallel data in the pretraining, which is usually regarded as important to improve the similarity. Nevertheless, the consistent improvement indicates that LANGSAMP effectively improves the language neutrality by decoupling language- and script-specific features into auxiliary embeddings.

C Results for Each Language Family

We report the aggregated results for each language family for each task in Table 6. We see consistent improvement for all language families in sentence retrieval and text classification tasks. For sequence tagging tasks, LANGSAMP achieves similar performance compared with the baseline. This trend is similar to the main results we report in §4.3.

D Complete Crosslingual Transfer Results

We report the complete results of English-centric zero-shot crosslingual performance of baseline and LANGSAMP for all tasks and languages in Table 7, 8 (SR-B), Table 9 (SR-T), Table 10, 11 (Taxi1500), 12 (SIB200), Table 13 (NER), and Table 14 (POS). Each result is the average over fine-tuning the baseline or LANGSAMP under five random seeds.

E Transfer Results Using English and Closest Donor Language

We report the complete results of the zero-shot crosslingual performance of LANGSAMP when using English and the closest donor language as the source language in Table 15, 16 (Taxi1500), 17 (SIB200), Table 18 (NER), and Table 19 (POS). Each result is directly obtained from a single run. We fine-tune the LANGSAMP using different donor languages under the same random seed.
Figure 5: Comparison between baseline (left) and LANGSAMP (right) in terms of the pairwise cosine similarity. LANGSAMP achieves better similarity for each pair, indicating improved language neutrality of the representations.

<table>
  <tr>
    <th colspan="13">SR-B</th>
  </tr>
  <tr>
    <th>(indo1319, 93)</th><th>(atla1278, 69)</th><th>(aust1307, 55)</th><th>(turk1311, 23)</th><th>(sino1245, 23)</th><th>(maya1287, 15)</th><th>(afro1255, 12)</th><th>(other, 79)</th><th>(all, 369)</th>
  </tr>
  <tr>
    <td>Baseline</td><td>61.4</td><td>37.3</td><td>42.9</td><td>60.9</td><td>31.6</td><td>15.5</td><td>29.5</td><td>31.3</td><td>42.9</td>
  </tr>
  <tr>
    <td>LANGSAMP</td><td>62.0</td><td>40.2</td><td>45.1</td><td>63.3</td><td>34.8</td><td>15.7</td><td>32.0</td><td>34.6</td><td>45.1</td>
  </tr>
  <tr>
    <th colspan="13">SR-T</th>
  </tr>
  <tr>
    <th>(indo1319, 54)</th><th>(atla1278, 2)</th><th>(aust1307, 7)</th><th>(turk1311, 7)</th><th>(sino1245, 3)</th><th>(maya1287, 0)</th><th>(afro1255, 5)</th><th>(other, 20)</th><th>(all, 98)</th>
  </tr>
  <tr>
    <td>Baseline</td><td>74.2</td><td>50.0</td><td>48.7</td><td>71.3</td><td>81.7</td><td>-</td><td>52.1</td><td>68.7</td><td>69.7</td>
  </tr>
  <tr>
    <td>LANGSAMP</td><td>75.2</td><td>50.6</td><td>50.2</td><td>74.6</td><td>83.0</td><td>-</td><td>54.2</td><td>70.5</td><td>71.1</td>
  </tr>
  <tr>
    <th colspan="13">Taxi1500</th>
  </tr>
  <tr>
    <th>(indo1319, 87)</th><th>(atla1278, 68)</th><th>(aust1307, 51)</th><th>(turk1311, 18)</th><th>(sino1245, 22)</th><th>(maya1287, 15)</th><th>(afro1255, 11)</th><th>(other, 79)</th><th>(all, 351)</th>
  </tr>
  <tr>
    <td>Baseline</td><td>60.8</td><td>42.6</td><td>51.6</td><td>60.3</td><td>49.5</td><td>42.4</td><td>35.7</td><td>46.1</td><td>50.3</td>
  </tr>
  <tr>
    <td>LANGSAMP</td><td>62.6</td><td>46.8</td><td>55.2</td><td>62.7</td><td>53.6</td><td>45.5</td><td>38.8</td><td>49.1</td><td>53.4</td>
  </tr>
  <tr>
    <th colspan="13">SIB200</th>
  </tr>
  <tr>
    <th>(indo1319, 71)</th><th>(atla1278, 33)</th><th>(aust1307, 17)</th><th>(turk1311, 10)</th><th>(sino1245, 5)</th><th>(maya1287, 0)</th><th>(afro1255, 13)</th><th>(other, 23)</th><th>(all, 172)</th>
  </tr>
  <tr>
    <td>Baseline</td><td>82.1</td><td>59.0</td><td>76.4</td><td>80.5</td><td>67.4</td><td>-</td><td>73.0</td><td>75.1</td><td>75.0</td>
  </tr>
  <tr>
    <td>LANGSAMP</td><td>82.7</td><td>60.5</td><td>78.0</td><td>81.8</td><td>68.7</td><td>-</td><td>73.1</td><td>75.7</td><td>75.9</td>
  </tr>
  <tr>
    <th colspan="13">NER</th>
  </tr>
  <tr>
    <th>(indo1319, 94)</th><th>(atla1278, 5)</th><th>(aust1307, 12)</th><th>(turk1311, 12)</th><th>(sino1245, 7)</th><th>(maya1287, 0)</th><th>(afro1255, 6)</th><th>(other, 28)</th><th>(all, 164)</th>
  </tr>
  <tr>
    <td>Baseline</td><td>66.8</td><td>60.4</td><td>58.8</td><td>62.1</td><td>37.1</td><td>-</td><td>53.8</td><td>56.7</td><td>62.2</td>
  </tr>
  <tr>
    <td>LANGSAMP</td><td>67.2</td><td>61.2</td><td>59.5</td><td>61.3</td><td>36.6</td><td>-</td><td>55.5</td><td>57.2</td><td>62.6</td>
  </tr>
  <tr>
    <th colspan="13">POS</th>
  </tr>
  <tr>
    <th>(indo1319, 54)</th><th>(atla1278, 2)</th><th>(aust1307, 4)</th><th>(turk1311, 5)</th><th>(sino1245, 3)</th><th>(maya1287, 1)</th><th>(afro1255, 6)</th><th>(other, 16)</th><th>(all, 91)</th>
  </tr>
  <tr>
    <td>Baseline</td><td>78.0</td><td>61.9</td><td>74.4</td><td>72.1</td><td>33.3</td><td>61.1</td><td>64.1</td><td>60.3</td><td>71.5</td>
  </tr>
  <tr>
    <td>LANGSAMP</td><td>78.0</td><td>61.0</td><td>74.4</td><td>71.9</td><td>35.8</td><td>58.8</td><td>64.9</td><td>60.6</td><td>71.6</td>
  </tr>
</table>

Table 6: Aggregated performance of the baseline and LANGSAMP for 7 major language families on all tasks. We report the average performance for indo1319 (Indo-European), atla1278 (Atlantic-Congo), aust1307 (Austronesian), turk1311 (Turkic), sino1245 (Sino-Tibetan), maya1287 (Mayan), and afro1255 (Afro-Asiatic). We classify the remaining languages into the group “other”. In addition, we report the average over all languages (group “all”). The number of languages in that family is shown in parentheses. Bold: best result for each task.
<table>
  <tr>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
  </tr>
  <tr><td>ace_Latin</td><td>43.8</td><td>49.4</td><td>ach_Latin</td><td>37.6</td><td>40.6</td><td>acr_Latin</td><td>17.6</td><td>18.6</td><td>afr_Latin</td><td>74.2</td><td>72.4</td></tr>
  <tr><td>agw_Latin</td><td>31.0</td><td>38.2</td><td>ahk_Latin</td><td>3.4</td><td>3.8</td><td>aka_Latin</td><td>41.8</td><td>48.4</td><td>aln_Latin</td><td>70.0</td><td>70.0</td></tr>
  <tr><td>als_Latin</td><td>54.4</td><td>54.4</td><td>alt_Cyril</td><td>53.8</td><td>57.0</td><td>alz_Latin</td><td>36.2</td><td>37.4</td><td>amh_Ethi</td><td>44.4</td><td>51.2</td></tr>
  <tr><td>aoj_Latin</td><td>15.6</td><td>18.6</td><td>arb_Arab</td><td>9.6</td><td>11.6</td><td>arm_Latin</td><td>18.2</td><td>23.0</td><td>ary_Arab</td><td>11.2</td><td>13.0</td></tr>
  <tr><td>arz_Arab</td><td>15.2</td><td>15.2</td><td>asm_Beng</td><td>59.2</td><td>59.0</td><td>ayr_Latin</td><td>37.6</td><td>46.0</td><td>azb_Arab</td><td>55.6</td><td>59.0</td></tr>
  <tr><td>aze_Latin</td><td>73.4</td><td>75.4</td><td>bak_Cyril</td><td>58.8</td><td>62.2</td><td>bam_Latin</td><td>38.4</td><td>44.8</td><td>ban_Latin</td><td>33.0</td><td>33.2</td></tr>
  <tr><td>bar_Latin</td><td>32.2</td><td>34.0</td><td>bha_Latin</td><td>26.2</td><td>31.0</td><td>bbc_Latin</td><td>60.8</td><td>58.8</td><td>bci_Latin</td><td>12.0</td><td>11.8</td></tr>
  <tr><td>bel_Latin</td><td>75.4</td><td>79.0</td><td>bel_Cyril</td><td>70.6</td><td>69.6</td><td>bcm_Latin</td><td>51.0</td><td>54.4</td><td>ben_Beng</td><td>53.4</td><td>55.4</td></tr>
  <tr><td>bbw_Latin</td><td>28.4</td><td>30.6</td><td>binm_Latin</td><td>31.4</td><td>42.8</td><td>bis_Latin</td><td>45.2</td><td>50.8</td><td>bod_Tibet</td><td>29.6</td><td>33.6</td></tr>
  <tr><td>bqc_Latin</td><td>27.4</td><td>29.2</td><td>bre_Latin</td><td>31.8</td><td>30.0</td><td>bts_Latin</td><td>62.4</td><td>62.0</td><td>btv_Latin</td><td>57.2</td><td>55.8</td></tr>
  <tr><td>bul_Cyril</td><td>79.8</td><td>80.0</td><td>bum_Latin</td><td>32.8</td><td>35.2</td><td>bjz_Latin</td><td>69.8</td><td>70.2</td><td>cab_Latin</td><td>11.6</td><td>11.8</td></tr>
  <tr><td>cac_Latin</td><td>10.8</td><td>11.8</td><td>cak_Latin</td><td>17.8</td><td>16.6</td><td>caq_Latin</td><td>26.0</td><td>29.8</td><td>cat_Latin</td><td>85.4</td><td>83.2</td></tr>
  <tr><td>cbk_Latin</td><td>54.8</td><td>56.2</td><td>cece_Latin</td><td>41.8</td><td>45.4</td><td>ceb_Latin</td><td>70.4</td><td>70.6</td><td>ces_Latin</td><td>68.2</td><td>67.0</td></tr>
  <tr><td>cfn_Latin</td><td>34.4</td><td>38.8</td><td>che_Cyril</td><td>10.2</td><td>11.2</td><td>chik_Latin</td><td>35.2</td><td>43.0</td><td>chv_Cyril</td><td>45.0</td><td>54.4</td></tr>
  <tr><td>ckb_Arab</td><td>31.2</td><td>32.8</td><td>cmn_Hani</td><td>41.4</td><td>40.8</td><td>cnh_Latin</td><td>38.2</td><td>43.2</td><td>chr_Cyril</td><td>67.2</td><td>70.0</td></tr>
  <tr><td>crs_Latin</td><td>85.6</td><td>84.4</td><td>csy_Latin</td><td>40.2</td><td>49.6</td><td>ctd_Latin</td><td>44.4</td><td>50.6</td><td>ctu_Latin</td><td>16.6</td><td>16.0</td></tr>
  <tr><td>cuk_Latin</td><td>17.0</td><td>17.0</td><td>cym_Latin</td><td>45.6</td><td>43.8</td><td>dan_Latin</td><td>72.4</td><td>71.8</td><td>deu_Latin</td><td>73.8</td><td>74.0</td></tr>
  <tr><td>djk_Latin</td><td>38.0</td><td>38.0</td><td>dln_Latin</td><td>46.6</td><td>51.4</td><td>dip_Latin</td><td>17.0</td><td>17.8</td><td>dyu_Latin</td><td>33.0</td><td>40.2</td></tr>
  <tr><td>dzo_Tibet</td><td>28.4</td><td>33.0</td><td>efi_Latin</td><td>41.6</td><td>53.6</td><td>ell_Greek</td><td>48.2</td><td>49.2</td><td>emn_Hani</td><td>69.4</td><td>69.4</td></tr>
  <tr><td>epo_Latin</td><td>67.4</td><td>65.8</td><td>est_Latin</td><td>66.4</td><td>66.0</td><td>eus_Latin</td><td>23.8</td><td>24.2</td><td>ewe_Latin</td><td>33.2</td><td>34.8</td></tr>
  <tr><td>fao_Latin</td><td>79.8</td><td>78.4</td><td>fas_Arab</td><td>80.2</td><td>84.2</td><td>fij_Latin</td><td>30.0</td><td>31.0</td><td>fi_latin</td><td>77.6</td><td>77.2</td></tr>
  <tr><td>fin_Latin</td><td>65.4</td><td>66.0</td><td>fon_Latin</td><td>20.2</td><td>25.2</td><td>fra_Latin</td><td>87.4</td><td>87.2</td><td>fry_Latin</td><td>47.0</td><td>44.0</td></tr>
  <tr><td>gaa_Latin</td><td>34.4</td><td>40.6</td><td>gil_Latin</td><td>30.0</td><td>31.6</td><td>giz_Latin</td><td>32.4</td><td>36.4</td><td>gkn_Latin</td><td>20.4</td><td>24.2</td></tr>
  <tr><td>gkp_Latin</td><td>13.2</td><td>14.6</td><td>gla_Latin</td><td>39.0</td><td>38.0</td><td>gle_Latin</td><td>41.2</td><td>38.4</td><td>glv_Latin</td><td>37.2</td><td>38.6</td></tr>
  <tr><td>gom_Latin</td><td>33.2</td><td>36.0</td><td>gor_Latin</td><td>21.8</td><td>23.0</td><td>grc_Greek</td><td>44.4</td><td>47.0</td><td>guac_Latin</td><td>9.8</td><td>8.2</td></tr>
  <tr><td>guy_Latin</td><td>28.2</td><td>31.2</td><td>guj_Gujr</td><td>69.8</td><td>67.6</td><td>gur_Latin</td><td>17.6</td><td>18.2</td><td>guw_Latin</td><td>36.8</td><td>45.4</td></tr>
  <tr><td>gya_Latin</td><td>27.6</td><td>32.6</td><td>gym_Latin</td><td>13.6</td><td>13.0</td><td>hat_Latin</td><td>76.4</td><td>74.6</td><td>hau_Latin</td><td>57.6</td><td>59.6</td></tr>
  <tr><td>haw_Latin</td><td>28.0</td><td>30.4</td><td>heb_Hebr</td><td>21.6</td><td>23.0</td><td>hif_Latin</td><td>33.2</td><td>34.6</td><td>hil_Latin</td><td>74.0</td><td>79.8</td></tr>
  <tr><td>hin_Deva</td><td>75.6</td><td>74.6</td><td>hin_Latin</td><td>34.2</td><td>36.2</td><td>hmo_Latin</td><td>44.2</td><td>57.0</td><td>hne_Deva</td><td>71.6</td><td>73.6</td></tr>
  <tr><td>hnj_Latin</td><td>39.6</td><td>46.6</td><td>hra_Latin</td><td>43.4</td><td>44.6</td><td>hrv_Latin</td><td>80.4</td><td>79.8</td><td>hui_Latin</td><td>19.8</td><td>22.0</td></tr>
  <tr><td>hun_Latin</td><td>65.6</td><td>69.0</td><td>hus_Latin</td><td>14.8</td><td>16.2</td><td>hye_Armn</td><td>62.8</td><td>65.6</td><td>iba_Latin</td><td>70.2</td><td>71.6</td></tr>
  <tr><td>ibo_Latin</td><td>32.4</td><td>31.6</td><td>ifa_Latin</td><td>26.2</td><td>29.0</td><td>fib_Latin</td><td>28.6</td><td>28.6</td><td>ikk_Latin</td><td>30.2</td><td>46.4</td></tr>
  <tr><td>ilo_Latin</td><td>53.4</td><td>54.4</td><td>ind_Latin</td><td>78.4</td><td>78.6</td><td>isl_Latin</td><td>71.0</td><td>71.8</td><td>ita_Latin</td><td>76.2</td><td>76.8</td></tr>
  <tr><td>ium_Latin</td><td>20.0</td><td>23.2</td><td>ixl_Latin</td><td>13.8</td><td>14.4</td><td>izz_Latin</td><td>19.6</td><td>22.6</td><td>jum_Latin</td><td>61.0</td><td>59.2</td></tr>
  <tr><td>jav_Latin</td><td>55.4</td><td>52.0</td><td>jpn_Jpan</td><td>65.8</td><td>67.6</td><td>kai_Cyril</td><td>71.2</td><td>75.0</td><td>kai_Latin</td><td>32.0</td><td>37.6</td></tr>
  <tr><td>kab_Latin</td><td>12.2</td><td>13.4</td><td>kac_Latin</td><td>22.2</td><td>27.0</td><td>kal_Latin</td><td>12.6</td><td>16.8</td><td>kna_Knda</td><td>50.0</td><td>52.8</td></tr>
  <tr><td>kat_Geor</td><td>49.6</td><td>52.4</td><td>kaz_Cyril</td><td>69.4</td><td>70.4</td><td>kbp_Latin</td><td>21.8</td><td>26.8</td><td>kek_Latin</td><td>16.6</td><td>18.6</td></tr>
  <tr><td>khm_Khmr</td><td>39.4</td><td>43.0</td><td>kia_Latin</td><td>24.6</td><td>28.8</td><td>kik_Latin</td><td>44.4</td><td>48.4</td><td>kin_Latin</td><td>56.6</td><td>60.2</td></tr>
  <tr><td>kir_Cyril</td><td>69.8</td><td>70.2</td><td>kjb_Latin</td><td>23.4</td><td>26.0</td><td>kjh_Cyril</td><td>45.6</td><td>50.6</td><td>kmm_Latin</td><td>33.8</td><td>38.0</td></tr>
  <tr><td>kmr_Cyril</td><td>42.0</td><td>40.2</td><td>kmr_Latin</td><td>60.2</td><td>60.4</td><td>knv_Latin</td><td>7.0</td><td>8.4</td><td>kor_Hang</td><td>60.8</td><td>64.0</td></tr>
  <tr><td>kpg_Latin</td><td>42.6</td><td>48.8</td><td>krc_Cyril</td><td>59.8</td><td>62.2</td><td>kri_Latin</td><td>61.4</td><td>62.6</td><td>ksd_Latin</td><td>31.4</td><td>41.0</td></tr>
  <tr><td>kss_Latin</td><td>5.2</td><td>6.0</td><td>ksw_Mymr</td><td>26.2</td><td>28.0</td><td>kua_Latin</td><td>43.0</td><td>43.8</td><td>lam_Latin</td><td>20.4</td><td>22.8</td></tr>
  <tr><td>lao_Laoo</td><td>41.6</td><td>47.2</td><td>lat_Latin</td><td>56.6</td><td>58.0</td><td>lav_Latin</td><td>69.8</td><td>71.2</td><td>ldi_Latin</td><td>22.4</td><td>22.0</td></tr>
  <tr><td>leh_Latin</td><td>46.8</td><td>45.8</td><td>luh_Latin</td><td>4.4</td><td>4.2</td><td>lin_Latin</td><td>64.6</td><td>71.0</td><td>lit_Latin</td><td>67.0</td><td>66.6</td></tr>
  <tr><td>loz_Latin</td><td>46.8</td><td>45.6</td><td>ltz_Latin</td><td>63.8</td><td>63.2</td><td>lug_Latin</td><td>37.2</td><td>40.8</td><td>luo_Latin</td><td>42.8</td><td>42.6</td></tr>
  <tr><td>lus_Latin</td><td>46.6</td><td>53.2</td><td>lzq_Hani</td><td>59.8</td><td>62.4</td><td>mad_Latin</td><td>42.6</td><td>44.6</td><td>mah_Latin</td><td>30.4</td><td>33.8</td></tr>
  <tr><td>mai_Deva</td><td>52.6</td><td>56.0</td><td>mal_Mlym</td><td>51.6</td><td>57.4</td><td>mam_Latin</td><td>10.2</td><td>10.2</td><td>mar_Deva</td><td>68.4</td><td>71.4</td></tr>
  <tr><td>mau_Latin</td><td>2.8</td><td>3.4</td><td>mbb_Latin</td><td>22.0</td><td>29.8</td><td>mck_Latin</td><td>55.6</td><td>53.4</td><td>mcr_Latin</td><td>34.2</td><td>40.8</td></tr>
  <tr><td>mco_Latin</td><td>6.6</td><td>6.4</td><td>mdy_Ethi</td><td>21.4</td><td>30.6</td><td>meu_Latin</td><td>48.8</td><td>52.0</td><td>mfe_Latin</td><td>77.4</td><td>77.4</td></tr>
</table>

Table 7: Top-10 accuracy of models on SR-B (Part I).
<table>
  <tr>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
  </tr>
  <tr>
    <td>mgh_Latn</td><td>17.4</td><td>20.8</td><td>mgr_Latn</td><td>48.6</td><td>47.2</td><td>mhr_Cyril</td><td>37.4</td><td>43.2</td><td>min_Latn</td><td>32.4</td><td>29.6</td>
  </tr>
  <tr>
    <td>miq_Latn</td><td>28.8</td><td>36.8</td><td>mkd_Cyril</td><td>78.4</td><td>78.8</td><td>mlg_Latn</td><td>60.2</td><td>61.2</td><td>mlt_Latn</td><td>48.0</td><td>50.4</td>
  </tr>
  <tr>
    <td>mos_Latn</td><td>32.2</td><td>32.8</td><td>mps_Latn</td><td>16.4</td><td>20.6</td><td>mri_Latn</td><td>43.6</td><td>55.0</td><td>mrw_Latn</td><td>34.0</td><td>40.6</td>
  </tr>
  <tr>
    <td>msa_Latn</td><td>43.6</td><td>44.2</td><td>nwmn_Latn</td><td>24.0</td><td>25.6</td><td>mxv_Latn</td><td>7.0</td><td>7.0</td><td>mya_Mymr</td><td>25.8</td><td>28.0</td>
  </tr>
  <tr>
    <td>myv_Cyril</td><td>26.6</td><td>30.6</td><td>mnzh_Latn</td><td>24.6</td><td>25.4</td><td>nan_Latn</td><td>13.2</td><td>13.6</td><td>naq_Latn</td><td>16.8</td><td>26.8</td>
  </tr>
  <tr>
    <td>nav_Latn</td><td>8.6</td><td>8.6</td><td>nbl_Latn</td><td>49.4</td><td>48.4</td><td>nch_Latn</td><td>21.6</td><td>21.6</td><td>ncj_Latn</td><td>18.8</td><td>19.4</td>
  </tr>
  <tr>
    <td>ndc_Latn</td><td>32.4</td><td>36.2</td><td>nde_Latn</td><td>51.0</td><td>54.8</td><td>ndo_Latn</td><td>41.0</td><td>44.0</td><td>nds_Latn</td><td>38.4</td><td>38.4</td>
  </tr>
  <tr>
    <td>nep_Deva</td><td>56.4</td><td>59.0</td><td>ngu_Latn</td><td>26.2</td><td>26.0</td><td>nia_Latn</td><td>25.6</td><td>28.0</td><td>nld_Latn</td><td>78.4</td><td>78.0</td>
  </tr>
  <tr>
    <td>nmf_Latn</td><td>25.6</td><td>28.2</td><td>nmb_Latn</td><td>33.2</td><td>38.8</td><td>nno_Latn</td><td>76.8</td><td>75.8</td><td>nob_Latn</td><td>85.4</td><td>85.0</td>
  </tr>
  <tr>
    <td>nor_Latn</td><td>85.8</td><td>83.4</td><td>opi_Deva</td><td>77.4</td><td>80.8</td><td>nse_Latn</td><td>48.4</td><td>51.8</td><td>nso_Latn</td><td>46.2</td><td>50.2</td>
  </tr>
  <tr>
    <td>nya_Latn</td><td>57.6</td><td>57.6</td><td>ynl_Latn</td><td>48.8</td><td>47.4</td><td>nyt_Latn</td><td>23.4</td><td>24.6</td><td>nzi_Latn</td><td>29.2</td><td>34.4</td>
  </tr>
  <tr>
    <td>ori_Orya</td><td>51.2</td><td>53.4</td><td>ory_Orya</td><td>46.4</td><td>49.8</td><td>oss_Cyril</td><td>41.4</td><td>56.4</td><td>ote_Latn</td><td>12.0</td><td>13.2</td>
  </tr>
  <tr>
    <td>pag_Latn</td><td>55.2</td><td>52.2</td><td>pam_Latn</td><td>37.4</td><td>41.2</td><td>pan_Guru</td><td>46.2</td><td>45.4</td><td>pap_Latn</td><td>72.8</td><td>75.0</td>
  </tr>
  <tr>
    <td>pau_Latn</td><td>17.0</td><td>23.4</td><td>pcm_Latn</td><td>69.8</td><td>69.4</td><td>pdt_Latn</td><td>69.4</td><td>66.0</td><td>pes_Arab</td><td>74.2</td><td>75.2</td>
  </tr>
  <tr>
    <td>pis_Latn</td><td>51.4</td><td>54.8</td><td>pls_Latn</td><td>27.0</td><td>31.8</td><td>plt_Latn</td><td>60.2</td><td>60.8</td><td>poh_Latn</td><td>10.6</td><td>11.4</td>
  </tr>
  <tr>
    <td>pol_Latn</td><td>73.8</td><td>75.6</td><td>pon_Latn</td><td>21.4</td><td>24.0</td><td>por_Latn</td><td>81.8</td><td>81.0</td><td>prk_Latn</td><td>42.0</td><td>47.4</td>
  </tr>
  <tr>
    <td>prs_Arab</td><td>84.6</td><td>87.0</td><td>pxm_Latn</td><td>18.2</td><td>19.8</td><td>qub_Latn</td><td>30.6</td><td>35.6</td><td>que_Latn</td><td>18.6</td><td>17.4</td>
  </tr>
  <tr>
    <td>qug_Latn</td><td>53.6</td><td>59.2</td><td>quh_Latn</td><td>40.2</td><td>43.8</td><td>quw_Latn</td><td>46.2</td><td>50.4</td><td>quy_Latn</td><td>47.4</td><td>54.4</td>
  </tr>
  <tr>
    <td>quz_Latn</td><td>59.4</td><td>63.6</td><td>qvi_Latn</td><td>49.2</td><td>57.6</td><td>rap_Latn</td><td>17.0</td><td>17.8</td><td>rar_Latn</td><td>20.4</td><td>19.8</td>
  </tr>
  <tr>
    <td>rmy_Latn</td><td>30.4</td><td>32.2</td><td>ron_Latn</td><td>69.4</td><td>69.0</td><td>rop_Latn</td><td>35.8</td><td>41.4</td><td>rug_Latn</td><td>37.8</td><td>38.4</td>
  </tr>
  <tr>
    <td>run_Latn</td><td>48.2</td><td>52.4</td><td>rus_Cyril</td><td>74.6</td><td>76.4</td><td>sag_Latn</td><td>39.6</td><td>45.4</td><td>sah_Cyril</td><td>43.4</td><td>45.8</td>
  </tr>
  <tr>
    <td>san_Deva</td><td>24.2</td><td>23.6</td><td>san_Latn</td><td>7.8</td><td>7.4</td><td>sha_Latn</td><td>28.0</td><td>29.2</td><td>seh_Latn</td><td>67.4</td><td>69.4</td>
  </tr>
  <tr>
    <td>sin_Sinh</td><td>45.6</td><td>49.0</td><td>silk_Latn</td><td>69.8</td><td>69.2</td><td>slv_Latn</td><td>61.2</td><td>60.8</td><td>sme_Latn</td><td>35.0</td><td>37.6</td>
  </tr>
  <tr>
    <td>smo_Latn</td><td>27.6</td><td>28.8</td><td>sna_Latn</td><td>38.4</td><td>41.2</td><td>snd_Arab</td><td>67.2</td><td>65.0</td><td>som_Latn</td><td>35.0</td><td>34.8</td>
  </tr>
  <tr>
    <td>sop_Latn</td><td>32.4</td><td>28.8</td><td>sot_Latn</td><td>48.4</td><td>52.4</td><td>spa_Latn</td><td>80.8</td><td>81.4</td><td>squ_Latn</td><td>62.2</td><td>64.8</td>
  </tr>
  <tr>
    <td>srn_Latn</td><td>28.2</td><td>26.6</td><td>srn_Latn</td><td>75.4</td><td>75.6</td><td>srp_Cyril</td><td>87.2</td><td>85.8</td><td>srp_Latn</td><td>85.8</td><td>85.4</td>
  </tr>
  <tr>
    <td>ssw_Latn</td><td>42.8</td><td>47.0</td><td>sun_Latn</td><td>52.0</td><td>54.0</td><td>suz_Deva</td><td>21.0</td><td>22.6</td><td>swe_Latn</td><td>78.6</td><td>77.0</td>
  </tr>
  <tr>
    <td>swh_Latn</td><td>71.6</td><td>71.4</td><td>ssn_Latn</td><td>20.6</td><td>20.8</td><td>tam_Tamil</td><td>47.0</td><td>50.6</td><td>tat_Cyril</td><td>68.2</td><td>70.4</td>
  </tr>
  <tr>
    <td>tbz_Latn</td><td>13.2</td><td>18.2</td><td>tca_Latn</td><td>10.0</td><td>13.8</td><td>tdt_Latn</td><td>50.0</td><td>53.6</td><td>tel_Telu</td><td>48.0</td><td>50.2</td>
  </tr>
  <tr>
    <td>teo_Latn</td><td>19.4</td><td>19.6</td><td>tgk_Cyril</td><td>69.2</td><td>69.4</td><td>tgl_Latn</td><td>79.6</td><td>78.0</td><td>tha_Thai</td><td>33.8</td><td>38.0</td>
  </tr>
  <tr>
    <td>tih_Latn</td><td>42.2</td><td>46.4</td><td>tr_Ethi</td><td>32.2</td><td>34.8</td><td>tlh_Latn</td><td>62.0</td><td>66.4</td><td>tob_Latn</td><td>11.6</td><td>11.4</td>
  </tr>
  <tr>
    <td>toh_Latn</td><td>36.8</td><td>41.8</td><td>toi_Latn</td><td>39.4</td><td>39.4</td><td>toi_Latn</td><td>14.8</td><td>12.6</td><td>ton_Latn</td><td>16.0</td><td>16.6</td>
  </tr>
  <tr>
    <td>top_Latn</td><td>6.6</td><td>6.0</td><td>tpi_Latn</td><td>58.0</td><td>62.2</td><td>tpm_Latn</td><td>27.4</td><td>23.0</td><td>tsn_Latn</td><td>32.6</td><td>34.6</td>
  </tr>
  <tr>
    <td>tso_Latn</td><td>50.0</td><td>51.0</td><td>tsz_Latn</td><td>21.2</td><td>25.8</td><td>tuc_Latn</td><td>25.6</td><td>32.4</td><td>tui_Latn</td><td>29.8</td><td>31.0</td>
  </tr>
  <tr>
    <td>tuk_Cyril</td><td>67.4</td><td>69.4</td><td>tuk_Latn</td><td>67.6</td><td>70.0</td><td>tum_Latn</td><td>58.4</td><td>57.0</td><td>tur_Latn</td><td>70.2</td><td>70.4</td>
  </tr>
  <tr>
    <td>twi_Latn</td><td>35.0</td><td>42.0</td><td>tyv_Cyril</td><td>44.2</td><td>43.4</td><td>tzh_Latn</td><td>19.0</td><td>19.8</td><td>tzo_Latn</td><td>14.2</td><td>13.6</td>
  </tr>
  <tr>
    <td>udm_Cyril</td><td>41.6</td><td>45.2</td><td>uig_Arab</td><td>47.4</td><td>50.8</td><td>uig_Latn</td><td>57.2</td><td>58.8</td><td>ukr_Cyril</td><td>67.0</td><td>68.0</td>
  </tr>
  <tr>
    <td>urd_Arab</td><td>60.4</td><td>61.4</td><td>uzb_Cyril</td><td>80.6</td><td>81.2</td><td>uzb_Latn</td><td>70.0</td><td>68.2</td><td>uzn_Cyril</td><td>82.4</td><td>83.0</td>
  </tr>
  <tr>
    <td>ven_Latn</td><td>37.2</td><td>42.0</td><td>vie_Latn</td><td>68.0</td><td>69.4</td><td>wal_Latn</td><td>35.0</td><td>43.4</td><td>war_Latn</td><td>42.6</td><td>44.0</td>
  </tr>
  <tr>
    <td>wbm_Latn</td><td>37.6</td><td>46.2</td><td>wol_Latn</td><td>31.8</td><td>33.2</td><td>xav_Latn</td><td>3.8</td><td>4.0</td><td>xho_Latn</td><td>42.6</td><td>44.2</td>
  </tr>
  <tr>
    <td>yan_Latn</td><td>16.4</td><td>27.2</td><td>yao_Latn</td><td>37.4</td><td>37.6</td><td>yap_Latn</td><td>15.8</td><td>19.6</td><td>yom_Latn</td><td>37.6</td><td>40.0</td>
  </tr>
  <tr>
    <td>yor_Latn</td><td>27.4</td><td>28.8</td><td>yua_Latn</td><td>13.2</td><td>12.8</td><td>yue_Hani</td><td>17.2</td><td>17.2</td><td>zai_Latn</td><td>29.0</td><td>30.6</td>
  </tr>
  <tr>
    <td>zho_Hani</td><td>41.6</td><td>41.8</td><td>zim_Latn</td><td>84.8</td><td>84.8</td><td>zom_Latn</td><td>39.6</td><td>45.0</td><td>zsm_Latn</td><td>90.0</td><td>91.0</td>
  </tr>
</table>

Table 8: Top-10 accuracy of models on SR-B (Part II).
<table>
  <tr>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
  </tr>
  <tr>
    <td>afr_Latn</td><td>77.9</td><td>80.4</td><td>umh_Ethi</td><td>47.0</td><td>52.4</td><td>ara_Arab</td><td>69.4</td><td>68.7</td><td>jarz_Arab</td><td>61.8</td><td>63.9</td>
  </tr>
  <tr>
    <td>ast_Latn</td><td>80.3</td><td>84.3</td><td>aze_Latn</td><td>82.6</td><td>84.1</td><td>bel_Cyril</td><td>83.6</td><td>83.0</td><td>ben_Beng</td><td>72.1</td><td>74.9</td>
  </tr>
  <tr>
    <td>bos_Latn</td><td>90.1</td><td>90.4</td><td>bre_Latn</td><td>17.4</td><td>18.2</td><td>bul_Cyril</td><td>87.5</td><td>89.2</td><td>cat_Latin</td><td>78.2</td><td>78.6</td>
  </tr>
  <tr>
    <td>cbk_Latn</td><td>49.4</td><td>48.0</td><td>ceb_Latn</td><td>39.0</td><td>42.5</td><td>ces_Latin</td><td>75.7</td><td>73.5</td><td>cnn_Hani</td><td>87.1</td><td>87.4</td>
  </tr>
  <tr>
    <td>csb_Latn</td><td>38.3</td><td>38.7</td><td>cym_Latin</td><td>52.2</td><td>55.0</td><td>dan_Latin</td><td>91.7</td><td>92.9</td><td>deu_Latin</td><td>95.5</td><td>95.7</td>
  </tr>
  <tr>
    <td>dtp_Latn</td><td>17.0</td><td>19.3</td><td>ell_Grek</td><td>79.3</td><td>82.7</td><td>epo_Latin</td><td>71.8</td><td>74.8</td><td>est_Latin</td><td>68.2</td><td>69.9</td>
  </tr>
  <tr>
    <td>eus_Latn</td><td>52.2</td><td>55.4</td><td>fao_Latin</td><td>77.1</td><td>75.6</td><td>fin_Latin</td><td>72.3</td><td>74.2</td><td>fra_Latin</td><td>85.3</td><td>85.2</td>
  </tr>
  <tr>
    <td>fry_Latn</td><td>75.1</td><td>79.2</td><td>gla_Latin</td><td>38.4</td><td>38.6</td><td>gle_Latin</td><td>44.8</td><td>48.3</td><td>glg_Latin</td><td>77.1</td><td>76.4</td>
  </tr>
  <tr>
    <td>gsw_Latn</td><td>58.1</td><td>63.2</td><td>heb_Hebr</td><td>71.4</td><td>74.9</td><td>hin_Deva</td><td>88.1</td><td>87.3</td><td>hrv_Latin</td><td>87.9</td><td>87.5</td>
  </tr>
  <tr>
    <td>hsb_Latn</td><td>49.7</td><td>49.7</td><td>hun_Latin</td><td>71.5</td><td>73.2</td><td>hye_Armen</td><td>79.1</td><td>81.3</td><td>ido_Latin</td><td>54.6</td><td>55.8</td>
  </tr>
  <tr>
    <td>ile_Latn</td><td>71.2</td><td>71.5</td><td>ina_Latin</td><td>89.2</td><td>90.7</td><td>ind_Latin</td><td>88.1</td><td>88.9</td><td>isl_Latin</td><td>84.0</td><td>84.5</td>
  </tr>
  <tr>
    <td>ita_Latin</td><td>84.1</td><td>85.7</td><td>jpn_Jpan</td><td>77.2</td><td>77.1</td><td>kab_Latin</td><td>10.8</td><td>11.0</td><td>kat_Geor</td><td>71.2</td><td>72.4</td>
  </tr>
  <tr>
    <td>kaz_Cyril</td><td>74.6</td><td>77.7</td><td>khm_Khmr</td><td>57.5</td><td>63.0</td><td>kor_Hang</td><td>80.8</td><td>81.1</td><td>kur_Latin</td><td>49.8</td><td>52.4</td>
  </tr>
  <tr>
    <td>lat_Latin</td><td>39.2</td><td>42.1</td><td>lfn_Latin</td><td>55.8</td><td>56.8</td><td>lit_Latin</td><td>70.4</td><td>72.9</td><td>lvs_Latin</td><td>76.2</td><td>78.1</td>
  </tr>
  <tr>
    <td>mal_Mlym</td><td>87.5</td><td>91.6</td><td>mar_Deva</td><td>79.8</td><td>81.6</td><td>mhr_Cyril</td><td>27.7</td><td>33.4</td><td>mkd_Cyril</td><td>79.6</td><td>79.4</td>
  </tr>
  <tr>
    <td>mon_Cyril</td><td>78.2</td><td>80.5</td><td>pds_Latin</td><td>71.3</td><td>72.5</td><td>nld_Latin</td><td>92.4</td><td>93.4</td><td>ano_Latin</td><td>85.5</td><td>87.4</td>
  </tr>
  <tr>
    <td>nob_Latin</td><td>94.5</td><td>95.3</td><td>oci_Latin</td><td>46.6</td><td>44.9</td><td>pam_Latin</td><td>10.2</td><td>10.2</td><td>pes_Arab</td><td>86.7</td><td>86.9</td>
  </tr>
  <tr>
    <td>pms_Latin</td><td>49.5</td><td>50.9</td><td>pol_Latin</td><td>84.3</td><td>83.4</td><td>por_Latin</td><td>90.2</td><td>90.7</td><td>ron_Latin</td><td>86.0</td><td>86.9</td>
  </tr>
  <tr>
    <td>rus_Cyril</td><td>91.6</td><td>92.1</td><td>sik_Latin</td><td>77.9</td><td>78.2</td><td>slv_Latin</td><td>76.2</td><td>75.9</td><td>spa_Latin</td><td>88.6</td><td>88.3</td>
  </tr>
  <tr>
    <td>squ_Latin</td><td>84.1</td><td>85.2</td><td>srp_Latin</td><td>89.7</td><td>89.6</td><td>swe_Latin</td><td>89.4</td><td>89.6</td><td>swh_Latin</td><td>45.1</td><td>44.9</td>
  </tr>
  <tr>
    <td>tam_Taml</td><td>50.2</td><td>45.0</td><td>tat_Cyril</td><td>71.2</td><td>74.6</td><td>tel_Telu</td><td>72.6</td><td>74.8</td><td>tgl_Latin</td><td>73.9</td><td>74.2</td>
  </tr>
  <tr>
    <td>tha_Thai</td><td>75.4</td><td>79.2</td><td>tuk_Latin</td><td>62.1</td><td>68.0</td><td>tur_Latin</td><td>79.1</td><td>82.0</td><td>tig_Arab</td><td>64.7</td><td>68.4</td>
  </tr>
  <tr>
    <td>ukr_Cyril</td><td>84.9</td><td>86.5</td><td>urd_Arab</td><td>78.5</td><td>81.7</td><td>uzb_Cyril</td><td>65.0</td><td>67.3</td><td>vie_Latin</td><td>88.9</td><td>88.8</td>
  </tr>
  <tr>
    <td>war_Latin</td><td>22.7</td><td>25.2</td><td>wuu_Hani</td><td>79.0</td><td>82.4</td><td>xho_Latin</td><td>54.9</td><td>56.3</td><td>yid_Hebr</td><td>65.8</td><td>67.6</td>
  </tr>
  <tr>
    <td>yue_Hani</td><td>79.0</td><td>79.3</td><td>zsm_Latin</td><td>90.2</td><td>91.0</td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 9: Top-10 accuracy of models on SR-T.
<table>
  <tr>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
  </tr>
  <tr><td>ace_Latin</td><td>66.6</td><td>ach_Latin</td><td>36.6</td><td>acr_Latin</td><td>45.6</td><td>afr_Latin</td><td>60.7</td></tr>
  <tr><td>agw_Latin</td><td>53.6</td><td>ahk_Latin</td><td>7.6</td><td>aka_Latin</td><td>43.3</td><td>aln_Latin</td><td>56.0</td></tr>
  <tr><td>als_Latin</td><td>56.6</td><td>alt_Cyril</td><td>48.5</td><td>alz_Latin</td><td>31.9</td><td>amh_Ethi</td><td>8.7</td></tr>
  <tr><td>aoj_Latin</td><td>36.8</td><td>ary_Latin</td><td>41.9</td><td>ary_Arab</td><td>33.8</td><td>arz_Arab</td><td>36.2</td></tr>
  <tr><td>asm_Beng</td><td>63.2</td><td>ayr_Latin</td><td>55.9</td><td>azb_Arab</td><td>63.7</td><td>aze_Latin</td><td>67.3</td></tr>
  <tr><td>bak_Cyril</td><td>60.8</td><td>bam_Latin</td><td>43.8</td><td>baq_Latin</td><td>42.6</td><td>bar_Latin</td><td>46.2</td></tr>
  <tr><td>bba_Latin</td><td>40.9</td><td>bci_Latin</td><td>31.3</td><td>bcu_Latin</td><td>55.0</td><td>bel_Cyril</td><td>59.8</td></tr>
  <tr><td>bem_Latin</td><td>45.5</td><td>ben_Beng</td><td>62.4</td><td>bhw_Latin</td><td>45.3</td><td>bim_Latin</td><td>49.4</td></tr>
  <tr><td>bis_Latin</td><td>67.2</td><td>bqc_Latin</td><td>32.3</td><td>bre_Latin</td><td>37.3</td><td>btn_Latin</td><td>54.0</td></tr>
  <tr><td>bul_Cyril</td><td>65.3</td><td>bum_Latin</td><td>39.5</td><td>bzh_Latin</td><td>66.2</td><td>cab_Latin</td><td>24.5</td></tr>
  <tr><td>cac_Latin</td><td>44.8</td><td>cak_Latin</td><td>53.2</td><td>caq_Latin</td><td>39.9</td><td>cat_Latin</td><td>63.6</td></tr>
  <tr><td>chk_Latin</td><td>62.2</td><td>cce_Latin</td><td>42.5</td><td>ceb_Latin</td><td>53.1</td><td>ces_Latin</td><td>61.6</td></tr>
  <tr><td>cfm_Latin</td><td>55.3</td><td>che_Cyril</td><td>17.6</td><td>chv_Cyril</td><td>56.7</td><td>cmn_Hani</td><td>67.8</td></tr>
  <tr><td>cnh_Latin</td><td>61.9</td><td>chr_Cyril</td><td>61.2</td><td>crs_Latin</td><td>65.9</td><td>csy_Latin</td><td>53.6</td></tr>
  <tr><td>ctd_Latin</td><td>53.5</td><td>ctu_Latin</td><td>52.2</td><td>cuk_Latin</td><td>40.2</td><td>cym_Latin</td><td>50.1</td></tr>
  <tr><td>dan_Latin</td><td>63.6</td><td>deu_Latin</td><td>53.4</td><td>djk_Latin</td><td>46.5</td><td>din_Latin</td><td>49.3</td></tr>
  <tr><td>dpf_Latin</td><td>50.2</td><td>dya_Latin</td><td>48.0</td><td>dzo_Tibetan</td><td>55.9</td><td>efi_Latin</td><td>52.4</td></tr>
  <tr><td>ell_Grek</td><td>59.8</td><td>eng_Latin</td><td>74.3</td><td>enn_Latin</td><td>72.2</td><td>epo_Latin</td><td>57.6</td></tr>
  <tr><td>est_Latin</td><td>56.8</td><td>eus_Latin</td><td>23.3</td><td>eue_Latin</td><td>43.6</td><td>fao_Latin</td><td>57.5</td></tr>
  <tr><td>fas_Arab</td><td>71.7</td><td>fiq_Latin</td><td>44.2</td><td>fil_Latin</td><td>57.5</td><td>fin_Latin</td><td>58.3</td></tr>
  <tr><td>fon_Latin</td><td>42.9</td><td>fra_Latin</td><td>65.4</td><td>fry_Latin</td><td>40.0</td><td>gaa_Latin</td><td>40.2</td></tr>
  <tr><td>gil_Latin</td><td>42.0</td><td>giz_Latin</td><td>45.1</td><td>gkn_Latin</td><td>38.3</td><td>gkp_Latin</td><td>32.1</td></tr>
  <tr><td>gla_Latin</td><td>48.3</td><td>gle_Latin</td><td>42.9</td><td>glv_Latin</td><td>39.7</td><td>gom_Latin</td><td>35.5</td></tr>
  <tr><td>gor_Latin</td><td>42.5</td><td>guc_Latin</td><td>33.4</td><td>gug_Latin</td><td>36.2</td><td>guj_Gujarati</td><td>68.8</td></tr>
  <tr><td>gur_Latin</td><td>34.1</td><td>guw_Latin</td><td>48.4</td><td>gya_Latin</td><td>40.6</td><td>gyn_Latin</td><td>41.1</td></tr>
  <tr><td>hat_Latin</td><td>62.8</td><td>hau_Latin</td><td>54.2</td><td>haw_Latin</td><td>30.2</td><td>heb_Hebrew</td><td>18.7</td></tr>
  <tr><td>hif_Latin</td><td>46.2</td><td>hil_Latin</td><td>65.1</td><td>hin_Deva</td><td>66.2</td><td>hmo_Latin</td><td>60.7</td></tr>
  <tr><td>hne_Deva</td><td>66.3</td><td>hni_Latin</td><td>63.4</td><td>hra_Latin</td><td>51.5</td><td>hrv_Latin</td><td>63.4</td></tr>
  <tr><td>hui_Latin</td><td>46.5</td><td>hun_Latin</td><td>64.2</td><td>hus_Latin</td><td>38.0</td><td>hye_Armn</td><td>70.0</td></tr>
  <tr><td>iba_Latin</td><td>57.6</td><td>ibo_Latin</td><td>58.2</td><td>ifa_Latin</td><td>49.1</td><td>ibb_Latin</td><td>50.3</td></tr>
  <tr><td>ikk_Latin</td><td>47.8</td><td>ilo_Latin</td><td>53.2</td><td>ind_Latin</td><td>76.4</td><td>isl_Latin</td><td>50.4</td></tr>
  <tr><td>ita_Latin</td><td>64.8</td><td>ium_Latin</td><td>57.5</td><td>ixl_Latin</td><td>32.2</td><td>izz_Latin</td><td>42.4</td></tr>
  <tr><td>jam_Latin</td><td>64.2</td><td>jav_Latin</td><td>45.9</td><td>jpn_Japan</td><td>64.9</td><td>kaa_Cyril</td><td>59.3</td></tr>
  <tr><td>kab_Latin</td><td>23.0</td><td>kac_Latin</td><td>49.0</td><td>kal_Latin</td><td>32.1</td><td>kan_Kanda</td><td>67.1</td></tr>
  <tr><td>kat_Geor</td><td>60.0</td><td>kaz_Cyril</td><td>65.2</td><td>kbp_Latin</td><td>35.3</td><td>kek_Latin</td><td>45.5</td></tr>
  <tr><td>khm_Khmr</td><td>69.0</td><td>kia_Latin</td><td>41.3</td><td>kik_Latin</td><td>42.7</td><td>kin_Latin</td><td>44.7</td></tr>
  <tr><td>kir_Cyril</td><td>67.4</td><td>kjb_Latin</td><td>46.8</td><td>kjh_Cyril</td><td>50.9</td><td>kmm_Latin</td><td>46.2</td></tr>
  <tr><td>kmr_Cyril</td><td>49.9</td><td>knv_Latin</td><td>44.5</td><td>kor_Hangul</td><td>70.2</td><td>kpg_Latin</td><td>64.4</td></tr>
  <tr><td>krc_Cyril</td><td>57.6</td><td>kri_Latin</td><td>59.4</td><td>ksd_Latin</td><td>54.8</td><td>kss_Latin</td><td>23.7</td></tr>
  <tr><td>ksw_Mynr</td><td>49.0</td><td>kua_Latin</td><td>42.6</td><td>lam_Latin</td><td>33.3</td><td>lao_Lao</td><td>72.7</td></tr>
  <tr><td>lat_Latin</td><td>58.5</td><td>lav_Latin</td><td>62.8</td><td>ldi_Latin</td><td>27.6</td><td>leh_Latin</td><td>45.1</td></tr>
  <tr><td>lhu_Latin</td><td>24.0</td><td>lin_Latin</td><td>47.6</td><td>lit_Latin</td><td>61.2</td><td>loz_Latin</td><td>51.2</td></tr>
  <tr><td>ltz_Latin</td><td>53.3</td><td>lug_Latin</td><td>44.0</td><td>luo_Latin</td><td>38.7</td><td>lus_Latin</td><td>48.1</td></tr>
  <tr><td>lzh_Hani</td><td>61.5</td><td>mad_Latin</td><td>60.6</td><td>mah_Latin</td><td>34.3</td><td>mai_Deva</td><td>65.1</td></tr>
  <tr><td>mal_Mlym</td><td>7.2</td><td>mam_Latin</td><td>29.2</td><td>mar_Deva</td><td>62.4</td><td>mau_Latin</td><td>7.0</td></tr>
  <tr><td>mbb_Latin</td><td>52.0</td><td>mck_Latin</td><td>41.1</td><td>mcn_Latin</td><td>36.6</td><td>mcq_Latin</td><td>24.4</td></tr>
  <tr><td>mld_Ethi</td><td>49.1</td><td>meu_Latin</td><td>49.4</td><td>mfe_Latin</td><td>68.8</td><td>mgh_Latin</td><td>32.9</td></tr>
  <tr><td>mgr_Latin</td><td>47.3</td><td>mhr_Cyril</td><td>41.6</td><td>min_Latin</td><td>51.2</td><td>miq_Latin</td><td>52.1</td></tr>
  <tr><td>mkd_Cyril</td><td>68.8</td><td>mlg_Latin</td><td>48.3</td><td>mil_Latin</td><td>51.3</td><td>mos_Latin</td><td>36.4</td></tr>
</table>

Table 10: F1 scores of models on Taxi1500 (Part I).
<table>
  <tr>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
  </tr>
  <tr>
    <td>mps_Latn</td><td>51.6</td><td>mri_Latn</td><td>56.2</td><td>mrv_Latn</td><td>43.2</td><td>mya_Mymr</td><td>49.8</td><td>nsm_Latn</td><td>47.9</td><td>msa_Latn</td><td>46.7</td><td>msa_Latn</td><td>48.9</td>
  </tr>
  <tr>
    <td>mwn_Latn</td><td>52.5</td><td>mxv_Latn</td><td>58.5</td><td>nav_Latn</td><td>16.0</td><td>ncj_Latn</td><td>27.6</td><td>myv_Cyril</td><td>57.2</td><td>myv_Cyril</td><td>57.8</td><td>nav_Latn</td><td>42.9</td><td>nav_Latn</td><td>48.1</td>
  </tr>
  <tr>
    <td>mzh_Latn</td><td>39.2</td><td>nan_Latn</td><td>41.9</td><td>naq_Latn</td><td>26.2</td><td>ndc_Latn</td><td>33.7</td><td>nav_Latn</td><td>40.3</td><td>nav_Latn</td><td>45.5</td><td>nav_Latn</td><td>22.2</td><td>nav_Latn</td><td>25.7</td>
  </tr>
  <tr>
    <td>nbl_Latn</td><td>46.7</td><td>nch_Latn</td><td>52.9</td><td>ndt_Latn</td><td>41.8</td><td>ndc_Latn</td><td>46.0</td><td>ndc_Latn</td><td>36.2</td><td>ndc_Latn</td><td>42.2</td><td>ndc_Latn</td><td>39.3</td><td>ndc_Latn</td><td>44.6</td>
  </tr>
  <tr>
    <td>nde_Latn</td><td>46.7</td><td>njo_Latn</td><td>52.9</td><td>nds_Latn</td><td>47.3</td><td>nds_Latn</td><td>51.2</td><td>nds_Latn</td><td>39.1</td><td>nds_Latn</td><td>48.6</td><td>nds_Latn</td><td>70.7</td><td>nds_Latn</td><td>72.5</td>
  </tr>
  <tr>
    <td>ngu_Latn</td><td>41.5</td><td>nld_Latn</td><td>44.1</td><td>nnf_Latn</td><td>62.7</td><td>nnf_Latn</td><td>62.1</td><td>nnf_Latn</td><td>43.1</td><td>nnf_Latn</td><td>47.8</td><td>nnf_Latn</td><td>38.5</td><td>nnf_Latn</td><td>46.0</td>
  </tr>
  <tr>
    <td>nno_Latn</td><td>63.1</td><td>nob_Latn</td><td>65.5</td><td>noh_Latn</td><td>60.6</td><td>noh_Latn</td><td>60.0</td><td>nor_Latn</td><td>61.7</td><td>opi_Deva</td><td>60.8</td><td>opi_Deva</td><td>69.7</td><td>opi_Deva</td><td>70.1</td>
  </tr>
  <tr>
    <td>nse_Latn</td><td>43.9</td><td>nso_Latn</td><td>45.1</td><td>nya_Latn</td><td>53.6</td><td>nya_Latn</td><td>52.4</td><td>nya_Latn</td><td>55.4</td><td>ory_Orya</td><td>61.9</td><td>ory_Orya</td><td>43.6</td><td>ory_Orya</td><td>47.0</td>
  </tr>
  <tr>
    <td>nyl_Latn</td><td>31.1</td><td>nzi_Latn</td><td>37.7</td><td>ori_Orya</td><td>34.4</td><td>ori_Orya</td><td>38.3</td><td>ori_Orya</td><td>70.3</td><td>ory_Orya</td><td>69.5</td><td>ory_Orya</td><td>71.4</td><td>ory_Orya</td><td>69.3</td>
  </tr>
  <tr>
    <td>oss_Cyril</td><td>48.0</td><td>ote_Latn</td><td>57.6</td><td>pag_Latn</td><td>35.6</td><td>pam_Latn</td><td>52.0</td><td>pam_Latn</td><td>54.3</td><td>pam_Latn</td><td>40.1</td><td>pam_Latn</td><td>45.4</td><td>pam_Latn</td><td>45.4</td>
  </tr>
  <tr>
    <td>pan_Guru</td><td>67.8</td><td>pap_Latn</td><td>66.4</td><td>pau_Latn</td><td>65.5</td><td>pau_Latn</td><td>66.2</td><td>pau_Latn</td><td>42.0</td><td>pcm_Latn</td><td>43.3</td><td>pcm_Latn</td><td>63.8</td><td>pcm_Latn</td><td>67.1</td>
  </tr>
  <tr>
    <td>pdi_Latn</td><td>58.1</td><td>pes_Arab</td><td>58.7</td><td>pis_Latn</td><td>70.5</td><td>pis_Latn</td><td>69.6</td><td>pis_Latn</td><td>67.5</td><td>pls_Latn</td><td>67.9</td><td>pls_Latn</td><td>46.5</td><td>pls_Latn</td><td>49.1</td>
  </tr>
  <tr>
    <td>plt_Latn</td><td>52.6</td><td>poh_Latn</td><td>50.1</td><td>pol_Latn</td><td>47.2</td><td>pol_Latn</td><td>48.2</td><td>pol_Latn</td><td>64.8</td><td>pon_Latn</td><td>68.8</td><td>pon_Latn</td><td>53.1</td><td>pon_Latn</td><td>53.9</td>
  </tr>
  <tr>
    <td>por_Latn</td><td>68.0</td><td>prk_Latn</td><td>72.3</td><td>prs_Arab</td><td>56.4</td><td>prs_Arab</td><td>68.9</td><td>prs_Arab</td><td>68.9</td><td>pxm_Latn</td><td>40.6</td><td>pxm_Latn</td><td>40.2</td><td>pxm_Latn</td><td>40.2</td>
  </tr>
  <tr>
    <td>qub_Latn</td><td>58.3</td><td>que_Latn</td><td>59.1</td><td>qui_Latn</td><td>51.0</td><td>qui_Latn</td><td>53.3</td><td>qui_Latn</td><td>64.5</td><td>quh_Latn</td><td>68.2</td><td>quh_Latn</td><td>62.4</td><td>quh_Latn</td><td>68.6</td>
  </tr>
  <tr>
    <td>quw_Latn</td><td>53.5</td><td>quy_Latn</td><td>55.6</td><td>quz_Latn</td><td>71.6</td><td>quz_Latn</td><td>70.6</td><td>quz_Latn</td><td>64.8</td><td>qvi_Latn</td><td>67.6</td><td>qvi_Latn</td><td>63.2</td><td>qvi_Latn</td><td>65.2</td>
  </tr>
  <tr>
    <td>rap_Latn</td><td>47.9</td><td>rar_Latn</td><td>49.4</td><td>rmy_Latn</td><td>52.7</td><td>rmy_Latn</td><td>45.0</td><td>rmy_Latn</td><td>47.9</td><td>ron_Latn</td><td>59.4</td><td>ron_Latn</td><td>59.4</td><td>ron_Latn</td><td>66.7</td>
  </tr>
  <tr>
    <td>rop_Latn</td><td>57.0</td><td>rug_Latn</td><td>58.1</td><td>run_Latn</td><td>51.2</td><td>run_Latn</td><td>55.1</td><td>run_Latn</td><td>48.2</td><td>rus_Cyril</td><td>53.6</td><td>rus_Cyril</td><td>69.4</td><td>rus_Cyril</td><td>72.3</td>
  </tr>
  <tr>
    <td>sag_Latn</td><td>44.7</td><td>sah_Cyril</td><td>47.1</td><td>sba_Sinh</td><td>59.0</td><td>sba_Sinh</td><td>61.6</td><td>sba_Sinh</td><td>38.7</td><td>seh_Latn</td><td>41.0</td><td>seh_Latn</td><td>47.5</td><td>seh_Latn</td><td>49.9</td>
  </tr>
  <tr>
    <td>sin_Sinh</td><td>67.5</td><td>slk_Latn</td><td>66.6</td><td>slv_Latn</td><td>60.3</td><td>slv_Latn</td><td>60.6</td><td>slv_Latn</td><td>62.4</td><td>sme_Latn</td><td>62.6</td><td>sme_Latn</td><td>36.8</td><td>sme_Latn</td><td>48.8</td>
  </tr>
  <tr>
    <td>smo_Latn</td><td>56.2</td><td>sna_Latn</td><td>61.6</td><td>snd_Arab</td><td>40.9</td><td>snd_Arab</td><td>45.4</td><td>snd_Arab</td><td>67.7</td><td>som_Latn</td><td>68.7</td><td>som_Latn</td><td>33.6</td><td>som_Latn</td><td>35.8</td>
  </tr>
  <tr>
    <td>sop_Latn</td><td>34.1</td><td>sot_Latn</td><td>39.0</td><td>spa_Latn</td><td>45.2</td><td>spa_Latn</td><td>48.6</td><td>spa_Latn</td><td>64.2</td><td>sqi_Latn</td><td>67.5</td><td>sqi_Latn</td><td>70.8</td><td>sqi_Latn</td><td>70.3</td>
  </tr>
  <tr>
    <td>srn_Latn</td><td>48.7</td><td>srn_Latn</td><td>52.4</td><td>srp_Latn</td><td>64.0</td><td>srp_Latn</td><td>64.4</td><td>srp_Latn</td><td>64.9</td><td>ssw_Latn</td><td>69.0</td><td>ssw_Latn</td><td>38.0</td><td>ssw_Latn</td><td>46.9</td>
  </tr>
  <tr>
    <td>sun_Latn</td><td>54.0</td><td>swd_Deva</td><td>56.6</td><td>swe_Latn</td><td>58.2</td><td>swe_Latn</td><td>60.3</td><td>swe_Latn</td><td>67.8</td><td>swh_Latn</td><td>69.0</td><td>swh_Latn</td><td>62.1</td><td>swh_Latn</td><td>64.2</td>
  </tr>
  <tr>
    <td>ssn_Latn</td><td>49.0</td><td>tam_Taml</td><td>51.8</td><td>tat_Cyril</td><td>72.6</td><td>tat_Cyril</td><td>73.7</td><td>tat_Cyril</td><td>65.5</td><td>thb_Latn</td><td>68.0</td><td>thb_Latn</td><td>36.0</td><td>thb_Latn</td><td>42.7</td>
  </tr>
  <tr>
    <td>tca_Latn</td><td>42.2</td><td>tdt_Latn</td><td>48.3</td><td>tel_Telu</td><td>58.2</td><td>tel_Telu</td><td>66.0</td><td>tel_Telu</td><td>70.9</td><td>teo_Latn</td><td>71.4</td><td>teo_Latn</td><td>24.0</td><td>teo_Latn</td><td>26.6</td>
  </tr>
  <tr>
    <td>tgk_Cyril</td><td>64.0</td><td>tgl_Latn</td><td>65.7</td><td>tha_Thai</td><td>57.5</td><td>tha_Thai</td><td>58.8</td><td>tha_Thai</td><td>65.3</td><td>thi_Latn</td><td>65.5</td><td>thi_Latn</td><td>56.6</td><td>thi_Latn</td><td>60.3</td>
  </tr>
  <tr>
    <td>tir_Ethi</td><td>51.9</td><td>tlh_Latn</td><td>52.5</td><td>tob_Latn</td><td>64.0</td><td>tob_Latn</td><td>65.3</td><td>tob_Latn</td><td>44.3</td><td>tol_Latn</td><td>44.0</td><td>tol_Latn</td><td>38.1</td><td>tol_Latn</td><td>40.3</td>
  </tr>
  <tr>
    <td>toi_Latn</td><td>39.5</td><td>toi_Latn</td><td>49.4</td><td>ton_Latn</td><td>36.7</td><td>ton_Latn</td><td>39.2</td><td>ton_Latn</td><td>48.2</td><td>top_Latn</td><td>50.9</td><td>top_Latn</td><td>22.9</td><td>top_Latn</td><td>26.6</td>
  </tr>
  <tr>
    <td>tpl_Latn</td><td>68.4</td><td>tpm_Latn</td><td>69.7</td><td>tsn_Latn</td><td>45.8</td><td>tsn_Latn</td><td>51.4</td><td>tsn_Latn</td><td>45.6</td><td>tsz_Latn</td><td>46.5</td><td>tsz_Latn</td><td>37.3</td><td>tsz_Latn</td><td>42.8</td>
  </tr>
  <tr>
    <td>tuc_Latn</td><td>56.3</td><td>tu_Latn</td><td>62.2</td><td>tuk_Latn</td><td>45.6</td><td>tuk_Latn</td><td>48.1</td><td>tuk_Latn</td><td>56.9</td><td>tum_Latn</td><td>63.5</td><td>tum_Latn</td><td>48.1</td><td>tum_Latn</td><td>50.1</td>
  </tr>
  <tr>
    <td>tur_Latn</td><td>62.6</td><td>twi_Latn</td><td>65.8</td><td>tyv_Cyril</td><td>42.1</td><td>tyv_Cyril</td><td>47.5</td><td>tyv_Cyril</td><td>58.2</td><td>uzh_Cyril</td><td>63.0</td><td>uzh_Cyril</td><td>38.8</td><td>uzh_Cyril</td><td>44.2</td>
  </tr>
  <tr>
    <td>tzo_Latn</td><td>38.3</td><td>udm_Cyril</td><td>42.5</td><td>ukr_Cyril</td><td>53.8</td><td>ukr_Cyril</td><td>54.2</td><td>ukr_Cyril</td><td>65.3</td><td>urd_Arab</td><td>67.5</td><td>urd_Arab</td><td>61.3</td><td>urd_Arab</td><td>60.4</td>
  </tr>
  <tr>
    <td>uzb_Latn</td><td>59.7</td><td>uzn_Cyril</td><td>58.0</td><td>ven_Latn</td><td>65.3</td><td>ven_Latn</td><td>65.8</td><td>ven_Latn</td><td>43.4</td><td>vie_Latn</td><td>46.3</td><td>vie_Latn</td><td>70.0</td><td>vie_Latn</td><td>69.3</td>
  </tr>
  <tr>
    <td>wal_Latn</td><td>42.1</td><td>war_Latn</td><td>49.0</td><td>wbm_Latn</td><td>45.6</td><td>wbm_Latn</td><td>52.4</td><td>wbm_Latn</td><td>57.5</td><td>wol_Latn</td><td>56.1</td><td>wol_Latn</td><td>34.0</td><td>wol_Latn</td><td>40.4</td>
  </tr>
  <tr>
    <td>xav_Latn</td><td>29.9</td><td>kho_Latn</td><td>33.1</td><td>yam_Latn</td><td>45.3</td><td>yam_Latn</td><td>48.5</td><td>yam_Latn</td><td>51.2</td><td>yao_Latn</td><td>53.6</td><td>yao_Latn</td><td>40.0</td><td>yao_Latn</td><td>46.5</td>
  </tr>
  <tr>
    <td>yap_Latn</td><td>39.3</td><td>yom_Latn</td><td>42.0</td><td>yor_Latn</td><td>36.2</td><td>yor_Latn</td><td>37.8</td><td>yor_Latn</td><td>47.4</td><td>yua_Latn</td><td>46.9</td><td>yua_Latn</td><td>36.7</td><td>yua_Latn</td><td>39.8</td>
  </tr>
  <tr>
    <td>yue_Hani</td><td>58.4</td><td>zai_Latn</td><td>60.4</td><td>zho_Hani</td><td>39.3</td><td>zho_Hani</td><td>44.2</td><td>zho_Hani</td><td>64.6</td><td>zlm_Latn</td><td>67.0</td><td>zlm_Latn</td><td>70.3</td><td>zlm_Latn</td><td>69.7</td>
  </tr>
  <tr>
    <td>zom_Latn</td><td>47.2</td><td>zsm_Latn</td><td>49.9</td><td>zul_Latn</td><td>69.7</td><td>zul_Latn</td><td>68.0</td><td>zul_Latn</td><td>50.6</td><td>zul_Latn</td><td>53.4</td><td>zul_Latn</td><td>53.4</td><td>zul_Latn</td><td>53.4</td>
  </tr>
</table>

Table 11: F1 scores of models on Taxi1500 (Part II).
<table>
  <tr>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
  </tr>
  <tr><td>ace_Latin</td><td>71.5</td><td><b>73.6</b></td><td>acm_Arab</td><td>82.2</td><td><b>83.0</b></td><td>afr_Latn</td><td>82.3</td><td><b>82.7</b></td><td>jap_Arab</td><td><b>83.4</b></td><td>81.8</td></tr>
  <tr><td>aka_Latin</td><td>62.2</td><td><b>67.2</b></td><td>als_Latin</td><td>82.4</td><td><b>84.4</b></td><td>amh_Ethi</td><td>74.2</td><td>73.6</td><td>apc_Arab</td><td><b>83.9</b></td><td>82.9</td></tr>
  <tr><td>arb_Arab</td><td><b>83.8</b></td><td>82.9</td><td>ary_Arab</td><td><b>81.5</b></td><td>80.2</td><td>arz_Arab</td><td><b>84.5</b></td><td>84.1</td><td>asm_Beng</td><td>83.6</td><td><b>84.2</b></td></tr>
  <tr><td>ast_Latin</td><td><b>88.4</b></td><td>88.0</td><td>ayr_Latin</td><td>51.1</td><td><b>53.8</b></td><td>azb_Arab</td><td>71.5</td><td><b>74.7</b></td><td>azi_Latn</td><td>87.0</td><td><b>88.0</b></td></tr>
  <tr><td>bak_Cyril</td><td>84.6</td><td><b>86.6</b></td><td>bam_Latin</td><td><b>47.9</b></td><td>47.6</td><td>ban_Latin</td><td>80.3</td><td><b>83.0</b></td><td>bel_Cyril</td><td>83.7</td><td>83.4</td></tr>
  <tr><td>bem_Latin</td><td>63.0</td><td><b>63.9</b></td><td>ben_Beng</td><td>83.3</td><td><b>84.3</b></td><td>bjn_Latn</td><td>77.1</td><td><b>78.5</b></td><td>bod_Tibt</td><td><b>73.5</b></td><td>69.2</td></tr>
  <tr><td>bos_Latin</td><td>86.5</td><td><b>88.2</b></td><td>bul_Cyril</td><td>86.1</td><td><b>87.5</b></td><td>cat_Latin</td><td>84.8</td><td><b>86.4</b></td><td>ceb_Latn</td><td>81.8</td><td><b>84.6</b></td></tr>
  <tr><td>ces_Latin</td><td><b>89.1</b></td><td>86.9</td><td>cjk_Latin</td><td>46.6</td><td><b>48.1</b></td><td>ckb_Arab</td><td><b>83.9</b></td><td>80.2</td><td>chr_Latin</td><td>74.0</td><td><b>76.2</b></td></tr>
  <tr><td>cym_Latin</td><td><b>75.9</b></td><td>75.4</td><td>dan_Latin</td><td>86.8</td><td><b>87.4</b></td><td>deu_Latin</td><td>86.5</td><td><b>87.8</b></td><td>dyu_Latn</td><td>42.6</td><td><b>44.5</b></td></tr>
  <tr><td>dzo_Tibt</td><td>68.7</td><td><b>72.6</b></td><td>ell_Grek</td><td>79.5</td><td><b>80.0</b></td><td>eng_Latin</td><td><b>90.8</b></td><td>90.0</td><td>epo_Latin</td><td><b>83.8</b></td><td>82.2</td></tr>
  <tr><td>est_Latin</td><td>80.6</td><td><b>81.6</b></td><td>eus_Latin</td><td>82.1</td><td><b>82.2</b></td><td>ewe_Latin</td><td>49.3</td><td><b>51.5</b></td><td>fao_Latin</td><td>83.7</td><td><b>84.9</b></td></tr>
  <tr><td>fij_Latin</td><td>56.1</td><td><b>58.0</b></td><td>fin_Latin</td><td>82.1</td><td><b>82.9</b></td><td>fon_Latin</td><td>41.7</td><td><b>44.6</b></td><td>fra_Latin</td><td>87.9</td><td><b>89.6</b></td></tr>
  <tr><td>fur_Latin</td><td>77.6</td><td><b>80.2</b></td><td>gla_Latin</td><td><b>57.6</b></td><td>54.3</td><td>gle_Latin</td><td>62.2</td><td><b>64.1</b></td><td>glg_Latin</td><td>87.8</td><td><b>89.0</b></td></tr>
  <tr><td>grn_Latin</td><td><b>75.0</b></td><td>74.5</td><td>guj_Gujr</td><td>83.9</td><td><b>84.7</b></td><td>hat_Latin</td><td>77.4</td><td><b>79.1</b></td><td>hau_Latin</td><td><b>62.7</b></td><td>62.1</td></tr>
  <tr><td>heb_Hebr</td><td>77.9</td><td><b>79.2</b></td><td>hin_Deva</td><td>84.1</td><td><b>84.4</b></td><td>hne_Deva</td><td>77.9</td><td><b>80.1</b></td><td>hrv_Latin</td><td>87.3</td><td><b>89.0</b></td></tr>
  <tr><td>hun_Latin</td><td>86.8</td><td><b>87.6</b></td><td>hye_Armn</td><td><b>83.0</b></td><td>82.5</td><td>ibo_Latin</td><td>72.3</td><td><b>74.1</b></td><td>ilo_Latin</td><td>75.8</td><td><b>79.6</b></td></tr>
  <tr><td>ind_Latin</td><td>88.7</td><td><b>89.1</b></td><td>isl_Latin</td><td>78.5</td><td><b>79.1</b></td><td>ita_Latin</td><td>87.7</td><td><b>89.2</b></td><td>jav_Latin</td><td>80.2</td><td><b>80.3</b></td></tr>
  <tr><td>jpn_Ipan</td><td>87.1</td><td><b>87.9</b></td><td>kab_Latin</td><td>31.1</td><td><b>36.9</b></td><td>kac_Latin</td><td>49.3</td><td><b>52.3</b></td><td>kam_Latin</td><td>49.1</td><td><b>49.5</b></td></tr>
  <tr><td>kan_Knda</td><td><b>83.2</b></td><td>82.0</td><td>kat_Geor</td><td>81.8</td><td><b>83.7</b></td><td>kaz_Cyril</td><td>84.2</td><td><b>84.9</b></td><td>kbp_Latin</td><td><b>45.1</b></td><td>44.2</td></tr>
  <tr><td>kea_Latin</td><td>75.4</td><td><b>77.0</b></td><td>khm_Khmr</td><td>84.3</td><td><b>84.4</b></td><td>kik_Latin</td><td>57.1</td><td><b>59.9</b></td><td>kin_Latin</td><td>69.5</td><td><b>70.5</b></td></tr>
  <tr><td>kir_Cyril</td><td><b>80.7</b></td><td>80.3</td><td>kmb_Latin</td><td>48.2</td><td><b>49.5</b></td><td>kmr_Latin</td><td><b>70.7</b></td><td>70.0</td><td>kon_Latin</td><td>65.3</td><td><b>69.2</b></td></tr>
  <tr><td>kor_Hang</td><td><b>85.2</b></td><td>83.9</td><td>lao_Laoq</td><td><b>85.1</b></td><td>84.2</td><td>lij_Latin</td><td>77.7</td><td><b>79.6</b></td><td>lim_Latin</td><td>74.7</td><td><b>75.2</b></td></tr>
  <tr><td>lin_Latin</td><td>69.3</td><td><b>71.4</b></td><td>lit_Latin</td><td><b>86.5</b></td><td>84.7</td><td>lmo_Latin</td><td>77.7</td><td><b>79.1</b></td><td>ltz_Latin</td><td>76.6</td><td><b>79.1</b></td></tr>
  <tr><td>lua_Latin</td><td><b>59.1</b></td><td>56.4</td><td>lug_Latin</td><td>55.5</td><td><b>59.1</b></td><td>luo_Latin</td><td>52.6</td><td><b>53.0</b></td><td>lus_Latin</td><td>65.3</td><td><b>67.9</b></td></tr>
  <tr><td>lvx_Latin</td><td><b>84.4</b></td><td>83.6</td><td>mai_Deva</td><td>83.4</td><td><b>84.0</b></td><td>mal_Mlym</td><td><b>80.6</b></td><td>79.9</td><td>mar_Deva</td><td><b>84.1</b></td><td>82.5</td></tr>
  <tr><td>min_Latin</td><td>77.7</td><td><b>79.6</b></td><td>mkl_Cyril</td><td>83.3</td><td><b>84.6</b></td><td>mlt_Latin</td><td>82.9</td><td><b>83.0</b></td><td>mos_Latin</td><td>44.9</td><td><b>46.6</b></td></tr>
  <tr><td>mri_Latin</td><td>54.4</td><td><b>59.3</b></td><td>mya_Mymr</td><td>80.1</td><td><b>81.6</b></td><td>nld_Latin</td><td><b>86.5</b></td><td>85.8</td><td>ano_Latin</td><td><b>86.6</b></td><td>86.4</td></tr>
  <tr><td>nob_Latin</td><td>85.8</td><td><b>86.1</b></td><td>npi_Deva</td><td><b>86.8</b></td><td>86.0</td><td>nso_Latin</td><td>61.3</td><td><b>61.9</b></td><td>aya_Latin</td><td>71.1</td><td><b>72.7</b></td></tr>
  <tr><td>oci_Latin</td><td>83.1</td><td><b>84.9</b></td><td>ory_Orya</td><td>79.7</td><td><b>80.3</b></td><td>pag_Latin</td><td>78.7</td><td><b>79.7</b></td><td>pan_Guru</td><td>77.4</td><td><b>79.0</b></td></tr>
  <tr><td>pap_Latin</td><td>77.2</td><td><b>79.0</b></td><td>pes_Arab</td><td>87.6</td><td><b>89.2</b></td><td>plt_Latin</td><td>68.4</td><td><b>68.5</b></td><td>pol_Latin</td><td>86.4</td><td><b>86.7</b></td></tr>
  <tr><td>por_Latin</td><td>87.3</td><td><b>88.6</b></td><td>prs_Arab</td><td>85.8</td><td><b>88.4</b></td><td>quy_Latin</td><td>63.7</td><td><b>64.0</b></td><td>ron_Latin</td><td>86.4</td><td>84.5</td></tr>
  <tr><td>run_Latin</td><td><b>68.3</b></td><td>67.2</td><td>rus_Cyril</td><td>87.6</td><td><b>87.9</b></td><td>sag_Latin</td><td>52.4</td><td><b>55.1</b></td><td>san_Deva</td><td><b>77.9</b></td><td>77.8</td></tr>
  <tr><td>sat_Olck</td><td>53.0</td><td><b>57.4</b></td><td>scn_Latin</td><td>77.6</td><td><b>78.2</b></td><td>sin_Sinh</td><td><b>84.5</b></td><td>84.1</td><td>sik_Latin</td><td>86.1</td><td><b>87.0</b></td></tr>
  <tr><td>slv_Latin</td><td><b>86.4</b></td><td>85.5</td><td>smo_Latin</td><td>73.4</td><td><b>74.1</b></td><td>sna_Latin</td><td>59.3</td><td>58.0</td><td>snd_Arab</td><td>72.1</td><td><b>76.9</b></td></tr>
  <tr><td>som_Latin</td><td><b>61.8</b></td><td>59.8</td><td>sot_Latin</td><td>65.3</td><td><b>67.6</b></td><td>spa_Latin</td><td><b>86.4</b></td><td>86.2</td><td>srd_Latin</td><td>74.0</td><td><b>75.8</b></td></tr>
  <tr><td>srp_Cyril</td><td><b>85.8</b></td><td>85.2</td><td>ssw_Latin</td><td>67.5</td><td><b>68.1</b></td><td>sun_Latin</td><td>84.0</td><td><b>85.2</b></td><td>swe_Latin</td><td>86.6</td><td><b>87.3</b></td></tr>
  <tr><td>swh_Latin</td><td>76.0</td><td><b>78.6</b></td><td>szl_Latin</td><td>74.3</td><td><b>75.5</b></td><td>tam_Taml</td><td>80.6</td><td><b>84.3</b></td><td>tat_Cyril</td><td>84.0</td><td><b>85.2</b></td></tr>
  <tr><td>tel_Telu</td><td>85.3</td><td><b>85.7</b></td><td>tgk_Cyril</td><td><b>81.6</b></td><td>80.9</td><td>tgl_Latin</td><td>81.9</td><td><b>83.0</b></td><td>tha_Thai</td><td>87.4</td><td><b>88.9</b></td></tr>
  <tr><td>tir_Ethi</td><td>59.9</td><td><b>61.4</b></td><td>tpi_Latin</td><td>80.6</td><td><b>82.3</b></td><td>tsn_Latin</td><td><b>59.1</b></td><td>55.2</td><td>iso_Latin</td><td>59.3</td><td><b>61.2</b></td></tr>
  <tr><td>tuk_Latin</td><td><b>78.3</b></td><td>78.2</td><td>tum_Latin</td><td>70.3</td><td><b>70.8</b></td><td>tur_Latin</td><td>82.9</td><td><b>83.6</b></td><td>wi_Latin</td><td>61.4</td><td><b>68.0</b></td></tr>
  <tr><td>uig_Arab</td><td>77.7</td><td><b>80.0</b></td><td>ukr_Cyril</td><td><b>84.7</b></td><td>84.5</td><td>umb_Latin</td><td><b>45.9</b></td><td>45.8</td><td>urd_Arab</td><td>81.3</td><td><b>81.9</b></td></tr>
  <tr><td>vec_Latin</td><td><b>82.0</b></td><td>81.1</td><td>vie_Latin</td><td>84.9</td><td><b>85.8</b></td><td>war_Latin</td><td>81.7</td><td><b>83.4</b></td><td>wol_Latin</td><td>49.2</td><td><b>52.1</b></td></tr>
  <tr><td>xho_Latin</td><td>62.4</td><td><b>64.0</b></td><td>yor_Latin</td><td>46.6</td><td><b>51.8</b></td><td>zsm_Latin</td><td><b>87.2</b></td><td>86.6</td><td>zul_Latin</td><td><b>73.8</b></td><td>73.6</td></tr>
</table>

Table 12: F1 scores of models on SIB200.
<table>
  <tr>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
    <th>Language</th>
    <th>Baseline</th>
    <th>LANGSAMP</th>
  </tr>
  <tr>
    <td>ace_Latin</td><td>42.8</td><td>42.6</td><td>afr_Latin</td><td>76.4</td><td>76.8</td><td>als_Latin</td><td>81.7</td><td>82.2</td><td>amh_Ethi</td><td>40.8</td><td>39.0</td>
  </tr>
  <tr>
    <td>ara_Arab</td><td>55.9</td><td>56.5</td><td>arg_Latin</td><td>78.9</td><td>80.0</td><td>arz_Arab</td><td>56.8</td><td>58.4</td><td>asm_Beng</td><td>66.0</td><td>64.8</td>
  </tr>
  <tr>
    <td>ast_Latin</td><td>82.3</td><td>84.5</td><td>aym_Latin</td><td>45.9</td><td>43.8</td><td>aze_Latin</td><td>65.4</td><td>65.9</td><td>bak_Cyril</td><td>61.3</td><td>60.2</td>
  </tr>
  <tr>
    <td>bar_Latin</td><td>68.2</td><td>70.0</td><td>bel_Cyril</td><td>74.4</td><td>74.6</td><td>ben_Beng</td><td>71.1</td><td>70.5</td><td>bih_Deva</td><td>55.9</td><td>55.2</td>
  </tr>
  <tr>
    <td>bod_Tibt</td><td>27.1</td><td>35.0</td><td>bos_Latin</td><td>73.0</td><td>72.6</td><td>bre_Latin</td><td>64.6</td><td>63.9</td><td>bul_Cyril</td><td>75.6</td><td>75.2</td>
  </tr>
  <tr>
    <td>cat_Latin</td><td>83.8</td><td>83.8</td><td>chk_Latin</td><td>53.9</td><td>51.6</td><td>ceb_Latin</td><td>54.0</td><td>57.5</td><td>ces_Latin</td><td>78.6</td><td>78.7</td>
  </tr>
  <tr>
    <td>che_Cyril</td><td>45.6</td><td>55.7</td><td>chv_Cyril</td><td>77.4</td><td>75.2</td><td>ckb_Arab</td><td>73.2</td><td>73.3</td><td>cos_Latin</td><td>59.1</td><td>59.7</td>
  </tr>
  <tr>
    <td>chh_Latin</td><td>51.9</td><td>50.2</td><td>csh_Latin</td><td>61.9</td><td>62.0</td><td>cym_Latin</td><td>62.2</td><td>60.2</td><td>dan_Latin</td><td>81.7</td><td>81.7</td>
  </tr>
  <tr>
    <td>deu_Latin</td><td>76.0</td><td>76.7</td><td>dig_Latin</td><td>51.7</td><td>53.4</td><td>div_Thaa</td><td>48.2</td><td>55.2</td><td>ell_Greek</td><td>73.0</td><td>72.9</td>
  </tr>
  <tr>
    <td>eml_Latin</td><td>42.0</td><td>42.1</td><td>eng_Latin</td><td>83.5</td><td>83.4</td><td>epo_Latin</td><td>68.1</td><td>69.0</td><td>est_Latin</td><td>72.9</td><td>73.6</td>
  </tr>
  <tr>
    <td>eus_Latin</td><td>57.5</td><td>59.1</td><td>ext_Latin</td><td>45.0</td><td>46.2</td><td>fao_Latin</td><td>70.9</td><td>69.6</td><td>fas_Arab</td><td>52.7</td><td>51.5</td>
  </tr>
  <tr>
    <td>fin_Latin</td><td>75.0</td><td>75.4</td><td>fra_Latin</td><td>76.4</td><td>76.8</td><td>frr_Latin</td><td>55.3</td><td>54.5</td><td>fry_Latin</td><td>77.2</td><td>76.3</td>
  </tr>
  <tr>
    <td>fur_Latin</td><td>57.5</td><td>57.1</td><td>gla_Latin</td><td>59.5</td><td>65.4</td><td>gle_Latin</td><td>72.7</td><td>72.8</td><td>glg_Latin</td><td>79.9</td><td>80.4</td>
  </tr>
  <tr>
    <td>grn_Latin</td><td>54.0</td><td>54.1</td><td>guj_Gujr</td><td>58.8</td><td>59.2</td><td>hbs_Latin</td><td>62.7</td><td>65.5</td><td>heb_Hebr</td><td>50.4</td><td>51.7</td>
  </tr>
  <tr>
    <td>hin_Deva</td><td>68.4</td><td>68.9</td><td>hrv_Latin</td><td>77.1</td><td>77.8</td><td>hsb_Latin</td><td>74.2</td><td>75.9</td><td>hun_Latin</td><td>76.2</td><td>76.7</td>
  </tr>
  <tr>
    <td>hye_Armn</td><td>54.1</td><td>57.2</td><td>ibo_Latin</td><td>57.3</td><td>57.3</td><td>ido_Latin</td><td>79.0</td><td>79.9</td><td>ilo_Latin</td><td>73.6</td><td>76.8</td>
  </tr>
  <tr>
    <td>ina_Latin</td><td>57.8</td><td>58.5</td><td>ind_Latin</td><td>62.2</td><td>64.0</td><td>isl_Latin</td><td>72.4</td><td>72.1</td><td>ita_Latin</td><td>78.4</td><td>78.6</td>
  </tr>
  <tr>
    <td>jav_Latin</td><td>58.9</td><td>57.8</td><td>jbo_Latin</td><td>25.0</td><td>25.1</td><td>jpn_Jpan</td><td>21.0</td><td>17.5</td><td>kan_Knda</td><td>58.5</td><td>58.7</td>
  </tr>
  <tr>
    <td>kat_Geor</td><td>67.5</td><td>68.2</td><td>kaz_Cyril</td><td>50.0</td><td>51.1</td><td>khm_Khmr</td><td>42.4</td><td>43.1</td><td>kin_Latin</td><td>63.5</td><td>69.0</td>
  </tr>
  <tr>
    <td>kir_Cyril</td><td>45.4</td><td>44.2</td><td>kor_Hang</td><td>52.7</td><td>51.2</td><td>ksh_Latin</td><td>58.0</td><td>59.2</td><td>kur_Latin</td><td>63.1</td><td>64.8</td>
  </tr>
  <tr>
    <td>lat_Latin</td><td>75.7</td><td>74.7</td><td>lav_Latin</td><td>72.4</td><td>74.9</td><td>lij_Latin</td><td>42.7</td><td>46.3</td><td>lim_Latin</td><td>70.3</td><td>71.0</td>
  </tr>
  <tr>
    <td>lin_Latin</td><td>51.5</td><td>49.2</td><td>lit_Latin</td><td>75.0</td><td>74.6</td><td>lmo_Latin</td><td>75.3</td><td>74.1</td><td>ltz_Latin</td><td>68.6</td><td>69.0</td>
  </tr>
  <tr>
    <td>lzh_Hani</td><td>14.6</td><td>11.5</td><td>mal_Mlym</td><td>62.8</td><td>61.7</td><td>mar_Deva</td><td>63.0</td><td>62.0</td><td>mht_Cyril</td><td>60.6</td><td>61.9</td>
  </tr>
  <tr>
    <td>min_Latin</td><td>41.5</td><td>42.9</td><td>mkd_Cyril</td><td>75.7</td><td>76.3</td><td>mlg_Latin</td><td>57.8</td><td>58.7</td><td>mlt_Latin</td><td>66.9</td><td>71.3</td>
  </tr>
  <tr>
    <td>mon_Cyril</td><td>66.0</td><td>68.9</td><td>mri_Latin</td><td>49.5</td><td>48.0</td><td>msa_Latin</td><td>68.7</td><td>69.0</td><td>mwL_Latin</td><td>48.9</td><td>51.1</td>
  </tr>
  <tr>
    <td>mya_Mymr</td><td>53.8</td><td>53.6</td><td>mzn_Arab</td><td>44.9</td><td>47.4</td><td>nan_Latin</td><td>83.6</td><td>85.0</td><td>nap_Latin</td><td>59.0</td><td>59.3</td>
  </tr>
  <tr>
    <td>nds_Latin</td><td>77.8</td><td>77.0</td><td>nep_Deva</td><td>63.1</td><td>61.7</td><td>nld_Latin</td><td>81.4</td><td>81.1</td><td>nno_Latin</td><td>76.5</td><td>77.3</td>
  </tr>
  <tr>
    <td>nor_Latin</td><td>76.6</td><td>77.7</td><td>oci_Latin</td><td>68.6</td><td>70.7</td><td>ori_Orya</td><td>31.7</td><td>31.9</td><td>oss_Cyril</td><td>53.5</td><td>51.3</td>
  </tr>
  <tr>
    <td>pan_Guru</td><td>49.7</td><td>48.1</td><td>pms_Latin</td><td>78.9</td><td>80.3</td><td>pnb_Arab</td><td>65.1</td><td>64.7</td><td>pol_Latin</td><td>78.0</td><td>78.1</td>
  </tr>
  <tr>
    <td>por_Latin</td><td>77.6</td><td>78.7</td><td>pus_Arab</td><td>40.5</td><td>42.0</td><td>que_Latin</td><td>66.1</td><td>67.4</td><td>roh_Latin</td><td>62.0</td><td>58.8</td>
  </tr>
  <tr>
    <td>ron_Latin</td><td>76.0</td><td>75.6</td><td>rus_Cyril</td><td>69.0</td><td>69.1</td><td>sah_Cyril</td><td>75.3</td><td>69.3</td><td>san_Deva</td><td>35.5</td><td>57.1</td>
  </tr>
  <tr>
    <td>scr_Latin</td><td>65.4</td><td>66.0</td><td>sco_Latin</td><td>84.5</td><td>87.3</td><td>sgs_Latin</td><td>60.7</td><td>64.0</td><td>sin_Sinh</td><td>53.0</td><td>49.4</td>
  </tr>
  <tr>
    <td>slk_Latin</td><td>77.9</td><td>77.3</td><td>slv_Latin</td><td>80.2</td><td>80.1</td><td>snd_Arab</td><td>44.3</td><td>41.9</td><td>som_Latin</td><td>52.2</td><td>55.8</td>
  </tr>
  <tr>
    <td>spa_Latin</td><td>74.0</td><td>76.5</td><td>sqi_Latin</td><td>77.7</td><td>77.6</td><td>srp_Cyril</td><td>63.5</td><td>64.3</td><td>sun_Latin</td><td>55.4</td><td>53.8</td>
  </tr>
  <tr>
    <td>swa_Latin</td><td>69.1</td><td>69.0</td><td>swe_Latin</td><td>70.3</td><td>73.0</td><td>szl_Latin</td><td>68.1</td><td>70.8</td><td>tam_Taml</td><td>54.9</td><td>55.3</td>
  </tr>
  <tr>
    <td>tat_Cyril</td><td>65.9</td><td>63.0</td><td>tel_Telu</td><td>50.2</td><td>49.1</td><td>tgk_Cyril</td><td>63.0</td><td>66.0</td><td>tgl_Latin</td><td>75.5</td><td>76.8</td>
  </tr>
  <tr>
    <td>tha_Thai</td><td>4.6</td><td>2.2</td><td>tuk_Latin</td><td>56.3</td><td>56.5</td><td>tur_Latin</td><td>76.1</td><td>76.6</td><td>uig_Arab</td><td>47.8</td><td>48.6</td>
  </tr>
  <tr>
    <td>ukr_Cyril</td><td>76.6</td><td>76.6</td><td>urd_Arab</td><td>66.2</td><td>65.1</td><td>uzb_Latin</td><td>73.1</td><td>74.5</td><td>vec_Latin</td><td>68.6</td><td>67.7</td>
  </tr>
  <tr>
    <td>vep_Latin</td><td>71.1</td><td>71.2</td><td>vie_Latin</td><td>73.0</td><td>73.1</td><td>vls_Latin</td><td>75.5</td><td>76.5</td><td>vol_Latin</td><td>59.6</td><td>59.7</td>
  </tr>
  <tr>
    <td>war_Latin</td><td>66.4</td><td>66.6</td><td>wuu_Hani</td><td>32.3</td><td>28.5</td><td>xmf_Geor</td><td>63.6</td><td>65.7</td><td>vid_Hebr</td><td>50.4</td><td>55.7</td>
  </tr>
  <tr>
    <td>yor_Latin</td><td>60.7</td><td>61.6</td><td>yue_Hani</td><td>23.7</td><td>21.6</td><td>zea_Latin</td><td>66.8</td><td>63.7</td><td>zho_Hani</td><td>24.7</td><td>20.9</td>
  </tr>
</table>

Table 13: F1 scores of models on NER.
<table>
  <tr>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
    <th>Language</th>
    <th>Baseline LANGSAMP</th>
  </tr>
  <tr>
    <td>afr_Latin</td><td>87.8</td><td>88.2</td><td>ajp_Arab</td><td>70.4</td><td>69.2</td><td>aln_Latin</td><td>52.9</td><td>50.7</td><td>lamh_Ethi</td><td>65.9</td><td>67.0</td>
  </tr>
  <tr>
    <td>ara_Arab</td><td>66.5</td><td>66.5</td><td>bam_Latin</td><td>42.2</td><td>41.5</td><td>bel_Cyril</td><td>86.0</td><td>85.5</td><td>ben_Beng</td><td>83.2</td><td>83.0</td>
  </tr>
  <tr>
    <td>bre_Latin</td><td>61.0</td><td>60.0</td><td>bul_Cyril</td><td>88.1</td><td>88.2</td><td>cat_Latin</td><td>86.5</td><td>86.7</td><td>ceb_Latin</td><td>65.2</td><td>65.2</td>
  </tr>
  <tr>
    <td>ces_Latin</td><td>84.7</td><td>84.7</td><td>cym_Latin</td><td>65.7</td><td>64.6</td><td>dan_Latin</td><td>90.4</td><td>90.4</td><td>deu_Latin</td><td>87.9</td><td>87.6</td>
  </tr>
  <tr>
    <td>ell_Grek</td><td>86.1</td><td>84.9</td><td>eng_Latin</td><td>96.0</td><td>96.0</td><td>est_Latin</td><td>83.7</td><td>83.7</td><td>eus_Latin</td><td>65.2</td><td>64.2</td>
  </tr>
  <tr>
    <td>fao_Latin</td><td>88.8</td><td>88.3</td><td>fas_Arab</td><td>71.3</td><td>71.9</td><td>fin_Latin</td><td>82.1</td><td>81.7</td><td>fra_Latin</td><td>86.2</td><td>85.8</td>
  </tr>
  <tr>
    <td>gla_Latin</td><td>57.9</td><td>57.5</td><td>gle_Latin</td><td>64.0</td><td>64.0</td><td>glg_Latin</td><td>83.1</td><td>83.5</td><td>glv_Latin</td><td>51.7</td><td>51.0</td>
  </tr>
  <tr>
    <td>gre_Grek</td><td>71.7</td><td>71.0</td><td>grn_Latin</td><td>19.7</td><td>21.3</td><td>gsw_Latin</td><td>79.7</td><td>80.0</td><td>hbo_Hebr</td><td>34.0</td><td>38.6</td>
  </tr>
  <tr>
    <td>heb_Hebr</td><td>68.5</td><td>68.0</td><td>hin_Deva</td><td>70.4</td><td>71.8</td><td>hrv_Latin</td><td>85.6</td><td>85.5</td><td>hsb_Latin</td><td>83.2</td><td>82.9</td>
  </tr>
  <tr>
    <td>hun_Latin</td><td>81.5</td><td>82.3</td><td>hye_Armn</td><td>84.0</td><td>84.4</td><td>hyw_Armn</td><td>81.2</td><td>81.4</td><td>ind_Latin</td><td>83.6</td><td>83.3</td>
  </tr>
  <tr>
    <td>isl_Latin</td><td>82.9</td><td>82.7</td><td>ita_Latin</td><td>87.8</td><td>88.7</td><td>jav_Latin</td><td>73.4</td><td>74.0</td><td>jpn_Jpan</td><td>22.6</td><td>28.9</td>
  </tr>
  <tr>
    <td>kaz_Cyril</td><td>76.5</td><td>76.0</td><td>kmr_Latin</td><td>74.1</td><td>74.2</td><td>kor_Hang</td><td>52.8</td><td>52.2</td><td>lat_Latin</td><td>72.8</td><td>71.7</td>
  </tr>
  <tr>
    <td>lav_Latin</td><td>83.6</td><td>83.6</td><td>lij_Latin</td><td>76.4</td><td>75.6</td><td>lit_Latin</td><td>81.5</td><td>81.3</td><td>lzh_Hani</td><td>23.7</td><td>21.3</td>
  </tr>
  <tr>
    <td>mal_Mlym</td><td>86.4</td><td>86.1</td><td>mar_Deva</td><td>81.4</td><td>80.7</td><td>mlt_Latin</td><td>79.5</td><td>80.0</td><td>myv_Cyril</td><td>64.4</td><td>64.6</td>
  </tr>
  <tr>
    <td>nap_Latin</td><td>70.6</td><td>75.9</td><td>nld_Latin</td><td>77.0</td><td>77.8</td><td>nld_Latin</td><td>88.2</td><td>88.3</td><td>nor_Latin</td><td>88.2</td><td>88.2</td>
  </tr>
  <tr>
    <td>pcm_Latin</td><td>57.3</td><td>57.3</td><td>pol_Latin</td><td>83.5</td><td>83.2</td><td>por_Latin</td><td>88.0</td><td>88.1</td><td>que_Latin</td><td>61.1</td><td>58.8</td>
  </tr>
  <tr>
    <td>ron_Latin</td><td>81.4</td><td>81.7</td><td>rus_Cyril</td><td>88.9</td><td>88.5</td><td>sah_Cyril</td><td>74.0</td><td>73.0</td><td>san_Deva</td><td>25.7</td><td>24.2</td>
  </tr>
  <tr>
    <td>sin_Sinh</td><td>56.0</td><td>56.5</td><td>sik_Latin</td><td>84.3</td><td>84.9</td><td>slv_Latin</td><td>77.2</td><td>77.0</td><td>sme_Latin</td><td>73.0</td><td>73.1</td>
  </tr>
  <tr>
    <td>spa_Latin</td><td>87.6</td><td>87.8</td><td>sqi_Latin</td><td>76.3</td><td>76.9</td><td>srp_Latin</td><td>85.3</td><td>85.3</td><td>swe_Latin</td><td>92.5</td><td>92.6</td>
  </tr>
  <tr>
    <td>tam_Tamil</td><td>73.6</td><td>73.7</td><td>tat_Cyril</td><td>70.2</td><td>71.1</td><td>tel_Telu</td><td>81.8</td><td>81.7</td><td>tgl_Latin</td><td>75.5</td><td>75.0</td>
  </tr>
  <tr>
    <td>tha_Thai</td><td>55.9</td><td>56.7</td><td>tur_Latin</td><td>71.3</td><td>71.1</td><td>uig_Arab</td><td>68.2</td><td>68.2</td><td>ukr_Cyril</td><td>85.0</td><td>85.2</td>
  </tr>
  <tr>
    <td>urd_Arab</td><td>62.0</td><td>64.6</td><td>vie_Latin</td><td>68.3</td><td>67.4</td><td>wol_Latin</td><td>61.6</td><td>59.5</td><td>xav_Latin</td><td>11.8</td><td>10.5</td>
  </tr>
  <tr>
    <td>yor_Latin</td><td>62.3</td><td>62.6</td><td>yue_Hani</td><td>37.4</td><td>41.3</td><td>zho_Hani</td><td>38.7</td><td>44.8</td><td></td><td></td><td></td>
  </tr>
</table>

Table 14: F1 scores of models on POS.
<table>
  <tr>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
  </tr>
  <tr><td>ace_Latn</td><td><b>63.3</b></td><td>60.1</td><td>ach_Latn</td><td>35.6</td><td><b>48.1</b></td><td>acr_Latn</td><td>48.8</td><td>46.7</td><td>afr_Latn</td><td><b>58.6</b></td><td>58.5</td></tr>
  <tr><td>ahk_Latn</td><td>5.4</td><td><b>8.3</b></td><td>aka_Latn</td><td><b>44.9</b></td><td>41.2</td><td>aln_Latn</td><td><b>56.2</b></td><td>54.7</td><td>als_Latn</td><td><b>57.1</b></td><td>57.1</td></tr>
  <tr><td>alz_Latn</td><td>34.1</td><td><b>43.0</b></td><td>aoj_Latn</td><td>40.9</td><td><b>46.2</b></td><td>arb_Arab</td><td><b>55.4</b></td><td>55.4</td><td>arn_Latn</td><td>43.1</td><td><b>44.4</b></td></tr>
  <tr><td>arz_Arab</td><td>33.7</td><td><b>40.3</b></td><td>ams_Beng</td><td>53.4</td><td><b>61.5</b></td><td>ayr_Latn</td><td>52.7</td><td><b>62.2</b></td><td>azb_Arab</td><td><b>61.0</b></td><td>61.0</td></tr>
  <tr><td>bak_Cyril</td><td>54.7</td><td><b>59.7</b></td><td>ham_Latn</td><td>48.9</td><td><b>55.6</b></td><td>ban_Latn</td><td><b>43.0</b></td><td>42.5</td><td>bar_Latn</td><td><b>47.8</b></td><td>43.3</td></tr>
  <tr><td>bei_Latn</td><td>34.6</td><td><b>37.1</b></td><td>bel_Latn</td><td>54.2</td><td><b>60.5</b></td><td>bel_Cyril</td><td>59.1</td><td><b>61.5</b></td><td>bem_Latn</td><td>44.2</td><td><b>49.4</b></td></tr>
  <tr><td>bhw_Latn</td><td><b>50.2</b></td><td>46.9</td><td>him_Latn</td><td>47.3</td><td><b>55.1</b></td><td>bis_Latn</td><td><b>68.4</b></td><td>68.1</td><td>hqc_Latn</td><td>33.2</td><td><b>41.6</b></td></tr>
  <tr><td>bit_Latn</td><td><b>56.7</b></td><td>53.8</td><td>bul_Cyril</td><td>62.5</td><td><b>62.6</b></td><td>bum_Latn</td><td>39.6</td><td><b>42.2</b></td><td>bzj_Latn</td><td><b>65.7</b></td><td>60.3</td></tr>
  <tr><td>cac_Latn</td><td>43.8</td><td><b>46.0</b></td><td>cak_Latn</td><td>51.0</td><td><b>57.9</b></td><td>caq_Latn</td><td>42.7</td><td><b>51.0</b></td><td>cat_Latn</td><td>61.2</td><td><b>62.3</b></td></tr>
  <tr><td>cee_Latn</td><td><b>43.8</b></td><td>38.0</td><td>ceb_Latn</td><td><b>49.8</b></td><td>49.1</td><td>ces_Latn</td><td>63.3</td><td><b>63.7</b></td><td>cfn_Latn</td><td><b>58.3</b></td><td>57.1</td></tr>
  <tr><td>chk_Latn</td><td><b>42.8</b></td><td>38.9</td><td>chv_Cyril</td><td>60.3</td><td><b>64.3</b></td><td>ckb_Arab</td><td>58.3</td><td><b>67.0</b></td><td>cmn_Hani</td><td>60.8</td><td><b>73.0</b></td></tr>
  <tr><td>chr_Cyril</td><td>61.4</td><td><b>67.7</b></td><td>crs_Latn</td><td>62.3</td><td><b>63.5</b></td><td>cys_Latn</td><td><b>58.3</b></td><td>56.7</td><td>ctd_Latn</td><td><b>56.6</b></td><td>55.8</td></tr>
  <tr><td>cuk_Latn</td><td>39.1</td><td><b>40.8</b></td><td>cym_Latn</td><td><b>51.9</b></td><td>46.0</td><td>dan_Latn</td><td><b>58.1</b></td><td>54.0</td><td>deu_Latn</td><td><b>51.5</b></td><td>51.5</td></tr>
  <tr><td>dln_Latn</td><td><b>54.4</b></td><td>54.4</td><td>dtp_Latn</td><td>51.5</td><td><b>51.6</b></td><td>dyu_Latn</td><td><b>55.6</b></td><td>48.2</td><td>dzo_Tibt</td><td>50.6</td><td><b>58.1</b></td></tr>
  <tr><td>ell_Grek</td><td><b>56.9</b></td><td>53.9</td><td>eng_Latn</td><td><b>78.0</b></td><td>78.0</td><td>enn_Latn</td><td><b>70.8</b></td><td>67.0</td><td>epo_Latn</td><td><b>58.3</b></td><td>58.3</td></tr>
  <tr><td>eus_Latn</td><td><b>25.2</b></td><td>21.4</td><td>ewe_Latn</td><td>46.4</td><td><b>52.1</b></td><td>fao_Latn</td><td>56.5</td><td><b>64.8</b></td><td>fas_Arab</td><td>69.6</td><td><b>70.2</b></td></tr>
  <tr><td>fil_Latn</td><td>56.7</td><td><b>58.7</b></td><td>fin_Latn</td><td><b>56.4</b></td><td>55.7</td><td>fon_Latn</td><td><b>36.8</b></td><td>35.4</td><td>fra_Latn</td><td><b>66.8</b></td><td>66.8</td></tr>
  <tr><td>gaa_Latn</td><td>36.9</td><td><b>47.7</b></td><td>gil_Latn</td><td>40.4</td><td><b>47.2</b></td><td>giz_Latn</td><td>48.4</td><td><b>48.5</b></td><td>gkn_Latn</td><td><b>40.0</b></td><td>34.1</td></tr>
  <tr><td>gla_Latn</td><td><b>45.6</b></td><td>45.6</td><td>gle_Latn</td><td>41.8</td><td><b>45.1</b></td><td>glv_Latn</td><td>37.3</td><td><b>48.7</b></td><td>gom_Latn</td><td>34.8</td><td><b>41.6</b></td></tr>
  <tr><td>guc_Latn</td><td><b>39.6</b></td><td>37.6</td><td>gug_Latn</td><td>39.0</td><td><b>46.0</b></td><td>guj_Gujr</td><td>67.1</td><td><b>70.4</b></td><td>gur_Latn</td><td>37.0</td><td><b>44.2</b></td></tr>
  <tr><td>gya_Latn</td><td>39.6</td><td><b>41.8</b></td><td>gym_Latn</td><td>45.4</td><td><b>52.9</b></td><td>hat_Latn</td><td><b>63.0</b></td><td>60.0</td><td>hau_Latn</td><td>54.0</td><td><b>59.6</b></td></tr>
  <tr><td>heb_Hebr</td><td>16.7</td><td>15.2</td><td>hif_Latn</td><td>42.4</td><td><b>53.6</b></td><td>hil_Latn</td><td>63.7</td><td>61.6</td><td>hin_Deva</td><td><b>64.8</b></td><td>64.8</td></tr>
  <tr><td>hne_Deva</td><td>64.1</td><td><b>67.5</b></td><td>hnj_Latn</td><td>61.5</td><td><b>63.2</b></td><td>hra_Latn</td><td>48.2</td><td><b>53.1</b></td><td>hrv_Latn</td><td><b>62.7</b></td><td>60.7</td></tr>
  <tr><td>hun_Latn</td><td>65.2</td><td><b>65.9</b></td><td>hus_Latn</td><td>37.6</td><td><b>40.7</b></td><td>hye_Armn</td><td>67.2</td><td><b>69.3</b></td><td>iba_Latn</td><td>57.9</td><td><b>59.2</b></td></tr>
  <tr><td>ifa_Latn</td><td>49.7</td><td><b>51.5</b></td><td>ibf_Latn</td><td><b>48.3</b></td><td>48.1</td><td>ikk_Latn</td><td>46.6</td><td><b>52.5</b></td><td>ilo_Latn</td><td><b>58.8</b></td><td>55.7</td></tr>
  <tr><td>isl_Latn</td><td>53.5</td><td><b>61.2</b></td><td>ita_Latn</td><td>62.8</td><td><b>67.1</b></td><td>iun_Latn</td><td>51.4</td><td><b>58.0</b></td><td>ixl_Latn</td><td>36.6</td><td><b>38.2</b></td></tr>
  <tr><td>jam_Latn</td><td><b>66.1</b></td><td>61.0</td><td>jav_Latn</td><td>43.9</td><td><b>47.6</b></td><td>jpn_Jpan</td><td><b>58.6</b></td><td>58.6</td><td>kaa_Latn</td><td>57.7</td><td><b>62.6</b></td></tr>
  <tr><td>kac_Latn</td><td>44.5</td><td><b>47.3</b></td><td>kal_Kaln</td><td>31.5</td><td><b>34.5</b></td><td>kan_Knda</td><td>60.6</td><td><b>67.5</b></td><td>kat_Geor</td><td>55.2</td><td>62.2</td></tr>
  <tr><td>kbp_Latn</td><td>34.9</td><td><b>39.5</b></td><td>kek_Latn</td><td><b>41.5</b></td><td>40.3</td><td>khm_Khmr</td><td><b>64.7</b></td><td>64.7</td><td>kia_Latn</td><td>48.0</td><td><b>51.7</b></td></tr>
  <tr><td>kin_Latn</td><td>47.2</td><td><b>52.5</b></td><td>kir_Cyril</td><td>61.1</td><td><b>64.7</b></td><td>kjb_Latn</td><td>44.7</td><td><b>48.1</b></td><td>kjh_Cyril</td><td><b>52.3</b></td><td>51.1</td></tr>
  <tr><td>kmr_Cyril</td><td>45.5</td><td><b>53.1</b></td><td>kiv_Latn</td><td><b>42.6</b></td><td>40.5</td><td>kor_Hang</td><td>69.8</td><td><b>71.3</b></td><td>kpg_Latn</td><td>64.1</td><td>57.4</td></tr>
  <tr><td>kri_Latn</td><td><b>63.2</b></td><td>56.0</td><td>ksd_Latn</td><td>54.2</td><td><b>54.4</b></td><td>kss_Latn</td><td>16.2</td><td><b>21.6</b></td><td>ksw_Mymr</td><td>50.4</td><td>50.3</td></tr>
  <tr><td>lam_Latn</td><td>34.7</td><td><b>35.6</b></td><td>lao_Laoo</td><td>69.1</td><td><b>72.7</b></td><td>lat_Latn</td><td>57.2</td><td><b>62.9</b></td><td>lav_Latn</td><td><b>60.4</b></td><td>57.7</td></tr>
  <tr><td>leh_Latn</td><td><b>43.5</b></td><td>37.2</td><td>luh_Latn</td><td>22.3</td><td><b>29.0</b></td><td>lin_Latn</td><td>47.1</td><td><b>54.7</b></td><td>lit_Latn</td><td>58.3</td><td><b>59.7</b></td></tr>
  <tr><td>ltz_Latn</td><td><b>48.2</b></td><td>48.2</td><td>lug_Latn</td><td><b>46.1</b></td><td>39.0</td><td>luo_Latn</td><td>40.6</td><td><b>41.2</b></td><td>lus_Latn</td><td><b>51.6</b></td><td>51.6</td></tr>
  <tr><td>mad_Latn</td><td>55.3</td><td><b>63.0</b></td><td>mah_Latn</td><td><b>41.6</b></td><td>38.3</td><td>mai_Deva</td><td><b>62.7</b></td><td>60.5</td><td>mam_Latn</td><td><b>33.9</b></td><td>33.2</td></tr>
  <tr><td>mau_Latn</td><td>5.5</td><td><b>8.4</b></td><td>mbb_Latn</td><td>52.6</td><td><b>53.1</b></td><td>mck_Latn</td><td><b>41.9</b></td><td>41.2</td><td>mcn_Latn</td><td>37.7</td><td><b>39.3</b></td></tr>
  <tr><td>mdy_Ethi</td><td>51.6</td><td><b>57.6</b></td><td>meu_Latn</td><td>54.9</td><td><b>55.8</b></td><td>mfe_Latn</td><td>66.0</td><td><b>66.2</b></td><td>mgh_Latn</td><td>30.3</td><td><b>33.1</b></td></tr>
  <tr><td>mhr_Cyril</td><td>36.0</td><td><b>38.5</b></td><td>min_Latn</td><td><b>49.9</b></td><td>40.7</td><td>miq_Latn</td><td><b>52.2</b></td><td>52.2</td><td>mkd_Cyril</td><td><b>71.2</b></td><td>70.3</td></tr>
  <tr><td>mlt_Latn</td><td><b>50.7</b></td><td>50.7</td><td>mos_Latn</td><td>40.3</td><td><b>41.2</b></td><td>mps_Latn</td><td><b>57.1</b></td><td>53.1</td><td>mri_Latn</td><td>50.9</td><td><b>52.6</b></td></tr>
</table>

Table 15: F1 scores of LANGSAMP on Taxi1500 using English and the closest donor language as source (Part I).
<table>
  <tr>
    <th>Language</th>
    <th>English</th>
    <th>Closest donor</th>
    <th>Language</th>
    <th>English</th>
    <th>Closest donor</th>
    <th>Language</th>
    <th>English</th>
    <th>Closest donor</th>
    <th>Language</th>
    <th>English</th>
    <th>Closest donor</th>
  </tr>
  <tr>
    <td>msa_Latn</td>
    <td>41.7</td>
    <td><b>42.0</b></td>
    <td>mwm_Latn</td>
    <td>55.1</td>
    <td>55.0</td>
    <td>mxv_Latn</td>
    <td>29.6</td>
    <td>27.4</td>
    <td>mya_Mymr</td>
    <td>54.4</td>
    <td>53.4</td>
  </tr>
  <tr>
    <td>mzh_Latn</td>
    <td>39.7</td>
    <td><b>45.1</b></td>
    <td>nan_Latn</td>
    <td>31.5</td>
    <td><b>31.8</b></td>
    <td>nag_Latn</td>
    <td>41.7</td>
    <td><b>43.7</b></td>
    <td>nav_Latn</td>
    <td>21.1</td>
    <td>29.5</td>
  </tr>
  <tr>
    <td>nch_Latn</td>
    <td><b>44.0</b></td>
    <td>36.6</td>
    <td>ncj_Latn</td>
    <td>38.6</td>
    <td><b>39.1</b></td>
    <td>ndc_Latn</td>
    <td>34.7</td>
    <td><b>36.6</b></td>
    <td>nde_Latn</td>
    <td>45.7</td>
    <td><b>49.8</b></td>
  </tr>
  <tr>
    <td>nds_Latn</td>
    <td><b>49.6</b></td>
    <td>44.0</td>
    <td>nep_Deva</td>
    <td>68.0</td>
    <td><b>72.1</b></td>
    <td>ngu_Latn</td>
    <td>43.4</td>
    <td><b>48.2</b></td>
    <td>nld_Latn</td>
    <td><b>61.1</b></td>
    <td>53.7</td>
  </tr>
  <tr>
    <td>nbb_Latn</td>
    <td>40.7</td>
    <td><b>46.1</b></td>
    <td>nno_Latn</td>
    <td><b>63.1</b></td>
    <td>63.1</td>
    <td>nob_Latn</td>
    <td>57.2</td>
    <td><b>58.2</b></td>
    <td>nor_Latn</td>
    <td>56.4</td>
    <td><b>57.8</b></td>
  </tr>
  <tr>
    <td>nse_Latn</td>
    <td>45.9</td>
    <td><b>48.5</b></td>
    <td>nso_Latn</td>
    <td>48.6</td>
    <td>48.6</td>
    <td>nya_Latn</td>
    <td><b>56.0</b></td>
    <td>47.4</td>
    <td>nyn_Latn</td>
    <td>43.0</td>
    <td><b>44.1</b></td>
  </tr>
  <tr>
    <td>nzi_Latn</td>
    <td>33.0</td>
    <td><b>33.8</b></td>
    <td>ori_Orya</td>
    <td>67.3</td>
    <td>67.3</td>
    <td>ory_Orya</td>
    <td>66.9</td>
    <td><b>70.7</b></td>
    <td>oss_Cyril</td>
    <td>55.5</td>
    <td><b>57.5</b></td>
  </tr>
  <tr>
    <td>pag_Latn</td>
    <td><b>55.5</b></td>
    <td>52.5</td>
    <td>pam_Latn</td>
    <td>42.0</td>
    <td>37.8</td>
    <td>pan_Guru</td>
    <td><b>64.1</b></td>
    <td>64.1</td>
    <td>pap_Latn</td>
    <td><b>65.6</b></td>
    <td>59.8</td>
  </tr>
  <tr>
    <td>pcm_Latn</td>
    <td><b>66.1</b></td>
    <td>65.9</td>
    <td>pdt_Latn</td>
    <td><b>60.0</b></td>
    <td>56.5</td>
    <td>pes_Arab</td>
    <td><b>69.0</b></td>
    <td>69.0</td>
    <td>pis_Latn</td>
    <td>64.3</td>
    <td><b>65.0</b></td>
  </tr>
  <tr>
    <td>plh_Latn</td>
    <td>46.8</td>
    <td><b>52.9</b></td>
    <td>poh_Latn</td>
    <td>44.3</td>
    <td><b>45.5</b></td>
    <td>pol_Latn</td>
    <td>64.8</td>
    <td><b>65.1</b></td>
    <td>pon_Latn</td>
    <td>50.5</td>
    <td><b>52.2</b></td>
  </tr>
  <tr>
    <td>prk_Latn</td>
    <td>52.9</td>
    <td><b>53.0</b></td>
    <td>prs_Arab</td>
    <td>69.2</td>
    <td><b>70.0</b></td>
    <td>pxm_Latn</td>
    <td>34.5</td>
    <td><b>41.5</b></td>
    <td>qub_Latn</td>
    <td>51.5</td>
    <td><b>56.3</b></td>
  </tr>
  <tr>
    <td>qug_Latn</td>
    <td><b>65.0</b></td>
    <td>61.3</td>
    <td>quh_Latn</td>
    <td><b>66.7</b></td>
    <td>58.8</td>
    <td>quw_Latn</td>
    <td>55.9</td>
    <td><b>56.0</b></td>
    <td>quy_Latn</td>
    <td>65.5</td>
    <td><b>67.7</b></td>
  </tr>
  <tr>
    <td>qvi_Latn</td>
    <td><b>62.0</b></td>
    <td>58.5</td>
    <td>rap_Latn</td>
    <td>48.9</td>
    <td><b>49.3</b></td>
    <td>rar_Latn</td>
    <td>48.9</td>
    <td><b>51.9</b></td>
    <td>rmy_Latn</td>
    <td>45.4</td>
    <td><b>49.1</b></td>
  </tr>
  <tr>
    <td>rop_Latn</td>
    <td><b>56.6</b></td>
    <td>54.7</td>
    <td>rug_Latn</td>
    <td>53.8</td>
    <td><b>55.1</b></td>
    <td>run_Latn</td>
    <td>48.0</td>
    <td><b>55.2</b></td>
    <td>rus_Cyril</td>
    <td><b>68.1</b></td>
    <td>68.1</td>
  </tr>
  <tr>
    <td>sah_Cyril</td>
    <td>55.1</td>
    <td><b>57.6</b></td>
    <td>sba_Latn</td>
    <td>39.1</td>
    <td><b>41.4</b></td>
    <td>seh_Latn</td>
    <td>45.0</td>
    <td><b>46.7</b></td>
    <td>sin_Sinh</td>
    <td>64.1</td>
    <td><b>66.9</b></td>
  </tr>
  <tr>
    <td>slv_Latn</td>
    <td><b>63.8</b></td>
    <td>60.7</td>
    <td>sme_Latn</td>
    <td><b>42.8</b></td>
    <td>37.6</td>
    <td>smo_Latn</td>
    <td><b>60.8</b></td>
    <td>54.2</td>
    <td>sna_Latn</td>
    <td>42.6</td>
    <td><b>44.9</b></td>
  </tr>
  <tr>
    <td>som_Latn</td>
    <td>33.9</td>
    <td><b>35.5</b></td>
    <td>sop_Latn</td>
    <td><b>36.4</b></td>
    <td>36.0</td>
    <td>sot_Latn</td>
    <td>43.5</td>
    <td><b>45.5</b></td>
    <td>spa_Latn</td>
    <td><b>64.2</b></td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>srm_Latn</td>
    <td>48.1</td>
    <td><b>48.4</b></td>
    <td>srn_Latn</td>
    <td><b>63.7</b></td>
    <td>62.8</td>
    <td>srp_Latn</td>
    <td>64.9</td>
    <td><b>65.2</b></td>
    <td>ssw_Latn</td>
    <td>43.7</td>
    <td><b>37.7</b></td>
  </tr>
  <tr>
    <td>suz_Deva</td>
    <td><b>58.0</b></td>
    <td>57.8</td>
    <td>swe_Latn</td>
    <td><b>66.8</b></td>
    <td>65.3</td>
    <td>swh_Latn</td>
    <td><b>59.8</b></td>
    <td>59.8</td>
    <td>ssn_Latn</td>
    <td><b>46.6</b></td>
    <td>40.2</td>
  </tr>
  <tr>
    <td>tat_Cyril</td>
    <td>62.2</td>
    <td><b>68.2</b></td>
    <td>thz_Latn</td>
    <td>36.4</td>
    <td><b>39.5</b></td>
    <td>tca_Latn</td>
    <td>43.3</td>
    <td><b>50.3</b></td>
    <td>thr_Latn</td>
    <td><b>60.3</b></td>
    <td>55.1</td>
  </tr>
  <tr>
    <td>teo_Latn</td>
    <td>23.7</td>
    <td><b>23.1</b></td>
    <td>tgk_Cyril</td>
    <td><b>60.9</b></td>
    <td>60.9</td>
    <td>tgl_Latn</td>
    <td>56.7</td>
    <td><b>58.7</b></td>
    <td>tha_Thai</td>
    <td><b>63.8</b></td>
    <td>63.8</td>
  </tr>
  <tr>
    <td>tir_Ethi</td>
    <td>50.1</td>
    <td>50.1</td>
    <td>tlh_Latn</td>
    <td><b>65.0</b></td>
    <td>65.0</td>
    <td>tob_Latn</td>
    <td>43.3</td>
    <td><b>50.4</b></td>
    <td>toh_Latn</td>
    <td>37.1</td>
    <td><b>39.0</b></td>
  </tr>
  <tr>
    <td>toj_Latn</td>
    <td><b>36.6</b></td>
    <td>34.1</td>
    <td>ton_Latn</td>
    <td>47.3</td>
    <td><b>51.5</b></td>
    <td>top_Latn</td>
    <td><b>21.9</b></td>
    <td>21.3</td>
    <td>tpi_Latn</td>
    <td>63.8</td>
    <td><b>67.6</b></td>
  </tr>
  <tr>
    <td>tsn_Latn</td>
    <td>39.8</td>
    <td><b>44.1</b></td>
    <td>tsz_Latn</td>
    <td>40.4</td>
    <td><b>41.0</b></td>
    <td>tuc_Latn</td>
    <td><b>57.4</b></td>
    <td>56.9</td>
    <td>tur_Latn</td>
    <td><b>43.7</b></td>
    <td>43.7</td>
  </tr>
  <tr>
    <td>tum_Latn</td>
    <td><b>47.6</b></td>
    <td>43.2</td>
    <td>tur_Latn</td>
    <td><b>62.1</b></td>
    <td>62.1</td>
    <td>twi_Latn</td>
    <td><b>41.4</b></td>
    <td>38.9</td>
    <td>tyv_Cyril</td>
    <td>59.8</td>
    <td><b>60.3</b></td>
  </tr>
  <tr>
    <td>tzo_Latn</td>
    <td><b>39.5</b></td>
    <td>39.5</td>
    <td>udm_Cyril</td>
    <td>49.6</td>
    <td><b>49.9</b></td>
    <td>ukr_Cyril</td>
    <td><b>62.4</b></td>
    <td>62.2</td>
    <td>uzb_Latn</td>
    <td>53.5</td>
    <td><b>57.7</b></td>
  </tr>
  <tr>
    <td>ven_Latn</td>
    <td>41.9</td>
    <td><b>48.6</b></td>
    <td>vie_Latn</td>
    <td>62.4</td>
    <td><b>65.4</b></td>
    <td>wal_Latn</td>
    <td>48.9</td>
    <td><b>42.7</b></td>
    <td>war_Latn</td>
    <td>47.7</td>
    <td><b>54.5</b></td>
  </tr>
  <tr>
    <td>wol_Latn</td>
    <td><b>37.2</b></td>
    <td>33.9</td>
    <td>xav_Latn</td>
    <td><b>25.5</b></td>
    <td>23.7</td>
    <td>xho_Latn</td>
    <td><b>44.9</b></td>
    <td>44.4</td>
    <td>yan_Latn</td>
    <td>50.3</td>
    <td><b>53.5</b></td>
  </tr>
  <tr>
    <td>yap_Latn</td>
    <td>42.8</td>
    <td><b>42.9</b></td>
    <td>yom_Latn</td>
    <td><b>37.6</b></td>
    <td>34.1</td>
    <td>yor_Latn</td>
    <td><b>41.8</b></td>
    <td>35.4</td>
    <td>yua_Latn</td>
    <td>40.1</td>
    <td><b>43.2</b></td>
  </tr>
  <tr>
    <td>zai_Latn</td>
    <td><b>42.6</b></td>
    <td>41.4</td>
    <td>zho_Hani</td>
    <td><b>60.7</b></td>
    <td>60.7</td>
    <td>zlm_Latn</td>
    <td><b>68.4</b></td>
    <td>65.5</td>
    <td>zom_Latn</td>
    <td><b>44.6</b></td>
    <td>44.4</td>
  </tr>
</table>

Table 16: F1 scores of LangSAMP on Taxi1500 using English and the closest donor language as source (Part II).
<table>
  <tr>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
  </tr>
  <tr><td>ace_Latn</td><td>69.9</td><td>72.4</td><td>acm_Arab</td><td>80.6</td><td>81.4</td><td>afr_Latn</td><td>81.4</td></tr>
  <tr><td>als_Latn</td><td>82.3</td><td>82.3</td><td>amh_Ethi</td><td>72.6</td><td>72.6</td><td>apc_Arab</td><td>81.7</td></tr>
  <tr><td>arz_Arab</td><td>82.1</td><td>84.4</td><td>asm_Beng</td><td>83.0</td><td>83.0</td><td>ast_Latn</td><td>87.1</td></tr>
  <tr><td>azj_Latn</td><td>86.5</td><td>84.0</td><td>bak_Cyril</td><td>84.3</td><td>86.5</td><td>bam_Latn</td><td>46.5</td></tr>
  <tr><td>bem_Latn</td><td>61.1</td><td>51.4</td><td>ben_Beng</td><td>83.7</td><td>84.0</td><td>bjn_Latn</td><td>75.9</td></tr>
  <tr><td>bul_Cyril</td><td>86.3</td><td>86.6</td><td>cat_Latn</td><td>85.7</td><td>85.2</td><td>ceb_Latn</td><td>81.2</td></tr>
  <tr><td>ckb_Arab</td><td>80.0</td><td>76.8</td><td>chr_Latn</td><td>76.8</td><td>75.7</td><td>cym_Latn</td><td>73.6</td></tr>
  <tr><td>dyu_Latn</td><td>43.6</td><td>42.4</td><td>dzo_Tibet</td><td>68.2</td><td>59.8</td><td>ell_Grek</td><td>79.5</td></tr>
  <tr><td>est_Latn</td><td>78.9</td><td>78.1</td><td>eus_Latn</td><td>78.8</td><td>80.7</td><td>ewe_Latn</td><td>49.9</td></tr>
  <tr><td>fin_Latn</td><td>80.9</td><td>81.5</td><td>fon_Latn</td><td>40.8</td><td>38.1</td><td>fra_Latn</td><td>87.8</td></tr>
  <tr><td>gle_Latn</td><td>61.5</td><td>64.4</td><td>gig_Latn</td><td>87.6</td><td>87.6</td><td>grn_Latn</td><td>71.6</td></tr>
  <tr><td>hau_Latn</td><td>59.3</td><td>64.2</td><td>heb_Hebr</td><td>76.8</td><td>80.2</td><td>hin_Deva</td><td>82.8</td></tr>
  <tr><td>hun_Latn</td><td>86.6</td><td>87.5</td><td>hye_Armn</td><td>81.3</td><td>80.3</td><td>ibo_Latn</td><td>71.4</td></tr>
  <tr><td>isl_Latn</td><td>78.0</td><td>78.3</td><td>ita_Latn</td><td>86.4</td><td>87.5</td><td>jav_Latn</td><td>79.9</td></tr>
  <tr><td>kac_Latin</td><td>48.9</td><td>46.6</td><td>kam_Latn</td><td>45.8</td><td>48.3</td><td>kan_Knda</td><td>82.9</td></tr>
  <tr><td>kbp_Latn</td><td>42.8</td><td>42.2</td><td>kea_Latn</td><td>73.1</td><td>73.1</td><td>khm_Khmr</td><td>82.7</td></tr>
  <tr><td>kir_Cyril</td><td>79.3</td><td>80.1</td><td>kmr_Latn</td><td>46.2</td><td>42.6</td><td>kmr_Latn</td><td>69.8</td></tr>
  <tr><td>lao_Laoo</td><td>83.4</td><td>82.9</td><td>lij_Latn</td><td>76.4</td><td>74.9</td><td>lim_Latn</td><td>74.1</td></tr>
  <tr><td>lmo_Latn</td><td>77.0</td><td>78.3</td><td>lit_Latn</td><td>76.4</td><td>76.4</td><td>lua_Latn</td><td>54.4</td></tr>
  <tr><td>lus_Latn</td><td>64.8</td><td>64.8</td><td>lvv_Latn</td><td>83.2</td><td>83.0</td><td>mai_Deva</td><td>82.9</td></tr>
  <tr><td>min_Latn</td><td>76.7</td><td>79.8</td><td>mkd_Cyril</td><td>83.6</td><td>82.8</td><td>mlt_Latn</td><td>81.3</td></tr>
  <tr><td>mya_Mymr</td><td>80.5</td><td>78.8</td><td>nld_Latn</td><td>85.1</td><td>86.4</td><td>nno_Latn</td><td>86.0</td></tr>
  <tr><td>nso_Latn</td><td>57.6</td><td>57.6</td><td>nya_Latn</td><td>69.2</td><td>70.9</td><td>oci_Latn</td><td>85.0</td></tr>
  <tr><td>pan_Guru</td><td>76.4</td><td>76.4</td><td>pap_Latn</td><td>76.9</td><td>78.1</td><td>pes_Arab</td><td>87.5</td></tr>
  <tr><td>por_Latn</td><td>85.3</td><td>86.8</td><td>prs_Arab</td><td>85.0</td><td>85.5</td><td>quy_Latn</td><td>62.6</td></tr>
  <tr><td>rus_Cyril</td><td>86.8</td><td>86.8</td><td>sag_Latn</td><td>51.3</td><td>50.2</td><td>san_Deva</td><td>72.9</td></tr>
  <tr><td>sin_Sinh</td><td>82.7</td><td>82.7</td><td>slk_Latn</td><td>85.4</td><td>85.1</td><td>slv_Latn</td><td>84.2</td></tr>
  <tr><td>snd_Arab</td><td>70.4</td><td>70.4</td><td>som_Latn</td><td>58.9</td><td>61.1</td><td>sot_Latn</td><td>64.1</td></tr>
  <tr><td>srp_Cyril</td><td>84.8</td><td>85.0</td><td>sws_Latn</td><td>64.1</td><td>65.2</td><td>sun_Latn</td><td>82.6</td></tr>
  <tr><td>szl_Latn</td><td>72.4</td><td>72.4</td><td>tam_Taml</td><td>81.2</td><td>81.2</td><td>tat_Cyril</td><td>83.6</td></tr>
  <tr><td>tgl_Latn</td><td>82.1</td><td>81.7</td><td>tha_Thai</td><td>85.4</td><td>85.7</td><td>tir_Ethi</td><td>60.3</td></tr>
  <tr><td>tso_Latn</td><td>57.3</td><td>60.3</td><td>tuk_Latn</td><td>78.1</td><td>78.5</td><td>tum_Latn</td><td>65.4</td></tr>
  <tr><td>uig_Arab</td><td>75.5</td><td>75.5</td><td>ukr_Cyril</td><td>84.3</td><td>83.8</td><td>umb_Latn</td><td>41.0</td></tr>
  <tr><td>vie_Latn</td><td>86.2</td><td>83.9</td><td>war_Latn</td><td>80.7</td><td>81.3</td><td>wol_Latn</td><td>50.5</td></tr>
  <tr><td>zho_Hans</td><td>89.6</td><td>89.2</td><td>zhg_Hant</td><td>88.8</td><td>88.8</td><td>zsm_Latn</td><td>86.4</td></tr>
  <tr><td>zul_Latn</td><td>86.0</td><td>86.0</td><td>zul_Latn</td><td>68.1</td><td>68.1</td><td>zul_Latn</td><td>68.1</td></tr>
</table>

Table 17: F1 scores of LANGSAMP on SIB200. using English and the closest donor language as source.
<table>
  <tr>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
  </tr>
  <tr>
    <td>ace_Latin</td><td>41.5</td><td>56.9</td><td>afr_Latin</td><td>75.8</td><td>80.3</td><td>als_Latin</td><td>80.9</td><td>80.9</td><td>amh_Ethi</td><td>39.7</td><td>39.7</td>
  </tr>
  <tr>
    <td>arg_Latin</td><td>82.2</td><td>88.8</td><td>arz_Arab</td><td>55.1</td><td>82.6</td><td>asm_Beng</td><td>69.0</td><td>45.9</td><td>ast_Latin</td><td>84.6</td><td>85.8</td>
  </tr>
  <tr>
    <td>aze_Latin</td><td>65.0</td><td>74.0</td><td>bak_Cyril</td><td>62.5</td><td>72.2</td><td>bar_Latin</td><td>68.2</td><td>62.8</td><td>bel_Cyril</td><td>74.9</td><td>79.7</td>
  </tr>
  <tr>
    <td>bih_Deva</td><td>56.2</td><td>67.6</td><td>bod_Tibetan</td><td>35.2</td><td>35.7</td><td>bos_Latin</td><td>70.1</td><td>75.2</td><td>bre_Latin</td><td>63.3</td><td>66.0</td>
  </tr>
  <tr>
    <td>cat_Latin</td><td>83.8</td><td>85.1</td><td>cbk_Latin</td><td>53.7</td><td>48.9</td><td>ceb_Latin</td><td>56.0</td><td>26.8</td><td>ces_Latin</td><td>77.9</td><td>69.6</td>
  </tr>
  <tr>
    <td>chv_Cyril</td><td>73.6</td><td>84.3</td><td>ckb_Arab</td><td>76.0</td><td>60.6</td><td>cos_Latin</td><td>63.0</td><td>61.9</td><td>chr_Latin</td><td>52.7</td><td>59.4</td>
  </tr>
  <tr>
    <td>cym_Latin</td><td>61.7</td><td>62.1</td><td>dan_Latin</td><td>81.4</td><td>81.3</td><td>deu_Latin</td><td>74.6</td><td>74.6</td><td>dig_Latin</td><td>54.0</td><td>72.2</td>
  </tr>
  <tr>
    <td>ell_Greek</td><td>71.9</td><td>72.0</td><td>emi_Latin</td><td>41.3</td><td>41.3</td><td>eng_Latin</td><td>83.5</td><td>83.5</td><td>epo_Latin</td><td>68.3</td><td>68.3</td>
  </tr>
  <tr>
    <td>eus_Latin</td><td>60.9</td><td>65.1</td><td>ext_Latin</td><td>44.2</td><td>48.6</td><td>fao_Latin</td><td>68.7</td><td>79.2</td><td>fas_Arab</td><td>55.0</td><td>53.6</td>
  </tr>
  <tr>
    <td>fra_Latin</td><td>76.5</td><td>76.5</td><td>frr_Latin</td><td>52.0</td><td>52.0</td><td>fry_Latin</td><td>74.6</td><td>73.9</td><td>fur_Latin</td><td>58.2</td><td>54.0</td>
  </tr>
  <tr>
    <td>gle_Latin</td><td>72.6</td><td>69.6</td><td>gpl_Latin</td><td>80.7</td><td>86.1</td><td>grn_Latin</td><td>55.1</td><td>59.8</td><td>guj_Gujarati</td><td>61.2</td><td>61.0</td>
  </tr>
  <tr>
    <td>heb_Hebr</td><td>52.0</td><td>52.9</td><td>hin_Deva</td><td>69.4</td><td>69.4</td><td>hrv_Latin</td><td>77.2</td><td>79.8</td><td>hsb_Latin</td><td>74.3</td><td>69.7</td>
  </tr>
  <tr>
    <td>hye_Armn</td><td>53.0</td><td>62.2</td><td>ibo_Latin</td><td>58.1</td><td>58.4</td><td>ido_Latin</td><td>82.6</td><td>81.5</td><td>ilo_Latin</td><td>80.0</td><td>74.9</td>
  </tr>
  <tr>
    <td>ind_Latin</td><td>67.6</td><td>67.6</td><td>isl_Latin</td><td>70.1</td><td>75.4</td><td>ita_Latin</td><td>78.2</td><td>79.5</td><td>jav_Latin</td><td>56.0</td><td>86.4</td>
  </tr>
  <tr>
    <td>jpn_Japan</td><td>22.0</td><td>22.0</td><td>kna_Knda</td><td>57.5</td><td>61.8</td><td>kat_Geor</td><td>68.7</td><td>60.1</td><td>kaz_Cyril</td><td>50.5</td><td>57.1</td>
  </tr>
  <tr>
    <td>kin_Latin</td><td>69.6</td><td>67.3</td><td>kir_Cyril</td><td>44.3</td><td>60.9</td><td>kor_Hang</td><td>50.4</td><td>51.2</td><td>ksh_Latin</td><td>59.7</td><td>51.4</td>
  </tr>
  <tr>
    <td>lat_Latin</td><td>71.9</td><td>81.4</td><td>lav_Latin</td><td>74.4</td><td>69.0</td><td>lij_Latin</td><td>45.2</td><td>54.2</td><td>lim_Latin</td><td>69.3</td><td>61.2</td>
  </tr>
  <tr>
    <td>lit_Latin</td><td>74.2</td><td>76.1</td><td>lin_Latin</td><td>73.6</td><td>65.5</td><td>ltz_Latin</td><td>67.9</td><td>67.9</td><td>lzh_Hani</td><td>14.8</td><td>14.8</td>
  </tr>
  <tr>
    <td>mar_Deva</td><td>62.5</td><td>76.6</td><td>mhr_Cyril</td><td>60.6</td><td>72.3</td><td>min_Latin</td><td>42.6</td><td>57.5</td><td>mkd_Cyril</td><td>72.2</td><td>73.1</td>
  </tr>
  <tr>
    <td>mlt_Latin</td><td>75.9</td><td>75.9</td><td>mon_Cyril</td><td>68.7</td><td>60.9</td><td>mri_Latin</td><td>50.0</td><td>47.0</td><td>msa_Latin</td><td>67.6</td><td>73.0</td>
  </tr>
  <tr>
    <td>mya_Mymr</td><td>55.3</td><td>56.3</td><td>mzn_Arab</td><td>43.3</td><td>47.2</td><td>nan_Latin</td><td>88.1</td><td>36.6</td><td>nap_Latin</td><td>63.0</td><td>55.3</td>
  </tr>
  <tr>
    <td>nep_Deva</td><td>56.9</td><td>60.4</td><td>nld_Latin</td><td>80.8</td><td>80.0</td><td>nno_Latin</td><td>77.6</td><td>77.6</td><td>nor_Latin</td><td>77.9</td><td>80.4</td>
  </tr>
  <tr>
    <td>ori_Orya</td><td>34.2</td><td>34.2</td><td>oss_Cyril</td><td>50.6</td><td>59.1</td><td>pan_Guru</td><td>51.5</td><td>51.5</td><td>pms_Latin</td><td>80.9</td><td>78.4</td>
  </tr>
  <tr>
    <td>pol_Latin</td><td>77.7</td><td>71.1</td><td>por_Latin</td><td>78.9</td><td>84.9</td><td>pus_Arab</td><td>42.6</td><td>45.3</td><td>que_Latin</td><td>70.4</td><td>55.5</td>
  </tr>
  <tr>
    <td>ron_Latin</td><td>77.8</td><td>75.5</td><td>rus_Latin</td><td>67.5</td><td>67.5</td><td>sah_Cyril</td><td>71.9</td><td>77.9</td><td>san_Deva</td><td>38.4</td><td>53.4</td>
  </tr>
  <tr>
    <td>sco_Latin</td><td>86.4</td><td>84.5</td><td>sgs_Latin</td><td>66.4</td><td>69.8</td><td>sin_Sinh</td><td>53.0</td><td>51.2</td><td>slk_Latin</td><td>76.4</td><td>55.9</td>
  </tr>
  <tr>
    <td>snd_Arab</td><td>41.8</td><td>41.8</td><td>som_Latin</td><td>57.5</td><td>56.2</td><td>spa_Latin</td><td>77.6</td><td>77.6</td><td>sqi_Latin</td><td>76.8</td><td>78.7</td>
  </tr>
  <tr>
    <td>sun_Latin</td><td>50.8</td><td>75.1</td><td>swa_Latin</td><td>71.8</td><td>71.8</td><td>swe_Latin</td><td>70.9</td><td>65.8</td><td>szl_Latin</td><td>70.9</td><td>70.9</td>
  </tr>
  <tr>
    <td>tat_Cyril</td><td>63.8</td><td>76.5</td><td>tel_Telu</td><td>48.1</td><td>49.0</td><td>tgl_Cyril</td><td>68.4</td><td>68.4</td><td>tgl_Latin</td><td>71.9</td><td>73.7</td>
  </tr>
  <tr>
    <td>tuk_Latin</td><td>54.4</td><td>57.3</td><td>tur_Latin</td><td>77.1</td><td>77.1</td><td>uiq_Arab</td><td>47.7</td><td>62.3</td><td>ukr_Cyril</td><td>76.6</td><td>85.3</td>
  </tr>
  <tr>
    <td>uzb_Latin</td><td>73.2</td><td>76.0</td><td>vec_Latin</td><td>68.0</td><td>75.1</td><td>vep_Latin</td><td>72.0</td><td>63.0</td><td>vie_Latin</td><td>72.3</td><td>49.7</td>
  </tr>
  <tr>
    <td>vol_Latin</td><td>61.0</td><td>36.5</td><td>war_Latin</td><td>64.9</td><td>56.1</td><td>wuu_Hani</td><td>35.7</td><td>66.7</td><td>xmt_Geor</td><td>69.3</td><td>55.7</td>
  </tr>
  <tr>
    <td>yor_Latin</td><td>69.3</td><td>41.7</td><td>yue_Hani</td><td>25.7</td><td>73.5</td><td>zea_Latin</td><td>62.9</td><td>75.4</td><td>zho_Hani</td><td>25.2</td><td>25.2</td>
  </tr>
</table>

Table 18: F1 scores of LANGSAMP on NER using English and the closest donor language as source.

<table>
  <tr>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
    <th>Language</th>
    <th>English Closest donor</th>
  </tr>
  <tr>
    <td>afr_Latin</td><td>88.5</td><td>79.5</td><td>ajp_Arab</td><td>71.1</td><td>41.9</td><td>ain_Latin</td><td>53.4</td><td>45.1</td><td>amh_Ethi</td><td>66.8</td><td>66.8</td>
  </tr>
  <tr>
    <td>bam_Latin</td><td>43.0</td><td>31.2</td><td>bel_Cyril</td><td>86.4</td><td>93.8</td><td>ben_Beng</td><td>87.5</td><td>80.2</td><td>bre_Latin</td><td>61.1</td><td>62.3</td>
  </tr>
  <tr>
    <td>cat_Latin</td><td>86.8</td><td>95.8</td><td>ceb_Latin</td><td>66.7</td><td>32.5</td><td>ces_Latin</td><td>85.4</td><td>73.3</td><td>cym_Latin</td><td>65.5</td><td>60.4</td>
  </tr>
  <tr>
    <td>deu_Latin</td><td>88.2</td><td>88.2</td><td>ell_Greek</td><td>84.9</td><td>75.5</td><td>eng_Latin</td><td>96.0</td><td>96.0</td><td>est_Latin</td><td>84.7</td><td>77.4</td>
  </tr>
  <tr>
    <td>fao_Latin</td><td>88.7</td><td>67.5</td><td>fas_Arab</td><td>72.2</td><td>69.1</td><td>fin_Latin</td><td>82.2</td><td>75.8</td><td>fra_Latin</td><td>85.8</td><td>85.8</td>
  </tr>
  <tr>
    <td>gle_Latin</td><td>64.6</td><td>65.5</td><td>gig_Latin</td><td>83.6</td><td>87.8</td><td>glv_Latin</td><td>51.9</td><td>57.8</td><td>grc_Greek</td><td>71.6</td><td>71.6</td>
  </tr>
  <tr>
    <td>gsw_Latin</td><td>82.7</td><td>82.7</td><td>hbo_Hebr</td><td>38.9</td><td>37.4</td><td>heb_Hebr</td><td>67.9</td><td>69.3</td><td>hin_Deva</td><td>77.2</td><td>77.2</td>
  </tr>
  <tr>
    <td>hsb_Latin</td><td>83.7</td><td>73.4</td><td>hun_Latin</td><td>82.2</td><td>42.0</td><td>hye_Armn</td><td>85.1</td><td>84.9</td><td>hyw_Armn</td><td>83.0</td><td>56.8</td>
  </tr>
  <tr>
    <td>isl_Latin</td><td>82.7</td><td>81.2</td><td>ita_Latin</td><td>88.9</td><td>92.4</td><td>jav_Latin</td><td>75.4</td><td>78.8</td><td>jpn_Japan</td><td>33.1</td><td>33.1</td>
  </tr>
  <tr>
    <td>kmr_Latin</td><td>76.6</td><td>61.6</td><td>kor_Hang</td><td>52.7</td><td>45.3</td><td>lat_Latin</td><td>72.8</td><td>74.2</td><td>lav_Latin</td><td>83.7</td><td>78.4</td>
  </tr>
  <tr>
    <td>lit_Latin</td><td>82.1</td><td>80.7</td><td>lzh_Hani</td><td>24.5</td><td>24.5</td><td>mal_Mlym</td><td>86.0</td><td>52.1</td><td>mar_Deva</td><td>84.1</td><td>81.7</td>
  </tr>
  <tr>
    <td>myv_Cyril</td><td>65.9</td><td>58.4</td><td>nap_Latin</td><td>82.4</td><td>70.6</td><td>nds_Latin</td><td>79.1</td><td>34.0</td><td>nld_Latin</td><td>88.2</td><td>82.2</td>
  </tr>
  <tr>
    <td>pcm_Latin</td><td>58.2</td><td>48.1</td><td>pol_Latin</td><td>84.2</td><td>89.1</td><td>por_Latin</td><td>87.9</td><td>92.0</td><td>que_Latin</td><td>63.3</td><td>52.6</td>
  </tr>
  <tr>
    <td>rus_Cyril</td><td>88.7</td><td>88.7</td><td>sah_Cyril</td><td>74.2</td><td>74.5</td><td>san_Deva</td><td>25.5</td><td>32.7</td><td>sin_Sinh</td><td>56.2</td><td>34.4</td>
  </tr>
  <tr>
    <td>slv_Latin</td><td>77.6</td><td>79.0</td><td>sme_Latin</td><td>74.8</td><td>60.6</td><td>spa_Latin</td><td>87.8</td><td>87.8</td><td>sqi_Latin</td><td>77.5</td><td>72.7</td>
  </tr>
  <tr>
    <td>swe_Latin</td><td>92.7</td><td>83.2</td><td>tam_Tamil</td><td>74.6</td><td>74.6</td><td>tat_Cyril</td><td>72.4</td><td>70.9</td><td>tel_Telu</td><td>80.9</td><td>55.9</td>
  </tr>
  <tr>
    <td>tha_Thai</td><td>58.3</td><td>27.5</td><td>tur_Latin</td><td>71.2</td><td>71.2</td><td>uiq_Arab</td><td>68.2</td><td>48.3</td><td>ukr_Cyril</td><td>85.6</td><td>91.7</td>
  </tr>
  <tr>
    <td>vie_Latin</td><td>68.4</td><td>32.4</td><td>wol_Latin</td><td>61.6</td><td>57.4</td><td>xav_Latin</td><td>16.7</td><td>11.2</td><td>yor_Latin</td><td>62.7</td><td>46.5</td>
  </tr>
</table>

Table 19: F1 scores of LANGSAMP on POS using English and the closest donor language as source.

Reviewer paper summaries: The paper proposes modifying multilingual encoder-style model (e.g. XLM-R) training by adding language- and script-embeddings (i.e. adding a learning script/language specific vector) to output hidden states. The output hidden states are then passed to the language modelling head as usual.

Adding the language/script embeddings at the output enables using these embeddings for pretraining, then discarding when it comes to using the model on a downstream task. This is desirable since it does not necessitate LangID / identifying the script for downstream usage.

Empirical results suggest that the language/script embeddings moderately improve performance and make the output representations more language-neutral. The authors also use the learned embeddings to estimate language similarity, which allows selecting improved source ("donor") languages in cross-lingual transfer.
Summary of strengths: - The method is simple and well motivated by addressing a clear concern (overspecialization of token embeddings to their script/language).
- The authors conduct qualitative and quantitative analysis of their method, yielding results which align with intuition to a large extent.
Summary of weaknesses: - The authors mention in Limitations that "we do not consider the possibility of introducing language and script embeddings before the Transformer blocks. Although this is also a possible architecture, it does not fulfill our aim and therefore is not relevant to us." (their aim being to create a model which does not need lang/script ID for downstream usage). I disagree. Although avoiding lang/script ID is convenient, it would be important to quantify how much is lost by avoiding it and how much better we could do by using lang/script ID for downstream tasks. This is very much related to the authors' motivation and experiments, so I don't quite buy the motivation for avoiding it.
Comments, suggestions and typos: - Figure 3 is hard to read. It should maybe show a subset of languages instead of trying to squeeze in all of them. 

Typos: 
 Line 137: "self-learning" is a bit odd; "self-supervised" would be more standard
Overall assessment scores: 3
Metareview score: 4
Metareview: The paper introduces LANGSAMP to improve multilingual pre-trained language models. It incorporates learnable language and script embeddings to encourage language neutrality, making the models learn more language-agnostic features. This design is supported by experimental results on zero-shot cross-lingual transfer, as well as qualitative and quantitative analysis.
Summary of reasons to publish: - The proposed method is well-justified with a strong underlying motivation.
- The experimental setup is comprehensive and complete.
- Extensive qualitative and quantitative analysis is provided to support the claim, along with detailed ablation studies.