MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment
Authors: Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schuetze
Accepted at: acl25

MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

Amir Hossein Kargarani1   Ali Modarressi1   Nafiseh Nikeghbal2
Jana Diesner2   François Yvon3   Hinrich Schütze1

1LMU Munich & Munich Center for Machine Learning
2Technical University of Munich
3Sorbonne Université & CNRS, ISIR
{amir, ali}@cis.lmu.de, nafiseh.nikeghbal@tum.de

Abstract
English-centric large language models (LLMs) often show strong multilingual capabilities. However, their multilingual performance remains unclear and is under-evaluated for many other languages. Most benchmarks for multilinguality focus on classic NLP tasks or cover a minimal number of languages. We introduce MEXA, a method for assessing the multilingual capabilities of pre-trained English-centric LLMs using parallel sentences, which are available for more languages than existing downstream tasks. MEXA leverages that English-centric LLMs use English as a pivot language in their intermediate layers. MEXA computes the alignment between English and non-English languages using parallel sentences to evaluate the transfer of language understanding from English to other languages. This alignment can be used to estimate model performance in different languages. We conduct controlled experiments using various parallel datasets (FLORES-200 and Bible), models (Llama family, Gemma family, Mistral, and OLMo), and established downstream tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute embeddings in decoder-only models. Our results show that MEXA, in its default settings, achieves an average Pearson correlation of 0.90 between its predicted scores and actual task performance across languages. This suggests that MEXA is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs.

Leaderboard cis-lmu-mexa.hf.space
Code github.com/cisnlp/MEXA

1 Introduction
Most state-of-the-art autoregressive large language models (LLMs) are “English-centric”, including closed-source models such as GPT-4 (OpenAI et al., 2023), open-weight models such as Llama 3 (Dubey et al., 2024); and open-source models such as OLMo (Groeneveld et al., 2024). English-centric refers to the majority of the pre-training data for these models being in English (Zhong et al., 2024; Kew et al., 2024). Even models labeled as heavily multilingual, such as BLOOM (Big-Science Workshop et al., 2023), have their major pre-training data in English (Laurençon et al., 2022).

Except for open-source models, where pre-training data is available and language distribution is transparent, there is still confusion about the language capabilities and coverage of other LLMs. Primarily, the focus in evaluating LLMs has been on developing benchmarks to assess their performance in English. Most benchmarks in multilingual settings consist of classical monolingual NLP tasks such as sequence labeling (Ahuja et al., 2023; Lai et al., 2023a), automatic translation of popular English benchmarks such as MMLU (Hendrycks et al., 2021) into a limited number of languages (Lai et al., 2023b; OpenAI, 2024), or the creation of language-specific benchmarks (Ghahroodi et al., 2024; Koto et al., 2024; Son et al., 2024; Yüksel et al., 2024; Li et al., 2024c).

Most LLMs are English-centric, either by choice or due to the availability of abundant data sources in English. For these models to be effective in other languages, the other languages must align with the dominant language, i.e., English. Given such alignment, English could act as a “rising tide that raises all ships,” meaning that improvements in English performance could benefit other languages, especially in tasks such as reasoning (Zhu et al., 2024). Contrarily, if a language does not align well with English, an English-centric LLM may not provide meaningful coverage for that language. Indeed, Wendler et al. (2024) have found that for Llama 2 (Touvron et al., 2023b), an English-centric LLM, English could be seen as a kind of “pivot”
language, enabling to solve complex semantic tasks in a foreign language through a detour into English. More precisely, they show that Llama 2 was able to decode semantically correct next tokens in the middle layers, assigning higher probabilities to the English tokens than to the foreign version, which is only selected in the upper layers. Zhao et al. (2024) present a hypothesis regarding the middle layers of English-centric LLMs, suggesting that these models use English as a means of reasoning while incorporating multilingual knowledge. Based on their analysis, the number of language-specific neurons in the middle layers decreases within the self-attention mechanism but remains consistent across the layers of the feed-forward structure when processing multilingual queries.

In this paper, we introduce MEXA, a method to estimate the actual multilingual coverage of English-centric LLMs. It builds on the observation that these models semantically use English as a pivot language in their middle layers, by measuring how well embeddings of non-English sentences align with their English counterparts.

We verify the MEXA estimation of language coverage for each LLM, using Pearson correlation between estimated and actual scores for various tasks. We use two parallel datasets: FLORES-200 (NLLB Team et al., 2022) and Bible (Mayer and Cysouw, 2014); nine LLMs: Llama family, Gemma family, Mistral, and OLMo; and three tasks: Belebele (Bandarkar et al., 2024), m-MMLU, and m-ARC (Lai et al., 2023b). Our results show that MEXA achieves a promising average Pearson correlation of 0.90 with established downstream tasks across nine models and two parallel datasets. In our study on the calculation of MEXA scores, we conduct multiple design analyses to examine the impact of token-level pooling for the embeddings (i.e., using the last token versus a weighted average) and layer-level pooling in computing alignment scores. While MEXA demonstrates a high correlation across most setups, we find that a weighted average based on tokens, combined with mean pooling, yields the best results. In summary, MEXA offers a scalable way to estimate the multilingual coverage of English-centric LLMs via alignment with English.

2 Background and Related Work

We discuss distribution of pre-training data in LLMs, and multilingual evaluation benchmarks in Appendices A.1 and A.2, and focus on cross-lingual alignment here. Research on cross-lingual alignment either aims to uncover the underlying mechanisms of alignment and assesses its impact on models and downstream tasks, or attempts to enhance model performance by enforcing alignment before, during, or after the pre-training phase. Most of these papers have focused on encoder-only models, such as XLM-R (Conneau et al., 2020a) and mBERT (Devlin et al., 2019), among others (Hämmerl et al., 2024). In this work, we focus on decoder-only models.

Understanding Alignment. Ye et al. (2023) show that English-centric models such as Llama 1 (Touvron et al., 2023a) possess multilingual transfer abilities (after fine-tuning on one source language, they can be applied to other languages) and may even surpass the multilingual transfer abilities of multilingual pre-trained models such as BLOOM (BigScience Workshop et al., 2023). Schäfer et al. (2024) find that GPT-2-style decoder-only models show strong cross-lingual generalization when trained on an imbalanced mix of languages. However, when trained on a balanced language set, they do not observe increased performance compared to monolingual settings. Wendler et al. (2024) perform single-token analysis to demonstrate that English-centered LLMs, such as Llama 2, use English semantically as an internal latent language in the middle layers when handling multilingual queries. Zhong et al. (2024) extend this analysis to multiple tokens, also showing that an LLM dominated by both English and Japanese uses both languages as internal latent languages. Zhao et al. (2024) explore how LLMs handle multilingualism. They hypothesize that LLMs initially interpret the query and convert multilingual inputs into English for task-solving. In the middle layers, the models rely on English with self-attention mechanisms for reasoning, while employing multilingual knowledge through feed-forward structures. In the final layers, LLMs generate responses consistent with the original query language. Li et al. (2025) and Li et al. (2024b) are even more closely related to ours. Li et al. (2025) uses absolute cosine similarity values between last token embeddings derived from parallel sentences with English to predict the ranking of language performance across various models. However, as we discuss in Section 3, relying solely on absolute cosine values can be misleading, and as shown in Section 5.3, absolute cosine values are less correlated
with downstream tasks than MEXA score. Li et al. (2024b) uses English probing tasks and their automatic translations to construct a multilingual evaluation. While they compare embedding similarity scores between high- and low-resource languages with corresponding evaluation results, similar to Li et al. (2025), they do not assess whether these correlations hold across other downstream tasks. In Section 5, we demonstrate that MEXA scores align closely with a broad range of downstream tasks.

Boosting Alignment. The idea to enforce alignment in encoder-only models using parallel sentences dates back to Conneau and Lample (2019), and has been explored under various guises, e.g., using mixed-language sentences and/or bilingual dictionaries (Huang et al., 2019; Conneau et al., 2020b; Cao et al., 2020; Kulshreshtha et al., 2020; Efimov et al., 2023; Zhang et al., 2023b). Recently, Li et al. (2024d) improved multilingual alignment by initializing the decoder-only models to generate similar representations of aligned words using contrastive learning and preserving this alignment using a code-switching strategy during pre-training. Liu et al. (2024a) propose a data allocation technique to select a core subset of languages for fine-tuning, better aligning the multilingual capabilities of decoder-only LLMs and making them more truthful in their responses. Li et al. (2024a) propose aligning internal sentence representations across different languages using multilingual contrastive learning and aligning outputs by following cross-lingual instructions in the target language for decoder-only models.

3 MEXA

We now describe the MEXA method for computing the alignment score of language \( L_1 \) with a pivot language \( L_2 \), given the language model \( m \). In this paper, we use the term cross-lingual alignment, or simply alignment to refer to the semantic similarity of multilingual embeddings across languages. \( L_2 \), for English-centric LLMs and in this paper, is English. To assess alignment, we use parallel sentences in two languages, \( L_1 \) and \( L_2 \). The goal of semantic similarity is to ensure that parallel sentences have sufficiently high similarity, reflecting alignment between the two languages. However, considering only the absolute cosine similarity value as the alignment score does not guarantee proper alignment. For some languages, even non-parallel sentences exhibit similarity scores comparable to those of parallel sentences (see §5.3). This is largely due to the anisotropy problem observed in transformer models, which can lead to so-called hubness issues, making it difficult to distinguish between similar and dissimilar embeddings (Ehtayarajh, 2019), especially in multilingual models (Hämmerl et al., 2023; Rajaee and Pilehvar, 2022). However, a direct comparative analysis of the cosine similarity between parallel and non-parallel sentence pairs across languages can help overcome these issues. Instead of using the absolute cosine similarity value for alignment, we assign binary values (1 or 0) based on whether a criterion for semantic similarity is satisfied. Our criterion imposes that (a) parallel sentences should have high cosine similarity, and (b) non-parallel pairs should also have low similarity values, ensuring the similarity is not random or biased. Specifically, if the cosine similarity for a pair of parallel sentences is higher than for any non-parallel sentences, we assign a value of 1 for this pair; otherwise, a value of 0. This approach sidesteps the hubness problem since the absolute cosine similarity values themselves are not directly used.

To compute MEXA, we first apply the cosine similarity function to the pairs of embeddings of parallel sentences in languages \( L_1 \) and \( L_2 \). In Section 3.1, we describe how embeddings can be computed for each layer \( l \) of the autoregressive language model \( m \). We generate a square matrix \( C(L_1, L_2, m, l) \) representing cosine similarities of embeddings at the output of layer \( l \) for all parallel sentences in languages \( L_1 \) and \( L_2 \). We denote \( c_{ij}(l) \) the element in the \( i \)-th row and \( j \)-th column of \( C(L_1, L_2, m, l) \). It represents the cosine similarity between the \( i \)-th sentence of \( L_1 \) and the \( j \)-th sentence of \( L_2 \) at layer \( l \) of language model \( m \). The diagonal elements of \( C \), denoted \( c_{ii}(l) \), represent the cosine similarity between parallel sentence pairs from \( L_1 \) and \( L_2 \). We define the MEXA alignment score \( \mu(C(L_1, L_2, m, l)) \) as follows:

\[
\frac{1}{n} \sum_{i=1}^{n} \mathbf{1}\left( c_{ii}(l) > \max_{j \in \{1, ..., n\} \setminus \{i\}} \{c_{ij}(l), c_{ji}(l)\} \right),
\]

where \( n \) is the number of diagonal elements (i.e., the dimension of the matrix), and \( \mathbf{1}(\cdot) \) is the indicator function, which equals 1 if its argument condition evaluates to true and 0 otherwise. This alignment score measures how often \( c_{ii}(l) \) is the maximum value in both its row and column. The
MEXA alignment score can alternatively be understood as a measure of sentence retrieval performance (Hu et al., 2020; Liu et al., 2024b; Hämmerl et al., 2024), with the metric of P@1 applied with queries in language \( L_1 \) and answers in \( L_2 \), and vice versa. We discuss other ways to calculate semantic similarity between languages in Appendix A.3.

**Layer-wise Pooling.** The MEXA alignment score \( \mu(C(L_1, L_2, m, l)) \) is computed for language \( L_1 \) respect to pivot language \( L_2 \) for each layer \( l \) of the language model \( m \). To compute a single MEXA alignment score given the language model \( m \) and \( L_1, L_2 \), we use mean and max pooling strategies over multiple layers.

3.1 Sentence Embeddings

We focus on autoregressive language models that use a decoder-only architecture. In this architecture, attention is not bidirectional; instead, it takes the form of causal attention (left-to-right). In bidirectional attention, each token has access to every other token in the sequence. However, in causal attention, the embedding of a token at position \( t \) is only influenced by the embedding of preceding tokens at positions \( 0, 1, \ldots, t-1 \). Therefore, simple averaging values biases the embeddings towards sentence-initial words. Instead, we consider alternative methods: using only the last token and weighted averaging. We use and compare both methods to acquire the sentence embeddings needed for MEXA.

A standard way to compute a sentence embedding uses only the last token of that sentence. Jiang et al. (2023b) show that using the last token in the format of a prompt template for a sentence \( s \), such as 'This sentence: {s} means in one word:', can be effective. Inspired by this, Li and Li (2024) used the prompt 'Summarize sentence {s} in one word:' to obtain the last token embedding as the sentence-level text embedding. However, not all models are instruction-tuned; some earlier works, such as Neelakantan et al. (2022); Wang et al. (2024); Ma et al. (2024), use the last token without any prompt. Since the models studied in this paper are only pre-trained and use multiple languages in the input, we decided to use the last token method without any preceding instruction. An alternative is weighted averaging, which relies on the intuition that using only the last token might not represent the entire sentence, as the influence of earlier tokens may have diminished. This implies that the tokens at the end of the sentence should contribute more to the overall embedding than those at the beginning. Another motivation for using weighted averaging is that sentence-final tokens are influenced by preceding tokens and contain more context, while the representation of sentence-initial tokens has significantly less contextual representation. To address this, Muennighoff (2022) proposes to assign weights to each token based on its position. Thus, the sentence embedding of layer \( l \) using position-weighted averaging is:

\[
e_l = \sum_{t=1}^T w_t h_{lt} \quad \text{with} \quad w_t = \frac{t}{\sum_{k=1}^T k},
\]

where \( T \) is the number of tokens in the given sentence, \( h_{lt} \) is the embedding of the \( t \)-th token at layer \( l \), and \( e_l \) is the sentence embedding at layer \( l \).

3.2 Robustness of MEXA

We show that the MEXA alignment score (\( \mu(.) \)) is very robust, and the odds of this score randomly achieving a high value are very slim. Recall that \( \mu(C(L_1, L_2, m, l)) \) measures the fraction of diagonal elements in matrix \( C(L_1, L_2, m, l) \) that have the maximum value in their respective rows and columns. If this condition is met \( k \) times out of \( n \) diagonal elements, then \( \mu(C(L_1, L_2, m, l)) \) is \( \frac{k}{n} \). In an \( n \times n \) random matrix, the probability of a diagonal element being the maximum in its row and column (a total of \( 2n - 1 \) elements) is \( p = \frac{1}{2n-1} \). The probability that at least \( k \) out of \( n \) independent variables are satisfied, given that the diagonal element is the maximum in its row and column, can be computed using the binomial distribution:

\[
P(X \geq \frac{k}{n}) = 1 - \sum_{i=0}^{k-1} \binom{n}{i} p^i (1-p)^{n-i}
\]

In Figure 1, we plot \( P(X \geq \frac{k}{n}) \). This plot illustrates that, given a sufficient number of parallel sen-

![The probability that at least k out of n diagonal elements in an n x n random matrix are the maximum elements in their respective rows and columns.](page_1016_180_377_246.png)

Figure 1: The probability that at least \( k \) out of \( n \) diagonal elements in an \( n \times n \) random matrix are the maximum elements in their respective rows and columns.
tences (\( n \)), the probability of achieving a high score by chance is very low. For example, with \( n = 100 \), the chance of obtaining MEXA alignment score larger than 0.05 (\( k = 5 \)) from a \( 100 \times 100 \) random matrix is \( P(X \geq 0.05) = 0.00016 \).

4 Experiments

We conduct experiments using various multi-parallel datasets (FLORES-200 and the Bible), models (Llama family, Gemma family, Mistral, and OLMo), and existing benchmarks/tasks (Belebele, m-MMLU, m-ARC). Our objective is to assess how well the MEXA alignment score from various parallel datasets correlates with the different benchmarks/tasks for different models.

4.1 Parallel Data

We calculate the MEXA score using the parallel datasets of FLORES-200 (NLLB Team et al., 2022) and the Bible (Mayer and Cysouw, 2014). While there are other high-quality parallel datasets, such as NTREX-128 (Federmann et al., 2022), IN22 (Gala et al., 2023), OPUS-100 (Zhang et al., 2020), Europarl (Koehn, 2005), OpenSubtitles (Lison and Tiedemann, 2016), TED2020 (Reimers and Gurevych, 2020), and Tatoeba (Tatoeba Community, 2006), we chose FLORES-200 due to its high quality and support for a wide range of languages, and the Bible dataset was selected for its extensive language coverage.

FLORES-200 is a parallel corpus with English sentences from Wikimedia translated into 204 language-script pairs, verified by humans. It includes 997 dev, 1012 dev-test, and 992 test sentences. As the test set isn’t public, we use the dev-test set as our test corpus, following prior work. For faster computation, we consider only the first 100 sentences from each language. As shown in Section 3.2, this is sufficient to ensure MEXA’s robustness, as the odds of the MEXA score randomly achieving a high value with 100 sentences are very slim. This choice also enables scaling to more languages, many of which lack enough parallel samples.

The Parallel Bible (Mayer and Cysouw, 2014) covers a very large number of languages. From this resource, we managed to create a subcorpus, a super parallel dataset of the Bible, with 1,401 language-script labels, each containing 103 sentences (i.e., Bible verses).\footnote{hf.co/datasets/cis-lmu/sPBC} This corpus includes many low-resource languages, many of which are not covered by existing language technologies (Joshi et al., 2020), and MEXA can be adopted since only parallel data is needed. We use all 103 sentences from each language.

4.2 Models

For our experiments, we select models with around 7B parameters, which are considered a base size in the LLM community. The state-of-the-art open-weight models in this range, as measured by performance on English-based tasks such as MMLU (Stanford CRFM, 2024), include Llama 1, 2, 3, and 3.1 (Touvron et al., 2023a,b; Dubey et al., 2024), Gemma 1 and 2 (Gemma Team et al., 2024a,b), Mistral 0.3 (Jiang et al., 2023a), and the open-source model OLMo 1.7 (Groeneveld et al., 2024). We also select a larger model, Llama 3.1 70B, to show that our findings hold even when scaling up further. To apply MEXA, we need to access model weights to compute input sentence embeddings for each layer. We use three popular open-weight model families: Llama, Gemma, and Mistral. As a less multilingual version of state-of-the-art LLMs, we include OLMo, which is trained on a more English-centric corpus of Dolma (Soldaini et al., 2024).

4.3 Benchmarks

Among the existing evaluation benchmarks in Table 5 from Appendix A.2, we chose Belebele (Bandarkar et al., 2024), m-ARC (Lai et al., 2023b), and m-MMLU (Lai et al., 2023b), which support the highest number of high-, medium-, and low-resource languages and are directly related to natural understanding tasks, which is the primary focus of this paper.

We use the entire test set for each of these benchmarks (\S A.4 for details) for more details) to evaluate the models, except in one case. For Llama 3.1 70B, we use the first 500 questions of m-MMLU instead of the whole set due to resource constraints. Since the selected LLMs used in our experiment are not instruction-tuned, we use 5-shot in-context learning with the lm-evaluation-harness framework, employing log-likelihood-based multiple-choice scoring. Other settings, such as prompt templates, are configured according to the framework’s default (Gao et al., 2023; Biderman et al., 2024).
<table>
  <tr>
    <th></th>
    <th>Gemma 2<br>9B</th>
    <th>Gemma 1<br>7B</th>
    <th>Llama 3.1<br>70B</th>
    <th>Llama 3.1<br>8B</th>
    <th>Llama 3<br>8B</th>
    <th>Llama 2<br>7B</th>
    <th>Llama 1<br>7B</th>
    <th>Mistral 0.3<br>7B</th>
    <th>OLMo 1.7<br>7B</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Belebele</td>
    <td>0.9178</td>
    <td>0.8467</td>
    <td><b>0.9456</b></td>
    <td>0.8767</td>
    <td>0.8689</td>
    <td>0.4822</td>
    <td>0.4156</td>
    <td>0.8389</td>
    <td>0.7711</td>
    <td>0.7737</td>
  </tr>
  <tr>
    <td>Task<sub>[eng]</sub></td>
    <td>0.6998</td>
    <td>0.6138</td>
    <td><b>0.7700</b></td>
    <td>0.6315</td>
    <td>0.6294</td>
    <td>0.4523</td>
    <td>0.3569</td>
    <td>0.5988</td>
    <td>0.5210</td>
    <td>0.5859</td>
  </tr>
  <tr>
    <td>m-MMLU</td>
    <td>0.6725</td>
    <td>0.5870</td>
    <td><b>0.7014</b></td>
    <td>0.5794</td>
    <td>0.5836</td>
    <td>0.5128</td>
    <td>0.5000</td>
    <td>0.5862</td>
    <td>0.4872</td>
    <td>0.5795</td>
  </tr>
  <tr>
    <td>m-ARC</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Belebele</td>
    <td>0.7093</td>
    <td>0.5633</td>
    <td><b>0.7684</b></td>
    <td>0.5705</td>
    <td>0.5533</td>
    <td>0.3028</td>
    <td>0.2755</td>
    <td>0.4457</td>
    <td>0.3627</td>
    <td>0.5057</td>
  </tr>
  <tr>
    <td>Task<sub>L\,[eng]</sub></td>
    <td>0.5582</td>
    <td>0.4734</td>
    <td><b>0.6384</b></td>
    <td>0.4720</td>
    <td>0.4664</td>
    <td>0.3260</td>
    <td>0.2807</td>
    <td>0.4207</td>
    <td>0.3390</td>
    <td>0.4416</td>
  </tr>
  <tr>
    <td>m-MMLU</td>
    <td>0.4779</td>
    <td>0.4220</td>
    <td><b>0.5054</b></td>
    <td>0.3941</td>
    <td>0.3892</td>
    <td>0.3174</td>
    <td>0.2970</td>
    <td>0.3662</td>
    <td>0.2731</td>
    <td>0.3825</td>
  </tr>
  <tr>
    <td>m-ARC</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>FLORES</td>
    <td><b>0.5088</b></td>
    <td>0.3815</td>
    <td>0.4110</td>
    <td>0.3963</td>
    <td>0.3939</td>
    <td>0.0866</td>
    <td>0.1946</td>
    <td>0.2642</td>
    <td>0.0413</td>
    <td>0.2976</td>
  </tr>
  <tr>
    <td>\( \mu_{Max} \)</td>
    <td>0.7194</td>
    <td>0.5872</td>
    <td><b>0.7775</b></td>
    <td>0.6538</td>
    <td>0.6520</td>
    <td>0.2464</td>
    <td>0.3579</td>
    <td>0.4716</td>
    <td>0.1965</td>
    <td>0.5175</td>
  </tr>
  <tr>
    <td>Bible</td>
    <td><b>0.3568</b></td>
    <td>0.2152</td>
    <td>0.3169</td>
    <td>0.2103</td>
    <td>0.2026</td>
    <td>0.1246</td>
    <td>0.0908</td>
    <td>0.1198</td>
    <td>0.0121</td>
    <td>0.1832</td>
  </tr>
  <tr>
    <td>\( \mu_{Max} \)</td>
    <td>0.6076</td>
    <td>0.4021</td>
    <td><b>0.6599</b></td>
    <td>0.4212</td>
    <td>0.4190</td>
    <td>0.2724</td>
    <td>0.2357</td>
    <td>0.2606</td>
    <td>0.0319</td>
    <td>0.3678</td>
  </tr>
</table>

Table 1: \( \mu_{pooling} \) shows MEXA scores for each pooling strategy using token-weighted embeddings. Results cover English-only tasks, non-English tasks (Belebele: 116 languages, m-MMLU: 33, m-ARC: 31), and MEXA scores from FLORES (116) and Bible (101). Top scores are in bold, second-best are underlined.

4.4 Evaluation Measures

We calculate the Pearson correlation coefficient to assess the strength of the correlation between MEXA and downstream performance on our evaluation benchmarks. This coefficient is a statistical measure of the strength and direction of the linear relationship between two variables. A high value would indicate that MEXA provides a reliable assessment of multilingual capabilities in English-centric LLMs.

5 Results

Table 1 presents the downstream performance of the selected models across three benchmarks, along with MEXA scores from two parallel datasets. Notably, among models with parameter sizes ranging from 7 to 9 billion, both Gemma 2 and Llama 3.1 outperform the other LLMs in terms of non-English downstream performance and MEXA scores. The Llama 3.1 and Llama 3 models exhibit similar alignment and downstream task performance, and both represent substantial advancements compared to Llama 2. Moreover, results for the Llama 3.1-70B model indicate that scaling can significantly enhance alignment when compared to its smaller version. Interestingly, while Mistral achieves comparable results to Gemma 1 on English benchmarks, it demonstrates inferior alignment, which likely accounts for its reduced performance on non-English tasks. Furthermore, the Llama 2 model achieves higher MEXA scores than OLMo, indicating better alignment. However, due to Llama 2’s weaker performance on English tasks, it fails to transfer this alignment effectively, leading to comparable non-English performance between Llama 2 and OLMo. This observation is further explored in Section 5.2, where we normalize the expected performance based on the pivot language, namely English.

<table>
  <tr>
    <th></th>
    <th colspan="2">Avg. across models</th>
  </tr>
  <tr>
    <td>FLORES weighted average</td>
    <td>\( \mu_{Mean} \leftrightarrow \) Belebele</td>
    <td>0.8994</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) Belebele</td>
    <td>0.9098</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-MMLU</td>
    <td>0.9513</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-MMLU</td>
    <td>0.9188</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-ARC</td>
    <td>0.9393</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-ARC</td>
    <td>0.8856</td>
  </tr>
  <tr>
    <td>Bible weighted average</td>
    <td>\( \mu_{Mean} \leftrightarrow \) Belebele</td>
    <td>0.9168</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) Belebele</td>
    <td>0.9058</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-MMLU</td>
    <td>0.9545</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-MMLU</td>
    <td>0.9134</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-ARC</td>
    <td>0.9195</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-ARC</td>
    <td>0.8685</td>
  </tr>
  <tr>
    <td>last token</td>
    <td>\( \mu_{Mean} \leftrightarrow \) Belebele</td>
    <td>0.8496</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) Belebele</td>
    <td>0.8811</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-MMLU</td>
    <td>0.8823</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-MMLU</td>
    <td>0.8210</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-ARC</td>
    <td>0.9018</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-ARC</td>
    <td>0.8354</td>
  </tr>
  <tr>
    <td>last token</td>
    <td>\( \mu_{Mean} \leftrightarrow \) Belebele</td>
    <td>0.8147</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) Belebele</td>
    <td>0.8070</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-MMLU</td>
    <td>0.7572</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-MMLU</td>
    <td>0.6998</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Mean} \leftrightarrow \) m-ARC</td>
    <td>0.7469</td>
  </tr>
  <tr>
    <td></td>
    <td>\( \mu_{Max} \leftrightarrow \) m-ARC</td>
    <td>0.6885</td>
  </tr>
</table>

Table 2: Pearson correlation between MEXA scores and performance on Belebele, m-MMLU, and m-ARC, averaged across models. Results use two embedding aggregation methods: weighted average and last-token. Best scores per dataset and benchmark are in bold, second-best are underlined.

5.1 MEXA Correlation Analysis

We compute sentence embeddings for the selected models using two methods: weighted average based on token positions and last token (see §3.1). We apply mean and max pooling on the MEXA alignment scores across all model layers to derive a single score for each language. In Table 2 (refer to Table 6 for the detailed table), we report the correlation between the MEXA scores (computed using
both mean- and max-pooling, for the two embedding methods) and task performances. Across all settings, the best overall result (higher correlation) is achieved when embeddings are computed using the weighted average, with mean pooling as the pooling method. We adopt this configuration as the default setting for MEXA.

FLORES vs Bible. In the default setting, the average Pearson correlation coefficient for the FLORES parallel dataset across different tasks is 0.9300, and while for the Bible parallel dataset, it is 0.8779. The reason the Bible scores are generally lower than FLORES is that FLORES data is cleaner and more aligned with modern, standardized texts, whereas the Bible data is older and more specialized. For example, for some languages, the orthography of Bible texts no longer matches today’s orthography. In the Bible, Arabic often includes diacritics, which are typically omitted in modern writing and tasks, making the text less familiar to models trained on contemporary data. Additionally, although the Bible dataset has been made parallel, sentence alignment can still be inconsistent due to translation nuances. In contrast, FLORES is carefully curated to ensure high-quality, sentence-level parallelism across languages for machine translation tasks.

Weighted Average vs. Last Token Embeddings. The use of last token embeddings shows promisingly high correlations with the FLORES parallel data; however, for the Bible dataset, the correlation is low in some cases. We believe this may stem from the high occurrence of Bible sentences (especially in English), which leads models to memorize these phrases. Using the WIMBD toolkit (Elazar et al., 2024), we found that, on average, there are 92 times more documents in Dolma 1.7 containing exact Bible sentences than those in FLORES. Consequently, when using Bible examples, the last token is biased towards predicting the specific memorized next token rather than incorporating context-related signals. Therefore, one should consider the hazard of memorized data when using last token embeddings. The weighted-average method, which takes into account the influence of multiple tokens, can mitigate the impact of a poor embedding for the last token and enable the model to capture useful information from the other tokens more robustly.

Max Pooling vs. Mean Pooling. In our comparison of mean pooling and max pooling on the Belebele benchmark, we found that mean pooling

![Relationship between Llama 3.1-8B MEXA scores from Bible and FLORES, adjusted by English task performance, on Belebele and m-ARC. Each point represents one language.](page_682_153_563_377.png)

Figure 2: Relationship between Llama 3.1-8B MEXA scores from Bible and FLORES, adjusted by English task performance, on Belebele and m-ARC. Each point represents one language.

underestimates low-resource languages (resulting in more MEXA scores near 0), while max pooling correlates better with the Belebele benchmark. This can be explained by the fact that Belebele is an easier task among the three evaluated, allowing even low-resource languages to achieve good scores. Conversely, based on our experiment with m-ARC, max pooling tends to overestimate low-resource languages, making mean pooling more aligned with m-ARC. This can be attributed to m-ARC being the most challenging task among the three, where even medium-resource languages do not achieve high scores. Changing the pooling method from mean to max can be considered when dealing with different levels of understanding.

5.2 Downstream Performance Estimation

A full score on Pearson correlation (i.e., \( \rho = 1.0 \)) indicates that a linear equation perfectly describes the relationship between MEXA and the evaluation benchmarks, with all data points lying on a line. Given the high correlation values shown in Table 6, it is reasonable to conclude that we can fit a line that closely approximates this linear relationship. This line converts the MEXA scores back to downstream task performances. We employed a linear model to predict this line by minimizing the residual sum of squares between the MEXA scores (multiplied by the performance on the English task) and the task performances. We needed to adjust the MEXA scores for this purpose, as the
MEXA score for language \( L_1 \) indicates how well \( L_1 \) is aligned with English but does not reflect the estimated task performance of the model for language \( L_1 \). Of course, this does not change the value of the correlation coefficient, as it is unaffected by linear transformations. The three tasks considered in this paper involve multiple-choice questions with four possible answers for each question, resulting in a chance of being randomly correct of \( \frac{1}{4} \). However, the minimum score for MEXA scores is 0. Thus, the ideal slope for the line would be \( \frac{3}{4} \) with an intercept of \( \frac{1}{4} \) (X-axis: adjusted MEXA scores, Y-axis: task performance). In Figure 2, we plot this relationship for Llama 3.1-8B using the Bible and FLORES parallel datasets for Belebele and m-ARC. We chose max pooling for Belebele and mean pooling for m-ARC, since these pooling methods yield a stronger correlation (see §5.1). The pairs of (slope, intercept) from left to right in the Figure 2 are: (0.6804, 0.2477), (0.6103, 0.1838), (0.6340, 0.3408), and (0.5726, 0.2423). With data points from both high-resource and low-resource languages, this line can be calculated; otherwise, the ideal line may be used as a reference.

Language Coverage. We present the adjusted MEXA score for all languages available in FLORES-200 in Table 7 from Appendix A.6 for the selected models. The languages are categorized into groups ranging from well-covered to not covered. In Table 7, we can clearly see that Llama 3.1-70B and Gemma 2-9B show a higher level of multilinguality than other models.

5.3 MEXA vs Absolute Cosine Similarity

We compare MEXA with the use of absolute cosine similarities. We used parallel data from FLORES and downstream task data from the Belebele benchmark, focusing on 116 common labels. For each non-English language, we computed the average absolute cosine similarity for parallel sentences with English, and for non-parallel sentences with English. Following the setup by Li et al. (2025), which uses absolute cosine similarity values to predict language performance and ranking, we computed sentence embeddings using the last-token method and applied mean pooling over layers {5, 10, 15, 20, 25}.

To evaluate the correlation of each method with downstream task performance, we report results using the Gemma 1 and Llama 1 7B models. Table 3 summarizes these results. For both models, MEXA consistently achieves a higher correlation with downstream performance compared to the absolute cosine similarity of parallel sentences. Moreover, the correlation between cosine similarity for parallel and non-parallel sentences is notably high, suggesting that absolute values may be less discriminative across sentence types. This discrepancy highlights a limitation of using absolute cosine similarity: for some languages, similarity scores may remain high even for non-parallel sentences. Conversely, a low overall similarity score does not necessarily imply weak alignment, as parallel sentence scores may still significantly exceed non-parallel ones. In contrast, MEXA offers a more robust and comparative measure across languages.

<table>
  <tr>
    <th></th>
    <th>Gemma 1 7B</th>
    <th>Llama 1 7B</th>
  </tr>
  <tr>
    <td>MEXA ↔ Belebele</td>
    <td>0.9260</td>
    <td>0.8365</td>
  </tr>
  <tr>
    <td>AC-P ↔ Belebele</td>
    <td>0.7651</td>
    <td>0.6473</td>
  </tr>
  <tr>
    <td>AC-P ↔ AC-NP</td>
    <td>0.9232</td>
    <td>0.9064</td>
  </tr>
</table>

Table 3: Pearson correlations between alignment metrics and Belebele performance. AC-P denotes absolute cosine similarity of parallel pairs; AC-NP, of non-parallel pairs.

5.4 Visualization of Layers

In Figure 3, we show the results of applying MEXA to 20 pairs of language_script from the FLORES parallel dataset for Llama 1-7B and Llama 3.1-8B across all 32 layers. We selected these languages from different families, writing systems, and both high- and low-resource categories. The embeddings are computed using a weighted average based on token positions. Figure 3 shows that high-resource languages (with more prevalence on the web; see §A.1) achieve higher alignment scores across different layers, while low-resource languages achieve lower scores. In the initial layers, embeddings are more in-language, resulting in lower alignment scores. As embeddings progress to the mid-layers, they become more aligned with the dominant language of the LLM, i.e., English.

MEXA is comparable between models as long as the same parallel dataset and setting is used to obtain the MEXA scores. Figure 3 shows that in many languages, particularly high-resource languages, Llama 3.1 achieves a significantly higher alignment score than its predecessor, Llama 1. Although Llama 3.1 leads to better alignment scores with English for medium and low-resource languages, there is still room for improvement. Comparing Arabic (arb_Arab) with its romanized ver-
Figure 3: Llama 1 vs. Llama 3.1 MEXA alignment score for different languages across all layers. Best performance markers in order: \( \triangle, \square, *, x, o, _ \)

Figure 4: Llama 3.1 t-SNE plots for 3 different layers. As shown, in the mid-layers, the embeddings become more language-neutral. The numbers shown in the mid-layers are the IDs of English sentences that are scattered.

sion (arb_Latn), we see that both Llama 1 and Llama 3.1 models perform better in the native script than in the Latin script, even though Llama 1’s tokenizer for Arabic is essentially a character-based tokenizer. In general, for very low-resource languages, those in Latin script tend to have higher alignment scores, likely because the tokenization is more favorable for Latin characters.

In Figure 4, we display the t-SNE (Van der Maaten and Hinton, 2008) plots of the embeddings of Figure 3 from 3 different layers of Llama 3.1: embedding layer 0, mid-layer 13, and last layer 32. We assign a different color to each language. For layers 0 and 32, the embeddings are more language-specific, while in the mid-layer, they become more language-neutral. Languages that maintain their language-specific embeddings in the mid-layer are clustered separately and, notably, correspond to the very low-resource languages that receive the lowest alignment scores from MEXA.

6 Conclusion

We introduce MEXA, a method for assessing the multilingual capabilities of English-centric large language models (LLMs). MEXA builds on the observation that English-centric LLMs semantically use English as a kind of pivot language in their intermediate layers. MEXA computes the alignment between non-English languages and English using parallel sentences, estimating the transfer of language understanding capabilities from English to other languages through this alignment. This metric can be useful in estimating task performance, provided we know the English performance in the task and the alignment score between languages derived from a parallel dataset. Through different studies with two parallel datasets (FLORES-200 and the Bible), different LLMs including the Llama family, Gemma family, Mistral, and OLMo, and three downstream tasks (Belebele, m-MMLU, and m-ARC), we demonstrated that MEXA provides a reliable estimation of multilingual performance. For MEXA score calculations, multiple design analyses are conducted to explore the impact of token-level pooling for embeddings and layer-level pooling in computing alignment scores. While MEXA shows high correlation across most configurations, a weighted average of tokens combined with mean pooling delivers the best results. The results reveal a promising average value for the Pearson correlation coefficient of 0.90 with established downstream tasks across nine models and two parallel datasets. Overall, MEXA proves to be a valuable method for practitioners aiming to assess the multilingual capabilities of English-centric LLMs, easing future efforts to expand these models to a wider range of underrepresented languages.
Limitations

We are aware of four main limitations of our work.

First, the scope of this paper is limited to non-generative tasks. Generation is generally more challenging than understanding, and it is unsurprising that these models for many languages may struggle to generate content in their language. While NLP has advanced toward generative capabilities, a significant portion of evaluation still focuses on non-generative tasks (e.g., Hugging Face Leaderboards)2 due to their convenience in multiple-choice question evaluation and standardized metrics. Assessing generated output remains challenging, even in English benchmarks. For example, model-based approaches (e.g., “LLM-as-a-judge” (Zheng et al., 2023)) require an LLM fully competent in the target language—a capability that is both questionable and the focus of our evaluation.

Second, MEXA provides a method of evaluation for open science, and only model weights are needed. Although, developers of closed-source models could use MEXA under the hood and report their multilingual results to provide insight of their model’s multilingual capabilities. We target the widely used settings where the LLM follows a decoder-only transformer architecture. For other architectures, as long as we can extract the embedding given a sentence for intermediate layers, MEXA can be calculated.

Third, we present a selection of tasks for multilingual evaluation in Table 5. As shown, for non-generative tasks, only a few benchmarks support a high number of languages, including low-resource ones. Benchmarks limited to around 10 languages, which mostly support high-resource languages, would not support our claims, as MEXA would achieve high results for all of them. Belebele includes the highest number of languages (except sequence labeling tasks), making it an ideal task to evaluate MEXA. Both m-MMLU and m-ARC are the next highest covered languages for non-generative tasks. However, since they are machine-translated tasks, they are not ideal and may bias some results for low-resource languages (or, more accurately, when the machine translation is poor). Yet, these translated versions are representative of the current state of automatic evaluation, as seen in multilingual leaderboards.3

Fourth, MEXA provides a rough estimate of the multilingual capabilities of pre-trained English-centric LLMs. Different tasks offer diverse perspectives on the abilities of LLMs, and MEXA cannot replace all of them. Our goal is to highlight the multilingual potential of English-centric LLMs and propose a simple way to evaluate them. We hope this encourages the development of more multilingual LLMs, even though they are likely to contain large shares of English data. Additionally, it is important to note that answers across languages do not always need to be fully aligned (Naous et al., 2024), and for such cases, language- and culture-specific evaluation benchmarks should be developed.

Acknowledgments

This work was funded by Deutsche Forschungsgemeinschaft (project SCHU 2246/14-1). François Yvon has been partly funded by the French National Funding Agency (ANR) under the France 2030 program (ref. ANR-23-IACL-0007).

References

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy BuzaaBa, Blessing Sibanda, Godson Kalipe, Jonathan Mukibi, Salomon Kabongo, Foutse Yuehgho, Mmasibidi Setaka, Lolwethu Ndolela, and 7 others. 2024. Irokobench: A new benchmark for African languages in the age of large language models. Preprint, arXiv:2406.03368.

Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022. IndicXNLI: Evaluating multilingual inference for Indian languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10994–11006. Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics.

Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng,
Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2024. MEGAVERSE: Benchmarking large language models across languages, modalities, models and tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2598–2637, Mexico City, Mexico. Association for Computational Linguistics.

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4623–4637.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. Preprint, arXiv:2308.16884.

Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanaka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, and 11 others. 2024. Lessons from the trenches on reproducible evaluation of language models. Preprint, arXiv:2405.14782.

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, and 1 others. 2023. BLOOM: A 176b-parameter open-access multilingual language model. Preprint, arXiv:2211.05100.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020a. GPT-3 dataset language statistics. https://github.com/openai/gpt-3/tree/master/dataset_statistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020b. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In International Conference on Learning Representations.

Grzegorz Chrupala and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2952–2962, Florence, Italy. Association for Computational Linguistics.

Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. Preprint, arXiv:1803.05457.

Common Crawl. 2024. Statistics of common crawl monthly archives. https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.

Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32, pages 7059–7069. Curran Associates, Inc.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.

Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6022–6034, Online. Association for Computational Linguistics.

Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12402–12426, Toronto, Canada. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and I others. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783.

Pavel Efimov, Leonid Boytsov, Elena Arslanova, and Pavel Braslavski. 2023. The impact of cross-lingual adjustment of contextual word representations on zero-shot transfer. In Advances in Information Retrieval, pages 51–67, Cham. Springer Nature Switzerland.

Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. What’s in my big data? In The Twelfth International Conference on Learning Representations.

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.

Christian Federmann, Tom Koemli, and Ying Xin. 2022. NTREX-128 – news test references for MT evaluation of 128 languages. In Proceedings of the First Workshop on Scaling Up Multilingual Evaluation, pages 21–24, Online. Association for Computational Linguistics.

Jay Gala, Pranjal A Chitale, AK Raghavan, Varun Gumma, Sumanth Doddapaneni, Janki Atul Nawale, Anupama Sujatha, Ratish Pudupully, Vivek Raghavan, Pratyush Kumar, and I others. 2023. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. Transactions on Machine Learning Research.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2023. A framework for few-shot language model evaluation.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Riviere, Mihir Sanjay Kale, Juliette Love, and I others. 2024a. Gemma: Open models based on Gemini research and technology. arXiv preprint arXiv:2403.08295.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Husseinot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and I others. 2024b. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.

Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. Khayyam challenge (PersianMMLU): Is your LLM truly wise to the Persian language? Preprint, arXiv:2404.06644.

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and I others. 2024. OLMo: Accelerating the science of language models. arXiv preprint arXiv:2402.00838.

Katharina Hämmerl, Alina Fastowski, Jindřich Libovický, and Alexander Fraser. 2023. Exploring anisotropy and outliers in multilingual language models for cross-lingual semantic sentence similarity. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7023–7037, Toronto, Canada. Association for Computational Linguistics.

Katharina Hämmerl, Jindřich Libovický, and Alexander Fraser. 2024. Understanding cross-lingual Alignment—A survey. In Findings of the Association for Computational Linguistics ACL 2024, pages 10922–10943, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubashir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-Sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR.

Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2485–2494, Hong Kong, China. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and I others. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825.

Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2023b. Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.

Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics.

Amir Hossein Kargaran, François Yvon, and Hinrich Schütze. 2024. GlotScript: A resource and tool for low resource writing system identification. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7774–7784, Torino, Italia. ELRA and ICCL.

Tannon Kew, Florian Schottmann, and Rico Sennrich. 2024. Turning English-centric LLMs into polyglots: How much multilinguality is needed? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13097–13124, Miami, Florida, USA. Association for Computational Linguistics.

Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519–3529. PMLR.

Fajri Koto, Haanon Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in arabic. Preprint, arXiv:2402.12840.

Saurabh Kulshreshtha, Jose Luis Redondo Garcia, and Ching-Yun Chang. 2020. Cross-lingual alignment methods for multilingual BERT: A comparative study. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 933–942, Online. Association for Computational Linguistics.

Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034–4048, Online. Association for Computational Linguistics.

Viet Lai, Nghia Ngo, Amir Pouran Ben Veysheh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023a. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics.

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023b. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, and I others. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809–31826.

Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315–7330.

Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024a. Improving in-context learning of
multilingual generative language models with cross-lingual alignment. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8058–8076, Mexico City, Mexico. Association for Computational Linguistics.

Daoyang Li, Mingyu Jin, Qingcheng Zeng, Haiyan Zhao, and Mengnan Du. 2024b. Exploring multilingual probing in large language models: A cross-language analysis. Preprint, arXiv:2409.14459.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2024c. CMMLU: Measuring massive multi-task language understanding in Chinese. Preprint, arXiv:2306.09212.

Jiahuan Li, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2024d. PreAlign: Boosting cross-lingual transfer by early establishment of multilingual alignment. Preprint, arXiv:2407.16222.

Xianming Li and Jing Li. 2024. AoE: Angle-optimized embeddings for semantic textual similarity. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1825–1839, Bangkok, Thailand. Association for Computational Linguistics.

Xiaochen Li, Zheng-Xin Yong, and Stephen H Bach. 2024e. Preference tuning for toxicity mitigation generalizes across languages. arXiv preprint arXiv:2406.16235.

Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. 2025. Language ranker: A metric for quantifying llm performance across high and low-resource languages. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28186–28194.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Sinig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, and 2 others. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Pierre Lison and Jörg Tiedemann. 2016. OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).

Weihao Liu, Ning Wu, Wenbiao Ding, Shining Liang, Ming Gong, and Dongmei Zhang. 2024a. Towards truthful multilingual large language models: Benchmarking and alignment strategies. Preprint, arXiv:2406.14434.

Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayooob Imani, Orgest Xhelli, Haotian Ye, Chunlan Ma, François Yvon, and Hinrich Schütze. 2024b. How transliterations improve crosslingual alignment. Preprint, arXiv:2409.17326.

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2421–2425.

Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet Üstün, Sara Hooker, and Sebastian Ruder. 2024. How does quantization affect multilingual LLMs? Preprint, arXiv:2407.03211.

Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel Bible corpus. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3158–3163, Reykjavík, Iceland. European Language Resources Association (ELRA).

Niklas Muenninghoff. 2022. SGPT: GPT sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904.

Niklas Muenninghoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bar, Sheng Shen, Zheng Xin Yong, Haley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.

Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamel Seddah. 2021. First align, then predict: Understanding the cross-lingual ability of multilingual BERT. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2214–2231, Online. Association for Computational Linguistics.

Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming
Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, and 6 others. 2022. Text and code embeddings by contrastive pre-training. Preprint, arXiv:2201.10005.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Hef-fernand, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejía Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

Odunayo Ogundepo, Tajudeen Gwadabe, Clara Rivera, Jonathan Clark, Sebastian Ruder, David Adelani, Bonaventure Dossou, Abdou Diop, Claytone Sika-sote, Gilles Hacheme, Happy Buzaaab, Ignatius Ezeani, Roowethier Mabuya, Salomey Osei, Chris Emezie, Albert Kahira, Shamsuddeen Muhammad, Akintunde Oladipo, Abraham Owodunni, and 25 others. 2023. Cross-lingual open-retrieval question answering for African languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14957–14972, Singapore. Association for Computational Linguistics.

OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt.

OpenAI. 2024. Multilingual massive multitask language understanding (MMMLU).

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayam Anadkat, and 1 others. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958.

Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common-sense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association for Computational Linguistics.

Sara Rajaei and Mohammad Taher Pilehvar. 2022. An isotropy analysis in the multilingual BERT embedding space. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1309–1316, Dublin, Ireland. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.

Anton Schäfer, Shauli Ravfogel, Thomas Hofmann, Tiago Pimentel, and Imanol Schlag. 2024. The role of language imbalance in cross-lingual generalisation: Insights from cloned language experiments. Preprint, arXiv:2404.07982.

Shivalika Singh, Freddie Vargas, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, and 14 others. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. Preprint, arXiv:2402.06619.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Author, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others. 2024. Dolma: an open corpus of three trillion tokens for language model pretraining research. Preprint, arXiv:2402.00159.

Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024. MMMLU: Measuring massive multitask language understanding in Korean. Preprint, arXiv:2402.11548.

Stanford CRFM. 2024. Holistic evaluation of language models - MMLU leaderboard. https://crfm.stanford.edu/helm/mmlu/latest/#leaderboard.

Tatoeba Community. 2006. Tatoeba collection. https://tatoeba.org/en/downloads.

Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. 2024. Exploring design choices for building language-specific LLMs. arXiv preprint arXiv:2406.14670.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwel Bhargava, Shruti Bhosale, and 1 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).

W3Techs. 2024. Usage statistics of content languages for websites. https://w3techs.com/technologies/overview/content_language

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11897–11916, Bangkok, Thailand. Association for Computational Linguistics.

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do Llamas work in English? on the latent language of multilingual transformers. Preprint, arXiv:2402.10588.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3687–3692, Hong Kong, China. Association for Computational Linguistics.

Jiacheng Ye, Xijia Tao, and Lingpeng Kong. 2023. Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability. Preprint, arXiv:2306.06688.

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, and Hinrich Schütze. 2024. Turkish-MMLU: Measuring massive multitask language understanding in Turkish. Preprint, arXiv:2407.12402.

Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1628–1639, Online. Association for Computational Linguistics.

Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023a. M3Exam: A multilingual, multimodal, multilevel benchmark for examining large language models. In Advances in Neural Information Processing Systems, volume 36, pages 5484–5505. Curran Associates, Inc.

Zhen-Ru Zhang, Chuanqi Tan, Songfang Huang, and Fei Huang. 2023b. VECO 2.0: Cross-lingual language model pre-training with multi-granularity contrastive learning. Preprint, arXiv:2304.08205.

Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and I others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623.

Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, and Sadao Kurohashi. 2024. Beyond english-centric LLMs: What language do multilingual language models think in? Preprint, arXiv:2408.10811.

Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024. Question translation training for better multilingual reasoning. Preprint, arXiv:2401.07817.

A Appendix

A.1 Distribution of Pre-training Data in LLMs

The distribution of languages in the training data of state-of-the-art LLMs is rarely fully documented. Llama 2 (Touvron et al., 2023b) is a counterexample and its authors have disclosed the language distribution use in pre-training. Their analysis uses the FastText (Bojanowski et al., 2017) language identification tool and a threshold of 0.5 for the language detection. We reproduce Touvron et al. (2023b, Table 10), which lists 27 languages with percentages greater than 0.005% in the Llama 2 pre-training data, in Table 4. English, with 89.70%, constitutes the vast majority of the training data.

All the languages listed in Table 4 have a presence of more than 0.10% (top 35 languages) on the web according to the W3Techs report (W3Techs, 2024) or more than 0.15% (top 36 languages) according to CommonCrawl (first three snapshots of 2024) (Common Crawl, 2024). However, not all of the most prevalent languages on the web appear in Table 4. The following 9 languages are missing, most of which use non-Latin writing systems: Turkish (tur_Latn), Persian (pes_Arab), Arabic (ara_Arab), Greek (ell_Grek), Hebrew (heb_Hebr), Thai (tha_Thai), Hindi (hin_Deva), Slovak (slk_Latn), and Lithuanian (lit_Latn).

The distribution of data in the training of English-centric LLMs is not the same as on the web, but it does have some correlation. The amount of English in LLM pre-training data is significantly larger than for other languages. This is also observable for GPT-3 (Brown et al., 2020b), where more than 92% of the training texts was in English (Brown et al., 2020a). The rest of the top languages in the data of such models are mostly high-resource languages,
<table>
  <tr>
    <th>Language</th>
    <th>Script</th>
    <th>Percent</th>
    <th>Language</th>
    <th>Script</th>
    <th>Percent</th>
  </tr>
  <tr>
    <td>English (eng)</td>
    <td>Latin</td>
    <td>89.70%</td>
    <td>Ukrainian (ukr)</td>
    <td>Cyril</td>
    <td>0.07%</td>
  </tr>
  <tr>
    <td>Unknown (unk)</td>
    <td></td>
    <td>8.38%</td>
    <td>Korean (kor)</td>
    <td>Hang</td>
    <td>0.06%</td>
  </tr>
  <tr>
    <td>German (deu)</td>
    <td>Latin</td>
    <td>0.17%</td>
    <td>Latin</td>
    <td>0.15%</td>
    <td></td>
  </tr>
  <tr>
    <td>French (fra)</td>
    <td>Latin</td>
    <td>0.16%</td>
    <td>Serbian (srp)</td>
    <td>Cyril/Latin</td>
    <td>0.04%</td>
  </tr>
  <tr>
    <td>Swedish (swe)</td>
    <td>Latin</td>
    <td>0.15%</td>
    <td>Indonesian (ind)</td>
    <td>Latin</td>
    <td>0.03%</td>
  </tr>
  <tr>
    <td>Chinese (zho)</td>
    <td>Hans/Hant</td>
    <td>0.13%</td>
    <td>Czech (ces)</td>
    <td>Latin</td>
    <td>0.03%</td>
  </tr>
  <tr>
    <td>Russian (rus)</td>
    <td>Cyril</td>
    <td>0.13%</td>
    <td>Hungarian (hun)</td>
    <td>Latin</td>
    <td>0.03%</td>
  </tr>
  <tr>
    <td>Dutch (nld)</td>
    <td>Latin</td>
    <td>0.12%</td>
    <td>Norwegian (nor)</td>
    <td>Latin</td>
    <td>0.03%</td>
  </tr>
  <tr>
    <td>Italian (ita)</td>
    <td>Latin</td>
    <td>0.11%</td>
    <td>Polish (pol)</td>
    <td>Latin</td>
    <td>0.03%</td>
  </tr>
  <tr>
    <td>Japanese (jpn)</td>
    <td>Japan</td>
    <td>0.10%</td>
    <td>Bulgarian (bul)</td>
    <td>Cyril</td>
    <td>0.02%</td>
  </tr>
  <tr>
    <td>Polish (pol)</td>
    <td>Latin</td>
    <td>0.09%</td>
    <td>Danish (dan)</td>
    <td>Latin</td>
    <td>0.02%</td>
  </tr>
  <tr>
    <td>Portuguese (por)</td>
    <td>Latin</td>
    <td>0.09%</td>
    <td>Slovenian (slv)</td>
    <td>Latin</td>
    <td>0.01%</td>
  </tr>
  <tr>
    <td>Vietnamese (vie)</td>
    <td>Latin</td>
    <td>0.08%</td>
    <td>Croatian (hrv)</td>
    <td>Latin</td>
    <td>0.01%</td>
  </tr>
</table>

Table 4: Language distribution in the pre-training data for Llama 2. The large “Unknown” category is partially composed of programming code data. Common scripts are sourced from the GlotScript resource (Kargaran et al., 2024).

which have the most available data on the web (top 36 languages). However, in some models, this could be adjusted by design, for example, to make writing systems with non-Latin languages less prominent (as seen in Llama 2). This weakens the correlation between LLMs’ pre-training data and the web.

A.2 Multilingual Evaluation Benchmarks

Multilingual evaluation methods and the development of benchmarks not only facilitate the assessment of diverse language representations in LLMs but also help in monitoring cross-lingual generalization, to assess the effect of quantization across multiple languages (Marchisio et al., 2024), the development of language-specific models (Tejaswi et al., 2024), and the optimization of safety preferences (Li et al., 2024e), among others. In Table 5, we list benchmarks with the largest language coverage. This list includes benchmarks referenced by MEGA (Ahuja et al., 2023), MEGAVERSE (Ahuja et al., 2024), xP3 (Muenighoff et al., 2023), the Aya collection (Singh et al., 2024), the Im-evaluation-harness framework (Gao et al., 2023; Biderman et al., 2024), and inter alia. These datasets comprise a mix of translated datasets, some human-translated or verified by native speakers such as AfriXNLI (Adelani et al., 2024) and some relying only on machine translation Lai et al. (2023b). Additionally, there are datasets created independently for each language, such as XLSum (Hasan et al., 2021), where the data is not parallel and the size of the data varies between languages. Despite the efforts reflected in Table 5, the community is still lacking highly multilingual benchmarks for tasks such as natural language understanding or text generation.

<table>
  <tr>
    <th>Dataset</th>
    <th>Task</th>
    <th># L</th>
  </tr>
  <tr>
    <td>XNL1 (Conneau et al., 2018)</td>
    <td>Natural Language Inference</td>
    <td>15</td>
  </tr>
  <tr>
    <td>IndicXNLI (Aggarwal et al., 2022)</td>
    <td>Natural Language Inference</td>
    <td>11</td>
  </tr>
  <tr>
    <td>AfriXNLI (Adelani et al., 2024)</td>
    <td>Natural Language Inference</td>
    <td>15</td>
  </tr>
  <tr>
    <td>m_HANS (Hansard et al., 2023b)</td>
    <td>Natural Language Inference</td>
    <td>31</td>
  </tr>
  <tr>
    <td>PAWS-X (Yang et al., 2019)</td>
    <td>Paraphrase Identification</td>
    <td>7</td>
  </tr>
  <tr>
    <td>XCOPA (Ponti et al., 2020)</td>
    <td>Commonsense Reasoning</td>
    <td>11</td>
  </tr>
  <tr>
    <td>XShore (Liu et al., 2023b)</td>
    <td>Commonsense Reasoning</td>
    <td>11</td>
  </tr>
  <tr>
    <td>m-ARC (Lai et al., 2023b)</td>
    <td>Commonsense Reasoning</td>
    <td>31</td>
  </tr>
  <tr>
    <td>TyDiQA (Clark et al., 2020)</td>
    <td>Question Answering</td>
    <td>11</td>
  </tr>
  <tr>
    <td>MLOA (Lewis et al., 2020)</td>
    <td>Question Answering</td>
    <td>7</td>
  </tr>
  <tr>
    <td>XQuAD (Liu et al., 2020)</td>
    <td>Question Answering</td>
    <td>11</td>
  </tr>
  <tr>
    <td>IndicQA (Doddapaneni et al., 2023)</td>
    <td>Question Answering</td>
    <td>10</td>
  </tr>
  <tr>
    <td>AfriQA (Ogundepo et al., 2023)</td>
    <td>Question Answering</td>
    <td>10</td>
  </tr>
  <tr>
    <td>m-TFQA (Lai et al., 2023b)</td>
    <td>MC General Question Answering</td>
    <td>31</td>
  </tr>
  <tr>
    <td>UDPOS (Zhang et al., 2021)</td>
    <td>Part-of-Speech Tagging</td>
    <td>104</td>
  </tr>
  <tr>
    <td>WikiaNN (Pan et al., 2021)</td>
    <td>Name Entity Recognition</td>
    <td>282</td>
  </tr>
  <tr>
    <td>XLSum (Hasan et al., 2021)</td>
    <td>Summarization</td>
    <td>44</td>
  </tr>
  <tr>
    <td>WikiQA (Zhang et al., 2020b)</td>
    <td>Summarization</td>
    <td>18</td>
  </tr>
  <tr>
    <td>Belebele (Bandarkar et al., 2024)</td>
    <td>MC Reading Comprehension</td>
    <td>115</td>
  </tr>
  <tr>
    <td>AfriMLU (Adelani et al., 2024)</td>
    <td>MC Knowledge Question Answering</td>
    <td>17</td>
  </tr>
  <tr>
    <td>m-MMLU (Lai et al., 2023b)</td>
    <td>MC Knowledge Question Answering</td>
    <td>31</td>
  </tr>
  <tr>
    <td>M3EExam (OpenAI, 2023a)</td>
    <td>MC Multimedia Question Answering</td>
    <td>15</td>
  </tr>
  <tr>
    <td>M3EExam (Zhang et al., 2023a)</td>
    <td>MC Multimedia Question Answering</td>
    <td>9</td>
  </tr>
</table>

Table 5: Multilingual evaluation benchmarks: MC stands for multiple-choice. # L shows the number of languages supported by each dataset.

A.3 Semantic Similarity in Multilingual Embeddings

There are other ways to compute similarity between languages, such as Representational Similarity Analysis (RSA) (Chrupala and Alishahi, 2019) and Central Kernel Alignment (CKA) (Kornblith et al., 2019). RSA involves first computing the cosine similarity for sentence embeddings within each language, then correlating these in-language similarities with those in other languages. CKA, another metric, is adopted by Conneau et al. (2020b) and Muller et al. (2021). Conneau et al. (2020b) show that the CKA similarity is highly correlated with sentence retrieval scores for four languages. In this paper, our focus is not on finding different ways to calculate similarity between languages, but on how helpful a properly defined alignment score can be in estimating the multilingual capabilities of LLMs across multiple languages.

A.4 Benchmark Details

Belebele is a multiple-choice reading comprehension task designed to assess language models across a range of high-, medium-, and low-resource languages. Each question in the dataset is paired with four possible answers and linked to a brief passage from the FLORES-200 dataset (NLLB Team et al., 2022). The human annotation process was carefully curated to generate questions that effectively differentiate between various levels of language comprehension, supported by rigorous quality checks. Belebele supports 122 distinct labels (language-script combinations) corresponding to 115 distinct languages. However, FLORES-200 does not sup-
port 5 of these labels, corresponding to Romanized versions of 5 Indic languages. Therefore, we conducted our analysis between the FLORES-200 and Belebele benchmarks on 117 common labels. Additionally, there are 102 common labels between the Bible parallel data and the Belebele benchmark.

Both ARC Challenge (Clark et al., 2018) and MMLU (Hendrycks et al., 2021) are also set up as multiple-choice question-answering tasks, but they focus on different types of knowledge and reasoning skills. ARC Challenge is classified as a common-sense reasoning task, consisting of grade-school level science questions, while MMLU consists of questions across a wide range of fields, including the humanities, social sciences, and more. Lai et al. (2023b) used GPT-3.5-turbo (OpenAI, 2022) and a translation prompt to translate examples from both datasets and create m-ARC and m-MMLU in 31 languages (excluding English). Later, m-MMLU was expanded to also include Icelandic (isl_Latn) and Norwegian (nob_Latn). The Icelandic portion was translated using the Mideind.is, while Norwegian was generated with DeepL.com.4 m-MMLU consists of 277 questions in its training set, 13,258 in the test set and 1,433 in the validation set. m-ARC consists of 1,116 questions in the training set, 1,169 in the test set, and 298 in the validation set.

A.5 Detailed Results

We show the detailed per model results of Table 2 in Table 6.

A.6 MEXA for FLORES-200

We compute MEXA with weighted average embedding and max pooling for the FLORES parallel data for 203 language labels, multiplied by the performance of Belebele for each model in English. We show the results in Table 7, and color the cells based on 0.2 intervals from green (well-covered) to red (not covered): (1.0-0.8), (0.8-0.6), (0.6-0.4), (0.4-0.2), (0.2-0). Note that although FLORES is a high-quality, human-translated dataset, we addressed two major issues before proceeding, as noted by Kargaran et al. (2023). First, the data labeled as Cantonese (Yue Chinese) is not actually Cantonese, so we removed it. Second, the code for Central Atlas Tamazight (tzm), which actually refers to Standard Moroccan Tamazight (zgh), was renamed accordingly. As Belebele is relatively an easy task since the models get good scores in English, and we are using max pooling, this gives a high estimate of the coverage the LLMs have. If the score for a language is not very high, it likely indicates that for more challenging tasks, it will remain low. In Table 7, we can clearly see that Llama 3.1-70B and Gemma 2-9B show a higher level of multilinguality than other models.

4hf.co/datasets/alexandrainst/m_mmlu
<table>
  <tr>
    <th rowspan="2">FLORES weighted average</th>
    <th colspan="2">Gemma 2</th>
    <th colspan="2">Gemma 1</th>
    <th colspan="2">Llama 3.1</th>
    <th colspan="2">Llama 3.1</th>
    <th colspan="2">Llama 3</th>
    <th colspan="2">Llama 2</th>
    <th colspan="2">Llama 1</th>
    <th>Mistral 0.3</th>
    <th>OLMs 1.7</th>
    <th>AVG</th>
  </tr>
  <tr>
    <th>9B</th><th>7B</th><th>9B</th><th>7B</th><th>8B</th><th>8B</th><th>8B</th><th>8B</th><th>7B</th><th>7B</th><th>7B</th><th>7B</th><th>7B</th><th>7B</th><th></th><th></th><th></th>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.9247</td><td>0.9421</td><td>0.8291</td><td>0.9478</td><td>0.9588</td><td>0.8364</td><td>0.8404</td><td>0.9732</td><td>0.8425</td><td>0.8994</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.9623</td><td>0.9676</td><td>0.9211</td><td>0.9392</td><td>0.9326</td><td>0.8362</td><td>0.7649</td><td>0.9448</td><td>0.9198</td><td>0.9998</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.9342</td><td>0.9697</td><td>0.9362</td><td>0.9689</td><td>0.9647</td><td>0.9223</td><td>0.9406</td><td>0.9857</td><td>0.9393</td><td>0.97513</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.9060</td><td>0.9596</td><td>0.8946</td><td>0.9003</td><td>0.8892</td><td>0.9386</td><td>0.8936</td><td>0.9311</td><td>0.9565</td><td>0.9188</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.9741</td><td>0.9706</td><td>0.9374</td><td>0.9515</td><td>0.9562</td><td>0.9052</td><td>0.9268</td><td>0.9693</td><td>0.8630</td><td>0.9393</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.9187</td><td>0.9499</td><td>0.8736</td><td>0.8582</td><td>0.8663</td><td>0.9297</td><td>0.8439</td><td>0.9001</td><td>0.8298</td><td>0.8856</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.8997</td><td>0.9326</td><td>0.8491</td><td>0.9494</td><td>0.9581</td><td>0.9141</td><td>0.8340</td><td>0.9679</td><td>0.9467</td><td>0.9168</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.9225</td><td>0.9309</td><td>0.9127</td><td>0.9244</td><td>0.9123</td><td>0.9125</td><td>0.7693</td><td>0.9460</td><td>0.9218</td><td>0.9058</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.9086</td><td>0.9637</td><td>0.9370</td><td>0.9687</td><td>0.9690</td><td>0.9771</td><td>0.9301</td><td>0.9659</td><td>0.9700</td><td>0.9545</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.8448</td><td>0.9297</td><td>0.8645</td><td>0.9224</td><td>0.9177</td><td>0.9699</td><td>0.8902</td><td>0.9161</td><td>0.9649</td><td>0.9134</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.9190</td><td>0.9541</td><td>0.9524</td><td>0.9356</td><td>0.9617</td><td>0.9390</td><td>0.9146</td><td>0.9451</td><td>0.7356</td><td>0.9195</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.8569</td><td>0.9447</td><td>0.9005</td><td>0.8944</td><td>0.8879</td><td>0.9464</td><td>0.8263</td><td>0.8859</td><td>0.7057</td><td>0.7685</td>
  </tr>
  <tr>
    <th>Bible weighted average</th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th colspan="2"></th>
    <th></th>
    <th></th>
    <th></th>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.8360</td><td>0.8530</td><td>0.7909</td><td>0.8781</td><td>0.8974</td><td>0.8982</td><td>0.8404</td><td>0.9118</td><td>0.7410</td><td>0.8496</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.8863</td><td>0.9001</td><td>0.8851</td><td>0.9242</td><td>0.9302</td><td>0.8926</td><td>0.8230</td><td>0.9337</td><td>0.7549</td><td>0.8811</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.8051</td><td>0.8886</td><td>0.8956</td><td>0.9096</td><td>0.8964</td><td>0.9252</td><td>0.9159</td><td>0.9093</td><td>0.7944</td><td>0.8823</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.8591</td><td>0.8813</td><td>0.7848</td><td>0.8604</td><td>0.8614</td><td>0.9088</td><td>0.9171</td><td>0.9171</td><td>0.8210</td><td>0.8820</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.8505</td><td>0.8998</td><td>0.9188</td><td>0.9267</td><td>0.9116</td><td>0.8940</td><td>0.9208</td><td>0.9317</td><td>0.8623</td><td>0.9018</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.6070</td><td>0.8803</td><td>0.8030</td><td>0.8769</td><td>0.8552</td><td>0.8684</td><td>0.8879</td><td>0.9178</td><td>0.8220</td><td>0.8354</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.7656</td><td>0.8005</td><td>0.5944</td><td>0.7934</td><td>0.8396</td><td>0.9046</td><td>0.8299</td><td>0.9177</td><td>0.8866</td><td>0.8147</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{Belebele}) \)</td>
    <td>0.7844</td><td>0.8299</td><td>0.5264</td><td>0.8000</td><td>0.8100</td><td>0.9047</td><td>0.8048</td><td>0.9235</td><td>0.8796</td><td>0.8070</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.7194</td><td>0.7646</td><td>0.6472</td><td>0.6068</td><td>0.6516</td><td>0.8827</td><td>0.8692</td><td>0.8672</td><td>0.8060</td><td>0.7572</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-MMLU}) \)</td>
    <td>0.7075</td><td>0.6866</td><td>0.5037**</td><td>0.5228**</td><td>0.4461**</td><td>0.9079</td><td>0.8576</td><td>0.8643</td><td>0.7994</td><td>0.6998</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.7411</td><td>0.7754</td><td>0.6592</td><td>0.5976</td><td>0.6494</td><td>0.8537</td><td>0.8537</td><td>0.8927</td><td>0.6997</td><td>0.7469</td>
  </tr>
  <tr>
    <td>\( \rho(\mu_{\text{MEXA}}, \text{m-ARC}) \)</td>
    <td>0.7293</td><td>0.7000</td><td>0.5190**</td><td>0.5335**</td><td>0.4853**</td><td>0.8494</td><td>0.8309</td><td>0.8624</td><td>0.6867</td><td>0.6885</td>
  </tr>
</table>

Table 6: Pearson correlation of MEXA using FLORES and Bible data across three tasks. \( \rho (\mu_{\text{Pooling}}, \text{Task}) \) is the correlation of MEXA for the corresponding pooling strategy and benchmark. In all settings except ***, the p-value is \( p < 0.001 \). The best average correlations for each task are in bold, and the second bests are underlined.
<table>
  <tr>
    <th></th>
    <th>Gemma 2<br>9B</th>
    <th>Gemma 1<br>7B</th>
    <th>Llama 3.1<br>70B</th>
    <th>Llama 3.1<br>8B</th>
    <th>Llama 3<br>8B</th>
    <th>Llama 2<br>7B</th>
    <th>Llama 1<br>7B</th>
    <th>Mistral<br>7B</th>
    <th>OLMo<br>7B</th>
    <th>AVG</th>
  </tr>
  <tr>
    <td>eng_Latn</td>
    <td>0.92</td>
    <td>0.85</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.48</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.77</td>
    <td>0.77</td>
  </tr>
  <tr>
    <td>fra_Latn</td>
    <td>0.92</td>
    <td>0.84</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.37</td>
    <td>0.41</td>
    <td>0.84</td>
    <td>0.70</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>por_Latn</td>
    <td>0.92</td>
    <td>0.84</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.41</td>
    <td>0.41</td>
    <td>0.84</td>
    <td>0.63</td>
    <td>0.75</td>
  </tr>
  <tr>
    <td>deu_Latn</td>
    <td>0.92</td>
    <td>0.84</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.35</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.65</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>spa_Latn</td>
    <td>0.92</td>
    <td>0.83</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.37</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.56</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>ita_Latn</td>
    <td>0.92</td>
    <td>0.83</td>
    <td>0.92</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.35</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.56</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>cat_Latn</td>
    <td>0.92</td>
    <td>0.82</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.39</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.50</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>nld_Latn</td>
    <td>0.92</td>
    <td>0.82</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.34</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.52</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>rus_Cyrl</td>
    <td>0.91</td>
    <td>0.82</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.34</td>
    <td>0.41</td>
    <td>0.83</td>
    <td>0.51</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>zho_Hans</td>
    <td>0.91</td>
    <td>0.80</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.32</td>
    <td>0.34</td>
    <td>0.81</td>
    <td>0.62</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>glg_Latn</td>
    <td>0.92</td>
    <td>0.83</td>
    <td>0.91</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.31</td>
    <td>0.41</td>
    <td>0.82</td>
    <td>0.52</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>swe_Latn</td>
    <td>0.92</td>
    <td>0.83</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.38</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.37</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>dan_Latn</td>
    <td>0.92</td>
    <td>0.83</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.31</td>
    <td>0.41</td>
    <td>0.82</td>
    <td>0.44</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>ces_Latn</td>
    <td>0.92</td>
    <td>0.82</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.26</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.43</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>ron_Latn</td>
    <td>0.92</td>
    <td>0.82</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.23</td>
    <td>0.41</td>
    <td>0.83</td>
    <td>0.48</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>nob_Latn</td>
    <td>0.91</td>
    <td>0.82</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.34</td>
    <td>0.39</td>
    <td>0.81</td>
    <td>0.39</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>zho_Hant</td>
    <td>0.91</td>
    <td>0.81</td>
    <td>0.94</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.31</td>
    <td>0.32</td>
    <td>0.79</td>
    <td>0.52</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>pol_Latn</td>
    <td>0.92</td>
    <td>0.81</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.22</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.38</td>
    <td>0.70</td>
  </tr>
  <tr>
    <td>ast_Latn</td>
    <td>0.90</td>
    <td>0.80</td>
    <td>0.91</td>
    <td>0.88</td>
    <td>0.86</td>
    <td>0.21</td>
    <td>0.40</td>
    <td>0.77</td>
    <td>0.49</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>ind_Latn</td>
    <td>0.92</td>
    <td>0.83</td>
    <td>0.93</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.22</td>
    <td>0.30</td>
    <td>0.82</td>
    <td>0.42</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>oci_Latn</td>
    <td>0.89</td>
    <td>0.75</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.22</td>
    <td>0.39</td>
    <td>0.81</td>
    <td>0.40</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>bos_Latn</td>
    <td>0.91</td>
    <td>0.81</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.19</td>
    <td>0.41</td>
    <td>0.84</td>
    <td>0.25</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>nno_Latn</td>
    <td>0.92</td>
    <td>0.82</td>
    <td>0.92</td>
    <td>0.84</td>
    <td>0.84</td>
    <td>0.26</td>
    <td>0.36</td>
    <td>0.78</td>
    <td>0.38</td>
    <td>0.68</td>
  </tr>
  <tr>
    <td>ukr_Cyrl</td>
    <td>0.92</td>
    <td>0.81</td>
    <td>0.95</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.22</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.15</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>zsm_Latn</td>
    <td>0.92</td>
    <td>0.83</td>
    <td>0.93</td>
    <td>0.88</td>
    <td>0.87</td>
    <td>0.17</td>
    <td>0.25</td>
    <td>0.81</td>
    <td>0.36</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>hrv_Latn</td>
    <td>0.91</td>
    <td>0.81</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.18</td>
    <td>0.41</td>
    <td>0.83</td>
    <td>0.23</td>
    <td>0.67</td>
  </tr>
  <tr>
    <td>sly_Latn</td>
    <td>0.91</td>
    <td>0.79</td>
    <td>0.93</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.20</td>
    <td>0.40</td>
    <td>0.84</td>
    <td>0.19</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>afr_Latn</td>
    <td>0.91</td>
    <td>0.81</td>
    <td>0.93</td>
    <td>0.87</td>
    <td>0.87</td>
    <td>0.20</td>
    <td>0.37</td>
    <td>0.79</td>
    <td>0.21</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>slk_Latn</td>
    <td>0.91</td>
    <td>0.80</td>
    <td>0.93</td>
    <td>0.86</td>
    <td>0.85</td>
    <td>0.12</td>
    <td>0.38</td>
    <td>0.82</td>
    <td>0.25</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>bul_Cyrl</td>
    <td>0.91</td>
    <td>0.80</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.12</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.14</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>jpn_Jpan</td>
    <td>0.90</td>
    <td>0.80</td>
    <td>0.93</td>
    <td>0.83</td>
    <td>0.82</td>
    <td>0.29</td>
    <td>0.25</td>
    <td>0.76</td>
    <td>0.24</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>hun_Latn</td>
    <td>0.91</td>
    <td>0.78</td>
    <td>0.92</td>
    <td>0.84</td>
    <td>0.83</td>
    <td>0.13</td>
    <td>0.39</td>
    <td>0.81</td>
    <td>0.18</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>vec_Latn</td>
    <td>0.87</td>
    <td>0.74</td>
    <td>0.93</td>
    <td>0.84</td>
    <td>0.83</td>
    <td>0.16</td>
    <td>0.35</td>
    <td>0.76</td>
    <td>0.28</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>srp_Cyrl</td>
    <td>0.91</td>
    <td>0.79</td>
    <td>0.90</td>
    <td>0.86</td>
    <td>0.86</td>
    <td>0.10</td>
    <td>0.42</td>
    <td>0.84</td>
    <td>0.06</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>tgl_Latn</td>
    <td>0.91</td>
    <td>0.74</td>
    <td>0.94</td>
    <td>0.82</td>
    <td>0.82</td>
    <td>0.16</td>
    <td>0.20</td>
    <td>0.77</td>
    <td>0.36</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>fin_Latn</td>
    <td>0.91</td>
    <td>0.79</td>
    <td>0.90</td>
    <td>0.85</td>
    <td>0.85</td>
    <td>0.14</td>
    <td>0.21</td>
    <td>0.74</td>
    <td>0.33</td>
    <td>0.64</td>
  </tr>
</table>

Continued on next page
<table>
  <tr>
    <th></th>
    <th>Gemma 2<br>9B</th>
    <th>Gemma 1<br>7B</th>
    <th>Llama 3.1<br>70B</th>
    <th>Llama 3.1<br>8B</th>
    <th>Llama 3<br>8B</th>
    <th>Llama 3<br>7B</th>
    <th>Llama 1<br>7B</th>
    <th>Mistral<br>7B</th>
    <th>OLMo<br>7B</th>
    <th>Avg</th>
  </tr>
  <tr><td>mkd_Cyril</td><td>0.90</td><td>0.77</td><td>0.94</td><td>0.87</td><td>0.86</td><td>0.07</td><td>0.38</td><td>0.80</td><td>0.11</td><td>0.63</td></tr>
  <tr><td>vie_Latn</td><td>0.91</td><td>0.81</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.22</td><td>0.08</td><td>0.79</td><td>0.16</td><td>0.63</td></tr>
  <tr><td>epo_Latn</td><td>0.87</td><td>0.76</td><td>0.95</td><td>0.86</td><td>0.85</td><td>0.14</td><td>0.26</td><td>0.67</td><td>0.11</td><td>0.61</td></tr>
  <tr><td>kor_Hang</td><td>0.88</td><td>0.74</td><td>0.91</td><td>0.84</td><td>0.83</td><td>0.22</td><td>0.15</td><td>0.71</td><td>0.15</td><td>0.60</td></tr>
  <tr><td>arb_Arab</td><td>0.91</td><td>0.80</td><td>0.94</td><td>0.86</td><td>0.85</td><td>0.05</td><td>0.17</td><td>0.70</td><td>0.10</td><td>0.60</td></tr>
  <tr><td>ars_Arab</td><td>0.91</td><td>0.80</td><td>0.93</td><td>0.86</td><td>0.85</td><td>0.04</td><td>0.17</td><td>0.69</td><td>0.08</td><td>0.59</td></tr>
  <tr><td>lim_Latn</td><td>0.76</td><td>0.65</td><td>0.89</td><td>0.83</td><td>0.83</td><td>0.21</td><td>0.25</td><td>0.59</td><td>0.21</td><td>0.58</td></tr>
  <tr><td>acq_Arab</td><td>0.91</td><td>0.78</td><td>0.92</td><td>0.83</td><td>0.82</td><td>0.04</td><td>0.13</td><td>0.67</td><td>0.09</td><td>0.58</td></tr>
  <tr><td>acm_Arab</td><td>0.90</td><td>0.76</td><td>0.90</td><td>0.86</td><td>0.82</td><td>0.04</td><td>0.14</td><td>0.67</td><td>0.09</td><td>0.57</td></tr>
  <tr><td>fur_Latn</td><td>0.73</td><td>0.60</td><td>0.91</td><td>0.81</td><td>0.77</td><td>0.16</td><td>0.27</td><td>0.59</td><td>0.26</td><td>0.57</td></tr>
  <tr><td>pes_Arab</td><td>0.91</td><td>0.79</td><td>0.88</td><td>0.85</td><td>0.85</td><td>0.05</td><td>0.08</td><td>0.59</td><td>0.07</td><td>0.56</td></tr>
  <tr><td>arz_Arab</td><td>0.88</td><td>0.74</td><td>0.90</td><td>0.84</td><td>0.83</td><td>0.03</td><td>0.10</td><td>0.63</td><td>0.09</td><td>0.56</td></tr>
  <tr><td>ajp_Arab</td><td>0.88</td><td>0.76</td><td>0.86</td><td>0.85</td><td>0.83</td><td>0.03</td><td>0.12</td><td>0.60</td><td>0.09</td><td>0.56</td></tr>
  <tr><td>lit_Latn</td><td>0.90</td><td>0.76</td><td>0.92</td><td>0.78</td><td>0.80</td><td>0.10</td><td>0.10</td><td>0.56</td><td>0.11</td><td>0.56</td></tr>
  <tr><td>apc_Arab</td><td>0.89</td><td>0.76</td><td>0.86</td><td>0.82</td><td>0.83</td><td>0.03</td><td>0.11</td><td>0.64</td><td>0.09</td><td>0.56</td></tr>
  <tr><td>ell_Grek</td><td>0.90</td><td>0.78</td><td>0.87</td><td>0.87</td><td>0.86</td><td>0.02</td><td>0.09</td><td>0.58</td><td>0.05</td><td>0.56</td></tr>
  <tr><td>tur_Latn</td><td>0.89</td><td>0.78</td><td>0.90</td><td>0.82</td><td>0.81</td><td>0.04</td><td>0.09</td><td>0.61</td><td>0.04</td><td>0.55</td></tr>
  <tr><td>est_Latn</td><td>0.90</td><td>0.77</td><td>0.90</td><td>0.82</td><td>0.83</td><td>0.12</td><td>0.09</td><td>0.45</td><td>0.10</td><td>0.55</td></tr>
  <tr><td>pap_Latn</td><td>0.79</td><td>0.60</td><td>0.89</td><td>0.75</td><td>0.73</td><td>0.18</td><td>0.22</td><td>0.56</td><td>0.23</td><td>0.55</td></tr>
  <tr><td>lmo_Latn</td><td>0.73</td><td>0.56</td><td>0.87</td><td>0.75</td><td>0.74</td><td>0.17</td><td>0.26</td><td>0.60</td><td>0.26</td><td>0.55</td></tr>
  <tr><td>szl_Latn</td><td>0.77</td><td>0.59</td><td>0.87</td><td>0.73</td><td>0.74</td><td>0.11</td><td>0.26</td><td>0.64</td><td>0.21</td><td>0.55</td></tr>
  <tr><td>prs_Arab</td><td>0.90</td><td>0.78</td><td>0.92</td><td>0.84</td><td>0.84</td><td>0.01</td><td>0.06</td><td>0.46</td><td>0.08</td><td>0.54</td></tr>
  <tr><td>scn_Latn</td><td>0.77</td><td>0.59</td><td>0.88</td><td>0.79</td><td>0.77</td><td>0.15</td><td>0.22</td><td>0.57</td><td>0.15</td><td>0.54</td></tr>
  <tr><td>heb_Hebr</td><td>0.91</td><td>0.81</td><td>0.89</td><td>0.83</td><td>0.83</td><td>0.02</td><td>0.05</td><td>0.47</td><td>0.06</td><td>0.54</td></tr>
  <tr><td>lv_s_Latn</td><td>0.90</td><td>0.75</td><td>0.90</td><td>0.81</td><td>0.79</td><td>0.05</td><td>0.05</td><td>0.55</td><td>0.08</td><td>0.54</td></tr>
  <tr><td>als_Latn</td><td>0.87</td><td>0.67</td><td>0.93</td><td>0.79</td><td>0.80</td><td>0.09</td><td>0.08</td><td>0.53</td><td>0.10</td><td>0.54</td></tr>
  <tr><td>lij_Latn</td><td>0.74</td><td>0.58</td><td>0.88</td><td>0.72</td><td>0.70</td><td>0.16</td><td>0.25</td><td>0.53</td><td>0.30</td><td>0.54</td></tr>
  <tr><td>ceb_Latn</td><td>0.83</td><td>0.59</td><td>0.89</td><td>0.73</td><td>0.72</td><td>0.16</td><td>0.15</td><td>0.49</td><td>0.24</td><td>0.53</td></tr>
  <tr><td>srd_Latn</td><td>0.73</td><td>0.59</td><td>0.86</td><td>0.75</td><td>0.72</td><td>0.16</td><td>0.23</td><td>0.55</td><td>0.21</td><td>0.53</td></tr>
  <tr><td>hin_Deva</td><td>0.90</td><td>0.74</td><td>0.91</td><td>0.80</td><td>0.79</td><td>0.03</td><td>0.05</td><td>0.44</td><td>0.06</td><td>0.53</td></tr>
  <tr><td>ltz_Latn</td><td>0.79</td><td>0.59</td><td>0.84</td><td>0.75</td><td>0.74</td><td>0.15</td><td>0.18</td><td>0.44</td><td>0.18</td><td>0.52</td></tr>
  <tr><td>tha_Thai</td><td>0.90</td><td>0.76</td><td>0.87</td><td>0.83</td><td>0.83</td><td>0.02</td><td>0.02</td><td>0.32</td><td>0.10</td><td>0.52</td></tr>
  <tr><td>aeb_Arab</td><td>0.82</td><td>0.67</td><td>0.86</td><td>0.78</td><td>0.75</td><td>0.04</td><td>0.10</td><td>0.55</td><td>0.08</td><td>0.52</td></tr>
  <tr><td>bel_Cyril</td><td>0.88</td><td>0.65</td><td>0.88</td><td>0.79</td><td>0.79</td><td>0.02</td><td>0.09</td><td>0.50</td><td>0.02</td><td>0.51</td></tr>
  <tr><td>isl_Latn</td><td>0.83</td><td>0.62</td><td>0.88</td><td>0.77</td><td>0.78</td><td>0.09</td><td>0.10</td><td>0.48</td><td>0.06</td><td>0.51</td></tr>
  <tr><td>swh_Latn</td><td>0.90</td><td>0.74</td><td>0.86</td><td>0.73</td><td>0.80</td><td>0.11</td><td>0.09</td><td>0.27</td><td>0.08</td><td>0.51</td></tr>
  <tr><td>mlt_Latn</td><td>0.88</td><td>0.63</td><td>0.87</td><td>0.74</td><td>0.74</td><td>0.12</td><td>0.11</td><td>0.38</td><td>0.12</td><td>0.51</td></tr>
  <tr><td>war_Latn</td><td>0.76</td><td>0.55</td><td>0.88</td><td>0.65</td><td>0.61</td><td>0.26</td><td>0.20</td><td>0.35</td><td>0.20</td><td>0.49</td></tr>
  <tr><td>cym_Latn</td><td>0.87</td><td>0.59</td><td>0.88</td><td>0.75</td><td>0.76</td><td>0.11</td><td>0.10</td><td>0.28</td><td>0.08</td><td>0.49</td></tr>
  <tr><td>fao_Latn</td><td>0.71</td><td>0.53</td><td>0.86</td><td>0.71</td><td>0.69</td><td>0.12</td><td>0.13</td><td>0.53</td><td>0.08</td><td>0.48</td></tr>
  <tr><td>urd_Arab</td><td>0.83</td><td>0.66</td><td>0.88</td><td>0.76</td><td>0.73</td><td>0.02</td><td>0.02</td><td>0.31</td><td>0.03</td><td>0.47</td></tr>
  <tr><td>jav_Latn</td><td>0.75</td><td>0.54</td><td>0.84</td><td>0.69</td><td>0.67</td><td>0.16</td><td>0.12</td><td>0.29</td><td>0.16</td><td>0.47</td></tr>
  <tr><td>eus_Latn</td><td>0.82</td><td>0.66</td><td>0.84</td><td>0.74</td><td>0.71</td><td>0.10</td><td>0.08</td><td>0.18</td><td>0.05</td><td>0.47</td></tr>
  <tr><td>sun_Latn</td><td>0.69</td><td>0.48</td><td>0.86</td><td>0.68</td><td>0.64</td><td>0.14</td><td>0.12</td><td>0.34</td><td>0.23</td><td>0.47</td></tr>
  <tr><td>kea_Latn</td><td>0.64</td><td>0.51</td><td>0.78</td><td>0.60</td><td>0.64</td><td>0.16</td><td>0.19</td><td>0.45</td><td>0.18</td><td>0.46</td></tr>
  <tr><td>ary_Arab</td><td>0.71</td><td>0.60</td><td>0.80</td><td>0.68</td><td>0.68</td><td>0.03</td><td>0.09</td><td>0.44</td><td>0.10</td><td>0.46</td></tr>
  <tr><td>hat_Latn</td><td>0.74</td><td>0.47</td><td>0.86</td><td>0.65</td><td>0.61</td><td>0.15</td><td>0.12</td><td>0.36</td><td>0.09</td><td>0.45</td></tr>
  <tr><td>mag_Deva</td><td>0.75</td><td>0.52</td><td>0.88</td><td>0.75</td><td>0.70</td><td>0.02</td><td>0.05</td><td>0.29</td><td>0.07</td><td>0.45</td></tr>
  <tr><td>min_Latn</td><td>0.56</td><td>0.45</td><td>0.81</td><td>0.68</td><td>0.68</td><td>0.16</td><td>0.12</td><td>0.34</td><td>0.17</td><td>0.44</td></tr>
  <tr><td>ban_Latn</td><td>0.48</td><td>0.32</td><td>0.82</td><td>0.62</td><td>0.63</td><td>0.17</td><td>0.15</td><td>0.42</td><td>0.30</td><td>0.43</td></tr>
  <tr><td>bjn_Latn</td><td>0.60</td><td>0.51</td><td>0.78</td><td>0.62</td><td>0.60</td><td>0.14</td><td>0.12</td><td>0.31</td><td>0.21</td><td>0.43</td></tr>
  <tr><td>azj_Latn</td><td>0.75</td><td>0.53</td><td>0.86</td><td>0.68</td><td>0.66</td><td>0.01</td><td>0.03</td><td>0.29</td><td>0.02</td><td>0.43</td></tr>
  <tr><td>npi_Deva</td><td>0.81</td><td>0.59</td><td>0.82</td><td>0.62</td><td>0.64</td><td>0.02</td><td>0.02</td><td>0.20</td><td>0.05</td><td>0.42</td></tr>
  <tr><td>mar_Deva</td><td>0.82</td><td>0.58</td><td>0.87</td><td>0.68</td><td>0.63</td><td>0.02</td><td>0.01</td><td>0.12</td><td>0.03</td><td>0.42</td></tr>
  <tr><td>awa_Deva</td><td>0.73</td><td>0.53</td><td>0.83</td><td>0.65</td><td>0.65</td><td>0.02</td><td>0.05</td><td>0.21</td><td>0.08</td><td>0.42</td></tr>
  <tr><td>ben_Beng</td><td>0.82</td><td>0.58</td><td>0.82</td><td>0.60</td><td>0.60</td><td>0.01</td><td>0.01</td><td>0.23</td><td>0.05</td><td>0.41</td></tr>
  <tr><td>uzn_Latn</td><td>0.70</td><td>0.47</td><td>0.84</td><td>0.60</td><td>0.62</td><td>0.04</td><td>0.05</td><td>0.23</td><td>0.05</td><td>0.40</td></tr>
  <tr><td>bho_Deva</td><td>0.51</td><td>0.46</td><td>0.89</td><td>0.68</td><td>0.65</td><td>0.02</td><td>0.03</td><td>0.26</td><td>0.08</td><td>0.40</td></tr>
  <tr><td>gle_Latn</td><td>0.68</td><td>0.31</td><td>0.82</td><td>0.64</td><td>0.64</td><td>0.09</td><td>0.08</td><td>0.22</td><td>0.09</td><td>0.40</td></tr>
  <tr><td>hye_Armn</td><td>0.85</td><td>0.59</td><td>0.79</td><td>0.60</td><td>0.58</td><td>0.01</td><td>0.02</td><td>0.10</td><td>0.01</td><td>0.40</td></tr>
  <tr><td>hne_Deva</td><td>0.67</td><td>0.44</td><td>0.80</td><td>0.65</td><td>0.63</td><td>0.01</td><td>0.02</td><td>0.24</td><td>0.08</td><td>0.40</td></tr>
  <tr><td>kaz_Cyril</td><td>0.62</td><td>0.47</td><td>0.87</td><td>0.61</td><td>0.60</td><td>0.02</td><td>0.06</td><td>0.27</td><td>0.03</td><td>0.39</td></tr>
  <tr><td>tpi_Latn</td><td>0.69</td><td>0.38</td><td>0.69</td><td>0.46</td><td>0.45</td><td>0.19</td><td>0.15</td><td>0.33</td><td>0.15</td><td>0.39</td></tr>
  <tr><td>hau_Latn</td><td>0.68</td><td>0.41</td><td>0.77</td><td>0.58</td><td>0.54</td><td>0.12</td><td>0.11</td><td>0.17</td><td>0.06</td><td>0.38</td></tr>
  <tr><td>mai_Deva</td><td>0.61</td><td>0.43</td><td>0.86</td><td>0.61</td><td>0.59</td><td>0.02</td><td>0.03</td><td>0.16</td><td>0.08</td><td>0.38</td></tr>
</table>
<table>
  <tr>
    <th></th>
    <th>Gemma 2<br>9B</th>
    <th>Gemma 1<br>7B</th>
    <th>Llama 3.1<br>70B</th>
    <th>Llama 3.1<br>8B</th>
    <th>Llama 3<br>8B</th>
    <th>Llama 3<br>7B</th>
    <th>Mistral<br>7B</th>
    <th>Mistral<br>7B</th>
    <th>OLMo</th>
    <th>AVG</th>
  </tr>
  <tr><td>crh_Latn</td><td>0.58</td><td>0.42</td><td>0.77</td><td>0.56</td><td>0.51</td><td>0.03</td><td>0.07</td><td>0.35</td><td>0.05</td><td>0.37</td></tr>
  <tr><td>ilo_Latn</td><td>0.61</td><td>0.32</td><td>0.74</td><td>0.47</td><td>0.46</td><td>0.17</td><td>0.15</td><td>0.24</td><td>0.13</td><td>0.37</td></tr>
  <tr><td>tat_Cyrl</td><td>0.56</td><td>0.38</td><td>0.83</td><td>0.58</td><td>0.55</td><td>0.03</td><td>0.04</td><td>0.21</td><td>0.02</td><td>0.36</td></tr>
  <tr><td>kat_Geor</td><td>0.73</td><td>0.45</td><td>0.72</td><td>0.53</td><td>0.50</td><td>0.01</td><td>0.04</td><td>0.18</td><td>0.01</td><td>0.35</td></tr>
  <tr><td>ydd_Hebr</td><td>0.74</td><td>0.45</td><td>0.78</td><td>0.48</td><td>0.47</td><td>0.03</td><td>0.02</td><td>0.05</td><td>0.02</td><td>0.34</td></tr>
  <tr><td>kir_Cyrl</td><td>0.53</td><td>0.33</td><td>0.81</td><td>0.58</td><td>0.57</td><td>0.01</td><td>0.04</td><td>0.13</td><td>0.02</td><td>0.34</td></tr>
  <tr><td>pag_Latn</td><td>0.33</td><td>0.23</td><td>0.63</td><td>0.35</td><td>0.39</td><td>0.25</td><td>0.21</td><td>0.36</td><td>0.24</td><td>0.33</td></tr>
  <tr><td>pan_Guru</td><td>0.78</td><td>0.50</td><td>0.75</td><td>0.47</td><td>0.40</td><td>0.01</td><td>0.00</td><td>0.03</td><td>0.05</td><td>0.33</td></tr>
  <tr><td>bak_Cyrl</td><td>0.56</td><td>0.36</td><td>0.82</td><td>0.51</td><td>0.51</td><td>0.01</td><td>0.04</td><td>0.13</td><td>0.02</td><td>0.33</td></tr>
  <tr><td>guj_Gujr</td><td>0.79</td><td>0.52</td><td>0.67</td><td>0.42</td><td>0.39</td><td>0.02</td><td>0.01</td><td>0.04</td><td>0.07</td><td>0.33</td></tr>
  <tr><td>tam_Taml</td><td>0.78</td><td>0.54</td><td>0.72</td><td>0.38</td><td>0.38</td><td>0.02</td><td>0.01</td><td>0.08</td><td>0.03</td><td>0.33</td></tr>
  <tr><td>pbt_Arab</td><td>0.50</td><td>0.23</td><td>0.82</td><td>0.57</td><td>0.57</td><td>0.02</td><td>0.03</td><td>0.10</td><td>0.07</td><td>0.32</td></tr>
  <tr><td>tgk_Cyrl</td><td>0.62</td><td>0.27</td><td>0.78</td><td>0.51</td><td>0.52</td><td>0.02</td><td>0.03</td><td>0.10</td><td>0.05</td><td>0.32</td></tr>
  <tr><td>tel_Telu</td><td>0.77</td><td>0.52</td><td>0.67</td><td>0.38</td><td>0.43</td><td>0.01</td><td>0.01</td><td>0.05</td><td>0.04</td><td>0.32</td></tr>
  <tr><td>snd_Arab</td><td>0.59</td><td>0.30</td><td>0.78</td><td>0.53</td><td>0.50</td><td>0.02</td><td>0.01</td><td>0.06</td><td>0.04</td><td>0.31</td></tr>
  <tr><td>kan_Knda</td><td>0.74</td><td>0.47</td><td>0.66</td><td>0.42</td><td>0.41</td><td>0.01</td><td>0.01</td><td>0.06</td><td>0.05</td><td>0.31</td></tr>
  <tr><td>mal_Mlym</td><td>0.76</td><td>0.50</td><td>0.68</td><td>0.32</td><td>0.30</td><td>0.01</td><td>0.01</td><td>0.03</td><td>0.03</td><td>0.29</td></tr>
  <tr><td>ckb_Arab</td><td>0.51</td><td>0.20</td><td>0.78</td><td>0.54</td><td>0.50</td><td>0.01</td><td>0.01</td><td>0.05</td><td>0.02</td><td>0.29</td></tr>
  <tr><td>gla_Latn</td><td>0.46</td><td>0.14</td><td>0.71</td><td>0.46</td><td>0.45</td><td>0.08</td><td>0.07</td><td>0.13</td><td>0.07</td><td>0.29</td></tr>
  <tr><td>asm_Beng</td><td>0.63</td><td>0.35</td><td>0.70</td><td>0.39</td><td>0.36</td><td>0.00</td><td>0.01</td><td>0.08</td><td>0.04</td><td>0.29</td></tr>
  <tr><td>tuk_Latn</td><td>0.49</td><td>0.31</td><td>0.63</td><td>0.43</td><td>0.41</td><td>0.05</td><td>0.04</td><td>0.14</td><td>0.02</td><td>0.28</td></tr>
  <tr><td>san_Deva</td><td>0.48</td><td>0.26</td><td>0.71</td><td>0.46</td><td>0.45</td><td>0.00</td><td>0.01</td><td>0.12</td><td>0.02</td><td>0.28</td></tr>
  <tr><td>kmr_Latn</td><td>0.38</td><td>0.15</td><td>0.69</td><td>0.48</td><td>0.50</td><td>0.05</td><td>0.06</td><td>0.13</td><td>0.05</td><td>0.28</td></tr>
  <tr><td>lus_Latn</td><td>0.53</td><td>0.09</td><td>0.56</td><td>0.34</td><td>0.33</td><td>0.14</td><td>0.10</td><td>0.24</td><td>0.11</td><td>0.27</td></tr>
  <tr><td>khk_Cyrl</td><td>0.44</td><td>0.18</td><td>0.73</td><td>0.42</td><td>0.43</td><td>0.01</td><td>0.02</td><td>0.08</td><td>0.03</td><td>0.26</td></tr>
  <tr><td>ltg_Latn</td><td>0.31</td><td>0.23</td><td>0.61</td><td>0.38</td><td>0.35</td><td>0.08</td><td>0.06</td><td>0.22</td><td>0.06</td><td>0.26</td></tr>
  <tr><td>azb_Arab</td><td>0.37</td><td>0.28</td><td>0.60</td><td>0.44</td><td>0.44</td><td>0.00</td><td>0.01</td><td>0.11</td><td>0.02</td><td>0.25</td></tr>
  <tr><td>plt_Latn</td><td>0.52</td><td>0.17</td><td>0.59</td><td>0.25</td><td>0.25</td><td>0.14</td><td>0.12</td><td>0.18</td><td>0.05</td><td>0.25</td></tr>
  <tr><td>ibo_Latn</td><td>0.35</td><td>0.15</td><td>0.64</td><td>0.38</td><td>0.37</td><td>0.09</td><td>0.08</td><td>0.12</td><td>0.07</td><td>0.25</td></tr>
  <tr><td>mri_Latn</td><td>0.35</td><td>0.11</td><td>0.60</td><td>0.38</td><td>0.35</td><td>0.12</td><td>0.10</td><td>0.18</td><td>0.07</td><td>0.25</td></tr>
  <tr><td>som_Latn</td><td>0.42</td><td>0.14</td><td>0.60</td><td>0.24</td><td>0.24</td><td>0.11</td><td>0.08</td><td>0.18</td><td>0.09</td><td>0.23</td></tr>
  <tr><td>ace_Latn</td><td>0.22</td><td>0.13</td><td>0.49</td><td>0.32</td><td>0.31</td><td>0.14</td><td>0.10</td><td>0.21</td><td>0.15</td><td>0.23</td></tr>
  <tr><td>xho_Latn</td><td>0.49</td><td>0.19</td><td>0.47</td><td>0.20</td><td>0.20</td><td>0.12</td><td>0.10</td><td>0.13</td><td>0.05</td><td>0.22</td></tr>
  <tr><td>nso_Latn</td><td>0.27</td><td>0.11</td><td>0.48</td><td>0.26</td><td>0.23</td><td>0.17</td><td>0.13</td><td>0.19</td><td>0.07</td><td>0.21</td></tr>
  <tr><td>sot_Latn</td><td>0.34</td><td>0.12</td><td>0.53</td><td>0.22</td><td>0.20</td><td>0.14</td><td>0.10</td><td>0.18</td><td>0.05</td><td>0.21</td></tr>
  <tr><td>zul_Latn</td><td>0.55</td><td>0.19</td><td>0.44</td><td>0.19</td><td>0.17</td><td>0.11</td><td>0.06</td><td>0.10</td><td>0.05</td><td>0.21</td></tr>
  <tr><td>kin_Latn</td><td>0.37</td><td>0.10</td><td>0.53</td><td>0.20</td><td>0.23</td><td>0.11</td><td>0.09</td><td>0.15</td><td>0.06</td><td>0.21</td></tr>
  <tr><td>sin_Sinh</td><td>0.56</td><td>0.27</td><td>0.49</td><td>0.26</td><td>0.18</td><td>0.01</td><td>0.01</td><td>0.03</td><td>0.02</td><td>0.20</td></tr>
  <tr><td>smo_Latn</td><td>0.28</td><td>0.09</td><td>0.66</td><td>0.20</td><td>0.20</td><td>0.10</td><td>0.08</td><td>0.13</td><td>0.06</td><td>0.20</td></tr>
  <tr><td>nya_Latn</td><td>0.36</td><td>0.13</td><td>0.41</td><td>0.19</td><td>0.19</td><td>0.13</td><td>0.10</td><td>0.17</td><td>0.06</td><td>0.19</td></tr>
  <tr><td>twi_Latn</td><td>0.23</td><td>0.08</td><td>0.46</td><td>0.22</td><td>0.22</td><td>0.14</td><td>0.13</td><td>0.19</td><td>0.07</td><td>0.19</td></tr>
  <tr><td>sna_Latn</td><td>0.41</td><td>0.17</td><td>0.40</td><td>0.19</td><td>0.20</td><td>0.10</td><td>0.08</td><td>0.12</td><td>0.07</td><td>0.19</td></tr>
  <tr><td>ulg_Arab</td><td>0.21</td><td>0.09</td><td>0.71</td><td>0.29</td><td>0.29</td><td>0.00</td><td>0.01</td><td>0.03</td><td>0.02</td><td>0.18</td></tr>
  <tr><td>bug_Latn</td><td>0.14</td><td>0.12</td><td>0.35</td><td>0.22</td><td>0.22</td><td>0.14</td><td>0.11</td><td>0.20</td><td>0.12</td><td>0.18</td></tr>
  <tr><td>luo_Latn</td><td>0.07</td><td>0.07</td><td>0.40</td><td>0.25</td><td>0.24</td><td>0.15</td><td>0.12</td><td>0.21</td><td>0.09</td><td>0.18</td></tr>
  <tr><td>tsn_Latn</td><td>0.24</td><td>0.10</td><td>0.42</td><td>0.18</td><td>0.18</td><td>0.12</td><td>0.11</td><td>0.20</td><td>0.06</td><td>0.18</td></tr>
  <tr><td>arb_Latn</td><td>0.29</td><td>0.07</td><td>0.46</td><td>0.24</td><td>0.20</td><td>0.05</td><td>0.05</td><td>0.17</td><td>0.08</td><td>0.18</td></tr>
  <tr><td>khm_Khmr</td><td>0.34</td><td>0.15</td><td>0.59</td><td>0.15</td><td>0.16</td><td>0.01</td><td>0.02</td><td>0.09</td><td>0.06</td><td>0.17</td></tr>
  <tr><td>lua_Latn</td><td>0.09</td><td>0.08</td><td>0.33</td><td>0.20</td><td>0.21</td><td>0.14</td><td>0.13</td><td>0.24</td><td>0.12</td><td>0.17</td></tr>
  <tr><td>lug_Latn</td><td>0.17</td><td>0.07</td><td>0.41</td><td>0.18</td><td>0.19</td><td>0.14</td><td>0.09</td><td>0.19</td><td>0.06</td><td>0.17</td></tr>
  <tr><td>grn_Latn</td><td>0.17</td><td>0.09</td><td>0.44</td><td>0.16</td><td>0.17</td><td>0.12</td><td>0.09</td><td>0.13</td><td>0.10</td><td>0.16</td></tr>
  <tr><td>ssw_Latn</td><td>0.27</td><td>0.10</td><td>0.37</td><td>0.17</td><td>0.17</td><td>0.11</td><td>0.08</td><td>0.14</td><td>0.05</td><td>0.16</td></tr>
  <tr><td>lin_Latn</td><td>0.11</td><td>0.08</td><td>0.43</td><td>0.16</td><td>0.18</td><td>0.12</td><td>0.11</td><td>0.16</td><td>0.08</td><td>0.16</td></tr>
  <tr><td>ory_Orya</td><td>0.28</td><td>0.03</td><td>0.66</td><td>0.18</td><td>0.19</td><td>0.01</td><td>0.01</td><td>0.03</td><td>0.03</td><td>0.16</td></tr>
  <tr><td>fij_Latn</td><td>0.13</td><td>0.06</td><td>0.38</td><td>0.18</td><td>0.16</td><td>0.14</td><td>0.11</td><td>0.15</td><td>0.08</td><td>0.15</td></tr>
  <tr><td>fuv_Latn</td><td>0.07</td><td>0.08</td><td>0.30</td><td>0.20</td><td>0.20</td><td>0.13</td><td>0.10</td><td>0.18</td><td>0.10</td><td>0.15</td></tr>
  <tr><td>kas_Arab</td><td>0.16</td><td>0.10</td><td>0.50</td><td>0.20</td><td>0.21</td><td>0.02</td><td>0.02</td><td>0.10</td><td>0.05</td><td>0.15</td></tr>
  <tr><td>quy_Latn</td><td>0.10</td><td>0.06</td><td>0.42</td><td>0.21</td><td>0.22</td><td>0.10</td><td>0.06</td><td>0.13</td><td>0.05</td><td>0.15</td></tr>
  <tr><td>aka_Latn</td><td>0.17</td><td>0.06</td><td>0.37</td><td>0.14</td><td>0.17</td><td>0.11</td><td>0.08</td><td>0.12</td><td>0.10</td><td>0.15</td></tr>
  <tr><td>mya_Mymr</td><td>0.36</td><td>0.13</td><td>0.46</td><td>0.14</td><td>0.16</td><td>0.00</td><td>0.00</td><td>0.02</td><td>0.02</td><td>0.15</td></tr>
  <tr><td>run_Latn</td><td>0.25</td><td>0.07</td><td>0.37</td><td>0.16</td><td>0.17</td><td>0.08</td><td>0.06</td><td>0.10</td><td>0.04</td><td>0.14</td></tr>
  <tr><td>bem_Latn</td><td>0.14</td><td>0.08</td><td>0.29</td><td>0.16</td><td>0.16</td><td>0.13</td><td>0.11</td><td>0.15</td><td>0.06</td><td>0.14</td></tr>
  <tr><td>kas_Deva</td><td>0.14</td><td>0.09</td><td>0.37</td><td>0.20</td><td>0.21</td><td>0.02</td><td>0.03</td><td>0.11</td><td>0.08</td><td>0.14</td></tr>
  <tr><td>wol_Latn</td><td>0.09</td><td>0.07</td><td>0.30</td><td>0.18</td><td>0.16</td><td>0.12</td><td>0.10</td><td>0.17</td><td>0.07</td><td>0.14</td></tr>
  <tr><td>kam_Latn</td><td>0.09</td><td>0.08</td><td>0.26</td><td>0.18</td><td>0.16</td><td>0.13</td><td>0.10</td><td>0.15</td><td>0.08</td><td>0.14</td></tr>
</table>
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">Gemma</th>
    <th colspan="3">Llama</th>
    <th colspan="3">Mistral</th>
    <th colspan="2">OLMo</th>
    <th>AVG</th>
  </tr>
  <tr>
    <th>9B</th>
    <th>7B</th>
    <th>70B</th>
    <th>8B</th>
    <th>8B</th>
    <th>7B</th>
    <th>7B</th>
    <th>7B</th>
    <th>7B</th>
    <th>7B</th>
    <th>7B</th>
  </tr>
  <tr>
    <td>tso_Latn</td>
    <td>0.14</td>
    <td>0.06</td>
    <td>0.35</td>
    <td>0.14</td>
    <td>0.13</td>
    <td>0.11</td>
    <td>0.08</td>
    <td>0.13</td>
    <td>0.06</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>kon_Latn</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.27</td>
    <td>0.15</td>
    <td>0.17</td>
    <td>0.09</td>
    <td>0.07</td>
    <td>0.13</td>
    <td>0.09</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>tum_Latn</td>
    <td>0.15</td>
    <td>0.07</td>
    <td>0.32</td>
    <td>0.11</td>
    <td>0.13</td>
    <td>0.09</td>
    <td>0.08</td>
    <td>0.11</td>
    <td>0.05</td>
    <td>0.13</td>
  </tr>
  <tr>
    <td>kik_Latn</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.32</td>
    <td>0.12</td>
    <td>0.13</td>
    <td>0.10</td>
    <td>0.09</td>
    <td>0.13</td>
    <td>0.12</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>taq_Latn</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.28</td>
    <td>0.14</td>
    <td>0.12</td>
    <td>0.11</td>
    <td>0.08</td>
    <td>0.14</td>
    <td>0.05</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>mos_Latn</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.25</td>
    <td>0.16</td>
    <td>0.14</td>
    <td>0.11</td>
    <td>0.09</td>
    <td>0.15</td>
    <td>0.07</td>
    <td>0.12</td>
  </tr>
  <tr>
    <td>yor_Latn</td>
    <td>0.13</td>
    <td>0.04</td>
    <td>0.30</td>
    <td>0.14</td>
    <td>0.14</td>
    <td>0.08</td>
    <td>0.06</td>
    <td>0.10</td>
    <td>0.04</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>amh_Ethi</td>
    <td style="background-color:#ffff99;">0.48</td>
    <td>0.16</td>
    <td>0.24</td>
    <td>0.04</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>sag_Latn</td>
    <td>0.05</td>
    <td>0.07</td>
    <td>0.22</td>
    <td>0.17</td>
    <td>0.17</td>
    <td>0.07</td>
    <td>0.07</td>
    <td>0.09</td>
    <td>0.06</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>cjk_Latn</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.21</td>
    <td>0.13</td>
    <td>0.12</td>
    <td>0.10</td>
    <td>0.08</td>
    <td>0.13</td>
    <td>0.07</td>
    <td>0.11</td>
  </tr>
  <tr>
    <td>umb_Latn</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.20</td>
    <td>0.15</td>
    <td>0.14</td>
    <td>0.10</td>
    <td>0.08</td>
    <td>0.11</td>
    <td>0.05</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>dyu_Latn</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.22</td>
    <td>0.13</td>
    <td>0.12</td>
    <td>0.06</td>
    <td>0.07</td>
    <td>0.10</td>
    <td>0.08</td>
    <td>0.10</td>
  </tr>
  <tr>
    <td>kac_Latn</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.22</td>
    <td>0.12</td>
    <td>0.14</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.12</td>
    <td>0.08</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>kmb_Latn</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.20</td>
    <td>0.11</td>
    <td>0.10</td>
    <td>0.10</td>
    <td>0.07</td>
    <td>0.09</td>
    <td>0.05</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>bam_Latn</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.18</td>
    <td>0.11</td>
    <td>0.09</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.12</td>
    <td>0.04</td>
    <td>0.09</td>
  </tr>
  <tr>
    <td>ayr_Latn</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.20</td>
    <td>0.11</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.05</td>
    <td>0.10</td>
    <td>0.06</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>lao_Laoo</td>
    <td>0.17</td>
    <td>0.04</td>
    <td>0.22</td>
    <td>0.07</td>
    <td>0.09</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.04</td>
    <td>0.09</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>dik_Latn</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.18</td>
    <td>0.06</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.07</td>
    <td>0.10</td>
    <td>0.05</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>ewe_Latn</td>
    <td>0.04</td>
    <td>0.03</td>
    <td>0.18</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.09</td>
    <td>0.07</td>
    <td>0.08</td>
    <td>0.04</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>knc_Latn</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.15</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.07</td>
    <td>0.06</td>
    <td>0.08</td>
    <td>0.05</td>
    <td>0.08</td>
  </tr>
  <tr>
    <td>kab_Latn</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.17</td>
    <td>0.09</td>
    <td>0.08</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.11</td>
    <td>0.04</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>sat_Olck</td>
    <td>0.19</td>
    <td>0.02</td>
    <td>0.32</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>gaz_Latn</td>
    <td>0.05</td>
    <td>0.03</td>
    <td>0.20</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.05</td>
    <td>0.04</td>
    <td>0.08</td>
    <td>0.03</td>
    <td>0.07</td>
  </tr>
  <tr>
    <td>bod_Tibt</td>
    <td>0.07</td>
    <td>0.01</td>
    <td>0.22</td>
    <td>0.08</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>fon_Latn</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.14</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.03</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.07</td>
    <td>0.06</td>
  </tr>
  <tr>
    <td>shn_Mymr</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.21</td>
    <td>0.06</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.07</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>kbp_Latn</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.14</td>
    <td>0.05</td>
    <td>0.04</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.08</td>
    <td>0.05</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>mni_Beng</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.12</td>
    <td>0.05</td>
    <td>0.06</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.08</td>
    <td>0.02</td>
    <td>0.05</td>
  </tr>
  <tr>
    <td>ace_Arab</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.15</td>
    <td>0.07</td>
    <td>0.07</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>knc_Arab</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.13</td>
    <td>0.04</td>
    <td>0.04</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.05</td>
    <td>0.05</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>bjn_Arab</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.11</td>
    <td>0.05</td>
    <td>0.08</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.05</td>
    <td>0.01</td>
    <td>0.04</td>
  </tr>
  <tr>
    <td>nus_Latn</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.07</td>
    <td>0.04</td>
    <td>0.03</td>
    <td>0.03</td>
    <td>0.02</td>
    <td>0.04</td>
    <td>0.05</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>min_Arab</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.13</td>
    <td>0.05</td>
    <td>0.04</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.03</td>
    <td>0.01</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>tir_Ethi</td>
    <td>0.10</td>
    <td>0.02</td>
    <td>0.05</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.03</td>
  </tr>
  <tr>
    <td>dzo_Tibt</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.08</td>
    <td>0.03</td>
    <td>0.03</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.02</td>
  </tr>
  <tr>
    <td>taq_Tfng</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.04</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.03</td>
    <td>0.01</td>
  </tr>
  <tr>
    <td>zgh_Tfng</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.02</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.01</td>
    <td>0.01</td>
    <td>0.01</td>
  </tr>
</table>

Table 7: Adjusted performance of MEXA using max pooling with the English performance of models on the Belebele benchmark.

Reviewer paper summaries: This paper introduces MEXA, a method for predicting the multilingual capabilities of pre-trained English-centric LLMs, given the weights of the model. It does this by computing the alignment for a given model between English and non-English languages using parallel sentences, and then uses this as a proxy for the multilingual capability of that model.

The authors first explain the MEXA alignment score and the reasoning behind their design decisions. They then describe the experiments they carried out to verify the correlation between MEXA and the multilingual capabilities of LLMs. For these, they used two parallel datasets, nine LLMs from four model families, and three multilingual NLU benchmarks. They carry out correlation analysis to verify their design decisions, as well as showing that downstream performance correlates strongly with downstream task performance and more strongly than absolute cosine similarity

This paper introduces MEXA, a novel method for evaluating cross-lingual alignment in LLMs between English and non-English languages. Based on the hypothesis that parallel sentences should exhibit high similarity in multilingual embedding spaces, MEXA computes an alignment score using cosine similarity between sentence embeddings from parallel datasets such as Flores-200. Experiments show that the MEXA score is strongly correlated with the downstream task performance, suggesting that multilingual capability depends on alignment with English. Additionally, the adjusted MEXA scores can be used to estimate performance on downstream tasks for other languages.
Summary of strengths: - The paper presents an intuitive and interesting method for predicting the multilingual capabilities of an LLM in a relatively inexpensive way.
- The analysis in the paper is rigorous and thorough, covering a wide range of models and a reasonable amount of test data.
- I particularly appreciated the clear explanation of MEXA in section 3 and the reasoning behind the various design decisions.
- The limitations of the method are discussed well, making it easier for a user to see where MEXA will be useful.
- Code and datasets are made available

- The MEXA score can more effectively evaluate the cross-lingual alignment than cosine similarity. It's interesting that the downstream task performance is strongly correlated with alignment with English.
- The adjusted MEXA score can estimate performance on downstream tasks for other languages.
Summary of weaknesses: 1) It wasn't clear to me (as a non-specialist in this area) if other methods aside from cosine similarity have already been developed to measure cross-lingual alignment in LLMs. If not, this should be made explicit to highlight the novelty of the method. If they have, these should be mentioned and compared with MEXA in the paper.

2) There is some discussion of how differences in resourcedness affect results (e.g. L 568-576), but results which take into account variation by language are not quantified in the text. Understanding how well metrics work for less-resourced languages is important for inclusion and should be discussed (e.g. by including standard deviation as well as average where appropriate).

3) Section 5.2 is currently missing a conclusion making the significance of the results explicit.

- The task type is related to understanding tasks. How does MEXA perform in generative tasks such as MGSM, where the questions are simply entered?
- Some sections (like Section 2, 4.1, 5.1) could be more concise. It's better to demonstrate the results in a table in Section 5.3.
- Some tables and figures are puzzling, which need refined captions. For example, the AVG column in Table 1 is weird. How to understand the value in this column? The bold and underlined values in Table 2 are also not well organized.
Comments, suggestions and typos: Suggestions:
- You should mention the Bible dataset (L415) in the contributions if it's a new resource, as well as what license it's under.
- L420: Mention how you chose the 103 sentences. Were they from diverse parts of the Bible?
- L458: I feel there should be a footnote for the source of lm-evaluation-harness
- Figure 1: You should explain the significance of the dotted line in the caption.
- Table 6: if possible, it would be good to make a figure summarising this table for the main body to support your conclusion that two models show higher multilinguality. Perhaps a cumulative plot showing scores on the x axis and the number of scores under that threshold on the y axis? 
 

Typos:
- L 339 and 341 have the ` the wrong way round at the start of the quote.

- Does each data point in Figure 1 represent one language?
- Does the cross-lingual alignment have other applications?
Overall assessment scores: 4

2