<figure><img src="image_1.png" /></figure>

HAL
open science

MEXA: Multilingual Evaluation of English-Centric
LLMs via Cross-Lingual Alignment

Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, Hinrich Schuetze

<figure><img src="image_2.png" /></figure>

► To cite this version:

Amir Hossein Kargaran, Ali Modarressi, Nafiseh Nikeghbal, Jana Diesner, François Yvon, et al. MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment. Findings of the Association for Computational Linguistics: ACL 2025, Association for Computational Linguistics, Jul 2025, Vienna, Austria. pp.27001-27023, 10.18653/v1/2025.findings-acl.1385. hal-05207048

HAL Id: hal-05207048

https://hal.science/hal-05207048v1

Submitted on 11 Aug 2025

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

Amir Hossein Kargaran¹ Ali Modarressi¹ Nafiseh Nikeghbal²

Jana Diesner² François Yvon³ Hinrich Schütze¹

¹LMU Munich & Munich Center for Machine Learning

²Technical University of Munich

³Sorbonne Université & CNRS, ISIR

{amir, ali}@cis.lmu.de, nafiseh.nikeghbal@tum.de

## Abstract

English-centric large language models (LLMs) often show strong multilingual capabilities. However, their multilingual performance remains unclear and is under-evaluated for many other languages. Most benchmarks for multilinguality focus on classic NLP tasks or cover a minimal number of languages. We introduce MEXA, a method for assessing the multilingual capabilities of pre-trained English-centric LLMs using parallel sentences, which are available for more languages than existing downstream tasks. MEXA leverages that English-centric LLMs use English as a pivot language in their intermediate layers. MEXA computes the alignment between English and non-English languages using parallel sentences to evaluate the transfer of language understanding from English to other languages. This alignment can be used to estimate model performance in different languages. We conduct controlled experiments using various parallel datasets (FLORES-200 and Bible), models (Llama family, Gemma family, Mistral, and OLMo), and established downstream tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute embeddings in decoder-only models. Our results show that MEXA, in its default settings, achieves an average Pearson correlation of 0.90 between its predicted scores and actual task performance across languages. This suggests that MEXA is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs.

et al., 2023), open-weight models such as Llama 3 (Dubey et al., 2024); and open-source models such as OLMo (Groeneveld et al., 2024). English-centric refers to the majority of the pre-training data for these models being in English (Zhong et al., 2024; Kew et al., 2024). Even models labeled as heavily multilingual, such as BLOOM (Big-Science Workshop et al., 2023), have their major pre-training data in English (Laurençon et al., 2022).

Except for open-source models, where pre-training data is available and language distribution is transparent, there is still confusion about the language capabilities and coverage of other LLMs. Primarily, the focus in evaluating LLMs has been on developing benchmarks to assess their performance in English. Most benchmarks in multilingual settings consist of classical monolingual NLP tasks such as sequence labeling (Ahuja et al., 2023; Lai et al., 2023a), automatic translation of popular English benchmarks such as MMLU (Hendrycks et al., 2021) into a limited number of languages (Lai et al., 2023b; OpenAI, 2024), or the creation of language-specific benchmarks (Ghahroodi et al., 2024; Koto et al., 2024; Son et al., 2024; Yüksel et al., 2024; Li et al., 2024c).

Most LLMs are English-centric, either by choice or due to the availability of abundant data sources in English. For these models to be effective in other languages, the other languages must align with the dominant language, i.e., English. Given such alignment, English could act as a *rising tide that raises all ships*, meaning that improvements in English performance could benefit other languages, especially in tasks such as reasoning (Zhu et al., 2024). Contrarily, if a language does not align well with English, an English-centric LLM may not provide *meaningful coverage* for that language. Indeed, Wendler et al. (2024) have found that for Llama 2 (Touvron et al., 2023b), an English-centric LLM, English could be seen as a kind of “pivot”

<figure><img src="image_4.png" /></figure>

Leaderboard cis-lmu-mexa.hf.space

<figure><img src="image_5.png" /></figure>

Code github.com/cisnlp/MEXA

## 1 Introduction

Most state-of-the-art autoregressive large language models (LLMs) are “English-centric”, including closed-source models such as GPT-4 (OpenAI
language, enabling to solve complex semantic tasks
in a foreign language through a detour into English.
More precisely, they show that Llama 2 was able
to decode semantically correct next tokens in the
middle layers, assigning higher probabilities to the
English tokens than to the foreign version, which
is only selected in the upper layers. Zhao et al.
(2024) present a hypothesis regarding the middle
layers of English-centric LLMs, suggesting that
these models use English as a means of reasoning
while incorporating multilingual knowledge. Based
on their analysis, the number of language-specific
neurons in the middle layers decreases within the
self-attention mechanism but remains consistent
across the layers of the feed-forward structure when
processing multilingual queries.

In this paper, we introduce MEXA, a method
to estimate the actual multilingual coverage of
English-centric LLMs. It builds on the observa-
tion that these models semantically use English as
a pivot language in their middle layers, by measur-
ing how well embeddings of non-English sentences
align with their English counterparts.

We verify the MEXA estimation of language
coverage for each LLM, using Pearson correlation
between estimated and actual scores for various
tasks. We use two parallel datasets: FLORES-
200 (NLLB Team et al., 2022) and Bible (Mayer
and Cysouw, 2014); nine LLMs: Llama family,
Gemma family, Mistral, and OLMo; and three
tasks: Belebele (Bandarkar et al., 2024), m-MMLU,
and m-ARC (Lai et al., 2023b). Our results show
that MEXA achieves a promising average Pearson
correlation of 0.90 with established downstream
tasks across nine models and two parallel datasets.
In our study on the calculation of MEXA scores,
we conduct multiple design analyses to examine the
impact of token-level pooling for the embeddings
(i.e., using the last token versus a weighted average)
and layer-level pooling in computing alignment
scores. While MEXA demonstrates a high corre-
lation across most setups, we find that a weighted
average based on tokens, combined with mean pool-
ing, yields the best results. In summary, MEXA
offers a scalable way to estimate the multilingual
coverage of English-centric LLMs via alignment
with English.

## 2 Background and Related Work

We discuss distribution of pre-training data in
LLMs, and multilingual evaluation benchmarks

in Appendices A.1 and A.2, and focus on cross-
lingual alignment here. Research on cross-lingual
alignment either aims to uncover the underlying
mechanisms of alignment and assesses its impact
on models and downstream tasks, or attempts to en-
hance model performance by enforcing alignment
before, during, or after the pre-training phase. Most
of these papers have focused on encoder-only mod-
els, such as XLM-R (Conneau et al., 2020a) and
mBERT (Devlin et al., 2019), among others (Häm-
merl et al., 2024). In this work, we focus on
decoder-only models.

**Understanding Alignment.** Ye et al. (2023) show that English-centric models such as Llama 1 (Touvron et al., 2023a) possess multilingual transfer abilities (after fine-tuning on one source language, they can be applied to other languages) and may even surpass the multilingual transfer abilities of multilingual pre-trained models such as BLOOM (BigScience Workshop et al., 2023). Schäfer et al. (2024) find that GPT-2-style decoder-only models show strong cross-lingual generalization when trained on an imbalanced mix of languages. However, when trained on a balanced language set, they do not observe increased performance compared to monolingual settings. Wendler et al. (2024) perform single-token analysis to demonstrate that English-centered LLMs, such as Llama 2, use English semantically as an internal latent language in the middle layers when handling multilingual queries. Zhong et al. (2024) extend this analysis to multiple tokens, also showing that an LLM dominated by both English and Japanese uses both languages as internal latent languages. Zhao et al. (2024) explore how LLMs handle multilingualism. They hypothesize that LLMs initially interpret the query and convert multilingual inputs into English for task-solving. In the middle layers, the models rely on English with self-attention mechanisms for reasoning, while employing multilingual knowledge through feed-forward structures. In the final layers, LLMs generate responses consistent with the original query language. Li et al. (2025) and Li et al. (2024b) are even more closely related to ours. Li et al. (2025) uses absolute cosine similarity values between last token embeddings derived from parallel sentences with English to predict the ranking of language performance across various models. However, as we discuss in Section 3, relying solely on absolute cosine values can be misleading, and as shown in Section 5.3, absolute cosine values are less correlated
with downstream tasks than MEXA score. Li et al. (2024b) uses English probing tasks and their automatic translations to construct a multilingual evaluation. While they compare embedding similarity scores between high- and low-resource languages with corresponding evaluation results, similar to Li et al. (2025), they do not assess whether these correlations hold across other downstream tasks. In Section 5, we demonstrate that MEXA scores align closely with a broad range of downstream tasks.

**Boosting Alignment.** The idea to enforce alignment in encoder-only models using parallel sentences dates back to Conneau and Lample (2019), and has been explored under various guises, e.g., using mixed-language sentences and/or bilingual dictionaries (Huang et al., 2019; Conneau et al., 2020b; Cao et al., 2020; Kulshreshtha et al., 2020; Efimov et al., 2023; Zhang et al., 2023b). Recently, Li et al. (2024d) improved multilingual alignment by initializing the decoder-only models to generate similar representations of aligned words using contrastive learning and preserving this alignment using a code-switching strategy during pre-training. Liu et al. (2024a) propose a data allocation technique to select a core subset of languages for fine-tuning, better aligning the multilingual capabilities of decoder-only LLMs and making them more truthful in their responses. Li et al. (2024a) propose aligning internal sentence representations across different languages using multilingual contrastive learning and aligning outputs by following cross-lingual instructions in the target language for decoder-only models.

## 3 MEXA

We now describe the MEXA method for computing the alignment score of language $L_1$ with a pivot language $L_2$, given the language model $m$. In this paper, we use the term *cross-lingual alignment*, or simply *alignment* to refer to the semantic similarity of multilingual embeddings across languages. $L_2$, for English-centric LLMs and in this paper, is English. To assess alignment, we use parallel sentences in two languages, $L_1$ and $L_2$. The goal of semantic similarity is to ensure that parallel sentences have sufficiently high similarity, reflecting alignment between the two languages. However, considering only the absolute cosine similarity value as the alignment score does not guarantee proper alignment. For some languages, even non-

parallel sentences exhibit similarity scores comparable to those of parallel sentences (see §5.3). This is largely due to the anisotropy problem observed in transformer models, which can lead to so-called hubness issues, making it difficult to distinguish between similar and dissimilar embeddings (Ethanyarajh, 2019), especially in multilingual models (Hämmerl et al., 2023; Rajaee and Pilehvar, 2022). However, a direct comparative analysis of the cosine similarity between parallel and non-parallel sentence pairs across languages can help overcome these issues. Instead of using the absolute cosine similarity value for alignment, we assign binary values (1 or 0) based on whether a criterion for semantic similarity is satisfied. Our criterion imposes that (a) parallel sentences should have high cosine similarity, and (b) non-parallel pairs should also have low similarity values, ensuring the similarity is not random or biased. Specifically, if the cosine similarity for a pair of parallel sentences is higher than for any non-parallel sentences, we assign a value of 1 for this pair; otherwise, a value of 0. This approach sidesteps the hubness problem since the absolute cosine similarity values themselves are not directly used.

To compute MEXA, we first apply the cosine similarity function to the pairs of embeddings of parallel sentences in languages $L_1$ and $L_2$. In Section 3.1, we describe how embeddings can be computed for each layer $l$ of the autoregressive language model $m$. We generate a square matrix $C(L_1, L_2, m, l)$ representing cosine similarities of embeddings at the output of layer $l$ for all parallel sentences in languages $L_1$ and $L_2$. We denote $c_{ij}(l)$ the element in the $i$-th row and $j$-th column of $C(L_1, L_2, m, l)$. It represents the cosine similarity between the $i$-th sentence of $L_1$ and the $j$-th sentence of $L_2$ at layer $l$ of language model $m$. The diagonal elements of $C$, denoted $c_{ii}(l)$, represent the cosine similarity between parallel sentence pairs from $L_1$ and $L_2$. We define the MEXA alignment score $\mu(C(L_1, L_2, m, l))$ as follows:

$$ \frac{1}{n} \sum_{i=1}^{n} 1 \left( c_{ii}(l) > \max_{j \in \{1, \dots, n\} \setminus \{i\}} \{c_{ij}(l), c_{ji}(l)\} \right), $$

where $n$ is the number of diagonal elements (i.e., the dimension of the matrix), and $1(\cdot)$ is the indicator function, which equals 1 if its argument condition evaluates to true and 0 otherwise. This alignment score measures how often $c_{ii}(l)$ is the maximum value in both its row and column. The
MEXA alignment score can alternatively be understood as a measure of sentence retrieval performance (Hu et al., 2020; Liu et al., 2024b; Hämmerl et al., 2024), with the metric of P@1 applied with queries in language $L_1$ and answers in $L_2$, and vice versa. We discuss other ways to calculate semantic similarity between languages in Appendix A.3.

**Layer-wise Pooling.** The MEXA alignment score $\mu(C(L_1, L_2, m, l))$ is computed for language $L_1$ respect to pivot language $L_2$ for each layer $l$ of the language model $m$. To compute a single MEXA alignment score given the language model $m$ and $L_1, L_2$, we use mean and max pooling strategies over multiple layers.

## 3.1 Sentence Embeddings

We focus on autoregressive language models that use a decoder-only architecture. In this architecture, attention is not bidirectional; instead, it takes the form of causal attention (left-to-right). In bidirectional attention, each token has access to every other token in the sequence. However, in causal attention, the embedding of a token at position $t$ is only influenced by the embedding of preceding tokens at positions $0, 1, \dots, t-1$. Therefore, simple averaging values biases the embeddings towards sentence-initial words. Instead, we consider alternative methods: using only the last token and weighted averaging. We use and compare both methods to acquire the sentence embeddings needed for MEXA.

A standard way to compute a sentence embedding uses only the last token of that sentence. Jiang et al. (2023b) show that using the last token in the format of a prompt template for a sentence $s$, such as 'This sentence: {s} means in one word:' can be effective. Inspired by this, Li and Li (2024) used the prompt 'Summarize sentence {s} in one word:' to obtain the last token embedding as the sentence-level text embedding. However, not all models are instruction-tuned; some earlier works, such as Neelakantan et al. (2022); Wang et al. (2024); Ma et al. (2024), use the last token without any prompt. Since the models studied in this paper are only pretrained and use multiple languages in the input, we decided to use the last token method without any preceding instruction. An alternative is weighted averaging, which relies on the intuition that using only the last token might not represent the entire sentence, as the influence of earlier tokens may have diminished. This implies that the tokens at the end of the sentence should contribute more to

<figure><img src="image_6.png" /><figcaption>Figure 1: The probability that at least $k$ out of $n$ diagonal elements in an $n \times n$ random matrix are the maximum elements in their respective rows and columns.</figcaption></figure>

the overall embedding than those at the beginning. Another motivation for using weighted averaging is that sentence-final tokens are influenced by preceding tokens and contain more context, while the representation of sentence-initial tokens has significantly less contextual representation. To address this, Muennighoff (2022) proposes to assign weights to each token based on its position. Thus, the sentence embedding of layer $l$ using position-weighted averaging is:

$$e_l = \sum_{t=1}^{T} w_t h_{lt} \quad \text{with} \quad w_t = \frac{t}{\sum_{k=1}^{T} k},$$

where $T$ is the number of tokens in the given sentence, $h_{lt}$ is the embedding of the $t$-th token at layer $l$, and $e_l$ is the sentence embedding at layer $l$.

## 3.2 Robustness of MEXA

We show that the MEXA alignment score ($\mu(.)$) is very robust, and the odds of this score randomly achieving a high value are very slim. Recall that $\mu(C(L_1, L_2, m, l))$ measures the fraction of diagonal elements in matrix $C(L_1, L_2, m, l)$ that have the maximum value in their respective rows and columns. If this condition is met $k$ times out of $n$ diagonal elements, then $\mu(C(L_1, L_2, m, l))$ is $\frac{k}{n}$. In an $n \times n$ random matrix, the probability of a diagonal element being the maximum in its row and column (a total of $2n-1$ elements) is $p = \frac{1}{2^{n-1}}$. The probability that at least $k$ out of $n$ independent variables are satisfied, given that the diagonal element is the maximum in its row and column, can be computed using the binomial distribution:

$$P(X \ge \frac{k}{n}) = 1 - \sum_{i=0}^{k-1} \binom{n}{i} p^i (1-p)^{n-i}$$

In Figure 1, we plot $P(X \ge \frac{k}{n})$. This plot illustrates that, given a sufficient number of parallel sen-
tences (n), the probability of achieving a high score by chance is very low. For example, with n = 100, the chance of obtaining MEXA alignment score larger than 0.05 (k = 5) from a 100 × 100 random matrix is P(X ≥ 0.05) = 0.00016.

# 4 Experiments

We conduct experiments using various multi-parallel datasets (FLORES-200 and the Bible), models (Llama family, Gemma family, Mistral, and OLMo), and existing benchmarks/tasks (Belebele, m-MMLU, m-ARC). Our objective is to assess how well the MEXA alignment score from various parallel datasets correlates with the different benchmarks/tasks for different models.

## 4.1 Parallel Data

We calculate the MEXA score using the parallel datasets of FLORES-200 (NLLB Team et al., 2022) and the Bible (Mayer and Cysouw, 2014). While there are other high-quality parallel datasets, such as NTREX-128 (Federmann et al., 2022), IN22 (Gala et al., 2023), OPUS-100 (Zhang et al., 2020), Europarl (Koehn, 2005), OpenSubtitles (Lison and Tiedemann, 2016), TED2020 (Reimers and Gurevych, 2020), and Tatoeba (Tatoeba Community, 2006), we chose FLORES-200 due to its high quality and support for a wide range of languages, and the Bible dataset was selected for its extensive language coverage.

FLORES-200 is a parallel corpus with English sentences from Wikimedia translated into 204 language-script pairs, verified by humans. It includes 997 dev, 1012 dev-test, and 992 test sentences. As the test set isn't public, we use the dev-test set as our test corpus, following prior work. For faster computation, we consider only the first 100 sentences from each language. As shown in Section 3.2, this is sufficient to ensure MEXA's robustness, as the odds of the MEXA score randomly achieving a high value with 100 sentences are very slim. This choice also enables scaling to more languages, many of which lack enough parallel samples.

The Parallel Bible (Mayer and Cysouw, 2014) covers a very large number of languages. From this resource, we managed to create a subcorpus, a super parallel dataset of the Bible, with 1,401 language-script labels, each containing 103 sentences (i.e., Bible verses).<fnref n="1" /> This corpus includes many low-resource languages, many of which are not covered by existing language technologies (Joshi et al., 2020), and MEXA can be adopted since only parallel data is needed. We use all 103 sentences from each language.

## 4.2 Models

For our experiments, we select models with around 7B parameters, which are considered a base size in the LLM community. The state-of-the-art open-weight models in this range, as measured by performance on English-based tasks such as MMLU (Stanford CRFM, 2024), include Llama 1, 2, 3, and 3.1 (Touvron et al., 2023a,b; Dubey et al., 2024), Gemma 1 and 2 (Gemma Team et al., 2024a,b), Mistral 0.3 (Jiang et al., 2023a), and the open-source model OLMo 1.7 (Groeneveld et al., 2024). We also select a larger model, Llama 3.1 70B, to show that our findings hold even when scaling up further. To apply MEXA, we need to access model weights to compute input sentence embeddings for each layer. We use three popular open-weight model families: Llama, Gemma, and Mistral. As a less multilingual version of state-of-the-art LLMs, we include OLMo, which is trained on a more English-centric corpus of Dolma (Sol-daini et al., 2024).

## 4.3 Benchmarks

Among the existing evaluation benchmarks in Table 5 from Appendix A.2, we chose Belebele (Bildarkar et al., 2024), m-ARC (Lai et al., 2023b), and m-MMLU (Lai et al., 2023b), which support the highest number of high-, medium-, and low-resource languages and are directly related to natural understanding tasks, which is the primary focus of this paper.

We use the entire test set for each of these benchmarks (§A.4 for details) for more details) to evaluate the models, except in one case. For Llama 3.1 70B, we use the first 500 questions of m-MMLU instead of the whole set due to resource constraints. Since the selected LLMs used in our experiment are not instruction-tuned, we use 5-shot in-context learning with the lm-evaluation-harness framework, employing log-likelihood-based multiple-choice scoring. Other settings, such as prompt templates, are configured according to the framework's default (Gao et al., 2023; Biderman et al., 2024).

<footnote n="1">¹hf.co/datasets/cis-lmu/sPBC</footnote>
<table><caption>Table 1: $\mu_{pooling}$ shows MEXA scores for each pooling strategy using token-weighted embeddings. Results cover English-only tasks, non-English tasks (Belebele: 116 languages, m-MMLU: 33, m-ARC: 31), and MEXA scores from FLORES (116) and Bible (101). Top scores are in **bold**, second-best are **underlined**.</caption><thead><tr><th></th><th></th><th>Gemma 2<br>9B</th><th>Gemma 1<br>7B</th><th>Llama 3.1<br>70B</th><th>Llama 3.1<br>8B</th><th>Llama 3<br>8B</th><th>Llama 2<br>7B</th><th>Llama 1<br>7B</th><th>Mistral 0.3<br>7B</th><th>OLMo 1.7<br>7B</th><th>AVG</th></tr></thead><tbody><tr><td rowspan="3">Task<sub>(eng)</sub></td><td>Belebele</td><td><u>0.9178</u></td><td>0.8467</td><td><b>0.9456</b></td><td>0.8767</td><td>0.8689</td><td>0.4822</td><td>0.4156</td><td>0.8389</td><td>0.7711</td><td>0.7737</td></tr><tr><td>m-MMLU</td><td><u>0.6998</u></td><td>0.6138</td><td><b>0.7700</b></td><td>0.6315</td><td>0.6294</td><td>0.4523</td><td>0.3569</td><td>0.5988</td><td>0.5210</td><td>0.5859</td></tr><tr><td>m-ARC</td><td><u>0.6775</u></td><td>0.5870</td><td><b>0.7014</b></td><td>0.5794</td><td>0.5836</td><td>0.5128</td><td>0.5000</td><td>0.5862</td><td>0.4872</td><td>0.5795</td></tr><tr><td rowspan="3">Task<sub>L\(eng)</sub></td><td>Belebele</td><td><u>0.7093</u></td><td>0.5633</td><td><b>0.7684</b></td><td>0.5705</td><td>0.5533</td><td>0.3028</td><td>0.2755</td><td>0.4457</td><td>0.3627</td><td>0.5057</td></tr><tr><td>m-MMLU</td><td><u>0.5582</u></td><td>0.4734</td><td><b>0.6384</b></td><td>0.4720</td><td>0.4664</td><td>0.3260</td><td>0.2807</td><td>0.4207</td><td>0.3390</td><td>0.4416</td></tr><tr><td>m-ARC</td><td><u>0.4779</u></td><td>0.4220</td><td><b>0.5054</b></td><td>0.3941</td><td>0.3892</td><td>0.3174</td><td>0.2970</td><td>0.3662</td><td>0.2731</td><td>0.3825</td></tr><tr><td rowspan="2">FLORES</td><td>μ<sup>Mean</sup></td><td><b>0.5088</b></td><td>0.3815</td><td><u>0.4110</u></td><td>0.3963</td><td>0.3939</td><td>0.0866</td><td>0.1946</td><td>0.2642</td><td>0.0413</td><td>0.2976</td></tr><tr><td>μ<sup>Max</sup></td><td>0.7194</td><td>0.5872</td><td><b>0.7725</b></td><td>0.6538</td><td>0.6520</td><td>0.2464</td><td>0.3579</td><td>0.4716</td><td>0.1965</td><td>0.5175</td></tr><tr><td rowspan="2">Bible</td><td>μ<sup>Mean</sup></td><td><b>0.3568</b></td><td>0.2152</td><td><u>0.3169</u></td><td>0.2103</td><td>0.2026</td><td>0.1246</td><td>0.0908</td><td>0.1198</td><td>0.0121</td><td>0.1832</td></tr><tr><td>μ<sup>Max</sup></td><td><u>0.6076</u></td><td>0.4021</td><td><b>0.6599</b></td><td>0.4212</td><td>0.4190</td><td>0.2724</td><td>0.2357</td><td>0.2606</td><td>0.0319</td><td>0.3678</td></tr></tbody></table>

## 4.4 Evaluation Measures

We calculate the Pearson correlation coefficient to assess the strength of the correlation between MEXA and downstream performance on our evaluation benchmarks. This coefficient is a statistical measure of the strength and direction of the linear relationship between two variables. A high value would indicate that MEXA provides a reliable assessment of multilingual capabilities in English-centric LLMs.

## 5 Results

Table 1 presents the downstream performance of the selected models across three benchmarks, along with MEXA scores from two parallel datasets. Notably, among models with parameter sizes ranging from 7 to 9 billion, both Gemma 2 and Llama 3.1 outperform the other LLMs in terms of non-English downstream performance and MEXA scores. The Llama 3.1 and Llama 3 models exhibit similar alignment and downstream task performance, and both represent substantial advancements compared to Llama 2. Moreover, results for the Llama 3.1-70B model indicate that scaling can significantly enhance alignment when compared to its smaller version. Interestingly, while Mistral achieves comparable results to Gemma 1 on English benchmarks, it demonstrates inferior alignment, which likely accounts for its reduced performance on non-English tasks. Furthermore, the Llama 2 model achieves higher MEXA scores than OLMo, indicating better alignment. However, due to Llama 2's weaker performance on English tasks, it fails to transfer this alignment effectively, leading to comparable non-English performance between Llama 2 and OLMo. This observation is further explored in Section 5.2, where we normalize the expected performance based on the pivot language, namely English.

<table><caption>Table 2: Pearson correlation between MEXA scores and performance on Belebele, m-MMLU, and m-ARC, averaged across models. Results use two embedding aggregation methods: weighted average and last-token. Best scores per dataset and benchmark are in **bold**, second-best are **underlined**.</caption><thead><tr><th></th><th></th><th>Avg. across models</th></tr></thead><tbody><tr><td rowspan="6">FLORES<br>weighted average</td><td>μ<sub>Mean</sub> ↔ Belebele</td><td>0.8994</td></tr><tr><td>μ<sub>Max</sub> ↔ Belebele</td><td><u>0.9098</u></td></tr><tr><td>μ<sub>Mean</sub> ↔ m-MMLU</td><td><u>0.9513</u></td></tr><tr><td>μ<sub>Max</sub> ↔ m-MMLU</td><td>0.9188</td></tr><tr><td>μ<sub>Mean</sub> ↔ m-ARC</td><td><b>0.9393</b></td></tr><tr><td>μ<sub>Max</sub> ↔ m-ARC</td><td>0.8856</td></tr><tr><td rowspan="6">last token</td><td>μ<sub>Mean</sub> ↔ Belebele</td><td><b>0.9168</b></td></tr><tr><td>μ<sub>Max</sub> ↔ Belebele</td><td>0.9058</td></tr><tr><td>μ<sub>Mean</sub> ↔ m-MMLU</td><td><b>0.9545</b></td></tr><tr><td>μ<sub>Max</sub> ↔ m-MMLU</td><td>0.9134</td></tr><tr><td>μ<sub>Mean</sub> ↔ m-ARC</td><td><u>0.9195</u></td></tr><tr><td>μ<sub>Max</sub> ↔ m-ARC</td><td>0.8685</td></tr><tr><td rowspan="6">Bible<br>weighted average</td><td>μ<sub>Mean</sub> ↔ Belebele</td><td><u>0.8496</u></td></tr><tr><td>μ<sub>Max</sub> ↔ Belebele</td><td><b>0.8811</b></td></tr><tr><td>μ<sub>Mean</sub> ↔ m-MMLU</td><td><b>0.8823</b></td></tr><tr><td>μ<sub>Max</sub> ↔ m-MMLU</td><td><u>0.8210</u></td></tr><tr><td>μ<sub>Mean</sub> ↔ m-ARC</td><td><b>0.9018</b></td></tr><tr><td>μ<sub>Max</sub> ↔ m-ARC</td><td><u>0.8354</u></td></tr><tr><td rowspan="6">last token</td><td>μ<sub>Mean</sub> ↔ Belebele</td><td>0.8147</td></tr><tr><td>μ<sub>Max</sub> ↔ Belebele</td><td>0.8070</td></tr><tr><td>μ<sub>Mean</sub> ↔ m-MMLU</td><td>0.7572</td></tr><tr><td>μ<sub>Max</sub> ↔ m-MMLU</td><td>0.6998</td></tr><tr><td>μ<sub>Mean</sub> ↔ m-ARC</td><td>0.7469</td></tr><tr><td>μ<sub>Max</sub> ↔ m-ARC</td><td>0.6885</td></tr></tbody></table>

## 5.1 MEXA Correlation Analysis

We compute sentence embeddings for the selected models using two methods: weighted average based on token positions and last token (see §3.1). We apply mean and max pooling on the MEXA alignment scores across all model layers to derive a single score for each language. In Table 2 (refer to Table 6 for the detailed table), we report the correlation between the MEXA scores (computed using
both mean- and max-pooling, for the two embedding methods) and task performances. Across all settings, the best overall result (higher correlation) is achieved when embeddings are computed using the **weighted average**, with **mean pooling** as the pooling method. We adopt this configuration as the default setting for MEXA.

**FLORES vs Bible.** In the default setting, the average Pearson correlation coefficient for the FLORES parallel dataset across different tasks is 0.9300, and while for the Bible parallel dataset, it is 0.8779. The reason the Bible scores are generally lower than FLORES is that FLORES data is cleaner and more aligned with modern, standardized texts, whereas the Bible data is older and more specialized. For example, for some languages, the orthography of Bible texts no longer matches today's orthography. In the Bible, Arabic often includes diacritics, which are typically omitted in modern writing and tasks, making the text less familiar to models trained on contemporary data. Additionally, although the Bible dataset has been made parallel, sentence alignment can still be inconsistent due to translation nuances. In contrast, FLORES is carefully curated to ensure high-quality, sentence-level parallelism across languages for machine translation tasks.

**Weighted Average vs. Last Token Embeddings.** The use of last token embeddings shows promisingly high correlations with the FLORES parallel data; however, for the Bible dataset, the correlation is low in some cases. We believe this may stem from the high occurrence of Bible sentences (especially in English), which leads models to memorize these phrases. Using the WIMBD toolkit (Elazar et al., 2024), we found that, on average, there are 92 times more documents in Dolma 1.7 containing exact Bible sentences than those in FLORES. Consequently, when using Bible examples, the last token is biased towards predicting the specific memorized next token rather than incorporating context-related signals. Therefore, one should consider the hazard of memorized data when using last token embeddings. The weighted-average method, which takes into account the influence of multiple tokens, can mitigate the impact of a poor embedding for the last token and enable the model to capture useful information from the other tokens more robustly.

**Max Pooling vs. Mean Pooling.** In our comparison of mean pooling and max pooling on the Belebele benchmark, we found that mean pooling

<figure><img src="image_7.png" /><figcaption>Figure 2: Relationship between Llama 3.1-8B MEXA scores from Bible and FLORES, adjusted by English task performance, on Belebele and m-ARC. Each point represents one language.</figcaption></figure>

underestimates low-resource languages (resulting in more MEXA scores near 0), while max pooling correlates better with the Belebele benchmark. This can be explained by the fact that Belebele is an easier task among the three evaluated, allowing even low-resource languages to achieve good scores. Conversely, based on our experiment with m-ARC, max pooling tends to overestimate low-resource languages, making mean pooling more aligned with m-ARC. This can be attributed to m-ARC being the most challenging task among the three, where even medium-resource languages do not achieve high scores. Changing the pooling method from mean to max can be considered when dealing with different levels of understanding.

## 5.2 Downstream Performance Estimation

A full score on Pearson correlation (i.e., $\rho = 1.0$) indicates that a linear equation perfectly describes the relationship between MEXA and the evaluation benchmarks, with all data points lying on a line. Given the high correlation values shown in Table 6, it is reasonable to conclude that we can fit a line that closely approximates this linear relationship. This line converts the MEXA scores back to downstream task performances. We employed a linear model to predict this line by minimizing the residual sum of squares between the MEXA scores (multiplied by the performance on the English task) and the task performances. We needed to adjust the MEXA scores for this purpose, as the
MEXA score for language $L_1$ indicates how well $L_1$ is aligned with English but does not reflect the estimated task performance of the model for language $L_1$. Of course, this does not change the value of the correlation coefficient, as it is unaffected by linear transformations. The three tasks considered in this paper involve multiple-choice questions with four possible answers for each question, resulting in a chance of being randomly correct of $1/4$. However, the minimum score for MEXA scores is 0. Thus, the ideal slope for the line would be $3/4$ with an intercept of $1/4$ (X-axis: adjusted MEXA scores, Y-axis: task performance). In Figure 2, we plot this relationship for Llama 3.1-8B using the Bible and FLORES parallel datasets for Belebele and m-ARC. We chose max pooling for Belebele and mean pooling for m-ARC, since these pooling methods yield a stronger correlation (see §5.1). The pairs of (slope, intercept) from left to right in the Figure 2 are: (0.6804, 0.2477), (0.6103, 0.1838), (0.6340, 0.3408), and (0.5726, 0.2423). With data points from both high-resource and low-resource languages, this line can be calculated; otherwise, the ideal line may be used as a reference.

**Language Coverage.** We present the adjusted MEXA score for all languages available in FLORES-200 in Table 7 from Appendix A.6 for the selected models. The languages are categorized into groups ranging from well-covered to not covered. In Table 7, we can clearly see that Llama 3.1-70B and Gemma 2-9B show a higher level of multilinguality than other models.

## 5.3 MEXA vs Absolute Cosine Similarity

We compare MEXA with the use of absolute cosine similarities. We used parallel data from FLORES and downstream task data from the Belebele benchmark, focusing on 116 common labels. For each non-English language, we computed the average absolute cosine similarity for parallel sentences with English, and for non-parallel sentences with English. Following the setup by Li et al. (2025), which uses absolute cosine similarity values to predict language performance and ranking, we computed sentence embeddings using the last-token method and applied mean pooling over layers {5, 10, 15, 20, 25}.

To evaluate the correlation of each method with downstream task performance, we report results using the Gemma 1 and Llama 1 7B models. Table 3 summarizes these results. For both models, MEXA consistently achieves a higher correlation

<table><caption>Table 3: Pearson correlations between alignment metrics and Belebele performance. AC-P denotes absolute cosine similarity of parallel pairs; AC-NP, of non-parallel pairs.</caption><thead><tr><th></th><th>Gemma 1 7B</th><th>Llama 1 7B</th></tr></thead><tbody><tr><td>MEXA ↔ Belebele</td><td>0.9260</td><td>0.8365</td></tr><tr><td>AC-P ↔ Belebele</td><td>0.7651</td><td>0.6473</td></tr><tr><td>AC-P ↔ AC-NP</td><td>0.9232</td><td>0.9064</td></tr></tbody></table>

with downstream performance compared to the absolute cosine similarity of parallel sentences. Moreover, the correlation between cosine similarity for parallel and non-parallel sentences is notably high, suggesting that absolute values may be less discriminative across sentence types. This discrepancy highlights a limitation of using absolute cosine similarity: for some languages, similarity scores may remain high even for non-parallel sentences. Conversely, a low overall similarity score does not necessarily imply weak alignment, as parallel sentence scores may still significantly exceed non-parallel ones. In contrast, MEXA offers a more robust and comparative measure across languages.

## 5.4 Visualization of Layers

In Figure 3, we show the results of applying MEXA to 20 pairs of language_script from the FLORES parallel dataset for Llama 1-7B and Llama 3.1-8B across all 32 layers. We selected these languages from different families, writing systems, and both high- and low-resource categories. The embeddings are computed using a weighted average based on token positions. Figure 3 shows that high-resource languages (with more prevalence on the web; see §A.1) achieve higher alignment scores across different layers, while low-resource languages achieve lower scores. In the initial layers, embeddings are more in-language, resulting in lower alignment scores. As embeddings progress to the mid-layers, they become more aligned with the dominant language of the LLM, i.e., English.

MEXA is comparable between models as long as the same parallel dataset and setting is used to obtain the MEXA scores. Figure 3 shows that in many languages, particularly high-resource languages, Llama 3.1 achieves a significantly higher alignment score than its predecessor, Llama 1. Although Llama 3.1 leads to better alignment scores with English for medium and low-resource languages, there is still room for improvement. Comparing Arabic (arb_Arab) with its romanized ver-
<figure><img src="image_8.png" /><figcaption>Figure 3: Llama 1 vs. Llama 3.1 MEXA alignment score for different languages across all layers. Best performance markers in order: △, □, *, ×, ○, _</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 4: Llama 3.1 t-SNE plots for 3 different layers. As shown, in the mid-layers, the embeddings become more language-neutral. The numbers shown in the mid-layers are the IDs of English sentences that are scattered.</figcaption></figure>

sion (arb_Latn), we see that both Llama 1 and Llama 3.1 models perform better in the native script than in the Latin script, even though Llama 1's tokenizer for Arabic is essentially a character-based tokenizer. In general, for very low-resource languages, those in Latin script tend to have higher alignment scores, likely because the tokenization is more favorable for Latin characters.

In Figure 4, we display the t-SNE (Van der Maaten and Hinton, 2008) plots of the embeddings of Figure 3 from 3 different layers of Llama 3.1: embedding layer 0, mid-layer 13, and last layer 32. We assign a different color to each language. For layers 0 and 32, the embeddings are more language-specific, while in the mid-layer, they become more language-neutral. Languages that maintain their language-specific embeddings in the mid-layer are clustered separately and, notably, correspond to the very low-resource languages that receive the lowest alignment scores from MEXA.

# 6 Conclusion

We introduce MEXA, a method for assessing the multilingual capabilities of English-centric large language models (LLMs). MEXA builds on the observation that English-centric LLMs semantically use English as a kind of pivot language in their intermediate layers. MEXA computes the alignment between non-English languages and English using

parallel sentences, estimating the transfer of language understanding capabilities from English to other languages through this alignment. This metric can be useful in estimating task performance, provided we know the English performance in the task and the alignment score between languages derived from a parallel dataset. Through different studies with two parallel datasets (FLORES-200 and the Bible), different LLMs including the Llama family, Gemma family, Mistral, and OLMo, and three downstream tasks (Belebele, m-MMLU, and m-ARC), we demonstrated that MEXA provides a reliable estimation of multilingual performance. For MEXA score calculations, multiple design analyses are conducted to explore the impact of token-level pooling for embeddings and layer-level pooling in computing alignment scores. While MEXA shows high correlation across most configurations, a weighted average of tokens combined with mean pooling delivers the best results. The results reveal a promising average value for the Pearson correlation coefficient of 0.90 with established downstream tasks across nine models and two parallel datasets. Overall, MEXA proves to be a valuable method for practitioners aiming to assess the multilingual capabilities of English-centric LLMs, easing future efforts to expand these models to a wider range of underrepresented languages.
## Limitations

We are aware of four main limitations of our work.

First, the scope of this paper is limited to non-generative tasks. Generation is generally more challenging than understanding, and it is unsurprising that these models for many languages may struggle to generate content in their language. While NLP has advanced toward generative capabilities, a significant portion of evaluation still focuses on non-generative tasks (e.g., Hugging Face Leaderboards)² due to their convenience in multiple-choice question evaluation and standardized metrics. Assessing generated output remains challenging, even in English benchmarks. For example, model-based approaches (e.g., “LLM-as-a-judge” (Zheng et al., 2023)) require an LLM fully competent in the target language—a capability that is both questionable and the focus of our evaluation.

Second, MEXA provides a method of evaluation for open science, and only model weights are needed. Although, developers of closed-source models could use MEXA under the hood and report their multilingual results to provide insight of their model’s multilingual capabilities. We target the widely used settings where the LLM follows a decoder-only transformer architecture. For other architectures, as long as we can extract the embedding given a sentence for intermediate layers, MEXA can be calculated.

Third, we present a selection of tasks for multilingual evaluation in Table 5. As shown, for non-generative tasks, only a few benchmarks support a high number of languages, including low-resource ones. Benchmarks limited to around 10 languages, which mostly support high-resource languages, would not support our claims, as MEXA would achieve high results for all of them. Belebele includes the highest number of languages (except sequence labeling tasks), making it an ideal task to evaluate MEXA. Both m-MMLU and m-ARC are the next highest covered languages for non-generative tasks. However, since they are machine-translated tasks, they are not ideal and may bias some results for low-resource languages (or, more accurately, when the machine translation is poor). Yet, these translated versions are representative of the current state of automatic evaluation, as seen in

multilingual leaderboards.³

Fourth, MEXA provides a rough estimate of the multilingual capabilities of pre-trained English-centric LLMs. Different tasks offer diverse perspectives on the abilities of LLMs, and MEXA cannot replace all of them. Our goal is to highlight the multilingual potential of English-centric LLMs and propose a simple way to evaluate them. We hope this encourages the development of more multilingual LLMs, even though they are likely to contain large shares of English data. Additionally, it is important to note that answers across languages do not always need to be fully aligned (Naous et al., 2024), and for such cases, language- and culture-specific evaluation benchmarks should be developed.

## Acknowledgments

This work was funded by Deutsche Forschungsgemeinschaft (project SCHU 2246/14-1). François Yvon has been partly funded by the French National Funding Agency (ANR) under the France 2030 program (ref. ANR-23-IACL-0007).

## References

David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O. Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, Chiamaka Chukwuneke, Happy Buzaaba, Blessing Sibanda, Godson Kalipe, Jonathan Mukiibi, Salomon Kabongo, Foutse Yuehgoh, Mmasibidi Setaka, Lolwethu Ndolela, and 7 others. 2024. Irokobench: A new benchmark for African languages in the age of large language models. Preprint, arXiv:2406.03368.

Divyanshu Aggarwal, Vivek Gupta, and Anoop Kunchukuttan. 2022. IndicXNLI: Evaluating multilingual inference for Indian languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10994–11006, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2023. MEGA: Multilingual evaluation of generative AI. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4232–4267, Singapore. Association for Computational Linguistics.

Sanchit Ahuja, Divyanshu Aggarwal, Varun Gumma, Ishaan Watts, Ashutosh Sathe, Millicent Ochieng,

²hf.co/spaces/open-llm-leaderboard/open_llm_leaderboard

³hf.co/spaces/uonlp/open_multilingual_llm_leaderboard
Rishav Hada, Prachi Jain, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. 2024. MEGAVERSE: Benchmarking large language models across languages, modalities, models and tasks. In *Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)*, pages 2598–2637, Mexico City, Mexico. Association for Computational Linguistics.

Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 4623–4637.

Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele benchmark: a parallel reading comprehension dataset in 122 language variants. *Preprint*, arXiv:2308.16884.

Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji, Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Hao nan Li, and 11 others. 2024. Lessons from the trenches on reproducible evaluation of language models. *Preprint*, arXiv:2405.14782.

BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, and 1 others. 2023. BLOOM: A 176b-parameter open-access multilingual language model. *Preprint*, arXiv:2211.05100.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. *Transactions of the Association for Computational Linguistics*, 5:135–146.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020a. GPT-3 dataset language statistics. https://github.com/openai/gpt-3/tree/master/dataset_statistics.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020b. Language models are few-shot learners. *Advances in neural information processing systems*, 33:1877–1901.

Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In *International Conference on Learning Representations*.

Grzegorz Chrupała and Afra Alishahi. 2019. Correlating neural and symbolic representations of language. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 2952–2962, Florence, Italy. Association for Computational Linguistics.

Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. *Transactions of the Association for Computational Linguistics*, 8:454–470.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. *Preprint*, arXiv:1803.05457.

Common Crawl. 2024. Statistics of common crawl monthly archives. https://commoncrawl.github.io/cc-crawl-statistics/plots/languages.html.

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020a. Unsupervised cross-lingual representation learning at scale. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 8440–8451, Online. Association for Computational Linguistics.

Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In *Advances in Neural Information Processing Systems*, volume 32, pages 7059–7069. Curran Associates, Inc.

Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In *Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing*, pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.

Alexis Conneau, Shijie Wu, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2020b. Emerging cross-lingual structure in pretrained language models. In *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, pages 6022–6034, Online. Association for Computational Linguistics.

Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. *Computational Linguistics*, 47(2):255–308.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *Proceedings of the 2019 Conference of*
*the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 12402–12426, Toronto, Canada. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. *The Llama 3 herd of models*. *Preprint*, arXiv:2407.21783.

Pavel Efimov, Leonid Boytsov, Elena Arslanova, and Pavel Braslavski. 2023. The impact of cross-lingual adjustment of contextual word representations on zero-shot transfer. In *Advances in Information Retrieval*, pages 51–67, Cham. Springer Nature Switzerland.

Yanai Elazar, Akshita Bhagia, Ian Helgi Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Evan Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, Hannaneh Hajishirzi, Noah A. Smith, and Jesse Dodge. 2024. What’s in my big data? In *The Twelfth International Conference on Learning Representations*.

Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 55–65, Hong Kong, China. Association for Computational Linguistics.

Christian Federmann, Tom Kocmi, and Ying Xin. 2022. NTREX-128 – news test references for MT evaluation of 128 languages. In *Proceedings of the First Workshop on Scaling Up Multilingual Evaluation*, pages 21–24, Online. Association for Computational Linguistics.

Jay Gala, Pranjal A Chitale, AK Raghavan, Varun Gumma, Sumanth Doddapaneni, Janki Atul Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, and 1 others. 2023. *Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages*. *Transactions on Machine Learning Research*.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h,

Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2023. *A framework for few-shot language model evaluation*.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, and 1 others. 2024a. Gemma: Open models based on Gemini research and technology. *arXiv preprint arXiv:2403.08295*.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024b. *Gemma 2: Improving open language models at a practical size*. *Preprint*, arXiv:2408.00118.

Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. Khayyam challenge (PersianMMLU): Is your LLM truly wise to the Persian language? *Preprint*, arXiv:2404.06644.

Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, and 1 others. 2024. OLMo: Accelerating the science of language models. *arXiv preprint arXiv:2402.00838*.

Katharina Hämmerl, Alina Fastowski, Jindřich Libovický, and Alexander Fraser. 2023. Exploring anisotropy and outliers in multilingual language models for cross-lingual semantic sentence similarity. In *Findings of the Association for Computational Linguistics: ACL 2023*, pages 7023–7037, Toronto, Canada. Association for Computational Linguistics.

Katharina Hämmerl, Jindřich Libovický, and Alexander Fraser. 2024. Understanding cross-lingual Alignment—A survey. In *Findings of the Association for Computational Linguistics ACL 2024*, pages 10922–10943, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021*, pages 4693–4703, Online. Association for Computational Linguistics.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. *Preprint*, arXiv:2009.03300.

Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411–4421. PMLR.

Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2485–2494, Hong Kong, China. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023a. Mistral 7b. arXiv preprint arXiv:2310.06825.

Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang. 2023b. Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293. Online. Association for Computational Linguistics.

Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schuetze. 2023. GlotLID: Language identification for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6155–6218, Singapore. Association for Computational Linguistics.

Amir Hossein Kargaran, François Yvon, and Hinrich Schütze. 2024. GlotScript: A resource and tool for low resource writing system identification. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7774–7784, Torino, Italia. ELRA and ICCL.

Tannon Kew, Florian Schottmann, and Rico Sennrich. 2024. Turning English-centric LLMs into polyglots: How much multilinguality is needed? In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13097–13124, Miami, Florida, USA. Association for Computational Linguistics.

Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of Machine Translation Summit X: Papers, pages 79–86, Phuket, Thailand.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3519–3529. PMLR.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024. ArabicMMLU: Assessing massive multitask language understanding in arabic. Preprint, arXiv:2402.12840.

Saurabh Kulshreshtha, Jose Luis Redondo Garcia, and Ching-Yun Chang. 2020. Cross-lingual alignment methods for multilingual BERT: A comparative study. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 933–942, Online. Association for Computational Linguistics.

Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034–4048, Online. Association for Computational Linguistics.

Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023a. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics.

Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. 2023b. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics.

Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, and 1 others. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. Advances in Neural Information Processing Systems, 35:31809–31826.

Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7315–7330.

Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. 2024a. Improving in-context learning of
multilingual generative language models with cross-
lingual alignment. In Proceedings of the 2024 Con-
ference of the North American Chapter of the Asso-
ciation for Computational Linguistics: Human Lan-
guage Technologies (Volume 1: Long Papers), pages
8058–8076, Mexico City, Mexico. Association for
Computational Linguistics.

Daoyang Li, Mingyu Jin, Qingcheng Zeng, Haiyan
Zhao, and Mengnan Du. 2024b. Exploring multi-
lingual probing in large language models: A cross-
language analysis. *Preprint*, arXiv:2409.14459.

Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai
Zhao, Yeyun Gong, Nan Duan, and Timothy Bald-
win. 2024c. CMMLU: Measuring massive multi-
task language understanding in Chinese. Preprint,
arXiv:2306.09212.

Jiahuan Li, Shujian Huang, Xinyu Dai, and Jiajun Chen.
2024d. PreAlign: Boosting cross-lingual transfer
by early establishment of multilingual alignment.
Preprint, arXiv:2407.16222.

Xianming Li and Jing Li. 2024. AoE: Angle-optimized embeddings for semantic textual similarity. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1825–1839, Bangkok, Thailand. Association for Computational Linguistics.

Xiaochen Li, Zheng-Xin Yong, and Stephen H Bach.
2024e. Preference tuning for toxicity mitigation generalizes across languages. *arXiv preprint*
*arXiv:2406.16235*.

Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ali Payani, Ninghao Liu, and Mengnan Du. 2025. Language ranker: A metric for quantifying llm performance across high and low-resource languages. In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pages 28186–28194.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu
Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na-
man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth
Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav
Chaudhary, Brian O'Horo, Jeff Wang, Luke Zettle-
moyer, Zornitsa Kozareva, Mona Diab, and 2 others.
2022. Few-shot learning with multilingual generative
language models. In *Proceedings of the 2022 Con-
ference on Empirical Methods in Natural Language
Processing*, pages 9019–9052, Abu Dhabi, United
Arab Emirates. Association for Computational Lin-
guistics.

Pierre Lison and Jörg Tiedemann. 2016. OpenSub-
titles2016: Extracting large parallel corpora from
movie and TV subtitles. In Proceedings of the Tenth
International Conference on Language Resources
and Evaluation (LREC’16), pages 923–929, Portorož,
Slovenia. European Language Resources Association
(ELRA).

Weihao Liu, Ning Wu, Wenbiao Ding, Shining Liang, Ming Gong, and Dongmei Zhang. 2024a. Towards truthful multilingual large language models: Benchmarking and alignment strategies. Preprint, arXiv:2406.14434.

Yihong Liu, Mingyang Wang, Amir Hossein Kargaran, Ayyoob Imani, Orgest Xhelili, Haotian Ye, Chun-lan Ma, François Yvon, and Hinrich Schütze. 2024b. How transliterations improve crosslingual alignment. Preprint, arXiv:2409.17326.

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage text retrieval. In *Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval*, pages 2421–2425.

Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis
Aumiller, Ahmet Üstün, Sara Hooker, and Sebastian
Ruder. 2024. How does quantization affect multilin-
gual LLMs? Preprint, arXiv:2407.03211.

Thomas Mayer and Michael Cysouw. 2014. Creating a massively parallel Bible corpus. In *Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)*, pages 3158–3163, Reykjavik, Iceland. European Language Resources Association (ELRA).

Niklas Muennighoff. 2022. SGPT: GPT sentence embeddings for semantic search. arXiv preprint arXiv:2202.08904.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hai-ley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Al-banie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generaliza-tion through multitask finetuning. In *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.

Benjamin Muller, Yanai Elazar, Benoît Sagot, and Djamé Seddah. 2021. First align, then predict: Understanding the cross-lingual ability of multilingual BERT. In *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume*, pages 2214–2231, Online. Association for Computational Linguistics.

Tarek Naous, Michael Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.

Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming
Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, and 6 others. 2022. Text and code embeddings by contrastive pre-training. Preprint, arXiv:2201.10005.

NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, and 20 others. 2022. No language left behind: Scaling human-centered machine translation. Preprint, arXiv:2207.04672.

Odunayo Ogundepo, Tajuddeen Gwadabe, Clara Rivera, Jonathan Clark, Sebastian Ruder, David Adelani, Bonaventure Dossou, Abdou Diop, Claytone Sikasote, Gilles Hacheme, Happy Buzaaba, Ignatius Ezeani, Rooweither Mabuya, Salomey Osei, Chris Emezue, Albert Kahira, Shamsuddeen Muhammad, Akintunde Oladipo, Abraham Owodunni, and 25 others. 2023. Cross-lingual open-retrieval question answering for African languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14957–14972, Singapore. Association for Computational Linguistics.

OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt.

OpenAI. 2024. Multilingual massive multitask language understanding (MMMLU).

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.

Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958.

Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common-sense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362–2376, Online. Association for Computational Linguistics.

Sara Rajaee and Mohammad Taher Pilehvar. 2022. An isotropy analysis in the multilingual BERT embedding space. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1309–1316, Dublin, Ireland. Association for Computational Linguistics.

Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4512–4525, Online. Association for Computational Linguistics.

Anton Schäfer, Shauli Ravfogel, Thomas Hofmann, Tiago Pimentel, and Imanol Schlag. 2024. The role of language imbalance in cross-lingual generalisation: Insights from cloned language experiments. Preprint, arXiv:2404.07982.

Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, and 14 others. 2024. Aya dataset: An open-access collection for multilingual instruction tuning. Preprint, arXiv:2402.06619.

Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogen, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, and 17 others. 2024. Dolma: an open corpus of three trillion tokens for language model pretraining research. Preprint, arXiv:2402.00159.

Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. 2024. KMMLU: Measuring massive multitask language understanding in Korean. Preprint, arXiv:2402.11548.

Stanford CRFM. 2024. Holistic evaluation of language models - MMLU leaderboard. https://crfm.stanford.edu/helm/mmlu/latest/#/leaderboard.

Tatoeba Community. 2006. Tatoeba collection. https://tatoeba.org/en/downloads.

Atula Tejaswi, Nilesh Gupta, and Eunsol Choi. 2024. Exploring design choices for building language-specific LLMs. arXiv preprint arXiv:2406.14670.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. *Journal of machine learning research*, 9(11).

W3Techs. 2024. Usage statistics of content lan-
guages for websites. https://w3techs.com/
technologies/overview/content_language.

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving text embeddings with large language models. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 11897–11916, Bangkok, Thailand. Association for Computational Linguistics.

Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do Llamas work in English? on the latent language of multilingual transformers. *Preprint*, arXiv:2402.10588.

Yinfei Yang, Yuan Zhang, Chris Tar, and Jason
Baldrige. 2019. PAWS-X: A cross-lingual adversar-
ial dataset for paraphrase identification. In *Proceed-
ings of the 2019 Conference on Empirical Methods
in Natural Language Processing and the 9th Inter-
national Joint Conference on Natural Language Pro-
cessing (EMNLP-IJCNLP)*, pages 3687–3692, Hong
Kong, China. Association for Computational Linguis-
tics.

Jiacheng Ye, Xijia Tao, and Lingpeng Kong. 2023. Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability. *Preprint*, arXiv:2306.06688.

Arda Yüksel, Abdullatif Köksal, Lütfi Kerem Şenel, Anna Korhonen, and Hinrich Schütze. 2024. Turkish-MMLU: Measuring massive multitask language understanding in Turkish. *Preprint*, arXiv:2407.12402.

Biao Zhang, Philip Williams, Ivan Titov, and Rico Sen-
nrich. 2020. Improving massively multilingual neu-
ral machine translation and zero-shot translation. In
*Proceedings of the 58th Annual Meeting of the Asso-
ciation for Computational Linguistics*, pages 1628–
1639, Online. Association for Computational Linguis-
tics.

Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. 2023a. M3Exam: A multilingual, multimodal, multilevel benchmark for examining large language models. In *Advances in Neural Information Processing Systems*, volume 36, pages 5484–5505. Curran Associates, Inc.

Zhen-Ru Zhang, Chuanqi Tan, Songfang Huang, and Fei Huang. 2023b. VECO 2.0: Cross-lingual language model pre-training with multi-granularity contrastive learning. Preprint, arXiv:2304.08205.

Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. *Advances in Neural Information Processing Systems*, 36:46595–46623.

Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng
Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki,
and Sadao Kurohashi. 2024. Beyond english-centric
LLMs: What language do multilingual language
models think in? *Preprint*, arXiv:2408.10811.

Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024. Question translation training for better multilingual reasoning. *Preprint*, arXiv:2401.07817.

A Appendix

A.1 Distribution of Pre-training Data in LLMs

The distribution of languages in the training data of
state-of-the-art LLMs is rarely fully documented.
Llama 2 (Touvron et al., 2023b) is a counter-
example and its authors have disclosed the lan-
guage distribution use in pre-training. Their analy-
sis uses the FastText (Bojanowski et al., 2017) lan-
guage identification tool and a threshold of 0.5 for
the language detection. We reproduce Touvron et al.
(2023b, Table 10), which lists 27 languages with
percentages greater than 0.005% in the Llama 2
pre-training data, in Table 4. English, with 89.70%,
constitutes the vast majority of the training data.

All the languages listed in Table 4 have a pres-
ence of more than 0.10% (top 35 languages) on the
web according to the W3Techs report (W3Techs,
2024) or more than 0.15% (top 36 languages) ac-
cording to CommonCrawl (first three snapshots
of 2024) (Common Crawl, 2024). However, not
all of the most prevalent languages on the web
appear in Table 4. The following 9 languages
are missing, most of which use non-Latin writing
systems: Turkish (tur_Latn), Persian (pes_Arab),
Arabic (ara_Arab), Greek (ell_Grek), Hebrew
(heb_Hebr), Thai (tha_Thai), Hindi (hin_Deva),
Slovak (slk_Latn), and Lithuanian (lit_Latn).

The distribution of data in the training of English-
centric LLMs is not the same as on the web, but it
does have some correlation. The amount of English
in LLM pre-training data is significantly larger than
for other languages. This is also observable for
GPT-3 (Brown et al., 2020b), where more than 92%
of the training texts was in English (Brown et al.,
2020a). The rest of the top languages in the data of
such models are mostly high-resource languages,
<table><caption>Table 4: Language distribution in the pre-training data for Llama 2. The large “Unknown” category is partially composed of programming code data. Common scripts are sourced from the GlotScript resource (Kargaran et al., 2024).</caption><thead><tr><th>Language</th><th>Script</th><th>Percent</th><th>Language</th><th>Script</th><th>Percent</th></tr></thead><tbody><tr><td>English (eng)</td><td>Latn</td><td>89.70%</td><td>Ukrainian (ukr)</td><td>Cyrl</td><td>0.07%</td></tr><tr><td>Unknown (unk)</td><td>-</td><td>8.38%</td><td>Korean (kor)</td><td>Hang</td><td>0.06%</td></tr><tr><td>German (deu)</td><td>Latn</td><td>0.17%</td><td>Catalan (cat)</td><td>Latn</td><td>0.04%</td></tr><tr><td>French (fra)</td><td>Latn</td><td>0.16%</td><td>Serbian (srp)</td><td>Cyrl/Latn</td><td>0.04%</td></tr><tr><td>Swedish (swe)</td><td>Latn</td><td>0.15%</td><td>Indonesian (ind)</td><td>Latn</td><td>0.03%</td></tr><tr><td>Chinese (zho)</td><td>Hans/Hant</td><td>0.13%</td><td>Czech (ces)</td><td>Latn</td><td>0.03%</td></tr><tr><td>Spanish (spa)</td><td>Latn</td><td>0.13%</td><td>Finnish (fin)</td><td>Latn</td><td>0.03%</td></tr><tr><td>Russian (rus)</td><td>Cyrl</td><td>0.13%</td><td>Hungarian (hun)</td><td>Latn</td><td>0.03%</td></tr><tr><td>Dutch (nld)</td><td>Latn</td><td>0.12%</td><td>Norwegian (nor)</td><td>Latn</td><td>0.03%</td></tr><tr><td>Italian (ita)</td><td>Latn</td><td>0.11%</td><td>Romanian (ron)</td><td>Latn</td><td>0.03%</td></tr><tr><td>Japanese (jpn)</td><td>Jpan</td><td>0.10%</td><td>Bulgarian (bul)</td><td>Cyrl</td><td>0.02%</td></tr><tr><td>Polish (pol)</td><td>Latn</td><td>0.09%</td><td>Danish (dan)</td><td>Latn</td><td>0.02%</td></tr><tr><td>Portuguese (por)</td><td>Latn</td><td>0.09%</td><td>Slovenian (slv)</td><td>Latn</td><td>0.01%</td></tr><tr><td>Vietnamese (vie)</td><td>Latn</td><td>0.08%</td><td>Croatian (hrv)</td><td>Latn</td><td>0.01%</td></tr></tbody></table>

which have the most available data on the web (top 36 languages). However, in some models, this could be adjusted by design, for example, to make writing systems with non-Latin languages less prominent (as seen in Llama 2). This weakens the correlation between LLMs' pre-training data and the web.

## A.2 Multilingual Evaluation Benchmarks

Multilingual evaluation methods and the development of benchmarks not only facilitate the assessment of diverse language representations in LLMs but also help in monitoring cross-lingual generalization, to assess the effect of quantization across multiple languages (Marchisio et al., 2024), the development of language-specific models (Tejaswi et al., 2024), and the optimization of safety preferences (Li et al., 2024e), among others. In Table 5, we list benchmarks with the largest language coverage. This list includes benchmarks referenced by MEGA (Ahuja et al., 2023), MEGAVERSE (Ahuja et al., 2024), xP3 (Muennighoff et al., 2023), the Aya collection (Singh et al., 2024), the lm-evaluation-harness framework (Gao et al., 2023; Biderman et al., 2024), and inter alia. These datasets comprise a mix of translated datasets, some human-translated or verified by native speakers such as AfriXNLI (Adelani et al., 2024) and some relying only on machine translation Lai et al. (2023b). Additionally, there are datasets created independently for each language, such as XLSum (Hasan et al., 2021), where the data is not parallel and the size of the data varies between languages. Despite the efforts reflected in Table 5, the community is still lacking highly multilingual benchmarks for tasks such as natural language understanding or text generation.

<table><caption>Table 5: Multilingual evaluation benchmarks: MC stands for multiple-choice. # L shows the number of languages supported by each dataset.</caption><thead><tr><th>Dataset</th><th>Task</th><th># L</th></tr></thead><tbody><tr><td>XNLI (Conneau et al., 2018)</td><td>Natural Language Inference</td><td>15</td></tr><tr><td>IndicXNLI (Aggarwal et al., 2022)</td><td>Natural Language Inference</td><td>11</td></tr><tr><td>AfriXNLI (Adelani et al., 2024)</td><td>Natural Language Inference</td><td>15</td></tr><tr><td>m_HellaSwag (Lai et al., 2023b)</td><td>Natural Language Inference</td><td>31</td></tr><tr><td>PAWS-X (Yang et al., 2019)</td><td>Paraphrase Identification</td><td>7</td></tr><tr><td>XCOPA (Ponti et al., 2020)</td><td>Commonsense Reasoning</td><td>11</td></tr><tr><td>XStoryCloze (Lin et al., 2022)</td><td>Commonsense Reasoning</td><td>11</td></tr><tr><td>m-ARC (Lai et al., 2023b)</td><td>Common Sense Reasoning</td><td>31</td></tr><tr><td>TyDiQA (Clark et al., 2020)</td><td>Question Answering</td><td>11</td></tr><tr><td>MLQA (Lewis et al., 2020)</td><td>Question Answering</td><td>7</td></tr><tr><td>XQuAD (Artetxe et al., 2020)</td><td>Question Answering</td><td>11</td></tr><tr><td>IndicQA (Doddapaneni et al., 2023)</td><td>Question Answering</td><td>10</td></tr><tr><td>AfriQA (Ogundepo et al., 2023)</td><td>Question Answering</td><td>10</td></tr><tr><td>m_TruthfulQA (Lai et al., 2023b)</td><td>MC General Question Answering</td><td>31</td></tr><tr><td>UDPOS 2.7 (de Marneffe et al., 2021)</td><td>Part of Speech Tagging</td><td>104</td></tr><tr><td>WikiANN (Pan et al., 2017)</td><td>Name Entity Recognition</td><td>282</td></tr><tr><td>XLSum (Hasan et al., 2021)</td><td>Summarization</td><td>44</td></tr><tr><td>WikiLingua (Ladhak et al., 2020)</td><td>Summarization</td><td>18</td></tr><tr><td>Belebele (Bandarkar et al., 2024)</td><td>MC Reading Comprehension</td><td>115</td></tr><tr><td>AfriMMLU (Adelani et al., 2024)</td><td>MC Knowledge Question Answering</td><td>17</td></tr><tr><td>m-MMLU (Lai et al., 2023b)</td><td>MC Knowledge Question Answering</td><td>31</td></tr><tr><td>MMMLU (OpenAI, 2024)</td><td>MC Knowledge Question Answering</td><td>15</td></tr><tr><td>M3Exam (Zhang et al., 2023a)</td><td>MC Multimodal Question Answering</td><td>9</td></tr></tbody></table>

## A.3 Semantic Similarity in Multilingual Embeddings

There are other ways to compute similarity between languages, such as Representational Similarity Analysis (RSA) (Chrupała and Alishahi, 2019) and Central Kernel Alignment (CKA) (Kornblith et al., 2019). RSA involves first computing the cosine similarity for sentence embeddings within each language, then correlating these in-language similarities with those in other languages. CKA, another metric, is adopted by Conneau et al. (2020b) and Muller et al. (2021). Conneau et al. (2020b) show that the CKA similarity is highly correlated with sentence retrieval scores for four languages. In this paper, our focus is not on finding different ways to calculate similarity between languages, but on how helpful a properly defined alignment score can be in estimating the multilingual capabilities of LLMs across multiple languages.

## A.4 Benchmark Details

Belebele is a multiple-choice reading comprehension task designed to assess language models across a range of high-, medium-, and low-resource languages. Each question in the dataset is paired with four possible answers and linked to a brief passage from the FLORES-200 dataset (NLLB Team et al., 2022). The human annotation process was carefully curated to generate questions that effectively differentiate between various levels of language comprehension, supported by rigorous quality checks. Belebele supports 122 distinct labels (language-script combinations) corresponding to 115 distinct languages. However, FLORES-200 does not sup-
port 5 of these labels, corresponding to Romanized
versions of 5 Indic languages. Therefore, we con-
ducted our analysis between the FLORES-200 and
Belebele benchmarks on 117 common labels. Ad-
ditionally, there are 102 common labels between
the Bible parallel data and the Belebele benchmark.

Both ARC Challenge (Clark et al., 2018) and MMLU (Hendrycks et al., 2021) are also set up as multiple-choice question-answering tasks, but they focus on different types of knowledge and reasoning skills. ARC Challenge is classified as a common-sense reasoning task, consisting of grade-school level science questions, while MMLU consists of questions across a wide range of fields, including the humanities, social sciences, and more. Lai et al. (2023b) used GPT-3.5-turbo (OpenAI, 2022) and a translation prompt to translate examples from both datasets and create m-ARC and m-MMLU in 31 languages (excluding English). Later, m-MMLU was expanded to also include Icelandic (isl_Latn) and Norwegian (nob_Latn). The Icelandic portion was translated using the Mideind.is, while Norwegian was generated with DeepL.com.<fnref n="4" /> m-MMLU consists of 277 questions in its training set, 13,258 in the test set and 1,433 in the validation set. m-ARC consists of 1,116 questions in the training set, 1,169 in the test set, and 298 in the validation set.

A.5 Detailed Results

We show the detailed per model results of Table 2
in Table 6.

A.6 MEXA for FLORES-200

We compute MEXA with weighted average em-
bedding and max pooling for the FLORES parallel
data for 203 language labels, multiplied by the per-
formance of Belebele for each model in English.
We show the results in Table 7, and color the cells
based on 0.2 intervals from green (well-covered)
to red (not covered): (1.0-0.8), (0.8-0.6), (0.6-0.4),
(0.4-0.2), (0.2-0). Note that although FLORES is
a high-quality, human-translated dataset, we ad-
dressed two major issues before proceeding, as
noted by Kargaran et al. (2023). First, the data
labeled as Cantonese (Yue Chinese) is not actually
Cantonese, so we removed it. Second, the code
for Central Atlas Tamazight (tzm), which actually
refers to Standard Moroccan Tamazight (zgh), was
renamed accordingly. As Belebele is relatively an

easy task since the models get good scores in En-
glish, and we are using max pooling, this gives a
high estimate of the coverage the LLMs have. If
the score for a language is not very high, it likely
indicates that for more challenging tasks, it will
remain low. In Table 7, we can clearly see that
Llama 3.1-70B and Gemma 2-9B show a higher
level of multilinguality than other models.

<footnote n="4">⁴hf.co/datasets/alexandrainst/m_mmlu</footnote>
<table><caption>Table 6: Pearson correlation of MEXA using FLORES and Bible data across three tasks. $\rho$ ($\mu_{Pooling}$, Task) is the correlation of MEXA for the corresponding pooling strategy and benchmark. In all settings except **, the p-value is $p < 0.001$. The best average correlations for each task are in **bold**, and the second bests are **underlined**.</caption><thead><tr><th></th><th></th><th>Gemma 2<br>9B</th><th>Gemma 1<br>7B</th><th>Llama 3.1<br>70B</th><th>Llama 3.1<br>8B</th><th>Llama 3<br>8B</th><th>Llama 2<br>7B</th><th>Llama 1<br>7B</th><th>Mistral 0.3<br>7B</th><th>OLMo 1.7<br>7B</th><th>AVG</th></tr></thead><tbody><tr><td rowspan="10">FLORES<br>weighted average</td><td>ρ (μ<sub>Mean</sub>, Belebele)</td><td>0.9247</td><td>0.9421</td><td>0.8291</td><td>0.9478</td><td>0.9588</td><td>0.8364</td><td>0.8404</td><td>0.9732</td><td>0.8425</td><td>0.8994</td></tr><tr><td>ρ (μ<sub>Max</sub>, Belebele)</td><td>0.9623</td><td>0.9676</td><td>0.9211</td><td>0.9392</td><td>0.9326</td><td>0.8362</td><td>0.7649</td><td>0.9448</td><td>0.9198</td><td><strong>0.9098</strong></td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-MMLU)</td><td>0.9342</td><td>0.9697</td><td>0.9362</td><td>0.9689</td><td>0.9647</td><td>0.9223</td><td>0.9406</td><td>0.9857</td><td>0.9393</td><td><strong>0.9513</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, m-MMLU)</td><td>0.9060</td><td>0.9596</td><td>0.8946</td><td>0.9003</td><td>0.8892</td><td>0.9386</td><td>0.8936</td><td>0.9311</td><td>0.9565</td><td>0.9188</td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-ARC)</td><td>0.9741</td><td>0.9706</td><td>0.9374</td><td>0.9515</td><td>0.9562</td><td>0.9052</td><td>0.9268</td><td>0.9693</td><td>0.8630</td><td><strong>0.9393</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, m-ARC)</td><td>0.9187</td><td>0.9499</td><td>0.8736</td><td>0.8582</td><td>0.8663</td><td>0.9297</td><td>0.8439</td><td>0.9001</td><td>0.8298</td><td>0.8856</td></tr><tr><td rowspan="10">last token</td><td>ρ (μ<sub>Mean</sub>, Belebele)</td><td>0.8997</td><td>0.9326</td><td>0.8491</td><td>0.9494</td><td>0.9581</td><td>0.9141</td><td>0.8340</td><td>0.9679</td><td>0.9467</td><td><strong>0.9168</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, Belebele)</td><td>0.9225</td><td>0.9309</td><td>0.9127</td><td>0.9244</td><td>0.9123</td><td>0.9125</td><td>0.7693</td><td>0.9460</td><td>0.9218</td><td>0.9058</td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-MMLU)</td><td>0.9086</td><td>0.9637</td><td>0.9370</td><td>0.9687</td><td>0.9690</td><td>0.9771</td><td>0.9301</td><td>0.9659</td><td>0.9700</td><td><strong>0.9545</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, m-MMLU)</td><td>0.8448</td><td>0.9297</td><td>0.8645</td><td>0.9224</td><td>0.9177</td><td>0.9699</td><td>0.8902</td><td>0.9161</td><td>0.9649</td><td>0.9134</td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-ARC)</td><td>0.9190</td><td>0.9541</td><td>0.9524</td><td>0.9536</td><td>0.9617</td><td>0.9390</td><td>0.9146</td><td>0.9451</td><td>0.7356</td><td><strong>0.9195</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, m-ARC)</td><td>0.8569</td><td>0.9147</td><td>0.9005</td><td>0.8944</td><td>0.8879</td><td>0.9464</td><td>0.8263</td><td>0.8859</td><td>0.7037</td><td>0.8685</td></tr><tr><td rowspan="10">Bible<br>weighted average</td><td>ρ (μ<sub>Mean</sub>, Belebele)</td><td>0.8360</td><td>0.8530</td><td>0.7909</td><td>0.8781</td><td>0.8974</td><td>0.8982</td><td>0.8404</td><td>0.9118</td><td>0.7410</td><td><strong>0.8496</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, Belebele)</td><td>0.8863</td><td>0.9001</td><td>0.8851</td><td>0.9242</td><td>0.9302</td><td>0.8926</td><td>0.8230</td><td>0.9337</td><td>0.7549</td><td><strong>0.8811</strong></td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-MMLU)</td><td>0.8051</td><td>0.8886</td><td>0.8958</td><td>0.9096</td><td>0.8964</td><td>0.9252</td><td>0.9159</td><td>0.9093</td><td>0.7944</td><td><strong>0.8823</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, m-MMLU)</td><td>0.5501</td><td>0.8831</td><td>0.7748</td><td>0.8683</td><td>0.8364</td><td>0.9180</td><td>0.9085</td><td>0.9107</td><td>0.7388</td><td><strong>0.8210</strong></td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-ARC)</td><td>0.8505</td><td>0.8998</td><td>0.9188</td><td>0.9267</td><td>0.9116</td><td>0.8940</td><td>0.9208</td><td>0.9317</td><td>0.8623</td><td><strong>0.9018</strong></td></tr><tr><td>ρ (μ<sub>Max</sub>, m-ARC)</td><td>0.6070</td><td>0.8803</td><td>0.8030</td><td>0.8769</td><td>0.8552</td><td>0.8684</td><td>0.8879</td><td>0.9178</td><td>0.8220</td><td><strong>0.8354</strong></td></tr><tr><td rowspan="10">last token</td><td>ρ (μ<sub>Mean</sub>, Belebele)</td><td>0.7656</td><td>0.8005</td><td>0.5944</td><td>0.7934</td><td>0.8396</td><td>0.9046</td><td>0.8299</td><td>0.9177</td><td>0.8866</td><td>0.8147</td></tr><tr><td>ρ (μ<sub>Max</sub>, Belebele)</td><td>0.7844</td><td>0.8299</td><td>0.5264</td><td>0.8000</td><td>0.8100</td><td>0.9047</td><td>0.8048</td><td>0.9235</td><td>0.8796</td><td>0.8070</td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-MMLU)</td><td>0.7194</td><td>0.7646</td><td>0.6472</td><td>0.6068</td><td>0.6516</td><td>0.8827</td><td>0.8692</td><td>0.8672</td><td>0.8060</td><td>0.7572</td></tr><tr><td>ρ (μ<sub>Max</sub>, m-MMLU)</td><td>0.7075</td><td>0.6886</td><td>0.5037**</td><td>0.5228**</td><td>0.4461**</td><td>0.9079</td><td>0.8576</td><td>0.8643</td><td>0.7994</td><td>0.6998</td></tr><tr><td>ρ (μ<sub>Mean</sub>, m-ARC)</td><td>0.7411</td><td>0.7754</td><td>0.6592</td><td>0.5976</td><td>0.6494</td><td>0.8537</td><td>0.8537</td><td>0.8927</td><td>0.6997</td><td>0.7469</td></tr><tr><td>ρ (μ<sub>Max</sub>, m-ARC)</td><td>0.7293</td><td>0.7000</td><td>0.5190**</td><td>0.5335**</td><td>0.4853**</td><td>0.8494</td><td>0.8309</td><td>0.8624</td><td>0.6867</td><td>0.6885</td></tr></tbody></table>
<table><thead><tr><th></th><th>Gemma 2<br>9B</th><th>Gemma 1<br>7B</th><th>Llama 3.1<br>70B</th><th>Llama 3.1<br>8B</th><th>Llama 3<br>8B</th><th>Llama 2<br>7B</th><th>Llama 1<br>7B</th><th>Mistral<br>7B</th><th>OLMo<br>7B</th><th>AVG</th></tr></thead><tbody><tr><td>eng_Latn</td><td>0.92</td><td>0.85</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.48</td><td>0.42</td><td>0.84</td><td>0.77</td><td>0.77</td></tr><tr><td>fra_Latn</td><td>0.92</td><td>0.84</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.37</td><td>0.41</td><td>0.84</td><td>0.70</td><td>0.75</td></tr><tr><td>por_Latn</td><td>0.92</td><td>0.84</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.41</td><td>0.41</td><td>0.84</td><td>0.63</td><td>0.75</td></tr><tr><td>deu_Latn</td><td>0.92</td><td>0.84</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.35</td><td>0.42</td><td>0.84</td><td>0.65</td><td>0.74</td></tr><tr><td>spa_Latn</td><td>0.92</td><td>0.83</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.37</td><td>0.42</td><td>0.84</td><td>0.56</td><td>0.74</td></tr><tr><td>ita_Latn</td><td>0.92</td><td>0.83</td><td>0.92</td><td>0.88</td><td>0.87</td><td>0.35</td><td>0.42</td><td>0.84</td><td>0.56</td><td>0.73</td></tr><tr><td>cat_Latn</td><td>0.92</td><td>0.82</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.39</td><td>0.42</td><td>0.84</td><td>0.50</td><td>0.73</td></tr><tr><td>nld_Latn</td><td>0.92</td><td>0.82</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.34</td><td>0.42</td><td>0.84</td><td>0.52</td><td>0.73</td></tr><tr><td>rus_Cyrl</td><td>0.91</td><td>0.82</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.34</td><td>0.41</td><td>0.83</td><td>0.51</td><td>0.72</td></tr><tr><td>zho_Hans</td><td>0.91</td><td>0.80</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.32</td><td>0.34</td><td>0.81</td><td>0.62</td><td>0.72</td></tr><tr><td>glg_Latn</td><td>0.92</td><td>0.83</td><td>0.91</td><td>0.88</td><td>0.87</td><td>0.31</td><td>0.41</td><td>0.82</td><td>0.52</td><td>0.72</td></tr><tr><td>swe_Latn</td><td>0.92</td><td>0.83</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.38</td><td>0.42</td><td>0.84</td><td>0.37</td><td>0.72</td></tr><tr><td>dan_Latn</td><td>0.92</td><td>0.83</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.31</td><td>0.41</td><td>0.82</td><td>0.44</td><td>0.71</td></tr><tr><td>ces_Latn</td><td>0.92</td><td>0.82</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.26</td><td>0.42</td><td>0.84</td><td>0.43</td><td>0.71</td></tr><tr><td>ron_Latn</td><td>0.92</td><td>0.82</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.23</td><td>0.41</td><td>0.83</td><td>0.48</td><td>0.71</td></tr><tr><td>nob_Latn</td><td>0.91</td><td>0.82</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.34</td><td>0.39</td><td>0.81</td><td>0.39</td><td>0.71</td></tr><tr><td>zho_Hant</td><td>0.91</td><td>0.81</td><td>0.94</td><td>0.88</td><td>0.87</td><td>0.31</td><td>0.32</td><td>0.79</td><td>0.52</td><td>0.71</td></tr><tr><td>pol_Latn</td><td>0.92</td><td>0.81</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.22</td><td>0.42</td><td>0.84</td><td>0.38</td><td>0.70</td></tr><tr><td>ast_Latn</td><td>0.90</td><td>0.80</td><td>0.91</td><td>0.88</td><td>0.86</td><td>0.21</td><td>0.40</td><td>0.77</td><td>0.49</td><td>0.69</td></tr><tr><td>ind_Latn</td><td>0.92</td><td>0.83</td><td>0.93</td><td>0.87</td><td>0.87</td><td>0.22</td><td>0.30</td><td>0.82</td><td>0.42</td><td>0.69</td></tr><tr><td>oci_Latn</td><td>0.89</td><td>0.75</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.22</td><td>0.39</td><td>0.81</td><td>0.40</td><td>0.68</td></tr><tr><td>bos_Latn</td><td>0.91</td><td>0.81</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.19</td><td>0.41</td><td>0.84</td><td>0.25</td><td>0.68</td></tr><tr><td>nno_Latn</td><td>0.92</td><td>0.82</td><td>0.92</td><td>0.84</td><td>0.84</td><td>0.26</td><td>0.36</td><td>0.78</td><td>0.38</td><td>0.68</td></tr><tr><td>ukr_Cyrl</td><td>0.92</td><td>0.81</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.22</td><td>0.42</td><td>0.84</td><td>0.15</td><td>0.67</td></tr><tr><td>zsm_Latn</td><td>0.92</td><td>0.83</td><td>0.93</td><td>0.88</td><td>0.87</td><td>0.17</td><td>0.25</td><td>0.81</td><td>0.36</td><td>0.67</td></tr><tr><td>hrv_Latn</td><td>0.91</td><td>0.81</td><td>0.90</td><td>0.86</td><td>0.86</td><td>0.18</td><td>0.41</td><td>0.83</td><td>0.23</td><td>0.67</td></tr><tr><td>slv_Latn</td><td>0.91</td><td>0.79</td><td>0.93</td><td>0.86</td><td>0.86</td><td>0.20</td><td>0.40</td><td>0.84</td><td>0.19</td><td>0.66</td></tr><tr><td>afr_Latn</td><td>0.91</td><td>0.81</td><td>0.93</td><td>0.87</td><td>0.87</td><td>0.20</td><td>0.37</td><td>0.79</td><td>0.21</td><td>0.66</td></tr><tr><td>slk_Latn</td><td>0.91</td><td>0.80</td><td>0.93</td><td>0.86</td><td>0.85</td><td>0.12</td><td>0.38</td><td>0.82</td><td>0.25</td><td>0.66</td></tr><tr><td>bul_Cyrl</td><td>0.91</td><td>0.80</td><td>0.90</td><td>0.86</td><td>0.86</td><td>0.12</td><td>0.42</td><td>0.84</td><td>0.14</td><td>0.65</td></tr><tr><td>jpn_Jpan</td><td>0.90</td><td>0.80</td><td>0.93</td><td>0.83</td><td>0.82</td><td>0.29</td><td>0.25</td><td>0.76</td><td>0.24</td><td>0.65</td></tr><tr><td>hun_Latn</td><td>0.91</td><td>0.78</td><td>0.92</td><td>0.84</td><td>0.83</td><td>0.13</td><td>0.39</td><td>0.81</td><td>0.18</td><td>0.64</td></tr><tr><td>vec_Latn</td><td>0.87</td><td>0.74</td><td>0.93</td><td>0.84</td><td>0.83</td><td>0.16</td><td>0.35</td><td>0.76</td><td>0.28</td><td>0.64</td></tr><tr><td>srp_Cyrl</td><td>0.91</td><td>0.79</td><td>0.90</td><td>0.86</td><td>0.86</td><td>0.10</td><td>0.42</td><td>0.84</td><td>0.06</td><td>0.64</td></tr><tr><td>tgl_Latn</td><td>0.91</td><td>0.74</td><td>0.94</td><td>0.82</td><td>0.82</td><td>0.16</td><td>0.20</td><td>0.77</td><td>0.36</td><td>0.64</td></tr><tr><td>fin_Latn</td><td>0.91</td><td>0.79</td><td>0.90</td><td>0.85</td><td>0.85</td><td>0.14</td><td>0.21</td><td>0.74</td><td>0.33</td><td>0.64</td></tr></tbody></table>

Continued on next page
<table><thead><tr><th></th><th>Gemma 2<br>9B</th><th>Gemma 1<br>7B</th><th>Llama 3.1<br>70B</th><th>Llama 3.1<br>8B</th><th>Llama 3<br>8B</th><th>Llama 2<br>7B</th><th>Llama 1<br>7B</th><th>Mistral<br>7B</th><th>OLMo<br>7B</th><th>AVG<br>7B</th></tr></thead><tbody><tr><td>mkd_Cyrl</td><td>0.90</td><td>0.77</td><td>0.94</td><td>0.87</td><td>0.86</td><td>0.07</td><td>0.38</td><td>0.80</td><td>0.11</td><td>0.63</td></tr><tr><td>vie_Latn</td><td>0.91</td><td>0.81</td><td>0.95</td><td>0.88</td><td>0.87</td><td>0.22</td><td>0.08</td><td>0.79</td><td>0.16</td><td>0.63</td></tr><tr><td>epo_Latn</td><td>0.87</td><td>0.76</td><td>0.95</td><td>0.86</td><td>0.85</td><td>0.14</td><td>0.26</td><td>0.67</td><td>0.11</td><td>0.61</td></tr><tr><td>kor_Hang</td><td>0.88</td><td>0.74</td><td>0.91</td><td>0.84</td><td>0.83</td><td>0.22</td><td>0.15</td><td>0.71</td><td>0.15</td><td>0.60</td></tr><tr><td>arb_Arab</td><td>0.91</td><td>0.80</td><td>0.94</td><td>0.86</td><td>0.85</td><td>0.05</td><td>0.17</td><td>0.70</td><td>0.10</td><td>0.60</td></tr><tr><td>ars_Arab</td><td>0.91</td><td>0.80</td><td>0.93</td><td>0.86</td><td>0.85</td><td>0.04</td><td>0.17</td><td>0.69</td><td>0.08</td><td>0.59</td></tr><tr><td>lim_Latn</td><td>0.76</td><td>0.65</td><td>0.89</td><td>0.83</td><td>0.83</td><td>0.21</td><td>0.25</td><td>0.59</td><td>0.21</td><td>0.58</td></tr><tr><td>acq_Arab</td><td>0.91</td><td>0.78</td><td>0.92</td><td>0.83</td><td>0.82</td><td>0.04</td><td>0.13</td><td>0.67</td><td>0.09</td><td>0.58</td></tr><tr><td>acm_Arab</td><td>0.90</td><td>0.76</td><td>0.90</td><td>0.86</td><td>0.82</td><td>0.04</td><td>0.14</td><td>0.67</td><td>0.09</td><td>0.57</td></tr><tr><td>fur_Latn</td><td>0.73</td><td>0.60</td><td>0.91</td><td>0.81</td><td>0.77</td><td>0.16</td><td>0.27</td><td>0.59</td><td>0.26</td><td>0.57</td></tr><tr><td>pes_Arab</td><td>0.91</td><td>0.79</td><td>0.88</td><td>0.85</td><td>0.85</td><td>0.05</td><td>0.08</td><td>0.59</td><td>0.07</td><td>0.56</td></tr><tr><td>arz_Arab</td><td>0.88</td><td>0.74</td><td>0.90</td><td>0.84</td><td>0.83</td><td>0.03</td><td>0.10</td><td>0.63</td><td>0.09</td><td>0.56</td></tr><tr><td>ajp_Arab</td><td>0.88</td><td>0.76</td><td>0.86</td><td>0.85</td><td>0.83</td><td>0.03</td><td>0.12</td><td>0.60</td><td>0.09</td><td>0.56</td></tr><tr><td>lit_Latn</td><td>0.90</td><td>0.76</td><td>0.92</td><td>0.78</td><td>0.80</td><td>0.10</td><td>0.10</td><td>0.56</td><td>0.11</td><td>0.56</td></tr><tr><td>apc_Arab</td><td>0.89</td><td>0.76</td><td>0.86</td><td>0.82</td><td>0.83</td><td>0.03</td><td>0.11</td><td>0.64</td><td>0.09</td><td>0.56</td></tr><tr><td>ell_Grek</td><td>0.90</td><td>0.78</td><td>0.87</td><td>0.87</td><td>0.86</td><td>0.02</td><td>0.09</td><td>0.58</td><td>0.05</td><td>0.56</td></tr><tr><td>tur_Latn</td><td>0.89</td><td>0.78</td><td>0.90</td><td>0.82</td><td>0.81</td><td>0.04</td><td>0.09</td><td>0.61</td><td>0.04</td><td>0.55</td></tr><tr><td>est_Latn</td><td>0.90</td><td>0.77</td><td>0.90</td><td>0.82</td><td>0.83</td><td>0.12</td><td>0.09</td><td>0.45</td><td>0.10</td><td>0.55</td></tr><tr><td>pap_Latn</td><td>0.79</td><td>0.60</td><td>0.89</td><td>0.75</td><td>0.73</td><td>0.18</td><td>0.22</td><td>0.56</td><td>0.23</td><td>0.55</td></tr><tr><td>lmo_Latn</td><td>0.73</td><td>0.56</td><td>0.87</td><td>0.75</td><td>0.74</td><td>0.17</td><td>0.26</td><td>0.60</td><td>0.26</td><td>0.55</td></tr><tr><td>szl_Latn</td><td>0.77</td><td>0.59</td><td>0.87</td><td>0.73</td><td>0.74</td><td>0.11</td><td>0.26</td><td>0.64</td><td>0.21</td><td>0.55</td></tr><tr><td>prs_Arab</td><td>0.90</td><td>0.78</td><td>0.92</td><td>0.84</td><td>0.84</td><td>0.01</td><td>0.06</td><td>0.46</td><td>0.08</td><td>0.54</td></tr><tr><td>scn_Latn</td><td>0.77</td><td>0.59</td><td>0.88</td><td>0.79</td><td>0.77</td><td>0.15</td><td>0.22</td><td>0.57</td><td>0.15</td><td>0.54</td></tr><tr><td>heb_Hebr</td><td>0.91</td><td>0.81</td><td>0.89</td><td>0.83</td><td>0.83</td><td>0.02</td><td>0.05</td><td>0.47</td><td>0.06</td><td>0.54</td></tr><tr><td>lvs_Latn</td><td>0.90</td><td>0.75</td><td>0.90</td><td>0.81</td><td>0.79</td><td>0.05</td><td>0.05</td><td>0.55</td><td>0.08</td><td>0.54</td></tr><tr><td>als_Latn</td><td>0.87</td><td>0.67</td><td>0.93</td><td>0.79</td><td>0.80</td><td>0.09</td><td>0.08</td><td>0.53</td><td>0.10</td><td>0.54</td></tr><tr><td>lij_Latn</td><td>0.74</td><td>0.58</td><td>0.88</td><td>0.72</td><td>0.70</td><td>0.16</td><td>0.25</td><td>0.53</td><td>0.30</td><td>0.54</td></tr><tr><td>ceb_Latn</td><td>0.83</td><td>0.59</td><td>0.89</td><td>0.73</td><td>0.72</td><td>0.16</td><td>0.15</td><td>0.49</td><td>0.24</td><td>0.53</td></tr><tr><td>srd_Latn</td><td>0.73</td><td>0.59</td><td>0.86</td><td>0.75</td><td>0.72</td><td>0.16</td><td>0.23</td><td>0.55</td><td>0.21</td><td>0.53</td></tr><tr><td>hin_Deva</td><td>0.90</td><td>0.74</td><td>0.91</td><td>0.80</td><td>0.79</td><td>0.03</td><td>0.05</td><td>0.44</td><td>0.06</td><td>0.53</td></tr><tr><td>ltz_Latn</td><td>0.79</td><td>0.59</td><td>0.84</td><td>0.75</td><td>0.74</td><td>0.15</td><td>0.18</td><td>0.44</td><td>0.18</td><td>0.52</td></tr><tr><td>tha_Thai</td><td>0.90</td><td>0.76</td><td>0.87</td><td>0.83</td><td>0.83</td><td>0.02</td><td>0.02</td><td>0.32</td><td>0.10</td><td>0.52</td></tr><tr><td>aeb_Arab</td><td>0.82</td><td>0.67</td><td>0.86</td><td>0.78</td><td>0.75</td><td>0.04</td><td>0.10</td><td>0.55</td><td>0.08</td><td>0.52</td></tr><tr><td>bel_Cyrl</td><td>0.88</td><td>0.65</td><td>0.88</td><td>0.79</td><td>0.79</td><td>0.02</td><td>0.09</td><td>0.50</td><td>0.02</td><td>0.51</td></tr><tr><td>isl_Latn</td><td>0.83</td><td>0.62</td><td>0.88</td><td>0.77</td><td>0.78</td><td>0.09</td><td>0.10</td><td>0.48</td><td>0.06</td><td>0.51</td></tr><tr><td>swh_Latn</td><td>0.90</td><td>0.74</td><td>0.86</td><td>0.73</td><td>0.80</td><td>0.11</td><td>0.09</td><td>0.27</td><td>0.08</td><td>0.51</td></tr><tr><td>mlt_Latn</td><td>0.88</td><td>0.63</td><td>0.87</td><td>0.74</td><td>0.74</td><td>0.12</td><td>0.11</td><td>0.38</td><td>0.12</td><td>0.51</td></tr><tr><td>war_Latn</td><td>0.76</td><td>0.55</td><td>0.88</td><td>0.65</td><td>0.61</td><td>0.26</td><td>0.20</td><td>0.35</td><td>0.20</td><td>0.49</td></tr><tr><td>cym_Latn</td><td>0.87</td><td>0.59</td><td>0.88</td><td>0.75</td><td>0.76</td><td>0.11</td><td>0.10</td><td>0.28</td><td>0.08</td><td>0.49</td></tr><tr><td>fao_Latn</td><td>0.71</td><td>0.53</td><td>0.86</td><td>0.71</td><td>0.69</td><td>0.12</td><td>0.13</td><td>0.53</td><td>0.08</td><td>0.48</td></tr><tr><td>urd_Arab</td><td>0.83</td><td>0.66</td><td>0.88</td><td>0.76</td><td>0.73</td><td>0.02</td><td>0.02</td><td>0.31</td><td>0.03</td><td>0.47</td></tr><tr><td>jav_Latn</td><td>0.75</td><td>0.54</td><td>0.84</td><td>0.69</td><td>0.67</td><td>0.16</td><td>0.12</td><td>0.29</td><td>0.16</td><td>0.47</td></tr><tr><td>eus_Latn</td><td>0.82</td><td>0.66</td><td>0.84</td><td>0.74</td><td>0.71</td><td>0.10</td><td>0.08</td><td>0.18</td><td>0.05</td><td>0.47</td></tr><tr><td>sun_Latn</td><td>0.69</td><td>0.48</td><td>0.86</td><td>0.68</td><td>0.64</td><td>0.14</td><td>0.12</td><td>0.34</td><td>0.23</td><td>0.47</td></tr><tr><td>kea_Latn</td><td>0.64</td><td>0.51</td><td>0.78</td><td>0.60</td><td>0.64</td><td>0.16</td><td>0.19</td><td>0.45</td><td>0.18</td><td>0.46</td></tr><tr><td>ary_Arab</td><td>0.71</td><td>0.60</td><td>0.80</td><td>0.68</td><td>0.68</td><td>0.03</td><td>0.09</td><td>0.44</td><td>0.10</td><td>0.46</td></tr><tr><td>hat_Latn</td><td>0.74</td><td>0.47</td><td>0.86</td><td>0.65</td><td>0.61</td><td>0.15</td><td>0.12</td><td>0.36</td><td>0.09</td><td>0.45</td></tr><tr><td>mag_Deva</td><td>0.75</td><td>0.52</td><td>0.88</td><td>0.75</td><td>0.70</td><td>0.02</td><td>0.05</td><td>0.29</td><td>0.07</td><td>0.45</td></tr><tr><td>min_Latn</td><td>0.56</td><td>0.45</td><td>0.81</td><td>0.68</td><td>0.68</td><td>0.16</td><td>0.12</td><td>0.34</td><td>0.17</td><td>0.44</td></tr><tr><td>ban_Latn</td><td>0.48</td><td>0.32</td><td>0.82</td><td>0.62</td><td>0.63</td><td>0.17</td><td>0.15</td><td>0.42</td><td>0.30</td><td>0.43</td></tr><tr><td>bjn_Latn</td><td>0.60</td><td>0.51</td><td>0.78</td><td>0.62</td><td>0.60</td><td>0.14</td><td>0.12</td><td>0.31</td><td>0.21</td><td>0.43</td></tr><tr><td>azj_Latn</td><td>0.75</td><td>0.53</td><td>0.86</td><td>0.68</td><td>0.66</td><td>0.01</td><td>0.03</td><td>0.29</td><td>0.02</td><td>0.43</td></tr><tr><td>npi_Deva</td><td>0.81</td><td>0.59</td><td>0.82</td><td>0.62</td><td>0.64</td><td>0.02</td><td>0.02</td><td>0.20</td><td>0.05</td><td>0.42</td></tr><tr><td>mar_Deva</td><td>0.82</td><td>0.58</td><td>0.87</td><td>0.68</td><td>0.63</td><td>0.02</td><td>0.01</td><td>0.12</td><td>0.03</td><td>0.42</td></tr><tr><td>awa_Deva</td><td>0.73</td><td>0.53</td><td>0.83</td><td>0.65</td><td>0.65</td><td>0.02</td><td>0.05</td><td>0.21</td><td>0.08</td><td>0.42</td></tr><tr><td>ben_Beng</td><td>0.82</td><td>0.58</td><td>0.82</td><td>0.60</td><td>0.60</td><td>0.01</td><td>0.01</td><td>0.23</td><td>0.05</td><td>0.41</td></tr><tr><td>uzn_Latn</td><td>0.70</td><td>0.47</td><td>0.84</td><td>0.60</td><td>0.62</td><td>0.04</td><td>0.05</td><td>0.23</td><td>0.05</td><td>0.40</td></tr><tr><td>bho_Deva</td><td>0.51</td><td>0.46</td><td>0.89</td><td>0.68</td><td>0.65</td><td>0.02</td><td>0.03</td><td>0.26</td><td>0.08</td><td>0.40</td></tr><tr><td>gle_Latn</td><td>0.68</td><td>0.31</td><td>0.82</td><td>0.64</td><td>0.64</td><td>0.09</td><td>0.08</td><td>0.22</td><td>0.09</td><td>0.40</td></tr><tr><td>hye_Armn</td><td>0.85</td><td>0.59</td><td>0.79</td><td>0.60</td><td>0.58</td><td>0.01</td><td>0.02</td><td>0.10</td><td>0.01</td><td>0.40</td></tr><tr><td>hne_Deva</td><td>0.67</td><td>0.44</td><td>0.80</td><td>0.65</td><td>0.63</td><td>0.01</td><td>0.02</td><td>0.24</td><td>0.08</td><td>0.40</td></tr><tr><td>kaz_Cyrl</td><td>0.62</td><td>0.47</td><td>0.87</td><td>0.61</td><td>0.60</td><td>0.02</td><td>0.06</td><td>0.27</td><td>0.03</td><td>0.39</td></tr><tr><td>tpi_Latn</td><td>0.69</td><td>0.38</td><td>0.69</td><td>0.46</td><td>0.45</td><td>0.19</td><td>0.15</td><td>0.33</td><td>0.15</td><td>0.39</td></tr><tr><td>hau_Latn</td><td>0.68</td><td>0.41</td><td>0.77</td><td>0.58</td><td>0.54</td><td>0.12</td><td>0.11</td><td>0.17</td><td>0.06</td><td>0.38</td></tr><tr><td>mai_Deva</td><td>0.61</td><td>0.43</td><td>0.86</td><td>0.61</td><td>0.59</td><td>0.02</td><td>0.03</td><td>0.16</td><td>0.08</td><td>0.38</td></tr></tbody></table>

Continued on next page
<table><thead><tr><th></th><th>Gemma 2<br>9B</th><th>Gemma 1<br>7B</th><th>Llama 3.1<br>70B</th><th>Llama 3.1<br>8B</th><th>Llama 3<br>8B</th><th>Llama 2<br>7B</th><th>Llama 1<br>7B</th><th>Mistral<br>7B</th><th>OLMo<br>7B</th><th>AVG<br>7B</th></tr></thead><tbody><tr><td>crh_Latn</td><td>0.58</td><td>0.42</td><td>0.77</td><td>0.56</td><td>0.51</td><td>0.03</td><td>0.07</td><td>0.35</td><td>0.05</td><td>0.37</td></tr><tr><td>ilo_Latn</td><td>0.61</td><td>0.32</td><td>0.74</td><td>0.47</td><td>0.46</td><td>0.17</td><td>0.15</td><td>0.24</td><td>0.13</td><td>0.37</td></tr><tr><td>tat_Cyrl</td><td>0.56</td><td>0.38</td><td>0.83</td><td>0.58</td><td>0.55</td><td>0.03</td><td>0.04</td><td>0.21</td><td>0.02</td><td>0.36</td></tr><tr><td>kat_Geor</td><td>0.73</td><td>0.45</td><td>0.72</td><td>0.53</td><td>0.50</td><td>0.01</td><td>0.04</td><td>0.18</td><td>0.01</td><td>0.35</td></tr><tr><td>ydd_Hebr</td><td>0.74</td><td>0.45</td><td>0.78</td><td>0.48</td><td>0.47</td><td>0.03</td><td>0.02</td><td>0.05</td><td>0.02</td><td>0.34</td></tr><tr><td>kir_Cyrl</td><td>0.53</td><td>0.33</td><td>0.81</td><td>0.58</td><td>0.57</td><td>0.01</td><td>0.04</td><td>0.13</td><td>0.02</td><td>0.34</td></tr><tr><td>pag_Latn</td><td>0.33</td><td>0.23</td><td>0.63</td><td>0.35</td><td>0.39</td><td>0.25</td><td>0.21</td><td>0.36</td><td>0.24</td><td>0.33</td></tr><tr><td>pan_Guru</td><td>0.78</td><td>0.50</td><td>0.75</td><td>0.47</td><td>0.40</td><td>0.01</td><td>0.00</td><td>0.03</td><td>0.05</td><td>0.33</td></tr><tr><td>bak_Cyrl</td><td>0.56</td><td>0.36</td><td>0.82</td><td>0.51</td><td>0.51</td><td>0.01</td><td>0.04</td><td>0.13</td><td>0.02</td><td>0.33</td></tr><tr><td>guj_Gujr</td><td>0.79</td><td>0.52</td><td>0.67</td><td>0.42</td><td>0.39</td><td>0.02</td><td>0.01</td><td>0.04</td><td>0.07</td><td>0.33</td></tr><tr><td>tam_Taml</td><td>0.78</td><td>0.54</td><td>0.72</td><td>0.38</td><td>0.38</td><td>0.02</td><td>0.01</td><td>0.08</td><td>0.03</td><td>0.33</td></tr><tr><td>pbt_Arab</td><td>0.50</td><td>0.23</td><td>0.82</td><td>0.57</td><td>0.57</td><td>0.02</td><td>0.03</td><td>0.10</td><td>0.07</td><td>0.32</td></tr><tr><td>tgk_Cyrl</td><td>0.62</td><td>0.27</td><td>0.78</td><td>0.51</td><td>0.52</td><td>0.02</td><td>0.03</td><td>0.10</td><td>0.05</td><td>0.32</td></tr><tr><td>tel_Telu</td><td>0.77</td><td>0.52</td><td>0.67</td><td>0.38</td><td>0.43</td><td>0.01</td><td>0.01</td><td>0.05</td><td>0.04</td><td>0.32</td></tr><tr><td>snd_Arab</td><td>0.59</td><td>0.30</td><td>0.78</td><td>0.53</td><td>0.50</td><td>0.02</td><td>0.01</td><td>0.06</td><td>0.04</td><td>0.31</td></tr><tr><td>kan_Knda</td><td>0.74</td><td>0.47</td><td>0.66</td><td>0.42</td><td>0.41</td><td>0.01</td><td>0.01</td><td>0.06</td><td>0.05</td><td>0.31</td></tr><tr><td>mal_Mlym</td><td>0.76</td><td>0.50</td><td>0.68</td><td>0.32</td><td>0.30</td><td>0.01</td><td>0.01</td><td>0.03</td><td>0.03</td><td>0.29</td></tr><tr><td>ckb_Arab</td><td>0.51</td><td>0.20</td><td>0.78</td><td>0.54</td><td>0.50</td><td>0.01</td><td>0.01</td><td>0.05</td><td>0.02</td><td>0.29</td></tr><tr><td>gla_Latn</td><td>0.46</td><td>0.14</td><td>0.71</td><td>0.46</td><td>0.45</td><td>0.08</td><td>0.07</td><td>0.13</td><td>0.07</td><td>0.29</td></tr><tr><td>asm_Beng</td><td>0.63</td><td>0.35</td><td>0.70</td><td>0.39</td><td>0.36</td><td>0.00</td><td>0.01</td><td>0.08</td><td>0.04</td><td>0.29</td></tr><tr><td>tuk_Latn</td><td>0.49</td><td>0.31</td><td>0.63</td><td>0.43</td><td>0.41</td><td>0.05</td><td>0.04</td><td>0.14</td><td>0.02</td><td>0.28</td></tr><tr><td>san_Deva</td><td>0.48</td><td>0.26</td><td>0.71</td><td>0.46</td><td>0.45</td><td>0.00</td><td>0.01</td><td>0.12</td><td>0.02</td><td>0.28</td></tr><tr><td>kmr_Latn</td><td>0.38</td><td>0.15</td><td>0.69</td><td>0.48</td><td>0.50</td><td>0.05</td><td>0.06</td><td>0.13</td><td>0.05</td><td>0.28</td></tr><tr><td>lus_Latn</td><td>0.53</td><td>0.09</td><td>0.56</td><td>0.34</td><td>0.33</td><td>0.14</td><td>0.10</td><td>0.24</td><td>0.11</td><td>0.27</td></tr><tr><td>khk_Cyrl</td><td>0.44</td><td>0.18</td><td>0.73</td><td>0.42</td><td>0.43</td><td>0.01</td><td>0.02</td><td>0.08</td><td>0.03</td><td>0.26</td></tr><tr><td>ltg_Latn</td><td>0.31</td><td>0.23</td><td>0.61</td><td>0.38</td><td>0.35</td><td>0.08</td><td>0.06</td><td>0.22</td><td>0.06</td><td>0.26</td></tr><tr><td>azb_Arab</td><td>0.37</td><td>0.28</td><td>0.60</td><td>0.44</td><td>0.44</td><td>0.00</td><td>0.01</td><td>0.11</td><td>0.02</td><td>0.25</td></tr><tr><td>plt_Latn</td><td>0.52</td><td>0.17</td><td>0.59</td><td>0.25</td><td>0.25</td><td>0.14</td><td>0.12</td><td>0.18</td><td>0.05</td><td>0.25</td></tr><tr><td>ibo_Latn</td><td>0.35</td><td>0.15</td><td>0.64</td><td>0.38</td><td>0.37</td><td>0.09</td><td>0.08</td><td>0.12</td><td>0.07</td><td>0.25</td></tr><tr><td>mri_Latn</td><td>0.35</td><td>0.11</td><td>0.60</td><td>0.38</td><td>0.35</td><td>0.12</td><td>0.10</td><td>0.18</td><td>0.07</td><td>0.25</td></tr><tr><td>som_Latn</td><td>0.42</td><td>0.14</td><td>0.60</td><td>0.24</td><td>0.24</td><td>0.11</td><td>0.08</td><td>0.18</td><td>0.09</td><td>0.23</td></tr><tr><td>ace_Latn</td><td>0.22</td><td>0.13</td><td>0.49</td><td>0.32</td><td>0.31</td><td>0.14</td><td>0.10</td><td>0.21</td><td>0.15</td><td>0.23</td></tr><tr><td>xho_Latn</td><td>0.49</td><td>0.19</td><td>0.47</td><td>0.20</td><td>0.20</td><td>0.12</td><td>0.10</td><td>0.13</td><td>0.05</td><td>0.22</td></tr><tr><td>nso_Latn</td><td>0.27</td><td>0.11</td><td>0.48</td><td>0.26</td><td>0.23</td><td>0.17</td><td>0.13</td><td>0.19</td><td>0.07</td><td>0.21</td></tr><tr><td>sot_Latn</td><td>0.34</td><td>0.12</td><td>0.53</td><td>0.22</td><td>0.20</td><td>0.14</td><td>0.10</td><td>0.18</td><td>0.05</td><td>0.21</td></tr><tr><td>zul_Latn</td><td>0.55</td><td>0.19</td><td>0.44</td><td>0.19</td><td>0.17</td><td>0.11</td><td>0.06</td><td>0.10</td><td>0.05</td><td>0.21</td></tr><tr><td>kin_Latn</td><td>0.37</td><td>0.10</td><td>0.53</td><td>0.20</td><td>0.23</td><td>0.11</td><td>0.09</td><td>0.15</td><td>0.06</td><td>0.21</td></tr><tr><td>sin_Sinh</td><td>0.56</td><td>0.27</td><td>0.49</td><td>0.26</td><td>0.18</td><td>0.01</td><td>0.01</td><td>0.03</td><td>0.02</td><td>0.20</td></tr><tr><td>smo_Latn</td><td>0.28</td><td>0.09</td><td>0.66</td><td>0.20</td><td>0.20</td><td>0.10</td><td>0.08</td><td>0.13</td><td>0.06</td><td>0.20</td></tr><tr><td>nya_Latn</td><td>0.36</td><td>0.13</td><td>0.41</td><td>0.19</td><td>0.19</td><td>0.13</td><td>0.10</td><td>0.17</td><td>0.06</td><td>0.19</td></tr><tr><td>twi_Latn</td><td>0.23</td><td>0.08</td><td>0.46</td><td>0.22</td><td>0.22</td><td>0.14</td><td>0.13</td><td>0.19</td><td>0.07</td><td>0.19</td></tr><tr><td>sna_Latn</td><td>0.41</td><td>0.17</td><td>0.40</td><td>0.19</td><td>0.20</td><td>0.10</td><td>0.08</td><td>0.12</td><td>0.07</td><td>0.19</td></tr><tr><td>uig_Arab</td><td>0.21</td><td>0.09</td><td>0.71</td><td>0.29</td><td>0.29</td><td>0.00</td><td>0.01</td><td>0.03</td><td>0.02</td><td>0.18</td></tr><tr><td>bug_Latn</td><td>0.14</td><td>0.12</td><td>0.35</td><td>0.22</td><td>0.22</td><td>0.14</td><td>0.11</td><td>0.20</td><td>0.12</td><td>0.18</td></tr><tr><td>luo_Latn</td><td>0.07</td><td>0.07</td><td>0.40</td><td>0.25</td><td>0.24</td><td>0.15</td><td>0.12</td><td>0.21</td><td>0.09</td><td>0.18</td></tr><tr><td>tsn_Latn</td><td>0.24</td><td>0.10</td><td>0.42</td><td>0.18</td><td>0.18</td><td>0.12</td><td>0.11</td><td>0.20</td><td>0.06</td><td>0.18</td></tr><tr><td>arb_Latn</td><td>0.29</td><td>0.07</td><td>0.46</td><td>0.24</td><td>0.20</td><td>0.05</td><td>0.05</td><td>0.17</td><td>0.08</td><td>0.18</td></tr><tr><td>khm_Khmr</td><td>0.34</td><td>0.15</td><td>0.59</td><td>0.15</td><td>0.16</td><td>0.01</td><td>0.02</td><td>0.09</td><td>0.06</td><td>0.17</td></tr><tr><td>lua_Latn</td><td>0.09</td><td>0.08</td><td>0.33</td><td>0.20</td><td>0.21</td><td>0.14</td><td>0.13</td><td>0.24</td><td>0.12</td><td>0.17</td></tr><tr><td>lug_Latn</td><td>0.17</td><td>0.07</td><td>0.41</td><td>0.18</td><td>0.19</td><td>0.14</td><td>0.09</td><td>0.19</td><td>0.06</td><td>0.17</td></tr><tr><td>grn_Latn</td><td>0.17</td><td>0.09</td><td>0.44</td><td>0.16</td><td>0.17</td><td>0.12</td><td>0.09</td><td>0.13</td><td>0.10</td><td>0.16</td></tr><tr><td>ssw_Latn</td><td>0.27</td><td>0.10</td><td>0.37</td><td>0.17</td><td>0.17</td><td>0.11</td><td>0.08</td><td>0.14</td><td>0.05</td><td>0.16</td></tr><tr><td>lin_Latn</td><td>0.11</td><td>0.08</td><td>0.43</td><td>0.16</td><td>0.18</td><td>0.12</td><td>0.11</td><td>0.16</td><td>0.08</td><td>0.16</td></tr><tr><td>ory_Orya</td><td>0.28</td><td>0.03</td><td>0.66</td><td>0.18</td><td>0.19</td><td>0.01</td><td>0.01</td><td>0.03</td><td>0.03</td><td>0.16</td></tr><tr><td>fij_Latn</td><td>0.13</td><td>0.06</td><td>0.38</td><td>0.18</td><td>0.16</td><td>0.14</td><td>0.11</td><td>0.15</td><td>0.08</td><td>0.15</td></tr><tr><td>fuv_Latn</td><td>0.07</td><td>0.08</td><td>0.30</td><td>0.20</td><td>0.20</td><td>0.13</td><td>0.10</td><td>0.18</td><td>0.10</td><td>0.15</td></tr><tr><td>kas_Arab</td><td>0.16</td><td>0.10</td><td>0.50</td><td>0.20</td><td>0.21</td><td>0.02</td><td>0.02</td><td>0.10</td><td>0.05</td><td>0.15</td></tr><tr><td>quy_Latn</td><td>0.10</td><td>0.06</td><td>0.42</td><td>0.21</td><td>0.22</td><td>0.10</td><td>0.06</td><td>0.13</td><td>0.05</td><td>0.15</td></tr><tr><td>aka_Latn</td><td>0.17</td><td>0.06</td><td>0.37</td><td>0.14</td><td>0.17</td><td>0.11</td><td>0.08</td><td>0.12</td><td>0.10</td><td>0.15</td></tr><tr><td>mya_Mymr</td><td>0.36</td><td>0.13</td><td>0.46</td><td>0.14</td><td>0.16</td><td>0.00</td><td>0.00</td><td>0.02</td><td>0.02</td><td>0.15</td></tr><tr><td>run_Latn</td><td>0.25</td><td>0.07</td><td>0.37</td><td>0.16</td><td>0.17</td><td>0.08</td><td>0.06</td><td>0.10</td><td>0.04</td><td>0.14</td></tr><tr><td>bem_Latn</td><td>0.14</td><td>0.08</td><td>0.29</td><td>0.16</td><td>0.16</td><td>0.13</td><td>0.11</td><td>0.15</td><td>0.06</td><td>0.14</td></tr><tr><td>kas_Deva</td><td>0.14</td><td>0.09</td><td>0.37</td><td>0.20</td><td>0.21</td><td>0.02</td><td>0.03</td><td>0.11</td><td>0.08</td><td>0.14</td></tr><tr><td>wol_Latn</td><td>0.09</td><td>0.07</td><td>0.30</td><td>0.18</td><td>0.16</td><td>0.12</td><td>0.10</td><td>0.17</td><td>0.07</td><td>0.14</td></tr><tr><td>kam_Latn</td><td>0.09</td><td>0.08</td><td>0.26</td><td>0.18</td><td>0.16</td><td>0.13</td><td>0.10</td><td>0.15</td><td>0.08</td><td>0.14</td></tr></tbody></table>

Continued on next page
<table><caption>Table 7: Adjusted performance of MEXA using max pooling with the English performance of models on the Belebele benchmark.</caption><thead><tr><th></th><th>Gemma 2<br>9B</th><th>Gemma 1<br>7B</th><th>Llama 3.1<br>70B</th><th>Llama 3.1<br>8B</th><th>Llama 3<br>8B</th><th>Llama 2<br>7B</th><th>Llama 1<br>7B</th><th>Mistral<br>7B</th><th>OLMo<br>7B</th><th>AVG</th></tr></thead><tbody><tr><td>tso_Latn</td><td>0.14</td><td>0.06</td><td>0.35</td><td>0.14</td><td>0.13</td><td>0.11</td><td>0.08</td><td>0.13</td><td>0.06</td><td>0.13</td></tr><tr><td>kon_Latn</td><td>0.07</td><td>0.08</td><td>0.27</td><td>0.15</td><td>0.17</td><td>0.09</td><td>0.07</td><td>0.13</td><td>0.09</td><td>0.13</td></tr><tr><td>tum_Latn</td><td>0.15</td><td>0.07</td><td>0.32</td><td>0.11</td><td>0.13</td><td>0.09</td><td>0.08</td><td>0.11</td><td>0.05</td><td>0.13</td></tr><tr><td>kik_Latn</td><td>0.07</td><td>0.04</td><td>0.32</td><td>0.12</td><td>0.13</td><td>0.10</td><td>0.09</td><td>0.13</td><td>0.12</td><td>0.12</td></tr><tr><td>taq_Latn</td><td>0.06</td><td>0.06</td><td>0.28</td><td>0.14</td><td>0.12</td><td>0.11</td><td>0.08</td><td>0.14</td><td>0.05</td><td>0.12</td></tr><tr><td>mos_Latn</td><td>0.04</td><td>0.04</td><td>0.25</td><td>0.16</td><td>0.14</td><td>0.11</td><td>0.09</td><td>0.15</td><td>0.07</td><td>0.12</td></tr><tr><td>yor_Latn</td><td>0.13</td><td>0.04</td><td>0.30</td><td>0.14</td><td>0.14</td><td>0.08</td><td>0.06</td><td>0.10</td><td>0.04</td><td>0.11</td></tr><tr><td>amh_Ethi</td><td>0.48</td><td>0.16</td><td>0.24</td><td>0.04</td><td>0.03</td><td>0.01</td><td>0.02</td><td>0.03</td><td>0.02</td><td>0.11</td></tr><tr><td>sag_Latn</td><td>0.05</td><td>0.07</td><td>0.22</td><td>0.17</td><td>0.17</td><td>0.07</td><td>0.07</td><td>0.09</td><td>0.06</td><td>0.11</td></tr><tr><td>cjk_Latn</td><td>0.06</td><td>0.06</td><td>0.21</td><td>0.13</td><td>0.12</td><td>0.10</td><td>0.08</td><td>0.13</td><td>0.07</td><td>0.11</td></tr><tr><td>umb_Latn</td><td>0.05</td><td>0.05</td><td>0.20</td><td>0.15</td><td>0.14</td><td>0.10</td><td>0.08</td><td>0.11</td><td>0.05</td><td>0.10</td></tr><tr><td>dyu_Latn</td><td>0.04</td><td>0.04</td><td>0.22</td><td>0.13</td><td>0.12</td><td>0.06</td><td>0.07</td><td>0.10</td><td>0.08</td><td>0.10</td></tr><tr><td>kac_Latn</td><td>0.02</td><td>0.03</td><td>0.22</td><td>0.12</td><td>0.14</td><td>0.06</td><td>0.06</td><td>0.12</td><td>0.08</td><td>0.09</td></tr><tr><td>kmb_Latn</td><td>0.05</td><td>0.06</td><td>0.20</td><td>0.11</td><td>0.10</td><td>0.10</td><td>0.07</td><td>0.09</td><td>0.05</td><td>0.09</td></tr><tr><td>bam_Latn</td><td>0.05</td><td>0.05</td><td>0.18</td><td>0.11</td><td>0.09</td><td>0.08</td><td>0.08</td><td>0.12</td><td>0.04</td><td>0.09</td></tr><tr><td>ayr_Latn</td><td>0.04</td><td>0.04</td><td>0.20</td><td>0.11</td><td>0.10</td><td>0.06</td><td>0.05</td><td>0.10</td><td>0.06</td><td>0.08</td></tr><tr><td>lao_Laoo</td><td>0.17</td><td>0.04</td><td>0.22</td><td>0.07</td><td>0.09</td><td>0.02</td><td>0.02</td><td>0.04</td><td>0.09</td><td>0.08</td></tr><tr><td>dik_Latn</td><td>0.05</td><td>0.06</td><td>0.18</td><td>0.06</td><td>0.07</td><td>0.08</td><td>0.07</td><td>0.10</td><td>0.05</td><td>0.08</td></tr><tr><td>ewe_Latn</td><td>0.04</td><td>0.03</td><td>0.18</td><td>0.08</td><td>0.08</td><td>0.09</td><td>0.07</td><td>0.08</td><td>0.04</td><td>0.08</td></tr><tr><td>knc_Latn</td><td>0.05</td><td>0.06</td><td>0.15</td><td>0.08</td><td>0.08</td><td>0.07</td><td>0.06</td><td>0.08</td><td>0.05</td><td>0.08</td></tr><tr><td>kab_Latn</td><td>0.04</td><td>0.02</td><td>0.17</td><td>0.09</td><td>0.08</td><td>0.06</td><td>0.06</td><td>0.11</td><td>0.04</td><td>0.07</td></tr><tr><td>sat_Olck</td><td>0.19</td><td>0.02</td><td>0.32</td><td>0.05</td><td>0.05</td><td>0.00</td><td>0.00</td><td>0.01</td><td>0.01</td><td>0.07</td></tr><tr><td>gaz_Latn</td><td>0.05</td><td>0.03</td><td>0.20</td><td>0.06</td><td>0.06</td><td>0.05</td><td>0.04</td><td>0.08</td><td>0.03</td><td>0.07</td></tr><tr><td>bod_Tibt</td><td>0.07</td><td>0.01</td><td>0.22</td><td>0.08</td><td>0.08</td><td>0.01</td><td>0.01</td><td>0.02</td><td>0.02</td><td>0.06</td></tr><tr><td>fon_Latn</td><td>0.03</td><td>0.02</td><td>0.14</td><td>0.06</td><td>0.06</td><td>0.03</td><td>0.04</td><td>0.05</td><td>0.07</td><td>0.06</td></tr><tr><td>shn_Mymr</td><td>0.02</td><td>0.01</td><td>0.21</td><td>0.06</td><td>0.06</td><td>0.01</td><td>0.02</td><td>0.03</td><td>0.07</td><td>0.05</td></tr><tr><td>kbp_Latn</td><td>0.03</td><td>0.02</td><td>0.14</td><td>0.05</td><td>0.04</td><td>0.03</td><td>0.02</td><td>0.08</td><td>0.05</td><td>0.05</td></tr><tr><td>mni_Beng</td><td>0.03</td><td>0.02</td><td>0.12</td><td>0.05</td><td>0.06</td><td>0.01</td><td>0.02</td><td>0.08</td><td>0.02</td><td>0.05</td></tr><tr><td>ace_Arab</td><td>0.03</td><td>0.02</td><td>0.15</td><td>0.07</td><td>0.07</td><td>0.00</td><td>0.00</td><td>0.03</td><td>0.01</td><td>0.04</td></tr><tr><td>knc_Arab</td><td>0.01</td><td>0.01</td><td>0.13</td><td>0.04</td><td>0.04</td><td>0.02</td><td>0.02</td><td>0.05</td><td>0.05</td><td>0.04</td></tr><tr><td>bjn_Arab</td><td>0.03</td><td>0.02</td><td>0.11</td><td>0.05</td><td>0.08</td><td>0.01</td><td>0.01</td><td>0.05</td><td>0.01</td><td>0.04</td></tr><tr><td>nus_Latn</td><td>0.02</td><td>0.02</td><td>0.07</td><td>0.04</td><td>0.03</td><td>0.03</td><td>0.02</td><td>0.04</td><td>0.05</td><td>0.03</td></tr><tr><td>min_Arab</td><td>0.02</td><td>0.01</td><td>0.13</td><td>0.05</td><td>0.04</td><td>0.01</td><td>0.00</td><td>0.03</td><td>0.01</td><td>0.03</td></tr><tr><td>tir_Ethi</td><td>0.10</td><td>0.02</td><td>0.05</td><td>0.02</td><td>0.02</td><td>0.01</td><td>0.01</td><td>0.02</td><td>0.02</td><td>0.03</td></tr><tr><td>dzo_Tibt</td><td>0.01</td><td>0.00</td><td>0.08</td><td>0.03</td><td>0.03</td><td>0.00</td><td>0.00</td><td>0.01</td><td>0.01</td><td>0.02</td></tr><tr><td>taq_Tfng</td><td>0.00</td><td>0.00</td><td>0.04</td><td>0.01</td><td>0.01</td><td>0.00</td><td>0.00</td><td>0.01</td><td>0.03</td><td>0.01</td></tr><tr><td>zgh_Tfng</td><td>0.00</td><td>0.00</td><td>0.02</td><td>0.01</td><td>0.01</td><td>0.00</td><td>0.00</td><td>0.01</td><td>0.01</td><td>0.01</td></tr></tbody></table>