Mexa: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual Alignment

## Abstract

Abstract English-centric large language models (LLMs) often show strong multilingual capabilities. However, the multilingual performance of these models remains unclear and is not thoroughly evaluated for many languages. Most benchmarks for multilinguality focus on classic NLP tasks, or cover a minimal number of languages.
We introduce Mexa , a method for assessing the multilingual capabilities of pre-trained English-centric LLMs using parallel sentences, which are available for more languages than existing downstream tasks. Mexa leverages the fact that English-centric LLMs use English as a kind of pivot language in their intermediate layers. It computes the alignment between English and non-English languages using parallel sentences to evaluate the transfer of language understanding from English to other languages. This alignment can be used to estimate model performance in other languages.
We conduct studies using various parallel datasets (FLORES-200 and Bible), models (Llama family, Gemma family, Mistral, and OLMo), and established downstream tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute embeddings in decoder-only models.
Our results show that Mexa , in its default settings, achieves a statistically significant average Pearson correlation of 0.90 with three established downstream tasks across nine models and two parallel datasets.
This suggests that Mexa is a reliable method for estimating the multilingual capabilities of English-centric LLMs, providing a clearer understanding of their multilingual potential and the inner workings of LLMs. Leaderboard hf.co/spaces/cis-lmu/Mexa Code github.com/cisnlp/Mexa

## 1 Introduction

Most state-of-the-art autoregressive large language models (LLMs) are English-centric, closed-source models such as Claude 3 Opus, GPT-4, and Gemini Pro (Anthropic, ; OpenAI et al., ; Gemini Team et al., ); open-weight models such as Llama 3.1, Gemma 2, and Mixtral  (Dubey et al., ; Gemma Team et al., ; Jiang et al., ); and open-source models such as OLMo (Groeneveld et al., ). Except for open-source models, where the data is available and thus the language distribution is transparent, there is confusion regarding the capabilities/language distribution of these LLMs in other languages.

Primarily, the focus in evaluating LLMs has been on developing benchmarks to assess their performance in English. Most benchmarks in multilingual setups consist of classical monolingual NLP tasks such as sequence labeling (Ahuja et al., ; Lai et al., ), the automatic translation of popular English benchmarks such as MMLU (Hendrycks et al., ) into a limited number of languages (Lai et al., ; OpenAI, ), or language-specific benchmarks for languages such as Persian (Ghahroodi et al., ), Arabic (Koto et al., ), Korean (Son et al., ), Turkish (Yüksel et al., ), and Chinese (Li et al., ).

Most LLMs are English-centric, either by choice or due to the availability of abundant data sources in English. Either way, for these models to be effective in other languages, it is important that the other languages align with the main language, i.e., English. Given such alignment, English could act as a “rising tide that raises all ships,” meaning that improvements in English performance could benefit other languages, especially in tasks such as reasoning (Zhu et al., ).
Contrarily, if a language does not align well with English, an English-centric LLM may not provide meaningful coverage for that language. Indeed, Wendler et al. () have found that for Llama 2 (Touvron et al., ), an English-centric LLM, English could be seen as a kind of “pivot” language, enabling to solve complex semantic tasks in a foreign language through a detour into English. More precisely, they show that Llama 2 was able to decode semantically correct next tokens in the middle layers, assigning higher probabilities to the English tokens than to the foreign version, which is only selected in the upper layers. Zhao et al. () present a hypothesis regarding the middle layers of English-centric LLMs, suggesting that these models use English as a means of reasoning while incorporating multilingual knowledge. Based on their analysis, the number of language-specific neurons in the middle layers decreases within the self-attention mechanism but remains consistent across the layers of the feed-forward structure when processing multilingual queries.

In this paper, we introduce Mexa, a method that uses the observation that English-centric LLMs semantically use English as a kind of pivot language in their middle layers to evaluate the actual multilingual coverage of LLMs. This is done by measuring how well the embeddings of parallel sentences in the middle layers of LLMs for non-English languages are aligned with the embedding of their corresponding English.
We extensively verify the Mexa estimation of language coverage for each LLM, using Pearson correlation between estimated and actual scores for various tasks. We use two parallel datasets: FLORES-200 (NLLB Team et al., ) and Bible (Mayer & Cysouw, ); nine LLMs: Llama family, Gemma family, Mistral, and OLMo; and three tasks: Belebele (Bandarkar et al., ), m-MMLU, and m-ARC (Lai et al., ).
Our results show that Mexa achieves a promising average Pearson correlation of 0.90 with established downstream tasks across nine models and two parallel datasets. In our study on the calculation of Mexa scores, we conduct multiple design analyses to examine the impact of token-level pooling for the embeddings (i.e., using the last token versus a weighted average) and layer-level pooling in computing alignment scores. While Mexa demonstrates a high correlation across most setups, we find that a weighted average based on tokens, combined with mean pooling, yields the best results.

## 2 Background and Related Work

We discuss the distribution of pre-training data in LLMs and multilingual evaluation benchmarks in Appendices A.1 and A.2 while focusing on cross-lingual alignment here.
Research in the cross-lingual alignment field either aims to uncover the underlying mechanisms of alignment and assess its impact on models and downstream tasks, or attempts to enhance model performance by enforcing alignment before, during, or after the pre-training phase. Most of these papers have focused on encoder-only models, such as XLM-R (Conneau et al., ) and mBERT (Devlin et al., ), among others (Hämmerl et al., ). In this work, we focus primarily on decoder-only models.

Understanding Alignment.
Ye et al. () show that English-centric models such as Llama 1 (Touvron et al., ) not only possess multilingual transfer abilities (after fine-tuning on one source language, they can be applied to other languages) but may even surpass the multilingual transfer abilities of multilingual pre-trained models such as BLOOM (BigScience Workshop et al., ).
Schäfer et al. () find that GPT-2-style decoder-only models show strong cross-lingual generalization when trained on an imbalanced mix of languages. However, when trained on a balanced language set, they do not observe increased performance compared to monolingual settings. Wendler et al. () perform single-token analysis to demonstrate that English-centered LLMs, such as Llama 2, use English semantically as an internal latent language in the middle layers when handling multilingual queries. Zhong et al. () extend this analysis to multiple tokens, also showing that an LLM dominated by both English and Japanese uses both languages as internal latent languages.
Zhao et al. () explore how LLMs handle multilingualism. They hypothesize that LLMs initially interpret the query, converting multilingual inputs into English for task-solving. In the middle layers, the models rely on English with self-attention mechanisms for reasoning, while employing multilingual knowledge through feed-forward structures. In the final layers, LLMs generate responses consistent with the original query language. Li et al. () and Li et al. () are even more closely related to ours.
Li et al. () employs absolute cosine similarity values between last token embeddings derived from parallel sentences with English to predict the ranking of language performance across various models. However, as we discuss in Section 3, relying solely on absolute cosine values can be misleading, and as shown in Section 5.3, absolute cosine values are less correlated with downstream tasks than Mexa score. Li et al. () uses English probing tasks and their automatic translations to construct a multilingual evaluation. While they compare embedding similarity scores between high- and low-resource languages with corresponding evaluation results, similar to Li et al. (), they do not assess whether these correlations hold across other downstream tasks. In Section 5, we demonstrate that Mexa scores align closely with a broad range of downstream tasks.

Boosting Alignment.
The idea to enforce alignment in encoder-only models using parallel sentences dates back to Conneau & Lample (), and has been explored under various guises e.g., using mixed-language sentences and/or bilingual dictionaries (Huang et al., ; Conneau et al., ; Cao et al., ; Kulshreshtha et al., ; Efimov et al., ; Zhang et al., ). Recently,
Li et al. () improve multilingual alignment by initializing the decoder-only models to generate similar representations of aligned words using contrastive learning and preserves this alignment using a code-switching strategy during pretraining. Liu et al. () propose a data allocation technique to select a core subset of languages for fine-tuning, better aligning the multilingual capabilities of decoder-only LLMs and making them more truthful in their responses. Li et al. () propose aligning internal sentence representations across different languages using multilingual contrastive learning and aligning outputs by following cross-lingual instructions in the target language for decoder-only models.

## 3 Mexa

We now describe the Mexa method for computing the alignment score of language $L_{1}$ with respect to the pivot language $L_{2}$, given the language model $m$. In this paper, we use the term cross-lingual alignment, geometric alignment, or simply alignment to refer to the semantic similarity of multilingual embeddings across languages. $L_{2}$, for English-centric LLMs and in this paper, is English. To assess alignment, we use parallel sentences in two languages, $L_{1}$ and $L_{2}$.

What defines semantic similarity in multilingual embeddings across languages? The goal of semantic similarity is to ensure that parallel sentences have sufficiently high similarity, reflecting alignment between the two languages. However, considering only the absolute cosine similarity value as the alignment score does not guarantee proper alignment. For some languages, even non-parallel sentences exhibit similarity scores comparable to those of parallel sentences (see §5.3).
This is largely due to the anisotropy problem observed in Transformer models, which can lead to so-called hubness issues, making it difficult to distinguish between similar and dissimilar embeddings (Ethayarajh, ), especially in multilingual models (Hämmerl et al., ; Rajaee & Pilehvar, ).
However, a direct comparative analysis of the cosine similarity between parallel and non-parallel sentence pairs across languages can help overcome these issues. Instead of using the absolute cosine similarity value for alignment, we assign binary values (1 or 0) based on whether a criterion for semantic similarity is satisfied. This criterion imposes that (a) parallel sentences should have high cosine similarity, and (b) non-parallel pairs should also have low similarity values, ensuring the similarity is not random or biased. Specifically, if the cosine similarity for a pair of parallel sentences is higher than for any non-parallel sentences, we assign a value of 1 for this pair; otherwise, a value of 0. This approach sidesteps the hubness problem since the absolute cosine similarity values themselves are not directly used.

To compute Mexa, we first apply the cosine similarity function to the pairs of embeddings of parallel sentences in languages $L_{1}$ and $L_{2}$. In Section 3.1, we describe how embeddings can be computed for each layer $l$ of the autoregressive language model $m$. We generate a square matrix $C(L_{1},L_{2},m,l)$ representing cosine similarities of embeddings at the output of layer $l$ for all parallel sentences in languages $L_{1}$ and $L_{2}$.
We denote $c_{ij}(l)$ the element in the $i$-th row and $j$-th column of $C(L_{1},L_{2},m,l)$. It represents the cosine similarity between the $i$-th sentence of $L_{1}$ and the $j$-th sentence of $L_{2}$ at layer $l$ of language model $m$. The diagonal elements of $C$, denoted $c_{ii}(l)$, represent the cosine similarity between parallel sentence pairs from
$L_{1}$ and $L_{2}$. We define the Mexa alignment score ($\mu(.)$) as follows:

$$ $\mu\big{(}C(L_{1},L_{2},m,l)\big{)}=\frac{1}{n}\sum_{i=1}^{n}\mathbf{1}\left(c _{ii}(l)>\max_{j\in\{1,\dots,n\}\setminus\{i\}}\{c_{ij}(l),c_{ji}(l)\}\right)$ (1) $$

where $n$ is the number of diagonal elements (i.e., the dimension of the matrix), and $\mathbf{1}(\cdot)$ is the indicator function, which equals 1 if its argument condition evaluates to true and 0 otherwise. This alignment score measures how often $c_{ii}(l)$
is the maximum value in both its row and column.
The Mexa alignment score can alternatively be understood as
a measure of sentence retrieval performance (Hu et al., ; Liu et al., ; Hämmerl et al., ), with the metric of P@1 applied with queries in language $L_{1}$ and answers in $L_{2}$, and vice versa. We discuss other ways to calculate semantic similarity between languages in Appendix A.3.

Layer-wise Pooling. The Mexa alignment score $\mu\big{(}C(L_{1},L_{2},m,l)\big{)}$ is computed for language $L_{1}$ respect to pivot language $L_{2}$ for each layer $l$ of the language model $m$. To compute a single Mexa alignment score given the language model $m$ and $L_{1}$, $L_{2}$, we use mean and max pooling strategies over multiple layers.

### 3.1 Sentence Embeddings

Sentence embeddings are a transformation of a sentence into a fixed-size vector that captures its meaning. The process of computing sentence embeddings can vary depending on the model architecture. Typically, sentence embeddings are used in encoder-based models such as BERT, which employ bidirectional attention. In these models, the hidden states of each token are first extracted, then aggregated, commonly by averaging the hidden states from the output layer. Since attention in these models is bidirectional, each token contributes equally to the final embedding. Alternatively one can use the output of the special [CLS] token as per the original BERT work (Devlin et al., ).

In this paper, we focus on autoregressive language models that use a decoder-only architecture. In this architecture, attention is not bidirectional; instead, it takes the form of causal attention (left-to-right). In bidirectional attention, each token has access to every other token in the sequence. However, in causal attention, the embedding of a token at position $t$ is only influenced by the embedding of preceding tokens at positions $0,1,\ldots,t-1$. Therefore, simple averaging biases the embeddings towards sentence-initial words. Instead, two alternative methods are considered: using only the last token and weighted averaging. We use and compare both methods to acquire the sentence embeddings needed for Mexa.

A standard way to compute a sentence embedding uses only the last token of that sentence. Jiang et al. () show that using the last token in the format of a prompt template for a sentence $s$, such as ’This sentence: {s} means in one word:’, can be effective. Inspired by this, Li & Li () used the prompt ’Summarize sentence {s} in one word:’ to obtain the last token embedding as the sentence-level text embedding. However, not all models are instruction-tuned; some earlier works, such as Neelakantan et al. (); Wang et al. (); Ma et al. (), use the last token without any prompt. Since the models studied in this paper are only pre-trained and use multiple languages in the input, we decided to use the last token method without any preceding instruction.
An alternative is weighted averaging. It relies on the intuition that using only the last token might not represent the entire sentence, as the influence of earlier tokens may have diminished. This suggests that tokens at the end of the sentence should contribute more to the overall embedding than those at the beginning. Another motivation is that sentence-final tokens are influenced by preceding tokens and contain more context, while the representation of sentence-initial tokens has significantly less contextual representation. To address this, Muennighoff () proposes to assign weights to each token based on its position. Thus, the sentence embedding of layer $l$ using position-weighted averaging is:

$$ $e_{l}=\sum_{t=1}^{T}w_{t}h_{lt}\quad\textrm{with}\quad w_{t}=\frac{t}{\sum_{k= 1}^{T}k}$ (2) $$

where $T$ is the number of tokens in the given sentence, $h_{lt}$ is the embedding of the $t$-th token at layer $l$, and $e_{l}$ is the sentence embedding at layer $l$.

## 4 Experiments

We conduct experiments using various multi-parallel datasets (FLORES-200 and the Bible), models (Llama family, Gemma family, Mistral, and OLMo), and existing benchmarks/tasks (Belebele, m-MMLU, m-ARC). Our objective is to assess how well the Mexa alignment score from various parallel datasets correlates with the different benchmarks/tasks for different models.

### 4.1 Parallel Data

We calculate the Mexa score using the parallel datasets of FLORES-200 (NLLB Team et al., ) and the Bible (Mayer & Cysouw, ). While there are other high-quality parallel datasets, such as NTREX-128 (Federmann et al., ), IN22 (Gala et al., ), OPUS-100 (Zhang et al., ), Europarl (Koehn, ), OpenSubtitles (Lison & Tiedemann, ), TED2020 (Reimers & Gurevych, ), and Tatoeba (Tatoeba Community, ), we specifically chose FLORES-200 due to its high quality and support for a wide range of languages, while the Bible dataset was selected for its extensive language coverage.

FLORES-200 is a parallel corpus, where the English subset is sourced from Wikimedia projects. Each English sentence has been translated into 204 distinct language-script combinations, these translations have been verified by humans. The dataset contains 997 sentences for development, 1012 sentences for dev-test, and 992 sentences for testing. As the FLORES-200 test set is not publicly accessible, we use the dev-test set as our FLORES parallel test corpus, in line with previous studies. For faster computation, we only consider the first 100 sentences from each language. As shown in Appendix A.4, the odds of the Mexa score randomly achieving a high value with 100 sentences are very slim.

The Parallel Bible (Mayer & Cysouw, ) covers a very large number of languages. From this resource, we managed to create a subcorpus, a super parallel dataset of the Bible, with 1,401 language-script labels, each containing 103 sentences (i.e., Bible verses).(^1^11[hf.co/datasets/cis-lmu/sPBC](https://huggingface.co/datasets/cis-lmu/sPBC)) This corpus includes many low-resource languages, many of which are not covered by existing language technologies (Joshi et al., ), and Mexa can be adopted since only parallel data is needed. We use all the 103 sentences from each language.

### 4.2 Models

For our experiments, we select models with around 7B parameters, which are considered a base size in the LLM community. The state-of-the-art open-weight models in this range include Llama 1, 2, 3, and 3.1 (Touvron et al., ; ; Meta, ; Dubey et al., ), Gemma 1 and 2 (Gemma Team et al., ; ), Mistral 0.3 (Jiang et al., ), and the open-source model OLMo 1.7 (Groeneveld et al., ).
We also select a larger model, Llama 3.1 70B, to show that our findings hold even when scaling further. To apply Mexa, we need to access model weights to compute input sentence embeddings for each layer. We use three popular open-weight model families: Llama, Gemma, and Mistral. As a less multilingual version of state-of-the-art LLMs, we include OLMo, which is trained on a more English-centric corpus of Dolma (Soldaini et al., ). Although there are multilingual models such as PolyLM (with support of 18 languages), XGLM (Lin et al., ) (with support of 30 languages) and BLOOM (BigScience Workshop et al., ) (with support of 46 languages) our focus here is on LLMs which are state-of-the-art in English based tasks such as MMLU (Stanford CRFM, ).

### 4.3 Benchmarks

Among the existing evaluation benchmarks in Table 4 from Appendix A.2, we chose the Belebele benchmark (Bandarkar et al., ), m-ARC (Lai et al., ), and m-MMLU (Lai et al., ), which support the highest number of high-, medium-, and low-resource languages and are directly related to natural understanding tasks, which is the primary focus of this paper.

Belebele is a multiple-choice reading comprehension task designed to assess language models across a range of high-, medium-, and low-resource languages. Each question in the dataset is accompanied by four possible answers and is linked to a brief passage from the FLORES-200 dataset (NLLB Team et al., ). The human annotation process was carefully curated to generate questions that effectively differentiate between various levels of language comprehension, supported by rigorous quality checks. Belebele supports 122 distinct labels (language-script combinations) corresponding to 115 distinct languages. However, FLORES-200 does not support 5 of these labels, corresponding to Romanized versions of 5 Indic languages. Therefore, we conducted our analysis between the FLORES-200 and Belebele benchmarks on 117 common labels. Additionally, there are 102 common labels between the Bible parallel data and Belebele benchmark.

Both ARC Challenge (Clark et al., ) and MMLU (Hendrycks et al., ) are also set up as multiple-choice question-answering tasks, but they focus on different types of knowledge and reasoning skills. ARC Challenge is classified as a common-sense reasoning task, consisting of grade-school level science questions, while MMLU consists of questions across a wide range of subjects, including humanities, social sciences, and more. Lai et al. () used GPT-3.5-turbo (OpenAI, ) and a translation prompt to translate examples from both datasets and create m-ARC and m-MMLU in 31 languages (excluding English).
Later, m-MMLU was expanded to also include Icelandic (isl_Latn) and Norwegian (nob_Latn). The Icelandic portion was translated using the Mideind.is, while Norwegian was generated with DeepL.com.(^2^22[hf.co/datasets/alexandrainst/m_mmlu](https://huggingface.co/datasets/alexandrainst/m_mmlu)) m-MMLU consists of 277 questions in its training set, 13,258 in the test set and 1,433 in the validation set.
m-ARC consists of 1,116 questions in the training set, 1,169 in the test set, and 298 in the validation set. We use the entire test set for each of these benchmarks to evaluate the models, except in one case. For Llama 3.1 70B, we use the first 500 questions of m-MMLU instead of the whole set due to resource constraints. Since the selected LLMs used in our experiment are not instruction-tuned, we use 5-shot in-context learning with the lm-evaluation-harness framework, employing log-likelihood-based multiple-choice scoring. Other settings, such as prompt templates, are configured according to the framework’s default (Gao et al., ; Biderman et al., ).

### 4.4 Evaluation Measures

We use Pearson correlation to assess the strength of the correlation between Mexa and downstream performance on our evaluation benchmarks. Pearson correlation is a statistical measure that calculates the strength and direction of the linear relationship between two variables. A high Pearson correlation would indicate that Mexa provides a reliable assessment of multilingual capabilities in English-centric LLMs.

## 5 Results

**Table 1: $\mu_{\text{pooling}}$ indicates the Mexa score for each corresponding pooling strategy. The embeddings are computed using weighted average based on token positions (Eq. 2). Top values are in bold, with second-best underlined.**
|  |  | Gemma 2 | Gemma 1 | Llama 3.1 | Llama 3.1 | Llama 3 | Llama 2 | Llama 1 | Mistral 0.3 | OLMo 1.7 | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | 9B | 7B | 70B | 8B | 8B | 7B | 7B | 7B | 7B |  |
| $\text{Task}_{\text{\scalebox{0.99}{\{eng\}}}}$ | Belebele | 0.9178 | 0.8467 | 0.9456 | 0.8767 | 0.8689 | 0.4822 | 0.4156 | 0.8389 | 0.7711 | 0.7737 |
| m-MMLU | 0.6998 | 0.6138 | 0.7700 | 0.6315 | 0.6294 | 0.4523 | 0.3569 | 0.5988 | 0.5210 | 0.5859 |  |
| m-ARC | 0.6775 | 0.5870 | 0.7014 | 0.5794 | 0.5836 | 0.5128 | 0.5000 | 0.5862 | 0.4872 | 0.5795 |  |
| $\text{Task}_{\text{\,\scalebox{0.99}{L$\setminus$\{eng\}} }}$ | Belebele (avg., $\lvert$L$\rvert$ = 116) | 0.7093 | 0.5633 | 0.7684 | 0.5705 | 0.5533 | 0.3028 | 0.2755 | 0.4457 | 0.3627 | 0.5057 |
| m-MMLU (avg., $\lvert$L$\rvert$ = 33) | 0.5582 | 0.4734 | 0.6384 | 0.4720 | 0.4664 | 0.3260 | 0.2807 | 0.4207 | 0.3390 | 0.4416 |  |
| m-ARC (avg., $\lvert$L$\rvert$ = 31) | 0.4779 | 0.4220 | 0.5054 | 0.3941 | 0.3892 | 0.3174 | 0.2970 | 0.3662 | 0.2731 | 0.3825 |  |
| FLORES | $\mu_{\text{Mean}}$ (avg., $\lvert$L$\rvert$ = 116) | 0.5088 | 0.3815 | 0.4110 | 0.3963 | 0.3939 | 0.0866 | 0.1946 | 0.2642 | 0.0413 | 0.2976 |
| $\mu_{\text{Max}}$ (avg., $\lvert$L$\rvert$ = 116) | 0.7194 | 0.5872 | 0.7725 | 0.6538 | 0.6520 | 0.2464 | 0.3579 | 0.4716 | 0.1965 | 0.5175 |  |
| Bible | $\mu_{\text{Mean}}$ (avg., $\lvert$L$\rvert$ = 101) | 0.3568 | 0.2152 | 0.3169 | 0.2103 | 0.2026 | 0.1246 | 0.0908 | 0.1198 | 0.0121 | 0.1832 |
| $\mu_{\text{Max}}$ (avg., $\lvert$L$\rvert$ = 101) | 0.6076 | 0.4021 | 0.6599 | 0.4212 | 0.4190 | 0.2724 | 0.2357 | 0.2606 | 0.0319 | 0.3678 |  |

Table 1 presents the downstream performance of the selected models across three benchmarks, along with Mexa scores from two parallel datasets. Notably, among models with parameter sizes ranging from 7 to 9 billion, both Gemma 2 and Llama 3.1 outperform the others in terms of non-English downstream performance and Mexa scores. The Llama 3.1 and Llama 3 models exhibit similar alignment and downstream task performance; yet, both represent substantial advancements compared to Llama 2.
Moreover, results for the Llama 3.1-70B model indicate that scaling can significantly enhance alignment when compared to its smaller version. Interestingly, while Mistral achieves comparable results to Gemma 1 on English benchmarks, it demonstrates inferior alignment, which likely accounts for its reduced performance on non-English tasks.
Furthermore, the Llama 2 model achieves higher Mexa scores than OLMo, indicating better alignment. However, due to Llama 2’s weaker performance on English tasks, it fails to transfer this alignment effectively, leading to comparable non-English performance between Llama 2 and OLMo. This observation is further explored in Section 5.2, where we normalize the expected performance based on the pivot language, namely English.

### 5.1 Mexa Correlation Analysis

**Table 2: Pearson correlation of Mexa using FLORES and Bible data across three tasks. $\rho$ ($\mu_{\text{Pooling}}$, Task) is the correlation of Mexa for the corresponding pooling strategy and benchmark. In all settings except **, the p-value is $p<0.001$. The best average correlations for each task are in bold, and the second bests are underlined.**
|  |  | Gemma 2 | Gemma 1 | Llama 3.1 | Llama 3.1 | Llama 3 | Llama 2 | Llama 1 | Mistral 0.3 | OLMo 1.7 | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | 9B | 7B | 70B | 8B | 8B | 7B | 7B | 7B | 7B |  |
| FLORES<br>weighted average | $\rho$ ($\mu_{\text{Mean}}$, Belebele) | 0.9247 | 0.9421 | 0.8291 | 0.9478 | 0.9588 | 0.8364 | 0.8404 | 0.9732 | 0.8425 | 0.8994 |
| $\rho$ ($\mu_{\text{Max}}$, Belebele) | 0.9623 | 0.9676 | 0.9211 | 0.9392 | 0.9326 | 0.8362 | 0.7649 | 0.9448 | 0.9198 | 0.9098 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-MMLU) | 0.9342 | 0.9697 | 0.9362 | 0.9689 | 0.9647 | 0.9223 | 0.9406 | 0.9857 | 0.9393 | 0.9513 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-MMLU) | 0.9060 | 0.9596 | 0.8946 | 0.9003 | 0.8892 | 0.9386 | 0.8936 | 0.9311 | 0.9565 | 0.9188 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-ARC) | 0.9741 | 0.9706 | 0.9374 | 0.9515 | 0.9562 | 0.9052 | 0.9268 | 0.9693 | 0.8630 | 0.9393 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-ARC) | 0.9187 | 0.9499 | 0.8736 | 0.8582 | 0.8663 | 0.9297 | 0.8439 | 0.9001 | 0.8298 | 0.8856 |  |
| FLORES<br>last token | $\rho$ ($\mu_{\text{Mean}}$, Belebele) | 0.8997 | 0.9326 | 0.8491 | 0.9494 | 0.9581 | 0.9141 | 0.8340 | 0.9679 | 0.9467 | 0.9168 |
| $\rho$ ($\mu_{\text{Max}}$, Belebele) | 0.9225 | 0.9309 | 0.9127 | 0.9244 | 0.9123 | 0.9125 | 0.7693 | 0.9460 | 0.9218 | 0.9058 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-MMLU) | 0.9086 | 0.9637 | 0.9370 | 0.9687 | 0.9690 | 0.9771 | 0.9301 | 0.9659 | 0.9700 | 0.9545 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-MMLU) | 0.8448 | 0.9297 | 0.8645 | 0.9224 | 0.9177 | 0.9699 | 0.8902 | 0.9161 | 0.9649 | 0.9134 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-ARC) | 0.9190 | 0.9541 | 0.9524 | 0.9536 | 0.9617 | 0.9390 | 0.9146 | 0.9451 | 0.7356 | 0.9195 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-ARC) | 0.8569 | 0.9147 | 0.9005 | 0.8944 | 0.8879 | 0.9464 | 0.8263 | 0.8859 | 0.7037 | 0.8685 |  |
| Bible<br>weighted average | $\rho$ ($\mu_{\text{Mean}}$, Belebele) | 0.8360 | 0.8530 | 0.7909 | 0.8781 | 0.8974 | 0.8982 | 0.8404 | 0.9118 | 0.7410 | 0.8496 |
| $\rho$ ($\mu_{\text{Max}}$, Belebele) | 0.8863 | 0.9001 | 0.8851 | 0.9242 | 0.9302 | 0.8926 | 0.8230 | 0.9337 | 0.7549 | 0.8811 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-MMLU) | 0.8051 | 0.8886 | 0.8958 | 0.9096 | 0.8964 | 0.9252 | 0.9159 | 0.9093 | 0.7944 | 0.8823 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-MMLU) | 0.5501 | 0.8831 | 0.7748 | 0.8683 | 0.8364 | 0.9180 | 0.9085 | 0.9107 | 0.7388 | 0.8210 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-ARC) | 0.8505 | 0.8998 | 0.9188 | 0.9267 | 0.9116 | 0.8940 | 0.9208 | 0.9317 | 0.8623 | 0.9018 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-ARC) | 0.6070 | 0.8803 | 0.8030 | 0.8769 | 0.8552 | 0.8684 | 0.8879 | 0.9178 | 0.8220 | 0.8354 |  |
| Bible<br>last token | $\rho$ ($\mu_{\text{Mean}}$, Belebele) | 0.7656 | 0.8005 | 0.5944 | 0.7934 | 0.8396 | 0.9046 | 0.8299 | 0.9177 | 0.8866 | 0.8147 |
| $\rho$ ($\mu_{\text{Max}}$, Belebele) | 0.7844 | 0.8299 | 0.5264 | 0.8000 | 0.8100 | 0.9047 | 0.8048 | 0.9235 | 0.8796 | 0.8070 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-MMLU) | 0.7194 | 0.7646 | 0.6472 | 0.6068 | 0.6516 | 0.8827 | 0.8692 | 0.8672 | 0.8060 | 0.7572 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-MMLU) | 0.7075 | 0.6886 | 0.5037** | 0.5228** | 0.4461** | 0.9079 | 0.8576 | 0.8643 | 0.7994 | 0.6998 |  |
| $\rho$ ($\mu_{\text{Mean}}$, m-ARC) | 0.7411 | 0.7754 | 0.6592 | 0.5976 | 0.6494 | 0.8537 | 0.8537 | 0.8927 | 0.6997 | 0.7469 |  |
| $\rho$ ($\mu_{\text{Max}}$, m-ARC) | 0.7293 | 0.7000 | 0.5190** | 0.5335** | 0.4853** | 0.8494 | 0.8309 | 0.8624 | 0.6867 | 0.6885 |  |

We compute sentence embeddings for the selected models using two methods: weighted average based on token positions and last token (see §3.1).
We apply mean and max pooling on the Mexa alignment scores across all model layers to derive a single score for each language. In Table 2, we report the correlation between the Mexa scores (computed using both mean- and max-pooling, for the two embedding methods) and task performances. Across all settings, the best overall result (higher correlation) is achieved when embeddings are computed using the weighted average, with mean pooling as the pooling method. We adopt this configuration as the default setting for Mexa.

FLORES vs Bible. In the default setting, the average Pearson correlation score for the FLORES parallel dataset across different tasks is 0.9300, while for the Bible parallel dataset, it is 0.8779. The reason the Bible scores are generally lower than FLORES is that FLORES data is cleaner and more aligned with modern, standardized texts, whereas the Bible data is older and more specialized. For example for some languages, the orthography of Bible texts no longer matches today’s orthography. In the Bible, Arabic often includes diacritics, which are typically omitted in modern writing and tasks, making the text less familiar to models trained on contemporary data. Additionally, although the Bible dataset has been made parallel, sentence alignment can still be inconsistent due to translation nuances. In contrast, FLORES is carefully curated to ensure high-quality, sentence-level parallelism across languages for machine translation tasks.

Weighted Average vs. Last Token Embeddings. The use of last token embeddings shows promisingly high correlations with the FLORES parallel data; however, for the Bible dataset, the correlation is low in some cases. We believe this may stem from the high occurrence of Bible sentences (especially in English), which leads models to memorize these phrases. Using the WIMBD toolkit (Elazar et al., ), we found that, on average, there are 92 times more documents in Dolma 1.7 containing exact Bible sentences than those in FLORES.
Consequently, when using Bible examples, the last token is biased towards predicting the specific memorized next token rather than incorporating context-related signals. Therefore, one should consider the hazard of memorized data when using last token embeddings. The weighted-average method, which takes into account the influence of multiple tokens, can mitigate the impact of a poor embedding for the last token and enable the model to capture useful information from the other tokens more robustly.

Max Pooling vs. Mean Pooling. In our comparison of mean pooling and max pooling on the Belebele benchmark, we found that mean pooling underestimates low-resource languages (resulting in more Mexa scores near 0), while max pooling correlates better with the Belebele benchmark. This can be explained by the fact that Belebele is an easier task among the three evaluated, allowing even low-resource languages to achieve good scores. Conversely, based on our experiment with m-ARC, max pooling tends to overestimate low-resource languages, making mean pooling more aligned with m-ARC. This can be attributed to m-ARC being the most challenging task among the three, where even medium-resource languages do not achieve high scores. Changing the pooling method from mean to max can be considered when dealing with different levels of understanding.

### 5.2 Downstream Performance Estimation

Figure: Figure 1: The relationship between Mexa scores of Llama 3.1-8B from the Bible and FLORES, adjusted by the English task performance, for tasks in Belebele and the m-ARC benchmark.
Refer to caption: x3.png

A complete Pearson correlation (i.e. $\rho=1.0$) indicates that a linear equation perfectly describes the relationship between Mexa and the evaluation benchmarks, with all data points lying on a line. Given the high correlation values shown in Table 2, it is reasonable to conclude that we can fit a line that closely approximates this linear relationship.
This line converts the Mexa scores back to downstream task performances. We employed a linear model to predict this line by minimizing the residual sum of squares between the Mexa scores (multiplied by the performance on the English task) and the task performances. We needed to adjust the Mexa scores for this purpose, as the Mexa score for language $L_{1}$ indicates how well $L_{1}$ is aligned with English but does not reflect the estimated task performance of the model for language $L_{1}$. Of course, this does not change the Pearson correlation, as it is unaffected by linear transformations.
The three tasks considered in this paper involve multiple-choice questions with four possible answers for each question, resulting in a chance of being randomly correct of $\frac{1}{4}$. However, the minimum score for Mexa scores is 0. Thus, the ideal slope for the line would be $\frac{3}{4}$ with an intercept of $\frac{1}{4}$ (X-axis: adjusted Mexa scores, Y-axis: task performance). In Figure 1, we plot this relationship for Llama 3.1-8B using the Bible and FLORES parallel datasets for Belebele and m-ARC. We chose max pooling for Belebele and mean pooling for m-ARC, since these pooling methods yield a stronger correlation (see §5.1). The pairs of (slope, intercept) from left to right in the Figure 1 are: (0.6804, 0.2477), (0.6103, 0.1838), (0.6340, 0.3408) and (0.5726, 0.2423). With data points from both high-resource and low-resource languages, this line can be calculated; otherwise, the ideal line may be used as a reference.

Language Coverage. We present the adjusted Mexa score for all languages available in FLORES-200 in Table LABEL:tab:whole-flores from Appendix A.5 for the selected models. The languages are categorized into groups ranging from well-covered to not covered. In Table LABEL:tab:whole-flores, we can clearly see that Llama 3.1-70B and Gemma 2-9B show a higher level of multilinguality than other models.

### 5.3 Mexa vs Absolute Cosine Similarity

We compare Mexa with the use of absolute cosine similarities. To begin with, cosine similarity scores are not always directly comparable across models. For example, if a language shows a higher cosine similarity with English for one model than another, it does not necessarily indicate better alignment in the former model. However, Mexa has the advantage of being directly comparable, as its score does not rely on absolute similarity values. To examine the correlation of both methods with downstream tasks, we conducted the following experiment.
We used parallel data from FLORES and downstream task data from the Belebele benchmark, focusing on 116 common labels. For each non-English language, we computed the average absolute cosine similarity for parallel sentences with English, and the average absolute cosine similarity for non-parallel sentences with English. Following the setup by Li et al. (), which employs absolute cosine similarity values to predict the performance and rank of languages, we computed the embeddings using the last-token method and applied mean pooling over layers {5, 10, 15, 20, 25}.
We report results using the Gemma 1 and Llama 1 7B models, which are commonly used in our experiments. For a fair comparison, this setup is applied to both absolute cosine similarity and Mexa. For the Gemma 1 model, Mexa achieves a correlation of 0.9260 with downstream task performance, while the absolute cosine similarity for parallel sentences achieves a correlation of 0.7651. Additionally, the correlation between the absolute cosine similarity for parallel and non-parallel sentences is 0.9232.
For the Llama 1 model, Mexa achieves a correlation of 0.8365 with downstream task performance, while the absolute cosine similarity for parallel sentences achieves a correlation of 0.6473. Additionally, the correlation between the absolute cosine similarity for parallel and non-parallel sentences is 0.9064.
In both models, the absolute cosine similarity method achieved significantly lower correlations with downstream tasks compared to Mexa.
This discrepancy arises primarily because, for some languages, the similarity score can be high regardless of whether the sentences are parallel or non-parallel. Furthermore, a low similarity score between two languages does not necessarily indicate weak alignment, as overall similarity scores may be low while parallel sentences still exhibit much higher scores than non-parallel ones.

### 5.4 Visualization of Layers

In Figure 2, we show the results of applying Mexa to 20 pairs of language_script from FLORES parallel dataset for Llama 1-7B and Llama 3.1-8B across all 32 layers. We selected these languages from different families, writing systems, and both high- and low-resource categories. The embeddings are computed using weighted average based on token positions. Figure 2 shows that high-resource languages (with more prevalence on the web; see §A.1) achieve higher alignment scores across different layers, while low-resource languages achieve lower scores. In the initial layers, embeddings are more in-language, resulting in lower alignment scores. As embeddings progress to the mid-layers, they become more aligned with the main language of the LLM, i.e., English.

Mexa is comparable between models as long as the same parallel dataset and setting is used to obtain the Mexa scores. Figure 2 shows that in many languages, particularly high-resource languages, Llama 3.1 achieves a significantly higher alignment score than its predecessor, Llama 1.
Although Llama 3.1 exhibits better alignment scores with English for medium and low-resource languages, there is still significant room for improvement. Comparing Arabic (arb_Arab) with its romanized version (arb_Latn), we see that both Llama 1 and Llama 3.1 models perform better in the native script than in the Latin script, even though Llama 1’s tokenizer for Arabic is essentially a character-based tokenizer. In general, for very low-resource languages, those in Latin script tend to have higher alignment scores, likely because the tokenization is more favorable for Latin characters.

Figure: Llama 1 7B
Refer to caption: x7.png

In Figure 3, we display the t-SNE (Van der Maaten & Hinton, ) plots of the embeddings of Figure 2 from 3 different layers of Llama 3.1: embedding layer 0, mid-layer 13, and last layer 32. We assign a different color to each language.
For layers 0 and 32, the embeddings are more language-specific, while in the mid-layer, they become more language-neutral. Languages that maintain their language-specific embeddings in the mid-layer are clustered separately and, notably, correspond to the very low-resource languages that receive the worst alignment scores from Mexa.

Figure: Layer 0
Refer to caption: x9.png

## 6 Conclusion

We introduce Mexa, a method for assessing the multilingual capabilities of English-centric large language models (LLMs). Mexa builds on the observation that English-centric LLMs semantically use English as a kind of pivot language in their intermediate layers. Mexa computes the alignment between non-English languages and English using parallel sentences, estimating the transfer of language understanding capabilities from English to other languages through this alignment. This metric can be useful in estimating task performance, provided we know the English performance in the task and the alignment score between languages derived from a parallel dataset. Through different studies with two parallel datasets (FLORES-200 and the Bible); a diverse range of LLMs including the Llama family, Gemma family, Mistral, and OLMo, and three downstream tasks (Belebele, m-MMLU, and m-ARC), we demonstrated that Mexa provides a reliable estimation of multilingual performance. For Mexa score calculations, multiple design analyses are conducted to explore the impact of token-level pooling for embeddings and layer-level pooling in computing alignment scores. While Mexa shows high correlation across most configurations, a weighted average of tokens combined with mean pooling delivers the best results. The results reveal a promising average Pearson correlation of 0.90 with established downstream tasks across nine models and two parallel dataset. Overall, Mexa proves to be a valuable method for practitioners aiming to assess the multilingual capabilities of English-centric LLMs, paving the way for future efforts to expand these models to a wider range of underrepresented languages.

## 7 Limitation

Mexa provides a rough estimate of the multilingual capabilities of pre-trained English-centric LLMs. Different tasks offer diverse perspectives on the abilities of LLMs, and Mexa cannot replace all of them. Our goal is to highlight the multilingual potential of English-centric LLMs and propose a simple way to evaluate them. We hope this encourages the development of more multilingual LLMs, even though they are likely to contain large shares of English data. Additionally, it is important to note that answers across languages do not always need to be fully aligned (Naous et al., ), and for such cases, language- and culture-specific evaluation benchmarks should be developed.

## Acknowledgment

This work was funded by Deutsche Forschungsgemeinschaft (project SCHU 2246/14-1).

## Appendix A Appendix

### A.1 Distribution of Pre-training Data in LLMs

The distribution of languages in the training data of state-of-the-art LLMs is rarely fully documented. Llama 2 (Touvron et al., ) is a counter-example and its authors have disclosed the language distribution use in pretraining.
Their analysis uses the FastText (Bojanowski et al., ) language identification tool and a threshold of $0.5$ for the language detection. We reproduce Touvron et al. (, Table 10), which lists 27 languages with percentages greater than 0.005% in the Llama 2 pre-training data, in Table 3. English, with 89.70%, constitutes the vast majority of the training data.

All the languages listed in Table 3 have a presence of more than 0.10% (top 35 languages) on the web according to the W3Techs report (W3Techs, ) or more than 0.15% (top 36 languages) according to CommonCrawl (first three snapshots of 2024) (Common Crawl, ).
However, not all of the most prevalent languages on the web appear in Table 3. The following 9 languages are missing, most of which use non-Latin writing systems: Turkish (tur_Latn), Persian (pes_Arab), Arabic (ara_Arab), Greek (ell_Grek), Hebrew (heb_Hebr), Thai (tha_Thai), Hindi (hin_Deva), Slovak (slk_Latn), and Lithuanian (lit_Latn).

The distribution of data in the training of English-centric LLMs is not the same as on the web, but it does have some correlation. The amount of English in LLM pre-training data is significantly larger than for other languages. This is also observable for GPT-3 (Brown et al., ), where more than 92%
of the training texts was in English (Brown et al., ). The rest of the top languages in the data of such models are mostly high-resource languages, which have the most available data on the web (top 36 languages). However, in some models, this could be adjusted by design, for example, to make writing systems with non-Latin languages less prominent (as seen in Llama 2).
This weakens the correlation between LLMs’ pretraining data and the web.

**Table 3: Language distribution in the pretraining data for Llama 2. The large “Unknown” category is partially composed of programming code data. Common scripts are sourced from the GlotScript resource (Kargaran et al., ).**
| Language | Common Script | Percent | Language | Common Script | Percent |
| --- | --- | --- | --- | --- | --- |
| English (eng) | Latn | 89.70% | Ukrainian (ukr) | Cyrl | 0.07% |
| Unknown (unk) | - | 8.38% | Korean (kor) | Hang | 0.06% |
| German (deu) | Latn | 0.17% | Catalan (cat) | Latn | 0.04% |
| French (fra) | Latn | 0.16% | Serbian (srp) | Cyrl/Latn | 0.04% |
| Swedish (swe) | Latn | 0.15% | Indonesian (ind) | Latn | 0.03% |
| Chinese (zho) | Hans/Hant | 0.13% | Czech (ces) | Latn | 0.03% |
| Spanish (spa) | Latn | 0.13% | Finnish (fin) | Latn | 0.03% |
| Russian (rus) | Cyrl | 0.13% | Hungarian (hun) | Latn | 0.03% |
| Dutch (nld) | Latn | 0.12% | Norwegian (nor) | Latn | 0.03% |
| Italian (ita) | Latn | 0.11% | Romanian (ron) | Latn | 0.03% |
| Japanese (jpn) | Jpan | 0.10% | Bulgarian (bul) | Cyrl | 0.02% |
| Polish (pol) | Latn | 0.09% | Danish (dan) | Latn | 0.02% |
| Portuguese (por) | Latn | 0.09% | Slovenian (slv) | Latn | 0.01% |
| Vietnamese (vie) | Latn | 0.08% | Croatian (hrv) | Latn | 0.01% |

### A.2 Multilingual Evaluation Benchmarks

Multilingual evaluation methods and the development of benchmarks not only facilitate the assessment of diverse language representations in LLMs but also help in monitoring cross-lingual generalization, to assess the effect of quantization across multiple languages (Marchisio et al., ), the development of language-specific models (Tejaswi et al., ), and the optimization of safety preferences (Li et al., ), among others. In Table 4, we list benchmarks with the largest language coverage. This list includes benchmarks referenced by MEGA (Ahuja et al., ), MEGAVERSE (Ahuja et al., ), xP3 (Muennighoff et al., ), the Aya collection (Singh et al., ), the lm-evaluation-harness framework (Gao et al., ; Biderman et al., ), and inter alia. These datasets comprise a mix of translated datasets, some human-translated or verified by native speakers such as AfriXNLI (Adelani et al., ) and some relying only on machine translation Lai et al. (). Additionally, there are datasets created independently for each language, such as XLSum (Hasan et al., ), where the data is not parallel and the size of the data varies between languages.
Despite the efforts reflected in Table 4, the community is still lacking highly multilingual benchmarks for tasks such as natural language understanding or text generation.

**Table 4: Multilingual evaluation benchmarks**
| Dataset | Task | # Languages |
| --- | --- | --- |
| XNLI (Conneau et al., ) | Natural Language Inference | 15 |
| IndicXNLI (Aggarwal et al., ) | Natural Language Inference | 11 |
| AfriXNLI (Adelani et al., ) | Natural Language Inference | 15 |
| m_HellaSwag (Lai et al., ) | Natural Language Inference | 31 |
| PAWS-X (Yang et al., ) | Paraphrase Identification | 7 |
| XCOPA (Ponti et al., ) | Commonsense Reasoning | 11 |
| XStoryCloze (Lin et al., ) | Commonsense Reasoning | 11 |
| m-ARC (Lai et al., ) | Common Sense Reasoning | 31 |
| TyDiQA (Clark et al., ) | Question Answering | 11 |
| MLQA (Lewis et al., ) | Question Answering | 7 |
| XQuAD (Artetxe et al., ) | Question Answering | 11 |
| IndicQA (Doddapaneni et al., ) | Question Answering | 10 |
| AfriQA (Ogundepo et al., ) | Question Answering | 10 |
| m_TruthfulQA (Lai et al., ) | Multiple Choice Question Answering | 31 |
| UDPOS 2.7 (de Marneffe et al., ) | Part of Speech Tagging | 104 |
| WikiANN (Pan et al., ) | Name Entity Recognition | 282 |
| XLSum (Hasan et al., ) | Summarization | 44 |
| WikiLingua (Ladhak et al., ) | Summarization | 18 |
| Belebele (Bandarkar et al., ) | Multiple Choice Reading Comprehension | 115 |
| AfriMMLU (Adelani et al., ) | Multiple Choice Knowledge Question Answering | 17 |
| m-MMLU (Lai et al., ) | Multiple Choice Knowledge Question Answering | 31 |
| MMMLU (OpenAI, ) | Multiple Choice Knowledge Question Answering | 15 |
| M3Exam (Zhang et al., ) | Multimodal Multiple Choice Knowledge Question Answering | 9 |

### A.3 Semantic Similarity in Multilingual Embeddings

There are other ways to compute similarity between languages, such as Representational Similarity Analysis (RSA) (Chrupała & Alishahi, ) and Central Kernel Alignment (CKA) (Kornblith et al., ). RSA involves first computing the cosine similarity for sentence embeddings within each language, then correlating these in-language similarities with those in other languages. CKA, another metric, is adopted by Conneau et al. () and Muller et al. (). Conneau et al. () show that the CKA similarity is highly correlated with sentence retrieval scores for four languages. In this paper, our focus is not on finding different ways to calculate similarity between languages, but on how helpful a properly defined alignment score can be in estimating the multilingual capabilities of LLMs across multiple languages.

### A.4 Robustness of Mexa

We show that the Mexa alignment score ($\mu(.)$) is very robust, and the odds of this score randomly achieving a high value are very slim. Recall that $\mu\big{(}C(L_{1},L_{2},m,l)\big{)}$ measures the fraction of diagonal elements
in matrix $C(L_{1},L_{2},m,l)$ that have the maximum value in their respective rows and columns. If this condition is met $k$ times out of $n$ diagonal elements, then $\mu\big{(}C(L_{1},L_{2},m,l)\big{)}$ is $\frac{k}{n}$.
In an $n\times n$ random matrix, the probability of a diagonal element being the maximum in its row and column (a total of $2n-1$ elements) is $p=\frac{1}{2n-1}$. The probability that at least $k$ out of $n$ independent variables are satisfied, given that the diagonal element is the maximum in its row and column, can be computed using the binomial distribution with Eq. 3.

$$ $P(X\geq\frac{k}{n})=1-\sum_{i=0}^{k-1}\binom{n}{i}p^{i}(1-p)^{n-i}$ (3) $$

Figure: Figure 4: The probability that at least $k$ out of $n$ diagonal elements in an $n\times n$ random matrix are the maximum elements in their respective rows and columns.
Refer to caption: x12.png

In Figure 4, we plot $P(X\geq\frac{k}{n})$. This plot illustrates that, given a sufficient number of parallel sentences ($n$), the probability of achieving a high score by chance is very low. For example, with $n=100$, the chance of obtaining Mexa alignment score larger than $0.05$ ($k=5$) from a $100\times 100$ random matrix is $P(X\geq 0.05)=0.00016$.

### A.5 Mexa for FLORES-200

We compute Mexa with weighted average embedding and max pooling for the FLORES parallel data for 203 langauge labels, multiplied by the performance of Belebele for each model in English. We show the results in Table LABEL:tab:whole-flores, and color the cells based on 0.2 intervals from green (well-covered) to red (not covered):
(1.0-0.8),
(0.8-0.6), (0.6-0.4),
(0.4-0.2),
(0.2-0).
Note that although FLORES is a high-quality, human-translated dataset, we addressed two major issues before proceeding, as noted by Kargaran et al. (). First, the data labeled as Cantonese (Yue Chinese) is not actually Cantonese, so we removed it. Second, the code for Central Atlas Tamazight (tzm), which actually refers to Standard Moroccan Tamazight (zgh), was renamed accordingly.
As Belebele is relatively an easy task since the models get good scores in English, and we are using max pooling, this gives a high estimate of the coverage the LLMs have. If the score for a language is not very high, it likely indicates that for more challenging tasks, it will remain low.
In Table LABEL:tab:whole-flores, we can clearly see that Llama 3.1-70B and Gemma 2-9B show a higher level of multilinguality than other models.

**Table 5: Adjusted performance of Mexa using max pooling with the English performance of models on the Belebele benchmark.**
|  | Gemma 2 | Gemma 1 | Llama 3.1 | Llama 3.1 | Llama 3 | Llama 2 | Llama 1 | Mistral | OLMo | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 9B | 7B | 70B | 8B | 8B | 7B | 7B | 7B | 7B |  |
| eng_Latn | 0.92 | 0.85 | 0.95 | 0.88 | 0.87 | 0.48 | 0.42 | 0.84 | 0.77 | 0.77 |
| fra_Latn | 0.92 | 0.84 | 0.94 | 0.88 | 0.87 | 0.37 | 0.41 | 0.84 | 0.70 | 0.75 |
| por_Latn | 0.92 | 0.84 | 0.94 | 0.88 | 0.87 | 0.41 | 0.41 | 0.84 | 0.63 | 0.75 |
| deu_Latn | 0.92 | 0.84 | 0.94 | 0.88 | 0.87 | 0.35 | 0.42 | 0.84 | 0.65 | 0.74 |
| spa_Latn | 0.92 | 0.83 | 0.95 | 0.88 | 0.87 | 0.37 | 0.42 | 0.84 | 0.56 | 0.74 |
| ita_Latn | 0.92 | 0.83 | 0.92 | 0.88 | 0.87 | 0.35 | 0.42 | 0.84 | 0.56 | 0.73 |
| cat_Latn | 0.92 | 0.82 | 0.94 | 0.88 | 0.87 | 0.39 | 0.42 | 0.84 | 0.50 | 0.73 |
| nld_Latn | 0.92 | 0.82 | 0.95 | 0.88 | 0.87 | 0.34 | 0.42 | 0.84 | 0.52 | 0.73 |
| rus_Cyrl | 0.91 | 0.82 | 0.94 | 0.88 | 0.87 | 0.34 | 0.41 | 0.83 | 0.51 | 0.72 |
| zho_Hans | 0.91 | 0.80 | 0.94 | 0.88 | 0.87 | 0.32 | 0.34 | 0.81 | 0.62 | 0.72 |
| glg_Latn | 0.92 | 0.83 | 0.91 | 0.88 | 0.87 | 0.31 | 0.41 | 0.82 | 0.52 | 0.72 |
| swe_Latn | 0.92 | 0.83 | 0.95 | 0.88 | 0.87 | 0.38 | 0.42 | 0.84 | 0.37 | 0.72 |
| dan_Latn | 0.92 | 0.83 | 0.94 | 0.88 | 0.87 | 0.31 | 0.41 | 0.82 | 0.44 | 0.71 |
| ces_Latn | 0.92 | 0.82 | 0.95 | 0.88 | 0.87 | 0.26 | 0.42 | 0.84 | 0.43 | 0.71 |
| ron_Latn | 0.92 | 0.82 | 0.94 | 0.88 | 0.87 | 0.23 | 0.41 | 0.83 | 0.48 | 0.71 |
| nob_Latn | 0.91 | 0.82 | 0.95 | 0.88 | 0.87 | 0.34 | 0.39 | 0.81 | 0.39 | 0.71 |
| zho_Hant | 0.91 | 0.81 | 0.94 | 0.88 | 0.87 | 0.31 | 0.32 | 0.79 | 0.52 | 0.71 |
| pol_Latn | 0.92 | 0.81 | 0.95 | 0.88 | 0.87 | 0.22 | 0.42 | 0.84 | 0.38 | 0.70 |
| ast_Latn | 0.90 | 0.80 | 0.91 | 0.88 | 0.86 | 0.21 | 0.40 | 0.77 | 0.49 | 0.69 |
| ind_Latn | 0.92 | 0.83 | 0.93 | 0.87 | 0.87 | 0.22 | 0.30 | 0.82 | 0.42 | 0.69 |
| oci_Latn | 0.89 | 0.75 | 0.95 | 0.88 | 0.87 | 0.22 | 0.39 | 0.81 | 0.40 | 0.68 |
| bos_Latn | 0.91 | 0.81 | 0.95 | 0.88 | 0.87 | 0.19 | 0.41 | 0.84 | 0.25 | 0.68 |
| nno_Latn | 0.92 | 0.82 | 0.92 | 0.84 | 0.84 | 0.26 | 0.36 | 0.78 | 0.38 | 0.68 |
| ukr_Cyrl | 0.92 | 0.81 | 0.95 | 0.88 | 0.87 | 0.22 | 0.42 | 0.84 | 0.15 | 0.67 |
| zsm_Latn | 0.92 | 0.83 | 0.93 | 0.88 | 0.87 | 0.17 | 0.25 | 0.81 | 0.36 | 0.67 |
| hrv_Latn | 0.91 | 0.81 | 0.90 | 0.86 | 0.86 | 0.18 | 0.41 | 0.83 | 0.23 | 0.67 |
| slv_Latn | 0.91 | 0.79 | 0.93 | 0.86 | 0.86 | 0.20 | 0.40 | 0.84 | 0.19 | 0.66 |
| afr_Latn | 0.91 | 0.81 | 0.93 | 0.87 | 0.87 | 0.20 | 0.37 | 0.79 | 0.21 | 0.66 |
| slk_Latn | 0.91 | 0.80 | 0.93 | 0.86 | 0.85 | 0.12 | 0.38 | 0.82 | 0.25 | 0.66 |
| bul_Cyrl | 0.91 | 0.80 | 0.90 | 0.86 | 0.86 | 0.12 | 0.42 | 0.84 | 0.14 | 0.65 |
| jpn_Jpan | 0.90 | 0.80 | 0.93 | 0.83 | 0.82 | 0.29 | 0.25 | 0.76 | 0.24 | 0.65 |
| hun_Latn | 0.91 | 0.78 | 0.92 | 0.84 | 0.83 | 0.13 | 0.39 | 0.81 | 0.18 | 0.64 |
| vec_Latn | 0.87 | 0.74 | 0.93 | 0.84 | 0.83 | 0.16 | 0.35 | 0.76 | 0.28 | 0.64 |
| srp_Cyrl | 0.91 | 0.79 | 0.90 | 0.86 | 0.86 | 0.10 | 0.42 | 0.84 | 0.06 | 0.64 |
| tgl_Latn | 0.91 | 0.74 | 0.94 | 0.82 | 0.82 | 0.16 | 0.20 | 0.77 | 0.36 | 0.64 |
| fin_Latn | 0.91 | 0.79 | 0.90 | 0.85 | 0.85 | 0.14 | 0.21 | 0.74 | 0.33 | 0.64 |
| mkd_Cyrl | 0.90 | 0.77 | 0.94 | 0.87 | 0.86 | 0.07 | 0.38 | 0.80 | 0.11 | 0.63 |
| vie_Latn | 0.91 | 0.81 | 0.95 | 0.88 | 0.87 | 0.22 | 0.08 | 0.79 | 0.16 | 0.63 |
| epo_Latn | 0.87 | 0.76 | 0.95 | 0.86 | 0.85 | 0.14 | 0.26 | 0.67 | 0.11 | 0.61 |
| kor_Hang | 0.88 | 0.74 | 0.91 | 0.84 | 0.83 | 0.22 | 0.15 | 0.71 | 0.15 | 0.60 |
| arb_Arab | 0.91 | 0.80 | 0.94 | 0.86 | 0.85 | 0.05 | 0.17 | 0.70 | 0.10 | 0.60 |
| ars_Arab | 0.91 | 0.80 | 0.93 | 0.86 | 0.85 | 0.04 | 0.17 | 0.69 | 0.08 | 0.59 |
| lim_Latn | 0.76 | 0.65 | 0.89 | 0.83 | 0.83 | 0.21 | 0.25 | 0.59 | 0.21 | 0.58 |
| acq_Arab | 0.91 | 0.78 | 0.92 | 0.83 | 0.82 | 0.04 | 0.13 | 0.67 | 0.09 | 0.58 |
| acm_Arab | 0.90 | 0.76 | 0.90 | 0.86 | 0.82 | 0.04 | 0.14 | 0.67 | 0.09 | 0.57 |
| fur_Latn | 0.73 | 0.60 | 0.91 | 0.81 | 0.77 | 0.16 | 0.27 | 0.59 | 0.26 | 0.57 |
| pes_Arab | 0.91 | 0.79 | 0.88 | 0.85 | 0.85 | 0.05 | 0.08 | 0.59 | 0.07 | 0.56 |
| arz_Arab | 0.88 | 0.74 | 0.90 | 0.84 | 0.83 | 0.03 | 0.10 | 0.63 | 0.09 | 0.56 |
| ajp_Arab | 0.88 | 0.76 | 0.86 | 0.85 | 0.83 | 0.03 | 0.12 | 0.60 | 0.09 | 0.56 |
| lit_Latn | 0.90 | 0.76 | 0.92 | 0.78 | 0.80 | 0.10 | 0.10 | 0.56 | 0.11 | 0.56 |
| apc_Arab | 0.89 | 0.76 | 0.86 | 0.82 | 0.83 | 0.03 | 0.11 | 0.64 | 0.09 | 0.56 |
| ell_Grek | 0.90 | 0.78 | 0.87 | 0.87 | 0.86 | 0.02 | 0.09 | 0.58 | 0.05 | 0.56 |
| tur_Latn | 0.89 | 0.78 | 0.90 | 0.82 | 0.81 | 0.04 | 0.09 | 0.61 | 0.04 | 0.55 |
| est_Latn | 0.90 | 0.77 | 0.90 | 0.82 | 0.83 | 0.12 | 0.09 | 0.45 | 0.10 | 0.55 |
| pap_Latn | 0.79 | 0.60 | 0.89 | 0.75 | 0.73 | 0.18 | 0.22 | 0.56 | 0.23 | 0.55 |
| lmo_Latn | 0.73 | 0.56 | 0.87 | 0.75 | 0.74 | 0.17 | 0.26 | 0.60 | 0.26 | 0.55 |
| szl_Latn | 0.77 | 0.59 | 0.87 | 0.73 | 0.74 | 0.11 | 0.26 | 0.64 | 0.21 | 0.55 |
| prs_Arab | 0.90 | 0.78 | 0.92 | 0.84 | 0.84 | 0.01 | 0.06 | 0.46 | 0.08 | 0.54 |
| scn_Latn | 0.77 | 0.59 | 0.88 | 0.79 | 0.77 | 0.15 | 0.22 | 0.57 | 0.15 | 0.54 |
| heb_Hebr | 0.91 | 0.81 | 0.89 | 0.83 | 0.83 | 0.02 | 0.05 | 0.47 | 0.06 | 0.54 |
| lvs_Latn | 0.90 | 0.75 | 0.90 | 0.81 | 0.79 | 0.05 | 0.05 | 0.55 | 0.08 | 0.54 |
| als_Latn | 0.87 | 0.67 | 0.93 | 0.79 | 0.80 | 0.09 | 0.08 | 0.53 | 0.10 | 0.54 |
| lij_Latn | 0.74 | 0.58 | 0.88 | 0.72 | 0.70 | 0.16 | 0.25 | 0.53 | 0.30 | 0.54 |
| ceb_Latn | 0.83 | 0.59 | 0.89 | 0.73 | 0.72 | 0.16 | 0.15 | 0.49 | 0.24 | 0.53 |
| srd_Latn | 0.73 | 0.59 | 0.86 | 0.75 | 0.72 | 0.16 | 0.23 | 0.55 | 0.21 | 0.53 |
| hin_Deva | 0.90 | 0.74 | 0.91 | 0.80 | 0.79 | 0.03 | 0.05 | 0.44 | 0.06 | 0.53 |
| ltz_Latn | 0.79 | 0.59 | 0.84 | 0.75 | 0.74 | 0.15 | 0.18 | 0.44 | 0.18 | 0.52 |
| tha_Thai | 0.90 | 0.76 | 0.87 | 0.83 | 0.83 | 0.02 | 0.02 | 0.32 | 0.10 | 0.52 |
| aeb_Arab | 0.82 | 0.67 | 0.86 | 0.78 | 0.75 | 0.04 | 0.10 | 0.55 | 0.08 | 0.52 |
| bel_Cyrl | 0.88 | 0.65 | 0.88 | 0.79 | 0.79 | 0.02 | 0.09 | 0.50 | 0.02 | 0.51 |
| isl_Latn | 0.83 | 0.62 | 0.88 | 0.77 | 0.78 | 0.09 | 0.10 | 0.48 | 0.06 | 0.51 |
| swh_Latn | 0.90 | 0.74 | 0.86 | 0.73 | 0.80 | 0.11 | 0.09 | 0.27 | 0.08 | 0.51 |
| mlt_Latn | 0.88 | 0.63 | 0.87 | 0.74 | 0.74 | 0.12 | 0.11 | 0.38 | 0.12 | 0.51 |
| war_Latn | 0.76 | 0.55 | 0.88 | 0.65 | 0.61 | 0.26 | 0.20 | 0.35 | 0.20 | 0.49 |
| cym_Latn | 0.87 | 0.59 | 0.88 | 0.75 | 0.76 | 0.11 | 0.10 | 0.28 | 0.08 | 0.49 |
| fao_Latn | 0.71 | 0.53 | 0.86 | 0.71 | 0.69 | 0.12 | 0.13 | 0.53 | 0.08 | 0.48 |
| urd_Arab | 0.83 | 0.66 | 0.88 | 0.76 | 0.73 | 0.02 | 0.02 | 0.31 | 0.03 | 0.47 |
| jav_Latn | 0.75 | 0.54 | 0.84 | 0.69 | 0.67 | 0.16 | 0.12 | 0.29 | 0.16 | 0.47 |
| eus_Latn | 0.82 | 0.66 | 0.84 | 0.74 | 0.71 | 0.10 | 0.08 | 0.18 | 0.05 | 0.47 |
| sun_Latn | 0.69 | 0.48 | 0.86 | 0.68 | 0.64 | 0.14 | 0.12 | 0.34 | 0.23 | 0.47 |
| kea_Latn | 0.64 | 0.51 | 0.78 | 0.60 | 0.64 | 0.16 | 0.19 | 0.45 | 0.18 | 0.46 |
| ary_Arab | 0.71 | 0.60 | 0.80 | 0.68 | 0.68 | 0.03 | 0.09 | 0.44 | 0.10 | 0.46 |
| hat_Latn | 0.74 | 0.47 | 0.86 | 0.65 | 0.61 | 0.15 | 0.12 | 0.36 | 0.09 | 0.45 |
| mag_Deva | 0.75 | 0.52 | 0.88 | 0.75 | 0.70 | 0.02 | 0.05 | 0.29 | 0.07 | 0.45 |
| min_Latn | 0.56 | 0.45 | 0.81 | 0.68 | 0.68 | 0.16 | 0.12 | 0.34 | 0.17 | 0.44 |
| ban_Latn | 0.48 | 0.32 | 0.82 | 0.62 | 0.63 | 0.17 | 0.15 | 0.42 | 0.30 | 0.43 |
| bjn_Latn | 0.60 | 0.51 | 0.78 | 0.62 | 0.60 | 0.14 | 0.12 | 0.31 | 0.21 | 0.43 |
| azj_Latn | 0.75 | 0.53 | 0.86 | 0.68 | 0.66 | 0.01 | 0.03 | 0.29 | 0.02 | 0.43 |
| npi_Deva | 0.81 | 0.59 | 0.82 | 0.62 | 0.64 | 0.02 | 0.02 | 0.20 | 0.05 | 0.42 |
| mar_Deva | 0.82 | 0.58 | 0.87 | 0.68 | 0.63 | 0.02 | 0.01 | 0.12 | 0.03 | 0.42 |
| awa_Deva | 0.73 | 0.53 | 0.83 | 0.65 | 0.65 | 0.02 | 0.05 | 0.21 | 0.08 | 0.42 |
| ben_Beng | 0.82 | 0.58 | 0.82 | 0.60 | 0.60 | 0.01 | 0.01 | 0.23 | 0.05 | 0.41 |
| uzn_Latn | 0.70 | 0.47 | 0.84 | 0.60 | 0.62 | 0.04 | 0.05 | 0.23 | 0.05 | 0.40 |
| bho_Deva | 0.51 | 0.46 | 0.89 | 0.68 | 0.65 | 0.02 | 0.03 | 0.26 | 0.08 | 0.40 |
| gle_Latn | 0.68 | 0.31 | 0.82 | 0.64 | 0.64 | 0.09 | 0.08 | 0.22 | 0.09 | 0.40 |
| hye_Armn | 0.85 | 0.59 | 0.79 | 0.60 | 0.58 | 0.01 | 0.02 | 0.10 | 0.01 | 0.40 |
| hne_Deva | 0.67 | 0.44 | 0.80 | 0.65 | 0.63 | 0.01 | 0.02 | 0.24 | 0.08 | 0.40 |
| kaz_Cyrl | 0.62 | 0.47 | 0.87 | 0.61 | 0.60 | 0.02 | 0.06 | 0.27 | 0.03 | 0.39 |
| tpi_Latn | 0.69 | 0.38 | 0.69 | 0.46 | 0.45 | 0.19 | 0.15 | 0.33 | 0.15 | 0.39 |
| hau_Latn | 0.68 | 0.41 | 0.77 | 0.58 | 0.54 | 0.12 | 0.11 | 0.17 | 0.06 | 0.38 |
| mai_Deva | 0.61 | 0.43 | 0.86 | 0.61 | 0.59 | 0.02 | 0.03 | 0.16 | 0.08 | 0.38 |
| crh_Latn | 0.58 | 0.42 | 0.77 | 0.56 | 0.51 | 0.03 | 0.07 | 0.35 | 0.05 | 0.37 |
| ilo_Latn | 0.61 | 0.32 | 0.74 | 0.47 | 0.46 | 0.17 | 0.15 | 0.24 | 0.13 | 0.37 |
| tat_Cyrl | 0.56 | 0.38 | 0.83 | 0.58 | 0.55 | 0.03 | 0.04 | 0.21 | 0.02 | 0.36 |
| kat_Geor | 0.73 | 0.45 | 0.72 | 0.53 | 0.50 | 0.01 | 0.04 | 0.18 | 0.01 | 0.35 |
| ydd_Hebr | 0.74 | 0.45 | 0.78 | 0.48 | 0.47 | 0.03 | 0.02 | 0.05 | 0.02 | 0.34 |
| kir_Cyrl | 0.53 | 0.33 | 0.81 | 0.58 | 0.57 | 0.01 | 0.04 | 0.13 | 0.02 | 0.34 |
| pag_Latn | 0.33 | 0.23 | 0.63 | 0.35 | 0.39 | 0.25 | 0.21 | 0.36 | 0.24 | 0.33 |
| pan_Guru | 0.78 | 0.50 | 0.75 | 0.47 | 0.40 | 0.01 | 0.00 | 0.03 | 0.05 | 0.33 |
| bak_Cyrl | 0.56 | 0.36 | 0.82 | 0.51 | 0.51 | 0.01 | 0.04 | 0.13 | 0.02 | 0.33 |
| guj_Gujr | 0.79 | 0.52 | 0.67 | 0.42 | 0.39 | 0.02 | 0.01 | 0.04 | 0.07 | 0.33 |
| tam_Taml | 0.78 | 0.54 | 0.72 | 0.38 | 0.38 | 0.02 | 0.01 | 0.08 | 0.03 | 0.33 |
| pbt_Arab | 0.50 | 0.23 | 0.82 | 0.57 | 0.57 | 0.02 | 0.03 | 0.10 | 0.07 | 0.32 |
| tgk_Cyrl | 0.62 | 0.27 | 0.78 | 0.51 | 0.52 | 0.02 | 0.03 | 0.10 | 0.05 | 0.32 |
| tel_Telu | 0.77 | 0.52 | 0.67 | 0.38 | 0.43 | 0.01 | 0.01 | 0.05 | 0.04 | 0.32 |
| snd_Arab | 0.59 | 0.30 | 0.78 | 0.53 | 0.50 | 0.02 | 0.01 | 0.06 | 0.04 | 0.31 |
| kan_Knda | 0.74 | 0.47 | 0.66 | 0.42 | 0.41 | 0.01 | 0.01 | 0.06 | 0.05 | 0.31 |
| mal_Mlym | 0.76 | 0.50 | 0.68 | 0.32 | 0.30 | 0.01 | 0.01 | 0.03 | 0.03 | 0.29 |
| ckb_Arab | 0.51 | 0.20 | 0.78 | 0.54 | 0.50 | 0.01 | 0.01 | 0.05 | 0.02 | 0.29 |
| gla_Latn | 0.46 | 0.14 | 0.71 | 0.46 | 0.45 | 0.08 | 0.07 | 0.13 | 0.07 | 0.29 |
| asm_Beng | 0.63 | 0.35 | 0.70 | 0.39 | 0.36 | 0.00 | 0.01 | 0.08 | 0.04 | 0.29 |
| tuk_Latn | 0.49 | 0.31 | 0.63 | 0.43 | 0.41 | 0.05 | 0.04 | 0.14 | 0.02 | 0.28 |
| san_Deva | 0.48 | 0.26 | 0.71 | 0.46 | 0.45 | 0.00 | 0.01 | 0.12 | 0.02 | 0.28 |
| kmr_Latn | 0.38 | 0.15 | 0.69 | 0.48 | 0.50 | 0.05 | 0.06 | 0.13 | 0.05 | 0.28 |
| lus_Latn | 0.53 | 0.09 | 0.56 | 0.34 | 0.33 | 0.14 | 0.10 | 0.24 | 0.11 | 0.27 |
| khk_Cyrl | 0.44 | 0.18 | 0.73 | 0.42 | 0.43 | 0.01 | 0.02 | 0.08 | 0.03 | 0.26 |
| ltg_Latn | 0.31 | 0.23 | 0.61 | 0.38 | 0.35 | 0.08 | 0.06 | 0.22 | 0.06 | 0.26 |
| azb_Arab | 0.37 | 0.28 | 0.60 | 0.44 | 0.44 | 0.00 | 0.01 | 0.11 | 0.02 | 0.25 |
| plt_Latn | 0.52 | 0.17 | 0.59 | 0.25 | 0.25 | 0.14 | 0.12 | 0.18 | 0.05 | 0.25 |
| ibo_Latn | 0.35 | 0.15 | 0.64 | 0.38 | 0.37 | 0.09 | 0.08 | 0.12 | 0.07 | 0.25 |
| mri_Latn | 0.35 | 0.11 | 0.60 | 0.38 | 0.35 | 0.12 | 0.10 | 0.18 | 0.07 | 0.25 |
| som_Latn | 0.42 | 0.14 | 0.60 | 0.24 | 0.24 | 0.11 | 0.08 | 0.18 | 0.09 | 0.23 |
| ace_Latn | 0.22 | 0.13 | 0.49 | 0.32 | 0.31 | 0.14 | 0.10 | 0.21 | 0.15 | 0.23 |
| xho_Latn | 0.49 | 0.19 | 0.47 | 0.20 | 0.20 | 0.12 | 0.10 | 0.13 | 0.05 | 0.22 |
| nso_Latn | 0.27 | 0.11 | 0.48 | 0.26 | 0.23 | 0.17 | 0.13 | 0.19 | 0.07 | 0.21 |
| sot_Latn | 0.34 | 0.12 | 0.53 | 0.22 | 0.20 | 0.14 | 0.10 | 0.18 | 0.05 | 0.21 |
| zul_Latn | 0.55 | 0.19 | 0.44 | 0.19 | 0.17 | 0.11 | 0.06 | 0.10 | 0.05 | 0.21 |
| kin_Latn | 0.37 | 0.10 | 0.53 | 0.20 | 0.23 | 0.11 | 0.09 | 0.15 | 0.06 | 0.21 |
| sin_Sinh | 0.56 | 0.27 | 0.49 | 0.26 | 0.18 | 0.01 | 0.01 | 0.03 | 0.02 | 0.20 |
| smo_Latn | 0.28 | 0.09 | 0.66 | 0.20 | 0.20 | 0.10 | 0.08 | 0.13 | 0.06 | 0.20 |
| nya_Latn | 0.36 | 0.13 | 0.41 | 0.19 | 0.19 | 0.13 | 0.10 | 0.17 | 0.06 | 0.19 |
| twi_Latn | 0.23 | 0.08 | 0.46 | 0.22 | 0.22 | 0.14 | 0.13 | 0.19 | 0.07 | 0.19 |
| sna_Latn | 0.41 | 0.17 | 0.40 | 0.19 | 0.20 | 0.10 | 0.08 | 0.12 | 0.07 | 0.19 |
| uig_Arab | 0.21 | 0.09 | 0.71 | 0.29 | 0.29 | 0.00 | 0.01 | 0.03 | 0.02 | 0.18 |
| bug_Latn | 0.14 | 0.12 | 0.35 | 0.22 | 0.22 | 0.14 | 0.11 | 0.20 | 0.12 | 0.18 |
| luo_Latn | 0.07 | 0.07 | 0.40 | 0.25 | 0.24 | 0.15 | 0.12 | 0.21 | 0.09 | 0.18 |
| tsn_Latn | 0.24 | 0.10 | 0.42 | 0.18 | 0.18 | 0.12 | 0.11 | 0.20 | 0.06 | 0.18 |
| arb_Latn | 0.29 | 0.07 | 0.46 | 0.24 | 0.20 | 0.05 | 0.05 | 0.17 | 0.08 | 0.18 |
| khm_Khmr | 0.34 | 0.15 | 0.59 | 0.15 | 0.16 | 0.01 | 0.02 | 0.09 | 0.06 | 0.17 |
| lua_Latn | 0.09 | 0.08 | 0.33 | 0.20 | 0.21 | 0.14 | 0.13 | 0.24 | 0.12 | 0.17 |
| lug_Latn | 0.17 | 0.07 | 0.41 | 0.18 | 0.19 | 0.14 | 0.09 | 0.19 | 0.06 | 0.17 |
| grn_Latn | 0.17 | 0.09 | 0.44 | 0.16 | 0.17 | 0.12 | 0.09 | 0.13 | 0.10 | 0.16 |
| ssw_Latn | 0.27 | 0.10 | 0.37 | 0.17 | 0.17 | 0.11 | 0.08 | 0.14 | 0.05 | 0.16 |
| lin_Latn | 0.11 | 0.08 | 0.43 | 0.16 | 0.18 | 0.12 | 0.11 | 0.16 | 0.08 | 0.16 |
| ory_Orya | 0.28 | 0.03 | 0.66 | 0.18 | 0.19 | 0.01 | 0.01 | 0.03 | 0.03 | 0.16 |
| fij_Latn | 0.13 | 0.06 | 0.38 | 0.18 | 0.16 | 0.14 | 0.11 | 0.15 | 0.08 | 0.15 |
| fuv_Latn | 0.07 | 0.08 | 0.30 | 0.20 | 0.20 | 0.13 | 0.10 | 0.18 | 0.10 | 0.15 |
| kas_Arab | 0.16 | 0.10 | 0.50 | 0.20 | 0.21 | 0.02 | 0.02 | 0.10 | 0.05 | 0.15 |
| quy_Latn | 0.10 | 0.06 | 0.42 | 0.21 | 0.22 | 0.10 | 0.06 | 0.13 | 0.05 | 0.15 |
| aka_Latn | 0.17 | 0.06 | 0.37 | 0.14 | 0.17 | 0.11 | 0.08 | 0.12 | 0.10 | 0.15 |
| mya_Mymr | 0.36 | 0.13 | 0.46 | 0.14 | 0.16 | 0.00 | 0.00 | 0.02 | 0.02 | 0.15 |
| run_Latn | 0.25 | 0.07 | 0.37 | 0.16 | 0.17 | 0.08 | 0.06 | 0.10 | 0.04 | 0.14 |
| bem_Latn | 0.14 | 0.08 | 0.29 | 0.16 | 0.16 | 0.13 | 0.11 | 0.15 | 0.06 | 0.14 |
| kas_Deva | 0.14 | 0.09 | 0.37 | 0.20 | 0.21 | 0.02 | 0.03 | 0.11 | 0.08 | 0.14 |
| wol_Latn | 0.09 | 0.07 | 0.30 | 0.18 | 0.16 | 0.12 | 0.10 | 0.17 | 0.07 | 0.14 |
| kam_Latn | 0.09 | 0.08 | 0.26 | 0.18 | 0.16 | 0.13 | 0.10 | 0.15 | 0.08 | 0.14 |
| tso_Latn | 0.14 | 0.06 | 0.35 | 0.14 | 0.13 | 0.11 | 0.08 | 0.13 | 0.06 | 0.13 |
| kon_Latn | 0.07 | 0.08 | 0.27 | 0.15 | 0.17 | 0.09 | 0.07 | 0.13 | 0.09 | 0.13 |
| tum_Latn | 0.15 | 0.07 | 0.32 | 0.11 | 0.13 | 0.09 | 0.08 | 0.11 | 0.05 | 0.13 |
| kik_Latn | 0.07 | 0.04 | 0.32 | 0.12 | 0.13 | 0.10 | 0.09 | 0.13 | 0.12 | 0.12 |
| taq_Latn | 0.06 | 0.06 | 0.28 | 0.14 | 0.12 | 0.11 | 0.08 | 0.14 | 0.05 | 0.12 |
| mos_Latn | 0.04 | 0.04 | 0.25 | 0.16 | 0.14 | 0.11 | 0.09 | 0.15 | 0.07 | 0.12 |
| yor_Latn | 0.13 | 0.04 | 0.30 | 0.14 | 0.14 | 0.08 | 0.06 | 0.10 | 0.04 | 0.11 |
| amh_Ethi | 0.48 | 0.16 | 0.24 | 0.04 | 0.03 | 0.01 | 0.02 | 0.03 | 0.02 | 0.11 |
| sag_Latn | 0.05 | 0.07 | 0.22 | 0.17 | 0.17 | 0.07 | 0.07 | 0.09 | 0.06 | 0.11 |
| cjk_Latn | 0.06 | 0.06 | 0.21 | 0.13 | 0.12 | 0.10 | 0.08 | 0.13 | 0.07 | 0.11 |
| umb_Latn | 0.05 | 0.05 | 0.20 | 0.15 | 0.14 | 0.10 | 0.08 | 0.11 | 0.05 | 0.10 |
| dyu_Latn | 0.04 | 0.04 | 0.22 | 0.13 | 0.12 | 0.06 | 0.07 | 0.10 | 0.08 | 0.10 |
| kac_Latn | 0.02 | 0.03 | 0.22 | 0.12 | 0.14 | 0.06 | 0.06 | 0.12 | 0.08 | 0.09 |
| kmb_Latn | 0.05 | 0.06 | 0.20 | 0.11 | 0.10 | 0.10 | 0.07 | 0.09 | 0.05 | 0.09 |
| bam_Latn | 0.05 | 0.05 | 0.18 | 0.11 | 0.09 | 0.08 | 0.08 | 0.12 | 0.04 | 0.09 |
| ayr_Latn | 0.04 | 0.04 | 0.20 | 0.11 | 0.10 | 0.06 | 0.05 | 0.10 | 0.06 | 0.08 |
| lao_Laoo | 0.17 | 0.04 | 0.22 | 0.07 | 0.09 | 0.02 | 0.02 | 0.04 | 0.09 | 0.08 |
| dik_Latn | 0.05 | 0.06 | 0.18 | 0.06 | 0.07 | 0.08 | 0.07 | 0.10 | 0.05 | 0.08 |
| ewe_Latn | 0.04 | 0.03 | 0.18 | 0.08 | 0.08 | 0.09 | 0.07 | 0.08 | 0.04 | 0.08 |
| knc_Latn | 0.05 | 0.06 | 0.15 | 0.08 | 0.08 | 0.07 | 0.06 | 0.08 | 0.05 | 0.08 |
| kab_Latn | 0.04 | 0.02 | 0.17 | 0.09 | 0.08 | 0.06 | 0.06 | 0.11 | 0.04 | 0.07 |
| sat_Olck | 0.19 | 0.02 | 0.32 | 0.05 | 0.05 | 0.00 | 0.00 | 0.01 | 0.01 | 0.07 |
| gaz_Latn | 0.05 | 0.03 | 0.20 | 0.06 | 0.06 | 0.05 | 0.04 | 0.08 | 0.03 | 0.07 |
| bod_Tibt | 0.07 | 0.01 | 0.22 | 0.08 | 0.08 | 0.01 | 0.01 | 0.02 | 0.02 | 0.06 |
| fon_Latn | 0.03 | 0.02 | 0.14 | 0.06 | 0.06 | 0.03 | 0.04 | 0.05 | 0.07 | 0.06 |
| shn_Mymr | 0.02 | 0.01 | 0.21 | 0.06 | 0.06 | 0.01 | 0.02 | 0.03 | 0.07 | 0.05 |
| kbp_Latn | 0.03 | 0.02 | 0.14 | 0.05 | 0.04 | 0.03 | 0.02 | 0.08 | 0.05 | 0.05 |
| mni_Beng | 0.03 | 0.02 | 0.12 | 0.05 | 0.06 | 0.01 | 0.02 | 0.08 | 0.02 | 0.05 |
| ace_Arab | 0.03 | 0.02 | 0.15 | 0.07 | 0.07 | 0.00 | 0.00 | 0.03 | 0.01 | 0.04 |
| knc_Arab | 0.01 | 0.01 | 0.13 | 0.04 | 0.04 | 0.02 | 0.02 | 0.05 | 0.05 | 0.04 |
| bjn_Arab | 0.03 | 0.02 | 0.11 | 0.05 | 0.08 | 0.01 | 0.01 | 0.05 | 0.01 | 0.04 |
| nus_Latn | 0.02 | 0.02 | 0.07 | 0.04 | 0.03 | 0.03 | 0.02 | 0.04 | 0.05 | 0.03 |
| min_Arab | 0.02 | 0.01 | 0.13 | 0.05 | 0.04 | 0.01 | 0.00 | 0.03 | 0.01 | 0.03 |
| tir_Ethi | 0.10 | 0.02 | 0.05 | 0.02 | 0.02 | 0.01 | 0.01 | 0.02 | 0.02 | 0.03 |
| dzo_Tibt | 0.01 | 0.00 | 0.08 | 0.03 | 0.03 | 0.00 | 0.00 | 0.01 | 0.01 | 0.02 |
| taq_Tfng | 0.00 | 0.00 | 0.04 | 0.01 | 0.01 | 0.00 | 0.00 | 0.01 | 0.03 | 0.01 |
| zgh_Tfng | 0.00 | 0.00 | 0.02 | 0.01 | 0.01 | 0.00 | 0.00 | 0.01 | 0.01 | 0.01 |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: Table 2 is complicated to read. Please consider reorganizing to improve readability.
soundness: 3
strengths: - The paper proposes a simple approach for prediction of multilingual performance, that requires few resources and is easy to implement. 
- The experiments are done on a wide range of languages, showing inclusivity and making the results solid. 
- The related work on understanding alignment is well-covered.
summary: In this paper, the authors propose a method to predict the performance of English-centric LLMs on other languages using information about the alignment between English and other languages in the LLMs. The central hypothesis here is that the LLMs performance on a task is primarily dependent on the English performance, and performance on other languages depends on how well its representations are aligned with English. The paper proposes a measure MEXA to compute the similarity of representations between languages and shows that MEXA score is high correlated to the language performance on a task. Results are shown on 3 classification tasks and multiple, diverse languages including low-resource languages. MEXA score only requires a small parallel corpus to compute the alignment scores between the languages.
weaknesses: - There is prior work on prediction of performance of multilingual NLP models based on various features like linguistic similarity, vocab overlap, embedding similarity, pre-training data size etc. These have not been discussed. This is directly relevant to the work, and in fact the proposed work can be considered a special case of many of these approaches, where only a single feature - the MEXA score is used for performance prediction. In the light of these works, the novelty is limited. The moot question to answer is how good is just a single feature - the MEXA score compared to using a plethora of features that previous works have used. A discussion with respect to previous work would be useful. 
- The work focusses only on classification tasks, and does not cover generation tasks. When the NLP landscape have moved towards generative capabilties, evaluating prediction of generation quality is also important. This has been mentioned as a limitation of the work in the paper. 
- The comparison with absolute cosine similarity is very limited (Section 5.3). To solidly establish the benefits of MEXA over cosine similarity, it would be good to report absolute cosine similarity results with the same default settings (weighted average, mean pooling) across all tasks and models.  

References: 

1. Ahuja, Kabir, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. "Multi task learning for zero shot performance prediction of multilingual models." ACL. 2022.
2. Ye, Zihuiwen, Pengfei Liu, Jinlan Fu, and Graham Neubig. "Towards more fine-grained and reliable NLP performance prediction." EACL. (2021).
3. Xia, Mengzhou, et al. “Predicting Performance for Natural Language Processing Tasks.” Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2020, pp. 8625–46. Crossref, https://doi.org/10.18653/v1/2020.acl-main.764.
4. Srinivasan, Anirudh, Sunayana Sitaram, Tanuja Ganu, Sandipan Dandapat, Kalika Bali, and Monojit Choudhury. "Predicting the performance of multilingual nlp models." arXiv preprint arXiv:2110.08875 (2021).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: - The authors use only 100 examples in their evaluation which is a very small number while they have more examples available. Even if they want to keep the compute small, they should try a few models on more examples and may show that their approach require a little as 100 parallel examples.

- The authors mention several tasks in Appendix but then decided to use only three tasks. I did not understand the relation behind this choice and why authors did not try to run of a larger number of tasks to show the generalization of their methodology. Authors also mentioned that the primary focus of their paper is natural understanding task. As far as I understood, their technique is general and should work for a diverse set of tsaks.

- Presentation of results in tables make it very hard to digest them. Authors may select one best strategy like mean/max and present the results. Table 1 and 2 both are very hard to understand. 

- What is the motivation of keeping the proposed method limited to monolingual models? What about multilingual models? Expanding the experiments to multilingual model will greatly increase the impact and breadth of the proposed method.

- I did not understand which of the final combination of layers work the best for MEXA? Is it a combination of all layers or a specific layer? From Figure 2, it seems that the alignment scores are best for middle layers. Is it correct? What is the optimal layer for MEXA?


- A few arguments need empirical support or support from literature. - Line 50-51: for these models to be effective in other languages, it is important that the other languages align with the main language. The basis of this assumption is unclear and authors should support it empirically or from literature. 

- What is the exact prompt given to models and what is their impact?

- Figures and tables are extremely small. I was unable to read them on paper. I am not sure this small size is even allowed by the conference. Authors should move a few models to appendix and present fewer models.
soundness: 2
strengths: The approach is simple and empirical results are quite strong.
summary: The paper presents a method, MEXA, to evaluate the multilingual strength or capability of English centric LLMs. MEXA is an intrinsic evaluation method that utilizes a parallel corpus and calculate the embedding distances between parallel sentences in comparison to two random sentences in two different languages. A number of evaluations are conducted using several 7B models and using approximately 100 parallel examples from two datasets. The results show that their proposed method strongly correlates with the multilingual performance of the under-observation models.
weaknesses: I put my questions and weaknesses in the questions section and would encourage authors to respond to them.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. How is the MEXA score for English in the first row of Table 1 calculated, given that the MEXA formulation is designed to compute alignment between English and a non-English language?

2. Prior works (https://arxiv.org/abs/2405.14782) have demonstrated that likelihood based evaluation and generative evaluation exhibit different trends, how would MEXA correlations change when switched to a generative evaluation?

3. Since the distribution of MEXA scores may vary depending on the model, the statement in lines 431-432 may not be appropriate to make.

4. How does one compare MEXA scores across different translation datasets to determine whether the trends remain consistent? If they are inconsistent, what criteria should guide the selection of the most appropriate translation dataset?

5. For different models, what is the optimal layer to compute MEXA scores? Does the choice of layer vary significantly between models, and for a new model, would it be necessary to conduct an independent analysis to determine the optimal layer, or could insights from existing experiments offer a reliable estimate?

6. Echo embeddings would serve as a valuable baseline and a useful ablation in this analysis (see: https://arxiv.org/abs/2402.15449).
soundness: 3
strengths: The problem is very well motivated, particularly given the lack of multilingual benchmarks and such alignment based methods would be valuable to estimate multilingual performance in a resource-efficient manner. Such methods would be useful for determining model readiness for production deployment in specific languages.

The proposed method is straightforward and does not require any creation of additional sophisticated benchmarks or extra resources and is largely model-agnostic.

The paper is well-structured and easy to follow.

The authors perform exhaustive experiments across a wide range of languages and models, offering valuable, in-depth insights.
summary: This paper presents MEXA, a method to evaluate the multilingual capabilities of English-centric LLMs. Given the lack of multilingual benchmarks available, multilingual capabilities of English-centric LLMs may not be well evaluated across diverse languages. 
MEXA relies on parallel sentences to calculate alignment scores between English and non-English languages to measure the transfer of language understanding from English to other languages. The authors ablate around optimal choices for layer and aggregation strategies in computing embeddings. The authors evaluate the MEXA across three multiple-choice style downstream tasks and observe that the MEXA scores demonstrates a strong correlation with downstream task performance.
weaknesses: 1. The MEXA formulation relies on relative scoring and does not account for the magnitude of cosine similarity. For example, in cases where scores are close to each other but also have low magnitudes, this minimal difference is inconsequential and does not necessarily imply alignment. Thus, the limitations of standard cosine similarity, as discussed in Section 5.3, are also relevant here. In the NMT literature, following bitext mining, which incorporates relative scores, an absolute scoring mechanism, such as a final cosine similarity threshold, is often applied to retain only high-quality pairs. Additionally, margin-based approaches have consistently shown better performance than standard cosine similarity methods (see: https://aclanthology.org/2021.acl-long.507/). Adding an absolute component to the formulation, would be beneficial.

2. The authors report correlation scores but omit absolute scores. Including absolute scores, at least for select tasks, would provide a more comprehensive evaluation and further strengthen the paper.

3. The authors’ evaluation is restricted to discriminative tasks, omitting open-ended text generation tasks. It is therefore unclear whether MEXA scores correlate with the LLM's multilingual text-generation capabilities.

4. FLORES (derived from Wikipedia) and BIBLE (potentially a component of the pre-training data) are used as parallel datasets; and many benchmarks may also be Wikipedia-based, however, it remains uncertain whether MEXA is generalizable in terms of demonstrating reliable correlations with other parallel datasets from unseen distributions. Further, the authors do not provide any method of determining the optimal choice of the parallel data to be utilized in the MEXA score computation.

5. The paper’s ablations are somewhat conventional, with previous studies having compared similar design choices. While the plots offer some valuable insights, the authors only discuss these superficially and omit in-depth analysis. For instance, since Belebele derives from FLORES, one would ideally expect it to exhibit a superior correlation with the prediction of downstream performance on Belebele. However, as illustrated in Figure 5.2, the Bible dataset exhibits a stronger correlation. The authors should explore and discuss such deviations from expected outcomes in their paper. Another example is the contrast between romanized versus native script performance, something that is actively being explored in literature, (see: https://aclanthology.org/2024.acl-long.833/, https://arxiv.org/abs/2201.12501). While the authors briefly discuss this aspect, they do not extend their analysis to other languages. Such detailed insights would add substantial value and enhance the overall assessment of the paper.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: The writing in some parts of the paper could be made a bit clearer, specifically lines 139--147 and 197--202. Sentences like “This criterion imposes…”, “This suggests…”, “Another motivation…” are somewhat ambiguous in the first reading. It’s not clear if the authors are referring to their motivations for introducing a new concept or citing the motivations of previous research.
soundness: 4
strengths: (1) The main finding – that Mexa serves as a way to estimate downstream task performance based on cross-lingual alignment – is robustly supported by the results. Experiments were conducted across several LLMs, languages, and tasks, and Mexa exhibits very high correlation with task performance (both in terms of absolute correlation values and compared to cosine similarity.

(2) The authors have explored a number of design decisions for Mexa (which token representations to use to obtain sentence embeddings, how to pool embeddings) and present a full analysis of their findings. This provides practitioners with a clear guide of how to use and interpret Mexa in different contexts.

(3) The study conducted by the authors represent a valuable contribution to our understanding of multilingual LLMs. They build on previous findings and produce new insights into the mechanisms underlying cross-lingual transfers in LLMs.
summary: This paper proposes Mexa, a metric for quantifying cross-lingual alignment between language representations in LLMs. Most LLMs are English-centric and achieve multilingual performance in part by using English as a pivot language. This occurs in the middle layers of LLMs, where sentence representations are less language-specific, exhibiting cross-lingual alignment (similarity in vector representations). The degree of cross-lingual alignment of parallel sentences indicates to what extent English is utilised as a pivot language. Mexa quantifies this by comparing the cosine similarity of parallel sentences to non-parallel sentences, producing a metric that captures to what extent cross-lingual parallel sentences are representationally aligned. The authors present Mexa as a way to quantify cross-lingual transfer and estimate task performance based on how aligned languages are with English. Their results support the viability of this, with Mexa achieving comfortably higher correlation coefficients with downstream task performance than cosine similarity. The authors test different ways of computing sentence representations for Mexa, finding that different strategies are more effective for different types of downstream tasks.
weaknesses: (1) The authors seem to assume at the outset that cross-lingual representational similarity WILL entail better cross-lingual transfer. This assumption is the motivating factor behind Mexa. Their results do strongly suggest this, but they should be more cautious in stating this as a hypothesis (or assumption) at the start of the paper, or provide a survey of previous work proving the relation between alignment and downstream performance. It’s possible that alignment helps to some extent, but that too much cross-lingual overlap can impede performance e.g. for high-resource languages, where using English as a pivot might not be optimal. This might or might not be the case. My point is that the authors gloss over the assumption that more alignment implies better downstream performance for other languages. I see it more as a hypothesis that their findings strongly support. 

(2) Mexa is not complex as a technical contribution. It is a simple way to encode relative cross-lingual similarity. While this in itself does not degrade its value as a contribution (its impact is clearly useful), it does not introduce impressive novelty from a technical standpoint.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: 1. The idea of an "English-centric" LLM is referenced several times in the paper, but I do not see a concrete definition of "English-centric" anywhere in the paper. From my understanding, the intent of Mexa is to evaluate only English-centric LLMs, so it would be good to define what you mean by this so that practitioners know whether or not to use Mexa for a given LLM.

2. L252 "Although there are multilingual models [...] our focus here is on LLMs which are **state-of-the-art in English based tasks**": Why is the focus only on evaluating SOTA LLMs? The intention of Mexa is to be a public benchmark and leaderboard, so it will be used for a variety of LLMs, right? (including e.g. some that are really focused on one task so aren't SOTA across a variety of tasks). I am thinking it would be good to evaluate on a greater variety of LLMs for this purpose.

3. More generally, as a suggestion, I wonder if positioning this work as a benchmark was the best strategy. It was only evaluated on some relatively strict cases (open-weights/open-source models that are English-centric and decoder-only/Llama-esque and state-of-the-art on English-based tasks) which I think will hinder its usability as a benchmark (if evaluations do indeed need to be limited to such models). It may be better to position it as analysis?
soundness: 2
strengths: 1. The proposed Mexa approach is interesting and novel. I could see this approach inspiring future work in analysis of multilingual LLM performance.

2. It is impressively accurate at predicting the performance of several decoder-only English-centric LLMs on 3 different multilingual datasets
summary: This paper proposes Mexa, which is a leaderboard for estimating the cross-lingual abilities of English-based large language models. Mexa works by using a large corpus of parallel sentences (e.g., FLORES or Bible corpora) and calculating the semantic similarity score between the given model's embeddings of the English and non-English parallel sentences. These are considered aligned if the similarity between the parallel sentences is greater than between any two non-parallel sentences. This score is aggregated over the whole corpus, and it is found to be a strong predictor of performance on downstream non-English tasks (modulo the corresponding English performance on the task). This is a useful benchmark because parallel sentences and English benchmarks are widely available, but multilingual benchmarks are less readily available.
weaknesses: 1. Lack of variety in LLMs considered. All LLMs evaluated are quite similar: they are all decoder-only models with very similar architectures. For a benchmark to be useful it needs to be reliable on arbitrary models including new ones that might be invented in the future, so it would be good to evaluate on a wider variety of architectures. Even something like Mixtral would be a good place to start. This is particularly important since some of the assumptions behind Mexa (e.g. in the paragraph at line 49-63, about English being the pivot language in the middle layers) seem based on analysis of Llama so may not hold as well when evaluating models with other architectures.

2. Benchmarks used. Mexa is quite good at predicting performance on the three multilingual benchmarks considered (table 2). However, it's not clear to me why only three benchmarks are considered; table 4 lists several more that are available. In particular, both m-MMLU and m-ARC rely on machine translation to generate the multilingual data, which means that they are more likely to align with the semantic similarity measures used, so I am not sure whether this generalizes to performance on realistic multilingual tasks. Also, there is a bit of tuning on these datasets (weighted average vs. last token, max pooling vs. mean pooling, FLORES vs. bible), and there is no held-out benchmark to truly evaluate generalization.

3. Inability to benchmark closed-source models. Unless I misunderstood, Mexa requires access to the weights to be able to evaluate a model. This makes it relatively impractical: practitioners may want to select a model by using Mexa to estimate performance on their desired task, but they will not be able to evaluate closed-source models.

Meta-review: The paper proposes Mexa, a method to assess the multilingual capabilities of English-centric large language models (LLMs) by evaluating the alignment of representations between English and other languages using parallel datasets. Mexa aims to predict task performance across multiple languages without relying on pretraining data or multilingual benchmarks, which are scarce for low-resource languages. The results demonstrate a strong correlation between Mexa scores and downstream task performance, achieving a Pearson correlation of 0.90 on average across nine models and two parallel datasets. Experiments are performed on decoder-only models, specifically the Llama, Gemma, Mistral, and OLMo families. The paper concludes that Mexa provides insights into cross-lingual transfer mechanisms and serves as a practical, resource-efficient measure for estimating multilingual performance.

The paper’s strengths lie in its motivation, simplicity, and comprehensiveness of experiments. The authors address an important problem of evaluating multilingual performance for English-centric LLMs, especially for low-resource languages. The approach is lightweight, requiring only parallel data, and achieves significant correlation with multilingual task performance. The experiments are exhaustive across languages and models, offering practical and valuable insights. The paper is also well-structured, making the contributions accessible to the broader NLP community.

However, several critical weaknesses limit the paper's overall contribution. First, the scope of evaluation is narrow, as the authors focus exclusively on classification tasks, ignoring generative tasks. Despite claiming the method’s generalizability, its applicability to text generation remains unclear. Second, the paper lacks sufficient comparison with absolute cosine similarity and other existing metrics for performance prediction. Although Mexa performs better, the evidence provided is incomplete. Third, the choice of parallel datasets (FLORES and Bible) raises concerns about generalizability. The FLORES dataset heavily relies on Wikipedia-based translations, while the Bible dataset may be partially memorized by LLMs. The paper fails to explore datasets from unseen distributions, which undermines the robustness of their claims. Fourth, the focus on decoder-only, open-weight LLMs limits Mexa's practical applicability, as closed-source models cannot be evaluated. Furthermore, the assumption that English serves as a latent pivot language across layers is presented without sufficient empirical support or justification, potentially reducing the generalizability of the findings. Finally, the presentation of results is overly complex and difficult to interpret, as highlighted by reviewers.

Given these limitations, the contributions of Mexa, though interesting, are incremental relative to prior work on predicting multilingual performance using alignment metrics or embedding similarity. While the authors provided detailed rebuttals, most responses were promises to address weaknesses in a revised version rather than concrete updates, and critical concerns remain unaddressed. Due to the limited novelty of the method, combined with the restricted evaluation scope and concerns about generalizability, the paper does not meet the bar for acceptance at ICLR. Thus, I recommend rejecting the paper in its current form.

Decision: Reject