MEXA: Multilingual Evaluation of English-Centric LLMs via Cross-Lingual
  Alignment
Authors: name: Amir Hossein Kargaran, name: Ali Modarressi, name: Nafiseh Nikeghbal, name: Jana Diesner, name: François Yvon, name: Hinrich Schütze
Organization: CIS, LMU Munich

English-centric large language models (LLMs) often show strong multilingual
capabilities. However, the multilingual performance of these models remains
unclear and is not thoroughly evaluated for many languages. Most benchmarks for
multilinguality focus on classic NLP tasks, or cover a minimal number of
languages. We introduce MEXA, a method for assessing the multilingual
capabilities of pre-trained English-centric LLMs using parallel sentences,
which are available for more languages than existing downstream tasks. MEXA
leverages the fact that English-centric LLMs use English as a kind of pivot
language in their intermediate layers. It computes the alignment between
English and non-English languages using parallel sentences to evaluate the
transfer of language understanding from English to other languages. This
alignment can be used to estimate model performance in other languages. We
conduct studies using various parallel datasets (FLORES-200 and Bible), models
(Llama family, Gemma family, Mistral, and OLMo), and established downstream
tasks (Belebele, m-MMLU, and m-ARC). We explore different methods to compute
embeddings in decoder-only models. Our results show that MEXA, in its default
settings, achieves a statistically significant average Pearson correlation of
0.90 with three established downstream tasks across nine models and two
parallel datasets. This suggests that MEXA is a reliable method for estimating
the multilingual capabilities of English-centric LLMs, providing a clearer
understanding of their multilingual potential and the inner workings of LLMs.
Leaderboard: https://huggingface.co/spaces/cis-lmu/Mexa, Code:
https://github.com/cisnlp/Mexa.