GlotEval: A Test Suite for Massively Multilingual Evaluation of Large
  Language Models
Authors: name: Hengyu Luo; name: Zihao Li; name: Joseph Attieh; name: Sawal Devkota; name: Ona de Gibert; name: Shaoxiong Ji; name: Peiqin Lin; name: Bhavani Sai Praneeth Varma Mantina; name: Ananda Sreenidhi; name: Raúl Vázquez; name: Mengjie Wang; name: Samea Yusofi; name: Jörg Tiedemann
Published: 2025-04-05 12:30:58

Abstract: Large language models (LLMs) are advancing at an unprecedented pace globally,
with regions increasingly adopting these models for applications in their
primary language. Evaluation of these models in diverse linguistic
environments, especially in low-resource languages, has become a major
challenge for academia and industry. Existing evaluation frameworks are
disproportionately focused on English and a handful of high-resource languages,
thereby overlooking the realistic performance of LLMs in multilingual and
lower-resource scenarios. To address this gap, we introduce GlotEval, a
lightweight framework designed for massively multilingual evaluation.
Supporting seven key tasks (machine translation, text classification,
summarization, open-ended generation, reading comprehension, sequence labeling,
and intrinsic evaluation), spanning over dozens to hundreds of languages,
GlotEval highlights consistent multilingual benchmarking, language-specific
prompt templates, and non-English-centric machine translation. This enables a
precise diagnosis of model strengths and weaknesses in diverse linguistic
contexts. A multilingual translation case study demonstrates GlotEval's
applicability for multilingual and language-specific evaluations.