MUG-Eval: A Proxy Evaluation Framework for Multilingual Generation
  Capabilities in Any Language
Authors: name: Seyoung Song; name: Seogyeong Jeong; name: Eunsu Kim; name: Jiho Jin; name: Dongkwan Kim; name: Jay Shin; name: Alice Oh
Published: 2025-05-20 14:14:00

Abstract: Evaluating text generation capabilities of large language models (LLMs) is
challenging, particularly for low-resource languages where methods for direct
assessment are scarce. We propose MUG-Eval, a novel framework that evaluates
LLMs' multilingual generation capabilities by transforming existing benchmarks
into conversational tasks and measuring the LLMs' accuracies on those tasks. We
specifically designed these conversational tasks to require effective
communication in the target language. Then, we simply use task success rate as
a proxy of successful conversation generation. Our approach offers two key
advantages: it is independent of language-specific NLP tools or annotated
datasets, which are limited for most languages, and it does not rely on
LLMs-as-judges, whose evaluation quality degrades outside a few high-resource
languages. We evaluate 8 LLMs across 30 languages spanning high, mid, and
low-resource categories, and we find that MUG-Eval correlates strongly with
established benchmarks (r > 0.75) while enabling standardized comparisons
across languages and models. Our framework provides a robust and
resource-efficient solution for evaluating multilingual generation that can be
extended to thousands of languages.