The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
Authors: name: Minghao Wu; name: Weixuan Wang; name: Sinuo Liu; name: Huifeng Yin; name: Xintong Wang; name: Yu Zhao; name: Chenyang Lyu; name: Longyue Wang; name: Weihua Luo; name: Kaifu Zhang
Published: 2025-04-22 01:47:37

Abstract: As large language models (LLMs) continue to advance in linguistic
capabilities, robust multilingual evaluation has become essential for promoting
equitable technological progress. This position paper examines over 2,000
multilingual (non-English) benchmarks from 148 countries, published between
2021 and 2024, to evaluate past, present, and future practices in multilingual
benchmarking. Our findings reveal that, despite significant investments
amounting to tens of millions of dollars, English remains significantly
overrepresented in these benchmarks. Additionally, most benchmarks rely on
original language content rather than translations, with the majority sourced
from high-resource countries such as China, India, Germany, the UK, and the
USA. Furthermore, a comparison of benchmark performance with human judgments
highlights notable disparities. STEM-related tasks exhibit strong correlations
with human evaluations (0.70 to 0.85), while traditional NLP tasks like
question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30).
Moreover, translating English benchmarks into other languages proves
insufficient, as localized benchmarks demonstrate significantly higher
alignment with local human judgments (0.68) than their translated counterparts
(0.47). This underscores the importance of creating culturally and
linguistically tailored benchmarks rather than relying solely on translations.
Through this comprehensive analysis, we highlight six key limitations in
current multilingual evaluation practices, propose the guiding principles
accordingly for effective multilingual benchmarking, and outline five critical
research directions to drive progress in the field. Finally, we call for a
global collaborative effort to develop human-aligned benchmarks that prioritize
real-world applications.