Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs
Authors: name: Sumanth Doddapaneni; name: Mohammed Safi Ur Rahman Khan; name: Dilip Venkatesh; name: Raj Dabre; name: Anoop Kunchukuttan; name: Mitesh M. Khapra
Published: 2024-10-17 09:45:32

Abstract: Evaluating machine-generated text remains a significant challenge in NLP,
especially for non-English languages. Current methodologies, including
automated metrics, human assessments, and LLM-based evaluations, predominantly
focus on English, revealing a significant gap in multilingual evaluation
frameworks. We introduce the Cross Lingual Auto Evaluation (CIA) Suite, an
extensible framework that includes evaluator LLMs (Hercule) and a novel test
set (Recon) specifically designed for multilingual evaluation. Our test set
features 500 human-annotated instructions spanning various task capabilities
along with human judgment scores across six languages. This would enable
benchmarking of general-purpose multilingual LLMs and facilitate
meta-evaluation of Evaluator LLMs. The proposed model, Hercule, is a
cross-lingual evaluation model that addresses the scarcity of reference answers
in the target language by learning to assign scores to responses based on
easily available reference answers in English. Our experiments demonstrate that
Hercule aligns more closely with human judgments compared to proprietary
models, demonstrating the effectiveness of such cross-lingual evaluation in low
resource scenarios. Further, it is also effective in zero-shot evaluation on
unseen languages. This study is the first comprehensive examination of
cross-lingual evaluation using LLMs, presenting a scalable and effective
approach for multilingual assessment. All code, datasets, and models will be
publicly available to enable further research in this important area.