Cross-lingual Evaluation of Multilingual Generative AI
Authors: Shamil Chollampatt, Minh-Quang PHAM, Sathish Reddy Indurthi, Marco Turchi


Reviewer paper summaries: This paper presents a novel, reference-free approach to evaluating the quality of multilingual text generation. The proposed protocol demonstrates a stronger correlation with ROUGE-2 than G-Eval in meta-evaluation, and its application to various language models reveals some insights into their multilingual generative capabilities.
Summary of strengths: **Motivation**: The paper's motivation is well presented, highlighting the limitations of existing evaluation methods that rely on human-written references or LLM scorers with limited multilingual capabilities.

**Originality and Implication**: The proposed approach is novel, eliminating the need for references and LLM scorers in evaluating text generation. This method can be applied to any language, provided there is a strong MT system for translation into English and a cross-lingual metric that supports the language.

**Substance**: The paper presents extensive experiments using the proposed evaluation protocol on various LLMs and languages, revealing interesting patterns regarding the multilingual abilities of these models and their training data.
Summary of weaknesses: **Soundness**: While the proposed approach is somewhat original, I have some concerns regarding its reliability. 
- Firstly, the paper only computes the correlation with the ROUGE-2 metric in meta-evaluation, which has been shown to have low correlation with human evaluations as LLMs improve [1]. Without a correlation with human evaluation, it is difficult to determine whether this approach can be trusted.
- Additionally, the protocol essentially involves obtaining a reference summary from a strong model (e.g., GPT4) and computing similarity between the model's generation and the reference generated by this strong model using cross-lingual metrics. This approach shares the drawbacks of traditional reference-based metrics, such as issues with reference quality and bias towards generations similar to the reference, etc.



[1] News Summarization and Evaluation in the Era of GPT-3. https://arxiv.org/abs/2209.12356
Comments, suggestions and typos: See strengths and weaknesses above.
Overall assessment scores: 2.5
Metareview score: 2
Metareview: The paper proposes a method to evaluate multi-lingual text-generation tasks by leveraging a multi-stage pipeline that translates inputs into English, uses a strong LLM (in English) to perform the task, and uses cross-eval evaluation metric to compare the English LLM-output to a target model's output in the non-English language. The methodology is evaluated text summarization, finding the method's scores correlate better than G-Eval (an external baseline) with ROUGE-2. The method is then applied across 90 languages to assess multi-lingual model ability at summarization.
Summary of reasons to publish: - Multi-lingual text generation evaluation is challenging and merits new ideas.
- The work proposes clever ideas on leveraging model performance in a well-represented language (English) to gain insights on less-represented languages, which is insightful.