Rethinking Efficient Multilingual Text Summarization Meta-Evaluation (2024)

Abstract: Evaluating multilingual summarization evaluation metrics, i.e., meta-evaluation, is challenging because of the difficulty of human annotation collection. Therefore, we investigate an efficient multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual meta-evaluation dataset into multilingual versions. To this end, we introduce a statistical test to verify the transformed dataset quality by checking the metaevaluation result consistency on the original dataset and back-translated dataset. With this quality verification method, we transform an existing English summarization meta-evaluation dataset, RoSE, into 30 languages, and conduct a multilingual meta-evaluation of several representative automatic evaluation metrics. In our meta-evaluation, we find that metric performance varies in different languages and neural metrics generally outperform classical textmatching-based metrics in non-English languages. Moreover, we identify a two-stage evaluation method with superior performance, which first translates multilingual texts into English and then performs evaluation. We make the transformed datasets publicly available to facilitate future research.

heading: 1 Introduction
text: The evaluation of automatic evaluation metrics, or meta-evaluation, is critical for the development and evaluation of text summarization models. However, conducting a meta-evaluation study is difficult and resource-intensive because it can require a large collection of human annotations for calculating the correlations between metric and human evaluation. This is especially true in multilingual contexts, where resources are scarce and recent advancements have predominantly centered around English. As a result, there are only a few human annotation benchmarks for summarization meta-evaluation in
* Equal contribution
recent years (Huang et al., 2020; Bhandari et al., 2020; Fabbri et al., 2021; Zhang and Bansal, 2021; Gao and Wan, 2022; Liu et al., 2023), and even fewer that are multilingual (Aharoni et al., 2023; Clark et al., 2023; Hada et al., 2023).
The limited availability of multilingual metaevaluation datasets makes it challenging to evaluate automatic metrics in multilingual settings. Therefore, we aim to explore an efficient multilingual meta-evaluation strategy based on monolingual data. Specifically, inspired Braun et al. (2022), we revisit and explore the possibility of transforming a monolingual (English) meta-evaluation dataset into a multilingual dataset through neural machine translation (NMT), which can avoid the difficulty of collecting human annotations in different languages, especially the low-resource languages. To this end, our study includes two main parts:
We first investigate the soundness and feasibility of our NMT-based dataset transformation approach. Specifically, we identify the key question to be whether the translated dataset is of good quality and supports meaningful multilingual summarization meta-evaluation. To verify this question, we propose a dataset quality measurement strategy based on neural back-translation, or round-trip translation (Mallinson et al., 2017). Specifically, we hypothesize that if a translated dataset is of good quality, then its back-translated dataset in English should yield consistent meta-evaluation results as the original English dataset for the same automatic metric. To measure this consistency, we adopt a statistical test, Two One-Sided Test (TOST) (Schuirmann, 1987), which allows us to assess this consistency for our use case. While Braun et al. (2022) also applies TOST for checking dataset quality, our test is fundamentally different since they aim to check whether metrics perform similarly on the translated dataset, while we focus on the back-translated dataset, which avoids the metric performance discrepancy in different
15739
languages. Furthermore, to better understand the effect of translation quality in our task, we compare two NMT systems for dataset transformation, including a domain-specific NMT model, M2M100 (Fan et al., 2021), and a generic large language model (LLM), GPT-3.5 (Ouyang et al., 2022). Our results indicate that translation quality is crucial in our context, and the data quality testing method we propose is sensitive to the performance of the translation systems employed during our dataset construction.
After verifying the soundness of our proposed dataset transformation approach, we conduct a multilingual summarization meta-evaluation on the dataset transformed into 30 languages from a recently introduced English summarization metaevaluation dataset, RoSE (Liu et al., 2023). This results in a much larger dataset compared with previous work (Braun et al., 2022) in terms of both the number of languages (30 v.s. 7) and data examples in the datasets. To ensure meaningful evaluation results, we use the TOST results we obtained as a filtering criterion – we accept only the evaluation results that can pass the consistency test we proposed. Our meta-evaluation highlights the metric performance discrepancy in different languages and reveals their limitations. Furthermore, we identify a two-stage automatic evaluation approach, which first translates multilingual texts into English and then applies automatic metrics to the translated text, achieving better average performance than applying the metrics to multilingual texts directly.
Our contributions are three-fold: (1) We conduct a thorough analysis of an efficient meta-evaluation method using NMT-based data transformation and design a statistical test procedure to verify the soundness of this approach.
(2) We conduct a multilingual meta-evaluation in 30 languages, shedding light on the limitations of current summarization evaluation metrics.
(3) We release the transformed datasets in all languages to facilitate future research.1

heading: 2 Dataset Creation
text: To circumvent the difficulty of collecting multilingual human annotations, we apply a data creation strategy transforming existing monolingual datasets into a multilingual one using NMT systems. Specifically, we choose RoSE (Liu et al.,
1The datasets are available at https://github.com/ yale-nlp/MRoSE.
2023), a recently introduced summarization metaevaluation dataset for our study. RoSE contains human annotations for reference-based summary salience evaluation based on a fine-grained protocol, Atomic Content Units (ACUs), which aims to reduce the subjectivity of reference-based summarization human evaluation. We use its subset from the CNN/DailyMail (Nallapati et al., 2016) test set in our experiments since it contains longer summaries with higher complexity. It contains 500 distinct documents, each accompanied by 12 humanannotated scores of system-generated summaries.
We use two NMT systems to transform RoSE into 30 languages: (1) M2M-100, a multilingual encoder-decoder model primarily intended for translation tasks, which is chosen because it supports a wide range of languages; (2) GPT3.5 (Ouyang et al., 2022),2 an LLM that has shown strong performance on machine translation especially when translating from and to a pivot language, e.g., English (Jiao et al., 2023). The 30 languages we selected are listed in Table 1. Representing 16 diverse language families, these languages ensure a thorough linguistic range.
To evaluate the translation quality, we use roundtrip translation to translate the translated dataset back to English. We then use a few standard automatic metrics to compare the similarity between the back-translated dataset and the original English dataset. We generally observe a high variance in performance, with mostly resource-rich languages like German, Italian, and French achieving a high score (e.g., Rouge-2 of 70+%), while some other languages achieving lower performance (e.g., Bengali, Japanese and Tamil with Rouge-2 of 30% or less). The full results are in Appendix A.
2gpt-3.5-turbo is used, described in https:// platform.openai.com/docs/models/gpt-3-5.

heading: 3 Dataset Analysis
text: We design and conduct statistical analyses to verify whether our dataset obtained through machine translation can support reliable evaluation of automatic evaluation metrics.
Our dataset quality verification process relies on a key assumption – the meta-evaluation performed on the original dataset and the backtranslated dataset should yield consistent results if the dataset transformation is reliable. With this assumption, we leverage the Two One-Sided Test (TOST) (Schuirmann, 1987) to check this consistency property, which is frequently utilized to test whether the difference between two groups is negligible. Specifically, a small difference in meta-evaluation results between the original dataset and its back-translated counterpart can indicate the maintained quality in our translated dataset. Here, the meta-evaluation results are the correlations between the metric and the human evaluation results (Bhandari et al., 2020; Deutsch et al., 2021).3 Therefore, we apply TOST to test the equivalence of the human-metric correlations on the original dataset (ρorig) and the back-translated dataset (ρback). To execute TOST, we formulated both null and alternative hypotheses:
H0 : ∣∣ρorig − ρback ∣∣ > ∆e, H1 : ∣∣ρorig − ρback ∣∣ < ∆e (1)
The null hypothesis, represented by H0, assumes that the absolute difference between two correlations cannot be ignored when considering a predefined equivalence margin ∆e. Conversely, the
3We report the summary-level correlations since it is more challenging than system-level correlations (Liu et al., 2023).
alternative hypothesis (H1) suggests that the difference is small enough to infer that the two correlations are nearly identical. By conducting TOST, our goal is to identify the equivalence margin (∆e) at which we can reject the null hypothesis and conclude that two correlations are sufficiently close. For each automatic metric we are going to evaluate, we execute TOST with bootstrapping (Tibshirani and Efron, 1993) on the datasets translated into 30 languages with Kendall rank correlation as the correlation coefficient, and search for the margin ∆e by identifying the point at which the null hypothesis can be rejected (p < 0.05).
We report the TOST results on the datasets translated by M2M-100 and GPT-3.5 in Figure 1. For M2M-100, we find that the majority of the metaevaluation results cannot be regarded as equivalent under an equivalence margin (∆e) of 0.1. On the other hand, for GPT-3.5, the majority of the results are considered equivalent with the same margin.
The test provides a systematic way to assess the translated dataset quality and highlights the importance of using a strong NMT system for the dataset transformation. Therefore, we propose a data-filtering method for the actual multilingual meta-evaluation – with a pre-defined margin (∆e), the meta-evaluation data point (i.e., one metric evaluated on one language) that cannot pass TOST will be discarded. As a case study, we set this margin to 0.1 in our study, noting that it can be adjusted based on the level of error tolerance.

heading: 4 Evaluating Multiligual Automatic Summarization Evaluation
text: We now conduct a meta-evaluation of different multilingual metrics on the dataset we obtained using GPT-3.5 as the NMT system.

heading: 4.1 Metrics
text: We evaluate a few widely used automatic metrics for evaluating textual similarity, including textmatching based metrics: (1) ROUGE-1,2,L (Lin, 2004), (2) BLEU (Papineni et al., 2002), (3) METEOR (Banerjee and Lavie, 2005); and neural metrics: (1) BERTScore (Zhang et al., 2019b), (2) BARTScore (Yuan et al., 2021), (3) MoverScore (Zhao et al., 2019). For each neural metric, we use their multilingual version powered by base multilingual language models. Specifically, we use the bert-base-multilingual-cased version for BERTScore and MoverScore, and mbart-large-50 for BARTScore which is based on MBART (Tang et al., 2021).4 We note that since RoSE contains system summary recall scores against the reference summary, we use the recall score of the metrics when available.
We also investigate a two-stage automatic evaluation strategy – using an NMT model to translate the text into English first, then applying the metrics to the translated English text. Therefore, for each metric we also evaluate their performance on English text translated by GPT-3.5.

heading: 4.2 Meta Evaluation
text: We evaluate the metric performance on the translated datasets by calculating the correlation between the metric scores and human-annotated
4The model checkpoints are from Hugging Face Models: https://huggingface.co/models.
scores. As mentioned above, we use the TOST results we obtained in §3 to filter out the invalid evaluation results. The results on the datasets translated by GPT-3.5 are in Figure 2, and the results with M2M-100 translations can be found in Appendix B. We note several key findings: (1) Metric performance varies across different languages, and in general they perform worse on non-alphabetical languages. For example, most metrics achieve worse performance in Chinese (zh) than in German (de), indicating their limitations in multilingual evaluation settings. (2) Different metrics show large performance gaps on certain languages. Specifically, textmatching based metrics such as ROUGE and BLEU are less robust than neural metrics such as BERTScore. We believe this is because neural metrics based on multilingual language models can better capture the semantic similarity than ROUGE. (3) NMT-based two-stage evaluation methods achieve strong performance. Specifically, all metrics we evaluated tend to perform better when evaluating the translated English summaries than evaluating the original summaries in different languages, with a few exceptions such as BERTScore in Chinese (zh) or ROUGE-2 in Vietnamese (vi).

heading: 5 Conclusion
text: In this work, we investigate an efficient multilingual summarization meta-evaluation approach by translating a monolingual dataset into a multilingual version. We develop a statistical testing procedure to verify the transformed dataset quality to ensure the validity of the meta-evaluation results. With this procedure, we compare the dataset quality translated by two NMT systems, and we found that the translation quality is critical for multilin-
gual meta-evaluation. We then present a case study of transforming a monolingual meta-evaluation dataset into 30 languages with high-quality translations, and conduct a multilingual meta-evaluation of representative automatic evaluation metrics on the transformed dataset.

heading: 6 Limitations
text: Our work utilizes the NMT systems to transform a monolingual meta-evaluation dataset into a multilingual version. While we have developed statistical tests to evaluate the quality of the translated dataset, we recognize that it is not a replacement for human evaluation of the dataset quality. However, we found it difficult to conduct such human evaluation in multilingual settings across many languages, which we leave for more comprehensive future work. Similarly, in our meta-evaluation study, we only report the quantitative metric performance by computing their correlations with the human evaluation results. We believe that a qualitative study with human evaluation would provide a more in-depth understanding of the metric performance and behavior, which is an important next step to reliability extend the summarization meta-evaluation research into multilingual settings.
We acknowledge that there are other metrics we did not include in our meta-evaluation, which could have made this meta-evaluation study more complete. Besides, we only investigate one quality dimension, the summary salience. It is possible that our data construction process can have a different impact on the other quality dimensions such as summary coherence or factuality.
We note that the (back-)translation technique we employed can lead to several concerns. First, our translated multilingual dataset may have different distributions/characteristics compared to datasets that are originally multilingual mostly because of issues in translationese (Wang et al., 2023). Moreover, since we did not employ human evaluations to check the translated dataset, it is possible that some of the translation errors (e.g., failing to translate from English to another language) are not captured in our data quality verification method.

heading: Acknowledgements
text: We thank the anonymous reviewers for their invaluable comments and suggestions. We are grateful to the OpenAI’s Researcher Access Program for API credit support.

heading: A Translation Quality
text: In §2, we apply two NMT systems to transform the English meta-evaluation dataset into multilingual datasets. To have an initial check of the translation quality, in Table 2 and Table 3 we report the metric scores between the original English reference summaries and the back-translated English reference summaries.

heading: B Additional Meta-Evaluation Results
text: In §4.2, we report the meta-evaluation results on the datasets translated by GPT-3.5. As a reference,
Language (Code) BLEU ROUGE-1 ROUGE-2
in Figure 3 we report the meta-evaluation results on datasets translated by M2M-100. We note that since most of the subsets translated by M2M-100 cannot pass our equivalence test, TOST, the metaevaluation results on these subsets are unreliable.

heading: C Additional Related Work
text: Summarization Evaluation Neural text summarization has seen great success in recent years (Liu and Lapata, 2019; Zhang et al., 2019a; Yang et al., 2020). Evaluation metrics like ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), and METEOR (Banerjee and Lavie, 2005) traditionally assess n-gram overlap, while neural metrics such as BERTScore (Zhang et al., 2019b), MoverScore (Zhao et al., 2019), and BARTScore (Yuan et al., 2021) utilize neural language models for similarity measurement. To verify the effectiveness of such summarization metrics, many related studies conduct meta-evaluation, which involves calculating the correlation between the automatic metric and human-annotated scores (Bhandari et al., 2020;
Huang et al., 2021; Fabbri et al., 2021; Pagnoni et al., 2021; Zhang and Bansal, 2021; Gao and Wan, 2022; Liu et al., 2023). Most of these studies are conducted on English corpora only, which is likely because of the limitations of datasets, summarization systems and metrics, and the difficulty of conducting multilingual human evaluation.
Multilingual Summarization Datasets Despite the rapid progress of text summarization research, the availability of multilingual text summarization datasets remains limited. MLSUM (Scialom et al., 2020) is a pioneering multilingual summarization dataset with 1.5 million article-summary pairs in five languages (French, German, Spanish, Russian, and Turkish). Another dataset, XLSUM (Hasan et al., 2021), contains news articles from the BBC news website and the associate summaries in 45 languages. WikiLingua (Ladhak et al., 2020) is a large-scale dataset focusing on cross-lingual abstractive summarization, with 141,000 English articles and parallel articles for 17 languages. There are even fewer multilingual summarization metaevaluation datasets. Among them, mFACE (Aharoni et al., 2023) evaluates summarization faithfulness on XLSum across 45 languages and releases the human annotations. SEAHORSE (Clark et al., 2023), another meta-evaluation dataset, includes 96,000 summaries across 6 languages and 9 summarization systems with multi-dimensional human annotations of summary quality. Hada et al. (2023) conducted a meta-evaluation of LLM-based evaluators under multilingual settings in 9 languages. They found that while LLMs can achieve high agreements with human annotators on referencefree summarization evaluation, the LLMs’ performance varies across different languages. There are also more recent cross-lingual summarization datasets, including CrossSum (Bhattacharjee et al., 2023) and PMIndiaSum (Urlana et al., 2023).

heading: Future Work
text: However, we found it difficult to conduct such human evaluation in multilingual settings across many languages, which we leave for more comprehensive future work. Similarly, in our meta-evaluation study, we only report the quantitative metric performance by computing their correlations with the human evaluation results. We believe that a qualitative study with human evaluation would provide a more in-depth understanding of the metric performance and behavior, which is an important next step to reliability extend the summarization meta-evaluation research into multilingual settings. We acknowledge that there are other metrics we did not include in our meta-evaluation, which could have made this meta-evaluation study more complete. Besides, we only investigate one quality dimension, the summary salience. It is possible that our data construction process can have a different impact on the other quality dimensions such as summary coherence or factuality. We note that the (back-)translation technique we employed can lead to several concerns. First, our translated multilingual dataset may have different distributions/characteristics compared to datasets that are originally multilingual mostly because of issues in translationese (Wang et al. , 2023). Moreover, since we did not employ human evaluations to check the translated dataset, it is possible that some of the translation errors (e. g. , failing to translate from English to another language) are not captured in our data quality verification method