An Empirical Study of Many-to-Many Summarization with Large Language Models
Authors: Jiaan Wang, Fandong Meng, Zengkui Sun, Yunlong Liang, Yuxuan Cao, Jiarong Xu, HAOXIANG SHI, Jie Zhou
Accepted at: acl25

An Empirical Study of Many-to-Many Summarization with Large Language Models

Jiaan Wang1*, Fandong Meng1*, Zengkui Sun4, Yunlong Liang1, Yuxuan Cao5
Jiarong Xu2, Haoxiang Shi3 and Jie Zhou1
1Pattern Recognition Center, WeChat AI, Tencent Inc, China   2Fudan University
3Inner Mongolia University of Technology   4Beijing Jiaotong University   5Zhejiang University
{torchwang,fandongmeng,yunlonliang,withtomzhou}@tencent.com

Abstract
Many-to-many summarization (M2MS) aims to process documents in any language and generate the corresponding summaries also in any language. Recently, large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform M2MS in real applications. This work presents a systematic empirical study on LLMs’ M2MS ability. Specifically, we first reorganize M2MS data based on eight previous domain-specific datasets. The reorganized data contains 47.8K samples spanning five domains and six languages, which could be used to train and evaluate LLMs. Then, we benchmark 18 LLMs in a zero-shot manner and an instruction-tuning manner. Fine-tuned traditional models (e.g., mBART) are also conducted for comparisons. Our experiments reveal that, zero-shot LLMs achieve competitive results with fine-tuned traditional models. After instruct-tuning, open-source LLMs can significantly improve their M2MS ability, and outperform zero-shot LLMs (including GPT-4) in terms of automatic evaluations. In addition, we demonstrate this task-specific improvement does not sacrifice the LLMs’ general task-solving abilities. However, as revealed by our human evaluation, LLMs still face the factuality issue, and the instruction tuning might intensify the issue. Thus, how to control factual errors becomes the key when building LLM summarizers in real applications, and is worthy to be noted in future research.

1 Introduction
Many-to-Many Summarization (M2MS) is proposed to generate a brief summary in any language given a document also in any language (c.f., Figure 1) (Wang et al., 2023c; Bhattacharjee et al., 2023). This task is extremely challenging since it requires the ability to summarize and translate across many languages. Meanwhile, many LLMs adopt the multi-lingual setting to share the language modeling across various languages (Touvron et al., 2023a; OpenAI, 2023), making it possible to become an advanced M2MS solver in theory. However, there is still a lack of practice in exploring LLMs’ M2MS performance.

In this paper, we try to investigate how LLMs can perform the M2MS task in real applications with multi-domain scenarios. Considering the limited diversity and the single-domain characteristic in each existing dataset, a single dataset cannot be directly used to benchmark LLMs in multi-domain scenarios. Thus, we first reorganize and select M2MS samples from eight existing multi-lingual summarization datasets (Ladhak et al., 2020; Fatima and Strube, 2021; Perez-Beltrachini and Lapata, 2021; Wang et al., 2022b; Chen et al., 2023; Bhattacharjee et al., 2023; Zheng et al., 2023). These datasets cover five domains, i.e., news, how-to guides, encyclopedia, dialogue, and technology, allowing the transformation of shareable knowledge across different domains. During sample selection, we consider the intrinsic quality metrics (coverage, redundancy and coherence), and balance the number of samples among different languages and domains. For the testing samples, we also consider data contamination to ensure fair evaluations. After that, there are 47.8K samples used to train and evaluate models in our study.

![Illustration of many-to-many summarization.](page_1012_362_393_246.png)

Figure 1: Illustration of many-to-many summarization.
Then, we benchmark 18 human-aligned LLMs, including open- and closed-source LLMs. We evaluate their zero-shot M2MS ability by prompting them with task-specific instruction and in-context examples. In this way, LLMs can leverage their instruction-following ability to perform M2MS without any parameter updating. Besides, we fine-tune two state-of-the-art traditional models (Tang et al., 2021; Wang et al., 2023c) for comparison. Our experiments reveal that the zero-shot LLMs could achieve competitive results with fine-tuned traditional models, showing the promising task-solving ability of LLMs. Furthermore, we train the open-source LLMs to perform M2MS via instruction tuning. We find that open-source LLMs can significantly improve their M2MS ability through instruct-tuning, and outperform the original models and traditional models by a large margin. Some tuned LLMs can even outperform zero-shot GPT-4 in terms of automatic metrics. In addition, we evaluate the original LLMs and the instruction-tuned LLMs on MMLU (Hendrycks et al., 2021). The results demonstrate the improvement brought by instruction-tuning does not sacrifice the LLMs’ general task-solving abilities.

Moreover, as revealed by recent work, hallucination is an obstacle when building LLMs in real applications (Zhang et al., 2023). We conduct fine-grained human evaluation to figure out whether the generated summaries involve factual errors. The results indicate that open-source LLMs have more factual errors than zero-shot GPT-4. Besides, the instruction tuning on LLMs might intensify factual errors, and make LLMs tend to generate hallucinations. This issue might come from the hallucination signals of ground truth references in existing summarization datasets (Wang et al., 2022a; Gao et al., 2023). Therefore, future work should strengthen the factual consistency when building LLM summarizers in real M2MS applications.

Our main contributions are concluded as follows:
• To our knowledge, we are the first to investigate how LLMs perform the M2MS task. To this end, we reorganize and select samples from previous multi-lingual summarization datasets to construct a multi-domain M2MS scenario.
• We conduct extensive studies on 18 LLMs. The evaluation process involves zero-shot prompting and instruction tuning. Fine-tuned traditional models are also conducted for comparison.
• In-depth analyses of the M2MS results on automatic evaluation and human evaluation provide a deeper understanding of the M2MS task-solving situations in the LLM era.

2 Related Work

2.1 Summarization in Multi-Lingual World.

To adapt text summarization to the multilingual world, the summarization research field proposes the following three branch tasks:

(1) Cross-lingual summarization (CLS) aims to generate a target-language summary for a document in a different source language (Wang et al., 2022c). Early work typically focuses on pipeline methods (Leuski et al., 2003; Orăsan and Chioorean, 2008). Some recent studies have demonstrated that such pipeline methods suffer from error propagation and inference latency, and their performance is worse than the end-to-end ones (Zhu et al., 2019; Perez-Beltrachini and Lapata, 2021). Meanwhile, with the availability of large-scale CLS datasets, many researchers shift the research attention to end-to-end CLS, using different techniques to deal with CLS, i.e., multi-task learning (Cao et al., 2020a; Bai et al., 2022; Liang et al., 2022a), knowledge distillation (Duan et al., 2019; Nguyen and Tuan, 2022) and different pre-training strategies (Xu et al., 2020; Chi et al., 2021; Wang et al., 2022b).

(2) Multi-lingual summarization (MLS) aims to process documents in multiple languages and generate their summaries in the corresponding language. Recently, large-scale MLS datasets (Scialom et al., 2020; Hasan et al., 2021; Liang et al., 2023) have been proposed one after another, facilitating further research on MLS. MultiSumm (Cao et al., 2020b) explores various knowledge-sharing strategies to train a MLS model among different languages. CALMS (Wang et al., 2021) proposes to train MLS models in the contrastive learning framework to share salient information extractive ability across different languages. Some studies (Aharoni et al., 2023; Qiu et al., 2023) aim to enhance the factual consistency of MLS models.

(3) Many-to-many summarization (M2MS) combines CLS and MLS into a more general setting that requires the model to summarize documents in any source language to a target language also in any language. CrossSum (Bhattacharjee et al., 2023) studies M2MS in the news domain, and shows M2MS model consistently outperforms CLS models, verifying the practicality of M2MS. Wang et al. (2023c) propose PISCES, a pre-trained M2MS model based on mBART. Besides, they explore
the summarization models trained with the settings of CLS, MLS and M2MS, and demonstrates the superiority of M2MS that allows task knowledge sharing across all languages. Different from existing work which generally uses traditional models, such as mBART (Liu et al., 2020), we first explore how well existing LLMs can perform M2MS in the zero-shot and the instruction-tuning manners.

2.2 Large Language Models.
Recently, there has been growing interest in LLMs for various NLP tasks (Zhao et al., 2023). A remarkable progress is the launch of ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023). LLMs show their powerful ability that serve as a general-purpose language task solver. Many powerful LLMs are proposed one after another to facilitate the LLM research, including LLaMa (Touvron et al., 2023a), LLaMa-2 (Touvron et al., 2023b), BaiChuan (Yang et al., 2023), Qwen (Bai et al., 2023; Yang et al., 2024), Vicuna (Chiang et al., 2023) and InternLM (Team, 2023). These LLMs generally adopt a three-stage training paradigm which first uses the next token prediction to learn the language modeling ability, and then leverages instruction tuning to enhance the model ability of following human instructions. Finally, an optional reinforcement learning with human feedback (RLHF) stage aligns LLMs’ values with humans.

3 Data
Dataset Selection. To ensure the data quality, the selected multi-lingual summarization datasets should meet the following requirements: (i) the datasets should be peer-reviewed and published; (ii) the datasets should provide cross-lingual alignments for their documents and summaries across different languages to support M2MS.1 After carefully comparing existing data, we finally choose the following eight datasets: (1) CrossSum (Bhattacharjee et al., 2023) is a news-domain dataset that collects document-summary pairs from the BBC news website. (2) XWikis (Perez-Beltrachini and Lapata, 2021) is an encyclopedia-domain dataset that collects summarization samples from Wikipedia. (3) XSAMSum (Wang et al., 2022b), (4) XMediaSum (Wang et al., 2022b) and (5) DialogSumX (Chen et al., 2023) are three dialogue-domain datasets, which are collected by manually translating the summaries of existing English dialogue summarization datasets into other languages. (6) WikiLingua (Ladhak et al., 2020) is a multi-lingual dataset in the domain of how-to guides. This dataset is collected from the WikiHow website. (7) Perseus (Zheng et al., 2023) is a technology-domain dataset that collects Chinese scientific articles with the corresponding Chinese and English summaries. (8) Spektrum (Fatima and Strube, 2021) is also a technology-domain dataset. This dataset collects samples from Spektrum der Wissenschaft (a German scientific journal).

Considering the involved languages of the chosen datasets, we make the used data support English (abbr. En), Czech (Cs), German (De), French (Fr), Chinese (Zh) and Ukrainian (Uk).

Intrinsic Metrics. After determining the datasets and the languages, we select M2MS samples from these datasets to use in our empirical study. Since the samples from a single dataset might be mixed-quality, we follow Grusky et al. (2018); Bommasani and Cardie (2020) and filter out low-quality samples based on three intrinsic quality metrics, i.e., coverage, redundancy and coherence. These metrics are all automatically calculated based on the text features of document-summary pairs. For more details of these metrics and the filtering thresholds, please refer to Appendix A.

Size and Contamination Controlling. To ensure the data diversity in our empirical study, given a dataset and a specific source-target language pair, we decide to randomly select a few hundred samples from the remaining samples. Following the success of the instruction tuning in LLaMa-2 (Touvron et al., 2023b) and Vicuna (Chiang et al., 2023), we control the number of training set to tens of thousands. During sample selection, we also consider the balance of each language as well as each domain. We make the selected data contain 19,530, 14,150 and 14,150 samples in the training, validation and testing sets. As LLMs are pre-trained on massive data, their downstream performances might be inflated due to data contamination (Dong et al., 2024; Golchin and Surdeanu, 2024). To alleviate this issue, during the selection of the testing samples, we follow Golchin and Surdeanu (2024) to calculate instance-level contamination for each M2MS sample, and control the proportion of contaminated samples is less than 1% in the testing set (more details are provided in Appendix B).

Data Statistics. As shown in Table 1, the final

1Note that not all datasets provide these alignments, some datasets only provide monolingual document-summary pairs in multiple languages, and thus do not support summarizing documents from a language into other languages.
<table>
  <tr>
    <th>Src</th>
    <th>Tgt</th>
    <th>En</th>
    <th>Cs</th>
    <th>De</th>
    <th>Fr</th>
    <th>Zh</th>
    <th>Uk</th>
  </tr>
  <tr>
    <td>En</td>
    <td></td>
    <td>3,900</td>
    <td>1,200</td>
    <td>3,400</td>
    <td>1,350</td>
    <td>3,050</td>
    <td>1,450</td>
  </tr>
  <tr>
    <td>Cs</td>
    <td></td>
    <td>1,200</td>
    <td>1,000</td>
    <td>1,200</td>
    <td>1,200</td>
    <td>1,200</td>
    <td>-</td>
  </tr>
  <tr>
    <td>De</td>
    <td></td>
    <td>1,900</td>
    <td>1,200</td>
    <td>2,000</td>
    <td>1,200</td>
    <td>1,200</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Fr</td>
    <td></td>
    <td>1,400</td>
    <td>1,200</td>
    <td>1,200</td>
    <td>1,050</td>
    <td>1,165</td>
    <td>700</td>
  </tr>
  <tr>
    <td>Zh</td>
    <td></td>
    <td>2,550</td>
    <td>1,200</td>
    <td>1,200</td>
    <td>1,165</td>
    <td>2,550</td>
    <td>1,100</td>
  </tr>
  <tr>
    <td>Uk</td>
    <td></td>
    <td>1,000</td>
    <td>-</td>
    <td>-</td>
    <td>700</td>
    <td>1,000</td>
    <td>1,000</td>
  </tr>
</table>

Table 1: The number of samples w.r.t different source-target language pairs. “Src” and “Tgt” denote the source and the target languages, respectively.

<table>
  <tr>
    <th>LLM</th>
    <th>Para.</th>
    <th>Max Len.</th>
    <th>Flores</th>
  </tr>
  <tr>
    <td>gpt-4o-0816</td>
    <td>-</td>
    <td>16K</td>
    <td><b>29.1</b></td>
  </tr>
  <tr>
    <td>gpt-4-1106</td>
    <td>-</td>
    <td>16K</td>
    <td>27.7</td>
  </tr>
  <tr>
    <td>gpt-3.5-turbo-1106</td>
    <td>-</td>
    <td>16K</td>
    <td>22.0</td>
  </tr>
  <tr>
    <td>LLaMa-2-13B-chat</td>
    <td>13B</td>
    <td>4K</td>
    <td>6.2</td>
  </tr>
  <tr>
    <td>LLaMa-2-7B-chat</td>
    <td>7B</td>
    <td>4K</td>
    <td>5.2</td>
  </tr>
  <tr>
    <td>LLaMa-3-8B-chat</td>
    <td>8B</td>
    <td>8K</td>
    <td>9.4</td>
  </tr>
  <tr>
    <td>Vicuna-13B-v1.5</td>
    <td>13B</td>
    <td>4K</td>
    <td>7.2</td>
  </tr>
  <tr>
    <td>Vicuna-13B-v1.5-16k</td>
    <td>13B</td>
    <td>16K</td>
    <td>6.8</td>
  </tr>
  <tr>
    <td>Vicuna-7B-v1.5</td>
    <td>7B</td>
    <td>4K</td>
    <td>6.1</td>
  </tr>
  <tr>
    <td>Vicuna-7B-v1.5-16k</td>
    <td>7B</td>
    <td>16K</td>
    <td>5.9</td>
  </tr>
  <tr>
    <td>Baichuan2-13B-Chat</td>
    <td>13B</td>
    <td>4K</td>
    <td>12.6</td>
  </tr>
  <tr>
    <td>Baichuan2-7B-Chat</td>
    <td>7B</td>
    <td>4K</td>
    <td>11.0</td>
  </tr>
  <tr>
    <td>Qwen-14B-Chat</td>
    <td>14B</td>
    <td>8K</td>
    <td>17.1</td>
  </tr>
  <tr>
    <td>Qwen-7B-Chat</td>
    <td>7B</td>
    <td>8K</td>
    <td>13.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Chat</td>
    <td>14B</td>
    <td>32K</td>
    <td>19.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Chat</td>
    <td>7B</td>
    <td>32K</td>
    <td>15.3</td>
  </tr>
  <tr>
    <td>Internlm2-chat-20B</td>
    <td>20B</td>
    <td>32K</td>
    <td>16.9</td>
  </tr>
  <tr>
    <td>Internlm2-chat-7B</td>
    <td>7B</td>
    <td>32K</td>
    <td>15.2</td>
  </tr>
</table>

Table 2: Comparisons among LLMs used in experiments, including their parameters (Para.), maximum support length (Max Len.), and multi-lingual performance on Flores.

data covers most language pairs among the six languages except for Cs↔Uk and De↔Uk. For more details, including the number of samples w.r.t each subset, the data sources w.r.t each language pair, length distribution and domain distribution, please refer to Appendix C.

4 Experimental Setup

4.1 Evaluation LLMs

We conduct experiments on the following three types of backbones, i.e., traditional models, closed-and open-source LLMs.

Traditional multi-lingual models. (1) mBART-50 (Tang et al., 2021) is a pre-trained multi-lingual model with transformer encoder-decoder architecture (Vaswani et al., 2017) and 610M parameters. (2) PISCES (610M) (Wang et al., 2023c) is an M2MS pre-trained model that extends mBART-50 by further pre-training.

Closed-source LLMs. (1) GPT-3.5-turbo (Chat-GPT) (OpenAI, 2022) is created by fine-tuning a GPT-3.5 series model via reinforcement learning from human feedback (RLHF). We use gpt-3.5-turbo-1106 in our experiments. (2) GPT-4 (OpenAI, 2023) is another advanced LLM that exhibits human-level performance on various benchmark datasets. We use gpt-4-1106 in our experiments. (3) GPT-4o (OpenAI, 2024) is an autoregressive omni model, which accepts multi-modal inputs and can generate multi-modal outputs. The model shows superiority performance in various NLP benchmarks. We use gpt-4o-2024-08-16 in our experiments.

Open-source LLMs. (1) LLaMa (Touvron et al., 2023b; Dubey et al., 2024) is a LLM family, which also shows remarkable performance as a general task solver. We use LLaMa-2-7B-chat, LLaMa-2-13B-chat and LLaMa-3-8B-chat in experiments. (2) Vicuna (Chiang et al., 2023) is created by fine-tuning LLaMa-series models on user-shared conversations collected from ShareGPT. Vicuna also provides 16k versions to support long text. We use Vicuna-7B-v1.5, Vicuna-7B-v1.5-16k, Vicuna-13B-v1.5 and Vicuna-13B-v1.5-16k in experiments. (3) BaiChuan-2 (Yang et al., 2023) is also trained with instruction-tuning and RLHF. This model shows its superior multi-lingual abilities in downstream tasks. We use Baichuan2-7B-Chat and Baichuan2-13B-Chat in experiments. (4) Qwen (Bai et al., 2023; Yang et al., 2024) is a LLM family, which shows great performance as a general task solver. We use Qwen-7B-Chat, Qwen-14B-Chat, Qwen2.5-7B-Chat and Qwen2.5-14B-Chat in experiments. (5) InternLM (Team, 2023) is a multi-lingual LLM pre-trained on multi-lingual corpora. We use Internlm2-chat-7B and Internlm2-chat-20B in experiments.

To provide a deeper understanding of the above LLMs, Table 2 compares their multi-lingual performance on Flores-101 (Goyal et al., 2021), parameters and maximum support lengths. To evaluate LLMs on M2MS, there are two settings should be considered: (1) In the zero-shot prompting setting, LLMs directly perform M2MS based on a carefully designed prompt with task-specific instruction and in-context examples (Appendix D). (2) In the instruction-tuning setting, the training samples will be used for tuning open-source LLMs. The above prompt is also used to formulate the M2MS sample into an instruction-response format.

4.2 Evaluation Metrics

We adopt ROUGE-1 (R1), ROUGE-2 (R2), ROUGE-L (RL) (Lin, 2004) and BERTScore (BS) (Zhang et al., 2020). The ROUGE scores measure the lexical overlap between the generated
summaries and the references. BERTScore measures the similarity between them from a semantics perspective. Besides, following Wang et al. (2023a); Liu et al. (2024), we prompt GPT-4o to score the generated summaries in terms of conciseness (Con.), coherence (Coh.), and relevance (Rel.) on a 5-point scale (more details are given in Appendix E.1).

4.3 Implementation Details

For all LLMs, including open- and closed-source LLMs, we use the sampling decoding strategy, and follow Liu et al. (2023) to set the temperature to 0.1. Besides, the maximum generation length is set to 400 tokens (the length of more than 97.8% of summaries in used data is less than 400 tokens). Since different LLMs have different maximum support lengths, we truncate the input source-language document to ensure the input length is within 3600 tokens to ensure fair comparison. Please refer to Appendix E.2 for more details of utilized model checkpoints, instruction-tuning LLMs, fine-tuning traditional models, and training hours.

5 Results and Analyses

Table 3 shows the experimental results in terms of R1, RL and BS. For the full results including R2, please refer to Appendix F. We analyze the results from the following aspects:

Comparing traditional models with LLMs. When comparing traditional multi-lingual language models (mBART-50 and PISCES) with zero-shot LLMs, the fine-tuned traditional models slightly outperform the best zero-shot LLM (i.e., GPT-4o). For example, GPT-4o achieves 26.0 R1 and 66.7 BS scores in overall performance, while the counterparts of mBART-50 are 27.4 and 67.8. PISCES outperforms mBART-50, and achieves 30.8 R1 and 68.6 BS scores. This finding is consistent with previous exploration of other summarization tasks (Wang et al., 2023c; Qin et al., 2023). The fine-tuned models learn the mapping from documents to summaries based on the whole training samples. In contrast, zero-shot LLMs only know a few in-context examples when performing M2MS. Without parameter updating, zero-shot LLMs could achieve competitive results with fine-tuned traditional models, showing their powerful instruction-following and in-context learning abilities.

Comparing traditional models with instruction-tuned LLMs, we find that instruction-tuned LLMs generally outperform the best traditional language model (i.e., PISCES) by a large margin. For example, the instruction-tuned Vicuna-13B-16k outperforms PISCES by 7.2 R1, 7.4 RL and 5.5 BS scores. We analyze this phenomenon from the following aspects: (1) The LLMs involve more parameters than the traditional models, thus having a more powerful ability to fit the tasks-specific data with small-scale instruction samples. As shown in previous work (Ladhak et al., 2020; Perez-Beltrachini and Lapata, 2021; Bhattacharjee et al., 2023), traditional models need a large number of multi-lingual summarization samples to learn how to generate a target-language summaries for the given source-language documents. Even in a single domain, traditional models generally need more than 100K samples during their training stage (Liang et al., 2022b). However, our study only uses 19.5K training samples from five domains, resulting in a great challenge to traditional models. (2) Another important aspect is the model’s maximum support length. Many documents in M2MS samples contain more than 2K tokens which is larger than the maximum support length of traditional models (c.f., Appendix C). For example, mBART-50 and PISCES only support input text within 1K tokens due to the limitation in their vanilla \( O(n^2) \) self-attention mechanism (Vaswani et al., 2017). In contrast, LLMs typically adopt RoPE (Su et al., 2024) that comes with valuable properties such as the flexibility of being expanded to any sequence length, and the capability of equipping the linear self-attention with relative position encoding. In this manner, LLMs could support long-document inputs and are more practical in real-world scenes.

Comparisons among zero-shot LLMs. Among all zero-shot LLMs, GPT-4o achieves the best results in overall performance while GPT-4 achieves the second results in most cases. Compared with other LLMs, GPT-4o shows its powerful ability to follow human instructions to perform MSMS and generally outperforms other LLMs. Among open-source LLMs, Vicuna-13B-16k works best and reaches 22.9 R1, 13.9 RL and 66.0 BS scores in overall performance, verifying the effectiveness of instruction-tuning LLaMa-series LLMs with the ShareGPT’s user conversations. Comparing Vicuna-series LLMs with other open-source LLMs, we find that though the multi-lingual ability of Vicuna-series LLMs is less than others (as demonstrated in Table 2), Vicuna-series LLMs typically
<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th colspan="3">Overall (R1 / RL / BS)</th>
    <th colspan="3">News (R1 / RL / BS)</th>
    <th colspan="3">Encyc. (R1 / RL / BS)</th>
    <th colspan="3">Dialogue (R1 / RL / BS)</th>
    <th colspan="3">Guide (R1 / RL / BS)</th>
    <th colspan="3">Tech. (R1 / RL / BS)</th>
  </tr>
  <tr>
    <th></th><th></th><th></th>
    <th></th><th></th><th></th>
    <th></th><th></th><th></th>
    <th></th><th></th><th></th>
    <th></th><th></th><th></th>
    <th></th><th></th><th></th>
  </tr>
  <tr>
    <th colspan="18">Setting 1: Zero-Shot LLMs</th>
  </tr>
  <tr>
    <td>GPT-4o</td><td>26.0 / 16.6 / 66.7</td><td>19.8 / 12.9 / 66.8</td><td>27.9 / 16.3 / 66.0</td><td>29.5 / 22.1 / 70.4</td><td>25.1 / 16.1 / 69.0</td><td>34.2 / 19.1 / 69.1</td>
  </tr>
  <tr>
    <td>GPT-4</td><td>25.7 / 16.4 / 66.4</td><td>19.5 / 12.5 / 65.9</td><td>26.9 / 14.5 / 64.8</td><td>28.9 / 21.6 / 70.0</td><td>24.0 / 15.5 / 68.4</td><td>33.8 / 18.8 / 68.9</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td><td>25.2 / 16.1 / 66.7</td><td>19.3 / 12.4 / 66.5</td><td>28.1 / 16.1 / 65.8</td><td>24.0 / 18.5 / 66.4</td><td>22.4 / 14.6 / 67.9</td><td>33.6 / 19.3 / 69.2</td>
  </tr>
  <tr>
    <td>LLaMa-2-13B</td><td>21.5 / 13.0 / 64.1</td><td>17.9 / 11.8 / 65.1</td><td>25.5 / 14.6 / 63.8</td><td>19.3 / 13.8 / 64.0</td><td>18.4 / 11.3 / 64.3</td><td>30.7 / 17.2 / 64.5</td>
  </tr>
  <tr>
    <td>LLaMa-2-7B</td><td>18.2 / 10.8 / 63.3</td><td>14.2 / 0.9 / 64.0</td><td>22.7 / 12.7 / 62.4</td><td>17.4 / 12.7 / 62.6</td><td>15.0 / 0.9 / 63.6</td><td>23.6 / 13.8 / 62.5</td>
  </tr>
  <tr>
    <td>LLaMa-3-8B</td><td>19.5 / 12.4 / 63.5</td><td>14.9 / 0.9 / 64.6</td><td>23.3 / 13.3 / 62.8</td><td>18.0 / 13.1 / 62.8</td><td>15.7 / 10.0 / 64.2</td><td>24.1 / 14.4 / 63.0</td>
  </tr>
  <tr>
    <td>Vicuna-13B</td><td>22.4 / 13.4 / 65.5</td><td>18.5 / 11.8 / 65.1</td><td>25.9 / 14.9 / 64.9</td><td>22.5 / 16.5 / 65.9</td><td>18.8 / 11.9 / 64.8</td><td>32.5 / 18.0 / 68.6</td>
  </tr>
  <tr>
    <td>Vicuna-13B-16k</td><td>22.9 / 13.9 / 66.0</td><td>19.0 / 11.9 / 65.3</td><td>27.2 / 15.5 / 65.3</td><td>22.6 / 17.1 / 66.1</td><td>20.3 / 12.9 / 65.9</td><td>33.0 / 19.2 / 69.1</td>
  </tr>
  <tr>
    <td>Vicuna-7B</td><td>22.3 / 13.7 / 65.0</td><td>17.8 / 11.6 / 65.5</td><td>26.0 / 15.4 / 64.9</td><td>22.1 / 16.2 / 67.1</td><td>18.5 / 11.3 / 65.3</td><td>31.1 / 17.5 / 67.4</td>
  </tr>
  <tr>
    <td>Vicuna-7B-16k</td><td>22.8 / 14.1 / 65.3</td><td>18.3 / 12.0 / 65.8</td><td>27.0 / 15.1 / 65.1</td><td>21.6 / 16.2 / 66.1</td><td>19.2 / 11.7 / 64.5</td><td>31.9 / 18.2 / 66.7</td>
  </tr>
  <tr>
    <td>Baichuan-2-13B</td><td>20.5 / 12.8 / 65.0</td><td>15.9 / 10.2 / 64.9</td><td>24.4 / 13.7 / 64.1</td><td>19.8 / 15.3 / 64.9</td><td>18.1 / 11.1 / 65.0</td><td>30.0 / 16.8 / 66.2</td>
  </tr>
  <tr>
    <td>Baichuan-7B</td><td>20.8 / 13.2 / 65.1</td><td>16.5 / 10.5 / 65.3</td><td>24.6 / 14.1 / 64.2</td><td>21.4 / 16.2 / 66.0</td><td>17.8 / 11.1 / 64.9</td><td>30.1 / 16.1 / 64.8</td>
  </tr>
  <tr>
    <td>Qwen-14B</td><td>21.6 / 13.0 / 65.2</td><td>17.9 / 11.5 / 65.6</td><td>25.3 / 14.3 / 64.7</td><td>21.8 / 16.3 / 66.5</td><td>18.1 / 11.1 / 64.7</td><td>32.0 / 17.8 / 64.8</td>
  </tr>
  <tr>
    <td>Qwen-7B</td><td>21.8 / 13.1 / 64.9</td><td>18.3 / 11.5 / 66.0</td><td>25.9 / 15.1 / 65.1</td><td>21.3 / 15.9 / 66.4</td><td>17.8 / 10.9 / 65.2</td><td>30.8 / 17.8 / 65.6</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td><td>22.1 / 13.1 / 65.4</td><td>18.4 / 11.7 / 65.8</td><td>25.8 / 14.8 / 65.2</td><td>22.0 / 16.6 / 66.8</td><td>18.5 / 11.6 / 64.9</td><td>32.6 / 18.1 / 65.2</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td><td>21.9 / 13.3 / 65.1</td><td>18.6 / 11.8 / 66.5</td><td>26.5 / 15.4 / 65.4</td><td>21.9 / 16.1 / 66.6</td><td>18.1 / 10.9 / 65.6</td><td>30.9 / 18.0 / 65.7</td>
  </tr>
  <tr>
    <td>Internlm-2-20B</td><td>19.2 / 12.0 / 62.9</td><td>14.9 / 0.9 / 66.2</td><td>24.0 / 13.9 / 63.8</td><td>11.6 / 0.8 / 59.1</td><td>16.2 / 10.1 / 63.0</td><td>30.6 / 17.5 / 66.6</td>
  </tr>
  <tr>
    <td>Internlm-2-7B</td><td>18.5 / 11.6 / 62.2</td><td>14.3 / 0.9 / 62.6</td><td>23.9 / 13.3 / 63.2</td><td>11.6 / 0.9 / 58.4</td><td>16.2 / 0.9 / 62.4</td><td>29.7 / 17.7 / 64.1</td>
  </tr>
  <tr>
    <th colspan="18">Setting 2: Fine-Tuned Traditional Multi-Lingual Language Models</th>
  </tr>
  <tr>
    <td>mBART-50</td><td>27.4 / 19.9 / 67.8</td><td>27.2 / 20.1 / 67.8</td><td>26.6 / 20.1 / 65.3</td><td>32.9 / 24.9 / 71.0</td><td>25.8 / 19.5 / 68.1</td><td>23.2 / 16.7 / 65.4</td>
  </tr>
  <tr>
    <td>PISCES</td><td>30.8 / 22.8 / 68.6</td><td>27.2 / 19.8 / 68.7</td><td>28.2 / 20.9 / 66.0</td><td>34.1 / 26.8 / 70.9</td><td>36.3 / 28.8 / 71.9</td><td>24.3 / 17.4 / 65.7</td>
  </tr>
  <tr>
    <th colspan="18">Setting 3: Instruction-Tuned LLMs</th>
  </tr>
  <tr>
    <td>LLaMa-2-13B</td><td>37.7 / 29.4 / 74.4</td><td>37.1 / 27.2 / 74.2</td><td>40.2 / 32.2 / 74.2</td><td>40.3 / 32.4 / 75.4</td><td>33.0 / 26.6 / 73.4</td><td>38.2 / 26.2 / 73.4</td>
  </tr>
  <tr>
    <td>LLaMa-2-7B</td><td>35.5 / 27.0 / 73.0</td><td>34.9 / 26.5 / 73.2</td><td>37.6 / 29.7 / 73.2</td><td>37.9 / 30.8 / 75.0</td><td>31.8 / 25.9 / 72.6</td><td>37.8 / 25.7 / 72.7</td>
  </tr>
  <tr>
    <td>LLaMa-3-8B</td><td>36.2 / 27.5 / 73.4</td><td>35.4 / 26.9 / 73.4</td><td>37.9 / 29.8 / 73.9</td><td>38.6 / 31.5 / 75.7</td><td>32.5 / 26.4 / 73.2</td><td>38.5 / 26.2 / 73.3</td>
  </tr>
  <tr>
    <td>Vicuna-13B</td><td>37.3 / 28.7 / 73.9</td><td>36.3 / 28.0 / 73.6</td><td>39.8 / 32.3 / 74.4</td><td>40.4 / 32.3 / 75.9</td><td>34.2 / 28.1 / 73.5</td><td>38.4 / 26.6 / 73.7</td>
  </tr>
  <tr>
    <td>Vicuna-13B-16k</td><td><b>38.0 / 30.2 / 74.1</b></td><td>36.9 / 28.6 / 74.7</td><td><b>40.4 / 32.9 / 74.2</b></td><td><b>41.2 / 33.6 / 75.9</b></td><td><b>34.5 / 28.5 / 73.9</b></td><td><b>38.3 / 26.4 / 73.5</b></td>
  </tr>
  <tr>
    <td>Vicuna-7B</td><td>35.6 / 28.0 / 73.1</td><td>34.5 / 26.2 / 73.8</td><td>38.3 / 29.1 / 73.3</td><td>38.9 / 32.3 / 75.4</td><td>31.2 / 25.7 / 72.9</td><td>36.8 / 24.5 / 72.4</td>
  </tr>
  <tr>
    <td>Vicuna-7B-16k</td><td>36.2 / 28.6 / 73.7</td><td>35.5 / 27.7 / 73.4</td><td>38.7 / 30.3 / 74.2</td><td>38.9 / 31.8 / 75.3</td><td>32.3 / 25.9 / 72.7</td><td>37.7 / 26.5 / 73.3</td>
  </tr>
  <tr>
    <td>Baichuan-2-13B</td><td>36.1 / 28.0 / 74.4</td><td>35.9 / 26.4 / 73.1</td><td>38.0 / 29.8 / 73.2</td><td>40.7 / 34.1 / 75.9</td><td>33.5 / 25.7 / 73.7</td><td>39.2 / 25.3 / 73.8</td>
  </tr>
  <tr>
    <td>Baichuan-7B</td><td>35.0 / 27.4 / 73.5</td><td>35.4 / 26.6 / 73.0</td><td>37.3 / 29.4 / 73.3</td><td>38.8 / 31.5 / 74.9</td><td>31.8 / 23.9 / 72.6</td><td>38.1 / 25.9 / 73.7</td>
  </tr>
  <tr>
    <td>Qwen-14B</td><td>37.1 / 28.4 / 74.2</td><td>36.0 / 26.8 / 73.2</td><td>38.4 / 30.8 / 73.4</td><td>40.2 / 33.4 / 75.5</td><td>34.4 / 28.5 / 74.4</td><td>39.1 / 26.4 / 74.4</td>
  </tr>
  <tr>
    <td>Qwen-7B</td><td>34.8 / 26.8 / 73.2</td><td>33.5 / 25.0 / 72.5</td><td>36.1 / 27.6 / 73.0</td><td>38.9 / 31.5 / 75.2</td><td>33.1 / 25.7 / 73.2</td><td>37.0 / 24.3 / 73.0</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td><td>37.8 / 29.3 / 74.3</td><td>36.7 / 28.0 / 74.0</td><td>39.4 / 31.6 / 73.9</td><td>40.8 / 33.6 / 75.8</td><td>34.4 / <b>28.7 / 74.7</b></td><td><b>39.4 / 26.6 / 74.8</b></td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td><td>35.2 / 27.3 / 73.7</td><td>34.2 / 25.6 / 72.9</td><td>36.6 / 28.2 / 73.7</td><td>39.5 / 32.3 / 76.0</td><td>33.4 / 26.5 / 73.6</td><td>37.7 / 24.9 / 73.6</td>
  </tr>
  <tr>
    <td>Internlm-2-20B</td><td>36.7 / 28.2 / 73.7</td><td>35.0 / 25.6 / 73.0</td><td>38.0 / 29.0 / 72.8</td><td>41.1 / <b>34.2 / 76.2</b></td><td><b>34.5 / 27.9 / 74.3</b></td><td><b>39.3 / 25.7 / 73.4</b></td>
  </tr>
  <tr>
    <td>Internlm-2-7B</td><td>35.7 / 27.2 / 73.5</td><td>34.1 / 25.6 / 72.6</td><td>36.2 / 28.4 / 72.8</td><td>40.7 / 33.5 / 75.9</td><td>33.3 / 27.0 / 73.3</td><td>37.8 / 25.6 / 73.0</td>
  </tr>
</table>

Table 3: Experimental results of the overall performance and fine-grained results in each domain. The bold denotes the best performance under each setting. Encyc.: Encyclopedia; Tech.: Technology.

outperform others in M2MS, showing its superior instruction-following and in-context learning abilities distilled from ShareGPT. In addition, we also find that Vicuna-13B-16k and Vicuna-7B-16k outperform Vicuna-13B and Vicuna-7B, respectively. Though the input length is truncated to 4K tokens for all LLMs, the Vicuna-16k models scale the support length from 4K to 16K and achieve better behaviors when processing long documents.

Comparisons among instruction-tuned LLMs. Vicuna-13B-16k also performs best among all instruction-tuned LLMs, and achieves 38.0 R1, 30.2 RL and 74.1 BS scores in overall performance. Qwen2.5-14B and LLaMa-2-13B achieve promising results following closely behind Vicuna-13B-16k. Besides, all instruction-tuned LLMs significantly outperform the fine-tuned traditional models as well as zero-shot LLMs by a large margin. This finding demonstrates that LLMs can improve their task-specific ability via instruction-tuning on small-scale task data. In addition to domain-wise performance, we also show the instruction-tuned LLM performance when using a specific source or target language during evaluation. The results are provided in Figure 2. Different LLMs might be good at different languages. For example, Vicuna-13B-16k performs best when processing English and Czech documents, while Qwen-14b performs best in German and Chinese documents. As for target languages, the LLaMa and Vicuna LLMs
are good at generating English, Czech, French and Ukrainian summaries. Baichuan2-13B, Qwen-14B and Internlm-20B do well in generating Chinese summaries.

LLM evaluation results. Table 4 shows the evaluation results using GPT-4o evaluation. The instruction-tuned LLMs significantly outperform traditional models in all metrics. For zero-shot LLMs, they tend to generate relatively lengthy summaries compared to traditional models or fine-tuned LLMs, resulting in low conciseness. This is because the alignment phase in LLMs emphasizes the usefulness of the models, making them provide detailed information. After tuning, LLMs can study to generate short summaries, and significantly improve their conciseness scores. In terms of coherence and relevance, instruction-tuning also brings improvements to LLMs. Moreover, we find that some tuned LLMs (e.g., LLaMa-2-7B and Vicuna-7B) achieve lower coherence and relevance than zero-shot GPT-4, though they outperform GPT-4 in terms of ROUGE scores and BERTScore.

The effects of training scales. In our main experiments, we use 19.5K training samples to fine-tune LLMs and traditional models. We further discuss the effects of the training scales on model performance, please refer to Appendix G.

<table>
  <tr>
    <th>Zero-shot Results</th>
    <th colspan="3">Ins-tuned Results</th>
  </tr>
  <tr>
    <th></th>
    <th>Con.</th>
    <th>Coh.</th>
    <th>Rel.</th>
    <th>Con.</th>
    <th>Coh.</th>
    <th>Rel.</th>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>3.30</td>
    <td><b>4.66</b></td>
    <td><b>4.83</b></td>
    <td>mBART-50</td>
    <td>3.82</td>
    <td>3.42</td>
    <td>4.02</td>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>3.27</td>
    <td>4.63</td>
    <td>4.75</td>
    <td>PiCES</td>
    <td>3.95</td>
    <td>3.58</td>
    <td>4.15</td>
  </tr>
  <tr>
    <td>GPT-3.5-turbo</td>
    <td>3.04</td>
    <td>4.56</td>
    <td>4.69</td>
    <td>LLaMA-2-13B</td>
    <td>4.59</td>
    <td>4.70</td>
    <td>4.72</td>
  </tr>
  <tr>
    <td>LLaMA-2-13B</td>
    <td>3.20</td>
    <td>4.34</td>
    <td>4.50</td>
    <td>LLaMA-2-13B</td>
    <td>4.59</td>
    <td>4.70</td>
    <td>4.72</td>
  </tr>
  <tr>
    <td>LLaMA-2-7B</td>
    <td>3.32</td>
    <td>4.31</td>
    <td>4.48</td>
    <td>LLaMA-3-8B</td>
    <td>4.50</td>
    <td>4.59</td>
    <td>4.68</td>
  </tr>
  <tr>
    <td>LLaMA-3-8B</td>
    <td>3.32</td>
    <td>4.31</td>
    <td>4.48</td>
    <td>Vicuna-13B</td>
    <td>4.70</td>
    <td>4.68</td>
    <td>4.70</td>
  </tr>
  <tr>
    <td>Vicuna-13B</td>
    <td>3.15</td>
    <td>4.35</td>
    <td>4.51</td>
    <td>Vicuna-13B-16k</td>
    <td>4.63</td>
    <td>4.71</td>
    <td>4.66</td>
  </tr>
  <tr>
    <td>Vicuna-13B-16k</td>
    <td>3.10</td>
    <td>4.39</td>
    <td>4.52</td>
    <td>Vicuna-7B</td>
    <td>4.47</td>
    <td>4.55</td>
    <td>4.62</td>
  </tr>
  <tr>
    <td>Vicuna-7B</td>
    <td>3.06</td>
    <td>4.27</td>
    <td>4.45</td>
    <td>Vicuna-7B-16k</td>
    <td>4.53</td>
    <td>4.58</td>
    <td>4.60</td>
  </tr>
  <tr>
    <td>Vicuna-7B-16k</td>
    <td>3.10</td>
    <td>4.30</td>
    <td>4.37</td>
    <td>Baichuan2-13B</td>
    <td>4.51</td>
    <td>4.65</td>
    <td>4.69</td>
  </tr>
  <tr>
    <td>Baichuan2-13B</td>
    <td>3.37</td>
    <td>4.34</td>
    <td>4.46</td>
    <td>Baichuan2-7B</td>
    <td>4.40</td>
    <td>4.51</td>
    <td>4.53</td>
  </tr>
  <tr>
    <td>Baichuan2-7B</td>
    <td>3.10</td>
    <td>4.29</td>
    <td>4.29</td>
    <td>Qwen-14B</td>
    <td>4.55</td>
    <td>4.55</td>
    <td>4.60</td>
  </tr>
  <tr>
    <td>Qwen-14B</td>
    <td>3.30</td>
    <td>4.23</td>
    <td>4.33</td>
    <td>Qwen-7B</td>
    <td>4.39</td>
    <td>4.51</td>
    <td>4.45</td>
  </tr>
  <tr>
    <td>Qwen-2.5-14B</td>
    <td>3.32</td>
    <td>4.37</td>
    <td>4.48</td>
    <td>Qwen-2.5-14B</td>
    <td>4.63</td>
    <td>4.69</td>
    <td>4.65</td>
  </tr>
  <tr>
    <td>Qwen-2.5-7B</td>
    <td>3.27</td>
    <td>4.29</td>
    <td>4.32</td>
    <td>Qwen-2.5-7B</td>
    <td>4.42</td>
    <td>4.56</td>
    <td>4.50</td>
  </tr>
  <tr>
    <td>Internlm2-20B</td>
    <td>3.17</td>
    <td>4.29</td>
    <td>4.37</td>
    <td>Internlm2-20B</td>
    <td>4.51</td>
    <td>4.60</td>
    <td>4.67</td>
  </tr>
  <tr>
    <td>Internlm2-7B</td>
    <td>3.04</td>
    <td>4.18</td>
    <td>4.28</td>
    <td>Internlm2-7B</td>
    <td>4.25</td>
    <td>4.47</td>
    <td>4.52</td>
  </tr>
</table>

Table 4: Overall performance in terms of GPT-4o evaluation (Ins-tuned: Instruction-tuned).

<table>
  <tr>
    <th>Model</th>
    <th>Rate</th>
    <th>Model</th>
    <th>Rate</th>
  </tr>
  <tr>
    <td>LLaMA-2-13B</td>
    <td>91.1<sup>†</sup>/98.7<sup>†</sup></td>
    <td>Baichuan2-13B</td>
    <td>98.7<sup>†</sup>/99.4<sup>†</sup></td>
  </tr>
  <tr>
    <td>LLaMA-2-7B</td>
    <td>81.6<sup>†</sup>/98.7<sup>†</sup></td>
    <td>Baichuan2-7B</td>
    <td>94.8<sup>†</sup>/99.4<sup>†</sup></td>
  </tr>
  <tr>
    <td>LLaMA-3-8B</td>
    <td>96.6<sup>†</sup>/98.8<sup>†</sup></td>
    <td>Qwen-14B</td>
    <td>98.7<sup>†</sup>/99.3<sup>†</sup></td>
  </tr>
  <tr>
    <td>Vicuna-13B</td>
    <td>98.7<sup>†</sup>/98.8<sup>†</sup></td>
    <td>Qwen-7B</td>
    <td>98.7<sup>†</sup>/99.1<sup>†</sup></td>
  </tr>
  <tr>
    <td>Vicuna-13B-16k</td>
    <td>98.7<sup>†</sup>/98.8<sup>†</sup></td>
    <td>Qwen-2.5-14B</td>
    <td>99.2<sup>†</sup>/99.4<sup>†</sup></td>
  </tr>
  <tr>
    <td>Vicuna-7B</td>
    <td>94.5<sup>†</sup>/99.2<sup>†</sup></td>
    <td>Qwen-2.5-7B</td>
    <td>98.5<sup>†</sup>/99.1<sup>†</sup></td>
  </tr>
  <tr>
    <td>Vicuna-7B-16k</td>
    <td>97.5<sup>†</sup>/98.7<sup>†</sup></td>
    <td>Internlm2-20B</td>
    <td>97.0<sup>†</sup>/98.7<sup>†</sup></td>
  </tr>
  <tr>
    <td>Internlm2-7B</td>
    <td>97.2<sup>†</sup>/98.5<sup>†</sup></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Correct language rate (%) of the summaries († and ‡ denote the results of zero-shot and instruction-tuned LLMs, respectively).

6 Discussion

Are generated summaries in the correct language? As reported by Wang et al. (2023c), using traditional models as multi-lingual summarizers might generate in the wrong languages instead of the given target language. We wonder whether LLM-generated summaries are in the correct language. To this end, we use fastlangid2 to detect the generated summaries, and calculate the rate of the generated summaries in the correct language, named correct language rate (CR). As shown in Table 5, we find that most zero-shot LLMs could follow the language requirements in the prompt and generate summaries in the right target language with ≥ 95 CR score. This is because the LLMs are trained with multi-lingual corpora that include a large number of parallel sentences across different languages, making the model already learn the multi-lingual skills. After tuning, LLMs generally improve their CR scores. Thus, using LLMs to serve as the backbones of the M2MS systems has great potential in real-world applications.

2https://pypi.org/project/fastlangid/
<table>
  <tr>
    <th>LLM</th>
    <th colspan="2">Before</th>
    <th>After</th>
    <th>LLM</th>
    <th colspan="2">Before</th>
    <th>After</th>
  </tr>
  <tr>
    <td>LLaMa-2-13B</td>
    <td>53.5</td>
    <td>54.3</td>
    <td>54.5</td>
    <td>54.1</td>
    <td>54.1</td>
    <td>54.1</td>
    <td>54.1</td>
  </tr>
  <tr>
    <td>LLaMa-2-7B</td>
    <td>47.2</td>
    <td>47.3</td>
    <td>52.9</td>
    <td>52.0</td>
    <td>52.0</td>
    <td>52.0</td>
    <td>52.0</td>
  </tr>
  <tr>
    <td>Vicuna-13B</td>
    <td>55.5</td>
    <td>55.8</td>
    <td>66.0</td>
    <td>65.9</td>
    <td>65.9</td>
    <td>65.9</td>
    <td>65.9</td>
  </tr>
  <tr>
    <td>Vicuna-13B-16k</td>
    <td>54.3</td>
    <td>54.5</td>
    <td>56.2</td>
    <td>57.0</td>
    <td>57.0</td>
    <td>57.0</td>
    <td>57.0</td>
  </tr>
  <tr>
    <td>Vicuna-7B</td>
    <td>49.8</td>
    <td>49.6</td>
    <td>65.0</td>
    <td>64.6</td>
    <td>64.6</td>
    <td>64.6</td>
    <td>64.6</td>
  </tr>
  <tr>
    <td>Vicuna-7B-16k</td>
    <td>48.0</td>
    <td>48.4</td>
    <td>59.1</td>
    <td>58.9</td>
    <td>58.9</td>
    <td>58.9</td>
    <td>58.9</td>
  </tr>
</table>

Table 6: The LLMs’ performance on MMLU. “Before”, and “After” denote the results of LLMs before and after the M2MS instruction tuning, respectively.

Does the instruction tuning on M2MS influence LLMs’ general ability? As shown in Section 5, LLMs can improve their M2MS ability by instruction-tuning on the collected training samples. An important question arises naturally, i.e., does this task-specific improvement sacrifice the general task-solving ability of LLMs? To figure out this question, we further evaluate the LLMs before and after M2MS instruction tuning on the MMLU evaluation dataset (Hendrycks et al., 2021). The MMLU dataset covers 57 tasks including elementary mathematics, US history, computer science, law, etc, and is designed to evaluate models’ world knowledge and problem-solving ability. We follow previous LLM work (Touvron et al., 2023b; Yang et al., 2023; Bai et al., 2023), and adopt the 5-shot evaluation strategy. The experimental results are provided in Table 6. As we can see, the M2MS instruction tuning on LLMs does not sacrifice their general task-solving ability. Some instruction-tuned LLMs (i.e., LLaMa-2-13B and Qwen-7B) even outperform their original models.

Can large model generate factually consistent summaries? As revealed by Maynez et al. (2020); Gao et al. (2023), the model-generated summaries might be inconsistent with the source documents. We want to know if LLMs can generate factually consistent summaries. To this end, we randomly select 100 samples from the testing set and conduct fine-grained human evaluation on the summaries generated by GPT-4, zero-shot & tuned LLaMa-2-13B, and zero-shot & tuned Vicuna-13B-16k. Following Gao et al. (2023), for each sample, we employ human evaluators to annotate the following four types of factual errors (if has): (1) hallucination error: a generated summary contains events not directly inferable from the given document. (2) particulars error: the summary contains the right events but some details are inaccurate or mistaken. (3) predicate error: the predicate in the summary is contradictory to the source document. (4) entity error: the entity of an event in the summary is wrong.

<table>
  <tr>
    <th></th>
    <th>Hallu.</th>
    <th>Parti.</th>
    <th>Predi.</th>
    <th>Entity</th>
  </tr>
  <tr>
    <td>GPT-4</td>
    <td>8</td>
    <td>3</td>
    <td>5</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Zero-shot LLaMa-2-13B</td>
    <td>17</td>
    <td>13</td>
    <td>9</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Tuned LLaMa-2-13B</td>
    <td>23</td>
    <td>18</td>
    <td>7</td>
    <td>9</td>
  </tr>
  <tr>
    <td>Zero-shot Vicuna-13B-16k</td>
    <td>12</td>
    <td>8</td>
    <td>8</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Tuned Vicuna-13B-16k</td>
    <td>17</td>
    <td>16</td>
    <td>10</td>
    <td>6</td>
  </tr>
</table>

Table 7: Fine-grained human evaluation results on factuality (Hallu.: hallucination; Parti.: particulars; Predi.: predicate).

More details about human evaluation are given in Appendix H. Table 7 reports the proportion of each error. The summaries generated by GPT-4 have the lowest error proportion in terms of all errors. Besides, among all types of factual errors, the hallucination error occurs more frequently, indicating it is a non-trivial issue when adapting LLMs as the summarizers. Another finding is that instruction tuning on LLMs might intensify their factual issue especially the hallucination and the particulars errors. We conjecture this because the summaries in the training samples are written by humans, thus they might involve more information (like background information) beyond the given documents. Such an information gap might encourage LLMs to generate hallucinations during tuning. As reported by previous studies (Wang et al., 2022a; Gao et al., 2023), the ground truth references in summarization data also have the hallucination error. Therefore, when building LLM M2MS summarizers in the real applications, this error should be carefully considered during instruction tuning.

7 Conclusion

In this paper, we explore how well off-the-shelf LLMs can deal with the many-to-many summarization (M2MS) task. Considering the limited diversity and the single domain characteristics in each single dataset, we reorganize M2MS data based on eight existing multi-lingual summarization datasets, and the used data covers five domains and six languages. Based on it, we conduct extensive experiments on various open- and closed-source LLMs. Our results indicate that the zero-shot LLMs could achieve competitive results with fine-tuned traditional models. Furthermore, through instruction tuning, open-source LLMs can significantly improve their M2MS ability, and not sacrifice their general capabilities. However, as shown in our human evaluation, LLMs still face the factuality issue, and the instruction tuning might intensify this issue, which is worth noting in future research.
Limitations

While we evaluate the performance of LLMs on the many-to-many summarization task, there are some limitations worth noting: (1) We only evaluate the lower threshold of these models’ M2MS performance. Prompts are important to guide LLMs to perform specific tasks, and future work could explore better prompts to obtain better results. (2) The used data in our empirical study only involves data from existing multi-lingual summarization datasets. Future work could extend it with more domains in the real scenes.

Ethical Considerations

In this paper, we use multiple LLMs (e.g., GPT-4o, GPT-4, GPT-3.5-turbo and LLaMa-2) and traditional language models (e.g., mBART) as the M2MS models in experiments. During instruction tuning and fine-tuning, the adopted M2MS samples mainly come from previous datasets, i.e., CrossSum (Bhattacharjee et al., 2023), XWikis (Perez-Beltrachini and Lapata, 2021), XSAMSum (Wang et al., 2022b), XMediaSum (Wang et al., 2022b), DialogSumX (Chen et al., 2023), WikiLingua (Ladhak et al., 2020), Perseus (Zheng et al., 2023) and Spektrum (Fatima and Strube, 2021). Therefore, the trained models might involve the same biases and toxic behaviors exhibited by these datasets.

References

Reoe Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2023. Multilingual summarization with factual consistency evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 3562–3591, Toronto, Canada. Association for Computational Linguistics.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Yu Bai, Heyan Huang, Kai Fan, Yang Gao, Yiming Zhu, Jiaao Zhan, Zewen Chi, and Boxing Chen. 2022. Unifying cross-lingual summarization and machine translation with compression rate. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1087–1097.

Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2023. CrossSum: Beyond English-centric cross-lingual summarization for 1,500+ language pairs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2541–2564, Toronto, Canada. Association for Computational Linguistics.

Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075–8096, Online. Association for Computational Linguistics.

Yue Cao, Hui Liu, and Xiaojun Wan. 2020a. Jointly learning to align and summarize for neural cross-lingual summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6220–6231, Online. Association for Computational Linguistics.

Yue Cao, Xiaojun Wan, Jinge Yao, and Dian Yu. 2020b. Multisumm: Towards a unified model for multi-lingual abstractive summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 34(01):11–18.

Yulong Chen, Huajian Zhang, Yijie Zhou, Xuefeng Bai, Yueguan Wang, Ming Zhong, Jianhao Yan, Yafu Li, Judy Li, Xianchao Zhu, and Yue Zhang. 2023. Revisiting cross-lingual summarization: A corpus-based study and a new benchmark with improved annotation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9352–9351, Toronto, Canada. Association for Computational Linguistics.

Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, and Furu Wei. 2021. mT6: Multilingual pretrained text-to-text transformer with translation pairs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1671–1683, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344–16359.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages
4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey on in-context learning. arXiv preprint arXiv:2301.00234.

Yihong Dong, Xue Jiang, Huanyu Liu, Zhi Jin, Bin Gu, Mengfei Yang, and Ge Li. 2024. Generalization or memorization: Data contamination and trustworthy evaluation for large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 12039–12050, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Xiangyu Duan, Mingming Yin, Min Zhang, Boxing Chen, and Weihua Luo. 2019. Zero-shot cross-lingual abstractive sentence summarization through teaching generation and attention. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3162–3172, Florence, Italy. Association for Computational Linguistics.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Mehwish Fatima and Michael Strube. 2021. A novel Wikipedia based dataset for monolingual and cross-lingual summarization. In Proceedings of the Third Workshop on New Frontiers in Summarization, pages 39–50, Online and in Dominican Republic. Association for Computational Linguistics.

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.

Mingqi Gao, Wenqing Wang, Xiaojun Wan, and Yue-mei Xu. 2023. Evaluating factuality in cross-lingual summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12415–12431, Toronto, Canada. Association for Computational Linguistics.

Shahriar Golchin and Mihai Surdeanu. 2024. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations.

Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. arXiv preprint arXiv:2106.03193.

Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.

Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubashir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In International Conference on Learning Representations.

Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020. WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4034–4048, Online. Association for Computational Linguistics.

Anton Leuski, Chin-Yew Lin, Liang Zhou, Ulrich Germann, Franz Josef Och, and Eduard H. Hovy. 2003. Cross-lingual c*st*rd: English access to hindi information. ACM Trans. Asian Lang. Inf. Process., 2:245–269.

Yunlong Liang, Fandong Meng, Jinan Xu, Jiaan Wang, Yufeng Chen, and Jie Zhou. 2023. Summary-oriented vision modeling for multimodal abstractive summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2934–2951, Toronto, Canada. Association for Computational Linguistics.

Yunlong Liang, Fandong Meng, Chulun Zhou, Jinan Xu, Yufeng Chen, Jinsong Su, and Jie Zhou. 2022a. A variational hierarchical model for neural cross-lingual summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2088–2099, Dublin, Ireland. Association for Computational Linguistics.

Yunlong Liang, Fandong Meng, Chulun Zhou, Jinan Xu, Yufeng Chen, Jinsong Su, and Jie Zhou. 2022b. A variational hierarchical model for neural cross-lingual summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2088–2099, Dublin, Ireland. Association for Computational Linguistics.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Ran Liu, Ming Liu, Min Yu, He Zhang, Jianguo Jiang, Gang Li, and Weiqing Huang. 2024. SumSurvey:
An abstractive dataset of scientific survey papers for long document summarization. In Findings of the Association for Computational Linguistics ACL 2024, pages 9632–9651, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.

Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Zhuoer Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, Xiaohan Zhang, Lichao Sun, Hongning Wang, Jing Zhang, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. Alignbench: Benchmarking chinese alignment of large language models. Preprint, arXiv:2311.18743.

Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742.

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics.

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Thong Nguyen and Luu Anh Tuan. 2022. Improving neural cross-lingual summarization via employing optimal transport distance for knowledge distillation. Proc. of AAAI.

OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt.

OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.

OpenAI. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Constantin Orăsan and Oana Andreea Chiorean. 2008. Evaluation of a cross-lingual Romanian-English multi-document summariser. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA).

Laura Perez-Beltrachini and Mirella Lapata. 2021. Models and datasets for cross-lingual summarisation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9408–9423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.

Chengwei Qin, Aston Zhang, Zhusheng Zhang, Jiao Chen, Michihiro Yasunaga, and Dijy Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476.

Yifu Qiu, Yftah Ziser, Anna Korhonen, Edoardo Ponti, and Shay Cohen. 2023. Detecting and mitigating hallucinations in multilingual summarisation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8914–8932, Singapore. Association for Computational Linguistics.

Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051–8067, Online. Association for Computational Linguistics.

Jianlin Su, Muradtha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063.

Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2021. Multilingual translation from denoising pre-training. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3450–3466, Online. Association for Computational Linguistics.

Yuting Tang, Ratish Puduppully, Zhengyuan Liu, and Nancy Chen. 2023. In-context learning of large language models for controlled dialogue summarization: A holistic benchmark and empirical analysis. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 56–67, Singapore. Association for Computational Linguistics.

InternLM Team. 2023. Internlm: A multilingual language model with progressively enhanced capabilities.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Bin Wang, Chen Zhang, Yan Zhang, Yiming Chen, and Haizhou Li. 2022a. Analyzing and evaluating faithfulness in dialogue summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4897–4908, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu, and Lei Li. 2021. Contrastive aligned joint learning for multilingual summarization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2739–2750, Online. Association for Computational Linguistics.

Jian Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is ChatGPT a good NLG evaluator? a preliminary study. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 1–11, Singapore. Association for Computational Linguistics.

Jian Wang, Yunlong Liang, Fandong Meng, Beiqi Zou, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023b. Zero-shot cross-lingual summarization via large language models. In Proceedings of the 4th New Frontiers in Summarization Workshop, pages 12–23, Singapore. Association for Computational Linguistics.

Jian Wang, Fandong Meng, Ziyao Lu, Duo Zheng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022b. Clid-Sum: A benchmark dataset for cross-lingual dialogue summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 7716–7729, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Jian Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022c. A survey on cross-lingual summarization. Transactions of the Association for Computational Linguistics, 10:1304–1323.

Jian Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023c. Towards unifying multi-lingual and cross-lingual summarization. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15127–15143, Toronto, Canada. Association for Computational Linguistics.

Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. 2024. Benchmark data contamination of large language models: A survey. arXiv preprint arXiv:2406.04244.

Ruochen Xu, Chenguang Zhu, Yu Shi, Michael Zeng, and Xuedong Huang. 2020. Mixed-lingual pre-training for cross-lingual summarization. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 536–541, Suzhou, China. Association for Computational Linguistics.

Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, et al. 2023. Baichuan 2: Open large-scale language models. arXiv preprint arXiv:2309.10305.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.

Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. 2023. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.

Shaohui Zheng, Zhixu Li, Jianan Wang, Jianfeng Qu, An Liu, Lei Zhao, and Zhigang Chen. 2023. Long-document cross-lingual summarization. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, pages 1084–1092.

Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Ji-ajun Zhang, Shaonan Wang, and Chengqing Zong. 2019. NCLS: Neural cross-lingual summarization. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3054–3064, Hong Kong, China. Association for Computational Linguistics.

A Details of Intrinsic Metrics

Following Grusky et al. (2018); Bommasani and Cardie (2020), we filter out low-quality summarization samples based on the following three intrinsic quality metrics:
(1) Coverage evaluates the percentage of words in a summary that are part of an extractive fragment from the document (Grusky et al., 2018):

\[
\text{Coverage}(D, S) = \frac{1}{|S|} \sum_{u \in F(D, S)} |u|
\]

where \( D \) and \( S \) denote a document and the corresponding summary, respectively. \( F(D, S) \) is the set of all extractive fragments that appear in both \( D \) and \( S \). A smaller number of coverage indicates the summary is more abstractive.

(2) Redundancy measures whether sentences in a summary are similar to each other (Bommasani and Cardie, 2020). Assuming a summary \( S \) has \( m \) sentences \( S = \{s_1, s_2, ..., s_m\} \), redundancy is formally calculated as follows:

\[
\text{Redundancy}(S) = \frac{1}{\binom{m}{2}} \sum_{1 \leq i < m - 1} \sum_{i+1 \leq j \leq m} \text{RL}(m_i, m_j)
\]

where \( \text{RL}(\cdot) \) denotes the ROUGE-L score (Lin, 2004). Generally, the lower the redundancy, the higher the sample quality.

(3) Coherence measures the semantic coherence of a summary \( S = \{s_1, ..., s_m\} \) by predicting the probability of each successive sentence conditioned on the previous one using a language model (Bommasani and Cardie, 2020):

\[
\text{Coherence}(S) = \frac{1}{m-1} \sum_{i=2}^m P_\theta(s_i|s_{i-1})
\]

where \( P_\theta \) denotes the predicted probability of a language model. Here, we adopt mBERT (Devlin et al., 2019) as \( P_\theta \). Generally, a high coherence score indicates the high quality of the sample.

For each intrinsic metric, we set a threshold to filter low-quality samples. Specifically, inspired by Bommasani and Cardie (2020), for each sample, its coverage should be less than \( \alpha_{cov} = 0.9 \), the redundancy should be less than \( \alpha_{red} = 0.2 \), and the coherence needs to be more than \( \alpha_{coh} = 0.9 \). Otherwise, the sample will be filtered out.

B Data Contamination

Data contamination is a potential major issue in measuring LLMs’ performance on downstream tasks, where the testing data might be in the pertaining corpora of LLMs (Xu et al., 2024). Golchin and Surdeanu (2024) propose an effective method using BLEURT & ROUGE-L metrics to measure instance-level contamination, i.e., identifying if an instance (usually a sentence or document) is contaminated for a given LLM. The method can be used in both open-source and closed-source LLMs.

When selecting the testing samples in our empirical study, after filtering samples via intrinsic metrics (Appendix A), we calculate the instance-level contamination for each sample w.r.t GPT-4o, Vicuna-7B, Baichuan2-7B, Qwen2.5-7B-Chat, LLaMa-3-8B-chat and Internlm2-7B.\footnote{For the same series of LLMs, we select one to measure the contamination.} The uncontaminated samples will be randomly selected to form the testing set. Some directions in the domain-specific datasets might contain only a few hundred samples (\emph{e.g.}, CrossSum only contains about 300 Fr⇒Zh samples), thus we cannot ensure data contamination and testing scale simultaneously in these directions. Therefore, an extremely small number of samples labeled as “contaminated” will also be included in our testing set, and they account for less than 1% of the whole testing set.

C Data Statistics

Language and Source Distribution. For a specific language pair, the number of samples in each subset (training, validation, and testing sets) and the corresponding data sources are provided in Table 8.

Length and Domain Distributions. To calculate the length of documents and summaries across different languages, we use tiktoken\footnote{https://github.com/openai/tiktoken} to tokenize the documents and summaries, and calculate their token-level length. As shown in Table 9, the average length of source documents typically reaches thousands of tokens, while the counterpart of target summaries is within 200 tokens. From the perspective of domains, Table 10 shows the average length

![Length distributions of M2MS samples w.r.t different domains.](page_1016_120_388_388.png)

Figure 3: Length distributions of M2MS samples w.r.t different domains.
<table>
  <tr>
    <th>Src</th>
    <th>Tgt</th>
    <th>En</th>
    <th>Cs</th>
    <th>De</th>
    <th>Fr</th>
    <th>Zh</th>
    <th>Uk</th>
  </tr>
  <tr>
    <td rowspan="2">En</td>
    <td>(CR, XW, XS, XM, DI, WI, SP)</td>
    <td>1400 / 1250 / 1250 (CR, XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>1300 / 1050 / 1050 (XW, XS, XM, WI, SP)</td>
    <td>500 / 425 / 425 (CR, XW, DI, WI)</td>
    <td>1150 / 950 / 950 (CR, XW, XS, XM, DI, WI)</td>
    <td>700 / 375 / 375 (CR, DI)</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Cs</td>
    <td>(XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>300 / 350 / 350 (XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">De</td>
    <td>(XW, WI, SP)</td>
    <td>800 / 550 / 550 (XW, WI, SP)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>900 / 550 / 550 (XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Fr</td>
    <td>(CR, XW, WI)</td>
    <td>650 / 375 / 375 (CR, XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>300 / 375 / 375 (CR, XW, WI)</td>
    <td>565 / 300 / 300 (CR, XW, WI)</td>
    <td>500 / 100 / 100 (CR)</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Zh</td>
    <td>(CR, XW, WI, PE)</td>
    <td>900 / 825 / 825 (CR, XW, WI, PE)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>500 / 350 / 350 (XW, WI)</td>
    <td>565 / 300 / 300 (CR, XW, WI)</td>
    <td>900 / 825 / 825 (CR, XW, WI, PE)</td>
    <td>500 / 300 / 300 (CR)</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Uk</td>
    <td>(CR)</td>
    <td>400 / 300 / 300 (CR)</td>
    <td>-</td>
    <td>-</td>
    <td>400 / 150 / 150 (CR)</td>
    <td>400 / 300 / 300 (CR)</td>
    <td>400 / 300 / 300 (CR)</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 8: The number of training/validation/testing samples and the data sources w.r.t different source-target language pairs. “Src” and “Tgt” denote the source and the target languages, respectively. (CR: CrossSum; XW: XWikis; XS: XSAMSum; XM: XMediaSum; DI: DialogSumX; WI: WikiLingua; PE: Perseus; SP: Spektrum)

<table>
  <tr>
    <th></th>
    <th>En→X</th>
    <th>CS→X</th>
    <th>De→X</th>
    <th>Fr→X</th>
    <th>Zh→X</th>
    <th>Uk→X</th>
  </tr>
  <tr>
    <th>Training</th>
    <td>Doc.</td>
    <td>869.05</td>
    <td>2418.82</td>
    <td>2111.35</td>
    <td>1385.84</td>
    <td>2534.56</td>
    <td>1693.38</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>65.64</td>
    <td>124.67</td>
    <td>196.98</td>
    <td>84.11</td>
    <td>150.68</td>
    <td>57.97</td>
  </tr>
  <tr>
    <th>Validation</th>
    <td>Doc.</td>
    <td>897.45</td>
    <td>2364.59</td>
    <td>2087.41</td>
    <td>1345.93</td>
    <td>2164.53</td>
    <td>1477.90</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>63.72</td>
    <td>123.44</td>
    <td>194.34</td>
    <td>81.56</td>
    <td>146.36</td>
    <td>56.76</td>
  </tr>
  <tr>
    <th>Testing</th>
    <td>Doc.</td>
    <td>791.15</td>
    <td>2361.89</td>
    <td>2074.92</td>
    <td>1280.33</td>
    <td>2269.90</td>
    <td>1346.57</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>52.32</td>
    <td>125.68</td>
    <td>198.89</td>
    <td>86.41</td>
    <td>134.35</td>
    <td>52.02</td>
  </tr>
  <tr>
    <th></th>
    <th>X→En</th>
    <th>X→CS</th>
    <th>X→De</th>
    <th>X→Fr</th>
    <th>X→Zh</th>
    <th>X→Uk</th>
  </tr>
  <tr>
    <th>Training</th>
    <td>Doc.</td>
    <td>2105.74</td>
    <td>2204.62</td>
    <td>2080.01</td>
    <td>1627.83</td>
    <td>2102.88</td>
    <td>1050.10</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>115.40</td>
    <td>159.83</td>
    <td>141.19</td>
    <td>58.17</td>
    <td>140.67</td>
    <td>89.05</td>
  </tr>
  <tr>
    <th>Validation</th>
    <td>Doc.</td>
    <td>1731.99</td>
    <td>2121.00</td>
    <td>2307.03</td>
    <td>1297.37</td>
    <td>1821.54</td>
    <td>982.90</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>98.52</td>
    <td>167.79</td>
    <td>124.08</td>
    <td>57.89</td>
    <td>129.91</td>
    <td>89.89</td>
  </tr>
  <tr>
    <th>Testing</th>
    <td>Doc.</td>
    <td>1769.09</td>
    <td>1807.71</td>
    <td>2092.57</td>
    <td>1502.02</td>
    <td>1883.59</td>
    <td>799.59</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>96.41</td>
    <td>163.11</td>
    <td>142.47</td>
    <td>39.25</td>
    <td>119.95</td>
    <td>79.27</td>
  </tr>
</table>

Table 9: The token-level average length of documents (Doc.) and summaries (Sum.) in the data w.r.t different source and target languages. En→X/X→En indicates all samples whose documents/summaries are in English.

<table>
  <tr>
    <th></th>
    <th>News</th>
    <th>Encyc.</th>
    <th>Dialogue</th>
    <th>Guide</th>
    <th>Tech.</th>
  </tr>
  <tr>
    <th>Training</th>
    <td>Num.</td>
    <td>4680</td>
    <td>4650</td>
    <td>2550</td>
    <td>4650</td>
    <td>3000</td>
  </tr>
  <tr>
    <td></td>
    <td>Doc.</td>
    <td>1350.21</td>
    <td>2020.66</td>
    <td>733.24</td>
    <td>772.6</td>
    <td>2857.5</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>63.36</td>
    <td>121.46</td>
    <td>42.04</td>
    <td>69.96</td>
    <td>235.14</td>
  </tr>
  <tr>
    <th>Validation</th>
    <td>Num.</td>
    <td>2300</td>
    <td>3425</td>
    <td>2600</td>
    <td>3425</td>
    <td>2400</td>
  </tr>
  <tr>
    <td></td>
    <td>Doc.</td>
    <td>1101.92</td>
    <td>1936.80</td>
    <td>728.72</td>
    <td>767.20</td>
    <td>2687.42</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>57.18</td>
    <td>122.57</td>
    <td>40.32</td>
    <td>67.10</td>
    <td>221.87</td>
  </tr>
  <tr>
    <th>Testing</th>
    <td>Num.</td>
    <td>2300</td>
    <td>3425</td>
    <td>2600</td>
    <td>3425</td>
    <td>2400</td>
  </tr>
  <tr>
    <td></td>
    <td>Doc.</td>
    <td>1055.44</td>
    <td>1994.10</td>
    <td>749.52</td>
    <td>732.96</td>
    <td>2882.51</td>
  </tr>
  <tr>
    <td></td>
    <td>Sum.</td>
    <td>54.72</td>
    <td>123.86</td>
    <td>36.18</td>
    <td>68.36</td>
    <td>245.10</td>
  </tr>
</table>

Table 10: The token-level average length of documents (Doc.) and summaries (Sum.) in the data w.r.t different domains. “Num.” indicates the number of samples in each domain. Encyc.: Encyclopedia; Tech.: Technology

of documents as well as summaries w.r.t different domains. As we can see, the average document length in the encyclopedia and technology domains is generally more than that in other domains. The average length of dialogue and guide documents is less than 800 tokens, making them the shortest document length among all domains. To provide a deeper understanding of the used data in our empirical study, Figure 3 shows the length distributions of different domains.

D M2MS Prompt

Inspired by previous LLM summarization studies (Wang et al., 2023b; Tang et al., 2023) and the in-context learning technique (Dong et al., 2022; Min et al., 2022), we attempt various M2MS prompts on GPT-3.5-turbo and GPT-4, and choose the prompt with the best results (using both automatic and human evaluation) on a pilot experiment. Specifically, as shown in Figure 4, the final chosen prompt is designed with task descriptions, domain information and a few output examples. [source language] and [target language] are selected from “English”, “Czech”, “German”, “French”, “Chinese” and “Ukrainian”. When the source and the target languages are the same, the content in parentheses will be omitted. [domain] indicates the domain of the input document, which is selected from “news”, “encyclopedia”, “dialogue”, “how-to guides” and “technology”. [example summary i] (\( i \in \{1,2,3\} \)) denotes a ground truth summary randomly selected from the training samples. [document] represents the current input document that needs to generate the corresponding summary.
System: You are a professional literary worker and proficient in [source language] (and [target language]). You are good at summarizing [source language] [domain] documents into [target language] summaries.

The following example summaries are written by you previously, and have received compliments:
Summary 1: [example summary 1]
Summary 2: [example summary 2]
Summary 3: [example summary 3]

Your task is to continue to summarize the [source language] [domain] documents into [target language] summaries. The newly generated summaries should have a similar style to your previously written summaries.

User: The [source language] document is provided as follows:
[document]
Please generate its [target language] summary.

Figure 4: Illustration of the used M2MS prompt that includes a system round and a user round.

E Implementation Details

E.1 Implementation Details of Evaluation Metrics

To calculate the ROUGE scores in the multi-lingual setting, we use the multi-lingual rouge5 toolkit. For BERTScore, we use the bert-score6 toolkit, and set the backbone to bert-base-multilingual-cased.

During evaluation via GPT-4o, the used prompt is “I will provide you with a summary of a document. Please rate the summary on a scale of one to five in terms of conciseness, coherence, and relevance.”. We use gpt-4o-2024-0816 as the evaluator. Since the testing set in our study contains more than 14K samples, it is a high cost to evaluate all model-generated summaries via GPT-4o. Thus, we randomly select 500 samples to conduct the LLM evaluation.

E.2 Implementation Details of LLMs

Instruction-Tuning Details. All LLMs are tuned on 8×NVIDIA A800 GPUs (80G) with 1e-5 learning rate and 32 (8×4 gradient accumulation) batch size. We follow the success of instruction tuning in LLaMa-2 (Touvron et al., 2023b), and set the training epochs to 2. We use the DeepSpeed optimization library7, and set ZeRO-2 optimization. For Internlm2-chat-20B, we also offload the optimizer into the CPU to avoid CUDA out-of-memory error. Flash attention (v2) (Dao et al., 2022) is also employed to save memory. During tuning, documents are also truncated to ensure the input length is within 3,600 tokens to ensure fairness. For the instruction-tuned LLMs, we use the same decoding strategy as the zero-shot ones.

Model Checkpoints. (1) For traditional multilingual language models, we use mBART-50 (610M)8 (Tang et al., 2021) and PISCES (610M)9 (Wang et al., 2023c). (2) For open-source LLMs, we use LLaMa-2-7B-chat10, LLaMa-2-13B-chat11, LLaMa-3-8B-chat12, Vicuna-7B-v1.513, Vicuna-7B-v1.5-16k14, Vicuna-13B-v1.515, Vicuna-13B-v1.5-16k16, Baichuan2-7B-Chat17, Baichuan2-13B-Chat18 Qwen-7B-Chat19, Qwen-14B-Chat20, Qwen2.5-7B-Chat21, Qwen2.5-14B-Chat22, Internlm2-chat-7B23 and Internlm2-chat-20B24 in experiments. All model checkpoints are available at the Huggingface community.

Fine-Tuning Details. To fine-tune traditional multi-lingual language models, i.e., mBART-50 and PISCES, we follow Wang et al. (2023c) and set the learning rate to 3e-5, batch size to 8×8, and epochs to 10. Experiments are conducted on 8×NVIDIA A800 GPUs (80G). Different from LLMs, the source-language documents are directly input into these models without any prompts. Following previous work (Wang et al., 2023c; Bhattacharjee et al., 2023), a language tag is appended

5https://github.com/csebuetnlp/xl-sum/tree/master/multilingual_rouge_scoring
6https://github.com/Tiiiger/bert_score
7https://github.com/microsoft/DeepSpeed
8https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt
9https://huggingface.co/Krystalan/PISCES
10https://huggingface.co/meta-llama/LLaMa-2-7B-chat-hf
11https://huggingface.co/meta-llama/LLaMa-2-13B-chat-hf
12https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
13https://huggingface.co/lmsys/Vicuna-7B-v1.5
14https://huggingface.co/lmsys/Vicuna-7B-v1.5-16k
15https://huggingface.co/lmsys/Vicuna-13B-v1.5
16https://huggingface.co/lmsys/Vicuna-13B-v1.5-16k
17https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat
18https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat
19https://huggingface.co/Qwen/Qwen-7B-Chat
20https://huggingface.co/Qwen/Qwen-14B-Chat
21https://huggingface.co/Qwen/Qwen2.5-7B-Instruct
22https://huggingface.co/Qwen/Qwen2.5-14B-Instruct
23https://huggingface.co/internlm/Internlm2-chat-7B
24https://huggingface.co/internlm/Internlm2-chat-20B
<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th colspan="3">Overall</th>
    <th colspan="3">News</th>
    <th colspan="3">Dialogue</th>
    <th colspan="3">Guide</th>
    <th colspan="3">Tech.</th>
  </tr>
  <tr>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
    <th>(R1 / R2 / RL / BS)</th>
  </tr>
  <tr>
    <td>GPT-4</td><td>26.0 / 12.3 / 16.6 / 66.7</td><td>19.8 / 0.9 / 12.9 / 66.8</td><td>27.9 / 12.5 / 16.3 / 66.0</td><td>29.5 / 17.3 / 22.1 / 70.4</td><td>25.1 / 11.3 / 16.1 / 69.0</td><td>34.2 / 16.3 / 19.1 / 69.1</td>
    <td>GPT-4</td><td>25.7 / 12.1 / 16.4 / 66.4</td><td>19.5 / 0.8 / 12.5 / 65.9</td><td>26.9 / 11.2 / 14.5 / 64.8</td><td>28.9 / 17.0 / 21.6 / 70.0</td><td>24.0 / 10.7 / 15.5 / 68.4</td><td>33.8 / 16.1 / 18.8 / 68.9</td>
    <td>GPT-3.5-turbo</td><td>25.2 / 11.3 / 16.1 / 66.7</td><td>19.3 / 0.8 / 12.4 / 66.5</td><td>28.1 / 12.1 / 16.1 / 65.8</td><td>24.0 / 13.5 / 18.5 / 66.4</td><td>22.4 / 0.9 / 14.6 / 67.9</td><td>33.6 / 14.9 / 19.3 / 69.2</td>
    <td>LLaMA-2-13B</td><td>21.5 / 0.9 / 13.0 / 64.1</td><td>17.9 / 0.8 / 11.8 / 63.5</td><td>25.5 / 10.8 / 14.6 / 63.8</td><td>19.3 / 0.9 / 13.8 / 64.0</td><td>18.4 / 0.6 / 11.3 / 64.3</td><td>30.7 / 12.6 / 17.2 / 64.5</td>
    <td>LLaMA-2-7B</td><td>18.2 / 0.6 / 10.8 / 63.3</td><td>14.2 / 0.5 / 0.9 / 60.0</td><td>27.8 / 0.3 / 12.7 / 62.4</td><td>17.4 / 0.7 / 12.7 / 62.6</td><td>15.0 / 0.4 / 0.9 / 63.5</td><td>26.3 / 0.8 / 13.8 / 62.5</td>
    <td>LLaMA-2-16B</td><td>19.5 / 0.7 / 11.9 / 64.6</td><td>15.8 / 0.6 / 11.3 / 64.6</td><td>25.9 / 0.6 / 14.9 / 63.9</td><td>18.0 / 0.7 / 11.1 / 65.9</td><td>18.8 / 0.7 / 11.0 / 64.8</td><td>31.5 / 13.7 / 18.0 / 66.6</td>
    <td>Vicuna-7B</td><td>22.9 / 0.9 / 13.7 / 65.0</td><td>17.8 / 0.7 / 11.6 / 65.5</td><td>26.0 / 10.4 / 15.4 / 64.9</td><td>22.1 / 10.7 / 16.2 / 67.1</td><td>18.5 / 0.6 / 11.3 / 65.3</td><td>31.1 / 12.8 / 17.5 / 67.4</td>
    <td>Vicuna-7B-16k</td><td>22.8 / 0.9 / 14.1 / 65.3</td><td>18.3 / 0.8 / 12.0 / 65.8</td><td>27.0 / 11.2 / 15.1 / 65.1</td><td>21.6 / 10.0 / 16.2 / 66.1</td><td>19.2 / 0.6 / 11.7 / 64.5</td><td>31.9 / 13.1 / 18.2 / 66.7</td>
    <td>Baichuan2-13B</td><td>20.5 / 0.8 / 12.8 / 65.0</td><td>15.9 / 0.6 / 10.2 / 64.9</td><td>24.4 / 0.9 / 13.7 / 64.4</td><td>19.8 / 0.9 / 15.3 / 64.9</td><td>18.1 / 0.6 / 11.1 / 65.0</td><td>30.0 / 12.3 / 16.8 / 66.2</td>
    <td>Baichuan2-7B</td><td>20.8 / 0.8 / 13.2 / 65.3</td><td>16.5 / 0.6 / 10.5 / 65.3</td><td>24.6 / 0.9 / 14.1 / 64.2</td><td>21.4 / 10.2 / 16.2 / 66.0</td><td>17.8 / 0.6 / 11.1 / 64.9</td><td>30.1 / 12.5 / 16.1 / 64.8</td>
    <td>Qwen-14B</td><td>21.6 / 0.9 / 13.0 / 65.1</td><td>17.9 / 0.8 / 11.5 / 65.6</td><td>25.3 / 10.8 / 14.3 / 64.7</td><td>21.8 / 0.9 / 13.6 / 66.6</td><td>18.1 / 0.7 / 11.1 / 64.7</td><td>32.0 / 13.5 / 17.8 / 64.8</td>
    <td>Qwen-14B-16k</td><td>21.9 / 0.9 / 13.1 / 65.4</td><td>18.4 / 0.8 / 11.7 / 65.8</td><td>25.8 / 1.2 / 14.8 / 65.2</td><td>22.0 / 11.1 / 16.6 / 65.8</td><td>18.5 / 0.7 / 11.3 / 64.9</td><td>32.6 / 13.8 / 17.5 / 65.2</td>
    <td>Qwen-2.5-14B</td><td>22.1 / 0.9 / 13.1 / 65.4</td><td>18.4 / 0.8 / 11.7 / 65.8</td><td>25.8 / 1.2 / 14.8 / 65.2</td><td>22.0 / 11.1 / 16.6 / 65.8</td><td>18.5 / 0.7 / 11.3 / 64.9</td><td>32.6 / 13.8 / 17.5 / 65.2</td>
    <td>Qwen-2.5-7B</td><td>21.9 / 0.8 / 13.3 / 65.1</td><td>18.6 / 0.8 / 11.8 / 66.5</td><td>26.5 / 10.9 / 15.4 / 65.4</td><td>21.9 / 11.0 / 16.1 / 66.6</td><td>18.1 / 0.6 / 11.0 / 65.6</td><td>30.9 / 12.9 / 18.0 / 65.7</td>
    <td>Internlm2-20B</td><td>19.2 / 0.7 / 12.0 / 62.9</td><td>14.9 / 0.6 / 0.9 / 66.7</td><td>24.0 / 10.0 / 13.9 / 63.8</td><td>16.1 / 0.5 / 0.8 / 59.1</td><td>16.2 / 0.6 / 11.0 / 63.0</td><td>30.6 / 13.9 / 17.5 / 66.6</td>
    <td>Internlm2-7B</td><td>18.5 / 0.7 / 11.6 / 62.2</td><td>14.3 / 0.6 / 0.9 / 65.2</td><td>23.9 / 0.7 / 13.3 / 63.2</td><td>11.6 / 0.5 / 0.9 / 58.4</td><td>16.2 / 0.6 / 0.9 / 62.4</td><td>29.7 / 13.2 / 17.7 / 64.1</td>
  </tr>
  <tr>
    <td>mBART-50</td><td>27.4 / 11.9 / 19.9 / 67.8</td><td>27.2 / 12.5 / 20.1 / 67.8</td><td>26.6 / 12.7 / 20.1 / 65.3</td><td>32.9 / 17.5 / 24.9 / 71.0</td><td>25.8 / 11.1 / 19.5 / 68.1</td><td>23.2 / 10.8 / 16.7 / 65.4</td>
    <td>PISCES</td><td>30.8 / 15.0 / 22.8 / 68.6</td><td>27.2 / 12.0 / 19.8 / 68.7</td><td>28.2 / 12.9 / 20.9 / 66.0</td><td>34.1 / 18.0 / 26.8 / 70.9</td><td>36.3 / 20.7 / 28.8 / 71.9</td><td>24.3 / 11.0 / 17.4 / 65.7</td>
  </tr>
</table>

Setting 1: Zero-Shot LLMs

Setting 2: Fine-Tuned Traditional Multi-Lingual Language Models

Setting 3: Instruction-Tuned LLMs

Table 11: Experimental results of the overall performance and fine-grained results in each domain. The bold denotes the best performance under each setting. Encyc.: Encyclopedia; Tech.: Technology.

on the decoder side and serves as the decoder start token to control which language should be generated in the summaries. Besides, we set the maximum number of tokens for input sequences to 1024 (mBART-50 and PISCES accept input text with a maximum length of 1K, and this is also a shortcoming of traditional models compared with LLMs). The fine-tuned traditional models use the same decoding strategy as the LLMs.

Training/Tuning Hours. All experiments are conducted on NVIDIA A800 GPUs with 80G memory, and we use its GPU hours to denote the consumption of computing resources. Each instruction-tuned 7B LLM needs 19 GPU hours, while each 13B LLM needs 32 GPU hours. For Internlm2-chat-20B, it costs 80 GPU hours since we offload the optimizer. To fine-tune the traditional multi-lingual language models, 3 GPU hours are cost.

F Full Results

Table 11 shows the full results in terms of R1, R2, RL and BS. Typically, the results in terms of R2 are consistent with the results in terms of other metrics.

G The Effects of Training Scales

To assess the impact of training scales, we randomly select the training samples into various sizes (2K, 4K, 8K, 12K, 16K, and 19.5K) and subse-

![Model performance (ROUGE-1) using different scales of training samples.](page_1012_1342_484_312.png)

Figure 5: Model performance (ROUGE-1) using different scales of training samples.
quently fine-tune the models mBART-50, PISCES, and Vicuna-13B-16k for each scale. During the randomly selection, we use probability sampling to ensure the balance of each language as well as each domain. As shown in Figure 5, compared with LLMs, the performance of traditional models is more sensitive with the training scale. Specifically, when decreasing the training data from 19.5K to 2K, mBART-50 and PISCES sacrifice 21.8 and 23.6 R1 (ROUGE-1) scores, respectively, while the counterpart of Vicuna-13B-16k is 12.4.

H  Human Evaluation

Following Gao et al. (2023), we employ three graduate students with high levels of fluency in both English and Chinese as our evaluators. We randomly select 100 English documents from the testing set, and let the evaluators judge whether factual errors in the Chinese summaries generated by GPT-4, zero-shot & tuned LLaMa-2-13B, and zero-shot & tuned Vicuna-13B-16k. If a generated summary has factual errors, evaluators also should label which types of factual errors in the summary. Finally, the Fleiss’ Kappa scores (Fleiss, 1971) of hallucination error, particulars error, predicate error and entity error are 0.78, 0.73, 0.81, 0.75, respectively, indicating a good inter-agreement among our evaluators.

Reviewer paper summaries: This paper presents an ample empirical evaluation of contemporary LLMs over a task of many-to-many summarization (M2MS) with a dataset newly assembled by the authors themselves (which will be released upon publication alongside the code).

In brief, the evaluation covers conventional multi-lingual Transformers, closed-source LLMs and open-source LLMs, both zero-shot and instruction-tuned, and uses a range of evaluation metrics (ROUGE, BERTScore, GPT-4o) and human evaluation to derive its findings.

This paper is the revision of a manuscript submitted to the ARR June 2024 cycle for which I had served as a reviewer.

The authors present a comprehensive empirical study of LLMs performance on many-to-many summarisation. The study includes the compilation of a new dataset, using a subset of good quality samples from 8 different existing dataset, that covers 5 domains and 6 languages. The authors compare the performance of 18 different models, grouped into 3 groups: 1) Traditional fine-tuned models (e.g., mBART, Pisces), closed-sourced LLMs (e.g., GPT-4) and open-sourced LLMs (e.g., Llama, Vicuna...). The closed-sourced LLMs are only tested in a zero-shot setting while the open-sourced models are also instruction tuned over the training data of the compiled dataset.

The experimental results show that traditional fine-tune models can still beat both closed-source and open-sourced LLMs in this task. However, instruction-tuned LLMs can outperform the traditional models by a large margin.

The paper investigates Large-Language Models capability to conduct many-to-many summarization in both zero-shot and an instruction-tuning manner. To this end they conduct an extensive Benchmark and combine and re-use existing datasets. They compare with traditional SOTA multilingual summarization models like mBART and PISCES and find that zero-shot LLMs are competitive with them, whereas LLMs become far superior after instruction tuning. Additionally, they find that their instruction tuning does not deteriorate other capabilities of the LLMs but LLM generated summaries are prone to errors in factuality which can be exacerbated by instruction tuning.
Summary of strengths: I had a very positive impression of the original submission of this paper, and I had recommended its acceptance. In my reading, the paper presented a very thorough and current analysis, with many interesting findings. In this revision, several other contributions have been added, including a human evaluation of factuality of the predicted summaries. Overall, these are its key strengths:  

•	The M2MS task is properly framed in the related literature.

•	The filters applied to select the samples (coverage, redundancy, coherence) seem sensible.

•	The number and diversity of languages pairs and the number of domains is sizable.

•	The number of evaluated models is remarkable, including conventional multi-lingual Transformers and closed-source and open-source LLMs.

•	The findings are very interesting, and confirm the usefulness of instruction-tuning.

•	The evaluation has now added a score obtained from GPT-4o (which is well aligned with the other scores), and a useful human evaluation.

- The paper is well written and the main contributions clearly presented.
- Comprehensive study that includes a large number of domains, languages and LLMs.
- Interesting analysis of the correct language rate.
- The authors show that the instruction tuning of the models does not affect their ability for other tasks.
- Insightful human evaluation to show that open-sourced models suffer from factuality errors more than the closed-source models, and that these errors get augmented by the instruction tuning.

This is a resubmission. The authors have improved their paper extensively by:
- Improving the motivation and clarifying misconceptions.
- Adding more SOTA LLMs and more test examples.
- Adding human and LLM based evaluation along with ROUGE and BERTScore.
- Adding controls for Data Contamination.
Summary of weaknesses: None that I can tell. Would it be possible to add PISCES to Table 7? It would be interesting to see where conventional models sit in this analysis. This may help better understand the reasons for the factuality errors. At the moment, the conjecture at line 573 is inevitably very speculative.

- What is the variance of these models? It would be good to understand in Table 4 what are the variances or some king of statistical significance test.
- In addition to the human evaluation, it would be interesting to use one of the existing factuality metrics available.

- For the LLM based evaluation, the definitions of conciseness, coherence and relevance should have been provided to the LLM so that the evaluation is more standardized.
- Only having 3 student annotators for Human Evaluation is too few to make general statements. The scope should be increased.
Comments, suggestions and typos: There is some non-idiomatic use of English, e.g. “(M2MS) is proposed to generate”, “Our main contributions are concluded as follows”, “aligns LLMs’ values with human”. They do not affect comprehension, but it would be nice if someone could proof edit them.

As a small thing, adjective “testing” has a specific meaning in English. Corresponding samples and sets should be referred as “test set” and “test samples”, respectively. It’s a rather common mistake; still a mistake.

instruct-tuning -> instruction-tuning

remove full stops from sections’ names

Comments after Author Discussion:
I have appreciated the authors' reply to my minor suggestions. It would be useful if they could include the factuality analysis with PISCES in the camera-ready, rather than future work, because there's much ongoing interest on the performance of small models.

A lot of the text in the paper is in blue font. Is this intentional?

N/A
Overall assessment scores: 4

4

3.5