Multi-Document Summarization with Centroid-Based Pretraining
Year: 2023

Abstract:
In Multi-Document Summarization (MDS), the input can be modeled as a set of documents, and the output is its summary. In this paper, we focus on pretraining objectives for MDS. Specifically, we introduce a novel pretraining objective, which involves selecting the ROUGEbased centroid of each document cluster as a proxy for its summary. Our objective thus does not require human written summaries and can be utilized for pretraining on a dataset consisting solely of document sets. Through zero-shot, few-shot, and fully supervised experiments on multiple MDS datasets, we show that our model Centrum is better or comparable to a state-ofthe-art model. We make the pretrained and finetuned models freely available to the research community1.

heading: 1 Introduction
text: In Multi-Document Summarization (MDS), the input is a set of documents, and the output is a summary that describes important information in a coherent and non-redundant manner (McKeown and Radev, 1995; Radev and McKeown, 1998). In recent years, there have been significant improvements in MDS due to the availability of MDS datasets (Fabbri et al., 2019; Gholipour Ghalandari et al., 2020; Liu* et al., 2018) and advances in pretraining approaches (Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020).
In particular, Xiao et al. (2022) introduced a pretraining approach called PRIMERA (Pyramidbased Masked Sentence Pretraining) adapted for MDS. To create synthetic summaries, they used the Pyramid scheme (Nenkova and Passonneau, 2004), incorporating a process of identifying and ranking entities, followed by grouping sentences containing these entities in the input documents. The
∗Part of the work was done when the author was at the University of Edinburgh
1https://github.com/ratishsp/centrum
sentences with the highest overlap with other documents (measured using ROUGE) in each group were masked in the input and integrated into the output, forming a synthetic summary. Xiao et al. (2022) show that an encoder-decoder model trained on such a corpus attains strong zero-shot, few-shot, and fully supervised results on multiple datasets.
However, these synthetic summaries may lack coherence as the sentences are derived from various positions within the input documents. Furthermore, there is potential for redundancy, as sentences encapsulating similar information could be selected for inclusion in the summary.
In this paper, we propose Centrum, a pretraining objective for MDS, which is conceptually simple and overcomes these problems. The key intuition is that among a set of documents in a news cluster, the document which shares the most content with the other documents in the cluster can serve as a proxy for the summary of the document set. Such a cluster centroid is inherently coherent as it is a humanwritten document. Furthermore, because it isn’t artificially assembled, it avoids content repetition.
In this paper, we pretrain Centrum on NewSHead (Gu et al., 2020) corpus and perform zero-shot, fewshot and fully-supervised experiments on various MDS datasets. We show that Centrum performs favorably compared to PRIMERA, especially in the zero-shot and few-shot settings, where there are none or very few training examples available.

heading: 2 Centroid-based Selection of Document as Summary
text: Background on PRIMERA Xiao et al. (2022) leveraged the NewSHead corpus (Gu et al., 2020), a compilation of 369,940 news clusters, for pretraining. Using the Pyramid scheme (Nenkova and Passonneau, 2004), they created synthetic summaries through a multi-step procedure. They gathered the entity mentions in the input documents and rank the entities by the count of documents in which
128
an entity is mentioned. Next, they divide the sentences from the documents into distinct groups, such that the sentences containing an entity belong to the same group. They then extracted the sentence with the highest overlap (as quantified by ROUGE (Lin, 2004)) with other documents from each group. This sentence was replaced with a mask token in the input, and copied to the output document. The idea here was to leverage information from other documents to reconstruct the masked sentence. The sentences thus obtained were concatenated to form a synthetic summary.
Xiao et al. (2022) applied the method to the NewSHead corpus (Gu et al., 2020) containing news articles clustered by topic. To accommodate long document lengths, they use Longformer EncoderDecoder (LED) architecture (Beltagy et al., 2020). LED supports sparse global attention along with dense local attention on the input. PRIMERA employs global attention on specialized tokens (<doc-sep>), which act as separators between the documents within the input cluster. The pretrained PRIMERA model was then used for zeroshot evaluation, few-shot or full finetuning across multiple MDS datasets.
Problems with PRIMERA pretraining PRIMERA’s reference summaries consist of sentences extracted from varying positions within different documents in a cluster. This method can yield incoherent summaries, as it can be unclear which entities the sentences refer to. We illustrate this with an example of a synthetic summary created using PRIMERA in Table 1. The first sentence about Lady Gaga originates from the first document, while the second sentence mentioning Donald Trump and Elton John comes from the second document. The lack of entity mentions within these sentences disrupts the overall coherence of the summary. We also note occurrences of redundant information in the synthetic summary. Our hypothesis is that pretraining using such noisy synthetic summaries could negatively impact model performance, particularly in zero-shot or few-shot experiments.
Our Model We propose an alternate pretraining objective for MDS called as Centrum. We hypothesize that a document exhibiting the highest similarity with the rest of the documents in a cluster could serve as a proxy for its summary. This method inherently filters out documents that bear
only a distant relation to other documents in the cluster. Furthermore, it addresses potential noise present in automatically created multi-document cluster datasets (Gu et al., 2020), for example, a document falsely associated with a cluster. The Centrum pretraining objective excludes such noise, as a mismatched document would not be chosen as the cluster centroid. Among the documents, a document may have more relevant content than others. The Centrum objective will select the more relevant document as the summary.
Drawing inspiration from Gu et al. (2020), we designate a document as the summary if it maximizes the semantic match with other documents in the cluster. Specifically, from each document set D in an instance, the best candidate summary ŷ is selected as:
ŷ = argmax x∈D
1/|D| ∑
x′∈D\{x} f(x, x′) (1)
where f(x, x′) represents the semantic match of summary x with document x′. A model can be trained to learn this function f . In our approach, we employ the average of ROUGE1, ROUGE2, and ROUGEL as this function. Our pretraining corpus is constructed by treating D \ {ŷ} as the
input and ŷ as the output. Vogler et al. (2022) recently employed a comparable strategy for unsupervised MDS. However, our approach differs from theirs by applying this strategy for MDS task-specific pretraining. Moreover, following Xiao et al. (2022), we employ the LED architecture for handling long document context in the input.

heading: 3 Experimental Setup
text: Model We utilize the Transformers (Wolf et al., 2020) library to conduct our experiments. Similar to Xiao et al. (2022), we adopt the large configuration of LED, comprising 459M parameters. We finetune the LED model on the NewSHead corpus (Gu et al., 2020) with our Centrum pretraining objective. Documents within a cluster are concatenated into a single text sequence, with <doc-sep> tokens employed as separators. We apply global attention to the <doc-sep> tokens, while local attention is used for the remaining tokens. Further details about the hyperparameter settings can be found in Appendix C.
Datasets We conduct our evaluation on the MultiNews (Fabbri et al., 2019), WCEP (Gholipour Ghalandari et al., 2020), and DUC 2007 datasets, comparing zero-shot, few-shot, and fully-supervised results. DUC 2007 comprises 45 examples, 20 of which we designate as the test set (Xiao et al., 2022).
Preprocessing of NewSHead Dataset We apply the following criteria when preprocessing the dataset:
• Minimum Document Count in a Cluster: We require that a news cluster must contain a minimum of three documents, allowing a document to serve as a summary for the remaining documents in the cluster. Clusters not meeting this requirement are excluded.
• Minimum Summary Size: We hypothesize that a significant variance in summary lengths during pretraining could hurt performance. Therefore, we ensure that candidate summaries during pretraining are not too short, setting a minimum requirement of 250 tokens. Clusters not meeting this requirement are also excluded. In contrast, Xiao et al. (2022) can control the length of their synthetic reference summaries, ensuring that the sentence count
in the synthetic summary constitutes at least 30% of the total sentences in the cluster.
Additional preprocessing steps are outlined in Appendix D. After applying these criteria, we retain 172K clusters, approximately 45% of the total clusters in the NewSHead corpus (Gu et al., 2020).
Comparison models In addition to the reported scores of PRIMERA (denoted as PRIMERA* in Table 2), we independently reproduce the PRIMERA model scores by running inference using the PRIMERA checkpoints available in the Transformers (Wolf et al., 2020) library. Similar to the findings of Giorgi et al. (2022), we note that our reproduced scores are lower than those reported by Xiao et al. (2022), an exception being the zeroshot results for the WCEP dataset. For a broader comparison, we also consider the Pegasus model proposed by Zhang et al. (2020). Pegasus is a pretrained model focusing on single-document summarization (SDS), which obtains strong results on multiple SDS datasets such as XSum (Narayan et al., 2018) and CNN-DailyMail (Hermann et al., 2015).

heading: 4 Results
text: We conduct experiments in three settings: zeroshot, few-shot, and fully supervised.
Zero-shot In the zero-shot setting, we evaluate our pretrained Centrum model on the test datasets of Multi-News, WCEP, and DUC 2007. Following Xiao et al. (2022), the output length of the summary is set as the average length of the gold summaries of the test datasets. As Table 2 illustrates, Centrum outperforms the PRIMERA model in terms of ROUGE scores across all three datasets.
Few-shot In the few-shot setting, we follow the approach of Xiao et al. (2022) by conducting two sets of experiments. We randomly select 10 and 100 examples from the training set for model finetuning, and an equivalent number of examples from the validation set. To account for potential variance in scores due to example selection, we repeat this process five times with different seeds.
We observe that the summaries generated by Centrum are, on average, longer than those produced by PRIMERA. This is primarily a result of the Centrum pretraining objective, which imposes a minimum summary length of 250 tokens. In contrast, PRIMERA synthetic summaries are restricted
to a maximum length equating to 30% of the input set. To ensure a fair comparison, we truncate the summaries in the few-shot setting to match the lengths assigned in the zero-shot setting.
Table 2 presents the average scores obtained over the five seeds. Given that the DUC 2007 dataset contains only 45 examples, results are reported for training and validation with 10 examples. From the results, we see that Centrum outperforms PRIMERA across all datasets when finetuned with 10 examples. Furthermore, Centrum maintains performance parity with PRIMERA when finetuned using 100 examples.
Fully supervised In this setting, the pretrained models are finetuned on the training split of the Multi-News dataset. As reported in Table 3, the results from the fully-supervised experiments demonstrate that Centrum performs on par with PRIMERA on the Multi-News dataset.
Human Evaluation To complement the automatic evaluation results, we conduct a human evaluation study. Three professional linguists are
tasked with comparing the outputs of Centrum, PRIMERA, and Pegasus using the DUC 2007 dataset, and are compensated at rates higher than local minimum wages. The evaluation focuses on three metrics as outlined by Angelidis et al. (2021): informativeness (which assesses the consistency between model output and the human reference summary), coherence (which evaluates the ordering of information in the summary), and non-repetition (where a higher-quality summary exhibits fewer repetitions of words, phrases, or sentences).
The evaluators are presented with three summaries from the three models, randomly ordered, along with the reference summary. They are then instructed to rank the summaries from best (+1) to worst (-1) for each of the three metrics. These rankings are summed and scaled by the number of examples (20), producing scores that range from 100% (best) to –100% (worst). The results of this human evaluation are presented in Table 4.
Our findings indicate that Centrum significantly outperforms Pegasus across all three metrics, as confirmed by a one-way ANOVA with a post-hoc Tukey test (p ≤ 0.05). In comparison to PRIMERA, Centrum is significantly better in terms of informativeness and performs comparably in terms of coherence and non-repetition. Pegasus, on the other hand, is marked by heavy repetition within its summaries, which likely accounts for its lower scores.

heading: 5 Conclusion
text: We propose a centroid-based pretraining objective for multi-document summarization. Through experiments, we see that our model Centrum outperforms the existing state-of-the-art model PRIMERA on zero-shot settings and is comparable with PRIMERA in few-shot and supervised settings.

heading: 6 Limitations
text: As mentioned in the main paper, one of the limitations of our Centrum model is that it tends to produce longer outputs in comparison to PRIMERA. This necessitates controlling the length of the summary by truncating to a desired length. Moreover, due to our requirement of at least three documents in a cluster for centroid computation, we are unable to utilize clusters of only two documents present in Gu et al. (2020). This constraint significantly reduces the utilizable corpus size, leading us to work with roughly 45% of the corpus size used by PRIMERA. Future research could explore the possibility of initializing Centrum with the gap sentence generation-based Pegasus (Zhang et al., 2020) single document summarization objective, potentially allowing for full utilization of the corpus size of Gu et al. (2020).

heading: Acknowledgements
text: This research was supported by funding from the Institute for Infocomm Research (I2R) under A*STAR ARES, Singapore, and by the National Research Foundation, Prime Minister’s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. The work was supported in part by ERC Advanced Fellowship GA 742137 SEMANTAX and the University of Edinburgh Huawei Laboratory. Parag is supported by Huawei and the
UKRI Centre for Doctoral Training in Natural Language Processing (grant EP/S022481/1). We thank the anonymous reviewers for their constructive feedback.

heading: A Potential Risks
text: Despite our model’s potential, there is a risk that the generated summaries might not accurately represent the input document due to noise present in the training and finetuning examples. At the same time, we believe that our Centrum pretraining strategy doesn’t affect the factuality of the model either positively or negatively compared to Xiao et al. (2022). Future research will aim to explicitly evaluate and improve the factuality of our model’s output.

heading: B Details of the Datasets
text: Table 5 provides detailed information about the datasets used in our study. The NewSHead, MultiNews, and DUC 2007 datasets all originate from the news domain, while the WCEP dataset is derived from the Wikipedia Current Events Portal.

heading: C Hyperparameter Details
text: Our hyperparameters are similar to Xiao et al. (2022). We train for 100K steps with a learning rate of 3e-5. We evaluate every 500 steps and earlystop on the validation perplexity with a patience of
50. Pretraining Centrum on a 4-node A100 GPU took around 4 days. We computed the results using ROUGE (Lin, 2004) library 2 with the default settings and ‘–use_stemmer’ argument.

heading: D Additional Preprocessing Steps
text: • Removing boilerplate text from summaries: We remove boilerplate text such as “Sorry, this video isn’t available any more.”, “Advertisement Story continues below” from the summary sentences using regular expression based cleaning.
• Truncation of documents: We truncate each document in the cluster to the maximum length of source context allowed in LED divided by the count of the documents in the cluster. Thus, each document has a proportional representation in the cluster, similar to Xiao et al. (2022).

heading: E Software and Licenses
text: Our model relies on datasets downloaded from HuggingFace datasets (Lhoest et al., 2021) (Apache 2.0). We release our models under the Apache 2.0 license.

heading: F Human Evaluation
text: Figures 1 and 2 show the screenshots of the user interface presented to the raters.
2https://github.com/google-research/ google-research/tree/master/rouge
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6

heading: 3 A2. Did you discuss any potential risks of your work?

heading: Appendix A

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3 Models and Datasets

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3 Models and Datasets, and Section B of Appendix

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section E

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.

heading: B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section B of Appendix
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section B
C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Section 3, Section C of Appendix
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3, Section C of Appendix

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section C of Appendix
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 4 (Human evaluation)

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 4 (Human evaluation) and Section F in Appendix

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 4 (Human evaluation)

heading: 3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Section F in Appendix

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Not applicable. Left blank.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Not applicable. Left blank.

heading: Future Work
text: Future research will aim to explicitly evaluate and improve the factuality of our model’s output