Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval
Authors: John Michael Giorgi
Luca Soldaini
BO WANG
Gary D. Bader
Kyle Lo
Lucy Lu Wang
Arman Cohan
Venue: EMNLP 2023 Findings (2023)
Keywords: summarization
multi-document summarization
retrieval
open-domain
TL;DR: We study a new, open-domain task definition for MDS where only a query is provided as input, and the relevant documents to summarize must be retrieved.

Abstract: Multi-document summarization (MDS) assumes a set of topic-related documents are provided as input. In practice, this document set is not always available; it would need to be retrieved given an information need, i.e. a question or topic statement, a setting we dub "open-domain" MDS. We study this more challenging setting by formalizing the task and bootstrapping it using existing datasets, retrievers and summarizers. Via extensive automatic and human evaluation, we determine: (1) state-of-the-art summarizers suffer large reductions in performance when applied to open-domain MDS, (2) additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents. Based on our results, we provide practical guidelines to enable future work on open-domain MDS, e.g. how to choose the number of retrieved documents to summarize. Our results suggest that new retrieval and summarization methods and annotated resources for training and evaluation are necessary for further progress in the open-domain setting.

Open Domain Multi-document Summarization: A Comprehensive Study of Model Brittleness under Retrieval
John Giorgi 1,23+f
Luca Soldaini Bo Wang!. 1.3
Gary Bader
Kyle Lo
Lucy Lu Wang4.5 Arman Cohan 4.6
University of Toronto
Terrence Donnelly Centre
Vector Institute for Al Yale University
Allen Institute for Al
á University of Washington
john. giorri@utoronto.c.. (lucas, kylelJ@allemai.org, lucylw@uw.edee arman. www.nnalle.eeedeuuu
Abstract
document index
information need
Open-domain MDS
Multi-document summarization (MDS) as- sumes a set of topie-related documents are pro- vided as input. In practice, this document set is not always available; it would need to be retrieved given an information need, i.e. a ques- tion or topic statement, a setting we dub "open- domain" MDS. We study this challenging setting by formalizing the task and bootstrap- ping using existing datasets, retrievers and summarizers. Via extensive automatic and hu- man evaluation, we determine: (I) state-of-the- art summarizers suffer large reductions in per- formance when applied to open-domain MDS, (2) additional training in the open-domain set- ting can reduce this sensitivity to imperfect re- trieval, and (3) summarizers are insensitive to the retrieval duplicate documents and the or- der of retrieved documents, but highly sensitive to other errors, like the retrieval of irrelevant documents. Based on our results, we provide practical guidelines enable future work open-domain MDS, e.g. how to choose the number of retrieved documents summarize. Our results suggest that new retrieval and sum- marization methods and annotated resources for training and evaluation necessary for further progress in the open-domain setting.'
xetxiever
Traditional MDS
input docunents
summarizer
sunnaxy
Figure 1: "Trrditional" MDS assumes a topic-related set of documents is given at train and test time. Here, investigate the more challenging "open-domain" setting, where the document set must be retrieved given a query.
and important applications, e.g. the summarization of news articles (Fabbri et al., 2019; Gholipour Gha- landari et al., 2020), scientific literature (Lu et al., 2020; Wallace et al., 2021; De Young et al., 2021), and legal documents (Shen al., 2022).
Existing MDS task definitions, including query- focused MDS (see $3 for detailed comparison), assume a ground-truth, topic-related document set is provided at train and test time. This document set is often an artifact of the dataset curation process; in many practical settings, it is not available priori and would need to be defined by information need, expressed as query. 2 Documents relevant to the query would need be retrieved from a large collection of (mostly irrelevant) documents and summarized (Figure I): a setting we refer to as open-domain MDS. Because even state-of-the-art information retrieval (IR) methods are imperfect, errors. like the retrieval of irrelevant documents, will occur. It is open question how existing summarizers behave under this more challenging but realistic setting. Our major contributions are:
1 Introduction
Summarization is an NLP task that aims to generate accurate and coherent summaries for some given text automatically. Multi ducti-document summarization (MDS) extends this task to provide multiple topic- related documents as input, with the goal of sum- marizing salient information while avoiding redun- dancy. MDS a popular research objective with many proposed approaches (Yasunaga et al., 2017;
We formalize the task definition of open-domain MDS ($2), and bootstrap its study using existing
Liao et al., 2018 8; Liu and Lapata, 2019; Li et al., 2020; Jin et al., 2020; Mao et al., 2020; Zhang et al., 2020a; Pasunuru et al., b021b; Xiao et al., 2022)
PE.g. question: "Does vitumin improve physical capu- bilities of elderly patients?" or topic statement: "Report on vulnerabilities of power grid efforts to improve it." 'Inspired by the QA literature, where "open-domain" also denotes the setting where only a query is provided as input
"Work performed during internship A12 Core contributors. See author contributions chttps://ggithhe com/a wwwellll/aaa// opennmmdm

---

In traditional MDS, model is given a set of topic- related documents D= = [d1..... d and must gen- erate summary S that accurately and coherently summarizes the information in D. Such models are typically trained in supervised fashion to minimize the difference between S and (usually human-written) reference summary R. The goals (and evaluation) remain the same in open-domain MDS, but instead of D, the inputs are a query q and a large index of documents Dindex, where iDindex| D). The specifies of q, Dindex, and S will depend on the application or use-case. For example, in automatic literature review (Wallace al., 2021; Youun et al., 2021), q would be research question or statement, e.g. "Is massage therapy effective for people with musculoskeletal disorders?", Dimuex would be a large corpus of sci- entific literature (e.g. PubMed), and S would be a literature review-style discussion, e.g. "Massage therapy, as stand-alone trealment, reduces pain and improves function compared to treatment..." datasets, retrievers and summarizers ($4);
which we will refer works as follows: retriever ranks all documents in Pindex from most-to-least relevant given q. The top-k documents are input to a summarizer; & is a parameter that may be tuned for a particular use case. This approach has some desirable properties: (1) Dindex can be updated with new documents with- out re-training the retriever summarizer, and (2) it provides provenance for model-generated sum- maries: the top-k documents. In the remainder of the paper, we focus our investigation on the seriive-then-summmar approach.
We evaluate summarizers in the open-domain setting and find, through automated and human evaluations, that performance degrades signifi- cantly ($5). Promisingly, we show that additional training in the open-domain setting can reduce this sensitivity to imperfect retrieval ($6);
Finally, we subject summarizers to an extensive suite of carefully designed input perturbations to determine which retrieval errors are driving the degradation in summarization performance ($7)
Based on our results, we provide detailed, practical guidelines for future work in open-domain MDS. We release all code, model & data artifacts, and ex- perimental results created during our investigation.
Related Work
Query-focused MDS In query-focused MDS (QMDS, Wang et al., 2013; Feigenblat et al., 2017; Xu and Lapata, 2020; Pasunuru et al., 2021 a), a query is provided alongside a set of topic-related input documents and used to guide summarization. For example, extractive QMDS methods use query relevance to select the sentences that form the sum- mary. No retrieval from document index is per- formed. Here, we propose and investigate the more challenging scenario where, given only a query, the input documents must be retrieved from a large index containing mostly irrelevant documents.
2 Task Definition: Open-domain MDS
Previous attempts at open-domain MDS Liu et al. (2018) proposed the WikiSum dataset. Given the title of a Wikipedia article and collection of non- Wikipedia reference documents, the goal to generate the first section of the article. The pro- posed extractive-absstrativ approach resembles open-domain MDS. However, the document index is small (10s to 100s of documents) and composed of relevant documents (references of the article plus ten pages of search results using the title query). We study the more challenging and more general setting where the index is large (> 10,000 documents, see Table 6) and contains many more irrelevant documents than relevant documents.
There are multiple ways to approach open- domain MDS. We could think of q as a prompt to large language model (LLM) capable of in- context learning (ICL, Brown et al. 2020); which case we can view Dindex as the LLMs training data and retrieval as happening implicitly during infer- ence. However, because all information is stored in the model's weights, this approach requires ex- tremely large models, cannot produce summaries for events outside the training data, and does not provide provenance for the generated summaries. Another approach is to introduce an explicit re- trieval step over an external knowledge source,
In Zhang et al. (2021), method similar to serreve-then-summaan
is proposed, using pre- trained dense passage retriever (Karpukhin et al., 2020) and T5 (Raffel et al., 2020) as summarizer." The model is trained & evaluated on a dataset con- structed from existing QMDS datasets. This dataset is small (~~90 training examples) and does not ap- pear to be publicly available. Here, we conduct a large-scale analysis on multiple datasets from dif- ferent domains (each consisting of thousands of ex-
*At the time of writing, this work is unpublished

---

amples) and evaluate several of the top-performing multi-document summarizers currently available ($4). We also extensively simulate document re- trieval errors to probe their relative impact on sum- marization ($7). Together, this allows us to draw broader conclusions about open-domain MDS and provide detailed practical advice for future work.
weighted counts of overlapping terms. Dense re- trievers embed documents & queries into shared vector space and use proximity to determine rele- vance. Retrievers from these families exhibit dif- ferent characteristics and limitations (Macevaney et al., 2022); therefore, we investigate represen- tative retriever from each: BM25 (sparse, Robert- son et al., 1994) and Contriever (dense, Izacard et al., 2022). Both achieve strong zero-shot perfor- mance, making them particularly suitable for our purposes. See Appendix B for details.
Open-domain QA Our open open-domain MDS pro- posal mirrors a similar trend in question answer- ing (QA). While earlier research focused on an- swering questions provided text passage (Ra- jpurkar et al., 2016, 2018), the now predominant approach, open-domain QA (ODQA), is to answer questions without this passage, usually by referenc- ing an external knowledge source (e.g. Wikipedia). Even broader are knowledge-intensive (KI) lan- guage tasks (Petroni et al., 2021), which include ODQA but also, for example, fact-checking. KI tasks are commonly approached with a retrieve- then-generate framework (Guu et al., 2020; Lewis et al., 2020b; Borgeaud et al., 2022). Retrieve-then- summarize is similar, except that the outputs are, on average, much longer and tend to be less extractive than the outputs of KI language tasks like ODQA.
4.3 Multi-document Summarization Models All MDS models we experiment with are transformeerbased encoder-decoders (Vaswani et al., 2017) trained for abstractive summarization, representing the state-of-the- approach. The in- put contains one or more documents concatenated with special tokens (c.g. <doc-sep>). As is typical, we truncate each document based on the maximum input length of the model divided by the total num- ber of documents. See Appendix C for details.
4,4 Operationalize Retriev-then-Summarize
To extend these datasets and models to the open- domain setting and operationalize the retrieve-then- summarize approach, we address the following:
4 Bootstrapping Open-domain MDS
Since no large-scale annotated datasets or trained models exist for open-domain MDS, we bootstrap this task using existing datasets 154.1) and mod- els ($4.2, $4.3). We describe operationalization considerations in $4.4 and evaluation in $4.5.
How to choose query? In open-domain MDS, query anything that defines the documents to summarize, e.g. a question or topic statement. Ide- ally, a -riteewritten query would be available for each example in our dataset; however, exist- ing MDS datasets do not provide these queries. Therefore, we use R,? the human-writter reference summaries, as pseudo-queries, describe the input documents of each example.*
4.1 Datasets
We investigate representative selection of 5 MDS datasets comprised of news articles, medical stud- ies, and scientific literature, deliberately choosing datasets for which high-performing summarizers exist (see Appendix for more details). The inputs of these datasets generally consist only of the doc- uments to summarize. However, Multi-XScience and MS*2 each provide additional text as input: the target article's abstract and the target review's 's background section. In our experiments, we always provide this additional text and do not retrieve it.
they naturally
How to assemble the document index? For our purposes, we take the set of all documents in the train, validation, and test splits of each dataset to form Dindex- This guarantees that the ground-trutt documents for each example are present in the in- dex while providing plenty of negative examples.
How many documents to summarize? The number of retrieved documents summarize, k,
4.2 Retrieval Models
Broadly speaking, retrievers are divided into two categories, sparse & dense. Sparse retrievers de- termine relevance of document to a query using
"See the BEIR (Thakur al., 2021) zero-shot benchmark "Except for MS'2, where we found the provided "back- ground" section to perform better as a query; see Appendix A "We also experimented with query generation using LLMs (c.g. GPT-3), but found that they significantly under- performed the reference summary as query, e.g. by at least 8 points P/R @K K on a sample of the Multi-News validation set
"Existing query-focused MDS datasets (e.g. DUC 2005, 2006 & 2007) are extremely small (10s of examples) and are therefore not suitable for the large-scale analysis we conducted

---

Table Results of the open-domain MDS experiments. We observe the following: retrieval performance ranges from high (Multi-News, WCEP-10, dark blue) to low (Multi-XScience, MS*2, Cochrane), (2) when summarizers trained on these datasets are provided retrieved documents, they suffer from significant drops in performance (dark red); more severe performance drops were observed in cases where baseline summarization performance was relatively high (dark green). Experiments here used sparse retriever (BM25) and max top-k strategy (see Table 9 for complete results with all top-k strategies). Similar results were observed using dense retriever (Contriever, see Table 10). Statistically significant results are underlined (paired t-test, p= 0.01).
Retrieval
Summarization
Dataset
Model
P@K
RØK
ROUGE-Avg F1
ROUGE-Avg Fl BERTScore FI
A BERTScore F!
Multi-News
PRIMERA
0.22
0.82
31.66
-7.39
31.78
-10333
PEGASUS
31.23
-8.99
29.88
.10.77
LSG-BARR-base
30,05
-6.44
26.57
-8.17
GPT-3.5-turbo
23.86
-2.46
21.68
-3.92
WCEP-10
PRIMERA
0.63
0.67
35.50
-1.02
48.26
-0.76
-SG-BART-buse
35.76
-1.15
48.17
-0.85
GPT-3.5 -turrbo
26.36
-0.22
32.72
-0.25
Multi-XScience
PRIMERA
0.06
0.40
18.31
-0.57
10.57
-1.82
MS"2
LED-base
0.16
-0.14
22.74
-0.47
Cochrane
LED-base
0.17
0.57
17.39
-0.28
23.12
-2.11
is a parameter that be tuned for different use cases. To determine its impact on summarization performance, investigate three strategies:
ROUGE- F1, the average Fl-score of ROUGE-
1/2/L. We also report BERTScore (Zhang al., 2020b), which has been shown better correlate with human judgment (Yuan et al., 2021; Fischer et al., 2022). For document retrieval, we report the precision and recall at (P@K K and R@ ())); which are suitable metrics when the input documents do not have an inherent order, as is usually the case in MDS. We evaluate on the test splits of each dataset, except for MS²2 and Cochrane, where evaluate the validation because the test split is blind.
Max: Choose as the maximum number of input given dataset.
documents for any example
Tends to select for recall at the cost of precision.
Mean: Choose as the mean number of input documents for examples in given dataset. Tends to select for precision at the cost recall.
Oracle: Choose l as the number of ground-truth input documents for each example. This mimics the scenario where all documents with rele- vance score (assigned by the retriever) above certain optimal threshold are retained.
Evaluating Open-domain MDS
Here we present the results of our open-domain MDS experiments. In general, we find existing summarizers suffer large reductions in performance when applied open-domain MDS, even when retrieval performance is high (Table 1). Below, we provide key observations on how the individ- ual components (retriever and summarizer) behave within serrive--hee-summmaan framework.
We note that this is highly idealized setting. Using R as query leaks information about the reference summary into the retrieval step, likely inflating retrieval and summarization performance. In prac- tice, Dindex will be much larger (e.g. PubMed-, Wikipedia-, or even Web-scale), making retrieval more difficult. Our intention to determine a lower-bound for the expected performance degra- dation of state-of-the-ar summarizers in the open- domain setting; as we will show in $5, even this idealized setting often leads to large reductions in performance. In $7. we extensively simulate docu- ment retrieval errors to determine how summarizers behave in both low- and high-performing retrieval settings across a variety of retrieval error types.
Strong summarizers more sensitive to imper- feet retrieval than weak ones We observe a rela- tionship between summarizer's (baseline) perfor- mance on a dataset and its sensitivity to imperfect document retrieval (Table 1). The largest reduc- tions in summarization performance were observed for the most performant summarizers, despite re- trieval performance being the highest these cases. However, this relationship is confounded by differ- ences between datasets. To control for this, we con- duct experiments comparing fine-tuned PRIMERA
4.5 Evaluation
We follow previous work by evaluating summariza- tion with ROUGE-1/211 scores (Lin, 2004). To provide a single metric for comparison, we report
"Results for sparse and dense retrievers were comparable and exhibited similar trends. We elect to show results for the sparse retriever; see Appendix E for dense retriever results

---

Table 2: Results of the open open-domain MDS experiments with zero-shot summarizers. Controls for differences datasets and models, isolating the relationship between summarization performance in the traditional and open- domain settings. Top-k strategy mean is used. Statistically significant results are underlined (paired t-test, p 0.01).
Retrieval
Summarization
Dataset
Model
Retriever
PøK
ReK
ROUGE-Avg F1
ROUGE-Ayg
BERTSeore
BEKTSoore
Mulri-Nows
PRIMERA
sparse (BM25)
0.64
0.74
31,66
-2.82
31.78
-4.08
dense (Contriever)
0.59
0.70
.3.31
..60
sere-shot
23.58
-0.09
18.66
-0.39
-0.4
WCEP-10
PRIMERA
sparse
0.66
0.64
35,50
-0.90
48.26
-0.68
dense
0.66
0.63
00.14
+0.68
3070-shol
spurse
21.43
+0.35
25.48
+0.72
dense
+1.00
42.19
XScience
sparse
0.27
10.57
-127
dense
0.16
0.24
0.81
-0.96
zete-shot
15.18
+0.69
6.02
-0.47
dense
+0.00
Table 3: Comparing ROUGE-AV Avg Fl scores of model goner--generated summaries to heuristic baselines. In some cases, heuristics perform surprisingly close trained summarizers. All Lead the concatenation of the first sentence from each input document. Oracle document is the document with the highest token overlap with the reference summary; oracle lead is the first sentence from this document. Background/ MS² and Multi Salti-eSSeeene. The best baseline for each dataset bolded.
srbouund/bbtracttt Abstract the additional input from
Baselines
Datasel
Best Sunmarizer
Random Summary
A All Lead
A Oracle Document
A Oracle Lead
A Background/ Ahstrac
Multi-News
31.7
-18.3
-15.3
.4.1
-21.8
WCEP-10
35.8
-2..8
-2.44
-15.3
-9,9
Multi-XScience
18.3
-6.2
-5.0
-0.8
-9.3
-2.3
MS"2
19.7
-10.4
-11.0
-7.6
-4.0
-0,2
Cochrane
17.4
-5.0
-422
-3.9
-3.5
PRIMERA A evaluated zero-shot (Table 2). 10 This allows to hold the dataset, model architecture, and retriever constant, isolating the relationship between summarization performance the tradi- tional and open-domain settings. Here, the trend is clear: "strong" summarizers are more sensitive to imperfect retrieval than "weak" summarizers."
for Multi-XSeient by copying the document with the highest token overlap to the reference summary. Future work should carefully establish that summa- rizers are performing adequately before attempting the more difficult open- open-domain setting.
Better retrieval performance o better summa- rization performance Performance of the sparse and dense retrievers was generally comparable (Ta- ble 7), with the sparse retriever performing better some datasets (Multi-News, WCEP-10, Multi- XSeience, see Table 9) and the dense retriever per- forming better on others (MS*2, Cochrane, see Ta- ble 10). Interestingly, however, better retrieval per- formance did not always correspond with smaller reductions in summarization performance. E.g. on WCEP-10, the sparse retriever performed slightly better, but the reduction in summarization perfor- mance considerably larger. On MS*2 and Cochrane, the petter-performing dense retriever led to a larger reduction in summarization perfor- mance. This suggests that the two types of retriev- ers are making characteristically different e2 that P/R @K do not completely capture. Future work should consider summarization performance
One explanation is that weak summarizers have less to lose from imperfect retrieval, perhaps be- cause they are not performing adequately even when trained and evaluated on ground-truth inputs. They may, to a greater degree than strong summa- only
rizers: hallucinate, exploit heuristics, fraction of the input documents (Kryscinski et al., 2019; Wolhandler al., 2022). To probe this, we construct several baselines that mimic these be- haviours. We find that, for example, copying the background section of MS*2 performs comparably to trained model, suggesting that the observed insensitivity to retrieval errors could be due to sum- marizers exploiting this heuristic (Table 3, see Ap- pendix E. for details). We observe similar result
10 Thhee only model we evaluate with zero-shot capabilities 1I We use "strong" and "weak" as shorthand to refer to cases where summarization performance is high (c.g. PRIMERA on Multi-News) and low (c.g. LED on Cochrane)
Previously noted by MacAvancy et al. (2022)

---

Table 4: Human evaluation on Multi-News. A bino- mial test three human annotators for 50 random test examples was conducted for each facet (exclud- ing ties). All results statistically significant (p < 0.01). Inter-annotator agreement reported as Fleiss' Kappa (x).
Prompt Template
Natural language instructions
You are an expert journalist. Given multiple news articles about particular event, write summary of Reepond
approximately inax, woxds| journalese' Cite sources and provide quotes from the source documents where appropriate, Do not refuse to answer. example summaries general guidonce about the expected length and style.
baseline
open-domain preferred
Facet
preferred
p
Coverage
5,97c-05
0.32
Informativeness
69
27
2. 15c50
0.47
In-context examples (up
Exomple summaries texamples]
itself, alongside IR metrics like P/R@K, when tun- ing retrieval pipelines for open-domain MDS.
Test input
The number of documents to retrieve matters We observe clear differences in the strategy for choosing k, the number of retrieved documents to summarize. Unsurprisingly, the oracle strategy almost always leads to the smallest reduction in summarization performance. This strategy closely mimics the setting of retaining all documents with relevance score (assigned by the retriever) over certain threshold but assumes a strong retriever and a well-calibrated threshold, both of which may be difficult to achieve in practice. Our results suggest that setting as the mean number of relevant doc- uments (if an accurate estimate can be made) is a second-best strategy. We note that, relative to max k, mean & tends to select for precision over recall (see P@K K vs. R@K scores in Table & Table 10); future work should consider tuning & for precision.
Source documents
Edocuments]
Summary:
Figure 2: Prompt template for our in-context learning (ICL) based approach. Each prompt includes natural language instructions, example reference summaries in-context examples, and unseen source documents as input, max_words set per dataset, 384 for Multi-News and 32 for WCEP-10 10. examples (2 for Multi-Ners News and for WCEP- 10) are randomly selected from train and the same examples are used for every input.
ting compares to the fine-tuned models that were the focus of our evaluation. We chose GPT-3.5- turbo as the LLM and designed a suitable prompt (tuned based on validation set performance) which contains some natural language instructions and a few example summaries (Figure 2). We omit exper- iments on the MS and Cochrane datasets, whose source documents, even when truncated to the first 25, exceed GPT-3.5's maximum input token length of 4096 (see Table 6). We also omit experiments on as a snapshot of arXiv is presum- ably included the model's 's training set; therefore we cannot control for possible train-test contami- nation as the reference summaries are the related work sections of arXiv papers. To fit within the maximum token limit of the model, we use only ex- ample summaries the in-context examples (omit- ting the source documents) and randomly choose example summaries from the train-set for Multi- News and for WCEP-10. 10. To make results as re- producible as possible, we set the tempera a and used the 03/01/2023 GPT-3.5-turbo snapshot. All other hyperparameters of the OpenAl API are left at their defaults. 14 Lastly, due associated costs in using the model, restrict experi-
Human evaluation confirms degradation of sum- marization performance Automatic evaluation metrics like ROUGE are imperfect and may not correlate with aspects of human judgment. 13 There- fore, we conducted a human evaluation to validate results. In short, human annotators have sta- tistically significant preference for summaries pro- duced by the "baseline" model (no retrieval) along two facets, coverage and informativeness (Table 4), corroborating the degradation of summarization performance in the open-domain setting as quanti- fied by the automatic metrics. See Appendix for full details, including a manual analysis of example summaries produced in the open-domain setting.
In-Context Learning with LLMs In-context learning (ICL.) with large language models (LLMs) has emerged as a viable approach to zero-shot sum- marization (Goyal et al., 2022). We conducted preliminary experiments using this approach to de- termine how its behaviour in the open-domain set-
19 phttps://platforrt api-ref api-reference// ooiirreferenne/ccomletiioos
13 "Se $8 for an extended discussion
completions

---

1C Mub--SScinno-
oddition
duplication
retrieved
ground-bruth
A
B
A
B
B
ROOUGE-AV F1
delation
replacement
-2
B
B
ABERTScore F1
sorting
backtranslation
B
A
B
backtzarslate
additional training epochs
Figure 4: Graphical depiction of perturbations. Dashed line indicates input documents. Uncoloured documents ground-truth green have been added, red removed and blue modified. Unique documents are lettered.
Figure 3: Fine-tuning summarizers in the open-domain. Additional training on relrieved documents can reduce sensitivity to imperfect retrieval; often at the cost of performance on ground-truth documents. Dashed grey line represents no change in performance.
WCEP-10 and Cochrane), and sometimes unsta- ble (e.g. WCEP, Cochrame). We note again that our retrieval setting is highly idealized ($4.4); nonethe- less, our results suggest that existing summariz- ers could be adapted to the open doen-domain setting if query- annettan training examples and appropriate document indices constructed.
ments to the sparse retriever. Results are presented in Table 1. The ICL-based approach significantly underperforms the fine-tuned models (by at least ~ points) but outperforms PRIMERA zero-shot. The general trends (and relative magnitude) of the degradation in performance in the open domain comparable. We note that previous work has found that human annotators often prefer GPT-generated summarics over those generated by smaller, finc- tuned models, even when automatic metrics like ROUGE disagree. However, this observation is restricted to the single-document setting (Goyal et al., 2022). Future work is needed to determine if LLMs exhibit different behaviour under the open- domain MDS setting than their smaller, fine-tuned pre-trained-languaal model (PLM) counterparts.
Simulating Document Retrieval Errors
In this section, we investigate what is driving the re- duction in summarization performance in the open- domain setting ($5). We begin by carefully cate- gorizing the various retrieval errors that can occur E.g., could erroneously retrieve documents irrel- evant to the query. For each error type, design corresponding "perturbation" that can be applied to the inputs of cxisting MDS datasets before they are fed to a summarizer. The perturbations are de- scribed below and depicted graphically in Figure 4:
6 Training in the Open-domain Setting
Addition: Add one more irrelevant docu- ments. This could occur if we retrieve all relevant documents but also retrieve irrelevant ones.
A natural question is whether summarizer's bustness document retrieval errors test time
can be improved by exposing the model to similar errors at train time, To explore this, we retrieve documents for examples the train set of each dataset and fine-tune summarizers on these exam- ples. We then evaluate them on both the retrieved document set and the original (ground-truth) docu- ment set (Figure 3, see Appendix F for details). We find clear cases where summarization performance in the open-domain benefits from the additional training (e.g. Multi-News, Multi-XScience), but that this benefit can come at the cost of performance on the ground-trutt truth documents (e.g. Multi-News,
Deletion: Remove one or more documents. This could occur we retrieve only a fraction of all relevant documents.
Replacement: Replace one or more relevant documents with irrelevant ones. This could occur if we retrieve the correct number of documents but substitute relevant ones for irrelevant ones.
Duplicate one or more documents. This could occur if duplicate (or, more likely, near-duplicate) documents exist in the index.!! 15
Duplication:
Deduplication is non-trivial (Lee et al., 2022) and near-

---

PRIMERA, Multi-News
LED- LED-Base, Cochrane
addition duplication doleton
replacement backFansiation
addition dupticadon- delation roplocament bacitrensiaaion
random
oracle
random
BERTScrre AROUGE Avg F1
&ROUGE Avg F1
-10"
-10
-10
A BERT F1
10
-10
-10
-10
0:20
20-40
40-60
66-80
80-100
0.20
20-40
40-60
60-80
80 10:
of input documents perturbed
% input documents perturedd
Figure 5: Results of the perturbation experiments Multi-News (left) and Cochrane (right). Mean change in summarization performance plotted against the percent of perturbed imput documents. Values above -0.49 ROUGE are shaded in green, and values below in red, the average difference in summarization performance reported in *CL conferences. Y-axis is displayed in symlog scale. 68% confidence intervals (CI) are plotted as errr bands.
Sorting: Modify order of documents. Input doc- uments for MDS are typically unordered. How- ever, many methods concatenate documents be- fore providing them as input, and it is unknown if models are sensitive to this order. Different or- derings could occur, e.g., if documents are sorted by order of relevance before concatenating.
Oracle: Select documents in way that mimics strong retriever. E.g. in deletion, we remove ground-truth documents in order of least-to-mosi similar to the reference summary R, in addi- tion, we add irrelevant documents in order of most-to-least similar." 18
For perturbations that require selecting irrelevant documents (addition & replacement), we select from the set of all documents in the train, valida- tion, and test splits (excluding documents from the example we are perturbing). We evaluate summa- rizers under increasing amounts of perturbation: from 0% of documents perturbed up to 100%.
Token-level perturbation It is well known that NLP models are sensitive to minor token-level changes in their inputs (Prabhakaran et al., 2019; Niu et al., 2020; Ribeiro et al., 2020; Moradi and Samwald, 2021). To compare and contrast the document-level sensitivity we investigate with this known sensitivity, we include token-level perturbation, backtranslation, as control. Here, we translate one or more input documents to an- other high-resource language and back again. This causes small changes, e.g. paraphrasing and syn- onym substitution, allowing us to create semantics- preserving, token-level changes to a document." !6
7.2 Results of Simulation Experiments
In Figure 5, we display the results of our exper- iments simulating document retrieval errors for two model-dataset pairs. To better contextual- ize results, we shade differences in ROUGE 0.5 (average difference in summarization performance reported in *CL conferences, Deutsch et al., 2022) in red and the rest in green. This serves as a rough yardstick to help identify large drops in perfor- mance. We symlog (Webber, 2012) the y-axis to make small changes in performance more apparent. In general, results are congruent with open- domain MDS experiments ($5): (1) large reduc-
7.1 Selecting Documents to Perturb
Each perturbation requires selecting one or more documents to perturb, e.g., in addition and deletion, we need to choose which documents to add and which to remove. We investigate two strategies:
Random: Select documents to perturb randomly, mimicking a (very) weak retriever.
Siimilrr $5, this leverages I a pseudo-query "Determined using Sentence Transformers (Reimers and Gurevych, 2019); specifically Al-MMiniLM-L--v
duplicates are not uncommon large document collections like (Dodge et al., 2021) or S20RC (Lo et al., 2020)
'These are exemplary of main trends observed across all model-dataset pairs; see Appendix G for complete results
"See Appendix G.1 for details

---

tions in summarization performance are observed even in cases of few simulated errors (< 20%) and (2) strong summarizers (Figure 5, left) are more sensitive to retrieval errors than weak summarizers (Figure 5, right). Below, we discuss major trends.
formers are biased toward earlier tokens in their input (Hofstätter al., 2021), so we might ex- pect improved performance by placing the most similar content to R first, (2) a strong retriever would assign a higher rank to the most relevant documents, and we might choose to input docu- ments to our summarizer in this order.
Summarizers insensitive to duplicates and small token-level changes A consistent trend across models & datasets was an insensitivity to duplicate documents, even in the extreme case of >80% dupli- cation, suggesting that deduplication efforts the document index are unlikely translate improve- ments in summarization performance. However, this assumes duplicate documents are included without replacing relevant documents, which is pos- sible if k is based on relevance threshold cutoff. Another trend is that models are not overly sensi- tive to minor token-level changes (exemplified by backtranslation) relative to other perturbations, fur- ther motivating our focus on document-level errors.
We find no significant difference (paired t-test, p 0.01) in summarization performance for any model-dataset pair, except for WCEP-10 (see Ap- pendix G.2). Here we find that both models we evaluate (PRIMERA & LSG-BART) are negatively affected by random sorting. One possible explana- tion is that, due to how WCEP-10 was constructed, the documents are (partially) sorted in order of rel- evance (see Appendix A). Models trained on this dataset may have learned to exploit this, e.g., by as- signing more weight to carlier documents in the in- put. After randomly shuffling input documents, this learned heuristic would no longer hold, and sum- marization performance might drop accordingly.
Erroneous additions vs. erroneous deletions In the random setting, small amounts of deletion led to large drops in summarization performance. Con-
Conclusion
versely, deletion has surprisingly little impact in the oracle setting until most documents (>60%) are removed. These results have two non-mutually exclusive explanations: (1) summarizers only con- sider select few input documents, (2) reference summaries have low coverage of the input docu- ments, both corroborated by recent work (Wolhan- dler et al., 2022). Based on the addition pertur- bation results in the oracle setting, summarizers appear to be more sensitive to erroneous additions than deletions, which, alongside our top-k strat- egy results in $5, suggests that retrieval pipelines should be tuned for precision in open-domain MDS.
We present a new, open-domain task definition for MDS. This reformulation is more challenging and potentially more useful, enabling users to specify their intent with only query. Via extensive au- tomatic and manual evaluation, find that: (1) summarization performance significantly degrades in the open-domain setting, even when retrieval performance is high, (2) additional training can reduce this sensitivity to imperfect retrieval, and (3) summarizers are insensitive to the retrieval of duplicate documents and the order of retrieved doc- uments but highly sensitive to other errors, like the retrieval of irrelevant documents. Based on our results, provide practical guidelines, e.g. that retrieval pipelines for open-domain MDS should be tuned for precision. Curating high-quality MDS datasets annotated with queries will be necessary enable further progress in the open-domain setting.
Summarizers are insensitive to document order As far as we know, prior work has yet to investigate whether multi-document summarizers are sensitive input document order. Although the documents are generally considered unordered, they are usu- ally concatenated before providing them as input. To determine if summarizers are sensitive to this or- der, we sorted the input documents of each dataset before concatenation and re-evaluated the summa- rizers. We investigate two sorting strategies:
Limitations
Automated evaluation metrics may not corre- late with human judgment Though established metrics such as ROUGE and BERTScore are im- perfect (Deutsch et al., 2022), they are frequently used in the summarization literature, do correlate with aspects of summary quality, and are useful for comparing system- level performance, especially in scenarios such as ours where performance differ-
Random: Shuffle documents randomly.
Sort documents by similarity to the ref- crence summary, R. This motivated from two perspectives: (1) prior work has found that trans-
Oracle:

---

ences can be several points below the baseline. To validate our findings, we also conduct a human eval- uation to better understand qualitative differences in summaries generated in the open-domain setting (see Appendix H). The investigation of better au- tomated metries for natural language generation is active field of research, and we hope to integrate novel and performant metrics in future work.
Kyle, Lucy, and Arman were project mentors, con- tributing equally to project scoping and experimen- tal design and providing the core ideas and direc- tion throughout the course of the project and paper writing. Additionally, Luca made technical con- tributions to model implementation. Bo and Gary provided high-level feedback and advice in later stages of the project.
Results conflate dataset features and model per- formance Our evaluation conflates several is- sues beyond the relative performance of retriev- ers and summarizers. Dataset quality, the "multi- document-ness" of each dataset, and the shortcom- ings of automatic metrics all contribute to noise in results. For example, a dataset whose reference summaries have low coverage of the input docu- ments (as characterized by Wolhandler et al., 2022) would not be expected to respond to retrieval errors in the same way as dataset where this coverage is high. By experimenting with multiple datasets, retrievers, and summarizers, as well as in the syn- thetic perturbation setting ($7), we expect our sults be more resilient to these confounders.
References
Iz Beltagy, Matthew E.. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. ArXiv preprint, 355500000000000
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy. Jacob Menick, Roman Ring. Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals. Simon Osindero, Karen Si- monyan, Jack W. Rae, Frich Elsen, and Laurent Sifre. 2022. Improving language models by retrieving from trillions of tokens. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Bal- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 2206-2244 PMLR.
Specialized retrievers may lead to better perfor- mance We experiment with standard sparse and dense retrievers in the zero-shot setting. More ef- fort tuning retrieval pipelines, c.g. by introducing re-rankers (Pradeep et al., 2021) by fine-tuning retrievers directly on MDS datasets, may improve retrieval performance and lead to smaller summa- rization performance reductions. Additionally, bet- ter summarization performance might be achieved by retrieving content at the span-level, (as opposed to full documents). We leave the development of retrieval pipelines purpose-built and tuned for open- domain MDS to future work.
Tom B. Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. 7iegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Syxtems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, viriual.
Acknowledgements
This research was enabled in part by support provided by the Digital Research Alliance of Canada (alliancecan.ca) and Compute Ontario (www.comptteottrro.caa)
Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg. Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. 2016. Ms marco: A human generated machine reading compre- hension dataset. ArXiv preprint, abs//611.0922
Author Contributions
Charles Condevaux and Sébastien Harispe. 2022. Lsg attention: Extrapolation of pretrained transformers to long sequences. ArXiv preprint, abs/2210. 15497.
John Giorgi made most of the technical contri- butions, including dataset collection and process- ing, model implementation, and running experi- ments. John also contributed to project scoping and idcation, wrote the paper with feedback from everyone, and led the project in general. Luca,
Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. Re- examining system-level correlations of automatic summarization evaluation metrics. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics:

---

Human Language Technologies, pages 6038-6052, Seattle, United States. Association for Computational Linguisties.
Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2022.
News summarization and evaluation in the era of gpt-3. ArXiv preprint, abs/2209. 12356.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 708-719 New Orleans, Louisiana. As- sociation for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming- Wei Chang. 2020. Realm: Retrieval- augmented language model pre-training. In Proceed- ings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org.
Jay De Douonn Iz Beltagy, Madeleine van Zuylen, Bailey Kuchl, and Lucy Lu Wang. 2021. MS Multi- document summarization of medical studies. Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 7494- 7513, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Sebastian Hofstätter, Aldo Lipani, Sophia Althammer, Markus Zlabinger, and Allan Hanbury. 2021. Mit- igating the position bias of transformer models in passage re-ranking. ArXiv preprint, abs/2101.06980
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilaaroo, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021. Documenting large webtext corpora: A case study on the colos- sal clean crawled corpus. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1286-1305, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kai Hong and Ani Nenkova. 2014. Improving the esti- mation of word importance for news multi-document summarization. In Proceedings of the 14th Confer- ence of the European Chapter of the Association for Computational Linguistics, pages 712-721, Gothen- burg, Sweden. Association for Computational Lin- guistics.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebas- tian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave 2022. Unsupervised dense informa- tion retrieval with contrastive learning. Transactions Machine Learning Research.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstrac- tive hicrarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Asso- ciation for Computational Linguistics.
Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020. Multi- huutt-graanularity interaction network for extractive and abstractive multi-document summarization. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6244- 6254, Online. Association for Computational Lin- guistics.
Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki. 2017. Unsupervised query- focused multi-document summarization using the cross entropy method. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 961-964. ACM.
Karen Spärck Jones. 2004. A statistical interpretation of term specificity and its application in retrieval. J. Documentation, 60:493-502.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781. Online. Association for Computational Linguistics.
Tim Fischer, Steffen Remus, and Chris Biemann. 2022. Measuring faithfulness of abstractive summaries. In Proceedings of the 18th Conference on Natural Lan- guage Processing (KONVENS 2022), pages 63-73, Potsdam, Germany. KONVENS 2022 Organizers.
Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020. A large-scale multi multi-document summarization dataset from the Wikipedia current events portal. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1302-1308, Online. Association for Computational Linguistics.
Wojeiech Kryscinski. Nitish Shirish Keskar, Bryan Mc- Cann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 540-551, Hong

---

Kong, China. Association for Computational Linguis- tics.
summarization. In Proceedings of the 27th Inter- national Conference on Computational Linguistics. pages 199,, Santa Fe, New Mexico, USA. As- sociation for Computational Linguisties.
J. Richard Landis and Gary G. Koch. 1977. The mea- surement of observer agreement for categorical data. Biometrics, 33(1): 159-17444
Chin-Yer Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Bure Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 8424-8445, Dublin, Ireland. Association for Computational Linguistics.
Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In 6th International Conference Learning Representations, ICLR 2018. Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequenc pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
Yang Liu and Mirella Lapata. 2019. Hierarchical trans- formers for multi-document summarization, In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguisties, pages 5070- 5081, Florence, Italy. Association for Computational Linguistics.
Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kin- ney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 4983, Online. Asso- ciation for Computational Linguistics.
Patrick S. H. Lewis. Ethan Perez, Aleksandra Pik- tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis. Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmenter generation for wewllede-iintesive NLP tasks. In Advances in Neu- ral Information Processing Systems 33: Annual Con- ference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations. ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Quentin Lhoest, Albert Villanova del Moral, Yacine Jemite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu. Lewis Tunstall, Joe Davison, Mario Saško, Gun- jan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Man Major. Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus- sière, Lysandre Debut, Stas Bekman, Pierric Cis- tac. Thibault Goehringer. Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi- XScience: A large-scale dataset for extreme multi- document summarization of scientific articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8068-8074, Online. Association for Computa- tional Linguistics.
Edward Ma. 2019. Nlp augmentation.
Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, and Arman Cohan, 2022. ABNIRMI: Analyzing the behavior of neural IR models. Trans- actions of the Association for Computational Linguis- tics, 10:224-239.
Craig Macdonald and Nicola Tonellotto. 2020. Declar- ative experimentation ininformation retrieval using pyterrier. In Proceedings of ICTIR 2020.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summariza- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6232-6243, Online. Association for Computational Linguistics.
Yuning Mao, Yanru Qu, Yiqing Xie, Xiang Ren, and Jiawei Han. 2020. Multi-document summarization with maximal marginal relevance- -eriddd reinforce- ment learning. In Proceedings of the 2020 Confer- ence Empirical Methods in Natural Language Processing (EMNLP), pages 1737-1751, Online. As- sociation for Computational Linguistics.
Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Ab- stract Meaning Representation for multi-document

---

Milad Moradi and Matthias Samwald. 2021. Evaluating the robustness of neural language models to input perturbations. In Proceedings of the 2021 Confer- ence on Empirical Methods Natural Language Pro- cessing, pages 1558-1570, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Fabio Petroni, Aleksandra Piktus. Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedingg of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2224,2544, Online. Association for Computational Linguisties.
Ani Nenkova, Sameer Maskey, and Yang Liu. 2011. Au- tomatic summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, page 3, Portland, Ore- gon. Association for Computational Linguistics.
Vinodkumar Prabhakaran, Ben Hutchinson, and Mar- garet Mitchell. 2019. Perturbation sensitivity analy- sis to detect unintended model biases. In Proceedings of the 2019 Conference Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IICNIP),, pages 5740-5745, Hong Kong, China. Association for Computational Linguistics.
Ani Nenkova and Rehecca Passonneau. 2004. Evaluat- ing content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chap- ter of the Association for Computational Linguistie:: HLT-NAACL 2004, pages 145-152, Boston, Mas- sachusetts, USA. Association for Computational Lin- guistics.
Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models. ArXiv preprint, abs/2101.055 056677
Xing Niu, Prashant Mathur, Georgiana Dinu, and Yaser Al-Onaizan. 2020. Evaluating robustness to input perturbations for neural machine translation. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8538- 8544, Online. Association for Computational Lin- guistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 31:140:1-140:6;
Ramakanth Pasunuru, Asli Celikyilmaz, Michel Gal- ley. Chenyan Xiong, Yizhe Zhang, Mohit Bansal, and Jianfeng Gao. 202 Data augmentation for ab- stractive query-focused multi-document summariza- tion. In Thirty-Fifth AAAI Conference on Artificial Inteiligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence. IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Vir- tual Event, February 2-9, 2021, pages 13666-13674, AAAI Press.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable ques- tions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guisties (Volume Short Papers). pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD; 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2393-2392, Austin, Texas, Association for Computational Linguisties.
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Su- jith Ravi, and Markus Dreyer. 202 Efficiently summarizing text and graph encodings of multi- document clusters. Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies. pages 4768 4779, Online. As- sociation for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
pages
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf,, Edward Yang. Zachary Vito,, Martin Raison, Alykhan Te- jani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style. high performance deep learning li- brary. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NewrlPS 2019, De- cember 8-14, 2019, Vancouver, BC, Canada, pages 8034-8035.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin. and Sameer Singh. 2020. Beyond accuracy: Be- havioral testing of NI.P models with CheckL In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4902- 4912, Online. Association for Computational Lin- guistics.
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at trec-3. In TREC.

---

Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg. Margo Schlanger, and Doug Downey. 2022. Multi- lexsum: Real-world summaries of civil rights law- suits at multiple granularities. Advances in Neural Information Processing Systems, 15:13158-13173.
pages 5761-5769, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. PRIMERA: Pyramid-based masked sentence pre-training for multi-document summariza- tion. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 5245-5263, Dublin, Ireland. Association for Computational Linguistics.
Luca Soldaini. 2022. Pyterrier sentence transformers.
Nandan Thakur, Nils Reimers, Andreas Ruckl'e, Ab- hishek Srivastava, and Iryna Gurevych. 2021, Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. ArXiv preprint, abs/2 104.08663.
Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the lead bias in news summarization via al- ternating adversarial learning, In Proceedings of the 59th Anmual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 2: Short Papers), pages 948-954, Online. Asso- ciation for Computational Linguistics.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS- MT - building open translation services for the world. In Praceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479-480, Lisboa, Portugal. European Associa- tion for Machine Translation.
Yumo Xu and Mirella Lapata. 2020. Coarse-to- fo-e query focused documet summarization. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3632-3645, Online, Association for Computa- tional Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA. USA, pages 5998 6008.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek. Krishnan Srinivasan, and Dragomir Radev. 2017. Graph-based neural multi-document summa- rization. In Proceedings of the 21 Conference on Computational Natural Language Learning (CoNLL 2017), pages 452-462, Vancouver, Canada. Associa- tion for Computational Linguistics.
Byron C Wallace, Sayantan Saha, Frank Soboczenski, and lain J Marshall. 2021. Generating (factual?) narrative summaries of RCTs: Experiments with neu- ral multi-document summarization. AMIA Summits Transl. Sci. Proc., 2021:605-614
Lu Wang, Hema Raghavan, Vittorio Castelli, Radu Flo- rian, and Claire Cardie. 2013. A sentence com- pression based framework query-focused multi- document summarization.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In Advances in Neural Information Processing Systems 34; Annual Conference on Neural Informa- tion Processing Systems 2021, NeurlPS 2021, De- cember 6-14, 2021, virtual, pages 27263-27277.
Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1384- 1394, Sofia, Bulgaria. Association for Computational Linguistics.
Jingqing Zhang. Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020a. PEGASUS: pre- pre-training with ex- tracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, 1CML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1 1228-11339. PMLR
Beau W Webber. 2012. A bi-symmetric log transfor- mation for wide-range data. Measurement Science and Technology, 24(20.227001.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-a natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 45, Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Eval- uating text generation with BERT. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Weijia Zhang, Svitlana Vakulenko, Thilina Rajapakse, and Evangelos Kanoulas. 2021. Scaling up query- focused summarization to meet open-domain ques- tion answering. ArXiv preprint, abs/21 12.07536.
Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Da- gan, 2022. How "multi" is multi-document summa- rization? Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing,

---

A Dataset Details
Table Evaluated multi-document summarizers and the datasets for which fine-tuned model is publicly available (or was trained by us).
All datasets were managed in the Hugginglaace Datasets library (Lhoest et al., 2021). The exam- plcs of each dataset consist of an input document set, D and human- written reference summary, S. Multi-XScienee and MS each have an additional input that is always provided (and never retrieved perturbed), the target articles abstract and the target reviews background section. See Table for dataset statistics. Below, we provide detailed descriptions of each dataset:
Model
Fine-tuned on
Max Input Len.
Zero-shox?
LED
MS"2, Cochrune Multi-New
16384
K
PEGASUS
1024
X
Multi-Newx, WCTP--10
PRIMERA
4096
Multi XSeience
LSG-BART
Multi-News, WCEP-10
4096
x
Cochrane (Wallace et al., 2021): Similar to MS^2, except a background statement not included as input. There are 3752 examples in the train set and 470 examples in the vali- dation set. Each example contains between 1 and 537 documents, with mean of ~10.9.
Multi-News (Fabbri et al., 2019): Consists of news articles and summaries collected from www.er..rrccom com. There are 44,972 examples in the train set and 5622 examples in the test set. Each example contains between I and 10 documents, with mean of ~2.7.
B Retrieval Details
Document retrieval and evaluation are conducted in the PyTerrier library (Macdonald and Tonellotto, 2020). In Table 7, we present the retrieval perfor- mance on the train, validation and test split for each dataset, retriever, and top-k strategy. Below, we provided detailed descriptions of all retrievers:
WCEP-10: Consists of news articles and summaries collected from the Wikipedia Cur- rent Events Portal (WCEP22). WCEP-1021 sub-samples the top 10 most relevant doc- uments from the original WCEP dataset (Gholipour Ghalandari et al., 2020). There are 8158 examples the train set and 1022 examples in the test set. Each example con- tains between 1 and 10 documents, with a mean of 9.1
BM25 (Robertson al., 1994): Like other sparse retrievers, BM25 represents queries and documents as sparse vectors, where each element of a vector corresponds to term in the vocabulary. BM25 is widely used weighting scheme that extends TF-IDF (Jones, 2004) to account for document length and term-frequency saturation. We use BM25 via PyTerrier with the default settings.
Multi-XScient (Lu et al., 2020): The target summary of each example is the related works section of a scientific article, and the input documents are the abstracts of the articles this section cites. Also included is the target ar- ticle's abstract. There are 30,369 examples in the train set and 5093 examples in the test set. Each example contains between I and 20 documents, with a mean of 4..1.
Contriever (Izacard et al., 2022): Contriever is an unsupervised dense retriever that uses a bi-encoder architecture. Documents and queries are encoded independently using the same BERT model (Devlin et al., 2019), and the final embedding is obtained by mean- pooling over the hidden representations of the model's last layer. The relevance score be- tween query and document is the dot prod- uct of their embeddings. Specifically, we use contriever-msmarco 22 which has been fine- tuned on the MS MARCO dataset (Campos et al., 2016). We use Contriever via the PyTer- rier Sentence Transformers plugin (Soldaini, 2022) with the default settings.
MS² (De Young al., 2021): The target sum- mary is few sentences from biomedical sys- temic review which summarize the main find- ings. The input documents are the included studies for that review. Also included is the target reviews background section. There are 14,188 examples in the train set and 2021 ex- amples in the validation set. Each example contains between and 401 documents, with mean of 23.2.
20 hhttps://er http///nwwwkipedi..... wttp://iee....roeao////rrrr..r. org/wiki/Portal: Current_events
22 hhttps://huggingfeee co/ /acebbook/ cuntriever-mmarrc
2'tttp:: http://////ggig...e.o/o////tt//s wwt.//aaasets//cc///ee....

---

Table 6: Dataset statistics, counting whitespace tokens and punctuation. "Following (De Young et al., 2021), we take the first 25 documents input (full statistics in parentheses). "Multi-XScience" and MS each have inputs that are always provided (and never retrieved), the target articles abstract and the target reviews background section.
Number of Documents
Average Number Tokens
Dataset
Domain
Max
Mean
Total
Per document
Per summary
Multi-News
News Articles
10
2.7
154,544
788
267
WCEP-10
News Articles
10
9.1
92,560
494
33
MultiXSScience
Scientific I. Literature
20
4.1
165,546
153
125
= =t
Medical Studies
25 (401)
17 (23)
415,333
332
Cochrane
Medical Studies
25 (537)
(11)
51,208
266
Table Retrieval performance. The precision and recall & for each retriever and top-k strategy is reported. The index for each dataset is the set of all documents in the train, validation and test sets; the reference summaries are used as queries, except for MS"2, where we use provided background section. The Cochrane test set is blind, so do not have access the reference summaries use queries and therefore do not evaluate on the test set.
Train
Validation
Test
Dataset
Retriever
Retriever Type
Top-k Strategy
P@K
R@K
P@K
R@K
P@K
Multi-News
BM25
sparse
max(10)
0.22
0.83
0.22
0.82
0.22
0.82
0.64
0.74
0.64
0.74
0.64
0.74
oracle
0.75
0.75
0.75
0.75
0.75
0.75
Contriever
dense
0.21
0.80
0.21
0.79
0.21
0.80
0.59
0.69
0.59
0.69
0.59
0.70
oracle
0.69
0.69
0.69
0.69
0.69
0.69
WCEP-10
BM25
sparse
(10)
0.59
0.66
0.60
0.63
0.63
0.67
mean
0.62
0.62
0.63
0.60
0.66
0.64
oracle
0.64
0.64
0.63
0.63
0.67
0.67
Contriever
dense
max
0.66
0.60
0.64
0.67
mean
0.62
0.63
0.63
0.60
0.66
0.63
oracle
0.65
0.65
0.63
0.63
0.66
Multi-XScience
BM25
sparse
max (20)
0.41
0.06
0.40
0.06
0.40
mean (4)
0.16
0.27
0.16
0.26
0.16
0.27
oracle
0.22
0.22
0.22
0.22
0.23
Contriever
dense
max
0.06
0.38
0.06
0.38
0.06
0.38
mean
0.16
0.24
0.16
0.24
0.16
0.24
oracle
0.20
0.20
0.20
0.20
0.21
0.21
MS*2
BM25
sparse
max (25)
0.17
0.26
0.16
0.22
0.17
0.22
mean (17)
0.21
0.22
0.18
0.18
0.20
0.18
oracle
0.22
0.22
0.18
0.18
0.19
0.19
Contriever
dense
max
0.19
0.29
0.18
0.25
0.19
0.26
mean
0.23
0.24
0.21
0.21
0.23
0.21
oracle
0.24
0.24
0.21
0.21
0.22
0.22
Cochrane
BM25
sparse
max (25)
0.17
0.55
0.17
0.57
mean (9)
0.30
0.42
0.31
0.44
oracle
0.38
0.38
0.40
0.40
Contriever
dense
max
0.20
0.63
0.20
0.64
mean
0.34
0.48
0.35
0.49
oracle
0.45
0.45
0.44
0.44
Model Details
self-attention with local windowed attention and global attention mechanisms that scale linearly with input sequence length, allow- ing for efficient processing of inputs up 16K tokens. Its parameters are initialized with the pretrained parameters of BART (Lewis et al., 2020a), its positional embeddings with 16 copies of BART's 1K position embeddings. The model is fine-tuned on MDS datasets in supervised fashion.
All models are implemented in PyTorch (Paszke et al., 2019 9), and pretrained weights are obtained from the HuggingFace Transformers library (Wolf et al., 2020). Models were trained and evaluated 1 1-4 NVIDIA A100 40GB GPUs. We list details about the models in Table 5. Below, we provide detailed descriptions of all models:
LED (Beltagy et al., 2020): LED replaces full

---

Table Reported versus reproduced ROUGE- 1/2/L scores for each model-dataset pair evaluated in the main paper. We also report zero-shot performance select datasets for PRIMERA. *Fine-tuned by us.
Reported
Reproduced
Model
PRIMERA
PEGASUS
LED-base
LSG-RART-base
PRIMERA
PRGASUS
LED-base
LSG-RART-base
Dataset
News
49.9/21.1/25.9
47.5/18.7/24.9
47.1/18.9/25.2
19.3/20.3/25.4
18.2/20.1/25.4
46.3/18.8/25.1
+- geio-shio
42.0/13.6/20.8
99.7/11.911 1//199
WCEP
46.1/25.2/37.9
16.0/242/37.4
45.1/24.7/36.7
45.9/24.1/37.2
Mulri-SScienoe
31.9/7.4/18. 4/1.8.0
31/7/6, 1/17.1
- zero-ahoo
29.174.6/15.7
27 27.0/3.9/14.6
2
26.48800119.4
28. 2/9.0/95/20.9
Cachrana*
23.9906/17.66
36.9/699188.4
Table Results of the open-domain MDS experiments. We observe the following: (1) retrieval performance ranges from high (Multi-News, WCEP-10, dark blue) to low AMulti-XScienm MS²2, Cochrane), (2) when summarizers trained on these datasets are provided retrieved documents, they suffer from significant drops in performance (dark red); more severe performance drops were observed in cases where baseline summarization performance relatively high (dark green). Experiments here used sparse retriever (BM22);; similar results were observed using a dense retriever (Contriever, see Table 10). Statistically significant results are underlined (paired I-test, p 0.01).
Retrieval
Summarization
Dataset
Model
Top-k Strategy
P@K
RØK
ROUGE-Avg F1
à ROUGE-Avg
BERTScore
BERTScore F1
MultiNNws
PRIMERA
max (10)
0.22
0.82
31.66
-7.39
31.78
-10.3
(3)
0.64
0.74
.2.82
.408
oracle
0.75
0.75
-1.61
-2.36
PEGASUS
max
31.23
8.49
29.88
10.87
-2.08
oracle
-1.15
-1.50
LSG-BART-base
max
30.05
-644
26:57
.2.17
2.35
oracle
-0.0
-0.99
GPT.3.5-turbo
23.86
.2.66
21.68
.3.92
mian
-1.59
-2.76
oracle
.0.7
1103
WCEP-10
PRIMERA
48.26
-0.76
mean (9)
0.66
0.64
0.90
-0688
oracle
0.67
0.67
40.53
-0.32
LSG.BART.
.115
meun
-1.19
-0.54
oracle
0.88
-0.54
5-trbo
max
26.36
-0.22
32.72
-0.25
mean
-0.06
-0.33
+0.10
Multi XScience
PRIMERA
max (20)
0.06
0.40
18.31
.0.57
10.57
.182
(4)
0.16
0.27
-0.25
-1.27
.0.97
MS"2
LED--bscc
max (25)
0.16
0.22
19.66
-0.14
22.74
-0.47
(17)
0.18
0.18
-0.10
-0.13
oracle
0.18
0.18
-0.01
-0.11
Cachranc
BDD-hsec
max (2S)
0.17
0.57
17.39
-0.28
23.12
-2.11
(9)
+0.34
-0.32
oracle
0.40
0.40
+0.10
+0.00
PEGASUS (Zhang et al., 2020a): PEGASUS is pretrained using a novel Gap Sentences Generation (GSG) objective, where whole sen- tences from each document are masked, and concatenated to form a pseudo-summary The model is then fine-tuned on MDS datasets in supervised fashion.
MDS datasets in supervised fashion used in zero-shot setting.
LSG-BART (Condevaux and Harispe, 2022): Like LED, LSG-BART replaces full self- attention with sparsified version, dubbed Local-Sparse-Globhe (LSG) attention, al- low for efficient processing of long inputs. It is initialized with the pretrained parameters of BART and ine-tuned on MDS datasets in a supervised fashion.
PRIMERA (Xiao al., 2022): Extends the GSG objective with novel masking strategy explicitly designed for multi-document inputs and pre-trains on a corpus of multi-document examples. The model is then fine-tuned on

---

Table 10: Results of the open-domain MDS experiments using a dense retriever (Contriever). Difference between a summarizers performance the fround-truth input documents and performance when the documents retrieved is shown. Statistically significant results are underlined (paired t-test, 0.01).
Retrieval
Summarization
Daraset
Model
Top-k Strategy
P@K
R
ROUGE-Avg FI
A ROUGE-Avg FI
BERT
A BERTScore
Multi-Nows
PRIMERA
max (10)
0.21
0.80
31.66
-2.77
31.78
-10.47
0.70
-3.31
oracle
0.69
0.69
-2.20
-3.07
PEGASUS
max
31.23
-8.69
29.88
-10.88
mean
-2.65
-3.45
oracle
-1.76
-2.28
LSG-BART-hase
max
30.05
-6.70
26.57
.8.15
mean
.2.26
:2.69
WCEP-10
PRIMERA
(10)
0.63
0.67
35,50
+0.10
48.26
+0.90
mean (9)
0.66
0.63
0.14
+0.68
oracle
0.66
+0.29
+0.86
LSG-BART-ban
35.76
0996
+0.10
oracle
-0.15
+0.66
Mulii-XScienne
PRIMERA
max (20)
0.06
0.38
18.31
-0.55
10.57
-0.96
-0.11
oracle
0.21
0.21
-0.28
-0.37
MS
LED-base
max (25)
0.18
0.25
19.66
-0.43
22.74
-0.70
miaa (17)
-0.37
-0.64
0.21
-0.32
-0.38
Cochnine
1 LDD-aaee
max (25)
0.20
0.64
17.39
-0.94
23.12
-2.77
mean (9)
0.35
0.49
-0.37
-0.93
C.1 Reproducing Reported Scores
E Extended Results from: section 5
Before experimentation, we attempt to reproduce the reported scores of each MDS model. The re- sults provided in Table 8. In general, reproduce the reported ROUGE scores (and some- times even improve upon them); however, in a few cases, there are differences as large as ~ ROUGE, with the largest differences being observed for PRIMERA, particularly in the zero-shot setting.
In $5, presented the results from our open- domain MDS experiments for the sparse retriever (BM25) and max top-k strategy only. In Table we present the complete set of results for the sparse retriever. The dense retriever (Contriever) results were comparable and exhibited the same general trends; they are presented in Table 10.
can
E.1 Summarization Baselines
In Table 3, present scores of heuristic baselines. Detailed descriptions of each baseline follow:
Evaluation Details
The evaluation metrics, ROUGE and BERTScore, called from the HuggingFace Evaluate library23. Before metrics are calculated, all text is lightly pre-processed by removing leading and trailing whitespace, newline characters and tabs. For ROUGE, we use the default settings besides "se_sttemmer=True."
Random (length-matched) summary: For each example, take the summary to be the reference summary of another example from the same dataset that is the same (or similar) length as the examples reference summary. This provides us with coherent (but likely ir- relevant) summaries of approximately the cor- rect length from the same domain.
BERTScore has many parameters which affect the final score; for repro- ducibility, hasheode is produced. Our hashcode is:
All lead: For each example, take the sum- mary to be the concatenation of the first sen- tence from each input document. This moti- vated by the notion of lead bias, namely that many summarization datasets (particularly those comprised news articles), sentences at the beginning of a document are more likely
no-idf_versee vo-idf_vversion003..1 (huggtttttttatrasss4... (atplug_traass4.2222 fastrrrrsseleedl_sttttooeeeee
2 hoatts://hugging. acee. co/ sosspaces/
wvaluate-meticc/ouu

---

addition
deletion
replacement
Multi Mull--Nows
WCEP-10
MultiXSSrionce
MS2
Cochrane
400-
30
750
300
80
Absolute Error Count (100s)
20
200
200
40
10
250
100
400-
30
750
80
20
200
200
40
10
260
100
maz
memn
oracio
max
oracte
max
mean
oracia
momm
orade
mat
maman
oracle
Figure 6: Absolute error counts for different retrieval systems (sparse and dense) and top-k selection strategies (max, mean, oracle). For each example in given dataset, retrieved document that does not exist in the ground-truth input document set is counted as addition and a ground-truth document that was not retrieved as a deletion. Instances one addition and one deletion are counted as a replacement.
to contain information that appears in the ref- erence summary (Nenkova et al., 2011; Hong and Nenkova, 2014; Xing et al., 2021).
each time both one addition and one deletion occur. In general, the sparse and dense retrievers make highly comparable number of errors of each type. Unsurprisingly, the oracle top-k strategy tends produce the lowest number of errors in total (pri- marily replacements), followed by mean (a mix all error types) and max (primarily additions).
Oracle document: For each example, take the summary be the input document with the highest ROUGE-1 Fl score with that ex- ample's reference summary. This provides us with relevant (but likely incomplete) sum- maries with high token overlap. A high score may indicate that dataset is less "multi" (Wolhandler et al., 2022).
Training in the Open-domain Setting
In $6, we presented the results of our experi- ments fine-tuning summarizers in the open-domain setting. We fine-tuned and evaluated the best- performing model from each dataset: PRIMERA for Multi-Nest and Base for WCEP-10 10 and LED-Base for MS and Cochrane. Each model was fine-tuned for an ad- ditional three epochs (we found additional epochs made little difference) using the original training hyperparameters. All models were fine-tuned with the AdamW optimizer (Loshchilov and Hutter, 2019) in PyTorch via the HuggingFace Transform- library. The learning rate was linearly increased for the first 10% of training steps and linearly de- cayed zero afterward. Documents were retrieved using the dense retriever (contriever) and the mean top-k strategy.
Oracle lead: The first sentence of the oracle document (see above). For MS Cochrane, this the title of the oracle document.
-G--BART-
Background/abstract: Applies only to MS*2 and Multi-XSeience For each example, take the summary to be the additional input from MS*2 (target reviews background section) and Multi-XScience (target articles abstract).
E.2 Document Retrieval Error Analysis
In Figure 6, we tally the total number of errors made by the sparse (BM25) and dense (Contriever) retrievers on each dataset. For each example, we count addition (i.e. erroneous inclusion) each time document not in the ground-truth input doc- ument retrieved, deletion (i.e. erroncous exclusion) each time a ground-truth document not retrieved and a replacement (i.e. an erroncous swap of a relevant document for an irrelevant one)
G Extended Results from:
section
In $7, we presented results from our experiments simulating document retrieval errors for two model- dataset pairs that exemplified the main trends in the

---

Original Document
Backtranslated Document
Relocation of endangered aninals carries Fisks but loss of half of then is highly unusual. Eight out o died
Movement of endangered animals carries Pisks, but he loss of half of then is very unusual. Eight out critically endangered black
critically endangered black thinos fter being noved to reserve in southern Kenya, idlife officials have revealedj in what one consery ationist described as complete disaster". Prelim inary Investigations pointed to salt polsoning as their
ed after being noved to reserve in southern Keny a, wildlife officials have revealed what conse described
complete disaster. Pr eliminary studies pointed to salt poisoning when th tried to adapt new home, the Kenyan Ministry of Tourism and Anin al Health said in statenent. It suspended nov ement of other rhinoceros and said that the survivo rs were being closely nonitored.
rvation
thinos tried adapt to saltier new home, the Kenyan Ministry of Tourisn and wild1 said statenent. It suspended noving other rhinos and said the surviving ones being closely monitored.
Figure 7: Graphical depiction of the backtranslation perturbation. A truncated document from the Multi-News (Fabbri et al., 2019) dataset is shown, and changes after backtranslation are highlighted.
Table 11: Results of the sorting perturbation experiments. Difference between summarizers performance the ground-truth input documents and performance when the documents were perturbed shown. Statistically significant results are underlined (paired t-test, p 0.01).
A ROUGE-Avg F1
A BERTScore FI
Dataset
Model
ROUGE-Avg
BERTScore
Random
Oracle
Random
Oracle
Multi-News
PRIMERA
31.66
31.78
+0.02
+0,02
PEGASUS
31.23
29.88
-0.05
+0.04
+0.16
WCEP-10
PRIMERA
35.50
48.26
-0.66
+0.11
+0.57
LSG-BART-buse
35.76
48.17
-0.98
+0.38
Multi-XScience
PRIMERA
18.31
10.57
+0.07
-0.04
+0.13
40.03
MS2
LED-base
19.66
22.74
+0.09
+0.24
+0.00
-0.01
Cochrane
LFD-bbsc
17.39
23.12
-0.41
-0.32
-0.42
+0.06
rest of the results. We present complete results for all model-dataset pairs in figures 8-15.
G.2
Sorting Perturbation Results
In Table 11, we present the tabulated results from the sorting perturbation experiments (see $7 for more details the experimental procedure and $7.2 for analysis of the results).
G.1 Backtranslation
In the main paper, backtranslation create token-level perturbations. The procedure involves selecting one or more documents from the input set and translating them to another language and back again, often creating small, token-level changes like paraphrasing and synonym substitution (this is sometimes called "round-trip translation". or RTT). We choose to translate documents to and from Danish, there exists freely available and nigh-performing EN-DA and DA-EN machine translation (MT) models. In particular, we use the models provided by the Language Technology Re- search Group at the University of Helsinki (Tiede- mann and Thottingal, 2020). We implement back- translation using the nlpaug library (Ma, 2019). In Figure 7, provide example of a backtrans- lated document demonstrating synonym substitu- tion (e.g. "highly" ""rrbl""""vvvr"""""""""" "veery"), paraphrasing (e.g. "said the surviving ones" "said that the survivors") and grammatical errors (c.g. "14 critically endan- gered black rhinoceros has died").
H Human Evaluation
To make human evaluation feasible, we chose single model-dataset pair with high summarization and retrieval performance: PRIMERA and Multi- News, 25 To conduct the evaluation, we randomly sampled 50 examples from the test set and pre- sented three human annotators with the generated summaries for these examples from the "baseline" model (no retrieval) and the open-domain model (with retrieval). Annotators were presented the model summaries in randomized order as "model summary A" and "model summary B" and in- structed to select which summary ("A", "B" "Neither") they preferred for each of two facets,
25 take results from the highest performing retriever (sparse) and non-oracle top-k strategy (mean) 26The three annotators are subset of the authors who did not interact with model outputs prior to annotation

---

Table 12: Examples of degradation of summarization performance in the open-domain setting. Shown is the output of the summarizer in the open-domain setting (truncated) and human annotator comments (paraphrased). The plausible reason for degradation is based on a manual analysis of summarizer inputs and outputs.
Open-domain model summary
Annotator comments
Plausible reason for degradation
For the second year in a row, the Academy of Motion Picture Arts and Sciences did not nominate any black actors any of the 20 slots in the four acting categories [...] The Hol- lywood Reporter calls it a "whiteout," and the president the African Ameri- can Film Critics Association says [...I There needs
The reference summary indicates that meaningful progress is heing made in improving diversity amongst members of the Academy of Motion Picture Arts and Sciences. The baseline model gets this correct, but the framing of the open-domain model summary is that little to no progress has been made.
The gold document set contains 2 documents, each about efforts to im- prove diversity among members of the Academy of Motion Picture Arts and Sciences, both written after the 2016 Oscars. The retrieved document set contains documents, from the gold document set, and 2 written before the 2016 Oscars, both criticizing the fact that no black actors were nominated for any acting category.
changes across the board. I...1 I The Academy, which is 94% white and 77% male, has been try- ing to diversify its membership I...I
A GoFundMe campaign has raised more than $400,000 for man who lost his wife in childbirth [...] Christian musician Nathan Johnson gave birth to his first child [...] (his wife started having complications later in the morn- ing f...] 1 Johnson is surrounded by friends and family who are helping him deal with the loss of his wife [...)
The reference summary is about a Go- FundMe campaign created on behalf Dawn Wells (hest known for play- ing Mary Ann Summers on Gilligan's 's Island) who is experiencing financial hardship. The baseline model gets this correct, but the open-domain model's s summary is about a completely differ- ent GoFundMe campaign.
The gold document set contains 2 doc- uments, both about GoFundMe cam- paign for Dawn Wells. The retrieved document set contains additional doe- ument, the story of a entirely different GoFundMe campaign about a musi- cian who lost his wife due to compli- cations during childbirth.
A British woman is "very lucky" to be alive after falling from a cruise ship 1..-I The cruise line says in a statement that the woman intentionally jumped overboard ]...] "the ship and charter compuny teams are providing support to the family and all impacted guests during this difficuls time," [...]
The summary produced by the open- domain model contains an insinua- tion of possible suicide (i.c. that the woman jumped intentionally) that is not present in the reference summary or baseline model summary.
The retrieved document set contains one additional document compared to the gold document set, ahout a differ- ent woman who did intentionally jump overhoard while on a cruise.
Prime Minister David Cameron suid he will step down in two days in favor of Theresa May [...] who will become Britain's is second female leader. [...] the will have the task of leading a divided country out of the EU L... I
The reference summary indicates that May and Andrea Leadsom are the final two candidates for Prime Minister of the UK, which is reflected in the base- line model summary. The summary produced by the open-domain model claims both that May is the winner and that the winner is yet be announced.
The retrieved document set contains one additional document compared to the gold set, about the surprise with- drawal of Leadsom from the race.
The winner will be announced Sept. 9 and will replace David Cameron I
coverage and informativeness, 27 relative to the pro- vided, human-written target summary R:
(IAA), are presented in Table 4. Human annotators have a statistically significant preference for the baseline model along both facets, with fair inter- annotator agreement (N > 0.21, Landis and Koch, 1977), providing further evidence for the degrada- tion of summarization performance in the open- domain setting observed throughout this work. In Table 12, we provide examples of summaries pro- duced in the open-domain setting alongside (para- phrased) human annotator comments noting issues with the summary. Based on a manual analysis of the inputs and outputs of the summarizer, we also provide plausible reasons for this observed degra- dation in summarization quality as it relates to the retrieved versus gold document sets.
Coverage (Grusky et al., 2018): How many semantic content units from the reference sum- mary are covered by the model summary.
Informativeness (Nenkova and Passonneau, 2004): How well does the model summary capture the key ideas of the reference sum- mary.
The results from binomial 28 on the human annotations, as well as inter-annotator agreement
"There are many facets for which human evaluation of summarization could be conducted: we choose coverage and informativeness as rough proxies for recall and precision Whttps:/ / //en.w/wwwwwwe wikipedia. a. .rgg/wkki /iikoiii//niallll tesst

---

additon
backrrnnsation
addiaon dupication deleton bacttrandation
random
orade
random
oracie
AROUGE-AVV F1
AROUGE
-10
-16
ABERTScore F1
ABERTScCor
-20
20
30
0.20
20-40
40-60
60-80
0.20
20-40
40-00
60-90
80
% of hppu documents perturbed
% of input documents perturbed
Figure 8: Results the perturbation experiments on the Multi-News set using PRIMERA. Mean change in summarization performance plotted against percent of perturbed input documents. 68% confidence intervals (CI) plotted error bands.
Figure 10: Results of the perturbation experiments on the Multi-News test using LSG-BART-bard Mean change in summarization performance plotted against percent perturbed input documents. 68% confidence intervals (CI) plotted emor bands.
-e- addiion - duplication - deledon --- replacement - bac Mrensinniooonn
-en additon -ee duplication - delatin replccement backtaasaation
random
oradle
random
oracle
AROUGE-NN F1
&ROUGE-AVO
-10
20
BERTTScrr F1
BERTTScrre
-00
-20
-30
-30
440
0:20
20-40
#0.60
60-80
80-100
0.20
20:40
40460
80.80
80 1gd
input documents perturbed
inpul documents perturbed
Figure 9: Results of the perturbation experiments the Multi-News test set using PEGASUS. Mean change in summarization performance plotted against percent perturbed input documents. 68% confidence intervals (CI) are plotted as error bands.
Figure II: Results of the perturbation experiments the WCEP-10 test set using PRIMERA. Mean change summarization performance plotted against percent of perturbed input documents. 68% confidence intervals (CI) are plotted as error bands.

---

additon
backrrnddation
addiaon dupication
bacttrandation
rendom
orade
random
oracie
BROUGE-AVS F1
0.5
-10
&ROUGE-AID
0.0
-0.5
-30
ABERTS F1
ABERTScore
0
-1
-40
0-20
20-40
40-60
60-80
80 100
0.20
20-40
40-00
60-80
0-100
% of hppt documents perturbed
% of input documents perturbed
Figure 12: Results of the perturbation experiments on the WCEP-10 test using LSG-BART-bar Mean change in summarization performance plotted against percent of perturbed input documents. 68% confidence intervals (CI) are plotted as error bands.
Figure 14: Results of the perturbation experiments on
the MS*2 validation set using LED-base. Mean change in summarization performance plotted against percent of perturbed input documents. 68% confidence intervals (CI) are plotted as error bands.
--- addiion - duplication -a-- deledon replacement - backnansleion
-en additon -ee duplication - delatinnon - replecement backtmanlationn
random
oradle
random
oracle
AROUGE-Avy F1
&ROUGE-AVI
-2
-10
ABERTScore F1
ABERTScrre
10
4
-20
0:20
20.0
#0ve0
60-80
10-100
0.20
20.40
4040
80.80
80 100
input documents perturbed
inpul documents perturbed
Figure 13: Results of the perturbation experiments the Multi- XScience test set using PRIMERA. Mean change in summarization performance plotted against percent of perturbed input documents. 68% confidence intervals (CI) are plotted as error bands.
Figure 15: Results of the perturbation experiments the Cochrane validation set using LED-base. Mean change in summarization performance plotted against percent of perturbed input documents. 68% confidence intervals (CI) are plotted as error bands.


Reviews:
[{"review_id": "9L0hI6itY6", "reviewer": "Reviewer_7Smv", "summary": "", "questions": "line 268: \"lower-bound\" -should it be \"higher bound\" or am I missing something?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for commenting that our paper is well written, points out a gap between current MDS research and real-world applications, and that our experimentation is extensive. We address any questions or concerns below:\n\n> The authors claim that the data is more difficult than other dataset as it contains many irrelevant documents that could hurt summarization. On the other hand, in a real-world setup there might be many \"difficult\" documents, as irrelevant documents that seem to be relevant as they are similar to relevant documents. Adding analysis quantifying this phenomenon in the new dataset may add additional insights.\n\nWe agree; we carefully designed experiments to analyze this phenomenon in __Section 7__. In the _oracle_ setting of the _addition_ perturbation, irrelevant documents were selected based on their similarity to the target reference summary (see __Section 7.1__). We believe this is exactly the experiment you are suggesting.\n\nIn short, we found that the performance degradation tends to be _similar_ or _slightly greater_ in the random setting. A possible explanation is that random additions introduce more _topically_ and _semantically_ irrelevant content than oracle additions and that the automatic metrics may be particularly sensitive to this content when it is included in the model’s output summary (e.g. names, places and events not present in the ground-truth summary). We have updated the main text to note this relationship and include the provided rationale.\n\n> In fact, the paper contribution focuses on pointing out the need for a new setup. In practice, the suggested dataset can be used as an upper bound only. A significant contribution could be by releasing a \"real\" dataset. But it might be okay to leave it for future work.\n\nWe agree that a new dataset for open-domain MDS would be an exciting contribution, however, our contributions are orthogonal. __By identifying the problem, confirming that existing summarizers are sensitive to retrieval, and providing a thorough set of guidelines, we have enabled future researchers to collect, curate and create a “real” dataset.__\n\n> line 268: \"lower-bound\" -should it be \"higher bound\" or am I missing something?\n\nWe consider our results a _lower_-bound on the expected performance _degradation_, not an _upper_ bound on performance. Several aspects of our experimental design are intentionally idealized, and performance degradation of summarization in the true setting is likely even worse, hence the _lower_ bound. Please see __Section 4.4__ where this is explained in detail.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and the flow is clear.\n- The paper points out a real lack in the current MDS setup which does not align with real-world applications.\n- The authors conducted extensive experiments, deriving several insights.\n- The authors added a thorough error analysis.", "reasons_to_reject": "- The authors claim that the data is more difficult than other dataset as it contains many irrelevant documents that could hurt summarization. On the other hand, in a real-world setup there might be many \"difficult\" documents, as irrelevant documents that seem to be relevant as they are similar to relevant documents. Adding analysis quantifying this phenomenon in the new dataset may add additional insights. \n- In fact, the paper contribution focuses on pointing out the need for a new setup. In practice, the suggested dataset can be used as an upper bound only. A significant contribution could be by releasing a \"real\" dataset. But it might be okay to leave it for future work.", "questions_to_authors": "line 268: \"lower-bound\" -should it be \"higher bound\" or am I missing something?", "paper_topic_and_main_contributions": "The paper establish the new task of open domain MDS. It creates a new dataset, shows that current MDS models perform lower on this task, while fine-tuning improves the results. The paper also makes a nice error analysis.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rP0QgUhBAa", "reviewer": "Reviewer_GCwU", "summary": "", "questions": "Q_A: Please, provide more details about the input size. See the last reject point.\nQ_B: Is it possible to combine the ranking of both retrieval models (sparse and dense)? Since they provide different rankings, as stated by the authors, the combination of rankings could lead to better retrieval performance.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing out the depth of our results and analysis and that we produce helpful guidelines for future work on open-domain MDS.\n\n> The paper has nine pages for the main contribution and ten for appendices!\n\nThe paper is 8 pages in length (the maximum page limit for EMNLP 2023).\n\n> This means two things: 1) the paper is hard to read because, at some points, you need to jump from contribution to appendix; 2) the contribution is more suitable as a journal paper than a conference one.\n> The paper is generally well written but hard to follow since many important content is provided in appendices.\n\nWe appreciate the reviewer's concern, however we were careful to place the most important content for understanding our contributions in the main paper. The appendix consists of experimental details and supporting experiments that are important but not critical to understanding our key contributions. __During a read, there should be little to no jumping between the main paper and the appendix__. We are open to moving some content to the main paper if you have specific suggestions. We will also include as much relevant content in the 9-page allotment provided to camera-ready papers if this manuscript were to be accepted.\n\n> The authors report the main drawback in the conclusions: \"Curating high-quality MDS datasets annotated with queries will be necessary to enable further progress in the open-domain setting.\". I think that manually created queries are fundamental for providing a novel and valuable contribution. Using the human-written reference summary as a query biases the retrieval performance since the IR model retrieves documents in which words in the summary occur in the retrieved document, mainly if a BM25 is used as the ranking model.\n\nUsing the human-written reference summaries as queries was deliberate (__Section 4.4__). Our study aimed to place a lower bound on the degradation of existing summarizers in the open-domain setting. As you say, using reference summaries as queries makes the retrieval problem easier; that we still find large reductions in summarization performance suggests that the degradation would be _worse_ given manually crafted queries. Our perturbation experiments (__Section 7__) generalize our findings, by not relying on specific queries or retrievers and by covering all major types of retrieval errors from 0 to 100% of input documents perturbed.\n\nWe also agree that creating a new dataset for MDS with carefully crafted examples and queries would be a major contribution. Our contributions are orthogonal: __we are the first to identify the need for such a dataset and enable future researchers who are better positioned to build it because of our work__.\n\n> Q_A: Please, provide more details about the input size.\n\nThe maximum input size of all models is listed in __Table 7__ of the appendix (referenced from __Section 4.3__, which also explains the truncation procedure). For convenience, we relist them below:\n\n- LED: 16,384\n- PEGASUS: 1024\n- PRIMERA/LSG-BART: 4096\n\nDataset sizes (both in terms of # of docs and # of tokens) are listed in __Table 4__ of the appendix.\n\n> ​​Since encoder-decoder architectures accept input of a limited size, the authors should provide a deep evaluation of how this affects the performance. Truncing the document to the first tokens may not be a good choice.\n\nOur truncation strategy is not unique to open-domain MDS, and the same or similar strategy is used by all baselines we cite.\n\nWe note that for 2/5 datasets (MS^2 and Cochrane), no truncation happens at all, as the maximum input size is less than the models’ maximum input size. For a different 2/5 datasets (Multi-News and Multi-XScience), there is no truncation in the _average_ case. For the final dataset (WCEP-10), only a small amount of total tokens are truncated, ~30%, distributed across the ~10 input documents. This can be quickly confirmed by referencing the dataset sizes (__Table 4__) and then models’ max input sizes (__Table 7__). __Given this, we have no reason to suspect that our results would change meaningfully with different truncation strategies__.\n\n> You can select the portion of the document that best fits the query.\n\nThis is the difference between _document_-level and _span_-level retrieval. We consider the latter to be beyond the scope of this study, but is certainly an interesting direction. We have already noted this in the __Limitations__ section: “Specialized retrievers may lead to better performance…”\n\n> Is it possible to combine the ranking of both retrieval models (sparse and dense)? Since they provide different rankings, as stated by the authors, the combination of rankings could lead to better retrieval performance.\n\nYes, a common strategy is to retrieve a large number of documents with a sparse retriever (e.g. BM25) and re-rank those with a dense retriever (e.g. Contriever). Our preliminary results showed marginal gains in retrieval performance (+0.05 P/R@K at best) with this pipeline. __Because of the marginal gains and because it complicates the relationship between retriever type and impact on summarization performance, we elected not to include these experiments in the paper__. We have updated the text in the __Limitations__ section (“Specialized retrievers may lead to better performance…”) to note this.\n\n---\n\n__RE the soundness score__: we are surprised to see a low soundness score. We were extensive and meticulous in our experimental design; something the other reviewers noted. Alongside our main experiments evaluating and training models in the open-domain setting, we included an extensive set of perturbations designed to mimic all main types of retrieval errors, from 0-100% of input documents modified. Finally, the appendix contains supplementary experiments that support our main claims. We were also careful to ensure everything was reproducible by including notebooks demonstrating how to run each experiment from the paper (this can be confirmed by checking the anonymous GitHub repository we linked to in our submission). We hope that the reviewer might consider raising the soundness score in light of this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new dataset for text summarization\n- Deep results analysis\n- Helpful guidelines for further investigation", "reasons_to_reject": "- A low novelty for an EMNLP paper\n- The paper has nine pages for the main contribution and ten for appendices! This means two things: 1) the paper is hard to read because, at some points, you need to jump from contribution to appendix; 2) the contribution is more suitable as a journal paper than a conference one.\n- The authors report the main drawback in the conclusions: \"Curating high-quality MDS datasets annotated with queries will be necessary to enable further progress in the open-domain setting.\". I think that manually created queries are fundamental for providing a novel and valuable contribution. Using the human-written reference summary as a query biases the retrieval performance since the IR model retrieves documents in which words in the summary occur in the retrieved document, mainly if a BM25 is used as the ranking model.\n- Since encoder-decoder architectures accept input of a limited size, the authors should provide a deep evaluation of how this affects the performance. Truncing the document to the first tokens may not be a good choice. You can select the portion of the document that best fits the query.", "questions_to_authors": "Q_A: Please, provide more details about the input size. See the last reject point.\nQ_B: Is it possible to combine the ranking of both retrieval models (sparse and dense)? Since they provide different rankings, as stated by the authors, the combination of rankings could lead to better retrieval performance.", "paper_topic_and_main_contributions": "The paper proposes a new dataset for open-domain multi-document summarization and reports a preliminary evaluation of several approaches.\nIn particular, the authors focus on the impact of retrieval performance on the quality of the final summarization.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is generally well written but hard to follow since many important content is provided in appendices.\nConsidering the type of contribution and the content length, in case of rejection, I suggest revising your paper and sending it to a journal."}}, {"review_id": "LnsIwnuwaZ", "reviewer": "Reviewer_ABNP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their succinct and cogent summary of our contributions and for pointing out the meticulousness of our experimental design.\n\n> It is debatable whether we should put the blame on MDSs for their not being able to cope with topically diverse documents, as they are intended to work under the assumption that documents that come their way are semantically aligned. There is nothing wrong with their breaking down where the assumption does not hold.\n\nThis is a fair point; we tried to be careful not to suggest this, by noting that the behaviour of existing summarizers in the open-domain setting is _unknown_ (lines 059-061 and elsewhere), motivating our exploratory study. We also fine-tuned the summarizers in this new setting, acknowledging, both implicitly and explicitly, that they should be adapted to the open-domain setting before use (__Section 6__).\n\nHowever, we understand how the current framing of the paper might muddy this message. We have updated the wording throughout the paper (particularly in the introduction and conclusion) to avoid this ambiguity. The tweaked introduction highlights that our motivation was to study the behaviour of these models in the novel open-domain setting using a _reasonable_ (and even idealized) setup to identify their weaknesses and enable future research on the open-domain setting.\n\n> There is a chance that the authors are asking a question that people in MDS are not even interested in asking.\n\nWe hope that the MDS community and broader community of researchers working in summarization will be interested in a task that is more technically challenging (due to the addition of retrieval) and potentially more helpful for an end user with an information need but not necessarily a set of topic-related input documents. Open-domain MDS is an exciting area for future research that can coexist alongside research on traditional MDS tasks without retrieval, much like how open-domain and closed-book QA coexist as research areas.\n\n> footnote 19: the period is missing.\n\nFixed, thanks.\n\n---\n\n__RE the reproducibility score__: we are surprised to see a middling reproducibility score. We were extra careful to provide scripts to reproduce all experiments, release all data and model artifacts that we created as part of our study, and we made all the raw experimental results available. We also have notebooks that demonstrate how to reproduce each experiment from the paper. This can be confirmed by checking the anonymous GitHub repository we linked to in our submission. We hope that the reviewer might consider raising the reproducibility score in light of this.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I will give them some credit for bringing in an interesting perspective for MDS; I like the authors’ breaking down of performance from different angles and discussing how it is influenced by what factors (Figure 4); I also like the way they designed experiments; it is quite meticulous.", "reasons_to_reject": "It is debatable whether we should put the blame on MDSs for their not being able to cope with topically diverse documents, as they are intended to work under the assumption that documents that come their way are semantically aligned. There is nothing wrong with their breaking down where the assumption does not hold.  There is a chance that the authors are asking a question that people in MDS are not even interested in asking.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The work is prompted by a concern that multi-document summarization (MDS) as has been practiced in the literature may not scale well to a real word scenario where one collects documents via information retrieval and conducts summarization. To examine what effect it may have on performance of MDS, the authors simulate the scenario, where summarizers have to work with groups of documents, not curated by humans but collected via conventional retrieval methods. What they found out from the exercise was that MDSs tend to break in the absence of topical coherence among documents.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "footnote 19: the period is missing."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates how multi-document summarization (MDS) models behave when the set of input documents is provided by an IR model. It formalizes this task as open-domain MDS, bootstraps it using existing datasets and summarization/retrieval models, and conducts extensive experiments. Reviewers agree that the paper makes meaningful contributions (recasting MDS in a more "realistic scenario", thorough experiments and analysis, opening potentially enabling further work in MDS) and I believe some of their concerns were addressed in the rebuttal period.
Decision: Accept-Findings