A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization
Authors: Chenhui Shen
Liying Cheng
Xuan-Phi Nguyen
Yang You
Lidong Bing
Venue: EMNLP 2023 Findings (2023)
Keywords: multi-document summarization
abstractive summarization
pre-trained language models
hierarchical

Abstract: Pre-trained language models (PLMs) have achieved outstanding achievements in abstractive single-document summarization (SDS). However, such benefits may not fully extend to multi-document summarization (MDS), where the handling of cross-document information is more complex. Previous works either design new MDS architectures or apply PLMs bluntly with concatenated source documents as a reformulated SDS task. While the former does not utilize previous pre-training efforts and may not generalize well across different domains, the latter may not sufficiently attend to the intricate cross-document relationships unique to MDS tasks. Instead, we enforce hierarchy on both the encoder and decoder to better utilize a PLM to facilitate multi-document interactions for the MDS task. Across 10 MDS benchmarks from various domains, our method outperforms or is competitive with the previous best models, including those with additional MDS pre-training or with more parameters. It outperforms its corresponding PLM backbone by up to 3 Rouge-L and is favored by humans.

A Hierarchical Encoding-Decoding Scheme for Abstractive Multi-document Summarization
Chenhui Shen" 1.2 Liying Cheng 1.3 Xuan-Phi Nguyen Yang You Lidong Bing
'DMMO Academy, Alibaba Group, Singapore NNational University of Singapore HHupan Lab, 310023, Hangzhou, China
[chenhui.shen) liying. - cheng, x.nguy,n, 1 J.bing] wealibaba-incccom youyêcomp. mus.edu sg
Abstract
repetitions and larger amounts of trivial informa- tion across documents (Zhou et al., 2021; Cui and Hu, 2021; Liu and Lapata, 2019a; Lebanoff et al., 2018; Li et al., 2017a, 2018; Yasunaga et al., 2017; Ganesan et al.. 2010).
Pre-trained language models (PLMs) have achieved outstanding achievements in abstrac- tive single-document summarization (SDS). However, such benefits may not fully extend to multi-document summarization (MDS), where the handling of cross- cross-document information is more complex. Previous works either design MDS architectures apply PLMs bluntly with concatenated source documents as refor- mulated SDS task. While the former does not utilize previous pre-training efforts and may not generalize well across different domains, the latter may not sufficiently attend the intricate cross-document relationships unique to MDS tasks. Instead, we enforce hierarchy on both the encoder and decoder to better utilize PLM to facilitate multi-locument interactions for the MDS task. Across 10 MDS benchmarks from various domains, method outperforms is competitive with the previous best models, in- cluding those with additional MDS pre-training with more parameters. It outperforms its cor- responding PLM backbone by up to 3 ROUGE- I. and is favored by humans.
While MDS could be superficially converted to SDS by concatenating multiple documents into a pseudo-single document (Fabbri al., 2019; Liu et al., 2018; Lebanoff et al., 2018), the irregular and complex multi-document information remains. As result, it continues to pose challenges due its large difference from the mostly coherent PL.M pre- training data continuous text segments (Baumel et al., 2018). On the other hand, many works design specialized MDS architectures (Xiao et al., 2022; Liu et al., 2021; Liu and Lapata, 201 9a; Fabbri et al., 2019). However, they require large super- vised MDS datasets the scale of 100K training data (Liu et al., 2021; Liu and Lapata, 2019a; Liu et al., 2018; Fabbri et al., 2019) and may not gen- eralize to unseen domains. Neither do they utilize the capabilities of large PLMs (Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020) that were pre-trained on coherent data.
1 Introduction
Given that MDS data are scarce and costly to obtain (Lebanoff et al., 2018), we propose sim- ple hierarchical encoding-decodiing scheme to ef- fectively fine-tune PLM smaller-scale MDS datasets ranging from about 2K to 45K training samples. We aim better adapt the PLM's knowl- edge gained during pre-training (which is more use- ful to SDS due to a smaller data shift) for MDS fine- tuning in both the encoder and the decoder. In the encoder, we preserve the original attention mecha- nism within the same document while re- purposing "<S> document-level representation for high- level interactions. This approach ensures that intra- document tokens can be well-represented using the pre-training knowledge, while the document will still interact with one other on a higher level. In the cross cros-attenntion layers of the decoder, we impose
Multi-document summarization (MDS) was first proposed by Barzilay et al. (1999), framed as a task of producing a single summary using set of related documents, and has been extensively stud- ied (Xiao et al., 2022; Song et al., 2022; Tu et al., 2022; Liu et al., 2021; Li et al., 2020: Liu and Lap- ata, 2019a; Fabbri et al., 2019). MDS is inherently much more complex than single-document sum- marization (SDS). Specifically, unlike SDS which requires the extraction of crucial information in single article, MDS requires handling not only the contradictions from multiple sources but also the
CChenhi is under the Joint PhD Program between A1- ibaba and National University of Singapore.
Correspouding author.
Our code and data are fully released at https: thg.. com/DAMO-NO -LPO---LLP--- ww/Hic/rHcceDe

---

document-level importance scaling on the attention weights, which allows each document to influence the output tokens differently but more appropri- ately. As a result, our method can still make use of the general language modeling capabilities of PLMs, while learning to handle the complex cross- document information during fine-tuning.
corpus (e.g., NewSHead (Gu et al., 2020) is much smaller in scale than the pre-training corpora, and re-training PLMs with much smaller-sized MDS corpus may lead to catastrophic forgetting (French, 1999) of the pre-trained knowledge.
PLMs for MDS PLMs (Lewis al., 2020; Raffel et al., 2020; Zhang et al., 2020) have achieved state-of-the-ari performances for SDS, Some works have directly used these PLMs for MDS by concatenating multiple documents (Guo et al., 2022; Xiao et al., 2022; Shen et al., 2022a,b), including the works that focus on longer context windows (Zaheer et al., 2020; Beltagy et al., 2020). However, concatenating the content of multiple documents together prevents the PLMs from distinguishing or processing the contents differently. By essentially processing the MDS inputs as if they were a single document, the PLMs may not handle the cross cross-document information well. An alternative is to conduct summarization through multiple stages (Song et al., 2022; Tu et al., 2022; Ernst et al., 2022; Hokamp et al., 2020a; Li et al., 2020; Lebanoff et al., 2018; Li et al., 2017b,c; Bing al., 2015), where salient contents are first extracted or summarized from multiple documents, then passed to a PLM for the final abstraction stage. However, the inputs to the PLM now consist of groups of incoherent sentences or phrases stitched together from multiple sources, presenting significant distributional shift from the PLMs original pre-training corpora.
We conduct experiments on various MDS datasets (Shen et al., 2022b; Lu et al., 2020; Ghalan- dari et al., 2020; Fabbri et al., 2019; Wang and Ling, 2016) across a vast range of domains, namely, news, scientific literature, movie critics, peer reviews, and specific Wikipedia topics. Our experimental results show that our hicrarchical scheme outperforms strong state-of-the-ar models (including those with additional MDS pre-training or larger model sizes). It also consistently improves the PLM backbone on all 10 datasets by up to 3 ROUGE-L and, on the manually inspected datasets, is preferred by humans. In addition, our detailed attention, con- tent analyses, and ablation reveal that our method ensures more coherent encoder representations for intra-document tokens, as well as obtains the wider cross-attention coverage of the different documents during decoding. In this manner, our method suffi- ciently utilizes the pre-training knowledge for en- coding, while encouraging the decoding process to consider all documents by more proper weights, rather than over-foc a few documents and ig- noring the rest.
2 Related Work
Our method is different from the previous notable works (Xiao et al., 2022; Beltagy et al., 2020) in both that we make novel use of the ofort-of-docummnt tokens as a document-level representation, as well as further utilizing these tokens during our hierarchical decoding scheme to enable cross-document interactions.
MDS Models Previous works have designed spe- cific neural abstractive summarization architectures for the MDS task (Liu et al.. 2021; Liu and Lap- ata, 2019a; Liu et al., 2018; Fabbri et al., 2019). However, it hard to adapt the existing PLMs to these specific designs. Thus, these models have to be trained from scratch using relatively large MDS datasets on the scale of hundreds of thousands of examples. The same applies to graph-based net- works (Chen et al., 2021; Li et al., 2020; Parveen and Strube, 2014; Christensen et al., 2013) that explicitly facilitate document interactions with the graphical designs. As result, all these models may work well in the specific domain it was trained in, but may not generalize well the smaller MDS datasets of other domains. Recent works experi- ment with additional MDS pre-training (Xiao et al., 2022: Puduppully and Steedman, 2022) with an existing PLM. Nevertheless, the MDS pre-training
3 Preliminaries and Notations
We focus on adapting encoder-decoder PLMs for the MDS task. The encoder-decoder Transformer (Vaswani et al., 2017) is a common underlying architecture for PLMs frequently used for summa- rization tasks (Beltagy et al., 2020; Lewis et al., 2020; Zhang et al., 2020). The encoder uses self- attention to encode the source tokens, whereas the decoder uses cross-attentin with source tokens and self-attention with the generated tokens for decod- ing. Our method targets the attention involving the

---

source tokens, i... the self-attention in the encoder and the cross-attention in the decoder.
encoder and decoder. For the second requirement. we use the PLM's "
" token for the hierarchi- cal representation of the documents in the encoder, then make further use of these tokens for hierarchi- cal attention scaling in the decoder.
Formally, the encoder updates the representation of each source token across all layers. The rep- resentation of the ith token in the jth layer, h,)) can be computed as a weighted sum of the hidden states of the context tokens in the previous layer:
4.1 Hierarchical Encoding Scheme
4.1.1 Leveraging PLM Knowledge
K Vh (--1)
To better leverage the knowledge for source token representations gained during pre-training, we ap- ply the following modifications to the encoder: re- stricted intra-document full attention, and position restart.
Σ k=1
wk
(1)
where V is the value projection matrix, K is the total number of input tokens, and the weight wk is obtained by a softmax calculation of the token attention scores:
Restricted Intra-Document Full Attention Since the PLM's self-attention is only applied to tokens from the same congruent source during pre- training (Lewis et al., 2020), it may not handle cross-document token representations well. Thus, to conform with the pre-training process, we re- strict the self-attention for each token to only its sibling tokens within the same document (Figure top left). In this way, the tokens are only allowed to see words within their own documents, which avoids leaking information or being influenced by off-context or contradictory information from other documents. Some works (Shen et al., 2022b; Fab- bri et al., 2019) have directly applied full attention to tokens from all documents (Figure 2a) or use local attention window (c.g., LED (Beltagy et al., 2020) and PRIMERA (Xiao et al., 2022)) (Figure 2b) where each token only attends small window of surrounding tokens. both approaches, the token is allowed to be influenced by tokens from other documents, thus potentially causing improper representations.
w (wo, wk) so softmar(ao- .... ok) (2)
The attention scores (oo. ax) are com-
puted with the query (Q) and key matrices (K):
(j-1) Kh x- a a=Qh [Khg
(3)
For the decoder, cross-attention is calculated be- tween the last generated token and the source to- kens. The weights calculations are similar to Equa- tion (2) and Equation (3) except that the matrix takes in the representation of the last generated token (after decoder self-attention), and the K ma- trix takes in the representations of the source tokens from the last encoder layer.
Lastly, for multi-document encoding, we fol- low the common approach (Xiao et al., 2022; Shen et al., 2022b; Fabbri et al., 2019) to feed multiple inputs into the model by concatenating document after another. With N documents, the input sequence can be represented as X (Do. DI..., Dx), where D, = {#n,0, In,1, ...) are the input tokens from the th document.
Position Restart We restart the positional encod- ing for each document (Figure bottom), in order to signal to our modified encoder that subsequent words are from the next document and are not con- tinuous paragraphs the previous document. As a result, the encoding process of each document will conform with the single-document encoding process that PLMs were pre-trained on.
4 Method
To better leverage the strong language modeling capabilitics of PLMs, we propose an encoding- decoding scheme to better equip the PLMs for the MDS setting. Our design fulfills the following re- quirements: (1) It leverages the language modeling capability of PLMs gained during pre- pre-training; and (2) It facilitates the model to better process cross- document information during fine-tuning with the MDS datasets.
4.1.2 Handling Cross-Document Information Leveraging the PLM knowledge alone only lead to the independent processing of each doc- ument. To equip the model with capabilities better learn cross-document information during the fine-tuning stage, we make use of global tokens for document-level information encapsulations.
For the first requirement, we preserve the token interactions within the same document for both the

---

Document
Document
Document
softmax
Pup
Document Documeet
softmax
Po.2
Hierarchical Attention
21.1
softmax
912
22.0
P2y
softmax
22.9
Hierarchical Self-Allention
Generate
Yo
Y2
Positional
Embedding
+
+
+
+
+
+
Input
Token
<g>>
Xap
Xo.3
<g>>
K
X
X1,
<>>>
X2,2
X2,3
Embedding
Document
Document
Document
Figure 1: Our model diagram that highlights the differences compared with the PLM backbone: 1. re-start the positional embedding for each document (Section 4. 1); 2. use hierarchical self-attention in the encoder (Section 4. 3. use hierarchical cross-attention in the decoder (Section 4.2). For "Hierarchical Self-Altention" we use bold black lines to indicate document borders, blue cells for local attention, and orange cells for document-lews attention. Best viewed in color.
SOD token. However, instead of letting the SOD tokens attend to all tokens in the documents and act global tokens, we restrict their attention to the corresponding document tokens that they represent. This is because the "
" commonly resides at the sequence's outset of the pre-training inputs of PLMs, and has also been conventionally employed to encapsulate text sequence in Transformer architectures. Consequently, the PLMs have already acquired the capability capture the document information with the "
" tokens. this way, leverage the "
" tokens as a high-level document representation to further facilitate cross-document interactions both the encoder and decoder (Section 4.2).
(b) Global local atin window Figure 2: Encoder self-attention patterns for different attention schemes. We use black lines to indicate docu- ment borders in the MDS setting, blue cells for local attention, and orange cells for global attention.
(a) Full atm
Start-of-Documen (SOD) Representations Previous approaches have employed either a single "
" token for retaining global information of the full input sequence through self-attention (e.g., LED) or have introduced multiple special "<doc-sep>" tokens between documents for the same purpose (e.g., PRIMERA). In contrast, adopt different strategy by utilizing the "
" token to capture information from individual documents. Specifically, we position the "
"
Document-Level Attention Unlike LED PRIMERA which enable full attention global tokens (shown with orange blocks in Figure 2b), only allow the SOD tokens attend to same- document tokens and SOD tokens from other doc- uments, as shown with the orange blocks in the "hierarchical self-attention" diagram Figure 1. This allows each SOD token to encode the informa-
token the start of each document, serving as the

---

tion in the same document, while only exchanging cross-document information with other SOD to- kens.
Dataset
Instances
Does
Lettare
Lettge
Train Steps
Multinews
56K
2.8
1793
217
1300000
WCEP
10K
9.1
3866
28
15500
Multi-Xscicnnes
40K
5.1
700
105
90000
Rotlen Tomatons
3K
100
2052
21
4500
4.2 Hierarchical Decoding Scheme
MReD
6K
3.3
1478
120
10500
MReD4
6K
6.3
3069
10500
We further design a hierarchical decoder to leverage the document encapsulations in the SOD tokens, while leveraging the existing pre-training knowl- edge for decoding. As discussed in Section 3, the PLM's decoder first carries out self-attention with the previously generated tokens, followed by cross- attention with the source tokens. We do not modify the self-attention mechanism which independent of the source tokens. Instead, we take advantage the document-level information in the SOD en- coder output tokens, by scaling the cross-attention attention weights towards the source tokens from the respective documents accordingly.
Film
37K
4.5
777
85000
MrunOTTrangporatiinn
10K
4.1
878
20000
Town
16K
4.7
582
52
37000
Software
15K
4.3
843
113
35000
Table I: The statistics of all datasets used in this paper. The number of instances includes all training, validation. and test set examples.
(Lu et al., 2020), Rotten tomatoes (Wang and Ling, 2016), and the recently released MReD dataset (Shen et al., 2022b). These datasets are gath- ered from wide range of domains, namely news, events, scientific literature, movie reviews, and peer reviews. Table provides the statistics of each dataset (see download links in Appendix A).
We illustrate the cross-attention mechanism in Figure/ 1 ("hierarchical "ross-attention" diagram). Formally, given N documents, we denote the cross- attention scores in the decoder toward each individ- ual document as an = (An,0: kn,k.), where kn is the number of tokens in the nhth document. Simi- lar to Equation (2), we calculate the cross-attention weights for tokens within each document:
Due the scarcity of MDS datasets, we further compile several datasets in different domains. the MReD peer-review dataset (Shen et al., 2022b), multiple reviews of the same research paper are used as the inputs, whereas the meta-review is used as the summary. We extend this dataset by includ- ing the corresponding rebuttals and refer to it "MReD+". In addition, compiled MDS datasets from four domains of Wikipedia articles?, namely, "Film", "MeunOTTransporattion"" "Software", and "Town". These datasets use the Wikipedia lead sec- tion text as the summary and the individual sections as multi-document inputs.
Wp, = (Wn,0s .... Wn.k.)) sof (oolmaa(aa)) (4)
Next, we rely on the SOD tokens to decide the relative importance of each document, then scale the cross-attention weights accordingly. To do so, we first obtain the normalized scaling factor for each document as:
So..., Sn = softmarraa..00.... an.0)
5.2 Evaluated Models
BART We fine-tune the the pre-trained "bart- large" (Lewis et al., 2020), which uses full attention (Figure 2a) for all source tokens.
where 0n E On is the attention score for the SOD token in the th document. We derive the normal- ized attention weights for tokens in each document:
LED LED (Beltagy et al., 2020) is competi- tive baseline for long text summarization. It is di- rectly initialized from "bart-large", but uses global- local attention to better handle long context in- puts. Specifically, each local token has sliding- window attention and also attends to the global tokens, whereas global tokens attend to all source tokens. This model contains slightly more parame- ters due to separate V. Q, and K (see Equation (1) and Equation (3)) for the global tokens and local tokens.
n (wn, W,,,k) Sn Wn
(6)
In this manner, our MDS decoder can better grasp the relative degree of importance of each document, while preserving relative cross-attention weights within the same document during decoding.
5 Experiments
5.1 Benchmark Datasets
We conduct experiments on wide range of MDS datasets as follows: Multinews (Fabbri et al., 2019), WCEP (Ghalandari et al., 2020),
Original dataset from WikiAsp (Hayashi al., 2021). "The model checkpoints are detailed in Appendix B.

---

LongTs LongT's (Guo et al., 2022) adopts sum- marization pre-training strategies (Zhang et al., 2020) with T3 (Raffel et al., 2020) for long context summarization. uses transient global attention, which differs from LED in that the global tokens are obtained by summing the embeddings of tokens in k non-overlapping continuous blocks which di- vides the full input source. Due to resource con- straints, we could only fine-tune the base model on our machines. Thus, we also provide our method results using the "bart-base" backhone smaller model size.
embeddings 4 times, similar how LED is de- rived from BART by Beltagy al. (2020). Follow- ing PRIMERA, for Multinews, WCEP, and Multi- Xscience, we truncate the end of each source docu- ment.. For the rest of the datasets, we follow Shen et al. (2022b) and truncate the end of the combined documents. See more explanations for our choices of truncation for different datasets in Appendix C. We use Adam optimizer with learning rate of 5e - 5, and without any warm-up or weight decay. All models are trained on a single A100- 80G GPU, for the same number of training steps on the same dataset as shown in Table 1. The number of training steps for each dataset is determined based on the development set losses on the "bart- large" baseline We evaluate all model outputs on the ROUGE (Lin, 2004) metric and provide the Fl values of ROUGE-I, ROUGE-2, and ROUGE-L. We also conduct human evaluations on selected benchmarks.
PRIMERA PRIMERA (Xiao et al., 2022) uses additional MDS pre-training on the NewSHead dataset top of the LED model. It introduces a special "<doc-sep>" token in between the docu- ments for the same function as the global token in LED. It also uses a moving context window, which may still capture tokens across two consecutive documents (see Figure 2b).
Additionally, compile results directly ported by other papers (Guo al., 2022; Song al., 2022: Liu et al., 2022; Tu et al., 2022: Shen et al., 2022b; Pasunuru et al., 2021) > in Appendix D's Ta- ble 6 to Table 10 to give better overview of the cur- rent performances on different datasets. However, as the settings for reported models differ vastly, the results are not strictly comparable.
HED (ours) We apply our Hierarchical Encoding Decoding scheme (HED) on top of "bart-large" to obtain "BART+HED". Since HED doesn't introduce new parameters, the "BART+HED" is strictly comparable with the BART baseline. Morcover, we apply HED with the "bart-large-cnn" checkpoint, resulting in the "BART-cnn+HHP model. The "bart-large-cnn" checkpoint involves additional pre-training with an SDS dataset in the news domain (i.c., CNN/DM) over the "bart-large" checkpoint. While we do not conduct additional pre-training ourselves, we can still compare "BART-cnn+HED" with "BART+HED" to assess the knowledge transferred during the pre-training stage using an SDS dataset to the MDS downstream tasks. Moreover, this can be juxtaposed with the comparison between PRIMERA and LED, which reveals the benefits derived from extra MDS pre-training on news dataset (i.e. NewSHead) for the MDS task.
Results
6.1 Main Results
We show the ROUGE-1 and ROUGE-L results Table 2 for all 10 datasets (see ROUGE-2 scores in Appendix Table 11). The upper section of Table 2 displays the performance of comparatively smaller models. Notably, our "BART+HED" method sur- passes LongT5 across nearly all benchmarks, de- spite having approximately half the model size. Moving to the lower section of Table 2, our "BART+HED" model consistently exhibits im- provement over the corresponding "BART" back- bone, demonstrating the effectiveness of our pro- posed method.
5.3 Experimental Setup
We fine-tune all evaluated models discussed above with cross-entropy loss on all datasets. Following PRIMERA (Xiao et al., 2022),
In addition, "BART-cnn+HED" generally out- performs "BART+HED". This shows that our pro- posed method can effectively facilitate knowledge
source and target truncation of 4096 and 1024 respectively. Amongst all evaluated models, LongT5 and LED can readily accept source input of 4096 tokens. BART can only accept a maximum 1024 to- kens, so we repeatedly copy BART's positional
"For instance, if there are documents, each document is truncated 4096 1024 tokens.
*We have experimented on a few datasets and observed that the same number of training steps is selected by the "hart-base" model.

---

Multinews
WCEP
M-XSc
RT
MRkeD
MRee)+
MeanOT
Town
Software
Film
System
Size
R-1/R-L
R-1/R-L
R-1/R-L
R-1/R-L
R-1/R-L
R-1/R-L
R-I/R-L
R-1/R-L
R-1/R-L
R-1/R-L
LongT5
250M
46,4//4.5
43.4/35.3
27.0V15.0
26.0200.5
32.0/20.1
32.7/20.6
41.2//3.77
60.2/56.7
37,5/28.4
42,4/35.5
BARD
139M
47.1/25.0
44.8/36.8
31.9/7.77
26.8/0088
32.2/20.6
32.7/20.8
40.6/33.7
61.4/57.7
37.2/28.6
42.8/35.9
LED
435M
50.1/25.0
46.5/37.6
31.2/16.6
27.3/20.7
33.0/19.1
34.3/20.3
45.4/35.1
62.M/5.3
42. 1/28.8
44.8/35.7
PRIMERA*
447M
49.9/25.9
4/.1/
31.9/18.0
PRIMERA
447M
49,0/25.6
46,2/37.4
31.9/18.0
27.4/21.1
29.6/17.0
29.2/16.5
44.1/35.6
62.1/58.3
39.0/2844 44,4/36.9
BART
406M
47.4/24.0
42.8/34.5
31.5/16.9
26.1/20.3
32.9/19.9 32.9/20.1 43.0/34.9 59.9/56.3
39.5/28.7 42.1/34.4
BART+HED
406M
50.0/25.8
46,4/37.8
32.1/17.6 27.3/21.1 33.9/20.9 34.0/20.7 43,5/35.2 61.9/57.7 40.5/29.7 43.8/36.3
BART-can+HED
406M 51.1/25,9
47,0/37.6 34.7/18.6 27.6/00.5 34.1/20.5 34.5/20.6 46.1/35.4 62.8/58.3 42.9/29.7 45.9/36.6
Table ROUGE-1 and ROUGE-L results on 10 datasets including: Multinews, WCEP, Multi-XSciencs (M-XSc).
Rotten Tomatoes (RT), MRRD, MReD+, and 4 Wikipedia domains. *; results reported by Xiao et al. (2022).
Multinews
MReD
model
Flu
Rel
Abs
Sal
Cov
Flu
Rel
Abs
Sal
Cov
BART 0,510
0.430
0.475
0.500
0.480
0.440
0.480
0.370
0.355
0.350
BART+HED
0.490
0.570*
0.525
0.500
0.520
0.550*
0.520
0.630*
0.645*
0.650
Table 3:
Head-to-head human evaluation scores for fluency (Flu), relevance (Rel), abstractiveness (Abs), salience (Sal), and coverage (Cov). *: statistically significant with a higher human preference for p < 0.05.
transfer, even for the knowledge gained with pre- training on an SDS news dataset. On the other hand, PRIMERA uses additional MDS pre-training with a news dataset on top of LED, but experi- ences some obvious performance drops, especially on the datasets of MReD, MRed+, and Software. As discussed previously, the MDS pre-training cor- pora characteristics deviate significantly from those used during the PLM's original pre-training. Thus, it is likely that additional MDS pre-training for PRIMERA on only one news dataset (i.c., New- Shead) has led to catastrophic forgetting of the other domains. As result, the model loses some knowledge gained during its previous pre-training stage, and performs especially badly on domains that deviate significantly from the news domain.
from a single document suffices for summarization. Consequently, although LED lacks an inherent ar- chitecture for handling cross-document informa- tion, it can still perform well because of its opti- mized capability to process long-context inputs.
6.2 Human Evaluation
We conduct human evaluations on the Multi- news and MReD datasets of vastly different do- mains. Specifically, we randomly sample from each dataset 50 test samples to be evaluated inde- pendently by 2 random evaluators among a pool of 5 volunteers in the research field of artificial intelli- gence. Each evaluator is given both outputs from "BART" and "BART+HED" in a double-blind man- ner, and asked to rate I point for the better summary and 0 for the worse, while giving both 0.5 for tic.
Nonetheless, our method does not guarantee im- provements with additional SDS pre-training, as evidenced by the drop in ROUGE-L for the Rotten Tomatoes. This dataset is very small, comprising of around 2K training examples. It is also unique in the sense that each sample consists of an average of 100 documents (see Table 1 "#Docs"). In this case, additional SDS pre- pre-training may negatively impact the model's s ability to handle inputs with a large number documents, especially with very limited MDS fine-tuning data.
The evaluation criteria are (1) Fluency the over- all flow and grammar of the summary; (2) Rel- evance the selection of important content from input documents; (3) A Abstractiveness the extent to which the summary consists of rephrased contents and avoids copying extensively from the input; (4) Salience the amount of salient information in- cluded in the summary; and (5) Coverage the minimal number of input documents required to supply sufficient information for the summary. The latter new criteria are newly proposed by us to better assess the MDS task (see more details in Appendix E).
Another interesting observation is that LED per- forms relatively strongly, particularly on the WCEP and Multinews datasets. According to Wolhan- dler (2022), the gold summaries of WCEP and Multinews indicate a relatively limited degree multi-text merging, suggesting that information
The results are reported in Table For both evaluated benchmarks, "BART+HED" out- performs "BART" overall. In terms of writing

---

BART RART-EED
x BART y RART-HI
Relative Attention
Relative CDS
0.8
Moltoness
WCBP
Mutti KSai--XSaraeee
AfRati
Wils Town
Wiksi Witi-Fita
Wilks Seffware
Wike WarraeantOThan
Maliines
WCBP
Muiti KSoraee
MiRal
Wils-tom
Wikk the
Wike Mea-Mearrrraam
(a) Relative document self attention of "BART+HED" over "BART" in the encoder. For better visualization, we exclude the result for Rotten Tomatoes, which is 19.5.
(h) Relative cross-dncument standard deviation of "BART+HED" over "BART" in the decoder. We exclude the result for Rotten Tomatoes, which is statistically insignificant.
Figure 3: Document-level attention analysis. Results statistically significant with p 0.001.
style, "BART+HED" has significantly better ab- stractiveness for both datasets. This suggests that "BART+HED" summarizes using its own words rather than copying long phrases and sentences verbatim from the source, thanks to the hierar- chical structure that encourages explicit cross- document information handling. While higher levels of abstraction may lead to fluency degra- dation, "BART+HED" still performs competi- tively with "BART" in fluency on the Multinews dataset, and even surpasses "BART" on the MReD modifi-
"BART" baseline in the encoder. Unsurprisingy. the self document token attention for "BART+HED" is significantly higher than the baseline across all datasets. This observation affirms that each token representation influenced more by the coherent context of its own document while ignoring the po- tentially misleading or irrelevant information from other documents.
Decoder Analysis During decoding, we propose the Cross-Document Standard deviation (CDS) metric (see Appendix G) to measure the standard deviation of each predicted token's normalized cross-attention weights toward different documents. We plot the relative CDS of our "BART+HED" model over "BART" Figure 3b. A higher CDS indicates that the current decoding step only pays concentrated attention to a few documents, whereas smaller CDS indicates that the attention is more evenly spread out across different documents. In Figure 3b, it is evident that model has signifi- cantly smaller CDS values across all datasets. This shows that during the decoding process, our model pays attention more evenly across documents rather than focusing on a specific document, which helps it produce more comprehensive summaries that consider more documents."
dataset. This indicates that despite cations, "BART+HED" still retains sufficient lan- guage modeling capability from "BART".
In terms of content, our method also achieves higher levels of relevance than "BART" for both datasets. This supports that our model is better at comparing input documents during decoding (also supported by Section 6.3 later) and extracting the most important information for the summary. Moreover, MReD from the peer-review do- main, the summaries, which are meta-reviews, are highly condensed (Shen et al., 2022b) and pose a stronger challenge due to frequently conflicting in- put documents from disagreeing reviewers. In this dataset, "BART+HED" is superior in both salience and coverage with much higher margins, suggest- ing that method is particularly effective for such complex MDS tasks.
6.4 Content Analysis
To further validate if our method indeed produces more salient summaries, we conduct entailment- based content analyses the generated summaries across all datascts. Inspired by Laban et al. (2022). we treat each sentence in the summary as one infor- mation unit, and then calculate the average Number
6.3 Attention Analysis
To better understand the generation process, conduct attention weights analysis on 200 test sam- ples per dataset for both the encoder and decoder.
Encoder Analysis Figure 3a presents the relative (normalized over the "BART" baseline) attention each source token toward its belonged document (a.k.a. self document) of "BART+HED" over the
"Rotten Tomatoes has near-zero absolute CDS values. which is likely due to the much higher numbers of documents per sample. These results are not included in Figure 3b due to statistical insignificance.

---

System
Multinews
WCEP
M-XSc
RT
MRel)
MReD)+
MeanOT
Town
Software
Film
BART
0.71
4.47
0.65*
1.19
0.82
0.84
0.24
0.36
BART-HED
0.72
4,70
0.46
1.32
0.83
1.07
0,27
0.27
0.27
0.40
Table 4: Sentence-mormalized NED for the generated summaries. *; significantly higher with p 0.05.
HAE
HAD
PR
A(-1))
A(R-2)
A(R-))
As compared to the baseline BART (row 0), sim- ply adding the additional "
" tokens (row 1) can result in large performance gains. Next, by adding HAE row 2, we gain some small improvements. Rows and 4 show the gains of adding either HAD or PR on top of row 2. Interestingly, adding HAD or PR separately has little impact, but combining them leads to significant gain (row 5). This shows that position restart matters more for HAD than for HAE, because helps HAD to distinguish the dif- ferent documents, while our HAE encoder already restricts the attention of each source token to the same document and is thus less affected.
+0.6
+0.7
+0,9
+0.8
+0.8
+1.0
+0.8
+0.7
+0.9
+1.0
+0.9
/
+1.5
+1.3
+1.3
Table 5: Ablation. Average performance gain of using various subsets of our components compared to the baseline BART model (row 0).
of Entailed source Documents (NED) per sentence (see Appendix H). The higher the NED, the more salient the summary potentially is, since it contains information that entailed by more documents. As shown in Table 4, on most datasets, "BART+HED" has statistically higher NED, sug- gesting that our method may generate more salient summaries. One special case is Multi-Xscient (M-XSc), which uses the related work section of paper as the target summary, and the abstract of the cited papers as input documents. Upon inspec- tion, we discover that the summaries generated by "BART+HED" are much more abstractive and suc- cinct (consistent with the human evaluation results on other datasets in Section 6.2), resulting in low scores below the threshold value for the entailment model used; on the other hand, the generations of "BART" copies extensively and are thus easily classified as positive entailments. When a smaller threshold is used (Appendix H), our method still outperforms the "BART" backbone in general, and the NED difference on the Multi-Xscient dataset for both models reduces to an insignificant level (p > 0.05).
Conclusion
In this paper, we work on the abstractive multi- document summarization (MDS) task. We propose a hierarchical encoding-decodiin scheme, which allows effective fine-tuning of the PLM on a spe- cific MDS task dataset without any new parameters. Our proposed scheme makes novel use of global tokens for document-level interactions in both the encoder and decoder. can leverage the general- izing capability of PL.M across wide variety of domains. Evaluation results on MDS datasets show that our approach consistently outperforms the previous best models and our PLM backbone.
Limitations
Theoretically, our encoding-decoding scheme can reduce the space complexity of MDS n= summarization from 0( n= nx)2)) to O(Mazz(no "I..,,,, n))))) The former is the square of the total input length from all documents, whereas the latter is simply the square of the longest document. This is because a significant amount of self-attention is no longer calculated due to our restricted intra-document full attention mechanism. However, we have not implemented such optimization in our work as the actual realization is more complicated. As a result, our work also faces the common challenge of computational resources for long document summarization. We leave the investigations for better computational efficiency to future work.
6.5 Ablation Study
To investigate the effectiveness of each of our pro- posed components, we present the average perfor- mance gain from the BART baseline by using only a subset of our proposed components, namely the SOD <>> token, encoder hierarchical attention (HAE), decoder hierarchical attention (HAD), and position restart (PR). Note that HAE and HAD depend on "
" and HAD depends on HAE (Sec- tion 4), so they cannot be used along without their dependent components. We show the averaged re- sults across 10 datasets in Table 5 (see full results in Appendix F).
Due to the inefficient computation for long doc-

---

ument summarization, we focus on smaller-sized MDS datasets for PLM fine-tuning. We did not conduct experiments on the much larger datasets WikiSum (Liu et al., 2018), Arxiv and PubMed (Cohan et al., 2), and GovReport (Huang et al., 2021) as they are much larger in scale and require much more computational resources. Nevertheless, we believe that our incoding-decoding scheme has demonstrated consistent trends of improvement across a wide range of MDS domains.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of EMNIP-IJCNNLP.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:200..05750.
Lidong Bing. Piji Li, Yi Liao, Wai Lam, Weiwei Guo. and Rebecca Passonneau. 2015. Abstractive multi- document summarization via phrase selection and merging. In Proceedings of ACL-IJCNLP.
Moye Chen, Wei Li, Jiachen Liu, Xinyan Xiao, Hua Wu, and Haifeng Wang. 2021. Sgsum: Transform- ing multi-document summarization into sub-graph selection. In Proceedings of EMNLP.
Ethics Statement
This paper involves the preparation of several datasets. For MReD+, we have obtained approval from the ICLR committee to use data collected from the OpenReview? portal. The authors of MReD (Shen al., 2022b) already released the data for non-commerciiaizee public usage.
Janara Christensen, Stephen Soderland, Oren Etzioni. et al. 2013, Towards coherent marization. In Proceedings of the NAACL-HLT.
sum-
Eric Chu and Peter J. Liu. 2018. Mcansum: A neural model for unsupervised multi-document abstractive summarization. In International Conference on Ma- chine Learning.
For the Wikipedia domain datasets, we pair sub- portions of the articles organized by WikiSum (Liu et al., 2018) and WikiAsp (Hayashi et al., 2021), which are publicly available datasets.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-an aware attention model for abstractive summarization long documents. Proceedings of NAACL-HLT (Short Papers).
Acknowledgements
Yang You is being sponsored by NUS startup grant (Presidential Young Professorship), Singa- pore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant.
Peng Cui and Le Hu. 2021. Topic-guided abstractive multi-document summarization. In Findings of the Association for Computational Linguistics: EMNLP P 2021, pages 1463-1472, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Ori Emst, Avi Caciularu, Ori Shapira. Ramakanth Pa- sunuru, Mohit Bansal, Jacob Goldberger, and Ido Dagan. 2022. Proposition-leve clustering multi- document summarization. In Proceedings of NAACL- HLT.
References
Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2020. Unsupervised opinion summarization with content planning. In Proceedings of AAAI.
Alexander Richard Fabbri, Trene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019, Multi-news: large- scale multi-documeet summarization dataset and ab- stractive hierarchical model. In ACL.
Reinald Kim Amplayo and Mirella Lapata. 2019. Infor- mative and controllable opinion summarization. In Proceedings of EACL.
Robert M French, 1999, Catastrophic forgetting in con- nectionist networks. Trends in cognitive sciences.
Reinald Kim Amplayo and Mirella Lapata. 2020. Un- supervised opinion summarization with noising and denoising. In Proceedings of ACI.
Kavita A. Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: graph
based approach to abstrac- tive summarization of highly redundant opinions. In Proceedings of Coling.
Regina Barzilay, Kathleen McKeown, and Michael EI- hadad. 1999. Information fusion in the context of multi-document summarization. In Proceedings of ACL.
Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018. Bottom-up abstractive summarization. In Proceedings of EMNLP.
Tal Baumel, Matan Eyal, and Michael Elhadad. 2018. Query focused abstractive summarization: Incorpo- rating query relevance, multi-document coverage, and summary length constraints into seq2seq models. arXiv preprint arXiv:1801.07704
Demian Gholipour Ghalandari. Chris Hokamp. Nghia The Pham, John Glover, and Georgiana Ifrim. 2020. A large-scale multi-document summarization dataset from the wikipedia current events portal. Proceedings of ACI..
"https://openrewe

---

Xiaotao Gu. Yuning Mao, Jiawei Han, Jialu Liu, You Wu, Cong Yu, Daniel Finnie, Hongkun Yu, Jiaqi Zhai, and Nicholas Zukoski. 2020. Generating represen- tative headlines for news stories. In Proceedings of The Web Conference.
Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summarization. In Proceedings of ACL.
Chin- Cew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branckes out.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. T5: Efficient text-to-text transformer for long sequences. In Findings of NAACL.
Peter J Liu, Mohammad Salch, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summariz- ing long sequences. In ICL.R
Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021. Wikiasp: A dataset for multi-domain aspect- based summarization. TACL.
Shuaiqi Liu, Jiannong Cao, Ruosong Yang, and Zhiyuan Leveraging key phrase aware attention to improve abstractive multi- document summarization. In Findings of ACL- IJCNLP.
Wen. 2021. Highlight-transfooe
Chris Hokamp. Demian Gholipour Ghalandari, Nghia The Pham, and John Glover. 2020a. Dyne: Dynamic ensemble decoding for multi-document summarization. arXiv preprint arxivv2006.08848.
Chris Hokamp. Demian Gholipour Ghalandari, Nghia The Pham, and John Glover. 2020b. Dyne: Dynamic ensemble decoding for multi-document summarization. ArXiv, abs/2006.08748.
Yang Liu and Mirella Lapata. 2019a. Hierarchical trans- formers for multi-document summarization. In Pro- ceedings of ACL.
Yang Liu and Mirella Lapata. 2019b. Text summariza- tion with pretrained encoders. In Proceedings of EMNLP-IJCNLP.
Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. 2021. Efficient attentions for long document summarization. Proceedings of NAACL- HLT.
Yixin Liu, Ansong Ni, Linyong Nan. Budhaditya Deb, Chenguang Zhu, Ahmed H Awadallah, and Dragomir Radev. 2022. Leveraging locality abstractive text summarization. In Proceedings of EMNLP.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NI.I-
based models for inconsistency detection in summa- rization. TACL.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi- xscience: A large-scale dataset for extreme multi- document summarization of scientific articles. In Proceedings of EMNLP.
Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018. Adapting the neural encoder-decoder framework from single to multi-document summarization. In Proceedings of EMNLP.
Daraksha Parveen and Michael Strube. 2014. Multi- document summarization using bipartite graphs. In Proceedings of TextGraphs-9.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Chazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sqquee ural language generation, translation, and compre- hension. In Proceedings of ACL.
Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Su- jith Ravi, and Markus Dreyer. 2021. Efficiently sum- marizing text and graph encodings of multi-document clusters. In Proceedings of NAACL-HLT.
pre-training for nat-
Piji Li, Lidong Bing, and Wai Lam. 2017a. Reader- aware multi-documeed summarization: An enhanced model and the first dataset. In Proceedings of the Workshop on New Frontiers in Summarization.
Laura Cerez-Beltrachin and Mirella Lapata. 2021. Multi-document summarization with determinantal point process attention. Journal of Artificial Intelli- gence Research.
Piji Li, Lidong Bing, and Wai Lam. 2018. Actor-critic based training framework for abstractive summariza- tion. arXiv preprint arXiv: 1803. 1070.
Ratish Puduppully and Mark Steedman. 2022. Multi- document summarization with centroid-based pre- training. arXiv preprint arXiv:2208.0100
Piji Li, Wai Lam, Lidong Bing, Weiwei Guo, and Hang Li. 20170 7b. Cascaded attention based unsupervised in- formation distillation for compressive summarization. In Proceedings of EMNLP.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. Journal of Machine Learning Research.
Piji Li, Zihao Wang, Wai Lam. Zhaochun Ren, and Lidong Bing. 2017c. Salience estimation via vari- ational auto-encoders for multi-document summa- rization. In Proceedings of the AAAI Conference on Artificial Intelligence.
Eva Sharma, Chen Li, and Lu Wang. 2019. BIG- PATENT: A large-scale dataset for abstractive and coherent summarization. In Proceedings of ACL.

---

Chenhui Shen, Liying Cheng, Lidong Bing. Yang You, and Luo Si. 2022a. Sentbs: Sentence-level beam search for controllable summarization. In EMNLP.
Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022b. Mred: A meta-review dataset for structure-controllat Findings of ACI.
text generation. In
Yun-Zhu Song, Yi-Syuan Chen, and Hong- Han Shuai. 2022. Improving multi- doltimdocument summarization through referenced flexible extraction with credit- awareness. In proceedings of NAACL-HLT.
Shangqing Tu, Jifan Yu, Fangwei Zhu, Juanzi Li, Lei Hou, and Jian-Yut Nie. 2022. UPER: Boosting multi-document summarization with an unsupervised prompt-basee extractor. In Proceedings of Coling.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Éukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceddings of NeurlPS.
Lu Wang and Wang Ling. 2016. Neural network-based abstract generation for opinions and arguments. In Proceedings of NAACL-HLT.
Ruben Wolhandler, Arie Cattan, Ori Ernst, and Ido Da- gan. 2022. How "multi" is multi-document summa- rization? Proceedings of EMNLP.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2022. Primera: syramid-based masked sen- tence pre-training for multi-document summarization. In Proceedings of ACL.
Michihiro Yasunaga, Rui Zhang, Kshitijh Meelu, Ayush Pareek, Krishna Parasuram Srinivasan, and Dragomir R. Radev. 2017. Graph-based neural multi- document summarization. In Proceddings of CoNLL.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. NeurIPS.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In Pro- ceedings of ICML.
Hao Zhou, Weidong Ren, Gongshen Liu, Bo Su. and Wei Lu. 2021. Entity-aware abstractive multi- document summarization. In Findings of ACL- IJCNLP.

---

A Datasets Downloads
System
R-1
R-2
R.L.
PG-BRNN (Gelmman et al., 2018)
43.8
15.4
20.8
We download
Multinews
from
https:
Hieruchical-- Transformet (Liu and Laputa, 2019s)
42.4
15.3
22.1
Hi-MAP (Fabbri 2019)
"rgithub.com/Aless iatpr/Alee-Fabbiii/i Faabrri/ultiiNeett
WCEP
GraphSum (Li al., 2020)
45.0
16.7
22.5
from
https://github: comm/comlmmmntt tizer/ from
Highlight Transformer (Liu et 2021)
44.6
15.6
GraphSum RoBERTa (Li
wcep-mds- -dd--ddddddtt,, Multi-Xsceince
CTF DPP (Perez: Beltrachini and Lapata. 2021)
45.8
15.9
21.0
https:///itthee wttp.//oooo//uuuliiiSSSttt..
Rot- .ew.bbbicc...hh..
Long' TS base
46.4
18.6
24.5
BARToaae)HEDD Long
47.1
19.4
25.0
ten Tomatoes from
2022)
http://wwnglluyppoulllaaatoooooo html, MReD from https:://iithub. httllub..coo//heeeeeee commShhn-Chentuu/Meaae For MReD, we use the uncontrolled version of MReD since controllable summarization is not focus. We will release the processed data of MReD+ and the Wikipedia domain datasets with our code.
LongTT-al (Geo 2022)
48.2
19.4
24.9
BAKE-Long (Pasunuru et al., 2021)
48.5
18.6
23.8
Long--oong-Graph (Pasunuru et al., 2021)
19.0
24.0
Pag-Sun- (Liu 2022)
PEGASUS (Zhang ol., 2020)
47.5
18.7
24.9
BigBirdt
47.8
19.1
24.7
(MLE) (Song al., 2022)
REFLECT (CASC) (Song 2022)
20.0
24.8
BART-HED
50.0
21.0
25.8
BART-can-HED
51.1
21.5
25.9
B Model Checkpoints
BART<snn-HED:
Table 6: Results on Multinews. Results with citations are reported by the cited papers. t: results using Page- Sum's 's evaluation script. PageSum does not truncate the source but split all into different pages. However, are unable to run their code due to missing environment in- formation. Bolded systems: systems that HED method.
Our "bart-base" checkpoints are downloaded from or "bart-large" from https://hugginngacee facebook/barrttargge. LED from https:
//guagingaace... httaee/cooleeeea/eeedlrrreeeee 16384, LongT5 from https://hugginffant htt:://uuggngfaae..cogggggg long t5- tattggllbllbbass and PRIMERA from http:/thwb..commllleaa///tttteee In addition, depending the model size specified in Table 2, our "BART" was ei- ther initialized from the "bart-base" (139M) or "bart-large" (406M) checkpoints. our "BART- con+HED" was initialized from the "bart-large-enn" checkpoint from bart- Larg-large-cnn.
System
R-1
R-2
R-L
BigBird?
34,0
12.8
25.6
DynE (Hokamp al., 20206)
25.6
LED al., 2022)
BART+IED
46.4
24.7
37.8
BART-con-HED
47.0
24.8
37.6
Table 7: Results on WCEP. Results with citations are reported by the cited paper. Bolded systems: systems that use HED method.
Truncation Settings
to the meta-review. For Rotten Tomatoes which contains mostly single-sentence documents, per- document truncation may lead to incomprehensible broken sentences. For the Wikipedia domains, the encyclopedia- based information presented may be of equal importance regardless of their passage po- sition. Thus for all these datasets, we follow Shen et al. (2022b) truncate the end of the combined source.
We follow Xiao al. (2022) to use a per-document truncation for Multinews, WCEP, and Multi- Xscience. This setting is reasonable for Multinews and WCEP in the news domain, because the lead- ing sentences of news article may account for the overall event and, thus contain more salient information for summarization. Also, for Multi- Xseience, rather than delving into specific tech- niques and solutions, the beginning of the abstracts may give the overall description of the task and key ideas, and thus may provide a more suitable back- ground for summarization into the related works sections.
D Additional Results
We gather the available results reported by other papers and present them from Table 6 to Table 10. For newer datasets such as MReD (Shen et al., 2022b), there have not been results reported except for the original paper. Also, small datasets such as Rotten Tomatoes are also less studied and we could not find very recent results. Naturally, for our newly compiled datasets from four Wikipedia
However, the above setting may be equally reasonable for other domains. For the peer-review domain like MReD, the beginning sentences of re- views often give an abstract of the paper instcad of discussing personal opinions that matter more

---

System
R-1
R-2 R-L
System
R-1
R-2
R-L
HIERSUMM" (Liu and Lapata, 2019a)
30.0 5.0
OPINOSIS (Canesan al., 2010)
15.0
3.1
12.2
Hi-MAP" (Fabbri al., 2019)
31.7 5.9
MEANSUM (Chu and Liu, 2018)
15.8
1.9
12.3
PONTER-GENERATOR (Iu et al., 2020)
34.1 6.8
CONDASUM (Amplayo and Lapai, 2019)
22.5
7.7
18.5
BERTABS' (Liu and Lapata, 20196)
31.6 5.0
DENOISESUM (Amplayo et al., 20201
21.3
4.6
16.3
SCIBERTABS' (Bellagy et 2019)
32.1 5.6
PLANSUM (Amplayo and Lapata, 2020)
21.8
6.2
17.0
BigBirdt
31.1 7.0
16.8
BigBidf
19.3
14.0
BART-large (Song ul., 2022)
BART-HED
REFLECT (MLE) (Song ol., 2022)
33.9 8.1
BART-emn-HED
27.6 9.4 20.5
REFLECT (CASC) (Song 2022)
34.2
8.2
17.4
BART (our baselinc)
31.5
7.1
16.9
Table 9: Results on Rotten Tomatoes. Results with cita-
BART-HED
32.1
6.7
17.6
tions are reported by the cited paper. Bolded systems: systems that use our HED method. Unfortunately, we cannot find other more recent reported results on this dataset.
BAKT-cmn-HED
34.7
7.8
18.6
Table 8: Results on Multi-XScience. Results marked with are taken from Lu al. (2020). Other results with citations are reported by the cited papers. Note that Song et al. (2022) may have used different hyper- parameters. Their reported BA baseline results are much higher than ours, Bolded systems: systems that use our HED method.
System
R-1
R-2
R-I.
BART-con
33,3
8.6
19.7
BigBirdf
33.0
9.4
20.6
BART-HED
33.7
9.3
20.5
EART-cmn-HED
34.1
20.5
Table 10: Results on MReD. *; Reported by Shen et al. (2022b). Bolded systems: systems that HED method.
domains, there are no previously reported results.
Note that the additionally reported results meant for a better understanding of the current best performances only, because many of the reported models are not directly comparable to our setting (or comparable with each other) due to different model sizes and experimental settings. For in- stance, LongTS-large (Guo et al., 2022) uses 780M parameters, and LongT5-xl (Guo al., 2022) uses 3B parameters, which are not the same scale as the BART models. Moreover, multi-stage summa- rization models use multiple PLMs. For instance, REFLECT (Song et al., 2022) uses 2 RoBERTa- base and BART-large, whereas UPPER + LED (Tu et al., 2022) uses GPT and LED for the extrac- tion and abstraction stages respectively. Not all the models use the same source truncation either. In extreme case, PageSum (Liu al., 2022) uses the full source input with multiple encoding stages. Lastly, certain models didn't use pre-training data and are trained directly on the MDS dataset, such as Hi-MAP (Fabbri et al., 2019) and Highlight- Transformer (Liu et al., 2021).
form reasonably on larger datasets such as Multi- news, Multi-Xscience, and WCEP, but it lags be- hind other more competitive models. However, for small datasets such as Rotten Tomatoes, there is serious performance degradation. Nevertheless. given that ArXiv's 's domain is quite close to MReD, Big can reach competitive performance on this dataset.
E Human Evaluation
We define evaluation criteria specifically for the MDS setting: salience and coverage. MDS in- volves distilling important information from mul- tiple input documents. Thus, we use salience and coverage to measure how the summary makes use of the input documents from 2 perspectives.
First, treat each sentence in the summary as information unit and count the total number of input documents that support the information units for the whole summary. If the summary in- cludes consensus or opinions commonly agreed on by most input documents, it should have larger document count as compared to another summary that overly focuses on trivial details or contains extensive hallucinations. We normalize the total count by the total number of sentences in the sum- mary and regard it as the salience of the summary.
In addition, we also experiment with BigBird (Zaheer et al., 2020) with the same experimental setup in Section on 5 datasets. Unfortunately, we could only find BigBird's checkpoints for summa- of the following datasets: ArXiv (Cohan et al., 2018), PubMed (Cohan et al., 2018), or BigPatent (Sharma al., 2019). We use the Huggingface checkpoint at https: https://hugginggace co/ booggool/bbibbrd-peeee Larguuss-arg--axiv As shown by Table 6, 7, 8, 9, and 10, BigBird can still per-
rization trained
Second, we measure the minimum number input documents required to generate the corre-

---

Multinews
WCEP
M XSc
RT
MReD
MReD+
McanOT
Town
Software
Film
System
Size
R-2
R-2
R-2
R-2
R-2
R-2
R-2
R-2
R-2
R-2
LongT5
250M
18.6
22.5
8.6
23.1
45.7
18.5
24.8
BART-HED
139M
19.4
23.9
6.8
9.5
23.1
47.1
18.8
25.2
LED
435M
20.5
24.7
7.3
9.9
8.5
9.4
24.7
47.9
19.2
25.0
PRIMERA"
447M
21.1
25.3
7.4
26.5
BART
406M
18.3
21.6
8.8
24.4
45.6
18.8
23.2
BART-HED
406M
21.0
24.7
6.7
9.7
9.7
9.7
24.7
47.7
20.0
25.7
Table II: ROUGB-2 results on 10 datasets including: Multinews, WCEP, Multi-XScience (M-XSc), Rotten Tomatoes (RT), MReD, MReD+, and 4 Wikipedia domains. " results reported by Xiao et al. (2022).
Multinews
WCEP
M-XSc
RT
MReD
System
R-1/R-2/RRLL
R-1/R-2/R-LL
R-1/R-2/R-LL
R-1/R-2/R-L
R-1/RR-//R-L
BART
47.4/18.3/24.0
6/.3/21.6/34.5
31.6/7.1/16.9
1/9.4/2//
8299.9
<SS>
48. 48.7/19.6/24.9
44.5/23.3/36.1
31.4/6./177.3
26.6/966220.6
32.9/9.0/20.
KG HAR
49.3/20.2/25.2
45.1/24.0/36.8
32.3/6.9/17.2
26.7/9.6/20.6
32.7/8.8/20.2
<S> HAB + PR
39.1/20.3/25.4
44.9/23.4/36.5
32.0/6.4/17.1
27.1/9.8/20.7
33.0/8.6/19.5
<>> HAD + HAD
48.8/20.0/2..1
45.7/24.5/37.4
32.1/7.3/17.4
26.4/9.3/20.5
33.4/9.4/20
<G HAB HAD PR
00.0/21.0/5.88
46.4/24.7/37.8
12../677/
27.3/9.//11.
33.9/9.7/20.9
MReD+
McanOT
Town
Software
Film
System
R-1/R-2/R-11
R-1/R-2/R-LL
R-1/R-2/R-LL
R-1/R-2/R-LL
R-1/R-2/R-LL
BART
1//0.0
43.0/24.4/4499
59.9/45.//56
39.5/18. 8/28777
42.1/23.2/3444
33.4/9.3/206 3/20.6
42.6/24.4/35.
61.1/4.2//7.77
39,8/19.6/29.4
43.0/25.3/36.00
HAE
33.4/8. 9/20.0
42.8/24. 2/34.7
61. 6/.56/6./55.55
40.4/19.629.4
43.9/25
HAB PR
33.3/9.0/19.7
43.3/2466/351
4/55777.4/
40.0/19.6/9.4
43.7/25 6/36.2
HAR HAD
33.6/9.5/20.7
44.5/24.8/35.5
60. 1/4.5/655666.
38.8/19.2 29.0
43.3/25.3/3..
HAB HAD PR
34.0997720.77
43.5/24.7/35.2
61.9/47.7/58.1
40.5/20.0/2.77
43.8/25.7/3..
Table 12: Detailed ablation results.
sponding summary. The summary should represent all input documents as much as possible, to provide holistic perspectives. We also normalize this count over the total number of sentences in the summary and regard it as the coverage of the summary.
F Ablation
In this section, we show the detailed ablation re- sults for all 10 datasets in Table 12. It can be seen that consistent improvements can be achieved by adding the HierEnc. Next, implementing decoder attention scaling with position restart make fur- ther use of the "
" tokens achieve the best results.
For instance, if document #1 contains informa- tion A and B, document #2 contains information A and C, document #3 contains information C and D. and the summary contains information A, B, and C in 3 sentences, then the salience of the sum- mary 1.67, and the coverage is 0.67. For salience, because A is supported by 2 documents, B is sup- ported by 2 documents, and C is supported by document, it is total of 5 documents divided by 3 summary sentences. For coverage, using docu- ments #1 and #3 alone is sufficient for the summary, and hence it is documents, divided by 3 summary sentences.
G Cross-Document Standard Deviation
We provide details of the Cross-Document Stan- dard Deviation (CDS) calculation below. First, during the model's inference stage, we can eas- ily obtain the normalized cross-attention weights of each decoded token toward the source tokens in all attention heads and layers. We average these weights for all attention heads and layers for each individual token, then aggregate the weights for the tokens belonging to same document. Thus, the cross-attention for the Thth document can be calculated as:
For each pair of summaries, we first convert the score from each annotator head-to-heal score, before calculating the final average score between the 2 annotators.
WDD
Un.0: .... Wn,ks
(7)

---

System
Multinews
WCEP
M-XSc
RT
MReD
MReD+
MeanOT
Town
Software
Film
5.22
1.43
0.40
BART+HED
0.82
5.45
0.74
4.42
1.19
1.68
0.48
0.42
0.45
Table 13: Sentence-normalizedd NED for the generated summaries using entailment threshold = 0.2. 8: significantly higher with < 0.05.
where n the total number of tokens in the th document. Next, we normalize the cross- attentinn for all documents as:
each system on the same dataset, and name this metric "NED". higher NED number indicates that the summary has on average more entailed documents for each sentence. Thus, it is likely that the summary includes more crucial information that is agreed upon by more documents.
(wD.....WD.)) } softmax (w Da......
WDD. WD.]
(8)
Then, obtain the CDS score for the 7th generated token over the source documents by:
In addition, we present the NED using threshold 0.2 in Table 13. The general trend is similar using threshold 0.5 (Table 4), that our method produces summarics with higher NED over most datasets. Moreover, with a lower NED require- ment, we can see that the gap between "BART" and "BART+HED" on Multi-Xscience reduces to an insignificant level. This supports our hypoth- esis that "BART+HED" has lower NED this particular dataset due to its much more abstractive generations. The NLI model may mistakenly clas- sify the much more abstractive summaries as not entailed to any document, and may not indicate that the salience of "BART+HED" is worse than "BART".
CDS; = Std WD.....
Note that for case of denotation, we do not spec- ify the i value in Equation (7) and Equation (8), but the set of weights { [tn,00- ... wn,k) are specific to the 8th token. Finally, we can obtain single CDS value for each test instance averaging the CDS values for all generated tokens.
We can calculate for both the "BART+HED" and "BART" models over the same test input, then di- vide the CDS value of "BART+HED" by that of "BART" to obtain the relative CDS value. Our rela- tive CDS value provided Figure 3b is the average of all 200 analyzed examples.
H Content Analysis
Inspired by Laban et al. (2022), use natu- ral language entailment (NLI) model to evaluate which documents each generated sentence in the summary makes use
Specifically, the code of Laban et al. (2022) can output a score for the entailment of each sentence towards an input document from range of -1 to 1, where 1 indicates total contradiction, and I indi- cates perfect entailment. We use a threshold of 0.5%, and if one generated sentence has higher entail- ment score than the threshold with one document, we consider this sentence relies on the information of that particular document.
In this way, for cach sentence, we can count the total number of documents entailed. We aver- age this number across all summaries generated by
*Following Laban et al. (2022). we use the Hug- gingface checkpoint https://hugginggacce wan.naaaccc.om co/tals/ momeemmnlaass our NLI model.
'Although from our experience with the code, a score as low as 0.2 can correctly identify correct entailment, we use a higher threshold just to be sure not to mistakenly include any wrong documents.


Reviews:
[{"review_id": "QiClcbiQzp", "reviewer": "Reviewer_MRbH", "summary": "", "questions": "[A] PRIMERA's reported numbers seem to be lower than LED on some of the datasets. However, PRIMERA uses LED as base so why is there such a discrepancy?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review! Let us clarify and address your concerns as follows:\n\n* [Guo et al., 2023] is public on June 26, 2023, while the EMNLP deadline was June 23, 2023. Our paper is also public before the EMNLP anonymity period (May 23, 2023). We seek the reviewer's understanding that a comparison with [Guo et al. 2023] is therefore **not warranted**. Regardless, our method is not similar to Guo et al. in that they use sliding window attentions closer to Longformer and BigBird, and used 100k examples to pre-train the model for code tasks; on the other hand, ours seeks to equip the pre-trained BART with document-level attentions for the MDS domain with as little as 3K examples during fine-tuning. \n\n* In the context of MDS, our method is not only novel but also achieves the SOTA and is superior to other methods that attempt to modify the attention mechanism, such as LongT5 and LED.\n\n* PRIMERA has lower results than LED on the smaller out-of-domain datasets. This is most likely because PRIMERA's additional MDS pre-training in the news domain leads to catastrophic forgetting of the general knowledge gained during the original pre-training stage (more details in lines 437-451). This may not be much of an issue if there is still sufficient training data (i.e. Multi-Xscience with 40K examples), but for MReD with only 6K examples, it struggles to keep on par.  ​This behavior can also be observed from our BigBird results (using the checkpoint pre-trained in the scientific domain) shown in the appendix (lines 1007-1024). While BigBird has acceptable performance on large out-of-domain datasets such as Multinews (56K), it lags behind other models on small out-of-domain datasets like WCEP (10K) and Rotten Tomatoes (3K).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed approach is simple.\n2. The performance numbers are good.\n\nBased on the provided clarification:\n1. > Regardless, our method is not similar to Guo et al. in that they use sliding window attentions closer to Longformer and BigBird, and used 100k examples to pre-train the model for code tasks; on the other hand, ours seeks to equip the pre-trained BART with document-level attentions for the MDS domain with as little as 3K examples during fine-tuning. \n2. > In the context of MDS, our method is not only novel but also achieves the SOTA and is superior to other methods that attempt to modify the attention mechanism, such as LongT5 and LED.", "reasons_to_reject": "Limited Novelty, similar approaches have been applied in other context. For example, [Guo et. al., 2023] introduce the concept of bridge and memory tokens in code, where the attention across various code blocks is restricted via these tokens. However, one could consider this application to MDS as novel.\n\n# References\n[Guo et. al., 2023], LongCoder: A Long-Range Pre-trained Language Model for Code Completion, ICML 2023", "questions_to_authors": "[A] PRIMERA's reported numbers seem to be lower than LED on some of the datasets. However, PRIMERA uses LED as base so why is there such a discrepancy?", "paper_topic_and_main_contributions": "Typically Multi-Document Summarization (MDS) has been handled in two ways: A] Proposing a completely new MDS architecture or B] Use PLMs and treat MDS as SDS task by concatenating the documents. The first approach cannot leverage existing pretraining information and the second cannot leverage intricate details unique to MDS task. The paper proposes a modification to the existing encoder-decoder models that better facilitates multi-document summarization while being able to leverage existing pre-training information.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "npJ4Xu5q4S", "reviewer": "Reviewer_VCWy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review and suggestions! Let us clarify and address your concerns as follows:\n\n* _\"minimal scientific contribution in terms of methodology\"_\n\nOur methodology is a significant contribution to the realm of MDS in the following ways\n\n‒ Unlike previous work, which either trains hierarchical models from scratch or applies PLM without considering the MDS structures, ​our method readily adapts an __existing pre-trained model__ to achieve __SOTA results__ without requiring a complete re-training job or additional MDS pre-train data.\n\n‒ Therefore, our method saves computational costs of approx. __25k V100 GPU hours__ [Liu et al., 2019] (__~100k USD__ on AWS compute service) with significantly less compute and training data.\n\n‒ Moreover, in terms of technical details, we are the __first__ to scale the same decoder's cross attention according to documents, and we also differ from previous works by using encoder intra-document attention (as contrasted to the sliding window) to distill document-level representations. Our detailed analysis shows that our method leads to more __coherent__ encoder representations of the same-document tokens and __wider cross-attention coverage__ of different documents  (Sec 6.3 and 6.4). \n\n* _\"Incremental gains over SOTA results\"_\n\nIt can be observed in Tab 1 that a SOTA model may be stronger in some datasets but weaker in others, yet our method generally outperforms all models and shows non-trivial improvements (&GreaterEqual; 1 Rouge-1 point) over __any single__  previous SOTA model. Using Rouge-1 as an example:\n\n‒ Our method outperforms PRIMERA on Multinews by 2.1, on M-Xsc by 2.8, and on MReD by 4.5, MReD+ by 5.3, MeanOT by 2.0, Software by 3.9, and Film by 1.5;\n\n‒ Our method outperforms LED on Multinews by 1.0, M-XSc by 3.5, MReD by 1.1, and Film by 1.1;\n\n‒ Our method (with only __half the model size__) outperforms LongT5 on WCEP by 1.4, M-Xsc by 4.9, and Town by 1.2; \n\n* _Presentation_\n\nWe appreciate the reviewer's suggestions! \n\n‒ We will revise 052 as __\"...MDS architectures require MDS supervised dataset on the scale of 100k examples to train from scratch for reasonable performance\"__  (citing the works in lines 109-110)\n\n‒ We will revise 063 as __\" ... more suitable for SDS because ....\"__  and elaborate with reasons in lines 231-233, and lines 246-248​. We also include [Li et al., 2020] and[Liu and Lapata, 2019] for citations.\n​​\n\nReferences:\n\n[Liu et al., 2019] RoBERTa: A Robustly Optimized BERT Pretraining Approach (arxiv 2019)\n\n[Li et al., 2020] Leveraging Graph to Improve Abstractive Multi-Document Summarization (ACL 2020)\n\n[Liu and Lapata, 2019] Hierarchical Transformers for Multi-Document Summarization (ACL 2019)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Very thorough evaluation including a broad set of evaluation datasets as well as human evaluation\n- clear presentation of results and inclusion of ablation study of components \n- straightforward methodology that should be easy to replicate\n", "reasons_to_reject": "- minimal scientific contribution in terms of methodology\n- incremental gains over SOTA results", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors introduce a method for multi document summarization which enforces a hierarchy upon both encoder and decoder components of pre-trained language models.\nThe authors work with transformer-based PLMs, and the approach uses the special token 
 at the beginning of each document to provide a document-level representation in the encoder, which is used to produce a document-level weighting that is applied to partitions of the attention weights in the decoder. In addition, the encoder attention is masked so that tokens within a document can only attend to other tokens from the same document. Decoder attention is scaled by document-level weights that the authors claim reflect the relative contribution of each document to the summary. \n", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "052 much large\nthe tone of the prose is not scientific in places, especially in introduction, consider revising\ni.e. Statements such as “(which is more suitable for SDS)” - 063 should either be supported with citations or evidence or deleted"}}, {"review_id": "l5cQLCAkY1", "reviewer": "Reviewer_62Tb", "summary": "", "questions": "How it is different from a combination of PRIMER and LED? \nRouge scores computed by a same script or scores carried forward?\nHow does the proposed model perform over other metrics?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful review! Let us clarify and address your concerns as follows:\n\n* _Our method is not similar to \"combining PRIMERA and LED\"._\n\n‒ We seek to clarify that PRIMER's methodology is to use carefully designed MDS __pre-training objectives__ (lines 363-365) on top of a minimally modified LED model. So we can say that PRIMERA is close to an LED with additional MDS pre-training.\n\n‒ Our work differs from PRIMERA by directly equipping BART with document-level understandings __without additional pretraining__.\n\n‒ Moreover, both PRIMERA and LED use __sliding local attention windows__ (line 241-248, and Fig. 2b) on the encoder __without any decoder modification__, where ours induces __document-level representation__ in the encoder (Fig.1 top left), and is the __first__ to scale the same decoder's cross attention according to documents (Fig.1 top right).\n\nWe will improve our writing to highlight all of the above in the paper. \n\n* _We believe our method is as convincing as it could be for this kind of research work,_ that is:\n\n‒ We evaluate our work using __both Rouge and human evaluations__ for our results, which is consistent with the __standard process in summarization literature__ ([Guo et al., 2022], [Xiao et al., 2022], [Beltagy et al., 2020]). The results demonstrate that our method consistently outperforms previous baselines and establishes the SOTA.\n\n‒ In addition, for __all 10 datasets__, we provide detailed __attention analysis__ on the encoder and decoder to show how our method differs (Sec 6.3),  use __content analysis__ to show our method generally produces more salient summaries (Sec 6.4), and also provide a detailed __ablation analysis__ to examine each of our proposed components.\n\n* _Whether all baselines are evaluated on the same ROUGE:_\n\nYes, we run experiments and use the same evaluation script on all baselines. We explicitly highlight places where we __additionally__ include reported values from cited papers (e.g., Tab 1's caption with * sign, Appendix D lines 976-978). \n\n* _Evaluation results with BARTScore, BERTScore, and QA metric:_\n\nWe seek the reviewer's understanding that many summarization papers evaluated on Rouge alone or with some human evaluation are accepted by the NLP community ([Guo et al., 2022], [Xiao et al., 2022], [Beltagy et al., 2020]). We provide additional results for the requested metrics below, which all show that our method is the SOTA in general:\n\n\nBERTScore (rescaled)  \n|          model                |   Multinews  |   WCEP    |   M-XSc      |  RT           |   MReD    |\n| ---------------------   | -----------   | -----------  | -----------   | ----------- | ----------- | \n| LED                             |     0.248     |    0.484      |      0.060    |     0.317     |     0.110   |\n| PRIMERA                    |     0.243     |     0.477     |      0.025    |     0.330     |    0.061    |\n| BART                           | 0.222        | 0.451         | 0.077         | 0.323         | 0.138        |\n| BART+HED _(ours)_        | 0.258        | __0.495__  | __0.103__  | __0.330__ | __0.168__ |\n| BART-cnn+HED _(ours)_ | __0.262__ | 0.490         | 0.055         | 0.321         | 0.134        |\n\nBARTScore (F1)\n\n|          model                |   Multinews  |   WCEP     |   M-XSc      |  RT           |   MReD    |\n| ---------------------      | -----------   | -----------   | -----------    | ----------- | ----------- | \n| LED                              | -4.47          | -4.11        | -5.84        |   -5.01        | -5.66         |\n| PRIMERA                     | -4.45          | -4.13         | -5.93         | __-4.93__ | -5.86        |\n| BART                            | -4.64          | -4.35         | -5.84         | -4.59        | -5.66         |\n| BART+HED _(ours)_        | -4.39        |         -4.11  | __-5.46__  | -4.99         | __-5.31__ |\n| BART-cnn+HED _(ours)_ | __-4.38__   | __-4.08__ | -5.97         | -5.06         | -5.53         |\n\nQuestEval: \n\nUnfortunately, this metric takes an excessively long time to run, as it uses 2 generation models. Multinews and Multi-Xscience are the 2 largest datasets in our study. Each dataset requires 10+ GPU hours for each test output file of 5.5k+ examples, so 5 models with 3 runs require 150+ GPU hours per dataset, which is infeasible to complete in a short time.\n\n|          model                |   WCEP      |  RT           |   MReD    |\n| ---------------------      | -----------   | ----------- | ----------- | \n| LED                              | 0.481          |   0.370       | 0.319       |\n| PRIMERA                     | 0.478          |     0.364     | 0.309        |\n| BART                           | 0.462          | 0.360        | 0.314       |\n| BART+HED _(ours)_      | __0.484__   | __0.371__   | __0.324__ |\n| BART-cnn+HED _(ours)_  | 0.483        | __0.371__    |  0.318      |\n\nReferences:\n\n[Guo et al., 2022] LongT5: Efficient Text-To-Text Transformer for Long Sequences (NAACL 2022)\n\n[Xiao et al., 2022] PRIMER: Pyramid-based Masked Sentence Pre-training for Multi-document Summarization (ACL 2022)\n\n[Beltagy et al., 2020] Longformer: The Long-Document Transformer (arxiv 2020)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a novel method that enforces hierarchy on both the encoder and decoder, aiming to better leverage PLMs for facilitating interactions between multiple documents in the context of MDS.\nThe paper claims to have achieved promising results across 10 MDS benchmarks from various domains. The method reportedly outperforms or at least competes well with the best-performing models in the field, including those with additional MDS pre-training or with more model parameters. \nThe arguments are sound and the paper is written well.", "reasons_to_reject": "The methodology looks like an extension of PRIMERA only. The method is not very convincing.\nIt is also unclear if all baselines and the proposed architecture are evaluated on same Rouge metric or reported results from other papers are carried forward.\nHow does your system perform over other metrics? for example BERT or BARTScore, QA based metrics? Rouge alone can not suffice as a improvement.", "questions_to_authors": "How it is different from a combination of PRIMER and LED? \nRouge scores computed by a same script or scores carried forward?\nHow does the proposed model perform over other metrics?", "paper_topic_and_main_contributions": "The paper proposes a new model which uses Hierarchical context for the task of MDS.The paper highlights the limitations of existing approaches, where some either introduce new MDS architectures or naively treat MDS as a concatenated SDS task. The former might not leverage the full potential of pre-training, while the latter may not effectively capture the intricate inter-document relationships inherent to MDS.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new model which uses Hierarchical context for the task of MDS. The hierarchical schema is designed not only for encoder, where one reviewer pointed with limited novelty, but also for decoder which is interesting. Based on the performance, almost all the reviewers agree that the method competes well with the best-performing models in the field, including those with additional MDS pre-training or with more model parameters. Therefore, I would suggest the paper as Findings in the conference, or  Main conference is also acceptable.
Decision: Accept-Findings