Venue: NAACL 2022

# Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness

Yun-Zhu Song and Yi-Syuan Chen and Hong-Han Shuai National Yang Ming Chiao Tung University, Taiwan {yunzhusong.eed07g,yscalen.eed09g,hhshuai}@nctu.edu.tw

###### Abstract

A notable challenge in Multi-Document Summarization (MDS) is the extremely-long length of the input. In this paper, we present an extract-then-abstract Transformer framework to overcome the problem. Specifically, we leverage pre-trained language models to construct a hierarchical extractor for salient sentence selection across documents and an abstractor for rewriting the selected contents as summaries. However, learning such a framework is challenging since the optimal contents for the abstractor are generally unknown. Previous works typically create _pseudo extraction oracle_ to enable the supervised learning for both the extractor and the abstractor. Nevertheless, we argue that the performance of such methods could be restricted due to the insufficient information for prediction and inconsistent objectives between training and testing. To this end, we propose a loss weighting mechanism that makes the model aware of the unequal importance for the sentences not in the pseudo extraction oracle, and leverage the fine-tuned abstractor to generate summary references as auxiliary signals for learning the extractor. Moreover, we propose a reinforcement learning method that can efficiently apply to the extractor for harmonizing the optimization between training and testing. Experiment results show that our framework substantially outperforms strong baselines with comparable model sizes and achieves the best results on the Multi-News, Multi-XScience, and WikiCatSum corpora.1

Footnote 1: The implementation code and trained models are available at [https://github.com/yunzhusong/NAACL2022-REFLECT](https://github.com/yunzhusong/NAACL2022-REFLECT)

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1667 - 1681

## 1 Introduction

Neural multi-document summarization has drawn a lot of attention due to the wide applications, e.g., Wikipedia generation Liu et al. (2018), news digest Fabbri et al. (2019), or related-work section generation Lu et al. (2020). Through concatenating the document clusters, it is applicable to adopt the approaches from single-document summarization Zhang et al. (2020); Fabbri et al. (2019) under multi-document settings. However, the input length from multiple documents is typically long, which might be computationally infeasible for Transformer-based models Vaswani et al. (2017). One possible solution is to set a fixed length and truncate the input sequence. However, the truncation leads to information loss and performance drop. Moreover, even if the model can take such a long sequence as input, the attentions could be dispersed over long sequences Yang et al. (2018) and further degrade the performance Jin et al. (2020); Liu and Lapata (2019); Cohan et al. (2018).

To tackle the long sequence problem, another possible solution is to leverage the extract-then-abstract architecture in single document summarization Pilault et al. (2020); Chen and Bansal (2018); Gehrmann et al. (2018). In such an architecture, an extractor first selects salient contents from the documents, and an abstractor is applied to rewrite the selected contents to coherent summaries. However, several issues arise for directly applying the procedure in MDS. 1) _Long-sequence problem for the extractor_. Although the input length for the abstractor can be controlled through the content selection and thus enables the usage of Transformer models Pilault et al. (2020), it is still inevitable for the extractor to process the complete input documents. Previous methods mainly use LSTM-based extractors Pilault et al. (2020); Chen and Bansal (2018); Gehrmann et al. (2018). However, this fails to leverage knowledge of pre-trained language models. 2) _Suboptimal pseudo oracles_. Most summarization corpora do not contain the oracle for extraction. As an alternative, previous works typically generate _pseudo extraction oracle_, or simply called _pseudo oracle_, through a greedy process Liu and Lapata (2019); Chen and Bansal (2018). Specifically, for each iteration in the process,

candidate sentences are individually concatenated with previously-selected sentences and compute the ROUGE Lin (2004) scores with the human-written summaries. The top-scored sentence is iteratively selected until no sentence could further improve the ROUGE score. In practice, there are different ways for scoring. For examples, Liu and Lapata (2019b) use the average of ROUGE-1 and ROUGE-2 F1 scores while Chen and Bansal (2018) use ROUGE-L recall. The variants of design could cause the extractor to behave differently. In our study, we also found that using the ROUGE precision metric leads to much less extraction than using the recall metric. This implies that the pseudo oracles are suboptimal, and learning the extractor fully relies on the pseudo oracles could restrict the performance. How to alleviate the negative effects of pseudo oracles remains open. 3) _Insufficient information for the extractor_. Even if the pseudo oracles are good enough to train the abstractor well, learning a precise extractor is still challenging. The problem lies in that a pseudo oracle is derived from a specific summary. However, there are potentially multiple valid summaries given the documents. To select salient sentences, the extractor is required to implicitly infer the underlying summary used for oracle construction, which is difficult due to the lack of evidence. 4) _Intconsistent objectives for the extractor_. With pseudo oracles, the extractor is learned to select a set of sentences that has high lexical similarity to the summaries without redundancy. However, the goal of the extractor in test-time is to provide inputs for the abstractor that non-overlapping lexical may still be valuable. In other words, the objective for the extractor in training is inconsistent with the one in testing.

To address these issues, we propose the **RE**ferenced **FLE**xible Extraction with **C**redi**T-Awareness (REFLECT) for MDS. For the first problem, we propose a Transformer-based hierarchical extractor that contains the token- and sentence-level feature encoders. Both the encoders are initialized with pre-trained language models to utilize the pretext knowledge. For the second problem, we propose Pseudo Oracle Relaxation (POR) to render the model aware of the unequal importance for the non-oracle sentences. This mechanism encourages the model to emphasize the precision for critical sentences with either high or low lexical similarity to the summaries, and avoids the confusion arising from the different labels for similar sentences. For the third problem, we propose Summary Referencing (SR) to leverage the fine-tuned abstractor for providing additional learning signals to evidence extraction prediction. The summary reference serves as an approximation for the human-written summary while being able to generalize for testing. For the fourth problem, we propose Credit-Aware Self-Critic (CASC) learning to fine-tune for matching the objective between training and testing. Different from previous methods that assign an identical reward for all actions, we reallocate the rewards based on the impacts of actor explorations.

The contributions are summarized as follows:

* We leverage pre-trained language models to propose an extract-then-abstract framework, which contains a hierarchical extractor that efficiently handles long inputs while utilizing pretext knowledge.
* We investigate the problems for typical learning paradigms of the extractor and propose a framework, named REFLECT, to further improve the extractor performance. The studies on pseudo oracles also provide valuable insights for extract-then-abstract frameworks.
* Experimental results on Multi-News, Multi-XScience, and WikiCatSum corpora demonstrate that REFLECT outperforms the state-of-the-art models with comparable sizes.

## 2 Related Work

Early attempts for MDS focus on extracting sentences through statistical methods Goldstein et al. (2000); Erkan and Radev (2004); Wan and Yang (2006); Zhou et al. (2008). For example, Goldstein et al. (2000) extend Maximal Marginal Relevance (MMR) method to select sentences that are relevant to the query and novel across different documents. Erkan and Radev (2004) leverage sentence relations with graph structures that represent pairwise sentence similarities, and apply PageRank Page et al. (1999) algorithm to extract sentences given the query document. However, extractive methods often suffer the coherence problem Wu and Hu (2018). Therefore, instead of directly extracting sentences from the articles, abstractive methods that can rewrite the articles achieve great success with the advantages of large annotated corpora Pang et al. (2021); Zhou et al. (2021); Liu et al. (2021a); Zhong et al. (2020); Li et al. (2020); Liu and Lapata (2019a) 

Directly operating on the long inputs in MDS often leads to model degradation Jin et al. (2020). One of the promising solutions is to leverage the extract-then-abstract architectures in single document summarization Chen and Bansal (2018); Gehrmann et al. (2018). For example, Gehrmann et al. (2018) propose an LSTM-based word-level extractor to choose phrases from the document, and apply an abstractor with the copy mechanism to generate summaries given the selected contents. Pilault et al. (2020) also apply an LSTM-based sentence extractor to select contents, but further leverage a Transformer decoder to improve the performance.

Although introducing Transformer architectures provides improvements, the input length is typically limited according to the computation and memory overhead. A recent line of studies has been proposed to alleviate such problems Brazinskas et al. (2021); Beltagy et al. (2020); Liu and Lapata (2019a); Jin et al. (2020). Liu and Lapata (2019a) propose to rank the candidate input paragraphs, and only concatenate the top few as the inputs for the abstractor. Jin et al. (2020) encode multiple documents in different granularity including token-level, sentence-level, and document-level. Pasunuru et al. (2021) design a graph encoder parallel with standard encoder to provide inter-document information and integrate the pre-trained BART Lewis et al. (2020) with local attention mechanism Beltagy et al. (2020) to overcome the long input problem. Brazilnskas et al. (2021) apply policy gradient optimization to learn a train-time selector using lexical features pre-computed from source texts and gold summaries as inputs. Due to the lack of gold summaries in testing, a test-time selector is learned using lexical features solely from source texts as inputs and predictions from the train-time selector as learning targets. Comparatively, our methods thoroughly leverage the language models with an extract-and-abstract framework that enjoys the full capacities for both the extractor and the abstractor. The introduced summary referencing further improves the extractor with additional signals in both train- and test-phase.

## 3 Methodology

Utilizing large pre-trained language models brings great benefits for text generation problems. However, the input length of multi-document summarization is typically long, which makes large pre-trained language models, _e.g._, Transformer-based models, inefficient for processing. To match the length constraint, document-level truncation Fabbri et al. (2019) has been widely-used. However, the truncation could inevitably cause information loss and degrade the performance. To solve the problem, we propose to leverage the extract-then-abstract framework. However, as described in the introduction, four challenges arise while learning such a framework. We first present our architecture to solve the long sequence problem and elaborate on the proposed methods to tackle the challenges.

### Hierarchical Summarizer

Figure 1 illustrates the proposed REFLECT framework. Specifically, based on the Transformer Vaswani et al. (2017), our architecture contains a hierarchical extractor (H-EXT) and an abstractor (ABS). The hierarchical extractor is composed of a token-level encoder (T-ENC), a sentence-level encoder (S-ENC), and a sentence selector (SS). Considering a training example \((x,y)\), \(x=\{x_{1},x_{2},...,x_{N}\}\) is the concatenation of multiple documents that jointly consists of \(N\) complete sentences and \(y\) is the corresponding human-written summary. Let \(M\) denote the allowed input length in Transformer-based models, we split all sentences into \(K\) disjoint sets \(\{h_{k}\}^{K}_{k=1}\) such that each set \(h_{k}\) consists of sentences whose total number of tokens matches the length constraint \(M\). The hierarchical extractor then predicts a set containing the indices of selected sentences (denoted by \(\hat{e}\)), which is used to retrieve salient contents from \(x\) for the abstractor to produce summaries. Specifically, the hierarchical extractor can be expressed as follows.

\[\begin{split}\hat{e}&=\text{H-EXT}(h)\\ &=\text{SS}(\text{S-ENC}(\bar{\oplus}\{\text{T-ENC}(h_{k})\}^{ K}_{k=1})),\end{split} \tag{1}\]

where \(\bar{\oplus}\) is the operation of taking average for hidden states of tokens within sentences and concatenating the averaged results to obtain the sentence-level representations. The hierarchical summarizer (H-SUM) can then be expressed as:

\[\text{H-SUM}(h)=\text{ABS}(\oplus\{w_{x_{i}}|i\in\hat{e}\}), \tag{2}\]

where \(\oplus\) is the concatenation operation.

### Pseudo Oracle Relaxation

The sentence selection process of the extractor is typically formulated as a supervised classification 

problem. However, most summarization corpora do not provide such annotations. As an alternative, it is common to create pseudo oracles through a greedy process with the human-written summaries Liu and Lapata (2019); Chen and Bansal (2018). However, such methods only provide suboptimal solutions since they are limited by the design of the greedy algorithm. The underlying patterns of true oracles could be too complicated to be designed precisely. The performance of the extractor is thus restricted since typical Maximal Likelihood Estimation (MLE) methods fully depend on pseudo oracles. For example, consider a case that there are only three sentences \(x_{a}\), \(x_{b}\) and \(x_{c}\), and the ROUGE scores between sentences and summary are ranked as \(x_{a}>x_{b}\gg x_{c}\). Assume that only \(x_{a}\) is chosen as the pseudo oracle. If the extractor further selects one additional sentence for improving the results, it is expected that the combination of \((x_{a},x_{b})\) could be better than \((x_{a},x_{c})\) generally. However, MLE-based methods do not consider such discrepancy according to the pseudo oracle.

Therefore, we propose a loss weighting mechanism to further consider the lexical similarity for the non-oracle sentences during the learning process. Consider an input \(x=\{x_{1},x_{2},...,x_{N}\}\) and the corresponding pseudo oracles \(e\), the input sentences can be separated into two sets of pseudo oracles and non-pseudo oracles, \(S=\{x_{i}|i\in e\}\) and \(S^{c}=\{x_{i}|i\in e^{c}\}\), respectively. We maintain the loss weights for sentences in \(S\) as one, while modifying the loss weights for the sentences in \(S^{c}\) as follows:

\[w_{i}=(1-\text{ROUGE-1}(x_{i},y))^{\gamma}, \tag{3}\]

where \(\gamma\) is a hyper-parameter controlling the weighting scales. To stabilize the training, we further shift loss weights of \(S^{c}\) such that the maximum value is one. This weighting mechanism emphasizes the predictions of both pseudo oracles and the low-ROUGE sentences, which have more impact on the performance. It relaxes the constraint from the binarized oracles to make the model aware of the differences between candidate sentences. The objective for the hierarchical extractor can then be written as:

\[L=-\sum_{x_{i}\in x}w_{i}\log(\frac{exp(z_{i}^{1s(i)})}{exp(z_{i}^{0})+exp(z_{ i}^{1})}), \tag{4}\]

where \(z_{i}^{0}\) and \(z_{i}^{1}\) are the binary logits for the \(i\)-th sentence, and \(\mathbf{1}_{S}(i)\) is the indicator function of oracle sentences \(S\).

### Summary Referencing

The pseudo oracle relaxation described in Section 3.2 makes the extractor focus on the sentences that are more important in terms of the lexical similarity. In other words, it considers the discrepancy between non-oracle sentences instead of treating them equally. However, the ambiguity could still exist between the oracle and non-oracle sentences. For example, consider a case where there are two sentences \(x_{a}\) and \(x_{b}\), and only \(x_{a}\) is chosen as the oracle during the greedy process. The ROUGE scores between the sentences and the summaries are roughly the same for the two sentences. Learning with such oracles may confuse the model since the positive and negative sentences are similar but with a large loss difference. Therefore, we propose to provide summary references for the extractor to further reason the selection. Since the summary is only available in the training stage, we leverage a fine-tuned abstractor to provide such summary references. With such a mechanism, the operations

Figure 1: Framework of REFLECT. The illustration of proposed Pseudo Oracle Relaxation (POR), Summary Referencing (SR), and Credit-Aware Self-Critic (CASC) learning are indicated with bold fonts.

 

of the hierarchical extractor are further revised as:

\[r =\text{ABS}(\oplus\{w_{x_{i}}|i\in\hat{e}\}), \tag{5}\] \[\hat{e} =\text{H-EXT}(h,r)\] \[=\text{SC}(\text{S-ENC}(\bar{\oplus}(\{\text{T-ENC}(r)\}+\] \[\qquad\qquad\qquad\quad\{\text{T-ENC}(h_{k})\}^{K}_{k=1}))).\]

Compared to directly using human-written summaries as references when training, which may cause the extractor highly rely on the reference signal, the generation results from the abstractor could serve as good approximations and provide the generalization from training to testing.

### Credit-Aware Self-Critic Learning

With the objective of maximal likelihood estimation, the extractor is learned to select sentences that are jointly have high lexical similarity to the human-written summary. However, in extract-then-abstract frameworks, the required objective of extractor is to provide salient contents that can maximize the generation quality after rewritten by the abstractor. In other words, the objective for the extractor is inconsistent between training and testing. Therefore, we further propose a reinforcement learning method that can directly optimize with the test-time objective to bridge the gaps.

Specifically, we formulate the extraction of sentences as a single-round Combinatorial Multi-Armed Bandit (CMAB) problem Chen et al. (2016). A general CMAB problem can be modeled as a tuple \((E,\mathcal{F},P,R)\), where \(E=\{1,,2...,N\}\) is a set of \(N\) arms, \(\mathcal{F}\subseteq 2^{E}\) is a set of subsets of \(E\), \(P\) is a probability distribution over \([0,1]^{N}\), and \(R\) is a reward function defined on \([0,1]^{N}\times\mathcal{F}\). At the \(t\)-th round, the agent pulls a subset of arms \(S^{t}\in\mathcal{F}\), and produce stochastic outcomes \(M^{t}=(M^{t}_{1},M^{t}_{2},...,M^{t}_{N})\sim P\), where \(M^{t}_{i}\) is the outcome of \(i\)-th arm. With a realization of outcomes \(m=(m_{1},m_{2},...,m_{N})\), the agents then receive a reward of \(R(m,S)\). The goal of the agent is to maximize the expected cumulative reward in \(T\) rounds, which is \(\mathbb{E}_{\{M^{t}\}_{t=1}^{T}}[\sum_{t=1}^{T}R(M^{t},S^{t})]\).

For the sentence extraction problem, we consider each sentence as an arm, and pull multiple arms (i.e., select multiple sentences) only for a single round. We solve this problem with the self-critic learning framework Rennie et al. (2017). Specifically, we consider the sentence-level encoder with the sentence selector that as the agent (jointly parameterized by \(\theta\)). The agent takes the sentence representations \(x=\{x_{1},x_{2},...,x_{N}\}\) from the token-level encoder, and selects a set of sentences \(S\) according to a policy \(\pi_{\theta}(\cdot)\) as:

\[m_{i} \sim\text{Bern}(\frac{exp(z_{i}^{1})}{exp(z_{i}^{0})+exp(z_{i}^{ 1})}), \tag{6}\] \[S =\pi_{\theta}(x)=\{i|m_{i}=1\},\]

where \(z_{i}^{0}\) and \(z_{i}^{1}\) are the binary logits for the \(i\)-th sentence. To reduce the variance during learning, we introduce a baseline term through another policy \(\tilde{\pi}_{\theta}\) as:

\[\tilde{m}_{i}=\frac{exp(z_{i}^{1})}{exp(z_{i}^{0})+exp(z_{i}^{1})}, \tag{7}\] \[\tilde{S}=\tilde{\pi}_{\theta}(x)=\{i|\tilde{m}_{i}>0.5\},\]

which is essentially a greedy policy as the taken actions are not explored. The two policies are derived from the agents with the same parameters, and the learning process is thus self-critical. We define the reward as the performance of generation from an abstractor using the selected sentences \(S\) as the input, _i.e._,

\[R(S)=\text{ROUEG-L}(\text{ABS}(S),y), \tag{8}\]

where \(S\) is produced from the outcomes \(m\). The advantage \(a\) can be written as:

\[a=R(S)-R(\tilde{S}). \tag{9}\]

We then optimize the agent through gradient descent with the objective function as:

\[L=-\sum_{i\in E}a\log(\frac{exp(z_{i}^{\mathbf{1}_{S}(i)})}{exp(z_{i}^{0})+ exp(z_{i}^{1})}), \tag{10}\]

where \(\mathbf{1}_{S}(i)\) is the indicator function of selected sentence set \(S\). However, in such a learning objective, the advantages are applied uniformly to update all actions, which could make learning difficult since the sentences number is typically large in our setting. Thus, we propose to specifically credit the advantages to the selections that are distinct between policies \(\pi_{\theta}\) and \(\tilde{\pi}_{\theta}\), and the objective function \(L^{credit}\) can be written as:

\[L^{credit}=\\ -\sum_{i\in E}\mathbf{1}_{S\cap\tilde{S}}(i)a\log(\frac{exp(z_{i}^ {\mathbf{1}_{S}(i)})}{exp(z_{i}^{0})+exp(z_{i}^{1})}). \tag{11}\]

Different with previous work Chen and Bansal (2018) that formulates the sentence selections as a sequence of decisions and uses LSTM-based agent for learning, our formulation enables the usage of Transformer-based models for a better efficiency 



## 4 Experimental Results

### Settings

We compare REFLECT with several strong baselines Liu and Lapata (2019); Gehrmann et al. (2018); Fabbri et al. (2019); Perez-Beltrachini and Lapata (2021); Li et al. (2020); Liu et al. (2021); Zhong et al. (2020); Zhang et al. (2020); Pasunuru et al. (2021) on **Multi-News**Fabbri et al. (2019), **Multi-XScience**Lu et al. (2020) and **WikiCatSum**Perez-Beltrachini et al. (2019) corpora, derived from news, academic domains and Wikipedia, respectively. Due to space limit, the results of Multi-XScience and WikiCatSum are provided in the Appendix A. For evaluation, we use ROUGE F1 metrics Lin (2004), BERTScore Zhang et al. (2020), and factual consistency evaluated with FactCC Kryscinski et al. (2020) to investigate performance from different perspectives. In our architecture, the hierarchical extractor is initialized by RoBERTa-base Liu et al. (2020) containing 12 attention layers. We take the first \(l\) layers and the rest layers (12-\(l\)) as the token- and sentence-level encoder, respectively. The input length limit \(M\) is set to 512. The abstractor is a sequence-to-sequence model initialized by BART Lewis et al. (2020). To make the CASC computationally efficient, we use the BART-base as the abstractor that provides rewards for the extractor during training, while exploiting the BART-large in the testing. We provide more implementation details in Appendix C.

### Main Results

Table 1 summarizes the performance of REFLECT with various baselines on Multi-News corpus with the rows sorted by the average of ROUGE 1, 2, and L scores Lin (2004). 2 The results demonstrate that REFLECT outperforms all baselines. PG-BRNN and Hi-MAP both use an LSTM-based point generator for sentence selection and apply a decoder for the generation. However, the performance is limited since LSTM could still suffer long-term dependency problem Trinh et al. (2018). Hierarchical-Transformer (row 1) further uses an LSTM-based ranker to select paragraphs through predicting the ROUGE scores with the summaries for each paragraph, and uses a Transformer-based hierarchical encoder to capture the local and global information. However, the estimation of the ROUGE scores could be difficult since there are multiple valid summaries. In contrast, REFLECT further applies pre-trained language models to extract the hierarchical features, and the proposed SR method explicitly provides the extractor with the evidence for selecting more salient contents to be rewritten from the abstractor. Moreover, GraphSum (row 5 & 6) leverages graph structures to explore the relations of paragraphs in the encoder, and Highlight-Transformer (row 7) specifically assigns higher attention weights for key phrases. Both the methods implicitly provide additional selective information for the decoder. REFLECT realizes such a content selection pro

\begin{table}
\begin{tabular}{p{34.1pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt} p{28.5pt}} \hline \hline Model & ROUGE-1 & ROUGE-2 & ROUGE-L & Average & Average Improvement \\ \hline Hierarchical-Transformer Liu and Lapata (2019) & 42.36 & 15.27 & 22.08 & 26.57 & +17.91\% \\ PG-BRNN Gehrmann et al. (2018) & 43.77 & 15.38 & 20.84 & 26.66 & +17.52\% \\ Hi-MAP Fabbri et al. (2019) & 44.17 & 16.05 & 21.38 & 27.20 & +15.18\% \\ CT-DPP Perez-Beltrachini and Lapata (2021) & 45.84 & 15.94 & 21.02 & 27.60 & +13.51\% \\ Graphsum Li et al. (2020) & 45.02 & 16.69 & 22.50 & 28.07 & +11.61\% \\ GraphSum + RoBERTa Li et al. (2020) & 45.87 & 17.56 & 23.39 & 28.94 & +8.26\% \\ Highlight-Transformer Liu et al. (2021) & 44.62 & 15.57 & - & - & - \\ MatchSum Zhong et al. (2020) & 46.20 & 16.51 & - & - & - \\ PEGASUS Zhang et al. (2020) & 47.52 & 18.72 & **24.91** & 30.38 & +3.13\% \\ BART-Long Pasunuru et al. (2021) & 48.54 & 18.56 & 23.78 & 30.29 & +3.43\% \\ BART-Long-Graph Pasunuru et al. (2021) & 49.24 & 18.99 & 23.97 & 30.73 & +1.95\% \\ \hline REFLECT (MLE) & 48.16\(\pm\)0.01 & 18.87\(\pm\)0.01 & 23.78\(\pm\)0.17 & 30.27 & +3.50\% \\ REFLECT (CASC) & **49.27\(\pm\)**0.06 & **19.96\(\pm\)**0.03 & 24.76\(\pm\)0.09 & **31.33** & - \\ \hline \hline \end{tabular}
\end{table}
Table 1: Performance of REFLECT with various baselines on Multi-News corpus. The results of Hierarchical Transformer, HiMAP, and PG-BRNN are copied from Li et al. (2020). The rest baseline results are from the original papers. Best ROUGE scores are bolded. Performance of REFLECT is reported with five runs.



cess in the data-level through the two-stage design. The proposed POR and SR methods alleviate the negative effects from the pseudo oracles, and thus provide complete information for the abstractor to rewrite. The results of PEGASUS (row 9) show the benefits of pre-trained sequence-to-sequence language models, which are also leveraged in REFLECT. The BART-Long-Graph (row 10 & 11) further combines the previous advantages on graph structure and pre-trained language models to achieve strong performance. REFLECT outperforms it in terms of the ROUGE-2 and ROUGE-L scores especially, probably due to the local attention mechanism used in BART-Long-Graph. Although such a design reduces the complexity to accommodate longer sequences, the trade-off for the attention capacity still restricts the model from generating more coherent summaries. Due to the space limit, please refer to Appendix A for results on Multi-XScience and WikiCatSum.

In addition to the evaluation of lexical overlaps, Table 2 shows the performance of semantic and factual consistency, which are important for summarization applications that will influence the public. The results demonstrate that REFLECT can improve factual consistency through the hierarchical architecture, _i.e._, the architecture enables selection of useful information from multiple documents, while still maintaining the semantics in generations.

### Ablations and Analyses

In this subsection, we perform ablations and analyses for REFLECT, and more results can be found in Appendix B.

**Effect of Pseudo Oracle Relaxation (POR).** We start the discussions with the results using MLE objectives. As shown in Table 3, when applying POR to the base method (row 1 & 2), the extraction recall increases while the precision decreases. Pseudo oracles generally contain sentences with high ROUGE scores to the summaries, and POR encourages the selection for such sentences by assigning smaller loss weights to increase the recall. From the abstraction performance, it shows that such selection preference could provide more concise information (even though not in pseudo oracles) as the input for improvements. Under the combination of SR (row 3 & 4), the precision is further improved and achieves a higher overall F1 score due to more information for selection. However, the abstraction results show that the performance is actually inferior to the one only using SR, which meets our suggestion that the training objective of the extractor is inconsistent in the test-time. The proposed CASC learning further overcomes such problems and improves the performance by integrating POR and SR (row 6 & 10).

**Effect of Summary Referencing (SR).** We first investigate the MLE results. With SR, the extractor is learned to select sentences given the approximations of summaries generated from the abstractor. The extraction recall is thus enhanced with such references. However, the trade-off between recall and precision still exists due to the discrepancy between ground-truth summaries and generation results (row 1 & 3). Although ground-truth can be used as references to increase extraction performance during training, it fails to generalize in testing due to the distributional difference between them. The abstraction performance also shows SR makes significant improvements (row 3) and also benefits from the combination with CASC (row 6 & 8).

**Effect of Credit-Aware Self-Critic (CASC) Learning.** Consider the case that applying CASC with POR and SR respectively (row 5 & row 6, row 7 & row 8), the results demonstrate that CASC substantially outperforms the Self-Critic (SC) learning methods, even when combining the usage of POR and SR (row 9 & row 10). We could further investigate the difference between MLE and RL methods through the visualization in Figure 2. It shows that, although having lower extraction performance, the RL methods can improve over the MLE methods in final abstraction performance due to the consistent objective between training and testing. CASC further improves over SC through explicitly assigning advantage to the actions that have the credits for exploration. Such design is critical for the long input sequences in multi-document summarization and can potentially be applied to other applications.

\begin{table}
\begin{tabular}{l c c} \hline \hline Model & BERTScore & Factual Consistency \\ \hline BART-base & 0.870 & 79.7 \\ CTF-DPP & 0.852 & 81.9 \\ \hline REFLECT & **0.871** & **82.2** \\ \hline \hline \end{tabular}
\end{table}
Table 2: BERTScore and factual consistency evaluated with FactCC on Multi-News corpus.

 



### Effects of Summary Reference

REFLECT uses the BART-large fine-tuned with truncated articles (BART-large-A) to provide summary references for both training and testing. Here, we study the effect of applying different summary referencing strategies in test-time. To investigate such effect across models, we leverage PEGASUS Zhang et al. (2020) and fine-tune it with Multi-News corpus to produce summary references. Furthermore, we also conduct experiments that use the BART-large fine-tuned with pseudo oracle sentences (BART-large-O) and the ground-truth summaries. _Note that these two methods are prohibited in practical usage since they require annotations on testing data._ The results in Table 4 manifest that using references from another model that is different from the training one does not significantly affect the performance. For the case using oracle inputs, the performance is similar to the ones directly using articles. This suggests that our method is stable for the practical scenario. The result of the ground-truth also shows that the reference quality could still affect the performance, which is left as a future research direction.

## 5 Conclusion

In this work, we present an effective extract-then-abstract framework, named REFLECT, for MDS. We utilize large pre-trained language models to construct a hierarchical extractor for solving the long sequence problem. Moreover, we investigate current learning paradigms for such frameworks and find that entirely relying on the pseudo oracles produced via a greedy process could hinder the performance. Therefore, we propose three corresponding techniques (POR, SR, and CASC) to overcome the issues. The experimental results not only show that REFLECT outperforms the state-of-the-art models on Multi-News, Multi-XScience, and WikiCatSum corpora, but also demonstrate that bridging the gap between training and testing is significant. Also, we present extensive studies to motivate more investigations. Finally, we consider further exploring the interactions between the extractor and abstractor, including iteratively providing a better reference or reusing the extraction predictions as the training signals for the abstractor, and study how to build a more efficient reference for providing the extraction evidence.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Reference & R-1 & R-2 & R-L & Average \\ Source & & & & \\ \hline PEGASUS & 49.28 & 19.99 & 24.84 & 34.78 \\ BART-large-A & 49.29 & 19.99 & 24.83 & 34.79 \\ \hline BART-large-O & 49.29 & 19.98 & 24.81 & 34.78 \\ Ground-Truth & 49.38 & 20.05 & 24.88 & 34.85 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of different summary referencing strategies in REFLECT. We also present the results of BART models trained by truncated articles (A) and pseudo oracles (O). ROUGE score is abbreviated as \(R\).

Figure 2: Comparisons of MLE and RL results. For clarity, the displayed range for average ROUGE scores and extraction F1 scores are limited in [33,35] and [0.5,0.59], respectively.





## Acknowledgements

This work is supported in part by the Ministry of Science and Technology (MOST) of Taiwan under the grants MOST-109-2221-E-009-114-MY3, MOST-110-2221-E-001-001 and MOST-110-2221-E-A49-164. This work was also supported by the Higher Education Sprout Project of the National Yang Ming Chiao Tung University and Ministry of Education (MOE), Taiwan. We are grateful to the National Center for High-performance Computing for computer time and facilities.

## References

* Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. _arXiv preprint arXiv:2004.05150_.
* Brazinskas et al. (2021) Arthur Brazinskas, Mirella Lapata, and Ivan Titov. 2021. Learning opinion summarizers by selecting informative reviews. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 9424-9442, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Chen et al. (2016) Wei Chen, Wei Hu, Fu Li, Jian Li, Yu Liu, and Pinyan Lu. 2016. Combinatorial multi-armed bandit with general reward functions. In _Advances in Neural Information Processing Systems_, volume 29. Curran Associates, Inc.
* Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 675-686, Melbourne, Australia. Association for Computational Linguistics.
* Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 615-621, New Orleans, Louisiana. Association for Computational Linguistics.
* Erkan and Radev (2004) Gunes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. _J. Artif. Int. Res._, 22(1):457-479.
* Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
* Gehring et al. (2017) Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In _Proceedings of the 34th International Conference on Machine Learning_, volume 70 of _Proceedings of Machine Learning Research_, pages 1243-1252. PMLR.
* Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
* Volume 4_, NAACL-ANLP-AutoSum '00, page 40-48, USA. Association for Computational Linguistics.
* Jin et al. (2020) Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020. Multi-granularity interaction network for extractive and abstractive multi-document summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6244-6254, Online. Association for Computational Linguistics.
* Kryscinski et al. (2020) Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9332-9346, Online. Association for Computational Linguistics.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li et al. (2020) Wei Li, Xinyan Xiao, Jiachen Liu, Hua Wu, Haifeng Wang, and Junping Du. 2020. Leveraging graph to improve abstractive multi-document summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6232-6243, Online. Association for Computational Linguistics.
* Lin (2004) Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In _Text Summarization Branches Out_, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
* Liu et al. (2018) Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. In _International Conference on Learning Representations_

Shuaiqi Liu, Jianhong Cao, Ruosong Yang, and Zhiyuan Wen. 2021a. Highlight-transformer: Leveraging key phrase aware attention to improve abstractive multi-document summarization. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 5021-5027, Online. Association for Computational Linguistics.
* Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5070-5081, Florence, Italy. Association for Computational Linguistics.
* Liu and Lapata (2019b) Yang Liu and Mirella Lapata. 2019b. Text summarization with pretrained encoders. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
* Liu et al. (2021b) Yang Liu, Sheng Shen, and Mirella Lapata. 2021b. Noisy self-knowledge distillation for text summarization. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 692-703, Online. Association for Computational Linguistics.
* Liu et al. (2020) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2020. RoBERTa: A robustly optimized BERT pretraining approach.
* Lu et al. (2020) Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8068-8074, Online. Association for Computational Linguistics.
* Page et al. (1999) Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. 1999. The pagerank citation ranking: Bringing order to the web. Technical Report 1999-66, Stanford InfoLab.
* Pang et al. (2021) Richard Yuanzhe Pang, Adam Lelkes, Vinh Tran, and Cong Yu. 2021. AgreeSum: Agreement-oriented multi-document summarization. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3377-3391, Online. Association for Computational Linguistics.
* Pasunuru et al. (2021) Ramakanth Pasunuru, Mengwen Liu, Mohit Bansal, Sujith Ravi, and Markus Dreyer. 2021. Efficiently summarizing text and graph encodings of multi-document clusters. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4768-4779, Online. Association for Computational Linguistics.
* Perez-Beltrachini and Lapata (2021) Laura Perez-Beltrachini and Mirella Lapata. 2021. Multi-document summarization with determinantal point process attention. _J. Artif. Int. Res._, 71:371-399.
* Perez-Beltrachini et al. (2019) Laura Perez-Beltrachini, Yang Liu, and Mirella Lapata. 2019. Generating summaries with topic templates and structured convolutional decoders. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5107-5116, Florence, Italy. Association for Computational Linguistics.
* Pilault et al. (2020) Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Chris Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 9308-9319, Online. Association for Computational Linguistics.
* Rennie et al. (2017) Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 1179-1195, Los Alamitatos, CA, USA. IEEE Computer Society.
* Trinh et al. (2018) Trieu Trinh, Andrew Dai, Thang Luong, and Quoc Le. 2018. Learning longer-term dependencies in RNNs with auxiliary losses. In _Proceedings of the 35th International Conference on Machine Learning (PMLR)_, volume 80, pages 4965-4974.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wan and Yang (2006) Xiaojun Wan and Jianwu Yang. 2006. Improved affinity graph based multi-document summarization. In _Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers_, pages 181-184, New York City, USA. Association for Computational Linguistics.
* Wan and Yang (2008) Xiaojun Wan and Jianwu Yang. 2008. Multi-document summarization using cluster-based link analysis. In _Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_, SIGIR '08, page 299-306, New York, NY, USA. Association for Computing Machinery.
* Wu and Hu (2018) Yuxiang Wu and Baotian Hu. 2018. Learning to extract coherent summary via deep reinforcement learning. _Proceedings of the AAAI Conference on Artificial Intelligence_, 32(1).
* Yang et al. (2018) Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 4449-4458, Brussels, Belgium. Association for Computational Linguistics.

 

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In _Proceedings of the 37th International Conference on Machine Learning_, volume 119 of _Proceedings of Machine Learning Research_, pages 11328-11339. PMLR.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with bert. In _International Conference on Learning Representations_.
* Zhong et al. (2020) Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6197-6208, Online. Association for Computational Linguistics.
* Zhou et al. (2021) Hao Zhou, Weidong Ren, Gongshen Liu, Bo Su, and Wei Lu. 2021. Entity-aware abstractive multi-document summarization. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 351-362, Online. Association for Computational Linguistics.
* Zhu et al. (2021) Fangwei Zhu, Shangqing Tu, Jiaxin Shi, Juanzi Li, Lei Hou, and Tong Cui. 2021. TWAG: A topic-guided Wikipedia abstract generator. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_, pages 4623-4635, Online. Association for Computational Linguistics.

## Appendix A Results on Other Corpora

### REFLECT on Multi-XScience Corpus

In this subsection, we additionally experiment REFLECT with Multi-XScience (Lu et al., 2020) corpus, which is a multi-document summarization corpus for scientific articles. The source of this corpus is a tuple of a paper abstract and abstracts of the citation papers, while the generation target is the corresponding related-work paragraph. The performances of baselines are from Lu et al. (2020)3. Table 5 shows that REFLECT achieves a better performance comparing to all baselines, and makes a substantial improvement especially in ROUGE-2 score. We also report the results of BART-large which serves as our abstractor to summarize from the extracted sentences. The improvement shows the benefit of the extraction process after the Credit-Aware Self-Critic Learning.

Footnote 3: We can not tell the version of score metric ROUGE-L used in Multi-XScience, and the evaluation code is also unavailable. Therefore, we do not report the ROUGE-L score from Lu et al. (2020)

### REFLECT on WikiCatSum Corpus

To further analyze the capability of our proposed model, we also apply REFLECT on a domain-specific corpus, WikiCatSum (Perez-Beltrachini et al., 2019) as shown in Table 6. WikiCatSum is a multi-document summarization dataset derived from WikiDum (Liu et al., 2018) and represents three distinct domains (Animal, Company, and Film). We compare our model with several baselines. TF-S2S is a Transformer sequence-to-sequence model of Liu et al. (2018) and CV-S2D+T is a variant of CV-S2S (Gehring et al., 2017) with a single sequence encoder and a structure decoder. Both CV-S2D+T (Perez-Beltrachini et al., 2019) and TWAG (Zhu et al., 2021) introduce the topic detection model to guide the generation. The decoder of CV-S2D+T is trained to predict the topics as an auxiliary task, while TWAG uses the topic information to group the input paragraphs and encodes them separately. Liu et al. (2021) apply knowledge distillation to alleviate the problem of single reference in maximum likelihood training, while our model leverages reinforcement learning to the train-test mismatch issue. The results of BART-large models are fine-tuned on WikiCatSum 4. REFLECT outperforms all baselines in all three domains, especially in the ROUGE-1 score, showing that our generated summaries carry more information. The results between REFLECT (MLE) and REFLECT (CASC) also manifest the effectiveness of Credit-Aware Self-Critic learning for bridging the gap between training and testing.

\begin{table}
\begin{tabular}{c c c c c} \hline \hline Model & R-1 & R-2 & R-L & R-LSum \\ \hline LEAD & 27.46 & 4.57 & - & - \\ LEXRANK & 30.19 & 5.53 & - & - \\ TEXTRANK & 31.51 & 5.83 & - & - \\ Hi-MAP & 31.66 & 5.91 & - & - \\ PG & 34.11 & 6.76 & - & - \\ \hline BART-large & 33.29 & 8.07 & 17.31 & 29.04 \\ REFLECT (MILE) & 33.87 & 8.13 & 17.20 & 29.69 \\ REFLECT (CASC) & **34.18** & **8.20** & **17.42** & **29.73** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performance of REFLECT and baselines on Multi-XScience corpus. We also present the results of BART models trained by truncated articles. The results of baselines are from Lu et al. (2020), where PG represents Pointer-Generator and the Hi-MAP represents the model proposed by Fabbri et al. (2019). ROUGE score is abbreviated as \(R\).

 



## Appendix B Additional Ablations & Analyses

### Effects of Hierarchical Architecture

In this subsection, we study the effect of hierarchical architecture in the extractor. As described in Section 4.1, the token- and sentence-level encoder jointly contain 12 layers in our settings. Thus, we experiment with different numbers of layers distributed for the token-level encoder. Also, we consider a case for no (0) layer, where we directly aggregate the word embeddings from the token-level encoder as sentence features. Table 7 demonstrates that hierarchical architecture improves the resulted abstraction performance through better sentence representations. The performance is enhanced with the increasing number of layers, while more than 3 layers only makes marginal improvements. Thus, we use 3 token-level layers in this paper.

### The Choice of Summary Reference at Training Stage

In this subsection, we study the effect of the summary reference at training stage. As mention in the subsection 3.3, directly using human-written summaries as reference may cause the severe train-test mismatch. To verify the idea, we take the human-written summaries as the summery referencing for the extractor during training, and perform the standard test process, that is utilizing generated summaries as the references. The results shown in Table 8 verify our assumption that training with such explicit signals reduces the ability of the model to generalize. Therefore, in our experiments, we take the generation results of BART-large as the summary referencing of the extractor.

### Effect of Input Settings for Abstract Fine-tuning

In REFLECT, we decouple the learning for the extractor and the abstractor. Therefore, we study the effect of the input settings for the fine-tuning of the abstractor. Table 9 shows the performance of BART under different model configurations with either using truncated articles or pseudo oracle sentences as inputs. We find that using pseudo oracles for BART-large fine-tuning can effectively improve the performance even when the testing input is truncated articles. In addition, the results of using pseudo oracles for both training and testing demonstrate the upper bound performance of the extractor

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & Animal & & \\ \hline Model & R-1 & R-2 & R-L & R-LSum \\ \hline TF-S2S & 44.0 & 28.8 & 40.0 & - \\ CV-S2D+T & 42.7 & 27.9 & 37.9 & - \\ TWAG & 43.1 & 24.4 & 40.9 & - \\ Liu et al. (2021b) & 45.9 & **32.2** & 41.4 & - \\ \hline BART-large & 46.3 & 29.2 & 39.6 & 44.1 \\ REFLECT (MLE) & 46.5 & 27.1 & 38.2 & 43.6 \\ REFLECT (CASC) & **48.6** & 30.2 & **41.5** & **46.0** \\ \hline \multicolumn{5}{c}{Company} \\ \hline Model & R-1 & R-2 & R-L & R-LSum \\ \hline TF-S2S & 26.0 & 9.5 & 20.4 & - \\ CV-S2D+T & 27.5 & 10.6 & 21.4 & - \\ TWAG & 34.1 & 11.9 & **31.6** & - \\ Liu et al. (2021b) & 33.5 & 15.0 & 25.9 & - \\ \hline BART-large & 36.8 & 15.1 & 25.6 & 33.6 \\ REFLECT (MLE) & 40.3 & 15.5 & 27.0 & 36.2 \\ REFLECT (CASC) & **40.8** & **15.8** & 27.5 & **36.6** \\ \hline \multicolumn{5}{c}{Film} \\ \hline Model & R-1 & R-2 & R-L & R-LSum \\ \hline TF-S2S & 36.5 & 18.8 & 31.0 & - \\ CV-S2D+T & 38.0 & 21.2 & 32.3 & - \\ TWAG & 40.8 & 21.2 & 34.3 & - \\ Liu et al. (2021b) & 42.7 & 26.1 & 36.8 & - \\ \hline BART-large & 44.3 & 25.5 & 35.9 & 41.7 \\ REFLECT (MLE) & 46.7 & 25.6 & 36.5 & 43.2 \\ REFLECT (CASC) & **47.6** & **26.8** & **37.9** & **44.1** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance of REFLECT with various baselines on WikiCatSum corpus. The results of Transformer sequence-to-sequence (TF-S2S) and CV-S2D+T are referenced from Perez-Beltrachini et al. (2019), and TWAG is from Zhu et al. (2021). REFLECT outperforms all baseline models in a large margin for all domains, especially for Company and Film in the ROUGE-1 score.

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Layer Number & R-1 & R-2 & R-L & R-LSum & Average \\ \hline
0 & 47.94 & 18.68 & 23.46 & 43.87 & 33.49 \\
1 & 48.01 & 18.63 & 23.53 & 43.95 & 33.53 \\
2 & 48.11 & **18.87** & **23.62** & 44.03 & 33.66 \\
3 & **48.16** & **18.87** & 23.61 & **44.06** & **33.68** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Performance of different extractor configurations with MLE learning. Note that all configurations share the same amount of learnable parameters. ROUGE score is abbreviated as \(R\).



under MLE training, which suggests a promising development of our framework as the extractor can make more precise predictions.

### Generation Examples

We demonstrate two examples of generation in Table 10 and Table 11. The results demonstrate that generations from REFLECT-(CASC) could provide more faithful information from multiple documents, which mainly resulted from the better sentence extraction strategy learned through CASC.

## Appendix C Implementation Details

All of our experiments are conducted on a single NVIDIA Tesla V100 32GB GPU with PyTorch. The hierarchical extractor is initialized by RoBERTa-base5, and the first three layers are exploited as token-level encoder and the rest layers are sentence-level encoder. The BART-base6 is used as the abstractor to provide the rewards during extractor training, while BART-large7 is used for generating the final results. We generate the initial summary reference (SR) by the BART-large model. The hyper-parameter \(\gamma\) in POR is set to 10. We use Adam with constant learning 3e-5 for optimization, and select the model with highest ROUGE-1 F1 score on validation set. For the pseudo extraction oracle, we greedily select at least 30 sentences from article as the pseudo oracle for the extractor during MLE learning. The selection criteria is based on the average of ROUGE-1 recall and ROUGE-2 recall.

Footnote 5: [https://huggingface.co/deepset/roberta-base-squad2](https://huggingface.co/deepset/roberta-base-squad2)

Footnote 6: [https://huggingface.co/facebook/bart-base](https://huggingface.co/facebook/bart-base)

For evaluations, we report ROUGE, BERTScore and factual consistency derived from FactCC framework. For ROUGE, we use _rouge_score_ package 8 to report ROUGE-1, 2, L, and LSum scores. For BERTScore, we use official implementation 9 and report the F1 scores. For factual consistency, we use official implementation of FactCC 10 and follow previous work (Perez-Beltrachini and Lapata, 2021) to calculate the factual consistency for multi-document summarization.

Footnote 7: [https://huggingface.co/facebook/bart-large](https://huggingface.co/facebook/bart-large)

Footnote 8: [https://pypi.org/project/rouge-score](https://pypi.org/project/rouge-score)

Footnote 9: [https://github.com/Tiiiger/bert_score](https://github.com/Tiiiger/bert_score)

Footnote 10: [https://github.com/salesforce/factCC](https://github.com/salesforce/factCC) 



**Article:** Wesley Snipes Begins Serving 3-Year Prison Stint. This Wesley Snipes began serving a three-year sentence at a federal prison in Pennsylvania on Thursday for failure to file income tax returns. Snipes, 48, arrived shortly before noon at the Federal Correctional Institution McKean in the tiny northwestern Pennsylvania town of Lewis Run, federal prisons spokesman Ed Ross said. He had been ordered to surrender by noon. The minimum security prison camp is worlds away from the harsh prison fortresses depicted in the Snipes' films 'Undisputed' and 'Brooklyn's Finest.' The minimum-security camp doesn't have fences around its perimeter. The 300 nonviolent innates live in barracks that feature two-man rooms, daily showers and double-feature movie showings Friday through Sunday. ...

* Wesley Snipes enters prison in Pennsylvania today to serve a three-year sentence for tax evasion--and though the minimum-security McKean prison camp isn't exactly a five-star resort, it's not as bad as it could be. The AP gives details on what his life will be like: No fences: But he will submit to five daily head counts, three of which are overnight. Living quarters: He'll have a two-man room in the barracks, where the other 300 nonviolent innates live. Daily schedule: Wake-up time is 6:35am, and jobs are performed for seven hours per day. Conjugal visits: He'll have to limit himself to just a kiss in the visitors room. Money: He can earn pennies an hour by doing laundry or other chores, and is allowed to spend $290 a month at the commissary. Entertainment: Double-feature movies are shown Friday through Sunday, but no R-, NC-17-, or X-rated films are screened. Exercise: Sand volleyball, indoor basketball, exercise machines, badminton, bocci, and bridge will be available to him. Medical facilities: The copay at the infirmary is only $2. Church: Almost any group you can think of, including Wickens, hold weekly meetings. Click for more, including why Snipes failed to pay taxes--or watch his Tuesday night appearance on Larry King Live.

* Wesley Snipes has begun his **three-year prison sentence for** failure to file income **tax** returns. The Blade: Trinity **star** arrived at the Federal Correctional Institution **McKean** in Lewis Run **in Pennsylvania today**, reports **the AP**. The **minimum security prison camp is** worlds away from the harsh prison fortresses depicted in Snipes' **films** Undisputed and Brooklyn's Finest, notes the AP, but the 48-year-old actor will **have to** adjust to earning just **pennies an hour** handling kitchen, **laundry**, and other campus **cheres." We recognize that he is high profile, but we treat all our **immates** the same," says a spokeswoman. Snipes, a ques-paying member of a tax-protest **group** that challenges the government's right to collect **taxes**, has been free on ball since his 2008 conviction, but he made a last-minute request for a new trial this week, arguing that the judge erred by not allowing defense attorneys to interview jurors about misconduct allegations. He also told **Larrey King** last night that he wasn't nervous about reporting to prison. "Don't send me up the river, yet," he said.

\begin{table} \end{table}
Table 10: A generation example for REFLECT on Multi-News corpus. Key overlaps between generations and the gold summary are bolded.





**Documents:** Vacant lot at 53 New York Avenue NE in Washington, where a government permit has been granted for work connected with Elon Musk's Hyperloop project. (Michael Laris/TWP) It's not much now, just a parking lot with a discarded gin bottle and an old extreminator receipt. But the slice of pavement near the Bureau of Alcohol, Tobacco, Firearms and Explosives in the District could be the gritty precursor to a tunnel network that could propel pods filled with people and speeding platforms topped with Tselas and Toyotas between the nation's capital and New York in 29 minutes. Or it could be just be a parking lot littered with dashed transportation dreams. Electric-car pioneer and space entrepreneur Elon Musk has been touting his vision for a high-speed transportation system since his tweeted announcement last summer that he had received "verbal govt approval" for his tunnel-digging firm, the Boring Company, to build a "NY-Phil-Balt-DC Hyperloop." The Boring Company team has received an early, and vague, building permit from the D.C. government...

**Gold Summary:** -- Elon Musk's dream of building a hyperloop that can move people between Washington, DC, and New York City in 29 minutes may be a small step closer to becoming a distant reality. A Nov. 29 permit issued by DC's Department of Transportation allows Musk's Boring Company to dig at an abandoned lct beside a McDonald's in northeast Washington that the company says might eventually become a station on a hyperloop connecting NYC, Philadelphia, Baltimore, and Washington. The system would speed up travel times, using electricity and electromagnetic levitation to send people and cars whipping through tunnels in pots. But before you ditch your car, the Washington Post points out the permit is only for "some preparatory and excavation work" at the site, and more permits are needed before any real construction begins. DC Mayor Muriel Bowser has visited a test tunnel Musk is constructing under Los Angeles. But though "we're open to the concept of moving people around the region more efficiently ... we're just beginning, in the mayor's office, our conversation to get an understanding of what the general vision is for Hyperloop," says Bowser's chief staff, John Falcicchio. Still, the permit "is comparable to an athlete stretching before a race" and shows Musk is serious about moving forward with his plans for high-speed travel since claiming "verbal" government support for an East Coast hyperloop last July, reports TechCrunch. He's also received a conditional permit to build a tunnel beneath Maryland Route 295 in Hawthorne, per the Post, with Maryland officials saying a leg linking Washington and Baltimore will come first. (Read about the hyperloop pod's first test.)

\begin{table} \end{table}
Table 11: A generation example for REFLECT on Multi-News corpus. Key overlaps between generations and the gold summary are bolded.