SlowGuess/NAACL_2022_OCR huggingface dataset:
Yun-Zhu Song and Yi-Syuan Chen and Hong-Han Shuai
National Yang Ming Chiao Tung University, Taiwan
{yunzhusong.eed07g,yschen.eed09g,hhshuai}@nctu.edu.tw
A notable challenge in Multi-Document Summarization (MDS) is the extremely-long length of the input. In this paper, we present an extract-then-abstract Transformer framework to overcome the problem. Specifically, we leverage pre-trained language models to construct a hierarchical extractor for salient sentence selection across documents and an abstractor for rewriting the selected contents as summaries. However, learning such a framework is challenging since the optimal contents for the abstractor are generally unknown. Previous works typically create pseudo extraction oracle to enable the supervised learning for both the extractor and the abstractor. Nevertheless, we argue that the performance of such methods could be restricted due to the insufficient information for prediction and inconsistent objectives between training and testing. To this end, we propose a loss weighting mechanism that makes the model aware of the unequal importance for the sentences not in the pseudo extraction oracle, and leverage the fine-tuned abstractor to generate summary references as auxiliary signals for learning the extractor. Moreover, we propose a reinforcement learning method that can efficiently apply to the extractor for harmonizing the optimization between training and testing. Experiment results show that our framework substantially outperforms strong baselines with comparable model sizes and achieves the best results on the Multi-News, Multi-XScience, and WikiCatSum corpora.<sup>1</sup>
Neural multi-document summarization has drawn a lot of attention due to the wide applications, e.g., Wikipedia generation (Liu et al., 2018), news digest (Fabbri et al., 2019), or related-work section generation (Lu et al., 2020). Through con
catenating the document clusters, it is applicable to adopt the approaches from single-document summarization (Zhang et al., 2020a; Fabbri et al., 2019) under multi-document settings. However, the input length from multiple documents is typically long, which might be computationally infeasible for Transformer-based models (Vaswani et al., 2017). One possible solution is to set a fixed length and truncate the input sequence. However, the truncation leads to information loss and performance drop. Moreover, even if the model can take such a long sequence as input, the attentions could be dispersed over long sequences (Yang et al., 2018) and further degrade the performance (Jin et al., 2020; Liu and Lapata, 2019a; Cohan et al., 2018).
To tackle the long sequence problem, another possible solution is to leverage the extract-thenabstract architecture in single document summarization (Pilault et al., 2020; Chen and Bansal, 2018; Gehrmann et al., 2018). In such an architecture, an extractor first selects salient contents from the documents, and an abstractor is applied to rewrite the selected contents to coherent summaries. However, several issues arise for directly applying the procedure in MDS. 1) Long-sequence problem for the extractor. Although the input length for the abstractor can be controlled through the content selection and thus enables the usage of Transformer models (Pilault et al., 2020), it is still inevitable for the extractor to process the complete input documents. Previous methods mainly use LSTM-based extractors (Pilault et al., 2020; Chen and Bansal, 2018; Gehrmann et al., 2018). However, this fails to leverage knowledge of pre-trained language models. 2) Suboptimal pseudo oracles. Most summarization corpora do not contain the oracle for extraction. As an alternative, previous works typically generate pseudo extraction oracle, or simply called pseudo oracle, through a greedy process (Liu and Lapata, 2019b; Chen and Bansal, 2018). Specifically, for each iteration in the process,
<sup>1</sup>The implementation code and trained models are available at https://github.com/yunzhusong/NAACL2022-REFLECT
Human Language Technologies, pages 1667 - 1681
candidate sentences are individually concatenated with previously-selected sentences and compute the ROUGE (Lin, 2004) scores with the human-written summaries. The top-scored sentence is iteratively selected until no sentence could further improve the ROUGE score. In practice, there are different ways for scoring. For examples, Liu and Lapata (2019b) use the average of ROUGE-1 and ROUGE-2 F1 scores while Chen and Bansal (2018) use ROUGE-L recall. The variants of design could cause the extractor to behave differently. In our study, we also found that using the ROUGE precision metric leads to much less extraction than using the recall metric. This implies that the pseudo oracles are suboptimal, and learning the extractor fully relies on the pseudo oracles could restrict the performance. How to alleviate the negative effects of pseudo oracles remains open. 3) Insufficient information for the extractor. Even if the pseudo oracles are good enough to train the abstractor well, learning a precise extractor is still challenging. The problem lies in that a pseudo oracle is derived from a specific summary. However, there are potentially multiple valid summaries given the documents. To select salient sentences, the extractor is required to implicitly infer the underlying summary used for oracle construction, which is difficult due to the lack of evidence. 4) Inconsistent objectives for the extractor. With pseudo oracles, the extractor is learned to select a set of sentences that has high lexical similarity to the summaries without redundancy. However, the goal of the extractor in test-time is to provide inputs for the abstractor that non-overlapping lexical may still be valuable. In other words, the objective for the extractor in training is inconsistent with the one in testing.
To address these issues, we propose the REferenced FLEXible Extraction with CrediT-Awareness (REFLECT) for MDS. For the first problem, we propose a Transformer-based hierarchical extractor that contains the token- and sentence-level feature encoders. Both the encoders are initialized with pre-trained language models to utilize the pretext knowledge. For the second problem, we propose Pseudo Oracle Relaxation (POR) to render the model aware of the unequal importance for the non-oracle sentences. This mechanism encourages the model to emphasize the precision for critical sentences with either high or low lexical similarity to the summaries, and avoids the confusion arising from the different labels for similar
sentences. For the third problem, we propose Summary Referencing (SR) to leverage the fine-tuned abstractor for providing additional learning signals to evidence extraction prediction. The summary reference serves as an approximation for the human-written summary while being able to generalize for testing. For the fourth problem, we propose Credit-Aware Self-Critic (CASC) learning to fine-tune for matching the objective between training and testing. Different from previous methods that assign an identical reward for all actions, we reallocate the rewards based on the impacts of actor explorations.
- We leverage pre-trained language models to propose an extract-then-abstract framework, which contains a hierarchical extractor that efficiently handles long inputs while utilizing pretext knowledge.
- We investigate the problems for typical learning paradigms of the extractor and propose a framework, named REFLECT, to further improve the extractor performance. The studies on pseudo oracles also provide valuable insights for extract-then-abstract frameworks.
- Experimental results on Multi-News, Multi-XScience, and WikiCatSum corpora demonstrate that REFLECT outperforms the state-of-the-art models with comparable sizes.
Early attempts for MDS focus on extracting sentences through statistical methods (Goldstein et al., 2000; Erkan and Radev, 2004; Wan and Yang, 2006, 2008). For example, Goldstein et al. (2000) extend Maximal Marginal Relevance (MMR) method to select sentences that are relevant to the query and novel across different documents. Erkan and Radev (2004) leverage sentence relations with graph structures that represent pairwise sentence similarities, and apply PageRank (Page et al., 1999) algorithm to extract sentences given the query document. However, extractive methods often suffer the coherence problem (Wu and Hu, 2018). Therefore, instead of directly extracting sentences from the articles, abstractive methods that can rewrite the articles achieve great success with the advantages of large annotated corpora (Pang et al., 2021; Zhou et al., 2021; Liu et al., 2021a; Zhong et al., 2020; Li et al., 2020; Liu and Lapata, 2019a).
Directly operating on the long inputs in MDS often leads to model degradation (Jin et al., 2020). One of the promising solutions is to leverage the extract-then-abstract architectures in single document summarization (Chen and Bansal, 2018; Gehrmann et al., 2018). For example, Gehrmann et al. (2018) propose an LSTM-based word-level extractor to choose phrases from the document, and apply an abstractor with the copy mechanism to generate summaries given the selected contents. Pilault et al. (2020) also apply an LSTM-based sentence extractor to select contents, but further leverage a Transformer decoder to improve the performance.
Although introducing Transformer architectures provides improvements, the input length is typically limited according to the computation and memory overhead. A recent line of studies has been proposed to alleviate such problems (Bražinskas et al., 2021; Beltagy et al., 2020; Liu and Lapata, 2019a; Jin et al., 2020). Liu and Lapata (2019a) propose to rank the candidate input paragraphs, and only concatenate the top few as the inputs for the abstractor. Jin et al. (2020) encode multiple documents in different granularity including token-level, sentence-level, and document-level. Pasunuru et al. (2021) design a graph encoder parallel with standard encoder to provide inter-document information and integrate the pre-trained BART (Lewis et al., 2020) with local attention mechanism (Beltagy et al., 2020) to overcome the long input problem. Bražinskas et al. (2021) apply policy gradient optimization to learn a train-time selector using lexical features pre-computed from source texts and gold summaries as inputs. Due to the lack of gold summaries in testing, a test-time selector is learned using lexical features solely from source texts as inputs and predictions from the train-time selector as learning targets. Comparatively, our methods thoroughly leverage the language models with an extract-and-abstract framework that enjoys the full capacities for both the extractor and the abstractor. The introduced summary referencing further improves the extractor with additional signals in both train- and test-phase.
Utilizing large pre-trained language models brings great benefits for text generation problems. However, the input length of multi-document summarization is typically long, which makes large pre
trained language models, e.g., Transformer-based models, inefficient for processing. To match the length constraint, document-level truncation (Fabbri et al., 2019) has been widely-used. However, the truncation could inevitably cause information loss and degrade the performance. To solve the problem, we propose to leverage the extract-then-abstract framework. However, as described in the introduction, four challenges arise while learning such a framework. We first present our architecture to solve the long sequence problem and elaborate on the proposed methods to tackle the challenges.
3.1 Hierarchical Summarizer
Figure 1 illustrates the proposed REFLECT framework. Specifically, based on the Transformer (Vaswani et al., 2017), our architecture contains a hierarchical extractor (H-EXT) and an abstractor (ABS). The hierarchical extractor is composed of a token-level encoder (T-ENC), a sentence-level encoder (S-ENC), and a sentence selector (SS). Considering a training example  $(x,y)$ ,  $x = \{x_{1},x_{2},\dots,x_{N}\}$  is the concatenation of multiple documents that jointly consists of  $N$  complete sentences and  $y$  is the corresponding human-written summary. Let  $M$  denote the allowed input length in Transformer-based models, we split all sentences into  $K$  disjoint sets  $\{h_k\}_{k=1}^K$  such that each set  $h_k$  consists of sentences whose total number of tokens matches the length constraint  $M$ . The hierarchical extractor then predicts a set containing the indices of selected sentences (denoted by  $\hat{e}$ ), which is used to retrieve salient contents from  $x$  for the abstractor to produce summaries. Specifically, the hierarchical extractor can be expressed as follows.
$$
\begin{array}{l} \hat {e} = \mathrm {H} - \operatorname {E X T} (h) \tag {1} \\ = \operatorname {S S} (\mathrm {S} - \operatorname {E N C} (\bar {\oplus} \{\mathrm {T} - \operatorname {E N C} (h _ {k}) \} _ {k = 1} ^ {K})), \\ \end{array}
$$
where  $\bar{\oplus}$  is the operation of taking average for hidden states of tokens within sentences and concatenating the averaged results to obtain the sentence-level representations. The hierarchical summarizer (H-SUM) can then be expressed as:
$$
\mathrm {H} - \operatorname {S U M} (h) = \operatorname {A B S} \left(\oplus \left\{w _ {x _ {i}} \mid i \in \hat {e} \right\}\right), \tag {2}
$$
where  $\oplus$  is the concatenation operation.
3.2 Pseudo Oracle Relaxation
The sentence selection process of the extractor is typically formulated as a supervised classification
Figure 1: Framework of REFLECT. The illustration of proposed Pseudo Oracle Relaxation (POR), Summary Referencing (SR), and Credit-Aware Self-Critic (CASC) learning are indicated with bold fonts.
problem. However, most summarization corpora do not provide such annotations. As an alternative, it is common to create pseudo oracles through a greedy process with the human-written summaries (Liu and Lapata, 2019b; Chen and Bansal, 2018). However, such methods only provide suboptimal solutions since they are limited by the design of the greedy algorithm. The underlying patterns of true oracles could be too complicated to be designed precisely. The performance of the extractor is thus restricted since typical Maximal Likelihood Estimation (MLE) methods fully depend on pseudo oracles. For example, consider a case that there are only three sentences  $x_{a}$ ,  $x_{b}$  and  $x_{c}$ , and the ROUGE scores between sentences and summary are ranked as  $x_{a} > x_{b} \gg x_{c}$ . Assume that only  $x_{a}$  is chosen as the pseudo oracle. If the extractor further selects one additional sentence for improving the results, it is expected that the combination of  $(x_{a}, x_{b})$  could be better than  $(x_{a}, x_{c})$  generally. However, MLE-based methods do not consider such discrepancy according to the pseudo oracle.
Therefore, we propose a loss weighting mechanism to further consider the lexical similarity for the non-oracle sentences during the learning process. Consider an input  $x = \{x_{1},x_{2},\ldots ,x_{N}\}$  and the corresponding pseudo oracles  $e$ , the input sentences can be separated into two sets of pseudo oracles and non-pseudo oracles,  $S = \{x_{i}|i\in e\}$  and  $S^c = \{x_i|i\in e^c\}$ , respectively. We maintain the loss weights for sentences in  $S$  as one, while modifying the loss weights for the sentences in  $S^c$  as follows:
$$
w _ {i} = \left(1 - \text {R O U G E - 1} \left(x _ {i}, y\right)\right) ^ {\gamma}, \tag {3}
$$
where  $\gamma$  is a hyper-parameter controlling the weighting scales. To stabilize the training, we fur
ther shift loss weights of  $S^c$  such that the maximum value is one. This weighting mechanism emphasizes the predictions of both pseudo oracles and the low-ROUGE sentences, which have more impact on the performance. It relaxes the constraint from the binarized oracles to make the model aware of the differences between candidate sentences. The objective for the hierarchical extractor can then be written as:
$$
L = - \sum_ {x _ {i} \in x} w _ {i} \log \left(\frac {\exp \left(z _ {i} ^ {\mathbf {1} _ {S (i)}}\right)}{\exp \left(z _ {i} ^ {0}\right) + \exp \left(z _ {i} ^ {1}\right)}\right), \tag {4}
$$
where  $z_{i}^{0}$  and  $z_{i}^{1}$  are the binary logits for the  $i$ -th sentence, and  $\mathbf{1}_S(i)$  is the indicator function of oracle sentences  $S$ .
3.3 Summary Referencing
The pseudo oracle relaxation described in Section 3.2 makes the extractor focus on the sentences that are more important in terms of the lexical similarity. In other words, it considers the discrepancy between non-oracle sentences instead of treating them equally. However, the ambiguity could still exist between the oracle and non-oracle sentences. For example, consider a case where there are two sentences  $x_{a}$  and  $x_{b}$ , and only  $x_{a}$  is chosen as the oracle during the greedy process. The ROUGE scores between the sentences and the summaries are roughly the same for the two sentences. Learning with such oracles may confuse the model since the positive and negative sentences are similar but with a large loss difference. Therefore, we propose to provide summary references for the extractor to further reason the selection. Since the summary is only available in the training stage, we leverage a fine-tuned abstractor to provide such summary references. With such a mechanism, the operations
of the hierarchical extractor are further revised as:
$$
\begin{array}{l} r = \operatorname {A B S} \left(\oplus \left\{w _ {x _ {i}} \mid i \in \hat {e} \right\}\right), \\ \begin{array}{l} \hat {e} = \mathrm {H} - \mathrm {E X T} (h, r) \\ \quad = \operatorname {S C} (\mathrm {S}, \operatorname {E N C} (\bar {\partial} ([ \mathrm {T}, \operatorname {E N C} (x) ] _ {+}) \end{array} \tag {5} \\ = \operatorname {S C} (\mathrm {S} - \operatorname {E N C} (\bar {\oplus} (\{\mathrm {T} - \operatorname {E N C} (r) \} + \{T - \operatorname {E N C} (h _ {k}) \} _ {k = 1} ^ {K}))) \\ \end{array}
$$
Compared to directly using human-written summaries as references when training, which may cause the extractor highly rely on the reference signal, the generation results from the abstractor could serve as good approximations and provide the generalization from training to testing.
3.4 Credit-Aware Self-Critic Learning
With the objective of maximal likelihood estimation, the extractor is learned to select sentences that are jointly have high lexical similarity to the human-written summary. However, in extract-then-abstract frameworks, the required objective of extractor is to provide salient contents that can maximize the generation quality after rewritten by the abstractor. In other words, the objective for the extractor is inconsistent between training and testing. Therefore, we further propose a reinforcement learning method that can directly optimize with the test-time objective to bridge the gaps.
Specifically, we formulate the extraction of sentences as a single-round Combinatorial Multi-Armed Bandit (CMAB) problem (Chen et al., 2016). A general CMAB problem can be modeled as a tuple  $(E,\mathcal{F},P,R)$ , where  $E = \{1,..,2\ldots,N\}$  is a set of  $N$  arms,  $\mathcal{F} \subseteq 2^{E}$  is a set of subsets of  $E$ ,  $P$  is a probability distribution over  $[0,1]^{N}$ , and  $R$  is a reward function defined on  $[0,1]^{N} \times \mathcal{F}$ . At the  $t$ -th round, the agent pulls a subset of arms  $S^t \in \mathcal{F}$ , and produces stochastic outcomes  $M^t = (M_1^t,M_2^t,\dots,M_N^t) \sim P$ , where  $M_i^t$  is the outcome of  $i$ -th arm. With a realization of outcomes  $m = (m_1,m_2,\dots,m_N)$ , the agents then receive a reward of  $R(m,S)$ . The goal of the agent is to maximize the expected cumulative reward in  $T$  rounds, which is  $\mathbb{E}_{\{M^t\}_{t=1}^T}[\sum_{t=1}^T R(M^t,S^t)]$ .
For the sentence extraction problem, we consider each sentence as an arm, and pull multiple arms (i.e., select multiple sentences) only for a single round. We solve this problem with the self-critic learning framework (Rennie et al., 2017). Specifically, we consider the sentence-level encoder with the sentence selector that as the agent (jointly parameterized by  $\theta$ ). The agent takes the sentence representations  $x = \{x_{1}, x_{2}, \dots, x_{N}\}$  from the token
level encoder, and selects a set of sentences  $S$  according to a policy  $\pi_{\theta}(\cdot)$  as:
$$
m _ {i} \sim \operatorname {B e r n} \left(\frac {\exp \left(z _ {i} ^ {1}\right)}{\exp \left(z _ {i} ^ {0}\right) + \exp \left(z _ {i} ^ {1}\right)}\right), \tag {6}
$$
$$
S = \pi_ {\theta} (x) = \{i | m _ {i} = 1 \},
$$
where  $z_{i}^{0}$  and  $z_{i}^{1}$  are the binary logits for the  $i$ -th sentence. To reduce the variance during learning, we introduce a baseline term through another policy  $\tilde{\pi}_{\theta}$  as:
$$
\tilde {m} _ {i} = \frac {\exp \left(z _ {i} ^ {1}\right)}{\exp \left(z _ {i} ^ {0}\right) + \exp \left(z _ {i} ^ {1}\right)}, \tag {7}
$$
$$
\tilde {S} = \tilde {\pi} _ {\theta} (x) = \{i | \tilde {m} _ {i} > 0. 5 \},
$$
which is essentially a greedy policy as the taken actions are not explored. The two policies are derived from the agents with the same parameters, and the learning process is thus self-critical. We define the reward as the performance of generation from an abstractor using the selected sentences  $S$  as the input, i.e.,
$$
R (S) = \operatorname {R O U E G} - \mathrm {L} (\operatorname {A B S} (S), y), \tag {8}
$$
where  $S$  is produced from the outcomes  $m$ . The advantage  $a$  can be written as:
$$
a = R (S) - R (\tilde {S}). \tag {9}
$$
We then optimize the agent through gradient descent with the objective function as:
$$
L = - \sum_ {i \in E} a \log \left(\frac {\exp \left(z _ {i} ^ {\mathbf {1} _ {S} (i)}\right)}{\exp \left(z _ {i} ^ {0}\right) + \exp \left(z _ {i} ^ {1}\right)}\right), \tag {10}
$$
where  $\mathbf{1}_S(i)$  is the indicator function of selected sentence set  $S$ . However, in such a learning objective, the advantages are applied uniformly to update all actions, which could make learning difficult since the sentences number is typically large in our setting. Thus, we propose to specifically credit the advantages to the selections that are distinct between policies  $\pi_{\theta}$  and  $\tilde{\pi}_{\theta}$ , and the objective function  $L^{credit}$  can be written as:
$$
\begin{array}{l} L ^ {c r e d i t} = \\ - \sum_ {i \in E} \mathbf {1} _ {S \cap \tilde {S}} (i) a \log \left(\frac {\exp \left(z _ {i} ^ {\mathbf {1} _ {S} (i)}\right)}{\exp \left(z _ {i} ^ {0}\right) + \exp \left(z _ {i} ^ {1}\right)}\right). \tag {11} \\ \end{array}
$$
Different with previous work (Chen and Bansal, 2018) that formulates the sentence selections as a sequence of decisions and uses LSTM-based agent for learning, our formulation enables the usage of Transformer-based models for a better efficiency.
Model	ROUGE-1	ROUGE-2	ROUGE-L	Average	Average Improvement
Hierarchical-Transformer (Liu and Lapata, 2019a)	42.36	15.27	22.08	26.57	+17.91%
PG-BRNN (Gehrmann et al., 2018)	43.77	15.38	20.84	26.66	+17.52%
Hi-MAP (Fabbri et al., 2019)	44.17	16.05	21.38	27.20	+15.18%
CTF-DPP (Perez-Beltrachini and Lapata, 2021)	45.84	15.94	21.02	27.60	+13.51%
GraphSum (Li et al., 2020)	45.02	16.69	22.50	28.07	+11.61%
GraphSum + RoBERTa (Li et al., 2020)	45.87	17.56	23.39	28.94	+8.26%
Highlight-Transformer (Liu et al., 2021a)	44.62	15.57	-	-	-
MatchSum (Zhong et al., 2020)	46.20	16.51	-	-	-
PEGASUS (Zhang et al., 2020a)	47.52	18.72	24.91	30.38	+3.13%
BART-Long (Pasunuru et al., 2021)	48.54	18.56	23.78	30.29	+3.43%
BART-Long-Graph (Pasunuru et al., 2021)	49.24	18.99	23.97	30.73	+1.95%
REFLECT (MLE)	48.16±0.01	18.87±0.01	23.78±0.17	30.27	+3.50%
REFLECT (CASC)	49.27±0.06	19.96±0.03	24.76±0.09	31.33	-
Table 1: Performance of REFLECT with various baselines on Multi-News corpus. The results of Hierarchical Transformer, HiMAP, and PG-BRNN are copied from Li et al. (2020). The rest baseline results are from the original papers. Best ROUGE scores are bolded. Performance of REFLECT is reported with five runs.