SlowGuess/NAACL_2021_OCR huggingface dataset:
Yixin Liu, Zi-Yi Dou, Pengfei Liu *
{yixinl2, zdou, pliu3}@cs.cmu.edu
Although some recent works show potential complementarity among different state-of-the-art systems, few works try to investigate this problem in text summarization. Researchers in other areas commonly refer to the techniques of reranking or stacking to approach this problem. In this work, we highlight several limitations of previous methods, which motivates us to present a new framework Refactor that provides a unified view of text summarization and summaries combination. Experimentally, we perform a comprehensive evaluation that involves twenty-two base systems, four datasets, and three different application scenarios. Besides new state-of-the-art results on CNN/DailyMail dataset (46.18 ROUGE-1), we also elaborate on how our proposed method addresses the limitations of the traditional methods and the effectiveness of the Refactor model sheds light on insight for performance improvement. Our system can be directly used by other researchers as an off-the-shelf tool to achieve further performance improvements. We open-source all the code and provide a convenient interface to use it: https://github.com/yixinL7/Refactoring-Summarization.
In neural text summarization, system designers commonly have flexible choices in model architectures (Rush et al., 2015; Kedzie et al., 2018), decoding strategies (Paulus et al., 2018) (e.g. beam search) and etc. As a result, even on the same dataset, different selection biases of these choices will lead to diverse system outputs (Kedzie et al., 2018; Hossain et al., 2020).
To combine complementarity of system's output under different setups, researchers have made some preliminary efforts on two-stage learning (Collins and Koo, 2005; Huang, 2008; González-Rubio
Figure 1: Illustration of two-stage learning. "Doc, Hypo, Ref" represent "input document, generated hypothesis, gold reference" respectively. "Hypo' represents texts generated during test phase.  $\Theta^{\mathrm{Base}}$  and  $\Theta^{\mathrm{Meta}}$  represent learnable parameters in two stages.
et al., 2011; Mizumoto and Matsumoto, 2016), consisting of (i) a base-stage: first generates different outputs under different setups, and (ii) a meta-stage: then aggregates them in diverse ways, exemplified by stacking that uses a high-level model to combine multiple low-level models (Ting and Witten, 1997), or reranking (Collins and Koo, 2005), which aims to rerank different outputs of one system. Although these methods each play a role in different scenarios, they suffer from following potential limitations:
(i) Ad-hoc Methods: most existing methods are designed for a specific scenario. For example, Li et al. (2015) and Narayan et al. (2018b) resort to reranking techniques to select summary-worthy sentences that are usually generated from one system. By contrast, Hong et al. (2015) focus on summaries generated from different systems and use a non-neural system combination method to make their complementary advantages. Few works explore if the complementarity existing in different scenarios could be utilized in a unified framework.
(ii) Base-Meta Learning Gap: parameterized models between two learning stages are relatively independent. For example, Zhou et al. (2017) and Huang et al. (2020) adapt the seq2seq (Sutskever et al., 2014) framework as the meta model for combination, which takes the outputs of multiple base
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1437-1448 June 6-11, 2021. ©2021 Association for Computational Linguistics
systems as a part of the inputs for machine translation. As a result, there is no parameter sharing between the meta model and base systems as shown in Fig. 1, which prevents the meta model from fully utilizing the knowledge encoded in the base systems.
(iii) Train-Test Distribution Gap: regarding the meta-learning stage, there is a distribution gap between the training and test distributions. Fig. 1 elucidates this phenomenon: the training distribution of Hypo differs from the test distribution of Hypo'. Although both two are outputs from the base stage, Hypo would be more accurate (closer to gold summaries) since it is the output during the training phase.
In this work, we aim to address these limitations by proposing a general framework, named Refactor, which can not only serve as a base system to construct a summary by selecting sentences from the source document but also act as a meta system to select the best system output from multiple candidates. The unification of base and meta systems allows them to share a set of parameters, thereby alleviating the "Base-Meta learning gap". Besides, we propose a pretrain-then-finetune paradigm for Refactor that mitigates the "Train-Test distribution gap". In practice, our proposed Refactor can be applied to different scenarios. For example, as a meta system, it can be used for multiple system combination or single system re-ranking.
Our contributions can be briefly summarized as:
(1) We dissect two major factors that influence the performance of two-stage learning when leveraging the complementarity among different systems: (i) Base-Meta Learning Gap (ii) Train-Test Distribution Gap;
(2) We show these two types of gaps can be alleviated by promoting communication between the two stages in §4, and therefore present a new paradigm where the base and meta learners are parameterized with shared parameters;
(3) We have made comprehensive experiments (twenty-two top-scoring systems, four datasets). In addition to achieving state-of-the-art results on CNN/DailyMail dataset (\$5) by a significant margin, the efficacy of the proposed Refactor opens up a thought-provoking direction for performance improvement: instead of pursuing a purely end-to-end system, a promising exploration is to incorporate different types of inductive biases stage-wisely with the same parameterized function. Our experi
imental results demonstrate that there exists complementarity introduced by decoding algorithms (e.g. beam search) §5.5 or system combination §5.6 among the current state-of-the-art summarization systems, which can be effectively utilized by our model for boosting the system performance.
Existing works commonly design systems in an end-to-end fashion (Sutskever et al., 2014; Sukhbaatar et al., 2015), which, though effective, also proves to be insufficient in some scenarios (Glasmachers, 2017; Webb et al., 2019). Instead of optimizing a system in an end-to-end fashion, one more flexible paradigm, stage-wise learning, is to break down the holistic process into different stages. The basic idea is to incorporate different types of inductive biases stage-wisely and two typical examples are: Stacking and Reranking.
Stacking Stacking (a.k.a, Stacked Generalization) is a general method of using a high-level model to combine lower-level models to achieve greater predictive accuracy (Ting and Witten, 1997). In NLP research, this method has been widely explored in machine translation (MT) task. Traditionally, it is used to improve the performance of statistical MT systems (González-Rubio et al., 2011; Watanabe and Sumita, 2011; Duh et al., 2011; Mizumoto and Matsumoto, 2016). Some recent work (Zhou et al., 2017; Huang et al., 2020) also extends this method to neural MT where the meta model and base systems are all neural models. There is a handful of works about system combination for summarization (Hong et al., 2015), in which a feature-based meta model is used for combining unsupervised text summarization systems.
Reranking. Reranking is a technique to improve performance by reranking the output of an existing system, which has been widely used across different NLP tasks, such as constituency parsing (Collins and Koo, 2005; Huang, 2008), dependency parsing (Zhou et al., 2016; Do and Rehbein, 2020), semantic parsing (Ge and Mooney, 2006; Yin and Neubig, 2019), machine translation (Shen et al., 2004; Mizumoto and Matsumoto, 2016).
Comparing reranking and stacking, both of them involve two-stage learning and the first stage would provide multiple candidate outputs as the input for the second stage. However, they differ in the way how multiple candidate outputs are generated at the
first stage. Specifically, reranking usually decodes  $k$ -most qualified results during inference, using one base system. By contrast, stacking generates multiple outputs that are usually from different base systems.
3 Summarization as Two-stage Learning
In what follows, we detail how to formulate summarization as a two-stage learning task.
Base system The system in the base stage aims to generate a summary based on the input text. Specifically, given a document  $D = \{s_1, \dots, s_n\}$  with  $n$  sentences, we refer to  $C$  as a candidate summary of  $D$  generated by a summarization system, which can be parameterized in diverse forms:
$$
C = \operatorname {B A S E} (D, \mathcal {T}, \mathcal {S}, \Theta^ {\text {b a s e}}) \tag {1}
$$
where  $\mathrm{BASE}(\Theta^{\mathrm{base}})$  represents a base system that can be instantiated either as an extractive model or abstractive model with a specific experimental setup: training method  $\mathcal{T}$ , decoding strategy  $S$ .
Meta system In practice, different choices of parameterized function  $\mathrm{BASE}(\cdot)$ , training method  $\mathcal{T}$  and decoding strategy  $S$  commonly lead to different candidate summaries,  $\mathcal{C} = \{C_1,\dots ,C_k\}$ , where  $\mathcal{C}$  represents a set of different candidate summaries. The goal of the meta system is to utilize complementarities among  $\mathcal{C}$  by popular techniques, such as reranking and system combination.
Specifically, given a set of candidate summaries  $\mathcal{C}$ , a meta system is used to re-construct a new candidate summary  $C^*$
$$
C ^ {*} = \operatorname {M E T A} (D, \mathcal {C}, \Theta^ {\mathrm {m e t a}}) \tag {2}
$$
where  $\Theta^{\mathrm{meta}}$  represents learnable parameters of the meta system.
4 Refactoring Text Summarization
Despite effectiveness of existing meta systems, they, as briefly mentioned in §1, suffer from two major problems: (i) Base-Meta Learning Gap and (ii) Train-Test Distribution Gap.
4.1 Refactoring
In this paper, we propose the model Refactor that unifies the goal of the base and meta systems by the view that a summary can be generated by selecting the best combination of document sentences. Therefore, both base and meta systems aim to select
an optimal candidate summary, and they only differ in how the candidate summary set is constructed. For example, Refactor can be a base system when the candidate summary set  $\mathcal{C}$  is formed by directly enumerating different combinations of document sentences and would be a meta system when  $\mathcal{C}$  represents summaries from different systems. This formulation is advantageous in two points:
(1) No matter where a system selects (from document sentences or multiple system outputs), the chosen criteria that define a good summary are shared. Therefore, the learning process of base and meta systems can be parameterized using a set of parameters, maximizing the information-sharing across two stages and mitigating the Base-Meta Learning Gap.
$$
C ^ {*} = \operatorname {R E F A C T O R} (D, \mathcal {C}, \Theta^ {\text {r e f a c t o r}}), \tag {3}
$$
where  $\mathrm{REFACTOR}(\cdot ,\Theta^{\mathrm{refactor}})$  is the Refactor model, and the candidate summaries  $\mathcal{C}$  can be constructed in different ways.
(2) Additionally, learning to select candidate summaries from document sentences enables the system to see more diverse candidates with different distributions. This is effective for solving the Train-Test Distribution Gap, where the distribution of the meta system outputs in training samples deviates from the test one.
Specifically, our proposed Refactor first learns to select candidate summaries from document sentences (pre-trained Refactor) and then learns to select candidate summaries from different system outputs (fine-tuned Refactor).
4.2 Pre-trained Refactor
Pre-trained Refactor takes as input a document  $D = \{s_1,\dots ,s_n\}$  as well as a set of candidate summaries  $\mathcal{C} = \{C_1,\dots ,C_m\}$ , which can be constructed by enumerating possible combinations of source sentences with heuristic pruning. For example, an extractive system could be used to prune unlikely sentences to control the number of candidates. REFACTOR(·, Θ $^{\text{refactor}}$ ) is instantiated as a score function which quantifies the degree to which a candidate summary  $C_i$  is matched with the source document  $D$ .
$$
\begin{array}{l} C ^ {*} = \operatorname {R E F A C T O R} (D, \mathcal {C}, \Theta^ {\text {r e f a c t o r}}) \\ = \underset {C _ {i} \in \mathcal {C}} {\operatorname {a r g m a x}} \left(\operatorname {S C O R E} \left(\mathbf {D}, \mathbf {C} _ {i}\right)\right) \tag {4} \\ \end{array}
$$
where  $\mathbf{D}$  and  $\mathbf{C}_i$  denote document and summary representations respectively, which are calculated
by a BERT (Devlin et al., 2019) model.  $\mathrm{SCORE}(\cdot)$  is a function that measures the similarity between a document and candidate summary.
Contextualized Similarity Function To instantiate  $\mathrm{SCORE}(\cdot)$ , we follow the forms as mentioned in Zhang et al. (2019b); Zhao et al. (2019); Gao et al. (2020), which have shown superior performance on measuring semantic similarity between documents and summaries.
Specifically,  $\mathrm{SCORE}(\cdot)$  is defined based on the greedy matching algorithm, which matches every word in one text sequence to the most similar word in another text sequence and vise versa. Given the document embedding matrix  $\mathbf{D} = \langle \mathbf{d}_1,\dots ,\mathbf{d}_k\rangle$  and the candidate embedding matrix  $\mathbf{C} = \langle \mathbf{c}_1,\dots ,\mathbf{c}_l\rangle$  encoded by BERT,  $\mathrm{SCORE}(\cdot)$  can be calculated as:
$$
\operatorname {S c o r e} (\mathbf {D}, \mathbf {C}) = 2 \frac {\operatorname {R} (\mathbf {D} , \mathbf {C}) \cdot \operatorname {P} (\mathbf {D} , \mathbf {C})}{\operatorname {R} (\mathbf {D} , \mathbf {C}) + \operatorname {P} (\mathbf {D} , \mathbf {C})} \tag {5}
$$
where the weighted recall R, precision P are defined as follows:
$$
\mathrm {R} (\mathbf {D}, \mathbf {C}) = \frac {\sum_ {i} w _ {i} \max _ {j} \cos (\mathbf {d} _ {i} , \mathbf {c} _ {j})}{\sum_ {i} w _ {i}} + 1, (6)
$$
$$
\mathrm {P} (\mathbf {D}, \mathbf {C}) = \frac {\sum_ {j} \max _ {i} \cos (\mathbf {d} _ {i} , \mathbf {c} _ {j})}{l} + 1, \tag {7}
$$
$w_{i}$  is the weight of the  $i$ -th token in the document. We use weighted recall R based on the assumption that for text summarization, tokens in the source document have different importance and the summary should capture the most important information of the source document. Therefore, we introduce a weighting module built by a two-layer Transformer (Vaswani et al., 2017) assigning weights  $w_{i}$ :
$$
w _ {i} = \frac {\exp (\operatorname {d o t} \left(\mathbf {d} _ {i} , \hat {\mathbf {d}} _ {0}\right) / \sqrt {d})}{\sum_ {j} \exp (\operatorname {d o t} \left(\mathbf {d} _ {j} , \hat {\mathbf {d}} _ {0}\right) / \sqrt {d})}, \tag {8}
$$
where  $\hat{\mathbf{D}} = \mathrm{Transformer}(\mathbf{D})$  and  $\hat{\mathbf{d}}_0 = \hat{\mathbf{D}}[0]$  represents the embedding of the “[CLS)” token which encodes the global information.  $d$  is the dimension of  $\mathbf{d}_i$ .
Learning Objective We use a ranking loss to learn the parameter  $\Theta^{\mathrm{refactor}}$ , inspired by the assumption (Zhong et al., 2020) that a good candidate summary should be as close with the source
Figure 2: ROUGE-1 distributions of the candidates in pretraining stage training set (pre-train), fine-tuning stage training set (meta-train) and fine-tuning stage test set (meta-test) on XSum dataset.
document as possible. Formally,
$$
\begin{array}{l} L = \sum_ {i} \sum_ {j > i} \max  (0, \operatorname {S C O R E} (\mathbf {D}, \mathbf {C} _ {j}) \tag {9} \\ - \operatorname {S C O R E} (\mathbf {D}, \mathbf {C} _ {i}) + (j - i) * \lambda_ {c}) \\ \end{array}
$$
where  $C_i$  and  $C_j$  denote the  $i$ -th and  $j$ -th sample of the candidate list which is descendingly sorted by the ROUGE (Lin, 2004) scores between the reference summary  $\hat{C}$  and candidates. That is,  $\mathrm{ROUGE}(C_i, \hat{C}) > \mathrm{ROUGE}(C_j, \hat{C})$  for  $i < j$ .  $\lambda_c$  is the corresponding margin set to 0.01.
4.3 Fine-tuned Refactor
In order to fit the distributions of the specific types of input, we then fine-tune Refactor using the outputs generated by the base systems. Specifically, fine-tuning is also based on Eq. 9 where the candidate summaries  $C$  are generated by the base systems under different application scenarios.
Why does Pre-train and Fine-tune matter? We elaborate on the proposed two-step training using a real case. Fig. 2 depicts the distribution of ROUGE-1 scores regarding the candidate summaries in the pre-training stage training set, fine-tuning stage training set and test set on the XSum dataset, where we sample the same number of {document, candidate summaries} pairs. We can observe that:
(i) there is a distribution gap between train and test samples in fine-tuning stage. (ii) in pre-training stage the pre-trained Refactor has seen a large number of candidate summaries with diverse performance (ROUGE value), which improves its generalization ability. In §5 we will show that the Pre-train and Fine-tune paradigm outperforms one-
<sup>1</sup>We found that adding 1 to the precision and recall helps to stabilize the training.
step training where the model is directly trained with data generated from the base systems.
4.4 Application Scenarios
Our Refactor can be used as different roles in different scenarios as follows.
4.4.1 Refactor as Base Learner
The pre-trained Refactor can not only be fine-tuned for a better selection of candidate summaries, but also be regarded as a base system, providing one system output. This feature of Refactor maximizes parameter sharing across the two training stages.
4.4.2 Refactor as Meta Learner
Both pre-trained Refactor and fine-tuned Refactor can be used as a meta system to select the best candidate when we have multiple system summaries. In this work, we explore the following settings:
(1) Single System: It considers re-ranking candidate summaries generated from a single abstractive system using beam search.
(2) Multi-system Summary-level: It is tasked to select the best candidate summary from the results of different systems.
(3) Multi-system Sentence-level: We also take a step towards the fine-grained fusion of summaries from extractive and abstractive systems. Specifically, here candidate summaries are generated by combining the results of different systems at the sentence level.
We mainly experiment on four datasets, whose statistics are shown in Tab. 1.
$\mathbf{CNNDM}^2$  (Hermann et al., 2015) is a widely used dataset containing news articles and the associated highlights which are used as the reference summaries. We follow the work of Nallapati et al. (2016) for data preprocessing.
$\mathbf{XSum}^3$  (Narayan et al., 2018a) contains online articles collected from BBC with highly abstractive one-sentence summaries.
PubMed4 (Cohan et al., 2018) contains scientific papers collected from PubMed.com.
Datasets	# Num	Avg. Len
Train	Valid	Test	Doc.	Sum.
CNNDM	287K	13K	11K	768.6	55.7
XSum	203K	11K	11K	429.2	23.3
PubMed	83K	4.6K	5K	468.7	210.3
WikiHow	168K	6K	6K	579.1	62.2
Table 1: Datasets Statistics. Len is the length of tokens.
WikiHow $^{5}$  (Koupae and Wang, 2018) is a large-scale dataset constructed from the articles using online WikiHow knowledge base.
5.2 Base Systems
Below, we mainly use BART, GSum and PEGASUS as the base systems since they have achieved state-of-the-art performance on at least one dataset.
BART (Lewis et al., 2020) is a large pre-trained sequence-to-sequence model that achieves strong performance on the abstractive summarization.
GSum (Dou et al., 2020) enhances the performance of BART using additional guidance information, which achieves the current state-of-the-art performance on the CNNDM dataset.
PEGASUS (Zhang et al., 2020) achieves competitive performance on various summarization datasets and is the current state-of-the-art on the XSum dataset.
To make a comprehensive evaluation of our proposed model, we additionally collect 19 top-scoring systems as base systems on CNNDM. In details, for §5.7 we use the following systems: pointer-generator+coverage (See et al., 2017), REFRESH (Narayan et al., 2018b), fastAbsRL-rank (Chen and Bansal, 2018), CNN-LSTM-BiClassifier (Kedzie et al., 2018), CNN-Transformer-BiClassifier (Zhong et al., 2019), CNN-Transformer-Pointer (Zhong et al., 2019), BERT-Transformer-Pointer (Zhong et al., 2019), Bottom-Up (Gehrmann et al., 2018), NeuSum (Zhou et al., 2018), BanditSum (Dong et al., 2018), twoStageRL (Zhang et al., 2019a), preSummAbs (Liu and Lapata, 2019), preSummAbs-ext (Liu and Lapata, 2019), HeterGraph (Wang et al., 2020), MatchSum (Zhong et al., 2020), Unilm-v1 (Dong et al., 2019), Unilm-v2 (Dong et al., 2019), T5 (Raffel et al., 2020).
$^{2}$ https://cs.nyu.edu/~kcho/DMQA/ $^{3}$ https://github.com/EdinburghNLP/XSum $^{4}$ https://github.com/ahohan/long-summarization
$^{5}$ https://github.com/mahNazkoupaew/WikiHow-Dataset  
 $^{6}$ Since CNNDM is the most popular dataset, we can collect more existing systems on it.
5.3 Baseline Systems
Neural system combinator: We use BERTScore (Zhang et al., 2019b) as an unsupervised baseline with neural models, which is an automatic evaluation metric computing the similarity of text pairs based on the corresponding BERT-encoded representations. We use it to directly compute the similarity score between the source documents and candidate summaries.
Non-Neural system combinator: We use RankSVM $^7$  (Joachims, 2002) as a non-neural baseline. We perform cross-validation on the development set for hyper-parameter searching and train the model on the development set. The set of features is listed in Appendix A.
Oracles: We compare our model with sample-wise Min, Max and Random oracles using ROUGE.
5.4 Training Details
For the following experiments in §5.5, §5.6 and §5.7 on CNNDM, we pre-train the Refactor model with a candidate set generated by enumerating combinations of sentences in the source documents. To reduce the number of candidates, we prune the sentences assigned with lower scores by an extractive model, BERTSum (Liu and Lapata, 2019), following Zhong et al. (2020). The maximum number of candidates for one data sample is 20. The pre-trained Refactor is also used as a base system in §5.6, whose outputs are used together with other base systems as candidate summaries. For different experiments, we fine-tune pre-trained Refactor on the base system's output, and name the model as fine-tuned Refactor. To analyze the effectiveness of the proposed two-stage training, we additionally train the model without the pre-training step, which is named as supervised Refactor.
The pre-trained BERT model we used is from Transformers library (Wolf et al., 2020). We use Adam optimizer (Kingma and Ba, 2015) with learning rate scheduling.
$$
l r = 0. 0 0 2 \cdot \min  (\text {s t e p} - \text {n u m} ^ {- 0. 5}, \tag {10}
$$
$$
\mathrm {s t e p \_ n u m} \cdot \mathrm {w a r m u p \_ s t e p s} ^ {- 1. 5}),
$$
where the warmup_steps is 10000. The model performance on the validation set is used to select the checkpoint. Pre-training takes around 40 hours
System	Method	R-1	R-2	R-L
BART	Base	44.26	21.12	41.16
Min	41.58	19.27	38.69
Max	47.22	23.28	43.90
Random	44.40	21.26	41.28
BERTScore	44.50	21.28	41.37
RankSVM	44.50	21.39	41.43
Supervised†	45.05	21.64	41.92
Pre-trained†	44.78	21.49	41.68
Fine-tuned†	45.15	21.70	42.00
GSum	Base	45.93	22.30	42.68
Min	44.37	21.25	41.29
Max	47.37	23.21	43.99
Random	45.84	22.22	42.61
BERTScore	45.84	22.25	42.64
RankSVM	46.04	22.29	42.78
Supervised †	46.11	22.32	42.85
Pre-trained	45.88	22.23	42.67
Fine-tuned†	46.18	22.36	42.91