SlowGuess/ACL_2020_OCR huggingface dataset:
Philippe Laban
Andrew Hsi
John Canny
Marti A. Hearst
UC Berkeley*
This work presents a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. It introduces a novel method that encourages the inclusion of key terms from the original document into the summary: key terms are masked out of the original document and must be filled in by a coverage model using the current generated summary. A novel unsupervised training procedure leverages this coverage model along with a fluency model to generate and score summaries. When tested on popular news summarization datasets, the method outperforms previous unsupervised methods by more than 2 R-1 points, and approaches results of competitive supervised methods. Our model attains higher levels of abstraction with copied passages roughly two times shorter than prior work, and learns to compress and merge sentences without supervision.
Summarization, or the task of condensing a document's main points into a shorter document, is important for many text domains, such as headlines for news and abstracts for research papers.
This paper presents a novel unsupervised abstractive summarization method that generates summaries directly from source documents, without the aid of example summaries. This approach simultaneously optimizes for the following important properties of a good summary:
- coverage of the keywords of the document,
- fluency of generated language, and
- brevity of generated summaries.
Original Document: Chilean President announced Wednesday that his country, which has been paralyzed by protests over the last two weeks, will no longer host two major international summits. [...] The President has now canceled the hosting of the economic APEC forum and COP25 environmental summit, which were both due to take place later this year. [...]
Masked Document: ____ announced Wednesday that his country, which has been ____ by ____ over the last two weeks, will no longer ____ two major international ____ [...]. The ____ has now ____ the ____ of the ____ and ____ which were both due to take place later this ____ [...].
Summary Loop [10 word constraint]: Pinera cancelled the APEC summit at Santiago.
Coverage Score: 0.22
Summary Loop [24 word constraint]: Pinera said Chileans have been canceled the hosting of the APEC summit, which was scheduled to take place in November. Coverage score: 0.33
Summary Loop [45 word constraint]: Sebastian Pinera announced Wednesday that his country will not hold the APEC summit, which was scheduled to take place in Santiago. Pinera said that Chileans had been paralyzed by protests over the last two weeks.
Coverage score: 0.39
Figure 1: Motivating example. A document from CNN.com (keywords generated by masking procedure are bolded), the masked version of the article, and generated summaries by three Summary Loop models under different length constraints.
One of the main contributions of this work is a novel method of inducing good coverage of important concepts from the original article. The coverage model we propose takes as input the original document with keywords masked out (see Figure 1). It uses the current best automatically generated summary to try to uncover the missing keywords. The more informative the current summary is, the more successful the coverage model is at guessing the blanked out keywords from the original document. A resulting coverage score is fed back into the training process of the summarization model
*Author emails: {phillab,canny,hearst} @berkeley.edu, ahsil@bloomberg.net
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5135-5150 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
with the objective of producing summaries with high coverage.
A second contribution is our unsupervised training procedure for summarization, the Summary Loop, which leverages the coverage model as well as a simple fluency model to generate and score summaries. During training, the procedure is conditioned on a desired summary length, forcing the Summarizer model to adapt to a length budget. Figure 1 shows Summary Loop summaries obtained for the same document under three different length budgets.
A third contribution is a set of specialized techniques employed during training to guide the model away from pathological behavior. These guard rails include a method for reducing repetition, for encouraging the model to complete sentences, and to avoid frame filling patterns.
The models trained through the Summary Loop outperform all prior unsupervised summarization methods by at least 2 ROUGE-1 points on common news summarization datasets (CNN/DM and Newsroom), and achieve within a few points of state-of-the-art supervised algorithms, without ever being exposed to any summaries. In addition, summaries generated by our method use  $50\%$  more summarization techniques (compression, merging, etc.) than prior automatic work and achieve higher levels of abstraction, reducing by almost half the gap between human-generated summaries and automatic summaries in terms of length of copied spans.
Supervised Abstractive Summarization.
Sequence-to-sequence (seq2seq) (Sutskever et al., 2014) models trained using teacher-forcing are the most common approach to abstractive summarization (Nallapati et al., 2016). A common architecture is the Pointer-Generator (See et al., 2017). Performance can further be improved by constraining the attention (Gehrmann et al., 2018; Gui et al., 2019; Wang et al., 2019) and using pretrained Transformer-based language models (Lewis et al., 2019; Chi et al., 2019; Edunov et al., 2019). Through architectural changes, the training procedure remains constant: using a large corpus of document-summary pairs, the model is trained to reproduce target summaries.
Unsupervised Summarization. Most unsupervised summarization work is extractive: sentences
deemed relevant are pulled out of the original document and stitched into a summary, based on a heuristic for a sentence's relevance (Mihalcea and Tarau, 2004; Barrios et al., 2015; West et al., 2019). Nikolov and Hahnloser (2019)'s abstractive approach is partially unsupervised, not requiring parallel data, but only a group of documents and a group of summaries. In contrast, our work does not require any summaries, and is trained using only documents. Radford et al. (2019) summarize documents using a language model (GPT2) in a Zeroshot learning setting. The model reads the document followed by a special token "TL/DR", and is tasked with continuing the document with a summary. Our work is an extension of this work: we initialize our Summarizer model with a GPT2 and specialize it with a second unsupervised method.
Summarization and Q&A. Eyal et al. (2019) and Arumae and Liu (2018) turn reference summaries into fill-in-the-blank (FIB) questions, either as an evaluation metric or to train an extractive summarization model. In this work, we directly generate FIB questions on the document being summarized, bypassing the need for a reference summary.
Scialom et al. (2019)'s work stays closer to a Q&A scenario, and uses a Question Generation module to generate actual questions about the document, answered by a Squad-based (Rajpurkar et al., 2018) model using the generated summary. We refrain from using actual questions because question generation remains a challenge, and it is unclear how many questions should be generated to assess the quality of a summary.
RL in Summarization. Paulus et al. (2018) introduced Reinforcement Learning (RL) to neural summarization methods by optimizing for ROUGE scores, leading to unreadable summaries. Since then, Reinforcement Learning has been used to select sentences with high ROUGE potential (Chen and Bansal, 2018), or optimize modified versions of ROUGE that account for readability (Pasunuru and Bansal, 2018). In all cases, the reward being computed relies on a reference summary, making the methods supervised. We craft a reward that does not require a target summary allowing our training process to remain unsupervised.
3 The Summary Loop
For this work, the definition of a summary is:
"A summary is a brief, fluent text that
covers the main points of an original document."
Brevity, fluency and coverage are the three pillars of a good summary. Under a length constraint, a good quality summary should contain as much information about the original document as possible while retaining fluent and coherent English.
Subsection 3.1 lays out the steps in the Summary Loop. Subsections 3.2-3.5 specify how each component is represented by a neural network. Section 4 shows how to train a summarizer model using this architecture in an unsupervised manner. $^{1}$
3.1 Summary Loop Steps
Numbers in Figure 2 correspond to the following steps:
1. Summarizer receives a document D and length-constraint L, and produces a summary S fulfilling the length constraint.
2. Using a Masking Procedure, D is modified into a masked document M, where important words have been replaced with blanks.
3. Coverage receives S and M, and uses them to fill in each blank in M with a word, producing F. F and D are compared, and the resulting fill-in accuracy is called the Coverage Score.
4. Fluency receives S, and gives a Fluency Score based on its assessment of the quality of the Summary's writing.
5. The Fluency Score is added to the Coverage Score (as a weighed sum) into a Summary Score for the (D, S) pair.
6. Reinforcement Learning is used to train the Summarizer to produce summaries with high Summary Score.
The Summary Loop does not rely on the use of a target/reference/human-written summary, but only the summaries produced by the Summarizer model. The process can therefore be iterated upon without supervision from Summarization datasets.
We use a Generative Transformer (Radford et al., 2019) as the model architecture of the summarizer. We make this choice for two reasons. First, Generative Transformers can produce text one word at a time, allowing the system to produce abstractive
Figure 2: The Summary Loop involves three neural models: Summarizer, Coverage and Fluency. Given a document and a length constraint, the Summarizer writes a summary. Coverage receives the summary and a masked version of the document, and fills in each of the masks. Fluency assigns a writing quality score to the summary. The Summarizer model is trained, other models are pretrained and frozen.
summaries. Second, we use the pretrained Generative Transformer to initialize the Summarizer.
Practically, the Summarizer first reads through the entire document, followed by a special START token, signaling summarization. The Summarizer produces a probability distribution over words in its vocabulary, and a word is picked from the distribution and fed back as an input into the model. This procedure is repeated and halts either when the summary reaches a length constraint, or when the Summarizer produces a special END token. See Appendix C for the model size and initialization used to train the summarization paper.
3.3 Masking Procedure
The Masking Procedure decides on a set of keywords that are important elements in the document that should be recoverable using a summary. The keywords are replaced with blanks, indirectly indicating which information should be present in the summary. We use a tfidf-based approach to decide on the set of masked keywords, as it is both simple and has been shown to represent word relevance to a document (Ramos, 2003). Masking procedure implementation details are presented in Section A of the Appendix.
We select the k words with highest tfidf score for the document to serve as the masked words. The k parameter represents a balance: if too many words are masked, the filling-in becomes impos
1The code, model checkpoints and other resources are available at https://github.com/CannyLab/summary_loop.
Figure 3: The Coverage model uses a finetuned BERT model. The summary is concatenated to the masked document as the input, and the model predicts the identity of each blank from the original document. The accuracy obtained is the raw coverage score.
sible, but if too few are masked, the Summarizer model will not be encouraged to include sufficient content in its summary. Varying the value of  $k$  (10,12,15,20) yielded only small discernible difference in the Summarizers produced, and we use  $k = 15$  in all our final experiments.
The masking procedure can be adapted to a specific domain. For instance, if summarizing financial documents, the masking procedure could systematically mask all numbers, encouraging the Summarizer model to add numbers to its summary.
3.4 Coverage Model
The Coverage Model receives a computationally generated summary and the masked document and attempts to fill in each blank word. The task of filling in blanks is similar to masked language modeling (MLM), used to pretrain BERT-like (Devlin et al., 2019) models. In MLM, some of the words are replaced with a special  $\text{MASK}$  token, and the model must use other information (unmasked words) to fill in the masked words. Because of the similarity to our task, we use a BERT-based neural network as the architecture for the coverage model. However, the coverage task differs from MLM in two ways. First, we modify the masking procedure: instead of masking a random percentage of the words (often  $15\%$  for BERT), we mask all appearances of the keywords selected by the masking procedure described in Section 3.3. Second, the input to the coverage model is a concatenation of the unmasked summary, a separator token and the masked document. The model can leverage un
masked information available in the summary to fill in the masked document. The Coverage Model is illustrated in Figure 3.
3.4.1 Computing a Coverage Score
Using the masking procedure, we obtain  $M = f(D)$ , the masked document. The coverage model produces the filled document  $F = g(M, S)$ . Raw coverage score is the fraction of correctly filled in words in F. Let  $D_i, F_i$  and  $M_i$  correspond to the  $i$ th word in their respective document,  $I_M$  the set indices of words that have been masked. Then:
$$
\operatorname {R a w C o v} (D, S) = \frac {\| i \in I _ {M} \text {i f} D _ {i} = F _ {i} \|}{\| I _ {M} \|} \tag {1}
$$
The model can use information in the unmasked (visible) words of M to predict the masked words. For instance, if the word "Chile" is visible, then "Santiago" would be a well-informed guess near the word "capital", which might not be masked out. This is undesirable, because coverage should account for what information the model can learn from the summary S, not what it can guess from the unmasked portion of D. To counteract this problem, we modify the raw coverage score by computing how much information the model can guess without the summary present, using an empty string summary:  $F_{\emptyset} = g(M, \dots)$ . We then normalize a summary's coverage by subtracting the empty string coverage from the raw coverage, leaving only filled-in words answerable using S, as shown in Equation 2.
$$
\operatorname {N o r m C o v} (D, S) = \tag {2}
$$
$$
\operatorname {R a w C o v} (D, S) - \operatorname {R a w C o v} (D, " ")
$$
In a nutshell, raw coverage score answers the question: "What fraction of blanked words can be correctly filled in with this summary?" and normalized coverage score answers: "What is the increase in the fraction of blanks that can be correctly filled in with this summary, compared to having no summary?" In the rest of this paper, Coverage Score refers to Normalized Coverage Score.
3.4.2 Training the Coverage Model
We train the Coverage Model once, and its weights are then fixed during the training of the Summarizer. In order to train the Coverage Model, we need pairs of documents (D) and summaries (S). However, we operate under the assumption that we do not have access to summaries (to keep the procedure unsupervised). In order to remove this dependency, we use the first 50 words of the unmasked
Summary Dataset	Summary Length	Raw Coverage	Norm. Coverage
Empty String	0	0.334	0
Headline	9.59	0.478	0.144
First 10 words	10.0	0.428	0.094
Newsroom	23.41	0.525	0.191
First 24 words	24.0	0.537	0.203
CNN/DM	45.75	0.726	0.392
First 46 words	46.0	0.649	0.315
Table 1: Analysis of the raw and normalized coverage of three existing human-written summary datasets, as well as first-k word baselines.
document  $(D[:50])$  as a proxy for document summaries. The Coverage Model is initialized with a trained BERT model (Devlin et al., 2019), and trained using  $(D,D[:50])$  pairs on the coverage task. Because BERT is already trained on the similar MLM task, the Coverage model is able to leverage knowledge accrued by BERT. The Coverage Model converges after roughly 5 hours of training on a Titan X GPU.
3.4.3 Analysis of Coverage
We present properties of the raw and normalized coverage through the analysis of existing human-written summary datasets. We focus our analysis on three datasets in the news domain: (1) a headline dataset obtained from common US news websites (Laban and Hearst, 2017), (2) the Newsroom dataset (Grusky et al., 2018), and (3) the CNN/DM dataset (Nallapati et al., 2016).
For each dataset, we take document/summary pairs and obtain raw and normalized coverage score through our Coverage model, reported in Table 1.
First, longer summaries obtain higher coverage scores: a CNN/DM summary with an average of 45 words can be used to fill in  $73\%$  of the blanks correctly, compared to  $48\%$  for a 9 word headline. Across datasets, the correlation between summary length and raw coverage score is 0.56, confirming that longer summaries contain more information, according to coverage.
Second, we simulate the first  $k$  words $^2$  of the document as a summary. We use  $k = 10, 24, 46$  to match average word length in the three datasets. For two of the three values (10 and 46), the coverage of human-written summaries is higher than the first- $k$  word counterpart. This is remarkable: even though the summary is farther away lexically (i.e.,
is not a subset of the original words), it obtains higher coverage, demonstrating that the coverage model can account for reworded information.
3.5 Fluency Model
A model solely trained to optimize coverage has no incentive to write in good English, use punctuation, determinants or pronouns, as these are not words removed by the masking procedure. The objective of a Fluency Model is to judge the writing quality of the summary, independent of its coverage.
Given the right corpus, we argue that a language model's probability can be modified into a Fluency Score. Therefore, we adapt a language model into the Fluency Model.
We choose the generative Transformer (Radford et al., 2019) architecture for our Fluency model, as it can be trained into a powerful language model. Just as with the Summarizer, by using a standardized architecture and model size, we can make use of pretrained models. However, it is important for Fluency to fine tune the language model on the target domain, so that the Summarizer is rewarded for generating text similar to target content.
To produce a uniform Fluency Score, we linearly scale the language model's log-probability of a given summary  $(LM(S))$  between an ideal value  $LP_{low}$  and a maximum value  $LP_{high}$ :
$$
\operatorname {F l u e n c y} (S) = 1 - \frac {L M (S) - L P _ {l o w}}{L P _ {h i g h} - L P _ {l o w}} \tag {3}
$$
This ensures that the Fluency  $(S)$  is usually in the range [0, 1].  $LP_{low}$  and  $LP_{high}$  are picked specifically for a particular language model, and ensure that the log-probability magnitudes of a specific language model do not affect the overall scores.
3.6 Summary Score
The final Summary Score is a weighed sum of the Coverage and Fluency Scores:
$$
\begin{array}{l} \operatorname {S u m m a r y S c o r e} (D, S) = \\ \alpha \cdot \operatorname {N o r m C o v} (D, S) + \beta \cdot \operatorname {F l u e n c y} (S) \tag {4} \\ \end{array}
$$
$\alpha, \beta$  are hyperparameters giving relative importance to Coverage and Fluency. We set  $\alpha = 5$ ,  $\beta = 1$  in all our experiments. Model choice, size, and initialization are summarized in Figure A1.
4 Training Procedure
We first outline the training procedure and then detail several guard-rail mechanisms used during
2We choose the first k words due to the similarity to Lede 3 (first 3 sentences), a common baseline in news.
training to prevent the Summarizer from learning pathological writing strategies. Figure A2 presents training plots of a Summary Loop model and interpretation of the different learning phases.
4.1 Training with Reinforcement Learning
We use Reinforcement Learning to train the Summarizer component (agent), such that it achieves high summary score (reward). Note that the Coverage and Fluency models are frozen, and their weights are not trained. We make this choice as allowing Fluency and Coverage models to evolve could enable the models to coordinate and cheat.
We use the Self-critical sequence training (SCST) method (Rennie et al., 2017), as it has been shown to perform well on similar text generation tasks optimizing BLEU for image captioning or ROUGE scores in summarization.
In SCST, the Summarizer is used to produce two summaries of document  $D$ : a greedy summary  $\hat{S}$ , using a decoding strategy that always picks the most likely next word, and a sampled summary  $S^s$ , picking the next word in the summary by sampling from the word distribution.
Summaries are scored using the Summary Loop:
$$
\begin{array}{r} \hat {R} = S u m m a r y S c o r e (D, \hat {S}) \\ R ^ {s} = S u m m a r y S c o r e (D, S ^ {s}) \end{array}
$$
Then we minimize the following loss:
$$
L = \left(\hat {R} - R ^ {s}\right) \sum_ {i = 0} ^ {N} \log p \left(w _ {i} ^ {s} \mid w _ {1} ^ {s}, \dots , w _ {i - 1} ^ {s}, D\right)
$$
Where  $p(w_i^s | \ldots)$  represent the probability of the  $i$ th word conditioned on previously generated word, according to the model.
Intuitively, if  $R^s > \hat{R}$ , minimizing L maximizes the likelihood of the sampled sequence — which is desired because it outperformed the greedy summary — and increases expected reward of the model.
4.2 Training guard rails
During training, the Summarizer model learns pathological summarization strategies. We build training guard rails to detect the pathological behavior and penalize the model during training.
A guard rail has a binary effect: if a pathology is detected in a summary, its Summary Score is reduced by a penalty amount  $\delta$ . We use  $\delta = 2$  for all experiments. We found three training guard rails to be useful: No-repetition, Finish-your-sentence, and No-frame-filling.
4.2.1 No-repetition
A common problem in neural text generation is repetition of text. Based on the observation that 3-grams seldom repeat in common summarization datasets, the "No-repetition" training guard rail raises a penalty on a summary when it contains any repeated 3-gram.
4.2.2 Finish-your-sentence
When generating a summary, the model can either produce the END token, or generate a number of words up to the length constraint. We observe that if the model does not produce the END token, it often generates partial sentences, which is undesirable. Because we want to encourage the model to generate an END token, the "Finish-your-sentence" raises a penalty if a summary has no END token.
4.2.3 No-frame-filling
During training, the model sometimes learns to overly rely on sentence patterns that achieves high reward as a one size fits all summary. In one example the model learns to produce summaries solely of the form: "X talks with Y about the Z". The model uses this frame, filling in the X, Y and Z slots with relevant keywords and entities to achieve a small but positive coverage. This form of frame-filling is undesirable, as the model often produces inaccurate information to fit the entities to the pattern.