SlowGuess/ACL_2020_OCR huggingface dataset:
Faisal Ladhak $^{1*}$ , Bryan Li $^{2*}$ , Yaser Al-Onaizan $^{3}$ , Kathleen McKeown $^{1,3}$
<sup>1</sup>Columbia University, <sup>2</sup>University of Pennsylvania, <sup>3</sup>Amazon AI
faisal.ladhak@columbia.edu, bryanli@seas.upenn.edu,
onaizan@amazon.com, kathy@cs.columbia.edu
We present a new summarization task, generating summaries of novel chapters using summary/chapter pairs from online study guides. This is a harder task than the news summarization task, given the chapter length as well as the extreme paraphrasing and generalization found in the summaries. We focus on extractive summarization, which requires the creation of a gold-standard set of extractive summaries. We present a new metric for aligning reference summary sentences with chapter sentences to create gold extracts and also experiment with different alignment methods. Our experiments demonstrate significant improvement over prior alignment approaches for our task as shown through automatic metrics and a crowd-sourced pyramid analysis.
When picking up a novel one is reading, it would be helpful to be reminded of what happened last. To address this need, we develop an approach to generate extractive summaries of novel chapters. This is much harder than the news summarization tasks on which most of the summarization field (e.g., (Cheng and Lapata, 2016; Grusky et al., 2018; Paulus et al., 2017)) focuses; chapters are on average seven times longer than news articles. There is no one-to-one correspondence between summary and chapter sentences, and the summaries in our dataset use extensive paraphrasing, while news summaries copy most of their information from the words used in the article.
We focus on the task of content selection, taking an initial, extractive summarization approach given the task difficulty. As the reference sum
maries are abstractive, training our model requires creating a gold-standard set of extractive summaries. We present a new approach for aligning chapter sentences with the abstractive summary sentences, incorporating weighting to ROUGE (Lin, 2004) and METEOR (Lavie and Denkowski, 2009) metrics to enable the alignment of salient words between them. We also experiment with BERT (Devlin et al., 2018) alignment.
We use a stable matching algorithm to select the best alignments, and show that enforcing one-to-one alignments between reference summary sentences and chapter sentences is the best alignment method of those used in earlier work.
We obtain a dataset of summaries from five study guide websites paired with chapter text from Project Gutenberg. Our dataset consists of 4,383 unique chapters, each of which is paired with two to five human-written summaries.
We experiment with generating summaries using our new alignment method within three models that have been developed for single document news summarization (Chen and Bansal, 2018; Kedzie et al., 2018; Nallapati et al., 2017). Our evaluation using automated metrics as well as a crowd-sourced pyramid evaluation shows that using the new alignment method produces significantly better results than prior work.
We also experiment with extraction at different levels of granularity, hypothesizing that extracting constituents will work better than extracting sentences, since summary sentences often combine information from several different chapter sentences. Here, our results are mixed and we offer an explanation for why this might be the case.
Our contributions include a new, challenging summarization task, experimentation that reveals potential problems with previous methods for creating extracts, and an improved method for creating gold standard extracts.
*Equal contribution. Work done while at Amazon.  
<sup>1</sup>We tried two abstractive models (Chen and Bansal, 2018; Liu and Lapata, 2019) but ROUGE was low and the output was poor with many repetitions and hallucinations.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5043-5054 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Relatively little work has been done in summarization of novels, but early work (Mihalcea and Ceylan, 2007) provided a dataset of novel/summary pairs drawn from CliffsNotes and GradeSaver and developed an unsupervised system based on Meade (Radev et al., 2001) and TextRank (Mihalcea and Tarau, 2004) that showed promise. More recently, Zhang et al. (2019) developed an approach for summarizing characters within a novel. We hypothesize that our proposed task is more feasible than summarizing the full novel.
Previous work has summarized documents using Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) to extract elementary discourse units (EDUs) for compression and more content-packed summaries (Daumé III and Marcu, 2002; Li et al., 2016; Arumae et al., 2019). Some abstractive neural methods propose attention to focus on phrases within a sentence to extract (Gehrmann et al., 2018). Fully abstractive methods are not yet appropriate for our task due to extensive paraphrasing and generalization.
While previous work on semantic textual similarity is relevant to the problem of finding alignments between chapter and summary text, the data available (Cer et al., 2017; Dolan and Brockett, 2005) is not suitable for our domain, and the alignments we generated from this data were of a poorer quality than the other methods in our paper.
We collect summary-chapter pairs from five online study guides: BarronsBookNotes (BB), BookWolf (BW), CliffsNotes (CN), GradeSaver (GS) and NovelGuide (NG). We select summaries from these sources for which the complete novel text can be found on Project Gutenberg.
Our initial dataset, for summaries with two or more sources, includes 9,560 chapter/summary pairs for 4,383 chapters drawn from 79 unique books. As our analysis shows a very long tail, two rounds of filtering were applied. First, we remove reference texts with  $>700$  sentences, as these are too large to fit into mini-batches ( $\sim 10\%$  of data). Second, we remove summaries with a compres
Summary Src	Mean (stdev)	Median	Total #
CN	442 (369)	347	1,053
BB	517 (388)	429	1,000
GS	312 (311)	230	1,983
BW	276 (232)	214	182
NG	334 (302)	244	2,070
All Sources	373 (339)	279	6,288
Chapter Text	5,165 (3,737)	4,122	6,288
Table 1: Train Split Statistics: World count statistics with total number for summaries and chapter text.
sion ratio of  $< 2.0$ , as such wordy summaries often contain a lot of commentary (i.e. phrases that have no correspondence in the chapter,  $\sim 5\%$ ).
This results in 8,088 chapter/summary pairs, and we randomly assign each book to train, development and test splits (6,288/938/862 pairs respectively). After filtering, chapters are on average seven times longer than news articles from CNN/Dailymail (5,165 vs 761 words), and chapter summaries are eight times longer than news summaries (372 vs 46 words).
Train split statistics are given in Table 1. These statistics reveal the large variation in length. Furthermore, we calculate word overlap, the proportion of vocabulary that overlaps between the summary and chapter. For novels, this is  $33.7\%$ ; for CNN/DailyMail news, this is  $68.7\%$ . This indicates the large amount of paraphrasing in the chapter summaries in relation to the original chapter.
In Figure 1, we show the first three sentences of a reference summary for Chapter 11, The Awakening which is paraphrased from several, nonconsecutive chapter sentences shown near the bottom of the figure. We also show a portion of the summaries from two other sources which convey the same content and illustrate the extreme level of paraphrasing as well as differences in detail. We show the full chapter and three full reference summaries in Appendix A.2.
4 Alignment Experiments
To train models for content selection, we need saliency labels for each chapter segment that serve as proxy extract labels, since there are no gold extracts. In news summarization, these are typically produced by aligning reference summaries to the best matching sentences from the news article. Here, we align the reference summary sentences with sentences from the chapter.
We address two questions for aligning chapter
2We do not have the rights to redistribute the data. To allow others to replicate the dataset, we provide a list of novel chapters we used at https://github.com/ manestay/novel-chapter-dataset
GS: In this chapter Mr. and Mrs. Pontellier participate in a battle of wills. When Mr. Pontellier gets back from the beach, he asks his wife to come inside. She tells him not to wait for her, at which point he becomes irritable and more forcefully tells her to come inside.
NG: Mr. Pontellier is surprised to find Edna still outside when he returns from escorting Madame Lebrun home. ... although he asks her to come in to the house with him, she refuses, and remains outside, exercising her own will.
BW: Leonce urges Edna to go to bed, but she is still exhilarated and decides to stay outside in the hammock...
Chapter sentences: He had walked up with Madame Lebrun and left her at the house. "Do you know it is past one o'clock? Come on," and he mounted the steps and went into their room. "Don't wait for me," she answered. "You will take cold out there," he said, irritably. "What folly is this? Why don't you come in?"
Figure 1: Portions of three reference summaries for The Awakening, Chapter 11 by Kate Chopin, along with chapter sentences they summarize.
and summary sentences to generate gold standard extracts: 1) Which similarity metric works best for alignment (Section 4.1)? and 2) Which alignment method works best (Section 4.2)?
4.1 Similarity Metrics
ROUGE is commonly used as a similarity metric to align the input document and the gold standard summary to produce gold extracts (Chen and Bansal, 2018; Nallapati et al., 2017; Kedzie et al., 2018). One drawback to using ROUGE as a similarity metric is that it weights all words equally. We want to, instead, assign a higher weight for the salient words of a particular sentence.
To achieve this, we incorporate a smooth inverse frequency weighting scheme (Arora et al., 2017) to compute word weights. The weight of a given word is computed as follows:
$$
W \left(w _ {i}\right) = \frac {\alpha}{\alpha + p \left(w _ {i}\right)} \tag {1}
$$
where  $p(w_{i})$  is estimated from the chapter text and  $\alpha$  is a smoothing parameter (here  $\alpha = 1e^{-3}$ ). N-gram and Longest Common Subsequence (LCS) weights are derived by summing the weights of each of the individual words in the N-gram/LCS. We take the average of ROUGE-1, 2, L using this weighting scheme as the metric for generating extracts,  $R$ -wtd, incorporating a stemmer to match
morphological variants (Porter, 1980).
Similarity Metrics Results: We compare R-wtd against ROUGE-L (Chen and Bansal, 2018) (R-L), and ROUGE-1, with stop-word removal and stemming (Kedzie et al., 2018) (R-1), for sentence alignment. To incorporate paraphrasing, we average METEOR (Banerjee and Lavie, 2005) scores with ROUGE-1,2,L for both un-weighted (RM) and weighted scores (RM-wtd). Given the recent success of large, pre-trained language models for downstream NLP tasks, we also experiment with BERT (Devlin et al., 2019) to compute alignment, using cosine similarity between averaged chapter segment and summary segment vectors. We compare the generated gold extracts using R-L F1 against reference summaries, to determine a shortlist for human evaluation (to save costs).
For the human evaluation, we ask crowd workers to measure content overlap between the generated alignments, and the reference summary, on a subset of the validation data. For each summary reference, they are shown a generated alignment and asked to indicate whether it conveys each of up to 12 summary reference sentences. An example task is shown in Appendix Figure 7. We then compute precision and recall based on the number of summary sentences conveyed in the extract.
Table 2 shows that humans prefer alignments generated using R-wtd by a significant margin.3 Sample alignments generated by R-wtd in comparison to the baseline are shown in Figure 2.
Method	RM	R-wtd	RM-wtd	R-1	R-L	BERT
R-L F1	41.2	40.6	39.3	37.1	35.1	35.4
H-F1	33.7	44.8	38.8	-	-	-
Table 2: ROUGE-L F1, and crowd-sourced F1 scores (H-F1) for content overlap.
4.2 Alignment Methods
Some previous work in news summarization has focused on iteratively picking the best article sentence with respect to the summary, in order to get the gold extracts (Nallapati et al., 2017; Kedzie et al., 2018), using ROUGE between the set of selected sentences and the target summary. In contrast, others have focused on picking the best article sentence with respect to each sentence in the summary (Chen and Bansal, 2018). We investigate which approach yields better alignments. We refer
3We suspect incorporating METEOR by averaging didn't work because the scale is different from ROUGE scores.
to the former method as summary-level alignment and the latter method as sentence-level alignment.
For sentence-level alignment, we note that the problem of finding optimal alignments is similar to a stable matching problem. We wish to find a set of alignments such that there exists no chapter segment  $a$  and summary segment  $x$  where both  $a$  and  $x$  would prefer to be aligned with each other over their current alignment match. We compute alignments based on the Gale-Shapley algorithm (1962) for stable matching and compare it with the greedy approach from prior work (Chen and Bansal, 2018).
For summary-level alignment (Nallapati et al., 2017; Kedzie et al., 2018), we compare two variants: selecting sentences until we reach the reference word count (WL summary), and selecting sentences until the ROUGE score no longer increases (WS summary).
Crowd-sourced evaluation results (Table 3) show that sentence-level stable matching is significantly better. We use this in the remainder of this work. These differences in alignments affect earlier claims about the performance of summarization systems, as they were not measured, yet have a significant impact.
Method	P	R	F1
Greedy Sent	48.4	48.7	48.5
Stable Sent	52.8	52.6	52.7
WL summary	34.5	36.6	36.7
WS summary	42.7	36.6	38.0
Table 3: Crowd sourced evaluation on content overlap for summary vs. sentence level on validation set.
Ref summary: He says he will, as soon as he has finished his last cigar.
R-L greedy: "You will take cold out there," he said, irritably.
R-L stable: He drew up the rocker, hoisted his slippered feet on the rail, and proceeded to smoke a cigar.
R-wtd stable: "Just as soon as I have finished my cigar."
Figure 2: A reference summary sentence and its alignments. R-L greedy and R-L stable are incorrect because they weight words equally (e.g. said, cigar, '').
5 Summarization Experiments
In order to assess how alignments impact summarization, we train three extractive systems – hierarchical CNN-LSTM extractor (Chen and Bansal, 2018) (CB), seq2seq with attention (Kedzie et al., 2018) (K), and RNN (Nallapati et al., 2017) (N). The target word length of generated summaries is based on the average summary length of similarly long chapters from the training set.
We also experiment with aligning and extracting at the constituent level, given our observation during data analysis that summary sentences are often drawn from two different chapter sentences. We create syntactic constituents by taking sub-trees from constituent parse trees for each sentence (Manning et al., 2014) rooted with S-tags. To ensure that constituents are long enough to be meaningful, we take the longest S-tag when one S-tag is embedded within others (see Appendix A.5).
Summary quality is evaluated on F1 scores for R-{1,2,L}, and METEOR. Each chapter has 2-5 reference summaries and we evaluate the generated summary against all the reference summaries. Part of a generated summary of extracted constituents for Chapter 11, The Awakening, is shown in Figure 3. The full generated summaries for this chapter (both extracted constituents and extracted sentences) are shown in Appendix A.2.
Generated Summary: |I thought I should find you in bed , ” ||said her husband , |when he discovered her |lying there . |He had walked up with Madame Lebrun and left her at the house . ||She heard him moving about the room ; |every sound indicating impatience and irritation .|
Figure 3: System generated summary, extracted constituents in teal, and separated by |.
We compare our method for generating extractive targets (ROUGE weighted, with stable matching at the sentence level) against the baseline method for generating extractive targets for each of the systems. Table 4 shows three rows for each summarization system: using the original target summary labels, and using either constituent or sentence segments. We see our proposed alignment method performs significantly better for all mod
<sup>5</sup>We do so by binning chapters into 10 quantiles by length.
Prior work has used EDUs, but automated parsers such as (Ji and Eisenstein, 2014) perform poorly in this domain.
4Bold text indicates statistical significance with  $p <   0.05$
Model	Seg	Method	R-1	R-2	R-L	METEOR
CB	sent	baseline	33.1	5.5	30.0	13.9
sent	R-wtd	35.8	6.9	33.4	15.2
const	R-wtd	36.2	6.9	35.4	15.2
K	sent	baseline	34.3	6.4	31.6	14.6
sent	R-wtd	35.6	6.9	33.2	15.0
const	R-wtd	36.2	6.9	35.2	15.1
N	sent	baseline	34.6	6.4	31.9	14.6
sent	R-wtd	35.7	7.0	33.3	15.1
const	R-wtd	35.9	7.0	35.2	15.0
els. ROUGE-L in particular increases  $10\%$  to  $18\%$  relatively over the baselines. Moreover, it would seem at first glance that the K and N baseline models perform better than the CB baseline, however this difference has nothing to do with the architecture choice. When we use our extractive targets, all three models perform similarly, suggesting that the differences are mainly due to small, but important, differences in their methods for generating extractive targets.
Human Evaluation: Given questions about the reliability of ROUGE (Novikova et al., 2017; Chaganty et al., 2018), we perform human evaluation to assess which system is best at content selection. We use a lightweight, sampling based approach for pyramid analysis that relies on crowd-sourcing, proposed by Shapira et al. (2019), and correlates well with the original pyramid method (Nenkova et al., 2007). We ask the crowd workers to indicate which of the sampled reference summary content units are conveyed in the generated summary.
We evaluated our best system + alignment on extraction of sentences and of constituents (CB R-wtd), along with a baseline system (CB K-align), using the crowd-sourced pyramid evaluation method. To produce readable summaries for extracted constituents, each extracted constituent is included along with the context of the containing sentence (black text in Figure 3). We find that CB Sent R-wtd has significantly higher content overlap with reference summaries in Table 5.
We present a new challenging task for summarization of novel chapters. We show that sentence
Table 4: ROUGE-F1, METEOR for generated summaries. "Baseline" is the method used for that model.
System	Pyramid Score
CB K-align	17.9
CB Sent R-wtd	18.9
CB Const R-wtd	18.1
Table 5: Crowd-sourced Pyramid Evaluation.
level, stable-matched alignment is better than the summary-level alignment used in previous work and our proposed  $R$ -wtd method for creating gold extracts is shown to be better than other similarity metrics. The resulting system is the first step towards addressing this task.
While both human evaluation and automated metrics concur that summaries produced with our new alignment approach outperform previous approaches, they contradict on the question of whether extraction is better at the constituent or the sentence level. We hypothesize that because we use ROUGE to score summaries of extracted constituents without context, the selected content is packed into the word budget; there is no potentially irrelevant context to count against the system. In contrast, we do include sentence context in the pyramid evaluation in order to make the summaries readable for humans and thus, fewer constituents make it into the generated summary for the human evaluation. This could account for the increased score on automated metrics.
It is also possible that smaller constituents can be matched to phrases within the summary with metrics such as ROUGE, when they actually should not have counted. In future work, we plan to experiment more with this, examining how we can combine constituents to make fluent sentences without including potentially irrelevant context.
We would also like to further experiment with abstractive summarization to re-examine whether large, pre-trained language models (Liu and Lapata, 2019) can be improved for our domain. We suspect these models are problematic for our documents because they are, on average, an order of magnitude larger than what was used for pretraining the language model (512 tokens). Another issue is that the pre-trained language models are very large and take up a substantial amount of GPU memory, which limits how long the input document can be. While truncation of a document may not hurt performance in the news domain due to the heavy lede bias, in our domain, truncation can hurt the performance of the summarizer.
7See the screen shot in Appendix A.4 8We use the best baseline alignment, Kedzie et al. (2018) with the CB model to keep model choice consistent.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Kristjan Arumae, Parminder Bhatia, and Fei Liu. 2019. Towards annotating and creating summary highlights at sub-sentence level. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 64-69, Hong Kong, China. Association for Computational Linguistics.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguistics.
Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643-653, Melbourne, Australia. Association for Computational Linguistics.
Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675-686.
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 484-494, Berlin, Germany. Association for Computational Linguistics.
Hal Daumé III and Daniel Marcu. 2002. A noisychannel model for document compression. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 449-456. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.