SlowGuess/ACL_2020_OCR huggingface dataset:
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
David Gale and Lloyd S Shapley. 1962. College admissions and the stability of marriage. The American Mathematical Monthly, 69(1):9-15.
Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098-4109, Brussels, Belgium. Association for Computational Linguistics.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 708-719, New Orleans, Louisiana. Association for Computational Linguistics.
Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 13-24.
Chris Kedzie, Kathleen McKeown, and Hal Daume III. 2018. Content selection in deep learning models of summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1818-1828.
Alon Lavie and Michael J Denkowski. 2009. The meteor metric for automatic evaluation of machine translation. Machine translation, 23(2-3):105-115.
Junyi Jessy Li, Kapil Thadani, and Amanda Stent. 2016. The role of discourse units in near-extractive summarization. In Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 137-147, Los Angeles. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of
the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3728-3738, Hong Kong, China. Association for Computational Linguistics.
William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: toward a functional theory of text organization. Text: Interdisciplinary Journal for the Study of Discourse, 8(3):243-281.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55-60.
R. Mihalcea and P. Tarau. 2004. TextRank: Bringing order into texts. In Proceedings of EMNLP-04 and the 2004 Conference on Empirical Methods in Natural Language Processing.
Rada Mihalcea and Hakan Ceylan. 2007. Explorations in automatic book summarization. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 380-389, Prague, Czech Republic. Association for Computational Linguistics.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In *Thirty-First AAAI Conference on Artificial Intelligence*.
Ani Nenkova, Rebecca Passonneau, and Kathleen McKeown. 2007. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Trans. Speech Lang. Process., 4(2).
Jekaterina Novikova, Ondrej Dusek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241-2252, Copenhagen, Denmark. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
Martin F Porter. 1980. An algorithm for suffix stripping. Program, 14(3):130-137.
Dragomir Radev, Sasha Blair-Goldensohn, and Zhu Zhang. 2001. Experiments in single and multi-document summarization using MEAD. In First Document Understanding Conference, New Orleans, LA.
Ori Shapira, David Gabay, Yang Gao, Hadar Ronen, Ramakanth Pasunuru, Mohit Bansal, Yael Amsterdamer, and Ido Dagan. 2019. Crowdsourcing lightweight pyramids for manual summary evaluation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 682-687, Minneapolis, Minnesota. Association for Computational Linguistics.
Weiwei Zhang, Jackie Chi Kit Cheung, and Joel Oren. 2019. Generating character descriptions for automatic summarization of fiction. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 7476-7483.
A.1 Acknowledgments
We would like to thank Spandana Gella for her contributions to the project. We would like to thank Jonathan Steuck, Alessandra Brusadin, and the rest of the AWS AI Data team for their invaluable feedback in the data annotation process. We would finally like to thank Christopher Hidey, Christopher Kedzie, Emily Allaway, Esin Durmus, Fei-Tzin Lee, Feng Nan, Miguel Ballesteros, Ramesh Nallapati, and the anonymous reviewers for their valuable feedback on this paper.
A.2 Example Chapter and Summaries
We show the full text of Chapter 11, The Awakening by Kate Chopin in Figure 4. We show three reference summaries in Figure 5, and two generated summaries using our best alignment method in Figure 6. While there are differences in length and level of detail, there are also clearly similarities in covered content.
A.3 Target Word Length for Summaries
The target word length for generated summaries is a function of the input chapter word count  $(wc_{\text{chapter}})$ .
We divide the train set into 10 quantiles, and in each quantile (or bin), associate it to the mean compression ratio  $(CR)$ :
$$
C R = \frac {w c _ {\text {c h a p t e r}}}{w c _ {\text {r e f . s u m m}}} \tag {2}
$$
$$
C R _ {\text {q u a n t i l e}} = \frac {1}{n} \sum_ {i = 1} ^ {n} C R _ {i} \tag {3}
$$
Where  $wc_{refsumm}$  is the word count of the reference summary, and  $CR_{i}$  is the compression ratio of the i-th quantile item.
The target word length for the generated summary  $(wc_{gen\_summ})$  is given by:
$$
w c _ {g e n \_ s u m m} = \frac {1}{C R _ {\text {q u a n t i l e}}} * w c _ {\text {c h a p t e r}} \tag {4}
$$
Generated summaries are created by extracting segments with the highest model probability until this budget is reached (without truncation). Oracle summaries also use this target word length, but may be shorter if the original summary had few segments (as we extract one chapter segment for each summary segment).
Quantile	Min wc	Max wc	CR
1	44	1,232	6.67
2	1,233	1,711	9.09
3	1,712	2,174	9.09
4	2,175	2,758	10.00
5	2,579	3,361	11.11
6	3,362	4,165	12.5
7	4,166	5,374	14.29
8	5,375	7,762	14.29
9	7,763	13,028	16.67
10	13,029	70,436	20
Table 6: Quantiles: For each quantile (bin), we show its max and min word words, and its compression ratio.
A.4 SCU Evaluation Task Setup
To obtain the distractors, we sample 2 SCUs from different chapters from the same book. We insert one of them, the positive distractor, into the generated summary, as well as into the list of statements, so it will always be correct. We insert the other, the negative distractor, only into the list of statements, so it will always be incorrect.
A.5 Constituent Extraction algorithm
Algorithm 1 extracts subtrees from a constituent parse tree. These subtrees are constituents, and break down sentences into meaningful spans of text. Constituents are one of
1. A relative clause
2. The highest level S or SBAR node in its subtree with (NP, VP) children
3. The highest level VP node above 2)
4. The remaining nodes in the tree that were not extracted with 1), 2) or 3)
"What are you doing out here, Edna? I thought I should find you in bed," said her husband, when he discovered her lying there. He had walked up with Madame Lebrun and left her at the house. His wife did not reply.
"Are you asleep?" he asked, bending down close to look at her.
"No." Her eyes gleamed bright and intense, with no sleepy shadows, as they looked into his.
"Do you know it is past one o'clock? Come on," and he mounted the steps and went into their room.
"Edna!" called Mr. Pontellier from within, after a few moments had gone by.
"Don't wait for me," she answered. He thrust his head through the door.
"You will take cold out there," he said, irritably. "What folly is this? Why don't you come in?"
"It isn't cold; I have my shawl."
"The mosquitoes will devour you."
"There are no mosquitoes."
She heard him moving about the room; every sound indicating impatience and irritation. Another time she would have gone in at his request. She would, through habit, have yielded to his desire; not with any sense of submission or obedience to his compelling wishes, but unthinkingly, as we walk, move, sit, stand, go through the daily treadmill of the life which has been portioned out to us.
"Edna, dear, are you not coming in soon?" he asked again, this time fondly, with a note of entreaty.
"No; I am going to stay out here."
"This is more than folly," he blurted out. "I can't permit you to stay out there all night. You must come in the house instantly."
With a writhing motion she settled herself more securely in the hammock. She perceived that her will had blazed up, stubborn and resistant. She could not at that moment have done other than denied and resisted. She wondered if her husband had ever spoken to her like that before, and if she had submitted to his command. Of course she had; she remembered that she had. But she could not realize why or how she should have yielded, feeling as she then did.
"Leonce, go to bed," she said, "I mean to stay out here. I don't wish to go in, and I don't intend to. Don't speak to me like that again; I shall not answer you."
Mr. Pontellier had prepared for bed, but he slipped on an extra garment. He opened a bottle of wine, of which he kept a small and select supply in a buffet of his own. He drank a glass of the wine and went out on the gallery and offered a glass to his wife. She did not wish any. He drew up the rocker, hoisted his slippered feet on the rail, and proceeded to smoke a cigar. He smoked two cigars; then he went inside and drank another glass of wine. Mrs. Pontellier again declined to accept a glass when it was offered to her. Mr. Pontellier once more seated himself with elevated feet, and after a reasonable interval of time smoked some more cigars.
Edna began to feel like one who awakens gradually out of a dream, a delicious, grotesque, impossible dream, to feel again the realities pressing into her soul. The physical need for sleep began to overtake her; the exuberance which had sustained and exalted her spirit left her helpless and yielding to the conditions which crowded her in.
The stillest hour of the night had come, the hour before dawn, when the world seems to hold its breath. The moon hung low, and had turned from silver to copper in the sleeping sky. The old owl no longer hooted, and the water-oaks had ceased to moan as they bent their heads.
Edna arose, cramped from lying so long and still in the hammock. She tottered up the steps, clutching feebly at the post before passing into the house.
"Are you coming in, Leonce?" she asked, turning her face toward her husband.
"Yes, dear," he answered, with a glance following a misty puff of smoke. "Just as soon as I have finished my cigar."
Figure 4: Full chapter text. Note that this is short at 847 words, as the median chapter length is 3168 words.
BookWolf summary: Léonce urges Edna to go to bed, but she is still exhilarated and decides to stay outside in the hammock. Léonce stays up with her and smokes his cigars. Edna feels defiant towards her husband and resents his control over her life. Eventually tiredness overcomes Edna and she goes to bed.
GradeSaver summary: In this chapter Mr. and Mrs. Pontellier participate in a battle of wills. When Mr. Pontellier gets back from the beach, he asks his wife to come inside. She tells him not to wait for her, at which point he becomes irritable and more forcefully tells her to come inside. Mrs. Pontellier resolves not to go in and thinks about how, on another occasion, she would have just done what her husband asked, simply because of inertia. Feeling stubborn and strong, she realizes that she had never taken such a stand against her husband before.
Mr. Pontellier then decides to join her outside. He drinks glasses of wine and smokes a number of cigars. After awhile, Mrs. Pontellier feels like she is being awakened from a dream and realizes that she is quite fatigued. It is almost dawn. Finally getting up from the hammock, Mrs. Pontellier asks her husband if he's going to join her. He replies that he will, after he finishes his cigar.
NovelGuide summary: Mr. Pontellier is surprised to find Edna still outside when he returns from escorting Madame Lebrun home. In a small but no doubt significant exchange-considering the events of the evening, and the novel's title-her distant and unperceiving husband asks her, "Are you asleep?" Edna, with eyes "bright and intense," definitively replies, "No." Although he asks her to come in to the house with him, she refuses, and remains outside, exercising her own will. As if trying to outlast his wife, Mr. Pontellier smokes cigar after cigar next to her. Gradually, Edna succumbs to her need for sleep. She feels "like one who awakens gradually out of a . . . delicious, grotesque, impossible dream . . . ." As described in Chapter VII, then, Edna is once again undergoing what might be called a "negative" "awakening"-an "awakening" to the realities of her present life-as opposed to the "positive" awakening to new possibilities and her own self-direction, to which the nighttime swim began to expose her. As if to underscore her failure to "awaken" to herself, the chapter ends with a scene of tables being turned: as Edna goes in, she asks her husband if he will be joining her. He says he will, as soon as he has finished his last cigar. While the narrator does not record Mr. Pontellier's tone of voice, the comments seem almost scornful, mockingly echoing Edna's earlier self-assertion.
Figure 5: Two reference summaries.
Constituent R-wtd: |I thought I should find you in bed , ” ||said her husband , |when he discovered her |lying there . |He had walked up with Madame Lebrun and left her at the house . ||She heard him moving about the room ; |every sound indicating impatience and irritation . |“ This is more than folly , ” |he blurted out . ‘ I can’t |permit you to stay out there all night .||But she could not realize |why or how she should have yielded , feeling as she then did . |He smoked two cigars ; |then he went inside and drank another glass of wine . She tottered up the steps , |clutching feebly at the post before passing into the house . |she asked , |turning her face toward her husband . |
Sentence R-wtd: | I thought I should find you in bed , ” said her husband , when he discovered her lying there . | | He had walked up with Madame Lebrun and left her at the house . | | His wife did not reply . | | “ This is more than folly , ” he blurted out . | | You must come in the house instantly . ” | | Edna began to feel like one who awakens gradually out of a dream , a delicious , grotesque, impossible dream , to feel again the realities pressing into her soul . | | She tottered up the steps, clutching feebly at the post before passing into the house . | | she asked , turning her face toward her husband . | | “ Just as soon as I have finished my cigar . ” |
Figure 6: Two generated summaries. Extracted segments are highlighted in teal, and delineated with |. Constituents are presented with context, whereas sentences extract all text.
Figure 7: An example HIT showing a segmented oracle summary, and two questions. Reading the summary, we see that we should answer "Present" for both questions. There can be up to 12 questions – we omit here for brevity. Note that in our evaluation, we counted both "Present" and "Partially Present" as a match.