SlowGuess/NAACL_2022_OCR huggingface dataset:
Provided summary quality did not impact the number of edits. Anticipating that participants might have needed to make more edits to improve on worse summaries, we compared the edit distance to provided summary quality (overall) using Spearman correlation. However, for neither Reddit nor XSum, did summary quality have a strong rela
Figure 3: Comparison between conditions for average time to summarize (per document) for Reddit and XSum. In general, participants in XSum took longer to complete the task, likely due to unfamiliarity with the domain.
tionship to edit distance. In some cases participants made many edits to both good and bad summaries, whereas in others, they made very few edits regardless of quality (see A.3.2 for correlation plots), due in part to participants' diverse editing styles, where some desired to make changes regardless of the provided summary quality; Moramarco et al. (2021) made similar observations. For example, P10 (Reddit, AI post-edit) "did not use [the provided summaries] at all" and P55 (Reddit, Human post-edit) edited all the summaries to match their preferred writing style, stating "I found the casual writing style confusing [...] I just did it my way."
Figure 4: User experience plots for task difficulty, "I found it difficult to summarize the article well", frustration, "Performing the summarization tasks was frustrating", and assistance utility, "The provided summaries were not useful to me when I was performing the summarization tasks" for Reddit (Left) and XSum (Right). Responses were collected using 7 point rating scales.
4.3 User Experience
We measured user experience with task difficulty, frustration, and assistance utility (Fig. 4). We also
Based on the trend, we would expect to see a significant result with more statistical power (more participants).
surfaced insights about participants' experiences from the qualitative analysis.
Participants found it harder to post-edit Reddit references. Summarization conditions significantly differed for task difficulty for Reddit  $(p = .04, F = 3.2)$ , but not for XSum  $(p = .72, F = .3)$ . Specifically, summarizing when provided a Reddit human reference (Human post-edit) was perceived significantly more difficult than summarizing when provided an AI-generated summary (AI post-edit,  $p = .04$ ). Other pairwise comparisons were not significant. Recall that post-editing human references also took longer than other conditions for Reddit; this difficulty might be due to the fact that Reddit human reference data consisted of poorly written TL;DRs, many of which add extra details not found in the original posts. As participants like P63 (Reddit, Human post-edit) and P58 (Reddit, Human post-edit) commented, some provided Reddit summaries were "really bad" or "off a bit."
Participants were mixed on whether the provided summaries were useful. Assistance utility did not significantly differ between summarization condition for Reddit  $(p = .12, F = 2.4)$  or XSum  $(p = .63, F = .2)$ , due in part to the high variability in participants' responses. However, participants provided mixed responses on the utility of the summaries: while many thought they were helpful "starting points" in their summarization process, others found they sometimes missed important points or contained information that was unneeded, incorrect, or incongruous with the original article.
Participants, therefore, used the provided summaries in different ways. Some, like P59 (Reddit, Human post-edit) used the summary as a starting point or guideline and made edits on top of it, "the provided summaries did the job pretty well, I just added some details." Others ignored the provided summary. As P61 (Reddit, Human post-edit) said, "it might have been easier to do blind summaries rather than having the provided examples." Some participants, like P31 (XSum, Human post-edit), chose to "read the passage, write my summary, and then look at the given summary."
Participants' had other concerns about post-editing the provided summaries during the task. Some thought it took high cognitive load to make edits and summarize at the same time, "it clouded my memory of what information the passage had actually provided" (P31 XSum, Human post-edit).
Others struggled with originality. While they perceived it "a bit like cheating" (P44 Reddit, AI post-edit) to use the provided summary instead of writing their own, many also "found it difficult to provide a better summary than what was provided" (P32 XSum, AI post-edit). Finally, some participants noticed that they had the tendency to over-rely on the provided summary. For example, P6 (XSum, Human post-edit) found that they were distracted from their own thinking and unlikely to challenge the provided summary, "the provided summaries deterred me from writing my own and gleaning my own major points from the articles. Instead, I would defer to the information given in the provided summaries and edit a few things, but not add anything major."
Comprehension of the original text can impact summarization. Participants found it challenging to summarize documents that lacked context, were overly detailed, or had poor quality. While we intentionally recruited participants with some expertise for the two data types (Reddit posts and British news), many were hindered by a lack of background knowledge, especially for the British news articles (XSum). For example, P18 (XSum, Manual) said it was difficult to summarize particular articles about "the British government" because "it's not something I am familiar with so it was hard to determine what information to include in the summary." Similarly, P15 (XSum, Manual) mentioned difficulty summarizing articles about Cricket, which contained "sport-specific jargon or proper nouns that I was wholly unfamiliar with." In fact,  $50\%$  of participants summarizing XSum described a lack of contextual knowledge compared to only  $8\%$  of participants summarizing Reddit posts. Post-editing can help in this case by providing a useful starting point so that users do not need to fully understand the document and write manually.
Participants also found it difficult to decide what was important from overly detailed original documents, particularly when summarizing manually. For example, P17 (XSum, Manual) stated, "some articles gave so many details and it took a while to decide which were important to keep in a summary." Finally, participants were hindered by the poor quality of the original documents. For example, P60 (Reddit, Human post-edit) viewed the "lack of capitalization and proper punctuation that is common with Reddit posts" as the greatest frustration in the summarization process. Participants
found it challenging to match the tone and style of these documents in their summaries. For example, in P46's (Reddit, AI post-edit) words, it is "hard to match in the winding, anecdotal writing style often found on Reddit."
4.4 Summary
Post-editing yielded better quality summaries than the automatic methods. However, compared to manual summarization, the results were mixed. For formal news articles, post-editing lead to similar quality summaries with improved efficiency, helping when participants lacked domain knowledge. However, post-editing produced worse summaries, more slowly for Reddit posts, likely due to the informal writing style and sometimes inaccurate TL;DR references provided in that case. We did not find a correlation between edit distance and provided summary quality, instead, some participants tended to make more edits—due to style or writing preferences—while others made fewer edits, instead of relying on the provided summaries—regardless of the quality of the summary they were editing.
This work is the first large-scale study of post-editing for text summarization, providing valuable insights on the benefits and drawbacks. We discuss these, as well as outline future research directions and design recommendations for post-editing summarization systems. Finally, we discuss the limitations of our experiments.
Post-editing was useful when domain context is needed. As our participants were not well-versed in the British news content of XSum, the provided summary was "useful" as a starting point (as described in qualitative responses), so that they did not have to write from scratch. This is similar to machine translation literature, which suggests that monolingual editors, despite lacking the knowledge of the other language, can still effectively improve the quality of translation via post-editing (Koehn, 2010). Beyond post-editing, systems could provide additional support when users lack domain knowledge or context, such as inline web searching to learn about unknown terms or phrases (e.g., the rules of Cricket).
Post-editing was less useful when the provided summaries were low quality. Low quality, particularly inaccurate or incoherent, provided sum
maries can be confusing and hard to edit, making them less "useful" as summarization assistance. Ideally, such summaries are not provided for postediting, as manual summarization would be better in those cases. Future systems should explore techniques for determining whether to provide a summary or not, based on desired summary qualities. Finally, systems might provide transparency, e.g., highlighting the important details in the original text (Lai and Tan, 2019). Then users can then decide for themselves whether or not those details are important and the summary should be trusted or ignored.
Post-editing can lead to over-reliance and stifle creativity. Humans have a tendency to over-rely on AI systems (Bussone et al., 2015; Buçinca et al., 2021): prior work on text generation found users consider the provided text as an "authority" and thus feel apprehensive to make significant edits (Bhat et al., 2021). In our experiment, some participants reported a similar tendency to over-rely on the provided summary or were distracted from writing their own version of the summary; some even developed their own combative strategies: writing their summaries manually first and then referring to the provided assistance. Therefore, future systems might allow different workflows, where summaries are shown before or after manual summarization (or not at all).
Post-editing systems should cater to varied users' preferences and needs. Users have varied summarization strategies and needs for assistance as a result of personal preferences and their experience with the domain. In our study, some preferred more control over the final summary, making lots of edits, while others made fewer edits, and a few did not use the provided summaries at all. These differences in writing style (and their effects on post-editing in text summarization) are also noted in prior work (Moramarco et al., 2021). Therefore, systems should give users control over the assistance they receive and alternative workflows, based on their preferences and needs.
Users' needs also vary by their target audience; users might desire summaries that are longer or shorter, or more formal or informal based on who they expect will read them. Post-editing can help, allowing users to tailor summaries to different audiences with the same underlying content. Future post-editing systems might provide multiple summary options, with diverse content and/or style,
from which users can choose.
Limitations. We note possible limitations of our results due to the length and nature of the task: participants only interacted with the summarization system for a short time (less than an hour) and for a task, for which they lacked ownership; also, including more participants would have given more statistical power for comparing conditions. Future work should perform experiments with more realistic and longer-term summarization engagements. Regarding our datasets, we chose two to differentiate between formal writing (i.e., news articles) and informal writing (i.e., social media posts). However, we did not experiment with more societally critical summarization tasks, such as medical or legal documents. While post-editing was useful when more domain context was needed, it is unclear how our findings would generalize to more high-risk scenarios.
To take advantage of the complimentary strengths of AI—which can produce summaries quickly—and humans—which can write summaries well—we explored how human-AI collaboration (i.e. post-editing) impacts summary quality, human efficiency, and user experience for text summarization. Through the first large-scale study on post-editing for text summarization, we provide valuable insights on the benefits and drawbacks: compared to summarizing manually, post-editing was helpful for formal news articles, where participants lacked domain knowledge, while post-editing was less helpful for informal social media posts, for which the reference TL;DR summaries sometimes included inaccurate information. We also observed differences in participants editing strategies and needs as well as concerns of over reliance, all of which deserve future exploration. We hope this initial exploration provides a starting point for future research on post-editing in text summarization.
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online. Association for Computational Linguistics.
Advait Bhat, Saaket Agashe, and Anirudha Joshi. 2021. How do people interact with biased text prediction models while writing? In Proceedings of the First Workshop on Bridging Human-Computer Interaction and Natural Language Processing, pages 116-121.
Rishi Bommasani and Claire Cardie. 2020. Intrinsic evaluation of summarization datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8075-8096, Online. Association for Computational Linguistics.
Zana Buçinca, Maja Barbara Malaya, and Krzysztof Z Gajos. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on ai in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1):1-21.
Adrian Bussone, Simone Stumpf, and Dympna O'Sullivan. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 international conference on healthcare informatics, pages 160-169. IEEE.
Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the original: Fact aware neural abstractive summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Florian Daniel, Pavel Kucherbaev, Cinzia Cappiello, Boualem Benatallah, and Mohammad Allahbakhsh. 2018. Quality control in crowdsourcing: A survey of quality attributes, assessment techniques, and assurance actions. ACM Computing Surveys (CSUR), 51(1):1-40.
Bonnie Dorr, David Zajic, and Richard Schwartz. 2003. Hedge trimmer: A parse-and-trim approach to headline generation. Technical report, MARYLAND UNIV COLLEGE PARK INST FOR ADVANCED COMPUTER STUDIES.
Wafaa S. El-Kassas, Cherif R. Salama, Ahmed A. Rafea, and Hoda K. Mohamed. 2021. Automatic text summarization: A comprehensive survey. Expert Systems with Applications, 165:113679.
Alexios Gidiotis and Grigorios Tsoumakas. 2021. Uncertainty-aware abstractive summarization. arXiv preprint arXiv:2105.10155.
Youxuan Jiang, Catherine Finegan-Dollak, Jonathan K. Kummerfeld, and Walter Lasecki. 2018. Effective crowdsourcing for a new type of summarization task.
In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 628-633, New Orleans, Louisiana. Association for Computational Linguistics.
Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive summarization of Reddit posts with multi-level memory networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2519-2531, Minneapolis, Minnesota. Association for Computational Linguistics.
Margaret R Kirkland and Mary Anne P Saunders. 1991. Maximizing student performance in summary writing: Managing cognitive load. *Tesol Quarterly*, 25(1):105-121.
Philipp Koehn. 2010. Enabling monolingual translators: Post-editing vs. options. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 537-545.
Maarit Koponen. 2012. Comparing human perceptions of post-editing effort with post-editing operations. In Proceedings of the seventh workshop on statistical machine translation, pages 181-190.
Maarit Koponen. 2016. Is machine translation post-editing worth the effort? a survey of research into post-editing and effort. The Journal of Specialised Translation, 25:131-148.
Anastassia Kornilova and Vladimir Eidelman. 2019. *BillSum: A corpus for automatic summarization of US legislation*. In *Proceedings of the 2nd Workshop on New Frontiers in Summarization*, pages 48-56, Hong Kong, China. Association for Computational Linguistics.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9332-9346, Online. Association for Computational Linguistics.
Vivian Lai and Chenhao Tan. 2019. On human predictions with explanations and predictions of machine
learning models: A case study on deception detection. In Proceedings of the conference on fairness, accountability, and transparency, pages 29-38.
Guy Lev, Michal Shmueli-Scheuer, Jonathan Herzig, Achiya Jerbi, and David Konopnicki. 2019. Talk-Summ: A dataset and scalable annotation method for scientific paper summarization based on conference talks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2125-2131, Florence, Italy. Association for Computational Linguistics.
Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of research and development, 2(2):159-165.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Francesco Moramarco, Alex Papadopoulos Korfiatis, Aleksandar Savkov, and Ehud Reiter. 2021. A preliminary study on evaluating consultation notes with post-editing. *EACL* 2021, page 62.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarrunner: A recurrent neural network based sequence model for extractive summarization of documents. In AAAI.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Ani Nenkova and Kathleen McKeown. 2012. A survey of text summarization techniques. In Mining text data, pages 43-76. Springer.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Mirko Plitt and François Masselot. 2010. A productivity test of statistical machine translation post-editing in a typical localisation context. In Prague Bull. Math. Linguistics.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In ACL (1), pages 1073-1083.
Nisan Stiannon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325.
Xiangru Tang, Alexander R Fabbri, Ziming Mao, Griffin Adams, Borui Wang, Haoran Li, Yashar Mehdad, and Dragomir Radev. 2021. Investigating crowdsourcing protocols for evaluating the factual consistency of summaries. arXiv preprint arXiv:2109.09195.
Oguzhan Tas and Farzad Kiyani. 2007. A survey automatic text summarization. *PressAcademia Procedia*, 5(1):205-213.
Lucas Nunes Vieira. 2019. Post-editing of machine translation. In *The Routledge Handbook of Translation and Technology*, pages 319–336. Routledge.
Michael Volske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59-63.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Justin Jian Zhang and Pascale Fung. 2012. Active learning with semi-automatic annotation for extractive speech summarization. ACM Transactions on Speech and Language Processing (TSLP), 8(4):1-25.
Our study explored how providing summaries for post-editing affects summary quality, efficiency, and user experience compared to fully manual or fully automatic approaches. The study involved two phases: (1) summary collection (Appendix A.1) and (2) human evaluation of the collected summaries (Appendix A.2). We also report on additional results (Appendix A.3)
A.1 Summary Collection
We collected summaries through a summarization task, where participants first reviewed documents (from either Reddit or XSum) and summarized them, either without any assistance or provided with a human-written or model-generated summary they could edit.
In the following, we describe details on the documents included in our study, how we piloted the task and interface, and more information about the study procedure.
A.1.1 Document Length Distribution
Participants summarized 120 documents from the test sets (10 documents per participant, per condition),<sup>11</sup> with length between the 25th and 75th percentile to balance task difficulty and time. Figure 5 gives the length distribution for both datasets.
Figure 5: This figure shows the document length of distribution of both datasets. The average length of the Reddit posts is 243.8 words, and the average length of the XSum articles is 222.3 words.
A.1.2 Piloting the Summarization Task and Interface
We performed two pilot study sessions (with researchers and Upwork pilot participants) for feedback on the web application, procedure, and to estimate session duration. The first was conducted among five researchers from our lab and the second
was conducted with 12 representative users from Upwork (see §3), which were later included in the main study.
Criteria	Explanation
Essence	The summary is a good representation of the post.
Clarity	The summary is reader-friendly. It expresses ideas clearly.
Accuracy	The summary contains the same information as the longer post
Purpose	The summary serves the same purpose as the original post.
Style	The summary is written in the same style as the original post.
Table 2: We showed this set of summary criteria to the participants in both tutorial and actual task.
A.1.3 Summarization Study Procedure