SlowGuess/NAACL_2022_OCR huggingface dataset:
Vivian Lai\*1, Alison Smith-Renner 2, Ke Zhang 2, Ruijia Cheng\*3, Wenjuan Zhang 2, Joel Tetreault 2, Alejandro Jaimes 2
<sup>1</sup> University of Colorado Boulder, vivian.lai@colorado.edu
<sup>3</sup> University of Washington, rcheng6@uw.edu
$^{2}$  Dataminr Inc., {arenner, kzhang, wzhang, jtetreault, ajaimes}@dataminr.com
Automatic summarization methods are efficient but can suffer from low quality. In comparison, manual summarization is expensive but produces higher quality. Can humans and AI collaborate to improve summarization performance? In similar text generation tasks (e.g., machine translation), human-AI collaboration in the form of "post-editing" AI-generated text reduces human workload and improves the quality of AI output. Therefore, we explored whether post-editing offers advantages in text summarization. Specifically, we conducted an experiment with 72 participants, comparing post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience on formal (XSum news) and informal (Reddit posts) text. This study sheds valuable insights on when post-editing is useful for text summarization: it helped in some cases (e.g., when participants lacked domain knowledge) but not in others (e.g., when provided summaries include inaccurate information). Participants' different editing strategies and needs for assistance offer implications for future human-AI summarization systems.
Text summaries provide short overviews of long documents or document collections, allowing readers to understand the content without the need to read full documents. For example, news summaries outline key points so that readers do not have to read the entire article. For scientific papers, abstracts allow readers to easily understand the extent of the work and decide whether the paper is relevant to them. While these human-written summaries are typically high quality, a human's time and energy is limited and such tasks require heavy cognitive load (Kirkland and Saunders, 1991).
Therefore, increasing research effort has explored machine models to generate summaries automatically (Tas and Kiyani, 2007; Nenkova and McKeown, 2012; El-Kassas et al., 2021). While recent advances in learning algorithms and data have resulted in models that can generate relatively high-quality summaries, human summarization is still the gold standard. Further, training models require large, high-quality summarization datasets that are expensive to curate.
Taking advantage of the complementary strengths of humans and AI, can they collaborate to improve summarization performance? In the area of machine translation, a common method of human-AI collaboration is human post-editing of AI-generated text, which increases human productivity and improves the quality of translation (Koponen, 2016; Vieira, 2019). However, in spite of its potential impact, studies of post-editing for summarization have been very limited, e.g. Moramarco et al. (2021), in the medical domain.
To bridge this gap, we performed a large-scale human subject experiment (72 participants) investigating the utility of post-editing provided summaries in text summarization for informal (Reddit) and formal (news) datasets. We expect tradeoffs in quality and efficiency (i.e., it might take longer to write better summaries), so we are interested in whether post-editing can actually improve efficiency or quality over manual methods, as well as the effects on users' experience. This work is an important step toward understanding the benefits and drawbacks of post-editing as opposed to manual text summarization.
Our main contributions can be summarized as follows: (1) we present the first large, human subjects experiment of post-editing for text summarization; (2) we show how post-editing impacts summary quality, efficiency, and user experience—where it is useful and where it is not; and (3) we create, and make public, two new datasets, each with
* Research work was done while authors were interning at Dataminr Inc.
Human Language Technologies, pages 502 - 519
360 human-evaluated summaries for news and Reddit posts—either written manually or post-edited on provided summaries.
2.1 Automatic Text Summarization
The field of automatic text summarization can be traced back to 1950s (Luhn, 1958); and since then much research has been devoted to developing algorithms, datasets, and evaluation metrics to developing summarization systems that can approach the quality of a human (El-Kassas et al., 2021). There are two primary automatic methods: (1) extractive (Dorr et al., 2003; Nallapati et al., 2017), where the model selects important sentences from the input document, and (2) abstractive (Rush et al., 2015; Paulus et al., 2018), where important parts of the input document are paraphrased to form new sentences. While recent deep learning-based summarization methods have significantly advanced the quality of AI-generated summaries, they face some common issues, including hallucination, also known as contextual inconsistency (Maynez et al., 2020) and factual inconsistency (Cao et al., 2018; Kryscinski et al., 2020). These critical issues limit the utility of automatic summarization if unaddressed; in fact, humans can be in the loop to manually fix identified mistakes, thus iteratively improving AI models (Zhang and Fung, 2012; Gidiotis and Tsoumakas, 2021).
2.2 Human Text Summarization
Summaries written by humans often serve as gold standard references to train and evaluate AI models (Bhandari et al., 2020). One natural source of human summaries is shared and collected on the web, e.g., titles of news articles (See et al., 2017; Narayan et al., 2018), TL;DR of Reddit posts (Volske et al., 2017; Kim et al., 2019), talk transcripts of scientific papers (Lev et al., 2019), and government bill summaries (Kornilova and Eidelman, 2019). Those are generated to serve specific goals and audiences, and often contribute datasets at scale to build AI models, although the data quality is not guaranteed (Kryscinski et al., 2019; Bommasani and Cardie, 2020). Alternatively, human summaries can be annotated by dedicated professionals or crowd-workers for domain-specific documents (Jiang et al., 2018). Annotators are often trained with guidance and summarization criteria, yet quality control (Daniel et al., 2018), due
to subjectivity and inconsistency between annotators (Tang et al., 2021), is a challenge. While the annotation process is costly and time-consuming, human summarization, often by domain experts, yields higher quality compared to automatic methods (Zhang et al., 2020). In this work, we turn to a common method of human-AI collaboration—human post-editing of AI-generated text—as an exploration for a viable solution.
2.3 Post-Editing AI-Generated Text
Post-editing is a common technique in machine translation, where translators edit the translations produced by automatic methods (opposed to completing the translations manually). It has been shown to increase productivity and improve translation quality (Plitt and Masselot, 2010; Koponen, 2016; Vieira, 2019), particularly when initial translations are good. However, post-editing longer segments can require more cognitive effort to identify errors and plan corrections (Koponen, 2012).
As summarization shares similarities to machine translation, post-editing is a promising paradigm, yet it is underexplored. One exception is Moramarco et al. (2021), who evaluated post-editing in the medical domain. In a study with 3 physicians, participants took less time to post-edit other physician's written notes as compared to AI-generated notes, and post-editing any type of notes was faster than writing an entire note from scratch. Participants' note-taking style differences also affected post-editing time. For example, Doctor A wrote shorter notes and only edited AI-generated notes when there were substantial issues while Doctor B was more meticulous and edited the AI-generated notes extensively. We build on prior work and explore post-editing for text summarization at a larger scale (72 participants) over two domains.
3 Evaluating Post-Editing for Text Summarization
We explored how providing summaries for post-editing affects (RQ1) final summary quality, (RQ2) efficiency, and (RQ3) user experience, compared to fully manual or fully automatic approaches for two domains: social media and news. Participants reviewed documents and summarized them, either without any assistance (manual) or provided with a human-written or AI-generated summary that they could edit (post-edit). A distinct set of annotators then evaluated the quality of the summaries.
We included both human-written and AI-generated summaries in our study to explore post-editing for different summary types and qualities.
3.1 Data and Model
We chose social media posts and news articles for our study as they could be understood by a general audience and are commonly experimented with automatic summarization literature. We also chose these datasets as they vary in writing formality, which might impact how humans understand and summarize text. Specifically, we used the Reddit-TIFU dataset (Kim et al., 2019) (informal, Reddit "Today I F'd Up" posts) and the Extreme Summarization (XSum) dataset (Narayan et al., 2018) (formal, British news articles). Each of these datasets includes human-written "reference" summaries for the original documents: Reddit-TIFU uses the "TL;DR" written by the author of the post<sup>1</sup> while XSum uses the introductory sentences—written by journalists—as the summaries (see Table 3).
For participants to summarize during our study, we randomly selected 120 documents from the test sets (10 documents per participant, per condition), $^{2}$  with length between the 25th and 75th percentile to balance task difficulty and time. The average length of the Reddit posts is 243.8 words, and the average length of the XSum articles is 223.3 words (see Appendix A.1.1 for length distribution).
We used the Pegasus model (Zhang et al., 2020) to generate summaries for the two datasets. Pegasus is a masked language model pre-trained with a novel self-supervised objective, gap-sentences generation, and fine-tuned on downstream abstractive summarization tasks. The model achieved state-of-the-art performance on multiple datasets, including XSum and Reddit-TIFU. We directly applied the off-the-shelf Pegasus models downloaded from HuggingFace, with one already finetuned on XSum<sup>3</sup> and the other on Reddit-TIFU<sup>4</sup>.
We did not introduce summaries from any other models besides Pegasus, as the goal of this paper was not to compare models but to understand how human post-editing of provided summaries compares to manual and automatic methods. And, while Pegasus is currently high-performing compared to other, weaker models, the summaries we
provided for post-editing in our study were of varied quality, particularly between datasets (see §4.1). This gave us an opportunity to explore how summary (or assistance) quality might affect human post-editing.
3.2 Study Design
This study consists of two phases: (1) summary collection and (2) human evaluation of the collected summaries. For summary collection, we used a between-subjects experimental design, with three conditions: (1) Manual, where participants wrote summaries without any assistance; (2) AI post-edit, where participants post-edited AI-generated summaries; and (3) Human post-edit, where participants post-edited human-written summaries. Participants summarized either informal Reddit posts or formal XSum news articles. For the Human post-edit condition, participants were provided the human written "reference" summaries from each of the datasets. In the following, we describe the participants and procedure for the summary collection phase, followed by details of the evaluation phase.
Condition	WPM (M, σ)
Reddit	Manual	334, 113
AI post-edit	374, 175
Human post-edit	363, 104
XSum	Manual	327, 112
AI post-edit	360, 241
Human post-edit	396, 178
Table 1: Average reading speed per condition per dataset. WPM represents words per minute. Strategic assignment to conditions ensured no significant differences between reading speed.
3.3 Summary Collection Participants
We recruited 72 participants (45 female, 22 male, 3 non-binary, 2 preferred not to disclose) from Upwork. They were on average 32 years old  $(\sigma = 12)$  and were required to be native or bilingual English speakers, have at least a  $90\%$  job success score, and possess expertise in writing, journalism, or communication. To ensure participants had some familiarity with the summarization domains, they described their experience reading or posting on Reddit and knowledge of British news. Specifically, participants rated the extent of the respective
<sup>5</sup>https://www.upwork.com; we used Upwork—as opposed to other crowdsourcing platforms—to recruit experienced participants and ensure higher quality summaries.
<sup>1</sup>Reddit users often self-summarize their posts with "TL;DR:" or "too long; didn't read:" statements.
2Reddit-TIFU, Xsum
$^{3}$ huggingface-pegasus-xsum
huggingface-pegasus-redit_tifu
knowledge based on a 7-point Likert scale, and were selected if they responded at a rating of 4 or above. Finally, participants reported their reading (in words per minute, WPM) and comprehension scores, which we used to (1) eliminate those with comprehension less than  $75\%$  and (2) account for reading speed when assigning conditions. To account for differences in participants' reading speed that could affect our results, we assigned participants into conditions, ensuring a similar average reading speed across conditions (Table 1). The average reading speed for participants in our study was 358 WPM  $(\sigma = 158)$ .
3.4 Summary Collection Procedure
Based on pilot studies (see Appendix A.1.2), we anticipated the summary collection task sessions would take an hour and we paid participants $20. Each participant took on average 33.9 minutes ( $\sigma = 15.0$ ) to perform the summarization task. Of the 72 participants, an equal number (12) were randomly assigned to each dataset (Reddit or XSum) and summarization condition (Manual, Human post-edit, AI post-edit), ensuring a similar average reading time for each condition (see Table 1). During the study, participants completed three phases: (1) instructions, tutorial and practice, (2) summarization task, and (3) post-task survey.
Participants first reviewed task instructions, the criteria for writing a good summary (from Stiennon et al. (2020)), and examples of good and bad summaries with explanations. For consistency, we used the same criteria when asking annotators (a distinct set of human evaluators) to evaluate the summary quality. Participants then reviewed 10 documents (either all Reddit posts or all XSum news articles, depending on their assignment) and summarized each, either manually (Manual) or post-editing a provided human summary (Human post-edit) or AI-generated summary (AI post-edit). Fig. 1 shows an example of the task interface for the AI post-edit condition and XSum. Participants were not made aware of the source of their provided summaries—whether human or AI. Per condition, each participant summarized a unique set of 10 documents. Participants had access to the summarization criteria as guidance while summarizing. After completing each summary, participants rated the difficulty for summarizing the original document. Finally,
News article 1/10
The album, currently untitled, will be released in the autumn by Motown Records. "If your a rapper I suggest you run because I'm about to come for all of you sucks," Hart wrote in a strongly worded post on Instagram. "I'll just knocked on the door of a small giant that loves chicken nuggets and long walks in the park." Hart achieved fame as a stand-up before enjoying further success as the star of such films as Get Hard, The Wedding Ringer and Ride Along. He has already been seen this summer in the action comedy Central Intelligence, alongside Dwayne "The Rock" Johnson. And he can be heard in hit animated film The Secret Life of Pets as the voice of a short-tempered rabbit named Snowball. The 37-year-old's previous albums have all been recordings of his stand-up shows.
Write summary here
Comedian and actor Kevin Hart is to release his first solo album.
save summary
Figure 1: Sample task interface for the AI post-edit condition for XSum, showing the provided, AI-generated summary in the text box.
after finishing all 10 summaries, participants completed a survey about their experience. See A.1.3 for details on the procedure and tutorial examples.
3.5 Human Evaluation of Summary Quality
To evaluate the quality of the summaries written during our study, we recruited a distinct set of annotators from Amazon Mechanical Turk. To ensure quality ratings, we only employed turkers who satisfied the following criteria: (1) completed 5000 HITs; (2)  $97\%$  HIT approval rate; (3) reside in the United States, Australia, and United Kingdom. Annotators underwent tutorials and multiple attention-check questions before performing the task (see A.2.2). We also eliminated annotators with validation procedures (see A.2.3). Annotators were paid $1.50 per HIT (see A.2.2) and were allowed to perform multiple HITs, assuming they would improve at the evaluation task over time. Each annotator performed 9.4 HITs on average.
The annotators evaluated six different summaries for each original document (XSum article or Reddit post) from our study: (1) the Manual summary written without any assistance, (2) the summary written in the AI post-edit condition, given (3) the AI-generated summary from the Pegasus model, (4) the summary written in the Human post-edit condition, given (5) the human reference from the dataset, and, finally, (6) a random summary generated by randomly selecting two sentences from the opposite dataset. See Table 3 in A.2.1 for examples of each summary type. The random summary was used as selection criteria to identify annotators who were not paying attention during the task. Following Stiennon et al. (2020), annotators evaluated each summary on four axes: coherence, accuracy, coverage, and overall. Each summary was evaluated by five annotators; we removed outliers then averaged the remaining annotators ratings to de
$^{6}$ https://swiftread.com
7 Participants spent additional time completing the posttask survey.
termine the final rating for each summary. Refer to A.2 for additional details on the human summary evaluation procedure, annotators, and quality control.
3.6 Measures
We report on summary quality, efficiency, user experience for the summarization task. Summary quality is measured using human ratings and efficiency is measured by the amount of time to read and write a summary for each document. We additionally measured user workload as edit distance or the difference between the provided summary and the final summary.
We also report on three subjective user experience measures collected using 7 point rating scales (from strongly disagree to strongly agree) either on the task-level (at the end of the task) or instance-level (after each summary): task difficulty, "I found it difficult to summarize the article well." (instance-level); frustration, or "performing the summarization tasks was frustrating." (task-level); and assistance utility, or "the provided summaries were not useful to me when I was performing the summarization tasks" (task-level). All task-level measures were paired with follow up "why did you feel this way" open-ended questions.
3.7 Data and Analysis
72 participants wrote 720 summaries (manually or post-editing provided summaries) and 113 annotators evaluated those summaries, resulting in 6360 summary quality ratings; after removing outliers (see A.2.3). Averaging resulted in one final quality rating (on four axes) for each summary. We make this dataset of summaries and their ratings public to promote future research.[9]
To find out if any statistical differences exist between the means of the conditions, we used one-way ANOVA for each objective and subjective (rating scale) measure. Using post-hoc Tukey's HSD, we also performed pairwise comparison to determine which two conditions are significantly different.
We qualitatively coded the open-ended responses related to the subjective measures of frustration, task difficulty, and assistance utility, as well as responses on likes, dislikes, and desired
Figure 2: Average overall quality ratings for the summaries by type and dataset. For Reddit, the human reference was the worst (aside from the Random summary). For XSum, the AI-generated summary was the worst.
improvements. One researcher read the data to identify emergent codes, followed by a discussion period to merge and update the themes in the codebook. Two researchers then independently coded all the open-ended responses respectively, achieving a high average inter-rater reliability (Cohen's  $\kappa > = .85$ ) for each of the open-ended responses. See A.2.4 for details on thematic coding. We refer to participants as P1-P72.
We report on the impact of post-editing on summary quality, efficiency, and user experience.
4.1 Summary Quality
We discuss quality ratings for the summaries for each dataset (Fig. 2). For simplicity, we report only on overall quality ratings from the human evaluation (see Appendix A.3.1 for other axes).
For Reddit, post-editing improved the quality of the provided summaries but manual summaries were the best. Reddit summaries produced by participants in the Manual condition were rated highest overall quality; the provided summaries, AI-generated followed by the Human reference were the lowest quality (Fig. 2). Interestingly, our evaluation finds that the AI-generated summaries are significantly higher quality than the human references  $(p = .02)$ . This is different from Zhang et al. (2020), for which the same Pegasus model achieved comparable performance to human references (but not better).
Comparing summarization conditions, we find significant differences for final summary quality  $(p < .01, F = 9.3)$ : Manual summaries outperformed summaries produced by participants in both the AI post-edit  $(p = .03)$  and Human post-edit
<sup>8</sup>Only participants in the post-editing conditions responded about the utility of the provided summaries. <sup>9</sup>https://github.com/vivlai/post-editing-effectiveness-summmarization
$(p < .01)$  conditions. Finally, summaries resulting from AI post-edit and Human post-edit were significantly better than the provided summaries for those conditions, Human reference  $(p < .01)$  and AI-generated  $(p < .01)$ , meaning participants improved the quality of the summaries they were given.
For XSum, post-editing improved the quality of the provided summaries and was just as good as manual. XSum summaries produced by participants in the Manual condition were rated the highest and the provided summaries (Human reference followed by AI-generated) were the lowest (Fig. 2). However, for XSum, summary quality was not significantly impacted by AI assistance  $(p = .08, F = 2.5)$ , meaning there was no significant difference in quality between the Manual, AI post-edit, or Human post-edit summaries.
Similar to Reddit, AI post-edit and Human post-edit summaries were significantly better than the provided summaries for those conditions  $(p < .01)$ . But, opposite of Reddit, the XSum Human reference summaries were significantly better than the AI-generated summaries  $(p = .02)$ .
4.2 Efficiency
Post-editing human references slowed Reddit summarization but post-editing AI-generated summaries was faster for XSum. Summarization conditions significantly differed in efficiency (Fig. 3) for both Reddit  $(p = .04, F = 3.2)$  and XSum  $(p = .02, F = 3.8)$ . For Reddit, it took significantly longer to post-edit human references (Human post-edit) than manually (Manual,  $p = .03$ ) or given AI-generated summaries (AI post-edit,  $p = .02$ ). We discuss possible reasons for this (e.g., problematic Reddit references) in §4.3.
For XSum, post-editing AI-generated summaries (AI post-edit) was faster than Manual  $(p = .11)$  or given human references (Human post-edit,  $p = .08$ ). However, no pairwise comparisons were significant after correction.[10]