SlowGuess/NAACL_2022_OCR huggingface dataset:
During the study, participants completed three phases: (1) instructions, tutorial, and practice; (2) summarization task; and (3) post-task survey. During the instructions, tutorial, and practice phase, participants reviewed task instructions, the criteria for writing a good summary (Table 2), and examples of good and bad summaries with explanations (Fig. 6 and Fig. 7). Participants then applied what they learned and practiced to write a good summary. To refrain from any confusion, the interface in the practice phase is exactly the same as the actual summarization task phase. Participants have access to the criteria as guidance at any time during the task (Fig. 8a). Participants then performed the summarization task as described in §3.4. After completing each summary, participants rated their agreement for the following statements (Fig. 8b): (1) I found it difficult to understand the content of the document; (2) I found it difficult to summarize the document well. Finally, participants responded to an exit survey before ending the study, where they answered questions regarding task ease, frustration, their familiarity with the domain (i.e., Reddit, British news and culture), if the provided assistance was useful, and what they liked and disliked about the summarization task and interface.
A.2 Human Evaluation of the Collected Summaries
We used human evaluation to assess the quality of the summaries we collected during our study. Each annotation task involved reading a news article or Reddit post and evaluating six different summaries for that document.
11Reddit-TIFU, Xsum
Reddit post #1
this was last year in august, i had just graduated high school and about to start college. i waited until the last very second to get a job because i was lazy and wanted to enjoy my last summer work free. so, i literally applied to every single place i could. i ended up getting an interview at chipotle, and they hired me. they told me i got  $50\%$  of all food when not working, so i came back another day that week to get my boyfriend and i some food.
this is when the fuckup happened. i haven't even had my first day yet either, keep in mind. i was driving to go park next to the building, about to pull into a parking space when wham. i pressed the gas instead of the break and rammed straight into the building. i looked in horror at the people sitting right beside the building in the outdoor seating. i slowly backed my car up off the building and curb.
then, not only was this the place i just got hired, but all the cops in my town were eating lunch there at that time. they all came out and surrounded me, and took my information. the building ended up having a big crack in it, and my car was just scratched up, surprisingly. they still kept me around and i still worked for them, thank goodness. but that was a hell of an embarrassing first day when i worked.
i apologize for grammar i'm on my mobile phone
Good summary
I drove into my work's building with my car right after I got hired. I was completely embarrassed but they kept me around and I still work for them..
The summary is succinct and it covers main points such as (1) the car ran into the building and (2) the author felt embarrassed but was not fired from the job. The summary matches the style of the original reddit post.
Bad summary
Applied to jobs late last summer, ran my car into the building, and all the firemen in town were having their lunch. I did not get fired though, but I was really embarrassed as it was my first day.
The summary is succinct and matches the style of the original reddit post, but it covers more points than are necessary for the story (e.g., the fact that the author applied to jobs late last summer) and it is factually incorrect as the original text mentions "policemen" instead of "firemen".
Figure 6: Reddit tutorial first example with good and bad explanations.
A.2.1 Example Summaries
We evaluated six types of summaries:
1. Manual. It was written without assistance.
2. AI-generated. The provided summary was generated by the Pegasus model.
News article #1
Malka Leifer, the former principal of a Jewish girls' school in Melbourne, fled to Israel in 2008 after accusations were raised against her. Ms Leifer says panic attacks stop her coming to court. Extradition hearings have been delayed for two years so far. Based on a psychiatric report, a judge has decided Ms Leifer will not face court until she completed treatment. Ms Leifer allegedly raped and indecently assaulted girls at the ultra-Orthodox Adass Israel School in Melbourne, Australia. The court in Jerusalem has also lifted her house arrest, meaning she will be able to move freely for the first time since Israeli police arrested her in 2014. Prosecutors are expected to appeal this decision. Ms Leifer will begin an initial six months of treatment before a committee assesses whether she is fit to face future extradition proceedings. "We are committed to seeing this woman extradited to Australia to face these very serious child sex abuse charges," Australia's ambassador to Israel, Dave Sharma, told the Australian Broadcasting Corp. "We are determined to be patient and persevere to this end with the view to seeing her extradited."
The former principal of a Jewish girls' school in Melbourne, facing child sex charges in Australia is mentally unfit to face extradition from Israel, a court says.
The summary is succinct and points out the following main points of the article: (1) a former principal is charged for alleged rape and assaults and (2) she is mentally unfit to face court.
(1) Woman allegedly raped and indecently assaulted girls in a Jewish girls' school, (2) due to her panic attacks, unable to go to court, (3) she will complete treatment before attending extradition hearings
While the summary covers main points, it is not written in a summarized method that matches the style of the original post, but instead just lists a few points. Some phrases are also not grammatically correct.
Figure 7: XSum tutorial first example with good and bad explanations.
3. Human reference. The provided summary is from the original dataset.
4. AI + post-edit. It was written by a human who was shown a AI-generated summary.
5. Human + post-edit. It was written by a human who was shown a reference summary.
6. Random. It was generated by randomly selecting two sentences from another dataset. This summary helped to weed out annotators who did not pay attention. For instance, if
Summarization task
Now you will begin performing the summarization task.
hide criteria
A good summary should satisfy the following criteria:
Essence: the summary is a good representation of the article.
- Clarity: the summary is reader-friendly. It expresses ideas clearly.
- Accuracy: the summary contains the same information as the longer article.
- Purpose: the summary serves the same purpose as the article.
- Concise: the summary is short and to-the-point.
- Style: the summary is written in the same style as the original article.
Many of Paris's tourist attractions, including the Louvre and the Musee d'Orsay, closed on Saturday amid heightened security. The institutions reopened at 13:00 local time following a minute's silence to honour those who were killed. However the Eiffel Tower will remain closed until further notice. A message on the tower's website added "the Eiffel Tower will not sparkle" on Monday night, during the period of national mourning, at the request of Paris Mayor Anne Hidalgo. A total of 129 people died in the attacks by Islamist militants. People were targeted in bars and restaurants, the Batacan concert hall and the Stade de France. In explaining the decision to reopen the venues, French Culture Minister Fleur Pellerin said although France had suffered a tragic event, "culture is more than ever this symbolic place of self-discovery". Ms Pellerin added the government would help to boost security measures at public cultural institutions in Ile-de-France, the region immediately surrounding Paris. Its landmarks include the Louvre, which houses Leonardo da Vinci's Mona Lisa painting, as well as the Eiffel Tower and the Notre Dame Cathedral. Eighty-nine people died at the Batacan venue, where US rock band Eagles of Death Metal had been performing. Many concerts scheduled to take place in Paris over the weekend, including those by U2, Motorhead, Foo Fighters and Deftones, were also cancelled. Cinemas were also ordered to close following Friday's attacks, but most reopened on Sunday.
Figure 8: Task interface and questions on the summarization task.
Click "save summary" when you have finished summarizing the article. Note: you will not be able to make changes to your summary after clicking to save.
(a) Task interface.
For the following statements, please rate your agreement on a scale from 1 (strongly disagree) to 7 (strongly agree).
I found it difficult to understand the content of the article.
I found it difficult to summarize the article well.
(b) Questions regarding the summarization task.
the participant sees a Reddit post, the Random summary is a summary generated from a news article from the XSum dataset.
Table 3 demonstrate the six example summaries for one document.
A.2.2 Summary Evaluation Procedure
The annotator went through two phases during the human evaluation task: (1) tutorial (Fig. 9a) and attention check (Fig. 9b); and (2) evaluating summaries (Fig. 9c and Fig. 9d). A tutorial with two examples was provided at the start of the task to teach the participant how to evaluate a summary. To further solidify their understanding, we also included two examples with ratings and explanations. Explanations were curated by the researchers and iterated a few times upon getting feedback. We included an attention check open-ended question to ensure that the participant read through and understood the tutorial. The task would only begin when they give the correct answer. Before the actual task, to ensure that annotators read the document, we asked them to write a short title after they read it. During the task, anchors for the original post and definition of axis are easily accessible, allowing annotators to refer to them whenever they wanted to. A HIT  $^{12}$  included reading a document and evaluating six different types of summaries of it. For each HIT an annotator does, they are paid $1.50. After removing annotators that failed the attention check, the average time taken to complete a HIT is 13.5 minutes (SD=5.7). Although the hourly rate may seem low, we learned from the annotators that turkers tend to open up to 25 tabs of HIT while working. As such, the time taken also included idle time, meaning that the actual average time taken could be less than 13.5 minutes. Additionally, annotators did 9.4 HITs on average.
A.2.3 Eliminating the bad apples.
To ensure high-quality evaluations, we performed quality control to eliminate annotators with validation procedures: attention check with a "random" summary, batched deployment, and removing outliers. We detail each of these procedures in the following.
Attention check: random summary. We incorporated an attention check to weed out annotators who did not pay attention and simply clicked through the rating scales. For this, we inserted a random
$^{12}$ Amazon Mechanical Turk Concepts
Table 3: Reddit and XSum example summaries.
summary to rate, which was generated by randomly selecting two sentences from the opposite dataset. Further, we randomize the order of summaries, ensuring that the random summary would not always appear in the same position. Since the random summary is created with sentences from a document from an entirely different dataset, it is fair to assume that the content will not cover nor be accurate as a summary. Therefore, we eliminated annotators (and discarded their responses) who did not give a
rating of 1 to both coverage and accuracy.
Batched deployments. To ensure only high-performing annotators participated in our evaluations and maintain the integrity of our results, we followed a batched deployment procedure opposed to deploying all evaluations at once. We deployed a total of 1200 evaluations (i.e., 120 HITs x 2 datasets x 5 samples) on Amazon Mechanical Turk, splitting the assignments into 10 batches. At the end of a batch deployment, annotators who failed the
attention check (random summary) had their qualification revoked and not allowed to accept future HITs for our evaluation task. A total of 113 annotators completed the 1200 evaluations.
Removing outliers. Finally, while we had 5 annotators evaluating a summary, not all five ratings were taken into consideration. We removed any outlier ratings following a standard approach: 1.5 more or less than the inter quartile range (IQR). Table 4 show the variance for each condition per dataset after removing outliers.
A.2.4 Thematic Coding
We performed thematic coding to analyze the open-ended responses in our study. A researcher in the team first manually coded the data and iteratively developed themes from answers of each question. For example, "lack of context knowledge" is one of the themes developed from the answers to the question on why participants rated the task as difficult. The researcher then discussed the themes with the team, merged and updated the themes, then recoded the data again. To validate the coding results, a second researcher also coded the data based on the themes developed by the first researcher.
A.2.5 Summary Quality Criteria: Coherence, Accuracy, Coverage, and Overall
Annotators were tasked to evaluate a summary according to four axes on a scale of 1 to 7 (Stiennon et al., 2020). The definition of each axis is listed as below:
1. Coherence. A summary is coherent if, when read by itself, it's easy to understand and free of English errors. A summary is not coherent if it's difficult to understand what the summary is trying to say. Generally, it's more important that the summary is understandable than it being free of grammar errors.
2. Accuracy. A summary is accurate if it doesn't say things that aren't in the article, it doesn't mix up people, and generally is not misleading. If the summary says anything at all that is not mentioned in the article or contradicts something in the article, it should be given a maximum score of 5.
3. Coverage. A summary has good coverage if it mentions the main information from the article that's important to understand the situation described in the article. A summary has
poor coverage if someone reading only the summary would be missing several important pieces of information about the situation in the article. A summary with good coverage should also match the purpose of the original article (e.g. to ask for advice).
4. Overall. This can encompass all of the above axes of quality, as well as others you feel are important. If it's hard to find ways to make the summary better, give the summary a high score. If there are lots of different ways the summary can be made better, give the summary a low score.
Condition	Rating (σ)
Reddit	Manual	0.81
AI post-edit	0.95
Human post-edit	1.05
AI-generated	0.63
Human reference	0.66
XSum	Manual	0.85
AI post-edit	0.84
Human post-edit	0.89
AI-generated	1.02
Human reference	0.63
Table 4: Variance between ratings for each condition per dataset for the Overall rating.
A.2.6 Research Experiment Ethics
Participants from Upwork and annotators from Amazon Mechanical Turk were aware of how the data collected would be used. They were assured that no personally identifiable information was collected from them. For participants on Upwork, the written summaries and exit survey responses were collected from them. Similarly, for annotators on Amazon Mechanical Turk, only responses and ratings were collected. Before working on the task, participants and annotators were made to read a description of the task and working on the task meant that they were aware of what was collected.
A.3.1 Ratings on Coherence, Accuracy, and Coverage
In the main paper, we reported only overall ratings. Fig. 10 shows the plots for coherence, accuracy, and coverage ratings. Both Reddit and XSum summaries produced by participants in the Manual condition were rated highest accuracy and coverage quality.
In Reddit, AI assistance significantly impacted coherence  $(p < .01, F = 15.4)$  and coverage  $(p < .01, F = 7.9)$  but not accuracy  $(p = .48, F = .7)$ : Manual summaries outperformed summaries produced by participants in both the AI post-edit  $(p = .01$  for coherence,  $p = .01$  for accuracy) and Human post-edit  $(p < .01$  for coherence,  $p = .01$  for accuracy) conditions.
In XSum, AI assistance significantly impacted accuracy  $(p = .03, F = 3.5)$  and coverage  $(p = .02, F = 3.8)$  but not coherence  $(p = .84, F = .17)$ : Manual summaries outperformed summaries produced by participants in Human post-edit  $(p = .03$  for accuracy) and AI post-edit  $(p = .01$  for coverage).
A.3.2 Correlation between Edit Distance and Summary Quality
We compared edit distance and overall summary rating and found weak to no correlation  $(p = 0.4, \rho = -0.05$  for Reddit and  $p = .02, \rho = -0.2$  for XSum) between the two factors. For XSum, while this suggests that the bigger the edit distance, the poorer the overall summary rating, the correlation score is very small. On the other hand, there is no correlation between edit distance and overall summary rating in Reddit. Fig. 11 shows the plots for both datasets.
A.3.3 User Experience
In the main paper, we reported insights from qualitative analysis for task difficulty, frustration, and assistance utility (§4.3). We also conducted thematic coding on why participants enjoyed working on the summarization task.
The summarization task was enjoyable and educational. Many participants enjoyed working on the task, describing the experience as educational (e.g., P14 (XSum, Manual), "it made me think about the information I had read and how to best condense it"). Others enjoyed reading the original text (e.g., P53 (Reddit, AI post-edit), "these stories are quite interesting, the summaries make me make sure I understood what I just read"), and felt a sense of achievement when finished (e.g., P36 (XSum, Human post-edit), "it was satisfying to reduce a block of text down to a succinct sentence or two").
Example summary #1   
I drove into my work's building with my car right after I got hired. I was completely embarrassed but they kept me around and I still work for them..   
Ratings for example summary #1   
The summary should be rated as follows:
(a) Tutorial interface.
Answer the following question to start the task. If you are unsure of the answer, review the rubrics above. The task section will appear when you've completed the test.
Which axis measures whether the summary contains all important information from the post? Enter your answer and click "Start task".
Figure 9: Annotation interfaces.
(b) Attention check interface.
Reddit post
so today I went to the zoomarine in algarve, portugal and my family decided to go watch the interactive pirate show (basically pirates play around with the crowd), so far so good... we sit down and to my right, this long hair, tattooed, bearded, baggy clothes, pirate looking guy sits down by himself, a bit after my mom without moving her head asks me if I have seen the pirate "hiding" which I quietly answer to with "yeah", here I am thinking "nicel he is gonna gonna let me take a photo with him", so I get my camera and ask him if I could take a picture with him. well, just when I am starting to pose for the picture, he stands up and starts cursing giberish in spanish (in other words: not in portuguese) and got everyone in the audience and my entire family looking at me and him like it is part of the show... at this point my mom looks even more confused than me thinking what I could have done. needless to say, later in the show the pirate my mom was talking about was 2 rows down from us. & The former Tottenham player, 29, is the Spanish club's first signing since Neymar left to join Paris St-Germain in a world record transfer. Meanwhile, Barca's Uruguay forward Luis Suarez would be out for "four to five weeks" after he was injured in the Super Cup defeat by Real Madrid. He will also miss World Cup qualifiers against Argentina and Paraguay. Paulinho joined Tottenham for £17m from Corinthians in 2013, before moving to China in 2015. He helped Evergrente win last season's Chinese Super League and leaves the team top of the table in the current campaign. Paulinho said: "You have to face challenges with courage. I will try to do my job and I am prepared. It's a very satisfying moment. The dream I have been looking for has come true. I will give everything." Barcelona are also keen on Liverpool midfielder Philippe Coutinho and Borussia Dortmund forward Ousmanne Dembele.
Jump to summary #1, #2, #3, #4, #5, #6
Have you finished reading the post?
Give a short title to the post to describe what it is about
pirate show at Portugal
You may now rate the summaries below.
For each summary, rate the coherence, accuracy, coverage, and overall quality (on a scale from 1 to 7), following the provided rubric (above).
Note: consult the rubric if you are unsure of a rating.
(c) Instructions are given before the actual task.
Summary #1
went to pirate saw, a pirate looking guy, tried to take selfie with him, it was an evil hipster. & Barcelona have signed Brazil midfielder Paulinho from Chinese club Guangzhou Evergrande for 40m euro (€36.4m).
view Reddit post
How coherent is the summary on its own? view rubric for Coherence
$$
\begin{array}{c c c c c c c} \text {I t i s i m p o s i b l e t o u n d a r y .} & \bigcirc & \bigcirc & \bigcirc & \bigcirc & \bigcirc & \bigcirc \\ & 1 & 2 & 3 & 4 & 5 & 6 & 7 \end{array} \quad \text {T h e s y m a r i z u n g i s p e r f o c t l y c r e a l}.
$$
Does the factual information in the summary accurately match the post? view rubric for Accuracy
The summary is completely wrong, made
$$
\begin{array}{c c c c c c c c} \circ & \circ & \circ & \circ & \circ & \circ & \circ \\ \text {T h e s y m a t h a r y h a s n o i n c o r r e s t s a t e m e n t s} \\ & \text {o r m i s l e a d i n g i m p l i c a t i o n s .} \end{array}
$$
How well does the summary cover the important information in the post? View rubric for Coverage
The summary contains no information relevant to the post. 1 2 3 4 5 6 7
How good is the summary overall at representing the post? view rubric for Overall quality
$$
\begin{array}{c c c c c c c c} \text {I t i s t e r b i c l e .} & \circ & \circ & \circ & \circ & \circ & \circ & \circ \\ & 1 & 2 & 3 & 4 & 5 & 6 & 7 \end{array} \quad \text {I t i s a n e x c e l l e n t r e p r e s e n t i o n o f t h e}
$$
(d) Evaluation interface.
(a) Coherence rating.
(b) Accuracy rating.
(c) Coverage rating.
Figure 10: Human evaluation rating on coherence, accuracy, and coverage.
(a) Reddit
(b) XSum
Figure 11: Edit distance vs. Overall rating