SlowGuess/NAACL_2022_OCR huggingface dataset:
In this task, workers were instructed to perform edit operations to an article version in anticipation of what the next version would look like. We recruited 5 workers: journalists who collectively had over a decade of experience working for outlets like The New York Times, Huffington Post, Vice, a local outlet in Maine, and freelancing.
We gave our workers the following instructions.
You will be adding, deleting and mov-
ing sentences around in a news article to anticipate what a future version looks like.
- Add a sentence either below or above the current sentence by pressing the Add  $\uparrow$  or Add  $\downarrow$  buttons. Adding a sentence means that you feel there is substantially new information, a novel viewpoint or quote, or necessary background information that needs to be present.
- Move a sentence by dragging it around on the canvas. Moving a sentence, (or what we're calling refactoring) means that the importance of a sentence should be either increased or decreased within the article. Please note: refactors are rare!
- Delete a sentence by hitting the Delete button. Deleting an Added sentence just reverses that action
Figure 8: Example of Sentence Matching Task. All lines represent sentences that have been matched. When the user hits "Submit", additional coloring is added to the unmatched sentences, which represent Addition (green, right) and Deletion (red, left) sentences.
we will not record this. Deleting a sentence that is present means you feel it needs to be (a) substantially rewritten (ergo: a new sentence should also be Added), or (b) the sentence no longer applies given new information that was added.
- Edit a sentence by hitting the Edit button. Editing a sentence means that the wording might change a little bit due to other changes happening around the sentence or events within the sentence being updated.
- Leaving a sentence unchanged means that you don't really expect the sentence to change at all in the next version of the article.
When you're ready to submit, please hit
Worker Id	Num Tasks Completed
ASQL7ZBXI7WF6	101
A2E8P5A3IKROKB	92
A17GX84A96WF6C	31
A1685VEOIJIUMR	13
A2USH7VYFMU1ME	5
A30BGCC8EC1NW	3
Table 15: Count of Tasks Completed per worker
the Submit button and please check to see what the actual edits were so you can improve for next task!
G.3 Annotator Analysis
We seek here to characterize the performance of different expert annotators. We see in Table 15 that there are three workers which do over 30 tasks each. We characterize the per-task accuracy by counting the number of edit-operations per document, and
Figure 9: Example of Editing Task. The gray boxes on the left serve as a reference for how the original article was written. The sandbox on the right is where annotators actually perform the task. The first sentence has been Edited, two sentences have been Added, the third has been Deleted and the fourth has been Refactored downwards.
Worker Id	Accuracy Across Tasks
A2E8P5A3IKROKB	76.6
A30BGCC8EC1NW	58.3
ASQL7ZBXI7WF6	46.0
A17GX84A96WF6C	38.7
A2USH7VYFMU1ME	35.0
A1685VEOIJIUMR	30.8
Table 16: Accuracy across document tasks (i.e. % bins correct across document-level subtasks: Added, Edited, Deleted,Refactored).
Figure 10: Worker Accuracy over time, by task
seeing if they got the same number as the true number of edits (each expressed as a binned count i.e. low: [0,1) operations, medium: [1,3) operations, high:  $[3,\infty)$  operations).
We show that there is a wide variety of performances, in Table 16, with some workers getting over  $75\%$  of the operations correct and others getting  $\approx 30\%$  correct.
Interestingly, we see that there is a learning process occurring. In Figure 10, we see that workers
get better over time as they do more tasks. This indicates that the training procedure of letting them see the edits that actually happened is successful at teaching them the style and patterns the edits will take.
G.4 Annotator Interview 1
This annotator was involved in the Editing task. They edited 50 stories.
1. What was your general thought process? Well, my first general though was: "how do I do this update?" Then I thought back to the instructions, and really tried to predict how the AP $^{44}$  would update.
I then had to decide what timespan I'd use—in general, I assumed a 24 hour update window, but sometimes it was different. If the story updates 2 hours after news breaks vs. 2 days, it will look very different
Sometimes, I would read the story, try to figure out what the story was about, ask what was missing, what I'd include in a story if I was reporting it fully. A lot of times what I felt were missing were more causal analysis, more quotes, more perspectives.
As I was going through, I almost always decided to edit the lede, and was almost always correct with that. Most leads, I thought, could be
44The AP, or The Associated Press, sets many standards for journalistic writing and reporting cycles.
more efficient, they could incorporate more details from further down in the story into the lede. Also, as stories unfolded, the actor responsible for the event becomes clear, that information will get added to the lede. For example, a building collapses in Manhattan  $->$  faulty beam causes the building collapse. This detail often only becomes apparent afterwards.
What I realized doing this was that there are different genres of breaking news article, and genre matters a lot for how it gets updated. These are the following categories:
(a) Stories where the future is contingent, and you're making predictions in realtime.
ex) A sailor went missing off the isle of Mann. This story is fundamentally about an unknown – will he be discovered or not? This is one of the harder ones to figure out how to update. How it plays out determines how it will be updated. If the search goes on for a long time, you'll have more details, you'll have quotes from his family, conditions on the water. If he's found, this stuff becomes irrelevant. You'll have information about how he gets found, then you'll have information about how many people get updated.
ex) A story was about "Trump is about to make a speech". "Trump expected to speak". I updated it as if event didn't happen yet. But the real update actually contained him speaking. Stories about when multiple futures can happen, without knowing the timescale of the update, are difficult to predict.
I determined whether an event was unfolding by looking for several clues. I looked for certain words: "expected", "scheduled", etc. Usually this signals an event-update. I looked for stories where there's a ton of uncertainty.
Another clue was that the only sources are official statements (ex. "Officials in Yemen say something happened"). The space of possible change increases. You're going to get conflicting reports, eye-witnesses contradicting official statements.
Some articles included direct appeals to readers—"don't use the A4 if you're traveling between London, etc." For crime articles: "if you have any information, please contact agency." This kind of direct appeal is not relevant in the next version.
(b) Past stories when the event is totally in the
For these stories, I looked for vagueness of the original article to determine what would be updated. If it's more specific, for example, with exact death toll numbers, information about specific actors and victims, the less it's going to be updated. For these stories, my tendency was to add at least 1-2 sentences of context towards the end of every story. If you're writing for Reuters, you might not need that.
In general, I wanted to see some background, people involved.
The quotes you're getting, are they press releases or are they directly from people? If they more official statements and press releases, then you'll see more updates in the form of specific victim quotes.
One general note: most breaking stories were about bad things. Disasters, crashes, missing people, etc. For a bombing, there's a pretty predictable pattern of expansion. Death toll will get added, more eyewitness accounts. It has an expansionary trajectory.
2. How did you determine if a sentence needed to be added? I decided to add anywhere I saw vagueness. I added a lot towards the beginning, right after the nut graf is where I added the most sentences. If I saw a sentence taken from a press release, I added after that, assuming that the journalist would get a more fleshed-out quote from someone.
Often I added [sentences] at the end to add context. I never added something before the lead.
Maybe a story has two ideas, then I'd add sentences to the second half to flesh out a second idea.
Sometimes I thought about different categories of information—quotes, analysis, etc.—and it was obvious if some of that was missing.
3. How did you determine if a sentence needed to be deleted?
I very rarely thought things needed to be deleted
One of the challenges of the experiment was that it was hard to indicate how to combine sentences. I got around this by hitting "edit" for sentences that needed to be combined. Then I'd delete ones below, assuming that the edited sentence would include a clause from the sentence
below it.
Structural sentences and cues got deleted often. Sentences like "More follows", etc. Nothing integral to the substance of the story.
I noticed that almost always, [informational content of sentences that had been deleted] had been reincorporated.
4. How did you determine if a sentence needed to be moved up/down?
I did this by feel, what seemed important. One example: A building collapse in Morocco. A sentence way towards the end had a report about weak foundations, that needed to be brought up. This indicated that the journalist became more confident about something
The inverted pyramid so widely used, in a breaking news it's fairly easy to weight the importance of different elements. Thus, I rarely felt the need to move items upwards.
Sometimes I saw examples of when what was initially a small quote from official was expanded in a later version. Then, it was brought up because the quote became more important. But usually, my instinct would not be to move quotes from officials up.
5. Did it help to see what actually happened after you finished the task?
Usually there was 1-2 things that we had done that were basically the same.
A couple of times, [I] was satisfied to see that the updated story made the same decision to switch sentences around.
6. Any general closing thoughts?
Most interesting thing was to see how formally constrained journalists and editors are, and how much these forms and genres shape your thought and your work.
There are assumptions get baked into the genres about who's credible, what kinds of things carry weight, sorts of outcomes deserve special attention, a whole epistemic framework.
Even though there's a lot of variation, there's a fair amount of consistency.
I was disappointed that, especially for rapidly expanding stories, the edits were mainly causes and main events. I saw very few structural, causal analyses added to breaking stories. There was some analysis that got added to one story about bombings in the Middle East, but still, not a whole lot about how the specific conflict originated.
G.5 Annotator Interview 2
This annotator was involved in both the editing task and the version-prediction task. They annotated over 100 examples of the first task, and 50 of the second.
1. What was your general thought process while doing the edits task?
First, before starting, I made the assumption that every story would need edits, because I think everything could always use more work. In reality, if the article wasn't updated the way it was, I was representing one option. My process was:
(a) Read the whole story, don't make any changes at first.
(b) Then, I would think about what I thought was the most important sentence.
(c) I would often pull that high up into the lede, and then I'd add a sentence before or after.
The factors that determined the most important part of the article were:
(a) Some indication of harm done or the most recent development. I always took "harm done" as the most important part of a story.
For example: Death count—20 people were killed in some explosion vs. a bomb went off here. Moved the "20 people killed" higher because that was a harm ex. Officials are investigating whether so-and-so doctored documents.
(b) Then, I would add/delete and edit based on these. So, I would create a new sentence and edit the next sentence to give more context.
2. How did you determine if a sentence needed to be added?
So, after identifying the lede that I described previously, I went through and looked through what parts I felt needed more context or a quote. Getting quotes was very important. Often I identified events that I thought warranted a reaction, acknowledgment, information from a source. If these weren't there, I added a sentence. I didn't keep a checklist of these elements (i.e. "quote", "context", etc.) It was more a gut feeling about what it needed. If I were going back and doing it again, I would write out a checklist.
Often, especially when the news was unpredictable, I would often add a sentence in the beginning saying "I don't know what this sentence is going to be, but it's going to be something". In other words, I was adding context to what the unknown would be. I was able to do this pretty
successfully, to predict what context would happen around the unpredictable event.
Where I tried to add more information to flesh out certain unknowns:
(a) If an official said something that needed to be followed up on, I would delete these and add new sentences
(b) I had hoped that the reporter would get that information themselves through eyewitnesses, court documents, etc.
(c) Sometimes an official would give filler quotes like: "we'll have more information later this afternoon". These would be replaced with the actual update.
(d) Context: I would add historical context. How often has something been occurring in this area, etc. Many of real updates did have these contextual sentences.
3. How did you decide whether a sentence needed to be edited?
After I decided what would be moved up, I looked at details (dates, people, etc.). Sentences with details were the ones that were most likely to be edited.
4. How did you determine if a sentence needed to be deleted?
I deleted sentences that were redundant. I identified filler quotes (e.g. officials saying they'll get more information soon.). These would be deleted when, presumably, more information did come in. Sometimes a quote was redundant to a sentence that was already there. One of the challenges was deciding when to delete or edit a sentence.
5. How did you determine if a sentence needed to be moved up/down?
I almost always moved sentences upwards, to the top. As we discussed previously, the top then needs to have room for an update. Again, as we discussed previously, I used harm and recent developments as a metric to decide where to move. The context was also moved around based on when the events took place.
I also tried to focus on recent developments. For example: "Officials are investigating whether so-and-so doctored documents". I would move that to the top. I pulled up the active part of the article to express what was actually happening.
6. What things did you get wrong?
I was really bad at predicting stories that were
"delete all", "replace all". I struggled more with stories that were about political leaders speaking at an event or speaking at a conference, because these ended up going different ways. Sometimes they made a big announcement that would make headlines, but it was hard to known beforehand what that announcement would be.
For crime, or spot news, it was clearer that an event was unfolding and would have specific updates. By "spot news", I mean stories about crimes, fires, rescues, weather events/disasters, etc. - something unexpected as opposed to articles about events that have been planned, like the example of a political figure speaking at a conference. It was these unexpected events that actually follow more predictable paths when they unfold.
I saw a lot of discrepancies between sentences I chose to edit, and then the actual result was that they got deleted. For example, the death toll was in a sentence, and I'd edit that sentence, but they chose to add a sentence with the same information. The sentence matching algorithm didn't do a good job with informational units that were not at the sentence level.
7. How did you assess uncertainty in an article?
Often it was topic-based. I can't think of key indicators that I used to assess uncertainty.
8. Was really helpful after I made the edits to see what actually happened?
I tried to balanced this with what my natural instincts were. I did get better over time. I did feel more confident over time. The changes would be more in my decisions to edit vs. add/delete. In my head, I had the same end result in mind, but they edited it and I added a new sentence. I never felt I was widely off
9. Did you see a lot of analytical pieces? Or mainly breaking news?
I saw a mix of stuff that was analytical vs. factual. There were certainly more breaking news events, events that were going to happen and change on the same day. However, I did see some day 2 stories. Sometimes, they were updates that were part of an ongoing investigation. The breaking stories and spot news, crime, were the easiest to do. Those ones seem much more formulaic.
10. What was your general thought process while doing the versioning task? How did you identify versions that updated?
This one was trickier because I would assume that everything would be updated, everything would be improved. The mindset change that I made was "Will this story itself be edited, or will they write a followup with more information?"
Once I made this separation this became easier 11. What patterns did you observe in this task?
The timing of when I thought an update would occur ended up mattering a lot. I paid closer attention to stories that would have updates within the same day or a short period of time. The longer the time-periods between updates, the more likely a new piece would be published instead of an update.
Again, crime and spot news it was clear—the person was on the scene at this minute, they'd get more information.
The other giveaways were "so and so is expected to deliver remarks later this afternoon." It wasn't quite a preview of the event but it would clearly be updated
The other thing that made me choose to mark a story as "would be updated" is if there was a key perspective missing or if there was no quotes at all. By "key perspective", I mean, a key quote from a participant that is usually present in this type of story. For a crime, for example, this included: Law enforcement perspective, witness, family. In general, it means that both sides are represented.
12. Were there examples that you thought would update that didn't?
There were some with stock figures, quarterly earnings, that I initially thought would be updated, but I had seen the examples that were filled out, but I'd be more accepting that this was a final report and that it's not going to have any quotes. I became better at identifying which types of pieces wouldn't have context or quotes.
13. Anything I may have missed?
I tried to flag a couple of articles that transferred over inaccurately. Sometimes there were cases of where one article published to the same URL was something completely different. Sometimes there were calls for subscribing to newsletters or related story links. I deleted ones that were repetitive. This might have influenced results on some articles. These structural updates were annoying.
14. Could you see solving this kind of prediction task as being useful in a newsroom?
I could see it being used as a people management tool. Newsrooms are desperate for any kind of methodology to guide the decisions they make. Deciding who should attack a new story, and who should stay put working on their old piece would help a lot!