SlowGuess/NAACL_2021_OCR huggingface dataset:
DAE (Goyal and Durrett, 2020): DAE is an entailment classification metric that operates on dependencies. The authors kindly provided outputs on the FRANK benchmark. We note that the model was trained with a max length of 128 after concatenating both article and summary. The CNN/DM articles can be significantly longer, thus the results reported for this metric involve truncating parts of the article.
FactCC (Kryscinski et al., 2020): FactCC is an entailment classification metric. We use the sentences of the model generated summary as input claims to the entailment classifier FactCC. For each sentence we obtain a binary factuality label. We take the average of these labels as the factuality score for the summary.
<sup>5</sup>We use the model and implementation from (Stanovsky et al., 2018) for OpenIE extraction.
$^{6}$ https://github.com/huggingface/neuralcoref
A.3 Summarization System Analysis Details
See Table 1 for more details.
A.4 Hotelling Williams Test
The correlation numbers in Table 2 should be read in combination with the pairwise Hotelling-Williams test Graham (2015) results in Table 4. The highlighted numbers indicate pairs of models for which the difference in correlation is statistically significant. We use partial correlations to run the test and compute metric-metric correlations.
A.5 Mutual Exclusiveness of typology:
To understand if our annotations are mutually exclusive, we study cases where two annotators agree on the error category (majority class) and one disagrees (minority class). In Figure 6, we report the confusion between majority and minority classes. For each category as majority, we report the distribution of other categories as minority.
We observe that all categories with the exception of OutE are frequently confused with NE which stands for no factual error. This primarily due to the noise in the annotations collected by crowd workers. However, for category CorefE (coreference errors) the confusion is significantly higher with  $69.7\%$ . We have noticed the same trend in practice tutorials: crowd annotators easily overlook situations where the correct pronoun is used (in terms of number and gender) but no antecedent appears in the summary. Intuitively after reading the article, unless paying particular attention, it is easy to subconsciously associate referring expressions with entities in the article without noticing their absence in the summary. The error persists despite stating the scenario explicitly in the instructions. This indicates an issue with annotators rather than annotation scheme.
The other trend that we observe is that categories PredE (wrong relation) and CircE (wrong modifier) are often confused with OutE (outside information). In our definition of OutE, outside information corresponds to the presence of entities not mentioned in the article or relations that cannot be verified based on the article. The confusion with PredE indicates that annotators can have different judgements on whether a relation is verifiable based on the article. Similarly, but to a lesser degree, wrong circumstantial information might be considered unverifiable given the article.
Finally, there were relatively few discourse con
Figure 6: Confusion matrix of different types of errors. Entry at row  $i$ , column  $j$  corresponds to the frequency of annotations that have  $\mathrm{Fi}$  as the majority class and for which disagreeing annotator selected  $\mathrm{Fj}$ .
text errors LinkE, so the analysis is less statistically significant. Discourse context errors correspond to using a wrong connectors between different facts, for example different logical links. These were confused with PredE and EntE (wrong relation). The distinction between the two errors lies in the confusion between what an entity and a fact are, since PredE occurs at the frame level while LinkE at the discourse level. Note, that there was no confusion in the other direction (PredE being confused with LinkE).
A.6 Annotation Setup Details
Below are more details on the annotation set up.
Clear Instructions We explain the annotation scheme without assuming linguistic knowledge and give several examples for each category. We also provide a practical ste-by-step to determine the category of the errors.
Training Every first-time user has to go through a tutorial which exercises the comprehension of the annotation scheme. The tutorial presents an article and several hand-crafted summaries of the article that need to be annotated. It is designed to be very similar to the actual annotation task and to contain at least one occurrence of each category of error. Feedback is provided when a user selects the wrong category of error. This tutorial is not used to evaluate users, only to help them understand the different categories in a practical setting.
Incorrect	F1	F2	F3	F4	F5	F6	F7	F8
Seq2Seq	74.8%	11%	46%	13%	15%	5%	14%	24%	0%
PGN	26.5%	4%	46%	0%	39%	0%	4%	21%	0%
Bottom Up	62.6%	6%	56%	6%	17%	9%	6%	21%	4%
BERTSum	27.2%	10%	37%	10%	23%	3%	13%	10%	0%
BART	23.8%	4%	25%	8%	33%	4%	17%	17%	4%
PGN	96.9%	16%	28%	11%	1%	1%	34%	13%	0%
TConvS2s	89.8%	10%	24%	18%	1%	0%	45%	1%	0%
TranS2S	96.9%	10%	32%	15%	0%	0%	44%	1%	0%
BERTS2S	83.7%	10%	25%	23%	0%	0%	38%	3%	0%
All models	60.0%	10%	36%	13%	10%	3%	27%	12%	1%
Table 3: Proportion of summaries that include factual errors, with breakdown of the categories of errors according to our human study. F8 corresponds to errors that are not captured by our typology. Full specification of categories of errors in Table 1.
B	MET	R-1	R-L	BS-P	OpIE	FEQA	QAGS	DAE	FCC
BLEU	-	0.83	0.77	0.85	0.04	0.26	0.03	-0.01	0.05	0.06
METEOR	0.83	-	0.87	0.85	0.04	0.28	0.02	-0.02	0.08	0.07
Rouge-1	0.77	0.87	-	0.89	0.04	0.21	0.01	-0.03	0.09	0.07
Rouge-L	0.85	0.85	0.89	-	0.03	0.21	0.01	-0.04	0.08	0.07
BERTS P	0.04	0.04	0.04	0.03	-	0.00	0.00	0.02	-0.02	-0.04
OpenIE	0.26	0.28	0.21	0.21	0.00	-	-0.01	0.09	0.10	0.15
FEQA	0.03	0.02	0.01	0.01	0.00	-0.01	-	-0.01	0.03	0.04
QAGS	-0.01	-0.02	-0.03	-0.04	0.02	0.09	-0.01	-	0.08	0.09
DAE	0.05	0.08	0.09	0.08	-0.02	0.10	0.03	0.08	-	0.10
FactCC	0.06	0.07	0.07	0.07	-0.04	0.15	0.04	0.09	0.10	-
Table 4: Pearson correlation between metrics. If value is in green, the metrics are not the same significant to the 0.05 threshold with the Hotelling Williams test.
Qualification test To participate in the annotation, users have to obtain a minimum score of  $85\%$  on a qualification test. The test comprehends an article and several summaries to be annotated. It contains at least one instance of each category of error. We use this test to verify that users can effectively recognize error categories. This ensures that users are able to perform the task correctly, but does not enforce that high standards of work quality are maintained throughout the annotation task.
Continuous evaluation We continuously evaluate a user by verifying that they read the text. For every article that is annotated, we ask to identify one of three entities that was not present in the article. We also monitor the annotations on artificially altered sentences that are randomly inserted at the end of summaries. Wrong sentences contain one of the following errors: negation of declarative sentences (PredE), pronoun swap (CorefE), sample sentence from another article (OutE), word scrambling (GramE). We immediately block users that fail the entity test or perform poorly on these sentences (less than  $50\%$  of correct answers on altered sentences) to ensure high quality annotations.
Bonusss All workers are paid  $50\%$  more than the average American minimum wage but we offer bonuses for scores of  $60\%$  or above on the continuous evaluation, and for completion a sequences of 10 annotations. We observe that bonuses increase the percentage of users with high continuous evaluation scores  $(<  10\%)$  blocked users with bonuses versus  $30\%$  without bonuses).
A.7 Correlation with Confounding Variables
Partial correlation measures the degree of association between two random variables, with the effect of a set of controlling random variables removed. Although we are unaware of the exact confounding variable, we use the categorical variable  $C$  of which system and dataset the summary was generated from.
Let  $M_{k}$  represent the output of metric  $k$  on the summaries. To compute partial correlation between  $M_{k}$  and human judgements  $H$  which we treat as random variables, we solve the two regression problems  $M_{k}|C = c\sim w_{M_{k}}c$  and  $H|C = c\sim w_{H}c$  and get the residuals:
$$
\Delta M _ {k} = M _ {k} - \hat {w} _ {M _ {k}} C
$$
$$
\Delta H = M _ {k} - \hat {w} _ {H} C
$$
And then calculate the correlation between these residuals  $\rho (\Delta M_k,\Delta H)$  instead of the original random variables. Since partial correlations are proper correlations between random variables, we can apply statistical significance tests without any modification.
A.8 Annotation Interface
We include screenshots of the annotation interface which we will make available.
Identifying Wrong Facts in Summaries of News Articles
Toggle Instructions
In this task you will read a news article and several summaries of the article. Afterward, you will be asked to report on whether the facts in these summaries are correct, and what kind of mistakes, if any, are present. You will have to identify specific sentences in the summary where information is inaccurate.
In the following passage we highlight entities. An entity is generally a thing, a person, an organization, a place, a number, etc. To help us identify wrong facts in the summaries, you will have to find out if the relationships between entities are sound. In other words, whether all the facts are correct.
Passage: The first vaccine for Ebola was approved by the FDA in 2019 in the US, five years after the initial outbreak in 2014. To produce the vaccine, scientists had to sequence the DNA of Ebola, then identify possible vaccines, and finally show successful clinical trials. Scientists say a vaccine for COVID-19 is unlikely to be ready this year, although clinical trials have already started. Over 200,000 people have died of COVID-19 in the US.
What is an error
Below, we list the potential errors present in the passage above as examples of types of errors. These errors are placed into categories depending on the type of error that is present. Later, you will be asked to identify the types of errors in other passages.
Information not in article:
The summary contains either an entity that was not in the article or a relation that cannot be verified using the article.
Example 1: China has already started clinical trials of the COVID-19 vaccine.
Example 2: The FDA is researching the COVID-19 vaccine.
Explanation 1: The entity "China" was never mentioned in the article. Even though the information might be true, since it was not contained in the article the fact should be considered wrong.
Explanation 2: Both entities "The FDA" and "the COVID-19 vaccine" appear in the article. However, the relation "is researching" cannot be verified based on the article (even though the information might be true) so the fact should be considered wrong.
- Grammatically meaningless:
The grammar of the sentence is so wrong that it becomes meaningless. Minor grammar errors should not be penalized if the meaning of the sentence is still clear.
Example: The Ebola vaccine accepted have already started.
Figure 7: Instructions can be toggled.
Main Task Part 3: Find Mistakes
Using the article as ground-truth, find mistakes in the highlighted sentence in the summary, and categorize the mistakes.
To help you, we underline the sentences in the article that are likely to contain relevant information. However, this is for guidance only: you should not restrict yourself to the highlighted sentences when answering the questions.
Are the facts in the highlighted sentence in the summary correct?
Article Text
Gaioz Nigalidze, the current Georgian champion, was expelled from the Dubai Open Chess tournament when he was found using his smartphone in the middle of a match A chess grandmaster has been thrown out of an international tournament and faces a 15-year ban after he was caught sneaking to the toilet to check moves on his mobile phone. Gaioz Nigalidze, the current Georgian champion, was expelled from the Dubai Open Chess tournament when he was found using his phone in the middle of a match. The two-time national champion was exposed when his opponent lodged a complaint when he grew suspicious about his frequent trips to the lavatory. Tournament organisers found Nigalidze had stored a mobile phone in a cubicle, covered in toilet paper. They announced their decision to expel Nigalidze on Sunday morning on their Facebook page. The complaint was made by Nigalidze's opponent in their sixth-round match in the tournament, Armenia's Tigran Petrosian. He said: "Nigalidze would promptly reply to my moves and then literally run to the toilet." I noticed that he would always visit the same toilet partition, which was strange, since two other partitions weren't occupied. I informed the chief arbiter about my
UNK UNK was expelled from the dubai open chess tournament . the two-time national championwas expelled from the dubai open chess tournament when he was found using his phone inthe middle of a match , the two-time national champion was expelled from the dubai open chesstournament when he was found using his phone in the middle of a match .
Figure 8: The sentences being annotated is highlighted in yellow. Relevant text is underlined in the article plain text.
Tip: Unsure about which category?
What kind of mistakes are present in the highlighted sentence? Select all that apply.
□ Information not in article: entity or relation were not mentioned in the text.
Grammatically meaningless: very wrong grammar cannot be understood.
Misuse of pronoun: wrong pronoun ("he", "she", etc.) or referring expression ("the former", etc.).
$\square$  Wrong relationship between entities: what happened is wrong (typically described by the verb).
Wrong entities in the relation: the "who", "what", "to whom", etc. is wrong. Relationship appears in the text but with different entities.
$\square$  Wrong circumstance: wrong location, time, date, goal, manner, adverbs etc.
$\square$  Wrong relationship between facts: logical or temporal link of facts is wrong.
Gaioz Nigalidze, the current Georgian champion, was expelled from the Dubai Open Chess tournament when he was found using his smartphone in the middle of a match A chess grandmaster has been thrown out of an international tournament and faces a 15-year ban after he was caught sneaking to the toilet to check moves on his mobile phone. Gaioz Nigalidze, the current Georgian champion, was expelled from the Dubai Open Chess tournament when he was found using his phone in the middle of a match. The two-time national champion was exposed when his opponent lodged a complaint when he grew suspicious about his frequent trips to the lavatory. Tournament organisers found Nigalidze had stored a mobile phone in a cubicle, covered in toilet paper. They announced their decision to expel Nigalidze on Sunday morning on their Facebook page. The complaint was made by Nigalidze's opponent in their sixth-round match in the
UNK UNK was expelled from the dubai open chess tournament the two-time national champion was expelled from the dubai open chess tournament when he was found using his phone in the middle of a match . the two-time national champion was expelled from the dubai open chess tournament when he was found using his phone in the middle of a match .
Main Task Part 1: Article
Read the following article fully. Note: article could take some time to load. We will ask a simple question to check you read it and you will not have access to the article.
Finished Reading
Article Webpage
Like 3.4M
Follow @
ISport D
Mail Online
Figure 9: After selecting that the sentence is not factual annotators choose the category of error.
U.K. News
U.S. Showbiz
Femail | Health
Science | Money
Transfer News
BoxingUFC
Tennis F1
Racing Rug
by Cricket
More Fantasy Football
At 18.
Bavern
I'll not
Van Gaal 1
Figure 10: Articles web pages are provided.
Clvne
Bonus: Maintain quality work to get bonuses and remain qualified
You will be awarded a $1 bonus for quality work per HIT and an extra $1 bonus every 10 quality HITs.
If your work quality is poor we will revoke your qualification and if it is very poor you will not be paid.
We will check your answers and ensure that your work quality remains high. The results of this HIT will be used to conduct research.
Main Task Part 2: Article Question
Answer the question below about the article that you just read.
Which of the following was not mentioned in the article?
Olivier Giroud
$\bigcirc$  Arsenal
Nairobi-born Mr Kantaria
Figure 11: Entity question to ensure annotators read the text.