BUMP: A Benchmark of Unfaithful Minimal Pairs for Meta-Evaluation of Faithfulness Metrics
Year: 2023

Abstract:
The proliferation of automatic faithfulness metrics for summarization has produced a need for benchmarks to evaluate them. While existing benchmarks measure the correlation with human judgements of faithfulness on modelgenerated summaries, they are insufficient for diagnosing whether metrics are: 1) consistent, i.e., indicate lower faithfulness as errors are introduced into a summary, 2) effective on human-written texts, and 3) sensitive to different error types (as summaries can contain multiple errors). To address these needs, we present a benchmark of unfaithful minimal pairs (BUMP), a dataset of 889 human-written, minimally different summary pairs, where a single error is introduced to a summary from the CNN/DailyMail dataset to produce an unfaithful summary. We find BUMP complements existing benchmarks in a number of ways: 1) the summaries in BUMP are harder to discriminate and less probable under SOTA summarization models, 2) unlike non-pair-based datasets, BUMP can be used to measure the consistency of metrics, and reveals that the most discriminative metrics tend not to be the most consistent, and 3) unlike datasets containing generated summaries with multiple errors, BUMP enables the measurement of metrics’ performance on individual error types.

heading: 1 Introduction
text: Although modern abstractive summarization systems have improved in their ability to produce fluent text (Lewis et al., 2020), their ability to generate text that is factually grounded in the source article remains an issue (Kryscinski et al., 2020). This phenomenon has inspired the NLP community to develop faithfulness evaluation metrics (Fabbri et al., 2022; Laban et al., 2022; Honovich et al., 2021; Scialom et al., 2021) that automatically measure the extent to which abstractive summarization systems produce summaries that contain information that cannot be verified by the source article.
As the number of these automatic faithfulness metrics has increased, there has arisen a corresponding need for benchmarks that evaluate their relative strengths. To satisfy this need, researchers have developed datasets such as FRANK (Pagnoni et al., 2021) and TRUE (Honovich et al., 2022) that are comprised of model-generated summaries along with human-annotated faithfulness levels. Although these datasets are useful for evaluating the degree to which faithfulness metrics correlate with human judgements and can discriminate unfaithful summaries, a number of factors limit the conclusions that can be drawn from them. For one, because model summaries can vary in terms of length, content, and number of errors, these benchmarks are ill-suited for drawing conclusions about the consistency (Gabriel et al., 2021) of metrics, i.e., whether their scores indicate lower faithfulness as summaries become increasingly unfaithful, as well as their sensitivity to specific types of errors since summaries can contain multiple errors. Furthermore, as the summaries are machine-generated, these benchmarks cannot evaluate whether metrics can detect human-written unfaithful summaries.
To enable research on these topics, we present BUMP—a benchmark of unfaithful minimal pairs— a dataset of 889 minimally different summary pairs where all unfaithful summaries are generated by human annotators. As illustrated in Figure 1, given an article and its reference summary, we ask a human annotator to edit the reference summary in a minimal way such that the edited summary exhibits one unfaithful error. We design two tasks for performance comparisons: 1) taxonomy-based edits, where a specific unfaithfulness error type is required according to our proposed taxonomy, and 2) freestyle edits, where no error type constraints are imposed. The motivation behind the first task setting is to ensure that different error types are adequately represented in our dataset, while the second task setting is important for understanding
12788
Article: Manchester City and Chelsea are set to battle it out for the signature of West Ham left-back Aaron Cresswell this summer... West Ham snapped up Cresswell for £2million from Ipswich last summer but he has adapted to the top tier with relative ease and attracted the eye of the division’s Champions League clubs... England stars Jack Wilshere, Jordan Henderson and Liverpool contract rebel Raheem Sterling are also on City’s wanted list to fulfil the quota. Premier League sides must name eight homegrown players in their 25-man squad registered at the beginning of the season. A player is deemed homegrown if they have spent at least three years at any English club before the age of 21, regardless of nationality.
Summary [Extrinsic Entity Error]: Aaron Cresswell has impressed during debut season in Premier League . The left back joined West Ham from Championship club Ipswich for £2m . Manchester City and Chelsea are both keen to sign the 25-year-old . Both clubs are mindful of boosting their quota of homegrown foreign players .
Faithfulness Metric Scores
Reference
Edited
QUESTEVAL
Consistent Reference
Edited
COCO
Consistent Reference
Edited
FACTCC
Inconsistent
Figure 1: Example from BUMP dataset. An annotator constructs an unfaithful summary containing an Extrinsic Entity Error (Section 3.2) by replacing the word “homegrown” in the reference summary with the word “foreign”. The reference and edited summary form a minimal unfaithful summary pair. Faithfulness metrics are evaluated on both the reference and edited summary and compared to measure whether the metric is consistent, e.g., in this example, QUESTEVAL and COCO are consistent, while FACTCC is not.
the completeness of our error type taxonomy as well as whether annotation difficulty is affected by instructing annotators to focus on specific error types.
We use BUMP to study the ability and performance consistency of faithfulness evaluation metrics in differentiating unfaithful summaries from faithful ones. Similar to how minimal pairs are used to diagnose linguistic knowledge of language models (Marvin and Linzen, 2018; Warstadt et al., 2020), the minimal summary pairs in BUMP allow targeted tests of a metric’s consistency on different types of errors (Table 1). This setup minimizes the effect of confounding factors that affect similar analyses (e.g., Pagnoni et al. (2021) and Tang et al. (2022)) such as text length, stylistic variation, and multiple errors occurring in the same summary. We evaluate standard and state-of-the-art faithfulness metrics on BUMP using meta-evaluation protocols that target two phenomena: 1) consistency, i.e. the fraction of unfaithful summaries that receive a lower score than their corresponding faithful summaries, and 2) discriminability, i.e., the metric’s ability to classify unfaithful vs. faithful summaries as measured by ROC AUC.
Our results (Section 5) yield a number of useful findings: 1) BUMP differs substantially from existing benchmarks: the summaries in BUMP are harder to discriminate (ROC AUC scores between 50–70% vs. 70–84%) and are less probable under SOTA summarization models. 2) Discriminability != consistency: interestingly, the most consistent
metrics (BARTSCORE, COCO) tend to have poor discriminability. 3) Some error types are harder than others: e.g., metrics seem to uniformly struggle with summaries containing Intrinsic Errors.
In sum, our contributions are three-fold: 1) We build a benchmark of human-generated unfaithful minimal pairs (BUMP) for evaluating faithfulness metrics. 2) We show human-generated unfaithful errors are substantially different from and more challenging than model-generated ones. 3) We demonstrate how BUMP provides insights on both the consistency and discriminative ability of faithfulness metrics on different error types than prior evaluation benchmarks that complement insights from existing benchmarks. The BUMP dataset is available at: https://github.com/ dataminr-ai/BUMP.

heading: 2 Related Work
text: Standard evaluation metrics for text generation tasks, e.g., BLEU and ROUGE, do not correlate well with human judgements of factual alignment in summarization settings (Kryscinski et al., 2019; Maynez et al., 2020). This has motivated the development of automated faithfulness metrics that quantify factual alignment through methods that either: use NLI to measure the entailment degree between the source article and summary (Kryscinski et al., 2020; Goyal and Durrett, 2020; Laban et al., 2022), compare summary probabilities when relevant information is removed from the source (Xie et al., 2021), or use question answering models to
measure if questions derived from the source can be answered by the summary and vice versa (Wang et al., 2020; Durmus et al., 2020; Scialom et al., 2021).
Existing faithfulness metric evaluations use one of two classes of benchmarks: 1) machinegenerated summaries paired with human-annotated faithfulness levels (Laban et al., 2022; Pagnoni et al., 2021; Tang et al., 2022), and 2) summary pairs pertaining to the same article where one summary is faithful and the other is unfaithful (Falke et al., 2019; Gabriel et al., 2021). While both classes can evaluate a metric’s ability to discriminate unfaithful summaries, the latter additionally allows for consistency tests, i.e., whether metrics assign higher values to more faithful summaries.
The BUMP dataset belongs to the second class of benchmarks; however, it has a number of unique properties. First, unlike both Falke et al. (2019) and Gabriel et al. (2021), the unfaithful summaries in BUMP are human-written. In addition, the unfaithful summaries in BUMP are minimally different, in the sense that only a single error differentiates the faithful and unfaithful summary. As shown in Section 5, this produces summary pairs that are substantially more challenging for metrics to differentiate. Inspired by the use of minimal pairs to diagnose linguistic knowledge of language models (Marvin and Linzen, 2018; Warstadt et al., 2020), the benefit of this approach is that it allows targeted tests of a metric’s consistency on different types of errors (Section 3.2) while minimizing the effect of confounding factors. Therefore, unlike other benchmarks with error type annotations (Pagnoni et al., 2021; Tang et al., 2022), results on BUMP are not complicated by issues such as multiple errors appearing in the same summary.

heading: 3 Benchmark of Unfaithful Minimal Pairs (BUMP)
text: Two annotation tasks are designed for BUMP, where Task 1 is taxonomy-based (a specific error type is required for the edited summary), and Task 2 allows freestyle edits (i.e., no error type constraints are imposed). In this section, we first describe how data sources are selected to build BUMP (3.1), and then describe the details of the two annotation tasks (3.2 and 3.3).

heading: 3.1 Dataset
text: For Task 1, we randomly select 100 articlesummary pairs from the test set of the CNN/DailyMail dataset (Hermann et al., 2015).1 For Task 2, we select an additional 100 random article-summary pairs. Both tasks are performed via Amazon Mechanical Turk.2

heading: 3.2 Task 1: Taxonomy-based Unfaithful Summaries
text: To obtain fine-grained performance evaluations of faithfulness metrics, it is critical to evaluate their sensitivity regarding various error types. Furthermore, benchmarks should contain sufficiently many instances associated with each error type to enable statistically significant comparisons to be made. To this end, we first define a taxonomy of unfaithful error types, and then ask annotators to introduce errors of a specific type in order to ensure each error type is adequately represented in the final dataset.
We note that existing taxonomies of error types may contain overlapped error types, e.g., grammatical vs. entity errors in FRANK (Pagnoni et al., 2021) or lack fine granularity, e.g., Tang et al. (2022). By considering the strengths and shortcomings of existing taxonomies, we define our own taxonomy in Table 1. Our taxonomy is first adapted from the one in FRANK (Pagnoni et al., 2021) by including semantic frame errors (Predicate Error, Entity Error, and Circumstance Error) and Coreference Error, and removing error types that might overlap with others. To further categorize each semantic frame error, we adopt the notions of Intrinsic and Extrinsic errors (Maynez et al., 2020; Goyal and Durrett, 2020; Tang et al., 2022). Note that we do not simply categorize errors into the Intrinsic and Extrinsic ones, as we believe semantic frame errors can better instruct annotators to create summaries with diverse unfaithful errors. In our taxonomy, the Intrinsic/Extrinsic distinction only applies to the Predicate, Entity, and Circumstance Error, since for a Coreference Error, it is generally ambiguous whether an erroneous pronoun/reference that does not exist in the source article should be regarded as intrinsic or extrinsic. In total, this results in seven different error types.
1We do not annotate samples from the XSum dataset (Narayan et al., 2018) since the reference summaries are frequently unfaithful (Maynez et al., 2020).
2https://www.mturk.com/; annotation guidelines and interfaces are detailed in Appendices A and B.
For each of the seven error types in this taxonomy, given an article-summary pair, we ask the annotator to introduce an error of the required type through a minimal edit to the reference summary. All <article, summary, error type> Human Intelligence Tasks (HITs) in Amazon Mechanical Turk are shuffled and there is no annotation repetition, i.e., one assignment per HIT. This increases the chance that edits of the same reference summary will be made by different annotators. Additional details regarding qualification tests and annotation instructions are presented in Appendix A.
After the data collection, we manually check the validity of each edit. For cases where the edits do not match the required error types, we relabel them with the corrected error types based on our taxonomy. The dataset statistics after correction are shown in Table 2. For this task, one common mistake is that annotators consider the quantity of a noun object as a circumstance and make edits to the quantity (the first example in Table 3), hence mistakenly treat Entity Errors as Circumstance Errors, which causes the total number of Circumstance Errors to be only 160 (much smaller than that of Entity Errors; see Table 2). Another frequent mistake is that the edited word actually exists in the original article for the required extrinsic error (the second example in Table 3), which results in a smaller number of Extrinsic Errors than intrinsic ones across all semantic frame errors, especially for Predicate Errors. Furthermore, Table 2 shows all edited summaries can be categorized by our taxonomy (no summaries are relabeled as “Other”), and the incorrect response rate is 16%, suggesting that, in general, annotators correctly respond with the required error types.

heading: 3.3 Task 2: Freestyle Unfaithful Summaries
text: In addition to the taxonomy-based Task 1, we also conduct a separate task, Task 2, where annotators can edit reference summaries in any way they want, i.e., freestyle editing, as long as only one error is introduced to the reference summary via minimal edits. The goal of Task 2 is to understand how human-generated unfaithful summaries may vary, and how the performance of faithfulness evaluation metrics changes accordingly, when there are no error type constraints. In particular, only annotators who did not participate in the qualification test of Task 1 are considered to participate in this task; in this way, we ensure the edited summaries in Task 2
are not constrained to any known error types. To post-process all data collected in Task 2, we manually assign an error type to each data point, based on our error type taxonomy in Task 1. Without informing annotators of any specific error types, we observe the rate that the “Other” label occurs is only 2.5% for Task 2 in Table 2. This confirms that the vast majority of errors produced by humans adhere to our proposed taxonomy. For more details on Task 2, please see Appendix B.
Remark. For both tasks, we ask annotators to introduce only one error (by editing the reference summary in a minimal way). We acknowledge that some reference summaries may be unfaithful in the first place; nevertheless, for both tasks, edited summaries are based on reference summaries, by which we ensure the edited summaries are always more unfaithful than reference summaries.

heading: 4 Meta-Evaluation of Faithfulness Evaluation Metrics
text: In this section, we first describe the faithfulness evaluation metrics benchmarked on BUMP (4.1).
Then meta-evaluation protocols are discussed (4.2).

heading: 4.1 Faithfulness Metrics
text: To cover diverse types of faithfulness metrics, in this section, we select metrics that are generally used for measuring generation quality (i.e., n-grambased metrics), recent metrics that are proposed specifically for faithfulness evaluations, as well as some pre-trained model based metrics, which are detailed as follows. We investigate their abilities to distinguish faithful summaries from their minimally edited counterparts.r n-Gram-Based Metrics: We evaluate the following 2 n-gram-based metrics: BLEU (Papineni et al., 2002) and ROUGE (ROUGE-2 Precision specifically) (Lin, 2004).r Faithfulness Evaluation Metrics: We evaluate the following 7 faithfulness evaluation metrics: QUESTEVAL (Scialom et al., 2021), Q2 (Honovich et al., 2021), QAFACTEVAL (Fabbri et al., 2022), FACTCC (Kryscinski et al., 2020), DAE (Goyal and Durrett, 2020), SUMMAC (Laban et al., 2022) and COCO (Xie et al., 2021) in this paper. To obtain a score for FACTCC, we take the classifier probability of the summary being faithful.rOther Metrics: We evaluate the following 3 pretrained model based metrics: BLEURT (Sellam et al., 2020) with the BLEURT-20 checkpoint (Pu et al., 2021), BERTSCORE (Zhang et al., 2020) (specifically the BERTSCORE-precision variant) using the DeBERTa-xl-MNLI model (He et al., 2021), and BARTSCORE (Yuan et al., 2021) with a BART (Lewis et al., 2020) model fine-tuned on the CNN/DailyMail dataset.
Note that for reference-based metrics, faithfulness scores are computed by treating the input article as the reference, and the reference/edited sum-
mary as the system output. We also normalize the direction of the metric score so that a higher score always corresponds to better faithfulness from the metric’s view, e.g., FACTCC predicts the probability that a summary is unfaithful, and so to obtain a faithfulness score, we take its complement.

heading: 4.2 Meta-Evaluation
text: Each faithfulness metric takes an article-summary pair and outputs a numerical faithfulness score. In our analysis, we measure faithfulness scores for both the reference summary as well as the humanannotated erroneous summary. We quantify the difference between faithfulness metrics on BUMP using two measurement protocols: consistency and ROC AUC. Originally introduced in Gabriel et al. (2021), consistency measures the success rate of a metric assigning a lower faithfulness score to the erroneous unfaithful summary. In contrast, ROC AUC instead measures the overall capability of a metric to discriminate faithful from unfaithful content for an input summary, and has previously been used by Honovich et al. (2022) for meta-evaluation. Although other metrics such as balanced accuracy have also been used to evaluate disciminability (Laban et al., 2022), we opt to use ROC AUC as it does not require determining a decision threshold.

heading: 5 Results
text: We report and analyze the performance of faithfulness metrics in this section using meta-evaluation protocol consistency and ROC AUC.
Consistency. The consistency studies of the two tasks3 for all the metrics are reported in Table 4. In terms of the difficulty per error type, 1) for Task 1,
3Note that for Task 2, the error types with only a few samples (e.g., Coreference and Other) are not analyzed separately.
Extrinsic Entity Errors are generally the easiest, while all but BARTSCORE struggle with Intrinsic Predicate Errors; 2) for Task 2, Intrinsic Entity Errors are the hardest. This implies that when annotators are not presented with any error types, the introduced error styles may differ from those in Task 1 (see Section 6), potentially causing inconsistencies for metrics in these two tasks. Nevertheless, we observe that for both tasks, Intrinsic Errors are more challenging than extrinsic ones across all but FACTCC in Task 2. This is likely because Intrinsic Errors can be derived from the original article, while Extrinsic Errors contain words that do not appear in the original article, making Intrinsic Errors more subtle to be identified than extrinsic ones.
For the overall performance (all error types are considered), BARTSCORE has the highest consistency in both tasks, though BARTSCORE has not been proposed specifically for faithfulness evaluations. Other metrics that rank top 4 in both tasks include QAFACTEVAL and COCO. By comparison, Q2 and FACTCC have the worst consistency, even worse than n-gram-based metrics ROUGE and BLEU; nevertheless, they exhibit different rankings in terms of ROC AUC (see the next section).
ROC AUC. ROC AUC scores are presented in Table 5. We observe that the overall ranking of faithfulness metrics according to ROC AUC sub-
stantially differs from the ranking according to consistency. In particular, the rank of BARTSCORE drops from the top one to the fifth, while Q2 improves significantly from second to last to second overall. QAFACTEVAL consistently exhibits high performance and even ranks first under ROC AUC, while n-gram based metrics, e.g., ROUGE-2 and BLEU consistently show the worst performance, as expected. In general, metrics that are specifically proposed for faithfulness evaluations rank higher than generic NLG evaluation metrics.
We additionally observe that the relative rankings of ROC AUC scores across error types and task settings are largely consistent with the relative rankings of consistency scores. Specifically, we again observe that on a per metric basis: 1) ROC AUC scores are generally lower for Task 2 than Task 1 (particularly for Entity Errors), and 2) metrics generally show worse performance on Intrinsic Errors than extrinsic ones.
For our two meta-evaluation protocols, consistency is suitable for the pairwise ranking of two summaries for a given input article, while ROC AUC is more adequate in evaluating the absolute capacity of unfaithful summary detection. If a metric has high consistency but low ROC AUC, it implies that the scores for predicted faithful and unfaithful summaries overlap frequently. Such overlap makes
it challenging to establish a clear decision boundary for classifications. Hence, to improve the classification capability of metrics with high consistency, more calibration is needed to increase the score gap between faithful and unfaithful summaries.

heading: 6 Analysis of BUMP
text: In this section, we conduct more analysis of BUMP by studying how BUMP differs from other benchmarks, followed by a qualitative analysis of the detection difficulty between Tasks 1 and 2.
Comparison with Model-Generated Unfaithful Summaries. We compare the generation probabilities of our edited summaries to those of summaries generated from beam search by a BARTbased summarizer (trained using the training data of CNN/DailyMail) for the same set of documents in our dataset. We report the difference of these generation probabilities normalized by the text length in Figure 2, where we find our edited summaries are much different from model generations in terms of the model generation probabilities. This highlights that existing metrics may not work well on summaries of various styles and experiments are needed to verify their effectiveness in humangenerated unfaithful summaries.
Furthermore, we compare our ROC AUC scores with those in existing datasets as shown in TRUE
(Honovich et al., 2022). In BUMP, faithful and unfaithful samples under each error type are balanced for both Tasks 1 and 2. Therefore, for a fair comparison, we pick QAGS-C (Wang et al., 2020) (also a balanced dataset on CNN/DailyMail) in TRUE. In Table 5, it shows that the ROC AUC scores evaluated on BUMP are generally much smaller (50–70% with many values close to random baseline), whereas most ROC AUC scores are 70–84% in QAGS-C (see Appendix C). This again
indicates that the human-generated errors in BUMP are more difficult for metrics to detect than modelgenerated errors in existing datasets, reinforcing the value of BUMP as a challenging benchmark for evaluating faithfulness metrics. In addition, we also compare the ROC AUC rankings of different faithfulness metrics under QAGS-C and BUMP. Specifically, we summarize the performance rankings under QAGS-C from Appendix C as well as those from Table 5 under Intrinsic/Extrinsic Error types in Tasks 1 and 2, and report them in Figure 3, where only faithfulness metrics used in both (Honovich et al., 2022) and Table 5 are presented. In Figure 3, we observe that for some faithfulness metrics, such as Q2 and BARTSCORE, their ROC AUC rankings are quite stable across all datasets. However, for other faithfulness metrics, the performance ranking under QAGS-C is very different from the ranking derived from BUMP, e.g., QUESTEVAL mostly exhibits high ROC AUC ranking in BUMP; by contrast, it experiences the worst performance in QAGS-C. Thus, we believe BUMP complements existing benchmarks and allows a more comprehensive analysis of faithfulness metrics in future studies.
Qualitative Analysis. We provide a qualitative analysis of examples that demonstrate the increased difficulty of Task 2. The examples are provided in Table 6. Each row contains edited summaries from Tasks 1 and 2 for the same original article and its reference summary. In addition, to compare edited summaries under the same error type, we pick examples where the corrected error type from Task 1
is the same as the exhibited error type from Task 2. As shown in Table 6, in the first example, for the Extrinsic Entity Error type, the annotator in Task 1 modifies the entity elevator shaft to another entity staircase. Whereas the annotator in Task 2 modifies the word communal to private (i.e., also an Extrinsic Entity Error) which requires commonsense knowledge to infer that private is contradictory to the fact that the elevator is used by several businesses in the building. In the second example, for the Extrinsic Entity Error type, the annotator in Task 1 modifies the entity name from Claire Nugent to a random name Tim Horton, whereas the annotator in Task 2 changes record player to black and white TV to fit the 60s theme, which again, requires additional knowledge. In the last example,
the annotator in Task 2 modifies the temporal state of the action sign from signed to is considering signing which is more challenging than changing the action urged to its antonym discouraged as the annotator in Task 1 does.
For the first two examples of Task 2 in Table 6, only 4 metrics (QAFACTEVAL, QUESTEVAL, BLEURT, and BERTSCORE for the first example; QAFACTEVAL, SUMMAC, BLEURT, and ROUGE-2 for the second example) succeed in giving a higher score to the reference summary. In comparison, 9 and 11 metrics succeed in giving a higher score to the reference summary in their Task 1 counterparts, respectively. For the last example, 8 metrics succeed in Task 2 and all 12 metrics succeed in Task 1. Thus, Table 6 shows that some unfaithful summaries in Task 2 are more challenging for faithfulness metrics to detect, which further exemplifies the challenges of Task 2 in BUMP.

heading: 7 Conclusion
text: In this paper, we presented a benchmark of unfaithful minimal pairs (BUMP) to evaluate faithfulness metrics. Unlike prior work where all unfaithful summaries are model generated, each unfaithful summary in BUMP is generated by minimal human edits to introduce one unfaithful error given a reference summary. Through our experiments, we found that BUMP complements existing benchmarks in a number of ways. First, we found that the summaries in BUMP are harder to discriminate and less probable under SOTA summarization models. Second, we used BUMP to measure the consistency of metrics, which cannot be readily measured using other benchmarks. This analysis revealed a discrepancy between the discriminability and consistency of existing metrics, highlighting an important area for future faithfulness metric research to address. Finally, we used BUMP to study faithfulness metrics’ performance on individual error types—where our minimal-pair-inspired setup helped control for conclusions being conflated across multiple error types—which revealed that sensitivity to intrinsic errors is another important area for future research to focus on.

heading: Acknowledgements
text: We would like to thank our colleague Aoife Cahill at Dataminr for her valuable comments, suggestions, and support for this paper. We also thank the anonymous reviewers for their feedback and
comments.
Limitations
Although BUMP is, to our knowledge, the first dataset on which to study the consistency of faithfulness metrics on human-written errors across different error types, there are some limitations regarding the conclusions that can be drawn from it. For one, because BUMP is comprised of minimal edits to reference summaries from CNN/DailyMail, it is not suitable for analyzing the consistency of faithfulness metrics when errors are added to reference summaries already containing many errors. In addition, due to a combination of resource constraints and human preferences for writing specific types of errors, the sample sizes for some error types in Task 2 (e.g., Coreference Error and Intrinsic Predicate Error) may not be sufficiently large to enable statistically significant comparisons between different metrics for specific error types.
Ethics Statement
The collection of BUMP involves human annotations. The human annotators are provided with clear task instructions and informed of the conditions where they would be qualified and disqualified. We compensate annotators with $3.00 per assignment in the qualification task and $0.50 per assignment in the full task for both Tasks 1 and 2. The final paid rate is $15 per hour which is over the US national minimum wage4 of $7.25. We are also aware that our shared datasets could be potentially misused as training samples, albeit a small number, to develop models to generate unfaithful content.

heading: A Details of Task 1: Taxonomy-based Unfaithful Summaries
text: A.1 Qualification Task The instructions and the task interface for the qualification task of Task 1 are shown in Figures A1 to A4.
In this qualification task, all US-based annotators are able to participate. Specifically, we ask annotators to read a news article and seven pairs of summaries. For each pair of summaries, the first summary is the correct reference summary, and the second summary is the unfaithfully edited summary that contains one of the seven error types in our taxonomy. We then ask the annotators to select one answer from the seven error types to indicate which type of error is introduced in the edited unfaithful summary. Only the annotators who answered 6 out of these 7 questions correctly passed the qualification task. We launched 3 batches in total with 9 assignments for each batch, and 9 annotators passed the qualification task.
A.2 Full Task The instructions and the task interface for the full task of Task 1 are shown in Figures A5 to A7.
In the full task for Task 1, different from the qualification task, we ask the annotators to read a news article from CNN/DailyMail (Hermann et al., 2015) and one reference summary for the article. We then ask the annotators to edit the reference summary to introduce the error type specified through a minimal edit. If they cannot introduce the error type based on the reference summary, they can write “N/A” to indicate that it is impossible to introduce the specified error type based on the provided reference summary. There are 18 samples in Task 1 dataset that are annotated as “N/A” by the annotators, all of which are reviewed by the authors of this paper and re-annotated with the correct edits (we note that the required error types can be provided for all these cases) as a post-processing step to ensure the completeness of the dataset.
In addition, for Task 1, to help reduce the confusion from annotators regarding Circumstance Errors and Entity Errors, we explicitly specify that the Circumstance Errors should only be erroneous edits concerning the time, duration, or location of
an event, and changing the quantity of a noun is not considered as a Circumstance Error.

heading: B Details of Task 2: Freestyle Unfaithful Summaries
text: B.1 Qualification Task
The instructions and the task interface for the qualification task of Task 2 are shown in Figures A8 to A9.
In this qualification task, all US-based annotators who did not participate in the qualification task of Task 1 are qualified to participate. Specifically, we show the annotators four pairs of news article and its summary from CNN/DailyMail, and ask them to answer if the summaries are faithful based on the original news articles. Among the four pairs, three of them are unfaithful and one is faithful. Only the annotators who answered correctly to all of these 4 pairs passed the qualification task. We launched 3 batches in total with 9 assignments for each batch, and 8 annotators passed the qualification task.
B.2 Full Task
The instructions and the task interface for the full task of Task 2 are shown in Figures A10 to A11.
In the full task of Task 2, unlike Task 1, we do not list any potential error types so as to achieve freestyle editing. The edited summary is valid as long as only one error is introduced based on the reference summary via a minimal edit. Furthermore, we also do the following to ensure the quality of edited summaries:
• For minimal edits, we explicitly ask annotators not to write from scratch, but to introduce only one error on top of the given reference summary.
• In the pilot study, we notice that some edited summaries are simply removing/adding sentences or phrases (such data points are removed in the final released data); we, therefore, add additional instructions that require the edited and the reference summaries to contain a similar amount of information about the given news article (i.e., similar coverage).
• The edited summaries should be grammatically correct.
• The edited summaries should be plausible and adhere to common sense.
• Some examples of edited summaries are given in the task instructions.

heading: C ROC AUC Results from Other Benchmarks
text: To compare BUMP with other benchmarks, we also report the ROC AUC scores from TRUE (Honovich et al., 2022). Specifically, in BUMP, faithful and unfaithful samples under each error type are balanced for both Tasks 1 and 2. Therefore, for a fair comparison, 1) in TRUE, we pick QAGSC (Wang et al., 2020), which is also a balanced dataset on CNN/DailyMail. The ROC AUC scores of QAGS-C are reported in Table A1; 2) for faithfulness metrics in Table A1, we use the same implementation and model checkpoints in this paper as those in TRUE (Honovich et al., 2022). Then according to Table A1, the metric performance ranking in terms of ROC AUC for QAGS-C is Q2 > SUMMAC > BARTSCORE > FACTCC > BLEURT > BERTSCORE > QUESTEVAL, which is very different from the ranking derived from our BUMP dataset, e.g., SUMMAC exhibits worse ROC AUC than QUESTEVAL for most error types in both Tasks 1 and 2 (see Table 5).
In addition to the balanced dataset QAGS-C in TRUE, we also report the ROC AUC scores of imbalanced FRANK (Pagnoni et al., 2021) and SummEval (Fabbri et al., 2021) datasets (two datasets containing CNN/DailyMail) from TRUE in Table A1. Although the FRANK and SummEval datasets are imbalanced, we have similar observations as those from the QAGS-C dataset: 1) their ROC AUC scores (mostly 70–90%) are much larger than the ROC AUC scores (50–70%) derived from our BUMP dataset; 2) in terms of the ROC AUC ranking, the top two remain Q2 and SUMMAC for both FRANK and SummEval, and SUMMAC always ranks higher than QUESTEVAL. By contrast, in Table 5, we show that SUMMAC mostly exhibits worse ROC AUC than QUESTEVAL.
Figure A1: Screenshot of the qualification task for Task 1 (1/4).
Figure A2: Screenshot of the qualification task for Task 1 (2/4).
Figure A3: Screenshot of the qualification task for Task 1 (3/4).
Figure A4: Screenshot of the qualification task for Task 1 (4/4).
Q2 SUMMAC BARTSCORE FACTCC BLEURT BERTSCORE QUESTEVAL
QAGS-C 83.5 80.9 80.9 76.4 71.6 69.1 64.2
FRANK 87.8 89.1 86.1 76.4 82.8 84.3 84.0 SummEval 78.8 81.7 73.5 75.9 66.7 77.2 70.1
Table A1: ROC AUC (%) of faithfulness evaluation metrics in TRUE (Honovich et al., 2022). All datasets contain CNN/DailyMail. Faithful and unfaithful samples in QAGS-C are balanced; however, in FRANK and SummEval, faithful and unfaithful samples are imbalanced.
Figure A5: Screenshot of the full task for Task 1 (1/3).
Figure A6: Screenshot of the full task for Task 1 (2/3).
Figure A7: Screenshot of the full task for Task 1 (3/3).
Figure A8: Screenshot of the qualification task for Task 2 (1/2).
Figure A9: Screenshot of the qualification task for Task 2 (2/2).
Figure A10: Screenshot of the full task for Task 2 (1/2).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: In the Limitations section

heading: 3 A2. Did you discuss any potential risks of your work?
text: In the section of Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? CNN/DailyMail is distributed under the MIT license. We plan on releasing our dataset under the MIT license as well pending legal approval. The LICENSE will be provided alongside the dataset as a text file on GitHub when the paper is published. UPDATE AFTER REVIEW: We recently learned that the data we were using is distributed under an Apache 2.0 license instead of MIT.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? CNN/DailyMail is distributed under the MIT license. Our BUMP dataset is derived from CNN/DailyMail. We plan on releasing BUMP under the MIT license as well pending legal approval. The LICENSE will be provided alongside the dataset as a text file on GitHub when the paper is published. Therefore, the data collection and distribution of BUMP is consistent with the license in CNN/DailyMail. UPDATE AFTER REVIEW: We recently learned that the data we were using is distributed under an Apache 2.0 license instead of MIT.

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We manually checked all collected human annotations; see Section 3.2 and 3.3

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3, and Appendix A and B
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Table 2 and its descriptions in Section 3.2 and 3.3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 4 and 5
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Not applicable. Left blank.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.1
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 3

heading: 3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Section 3, Appendix A and B

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section of Ethics Statement, and Appendix A.1 and B.1
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? The BUMP dataset is derived from CNN/DM. Attribution is provided in Section 3.1. Since CNN/DM is distributed under the MIT license (which allows modifications), we did not discuss with annotators on how their modified summaries will be used. UPDATE AFTER REVIEW: We recently learned that the data we were using is distributed under an Apache 2.0 license instead of MIT.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? We used a review process internal to our organization with HCI research scientists.

heading: 3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Appendix A.1 and B.1

heading: Future Work
text: This analysis revealed a discrepancy between the discriminability and consistency of existing metrics, highlighting an important area for future faithfulness metric research to address