SlowGuess/NAACL_2022_OCR huggingface dataset:
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Artidoro Pagnoni, Vidhisha Balachandran, and Yulia Tsvetkov. 2021. Understanding factuality in abstractive summarization with FRANK: A benchmark for factuality metrics. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4812-4829, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624-643, Online. Association for Computational Linguistics.
Thomas Scialom, Paul-Alexis Dray, Gallinari Patrick, Lamprier Sylvain, Piwowarski Benjamin, Staiano Jacopo, and Wang Alex. 2021. Questeval: Summarization asks for fact-based evaluation. ArXiv preprint, abs/2103.12693.
Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3246-3256, Hong Kong, China. Association for Computational Linguistics.
Oleg Vasilyev, Vedant Dharnidharka, and John Bohannon. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. In Proceedings
of the First Workshop on Evaluation and Comparison of NLP Systems, pages 11-20, Online. Association for Computational Linguistics.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020b. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv preprint, abs/1910.03771.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 11328-11339. PMLR.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A Additional Data and Model Details
In this section, we provide details regarding statistical testing, benchmark statistics, and miscellaneous details regarding our QA-based experiments.
A.1 Statistical Testing
To determine whether the improvements on the SummaC benchmark are statistically significant, we perform significance tests using bootstrap resampling (Efron, 1982), following Laban et al. (2021). We compare our best model to the best-performing model from prior work on a given subset of the benchmark. We compare confidence intervals at significance levels of 0.05 and 0.01 and apply the Bonferroni correction (Bonferroni, 1935). Statistically significant differences at the 0.01 level exist between QAFACTEVAL-NLI (supervised) and the best prior work on the FRANK subset and for the overall benchmark result. We do not see statistically significant differences on the other datasets in the benchmark. However, the statistically significant difference at the overall benchmark is notable; while other metrics may perform comparably or better on a given dataset, our metric demonstrates consistent good performance across datasets.
A.2 Benchmark Statistics
For completeness, we provide additional statistics for the SummaC benchmark in Table 6. Due to the exclusion of Omission and Addition as factual consistency errors in the Polytope dataset, our dataset contains benchmark replication contains many more positive examples for that dataset. For XSF, we restrict the dataset to those examples with labels for factual consistency with respect to the source, as opposed to more general factuality labels which take into account world knowledge, which results in fewer examples than the original SummaC benchmark. This is the same subset as was used in Koto et al. (2021).
Please see the following links for the licenses of the datasets and annotations:  $\mathrm{CGS}^2$ ,  $\mathrm{XSF}^3$ ,  $\mathrm{FactCC}^4$ ,  $\mathrm{SummEval}^5$ . We did not find licenses for the remaining datasets analyzed in our study. The intended uses of these licenses align with our use for research purposes.
$^{2}$ https://tudatalib.ulb.tu-darmstadt.de/handle/tudatalib/2002 $^{3}$ https://github.com/google-research-datasets/xsum_hallucination_annotations#license $^{4}$ https://github.com/salesforce/factCC/blob/master/License.txt $^{5}$ https://github.com/Yale-LILY/SummEval/blob/master/License
Dataset	# Valid	# Test	% Positive
CGS	1281	400	49.7
XSF	996	996	9.4
Polytope	634	634	87.2
FactCC	931	503	85.8
SummEval	850	850	90.6
FRANK	671	1575	33.2
Table 6: Statistics of the six datasets in the SummaC benchmark. We provide the number of validation and test set examples and the percentage of positive examples in the validation set.
A.3 Model Parameters
Ablation experiments started from a combination that provided good initial validation results and then swapped components. Running every combination of QA-based metric components is expensive. We experimented with running an ablation of the QA models with a 2nd-best performing answer selection component ALL. This reduced all scores compared to using the NP Chunks component. This experiment supports our setup of keeping the best component constant when running ablations in order to determine the highest-performing combination of components, rather than experimenting with every combination.
Inference for the MADE QA model is run using the average of the six MADE adapters' parameters.
For Question Filtering with the AnsweredSumm Filter, in addition to if the Electra-large QA model labels the question as unanswerable, if the  $F1$  overlap score between the selected answer and the QA model output is less than 0.60, we remove this question. This filter was added only to AnsweredSumm and not AnsweredInput as answering questions based on the summary, from which the question was generated, should be an easy task. We reached this threshold based on a qualitative analysis of model outputs, although this number could have also been further tuned on the validation set.
B Additional Correlation Results
We provide additional correlation coefficients as a point of reference for future work. Instance-level correlations calculate the correlation between all instances, while the summary-level correlation computes the correlation between scores for each summary of the same input and then averages over inputs. Summary-level correlations are excluded
for QAGS as this dataset does not contain annotations for multiple models, which is necessary to compute this score.
Model Type	Model Name	XSF	SummEval	FRANK-CNNDM	FRANK-XSum	QAGs-CNNDM	QAGs-XSum
Misc	BARTScore	0.25	0.34	0.54	0.14	0.68	0.17
BLANC	0.07	0.20	0.33	0.06	0.30	0.03
Entailment	FactCC	0.05	0.37	0.41	0.05	0.49	0.26
BertScore-FFCI	0.45	0.26	0.34	0.15	0.50	0.20
DAE	0.00	0.40	0.49	0.20	0.58	-0.14
ANLI	0.18	0.35	0.46	0.08	0.60	0.36
MNLI	0.16	0.39	0.49	0.11	0.61	0.35
DocNLI	0.01	0.34	0.11	0.21	0.21	-0.38
SCZeroShot	0.06	0.39	0.48	0.23	0.52	0.44
QA	QuestEval	0.43	0.33	0.47	0.24	0.45	0.24
QAFACTVAL	0.30	0.43	0.54	0.26	0.64	0.44
Learned	SCConv (synthetic)	0.19	0.41	0.54	0.22	0.04	0.04
QAFACTVAL-NLI(synthetic)	0.16	0.47	0.60	0.21	0.64	0.47
Table 7: Instance-level Spearman correlation coefficients across factual consistency evaluation datasets. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. The two highest-correlated metrics for each dataset are shown in bold.
Model Type	Model Name	XSF	SummEval	FRANK-CNNDM	FRANK-XSum	QAGs-CNNDM	QAGs-XSum
Misc	BARTScore	0.17	0.27	0.42	0.12	0.55	0.14
BLANC	0.05	0.15	0.25	0.05	0.24	0.02
Entailment	FactCC	0.03	0.29	0.31	0.04	0.38	0.21
BertScore-FFCI	0.31	0.20	0.25	0.12	0.39	0.16
DAE	0.00	0.32	0.38	0.16	0.47	-0.11
ANLI	0.12	0.28	0.36	0.07	0.48	0.30
MNLI	0.11	0.31	0.38	0.09	0.49	0.28
DocNLI	0.01	0.27	0.08	0.17	0.17	-0.31
SCZeroShot	0.04	0.31	0.37	0.18	0.41	0.36
QA	QuestEval	0.30	0.26	0.36	0.20	0.35	0.20
QAFACTVAL	0.22	0.34	0.43	0.23	0.51	0.36
Learned	SCConv (synthetic)	0.13	0.33	0.42	0.18	0.03	0.03
QAFACTVAL-NLI(synthetic)	0.11	0.37	0.47	0.17	0.51	0.38
Table 8: Instance-level Kendall correlation coefficients across factual consistency evaluation datasets. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. The two highest-correlated metrics for each dataset are shown in bold.
Model Type	Model Name	XSF	SummEval	FRANK-CNNDM	FRANK-XSum
Misc	BARTScore	0.18	0.40	0.65	0.29
BLANC	0.12	0.27	0.47	0.01
Entailment	FactCC	-0.02	0.39	0.40	-0.07
BertScore-FFCI	0.21	0.37	0.44	0.19
DAE	0.01	0.51	0.54	0.32
ANLI	0.09	0.49	0.53	0.18
MNLI	0.10	0.48	0.51	0.17
DocNLI	0.00	0.52	0.21	0.47
SCZeroShot	0.11	0.57	0.60	0.52
QA	QuestEval	0.30	0.45	0.54	0.44
QAFACTVAL	0.24	0.64	0.68	0.53
Learned	SCConv (synthetic)	0.17	0.54	0.60	0.46
QAFACTVAL-NLI(synthetic)	0.16	0.64	0.70	0.48
Table 9: Summary-level Pearson correlation coefficients across factual consistency evaluation datasets. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. The two highest-correlated metrics for each dataset are shown in bold.
Model Type	Model Name	XSF	SummEval	FRANK-CNNDM	FRANK-XSum
Misc	BARTScore	0.18	0.38	0.59	0.28
BLANC	0.12	0.25	0.43	0.06
Entailment	FactCC	0.00	0.37	0.42	-0.01
BertScore-FFCI	0.21	0.34	0.40	0.20
DAE	0.00	0.40	0.47	0.30
ANLI	0.10	0.39	0.47	0.17
MNLI	0.08	0.38	0.48	0.15
DocNLI	-0.02	0.39	0.19	0.41
SCZeroShot	0.11	0.41	0.51	0.50
QA	QuestEval	0.27	0.35	0.47	0.45
QAFACTVAL	0.22	0.45	0.59	0.47
Learned	SCConv (synthetic)	0.16	0.43	0.55	0.44
QAFACTVAL-NLI(synthetic)	0.17	0.47	0.63	0.49
Table 10: Summary-level Spearman correlation coefficients across factual consistency evaluation datasets. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. The two highest-correlated metrics for each dataset are shown in bold.
Model Type	Model Name	XSF	SummEval	FRANK-CNNDM	FRANK-XSum
Misc	BARTScore	0.15	0.32	0.51	0.25
BLANC	0.11	0.21	0.38	0.05
Entailment	FactCC	0.00	0.30	0.35	-0.01
BertScore-FFCI	0.17	0.28	0.34	0.18
DAE	0.00	0.33	0.41	0.27
ANLI	0.08	0.32	0.41	0.16
MNLI	0.07	0.31	0.41	0.14
DocNLI	-0.01	0.32	0.17	0.37
SCZeroShot	0.10	0.34	0.44	0.45
QA	QuestEval	0.23	0.29	0.41	0.41
QAFACTVAL	0.19	0.37	0.51	0.45
Learned	SCConv (synthetic)	0.14	0.36	0.49	0.41
QAFACTVAL-NLI(synthetic)	0.14	0.39	0.55	0.44
Table 11: Summary-level Kendall correlation coefficients across factual consistency evaluation datasets. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. The two highest-correlated metrics for each dataset are shown in bold.