SlowGuess/NAACL_2022_OCR huggingface dataset:
Component	Model Choice	Benchmark
QAFACTEVAL	77.5
Answer Selection	NP Chunks	-
Max NP	75.7
NER	66.4
ALL	75.7
Question Generation	BART-large (QA2D)	-
BART-large (SQuAD)	74.3
T5-base (SQuAD)	67.0
MixQG-base	75.1
MixQG-large	74.9
Question Answering	Electra-large	-
Electra-base	77.0
MADE	77.4
T5-base	76.1
UnifiedQA-base	75.7
Answer Overlap	LERC (QuIP)	-
EM	68.4
F1	71.7
IsAnsweredInput	73.3
LERC (orig)	71.8
LERC (RoBERTa)	77.3
Filtering/Answerability	Both	-
No AnsweredSumm Filter	73.8
No Answerability Penalty	72.1
Neither	67.4
Table 2: Results of ablation studies on the SummaC benchmark validation set, showing the effect of the individual components of QAFACTEVAL. The first row represents the performance of the best combination of components. Ablations are performed by swapping a given component while holding all others consistent with the best overall model, and the best setting is bolded.
or ALL answers together. Named entity selection likely performs worse due to the scarcity of extracted answers; only three entities are extracted on average across the benchmark, while all other approaches extract over 10 answers per summary.
Effect of QG Models The choice of the QG model notably affects downstream performance. BART-large (QA2D) works the best and produces much longer questions, about 17 tokens on average, versus about 10 from the other models. Deutsch et al. (2020) note how humans tend to produce shorter questions. However, longer questions may be preferable for this task to facilitate the QA model's ability to understand and answer the question. BART-large (QA2D) also is the most extractive, with only about  $20\%$  novel unigrams in the question, while T5-base (SQuAD) model is the most abstractive with about  $47\%$  novel unigrams, resulting in occasional hallucinations and questions that the QA model struggles to answer. As seen in Table 4, MixQG models do often produce highly-fluent questions, but the longer, highly-extractive output of BART-large (QA2D) improves downstream factual consistency performance.
Effect of QA Model Surprisingly, we do not find a large difference in the QA model compo
Model Type	Model Name	CGS	XSF	Polytope	FactCC	SummEval	FRANK	Benchmark
Misc	BARTScore	63.3	53.3	80.4	66.8	69.8	80.0	68.9
BLANC	51.6	54.5	72.2	53.0	63.0	76.2	61.8
Entailment	FactCC	64.8	56.6	80.2	77.1	73.6	70.3	70.4
BertScore-FFCI	56.9	68.8	69.2	57.9	67.4	71.9	65.4
DAE	71.3	49.7	78.9	80.7	74.7	81.0	72.7
ANLI	74.9	53.0	77.6	85.8	75.9	78.9	74.4
MNLI	67.6	61.5	77.3	89.8	78.7	79.6	75.7
DocNLI	49.6	57.0	84.7	73.0	75.6	70.9	68.5
SCZeroShot	59.6	56.1	81.5	83.2	77.9	78.5	72.8
QA	QuestEval	59.4	61.9	73.1	66.5	68.4	79.8	68.2
QAFACTEVAL	75.1	63.1	79.8	84.1	80.9	83.9	77.8
Learned	SCConv (synthetic)	60.8	60.9	76.0	88.1	78.1	81.6	74.3
QAFACTEVAL-NLI (synthetic)	74.2	59.1	82.1	91.1	80.2	83.4	78.3
QAFACTEVAL-NLI (supervised)	78.1	60.9	83.7	89.3	80.5	84.3*	79.5*
Table 3: Balanced accuracy on the test set of the six SummaC benchmark datasets, and the average over the benchmark. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. An improvement over prior work with a  $99\%$  confidence interval is indicated by  $*$ .
Document	Paul Merson has restarted his row with Andros Townsend. ... '... it was a great goal,' Merson said. 'It's just a matter of opinion, and ... he got pulled off after half an hour .... in front of Roy Hodgson, so he shouldn't have been in the squad. ...' ... Sky Sports pundits Merson (centre) criticised Townsend's call-up to the England squad last week ....	They're not gonna take it anymore. Really. Twisted Sister says that its 2016 tour will be its last, according to a press release. ... The band will also perform two shows in Pero's honor: one at Las Vegas Hard Rock Hotel and Casino, the other at the Starland Ballroom in Sayreville, New Jersey.
Summary	Paul Merson is not happy with Andros Townsend's call-up to the England squad last week	The band will perform two shows.
Selected Answer	Andros Townsend's call-up	the band
Question Generation	BART-QA2D What is Paul Mer-son not happy with to the England squad last week?	MixQG-large What is Paul Merson not happy with?	BART-QA2D Question Who will perform two shows?
QA Output	Townsend's call-up	he shouldn't have been in the squad	Unanswerable (Twisted Sister)
Answer Overlap	1.00	0.30	0.00 (0.80)
Table 4: Example source documents and summaries along with QA-based metric component outputs. Left: This example illustrates that the fluency of the QG model does not necessarily improve downstream factual consistency evaluation performance; the less fluent, more extractive BART-QA2D question is more-easily answerable by the QA model. Not shown, the entailment-based SCConv metric incorrectly labels this entity-centric example, likely due the introduction of novel unigrams. Right: The QA model incorrectly labels this question as unanswerable, perhaps due to the generality of the question or due to noise in the input document. The QA output and our learned overlap score if forced to extract an answer are in parenthesis. SCConv correctly labels this highly extractive example.
nent across model sizes or between extractive and abstractive QA models, implying that QA ability is not the bottleneck of our task. In this setting, we keep IsAnsweredInput from Electra-large constant, as not all QA models are trained with unanswerable questions; thus the only differences are in the answers to questions marked as answerable.
Effect of Answer Overlap Metric We observe a large difference between EM and other overlap metrics. We also see a notable gap between LERC (orig) and LERC (RoBERTa) along with a further
slight improvement with LERC (QuIP), showing the effect of the underlying model of the learned metric on factual consistency performance.
Effect of Question Filtering and Answerability Not filtering questions according to the QA model's ability to answer them conditioned upon the summary decreases performance. Furthermore, not applying the Answerability Penalty, and using the answer overlap score for the most probable answers for all questions, even those judged unanswerable by the QA model, also decreases perfor
Model Type	Model Name	XSF	SummEval	FRANK-CNNDM	FRANK-XSum	QAGs-CNNDM	QAGs-XSum
Misc	BARTScore	0.25	0.37	0.58	0.15	0.73	0.17
BLANC	0.03	0.20	0.33	0.07	0.33	0.02
Entailment	FactCC	0.04	0.37	0.38	0.06	0.40	0.30
BertScore-FFCI	0.45	0.27	0.36	0.16	0.53	0.21
DAE	0.02	0.45	0.50	0.22	0.63	-0.20
ANLI	0.16	0.43	0.53	0.18	0.65	0.39
MNLI	0.18	0.44	0.52	0.18	0.66	0.35
DocNLI	0.01	0.41	0.12	0.26	0.16	-0.34
SCZeroShot	0.06	0.50	0.55	0.27	0.57	0.44
QA	QuestEval	0.45	0.41	0.52	0.24	0.51	0.23
QAFACTEVAL	0.29	0.61	0.66	0.32	0.68	0.44
Learned	SCConv (synthetic)	0.12	0.50	0.59	0.30	0.03	0.06
QAFACTEVAL-NLI(synthetic)	0.19	0.61	0.66	0.25	0.65	0.48
Table 5: Instance-level Pearson correlation coefficients across factual consistency evaluation datasets. Metrics are divided into entailment-based, QA-based, and learned metrics that are fine-tuned on synthetic or supervised data. The two highest-correlated metrics for each dataset are shown in bold.
mance. While the answer overlap metric should capture unanswerable questions for information not found in the input (extrinsic error), the selected answer may appear in both the summary and source but in different contexts (intrinsic error). The QA model may return this as the most probable answer and be scored as correct by the answer overlap component despite a factual inconsistency. This finding demonstrates the importance of determining question answerability, a point also emphasized in Deutsch et al. (2020) for QA-based metrics of relevance. Removing both of these components results in a drastic performance decrease.
We present the results on the test set of SummaC in Table 3. QAFACTEVAL shows a substantial improvement over the previous state-of-the-art QA metric for factual consistency, QuestEval. Furthermore, it outperforms all other entailment-based metrics. QAFACTEVAL-NLI shows slight improvements on the synthetic data. Notable improvements in this synthetic setting can be observed on the FactCC dataset, likely as the synthetic FactCC data the model is trained on was designed to mirror the errors captured in annotations. This performance boost on FactCC motivated our use of supervised data for fine-tuning our learned metric. Supervised fine-tuning on validation data helps in most cases and QAFACTEVAL-NLI (supervised) improves on the overall benchmark by a statistically significant margin, using bootstrap resampling (Efron, 1982) with Bonferroni correction (Bonferroni, 1935) to obtain  $99\%$  confidence intervals (see Appendix for details). The performance drop on FactCC could be due to the proximity of the synthetic data to the labeled data and
the data size difference. BertScore-FFCI performs best on XSF perhaps due to the closeness between BertScore's token-level metric and XSF's word-level annotations, and DocNLI's Polytope performance may also be from training data similarity.
We find that QAFACTEVAL and SCConv do offer complementary signals that can be learned from supervised data. Individually fine-tuning the learned SCConv or a learned variation of QAFACTEVAL on supervised data did not improve results over the non-supervised metrics; this result suggests the necessity of combining the two for further improvements. Training on the validation sets combined, rather than on each individual dataset separately, did not give an improvement, likely due to the learnable combination of NLI and QAFACTEVAL being dataset dependent.
5.3 Correlation Analysis
We provide instance-level Pearson correlation between aggregated human judgments and metric scores for each model to compare to previous work in factual consistency that reports correlation analysis. Results are shown in Table 5. We split FRANK into CNN/DailyMail and XSum subsets for finer-grained analysis, as substantial differences have been noted in correlation performance across the two datasets (Durmus et al., 2020). We exclude Polytope, FactCC, and CGS here as prior work has only studied these datasets for binary classification.
We find that QAFACTEval performs well across most datasets. As in the classification results, BertScore-FFCI's performs well on XSF, and we note that QuestEval's answerability classifier correlates more so with these fine-grained annotations than on other datasets. QAFACTEval-NLI performs well on most datasets except XSF. Fine
tuning on FactCC synthetic data for binary classification likely does not capture the aggregated, word-level factuality scores of XSF. We leave a study of fine-tuning this model on supervised data with a regression loss for future work.
In this work, we demonstrated that QA-based metrics, when its components are properly optimized, outperform entailment-based metrics on a comprehensive factual consistency evaluation benchmark. We identify question generation and answerability detection as key components for improving QA-based metrics in future work. Furthermore, we show that entailment and QA-based metrics offer complementary signals through a combined metric that achieves state-of-the-art performance on this benchmark. We believe that our work lays the foundation for future work in QA-based metrics for factual consistency by offering a fairer comparison to other metrics across datasets and settings.
Dataset Biases The underlying models of the metrics presented in this work are trained on documents in English and thus mainly represent the culture of the English-speaking populace. Political or gender biases may also exist in the datasets, and models, and subsequently the metrics, trained on these datasets may propagate these biases. We did not stress test these metrics for such biases and request that the users of these metrics be aware of these potential issues in applying them.
Misuse Potential and Failure Mode When properly used, the metrics described in this paper can be a useful tool for detecting summarization model errors. However, the current metrics fail to detect all factual inconsistencies, which must be remembered when applying these metrics as a filter for downstream applications. Factual inconsistencies in summaries could contribute to misinformation on the internet.
Environmental Cost The experiments described in the paper primarily make use of A100 GPUs. Most of the metrics have already been trained, in which case we simply ran inference using the existing models. We typically used a single GPU per experiment. Training learned answer overlap components can take a couple of hours, while experiments for learned metrics on SummaC take
less than 10 minutes. These are the base models used in these experiments, with the number of parameters, in millions, in parentheses: BERT-base (110), BART-large (400), Electra-base (110), Electra-large (335), RoBERTa-large (355), T5-base (220), T5-large (770). Future work may analyze the effect of using distilled backbone models on factual consistency evaluation.
Mario Barrantes, Benedikt Herudek, and Richard Wang. 2020. Adversarial nli for factual correctness in text summarisation models. ArXiv preprint, abs/2005.11739.
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. 2020. Reevaluating evaluation in text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9347-9359, Online. Association for Computational Linguistics.
Carlo E Bonferroni. 1935. Il calcolo delle assicurazioni su gruppi di teste. Studi in onore del professore salvatore ortu carboni, pages 13-60.
Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2020. MOCHA: A dataset for training and evaluating generative reading comprehension metrics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6521-6532, Online. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv preprint, abs/1809.02922.
Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2020. Towards question-answering as an automatic metric for evaluating the content quality of a summary.
Daniel Deutsch and Dan Roth. 2020. SacreROUGE: An open-source library for using and developing summarization evaluation metrics. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS), pages 120-125, Online. Association for Computational Linguistics.
Daniel Deutsch and Dan Roth. 2021. Understanding the extent to which content quality metrics measure the information quality of summaries. In Proceedings of
the 25th Conference on Computational Natural Language Learning, pages 300-309, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Bradley Efron. 1982. The Jackknife, the bootstrap and other resampling plans. CBMS-NSF regional conference series in applied mathematics. SIAM, Philadelphia, PA. Lectures given at Bowling Green State Univ., June 1980.
Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3938-3948, Minneapolis, Minnesota. Association for Computational Linguistics.
Alexander R. Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir R. Radev. 2021. Summeval: Re-evaluating summarization evaluation. Trans. Assoc. Comput. Linguistics, 9:391-409.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
Dan Friedman, Ben Dodge, and Danqi Chen. 2021. Single-dataset experts for multi-dataset qa. In Empirical Methods in Natural Language Processing (EMNLP).
Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021. GO FIGURE: A meta evaluation of factuality in summarization. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 478-487, Online. Association for Computational Linguistics.
Tanya Goyal and Greg Durrett. 2020. Evaluating factuality in generation with dependency-level entailment. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3592-3603, Online. Association for Computational Linguistics.
Karl Moritz Hermann, Tomás Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 1693-1701.
Dandan Huang, Leyang Cui, Sen Yang, Guangsheng Bao, Kun Wang, Jun Xie, and Yue Zhang. 2020. What have we achieved on text summarization? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 446-469, Online. Association for Computational Linguistics.
Robin Jia, Mike Lewis, and Luke Zettlemoyer. 2021. Question answering infused pre-training of general-purpose contextualized representations.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Fajri Koto, Timothy Baldwin, and Jey Han Lau. 2021. Ffci: A framework for interpretable automatic evaluation of summarization.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2020. FFCI: A framework for interpretable automatic evaluation of summarization. ArXiv preprint, abs/2011.13662.
Wojciech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China. Association for Computational Linguistics.
Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. 2020. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
pages 9332-9346, Online. Association for Computational Linguistics.
Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2021. Summac: Re-visiting nlibased models for inconsistency detection in summarization.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, and Soumith Chintala. 2020. Pytorch distributed: Experiences on accelerating data parallel training. Proc. VLDB Endow., 13(12):3005-3018.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3730-3740, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Klaus-Michael Lux, Maya Sappelli, and Martha Larson. 2020. Truth or error? towards systematic analysis of factual errors in abstractive summaries. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 1-10, Online. Association for Computational Linguistics.
Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906-1919, Online. Association for Computational Linguistics.
Lidiya Murakhovs'ka, Chien-Sheng Wu, Tong Niu, Wenhao Liu, and Caiming Xiong. 2021. Mixqg: Neural question generation with mixed answer types.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.