SlowGuess/NAACL_2022_OCR huggingface dataset:
SUMMAC (Summary Consistency; Laban et al., 2021) is focused on evaluating factual consistency in summarization. They use NLI for detecting inconsistencies by splitting the document and summary into sentences and computing the entailment probabilities on all document/summary sentence pairs, where the premise is a document sentence and the hypothesis is a summary sentence. They aggregate the NLI scores for all pairs by either taking the maximum score per summary sentence and averaging  $(\mathrm{SC}_{\mathrm{ZS}})$  or by training a convolutional neural network to aggregate the scores  $(\mathrm{SC}_{\mathrm{Conv}})$ . We use the publicly available implementation. $^{12}$
3.4 QG-QA Based Metrics
Durmus et al. (2020) and Wang et al. (2020) proposed to use Question Generation (QG) and Question Answering (QA) models to automatically evaluate factual consistency in abstractive summarization, showing promising results. Honovich et al. (2021) employed a similar approach for evaluating knowledge-grounded dialogue generation.
The steps of the QG-QA approach are as follows: (1) Questions are automatically generated for spans in the generated text, such that the answer to a question is its respective input span. (2) The generated questions are answered using a QA model on the grounding text, resulting in an answer span or a "no-answer" output. (3) For each question, the two answer spans from the grounding and the generated text are compared to get a score. (4) The scores for
all questions are aggregated into a final score.
$\mathbf{Q}^2$  (Honovich et al., 2021) is a QG-QA method that employs an NLI model to compare the two answers for each question, where the grounding text answer is the premise and the generated text answer is the hypothesis. We report results for a re-implementation of  $Q^2$  using T5-11B as the backbone for the QG, QA and NLI models. While Honovich et al. (2021) validate each generated question by answering it using a QA model and comparing to the original extracted answer candidate using exact match, we relax this and instead use F1 token-overlap with a predefined threshold. $^{13}$
QuestEval (Scialom et al., 2021) is a QG-QA method that measures both factual consistency and relevance (by reversing the roles of the generated and grounding texts). The authors trained a model that weights each generated question according to the relevance of its answer to appear in the generated text. Their results showed high correlation with human judgments in comparison to prior work on the SummEval benchmark (Fabbri et al., 2021a). We use the publicly available version. $^{14}$
We report the ROC AUC $^{15}$  of various metrics on the standardized datasets in Table 3. The ROC curves can be found in Figure 2 in the appendix.  $\mathrm{SC}_{\mathrm{ZS}}$  was trained on VitaminC which includes examples from FEVER, so we exclude those datasets from the average AUC calculation for a more fair comparison. As all metrics operate in a "zero-shot" manner on all datasets (except for  $\mathrm{SC}_{\mathrm{ZS}}$  on VitaminC and FEVER) and no threshold tuning is required, we report results on the development sets. $^{16}$
The results show that the NLI-based models (ANLI,  $\mathrm{SC}_{\mathrm{ZS}}^{17}$ ) outperformed the other approaches on 6 datasets, with average AUC of 81.5 and 81.4 for ANLI and  $\mathrm{SC}_{\mathrm{ZS}}$ , respectively.  $Q^2$  outperformed the other approaches on 4 datasets, with an average AUC of 80.7. The next best method, BARTScore,
13More implementation details are available in Section B in the appendix.
$^{14}$ https://github.com/ThomasScialom/QuestEval
15 Multiplied by 100 for better readability.
$^{16}$ AUC for the test sets and accuracy for the dev and test sets are provided in Tables 10, 11 and 12 in the appendix.
<sup>17</sup>We report results for  $\mathrm{SC}_{\mathrm{ZS}}$  as it performed better in our experiments. Results for  $\mathrm{SC}_{\mathrm{Conv}}$  are available in Table 10 in the appendix.
10One example where the definition of NLI and factual consistency differs is in dialog, where subjective or opinionated statements of the dialog agent are not evaluated as factual statements, while NLI models consider all the text in the premise. 11More implementation details on the NLI model are available in Section B in the appendix. 12https://github.com/tingofurro/summac
Ensemble	Q2metric	ANLI	SCZS	F1	BLEURT	QuestEval	FactCC	BARTscore	BERTscore
FRANK	91.2	87.8	89.4	89.1	76.1	82.8	84.0	76.4	86.1	84.3
SummEval	82.9	78.8	80.5	81.7	61.4	66.7	70.1	75.9	73.5	77.2
MNBM	76.6	68.7	77.9**	71.3	46.2	64.5	65.3	59.4	60.9	62.8
QAGS-C	87.7	83.5	82.1	80.9	63.8	71.6	64.2	76.4	80.9	69.1
QAGS-X	84.8	70.9	83.8	78.1	51.1	57.2	56.3	64.9	53.8	49.5
BEGIN	86.2	79.7	82.6	82.0	86.4	86.4	84.1	64.4	86.3	87.9
Q2dataset	82.8	80.9*	72.7	77.4	65.9	72.4	72.2	63.7	64.9	70.0
DialFact	90.4	86.1**	77.7	84.1	72.3	73.1	77.3	55.3	65.6	64.2
PAWS	91.2	89.7**	86.4	88.2	51.1	68.3	69.2	64.0	77.5	77.5
FEVER	94.7	88.4	93.2**	93.2	51.8	59.5	72.6	61.9	64.1	63.3
VitaminC	96.1	81.4	88.3**	97.9	61.4	61.8	66.5	56.3	63.2	62.5
Avg. w/o VitC, FEVER	86.0	80.7	81.5	81.4	63.8	71.4	71.4	66.7	72.2	71.4
Table 3: ROC AUC results for the different metrics on the TRUE development set. We exclude VitaminC and FEVER from the average calculation as  $\mathrm{SC}_{\mathrm{ZS}}$  was trained on VitaminC that includes examples from FEVER. The highest score in each row (excluding the Ensemble) is in bold and the aforementioned SC results are in strikingthrough. Statistically significant results are indicated using * and ** for  $p < 0.05$  and  $p < 0.01$  respectively.
had lower average AUC of 72.2. All other approaches scored 72 or lower on average across all datasets (excluding FEVER and VitaminC). As expected, the simple token-matching based metrics did not perform well, and for completeness, we report their performance in Table 9 in the appendix. We keep the F1 score in Table 3 for convenient comparison to the other metrics.
One outlier is BEGIN, which is the only dataset where simple metrics like F1 token overlap achieved scores higher than 80. We measured the average overlap between the grounding and target texts per dataset, and found that BEGIN exhibits a high difference between grounded and ungrounded texts in comparison to other datasets (Table 8 in appendix A), which explains this.
We follow Laban et al. (2021) and perform significance testing through bootstrap resampling (Efron, 1982), comparing the best method to the second-best method on each dataset. We perform interval comparison at  $p = 0.05$  and  $p = 0.01$  and find significantly best results on 6 datasets, 3 achieved by  $Q^2$  and 3 by the ANLI-based model.
Given that no single method outperformed the rest on all datasets, we hypothesize that the NLI and QG-QA based metrics are complementary. We test this by averaging the  $Q^2$ , ANLI and  $\mathrm{SC}_{ZS}$  scores per example[18] (Ensemble in Table 3). Indeed, averaging the three methods yields better results on most datasets and on average, with an increase of 4.5 in ROC AUC from the best single-metric result.
Our results show that a single metric can do well across all tasks and datasets, with all 3 best metrics scoring higher than 80 on average on the 11 datasets. This corroborates our hypothesis that
evaluating factual consistency can be unified, and we hope such unified perspective will be adopted in future work to accelerate progress on the subject.
Input Length. As QA and NLI models may struggle with long inputs (Kočiský et al., 2018; Pang et al., 2021; Yin et al., 2021; Shaham et al., 2022), metrics based on them may fail when handling long text. To study the effect of input length on the metrics performance, we unify all datasets<sup>19</sup> and split examples into 6 bins according to the grounding length.<sup>20</sup> We focus on the grounding as the target texts are usually short (see Table 7 in Appendix A). We measure AUC of the best 3 metrics according to their overall score for each length bin, sampling 1,000 examples per bin.
The results are shown in Figure 1. We find that there is a consistent degradation for texts longer than 200 tokens for all metrics, including  $\mathrm{SC_{ZS}}$  which is designed to better handle long text. We find it surprising that the ANLI-based model and  $Q^2$  still do relatively well on the longest bin (with AUC  $>0.825$ ) as they perform end-to-end QA and NLI on text with more than 500 tokens.
Model Size. Model-based metrics are expected to benefit from increasing the model size. To quantify this we study the effect of using smaller models for the ANLI, BLEURT and BERTScore metrics. We compare the average ROC AUC of larger and
19Excluding VitaminC as it is much larger than other datasets and might therefore distort results. Statistics regarding the grounding and target text lengths per dataset is in Appendix A.
20We measure length in tokens (before subword splitting) as different metrics use different subword tokenizations.
Pairwise ensembles are reported in the appendix, Table 9.
Figure 1: ROC AUC when splitting TRUE's data according to the grounding length.
smaller model variants for each metric. The ablation results are in Table 4. We find an advantage of 4.7, 3.7 and 1.3 average ROC AUC for the larger ANLI, BLEURT and BERTScore variants respectively, showing that larger models indeed allow for better factual consistency evaluation metrics, and hinting at potential improvements from using even larger models.
Model	Avg. ROC AUC
ANLI-T5-11B	81.5 (+4.7)
ANLI-T5-Large	76.8
BLEURT-20	71.4 (+3.7)
BLEURT-20-D6	67.7
BERTScore P - deberta-xl-mnli	71.4 (+1.3)
BERTScore P - roberta-large	70.1
Table 4: Ablation study comparing the average ROC AUC results for models with different sizes. "BERTScore P" stands for BERTScore Precision.
Qualitative Analysis. We conduct manual error analysis to point at weaknesses of the different metrics and present challenges posed by the task. We analyze 80 examples that were misclassified by all three best metrics, as well as 100 examples that were correctly classified by one or two of the three.
Out of the analyzed examples, many seem to have a wrong label. This is especially true for cases in which all best metrics failed, with annotation errors in 35/80 cases. For the cases where one or two metrics failed, we found annotation errors in 27/100 cases. To verify that the high annotation error rate is indeed a result of inspecting the "hardest" examples and not a general issue in the datasets we used, we uniformly sample 100 additional examples, finding that only 10 had annotation errors. We therefore stress that the high misannotation rate indeed characterizes "hard" examples only, and is not a general property of the datasets we used. This is inline with the findings of Freitag et al. (2021), who showed that in some cases, metrics may be
"better" than non-expert annotators. These findings demonstrate the potential of automatic methods in "cleaning" training data by filtering factually inconsistent examples.
Despite showing impressive results, the best-performing metrics fail to detect subtle inconsistencies, as presented in Table 5. This was the case for 21/180 analyzed examples. Metrics that aggregate scores across parts of a target text, such as  $Q^2$  or  $\mathrm{SC}_{\mathrm{ZS}}$ , might assign a high score for texts in which all but a small part is consistent. End-to-end NLI should predict "contradiction" even when only a small part of the text contradicts the grounding, but it may fail to do so. Applying a strict approach in the aggregation step, like taking the minimum instead of the average, could potentially remedy this – with the price of having more false-negatives. Other errors are caused by domain-specific challenges, such as handling personal statements in dialogues. As shown in Table 5, such statements may be falsely classified as ungrounded. This was the case for 10/62 analyzed dialogue responses. A possible way to alleviate this would be to automatically exclude non-factual parts from the evaluation.
Ensemble Analysis. As shown in §4, a simple averaging ensemble using the three best metrics achieves strong results, outperforming individual metrics on most datasets. To understand this further, we analyze cases in which at least one of the best three metrics failed, while the ensemble succeeded. Overall, there were 25,761 such cases, where in  $85.2\%$  of these cases, two out of the three metrics succeeded, and only one failed. In  $14.6\%$  of these cases, one metric succeeded while the other two failed, and only in  $0.2\%$  of the cases, the ensemble succeeded while all metrics failed. These cases are a result of the different threshold used for the ensemble model vs. the thresholds for the individual metrics. We sample 100 of these examples and manually analyze them. Out of the sampled examples,  $47\%$  were misclassified by one metric only, where this metric assigned a borderline score - i.e., close to the decision threshold.  $36\%$  of these examples were misclassified by one metric only, and also with a non-borderline score - i.e., the metric was far from a correct prediction. Other cases include two, or even three, erroneous metrics.
Adding to the related work mentioned throughout the paper, works on unified evaluation of text
Grounding	Generated Text	Explanation
The word "philately" is the English version of the French word "philatélie", coined by Georges Herpin in 1864.	The word philately is actually a french word coined by george herpin.	The word philately is an English word based on a French word, but not French. All best metrics misclassified this.
French police have interviewed presidential candidate francois fillon and his wife penelope over claims she was paid for fake work. They provided information that would help find the "truth", mr fillon said...	French presidential candidate francois filon has said he and his wife penelope have been questioned by police over claims she worked illegally.	Most details are correct and the hallucination is subtle. In the case of Q2, most of the generated questions have the same answer based on the grounding and the generated text, therefore the overall score was high.
Stamp collecting is generally accepted as one of the areas that make up the wider subject of philately, which is the study of stamps.	I've never heard of stamps, but I do know that the word "philately" refers to the study of stamps.	The personal statement "I've never heard of stamps" is not factual and should not be evaluated.
Evidence suggests that cognitive behavioral therapy and a gradual increase in activity suited to individual capacity can be beneficial in some cases.	It has been suggested that cognitive behavioral therapy and gradual increase in exercise could help in some cases so I'm going to try that for now.	Similar to the previous examples - SummaC and ANLI falsely marked the text as inconsistent, probably due to the personal statement.
Table 5: Examples for the error analysis. The first two rows show cases of challenging inconsistencies, while the last two show dialogue responses containing non-factual personal statements.
generation across tasks include GEM (Gehrmann et al., 2021), where the focus is on evaluating system outputs and not the factual consistency evaluation methods as in TRUE. BEAMetrics (Scialom and Hill, 2021) proposes meta-evaluation protocols across tasks, but does not focus on factual consistency. When discussing consistency ("correctness") they measure correlations, which are not sufficient as mentioned in Section 2.3. Chen et al. (2021) present an adversarial meta-evaluation for factual consistency evaluators, focused on summarization. Other works on meta-evaluation of factual consistency across datasets include GO-FIGURE (Gabriel et al., 2021) FRANK (Pagnoni et al., 2021) SummaC (Laban et al., 2021) and QAFactEval (Fabbri et al., 2021b), however they all focus solely on summarization. Yeh et al. (2021) conduct a thorough assessment of dialog metrics, however not specifically around factual consistency. To the best of our knowledge, our work is the first to generalize the discussion on evaluating factual consistency across tasks and datasets, and the first to show that large-scale QG-QA and NLI are strong and highly complementary – setting better baselines and meta-evaluation methodology for future work.
We discuss the main takeaways of the TRUE study, pointing at actionable insights for future work. First, as QG-QA and NLI-based methods show better performance than other approaches, especially when combined together, we recommend model developers to use those methods for evaluation when factual consistency is a priority. As for metric developers, we recommend using those methods and the datasets in TRUE when evaluating new metrics.
We also suggest reporting ROC AUC rather than correlations, as it is more interpretable and actionable. Our proposed binary annotation scheme allows to easily test new metrics across tasks and datasets, which would be useful for future work.
Finally, we encourage data curators to use the binary annotation scheme, which is inline with the recommendations of Rashkin et al. (2021a). Having said that, we do not rule out more detailed labeling schemes – but rather ask to provide a protocol for converting such labels into the more general binary format. Future work may also address the challenges of long inputs and personal statements in dialogue, which we point out in our analysis.
We presented TRUE, a survey and assessment of automatic factual consistency evaluation methods. We standardized various datasets from diverse tasks into a unified labeling scheme to perform a thorough comparison of automatic evaluation methods, showing that large scale NLI and QG-QA based approaches perform well across multiple tasks and datasets. We further show that these methods are highly complementary – hinting at additional headroom for improvement while pointing on current limitations. We hope our results and methodology will encourage a more unified perspective in future work to foster progress towards more factually-consistent NLP applications.
We thank Dipanjan Das, Sebastian Gehrmann and Joshua Maynez for their valuable comments and suggestions for this work.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Gillian Brown and George Yule. 1983. Discourse Analysis. Cambridge University Press.
Yiran Chen, Pengfei Liu, and Xipeng Qiu. 2021. Are factuality checkers reliable? adversarial metaevaluation of factuality in summarization. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2082-2095, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177-190, Berlin, Heidelberg. Springer Berlin Heidelberg.
Mingkai Deng, Bowen Tan, Zhengzhong Liu, Eric Xing, and Zhiting Hu. 2021. Compression, transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Emily Denton, Mark Diaz, Ian Kivlichan, Vinodkumar Prabhakaran, and Rachel Rosen. 2021. Whose ground truth? accounting for individual and collective identities underlying dataset annotation. arXiv preprint arXiv:2112.04554.
Daniel Deutsch, Tania Bedrax-Weiss, and Dan Roth. 2021. Towards question-answering as an automatic metric for evaluating the content quality of a summary. Transactions of the Association for Computational Linguistics, 9:774-789.
Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. Re-examining system-level correlations of automatic summarization evaluation metrics. arXiv preprint arXiv:2204.10216.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of Wikipedia: Knowledge-powered conversational agents. In Proceedings of the International Conference on Learning Representations (ICLR).
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Nouha Dziri, Hannah Rashkin, Tal Linzen, and David Reitter. 2021. Evaluating groundedness in dialogue systems: The begin benchmark.
Bradley Efron. 1982. The jackknife, the bootstrap and other resampling plans. SIAM.
Alexander R Fabbri, Wojciech Krysciński, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2020. Summeval: Re-evaluating summarization evaluation. arXiv preprint arXiv:2007.12626.
Alexander R Fabbri, Wojciech Kryscinski, Bryan McCann, Caiming Xiong, Richard Socher, and Dragomir Radev. 2021a. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391-409.
Alexander R. Fabbri, Chien-Sheng Wu, Wenhao Liu, and Caiming Xiong. 2021b. Qafacteval: Improved qa-based factual consistency evaluation for summarization.
Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214-2220, Florence, Italy. Association for Computational Linguistics.
C. Fillmore. 1976. Frame semantics and the nature of language *. Annals of the New York Academy of Sciences, 280.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. arXiv preprint arXiv:2104.14478.
Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2021. GO FIGURE: A meta evaluation of factuality in summarization. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 478-487, Online. Association for Computational Linguistics.