SlowGuess/NAACL_2022_OCR huggingface dataset:
Alexander R. Fabbri Chien-Sheng Wu
Wenhao Liu Caiming Xiong
{afabbri, wu.jason, wenhao.liu, cxiong} @salesforce.com
Factual consistency is an essential quality of text summarization models in practical settings. Existing work in evaluating this dimension can be broadly categorized into two lines of research, entailment-based and question answering (QA)-based metrics, and different experimental setups often lead to contrasting conclusions as to which paradigm performs the best. In this work, we conduct an extensive comparison of entailment and QA-based metrics, demonstrating that carefully choosing the components of a QA-based metric, especially question generation and answerability classification, is critical to performance. Building on those insights, we propose an optimized metric, which we call QAFACTEVAL, that leads to a  $14\%$  average improvement over previous QA-based metrics on the SummaC factual consistency benchmark, and also outperforms the best-performing entailment-based metric. Moreover, we find that QA-based and entailment-based metrics can offer complementary signals and be combined into a single metric for a further performance boost.
Text summarization aims to compress long document(s) into a short and fluent form that preserves salient information. The field has benefited from the application of pretrained methods (Liu and Lapata, 2019; Lewis et al., 2020; Zhang et al., 2020a). However, state-of-the-art models are not always factually consistent with the source documents they are conditioned on (Maynez et al., 2020; Fabbri et al., 2021). Thus, determining the factual consistency of a summary remains an essential task.
Recent metrics for summarization factual consistency can be broadly split into two categories: 1) Entailment-based metrics that determine whether the content in the summary is entailed by the input document (Kryscinski et al., 2020; Koto et al.,
Document
The Knicks beat the Rockets. The fans were excited.
Summary
The Knicks beat the Bucks.
Entailment Matrix
[Contra, Neutral, Support]	Selected Answer
the Bucks
Generated Question
Who did the Knicks beat?
QA Output
the Rockets
Max Support Score
0.08	Answer Overlap Score
0.20
Table 1: Toy example of a factual inconsistency between a summary and a source document. Left: The entailment-based metric computes the level of contradiction, neutrality, and support between the summary and each source document sentence. The final factual consistency metric is calculated as the maximum support score over all source sentences. Right: The QA-based metric first selects a noun-phrase answer from the summary. A QG model then generates an associated question that a QA model answers based on the source document. The answer overlap score of the QA-based metric measures the semantic overlap between the QA model output and the selected answer as the final metric score.
2020) and 2) QA-based metrics that compute a factual consistency score based on a QA model's ability to answer, using the input document, questions generated from the summary (Wang et al., 2020a; Durmus et al., 2020). We provide an illustrative example in Table 1 in which both metric types correctly identify the factual inconsistency and output a low score.
Quantitative comparisons among entailment-based and QA-based metrics, however, often differ in their choices of baseline model and input granularity, evaluating on single datasets and drawing differing conclusions as to the best paradigm. For example, some work reports entailment-based metrics as performing best (Koto et al., 2020; Maynez et al., 2020), while other work argues for QA metrics (Durmus et al., 2020; Wang et al., 2020b; Scialom et al., 2021). Recently, Laban et al. (2021) pro
Human Language Technologies, pages 2587 - 2601
posed a benchmark called SummaC to compare metrics across six factual consistency datasets for the task of binary factual consistency classification, whether a summary is entirely factually consistent or not. This work unifies prior work on entailment-based metrics by studying the effect of input granularity, pretrained entailment model, and other hyperparameter choices on downstream evaluation performance. However, it does not study the components of QA-based metrics, which are more interpretable by their inherent decomposability.
To unify work in QA-based factual consistency evaluation, we do an extensive hyperparameter analysis of current metrics. We break down these metrics into four constituent components: 1) the selection of answers to ask questions about, 2) question generation (QG) conditioned upon these answers, 3) question answering (QA) based on the source document, and 4) answer overlap evaluation between QA model output and selected answers. We study the effect of each of these components on metric performance. Based on our insights, we propose an optimized metric, which we call QAFACTEVAL, that outperforms the entailment-based metrics of Laban et al. (2021).
Our contributions are the following: 1) We analyze all components of the QA-based metric pipeline, and our proposed solution improves performance over prior QA-based metrics by over  $14\%$  on a factual consistency benchmark consisting of 6 individual datasets, achieving state-of-the-art results. 2) We show that QA-based metrics and NLI-based metrics offer complementary signals and combine them into a new metric via a simple learned network, further improving performance. 3) We report results for 10 additional metrics across classification and correlation analysis, providing the most comprehensive benchmark results for factual consistency metrics and highlighting areas for future work in QA-based metrics<sup>1</sup>.
Evaluating Factual Consistency Within entailment-based factual consistency evaluation, Falke et al. (2019) propose the task of ranking summary pairs for factual consistency based on entailment models, while Kryscinski et al. (2020) explore factual consistency classification jointly with source support or contradiction span extrac
tion. Other work on entailment-based metrics has examined input granularity (Goyal and Durrett, 2020), trained on adversarial datasets (Barrantes et al., 2020), and explored entailment-based models as the backbone of others metrics such as BERTScore (Zhang et al., 2020b) as in Koto et al. (2021). Metric comparisons, however, were often conducted on isolated datasets. Laban et al. (2021) unify work in entailment-based metrics for factual consistency, showing the effect of granularity, base models, and other hyperparameter choices. This work also proposes a learned metric built on top of the output of an entailment model, with parameters fine-tuned on synthetic data. While this work fills a gap in the use of entailment-based metrics for factual consistency, our work analogously unifies QA-based metrics for factual consistency and proposes to combine entailment and QA-based metrics in a single learned metric.
QA-based evaluation metrics have received attention for summary quality dimensions beyond factual consistency (Eyal et al., 2019; Scialom et al., 2019; Deutsch et al., 2020). Recent work has shown that QA-based metrics better measure the overlap of information units for determining summary relevance over embedding-based metrics (Deutsch and Roth, 2021), further driving our study of QA-based metrics for factual consistency. While several QA-based metrics with similar structures have been applied for factual consistency, (Durmus et al., 2020; Wang et al., 2020b; Scialom et al., 2021), they differ in their underlying answer selection, question generation, question answering, and answer overlap components, reporting different performances. We perform a comprehensive evaluation of QA-based metric components and propose improved model components for the task of answer overlap and question filtering.
Summarization Benchmarking A recent line of work aims to take stock of the current state of summarization models and progress, both within factual consistency and across summarization more broadly. Kryscinski et al. (2019) note biases and failure modes of abstractive summarization models, while other work analyzes and collects annotations over the output of recent summarization models across multiple dimensions, including factual consistency (Fabbri et al., 2021; Bhandari et al., 2020; Huang et al., 2020). Lux et al. (2020) propose a typology of errors found in summarization models, while Gabriel et al. (2021) propose a framework for
$^{1}$ Code and metric outputs will be made publicly available: https://github.com/salesforce/QAFactEval
meta-evaluation of factual consistency metrics. Laban et al. (2021) propose to combine recent work in factual consistency evaluation for summarization through a single benchmark. Our work directly makes use of this benchmark while emphasizing QA-based metrics. We also include correlation analysis for a more comprehensive understanding of current factual consistency metrics.
3 Evaluation Metrics
In this section, we introduce the factual consistency metrics studied, which we divide into entailment metrics, QA-based metrics, and learned metrics.
3.1 Entailment-based Metrics
We include the following entailment-based metrics due to further understand differences in granularity and base entailment models. The metrics below produce a score for each summary sentence that is then averaged to compute the final metric score.
MNLI applies a RoBERTa large (Liu et al., 2019) model trained on MNLI (Williams et al., 2018). The score of a summary sentence is the maximum entailment score over all input sentences.
ANLI Barrantes et al. (2020) uses the same method as the MNLI metric with a model trained on the ANLI (Nie et al., 2020) dataset consisting of adversarial datapoints.
SCZeroShot Laban et al. (2021) works analogously to the above metrics with a base model trained on both MNLI and Vitamin-C data (Schuster et al., 2021), consisting of closely-related contrastive entailment examples.
BertScore-FFCI Koto et al. (2021) applies BertScore (Zhang et al., 2020b) with a backbone RoBERTa-MNLI model, averaging the three highest BertScore F1 scores over the input sentences.
DAE Goyal and Durrett (2020) computes entailment scores between a source document and summary dependency arcs, applying an entailment model trained on synthetic data.
FactCC Kryscinski et al. (2020) is a RoBERTa-base model trained on FactCC synthetic data to compute a document-level score, and thus the scores need not be aggregated over input sentences.
DocNLI Yin et al. (2021) train a document-level entailment model, similar to the FactCC metric.
3.2 QA Metric Components
We now describe the components that constitute the QA-based pipeline for factual consistency. We refer to our metric, consisting of the best combination of the below components, as QAFACTVAL.
Answer Selection QA-based metrics compare information units between the summary and source, so it is thus necessary to first extract such units, or answers, from the given summary. We follow the protocols from Deutsch et al. (2020) and compare extracting the following answer types: named entities (NER), noun phrase chunks (NPChunks), maximally sized noun phrases (Max NP), whereby the dependency subtrees of nouns reached by traversing a given sentence's dependency parse from the root are chosen as answers, and All, which combines answers from the above three techniques.
Question Generation Having selected answers, questions are generated conditioned upon these answers using the summary as context. Typically, this is an encoder-decoder model which inputs the answer and context separated by a special token. On the modeling side, we examine BART (Lewis et al., 2020) and T5 (Raffel et al., 2019) as the underlying generators. On the data side, we experiment with models trained for question generation on SQuAD (Rajpurkar et al., 2016), a standard QA dataset consisting of questions on Wikipedia articles, and on QA2D (Demszky et al., 2018), a dataset of declarative sentences with associated question/answer pairs derived from SQuAD. Furthermore, we experiment with the recently-introduced MixQG models (Murakhovs'ka et al., 2021), which are T5 models trained on a combination of nine QA datasets with diverse answer types and which outperform other QG models across several tasks. We apply both the small and large versions of MixQG to better understand the effect of QG model size.
Question Answering The QA component answers questions from the previous steps using the input document as context. We experiment with both extractive QA models, which extract a text span from the input as an answer, and abstractive QA models, which generate an answer token-by-token. For extractive models, we ablate Electra (Clark et al., 2020), a model architecturally similar to BERT (Devlin et al., 2019) that achieves strong performance on the SQuAD 2.0 dataset and was previously used in measuring summary relevance (Deutsch et al., 2020). We also include MADE
(Friedman et al., 2021), which models multi-dataset QA with a collection of dataset-specific adapter modules sharing the same underlying RoBERTabase model. For abstractive QA, we experiment with T5 fine-tuned on SQuAD and UnifiedQA (Khashabi et al., 2020), an approach that trains a T5 QA model on 8 diverse, seed datasets and was shown to generalize across 20 datasets and 4 input formats. All QA models except MADE are trained on data containing unanswerable questions. Additional QA models can be included, although the above set of models allows us to inspect the aspects of interest in this study, namely extractive vs abstractive performance and multi-dataset training.
Answer Overlap Evaluation An answer overlap metric must be computed to determine the match between the initial answer selected in the first component and the QA model output. Typically, answer overlap in QA is measured through exact match (EM) score or word  $F1$  score. We also test a learned metric, the LERC score proposed by Chen et al. (2020). This metric outputs a 1-5 answer overlap score conditioned on a question and context. The scorer is trained on their MOCHA dataset, consisting of 40k crowdsourced judgments on QA model outputs. We include the BERT-base (Devlin et al., 2019) model from the original paper, which we call LERC (orig). We additionally experiment with two models trained from RoBERTa-large checkpoints, one trained from the original checkpoint, LERC (RoBERTa), and one initialized from Jia et al. (2021), which we call LERC (QuIP), for the task of jointly encoding passages and answers with question-infused pretraining. Lastly, we experiment with the IsAnsweredInput answer metric, which is a 0/1 score of whether the question is answerable using the input document according to the QA model. We use the Electra-large QA model to determine whether a question is answerable, as this model shows strong performance on identifying unanswerable questions on SQuAD.
Question Filtering Model-generated questions may contain noise from the QG model itself or from disfluencies in the summary the QG model conditions upon. Such noisy questions can skew the overall metric score, as the QA component may be unable to correctly answer the question, regardless of the summary's factual consistency. We filter such questions through a step called IsAnswered-Summ Filter: the same Electra-large QA model
returns a 0/1 score of whether the question is answerable, now using the summary as context, and questions labeled as unanswerable are filtered.
Overall For a given question, if IsAnsweredInput returns 0, the question is unanswerable using the input, we label all the above answer overlap scores as 0, and otherwise use the answer overlap score. We refer to this scoring of unanswerable questions as 0 as the Answerability Penalty. We also experiment with not setting the overlap score of these unanswerable questions to 0 but rather using the answer overlap score of the most probable answer from the QA model. Finally, the overall factual consistency score for each metric is computed as its average scores over all questions remaining following Question Filtering.
3.3 Learned Metrics
SCConv is a model introduced by Laban et al. (2021) that learns to aggregate entailment-model output scores across input sentences into a single score. More concretely, for a document consisting of  $M$  sentences and a summary consisting of  $N$  sentences, the entailment-based model produces an  $M \times N$  matrix of entailment scores. The  $M \times N$  matrix is then transformed to an  $H \times N$  matrix by binning the  $M$  sentences to create a histogram, where  $H$  is the number of bins. This matrix is input to a 1-D convolution layer to produce a score for each summary sentence, and the scores are averaged across summary sentences. The parameters of this model are fine-tuned on synthetic data, detailed in Section 4.2
QAFACTEVAL-NLI While SCConv captures sentence-level support, QAFACTEVAL measures finer-grained answer overlap between the source and summary. Thus, we are able to combine these two into a single factual consistency metric, QAFACTEVAL-NLI. Assume that  $K$  answers are extracted from the summary. The pipeline described above will then output a single score per answer for the entire summary, resulting in an array of length  $K$ . We convert this to a histogram of size  $H$  in a similar manner as SCConv and pass this histogram through a 1-D convolution layer to produce a single QA score. This score is concatenated with the NLI score produced by SCConv and input to a linear layer to produce the final metric score. The linear layer can be trained in either synthetic or supervised ways, detailed in Section 4.2.
3.4 Additional Metrics
We include the following metrics for completeness.
BARTScore Yuan et al. (2021) calculates the log-likelihood from BART fine-tuned on CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016) of the summary conditioned upon the source text as a metric for factual consistency.
BLANC Vasilyev et al. (2020) is a reference-less metric of summary quality that measures the difference in masked language modeling performance with and without access to the summary.
QuestEval (Scialom et al., 2021) is the prior state-of-the-art QA-based metric for factual consistency. The T5-base (SQuAD) QG and T5-base QA models described above are applied directly from the QuestEval metric. QuestEval generates questions based on the input document and answers them using the summary in addition to following the above QA metric pipeline. QuestEval aggregates the score from these two pipelines. We believe that our described pipeline more closely measures factual consistency, while generating questions from the source may confound factual consistency with relevance.
We present the datasets explored for binary classification and correlation analyses. We also describe settings for reporting ablation and final results.
The SummaC benchmark (Laban et al., 2021) introduces a collection of datasets for binary factual consistency evaluation. A data point is labeled as positive if it contains no factual inconsistencies or is rated the highest possible score in the case of Likert scaling, and as negative otherwise. We now briefly describe the datasets in the benchmark and any departures from the original benchmark, and additional datasets we use for correlation analysis. We refer the reader to Laban et al. (2021) for further details regarding the benchmark creation.
CGS Falke et al. (2019) consists of paired summary sentences from CNN/DailyMail (Hermann et al., 2015; Nallapati et al., 2016), one correct sentence and one containing an error. Laban et al. (2021) treats the correct summaries as positive examples and the others as negative examples.
XSF Maynez et al. (2020) consists of summaries from the XSum dataset (Narayan et al., 2018) annotated for word-level factual consistency errors.
Polytope Huang et al. (2020) propose a typology of eight summarization errors consisting of both content and stylistic errors and annotate model outputs from 10 systems on CNN/DailyMail data. The original SummaC benchmark included the Omission and Addition errors of this proposed typology as factual inconsistencies, but these are largely extractive, factually consistent summaries. We thus label these examples as factually consistent and report results on this modified dataset.
FactCC Kryscinski et al. (2020) introduce a factual consistency dataset on CNN/DailyMail annotated by the authors of the paper to ensure the quality of the annotations.
SummEval Fabbri et al. (2021) analyze summaries from 17 models on CNN/DailyMail across the dimensions of factual consistency, coherence, fluency, and relevance.
FRANK Pagnoni et al. (2021) introduce an extensive typology of errors made by summarization systems across CNN/DailyMail and XSum.
QAGs Wang et al. (2020b) crowdsourced sentence-level summary annotations for factual consistency across CNN/Daily Mail and XSum data. We only report correlation analysis for this dataset as it was not a part of SummaC.
Metric Implementation Metrics were applied directly from the original GitHub repository or by using the SacreRouge Library (Deutsch and Roth, 2020), which was also used in correlation analysis. The learned metrics make use of code released from Laban et al. (2021) for training, and all models are implemented in PyTorch (Li et al., 2020) and in the Transformers library (Wolf et al., 2019). The BART-large (QA2D) QG and Electra-large QA models are applied from the QAEval relevance modeling metric (Deutsch et al., 2020).
Ablation Settings Following Laban et al. (2021), a metric threshold score for binary classification is determined from the validation set of SummaC and applied to the test set. This threshold score is determined for every metric studied. Furthermore, we note that hyperparameter choices for several of
the strong entailment baselines, namely SCConv, SCZeroShot, and MNLI are derived from Laban et al. (2021), thus providing a reasonable comparison to QAFactEval, whose hyperparameters we tune on the SummaC validation set. For ablation studies, we both perform thresholding and evaluation on the validation set to preserve the integrity of the test set. For each benchmark dataset, we sample a random subset of  $80\%$  of the validation set to determine the threshold and evaluate on the remaining  $20\%$  of the validation set. The best performing combination of QA metric components constitutes our QAFACTEVAL metric. We take the best performing combination of QA metric components and vary a given component, such as answer selection, while holding all other components constant and consistent with the best component combination.
Training Settings To tune the parameters of the learned metrics, we train on a subset of 50k synthetic data points from FactCC, following Laban et al. (2021). We name these runs synthetic setting due to the lack of human-labeled data. We also experiment with a supervised setting by fine-tuning the parameters on the SummaC validation set for each individual dataset, choosing the threshold on this validation data, and applying the model to the test set. Training on such a small amount of data is feasible due to the small number of parameters of the learned metrics. Cross entropy loss with Adam (Kingma and Ba, 2015) optimizer is used, with a batch size of 32 and a learning rate of 1e-2.
In this section, we first study the effects of model component choices on QAFACTVAL. We then compare metric results across both the SummaC binary classification task and correlation analysis.
5.1 Ablation Results
We provide the results of our ablation studies on the components of QA-based metrics in Table 2 and show two illustrative examples in Table 4.
Effect of Answer Selection Selecting NP Chunks performs best, aligning with Deutsch et al. (2020), which shows that NP Chunks obtain the largest coverage of information units while retaining high precision. We find a large decrease in performance when selecting NER and only a slight decrease in performance when choosing Max NP