RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
The objective of summary evaluation is to quantify the quality of summaries, either on a relative or an absolute scale. Accurate and reliable automatic summary evaluation systems are useful to researchers, as they provide an easy and cheap way to compare new summarization models to existing ones. Although current summarization systems have improved dramatically in the last decade, and are capable of generating highly fluent outputs Lewis et al. ( 2020 ); Zhang et al. ( 2020a ); Brown et al. ( 2020 ) , it has been shown that generated summaries are prone to exhibit factual errors or hallucinations Kryscinski et al. ( 2019 ); Huang et al. ( 2021 ); Cao et al. ( 2022 ); Ji et al. ( 2022 ) . Thus, information consistency between the summary and source is an important assessment criterion.

Existing methods that measure information consistency generally perform lexical matching, either directly such as ROUGE Lin ( 2004 ) and BLEU Papineni et al. ( 2002 ) , or indirectly using more complex representations such as triple matching Goodrich et al. ( 2019 ) . Some recent approaches adopt question answering (QA) pipelines to detect factual inconsistencies Chen et al. ( 2018 ); Wang et al. ( 2020 ); Durmus et al. ( 2020 ); Deutsch et al. ( 2021 ); Nan et al. ( 2021 ) . They are based on the assumption that if the source extracted answer is consistent with the summary extracted answer then the summary and source are consistent. The answers are compared using either lexical matching Scialom et al. ( 2019 ); Wang et al. ( 2020 ); Durmus et al. ( 2020 ); Scialom et al. ( 2021 ) or representation-based matching Deutsch and Roth ( 2022 ) . These span-based QA approaches may have lexical biases, and struggle with highly abstractive summaries or when dealing with multiple answer spans.

In this work, a measure of consistency between the source and summary is defined from an information-theoretic perspective. We propose a Multiple-choice Question Answering and Generation framework, MQAG, where instead of comparing text-based answer spans, multiple-choice questions are generated and the resulting answer distributions from the source and summary are compared. The main contributions of this paper are:

We provide an alternative and novel question answering-based approach for assessing information consistency. Our approach can represent the answers via probability distributions instead of lexical or embeddings.

We show that our approach, MQAG, achieves state-of-the-art performance on four out of six summary evaluation tasks.
Background and Related Work
Standard summary evaluation metrics such as ROUGE Lin ( 2004 ) and METEOR Banerjee and Lavie ( 2005 ) are designed to assess summaries against ground-truth summaries, i.e. reference summaries. However, these metrics have been shown to have a low correlation with human judgements Fabbri et al. ( 2021 ) . In practice, there is no ground-truth summary to be used as the reference, and evaluation methods need to compare the summary against the source. Therefore, the scope of this work is assessing the summary against the source.

Although there are several aspects of good summaries, including fluency, coherency, coverage or consistency, generation systems are becoming much more capable of generating fluent texts, so the fluency/coherency aspects are less of a concern compared to consistency and hallucination problems Ji et al. ( 2023 ) . Thus, this work focuses on consistency . Because the definition of consistent information can depend on one’s interpretation, we follow the definition of ‘faithfulness‘ in Maynez et al. ( 2020 ) such that we determine if the information in the summary is consistent with information in the source, and we do not consider ‘factuality’ where valid external facts are acceptable. Existing unsupervised evaluation methods are categorized and explained in the following part. 2 2 2 Supervised approaches, with systems trained on human evaluation annotations, are outside the scope of this work.
Textual overlap scores
n -gram based metrics, including BLEU Papineni et al. ( 2002 ) , ROUGE Lin ( 2004 ) , and METEOR Banerjee and Lavie ( 2005 ) measure n -gram overlap between two texts. Instead of n -grams, BERTScore Zhang et al. ( 2020b ) and BLEURT Sellam et al. ( 2020 ) compare texts in their representation space. These metrics measure textual similarity, so they are not necessarily a good measure of consistency. We note that the original works that proposed these metrics compare the summary against the ground-truth summary, but this work focuses on the scenario where there is no ground-truth summary, and these metrics are used as baselines to compare the summary against the source.
Knowledge representation
Goodrich et al. ( 2019 ) assess factual consistency by comparing relation triples from the source and the summary. The relation triples are in the format of Subject-Relation-Object and can be obtained using a model-free method such as OpenIE Etzioni et al. ( 2008 ) or using a trained relation extraction model. The factual accuracy score based on the triple matching approach is then defined as,

Score=\frac{|{\tt T}_{x}\cap{\tt T}_{y}|}{|{\tt T}_{y}|}

where {\tt T}_{x} and {\tt T}_{y} are relation triples extracted from the source and the summary, respectively.
Textual Entailment
Simulated data, such as real or fake summaries created by pre-defined transformations, have been used to train classifiers to detect inconsistent summaries Kryscinski et al. ( 2020 ); Bao et al. ( 2022 ) . Alternatively, Maynez et al. ( 2020 ) trained a textual entailment classifier on the Multi-NLI (MNLI) dataset Williams et al. ( 2018 ) . Given a context, the entailment model is to classify the hypothesis into one of the three classes (entail/neutral/contradict). When applied to assess summaries, the context is the source document and the hypothesis is the summary. The probability of being the entail class is then used as the consistency score,

Score=P(entail|x,y) (1)
Span-based Question Answering (SpanQAG)
A question-answering approach consists of a question-generation model and an answering model. Given automatically generated questions, the first answer is derived from the source and the second answer is derived from the evaluated summary, and then the two answers are compared.

For example, Eyal et al. ( 2019 ) proposed a QA-based method where questions are generated from the ground-truth summary. QAGS Wang et al. ( 2020 ) and FEQA Durmus et al. ( 2020 ) generate questions from the evaluated summary, so these two methods are designed to measure the amount of information in the summary that is consistent with the source. In contrast, SummaQA Scialom et al. ( 2019 ) generates questions from the source document, so it assesses the coverage of the summary. As an extension to the ideas in QAGS/FEQA and SummQA, QuestEval Scialom et al. ( 2021 ) generates questions from both the source and the summary separately to obtain a precision score and a recall score. QuestEval also assigns a weighting function to take into account the importance of each query/question.

Nevertheless, existing QA methods are span-based where the answering system extracts answer spans before two answer spans are compared. Due to the nature of span-based answers, answer verification (i.e. answer comparison) is typically through exact matching, token F1, BERTScore, or a learned metric Deutsch and Roth ( 2022 ) . This answer verification illustrates a drawback of the existing QA methods that they have to compare the similarity between two texts. To avoid span-based answer verification, we propose an alternative question answering-based approach where multiple-choice question generation and answering systems are used where the answers are now in the form of probability distributions rather than text spans.
Motivation and Theory
Since current summarization systems generate highly fluent summaries, this work focuses on assessing whether summaries contain the same information as that of the source, or whether it is contradictory. One way to view information would be to consider the set of questions that are answerable given a certain passage. If a summary is consistent with the source, then one would expect the set of answerable questions by the summary to overlap with those of the source and yield similar answers. Though span-based QA approaches are similarly motivated, existing span-based frameworks use text similarity measures, either in the form of lexical or representation space. In contrast, we attempt to measure information using multiple-choice questions, which allows for a more abstract understanding of information and enables convenient use of standard information-theoretic measures.
MQAG Score
Let x = source, y = summary, q = question, and \mathbf{o} = options associated with the question q . We define information inconsistency as,

\mathcal{I}(x,y)= \int\limits_{q,\mathbf{o}}D\left(P_{{\tt A}}(\mathbf{o}|q,x),P_{{\tt A}}(\mathbf{o}|q,y)\right)P_{{\tt G}}(q,\mathbf{o}|y)\mathrm{d}\mathbf{o}\mathrm{d}q \approx\frac{1}{N}\sum\limits_{i=1}^{N}D\left(P_{{\tt A}}(\mathbf{o}^{(i)}|q^{(i)},x),P_{{\tt A}}(\mathbf{o}^{(i)}|q^{(i)},y)\right) (2)

where \{q^{(i)},\mathbf{o}^{(i)}\} is sampled from P_{{\tt G}}(q,\mathbf{o}|y) , the question-option generation model, P_{{\tt A}}(\mathbf{o}^{(i)}|q^{(i)},x) and P_{{\tt A}}(\mathbf{o}^{(i)}|q^{(i)},y) are the option distributions given the source and summary respectively, and D is a statistical distance such as KL-divergence. Based on the information inconsistency score in Equation 2 , we define the MQAG score as, 3 3 3 If D>1 , for example, when using KL-divergence, the MQAG score can be negative, but the maximum value is 1.0.

MQAG-Score(x,y)=1-{\mathcal{I}}(x,y) (3)

We refer to Equation 3 as the MQAG-Sum score as the questions are generated from the summary. Furthermore, it is possible to generate questions, \{q,\mathbf{o}\} using the source x instead of the summary y , \{q^{(i)},\mathbf{o}^{(i)}\} is sampled from P_{{\tt G}}(q,\mathbf{o}|x) . We will refer to this variant as the MQAG-Src score. MQAG-Src is expected to measure the amount of source information present in the summary, i.e. the coverage of the summary, while MQAG-Sum is expected to measure the consistency of the summary with respect to the source. To account for consistency and coverage, we also consider a simple combination,

MQAG-F1=2\cdot\frac{MQAG-Sum\timesMQAG-Src}{MQAG-Sum+MQAG-Src} (4)
Statistical Distances D
Given two probability distributions over options \mathbf{o} (e.g. one conditioned on source x , and the other conditioned on summary y ), a statistical distance D measures the distance between the probability distributions. There are multiple distances, which can be used, and in this work, we consider some of the main distances and investigate their properties as well as their empirical performance in our MQAG framework as follows,

KL-Divergence:

D_{\tt KL}=\sum_{o\in\mathbf{o}}P_{{\tt A}}({o}|q,x)\log\left(\frac{P_{{\tt A}}({o}|q,x)}{P_{{\tt A}}({o}|q,y)}\right)

One-Best (i.e. argmax matching):

D_{\tt OB}=\begin{cases}0,&if o_{x}=o_{y}\ 1,&otherwise\end{cases}

where o_{x}=\operatorname*{arg\,max}_{o}P_{{\tt A}}({o}|q,x) and o_{y}=\operatorname*{arg\,max}_{o}P_{{\tt A}}({o}|q,y) . D_{\tt OB} simply determines whether the two answers match or not.

Total Variation:

D_{\tt TV}=\frac{1}{2}\left\|P_{{\tt A}}(\mathbf{o}|q,x)-P_{{\tt A}}(\mathbf{o}|q,y)\right\|_{1}

Hellinger:

D_{\tt HL}=\frac{1}{\sqrt{2}}\left\|\sqrt{P_{{\tt A}}(\mathbf{o}|q,x)}-\sqrt{P_{{\tt A}}(\mathbf{o}|q,y)}\right\|_{2}

KL divergence is unbounded, which means the value can be exceedingly large. In contrast, one-best is bounded but discontinuous. Both total variation and Hellinger distance are bounded and continuous. We illustrate examples of the properties of these statistical distances on Bernoulli distributions in Figure 4 in the appendix.
System Development Data
RACE Lai et al. ( 2017 ) is a multiple-choice reading comprehension dataset where each example consists of context, question, answer, and 3 distractors (i.e. incorrect options). SQuAD Rajpurkar et al. ( 2016 ) is a collection of question-answer pairs derived from Wikipedia articles, and the correct answers can be any sequence of tokens in the given context. The statistics are provided in Table 1 where abstractiveness is measured by 1.0 minus the length of the longest sequence that exists in both the context and the answer per the answer length, i.e. 1.0-\text{ROUGE-L}_{\text{Precision}}(\text{Answer},\text{Context}) .
Evaluation Data
We evaluate the performance by measuring the correlation against human judgements at the summary level on QAG-(CNNDM Hermann et al. ( 2015 ) , XSum Narayan et al. ( 2018 ) ), XSum-Hallucination and at the system level on Podcast Assessment and SummEval, and the definitions of summary-level and system-level correlations are provided in Appendix C . The statistics are provided in Table 2 .

QAG . Wang et al. ( 2020 ) annotated 235 CNNDM summaries of the system in Gehrmann et al. ( 2018 ) and 239 XSum summaries of fine-tuned BART Lewis et al. ( 2020 ) . The annotation was performed at the sentence level indicating if hallucination occurs or not. Subsequently, for each summary, the faithfulness (or consistency) score is then obtained by averaging all sentence-level human scores.

XSum-Hallucination (XSum-H) . Maynez et al. ( 2020 ) annotated 2500 XSum summaries using 3 crowd-sourced workers on two metrics: 1) Faithfulness = whether the information is faithful w.r.t. the source at the token level. The judgements are then averaged; 2) Factuality = whether the summary level is factual w.r.t source and external knowledge.

Podcast Assessment . Manakul and Gales ( 2022 ) compiled 3580 podcast summaries of abstraction and extractive summarization systems from Spotify Podcast Challenge 2020 Jones et al. ( 2021 ) . The human evaluation was performed on a 4-point scale considering a combination of consistency, coverage, and fluency.

SummEval . Fabbri et al. ( 2021 ) assessed 1600 CNNDM summaries from 16 different summarization systems on four aspects, including relevancy, consistency, coherency, and fluency. In this work, we use the consistency scores.
Baselines
All of the considered methods compare the summary y against the source document x without the ground-truth summary, and we implement these methods as described in Section 2 using code/repository from the relevant previous works.

ROUGE . We use the ROUGE-1 (F1) score in the rouge-score Python package.

OpenIE-TripleMatch . The relation extraction is based on an open scheme, and we use the implementation in FactSumm Heo ( 2021 ) .

BERTScore . We use DeBERTa-base He et al. ( 2021 ) fine-tuned to MNLI as the backbone.

Entailment model . Following the method in Maynez et al. ( 2020 ) , we trained BERT-large Devlin et al. ( 2019 ) on MNLI and we use the probability of the source being entailed by the summary as the assessment score as shown in Equation 1 .

Span-based QAG Baselines . We use three existing span-based question-answering methods as our baselines: QAGS proposed by Wang et al. ( 2020 ) , FEQA proposed by Durmus et al. ( 2020 ) , and QuestEval proposed by Scialom et al. ( 2021 ) .
Question Generation ( G1 , G2 )
The multiple-choice question generation is implemented in two stages. 4 4 4 The motivation is based on our initial experiments that a single generation system (generating the question and 4 options together) often gave low-quality distractors, and using two generation systems improved the quality of distractors. First model {\tt G1} generates the question q and answer a , then model {\tt G2} generates the distractors \mathbf{o}_{\setminus a} given q and a .

P_{{\tt G}}(q,\mathbf{o}|y)=P_{{\tt G2}}(\mathbf{o}_{\setminus a}|q,a,y)P_{{\tt G1}}(q,a|y) (5)

where \mathbf{o}=\{a,\mathbf{o}_{\setminus a}\} denotes all options/choices. We set the number of options to four. Both G1 and G2 are sequence-to-sequence T5-large models Raffel et al. ( 2020 ) . The question-answer generation system G1 is fine-tuned to either RACE or SQuAD, and the distractor generation system G2 is fine-tuned to RACE.
Question Answering ( A )
The answering stage contains one model {\tt A} , which is Longformer-large Beltagy et al. ( 2020 ) with a multiple-choice setup following Yu et al. ( 2020 ); Raina and Gales ( 2022 ) . The input to the model is a concatenation of context, question and option. The answering model {\tt A} is fine-tuned to RACE.
Answerability of Generated Questions
Because not all generated questions are of high quality, we consider filtering out low-quality questions through question-context answerability measures Kundu and Ng ( 2018 ); Hu et al. ( 2019 ) . We consider a simple answerability measure based on the entropy of the probability distribution over the options. We define the effective number of options,

\mathcal{N}_{y}(q,\mathbf{o})=2^{\mathcal{H}[P_{{\tt A}}(\mathbf{o}|q,y)]} (6)

where \mathcal{H}(.) is base-2 entropy, so \mathcal{N}_{y}(q,\mathbf{o}) ranges from 1.0 to the number of options, e.g. 4.0. When q is generated from y but \mathcal{N}_{y}(q,\mathbf{o}) is high, this question q should be deemed unanswerable as it is not answerable even when using the same context. As a result, we use \mathcal{N}_{y}(q,\mathbf{o}) as an answerability criterion to reject questions which have \mathcal{N}_{y}(q,\mathbf{o}) higher than a threshold denoted by \mathcal{N}^{\tau}_{y} .
Analysis of the Components in MQAG
In this subsection, we carry out experiments to find the best configuration of MQAG, including the analysis of statistical distances, variants of MQAG, and answerability. We build two MQAG variants: MQAG {}_{\text{SQuAD}} and MQAG {}_{\text{RACE}} , which differ in the training data of the question+answer generator G1 , while the distractor generator G2 and answering system A are both trained on RACE.
Statistical Distances
In Table 3 , our results compare statistical distances. It can be seen that in both configurations, KL-divergence yields lower correlations than other distances, and on average total variation slightly outperforms Hellinger and one-best distances. Hence, total variation will be used as the main distance. The next observation is that MQAG {}_{\text{SQuAD}} , despite generating more extractive questions, achieves higher correlations than MQAG {}_{\text{RACE}} on most tasks except on Podcast and SummEval.
MQAG-Sum, MQAG-Src, MQAG-F1
Here, we compare three variants of MQAG scores. Our results in Table 4 show that MQAG-Src, which assesses how much source information is contained in the summary by generating questions from the source, achieves lower PCCs than MQAG-Sum on all datasets. This finding aligns with our expectation, as the summaries were graded by humans predominantly on the consistency aspect (which MQAG-Sum was designed to measure) rather than the quantity of source information present (which MQAG-Src measures). When combining MQAG-Src and MQAG-Sum into MQAG-F1, we only observe a small gain on two test settings. Therefore, MQAG-Sum is selected as our main MQAG configuration for the remaining investigations.
Answerability
In Figure 2 , the answerability is swept from 4.0 (keeping all questions) to 1.0 (only keeping those that the answering system A is highly confident). It can be seen that as we filter out high-entropy questions, there is an upward trend in performance across all tasks. In addition, as shown in the figure, setting \mathcal{N}^{\tau}_{y} at 2.0 seems to be a reasonable answerability threshold. At this threshold, \mathcal{N}^{\tau}_{y}=2.0 , out of 50 automatically generated questions, about 36 questions are kept for MQAG {}_{\text{SQuAD}} and about 30 questions are kept for MQAG {}_{\text{RACE}} . The number of remaining questions is similar across all datasets as shown in Table 9 in the appendix. Thus, we set \mathcal{N}^{\tau}_{y}=2.0 , and the performance of MQAG using this answerability criterion is presented and compared against baseline systems in Table 5 .
Comparison Against Existing Baselines
The baseline and MQAG results are shown in Table 5 . The observation is that MQAG achieves a higher correlation than the best SpanQAG on 5 out of 6 tasks. When compared to all existing baselines, MQAG achieves state-of-the-art performance on 4 out of 6 tasks. To investigate the impact of the abstractiveness of summaries on the performance, we split QAG-XSum and XSum-H datasets 5 5 5 XSum summaries are more abstractive than CNNDM summaries, so using XSum should enable us to investigate the impact of abstractiveness better than CNNDM. into two portions of the same size by abstractiveness as measured by the longest sequence in the summary that exists in the source per the summary length (i.e. ROUGE-L precision of summary y using source x as the reference). The results in Table 6 show that although MQAG {}_{\text{RACE}} achieves lower PCCs than MQAG {}_{\text{SQuAD}} (in Table 5 ), when evaluated on the more abstractive split, the performance MQAG {}_{\text{RACE}} is much closer to that of MQAG {}_{\text{SQuAD}} . In addition, compared to MQAG, SpanQAG methods show a larger drop in PCCs in the more abstractive split. This finding further illustrates the benefits of comparing answer distributions rather than text spans.
Number of Questions ( N )
We analyse the impact of the number of generated questions on the performance of MQAG. The mean and standard deviation are presented in Figure 3 . The results show a smooth increase in correlation, which is as expected because the framework is based on a Monte-Carlo approximation (in Equation 2 ), and a similar finding was also observed in QAGS Wang et al. ( 2020 ) . Figure 3 also shows that the variance decreases with N , showing the stability of the approach. Though the performance curve has not completely plateaued at N =50, since the computational cost of MQAG scales linearly with N , 50 questions seem to be a reasonable compromise between computational efficiency and performance. An interesting next step would be to investigate if the same or similar performance can be achieved with as low N as possible, for example, by generating a smaller but more diverse set of questions and options such as varifocal question generation where questions are generated based on different focal points Ousidhoum et al. ( 2022 ) .
Pre-trained Backbone
We investigate model choices by swapping to less capable models, e.g. T5-large \rightarrow T5-base for generation, and Longformer(4096) \rightarrow RoBERTa(512) Liu et al. ( 2019 ) for answering. The results in Table 8 in the appendix show: (1) For generation stage, using a smaller model does not result in lower performance. This could be because T5-base has higher perplexity, and yields more diverse questions. (2) In contrast, for answering stage, when using RoBERTa, with a shorter input length, the performance on SummEval (the input length is mostly shorter than 512) remains almost the same. However, as the input length is longer in other datasets, we observe a drop in PCC when using RoBERTa.
Zero-shot Multiple-choice Question Generation
Given the impressive results of large language models (LLMs) across natural language generation tasks, we investigate the performance of LLMs in a zero-shot fashion instead of using fine-tuned T5 for multiple-choice question generation. Specifically, we use OpenAI GPT-3 Brown et al. ( 2020 ) (text-davinci-003) where we query 50 questions and 4 options using the following prompt format: Write 50 diverse multiple-choice questions with 4 options from the following context: {context}.

We found that GPT-3 generated 50 questions as specified in the prompt around 26% of the examples and the remaining only have 20 questions. The majority of questions (more than 95%) have 4 options, while the remaining have 2 options. In Table 7 , the results show that zero-shot GPT-3 performs worse than our fine-tuned T5 systems in both multiple-choice question generation tasks. This illustrates that there is some sensitivity due to the quality of generated questions, and using our fine-tuned T5 is a better option than zero-shot GPT-3.