RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Conclusion
This work proposes MQAG – a novel scheme for assessing information consistency between source and summary based on the distance between multiple-choice answer distributions instead of text-based answer spans in existing question-answering methods. Our experiments demonstrate the potential of this alternative approach which outperforms existing techniques on various datasets. The realization of the framework exploits current multiple-choice question generation and answering systems. Its performance is expected to increase as backbone systems improve, for example, the diversity of questions generated and the selection of options. Also, the framework is highly interpretable, allowing more insight into summary assessment.
Limitations
Domain . Our approach is designed to assess the information content, so it may not work well with other aspects of summary evaluation such as fluency or coherency. Our analysis is based on the systems trained on RACE, which is collected from English examinations in China. Hence, the generated questions and answer distributions could be biased towards the style of the examinations.

Efficiency . Given the realization of the MQAG framework where two generators G1 and G2 are adopted, the MQAG framework can be slow when using old infrastructure, for example, it takes around 3 seconds per question on one NVIDIA P100 GPU. To address this issue, future work could explore a more efficient realization of MQAG.
Acknowledgments
This work is supported by Cambridge University Press & Assessment (CUP&A), a department of The Chancellor, Masters, and Scholars of the University of Cambridge, and the Cambridge Commonwealth, European & International Trust. We would like to thank the anonymous
reviewers for their helpful comments.
Training QG and QA systems
We train the question+answer generation model ( G1 ) on RACE or SQuAD, and train the distractor generation model ( G2 ) and the answering model ( A ) on RACE. We do early stopping when the performance on the validation set does not improve. We use batch size 8 for G1 and G2 models (T5) and 2 for A model (Longformer). The learning rate is set to 1e-6, and we use the Adam optimizer. We carried out training on one NVIDIA A100-80GB GPU. Training one generation model (T5-large) takes around 8 hours, and training the answering model (Longformer-4096) takes up to 2 days. Running MQAG inference with generation=T5-large and answering=Longformer-4096 on one NVIDIA P100 GPU takes around 3 seconds per question.
Licenses
The licenses of the datasets are CC-BY-4.0 for XSum-Hallucination and Podcast Assessment, and MIT license for SummEval. For QAG, we were unable to find its license. The licenses of T5 and Longformer backbone models are apache-2.0.
Open-Sourcing Trained Models
To allow the trained models in MQAG to be used for research purposes in other question generation and answering tasks, we have made them available online. The links to these models on HuggingFace can be found on our project page at https://github.com/potsawee/mqag0 .
D.1 Ablation: Model Choices
For generation models, we measure cross-entropy losses on RACE-testset:

T5-base (223M): G1 = 1.612, G2 = 1.875

T5-large (738M): G1 = 1.478, G2 = 1.741

where G1 denotes question+answer generation, and G2 denotes distractor generation. For answering models, we measure accuracy on RACE-testset:

Roberta (355M): Accuracy = 84.84

Longformer (435M): Accuracy = 81.67
D.2 MQAG Results
Here, we provide results that are complementary to those presented in the main text. Figure 5 illustrates the answerability results on QAG-XSum and Podcast, and Figure 6 illustrates the impact of N on the remaining datasets not presented in the main text. Table 10 shows the results of all MQAG configurations. Table 11 shows the Spearman’s rank correlation coefficient of the main results.
Figure 1: Multiple-choice Question Answering and Generation (MQAG) framework. The answers are represented by probability distributions over choices instead of text spans in existing question-answering approaches.; Multiple-choice Question Answering and Generation (MQAG) framework. The answers are represented by probability distributions over choices instead of text spans in existing question-answering approaches.
Figure 2: \Delta PCC of MQAG-Sum with total variation (i.e. PCC - PCC {}_{N^{\tau}_{y}=4.0} ) against the answerability threshold \mathcal{N}^{\tau}_{y} on X-axis. MQAG without answerability is equivalent to setting {N^{\tau}_{y}=4.0} , and the results at this operating point can be seen on the right-most point in each plot. As we reduce the threshold ( N^{\tau}_{y}\downarrow ), more questions are rejected. The results on QAG-XSum and Podcast are provided in Figure 5 in the appendix.; \Delta PCC of MQAG-Sum with total variation (i.e. PCC - PCC {}_{N^{\tau}_{y}=4.0} ) against the answerability threshold \mathcal{N}^{\tau}_{y} on X-axis. MQAG without answerability is equivalent to setting {N^{\tau}_{y}=4.0} , and the results at this operating point can be seen on the right-most point in each plot. As we reduce the threshold ( N^{\tau}_{y}\downarrow ), more questions are rejected. The results on QAG-XSum and Podcast are provided in Figure 5 in the appendix.
Figure 3: Mean and standard deviation of Pearson correlation (Y-axis) of MQAG {}_{\text{RACE}} on QAG-CNNDM when the number of generated questions N is varied from 1 to 50 (X-axis). Standard deviation is obtained via bootstrapping. The results on other datasets are provided in Figure 6 in the appendix.; Mean and standard deviation of Pearson correlation (Y-axis) of MQAG {}_{\text{RACE}} on QAG-CNNDM when the number of generated questions N is varied from 1 to 50 (X-axis). Standard deviation is obtained via bootstrapping. The results on other datasets are provided in Figure 6 in the appendix.
Figure 4: Statistical distances between two Bernoulli distributions \mathbf{p}_{1}=[p_{1};1-p_{1}] and \mathbf{p}_{2}=[p_{2};1-p_{2}] at different values of p_{1} . We show 4 plots of different values of p_{1} = 0.00, 0.25, 0.50, 0.75, and Y-axis represents distance D and X-axis represents p_{2} . It can be seen that KL divergence is unbounded, which means the value can be exceedingly large. One-best, in contrast, is bounded between 0.0 and 1.0; however, one-best is discontinuous. Total variation and Hellinger distance are continuous and bounded between 0.0 and 1.0.; Statistical distances between two Bernoulli distributions \mathbf{p}_{1}=[p_{1};1-p_{1}] and \mathbf{p}_{2}=[p_{2};1-p_{2}] at different values of p_{1} . We show 4 plots of different values of p_{1} = 0.00, 0.25, 0.50, 0.75, and Y-axis represents distance D and X-axis represents p_{2} . It can be seen that KL divergence is unbounded, which means the value can be exceedingly large. One-best, in contrast, is bounded between 0.0 and 1.0; however, one-best is discontinuous. Total variation and Hellinger distance are continuous and bounded between 0.0 and 1.0.
Figure 5: \Delta PCC of MQAG-Sum with total variation against the answerability threshold \mathcal{N}^{\tau}_{y} on the X-axis. This figure extends Figure 2 in the main text.; \Delta PCC of MQAG-Sum with total variation against the answerability threshold \mathcal{N}^{\tau}_{y} on the X-axis. This figure extends Figure 2 in the main text.
Figure 6: Mean (top row) and standard deviation (bottom row) of Pearson correlation (Y-axis) of MQAG {}_{\text{RACE}} when the number of generated questions N is varied from 1 to 50 (X-axis). This figure extends Figure 3 in the main text.; Mean (top row) and standard deviation (bottom row) of Pearson correlation (Y-axis) of MQAG {}_{\text{RACE}} when the number of generated questions N is varied from 1 to 50 (X-axis). This figure extends Figure 3 in the main text.
Statistics of datasets for training MQAG systems. Length = #words. Abstractiveness of 0% indicates that in SQuAD the answer always exists in the context.
Statistics of evaluation datasets. Length is the number of words calculated using the NLTK tokenizer. ∗ #systems \times documents.
Comparison of Statistical Distances using MQAG-Sum without answerability.
Comparison of MQAG-Src, MQAG-Sum, and MQAG-F1 without answerability.
Pearson Correlation Coefficient (PCC) between the scores of summary evaluation methods and human judgements. PCCs are computed at the summary level on QAG and XSum-H, and at the system level on Podcast and SummEval. PCCs on Podcast are computed on 15 abstractive systems. Our best performing MQAG configuration consists of (i) generation stage G generates questions from summary y (i.e. MQAG-Sum), (ii) statistical distance is total variation, (iii) the answerability threshold \mathcal{N}^{\tau}_{y} is set to 2.0. Underline denotes where MQAG outperforms the best SpanQAG system, which is 5 out of 6 tasks. When compared to all baselines, MQAG achieves the highest PCC on 4 out of 6 tasks. The results of all MQAG configurations are provided in Table 10 , and Spearman’s correlation results are provided in Table 11 in the appendix.
Performance as measured by Pearson correlation coefficient on the low abstractiveness and high abstractiveness of QAG-XSum and XSum-H (Faithful). The results on the entire datasets are in Table 5 .
GPT-3 versus fine-tuned T5 using D_{\tt TV} without answerability for multiple-choice question generation.
Ablation on model choices in MQAG using N =20. SumE = SummEval (Consistency aspect), QAG-X = QAG-XSum, Podc = Podcast Assessment.
The number remaining questions at \mathcal{N}^{\tau}_{y}=2.0 .
Pearson correlation coefficients of all MQAG configurations. Our MQAG results are based on N =50. When applying the answerability mechanism, the threshold \mathcal{N}^{\tau}_{y} is set to 2.0.
Spearman’s rank correlation coefficient between the scores of summary evaluation methods and human judgements. This table is complementary to Table 5 which reports Pearson’s correlation coefficient results.
Example from QAG-XSum (documentID=1). Factual inconsistency in the summary is highlighted in red.