RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Question
Year: 2023

Abstract:
Existing metrics for evaluating the quality of automatically generated questions such as BLEU, ROUGE, BERTScore, and BLEURT compare the reference and predicted questions, providing a high score when there is a considerable lexical overlap or semantic similarity between the candidate and the reference questions. This approach has two major shortcomings. First, we need expensive human-provided reference questions. Second, it penalises valid questions that may not have high lexical or semantic similarity to the reference questions. In this paper, we propose a new metric, RQUGE, based on the answerability of the candidate question given the context. The metric consists of a question-answering and a span scorer modules, using pre-trained models from existing literature, thus it can be used without any further training. We demonstrate that RQUGE has a higher correlation with human judgment without relying on the reference question. Additionally, RQUGE is shown to be more robust to several adversarial corruptions. Furthermore, we illustrate that we can significantly improve the performance of QA models on out-of-domain datasets by fine-tuning on synthetic data generated by a question generation model and reranked by RQUGE.1

heading: 1 Introduction
text: Given the context (e.g. paragraph), the goal of question generation (QG) is to generate questions with or without providing the answer spans. Automatic question generation can be used in several applications: improving the question answering (QA) task (Duan et al., 2017; Du and Cardie, 2018; Puri et al., 2020; Cheng et al., 2021), automatic assess-
∗Work done during an internship at Meta AI. † This work is done when the author was on leave from BYJU’s LAB. 1The implementation and annotated data are provided at https://github.com/alirezamshi/RQUGE
ments (Rebuffel et al., 2021; Lee et al., 2021), especially for the educational domain (Chen et al., 2018), and the evaluation of factual consistency in the text generation tasks (Scialom et al., 2019a, 2021; Fabbri et al., 2022).
Previous work (Hosking and Riedel, 2019; Scialom et al., 2019b; Zhang and Bansal, 2019; Laban et al., 2022) has shown that QG models can gen-
6845
erate questions inconsistent with the corresponding context and the answer span. So, measuring the acceptability of candidate questions is a critical challenge. Human judgment is the most accurate method in natural language generation, but it is expensive, time-consuming, and not scalable. Consequently, several metrics e.g. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), BERTScore (Zhang et al., 2020) are proposed to automatically measure the quality of the generated text.
Specifically for the question generation task, previous work has utilised reference-based metrics e.g. BLEU, ROUGE, BERTScore, and BLEURT (Sellam et al., 2020; Ushio et al., 2023a,b, 2022) to evaluate the quality of the candidate question given the reference question. However, these methods highly depend on the diversity of the reference questions for a given answer span. Due to the huge cost of human annotations, existing QA/QG datasets mostly provide one reference question for the given context and answer, which results in wrongly penalising some valid questions. In Figure 1, the first candidate question (Q1) is generated by paying attention to different evidence in the context, and Q2 is a paraphrase of the reference, but both BLEU and BERTScore fail to assign high scores to them. Furthermore, reference-based metrics are not sensitive to very small corruptions of the reference questions, which makes the candidate question unacceptable (Q3).
In this paper, we propose RQUGE , a Referencefree QUestion Generation Evaluation metric that can compute the quality of the candidate question without requiring a reference question. Given the corresponding context and answer span, our metric calculates the acceptability score by applying a general question-answering module, followed by a span scorer. The former module generates the answer span for the given candidate question, and the latter computes the semantic similarity of the predicted and gold answer spans. Our metric is extremely valuable in cases where the reference question is not well-formed 2 or there is one (or no) reference for a given context and answer span. We evaluate our metric on several datasets, including SQuAD (v1) (Rajpurkar et al., 2016), Natural Questions (NQ) (Kwiatkowski et al., 2019), and MS-MARCO (Bonifacio et al., 2021), and show
2For instance, many questions in the Natural Question dataset (Kwiatkowski et al., 2019) are not well-formed. Example: “in a deep mine temperatures increase with depth at the rate of”
that it consistently has a better correlation with human judgment compared to previous QG metrics. We also integrate RQUGE into the decoding step by re-ranking candidate questions of each instance by our metric, leading to a better correlation with the human evaluation. Additionally, we demonstrate that RQUGE is more robust to adversaries than previous metrics with +13.1% relative improvement. Finally, we improve the performance of question answering models on an out-of-domain dataset by fine-tuning them on synthetic data generated by a question generation model, then re-ranked with RQUGE to choose the best candidate question for the given answer span, resulting in an +18.3% F1 and +22.2% EM relative improvement. To sum up, our contributions are as follows:
• We propose RQUGE, an evaluation metric for measuring the quality of the automatically generated questions, without requiring access to any reference questions.
• We show that our metric has a significantly higher correlation with human judgment in terms of the acceptability of the candidate questions on SQuAD (v1), NQ, and MSMARCO datasets. Also, re-ranking candidate questions with RQUGE leads to a better correlation with human judgment.
• We demonstrate that RQUGE metric is more robust compared to previous work on several adversarial strategies such as negation, entity swapping, gender reversing, or paraphrasing the reference questions.
• Finally, we illustrate that the performance of QA models significantly improves on the outof-domain datasets by fine-tuning them on the synthetic data, created by applying a question generator model, then re-ranking with RQUGE metric.

heading: 2 Related Work
text: Previous work on automatic evaluation of Natural Language Generation (NLG) tasks have been categorized as follows:
Unsupervised Metrics. It contains the most commonly used metrics e.g. BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), chrF (Popović, 2015), and METEOR (Denkowski and Lavie, 2010). These metrics calculate the correlation of reference
and predicted sequences in a discrete space by utilising token-level matching functions. Then, recent work e.g. BERTScore (Zhang et al., 2020) and MoverScore (Zhao et al., 2019) use BERT (Devlin et al., 2019) embeddings to provide a soft token-level matching instead of the hard n-gram overlap. These metrics have been applied to various NLG tasks (Du et al., 2017; Zhou et al., 2017; Xiong et al., 2019; Pan et al., 2020; Lewis et al., 2020; Cheng et al., 2021; Mohammadshahi et al., 2022a,b). Specifically for QG evaluation, Nema and Khapra (2018) propose a scoring function, focusing on the answerability of the candidate question, which improves the human correlation when integrated with existing unsupervised metrics.
Regression-based Metrics. These metrics e.g. COMET (Rei et al., 2020), BLEURT (Sellam et al., 2020), S3 (Peyrard et al., 2017), and VRM (Hirao et al., 2007) train a regression layer in a supervised manner to mimic the human judgment.
Ranking-based Metrics. The aim of these metrics is to assign a higher score to a better candidate compared to worse predictions. The most popular ones include BEER (Stanojević and Sima’an, 2014) and COMET (Rei et al., 2020).
Generation-based Metrics. The idea is to formulate the evaluation of NLG, as a text generation problem from pre-trained language models. Given the source sequence, the better candidate should be generated with a higher score (probabil-
ity) compared to the worse ones. The most popular ones are BARTScore (Yuan et al., 2021) and PRISM (Thompson and Post, 2020).
Additionally, we include CTC (Deng et al., 2021) and QRelScore (Wang et al., 2022) as referencefree metrics for better comparison. CTC (Deng et al., 2021) proposes an evaluation framework for NLG tasks by providing several reference-free metrics, which are computed by aggregating the alignment scores between the input, context and the predicted sequence. To measure the alignment, CTC (Deng et al., 2021) uses BERT (Devlin et al., 2019) embedding matching, discriminative, and regression models. 3 QRelScore computes the answerability of the candidate question by applying word-level hierarchical matching and sentence-level prompt-based generation. Different from previous work, RQUGE combines question answering and span scoring modules to compute the acceptability of the candidate, which leads to a significantly better correlation with human judgement in multiple datasets with different domains and answer lengths.

heading: 3 RQUGE Architecture
text: RQUGE architecture is illustrated in Figure 2. It consists of two components: question answering and span scorer modules. Given the context, gold answer span, and the candidate question, generated by a question generation model (QG), RQUGE
3CTC can be considered as an unsupervised metric when BERT embeddings are used to compute the alignment.
computes the acceptance score (κ) of the candidate question as follows:
{ ac = QA(qc, D)
κ = S(qc, ac, ar, D) (1)
where the qc = QG(ar, D) is the generated candidate question for the gold answer span ar and context D. To calculate the score, the question answering model QA(. ) predicts the answer span ac, given the candidate question qc and the context D. Finally, the span scorer S(. ) computes the acceptance score κ, conditioned on the candidate question, predicted answer, gold answer, and context. In the following, we will describe each module in detail.

heading: 3.1 Question Answering Module
text: Given the context and the candidate question, the question answering model predicts the answer span. To make our metric general to several domains, we use UnifiedQAv2 (Khashabi et al., 2022) model to generate the answer span. UnifiedQAv2 is a T5based encoder-decoder model, which is trained on 20 QA datasets, and achieves competitive performance with the state-of-the-art models in several in-domain and out-of-domain datasets.4 The input to the model is the concatenation of the candidate question and corresponding context.

heading: 3.2 Span Scorer Module
text: Given the predicted answer span ac of the candidate question qc, the span scorer calculates the score (ranging from 1 to 5) of the candidate question. Inspired by Chen et al. (2020) and Fabbri et al. (2022), we use an encoder-only BERT-based model to calculate the acceptance score. Specifically, we first encode the input sequence, then pass the vector representation of [CLS] to the regression layer to compute the acceptance score κ. The input to the module is:
[CLS] cand. question [q] gold answer [r] pred answer [c] context
We employ pre-trained RoBERTa model, provided by Fabbri et al. (2022). The model is first pretrained with a QA-infused pre-training objective,5
4We refer to Khashabi et al. (2022) for further details. A list of evaluated datasets is provided in Appendix A. We use unifiedqa-v2-t5-large checkpoint, provided in https: //github.com/allenai/unifiedqa.
5It includes pre-training contextual embeddings with a bi-encoder extractive QA loss, which results in encoding
then fine-tuned on MOCHA human ratings QA dataset (Chen et al., 2020). MOCHA is a dataset of human judgment scores for training and testing reading comprehension metrics, where annotators are asked to score candidate spans, given the context, gold answer, and the corresponding question.

heading: 4 Experimental Setup
text: Datasets. We evaluate metrics on three widelyused QA datasets, including SQuAD(v1) (Rajpurkar et al., 2016), NQ (Kwiatkowski et al., 2019), and MS-MARCO (Bonifacio et al., 2021). NQ is used to demonstrate the benefit of our metric in cases where reference questions are not wellformed and are derived from the Google engine. 6 For MS-MARCO, we use DESCRIPTION type of the dataset to show the effectiveness of our metric on candidate questions with long answer spans (13 tokens on average). Unlike SQuAD and NQ, MSMARCO is not included in the training data of the question answering module of RQUGE (i.e. UnifiedQAv2 (Khashabi et al., 2022)). We use MS-MARCO to demonstrate that RQUGE can be generalised to out-of-domain datasets. 7
Question Generators. We fine-tune two commonly used QG models, including GPT2 (Radford et al., 2019), trained with causal language modelling objective, and MixQG (Murakhovs’ka et al., 2022), which is the state-of-the-art question generator and is a T5-based (Raffel et al., 2020) sequenceto-sequence model. We choose GPT2 and MixQG as our question generators as there is a significant gap in their performance, making them suitable for evaluating the metrics. 8
Baselines. We include BLEU-4 (Papineni et al., 2002), ROUGE-1, ROUGE-L (Lin, 2004), METEOR (Denkowski and Lavie, 2010), MoverScore (Zhao et al., 2019), and BERTScore (Zhang et al., 2020)9 as unsupervised metrics, that are commonly used for the question generation task. We additionally use QBLEU, which is specific for
information about questions that can be answered by each text span. The pre-trained model is available at https: //github.com/salesforce/QAFactEval.
6We use the subset of NQ with short-form answer spans, similarly to the previous work (Murakhovs’ka et al., 2022).
7Further details of evaluated datasets are provided in Appendix B.1.
8Hyper-parameters for fine-tuning QG models are provided in Appendix B.2.
9We use the better-performing BERTScore, initialised with DeBERTa-xlarge model (He et al., 2021).
QG evaluation. Furthermore, we utilise BLEURT20 (Sellam et al., 2020), and COMET (Rei et al., 2020) as regression-based and ranking-based metrics. Finally, we include BARTScore (Yuan et al., 2021), fine-tuned on ParaBank2 (Hu et al., 2019). In all aforementioned metrics, scores are calculated between the candidate and reference questions. As reference-free baselines, we use QRelScore (Wang et al., 2022), and also adopt the factual consistency scorer of CTC (Deng et al., 2021) to calculate the consistency score as:
κctc = mean(align([ar, D] → qc))
where ar, D, and qc are answer span, context, and candidate question, respectively. The function align(.) estimates the alignment for tokens of the candidate question with the given context and answer. we use albert-xlarge-vitaminc-mnli, which uses a discriminative model to compute the alignment.10
Human Annotation. We use 3 volunteer annotators to rate the candidate questions of QG models.11 All annotators are fluent English speakers.
10We also used faithfulness aspect of BARTScore (Yuan et al., 2021) as a reference-free metric for measuring the quality of the candidate questions, but preliminary experiments result in a poor correlation with human judgment.
11Human evaluation of the quality of the generated questions is not available in previous work.
Inspired by previous work (Rus et al., 2010; Nema and Khapra, 2018), we ask annotators to score each candidate question based on three criteria: grammaticality, answerability, and relevance. Grammaticality measures the syntactic structure of the question. Answerability checks whether the question contains all the important entities, and relevance checks the relatedness of the generated questions with the given answer span. Grammaticality and answerability scores are on a 3-point scale (3 as acceptable, and 1 as rejection), and relevance is on a 2-point scale. We sample 600 questions generated from fine-tuned QG models on SQuAD(v1) (Rajpurkar et al., 2016), NQ (Kwiatkowski et al., 2019), and MS-MARCO (Bonifacio et al., 2021) datasets. We then randomly shuffle and anonymise them for annotators. Further details of the human annotation procedure are provided in Appendix C.12

heading: 5 Results and Discussion
text: We evaluate our RQUGE and previous metrics on various datasets and tasks. First, we evaluate the correlation of metrics with human judgment in Sections 5.1 and 5.2. We then demonstrate their robustness on the adversarial subset in Section 5.3.
12The human-evaluated data is available at https:// github.com/alirezamshi/RQUGE.
Finally, Section 5.4 illustrates that fine-tuning QA models on the synthetic data, created by our metric, improves their performance on out-of-domain datasets.

heading: 5.1 Correlation with Human Judgment
text: Annotator Agreement. The pairwise interannotator agreements, calculated using Cohen’s Kappa are 88.91%, 85.32%, and 83.54%. 13 We use the average score of three annotators for the remaining experiments.
Metric-to-Human Correlation. Table 1 illustrates the correlations of automatic metrics with the human judgment, averaged over all datasets.14 RQUGE metric has a considerably higher correlation with human judgment on all criteria. For instance, it outperforms the best previous work with +7.1%, +23.8%, +18.8% absolute improvement (based on Pearson (r) score) for grammaticality, answerability, and relevance, respectively. Appendix D illustrates the result of correlation with the human judgment for each dataset, separately. In in-domain evaluation sets, RQUGE results in
13Calculated on the summation of grammaticality, answerability, and relevance.
14For a fair comparison on grammaticality, we just evaluate on SQuAD evaluation set, where reference questions are mostly well-formed.
+29.7% and +24.8% absolute point improvement for SQuAD and NQ datasets, respectively, based on answerability measurement. For MS-MARCO as the out-of-domain dataset, RQUGE reaches +12.2% absolute improvement for the relevancy criterion, while having competitive results with CTC on answerability measurement. These results show the effectiveness of our metric in different domains, and question structures (well-formedness) and confirm the generalisation of our metric to outof-domain settings.

heading: 5.2 Re-Ranking with RQUGE
text: To further demonstrate the effectiveness of RQUGE, we use it to re-rank the output predictions of the question generation model to choose the best generated question. Given the context and answer span, QG model 15 generates a bag of candidate questions (here, we apply Nucleus sampling (Holtzman et al., 2020) to increase the diversity)16, that are sorted based on the perplexity (PPL) of the question generator. At each step, we choose K-
15We use MixQG (Murakhovs’ka et al., 2022) model as the question generation model, as it is the state-of-the-art QG model for the evaluated dataset.
16We use temperature sampling of 1, and top summation probability (topp) of 0.94.
first candidates of each sample and re-rank them with RQUGE. Then, other automatic metrics are computed for the best one chosen by RQUGE. Figure 3 demonstrates the relative score gains of QG metrics compared to K = 1 (best candidate by PPL) for different values of K. 17 Interestingly, it illustrates that other metrics drop as the number of candidate outputs of each sample (K) is increasing, meaning that the best candidates chosen by our metric are not correlated with other metrics. To confirm these results, annotators are asked to score 250 samples 18 from the evaluation dataset. For each sample, the best candidate questions of K = 1 (best predictions of PPL), K = 5 (where the RQUGE is starting to become plateau), and K = 50 (best predictions of RQUGE) are chosen, and annotators are asked to score these three generated questions based on criteria defined in Section 4.19
Figure 4 depicts the relative difference of NLG metrics compared to the score of K = 1, alongside human scores. We can see from the figure that annotators significantly prefer highest ranking questions of K = 5 and K = 50 chosen by RQUGE compared to the best candidate questions picked based on PPL, while the average scores of other automatic metrics drop as the number of candidate questions (K) increases. For K = 5, the average human score of highest ranking questions is rela-
17The relative performances of the remaining metrics for different K are provided in Appendix E.
18We filter redundant instances (samples that all of its best candidates are the same) during the sampling process.
19Summation of scores for grammaticality, answerability, and relevance is defined as the overall score.
tively +7.49% better than best candidate questions, chosen by PPL of question generator (K = 1). It confirms the effectiveness of RQUGE , when integrated into the decoding step. Additionally, there is not a significant difference (-1.1%) between the average human scores of candidate questions chosen based on RQUGE for K = 5 and K = 50. This is correlated with Figure 3, as RQUGE also becomes plateau from K = 5. Figure 5 illustrates an example in which annotators prefer the second candidate question, while BLEU-4 and BERTScore compute higher scores for the first candidate question. 20

heading: 5.3 Robustness Analysis
text: To further assess the robustness of the QG metrics on adversarial corruptions of reference questions, we evaluate metrics on a subset of positive and negative samples, created from SQuAD (Ra-
20For transparency, we will provide the human evaluation of the re-ranking experiment.
jpurkar et al., 2016) evaluation set, as shown in
Table 2.21 The remaining subset contains 2,500 samples equally selected from positive and negative questions. Inspired by Chen et al. (2021) and Honovich et al. (2022), positive questions are paraphrases of references, created by two methods, either translating to a high-resource language, then back-translating to English, or applying a T5 (Raffel et al., 2020) model fine-tuned on Quora paraphrasing task.22 For negative samples, we use three strategies: negation, reversing the gender, and swapping the entities of the reference question with relevant entities in the corresponding context. Further details of the adversarial evaluation set are provided in Appendix F.
Results and Discussion. We use RQUGE and previous metrics on the adversarial subset to classify the corrupted candidate questions based on their acceptability score. Table 3 illustrates the area of the ROC curve of QG metrics on the adversarial subset. 23 Overall, RQUGE metric significantly
21For a fair comparison, we omit NQ, and MS-MARCO evaluation sets as their reference questions are not always well-formed.
22https://www.kaggle.com/competitions/ quora-question-pairs/data.
23Number of instances for negation, gender reversing, and entity swapping are 1000, 150, and 100, respectively.
outperforms BLEURT-20 (the best previous work) by +13.1% relative improvement. Previous unsupervised metrics drop significantly for all types of negative samples, while BLEURT-20, BARTScore, and reference-free metrics perform better comparatively, especially for negation. Our RQUGE metric decreases the error relatively by +22.2% and +7.5% for negation, and entity swapping compared to previous work and has the second-best results on reversing the gender. This confirms the robustness of our metric for different adversarial corruption.

heading: 5.4 Domain Adaptation of QA Task
text: Generated questions using a QG model can be used to improve the performance of a question answering model on out-of-domain datasets. In this section, we show that fine-tuning on the generated synthetic data, re-ranked with RQUGE improves the performance of the question answering model.
Implementation Details. For the out-of-domain dataset, we choose MS-MARCO (Bonifacio et al., 2021) dataset, since the UnifiedQAv2 (Khashabi et al., 2022) (utilised in the calculation of RQUGE) has not used it for training. 24 Given the context, we apply Stanza (Qi et al., 2020a) Named-Entity Recognition (NER) model to extract candidate answer spans. A QG model is then applied to a randomly chosen candidate span, creating a bag of output predictions, using Nucleus sampling (Holtzman et al., 2020). Then, we apply the same reranking mechanism, described in Section 5.2 using RQUGE , CTC, QRelScore, and the PPL of the QG. We also use a beam search of size 5 with no re-ranking as a baseline. We use MixQG (Murakhovs’ka et al., 2022) to generate questions. For QA, we first fine-tune T5-small (Raffel et al., 2020) on SQuAD (Rajpurkar et al., 2016) (zero-shot for our setting), then fine-tune it on the generated synthetic data. Further implementation details and hyper-parameters are provided in Appendix G.
Results and Discussion. Figure 6 demonstrates the performance of the QA model on the out-ofdomain dataset, fine-tuned for different amounts of synthetic data. Generally, fine-tuned QA model reaches significantly better performance compared to the zero-shot setting. This is important for domains in which we do not have annotated QA data. Furthermore, fine-tuning on the re-ranked data with
24For compatibility with the NER model, we use instances with short-form answer span (less than 4 tokens).
RQUGE consistently improves the performance of the QA model for a different amount of synthetic data, compared to other baselines. Specifically, it significantly outperforms baselines by +18.3% F1, and +22.2% EM, on average. It again shows the effectiveness of our RQUGE by employing it in the domain adaptation of QA models for the outof-domain dataset.

heading: 6 Conclusion
text: We propose RQUGE , Reference-free QUestion Generation Evaluation metric to measure the quality of the generated questions, by better encoding the relevant context and answer without requiring a reference question. It consists of two modules, a question answering model, and a span scorer, which are existing pre-trained models without further fine-tuning. We compare the performance of RQUGE with existing QG metrics on SQuAD, MSMARCO, and NQ datasets, and show that RQUGE achieves a significantly better correlation with human judgment. Additionally, we integrate RQUGE into the decoding step by using it to re-rank the candidate questions, which leads to a better correlation with human. For robustness, we evaluate QG metrics on adversarial data by corrupting the reference questions and show that RQUGE achieves significantly better performance compared to previous work. Finally, we show that fine-tuning QA models
on the synthetic data, generated with a QG model and re-ranked with RQUGE , improves the performance of QA models on out-of-domain datasets.

heading: Acknowledgement
text: We thank Parth Pathak, Yatharf Saraf, and Omprakash Sonie for their helpful discussion and support. We are grateful to anonymous reviewers for their fruitful comments and corrections.
Limitations
The main limitation of our work is that we have applied and verified the effectiveness of our metric on the English question answering datasets. Since RQUGE depends on a strong question answering module, one has to find an alternative model to the UnifiedQA (Khashabi et al., 2022) we have used in calculation of RQUGE. Additionally, we did an error analysis on the subset that RQUGE and human evaluation have a significant difference in Appendix H, which shows that mistakes are categorised into syntactic-based and knowledge-based errors. It gives us directions for future improvement of RQUGE metric.

heading: Appendix A Evaluated Datsets of UnifiedQAv2 Model
text: UnifiedQAv2 is evaluated on SQuAD (v1) (Rajpurkar et al., 2016), SQuAD (v2) (Rajpurkar et al., 2018), NewsQA (Trischler et al., 2016), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019), NarrativeQA (Kočiský et al., 2018), DROP (Dua et al., 2019), NaturalQuestions (Kwiatkowski et al., 2019), MCTest (Richardson et al., 2013), RACE (Lai et al., 2017), OpenBookQA (Mihaylov et al., 2018), ARC (Clark et al., 2018), CommonsenseQA (Talmor et al., 2019), QASC (Khot et al., 2020), PhysicalIQA (Bisk et al., 2019), SocialIQA (Sap et al., 2019), Winogrande (Sakaguchi et al., 2021), BoolQ (Clark et al., 2019), MultiRC (yes/no) (Khashabi et al., 2018), and BoolQ-NP as in-domain datasets. Additionally, it is evaluation on AdversarialQA (Bartolo et al., 2020), ReCoRD (Zhang et al., 2018), RACE-C (Liang et al., 2019), HeadQA (Vilares and Gómez-Rodríguez, 2019), MMMLU (Hendrycks et al., 2020), ReClor (Yu et al., 2020), Quail (Rogers et al., 2020), OneStopQA (Cui et al., 2021), MCScript (Ostermann et al., 2018), MCScript 2.0 (Ostermann et al., 2019), CosmosQA (Huang et al., 2019), DREAM (Sun et al., 2019), ProcessBank (Berant et al., 2014), PROST (Aroca-Ouellette et al., 2021), StrategyQA (Geva et al., 2021), PubmedQA (Jin et al., 2019), QAConv (Wu et al., 2022), and TweetQA (Xiong et al., 2019) as out-of-domain evaluation sets.

heading: Appendix B Implementation Details
text: B.1 Details of Evaluated Datasets We evaluate QG metrics on three datasets, SQuAD (v1) (Rajpurkar et al., 2016) (under CC BY-SA 4.0 license), Natural Questions (Kwiatkowski et al., 2019) (under Creative Commons Share-Alike 3.0 license), and MS-MARCO (Bonifacio et al., 2021) (fully open-source, no license) datasets. Table 4 illustrates the number of samples in training and evaluation sets.
B.2 Hyper-parameters for Fine-tuning QG Models All models are trained on NVIDIA A100-SXM4-40GB GPUs. T5 (Raffel et al., 2020) is under Apache License 2.0. GPT2 (Radford et al., 2019) is under modified MIT License. We use AdamW optimiser (Loshchilov and Hutter, 2019), used in several previous works (Mohammadshahi et al., 2019; Devlin et al., 2019; Mohammadshahi and Henderson, 2021b,a, 2020).

heading: Appendix C Instruction of Human Evaluation
text: Annotators are asked to evaluate the quality of a question, given the context and answer span. An input example is shown in Figure 7. They should provide 3 scores for grammaticality, answerability, and relevance. For grammar, the syntactic structure of the sentence is evaluated. They should score 3 for "no grammatical errors", 2 for "not grammatically acceptable but able to get the meaning", and 1 for "unacceptable" questions. For answerability, the score should express the completeness of the candidate question and its consistency with the given answer. So, annotators are required to consider two criteria while scoring; the question should contain question words (e.g. wh-words) and necessary entities, and it should not include the answer. They should score 3, if the question contains all important information, and is consistent with the answer. Score 2 is for the cases, in which some important information is missing in the question or it contains the answer. They should score 1 if all important information is missing in the question and the question is not consistent with the answer. For relevance, annotators should score the relatedness of the question to the answer, given the context. They should score 2 if the question is answerable by the context and related to the given answer. They should score 1, if the question is out-of-context, or not related to the given answer. We sample 600 examples (200 for each dataset) from the evaluation sets of SQuAD (v1), NQ, and MS-MARCO. Samples are shuffled and anonymized. All annotators are fluent English speakers.

heading: Appendix D Correlation with Human Evaluation

heading: Appendix E Re-Ranking with RQUGE

heading: Appendix F Adversarial Evaluation Set
text: As discussed in Section 5.3, we create positive samples by two mechanisms:
• Back-Translation. Translating the reference question to an intermediate language, then translating it back to English. We apply Marian model (Junczys-Dowmunt et al., 2018), and use Chinese and French as intermediate languages, as Marian model has reasonable performance for these language directions.
• Quora Paraphrasing. We first train a T5-small (Raffel et al., 2020) model on Quora paraphrasing dataset, 25 and use it for paraphrasing the reference question.
Outputs of both methods are questions that are semantically similar to the reference questions with a few lexical differences. For the negative samples, as shown in Table 2, we apply the following methods:
• Negation. We first scan the reference question to find auxiliary and modal verbs. Then, we randomly either add not to the sentence or replace the verb with its antonyms by using WordNet (Miller, 1995) inside the NLTK package (Bird et al., 2009).
• Reverse Gender. The reference question is first scanned to find pronouns, and then pronouns are replaced with pronouns with the opposite gender.
25https://www.kaggle.com/competitions/quora-question-pairs/data
• Swap Entity. Stanza (Qi et al., 2020b) named-entity recognition model is applied to the reference question and the context. Then, we randomly select one extracted entity of the reference question and replace it with a random entity of the context with the same entity type.

heading: Appendix G Implementation Details of Fine-tuning QA models
text: All models are trained on NVIDIA A100-SXM4-40GB GPUs.

heading: Appendix H Error Analysis
text: We investigate on cases, in which there is a substantial difference between the human evaluation and RQUGE score. The errors are categorised into syntactic and knowledge-based types, as shown in Table 10. For the syntactic error, RQUGE sometimes computes unacceptable scores for sentences that either miss the question word (e.g. wh-words) or have wrong word order, as QA module of RQUGE focuses more on the semantic aspect of the candidate question to predict the answer span. For the knowledge-based mistakes, RQUGE requires further domain specific and commonsense knowledge to compute the correct score e.g. full moon is instant, not period of time as illustrated in the sample of Table 10. As shown in Table 11, RQUGE computes wrong values for some samples in "reversing gender" and "swapping entities" categories of evaluation set in Section 5.3. These errors shows the limitations of RQUGE metric, and lead the future work to apply larger and better QA and span scorer modules.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitation section and Appendix H.
A2. Did you discuss any potential risks of your work? Not applicable. We provide an evaluation metric for QG task. limitations are provided in Conlusion and Limitations sections.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: We include our contributions in both abstract and introduction sections. Specifically, we provide them at the end of introduction section.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? For the model, we use T5 and GPT2 in our paper (section 3 and section 5). For the metric, we use different automatic evaluation metric (sections 2,4,5). For the dataset, we use SQuAD, NQ, and MS-MARCO (sections 4 and 5). We also create the human annotation for the evaluation (sections 4 and 5, Appendix C).

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sections 4, 2, 3.1 and 3.2.

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Appendices B.1 and B.2

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Since our annotations are created from SQuAD, NQ, and MS-MARCO datasets, we use the same license for the distribution of our human annotations (Appendix C). For pre-trained models, we use publicly available models (Appendix B.2).
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We use SQuAD, NQ, and MS-MARCO datasets, which organizers already checked these concerns before making them available.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendices B.1, C, and F.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendices B.1, C, and F.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 5.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendices B.2 and G.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendices B.2 and G.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Not applicable. For the evaluation of metrics, we use pearson, spearman, and kendall metrics to find the correlation with the human judgment. For QA experiment, we run F1 and EM metrics for the evaluation. In both cases, there were a significant different between our model and previous works. For both QG and QA models, results are single-run, as mentioned in the paper.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 and Appendix F.
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? We collect the data for the evaluation of question generation metrics: Section 4 and Appendix C.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.

heading: 3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? The annotators were volunteer fluent English speaking students (mentioned in Appendix C), and we created our internal website to get the annotations.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. We inform them that the data will be used for the evaluation of question generation task (Appendix C).

heading: 3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: We use the same protocol as previous work (which was approved) in the question generation task.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: It gives us directions for future improvement of RQUGE metric