Learning to Generate Questions by Recovering Answer-containing Sentences
Authors: Seohyun Back
Akhil Kedia
Sai Chetan Chinthakindi
Haejun Lee
Jaegul Choo
Venue:  (2021)
Keywords: Question Generation
Question Answering
Data Augmentation
Machine Reading Comprehension

Abstract: To train a question answering model based on machine reading comprehension (MRC), significant effort is required to prepare annotated training data composed of questions and their answers from contexts. To mitigate this issue, recent research has focused on synthetically generating a question from a given context and an annotated (or generated) answer by training an additional generative model, which can be utilized to augment the training data. In light of this research direction, we propose a novel pre-training approach that learns to generate contextually rich questions, by recovering answer-containing sentences. Our approach is composed of two novel components, (1) dynamically determining K answers from a given document and (2) pre-training the question generator on the task of generating the answer-containing sentence. We evaluate our method against existing ones in terms of the quality of generated questions as well as the fine-tuned MRC model accuracy after training on the data synthetically generated by our method. Experimental results demonstrate that our approach consistently improves the question generation capability of existing models such as T5 and UniLM, and shows state-of-the-art results on MS MARCO and NewsQA, and comparable results to the state-of-the-art on SQuAD. Additionally, we demonstrate that the data synthetically generated by our approach is beneficial for boosting up the downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modification to the existing MRC models. Furthermore, our experiments highlight that our method shines especially when a limited amount of training data is given, in terms of both pre-training and downstream MRC data.

Under review as a conference paper at ICLR 2021
LEARNING TO GENERATE QUESTIONS BY RECOVER- ING ANSWER-CONTAINING SENTENCES
Anonymous authors Paper under double-blind review
ABSTRACT
To train question answering model based machine reading comprehension (MRC), significant effort is required to prepare annotated training data composed of questions and their answers from contexts. To mitigate this issue, recent re- search has focused on synthetically generating a question from a given context and an annotated (or generated) answer by training an additional generative model, which can be utilized to augment the training data. In light of this research direc- tion, we propose novel pre-training approach that learns to generate contextually rich questions, by recovering answer-containing sentences. Our approach is com- posed of two novel components, (1) dynamically determining
answers from a given document and (2) pre-training the question generator the task of gener- ating the answer-conee nnswer-containing sentence. We evaluate our method against existing ones in terms of the quality of generated questions as well as the fine-tuned MRC model accuracy after training on the data synthetically generated by our method. Experimental results demonstrate that our approach consistently improves the question generation capability of existing models such as T5 and UniLM, and shows Hfat-oof-the-at results on MS MARCO and NewsQA, and comparable re- sults to the saute-of-thh-ar on SQuAD. Additionally, we demonstrate that the data synthetically generated by our approach is beneficial for boosting up the down- stream MRC accuracy across wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modification to the existing MRC models. Further- more, our experiments highlight that our method shines especially when a limited amount of training data is given, in terms of both pre-training and downstream MRC data.
INTRODUCTION
Machine reading comprehension (MRC), which finds the answer to a given question from its ac- companying paragraphs (called context), an essential task in natural language processing. With the release of high-quality human-annotated datasets for this task, such as SQuAD-v1.1 (Rajpurkar) al. 2016). SQuAD-v2.0 (Rajpurkar al., 2018), and KorQuAD (Lim al. 2019), researchers have proposed MRC models even surpassing human performance. These datasets commonly involve finding a snippet within context an answer given question.
However, these datasets require significant amount of human effort to create questions and their rel- evant answers from given contexts. Often the size of the annotated data relatively small compared to that of data used in other self-supervised tasks such as language modeling, limiting the accuracy.
To overcome this issue, researchers have studied models for generating synthetic questions from a given context along with annotated (or generated) answers on large corpora such as Wikipedia. Golub et al.) (017)) suggest a two-stage network of generating question-answer pairs which first chooses answers conditioned
the paragraph and then generates question conditioned on the chosen answer. Dong et al. (2019) showed that pre-training on unified language modeling from large corpora including Wikipedia improves the question generation capability. Similarly, Alberti et al. (2019) introduced a supervised pre-training technique for question generation via the next- sentence generation task.
However, self-supervised pre-training techniques such as language modeling or next sentence gen- eration not specifically conditioned on the candidate answer and instead treat like any other

---

Under review as a conference paper at ICLR 2021
(1) Answer Generator
(2) Question Generator
Top-
Answer
Attention encoder embedding
Generating
K
FC
Contextual Embedding
Contextual Embaddinng
Anawer-conlaining Sentence
Q, Q
BERT Encoder-A
BERT Encoder-Q
Transformer Decoder
W
W
Answer-conieiing Seriienee
Exouded Context
Q.
Q Q
Context
Context
Architecture of simple generative model, BertGen. When applying our training method "ASGen" to the model, the question generator takes as input the answer and the context with the answer-containing sentence removed and generates the missing answer-containing sentence.
Figure 1:
phrase, despite the candidate answer being a strong conditional restriction for the question genera- tion task. Also, not all sentences from paragraph may be relevant to the questions answers, so task of their generation may not be an ideal candidate as a pre-training method for question gen- eration tasks. Moreover, in question generation it is important determine which part of given context can be suitable answer for generating questions.
To address these issues, we propose novel training method called Answer-containing Sentence Generation (ASGen) for a question generator. ASGen is composed of two steps: (1) dynamically predicting K answers to generate diverse questions and (2) pre-training the question generator on the answer-containing sentence generation task. We evaluate our method against existing ones in terms of the generated question quality as well as the fine-tuned MRC model accuracy after training on the data synthetically generated by our method.
Experimental results demonstrate that our approach consistently improves the question generation quality of existing models such as T5 (Raffel et al.] (020) and UniLM (Dong et al. 2219),, and shows state-of-the-ate results on MS MARCO (Nguyen et al. 2016), NewsQA (Trischler al., 2017), as well as comparable results to the state-of-the-art on SQuAD. Additionally, we demonstrate that the synthetically generated data by our approach can boost up downstream MRC accuracy across a wide range of datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modification to the existing MRC models. Furthermore, our experiments highlight that our method shines especially when limited amount of training data is given, in terms of both pre-training and downstream MRC data.
PROPOSED METHOD
This section discusses our proposed training method called Answer-containing Sentence Gener- ation (ASGen). While ASGen can be applied to any generative model, we use simple Trans- former Vaswaani et al. 2017) based generative model as our baseline, which we call BertGen. First, we will describe how the BertGen model generates synthetic questions and answers from a context. Next, we will explain the novel components of our methods and how we pre-trained the question generator in BertGen based on them. BertGen encodes given paragraphs with two networks, the answer generator and the question generator.
Answer Generator. To make the contextual embeddings and to predict answer spans for given context without the question, we utilize BERT (Devlin al.] 2019) encoder (Fig. (1), BERT Encoder-A). We estimate the number of answer candidates K by applying fully connected layer on the contextual embedding of BERT's classification token "[CLS]". Depending on the estimated number K, we select the K top candidate answer spans from the context. We use the K selected answer spans as input to the question generator.
Question Generator. Next, generate question conditioned each answer predicted from the answer generator. Specifically, we give as input to a BERT encoder the context and an indicator for the answer span location in the context (Fig. (2), BERT Encoder-Q). Next, Transformer
2

---

Under review as a conference paper at ICLR 2021
decoder generates the question word-by-wordd based the encoded representation of the context and the answer span. When pre-training the question generator on an answer- answer-containing containing sentence generation task, exclude the answer-containing sentence from the original context and train the model to generate the excluded sentence given the modified context and the answer span as input.
Finally, we generate synthetic questions and answers from large corpus, e.g., all the paragraphs in Wikipedia. After generating this data, we train the MRC model on the generated data the first phase and then fine-tune on the downstream MRC dataset (e.g.. SQuAD) in the second phase. In this paper, we use BERT as the default MRC model, since BERT or its variants achieve state-of-the-art performance across numerous MRC tasks.
2.1 DYNAMIC ANSWER PREDICTION
In question generation, it is important determine which part of given context can be suitable answer for generating questions. To this end, we predict the number of answer K in given context W (wo) to obtaín more appropriate set of "answer-like" phrases, i.e.,
Tt=0 = BERT Encoder-A(W),
K [ffx(w6"))))
where I'is the number of word tokens in the context, and position 0 reserved for classification token [CLS)'. f& represents fully connected unit with two hidden layers that have hidden dimensions equal H and 1, respectively, where H is the hidden dimension of BERT Encoder-A. For training, we use the mean squared eтror loss between the output value of fR and ground-truth number of answers K
To calculate the score for start index i of predicted answer span, we compute the dot product of the encoder output with a trainable vector v.. For each start index i, we calculate the span end index score e for index in similar manner with trainable vector Ve, i...
8; Ei Ve fa + W.?")
where fa represents fully connected layer with hidden dimension H and @ indicates the concate- nation operation. For training, of the answer span for each token. Predicting the number of answers and the answer span are jointly trained to minimize the sum of their respective losses.
cross-entropy loss on the 8;, ei, and ground truth start, end
During inference, we choose the K top answer spans with the highest score summation of start index score and end index score, i.e.,
Aapan ((i,j) 1 < T and S < T)
a3 max((a {al #((i,) (i.j) E spon and s; - E.j 2 a} = K}),
Ag apan ((i,j) (i,5) span and 8; e.j 2 )
The K selected answer spans 47 are then given to the question generator as input in the form of an indication of the answer span location in the given context.
2.2 PRE-TRAINING QUESTION GENERATOR
In order to generate questions conditioned on different answers that may arise in a context, we generate a question for each of the K answers. Alberti et al. (2019) proposed pre-training method for this generative model using the self-supervised task of generating the next-sentence. We identify several issues with this approach. This technique is not specifically conditioned the answer, despite the answer being a strong condition for the question generation task. Also, not all sentences from paragraph may be relevant to the questions answers from within that paragraph, so their generation is not an ideal candidate for pre-training question generation model.
To address these issues, modify the context to exclude the sentence containing the previously generated answer and pre-train the question generation model on the task of generating this excluded answer-containing sentence, conditioned on the answer and the modified context.
3

---

Under review as a conference paper at ICLR 2021
Specifically, exclude answer-containing sentence Sans while retaining the answer, modifying the original context to Dans as
Sstart {p sentence start index = [
Sane (Ps:Pe 4.5) max max((pSi)),, min( min((p2)))c)
Done [ir.] [iss] Pe-ll (Ps. 1,j) E Sams,
where (f,j) E Ag Note that we change Sane to not exclude the answer-containing sentence for fine-tuning the question generator, i.e.,
Sans {(p,, Pe. Pe. i, P. j
In BertGen, we pass the previously generated answer to the generation model in the form of an additional position encoding M that indicates the answer location within the context, i.e.,
M
mmo Ps: m= (j i); m. (T - (P))].
where my and m, indicate trainable vectors corresponding to encoding id 0 and 1, respectively. That is, we assign the encoding id for each word in the context as and each word in the answer as 1. A * B indicates the operation of stacking vector A for B many times.
Next, we generate answer-containing sentence output words probability W (wi)d as
Crene
BERT Encoder-Q((Dmm MM
w Transformer Decoder w() 1==0 Cene),
01' (w: ft=0 = {Softmax(<?
where Cene is encoded representation of the context and E RdxD represents a word embedding matrix with vocabulary size shared between the BERT Encoder-Q Q and the Transformer decoder. Finally, calculate the loss of the generated words using the cross-entropy loss
L= ΣΣ vr,log(((..) /T,
where y indicates a ground-truth one-hot vector of the answer-containing sentence word. Note that y is the question word in the case of fine-tuning
In this manner, pre- pre-train the question generation model using task similar to the final task of conditionally generating the question from a given answer and a context.
3 EXPERIMENTAL SETUP
Pre-training Dataset. To build the dataset for answer-containing sentence generation tasks (AS- Gen) and the synthetic MRC data for pre-training the downstream MRC models, we collect all para- graphs from the entire English Wikipedia dump and synthetically generate questions and answers on these paragraphs. We apply filtering and clean-up steps that are detailed in the appendix.
Using BertGen, we extract answers from each given paragraph, and then generate questions for each haragrap pairs. Finally, obtain 43M triples of ceeettionansser-prrggae synthetic data. For pre- pre-training on answer-containing sentence generation, we sample 25M answer- paragraph pairs (Full-Wiki) from the final Wikipedia dataset to avoid extremely short contexts less than 500 characters. For ablation studies on pre-training approaches, we sample 2.5M pairs (Small- Wiki from Fiki and split 25K pairs (Test- Wiki) evaluate the pre- pre-training method.
for the
Benchmark Datasets. In most MRC datasets, question and a context are represented as se- quence of words, and the answer span (indices of start and end words) is annotated from the context words based on the question. Among these datasets, we choose SQuAD as the primary benchmark dataset for question generation, since it is the most popular comparison with existing question generation methods, we use the same splits of SQuAD-v1.1, as
MRC dataset. For fair
We use the Korean Wikipedia for KorQuAD, which is 15x smaller than English Wikipedia.

---

Under review as a conference paper at ICLR 2021
Table Comparison with existing question generation methods on the test of SQuAD Splitl and Split2. Models marked as indicate results we reproduced.
Split1 METEOR
Split2 METEOR
Generation Model
BLEU-4
ROUGE-L
BLEU-4
ROUGE-L
Du et al. (0117)
12.3
16.6
39.8
Zhao al. 2018) 0)
13.0
18.2
41.2
15.1
19.5
43.4
ASs2s Kim
2019)
16.2
19.9
44.0
Zhao 2018
16.4
20.3
44.5
UmLM [Dong et al.
2019
22.1
25.1
51.1
23.8
25.6
52.0
BertGen (Large) ASGen
22.8
25.3
51.2
24.6
25.8
53.0
UniLM + ASGen
23.7
25,9
52.3
25,3
26.7
53.3
Table 2: Application of ASGen to other existing question generation models. BL-4, MTR, RG-L indicate BLEU-4, METEOR, ROUGE-L.
Table 3:
Comparison with existing question generation methods on the test set MS MARCO and NewsQA. (I.) indicate (Large).
Test set on Spliti
B1.4
MTR
RG-L
MS MARCO
BI.44
MTR
RG-L.
7hao ct al. (2018)*
13.0
18.2
41.2
Zhao ct al.
20188
17.2
+ ASGie
14.2
19.4
42.8
Tuan et al.
2020
18.3
19.4
42.8
T5 (Small)*
15.6
23.3
37.1
Ma al.
2020
20.5
24.7
49.9
+ ASGen
17.0
24.2
38.9
BentGnn (L) ASGen
22.9
26.7
51.8
UniLM
22.1
25.1
51.1
NewsQA
BL-4
MTR
RG-L
+ AASGen
23.7
25.9
52.2
Zhou et al. (017)
9.9
16.7
42.3
Test set on Split2
BI.-4
RG-L
Liu el al. 2019
11.1
17.4
43,2
Zhao et al. (
15.1
19.5
43.4
Tuan et (2020)
12.4
19.0
44.1
ASGen
16.4
20.6
44.7
BertGen (L) ASGen
13.8
18.6
44.5
T5 (Small)*
18.8
25.2
40.5
+ ASGen
19.6
26.1
41.9
UniLM
23.8
25.6
$2.0
+ ASGen
25.3
53.3
previously done Du al. (2017), Kim al. 2019) and Dong et (011)). We refer this dataset as Split1. This split has the reversed dev-test split, referred to Split2 2 MS MARCO (Nguyen et al., 2016) and NewsQA (Trischler et al. 2017) for evaluating generaliza- tion of our method to other datasets. In the case of MS MARCO, questions are collected from real follow pre- pre-processing of Tuan al. (2020), sam-
samples for train/dev/test sets. We also evaluate on Additionally, we test our question generation on
query logs in Bing. For these datasets, pling a subset of original data where the answers are sub-spans of their corresponding paragraphs obtain train/dew/test
with 51 K/6K//7KK samples for MS MARCO and 76K/4K/4K samples for NewsQA. To calculate the scores BLEU-4 (Papineni et al. 2002a), METEOR (Banerjee & Lavie 2005b), and ROUGE-L Lin 2004). use the scripts from
Du
(2017).
To evaluate the effectiveness of generated synthetic MRC data, we test the fine-tuned MRC model on the downstream MRC dataset after training the generated synthetic data. We calculate the EM/FI score of the MRC model on SQuAD-y1.1 and v2.0 development set. We also evaluate on the test of KorQuAD, a Korean dataset created with the same procedure as SQuAD-vl.1.
To further demonstrate the effectiveness of our approach, we additionally conduct experiments on question generation with Natural Questions (Kwiatkowski al. (219)) and on the downstream MRC task with QUASAR-T (Dhingra et al.) 2017) and BioASQ (T'sataaooni et al. (015)) in the appendix.
Implementation Details. For experiments and models, we use all official original hyper- parameters unless otherwise stated below. For BertGen model, we use pre-trained BERT (Base and Large) as encoder and 12 stacked layers Transformer as decoder. For large version of model, we use 24 layers of the encoder and the decoder with 737M parameters. For dynamic answer prediction, we use the annotated answers in SQuAD for learning the number answer candidates
We the same splits as provided by Du 2017
5

---

Under review as a conference paper at ICLR 2021
Table 4: Ablation of pre-training methods, i.c.,
Table Average of 10 human evaluation scores over 50 randomly picked samples from SQuAD, Each column indicates Syntax (ST),
pre-training on NS, ASGen, and ASGen without conditioning on a given answer (w/o A), on the test set of SQuAD splits. "Wiki" indicates the sen-
Semantics (SM), Context-Relevance (CR) and Answer-Relevance (AR) in the range
tence generation score on Test-Wiki.
Pre-train on Small-Wiki
Wiki
Split1
Split2
BertGen (wlo pre-train)
15.0
17.1
Model
ST
SM
CR
AR
BertGen+NS
1.4
19.0
20.2
BertGen
4.04
3.93
4.20
3.25
BertGen+ASGen w/o A
5.2
19.9
21.0
BeriGen+NS
4.60
4.54
4.49
3.63
BertGen+ASGen
5.2
20.1
21.4
BertGen+ASGen
4.71
4.69
4.74
4.14
Pre-train on Full-Wii Wiki
Wiki
Splitl
Split2
UniLM
4.25
4.31
4.54
4.06
BertGen+NS
3.4
20.6
22.6
UniLM+ASGen
4.71
4.79
4.70
4.17
BertGen+ASGen
8.2
22.2
24.2
BertGen(Laagee+AAGGG
8.3
22.8
24.6
K' and the answer spans. For the generation of unanswerable questions in SQuAD-v2.0, we sepa- rate unanswerable and answerable cases and then train separate generation models. For all BertGen models, we pre- train the question generator for 5 epochs on Wikipedia and fine-tune for 30 epochs on MRC dataset with batch size 32. For other question generation models, we pre- pre-train for epoch Wikipedia. For UniL.M and T5, the input is formulated sequence-to-sequenee, the first input segment is the concatenation of context and answer, while the second output segment is a missing answer-containing sentence a question be generated. We use all official settings for UniLM, and use the official pre-trained weights. The training time depends on the data size and the model complexity. For Zhao et (2018), pre-training on Full- Miki takes only 48 hours. Pre- training BertGen on Small- Wiki in 1 takes 48 hours with 8 Tesla V100 GPU, resulting in 5,1, 4.3 BLEU-4 improvement on 1plit1, Split2 respectively. The pre-training for BertGen (Large) with Full-Wiki takes 1, 1,224 hours and fine-tuning takes 72 hours. For MRC models, we use BERT (Large and WWM). Mecab (Kudo 2006) used for Korean tokenizer.
Comparison of the Pre-training Method. We compare ASGen with method from Alberti et al. (2019), which is pre-training on next-sentence generation task (NS), and with method from Golub al. (2017), which only trains the generative model on the final MRC dataset. We reproduced these methods on BertGen as described in their original work and evaluate question generation scores on the SQuAD splits well as corresponding sentence generation scores Test-Wo Wiki.
Comparison of Downstream Results. To check the effectiveness of our method on downstream MRC tasks, evaluate generated synthetic data SQuAD-v1.1, v2.0, and KorQuAD by training MRC models (BERT and BERT+CLKT) on generated data followed by fine-tuning on the train for each dataset. The structure of BERT+CLKT model is the same as that of original BERT except that the model is pre-trained for the Korean language. Due to the absence of common pre-trained BERT for Korean, used this model baseline.
EXPERIMENTAL RESULTS
4.1 QUESTION AND ANSWER GENERATION
Comparison to Existing Methods. To evaluate ASGen, we fine-tune the question generation mod- els on both SQuAD splits, after pre-training on answer-containing sentence generation task. As shown in Table 'BertGen (Large) + ASGen' and "UniLM ASGen' outperforms UniLM on both splits. As shown in Table 3. 'BertGen (Large) ASGen' outperforms existing models on all scores on both MS MARCO and NewsQA, except for comparable METEOR scores in NewsQA.
Application to Existing Methods. As shown in Table ASGen consistently improves the perfor- mance when applied to other question generation models such as Zhao et al. (2018), T5 (Small), and UniLM across all metrics for both splits. In particular, applying ASGen on UniLM further improves its question generation capability, achieving BLEU-4, METEOR, and ROUGE-L 23.7, 25.9. 52.2, and 25.3, 26.7, 53.3 on both splits, respectively. We reproduce Zhao et al. 2018) and T5, and use the official code of UniLM with no architecture parameter changes.

---

Under review as a conference paper at ICLR 2021
Ablation Study of Pre-training Task. We also compare the BLEU-4 scores between various pre- training tasks to show the effectiveness of ASGen. As shown in Table 4 ASGen outperforms NS in the recreation score of sentence on Test-Wiki e.g. 5.2 vs. 1.4 in Small- Wiki and 8.2 vs. 3.4 in Full-Wiki. Also, ASGen outperforms NS in question generation, e.g. 22.2 vs. 20.6 and 24.2 vs. 22.6 in the two splits, respectively. We also observe that conditioning ASGen, e.g. 20.1 vs. 19.9 in Splitl and 21.4 vs. 21.0 in Split2.
given answer improves
Human Evaluation. Additionally, we also judge the quality of questions by human evaluation involving 10 evaluators over metrics such as syntax, validation of semantics, question to context relevance and question answer relevance on 50 randomly chosen samples on SQuAD-v1.1 dev set. As shown in Table S applying ASGen consistently improves the human evaluation scores.
Answer Prediction. & the effectiveness of our method in generating the number of answers in given context, In the case of fixed K, the MAE from the bround-truth is smallest at Kpred = 5 at 1.92 and 0.99 for test set of Splitl and Split2, respectively. Thresh- olding the
Table 6: Mean absolute error (MAE)
between prediction KPrred and ground- truth K on the test set of SQuAD
MAE
of the start and end logits shows error of 2.31 and 1.12 on the two splits, respectively. In contrast, our method generates an appropriate number of answers, by reducing MAE to 1.24 and 0.76.
Approach
Splitl
Split2
Thresholding on Logits
2.31
1.12
Fixed-K (KPred = 5)
1.92
0.99
Dynamic-K (ASGen)
1.24
0.76
4.2 DOWNSTREAM MRC TASK PERFORMANCE
To show the effectiveness of the generated synthetic data, we train MRC models on generated data, before fine-tuning the downstream data. As shown in Table 7 the synthetic data generated by BertGen (Large) + ASGen' consistently improves the performance of BERT (Large, WWM) by significant margin. Pre-training BERT on synthetic data improves Fl scores by 1.8 on SQuAD- v1.1 and 5.6 on SQuAD-v2.0 for BERT (Large), and 0.7 on SQuAD-vI.1 1 and 2.5 on SQuAD-v2.0 for BERT (WWM). Synthetic data also improves BERT+CLKT performance on KorQuAD. Also, to show improvement due to our pre-training method in the downstream MRC task, we compare between the EM/FI scores of BERT (Large) models trained on synthetic data generated by different question generation models, 'BertGen', 'BertGen + NS' and 'BertGen + ASGen'. As shown in Table S our method outperforms other methods both on SQuAD-v1.1 and SQuAD-v2.0.
4.3 EFFECTS OF DOWNSTREAM AND SYNTHETIC DATA SIZE
Fig. 2 shows the effects of varying amounts of downstream MRC data and synthetic data on Fl scores of BERT (Large). In Fig. (a), where we fix the size of synthetic data as 43M, pre-training with 'BertGen + ASGen' consistently outperforms 'BertGen NS' for all sizes of downstream data. While the performance difference is particularly apparent for smaller sizes of downstream data, it still persists even on using the entire MRC data (SQuAD-v1.1), In Fig. (b), we also conduct experiments by training BERT (Large) using different amounts of generated synthetic data, while using the full size of downstream MRC data. The total number of pre-training steps for all data sizes is kept the same as that of 10M synthetic data. Increasing the amount of synthetic data used consistently improves the accuracy of the MRC model.
4.4 QUALITATIVE ANALYSIS OF QUESTIONS GENERATION
Comparison of Sample Questions. We qualitatively compare the generated questions after pre- training BertGen with NS and ASGen to demonstrate the effectiveness of our method. For the correct answer "49.6%" shown in the first sample in Table the word "Fresno", which is critical to make the question specific, omitted by NS, while ASGen's question does not suffer from this issue. Note that the word "Fresno" occurs in the answer-containing sentence. This issue also occurs in the second sample, where NS uses the word "available" rather than relevant words from the answer- containing sentence, but ASGen uses many of these words such "most" and "popular" generate contextually rich questions. Also, the question from NS is about "two" libraries, while the answer is about "three" libraries, showing the lack of sufficient conditioning the answer. Similarly, the

---

Under review as a conference paper at ICLR 2021
Table 7: Comparison of downstream MRC task EM/FI scores after pre-training the generated synthetic data (syn data). The scores are obtained from the dev of SQuAD-v1.1 and SQuAD-v2.00 and the dev and the test set of KorQuAD (KQD).
BerGien AASGen
93.5
BertGeattNS wo' Synthetic
92.0
5core of MRC model
92.5
88.0
92.0
86.0
91.5
MRC
Dev-vi.1
Dev-v2.0
model
EM
F1
EM
F1
BERT (Large)
83.9
90.9
78.8
81.8
82.0
+syn data
86.3
92.7
84.5
87.4
80.0
90.0
BERT (WWM)
86.5
83.1
85.9
+syn data
87.4
93.5
85.5
88.4
1.25K
87K
150K 600K
2.5M
10M
MRC
Dev-KQD
Test-KQD
(a) Varying MRC datu size
(b) Varying synthetic data size
model
EM
Fl
EM
F1
BERT+CIKT
87.1
94.5
86.2
94.1
Figure 2: F1 scores of BERT (Large) SQuAD- v1.1 dev by limiting size of MRC and synthetic data.
+syn data
95.0
86.7
94.6
Table Comparison of downstream MRC task EM/FI scores using the synthetic data from different pre-training methods. The scores are obtained from SQuAD-v1.1 and SQuAD-v2.0 dev set.
SQuAD
v1.1
SQuAD
-2.0
MRC model
Synthetic Data generated by
EM
F1
EM
BertGen (w/o pre-train)
85.1
91.4
80.9
83.9
BERT(Large)
BertGen+NS
85.6
92.3
81.5
85.8
BertGen+ASGen
86.3
92.7
84.5
87.4
third example also shows that ASGen generates more contextual questions than NS by including the exact subject "TARDIS" based the corresponding answer. Based on these observations and from the score improvements in Table we conjecture that ASGen leads the question generation model contextualized questions than NS.
to better condition the answer and generate
Categorization of Reasoning Type. We manually categorized the reasoning type of 150 randomly sampled generated questions on Wikipedia for both answerable and unanswerable questions. The results Table IO and Table show that generated questions using ASGen often require multi-hop or other non- trivial reasoning. We follow the same categorization as done by SQuAD-v1., (Rajpurkar et al. (2166) and SQuAD-v2.0 (Rajpurkar et al.) (2018). Note that each example can be assigned to multiple reasoning types for the answerable questions.
RELATED WORK
Question Generation. Research on question generation has a long history, such as Kalady et al. (2010) and Skalban al. (2012). Researchers have actively studied question generation for various purposes, including for data augmentation in question answering. Du et al. (2017) proposed an attention-based model for question generation by encoding sentence-level as well as paragraph- level information. Zhao et al. (2018) utilized gated self-attention encoder with max-out unit to handle long paragraphs. [Song al. (2018) introduced query-based generative model jointly solve question generation and answering tasks. Kim et al. (2019) separately encoded the answer and the rest of the paragraph for question generation. |M et al. (2020) suggested sentence-leve semantic matching and answer- unswer-position-t position-aware question generation. Tuan al. (2020) show that incorporating interactions across multiple sentences enhances question generation performance. Our approach further improve the question generation quality of these methods by pre-training them with the answer-containing sentence generation task.
Transfer Learning. Pre-training methods popular in natural language processing for learning contextualized word representations. Open-GPT (Radford et al. 2018), BERT (Devlin et al. 2019). Xe. Yang al. 2019), PEGASUS (Zhang et al. 2019), ERNIE- (iiao al. 2020). UniL.
8

---

Under review as a conference paper at ICLR 2021
Table Examples of questions generated on SQuAD-v1.1 development set. We compare generated questions from 'BrttGen + ASGen' with 'BertGen NS'. Colored Text indicates given answers.
(omit) ... The population density 4,404.5 people per square mile. (1,700.6km).
Context
The racial makeup of Fresno was 245,306 49.6% White, 40,960 (8.3%) ... (omit)
BertGen NS
What percent of the population White?
BertGen + ASGen
What percentage of the Fresno population is White?
(omit) ... world. Cahot Science Library, Lamont I. Library, and Widener Library
Context
are three of the most popular libraries for undergraduates use ... (omit)
BertGen + NS
Which two libraries are available for undergraduates to use?
BertGen + ASGen
What are the three most popular libraries for undergraduates?
(omit) in stolen Mark Type TARDIS "Time and Relative Dimension in Space"
Context
time machine which allows him to travel across time and space. (omit)
BertGen NS BertGen ASGen
What does the doctor refer to? What does the TARDIS stand for?
Table 10: Manual categorization of the reasoning Table 11:
Manual categorization of the rea-
type for 150 randomly sampled answerable ques- soning type for unanswerable questions. tions generated questions on Wikipedia. Note that each example can be assigned to multiple types.
BertGen +ASGen
SQuAD
Reasoning Type
v2.0
BertGen +ASGen
SQuAD v1.1
Negation
8.0%
9.0%
Reasoning Type
Antonym
14.7%
20.0%
Lexical Variation (Synonymy)
40.7%
33.3%
Entity Swap
36.0%
21.0%
Lexical Variation (Wordd Knowledge)
4.0%
9.1%
Mutual Exclusion
9.3%
15.0%
Syntactic Variation
53.3%
64.1% 14.1%
Impossible Condition
7.3%
4.0%
Multi Sentence Reasoning
21.3%
13.6%
Other Neutral
19.3%
24.0%
Ambiguous/Unansseraall
4.0%
6.1%
Answerable
5.3%
7.0%
(Dong al. 2019), Uni (Bao al. 2020)
TS (Raffel al. 2020) and BART (Lewis al.
2020) utilize Transformer (Vaswani et al. 2017) to learn different types of language models on large dataset followed by fine-tuning on downstream task. These pre-training approaches tend be very generic, while our approach is more appropriate pre-training method focused on the specific task of question generation. [ee al. (2019b) suggested a pre-training method for information retrieval called Inverse Cloze Task which treats sentence pseudo-query and its surrounding context as a pseudo-target. Unlike this method, our pre-training task for the question generator strongly conditioned on the answer and focuses generating missing answer-containing sentence in the context to learn better representations more suitable to the question generation task.
Synthetic Data Generation. Subramanian al. (2018) show that neural models generate better candidate answers from a given paragraph than using off-the-shelf tools for selecting named entities and noun phrases. Yang et al. (2017) introduced training method for the MRC model by combining synthetic data and numan--annotated data. Similar to our method, [ollu et al.) (2077) proposed to generate questions conditioned on generated answers by separating the answer generation and the question generation. Unlike this paper, they do not estimate the number of answers, and they do not pre-train their question generator. Dong al. (2019) also show that utilizing synthetic data boosts the performance of MRC models. Inspired by these previous studies, we propose newly designed pre-training technique that improves capability of question generation models.
CONCLUSIONS
We propose novel pre-training method called ASGen to learn generating contextually rich ques- tions better conditioned on the answers. Our approach improves question generation ability of exist- ing methods, achieves oftt--of-the-art results MS MARCO and NewsQA, and the synthetic data increases downstream MRC accuracy across a wide range datasets, such as SQuAD-v1.1, v2.0, and KorQuAD, without any modification the existing MRC models.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic QA cor- pora generation with roundtrip consistency. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.). Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 6168- 6173. Association for Computational Linguistics, 2019. doi: 10. 10.18653/11/119-16200 URL 1620
Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Yew Lin, and Clare R. Voss (eds.), Proceedings of the Workshop on Intrinsic and Extrinsic Evalua- fion Measures for Machine Translation and/or Summarization @ ACL 2005, Ann Arbor, Michi- gan, USA, Jume 29, 2005, pp. 65-72. Association for Computational Linguistics, 2005a. URL https: www. wwwwwb.www........... org/ /ttg//aatooooloo////000003
Satanjeev Banerjee and Alon Lavie. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Yein Lin, and Clare R. Voss (eds.). Proceedings of the Workshop on Intrinsic and Extinsic Evalua- tion Measures for Machine Translation and/or Summarization @ 2005, Ann Arbor, Michi- gan, USA, June 29, 2005, pp. 65-72. Association for Computational Linguistics, 2005b. URL https: www. ac web. org/ antho wwwechhoogyyyoooo ogy N95-0999/
Hangbo Bao. Li Dong. Furu Wei, Wenhui Wang. Nan Yang. Xiaodong Liu, Yu Wang. Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao- Hon. Unilmv2: Pseudo-masked language models for unified language model pre-training. CoRR, abs/2002. 11022002.1204,, 12804, 2020. URL https:
flarxiv.rrg/a abs 2002. 12804
Kevin Clark, Minh- Thang Luong, Quoe V. Le, and Christopher D. Manning. ELECTRA: pre- training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020.
net./ www.m/orummm/mmm fooormmiddrr" XMV
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171- doi: 10.18653/v1/n19-1
4186. Association for Computational Linguistics, 2019.
URL
Bhuwan Dhingra, Kathryn Mazaitis, and William W. Cohen. Quasar: Datasets for question answer- ing by search and reading. CoRR, abs/ 10s//700,,,,94, .39044 2017. URL
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Hon. Unified language model pre-training for natural language understanding and generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché- Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Sys- tems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 15002-13055 2019.
Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties, ACL 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 3342-1352, Association for Computational Linguistics, 2017. doi: 10. 10.18653/w1/P17-1222 123.. URL https:/ /tttpp:://di.... http:///eeooooooo..oooo//ooo 11t.///111113//111111111111
David Golub, Po-Sen Huang, Xiaodong He, and Li Deng. Two-stage synthesis networks for transfer learning in machine comprehension. In Martha Palmer, Rebecca Hwa, and Sebas- tian Riedel (eds.), Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pp. 835-
10

---

Under review as a conference paper at ICLR 2021
844. Association for Computational Linguistics, 2017. doi: 10.18653/v1//11110000 URL 1150///11...53/////1111111100000
Saidalavi Kalady, Ajeesh Elikkottil, and Rajarshi Das. Natural language question generation using syntax and keywords. In Proceedings of QG2010: The Third Workshop on Question Generation, volume 2, pp. 5-14, 2010.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. Improving neural question gen- eration using answer separation. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thisty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Inteligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pp. 1660266699 AAAI Press, 2019. doi: 10. 1..1///aaai..3311...22.. (33101.33016600 (RLhhtt::://dbl org 1009/ 1aa/ v33101.
33C16602
Taku Kudo. Mecab: Yet another part-of-speech and morphological analyzer. http://mecab source- forge. jp, 2006.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le. and Slav Petrov. Natural questions: a benchmark for question answering research. Trans, Assoc. Comput, Linguistics, 7:452-466, 2019. URL article/view/1e
wwwp/////ooo/oooooooo/ooooooeeot wwwps/iidxxx..cm php/tacl/
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jae- Kang. BioBERT: pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 09 2019a. ISSN 1367-4803. doi: 10. 00.1009/boonformatiis//22222. URL https::/ttto..ooo
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Associa- fion for Computational Linguistics, pp. 6086 6096, Florence, Italy, July 2019b. Association for Computational Linguisiccs. doi: 10.18653/v1/P19-1660 1612. URLhttpss /tt:://wwwwww /twp:w awwweb.lw.b... org/
anthology/P19-1165
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-squeene pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line, July 2020. Association for Computational Linguistics. doi: 10. 1/053///2002.aclmmmaaa URL http:: /wwww//www../m.... acl web. org/ wwwp///tttoooooooooooooeee
acl in. 703
Seunyyoung Lim, Myungji Kim, and Jooyoul Lec. Korquad1.0. 1.0: Korean QA dataset for machine reading comprehension. CoRR, abs/1909.07000 2019. URL
1909.07000
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, 2004.
Bang Liu, Mingjun Zhao, Di Niu, Kunfeng Lai, Yancheng He, Haojie Wei, and Yu Xu. Learn- ing to generate questions by learningwhat not to generate. In Ling Liu, Ryen W. White, Amin Mantrach, Fabrizio Silvestri, Julian J. McAuley, Ricardo Baeza- Yates, and Leila Zia (eds.), The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, pp. 1106- 1118. ACM, 2019. doi: 10.1145/3308558.331777 33C8558. 3313737
URL bttpps:///cc.... 1145/
Xiyao Ma, Qile Zhu, Yanlin Zhou, and Xiaolin Li. Improving question generation with sentence- level semantic matching and answer position inferring, In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Secont Innovative Applications of Artificial Intelli- gence Conference, IAAI 2020, The Tenth AAAI Symposium Educational Advances in Artificial Intelligence, EAA/ 2020, New York, NY, USA, February 7-12, 2020, pp. 8471. AAAI Press, 2020. URL https::///aaat www. index wwwph///wAAA//// http://aaaaa.ragojj././/////////

---

Under review as a conference paper at ICLR 2021
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. MS MARCO: A human generated machine reading comprehension dataset. In Tarek Richard Besold, Antoine Bordes, Artur S. d' Avila Garcez, and Greg Wayne (eds.), Proceed- ings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, December 9, 2016, volume 1773 of CEUR Workshop Proceed- ings. CEUR-WS org, 2016. UtLhttpp:// ceur ws org.oogg///ooo /Vol 1773 wwt.--11111//COCISS.. ..verr.pdd5
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Associa- fion for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 31 3-111 ACL, 2002a. doi: 10.31155100333331073111 URL https: / /www www.ww.wbbccccccccc.......... www.www...w..tto.oggggg/oo P02 10407
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pp. 311-318. ACL, 2002b. doi: 10.3115/107308330031155 URL URLhttts: httph:tts///cc..... /ttp:://wwwwww. wwwwwww... acl www.ww.wwwb....cccc.........w... org hwt.z//g/noooog..... P02 10407]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training- URL wtpp:://3-uu-west-2.oo wamauomaws.com/oppeee understanding paper.pdf. 2018.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140): 2020. URL http:
074. html
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh (eds.). Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA. November 1-4, 2016, pp. 2383 2392. The Association for Computational Linguistics, 2016. doi: 10.18653/v1/d1--1 1264. URL https: org/ 10.100101 1t..g//...165/11.... 116-1264
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable ques- tions for squad. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pp. 784-789. Association for Computational Linguis- tics, 2018. doi: 10.1865331/P18--1144, URL https: /www.wwwoooomoom...oo..//. aclweb. org/anthology/
P18 118-211 21247
Yvnnne Skalban, Le An Ha, Lucia Specia, and Ruslan Mitkov. Automatic question generation in multimedia-based learning. In Martin Kay and Christian Boitet (eds.), COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Posters, 8-15 December 2012, Mumbai. India, pp. 1151-1160, Indian Institute of Technology Bombay, 2012. URL https: www. aclweb org/ anthology/C1222 21127
Linfeng Song, Zhiguo Wang, and Wael Hamza. A unified query-based generative model for ques- tion generation and question answering. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018.
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Adam Trischler, and Yoshua Bengio. Neural models for key phrase extraction and question generation. In Eunsol Choi, Min- joon Seo, Danqi Chen, Robin Jia, and Jonathan Berant (eds.), Proceedings of the Workshop on Machine Reading for Question Answering @ACL 2018. Melbourne, Australia, July 19, 2018, pp. 78 78-88. Association for Computational Linguistics, 2018. doi: 10.18653/v1/W 10/553//1/W1826110 URL https: wwwwwwwww./mmmmm a web. leabb.ogg/antee www.enooooooooooooooooooooo 8-
12

---

Under review as a conference paper at ICLR 2021
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. In Phil Blunsom, Antoine Bordes, Kyunghyun Cho, Shay B. Cohen, Chris Dyer, Edward Grefenstette, Karl Moritz Hermann, Laura Rimell, Jason Weston, and Scott Yih (eds.), Proceedings of the 2nd Workshop on Representation Learning for NLP, Cep4NIP@AAT. 2017, Vancouver, Canada, August 3. 2017, pp. 191-200. Association for Computational Linguistics. 2017. doi: 10. 10.18653/11/w11-22222 URL https:
//doi.org/1i 10. 10.510...66vvv11 w 2623
George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers. Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiotis, Patrick Gallinari, Thierry Artieres, Axel Ngonga, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliouras. An overview of the bioasq large-scale biomedical semantic indexing and question answering competition. BMC Bioinformatics, 16: 138, 2015, doi: 10. 12599005--0---066-6666 URLhttpp www. bi omedcentral.......momo.......... com/ content/ pttp:////..1121--11-11----------
Luu Anh Tuan, Darsh J. Shah, and Regina Barzilay. Capturing greater context for question gener- ation. In The Thirty- AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty- Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 9065-9072. AAAI Press, 2020. URL https:://aaat org/ojs/ index. Awt.t//ttaattteeeteeaaaaaaaaeaaa
6440
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, UI- rike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: An- nual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA, pp. 5998 -008, 2017. URL htttp://pppers nips cc cc/paper/
7181-attention is all you-yo--eee need
Dongling Xiao, Han Zhang, Yu-Kun Li, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang- ERNIE-GEN: an enhanced multi-flow pre-training and fine-tuning framework for natural lan- guage generation. In Christian Bessiere (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 3997-4003. ijcai.org, 2020. doi: WRLLtttps://bo..orgp 212.11...4933331 htt.l/aa....22.0552 2020/553
10.24963/ijcai
Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William Cohen. Semi-supervised QA with gen- erative domain-adaptive nets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1040-1050, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/.//117-10096 URL https:/ www www.aclweb.cr org wttp./www.....aaa.ooooooo////--.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' A'Aché--Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pp. 5754-5764, 2019.
Wonjin Yoon, Jinhyuk Lee, Donghyeon Kim, Minbyul Jeong, and Jaewoo Kang. Pre-trained lan- guage model for biomedical question answering. In Peggy Cellier and Kurt Driessens (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 727-740, Cham, 2020. Springer International Publishing. ISBN 977--3-03--488876
Jingging Zhang, Yao Zhao, Mohammad Salch, and Peter J. Liu. PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. CoRR, abs/1912.08777,, 2019. URL http://
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. Paragraph-Ievel neural question genera- tion with maxout pointer and gated self-attention networks. In Ellen Riloff, David Chiang, Julia
13

---

Under review as a conference paper at ICLR 2021
Hockenmaier, and Jun 'cchi Tsujii (eds.), Proceedings of the 2018 Conference Empirical Meth- ods in Natural Language Processing. Brussels, Belgium. October 31 November 4, 2018, pp. 3901-3910. Association for Computational Linguisties, 2018. doi: 10.18653/v1/d18-1422 URL
httpps:/ddft http://ooooorggg//////// .org/10. 1ww.11166/lllll 1424
Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. Neural question generation from text: A preliminary study. In Xuanjing Huang, Jing Jiang, Dongyan Zhao, Yan- song Feng, and Yu Hong (eds.), Natural Language Processing and Chinese Computing- 6th CCF International Conference, NLPCC 2017, Dalian, China, November 8- 12. 2017. Proceedings, vol- ume 10619 of Lecture Notes in Computer Science, pp. 662 67-1671. Springer, 2017. doi: 10. 1007/ URL https: /ww.ddi.ooooooo org 978-3-319-73618- 1,.66
56
APPENDIX
A QUESTION GENERATION ON MORE MRC DATASETS
We also evaluate the question generation model on another data split (Split3) from Zhao et al. (2018) Split3 is obtained by dividing the original development set in SQuAD-v1.1 into two equal halves randomly and choosing one of them as the development set and the other as test set while retaining the train set in SQuAD-v1.1 As shown in Table 12, applying ASGen to the reproduced question generation model from Zhao et al.] (2018) improves BLEU-4, METEOR, and ROUGE-L score on Split3 by 1.3, 0.9, and 1.3, respectively.
We also test generalization capability of our method by evaluating on Natural Ques- tions (Kwiatkowski et al., 2019) dataset, where questions are collected from real user query logs on Google and may have less biased questions than other datasets. As shown in Table applying ASGen to BertGen shows improvement the question generation score on Natural Questions short answer dataset by 3.8 BLEU-4, 2.5 METEOR and 1.1 ROUGE-L.
Table 12: Additional experiments on the effectiveness of ASGen the test set of SQuAD Split3. Small-Wike is used to pre-train the models. Models with *** indicate those results we reproduced.
Model pre- pre-training method
BLEU-4
METEOR
ROUGE-L
Zhao al.
2018
16.8
20.6
44.9
Zhao et al.
2018
16.3"
20.3
"44.5
Zhao el al.
2018
ASGen
17.6
21.2
45.8
Table 13: Ablation study of applying ASGen to question generation model on Natural Ques-
tions (Kwiatkowski al. 2019) short answer dataset. The scores are obtained from the dev set.
Model + pre-training method
BLEU-4
METEOR
ROUGE-L
BertGen (Large)
31.5
30.4
60.2
BertGen (Large) ASGen
35.3
32,9
61.3
B TRAINING ELECTRA MRC MODEL WITH GENERATED SYNTHETIC DATA
We also apply our synthetic data generated from Small-Wiki another MRC model, ELEC- TRA (Clark et al. 2020), which shows the state-of-the-arr results. In Table we report the mean EM/FI score on SQuAD 2.0 development set of four runs by using official Electra source " and the pre-trained checkpoint. Pre-training ELECTRA on the generated synthetic data using ASGen improves 0.5 EM and 0.6 F1 score on the downstream MRC dataset, SQuAD-v2.0, even when using only Small-Wiki.
14

---

Under review as a conference paper at ICLR 2021
Table 14:
Ablation study of applying our method ELECTRA (Clak al. 2020) on SQuAD-v2.0
dev set after pre-training on the generated synthetic data using ASGen with Small-Wiki.
Dev set
MRC model
Synthetic Data
EM
ELECTRA
87.4
90.2
(Large)
SSmallWWiki
87.9
90.8
C TRANSFER LEARNING TO OTHER MRC DATASET (QUASAR-T)
To show that our generated data useful for other MRC datasets, we fine-tune and test the MRC model on QUASAR-T (Dhingra et al. 2017), which is another large-scale MRC dataset, after train- ing on the synthetic data generated from SQuAD-v1.1. In this experiment, we first fine-tune 'Bert- Gen ASGen' using SQuAD-v1.1 I, and using synthetic data generated by this model, we train the BERT (Large) MRC model. Afterwards, we fine-tune BERT (Large) for the downstream MRC task using QUASAR-T data. QUASAR-T has two separate datasets, one with short snippets context, and the other with long paragraphs as context. As shown in Table [5] training with our synthetic data improves the F1 score the test by 2.2 and 1.7 for the two cases, respectively.
Table 15: EM/F1 scores of the BERT (Large) fine-tuned on QUASAR-T dataset. The used synthetic data is generated from ASGen trained on SQuAD-v1.1 (Full-Wiki).
Short(Dev)
Short( Test)
MRC model
Synthetic Data
EM
EM
F1
74.3
78.6
74.1
77.8
BERT
Full-Wiki
76.5
80.1
76.5
80.0
Long(Devv
Long(
MRC model
Synthetic Data
EM
EM
F1
72.1
75.6
72.1
74.8
BERT
Full-Wiki
74.2
77.4
73.8
76.5
D COMPARISON OF ANSWER GENERATION APPROACHES ON MRC TASK
We also evaluate the effectiveness of dynamic-K answer prediction by pre-training the BERT (Large) (Devlin et al. (201)) MRC model tuning on the downstream MRC dataset, SQuAD-v2.0. As shown in Table 16 dynamic- K answer prediction shows 0.3 EM and 0.2 F1 score improvements from the baseline approach, fixed-K.
synthetic data from Small- Wiki followed by fine-
Table 16: Comparison predicting answers with downstream BERT (Large) MRC results on SQuAD-v2.0 dev set after pre-training on each generated synthetic data using corresponding answer generation approach with Small-Wiki
Dev set
Answer generation approach
EM
F1
Fixed-K K (Krred = 5) Dynamic-K K (ASGen)
81.38 (+00009)
84.36 (=0007)
81.73 (+0006)
84.62 (=0.04)
E DETAILS OF WIKIPEDIA PREPROCESSING
To build the answer-containing sentence generation data and the synthetic MRC data for SQuAD (Rajpurkar et al.; 2016), collect all paragraphs from all articles of the entire English Wikipedia dump and generate questions and answers on these paragraphs. We apply extensive fil- tering and clean-up only retain the highest-quality paragraphs from Wikipedia, as follows.
15

---

Under review as a conference paper at ICLR 2021
To filter out low-quality articles, remove those with less than 200 cumulative page- views includ- ing all re-directions in a two-month period. In order to calculate the number of page-views, official Wikipedia page-vier view dumps were used. Of the 5.4M original Wikipedia articles, filtering by page- views leaves 2.8M articles. We also remove those articles with less than 500 characters, as they are often low-quality stub articles, which further removes additional 16% of the articles. We remove all "meta" namespace pages such as talk, disambiguation, user pages, portals, etc. as they often con- tain irrelevant text casual conversations between editors. order to extract clean text from the wiki-markup format of the Wikipedia articles, we remove extraneous entities from the markup in- cluding table of contents, headers, footers, links/URLs, image captions, IPA double parentheticals, category tables, math equations, unit conversions, HTML escape codes, section headings, double brace templates such as info-boxes, image galleries, HTML tags, HTML comments, and all tables.
We then split the cleaned text into paragraphs and remove all paragraphs with less than 150 charac- ters or more than 3,500 characters. Paragraphs with the number of characters between 150 to 500 were sub-sampled such that these paragraphs make up 16.5% of the final dataset, as originally done for the SQuAD dataset. Since the majority of the paragraphs Wikipedia are rather short, out the 60M paragraphs from the final 2.4M articles, our final Wikipedia dataset contains 8.3M paragraphs. Finally, we generate 43M answer wawaerrp pairs from the final Wikipedia dataset with the answer generator of BertGen in this paper.
F TRANSFER LEARNING TO OTHER LIMITED DOMAIN DATA (BIOASQ)
We conduct experiments BioASQ ( satsaronis et al. 2015) dataset to show the effectiveness of our model in limited-data domains having less annotated data. As shown in Table 17 ASGen im- proves the question generation scores by 6.0 BLEU-4, 7.8 METEOR and 6.9 ROUGE-L on BioASQ factoid-type 6b. Moreover, using 'Full-Wiki' data enhances the performance of BERT(Large) by a large margin and outperforms BioBERT (Lee et al. 2019a), by 0.95 Macro F1 (Yes/No) and 1.63 Fl (List), Note that BioBERT is specifically pre-trained on medical corpus (PubMed) whereas we use generic corpus Wikipedia, FFull-Wiki', with our generation models fine-tuned on SQuAD.
Table 17: The performance of our method on limited-datt domain (BioASQ). Note that the scores of question generation are obtained from BioASQ factoid-type 6b. All experiments were conducted using the official source code of Yoon et al. (2020).
Question Generation Model
BLEU-4
METEOR
ROUGE-I.
BertGen (Large)
6.6
10.0
33.1
BertGen (Large) ASGen Wiki)
12.6
17.8
40.0
MRC model
Pre-training Data
Factoid (MRR)
Yes/No (Macro FI)
List-Type (F1)
BERT(Large)
34.3
53,8
36.1
BERT(Large)
ASGen (Full-Wiki)
49.2
81.1
39.8
BioBERT((arge)
PubMed
52.3
80.1
38.1
G APPLICATION OF ASGEN TO T5 WITH LIMITED
DATA
As shown in Table 18 our pre-training method, ASGen, increases question generation scores of T5 (Small) (Raffll et al. 2020) model even using limited pre-training data Small- Wiki. We expect pre- training may show similar effect in other sized T5 models well. Results for T5 pre- training with Full-Wiki data are in the main paper.
H CENTRAL TENDENCY AND VARIATION FOR HUMAN EVALUATION
Human evaluation involves 10 evaluators over metrics such as syntax (ST), validation of semantics (SM), question to context relevance (CR) and question answer relevance (AR) on 50 randomly chosen samples on SQuAD-v 1.1 development set. Each score is in the range to Central tendency and variation can be found in Table 19
16

---

Under review as a conference paper at ICLR 2021
Table 18: Application of ASGen T5 Model with Limited Pre- Fre-Trainnng data
Test set on Splitl
BLEU-4
METEOR
ROUGE-L
T5 (Small)
15.6
23.3
37.1
T5 (Small) ASGen (Small-Wiki)
16.5
24.0
38.4
Test on Split2
BLEU-4
METEOR
ROUGE-L
T5 (Small)
18.8
25.2
40.5
T5 (Small) ASGen (Small-Wiki)
19.2
25.9
41.3
Table 19: Central tendency and variation for human evaluation scores. + 95% confidence interval.
Model
ST
SM
CR
AR
4.04
3.93
4.20
3.25
BertGen
+0.18
+0.19
+0.16
+0.22
4.60
4.54
4.49
3.63
BertGen NS
-0.12
10.13
10.14
10.22
4.71
4.69
4.74
4.14
BertGen ASGen
+0.10
#0.11
+0.09
10.18
4.25
4.31
4.54
4.06
UniLM
10.16
10.16
10.12
10.19
4.71
4.79
4.70
4.17
UniLM ASGen
#0.11
+0.09
#0.11
10.18
I CENTRAL TENDENCY AND VARIATION FOR THE DOWNSTREAM TASKS
For the EM and scores on downstream SQuAD-v1. and v2.0 development set in Table 7 of main paper, we selected 5 model checkpoints from the same pre-training the synthetic data in different numbers of training steps. We then fine-tuned each of these models on the final downstream data three times each, chose the best performing model the development set, and reported its score. Central tendency and variation can be found in Table 20
Table 20: Central tendency and variation for the score of our approach, BertGen(Large) + ASGen, on downstream SQuAD-v1.1 and v2.0 dataset. + is standard deviation.
Dev-v1.1
Dev-v2.0
MRC model
Synthetic Data
EM
F1
EM
Fl
86.2
92.7
84.4
87.3
BERT (Large)
Full-Wiki
+0.1
+0.1
#0.2
+001
87.4
93.4
85.5
88.3
BERT (WWM)
Full-Wiki
10.1
10.1
10.1
10.1
J DETAILS OF GENERATING UNANSWERABLE QUESTIONS
The mechanism of generating questions may differ in generating answerable and unanswerable ques- tions. For example, the model could exploit mismatching phrase to make question plausible but unanswerable. order to reflect these characteristics, we train answerable and unanswerable mod- els separately. We first take the BertGen model pre-trained the ASGen task and then fine-tune this model on the no-answer question generation on SQuAD-v2.0. We infer with this model on the entire Wikipedia to make negative examples for un-answerble synthetic data for pre-training MRC models on SQuAD-v2.0.
K DISCUSSION ON WEAK SUPERVISION FOR DYNAMIC-K K PREDICTION
In question generation, is important to find which elements of given context are suitable answer. To do this, predict the number of answers to obtain appropriate set of "answer-like"
17

---

Under review as a conference paper at ICLR 2021
phrases that humans tend choose when they are preparing question, rather than all possible entity phrases. This tendency can also be found the SQuAD dataset, which has varying number of annotated answers per context, even though the annotators were recommended to create up to five answers, as shown in Table 21] While we do not have the ground-truth number of answers for all contexts, this characteristic of SQuAD annotation can still be useful weak supervision for learning the number of answer candidates.
Table 21: Distribution over the number of answers in SQuAD-v1.1 dataset.
Number Answers
5
6+
Percentage of Sample
0.5
0.9
9.1
21.9
60.1
7.5
L BLEU-4, METEOR, AND ROGUE-L
BLEU (Papineni et al.) 2002b), METEOR (Banerjee Lavie 2005a) and ROUGE (Lin 2004) are widely-used metrics for evaluating the quality of generated text, where the quality indicates the de- gree of correspondence between generated text and reference texts. BLEU uses modified precision to compare a generated text against the reference texts. BLEU-4 calculates a weighted score of unigram, bigram, trigram, and 4-gram based matching. METEOR uses harmonic mean between precision and recall of unigrams, but with for recall given more importance than precision. Unlike BLEU, METEOR also tries to match synonyms and performs stemming instead of just relying on exact word matching. ROUGE-L is the longest common sub-sequence based word matching. The longest co-occurrence in sequences of n-grams between generated text and reference texts are con- sidered for calculating the score. To calculate these evaluation scores, follow the script from Du et al. (2017), except for the corresponding scripts from other question generation models when ASGen is applied to them.
M LINKS TO DOWNLOADABLE COMPONENTS
For Wikipedia data, downloaded English Wikipedia dump in Feb 2019 from https:
7/dumps. wi www.iiiiii......m.... http:///www..aaaaaaatt//////.ii.
xml xml.bz2). Page views obtained from
and fttcppss//dmmsss.iiimiiiiie..e
pageviees/2211/2219--1
For applying our method other existing question generation
models, we reproduce /hao et al. (2018) using publicly available code (https: (https://g/thub
com/
Raffel et al. (2020) using publicly
available code http:////gthuuoooo//aaallssssttr wttpreessttiooneeeeeiiiioioi generation) and
use the official code of Dong et al. 2019) http0//00000////aaaaaaoooooaaoo.
18


Reviews:
[{"review_id": "up--IA1trW", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis work presents a new multi-step method to pre-train a question generation system, which can then be used to create synthetic data to improve a Machine Reading Comprehension system.\n\nFirst, the authors train a system to identify spans in a text paragraphs which would constitute likely answers for questions about the paragraph. Then, they pre-train a system to generate those questions by taking the selected answer and surrounding sentences as input, and generating the sentence which contained the answer. Finally, they use real MRC data to fine-tune the question generation system.\n\nThe authors first provide a direct evaluation of their method by showing a consistent improvement in reference-based metrics comparing to gold questions for a given paragraph answer, then shows that the generated synthetic data also leads to improvement on the downstream MRC task for the SQuAD dataset, especially when using less annotated data.\n\nThe results provided are encouraging and accompanied by a number of enlightening supporting experiments. The paper could still be improved by clarifying specific points.\n\n### Clarity\n\nWhile the experimental setting description gives a broad idea of what the authors did, some details are missing for full reproducibility. Most notably, there is very little information on the first step setup, training the answer selection system. The authors also describe their setting based on BERT but not the one with UniLM\n\nWhat is the dimension of the indicator vector that tells the model where the answer is located? Is there a particular reason why the authors decided to use an indicator vector rather than e.g. use a [SEP] token?\n\nDynamic prediction of the number of answers in a paragraph doesn't seem to account for much improvement. Can the authors provide some measure of statistical significance? Otherwise, the method really is novel enough without that part.\n\nDid the model evaluate the effect of using synthetic data on the other tasks?\n\n### Correctness\n\nThe claims are mostly well supported by the experiments (to the exception of the dynamic K prediction).\n\n##### Additional citations\nThe pre-training sentence prediction method is related to:  \n**Latent retrieval for weakly supervised open domain question answering**, Lee et al., *ACL 2019*\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fTCbZfEd7Db", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission introduces a new approach for pre-training models for question generation conditioned on an answer and a document. Given a document, a set of k answers is chosen by a classifier. Then, the sentence containing the answer is removed, and a sequence to sequence model is used to reconstruct it. This pre-training task more closely matches the desired end task. Results show that this method improves question generation performance compared to previous work, and that the synthetic questions can be used to improve question answering models. Overall, I think the approach makes a lot of sense. However, I think it needs to compare with stronger pre-training baselines, and needs a more thorough comparison of the effect on question answering performance.\n\nThe paper primarily evaluates on question generation for a range of datasets, and shows improvements on both automated and human evaluation metrics. I think it's fair to say that the question generation is quite a niche task, perhaps because of limited applications. Indeed, the abstract of the paper pitches the task as being primarily useful for generating synthetic data for question answering. That's fine, but given this motivation, I think the evaluation should focus much more on the downstream impact on question answering.\n\nI'd really like to see if the method still improves question generation performance on top of more recent sequence to sequence models, such as T5 or BART, which have both been available for almost a year. As far as I know, these both significantly outperform UniLM on all the comparisons I've seen. The pre-training objectives used there, which involve predicting masked spans, seem closely related to the proposed method. This experiment would help understand whether the proposed method adds anything on top of more general pre-training approaches.\n\nThe paper convincingly demonstrates that synthetic questions improve a baseline BERT question answering model, which was already shown by e.g. Alberti et al (2019). However, I don't think the paper does much to suggest that the synthetic questions from the proposed method are better for QA than other approaches - for example, Alberti et al. appear to report similar numbers with their method. It's important to include a baseline set of synthetic questions, as it's not at all clear that improvements on question generation metrics will correlate with usefulness for QA training.\n\nAgain, the question answering results would be more convincing if they hold up with more recent models than BERT, which achieve better scores without synthetic questions. Table 11 does give a result using Electra - but gives the baseline Electra model an exact match score of 87.4. The Electra paper instead lists 88.0, slightly higher than the results using synthetic questions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SA50RZ4IeHr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a pretraining technique for question-generation models. The pretraining involves generating the sentence which contains an answer candidate from the context around it. Through several well-executed experiments the paper shows that this type of pretraining can improve performance of several existing question generation models, and the resulting synthetic questions generated help in augmenting reading comprehension datasets.\n\nStrengths:\n- This is solid empirical work, with several detailed experiments showing the utility of the pretraining method on multiple benchmarks and with multiple models. It was particularly nice to see evaluation on Korean as well as English.\n\n- The authors clearly put effort in explaining all the methods and experiments precisely and with details.\n\nWeaknesses:\n- The technical contributions of the paper are rather limited. Pretraining has been of much interest in the NLP community recently, and this paper follows a line of related works which have proposed similar techniques (e.g. Inverse Cloze Task, Lee et al, ACL 2019; T5, Raffel et al, JMLR 2020). There isn't much discussion of these existing papers either.\n\n- It is not clear how much of the benefit of pretraining comes from the specific approach used here, versus the fact that there is some pretraining on the decoder which generates the questions. We could learn more about this if there was a comparison to other pretrained models which have a decoder, e.g. T5, BART (Lewis et al, 2020).\n\n- In terms of the question generation task, it is well known now that Squad questions have a bias towards high lexical overlap with the passage (due to the manner in which they were constructed, see Lee et al above). This raises the question whether the approach in this paper can generalize to datasets where this bias does not exist, e.g. Natural Questions. This seems to be a limitation of not just paper, but the prior works as well.\n\nOther comments / questions:\n- Where do the ground truth answers come from during pretraining? Is the answer prediction model also pretrained?\n\n- In section 2.1, it says that the MSE loss is computed using K, but K comes from the floor function which does not support backpropagation. Do you use the output f_k instead to compute the MSE?\n\n- In the same section, I assume a softmax is applied on s_i, e_i,j before taking the cross-entropy loss. In this case, is e_i,j normalized over all spans in the passage or only the ones which start at i?\n\n- Is w^o_t an embedding or a probability? There seems to be confusion at the end of section 2.\n\n- Some more discussion of the UniLM baseline would be good for people not familiar with that work.\n\n- Missing discussion of related work: \"Yang, Zhilin, et al. \"Semi-supervised qa with generative domain-adaptive nets.\" ACL (2017).\"\n\n- How does the model generate unanswerable questions for Squad 2.0?", "extra_scores": {}, "extra_text": {}}, {"review_id": "RmHSQDuvaz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a model for unsupervised pre-training for the task of question generation. The model first predicts the number of answer present in a given paragraph and then selects the top-K answer spans from the paragraph. After selecting the answer spans, the model, then tries to generate the answer containing sentence by inputting the paragraphs less the answer-containing sentence and also the answer span. The key idea is that this unsupervised pre-training strategy is close to the actual task of generating question given the context and the answer. This is also the key differentiator between this work and other existing pre-training strategies for question generation (e.g. Alberti et al 2019). \n\nThe paper outperforms other existing methods of question generation on several datasets (two splits of Squad, MS Marco, NewsQA) both on automatic and a small scale human evaluation. Efficacy of the pre-training scheme is shown via the fact that the pre-training scheme also improves other question generation model. Lastly, training a QA model on the synthetic generated questions improves downstream performance of a QA model and the difference is greater in low-data setting suggesting the applicability of this pre-training scheme in low-data regime (although no experiments in a specific low-data domain is reported).\n\nStrengths:\n* The pre-training scheme is closer to the original task than other existing methods and can be easily scaled.\n* The paper is well written and easy to follow and the experiments and ablations were exhaustive.\n* Improvements in low-data QA setting is promising and shows the scope of usefulness of this work\n\nWeaknesses\n* I am not sure about the need to predict the number of unique answers in a paragraph or that if it makes sense to do so. I think it is hard to estimate the number of questions that can be generated from a given text and regressing to a value which is present in current datasets might be suboptimal. This is because, the current QA datasets do not ask annotators to generate all possible question from a paragraph. Instead, I believe you could have taken a principled approach of considering named-entities as answers, and trying to generate questions for each entity (or a set of entities together). This would also eliminate the need to predict the number of answer spans in a paragraph at once.\n* I think it would be useful for the paper to categorize the kinds of “reasoning” that the generated question requires to answer. For example, is it just fuzzy pattern matching kind of questions or does answering the question require any kind of reasoning (such as multi-hop or numerical reasoning). \n* It was not clear to me, how you generate questions which are unanswerable (e.g. those in Squad 2.0)\n* Although, it is good to see that training on the synthetically generated questions help in low-data settings, I think the paper would be stronger and more convincing. if an actual experiment was done on a domain which has less data. For example, you could try the BioASQ datasets which have very less annotations and is in the bio-medical domain. It would be interesting to see if pre-training on a scientific corpus is actually helpful. Moreover, some questions in BioASQ need reasoning such as handling list questions, counting and it would be interesting to see if the performance on those questions improve.\n\nOverall, I think the current experiments are reasonably well-done and I think the paper would be much stronger if it was tested on an actual domain which has low-data and also if the paper discusses / categorizes the kind of reasoning that the generated questions require. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers appreciate the good quality of this submission with a good idea and solid execution (as said by R3). The paper is clearly written and the addition during the discussion have greatly improved it as acknowledged by all reviewers.

However, a major weakness of the submission still needs to be addressed before a publication at ICLR. 

As said in the paper, the task of question generation is a task whose main impact is to improve downstream tasks, and primarily QA. The evaluations follow that and extra-experiments (e.g. BioASQ) and discussion wrt state of the art (e.g. Alberti et al.) reinforce them. Yet, as pointed out by R1 & R4, the effect on downstream QA performance is only shown for weaker models than the current state of the art (e.g. T5, BART). Since the rebuttal period was not long enough to run these experiments, it is impossible to assess how the proposed approach compare to them with the current draft. Adding the experiments on T5 (Small) is a step in the right direction but it is not enough for that. Without those experiments, one can not conclude that this pretraining strategy would also help over the strongest existing pretrained model.

The authors should run those experiments to make the arguments presented in the submission much stronger.










Decision: Reject