ASGen: Answer-containing Sentence Generation to Pre-Train Question Generator for Scale-up Data in Question Answering
Authors: Akhil Kedia
Sai Chetan Chinthakindi
Seohyun Back
Haejun Lee
Jaegul Choo
Venue:  (2020)
Keywords: Question Answering
Machine Reading Comprehension
Data Augmentation
Question Generation
Answer Generation

Abstract: Numerous machine reading comprehension (MRC) datasets often involve manual annotation, requiring enormous human effort, and hence the size of the dataset remains significantly smaller than the size of the data available for unsupervised learning. Recently, researchers proposed a model for generating synthetic question-and-answer data from large corpora such as Wikipedia. This model is utilized to generate synthetic data for training an MRC model before fine-tuning it using the original MRC dataset. This technique shows better performance than other general pre-training techniques such as language modeling, because the characteristics of the generated data are similar to those of the downstream MRC data. However, it is difficult to have high-quality synthetic data comparable to human-annotated MRC datasets. To address this issue, we propose Answer-containing Sentence Generation (ASGen), a novel pre-training method for generating synthetic data involving two advanced techniques, (1) dynamically determining K answers and (2) pre-training the question generator on the answer-containing sentence generation task. We evaluate the question generation capability of our method by comparing the BLEU score with existing methods and test our method by fine-tuning the MRC model on the downstream MRC data after training on synthetic data. Experimental results show that our approach outperforms existing generation methods and increases the performance of the state-of-the-art MRC models across a range of MRC datasets such as SQuAD-v1.1, SQuAD-v2.0, KorQuAD and QUASAR-T without any architectural modifications to the original MRC model.

Under review as a conference paper at ICLR 2020
ASGEN: ANSWER-CONTAINING SENTENCE GENERATION TO PRE-TRAIN QUESTION GENERATOR FOR DATA AUGMENTATION IN QUESTION ANSWERING
Anonymous authors Paper under double-blind review
ABSTRACT
Numerous machine reading comprehension (MRC) datasets often involve ual annotation, requiring enormous human effort, and hence the size of MRC data remains significantly smaller than that of those available for unsupervised learning, limiting the generalization capability. To overcome this issue, a new ap- proach, which can generate synthetic question-and-answe data from large cor- pora such as Wikipedia, has been recently proposed. Such synthetic data can be utilized as additional data pre-train the main MRC model before fine-tuning it using real, existing MRC data. However, the quality of generated questions and answers still far from being satisfactory, so previous work introduced a pre- training technique for the question generator by pre-training the generation of the next sentence that follows a paragraph. However, the next sentence that follows paragraph may have little relevance to the questions answers from within the paragraph, and thus it is not the ideal candidate for pre-training question genera- tion. In response, we propose novel method called knswer-containie Sentence Generation (ASGen). Our approach is composed of multiple stages, involving two advanced techniques, (1) dynamically determining K answers from given doe- ument and (2) pre-training the question generator using the task of generating the unsternconaining sentence. We evaluate the question generation capability our method by comparing the BLEU score with existing methods and test our method by fine-tuning the MRC model using the downstream MRC data after training on synthetic data. Experimental results show that our approach achieves outperforms existing methods achieving new state-of-the-ar results on SQuAD question gen- eration and increases the performance of the state-of-the-art MRC models across a range of datasets such as SQuAD-v 1.1, SQuAD-v2.0 KorQuAD, and QUASAR- T with no architectural modifications to the original MRC model.
INTRODUCTION
Machine reading comprehension (MRC), which finds an answer to given question from given para- graphs called context, is an essential task in natural language processing. With the use of high-quality numan-annotatee datasets for this task, such as SQuAD-vl.J (Rajpurkar et al., 2016), SQuAD-v2.0 (Rajpurkar et al., 2018). and KorQuAD (Lim et al. 2019), researchers have proposed MRC models, often surpassing human performance on these datasets. These datasets commonly involve finding short snippet within paragraph as an answer to a given question.
However, these datasets require a significant amount of human annotation to create pairs of a ques- tion and its relevant answer from a given context. Often the size of the annotated data relatively small compared to that of data used in other unsupervised tasks such as language modeling. Hence, researchers often rely on the two-phase training method of transfer learning, i.c., pre-training the model using large corpora from another domain in the first phase, followed by fine-tuning it using the main MRC dataset in the second phase.
Most sfate-of-the-as models for MRC tasks involve such pre-training methods. Peters et al. (2018) present a bidirectional contextual word representation method called ELMo, which is pre-trained large corpus, and its learned contextual embedding layer has been widely adapted to many

---

Under review as a conference paper at ICLR 2020
Synthetic data
Answer generation task
Answer Wikipedia
Answer-containing sentence generation task
Question generation
Question
Figure 1: Overview of our Answer-containing Sentence Generation (ASGen) method.
other MRC models. Devlin et al. (2019a) show that pre-training with masked language model on large corpus and then fine-tuning on downstream dataset results in significant performance improvements.
However, pre-training on another domain task and then fine-tuning on downstream task may suf- fer from performance degradation, depending on which pre-training task is used in the first phase. For example, Yang al. (2019) show that the pre-training task of next sentence classification de- creases performance the downstream MRC tasks. To handle this problem, generating synthetic data similar to the those of downstream task crucial to obtain a properly pre-trained model. Recently, researchers have studied model for generating synthetic MRC data from large corpora such as Wikipedia. This is essentially a form of transfer learning, by training a generation model and using this model create synthetic data for training the MRC model, before fine-tuning on the downstream MRC dataset.
Golub et al. 2017) suggest two-stage synthesis network that decomposes the process of generating question-answer pairs into two steps, generating a fixed number (K) ) of answers conditioned on the paragraph, and question generation conditioned the paragraph and the generated answer. Devlin it al. (2019b) introduced a pre-training technique for the question generator of this method by pre- training on the generation of next-sentence that follows the paragraph.
However, choosing fixed number (K) of candidate answers from each paragraph will lead to miss- ing candidates if K is too small, and will lead having lower-quality candidates if K is too big. Moreover, the next sentence generation task is not conditioned on the answer, despite the answer being strong conditional restriction for question generation task. Also, the next sentence that fol- lows a paragraph may have little relevance to the questions or answers from within the paragraph, and hence is not the ideal candidate for pre-training question generation.
To address these issues, we propose Answer-containing Sentence Generation (ASGen), novel method for synthetic data generator with two novel processes, (1) dynamically predicting K an- swers to generate diverse questions and (2) pre-training the question generator answer-containing sentence generation task. We evaluate the question generation capability of our method by compar- ing the BLEU score with existing methods and test method by fine-tuning the MRC model on downstream MRC datasets after training on the generated data. Experimental results show that our approach outperforms existing generation methods, increasing the performance of the state-of- the-art MRC models across a wide range of MRC datasets such as SQuAD-v1.1 SQuAD-v2.0, KorQuAD, and QUASAR-T DDhinrra al. 2017) without any architectural modifications to the MRC model.
2 PROPOSED METHOD
This section discusses the details of our proposed ASGen method. ASGen consists of BERT-baaed generative model (BertGen) and answer-containing sentence generation pre-training (AS). First, we will describe how BertGen model generates synthetic data from Wikipedia. Next, we will explain the novel components of our methods and how we pre-trained the question generator in BertGen based them. BertGen encodes paragraphs in Wikipedia with two separate generation networks, the answer generator and the question generator.
Answer Generator. As shown in Fig. (1), generate the number of answer candidates K for given context without the question by applying a fully connected cordwrd layer on the contex- tual embedding of classification token "[CLS]". To make the contextual embeddings and predict
2

---

Under review as a conference paper at ICLR 2020
(1) Answer Generator (AG)
(2) Question Generator (QG)
Top*
Attention on encoder embedding
Arswer
Contextual Embedding
Contextual Embedding
K
Generating
Q,
Answer-containn
Sentence
BERT Encoder-A
BERT Encoder-B
Transformer
containing
Q.
Q
Context
[C.5]] W,
W.W.
W
W
Figure 2: Architecture of ASGen (BentGen+AS), (1) Answer Generator and (2) Question Generator. In the case of answer-containing sentence generation task (AS), the question generator takes the answer and the context without the answer-containing sentence as input and generates the answer- containing sentence.
BERT
AG
AG
Fine-tuning answer generation
Generating answer
MRC data
Fine-tuning question änswering
Synthetic data
Answer
Answer
BERT
Wikipedia
Wikipedia
Wikipedia
Training on synthetic data
Question
qquestion
Generating
QG
QG Fine-tuning question generation
QG
BERT
Pre-training answer-containing sentence generation
Figure 3: Detailed procedure of generating the synthetic MRC data from Wikipedia using ASGen.
answer spans, we utilize BERT (Devlin et al. 2019a) encoder (Fig. BERT Encoder-A). Depend- ing on the predicted number K, we select the K top candidate answer spans from the context. As shown Fig. (2). we use the K selected candidate answer spans as input the question generator.
Question Generator. Next, as shown in Fig. (2),, we generate a question conditioned on each answer predicted from the answer generator. Specifically, we pass input BERT encoder the context and an indicator for the answer span location in the context (Fig. 2 BERT Encoder-OP Q). Next, Transformer decoder (Vaswani et al. 2017) generates the question word-by-word based on the en- coded representation of the context and the answer span. For pre-training such a question generator on an answer-come contrr-cottiining sentence generation task, we exclude the answer-containing sentence from the original context and train the model to generate the excluded sentence given the modified context and the answer span as input.
Finally, generate questions and answers from large corpus, e.g., all the paragraphs in Wikipedia in this paper. After generating such data, we train the MRC model on the generated data the first phase and then fine-tune the downstream MRC dataset (such SQuAD) in the second phase. In this paper, we use BERT as the default MRC model, since exhibits sfate-of taee--of-hhattt the-art performance in many MRC datasets.

---

Under review as a conference paper at ICLR 2020
2.1 DYNAMIC ANSWER PREDICTION
The most natural method for humans to create a question-answw pair from given context is lect the answer first and then create correspond- ing question. this situation, conjecture that a human is more likely to choose as answer phrase that is "answer-like" such as keyphrases, nouns, dates, names, etc. There may be several an- swers in the context that are likely to selected by humans answers, especially if the context is lengthy or if it contains multiple nouns, dates, names, etc.
# of samples
9k
5k
1k
2
5
7
8
10
# of answers
Figure Distribution of the number of an- swers in each paragraph in SQuAD-v1.1.
For example, the context "Barack Hussein Obama II is an American attomny and politician who served as the 44th president of the United States from 2009 to 2017" can have as possible answers "Barack Hussein Obama", "44th", "United States", "2009 2017", etc. As shown in Fig. to these characteristics, we examine the distribution of the number of answers in the SQuAD dataset and hypothesize that there exists underlying pattern in the number of answers that occur in context. The conventional method to generate multiple answers from a context is to draw a fixed number (K) of answers. However, this approach can generate low-quality answers if K is big. and it can impact the number and diversity of the generated answers if K is too small.
Therefore, we predict the number answers K given context W {weJō W using regression as,
(w;
eeenn T 1==
BERT Encoder-A(W),,
K [w(")))
where T is the number of word tokens in the context with position reserved for classification token |CCSS),, and fü represents fully connected unit with two hidden layers that have hidden dimensions equal to and 1, respectively, where is the hidden dimension BERT Encoder-A.
To calculate the score s; for start index i of predicted answer span, we compute the dot product of the encoder output with a trainable start vector S. For each start index i, we calculate the span end index score ę for end index in similar manner with trainable end vector E, but conditioned on i, i.e.,
{s = S W. CNE
(er T.
E fa( @ (""
where fa represents fully connected layer with hidden dimension H and indicates the catenation operation. For training, we use the mean squared error loss between K and ground-truth number of answers. We also cross-entropy loss the sy,t, and ground truth start/end of the answer span for each token. Predicting the number of answers and predicting the span are jointly trained to minimize the sum of their respective losses.
During inference, we choose the K top answer spans with the highest score summation of start index score and end index score, i.e.,
Apan = {(i,j) i < T and i S < T)
Øg max( (a #((..) (i,j) E A spon and 8; + G., 2 a) K}),
A span ((i.j) (i,5) A span and s; + e 2 ax).
The K selected answer spans A Aspon are then given to the question generator as input in the form of an indication of the answer span location.
2.2 PRE-TRAINING QUESTION GENERATOR
In order to generate questions conditioned on different answers that may arise in a context, we generate question for each of the K answers. Devlin et al. (2019b) previously proposed pre-train this generation model with an unsupervised task that generates the next sentence following a given paragraph improve generation performance. We identify several issues with this approach. The
4

---

Under review as a conference paper at ICLR 2020
final question generation task has the form of sentence generation given while the next-sentence generation has no answer component. The next-sentence generation task not conditioned the answer, despite the answer being strong conditional constraint for the question generation task. Also, the next sentence that follows a paragraph may have little relevance the questions answers from within the paragraph, and hence is not the ideal candidate for pre-training question generation.
and context,
To address these issues, modify the context exclude the sentence containing our previously generated answer and pre-train our generator on the task of generating this excluded answer- containing sentence, conditioned on the answer and the modified context.
Specifically, we exclude answer answer-containing sentence Sans while leaving the answer and modify the original context D Dans
gatart {p sentence start index = p),
Same {p, q, i, max((plpsi)) min min((dlg2/))) (1,J) E apan P E Satart E Setart).
Dane [D(p] D(gj] D (w)] (p. 9, i.j) guns.
Note that we change Sans not exclude the answer-containing sentence in the case of fine-tuning on the question generation, i.e., gans ((p.g.,,)p==.
Afterwards, pass the previously generated answer the seuuence-o--seueeece generation model as segmentation encoding Mans that identifies the answer part within the context, i.e.,
M
mmo p: m. (j i): mo (T 4)),
Sans
where mo and m, indicate trainable vectors corresponding to segmentation id and respectively. Here we tag the segmentation id for each word in the context as 0 and each word in the answer as 1. A B indicates the operation of concatenating vector for / many times.
Next, we generate answer-containing sentence embedding W {w/]d 1 using Transformer sequence-to- to-ssuuen-- model (the encoder part is initialized with BERT) as
w! Transformer Decoder( (w]) (w N=0 BERT Encoder- (accode--((Dme) D " M))s)).
Finally, we calculate the loss of the generation model with cross-entropy over generated sentence words, i.c.
(wi) = (Softmax E)
T D
IL = ΣΣ Vo.oo(w.] + (r,))og(((111 - ww.)) /T, (=1
where y indicates a ground-truth one-hot vector of the answer-containing sentence word (the ques- tion word in the case of fine-tuning), D is the vocabulary size, and E RRdD represents a word embedding matrix shared between the BERT Encoder-Q and the Transformer decoder.
In this manner, we pre-train the question generation model using a task similar to the final task of conditionally generating the question from given answer and context.
EXPERIMENTAL SETUP
Pre-training Dataset. To build the dataset for answer-containing sentence generation tasks (AS) and the synthetic MRC data for pre-training the downstream MRC model, we collect all paragraphs from the entire English Wikipedia dump (Korean Wikipedia dump for KorQuAD) and synthetically generate questions and answers on these paragraphs. We apply extensive filtering and cleanup to only retain high quality collected paragraphs from Wikipedia. Detailed proocceessing steps for obtaining the final Wikipedia dataset can be found in the supplemental material.
Using the answer generator in ASGen (BertGen+AS),, FulliWWkii from the final Wikipedia dataset for pre-training on answer- contarinn sentence gen- eration. For ablation studies pre-training approaches, also sample 2.5M paraggah
generate 43M answer-paragraph pairs
5

---

Under review as a conference paper at ICLR 2020
pairs (Small-Wike from Full- and 25K answer-paragraig pairs (Test-Wiki) to evaluate the pre- training method. Finally, using the question generator in ASGen (BentGenAAS,, we generate one question for each answer-pane waasarrp pair in Full-Wike and create the final synthetic MRC data con- taining 43M triples of a paragraph, its question and its answer.
Benchmark Datasets. In most MRC datasets, question and context are represented sequence of words, and the answer span (indices of start and end words) is annotated from the context words based on the question. Among these datasets, we choose SQuAD as the primary benchmark dataset for question generation, since it is the most popular buman-annotatee MRC dataset. SQuAD-v 1.1 (Rajpurkar al.] 2016) consists of crowd-sourced questions and answers based on contexts from Wikipedia articles. We compare our question generation capability with existing question genera- tion methods such as UniLM (Dong et al. 2019). For fair comparison, we split the training set of SQuAD-v1.1 data into our own training and test sets, and keep the original development set as our dev set, as previously done in Du et al. (2017). Kim et al.) (2019), and Dong et al. (2019). We call this dataset as Test Split |
We also evaluate on the reversed dev-test split, called Test Split2.
To evaluate the effect of generated synthetic MRC data, we evaluate the fine-tuned MRC model on the downstream MRC dataset after training on the generated synthetic data. We perform this on SQuAD-v1.J and SQuAD-v2.0 (Rajpurkar et al.) 2018). We also evaluate on KorQuAD (Lim ct al. 2019) which is another dataset created with the same procedure as SQuAD-v1.J 1.1 for Korean language. To show that our generated data is useful for other MRC datasets, we fine-tune and test the MRC model on QUASAR-T (Dhingra et al. (017) which is large-scale MRC dataset, after training on the synthetic data that generated from SQuAD-vL.1.
Implementation Details. For the answer generator, use BERT (Devlin al.) 2019a) and two fully connected layers to predict the number of answers K. For the BertGen model, we use pre- trained uncased BERT (Base) as encoder and 12 layers of Transformer as decoder. For the gener- ation of unanswerable questions as in SQuAD-v2.0, we separate unanswerable cases and answer- able cases and train separate generation models. For the final MRC model, we use BERT (Large) which is the state-of-the-art model on multiple datasets with all official hyper-parameters We use the Mecab (Kudo) 2006) tokenizer for Korean to separate postposition words which do not exist in English.
Comparison of the Pre-training Method. We compare our question generation pre-training method, which is pre-training answer-containing sentence generation task (AS), with a method from Devlin et al. (2019b), which is pre-training on next-sentence generation task (NS), and with a method from Golub et al. (0117), which only trains question generation on final MRC dataset. We reproduced these methods on BertGen as they were described in their original work for comparison. Note that 'BertGen+ASS' is equivalent to 'ASGen'. We generate synthetic data from Wikipedia using these approaches which are trained on the target downstream MRC datasets except for QUASAR-T. In the case of QUASAR-T, we use synthetic data which is generated by ASGen trained on SQuAD- v1.1. To check the effectiveness of our method on downstream MRC tasks, we evaluate our gen- erated data on SQuAD-v1.1, SQuAD-v2.0. KorQuAD and QUASAR-T by training state-of-the arrt models (BERT and BERT+CLKT?) on generated data followed by fine-tuning on the train set for each dataset. The structure of 'BERT CLKTT model is the same as that of original BERT except that the model is pre-trained for the Korean language. Due to the absence of common pre- praitrd BERT for Korean, we used this model as a baseline to demonstrate the effectiveness of our method.
QUANTITATIVE RESULTS
Dynamic Answer Prediction. We conducted an experiment to demonstrate the performance of our method in generating the number of answers in a given context. As shown in Table [ in the case of fixed K, the mean absolute error from the ground-truth Kod is the smallest at Kored = 5 and the values are 1.92 and 0.99 for Test Splitl and Test Split2, respectively. Thresholding on the sum of the start and end logits with a fixed threshold value which minimizes the mean absolute error results in
'We use the identical splitting of SQuAD data provided by UniLM from its publicly available website https://gi www.momem//oooeeeee http::ttwwwwomomcoooootttttctttt www.iiiisooooooooooooooo
*BERT+CLKT with ASGen' model can he found as -BERT-CI.T--IDDDA on the leaderboard httpss:
//korquad.gi thub. i httpbb/ioooooguu211.....

---

Under review as a conference paper at ICLR 2020
Table I: Mean absolute error of prediction Kpred with respect to ground-truth Kst, The results are obtained on SQuAD Test Splitl and Test Split2.
Mean Absolute Error
Model
Test Splitl
Test Split2
Thresholding on Answer Logits
2.31
1.12
Fixed-K K (KPred = 5)
1.92
0.99
Dynamic-K Answer Prediction
1.24
0.76
Table Comparison of BLEU-4 scores with existing models on SQuAD Test Splitl and Test Split2. ASGen (Large) has 24 layers of encoder and decoder. Those models with are reproduced.
BLEU-4 score
Question Generation Model
Test Split1
Test Split2
Du et al. (2017)
12.3
Zhao ct al. 2018
12.8
14.9
ASs2s Kim ct al.
2019
16.2
Zhao et al. 2018)
16.4
UmLM (Dong et al.
2019)
22.1
23.8
ASGen (Full-Wiki)
21.5
24.7
ASGen (Full-Wiki) (Large)
25.4
28.0
an error of 2.31 and 1.12, respectively in the two splits. In contrast, our answer generator generates more appropriate number of answers than the fixed K approach, by reducing the mean absolute error between the ground-truth Ka and the prediction Krrind of 1.24 and 0.76, respectively for the two splits.
Question Generation. To evaluate our question generator, we fine-tune the model on both Test Split1 and Test Split2, after pre-training answer-containing sentence generation on Full-Wiki. As shown in Table 2 ASGen outperforms existing methods by 0.9 BLEU-4 score on Split2, 24.7 for ASGen vs. 23.8 for UniLM. Moreover, our final question generation model, ASGen (Large), outper- forms existing methods by large margin in BLEU-4 score on both splits, 25.4 for ASGen (Large) vs. 22.1 for UniLM for Splitl and 28.0 for ASGen (Large) vs. 23.8 for UniLM for Split2.
To show the effectiveness of our answer-containing sentence pre-training task (AS), we compare between various pre-training tasks. As shown in Table AS is shown to perform better than NS, e.g. 21.5 vs. 18.2 and 24.7 vs. 19.7 in the two splits, respectively. Note that conditioning on a given answer has only small effect on AS, e.g. 19.4 19.5. This implies the performance gain is largely due to pre-training on the answer-containing sentence generation task rather than conditioning on given answer.
We also compare the BLEU-4 scores between before and after applying AS on other existing ques- tion generation models. We reproduce Zhao al. (2018) and use the official code of Dong et al. (211)) As shown in Table g AS consistently improves the performance of other question genera- tion models with no architecture changes or parameter tuning.
Downstream Task Performance. We conduct experiments by training MRC models on the syn- thetic data generated by ASGen from Wikipedia before fine-tuning the model on the downstream dataset to show the effectiveness of our synthetic data generation. For each dataset, the MRC model pre-trained on the corresponding generated synthetic data and fine-tuned on the downstream data. As shown in Table 5 the MRC model pre-trained on the synthetic data generated by ASGen shows an improvement of 1.9 Fl score on SQuAD-v1.I 1, 4.0 Fl score on SQuAD-v2.00 and 0.5 Fl score on KorQuAD from the state-of-the-ar baseline models. Moreover, using the synthetic data generated from ASGen shows better performance than using the synthetic data generated from 'BertGen+NS' on both SQuAD-v1.1 and SQuAD-v2.0 downstream data.
Effects of MRC and Synthetic Data Size. Fig. 5 shows the effects of synthetic data with respect to the size of the synthetic and real MRC data. In Fig. (a). where we fix the size of synthetic data as
7

---

Under review as a conference paper at ICLR 2020
Table 3: Comparison among pre-training methods of the question generator in ASGen, i.c, without pre-training, pre-training NS, pre-training on AS, pre-training on AS without conditioning on a given answer. Note that we use Small-Wike for comparison of pre-training except those entries including "(Full-Wiki)".
BLEU-4 score
Model + pre-training method
Wikipedia Test-Wii Wiki
SQuAD v1.1
SQuAD v1.1 Test Split2
Test Splitl
BertGien
14.4
16.9
BertGien + NS
1.4
17.5
19.2
BertGen AS wol condition to given answer
6.8
19.4
21.2
BertGen + AS [ASGen]
7.1
19.5
22.2
BertGen NSS (Full-Wiki)
3.4
18.2
19.7
BertGen AS (Full-Wiki) [ASGen]
11.3
21.5
24.7
Table 4: Effects of pre-training on answer-containing sentence generation (AS) on other existing methods. We use Small-Wins data to pre-train existing models. Those models with * are reproduced.
BLEU-4 score
Model + pre-training method
Wikipedia Test- Diki
SQuAD v1.1
SQuAD v1.1 Test Split2
Test Splitl
Zhao al.
2018 *
12.8
14.9
Zhao et al.
2018
AS
6.8
14.0
16.2
UniLM Dong et al.
2019
22.1
23.8
UniLM Dong al.
2019 AS
8.9
22.9
24.5
43M, the Fl score of MRC model pre-trained the synthetic data generated by ASGen consistently outperforms that of BertGen+NS. In particular, performance difference becomes apparent for a small size of real MRC data, while the performance gap diminishes for large size. Such gap may become insignificant for sufficient size of real MRC data, but for the current size of SQuAD data (87K in total) AS still improves the performance.
As shown in Fig. (b),, we also conducted experiments by training the MRC model using a different amounts of generated synthetic data for the same number of iterations, while using the full size of real SQuAD data. The total number of training steps for all data sizes is kept the same as that of 10M synthetic data. A larger size of generated data consistently gives better performance.
Transfer Learning to Other Datasets. In this experiment, we first fine-ture ASGen using SQuAD- v1.1, and using synthetic data generated by this ASGen, train BERT MRC model. Afterwards, we fine-tune BERT for the downstream MRC task using QUASAR-T T, in order to verify that the data generated in this manner useful for other MRC datasets. QUASAR-T has two separate datasets, one with short snippets as context, and the other with long paragraphs context. As shown in Ta training with our synthetic data shown to improve the Fl score by 2.2 and 1.7 for the two cases, respectively.
QUALITATIVE RESULTS
Comparison of Question Generation. We qualitatively compare the generated questions after pre- training with NS and AS to demonstrate the effectiveness of our method. For the correct answer "49.6%" as shown in the first sample in Table NS omitted "Fresno", which is critical word to make the question specific, while AS's question does not suffer from this issue. Note that the word "Fresno" occurs in the answer-containing sentence. This issue also occurs in the second sample, where NS uses the word "available" rather than the more relevant words from the answer-con inswer-containing sentence, but AS uses many of these words such "most" and "popular" generate contextually rich questions. Also, the question from NS asks about "two" libraries, while the answer has "three" libraries, showing the lack of sufficient conditioning the answer. The third sample also shows that
8

---

Under review as a conference paper at ICLR 2020
Table 5: Comparison of EM/FI scores fine-tuned MRC model on SQuAD vI.1, SQuAD v2.0,
and KorQuAD dev sets using their corresponding synthetic data for pre-training.
SQuAD
v1.1
SQuAD
v2.0
KorQuAD
Model
Synthetic data
EM
F1
EM
F1
EM F1
83.9
90.9
78.8
81.8
BertGen
85.1
91.4
80.9
83.9
BERT
BertGen + NS
85.6
92.3
81.5
85.1
BertGen AS [ASGen]
86.3
92.8
82.5
85.8
87.1
94.5
BERT + CLKT
BertGen AS [ASGen]
87.8
95.0
seare MRMRC moddel on SQUAD deeset
94
Fl scoree MR model on SQUAD devoet
93.5
92
93.0
90
92.5
88
92.0
86
91.5
84
91.0
90.5
ASGen BertGen-NS wof Synthetic
80
90.0
1.25K
SK
20K
87K(Max)
150K
600K
2.5M
10M
(a) Used amount of MRC data (SQuAD-vI.I).
(b) Used amount of synthetic data.
Figure 5: Comparison of Fl of the BERT model on SQuAD-v1.1 dev set with different data sizes.
AS draws context-related questions than NS by including the exact subject "TARDIS" to for the corresponding answer in a similar vein.
RELATED WORK
Machine Reading Comprehension. For MRC tasks, a large number of datasets have been pro- posed, most often focused on finding an answer span for a question from given paragraph. Popular and fully human-annotated datasets include SQuAD-v1.1 (Rajpurkar et al. 2016), SQuAD-v2.0 (Rajpurkar al. 2018), KorQuAD (Lim et al. 2019), and HotpotQA (Yang al. 2018). However, these datasets are relatively small with around 100K samples each, which is far smaller than those datasets used for unsupervised tasks such language modeling.
Question Generation. Question generation methods have been actively studied for various purposes including data augmentation in question answering. Du al. (2017) proposed an attention-based model for question generation by encoding sentence-level as well as paragraph-Ievel information. Song et al. (2018) introduced query-based generative model jointly solve question generation and answering tasks. Kim et al. (2019) separately encoded the answer and the rest of the paragraph for proper question generation. Zhao et al.) (2018) utilized a gated self-attention encoder with max-out unit to handle long paragraphs. Our proposed method (AS) can further improve the ques- tion generation quality of these methods by pre-training them with an answer-containing sentence generation task.
Transfer Learning. Pre- Pre-training methods have been increasingly popular in natural language pro- cessing obtain contextualized word representations. Open-GPT (Radford et al. 2018), BERT (Devlin et al. (019a). XLNet Yang et al. 2019), and UniLM (Dong et 20,])019) use a Transformer module (Vaswani al. 2017) learn different styles of language models large dataset fol-

---

Under review as a conference paper at ICLR 2020
Table 6: Comparison of EM/FI scores of the BERT fine-tuned on QUASAR-T dataset. The synthetic data are generated from ASGen trained on SQuAD-v1.1.
Short(Dev)
Short(Test)
Len)(Dev)
Lens((Tsst
Model
Synthetic data
EM
F1
EM
F1
EM
F1
EM
F1
74.3
78.6
74.1
77.8
72.1
75.6
72.1
74.8
BERT
ASGen (SQuAD-v1.1)
76.5
80.1
76.5
80.0
74.2
77.4
73.8
76.5
Table 7: Examples from SQuAD-v1.1 dev set demonstrating generated questions. We compare our method (AS) with NS. Colored Text indicates a given answer.
The 2010 United States Census reported that Fresno had a population of
494,665, The population density was 4,404.5 people per square mile.
Context
(1,700.6km). The racial makeup of Fresno was 245,306 ( (49.6% White,
40,960 (8.3%) African American, 8525 (1.7%) Native American (omit)
Question by NS
What percent of the population is White?
Question by AS
What percentage of the Fresno population is White?
(omit) ... According to the American Library Association, this makes the largest academic library in the United States, and one of the largest in the world. Cabot Science Library, Lamont Library, and Widener Library are
Context
three the most popular libraries for undergraduates use, with ... (omit)
Question by NS
Which two libraries are available for undergraduates to use?
Question by AS
What are the three most popular libraries for undergraduates?
(omit) He fled from Gallifrey in stolen Mark Type TARDIS
Context
"Time and Relative Dimension in Space" time machine which allows him to travel across time and space. The TARDIS has a "chamelcon circuit"...(omit))
Question by NS
What does the doctor refer to?
Question by AS
What does the TARDIS stand for?
lowed by fine-tuning the downstream task. While our approach is similar to these approaches, our pre-training task for question generator generates answer-containing sentences to learn better representations for the question generation task.
Synthetic Data Generation. Subramanian et al. 2018 show that neural models generate better answers than using off-the-shelf tools for selecting named entities and noun phrases. Golub et al. (017)) proposed to separate the answer generation and the question generation. This model generates questions conditioned on generated answers, and then they evaluate the quality of the synthetic data by training an MRC model with them before fine-tuning on SQuAD. Inspired by the observations from previous studies, we improved the performance of answer generation and question generation by using a newly designed models as well as novel pre-training technique.
CONCLUSIONS
We propose two advanced training methods for generating high-quality and diverse synthetic data for MRC. First, we dynamically choose the top answer spans from an answer generator and then generate the sentence containing the corresponding answer span pre-training task for the ques- tion generator. Using the proposed methods, we generate 43M synthetic training samples and train the MRC model before fine-tuning the downstream MRC dataset. Our proposed method out- performs existing questions generation methods achieving new state-of-the-an the-art results on SQuAD question generation and consistently shows the performance improvement for the state-of state-of-the-aar -he-art models on SQuAD-v1.1. SQuAD-v2.0, KorQuAD, and QUASAR-T datasets without any architec- tural modification the MRC model.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Jacob Devlin, Ming-Wad Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019a.
Jacob Devlin, Ming-Wai Chang, Kenton Lee, and Kristina Toutanova. training of deep bidirectional transformers for language understanding. URL.
Bert: Pre-
https://mlp.ssaee
https//ntpossnnortlledusseeeeiee www.a//wwddddueeeiiiiiaraasiiiie pdf, 2019b.
Bhuwan Dhingra, Kathryn Mazaitis, and William Cohen. Quasar: Datasets for question answer- ing by search and reading, arXiv preprint arXiv:1707.03904, 2017.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. In Proceedings of Annual Meeting of the Association for Computational Linguis- tics (ACL), pp. 1342-1 1352, 2017.
David Golub, Po-Sen Huang, Xiaodong He, and Li Deng. Two-stage synthesis networks for transfer learning in machine comprehension. In Proceedings of the conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 835-844, 2017.
Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. Improving neural question gener- ation using answer separation. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), volume 33, pp. 6602-6609, 2019.
Taku Kudo. Mecab: Yet another part-of-speech and morphological analyzer. http://meccb. source- forge. jp, 2006.
Seungyoung Lim, Myungji Kim, and Jooyoul Lee. Korquad1 .O: Korean qa dataset for machine reading comprehension. arXiv preprint arXiv: 2019.
Matthew E Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HL.T), pp. 2227-2237, 2018
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language
wtw-wwettt...maaaaeeee understanding by generative pre-training. URL understanding paper.dff. 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the conference Empirical Methods in Natural Language Processing (EMNLP), pp. 2383-2392 2016.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), pp. 784-789, 2018
Linfeng Song, Zhiguo Wang, and Wael Hamza. A unified query-based generative model for ques- tion generation and question answering. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2018
Sandeep Subramanian, Tong Wang, Xingdi Yuan, Saizheng Zhang, Yoshua Bengio, and Adam Trischler. Neural models for key phrase extraction and question generation. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), pp. 78, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems (NIPS), pp. 5998-6008, 2017.

---

Under review as a conference paper at ICLR 2020
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the conference on Empirical Methods in Natural Language Processing
(EMNLP), pp. 2369-2380 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:/906.0880 1906.0827. 2019.
Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. Paragraph-Ievel neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3901-3910, 2018.
A DETAILS OF WIKIPEDIA PREPROCESSING
To build the answer-containing sentence generation data and the synthetic MRC data, we collect all paragraphs from all articles of the entire English Wikipedia dump (Korean Wikipedia dump for KorQuAD) and generate questions and answers these paragraphs. We apply extensive filtering and cleanup to only retain the highest-quality paragraphs from Wikipedia.
To filter out low-quality obscure pages, we remove all pages that received less than 200 cumulative page-views including all re-directions in 2-month period. In order to calculate the number of page- views, official Wikipedia page-vies vage-view dumps were used. Of the 5.4M original Wikipedia articles, filtering by page-views leaves 2.8M articles.
We also remove all pages with less than 500 characters, these pages are often low-quality stub articles, which removes a further 16% of the articles. We remove all "meta" namespace pages such as talk, disambiguation, user pages, portals, etc. as these often contain irrelevant text or casual con- versations between editors.
In order to extract usable text from the wiki-markup format of the Wikipedia articles, we remove extraneous entities from the markup including table of contents, headers, footers, links/URLs, im- age captions, IPA double parentheticals, category tables, math equations, unit conversions, HTML escape codes, section headings, double brace templates such info-boxes, image galleries, HTML tags, HTML comments and all other tables.
We then split the cleaned text from the pages into paragraphs, and remove all paragraphs with less than 150 characters or more than 3500 characters. Paragraphs with the number of characters between 150 to 500 were sub-sampled such that these paragraphs make up 16.5% of the final dataset, as originally done for the SQuAD dataset. Since the majority of the paragraphs in Wikipedia are rather short, of the 60M paragraphs from the final 2.4M articles, our final Wikipedia dataset contains 8.3M paragraphs.
B ADDITIONAL EXPERIMENT ON ANOTHER SQUAD SPLIT
We also evaluate the question generation model from Zhao et al. (2088) on another data split. We call this Test-Split3. Test Split3 is obtained by dividing the original development set in SQuAD-v into two equal halves randomly and choosing one of them as development set and the other as test set while retaining the train set in SQuAD-v 1. As shown in Table 8 the question generation model from Zhao et al. 2018) improves the BLEU-4 score on Test-Split3 by 1.3 (w.rt the reproduced score).
C STANDARD ERRORS OF EVALUATION DOWNSTREAM MRC TASKS
As shown in Table 9 in the case of downstream MRC results (EM/FI) which we dicussed in Sec- tion 4 for SQuAD v1.1 and SQuAD v2.0, we selected 5 model checkpoints from the same pre- training at varying numbers of pre-training steps. We then fine-tune each of these models the final downstream data 3 times each, picked the best performing model and reported it's score. For KorQuAD, only 1 finetuning was performed with the final pre-trained model.
12

---

Under review as a conference paper at ICLR 2020
Table Additional experiments for effectiveness of AS on SQuAD Test-Split3. We use Small-Wiki data to pre-train existing models. Those models with are reproduced.
BLEU-4 score
Model + pre- pre-training method
Wikipedia Test-Wii Wiki
SQuAD v1.1
Test Split3
Zhao al.
2018)
16.8
Zhao et al.
2018)
16.2
Zhao et al.
(2018) * + +S
6.8
17.5
Table 9: Standard errors of EM/F1 scores in downstream MRC tasks with ASGen.
Dev set
Test set
Dataset
EM
F1
EM
F1
SQuAD-v1.1
86.1(=0.2)
92.6(E0.1)
SQuAD-v2.0
82.3(=0.3)
85.6(+0.2)
QUASAR-T(Shor)
76.6(=0.1)
80.3(+0.2)
76.6(+0.2)
80.0(+0.3)
QUASAR-T(Long
74.9(=0.6)
78.1(+0.77
74.0(++0.8)
76.9(+0.9)
13


Reviews:
[{"review_id": "SyxuXwNFYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a pretraining technique for question generation, where an answer candidate is chosen beforehand, and the objective is to predict the answer containing sentence given a paragraph excluding this sentence and the target answer candidate. The intuition of this method is that question generation requires to generate a sentence which contains the information about the answer while being conditioned on the given paragraph. In particular, the paper compares its approach to Devlin’s presentation (https://nlp.stanford.edu/seminar/details/jdevlin.pdf according to the references; is it not a published work?) which uses next sentence generation for pretraining, that is less related to the downstream question generation task.\nThe proposed pretrained model is then used to finetune on a standard question generation task, which is then used to generate synthetic QA pairs for data augmentation in QA. The proposed method is evaluated on four datasets (SQuAD v1, SQuAD v2, KorQuAD, QUASAR-T) and have shown substantial performance gain.\n\nThe strength of this paper is clear to me: the idea in the paper is new and interesting, and they provide a good intuition of why the proposed learning objective is helpful compared to the previous methods.\n\nHowever, I have several concerns as follows.\n\nFirst, the performance improvements are marginal (less than 1.0 on three datasets, except QUASAR-T, which is actually interesting given it is under transfer learning setting).\n\nSecond, there is a very related work, “Dong et al. Unified Language Model Pre-training for Natural Language Understanding and Generation NeurIPS 2019” which also proposes a new pretraining approach for question generation and data augmentation for question answering. Not only this submission did not discuss this paper (it was posted in May 2019, which is substantially ahead of ICLR deadline), the result is substantially worse in both BLEU score of question generation and EM/F1 of end QA performance after data augmentation on SQuAD v2.\n\nThird, the paper does not discuss any work in question generation overall, although the task of question generation was studied for decades, including question generation as data augmentation for question answering. I believe this paper should discuss previous work in question generation and compare the performance with them.\n\nSome of the work on question generation.\n- Heilman & Smith. Good Question! Statistical Ranking for Question Generation. NAACL 2010.\n- Wang et al. Learning to ask questions in open-domain conversational systems with typed decoders. ACL 2018.\n- Zhao et al. Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Network. EMNLP 2018.\n- Kim et al. Improving Neural Question Generation Using Answer Separation. AAAI 2019.\n\nSome of the work on question generation for question answering.\n- Du et al. Learning to ask: Neural question generation for reading comprehension. ACL 2017.\n- Song et al. A Unified Query-based Generative Model for Question Generation and Question Answering. AAAI 2018.\n- Tang et al. Learning to Collaborate for Question Answering and Asking. NAACL 2018.\n- Sachan & Xing, Self-training for jointly learning to ask and answer questions. NAACL 2018.\n- Tang et al. Question Answering and Question Generation as Dual Tasks. AAAI 2018.\n- Lewis et al. Unsupervised Question Answering by Cloze Translation. ACL 2019.\n\nFourth, it looks like the paper uses the development set for both development and the final evaluation. The paper should either use a portion of the train set for development and treat the original development set as the test set, or just show the final model’s performance on the hidden test set by submitting to the leaderboard. (Especially, for Table 3, did authors reimplement BERT+NS and evaluate on the dev set? Because Devlin’s presentation only shows the result on the test set from the official leaderboard.)\n\n\n(This is not a weakness of the paper, but more a subjective opinion about the paper's claim) The paper claims answer-containing sentence generation is close to question generation. However, unlike question generation where core information about the answer is given as input text, answer containing sentence generation should generate the information about the answer itself with no given information. Consider the first example in Table 5. Question generation requires to read “The racial makeup of Fresno was 245,306 (49.6%)” and generate question “What percentage of the Fresno population is White?” However, in the proposed pretraining technique, the generation model is supposed to generate “The racial makeup of Fresno was 245,306 (49.6%)” with no information. In fact, I think the fact that answer candidate is given makes it closer to question generation task, rather than “generating answer-containing sentence” is the key.\n\n(By the way, I am giving 3 as a rating although my actual rating is closer to 4 or 5, because 4 or 5 is blocked in the review system. I am happy to increase the score after rebuttals.)\n\n--------------------------------------------------------------------------------------------------------------------------------------------\n\nNow, here are clarification questions.\n\nRegarding Section 2.1\n1) Did you attempt to predict ‘number of answer candidates’ by regression or classification? Which objective function did you use? Based on the current description, it looks like it’s neither regression nor classification which makes me confused.\n2) I believe the second last formula in this section should say “#{...}=k” instead of “#{...}”?\n3) Have you compared with thresholding instead of predicting the number of answer candidates?\n4) What is the objective of training the span selection model? There are multiple candidate spans given a single sentence, and hopefully you do not want to discourage candidates except only one.\n5) How was this model trained (both predicting the number and predicting the span)? Was it trained on QA datasets such as SQuAD? Then, did you train this model & go through preprocessing separately for each dataset? (Except QUASAR-I which uses synthetic data from SQuAD v1.)\n6) I believe that the most popular approach in question generation literature is selecting named entities & noun phrases using off-the-shelf tools, and wonder if authors have compared their method with this baseline.\n\nRegarding Section 2.2\n1) Although this section is the most important section in the paper, the description is a bit confusing to me. My understanding is that, if the initial paragraph has three sentences, <Sentence 1>, “The racial makeup of Fresno was 245,306 (49.6%).”, and <Sentence 3>, and the answer prediction model (described in Section 2.1) chooses “49.6%”, the question generation model is given “<Sentence 1> 49.6% <Sentence 3>”, and the model is supposed to generate  “The racial makeup of Fresno was 245,306 (49.6%).”. Is it correct?\n2) Did you insert any special token to indicate whether it is the sentence or the target answer candidate, or which sentence is before or after the target sentence?\n3) Have you tried the formulation of the masked language model, instead of the proposed model?\n\nRegarding Section 3\nIs preprocessing the same for all Golub et al 2017, NS and AS?\n\nRegarding Section 4\n1) I am curious why the improvement in BLEU-4 score (in Table 2) is significant but the improvement in the end task (in Table 3) is marginal. Do you have any intuition on it?\n2) For Figure 5(a), do you have results with BERT without any data augmentation as well? I wonder if using too small number of SQuAD annotations makes question generation inaccurate, which makes synthetic data be bad quality and hurts the performance compared to not using any synthetic data.\n3) How many synthetic QA pairs are used for Table 3? Curious since the model gets around 92.5 with the largest amount of data shown in Figure 5(b) but the number reported in Table 3 is 92.8.\n\n\n\n\n**** Update on Nov 10 **** Increasing the score to 6.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1xH2fEQcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper in the field of machine reading comprehension. The authors address the issue of generating labeled data of question-answer tuples, without the need of manual annotation. Specifically, the authors propose a method that dynamically generates K answers given a paragraph  in order to generate diverse questions and, secondly, pre-training the question generator on answers in a sentence generation task. The authors then show that this method is superior to existing baseline methods.\n\nOverall the paper is written rather well, however,  at times  it is tough  to understand because of the technical writing and heavy use of abbreviations.  The experiments make sense given the research question. \n\nI have a couple of questions:\n\n1.  Section 4&5: How often where these experiments repeated? What are the standard errors?\n2. How sensitive is the architecture w.r.t. its hyper-parameters?\n3. What was the training time/training cost when running the method? How does the number of parameters compare to previous state of the art?\n4. In Figure 5a the F1 scores of Bert+NS and Bert+AS seem to converge with increasing  data size. Why does the difference between those two methods seem to vanish when data set size increases?\n\n\nAdditional comment:  \"[..] researchers have proposed MRC models, often surpassing human performance.\"  This is a bold statement.  Machines definitely do not surpass humans in reading comprehensions.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for an interesting discussion. The paper introduces a sound question generation technique for QA. Reviewers are moderately positive, with low confidence. Some issues remain unresolved, though: While the UniLM comparison is currently not apples-to-apples, for example, nothing prevents the authors from using their method to pretrain UniLM. Currently, QA results are low-ish, and it is hard to accept a paper based solely on BLEU scores (questionable metric) for question generation (the task is but a means to an end). Moreover, the authors do not really discuss how their method relates to previous work (see Review 2 and the related work cited there; there's more, e.g., [0]). I also find it a little problematic that the paper completely ignores all work prior to 2017: The NLP community started organizing workshops on question generation in 2010. [1]
Decision: Reject