Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper in the field of machine reading comprehension. The authors address the issue of generating labeled data of question-answer tuples, without the need of manual annotation. Specifically, the authors propose a method that dynamically generates K answers given a paragraph  in order to generate diverse questions and, secondly, pre-training the question generator on answers in a sentence generation task. The authors then show that this method is superior to existing baseline methods.

Overall the paper is written rather well, however,  at times  it is tough  to understand because of the technical writing and heavy use of abbreviations.  The experiments make sense given the research question. 

I have a couple of questions:

1.  Section 4&5: How often where these experiments repeated? What are the standard errors?
2. How sensitive is the architecture w.r.t. its hyper-parameters?
3. What was the training time/training cost when running the method? How does the number of parameters compare to previous state of the art?
4. In Figure 5a the F1 scores of Bert+NS and Bert+AS seem to converge with increasing  data size. Why does the difference between those two methods seem to vanish when data set size increases?


Additional comment:  "[..] researchers have proposed MRC models, often surpassing human performance."  This is a bold statement.  Machines definitely do not surpass humans in reading comprehensions.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a pretraining technique for question generation, where an answer candidate is chosen beforehand, and the objective is to predict the answer containing sentence given a paragraph excluding this sentence and the target answer candidate. The intuition of this method is that question generation requires to generate a sentence which contains the information about the answer while being conditioned on the given paragraph. In particular, the paper compares its approach to Devlin’s presentation (https://nlp.stanford.edu/seminar/details/jdevlin.pdf according to the references; is it not a published work?) which uses next sentence generation for pretraining, that is less related to the downstream question generation task.
The proposed pretrained model is then used to finetune on a standard question generation task, which is then used to generate synthetic QA pairs for data augmentation in QA. The proposed method is evaluated on four datasets (SQuAD v1, SQuAD v2, KorQuAD, QUASAR-T) and have shown substantial performance gain.

The strength of this paper is clear to me: the idea in the paper is new and interesting, and they provide a good intuition of why the proposed learning objective is helpful compared to the previous methods.

However, I have several concerns as follows.

First, the performance improvements are marginal (less than 1.0 on three datasets, except QUASAR-T, which is actually interesting given it is under transfer learning setting).

Second, there is a very related work, “Dong et al. Unified Language Model Pre-training for Natural Language Understanding and Generation NeurIPS 2019” which also proposes a new pretraining approach for question generation and data augmentation for question answering. Not only this submission did not discuss this paper (it was posted in May 2019, which is substantially ahead of ICLR deadline), the result is substantially worse in both BLEU score of question generation and EM/F1 of end QA performance after data augmentation on SQuAD v2.

Third, the paper does not discuss any work in question generation overall, although the task of question generation was studied for decades, including question generation as data augmentation for question answering. I believe this paper should discuss previous work in question generation and compare the performance with them.

Some of the work on question generation.
- Heilman & Smith. Good Question! Statistical Ranking for Question Generation. NAACL 2010.
- Wang et al. Learning to ask questions in open-domain conversational systems with typed decoders. ACL 2018.
- Zhao et al. Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Network. EMNLP 2018.
- Kim et al. Improving Neural Question Generation Using Answer Separation. AAAI 2019.

Some of the work on question generation for question answering.
- Du et al. Learning to ask: Neural question generation for reading comprehension. ACL 2017.
- Song et al. A Unified Query-based Generative Model for Question Generation and Question Answering. AAAI 2018.
- Tang et al. Learning to Collaborate for Question Answering and Asking. NAACL 2018.
- Sachan & Xing, Self-training for jointly learning to ask and answer questions. NAACL 2018.
- Tang et al. Question Answering and Question Generation as Dual Tasks. AAAI 2018.
- Lewis et al. Unsupervised Question Answering by Cloze Translation. ACL 2019.

Fourth, it looks like the paper uses the development set for both development and the final evaluation. The paper should either use a portion of the train set for development and treat the original development set as the test set, or just show the final model’s performance on the hidden test set by submitting to the leaderboard. (Especially, for Table 3, did authors reimplement BERT+NS and evaluate on the dev set? Because Devlin’s presentation only shows the result on the test set from the official leaderboard.)


(This is not a weakness of the paper, but more a subjective opinion about the paper's claim) The paper claims answer-containing sentence generation is close to question generation. However, unlike question generation where core information about the answer is given as input text, answer containing sentence generation should generate the information about the answer itself with no given information. Consider the first example in Table 5. Question generation requires to read “The racial makeup of Fresno was 245,306 (49.6%)” and generate question “What percentage of the Fresno population is White?” However, in the proposed pretraining technique, the generation model is supposed to generate “The racial makeup of Fresno was 245,306 (49.6%)” with no information. In fact, I think the fact that answer candidate is given makes it closer to question generation task, rather than “generating answer-containing sentence” is the key.

(By the way, I am giving 3 as a rating although my actual rating is closer to 4 or 5, because 4 or 5 is blocked in the review system. I am happy to increase the score after rebuttals.)

--------------------------------------------------------------------------------------------------------------------------------------------

Now, here are clarification questions.

Regarding Section 2.1
1) Did you attempt to predict ‘number of answer candidates’ by regression or classification? Which objective function did you use? Based on the current description, it looks like it’s neither regression nor classification which makes me confused.
2) I believe the second last formula in this section should say “#{...}=k” instead of “#{...}”?
3) Have you compared with thresholding instead of predicting the number of answer candidates?
4) What is the objective of training the span selection model? There are multiple candidate spans given a single sentence, and hopefully you do not want to discourage candidates except only one.
5) How was this model trained (both predicting the number and predicting the span)? Was it trained on QA datasets such as SQuAD? Then, did you train this model & go through preprocessing separately for each dataset? (Except QUASAR-I which uses synthetic data from SQuAD v1.)
6) I believe that the most popular approach in question generation literature is selecting named entities & noun phrases using off-the-shelf tools, and wonder if authors have compared their method with this baseline.

Regarding Section 2.2
1) Although this section is the most important section in the paper, the description is a bit confusing to me. My understanding is that, if the initial paragraph has three sentences, <Sentence 1>, “The racial makeup of Fresno was 245,306 (49.6%).”, and <Sentence 3>, and the answer prediction model (described in Section 2.1) chooses “49.6%”, the question generation model is given “<Sentence 1> 49.6% <Sentence 3>”, and the model is supposed to generate  “The racial makeup of Fresno was 245,306 (49.6%).”. Is it correct?
2) Did you insert any special token to indicate whether it is the sentence or the target answer candidate, or which sentence is before or after the target sentence?
3) Have you tried the formulation of the masked language model, instead of the proposed model?

Regarding Section 3
Is preprocessing the same for all Golub et al 2017, NS and AS?

Regarding Section 4
1) I am curious why the improvement in BLEU-4 score (in Table 2) is significant but the improvement in the end task (in Table 3) is marginal. Do you have any intuition on it?
2) For Figure 5(a), do you have results with BERT without any data augmentation as well? I wonder if using too small number of SQuAD annotations makes question generation inaccurate, which makes synthetic data be bad quality and hurts the performance compared to not using any synthetic data.
3) How many synthetic QA pairs are used for Table 3? Curious since the model gets around 92.5 with the largest amount of data shown in Figure 5(b) but the number reported in Table 3 is 92.8.




**** Update on Nov 10 **** Increasing the score to 6.

review_title: Official Blind Review #2

Meta-review: Thanks for an interesting discussion. The paper introduces a sound question generation technique for QA. Reviewers are moderately positive, with low confidence. Some issues remain unresolved, though: While the UniLM comparison is currently not apples-to-apples, for example, nothing prevents the authors from using their method to pretrain UniLM. Currently, QA results are low-ish, and it is hard to accept a paper based solely on BLEU scores (questionable metric) for question generation (the task is but a means to an end). Moreover, the authors do not really discuss how their method relates to previous work (see Review 2 and the related work cited there; there's more, e.g., [0]). I also find it a little problematic that the paper completely ignores all work prior to 2017: The NLP community started organizing workshops on question generation in 2010. [1]
Decision: Reject