Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper presents a model for unsupervised pre-training for the task of question generation. The model first predicts the number of answer present in a given paragraph and then selects the top-K answer spans from the paragraph. After selecting the answer spans, the model, then tries to generate the answer containing sentence by inputting the paragraphs less the answer-containing sentence and also the answer span. The key idea is that this unsupervised pre-training strategy is close to the actual task of generating question given the context and the answer. This is also the key differentiator between this work and other existing pre-training strategies for question generation (e.g. Alberti et al 2019). 

The paper outperforms other existing methods of question generation on several datasets (two splits of Squad, MS Marco, NewsQA) both on automatic and a small scale human evaluation. Efficacy of the pre-training scheme is shown via the fact that the pre-training scheme also improves other question generation model. Lastly, training a QA model on the synthetic generated questions improves downstream performance of a QA model and the difference is greater in low-data setting suggesting the applicability of this pre-training scheme in low-data regime (although no experiments in a specific low-data domain is reported).

Strengths:
* The pre-training scheme is closer to the original task than other existing methods and can be easily scaled.
* The paper is well written and easy to follow and the experiments and ablations were exhaustive.
* Improvements in low-data QA setting is promising and shows the scope of usefulness of this work

Weaknesses
* I am not sure about the need to predict the number of unique answers in a paragraph or that if it makes sense to do so. I think it is hard to estimate the number of questions that can be generated from a given text and regressing to a value which is present in current datasets might be suboptimal. This is because, the current QA datasets do not ask annotators to generate all possible question from a paragraph. Instead, I believe you could have taken a principled approach of considering named-entities as answers, and trying to generate questions for each entity (or a set of entities together). This would also eliminate the need to predict the number of answer spans in a paragraph at once.
* I think it would be useful for the paper to categorize the kinds of “reasoning” that the generated question requires to answer. For example, is it just fuzzy pattern matching kind of questions or does answering the question require any kind of reasoning (such as multi-hop or numerical reasoning). 
* It was not clear to me, how you generate questions which are unanswerable (e.g. those in Squad 2.0)
* Although, it is good to see that training on the synthetically generated questions help in low-data settings, I think the paper would be stronger and more convincing. if an actual experiment was done on a domain which has less data. For example, you could try the BioASQ datasets which have very less annotations and is in the bio-medical domain. It would be interesting to see if pre-training on a scientific corpus is actually helpful. Moreover, some questions in BioASQ need reasoning such as handling list questions, counting and it would be interesting to see if the performance on those questions improve.

Overall, I think the current experiments are reasonably well-done and I think the paper would be much stronger if it was tested on an actual domain which has low-data and also if the paper discusses / categorizes the kind of reasoning that the generated questions require. 
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a pretraining technique for question-generation models. The pretraining involves generating the sentence which contains an answer candidate from the context around it. Through several well-executed experiments the paper shows that this type of pretraining can improve performance of several existing question generation models, and the resulting synthetic questions generated help in augmenting reading comprehension datasets.

Strengths:
- This is solid empirical work, with several detailed experiments showing the utility of the pretraining method on multiple benchmarks and with multiple models. It was particularly nice to see evaluation on Korean as well as English.

- The authors clearly put effort in explaining all the methods and experiments precisely and with details.

Weaknesses:
- The technical contributions of the paper are rather limited. Pretraining has been of much interest in the NLP community recently, and this paper follows a line of related works which have proposed similar techniques (e.g. Inverse Cloze Task, Lee et al, ACL 2019; T5, Raffel et al, JMLR 2020). There isn't much discussion of these existing papers either.

- It is not clear how much of the benefit of pretraining comes from the specific approach used here, versus the fact that there is some pretraining on the decoder which generates the questions. We could learn more about this if there was a comparison to other pretrained models which have a decoder, e.g. T5, BART (Lewis et al, 2020).

- In terms of the question generation task, it is well known now that Squad questions have a bias towards high lexical overlap with the passage (due to the manner in which they were constructed, see Lee et al above). This raises the question whether the approach in this paper can generalize to datasets where this bias does not exist, e.g. Natural Questions. This seems to be a limitation of not just paper, but the prior works as well.

Other comments / questions:
- Where do the ground truth answers come from during pretraining? Is the answer prediction model also pretrained?

- In section 2.1, it says that the MSE loss is computed using K, but K comes from the floor function which does not support backpropagation. Do you use the output f_k instead to compute the MSE?

- In the same section, I assume a softmax is applied on s_i, e_i,j before taking the cross-entropy loss. In this case, is e_i,j normalized over all spans in the passage or only the ones which start at i?

- Is w^o_t an embedding or a probability? There seems to be confusion at the end of section 2.

- Some more discussion of the UniLM baseline would be good for people not familiar with that work.

- Missing discussion of related work: "Yang, Zhilin, et al. "Semi-supervised qa with generative domain-adaptive nets." ACL (2017)."

- How does the model generate unanswerable questions for Squad 2.0?
review_title: Solid empirical work but limited technical contributions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The submission introduces a new approach for pre-training models for question generation conditioned on an answer and a document. Given a document, a set of k answers is chosen by a classifier. Then, the sentence containing the answer is removed, and a sequence to sequence model is used to reconstruct it. This pre-training task more closely matches the desired end task. Results show that this method improves question generation performance compared to previous work, and that the synthetic questions can be used to improve question answering models. Overall, I think the approach makes a lot of sense. However, I think it needs to compare with stronger pre-training baselines, and needs a more thorough comparison of the effect on question answering performance.

The paper primarily evaluates on question generation for a range of datasets, and shows improvements on both automated and human evaluation metrics. I think it's fair to say that the question generation is quite a niche task, perhaps because of limited applications. Indeed, the abstract of the paper pitches the task as being primarily useful for generating synthetic data for question answering. That's fine, but given this motivation, I think the evaluation should focus much more on the downstream impact on question answering.

I'd really like to see if the method still improves question generation performance on top of more recent sequence to sequence models, such as T5 or BART, which have both been available for almost a year. As far as I know, these both significantly outperform UniLM on all the comparisons I've seen. The pre-training objectives used there, which involve predicting masked spans, seem closely related to the proposed method. This experiment would help understand whether the proposed method adds anything on top of more general pre-training approaches.

The paper convincingly demonstrates that synthetic questions improve a baseline BERT question answering model, which was already shown by e.g. Alberti et al (2019). However, I don't think the paper does much to suggest that the synthetic questions from the proposed method are better for QA than other approaches - for example, Alberti et al. appear to report similar numbers with their method. It's important to include a baseline set of synthetic questions, as it's not at all clear that improvements on question generation metrics will correlate with usefulness for QA training.

Again, the question answering results would be more convincing if they hold up with more recent models than BERT, which achieve better scores without synthetic questions. Table 11 does give a result using Electra - but gives the baseline Electra model an exact match score of 87.4. The Electra paper instead lists 88.0, slightly higher than the results using synthetic questions.
review_title: Method makes sense, but needs more experiments

final_score: rating: 7: Good paper, accept
review_content: ### Summary

This work presents a new multi-step method to pre-train a question generation system, which can then be used to create synthetic data to improve a Machine Reading Comprehension system.

First, the authors train a system to identify spans in a text paragraphs which would constitute likely answers for questions about the paragraph. Then, they pre-train a system to generate those questions by taking the selected answer and surrounding sentences as input, and generating the sentence which contained the answer. Finally, they use real MRC data to fine-tune the question generation system.

The authors first provide a direct evaluation of their method by showing a consistent improvement in reference-based metrics comparing to gold questions for a given paragraph answer, then shows that the generated synthetic data also leads to improvement on the downstream MRC task for the SQuAD dataset, especially when using less annotated data.

The results provided are encouraging and accompanied by a number of enlightening supporting experiments. The paper could still be improved by clarifying specific points.

### Clarity

While the experimental setting description gives a broad idea of what the authors did, some details are missing for full reproducibility. Most notably, there is very little information on the first step setup, training the answer selection system. The authors also describe their setting based on BERT but not the one with UniLM

What is the dimension of the indicator vector that tells the model where the answer is located? Is there a particular reason why the authors decided to use an indicator vector rather than e.g. use a [SEP] token?

Dynamic prediction of the number of answers in a paragraph doesn't seem to account for much improvement. Can the authors provide some measure of statistical significance? Otherwise, the method really is novel enough without that part.

Did the model evaluate the effect of using synthetic data on the other tasks?

### Correctness

The claims are mostly well supported by the experiments (to the exception of the dynamic K prediction).

##### Additional citations
The pre-training sentence prediction method is related to:  
**Latent retrieval for weakly supervised open domain question answering**, Lee et al., *ACL 2019*


review_title: A new method to pre-train question generation systems for MRC data augmentation which leads to consistent improvements. The initial selection step needs to be exposed better but the findings are otherwise supported by thorough experiments.

Meta-review: All reviewers appreciate the good quality of this submission with a good idea and solid execution (as said by R3). The paper is clearly written and the addition during the discussion have greatly improved it as acknowledged by all reviewers.

However, a major weakness of the submission still needs to be addressed before a publication at ICLR. 

As said in the paper, the task of question generation is a task whose main impact is to improve downstream tasks, and primarily QA. The evaluations follow that and extra-experiments (e.g. BioASQ) and discussion wrt state of the art (e.g. Alberti et al.) reinforce them. Yet, as pointed out by R1 & R4, the effect on downstream QA performance is only shown for weaker models than the current state of the art (e.g. T5, BART). Since the rebuttal period was not long enough to run these experiments, it is impossible to assess how the proposed approach compare to them with the current draft. Adding the experiments on T5 (Small) is a step in the right direction but it is not enough for that. Without those experiments, one can not conclude that this pretraining strategy would also help over the strongest existing pretrained model.

The authors should run those experiments to make the arguments presented in the submission much stronger.










Decision: Reject