Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors propose a scheme to generate questions based on some answer sentences, topics and question types. Topics are extracted from questions using similar words in question-answer pairs. It is similar to what we find in some Q&A systems (like lexical answer types in Watson). A sequence classifier is also used to tag the presence of topic words. Question types correspond mostly to salient questions words. LSTMs are used to encode the various inputs and generate the questions. 

The paper is well written and easy to follow. I would expect more explanations why sentence classification and labeling results presented in Table 2 are so low. 

Experimental results on question generation are convincing and clearly indicate that the approach is effective to generate relevant and well-structured short questions. 

The main weakness of the paper is the selected set of question types that seems to be a fuzzy combination of answer types and question types (for ex. yes/no). Some questions type can be highly ambiguous; for instance “What” might lead to a definition, a quantity, some named entities... Hence I suggest you revise your qt set. 

I would also suggest, for your next experiments, that you try to generate questions leading to answers with list of values. 
review_title: Interesting paper on a question generation approach focusing on some topics and question type with promising experimental results. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposed a topic-based question generation method, which requires the input of target topic in addition to the descriptive text. In the proposed method, the authors first extract the topic based on the similarity of the target question token and answer token using word embedding. Then, the author proposed a topic-specific question generation model by encoding the extracted topic using LSTM and a pre-decode technique that the second decoding is conditioned on the hidden representation of the first decoding result. The authors performed the experiment on AQAD dataset, and show their performance achieve state-of-the-art result when using automatically generated topic, and perform better when using the ground truth topic. 

[Strenghts]

This paper introduced a topic-based question generation model, which generate question conditioned on the topic and question type. The authors proposed heuristic method to extract the topic and question type without further annotation. The proposed model can generate question with respect to different topic and pre-decode seems a useful trick. 

[Weaknesses]

This paper proposed an interesting and intuitive question generation model. However, there are several weaknesses existed:

1: It's true that given a descriptive text, it is often possible to ask many types of questions. But it also leads to different answers. In this paper, the authors treat the descriptive text as answers, is this motivation still true if the question generation is conditioned on answers, not descriptive text? Table 4 shows some examples, given the sentence, even conditioned on different topics, the generated question is similar. 

2: In terms of the experiment,  the authors use AQAD to evaluate proposed method. When the ground truth topic is provided, it's not fair to compare with the previous method, since knowing the similar word present in the answer will have great benefits to question generation. 

If we only consider the automatically generated topic, the performance of the proposed model is similar to the previous method (Du et al). Without the pre-decode technique, the performance is even worse. 

3: In section 4.2, the authors claim this is the theoretical explanation of the generalization capability of the proposed model (also appear in topic effect analysis). It is true that the proposed method may have better compositionality, but I didn't see any **theoretical** explantation about this. 

4: The automatically extracted topic can be very noisy, but the paper didn't mention any of the extracted topics on AQAD dataset. 

[Summary]

a topic-based question generation method, which requires the input of target topic in addition to the descriptive text. However, as I pointed out above, there are several weaknesses in the paper. Taking all these into account, I think this paper still needs more works to make it solid and comprehensive before being accepted.
Add Comment
review_title: Review from AnonReviewer1

final_score: rating: 3: Clear rejection
review_content: This paper presents a neural network-based approach to generate topic-specific questions with the motivation that topical questions are more meaningful in practical applications like real-world conversations. Experiments and evaluation have been conducted on the AQAD corpus to show the effectiveness of the approach.

Although the main contributions are clear, the paper contains numerous typos, grammatical errors, incomplete sentences, and a lot of discrepancies between text, notations, and figures making it ambiguous and difficult to follow. 

Authors claim to generate topic-specific questions, however, the dataset choice, experiments, and examples show that the generated questions are essentially keyword/key phrase-based. This is also apparent in Section 4.1 where authors present some observation without any supporting proof or empirical evidence. Moreover, the example in Figure 1 shows a conversation, but, typically, in an ongoing multi-round conversation people do not tend to repeat the keywords or key phrases or named entities, and topic shifts might occur at any time. 

Overall, a misconception about topic vs. keywords might have led the authors to claim that their work is the first to generate topic-specific questions whereas this has been studied before by Chali & Hasan (2015) in a non-neural setting. "Topic" in general has a broader meaning, I would suggest authors to see this to get an idea about what topic entails to in a conversational setting: https://developer.amazon.com/alexaprize/contest-rules . I think the proposed work is mostly related to: 1) "Towards Natural Question-Guided Search" by Kotov and Zhai (2010), and 2) "K2Q: Generating Natural Language Questions from Keywords with User Refinements" by Zheng et al. (2011), and other recent factoid question generation papers where questions are generated from a given fact (e.g. "Generating Factoid Questions With Recurrent Neural Networks: The 30M Factoid Question-Answer Corpus" by Serban et al. (2016)).

It is not clear how the question types are extracted from the given sentences. Please provide details. Which keywords are employed to accomplish this? Also, please explain the absence of the "why" type question. 

Figure 3 and the associated descriptions are very hard to follow. Please draw the figure by matching it with the descriptions.  Where are the bi-LSTMs in the figure? What are ac_t and em_t? 

My major concern is with the experiments and evaluation. The dataset essentially contains questions about product reviews and does not match authors motivation/observation about real-world conversations. Moreover, evaluation has been conducted on a very small test set (just about 1% of the selected corpus), making the results unconvincing. More details are necessary about how exactly Kim's and Liu's models are used to get question types and topics. 

Human evaluation results per category would have been more useful. How did you combine the scores of the human evaluation categories? Also, automatic evaluation and human evaluation results do not correlate well. Please explain.








review_title: Motivation, experiments, and evaluation are flawed

Meta-review: The pros and cons of the paper under consideration can be summarized below:

Pros:
* Reviewers thought the underlying model is interesting and intuitive
* Main contributions are clear

Cons:
* There is confusion between keywords and topics, which is leading to a somewhat confused explanation and lack of clear comparison with previous work. Because of this, it is hard to tell whether the proposed approach is clearly better than the state of the art.
* Typos and grammatical errors are numerous

As the authors noted, the concerns about the small dataset are not necessarily warranted, but I would encourage the authors to measure the statistical significance of differences in results, which would help alleviate these concerns.

An additional comment: it might be worth noting the connections to query-based or aspect-based summarization, which also have a similar goal of performing generation based on specific aspects of the content.

Overall, the quality of the paper as-is seems to be somewhat below the standards of ICLR (although perhaps on the borderline), but the idea itself is novel and results are good. I am not recommending it for acceptance to the main conference, but it may be an appropriate contribution for the workshop track.
Decision: Invite to Workshop Track