Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Thanks for the nice paper. I respect its technical contributions although I am still wondering the objectives of this research.

1. Long vs short answers. I believe the key for a good answer is not about the length of the answer, but how much information and/or how interesting the answer is. At the first glance people may feel impressed by long answers from the machine, but in practice if the answer is not 100% relevant the audience may be more annoyed. In reality, people may prefer to give both long and short conversations in one conversation. Shouldn't machines do that too?

2. It seems to me harder to evaluate the long answer than the short answers. Shall we consider the context in addition to correctness? 

3. In the conversation, what will happen if the questionnaire is composed with more than one questions?

review_title: a few concerns

final_score: rating: null
review_content: The description of the "target-glimpse model" on page 4 and figure 1(b) do not seem to fit together. In particular which symbols y_i are on the decoder side do not seem to fit between them. Please clarify - maybe I am not reading the figure right - but since the figure uses different letters it is not clear what the issue is. 
review_title: Clarification question

final_score: rating: 7: Good paper, accept
review_content: The paper is clearly interesting in that it does address important problems (length and diversity of responses) in sequence-to-sequence models. The two ideas put forward (glimpse model and segment-based stochastic decoding) both seem ideas in the right direction. I was however not so sold on the argument that these are particularly suitable for conversations.

The results indicate that the ideas do indeed generate longer and also somewhat more sensible target sequences and as such the paper makes progress w.r.t these important problems. So overall I would suggest accepting the paper even though the flavor of the proposed ideas are somewhat "small steps". 
review_title: Interesting ideas and interesting results

final_score: rating: 7: Good paper, accept
review_content: This paper considers the problem of generating long and diverse responses in dialog systems. Two techniques are proposed to the seq-to-seq framework: (1) glimpse model that trains on fixed-length segments of the target side at a time, and (2) a segment-based stochastic decoding technique which injects diversity earlier in the generated responses. The large scale experiments on 2.3B conversation messages are quite impressive.  Experiments on human evaluation should also be encouraged. 

With all these said, I am still not 100% convinced that machine generated long sequence is the right direction for dialog systems. As shown in Figure 3 (a), human evaluation shows that the proposed system is not significantly better than the baselines. I think more analysis and user preference mining should be done in the future to help us understand the nature of this problem. 
review_title: Good efforts

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes modification to seq2seq model to help it handle the problems when long responses are needed. 
Though the technical contributions may be of value, the work in my personal opinion is not in the right direction towards helping dialog systems. Essentially we try to generate long responses that sound ``nice" yet are not grounded to any reality, they just need to be related to the question and not suffers from obvious mistakes. Yet, the architectural innovations proposed may be of merit.
review_title: A method to produce long and diverse responses to questions, yet not a method that pushes forward dialog systems

final_score: rating: null
review_content: The paper advocated a conversation model but the proposed model seems to be used for single question answering only. I believe for conversational agents the main challenge is not how to produce long and diverse responses but responses that are coherent and not contradicting as conversation advances and as more context is built. Is this true?

If we assume the proposed model is for single question answering, would you motivate an application for this model which requires long and diverse answers? E.g., if a conversation agent is to be used a mobile phone and the query from the user is: “do you think I should wear boots today?” then the answer better be related to the weather outside, instead of irrelevant to context but long and diverse. Could you motivate the problem of producing long and diverse responses to single,  unrelated to each other, questions?

Else, would you mind testing your proposed model to a sequence of three consecutive related to each other question to see how it performs (in a short dialog that is)? 

One last question is: do the same questions appear in the training set? I would expect yes. Would you like to show the corresponding responses, to evaluate against a nearest neighbor baseline, or even just have a qualitative comparison against a nearest neighbor baseline?

review_title: Motivation and nearest neighbor baseline

Meta-review: The reviewers agree that the work presents interesting, but incremental, results. They are also unconvinced that this is the right direction for research on dialogue systems to go in or that the methods presented will appeal to a broader audience. It seems the authors have some work to do on the larger argument, before this work can be accepted.
Decision: Reject