Title: Generating Long and Diverse Responses with Neural Conversation Models
Authors: Louis Shao
Stephan Gouws
Denny Britz
Anna Goldie
Brian Strope
Ray Kurzweil
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning

Abstract: Building general-purpose conversation agents is a very challenging task, but necessary on the road toward intelligent agents that can interact with humans in natural language. Neural conversation models -- purely data-driven systems trained end-to-end on dialogue corpora -- have shown great promise recently, yet they often produce short and generic responses. This work presents new training and decoding methods that improve the quality, coherence, and diversity of long responses generated using sequence-to-sequence models. Our approach adds self-attention to the decoder to maintain coherence in longer responses, and we propose a practical approach, called the glimpse-model, for scaling to large datasets. We introduce a stochastic beam-search algorithm with segment-by-segment reranking which lets us inject diversity earlier in the generation process. We trained on a combined data set of over 2.3B conversation messages mined from the web. In human evaluation studies, our method produces longer responses overall, with a higher proportion rated as acceptable and excellent as length increases, compared to baseline sequence-to-sequence models with explicit length-promotion. A back-off strategy produces better responses overall, in the full spectrum of lengths.

TL;DR: We generate high quality and informative open-domain conversation responses using seq2seq model with target-side attention and stochastic beam search with segment-by-segment reranking. 

Review discussion:
The description of the "target-glimpse model" on page 4 and figure 1(b) do not seem to fit together.
In particular which symbols y_i are on the decoder side do not seem to fit between them. Please clarify - maybe I am not reading the figure right - but since the figure uses different letters it is not clear what the issue is.
Yes, the symbols used in the figure are different from those in the formula for visual purposes. The correspondence is as follows. In the example given, X = (A, B, C, D, E), Y_1 = (>, a, b) and Y_2 = (c, d, e), as indicated by the colors in the figure. The decoder is trained using teacher forcing, i.e. with (d, e, f) as labels and fixed next inputs. We will update the figure description to make the correspondence clear.
Thank you for the comment.
Adding to Denny's comment, in case source is (A, B, C, D, E) and target is (a, b, c, d, e), the original seq2seq model would train with
Encoder Input: (A, B, C, D, E, eos)
Decoder Input: (bos, a, b, c, d, e)
Decoder Label: (a, b, c, d, e, eos)
The glimpse model with K=3 will train with 2 examples:
Example 1:
Encoder Input: (A, B, C, D, E, eos)
Decoder Input: (bos, a, b)
Decoder Label: (a, b, c)
Example 2:
Encoder Input: (A, B, C, D, E, bos, a, b, eos)
Decoder Input: (c, d, e)
Decoder Label: (d, e, eos)
We didn't mention that the eos in the encoder input is always put at the end, because we think it is a minor issue. If it is not confusing we will revise this part.
The paper advocated a conversation model but the proposed model seems to be used for single question answering only. I believe for conversational agents the main challenge is not how to produce long and diverse responses but responses that are coherent and not contradicting as conversation advances and as more context is built. Is this true?
If we assume the proposed model is for single question answering, would you motivate an application for this model which requires long and diverse answers? E.g., if a conversation agent is to be used a mobile phone and the query from the user is: “do you think I should wear boots today?” then the answer better be related to the weather outside, instead of irrelevant to context but long and diverse. Could you motivate the problem of producing long and diverse responses to single,  unrelated to each other, questions?
Else, would you mind testing your proposed model to a sequence of three consecutive related to each other question to see how it performs (in a short dialog that is)?
One last question is: do the same questions appear in the training set? I would expect yes. Would you like to show the corresponding responses, to evaluate against a nearest neighbor baseline, or even just have a qualitative comparison against a nearest neighbor baseline?
Thank you for your thoughts! I agree that modeling context is important and challenging. We are working on that too.
However, generating interesting and informative responses, I believe, is another largely orthogonal issue that we have to solve, in order to make the user experience engaging. If the goal is to “answer” queries in a more task-oriented setting, then a short and precise answer is sufficient and efficient. If instead the goal is to make an assistant that interact with you for a long time, you may also want it to be interesting and inspiring, like a human friend. The effort to better model context happens more on the encoder-side.
Our work assumes that this is given, and that the model indeed gets the idea correctly, and then we look into how we can generate a more informative response. For the query you mentioned, “do you think I should wear boots today? ” the bot could answer “No”, or “No it may be too warm for you today. ” However I believe a real friend may say something like the following (generated by our model in 30 runs, and deduplicated).
I have no idea. I don't wear any of the boots I have in the closet.
I don't want to wear my boots with my pants on, but I will dress up and wear them every day.
I don't wear them but I think they look good.
I think you should. I don't think I've ever seen a woman wear boots in her entire life.
I don't know... I just got back from my first day of work and I couldn't resist. It was a good day.
Yes you do.
I think you should dress them up a bit.
I don't think so. I wear them all the time.
I think you look great.
Yes.
I think you should.
I think you should, it really depends on how you dress
I don't think so.
i think you should
No.
you can wear them if you want to
I would probably wear them day after day.
Yes, please
Yes, I think you should. You look like you could use a walk. :)
I think you should!
I would go for it.
These responses are not consistent regarding how it understands itself and how it understands the recipient, and in case there are further conversation history, our model is not modeling that either.
However, our model does a good job of giving informative and diverse response in the context-free setting. The next step would be to enhance the encoder to incorporate more context. That being said, in this paper we prefer not to test our model in the case of three consecutive turns, since we are not training the model in that way.
For the last question, yes we will try to put comparisons with the response-retrieval methods. The most relevant work may be this one (https://arxiv.org/abs/1606.00372) done by our colleagues. From that paper you can also see how context can be incorporated to help the retrieval. The current situation is that when we use our generative model as a ranker / scorer, it does a much better job than the ranking model in the above paper, in terms of the 100-choose-1 metric mentioned in our paper, when both models are trained with only the single query as input.
However, the retrieval models are cheaper and can incorporate more information easily. The drawback is that the retrieval method works well only if the required answer is short.
When the answer needed is informative, the chance of having an answer in the data set which is relevant in all the ways is low, due to the combinatorial nature. A generation approach could resolve that issue and our paper is an effort towards that. Adding extra context to the encoder in the form of similar sequence components is expensive. We are experimenting with different simplifications like those did in the above paper and will likely update you in the future works.
Hi, I worked on the last question you raised.
In terms of response retrieval baselines, there are two main approaches.
The first one is to learn a model (such as a feedforward network) which takes as input the prompt and the response, and output a score indicating how much the response is favored. Our seq2seq model outputs the log probability of the response given the prompt, therefore it can also be used for this scoring purpose.
While in [1] a deep network is learned on top of the bag-of-words representations of the prompt and the response. In that case, the classifier is trained by taking the true response as the positive examples and random responses as negative examples.
The second approach is to define or learn a latent representation of the prompt and the response separately, such that the prompt is closer to the true response rather than a randomly-chosen response (e.g., [2], [3]). In this case a nearest neighbor search can be adopted in the testing time to find the response given a prompt. We have experimented with learning such a representation using deep feedforward networks similar to [1] and found that the corresponding classification network always perform better in terms of the N-choose-1 metric, when the network size is similar.
Furthermore, we have tested that a network like [1], given only the prompt and the response as input, without additional context, gives 27% in the 100-choose-1 metric. It is quite a bit worse than our seq2seq model.
We will briefly mention these results in our next revision.
However, a detailed comparison may be out of the scope of this paper because both approaches mentioned above have many variants that we can try.
[1] https://arxiv.org/abs/1606.00372
[2] https://arxiv.org/abs/1408.6988
[3] http://dl.acm.org/citation.cfm?id=2911542
Thanks for the nice paper. I respect its technical contributions although I am still wondering the objectives of this research.
1. Long vs short answers. I believe the key for a good answer is not about the length of the answer, but how much information and/or how interesting the answer is. At the first glance people may feel impressed by long answers from the machine, but in practice if the answer is not 100% relevant the audience may be more annoyed.
In reality, people may prefer to give both long and short conversations in one conversation. Shouldn't machines do that too?
2. It seems to me harder to evaluate the long answer than the short answers. Shall we consider the context in addition to correctness?
3. In the conversation, what will happen if the questionnaire is composed with more than one questions?
Thank you for the comments! I'll try to answer separately.
1. For the last part, yes our model does give short answers when necessary. You can look at the responses I put in the last comment for AnonReviewer1. The duplicates removed there are all "yes" and "no".
The majority of our training data consists of reddit comments. These comments are long for most of the time (average 38 word-piece tokens). And the nature of these comments is that people at least try to make them interesting. That means, if we are able to get unbiased samples from the true p(response | prompt), we would get responses with the similar average length and a certain amount of interestingness.
However, due to many problems we mentioned in the introduction, the generated responses from the existing seq2seq models often tend to be shorter, generic, less informative, and therefore less interesting. There are two orthogonal approaches we can use to improve the quality of the responses: (1) we improve the model to be closer to the true model and try to make the samples more unbiased. (2) we use heuristics to alter the training and sampling, e.g., we could train on the comments whose numbers of up-votes are above a threshold. Our paper focuses more on the first approach. We believe that once we addressed (1), we have more confidence when working on (2).
I totally agree with your concern that long answers are harder to get right in all aspects. From a product perspective, it is a valid choice to avoid this problem and go with short answers only, and then apply heuristics to make them more interesting.
However in this paper we try to answer whether it is at all possible to generate long, informative answers that are self-consistent. Interestingness is a separated issue which is not the highest priority in this paper.
However, when we perform the human evaluation study, the human evaluation is affected by their judgement of interestingness.
2. By "considering context in addition to correctness", do you mean to spend effort on modeling the context when generating the response? As I answered in the last comment to AnonReviewer1, the effort to model context is orthogonal to the effort of generating informative and consistent responses. In another word, once we addressed the second issue, we can apply the same technique in the models that does take more context into account. I totally agree with your concern that modeling context is also important.
3. I think it would work fine because in the training set the reddit messages may have more than one questions. In the following I picked out the responses from our model which respond to the prompts that contain multiple questions or statements that contain different ideas (from the same collection of responses we sent for human evaluation).
- Do you have a favorite meal? Or any favorite place to go out to eat?
- I like to go to the grocery store because it's really easy to make and you can pick up a bunch of stuff. I also like to go out drinking and watch movies with friends.
- Nice to meet you, tell me about yourself
- I am a little bit of an introvert, I'm not really into the whole "I am a bit introvert" thing. I have a lot of friends and I really like to hang out and have conversations with people.
- Sometimes flying messes my ears, because of the air pressure. Especially if I have a cold or something, flying gets me messed up.
- I have no idea why, but I feel exactly the same way. I don't know why, but my brain is so sensitive to air that it makes my nose twitch.
- I don't think I'm that affected by the weather. If it rains sometimes I get a little down, but it doesn't usually affect me that much. How about you?
- I'm in the same boat as you. I love the weather, but the weather is so bad I don't even know what I'm going to do about it.
- I live alone, and if I eat at home I eat whatever I can put in the microwave that night. A lot of times it's just that, or I'll trade with friends; I'll cook and they'll cook and then we'll swap leftovers.
- I live alone, and if I eat at home, I eat whatever I can put in the microwave that night. I do this all the time. I eat the same thing over and over again.
- how was your weekend? what'd you do?
- I had a good weekend and I had a good time. I just wanted to get some rest.
- I'd like to read a good science fiction novel. Haven't read scifi in a while.
- If you want to know what sci-fi is, read the first book by Neil Gaiman. It's a sci-fi sci-fi sci-fi novel. It is really good and I recommend it to all of you. It has a great premise.
- I don't do chatting. I look for meaningful conversations.
- I do that too. I don't know what to tell people when I say I haven't done it yet.
- How much time would you say you spend attached to a computer? 2 hours? 10 hours?
- I have a gaming computer that I'd rather not have to spend hours on, so I spend all day playing games on it on a regular basis.
- Actually I like the snow too, except for today. It's all slushy and it's all over the sidewalks and it's gross.
- I'm in the same boat as you. I hate it when it's snowy and I don't like to shovel it all day. :( So this is a perfect excuse to not get out of bed and go somewhere else.
This paper considers the problem of generating long and diverse responses in dialog systems. Two techniques are proposed to the seq-to-seq framework: (1) glimpse model that trains on fixed-length segments of the target side at a time, and (2) a segment-based stochastic decoding technique which injects diversity earlier in the generated responses. The large scale experiments on 2.3B conversation messages are quite impressive.  Experiments on human evaluation should also be encouraged.
With all these said, I am still not 100% convinced that machine generated long sequence is the right direction for dialog systems. As shown in Figure 3 (a), human evaluation shows that the proposed system is not significantly better than the baselines. I think more analysis and user preference mining should be done in the future to help us understand the nature of this problem.
Thank you for the reviews!
I think you are right, we should have expressed our goal as the well-spoken-ness and consistency of long and informative responses, when generated from a sequence decoder network, and we take just this as a research problem without considering the actual end-user experience. The latter is mixed with too many other factors that we didn't touch.
For example, short and accurate answers may be more effective when the conversation is answer-seeking.
Also, a well-spoken response may exhibit negative sentiment in which case the evaluation tend to be negative. We'll try to revise the paper accordingly. These affects the human eval results which causes the overall improve not be not great enough.
The paper is clearly interesting in that it does address important problems (length and diversity of responses) in sequence-to-sequence models. The two ideas put forward (glimpse model and segment-based stochastic decoding) both seem ideas in the right direction. I was however not so sold on the argument that these are particularly suitable for conversations.
The results indicate that the ideas do indeed generate longer and also somewhat more sensible target sequences and as such the paper makes progress w.r.t these important problems.
So overall I would suggest accepting the paper even though the flavor of the proposed ideas are somewhat "small steps".
Thank you for the reviews! We appreciate the thoughts! You are right, our work is more like taking the conversation as a problem setting to be grounded and addressing the sequence generation problem.
The paper proposes modification to seq2seq model to help it handle the problems when long responses are needed.
Though the technical contributions may be of value, the work in my personal opinion is not in the right direction towards helping dialog systems. Essentially we try to generate long responses that sound ``nice" yet are not grounded to any reality, they just need to be related to the question and not suffers from obvious mistakes.
Yet, the architectural innovations proposed may be of merit.
Thank you for the review! As we mentioned below, the problem of "grounding to reality", i.e., modeling context, is orthogonal to the problem of generating long and informative responses. That says, when new advances are made to context modeling, we can apply the same technique we proposed, because our method is mainly on the decoder side. The two problems are both non-trivial and therefore worth separated efforts. Potential approaches towards context modeling spans from extensive feature engineering (e.g., [1] in my comments below) to hierarchical memory modeling, commonsense modeling, etc., and is likely an AI-complete problem.
Therefore we believe that it is not supposed to be addressed as part of a paper which do not focus on context modeling.
Thank you for the review! In light of your concerns, we'd like to offer some more clarification:
We agree that a good dialog system should not merely generate vacuous responses that ``sound'' nice, but are indeed grounded and sensible.
However, in the current dominant end-to-end encoder-decoder framework (i.e. "seq2seq"), one can argue that grounding and KB-integration can be modeled as an encoder-side problem, by making the source-side encoding arbitrarily rich.
Instead, in this work we focus on the decoder and we ask: *Given such a representation of the source/query* (which may involve grounding in the real world or some KB -- or, as in this case, only the source text), can we improve the quality of the generated longer responses? This is an important problem, since even a state-of-the art seq2seq model tends to generate very short and dull outputs, and they quickly lose coherence, even as one artificially encourages longer responses using for example "length-normalization" techniques.
Our experimental results show that the proposed techniques (training on glimpses of the target-side, and decoding segment-by-segment while injecting diversity earlier during the decoding process) are indeed more effective at generating longer responses that are more coherent and less dull than a seq2seq+attention model with length-normalization (the best alternative we know of). We believe this result may also carry over to other application domains where longer outputs may be required. We furthermore believe our results demonstrating that the glimpse-model (a simple data preprocessing technique) allows training to scale to much larger datasets (we train on 2.3 Billion message pairs) is another valuable contribution, and might also be useful in other application domains besides dialog modeling, e.g. machine translation -- since in all end-to-end trained systems more training data is almost always better.

Meta-review:
role: PC
data: comment: The reviewers agree that the work presents interesting, but incremental, results. They are also unconvinced that this is the right direction for research on dialogue systems to go in or that the methods presented will appeal to a broader audience. It seems the authors have some work to do on the larger argument, before this work can be accepted.

Decision: Reject