Abstract: Many sequence-to-sequence dialogue models tend to generate safe, uninformative responses. There have been various useful efforts on trying to eliminate them. However, these approaches either improve decoding algorithms during inference, rely on hand-crafted features, or employ complex models. In our work, we build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. Specifically, we start with a simple yet effective automatic metric, AvgOut, which calculates the average output probability distribution of all time steps on the decoder side during training. This metric directly estimates which tokens are more likely to be generated, thus making it a faithful evaluation of the model diversity (i.e., for diverse models, the token probabilities should be more evenly distributed rather than peaked at a few dull tokens). We then leverage this novel metric to propose three models that promote diversity without losing relevance. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch; the second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level; the third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal. Moreover, we experiment with a hybrid model by combining the loss terms of MinAvgOut and RL. All four models outperform their base LSTM-RNN model on both diversity and relevance by a large margin, and are comparable to or better than competitive baselines (also verified via human evaluation). Moreover, our approaches are orthogonal to the base model, making them applicable as an add-on to other emerging better dialogue models in the future.

Introduction: Many modern dialogue generation models use a sequence-to-sequence architecture as their backbone BIBREF0, following its success when applied to Machine Translation (MT) BIBREF1. However, dialogue tasks also have a requirement different from that of MT: the response not only has to be "correct" (coherent and relevant), but also needs to be diverse and informative. However, seq2seq has been reported by many previous works to have low corpus-level diversity BIBREF2, BIBREF3, BIBREF0, BIBREF4, as it tends to generate safe, terse, and uninformative responses, such as "I don't know.". These responses unnecessarily make a dialogue system much less interactive than it should be. To increase the diversity of dialogue responses, the first step is to faithfully evaluate how diverse a response is. There are metrics used by previous work that are correlated to diversity, but not strongly, such as ratio of distinct tokens BIBREF2 and response length BIBREF5. However, a response can be long but extremely boring in meaning, such as "I am sure that I don't know about it.", or short but interesting (i.e., contains a lot of information), such as "Dad was mean.". Only investigating discrete token output by the model is also not ideal, because these tokens are only a single realization of the model's output probability distribution at each time step, which unavoidably loses valuable information indicated by the whole distribution. BIBREF6 (BIBREF6) manually collect a shortlist of dull responses, and during training discourage the model from producing such utterances. However, an important drawback of hand-crafted rules is that the set of dull tokens or utterances is static, while in fact it usually evolves during training: when the current dull tokens are eliminated, another set of them might reveal themselves. In our work, we begin with a simple yet effective approach to measure how diverse a response is. This metric, which we name "Average Output Probability Distribution", or AvgOut, draws information directly from the training-in-session model itself. We calculate it by keeping track of the exponential average of all output probability distributions on the decoder side during training. This metric dynamically measures which tokens the model is biased toward without any hand-crafted rules, thus making it a faithful evaluation of the model diversity (i.e., for diverse models, the token probabilities should be more evenly distributed rather than peaked at a few dull tokens). In addition, since AvgOut is a one-dimensional categorical distribution rather than a dimensionless numerical value like entropy, it naturally carries and conveys more information about model diversity. We then propose three models that leverage our novel metric to promote diversity in dialogue generation. The first MinAvgOut model minimizes the dot product of current batch AvgOut and the exponential average AvgOut across batches, which encourages low-frequency tokens to be generated. The second LFT model uses a labeled transduction method and scales a "diversity label" by the diversity score of the ground-truth target sequence during training, while during testing can generate responses of different levels of diversity by tweaking the intended diversity score. The third RL model leverages reinforcement learning, where our novel metric is applied to discrete tokens and serve as a reward signal. In addition, since MinAvgOut regularizes directly on the continuous distribution while RL calculates its reward based on discrete sampled tokens, we simply add up the loss terms of the two models, creating an even stronger hybrid model. We first employ diverse automatic metrics, including Distinct-1 and -2 from previous work BIBREF2 and our novel metric Diveristy-iAUC (which calculates one minus the sum of normalized frequencies of the most frequent tokens produced by the model), plus activity/entity F1s, to evaluate the diversity and relevance of the generated responses. We then conduct human evaluations to verify that these models not only outperform their base model LSTM by a large margin, but are also comparable to or better than an advanced decoding algorithm MMI BIBREF2 and a very competitive model VHRED BIBREF7 on the Ubuntu dataset.

Question: How much better were results of the proposed models than base LSTM-RNN model?
Answer: on diversity 6.87 and on relevance 4.6 points higher