Introduction
Dialogue systems have become a very popular research topic in recent years with the rapid improvement of personal assistants and the growing demand for online customer support. However, research has been split in two subfields BIBREF2: models presented for generation of open-ended conversations BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7 and work on solving goal-oriented dialogue through dialogue management pipelines that include dialogue state tracking and dialogue policy BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14.
Dialogue state tracking has often been limited to detection of user intention, as well as learning a dialogue policy to determine what actions the system should take based on the detected user intent. Dialogue generation for open ended conversation, in contrast, has largely relied on transduction architectures originally developed for machine translation (MT) BIBREF15, BIBREF16, BIBREF17. Such architectures offer flexibility because of their ability to encode an utterance into a fixed-sized vector representation, and decoding it into a variable length sequence that is linguistically very different from the input utterance. However, MT-based approaches often lack the ability to encode the context in which the current utterance occurs. This can lead to repetitive and meaningless responses BIBREF18, BIBREF19, BIBREF17.
This observation has led researchers to extend simple encoder-decoder models to include context in order to deal with generation of larger structured texts such as paragraphs and documents BIBREF18, BIBREF20, BIBREF21. Many of these models work by encoding information at multiple levels, i.e., using both a context encoder and a last-utterance encoder, passing both encodings to a decoder that predicts the next turn. Such hierarchical methods have proven to be useful for open-ended chit chat, but were not designed for goal-oriented dialogue, where responses need not only be coherent, but also relevant.
In goal-directed dialogue generation, there is often one (context-dependent) right answer to a question (e.g., How many types of insurance do you offer?); in chit-chat, there are many good answers to questions (e.g., What do you want to talk about today?). We therefore hypothesize that in personal assistants and customer support, it is beneficial to increase the inductive bias of the dialogue generation model and its dependency on past conversations in order to keep responses relevant. We do so by designing a novel, hybrid dialogue generation model that conditions decoding on retrieved examplars from past history.
Although retrieval approaches to dialogue generation have been introduced before, they have typically been used to add more variety to the kind of answers the model can generate in open ended conversations BIBREF7, BIBREF22. Our model, in contrast, is designed for the purpose of goal oriented dialogue in the customer support domain. It is a hierarchical neural model with an information retrieval component that retrieves the most informative prior turns and conditions on those. This simple approach increases inductive bias and alleviates problems arising from long-context dependencies.
We show that an information retrieval step leads to improvements over traditional dialogue generation models intended for open ended chit chat when evaluated on BLEU and different embedding metrics. In addition, we evaluate our generated responses based on the Request/Inform success rate typically used in dialogue state tracking, and again, we show performance close to the more complex state-of-the-art model which contrary to our proposed model, makes use of annotated labels. Finally, based on our human evaluations, we show that a simple retrieval step leads to system responses that are more fluent and appropriate than the responses generated by the hierarchical encoder-decoder model.
Model Description
We want to adapt commonly used models for dialogue generation to the task of goal-oriented dialogue by providing a simple way of integrating past examples in a context-aware dialogue generation model. We extend the Hierarchical Recurrent Encoder-Decoder (HRED) model presented by BIBREF23 for query suggestion, subsequently adopted for dialogue by BIBREF20, which has shown to be a strong baseline for the task of dialogue generation. In line with previous research, we consider a dialogue $D$ between two speakers composed of $n$ utterances so that $D = [U_1, ... , U_n]$ and each utterance $U_i$ composed of $k_i$ tokens so that $U_n = [t_{i,1}, t_{i, 2}, ... , t_{i,k_i}]$. Each token $t_{i, k_i}$ represents a word from a set vocabulary. Preliminary experiments showed that limiting contexts to three utterances gave the best results, therefore all results presented use $n=3$. This is also in line with most previous work on dialogue generation and classification BIBREF20.
Model Description ::: HRED
HRED BIBREF23 consists of an encoder RNN, which maps an utterance to a vector, a context RNN, which summarizes the dialogue history by keeping track of the previous hidden states, and a decoder RNN, which decodes the hidden state of the context RNN. Given a dialogue consisting of three utterances – a system response, a user response, and a second system response, $\langle s_1,u,s_2\rangle $ – the goal is to predict the system utterance $s_2$ given the context $\langle s_1,u\rangle $. The utterance encoder takes in a single utterance and outputs a vector representation . The representations of each utterance are concatenated so that we end up with an array that is then fed to the context encoder. The context encoder outputs a global context, which is then fed into the decoder. Just as in previous work BIBREF21, BIBREF20, BIBREF3, BIBREF24, we use GRUs BIBREF25 for our encoder, context and decoder RNNs. All modules share parameters.
Model Description ::: Exemplar-HRED
In this study, we want to enhance HRED with a simple yet efficient information retrieval step. As already mentioned, similar approaches have been presented with the goal of incorporating factual information into open-ended conversations BIBREF22, to add variety and more topics to the conversation.
We hypothesize that using exemplar information is also beneficial for multi-domain goal-oriented systems. More specifically, we want to be able to inform our generation model about previous responses to similar utterances, biasing it towards past responses. For each user utterance, we extract the ten most similar past user utterances from the training set using approximate nearest neighbor search BIBREF26. We approximate a point $p \in S$ by specifying some error margin $\epsilon > 0 $ so that $ dist(p,q) \le (1+\epsilon )(dist(p*, q)) $, where $p*$ is the real nearest neighbor. Because we use approximate search, we rerank the retrieved utterances using a feed-forward ranking model, introduced in BIBREF27. Their ranking model is a multi-task model, which relies on simple textual similarity measures combined in a multi-layered perceptron architecture. The model nevertheless achieves state-of-the-art performance on question relevancy ranking. In the end, we take the top user utterance, and return its response as the example to be used in our model.
For goal-oriented dialogue generation, our proposed model uses the same architecture as the HRED baseline, however, we include an additional RNN, which encodes the top example response. We feed the representation of the example RNN context into the context RNN and feed this representation into the decoder. Just as in the baseline model, the utterance encoder outputs a vector representation. Additionally, we encode the exemplar into a vector using the example encoder. The representations of each utterance are concatenated so that we end up with an array that includes dialogue context and exemplar information, all of which is then fed to the context encoder. The global context is then fed into the decoder.
For all experiments, we use the MultiWoz dataset for goal oriented dialogue BIBREF0, which we describe in more detail in the next section. Our model uses the Adam optimizer BIBREF28 for all encoders. All our encoders are one layer RNNs. In addition, we use a dropout rate of 0.3, and a learning rate of 0.001. We set a maximum of 50 epochs, however, we use early stopping with a patience of 10. Most of our models converge by epoch 30. We use greedy search to generate the response during testing. More implementation details as well as our predicted utterances for each system can be found in the link provided
Experiments ::: Dataset
We use the MultiWoz dialogue corpus BIBREF0, which consists of 10,438 dialogues spanning several domains and annotated with dialogue states and acts. We train on 8,438 dialogues, and use 1000 dialogues for development and 1000 dialogues for testing. Although the data is primarily intended for dialogue state tracking and learning a dialogue policy, BIBREF0 also mention its potential as a benchmark for end-to-end dialogue due to the fact that it contains about 115k turns in total, which is larger than many structured dialogue corpora available. This makes it a good choice for hybrid approaches in generation and goal oriented dialogue. The MultiWOZ dataset is also a much more difficult dataset than the current benchmarks for goal oriented dialogue, as it spans about 7 different customer support domains and conversations are not limited to a single domain. In line with previous work in goal oriented dialog and recent work using this dataset, we delexicalize the utterances to remove phone numbers, reference numbers and train ids. As opposed to other studies, we only delexicalize these three slots since these were significantly increasing the size of the vocabulary. For delexicalizing, we use the ontology provided with the data and replace the value with the slot names using regular expressions. We do not delexicalize times, prices, postcodes and distinct names of restaurants and hotels. This also makes our generation task more difficult.
Experiments ::: Baselines
In this study, we are interested in simple ways of providing our dialogue generation model with enough inductive bias in order to generate fluent and on-topic responses for goal oriented dialogue. Encoder-decoder architectures work well when it comes to providing generic fluent responses for open ended conversations, however, in goal-oriented dialogue, it is also necessary for the system to remain on topic. Additionally, when training a single model on different domains, this becomes more difficult. The original HRED model BIBREF23, BIBREF21 adapted for dialogue performs well when it comes to open ended conversations as well as when trained on very large corpora (millions of utterances) BIBREF29, BIBREF30. In our setup however, we train the HRED model using a smaller dataset containing goal oriented dialogues in 7 different domains. We compare this model with our proposed exemplar-based model. In addition, for the BLEU metric we include the results of a transformer modelBIBREF31 that uses dialogue context to condition the decoder as well as a LSTM that uses dialogue context and incorporates belief state and KB results as additional inputs BIBREF0.
Results
Overall, we found that in most cases, our simple model leads to significant improvements over the standard metrics BIBREF32; see Table 1 for the results. Although we are tackling goal-oriented dialogue, traditional metrics for goal oriented dialogue rely on human-generated supervision i.e. slot-value pair labels or dialogue act labels. Word overlap metrics such as the ones used for machine translation are often used to evaluate the quality of dialogue generation, however, these standard metrics tend to have very weak correlation with human judgment. In any case, we include some of these, as well as word embedding metrics for comparison. For the standard metrics, we use the evaluation scripts from BIBREF20 . We observe that the retrieval model is consistently better across all scenarios and metrics. In addition to these metrics, we assess our performance using the dialogue success metrics typically used in belief tracking BIBREF0. We briefly explain these metrics further.
Results ::: BLEU
BLEU BIBREF33 is typically used for machine translation and has subsequently been used to evaluate the performance of many dialogue generation systems BIBREF34, BIBREF21, BIBREF20. BLEU analyzes co-occurrences of n-grams in a reference sequence and a hypothesis. For all datasets, we see improvements with BLEU. It uses a modified precision to account for the differences in length between reference and generated output. Given a reference sentence $s$ and a hypothesis sentence $\hat{s}$, we can denote the n-gram precision $P_{n}(s, \hat{s})$ as:
where q is the index of all possible n-grams, and h(q,s) is the number of n-grams in s.
Results ::: Average Word Embedding Similarity
We follow BIBREF32 and obtain the average embedding $e_s$ for the reference sentence $s$ by averaging the word embeddings $e_w$ for each token $w$ in $s$. We do the same for the predicted output $\hat{s}$ and obtain the final similarity score by computing cosine similarity of the two resulting vectors. Again, Exemplar-HRED is consistently superior yielding almost a 2 percent improvement over the best baseline model.
Results ::: Vector Extrema
We also compute the cosine similarity between the vector extrema of the reference and the hypothesis, again following BIBREF32. The goal of this metric as described by the authors is to consider informative words rather than common words, since the vectors for common words will tend to be pulled towards the zero vector. Our exemplar model achieves the largest improvement for this metric, with a gain of 6 percent over the baseline model.
Results ::: Greedy Matching
In greedy matching BIBREF32, given two sequences $s$ and $\hat{s}$, each token $w \in s$ is matched with each token $\hat{w} \in \hat{s}$ by computing the cosine similarity of the corresponding word embeddings $emb_w$ and $emb_{\hat{w}}$. The local match $g(s, \hat{s})$ is the word embedding with the maximum cosine similarity. We compute in both directions and the total score is:
This metric is used to favour key words. Our best model shows only small improvements on this metric.
Results ::: Human Evaluation
In addition to the previously mentioned standard metrics, we also evaluate the performance of our baseline and the exemplar-based models using human evaluations. We extract 100 baseline and exemplar model system responses at random. We ask the 7 evaluators to 1) pick the response that is more fluent and grammatically correct and 2) pick the response that achieves the goal given the context of the conversation. We provide the context of System and User utterances, and ask the evaluators to pick one of 4 options: 1) the output of the baseline, 2) the output of the exemplar model, 3) both, 4) none. The order of the options was shuffled.
Overall, we found that when it came to fluency, the evaluators perceived that 58% of the time, the exemplar response was better. The baseline beat the exemplar based response for 19 percent of the evaluated dialogs and the rest of the dialogs either both or none were picked. For appropriateness we see a similar pattern. Evaluators perceived the response produced by the exemplar model as the more appropriate one given the context, for 59 percent of the evaluated dialogs. The baseline beat the proposed model only 14 percent of the time. These results can also be found on table TABREF7
Results ::: Dialogue success: inform/request
Traditional goal-oriented dialogue systems based on prediction of slots and dialogue acts are typically evaluated on the accuracy of predicting these as labels, as well as their success at the end of the dialogue. Dialogue success is measured by how many correct inform/request slots a model can generate in a conversation in comparison to the ground truth. An inform slot is one that provides the user with a specific item for example the inform slots "food" and "area" i.e. (food=“Chinese”, area="center”) informs the user that there is a Chinese restaurant in the center. On the other hand, a request slot is a slot that specifies what information is needed for the system to achieve the user goal. For example, for booking a train, the system needs to know the departure location and the destination. The slots "departure" and "destination" would be the request slots in this case.
For goal-oriented generation, many of the models evaluated using the Inform/Request metrics have made use of structured data to semantically condition the generation model in order to generate better responses BIBREF35, BIBREF1, BIBREF0. BIBREF35 proposed to encode each individual dialog act as a unique vector and use it as an extra input feature, in order to influence the generated response. This method has been shown to work well when tested on single domains where the label space is limited to a few dialog acts. However, as the label space grows, using a one-hot encoding representation of the dialog act is not scalable. To deal with this problem, BIBREF1 introduced a semantically conditioned generation model using Hierarchical Disentangled Self-Attention (HDSA) . This model deals with the large label space by representing dialog acts using a multi-layer hierarchical graph that merges cross-branch nodes. For example, the distinct trees for hotel-recommend-area and attraction-recommend-area can be merged at the second and third levels sharing semantic information about actions and slots but maintaining specific information about the domains separate. This information can then be used as part of an attention mechanism when generating a response. This model achieves the state-of-the-art result for generation in both BLEU and Inform/Request metrics.
As we are concerned with improving dialogue generation for goal oriented dialogue, we are interested in assessing how our simple approach compares to models explicitly using dialog acts as extra information. We compute the inform/request accuracy and compare to the state-of-the-art BIBREF1 as well as other baseline models. BIBREF1 present experiments conditioning both on predicted acts as well as ground truth past acts. We include both of these as well as the performance of our baseline and proposed model in table TABREF15. We divide the results into models using act information to condition the language generation and models that do not.
Discussion
As shown in table TABREF7, our simplest proposed model achieved the largest improvements over the baseline when it came to the average embedding similarity and vector extrema similarity. As it is hard to interpret what the difference in performance of each model is based on standard dialogue metrics we examined the output to spot the major differences in response generation of our proposed models versus the baseline. We looked at the responses generated by our proposed models that had the highest score of these metrics and compared to the response generated by the baseline for that same dialogue. Overall we found that the baseline models tend to generate responses containing slots and values for the wrong domain. In addition, by examining the outputs we could see that the vector extrema metric is very sensitive when it comes to slight differences in the references and prediction. We found that this metric was more indicative of model performance than embedding similarity. We present some example outputs in table TABREF16.
As mentioned earlier, from manual inspection of the outputs we observed that the the exemplar model is able to stay within the correct domain of the conversation and returns information within that domain that is more appropriate given the conversation context. This was confirmed by the human evaluations and also the Inform/Request metrics. When comparing the performance of the exemplar-based model to models that do not use information about past acts to condition the decoder, we observe that including a simple retrieval step leads to very large gains in the success of providing the inform/request slots.. In addition, the exemplar model performs better than BIBREF0, which uses information of the belief state of the conversation as extra features. More interestingly, our proposed model performs better than the state-of-the-art when it comes to providing the request slots. It also outperforms this same model when evaluated on BLEU; however, it still falls behind the state-of-the-art when it comes to providing inform slots. Overall, we find that our model remains competitive without requiring turn labels.
Related Work
Dialogue generation has relied on transduction architectures originally developed for machine translation (MT) BIBREF15, BIBREF16, BIBREF17. Open domain dialogue systems aim to generate fluent and meaningful responses, however this has proven a challenging task. Most systems are able to generate coherent responses that are somewhat meaningless and at best entertaining BIBREF19, BIBREF17, BIBREF20. Much of the research on dialogue generation has tried to tackle this problem by predicting an utterance based on some dialogue history BIBREF36, BIBREF37, BIBREF38, BIBREF20. We extend such an architecture to also include past history, in order to avoid generating too generic responses.
Most research on goal-oriented dialogue has focused almost exclusively on dialogue state tracking and dialogue policy learning BIBREF12, BIBREF39, BIBREF40, BIBREF41, BIBREF42, BIBREF43, BIBREF9, BIBREF10. Dialogue state tracking consists of detecting the user intent and tends to rely on turn-level supervision and a preset number of possible slot and value pairs which limits the flexibility of such chatbots, including their ability to respond to informal chit chat, as well as transferring knowledge across domains. There has been some work in the past few years that has attempted to address these problems by introducing methods that focus on domain adaptation as well as introducing new data to make this task more accessible BIBREF43, BIBREF44, BIBREF0, BIBREF45. Recent approaches have also introduced methods for representing slot and value pairs that do not rely on a preset ontology BIBREF8, BIBREF46, in an attempt to add flexibility. In our research, we acknowledge the importance of this added flexibility in goal oriented dialogue and propose a method for generating goal oriented responses without having turn level supervision.
The idea of combining text generation with past experience has been explored before. BIBREF47 used a set of hand crafted examples in order to generate responses through templates. More recently, BIBREF48 also explored a hybrid system with an information retrieval component, but their system is very different: It uses a complex ranking system at a high computational cost, requires a post-reranking component to exploit previous dialogue turns (about half of which are copied over as predictions), and they only evaluate their system in a chit-chat set-up, reporting only BLEU scores. In a similar paper, BIBREF22 tried to move away from short generic answers in order to make a chit-chat generation model more entertaining by using an information retrieval component, to introduce relevant facts. In addition, a similar method was recently shown to improve other generation tasks such as summarization. In BIBREF49, the authors show that a simple extractive step introduces enough inductive bias for an abstractive summarization system to provide fluent yet precise summaries. In contrast to these works, we integrate a retrieval based method with a context-aware neural dialogue generation model in order to introduce relevant responses in a goal oriented conversation.
Conclusion
In this study, we have experimented with a simple yet effective way of conditioning the decoder in a dialogue generation model intended for goal oriented dialogue. Generating fluent and precise responses is crucial for creating goal-oriented dialogue systems, however, this can be a very difficult task; particularly, when the system responses are dependent on domain-specific information. We propose adding a simple retrieval step, where we obtain the past conversations that are most relevant to the current one and condition our decoder on these. We find that this method not only improves over multiple strong baseline models on word overlap metrics, it also performs better than the state-of-the-art on BLEU and achieves competitive performance for inform/request metrics without requiring dialog act annotations. Finally, by inspecting the output of the baseline versus our proposed model and through human evaluations, we find that a great advantage of our model is its ability to produce responses that are more fluent and remain on topic.

Question: what semantically conditioned models did they compare with?
Answer: Hierarchical Disentangled Self-Attention