Venue: NAACL 2019

# Multi-Level Memory for Task Oriented Dialogs

Revanth Reddy\({}^{1}\), Danish Contractor\({}^{2}\), Dinesh Raghu\({}^{2}\), Sachindra Joshi\({}^{2}\)

\({}^{1}\)Indian Institute of Technology, Madras \({}^{2}\)IBM Research AI, New Delhi

g.revanthreddy111@gmail.com

{dcontrac,diraghui,jsachind}@in.ibm.com

Work done during internship at IBM Research AI

###### Abstract

Recent end-to-end task oriented dialog systems use memory architectures to incorporate external knowledge in their dialogs. Current work makes simplifying assumptions about the structure of the knowledge base (such as the use of triples to represent knowledge) and combines dialog utterances (context), as well as, knowledge base (KB) results, as part of the same memory. This causes an explosion in the memory size, and makes reasoning over memory, harder. In addition, such a memory design forces hierarchical properties of the data to be fit into a triple structure of memory. This requires the memory reader to learn how to infer relationships across otherwise connected attributes.

In this paper we relax the strong assumptions made by existing architectures and use separate memories for modeling dialog context and KB results. Instead of using triples to store KB results, we introduce a novel multi-level memory architecture consisting of cells for each query and their corresponding results. The multi-level memory first addresses queries, followed by results and finally each key-value pair within a result. We conduct detailed experiments on three publicly available task oriented dialog data sets and we find that our method conclusively outperforms current state-of-the-art models. We report a 15-25% increase in both entity F1 and BLEU scores.

## 1 Introduction

Task oriented dialog systems are designed to complete a user specified goal, or service an information request using natural language exchanges. Unlike open domain end-to-end neural dialog models, task oriented systems rely on external knowledge sources, outside of the current conversation context, to return a response Henderson et al. (2014); Su et al. (2016); Bordes and Weston (2017); Eric and Manning (2017); El Asri et al. (2017). For instance, in the example shown in Table 1, a dialog agent giving tour package recommendations needs to be able to first query an external knowledge source to determine packages that meet a user's requirement, and then respond accordingly.

In order to enable end-to-end goal oriented dialog tasks, current state of the art methods use neural memory architectures to incorporate external knowledge Su et al. (2016); Eric and Manning (2017); Madduto et al. (2018). As can be seen in Table 1, agent responses may also include entity values present only in the dialog context (eg: "Munch" in the Agent response in Turn 2). In order to support such utterances, models also include tokens from the input dialog context in the same memory Madduto et al. (2018).

\begin{table}
\begin{tabular}{l l c c c c}
**Origin** & **Dest.** & **Hedel** & **Price** & **Cat.** & **Raling** \\ \hline Dallas & Mannebula & Segal House & Segal House & Segal House & Segal House \\ Toronto & Calapay & Annecourt & S1664.4 & 0.0 & 8.91 \\ Dallas & Mannequin & Varies Inc & S31592.8 & 3.0 & 7.15 \\ Dallas & Samica & Grooteauwer & S2008.0 & 8.37 & – \\ Dallas & Mannebula & Staplight & S4018.1 & 4.0 & 8.56 \\ \hline \hline \multicolumn{1}{c}{} & & & & \\ \hline \hline \multirow{2}{*}{**Rate**} & **Turn** & **Utremae** & & \\ \cline{2-6}  & \multicolumn{4}{c}{} & & \multicolumn{1}{c}{} \\ \hline \hline Agost & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline View & \multicolumn{4}{c}{} & \multicolumn{1}{c}{} \\ \hline \end{tabular}
\end{table}
Table 1: A goal oriented dialog based on the Frames dataset El Asri et al. (2017) along with an external knowledge source with each row containing a tour package.



Existing memory based architectures for task oriented dialog suffer from multiple limitations. First, the creation of a shared memory for copying values from dialog context, as well as the knowledge base (KB) results, forces the use of a common memory reader for two different types of data. This makes the task of reasoning over memory, harder - not only does the memory reader need to determine the right entries from a large memory (since each word from context also occupies a memory cell), it also needs to learn to distinguish between the two forms of data (context words and KB results) stored in the same memory.

Second, all current neural memory architectures store results, returned by a knowledge source, in the form of triples (eg. \(subject-relation-object\)). This modeling choice makes it hard for the memory reader to infer relationships across otherwise connected attributes. For instance, consider the example triple store in Table 2 showing results for a query executed for packages between "Dallas" and "Mannheim". If the user asks the dialog agent to check the price of stay at a 5 star hotel, the memory reader needs to infer that the correct answer is $2800 by learning that the price, category and hotel need to be linked inorder to return an answer (shown in blue).

Lastly, current models treat conversations as a sequential process, involving the use of KB results from only the most recent information request/query. In contrast, in real world dialogs such as the one shown in Table 1, the agent may have to refer to results (_to Mannheim_) from a previously executed query (see Turn 7). Thus, at each turn, the system has to memorize all the information exchanged during the dialog, and infer the package being referred to, by the user. In order to support such dialogs, the memory needs to store results of all queries executed during the course of the dialog. The problem of inferring over such results (which may be from multiple queries) is exacerbated when memory is represented in the form of triples.

In this paper, we present our novel multi-level memory architecture that overcomes the limitations of existing methods: (i) We separate the memory used to store tokens from the input context and the results from the knowledge base. Thus, we learn different memory readers for context words as well for knowledge base entities (ii) Instead of using a \(subj-rel-obj\) store, we develop a novel multi-level memory architecture which encodes the natural hierarchy exhibited in knowledge base results by storing queries and their corresponding results and values at each level. We first attend on the queries, followed by the results in each query to identify the result being referred to, by the user. We then attend on the individual entries in the result to determine which value to copy in the response. Figure 0(c) shows our multi-level memory storing the results from queries executed as part of the dialog in Table 1.

Our paper makes the following contributions:

**1.** We propose the use of separate memories for copying values from context and KB results. Thus, the model learns separate memory readers for each type of data.

**2.** Our novel multi-level memory for KB results, models the queries, results and their values in their natural hierarchy. As our experiments show, the separation of memory as well as our multi-level memory architecture, both, contribute to significant performance improvements.

**3.** We present detailed experiments demonstrating the benefit of our memory architecture along with model ablation studies. Our experiments on three publicly available datasets (**CamRest676**Su et al., 2016), **InCar assistant**Eric and Manning (2017), **Maluuba Frames**El Asri et al. (2017)) show a substantial improvement of 15-25 % in both entity F1 scores, and BLEU scores as compared to existing state of the art architectures. To the best of our knowledge, we are the first to attempt end-to-end modeling of task oriented dialogs with non-sequential references as well as multiple queries, as seen in the Maluuba Frames dataset. A human evaluation on model outputs also shows our model is preferred by users over existing systems such as KVRet Eric and Manning (2017) and Mem2Seq Madotto et al. (2018).

## 2 Related work

Recent methods, such as Vinyals and Le (2015); Serban et al. (2016, 2017), proposed for end-to-end learning of dialogs were aimed at modeling

\begin{table}
\begin{tabular}{l l c|l||l l}
**Subject** & **Relation** & **Object** & **Subject** & **Relation** & **Object** \\ \hline Verse Item & Price & $3952.0 & Verste Item & Category & 3.0 \\ Regal Resort & Price & $2800.0 & Regal Resort & Rating & 5.98 \\ Regal Resort & Category & $.0 & Starlight & Price & $4018.1 \\ Statlight & Raising & $.56 & Starlight & Category & 4.0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results from Dallas to Mannheim stored in the form of triples.

 

open-domain dialogs. While they can be used for learning task oriented dialogs, they are not well suited to interface with a structured KB. To better adapt them to handle task oriented dialogs: 1) Bordes and Weston (2017) proposed a memory network based architecture to better encode KB tuples and perform inferencing over them and 2) Madotto et al. (2018) incorporated copy mechanism to enable copying of words from the past utterances and words from KB while generating responses. All successful end-to-end task oriented dialog networks Eric and Manning (2017); Bordes and Weston (2017); Madotto et al. (2018) make assumptions while designing the architecture: 1) KB results are assumed to be a triple store, 2) KB triples and past utterances are forced to be represented in a shared memory to enable copying over them. Both these assumptions make the task of inferencing much harder. Any two fields linked directly in the KB tuple are now linked indirectly by the subject of the triples. Further, placing the KB results and the past utterances in same memory forces the architecture to encode them using a single strategy. In contrast, our work uses two different memories for past utterances and KB results. The decoder is equipped with the ability to copy from both memories, while generating the response. The KB results are represented using a multi-level memory which better reflects the natural hierarchy encoded by sets of queries and their corresponding result sets.

Memory architectures have also been found to be helpful in other tasks such as question answering. Work such as Xu et al. (2016) defines a hierarchical memory architecture consisting of sentence level memory followed by word memory for a QA task while Chandar et al. (2016) defines a memory structure that speeds up loading and inferencing over large knowledge bases. Recent work by Chen et al. (2018) uses a variational memory block along with a hierarchical encoder to improve diversity of open domain dialog responses.

## 3 Multi-Level Memory Network

In this section, we describe our end-to-end model for task oriented dialogues. Our model1 (Figure 1a) consists of: (i) a hierarchical encoder which encodes the current input context consisting of the user and agent utterances (ii) a multi-level memory that maintains the queries and knowledge base results seen so far in the course of the dialogue, and (iii) copy augmented sequence decoder that uses separate context and multi-level memory. The queries and their corresponding results are maintained in a multi-level memory. The decoder uses a gating mechanism for memory selection while generating a response.

Footnote 1: Code is available at Multi-Level Memory

### Encoder

Our model uses a standard hierarchical encoder as proposed by Sordoni et al. (2015). The encoder takes a sequence of utterances as input. For the \(t^{th}\) turn, the dialogue context can be represented as \((c_{1},c_{2},...c_{2t-1})\), which consists of \(t\) user utterances and \(t-1\) system utterances. Each utterance \(c_{i}\) is further a sequence of words \((w_{i1},w_{i2},...w_{im})\). We first embed each word \(w_{ij}\) using a word embedding function \(\phi^{emb}\) that maps each word to a fixed-dimensional vector. We then generate utterance representations, \(\varphi(c_{i})\) using a single layer bi-directional GRU. \(h^{e}_{ij}\) denotes the hidden state of word \(w_{ij}\) in the bi-directional GRU. The input representation \(c\) is generated by passing each utterance representation \(\varphi(c_{i})\) through another single layer GRU.

### Multi-level Memory

**Motivation:** Current approaches break down KB results by flattening them into (_subj-rel-obj_) triples. However, converting KB results into triples leads to loss of relationship amongst attributes in the result set. This makes the reasoning over memory difficult as model now has to infer relationships when retrieving values from memory. Instead, we use a multi-level memory which keeps the natural hierarchy of results intact (without breaking them into triples). We store the queries and their corresponding results and individual values at different _levels_. We first attend on the queries and then on the results for each query to identify which result the user is referring to. This also enables us to handle user requests that refer to results from a previously executed query. We propose that a representation of all the values in the result, and not just one of the values (designated as _subj_), should be used while attending over a result in KB. We attend on this compound representation of the result before attending on the individual key-value pairs in each result, to determine which value to copy into the generated response.

 



#### 3.2.1 Memory Representation

Let \(q_{1},q_{2},...q_{k}\) be the queries fired to the knowledge base till the current state of the dialogue. Every query \(q_{i}\) is a set of key-value pairs \(\{k^{qi}_{a}:v^{qi}_{a},1<a<n_{q_{i}}\}\), corresponding to the query's slot and argument where \(n_{q_{i}}\) is the number of slots in query \(q_{i}\). For example, after the user utterance at Turn 3 in Table 1, the query fired by the system on the knowledge base would be {_'origin':'Dallas',destination':'Manheim',Start':__Aug 26', 'end': 'Aug 31', 'Adults':1_}. The execution of a query on an external knowledge base, returns a set of results. Let \(r_{ij}\) be the \(j^{th}\) result of query \(q_{i}\). Each result \(r_{ij}\) is also a set of slot-value pairs \(\{k^{r_{ij}}_{a}:v^{r_{ij}}_{a},1<a<n_{r_{ij}}\}\) where \(n_{r_{ij}}\) is the number of attributes in result \(r_{ij}\). A visualization of the memory with queries and their corresponding results can be seen in Figure 0(c).

The first level of memory contains the query representations. Each query \(q_{i}\) is represented by \(q_{i}^{v}\) = Bag of words over the word embeddings of values (\(v^{q_{i}}_{a}\)) in \(q_{i}\). The second level of memory contains the result representations. Representation of each result \(r_{ij}\) is given by \(r_{ij}^{v}\) = Bag of words over the word embeddings of values (\(v^{r_{ij}}_{a}\)) in \(r_{ij}\). The third level of memory contains the result cells which have the key-value pairs (\(k^{r_{ij}}_{a}:v^{r_{ij}}_{a}\)) of the results. The values (\(v^{r_{ij}}_{a}\)) which are to be copied into the system response are thus present in the final level of memory . We now describe how we apply attention over the context and multi-level memory.

### Decoder

The model generates the agent response word-by-word; a word at time step \(t\) is either generated from the decode vocabulary or is a value copied from one of the two memories (knowledge base or context memory). A soft gate \(g_{1}\) controls whether a value is generated from vocabulary or copied from memory. Another gate \(g_{2}\) determines which of the two memories is used to copy values.

#### 3.3.1 Generating words:

Let the hidden state of the decoder at time \(t\) be \(h_{t}\).

\[h_{t}=GRU(\phi^{emb}(y_{t-1}),h_{t-1}) \tag{1}\]

The hidden state \(h_{t}\) is used to apply attention over the input context memory. Attention is applied over the hidden states of the input bi-directional (BiDi) GRU encoder using the "_concat_" scheme as given in Luong et al. (2015). The attention for the \(j\)th word in the \(i\)th utterance is given by:

\[a_{ij}=\frac{exp(w_{1}^{T}tanh(W_{2}tanh(W_{3}[h_{t},h^{e}_{ij}])))}{\sum_{ ij}exp(w_{1}^{T}tanh(W_{2}tanh(W_{3}[h_{t},h^{e}_{ij}])))} \tag{2}\]

Figure 1: Model architecture (a) along with schematic representation of context memory (b) and multi-level KB memory (c)

The attention scores \(a_{ij}\) are combined to create an attended context representation \(d_{t}\),

\[d_{t}=\sum_{i,j}a_{i,j}h^{e}_{ij} \tag{3}\]

and similar to Luong et al. (2015), the decoder word-generation distribution is given by :

\[P_{g}(y_{t})=softmax(W_{1}[h_{t},d_{t}]+b_{1}) \tag{4}\]

#### 3.3.2 Copying words from context memory:

The input context memory is represented using the hidden states \(h^{e}_{ij}\) of the input Bi-Di GRU encoder. Similar to Gulcehre et al. (2016), the attention scores \(a_{ij}\), are used as the probability scores to form the copy distribution \(P_{con}(y_{t})\) over the input context memory.

\[P_{con}(y_{t}=w)=\sum_{ij:w_{ij}=w}a_{ij} \tag{5}\]

#### 3.3.3 Copying entries from KB memory:

The context representation \(d_{t}\), along with the hidden state of decoder \(h_{t}\), is used to attend over the multi-level memory. The first level attention, \(\alpha\)., is applied over the queries \(q\).

\[\alpha_{i}=\frac{exp(w_{2}^{T}tanh(W_{4}[d_{t},h_{t},v_{i}^{v}]))}{\sum_{i} exp(w_{2}^{T}tanh(W_{4}[d_{t},h_{t},q_{i}^{v}]))} \tag{6}\]

The second level attention, \(\beta_{i.}\), is the attention over the results \(r_{i.}\) of query \(q_{i}\).

\[\beta_{ij}=\frac{exp(w_{3}^{T}tanh(W_{5}[d_{t},h_{t},r_{ij}^{v}]))}{\sum_{j} exp(w_{3}^{T}tanh(W_{5}[d_{t},h_{t},r_{ij}^{v}]))} \tag{7}\]

The product of first level attention and second level attention is the attention over results of all the queries in the multi-level memory. The weighted sum of the first level attention, second level attention and result representations gives us the attended memory representation, \(m_{t}\).

\[m_{t}=\sum_{i}\sum_{j}\alpha_{i}\beta_{ij}r_{ij}^{v} \tag{8}\]

Each result is further composed of multiple result cells. On the last level of memory, which contains the result cells, we apply key-value attention similar to Eric and Manning (2017). The key of the result cell is the word embedding of the slot, \(k_{a}^{r_{ij}}\), in the result. The attention scores, \(\gamma_{ij.}\), for the keys represent the attention over the result cells of each result \(r_{ij}\).

\[\gamma_{ijl}=\frac{exp(w_{4}^{T}tanh(W_{6}[d_{t},h_{t},\phi^{emb}(k_{l}^{r_{ ij}})]))}{\sum_{l}exp(w_{4}^{T}tanh(W_{6}[d_{t},h_{t},\phi^{emb}(k_{l}^{r_{ij}})]))} \tag{9}\]

The product of first level attention \(\alpha_{i}\), second level attention \(\beta_{ij}\) and third level attention \(\gamma_{ijl}\) gives the final attention score of the value \(v_{l}^{r_{ij}}\) in the KB memory. These final attention scores when combined (Eq. 10), form the copy distribution, \(P_{kb}(y_{t})\), over the values in KB memory.

\[P_{kb}(y_{t}=w)=\sum_{ijl:v_{l}^{r_{ij}}=w}\alpha_{i}\beta_{ij}\gamma_{ijl} \tag{10}\]

#### 3.3.4 Decoding

Similar to Gulcehre et al. (2016), we combine the generate and copy distributions - we use gate \(g_{2}\) (Eq. 11) to obtain the copy distribution \(P_{c}(y_{t})\) (Eq. 12) by combining \(P_{kb}(y_{t})\) and \(P_{con}(y_{t})\).

\[g_{2}=sigmoid(W_{7}[h_{t},d_{t},m_{t}]+b_{2}) \tag{11}\]

\[P_{c}(y_{t})=g_{2}P_{bb}(y_{t})+(1-g_{2})P_{con}(y_{t}) \tag{12}\]

Finally, we use gate \(g_{1}\) to obtain the final output distribution \(P(y_{t})\), by combining generate distribution \(P_{g}(y_{t})\) and copy distribution \(P_{c}(y_{t})\) as shown below:

\[g_{1}=sigmoid(W_{8}[h_{t},d_{t},m_{t}]+b_{3}) \tag{13}\]

\[P(y_{t})=g_{1}P_{g}(y_{t})+(1-g_{1})P_{c}(y_{t}) \tag{14}\]

We train our model by minimizing the cross entropy loss \(-\sum_{t=1}^{T}log(P(y_{t}))\).

## 4 Experiments

### Datasets

We present our experiments using three real world publicly available multi-turn task oriented dialogue datasets: the InCar assistant Eric and Manning (2017), CamRest Su et al. (2016) and the Maluuba Frames dataset El Asri et al. (2017). All three datasets contain human-human task oriented dialogues which were collected in a Wizard-of-Oz Wen et al. (2017) setting.

(i) **InCar assistant dataset** consists of \(3031\) multi-turn dialogues in three distinct domains: calendar scheduling, weather information retrieval, and point-of-interest navigation. Each dialogue has it's own KB information provided and thus, the system does not have to make any queries 

(ii) **CamRest dataset**, consists of \(676\) human-to-human dialogues set in the restaurant reservation domain. There are three queryable slots (food, price range, area) that users can specify. This dataset has currently been used for evaluating slot-tracking systems. Recent work by Lei et al. (2018) uses an end-to-end network without a KB and substitutes slot values with placeholders bearing the slot names in agent responses. However, we formatted the data to evaluate end-to-end systems by adding API call generation from the slot values so that restaurant suggestion task can proceed from the KB results.

(iii) **Maluuba Frames dataset**, consists of \(1369\) dialogues developed to study the role of memory in task oriented dialogue systems. The dataset is set in the domain of booking travel packages which involves flights and hotels. In contrast to the previous two datasets, this dataset contains dialogs that require the agent to remember all information presented previously as well as support results from multiple queries to the knowledge base. A user's preferences may change as the dialogue proceeds, and can also refer to previously presented queries (non-sequential dialog). Thus, to store multiple queries, we require \(3\) levels in our multi-level memory as compared to \(2\) levels in the other datasets, since they don't have more than one query. We do not use the dialogue frame annotations and use only the raw text of the dialogues. We map ground-truth queries to API calls that are also required to be generated by the model. Recent work has used this dataset only for frame tracking Schulz et al. (2017) and dialogue act prediction Peng et al. (2017); Tang et al. (2018). To the best of our knowledge we are the first to attempt the end-to-end dialog task using this dataset. Table 3 summarizes the statistics of the datasets.

### KB API Call Generation

In this section, we briefly describe how the knowledge base queries are generated as API calls as part of the model response. The InCar assistant dataset has a fixed KB for each dialogue whereas the CamRest and Malauba datasets require queries to be fired on a global KB. Queries in CamRest dataset can have 3 slots namely cuisine, area and precrange, whereas those in Maluuba can have 8 slots, which are destination, origin, start date, end date, budget, duration, number of adults and children. Any query that is to be fired on the KB is expected to be generated by the model as an API call, by considering a fixed ordering of slots in the generated response. For eg., in CamRest dataset, ApiCall(area=south, pricerange=cheap) would be generated by the model as _api_call dontcare south cheap_, with _dontcare_ meaning that the user does not have any preference for cuisine and, _south_, _cheap_ being the user constraints for area and pricerange respectively. Therefore, the task of API call generation typically involves copying relevant entities that are present in dialog context.

### Training

Our model is trained end-to-end using Adam optimizer Kingma and Ba (2014) with a learning rate of \(2.5e^{-4}\). The batch-size is sampled from [8,16]. We use pre-trained Glove vectors Pennington et al. (2014) with an embedding size of 200. The GRU hidden sizes are sampled from [128, 256]. We tuned the hyper-parameters with grid search over the validation set and selected the model which gives best entity F1.

### Evaluation Metrics

#### 4.4.1 Bleu

We use the commonly used BLEU metric Papineni et al. (2002) to study the performance of our systems as it has been found to have strong correlation Sharma et al. (2017) with human judgments in task-oriented dialogs.

#### 4.4.2 Entity F1

To explicitly study the behaviour of different memory architectures, we use the entity \(F1\) to measure how effectively values from the knowledge base are used in the dialog. To compute the entity F1, we micro-average the precision and recall over the entire set of system responses to compute the micro F12. For the InCar Assistant dataset, we compute a per-domain entity F1 as well as the aggregated entity F1. Since our model does not have slot-tracking by design, we evaluate

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline  & IncAr & CamRest & Maluuba Frames \\ \hline Train Dialogs & 2425 & 406 & 1095 \\ \hline Val Daibags & 302 & 135 & 137 \\ \hline Test Dialogs & 304 & 135 & 137 \\ \hline Avg. no. of turns & 2.6 & 5.1 & 9.4 \\ \hline Avg length, of sys. resp. & 8.6 & 11.7 & 14.8 \\ \hline Avg no. of yrs. entities & 1.6 & 1.7 & 2.9 \\ \hline Avg no. of queries & 0 & 1 & 2.4 \\ \hline Avg no. of KB entries & 66.1 & 13.5 & 141.2 \\ \hline \end{tabular}
\end{table}
Table 3: Statistics for \(3\) different datasets.

 

on entity F1 instead of the slot-tracking accuracy as in Henderson et al. (2014); Wen et al. (2017)

### Baselines

We experiment with the following baseline models for comparing the performance of our Multi-Level Memory architecture:

**Attn seq2seq3**Luong et al. (2015): A model with simple attention over the input context at each time step during decoding.

Footnote 3: We use the implementation provided by Madotto et al. (2018) at [https://github.com/HLTCHKUST/Mem2Seq](https://github.com/HLTCHKUST/Mem2Seq)

**Ptr-UNK\({}^{3}\)Gulcehre et al. (2016): The model augments a sequence-to-sequence architecture with attention-based copy mechanism over the encoder context.**

**KVRet**Eric and Manning (2017): The model uses key value knowledge base in which the KB is represented as triples in the form of \(subject-relation-object\). This model does not support copying words from context. The sum of word embeddings of \(subject\), \(relation\) is used as the key of the corresponding \(object\).

**Mem2Seq\({}^{3}\)Madotto et al. (2018): The model uses a memory networks based approach for attending over dialog history and KB triples. During decoding, at each time step, the hidden state of the decoder is used to perform multiple hops over a single memory which contains both dialog history and the KB triples to get the pointer distribution used for generating the response.**

### Results

Table 4 shows the performance of our model against our baselines. We find that our multi- level memory architecture comprehensively beats all existing models, thereby establishing new state-of-the- art benchmarks on all three datasets. Our model outperforms each baseline on both BLEU and entity F1 metrics.

**InCar:** On this dataset, we show entity F1 scores for each of the scheduling, weather and navigation domains. Our model has the highest F1 scores across all the domains. It can be seen that our model strongly outperforms Mem2Seq on each domain. A detailed study reveals that the use of triples cannot handle cases when a user queries with _non-subject_ entries or in cases when the response requires inferencing over multiple entries. In contrast, our model is able to handle such cases since we use a compound representation of entire result (bag of words over values) while attending on that result.

**CamRest:** Our model achieves the highest BLEU and entity F1 scores on this dataset. From Table 4, we see that simpler baselines like Ptr-UNK show competitive performance on this dataset because, as shown in Table 3, CamRest dataset has relatively fewer KB entries. Thus, a simple mechanism for copying from context results in good entity F1 scores.

**Maluuba Frames:** The Maluuba Frames dataset was introduced for the frame tracking task. Here, a dialog frame is a structured representation of the current dialog state. Instead of explicitly modeling the dialog frames, we use the context representation \(d_{t}\) to directly attend on the Multi-level memory. As Table 3 shows, this dataset contains significantly longer contexts as well as larger number of entities, as compared to the other two datasets. In addition, unlike other datasets, it also contains non-linear dialog flows where a user may refer to previously executed queries and results. The complexity of this dataset is reflected in the relatively lower BLEU and F1 scores as compared to other datasets.

### Analysis

#### 4.7.1 Entity source-wise performance

To further understand the effect of separating context memory from KB memory and using a multi

\begin{table}
\begin{tabular}{l||c c c|c c|c c c} \hline  & \multicolumn{4}{c}{InCar} & \multicolumn{4}{c}{CamResNet} & \multicolumn{2}{c}{Maluuba Frames} \\ \cline{2-9} Model & BLEU & F1 & CLendar & Weather & Navigate & \multirow{2}{*}{BLEU} & F1 & BLEU & F1 \\ \hline Attn seq2seq Luong et al. (2015) & 11.3 & 28.2 & 36.9 & 35.7 & 10.1 & 7.7 & 25.3 & 3.7 & 16.2 \\ Pre-UNK Gulcehre et al. (2016) & 5.4 & 20.4 & 22.1 & 24.6 & 14.6 & 5.1 & 40.3 & 5.6 & 25.8 \\ KVNet Eric and Manning (2017) & 13.2 & 48.0 & 62.9 & 47.0 & 41.3 & 13.0 & 36.5 & 10.7 & 31.7 \\ Mem2Seq Matotto et al. (2018) & 11.8 & 40.9 & 61.6 & 39.6 & 21.7 & 14.0 & 52.4 & 7.5 & 28.5 \\ \hline Multi-level Memory Model (MM) & **17.1** & **55.1** & **68.3** & **53.3** & **44.5** & **15.9** & **61.4** & **12.4** & **39.7** \\ \hline \end{tabular}
\end{table}
Table 4: Comparison of our model with baselines

\begin{table}
\begin{tabular}{l||c c|c c|c c} \hline  & \multicolumn{2}{c|}{**InCar**} & \multicolumn{2}{c|}{**CamRest**} & \multicolumn{2}{c}{**Maluuba**} \\ \cline{2-7}  & Ctxt. & KB & Ctxt. & KB & Ctxt. & KB \\ \hline \hline Mem2Seq & 66.2 & 25.3 & 63.7 & 36.5 & 17.7 & 8.9 \\ Multi-level Mem & 81.6 & 37.5 & 70.1 & 53.4 & 27.2 & 14.6 \\ \hline \end{tabular}
\end{table}
Table 5: Percentage (%) of category-wise (context vs KB) ground truth entities correctly captured in generated response. Abbreviation Ctxt denotes context.

 

level memory for KB, Table 5 shows the percentage of ground-truth entities, according to their category, which were also present in the generated response. For example, on the InCar dataset, out of the \(930\) entities in ground-truth response that were to be copied from the KB, our model was able to copy \(37.5\%\) of them into the generated response. From Table 5, it can be seen that our model is able to correctly copy a significantly larger number of entities from both, KB and context, as compared to the recent Mem2Seq model in all datasets.

#### 4.7.2 Model ablation study

We report results from ablation studies on all three datasets. Table 6 shows the incremental benefit obtained from individual components used in our model. We investigate the gains made by (i) Using separate memory for context and KB triples (ii) Replacing KB triples with a Multi-level memory. We use the recent Mem2Seq model for comparison with a unified context and KB memory model.

As can be seen from Table 6, the separation of context memory and KB memory leads to a significant improvement in BLEU and F1 scores on all datasets. This validates our hypothesis that storing context words and KB results in a single memory confuses the memory reader. The use of a multi-level memory instead of triples leads to further gains. This suggests, better organization of KB result memory by keeping the natural hierarchy intact is beneficial.

#### 4.7.3 Error Analysis

We analyzed the errors made by our dialog model on 100 dialog samples in test set of Maluuba Frames. We observed that the errors can be divided into five major classes: (i) Model outputs wrong KB result entry due to incorrect attention (27%), (ii) Model returns package details instead of asking for more information from the user (16%), (iii) Model incorrectly captures user intent (13%), (iv) Model makes an error due to non-sequential nature of dialog (22%). In such errors, our model either generates an API call for a result already present in memory, or our model asks for a query-slot value that was already provided by the user, (v) Data specific characteristics such as insufficient samples for certain classes of utterances (eg: more than one package returned) or returning different, but meaningful package attributes as compared to ground-truth data, contribute to 22% of the errors.

### Human Evaluation

We also conducted a blind user study that compared outputs from our model, Mem2Seq and KVRet systems. We used \(96\) randomly selected examples from each test split of Maluuba and CamRest datasets resulting in a total of 192 examples. Our study was split across 8 users who were provided with results fetched from the KB, current dialog context, gold response and the outputs of each of the models. Model outputs were shuffled in each example and users were asked to score each output between 1 (lowest) to 5 (highest) in terms of its accuracy of information in response and the quality of language. The results of this study are presented in Table 7. We also report the MRR (mean-reciprocal rank) for model preference along with other scores.

It can be seen that our model consistently ranks high for both information accuracy and language quality as well as reports a higher MRR. To further understand the quality of model performance, we asked the human evaluators whether their best ranked model output was a useful response. We saw that the evaluators agreed in 76.04% and 58.33% of the cases for CamRest and Maluuba datasets respectively. We observe that the results from human evaluation go hand-in-hand with automatic evaluation and reinforce our claim that separating context, KB memory and using a multi-level representation for the KB memory are useful for improving dialog modeling.

\begin{table}
\begin{tabular}{l||c c c c|c||c c c} \hline  & \multicolumn{4}{c||}{InCar} & \multicolumn{4}{c}{CamBest} & \multicolumn{2}{c}{Maluba Frames} \\ \hline \hline Model & BLEU & F1 & Calendar & Weather & Navigate & BLEU & F1 & BLEU & F1 \\ \hline Unified Context and KB memory (Mem2Seq) & 11.8 & 40.9 & 61.6 & 39.6 & 21.7 & 14.0 & 52.4 & 7.5 & 28.5 \\ \hline Separate Context and KB Memory & 14.3 & 44.2 & 56.9 & 54.1 & 24.0 & 14.3 & 55.0 & 12.1 & 36.5 \\ +Replace KB Triples with Multi-level memory & 17.1 & 55.1 & 68.3 & 53.3 & 44.5 & 15.9 & 61.4 & 12.4 & 39.7 \\ \hline \end{tabular}
\end{table}
Table 6: Model ablation study : Effect of (i) separate memory and (ii) multi-level memory design.

\begin{table}
\begin{tabular}{c||c c c|c c c} \hline  & \multicolumn{4}{c|}{**CamRest**} & \multicolumn{4}{c}{**Maluba**} \\ \hline  & Info. & Lang. & MRR & Info. & Lang. & MRR \\ \hline KVRet & 2.49 & 4.38 & 0.57 & 2.42 & 3.31 & 0.64 \\ \hline Mem2Seq & 2.48 & 3.72 & 0.51 & 1.78 & 2.55 & 0.50 \\ \hline Our Model & **3.62** & **4.48** & **0.76** & **2.45** & **3.93** & **0.69** \\ \hline \end{tabular}
\end{table}
Table 7: Informational accuracy and language quality scores using a human study.

 



### Attention Visualization

Analyzing the attention weights is a useful way to understand how the model is inferencing over the memory to copy entities from it. Table 8 shows an example of a dialog from the Maluuba Frames dataset and the outputs generated by different models. Here, the user first wants to know about packages to 'Manas' and then requests for trips to 'Pittsburgh'. Later, the user becomes interested in the 3.5 star hotel in Pittsburgh which was suggested by the agent and wants to know its guest rating. It can be seen from Table 8 that our model outputs the correct guest rating (8.86) of the hotel. Mem2Seq fails to understand the context and generates an irrelevant response. KVRet generates a readable response but points to the guest rating of a different hotel.

The attention over the memory while generating the word \(`8.86^{}\) for this example is shown in Fig 10. Fig 11(a) shows that the query with destination as 'Pittsburgh' gets the highest attention and among the results of this query, the package with the 3.5 star rated hotel gets highest attention. Within this result, the model gives highest score to the result cell with guest rating as the key. To further understand why the correct result hotel gets higher attention, Fig 11(b) shows the attention scores over the words in context memory. The context representation \(d_{t}\) captures the important words (3.5, guest, rating) in context which are in-turn used to apply attention over the multi-level memory.

Lastly, studying the values of the gates \(g_{1}\) (prob. of generating from vocabulary) and \(g_{2}\) (prob. of copying from KB), we found that gate \(g_{1}\) had a probability value of _O.08_ thereby driving the model to copy from memory instead of generating from output vocabulary and gate \(g_{2}\), with a probability value of _O.99_, was responsible for selecting KB memory over context memory.

## 5 Conclusion

In this paper, we presented an end-to-end trainable novel architecture with multi-level memory for task oriented dialogues. Our model separates the context and KB memory and combines the attention on them using a gating mechanism. The multi-level KB memory reflects the natural hierarchy present in KB results. This also allows our model to support non-sequential dialogs where a user may refer to a previously suggested result. We find that our model beats existing approaches by 15-25% on both entity F1 and BLEU scores, establishing state-of-the-art results on three publicly available real-world task oriented dialogue datasets. In a user study comparing outputs from our system against recent models, we found that our model consistently scored higher for both language quality as well as correctness of information in the response. We also present the benefits of each of our design choices by performing an ablation study. In future work, we would like to incorporate better modeling of latent dialog frames so as to improve the attention signal on our multi-level memory. As our error analysis suggests, nearly 22% of the errors could possibly be reduced by improved modeling of the dialog context. We believe that model performance can also be improved by capturing user intent better in case of non-sequential dialog flow.

Figure 10: Visualization of attention over memory while generating the word ‘8.86’ for the example in Table 8.

\begin{table}
\begin{tabular}{l l l} \hline \hline
**Bade** & **Turn** & **Uiterance** \\ \hline \hline Agent & 1 & belids ’how cat’ help you today \\ \hline User & 2 & _if not every token or slot. I have 4500 dollars per two scenarios days, if we consider what a prejudied yet to humans would be life_ \\ \hline Agent & 2 & 1 s cash out that there were no operating from the user \\ \hline User & 3 & _run Disparing, set constraints since some continuously thought_ \\ \hline Agent & 5 & [MPICAL names tiluents 28/86/9 4500...] \\ \hline User & 7 & third’s free, what other pitchards \\ \hline Agent & 7 & [APICAL pittsquurthijnama ... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... ........ .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... .... ......... .... .... .... .... .... .... .





## References

* Bordes and Weston (2017) Antoine Bordes and Jason Weston. 2017a. Learning end-to-end goal-oriented dialog. ICLR, abs/1605.07683. Cited by: SS4.1. A. Bordes and J. Weston (2017b)Learning end-to-end goal-oriented dialog. In International Conference on Learning Representations, External Links: Link Cited by: SS1, SS2.1. S. Chandar, S. Ahn, H. Larochelle, P. Vincent, G. Tesauro, and Y. Bengio (2016)Hierarchical memory networks. CoRRabs/1605.07427. Cited by: SS1.
* H. Chen, Z. Ren, J. Tang, Y. E. Zhao, and D. Yin (2018)Hierarchical variational memory network for dialogue generation. In Proceedings of the 2018 World Wide Web Conference, WWW '18, Republic and Canton of Geneva, Switzerland, pp. 1653-1662. External Links: ISBN 978-3-319-

Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In _Proceedings of the International Conference on Machine Learning_.
* Wen et al. (2017) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M. Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, pages 438-449. Association for Computational Linguistics.
* Xu et al. (2016) Jiaming Xu, Jing Shi, Yiqun Yao, Suncong Zheng, Bo Xu, and Bo Xu. 2016. Hierarchical memory networks for answer selection on unknown words. In _COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan_, pages 2290-2299.