Venue: NAACL 2019

# Disentangling Language and Knowledge in Task-Oriented Dialogs

Dinesh Raghu\({}^{1}\)\({}^{2}\), Nikhil Gupta\({}^{1}\) and Mausam\({}^{1}\)

\({}^{1}\) IIT Delhi, New Delhi, India

\({}^{2}\) IBM Research, New Delhi, India

_diraghui@in.ibm.com, nikhilgupta1997@gmail.com, mausam@cse.iitd.ac.in_

 D. Raghu is an employee at IBM Research. This work was carried out as part of PhD research at IIT Delhi.

###### Abstract

The Knowledge Base (KB) used for real-world applications, such as booking a movie or restaurant reservation, keeps changing over time. End-to-end neural networks trained for these task-oriented dialogs are expected to be immune to any changes in the KB. However, existing approaches breakdown when asked to handle such changes. We propose an encoder-decoder architecture (BoSSNet) with a novel Bag-of-Sequences (BoSs) memory, which facilitates the disentangled learning of the response's language model and its knowledge incorporation. Consequently, the KB can be modified with new knowledge without a drop in interpretability. We find that BoSsNet outperforms state-of-the-art models, with considerable improvements (>10%) on bAbI OOV test sets and other human-human datasets. We also systematically modify existing datasets to measure disentanglement and show BoSsNet to be robust to KB modifications.

## 1 Introduction

Task-oriented dialog agents converse with a user with the goal of accomplishing a specific task and often interact with a knowledge-base (KB). For example, a restaurant reservation agent Henderson et al. (2014) will be grounded to a KB that contains the names of restaurants, and their details.

In real-world applications, the KB information could change over time. For example, (1) a KB associated with a movie ticket booking system gets updated every week based on new film releases, and (2) a restaurant reservation agent, trained with the knowledge of eateries in one city, may be deployed in other cities with an entirely different range of establishments. In such situations, the system should have the ability to conform to new-found knowledge unseen during its training. Ideally, the training algorithm must learn to disentangle the language model from the knowledge interface model. This separation will enable the system to generalize to KB modifications, without a loss in performance.

Moreover, for achieving good progress towards the user's task, the agent must also retain the ability to draw inferences based on past utterances and the KB. Notably, we find that existing approaches either achieve this disentanglement or effective progress towards the task, but not both.

For instance, Mem2Seq Madotto et al. (2018) exhibits satisfactory performance when tested on the training KB. It represents the dialog history and the KB knowledge as a _bag of words_ in a flat memory arrangement. This enables Mem2Seq to revisit each word several times, as needed, obtaining good performance. But at the same time, flat memory prevents it from capturing any surrounding context - this deteriorates its performance rapidly when the amount of new unseen information in the KB increases, as shown in Figure 1. On the other hand, the performance of copy augmented sequence-to-sequence network Seq2Seq+Copy) Eric and Manning (2017), is robust to changes in the KB, but fails to achieve acceptable task-oriented performance. It captures context by representing the entire dialog history as one continuous _sequence_. However, it can be difficult for a sequence encoder to reason

Figure 1: Performance of various task-oriented dialog systems on the CamRest dataset as the percentage of unseen information in the KB changes.



over long dialogs found in real-world datasets and its ability to learn the task gets hampered.

We propose BoSsNet, a novel network that effectively disentangles the language and knowledge models, and also achieves state-of-the-art performance on three existing datasets.

To achieve this, BoSsNet makes two design choices. First, it encodes the conversational input as a _bag of sequences_ (BoSs) memory, in which the input representation is built at two levels of abstraction. The _higher level_ flat memory encodes the KB tuples and utterances to facilitate effective inferencing over them. The _lower level_ encoding of each individual utterance and tuple is constructed via a sequence encoder (Bi-GRU). This enables the model to maintain the sequential context surrounding each token, aiding in better interpretation of unseen tokens at test time. Second, we augment the standard cross-entropy loss used in dialog systems with an additional loss term to encourage the model to only copy KB tokens in a response, instead of generating them via the language model. This combination of sequence encoding and additional loss (along with dropout) helps in effective disentangling between language and knowledge.

We perform evaluations over three datasets - bAbI Bordes and Weston (2017), CamRest Wen et al. (2016), and Stanford Multi-Domain Dataset Eric et al. (2017). Of these, the last two are real-world datasets. We find that BoSsNet is competitive or significantly better on standard metrics in all datasets as compared to state-of-the-art baselines. We also introduce a _knowledge adaptability_ (KA) evaluation, in which we systematically increase the percentage of previously unseen entities in the KB. We find that BoSsNet is highly robust across all percentage levels. Finally, we also report a human-based evaluation and find that BoSsNet responses are frequently rated higher than other baselines.

Overall, our contributions are:

1. We propose BoSsNet, a novel architecture to disentangle the language model from knowledge incorporation in task-oriented dialogs.
2. We introduce a _knowledge adaptability_ evaluation to measure the ability of dialog systems to scale performance to unseen KB entities.
3. Our experiments show that BoSsNet is competitive or significantly better, measured via standard metrics, than the existing baselines on three datasets.

We release our code and _knowledge adaptability_ (KA) test sets for further use by the research community.1

Footnote 1: [https://github.com/dair-iitd/BossNet](https://github.com/dair-iitd/BossNet)

## 2 The BoSsNet Architecture

The proposed Bag-of-Sequences Memory Network has an encoder-decoder architecture that takes as input (1) dialog history, which includes a sequence of previous user utterances \(\{c^{u}_{1},\dots,c^{u}_{n}\}\) and system responses \(\{c^{s}_{1},\dots,c^{s}_{n-1}\}\), and (2) KB tuples \(\{kb_{1},\dots,kb_{N}\}\). The network then generates the next system response \(c^{s}_{n}=\langle y_{1}y_{2}\ldots y_{T}\rangle\) word-by-word. The simplified architecture of BoSsNet is shown in Figure 2.

In this section, we first describe the BoSs memory which contains the dialog history and KB tuples, followed by how the memory is consumed by the encoder and the decoder. We finally define the loss function, which, along with dropout, enables disentangled learning of language and knowledge.

### Bag-of-Sequences Memory

The memory \(M\) contains the dialog history \(\{c^{u}_{1},c^{s}_{1},\dots,c^{u}_{n-1},c^{s}_{n-1}\}\) and the KB tuples \(\{kb_{1},\dots,kb_{N}\}\). Each utterance in the dialog history and each KB tuple is placed in a memory cell. As utterances and tuples are inherently a sequence, we represent each memory cell \(m_{i}\) as

Figure 2: The dialog history and KB tuples stored in the memory have memory cell representations and token representations. The encoder understands the last user utterance using only the memory cell representations. The decoder generates the next response using both representations.

 

an ordered sequence of tokens \(\langle w_{i}^{1}w_{i}^{2}\dots w_{i}^{|m_{i}|}\rangle\). For an utterance, the word tokens are followed by a temporal indicator and a speaker indicator {Su, SSs}. For example, {good, morning,#1, SSs} indicates this was the first utterance by the system. For a KB tuple, the tokens are sequenced as {_subject, predicate, object_} followed by temporal indicator and a kb indicator ($db).

Token representation is generated using a bidirectional GRU. Let the outputs of the forward and backward GRUs for the token \(w_{i}^{j}\) be denoted as \(\overrightarrow{h_{i}^{j}}\) and \(\overleftarrow{h_{i}^{j}}\) respectively. Then the token representation \(\phi(w_{i}^{j})\) is given by Eq. 1. Memory cell representation \(\psi(m_{i})\) is computed by concatenating the forward GRU output of its last token and the backward GRU output of its first token as in Eq. 2.

\[\phi(w_{i}^{j})=\overrightarrow{[h_{i}^{j};h_{i}^{j}]} \tag{1}\] \[\psi(m_{i})=[\overrightarrow{[h_{i}^{|m_{i}|};h_{i}^{1}]} \tag{2}\]

### The BoSsNet Encoder

The encoder used in BoSsNet is similar to the multi-hop attention encoder with layer-wise weights proposed by Sukhbaatar et al. (2015). The encoder in Sukhbaatar et al. (2015) uses two different embedding matrices, whereas we use just one to reduce the number of parameters. The encoder considers the last user utterance as the query \(q=\psi(c_{n}^{u})\) and computes the reduced representation \(q_{r}\) using the memory \(M\) as follows:

\[p_{i} = \text{softmax}(q^{T}\psi(m_{i})) \tag{3}\] \[o = W_{r}\sum\nolimits_{i}p_{i}\psi(m_{i})\] (4) \[q_{r} = o+W_{o}q \tag{5}\]

where \(W_{r},W_{o}\in\mathbb{R}^{d\times d}\) are learnable parameters. The hop step can be re-iterated, by assigning the output of the previous hop as the new input query, i.e., setting \(q=q_{r}\). The output of the encoder after \(K\) hops, \(q_{r}^{k}\), is assigned as the initial state of the BoSsNet decoder.

### The BoSsNet Decoder

BoSsNet models a copy-augmented sequence decoder, which generates the response one word at a time. At any decode time step \(t\), the decoder can either _generate_ a word from the decode vocabulary or _copy_ a word from the memory. Consequently, the decoder computes: (1) generate distribution \(P_{g}(y_{t})\) over the decode vocabulary, and (2) copy distribution \(P_{c}(y_{t})\) over words in the memory.

The generate distribution is computed using a standard sequence decoder (Sutskever et al., 2014) by attending (Luong et al., 2015) over the memory cell representations \(\psi\). The copy distribution is generated by using a _two-level attention_. Given the decoder state \(s_{t}\), it first computes attention \(\alpha_{t}\) over the memory cells. Then it computes attention over the tokens in each memory cell \(m_{i}\). Finally it multiplies both these attentions to compute \(P_{c}(y_{t})\) as follows:

\[\alpha_{i}^{t}=\text{softmax}(s_{t}\psi(m_{i})) \tag{6}\] \[e_{ij}^{t}=s_{t}\phi(w_{i}^{j})\] (7) \[\beta_{ij}^{t}=\alpha_{i}^{t}*\frac{\exp(e_{ij}^{t})}{\sum_{k} \exp(e_{ik}^{t})}\] (8) \[P_{c}(y_{t}=w)=\sum_{ij:w_{i}^{j}=w}\beta_{ij}^{t} \tag{9}\]

The copy and generate distributions are combined using a soft gate \(g_{s}^{t}\in[0,1]\) as in See et al. (2017). \(g_{s}^{t}\) is a function of the decoder state at time \(t\) and the word decoded in the previous time step.

### Loss

The decoder is trained using cross-entropy loss. The loss per response is defined as:

\[\mathcal{L}_{ce}=-\sum_{t=1}^{T}\text{log}\Big{(}g_{s}^{t}P_{g}(y_{t})+(1-g_{ s}^{t})P_{c}(y_{t})\Big{)} \tag{10}\]

where \(T\) is the number of words in the sequence to be generated and \(y_{t}\) is the word to be generated at time step \(t\). The decision to generate or copy is learnt implicitly by the network. However, to attain perfect disentanglement, the KB words should be copied, while the language should be generated. In other words, any word in the response that is present in the BoSs KB memory should have a low \(g_{s}\). To obtain this behavior, we define a disentangle label \(D_{l}\) for each word in the response. This label is set to \(1\) if the word is present in the BoSs KB memory and \(0\) otherwise. We define a disentangle loss as follows:

\[\mathcal{L}_{d}=-\sum_{t=1}^{T}g_{s}^{t}\text{log}D_{l}^{t}+(1-g_{s}^{t})\text {log}(1-D_{l}^{t}) \tag{11}\]

We randomly drop some words with disentangle label set to \(1\). This _Disentangle Label Dropout (DDD)_ works in tandem with the disentangle loss and BoSs memory - it encourages the model to 

copy KB words whenever possible, based on their surrounding words. The overall loss is given as:

\[{\mathcal{L}}={\mathcal{L}_{ce}+\gamma{\mathcal{L}_{d}}} \tag{12}\]

The relative weight of \({\mathcal{L}_{d}}\) in the overall loss is controlled using a hyper-parameter (\(\gamma\)). The dropout rate is also a hyper-parameter.

## 3 Experimental Setup

We perform experiments on three task-oriented dialog datasets: bAbI Dialog Bordes and Weston (2017), CamRest Wen et al. (2016), and Stanford Multi-Domain Dataset Eric et al. (2017).

**bAbI Dialog** consists of synthetically generated dialogs with the goal of restaurant reservation. The dataset consists of five different tasks, all grounded to a KB. This KB is split into two mutually exclusive halves. One half is used to generate the train, validation, and test sets, while the other half is used to create a second test set called the OOV test set.

**CamRest** is a human-human dialog dataset, collected using the Wiz-of-Oz framework, also aimed at restaurant reservation. It is typically used to evaluate traditional slot filling systems. In order to make it suitable for end-to-end learning, we stripped the handcrafted state representations and annotations in each dialog, and divided the 676 available dialogs into train, validation, and test sets (406, 135, and 135 dialogs, respectively).

**Stanford Multi-Domain Dataset (SMD)** is another human-human dialog dataset collected using the Wiz-of-Oz framework. Each conversation is between a driver and an in-car assistant. The other datasets consist of dialogs from just one domain (restaurant reservation), whereas SMD consists of dialogs from multiple domains (calendar scheduling, weather information retrieval, and navigation).

### Knowledge Adaptability (KA) Test Sets

Each bAbI dialog task has an additional OOV test set, which helps to evaluate a model's robustness to change in information in the KB. A model that perfectly disentangles language and knowledge should have no drop in accuracy on the OOV test set when compared to the non-OOV test set. To measure the degree of disentanglement in a model, we generated 10 additional test sets for each real-world corpus by varying the percentage (in multiples of 10) of unseen entities in the KB. We systematically picked random KB entities and replaced all their occurrences in the dialog with new entity names. We will refer to these generated dialogs as the _Knowledge Adaptability_ (KA) test sets.

### Baselines

We compare BoSsNet against several existing end-to-end task-oriented dialog systems. These include retrieval models, such as the query reduction network (QRN) Seo et al. (2017), memory network (MN) Bordes and Weston (2017), and gated memory network (GMN) Liu and Perez (2017). We also compare against generative models such as a sequence-to-sequence model (Seq2Seq), a copy augmented Seq2Seq (Seq2Seq+Copy) Gulcehre et al. (2016), and Mem2Seq Madotto et al. (2018).2 For fairness across models, we do not compare against key-value retrieval networks Eric et al. (2017) as they simplify the dataset by canonicalizing all KB words in dialogs.

Footnote 2: We thank the authors for releasing a working code at [https://github.com/HLTCHKUST/Mem2Seq](https://github.com/HLTCHKUST/Mem2Seq)

We noticed that the reported results in the Mem2Seq paper are not directly comparable, as they pre-processed3 training data in SMD and bAbI datasets. For fair comparisons, we re-run Mem2Seq on the original training datasets. For completeness we mention their reported results (with pre-processing) as Mem2Seq*.

Footnote 3: Mem2Seq used the following pre-processing on the data: 1) The subject (restaurant name) and object (rating) positions of the rating KB tuples in bAbI dialogs are flipped 2) An extra fact was added to the navigation tasks in SMD which included all the properties (distance, address, etc.) combined together as the subject and _poi_ as the object. See Appendix.

### Evaluation Metrics

We evaluate BoSsNet and other models based on their ability to generate valid responses. The per-response accuracy Bordes and Weston (2017) is the percentage of generated responses that exactly match their respective gold response. The per-dialog accuracy is the percentage of dialogs with all correctly generated responses. These accuracy metrics are a good measure for evaluating datasets with boilerplate responses such as bAbI.

To quantify performance on other datasets, we use BLEU Papineni et al. (2002) and Entity F1 Eric and Manning (2017) scores. BLEU measures the overlap of n-grams between the generated response and its gold response and has become a popular measure to compare task-oriented dialog systems. Entity F1 is computed by micro-F1 over KB entities in the entire set of gold responses.

 



### Human Evaluation

We use two human evaluation experiments to compare (1) the _usefulness_ of a generated response with respect to solving the given task, and (2) the _grammatical correctness_ and _fluency_ of the responses on a 0-3 scale. We obtain human annotations by creating Human Intelligence Tasks (HITs) on Amazon Mechanical Turk (AMT). For each test condition (percentage of unseen entities), we sampled 50 dialogs from Camrest and SMD each, and two AMT workers labeled each system response for both experiments, resulting in 200 labels per condition per dataset per system. We evaluate four systems in this study, leading to a total of 1600 labels per condition. The detailed setup is given in the Appendix.

### Training

We train BoSsNet using an Adam optimizer Kingma and Ba (2014) and apply gradient clipping with a clip-value of 40. We identify hyper-parameters based on the evaluation of the held-out validation sets. We sample word embedding, hidden layer, and cell sizes from {64, 128, 256} and learning rates from {10\({}^{-3}\), 5\(\times\)10\({}^{-4}\), 10\({}^{-4}\)}. The hyper-parameter \(\gamma\) in the loss function is chosen between [0-1.5]. The Disentangle Label Dropout rate is sampled from {0.1, 0.2}. The number of hops for multi-hop attention in the encoder is sampled from {1, 3, 6}. The best hyper-parameter setting for each dataset is reported in the Appendix.

## 4 Experimental Results

Our experiments evaluate three research questions.

1. _Performance Study_: How well is BoSsNet able to perform the tasks of our three datasets as compared to the baseline models?
2. _Disentanglement Study_: How robust are the models in generalizing on the KA test sets?
3. _Ablation Study_: What is the performance gain from each novel feature in BoSsNet?

### Performance Study

Table 1 reports the per-response and per-dialog (in parentheses) accuracies on the bAbI dialog tasks. The multi-hop retrieval-based models such as QRN, MN and GMN perform well on the non-OOV test sets for tasks 1, 2, and 5, but fail to exhibit similar performance on the corresponding OOV test sets. This result is expected as these models are trained to retrieve from a pre-defined set of responses. Their poor non-OOV performance on tasks 3 and 4 is attributed to an error in the bAbI dataset construction, due to which, the non-OOV and OOV test conditions are the same for these tasks (see Appendix).

A simple generative model (Seq2Seq) achieves accuracies comparable to the multi-hop retrieval models. Enabling it with the ability to copy from the context (Seq2Seq+Copy) shows a considerable increase in performance, especially on the OOV test sets (and non-OOV tests for tasks 3 and 4).

The strong performance of simple sequence encoders when compared with multi-hop encoders (in retrieval models) raises a question about the value of multi-hop inference. Mem2Seq answers this question, by obtaining improvements in several tasks, specifically on their OOV test sets. This clearly shows that multi-hop inference and the copy mechanism are essentials for task-oriented dialogs.

Despite gains from the Mem2Seq model, the performance difference between the non-OOV and OOV test sets remains large. BoSsNet succeeds to bridge this gap with its ability to better interpret unseen words, using their surrounding context. It obtains significant improvements on average of about 34% per-dialog accuracy and 10% per-response accuracy for the bAbI OOV test sets.

In Table 2, we report results on the real-world datasets. BoSSNet greatly outperforms other models in both Entity F1 metric and BLEU scores on CamRest. On SMD, BoSSNet achieves the best only in Entity F1. On further analysis of the generated responses we observe that BoSSNet responses often convey the necessary entity information from the KB. However, they consist of meaningful phrases with little lexical overlap with the gold response, reducing the BLEU scores. We investigate this further in our human evaluation.

**Human Evaluation:** We summarize the human evaluation results for real-world datasets in Table 3. BoSSnet shows the best performance on Camrest, and is judged useful 77 times out of 100. Also, it has the highest average grammatical correctness score of 2.28 (very close to Seq2Seq and Mem2Seq). BoSsNet performs on par with Mem2Seq and Seq2Seq in its ability to relay appropriate information to solve SMD dialog tasks, and has a slightly higher grammaticality score.

### Disentanglement Study

We use our generated knowledge adaptability (KA) test sets to measure the robustness of BoSsNet

and the other baselines to changes in the KB. We perform this experiment on 4 different tasks, namely bAbI tasks 1 and 5, CamRest, and SMD.

Figures 3 and 4 show the per-response accuracies of the two bAbI dialog tasks plotted against the percentage of unseen entities in KA sets. From Figure 3 we observe that BoSSNet remains immune to any variablity in the KB content, whereas the performance of Mem2Seq and Seq2Seq models drops drastically due to their inability to capture semantic representations of the injected KB entities. We see a similar trend in Figure 4, but here all the models show a drop in performance, with BoSsNet appearing the most steady. We explain this trend using the example dialog in Table 4. In the current dialog context, the system is required to provide the address of the selected restaurant, but since more than one restaurant in the KB is unseen, it becomes ambiguous for the network to identify the correct restaurant and infer its address. In the end, the system is forced to pick a random address - the probability of which being correct decreases as more restaurants become unseen.

The performance on the CamRest KA test sets is illustrated in Figures 1 and 5. BoSSNet has the best performance with even a slight increase in both BLEU and Entity F1 metrics as more OOV content is injected in the dialog, probably because it is clear that it needs to copy when processing unseen entities. Seq2Seq+Copy is unable to perform well in CamRest as the length of the input (dialog history + KB tuples) is long and the size of the training set is also small. We believe that Seq2Seq+Copy works best in an environment with an abundance of short dialog training data (e.g., bAbI task 1 in Figure 3).

 

its ability to inference efficiently on the large KB (Figure 6). Mem2Seq shows the best BLEU score performance on the original test set, but its performance drop of 42.5%, from 10.3 at 0% unseen to 5.93 at 100% unseen, is a lot heavier than that of BoSsNet which only drops 7.6% - 8.27 at 0% unseen to 7.64 at 100% unseen.

**Human Evaluation:** We summarize the human evaluation results for real-world datasets on the 50% unseen KA test set in Table 5. BoSsNet again outperforms the baselines and is labeled _successful_ twice more often than the next best model on both Camrest and SMD. Seq2Seq appears to produce better sentence structures on the SMD dataset, primarily because it does not attempt to learn inference on the KB, allowing it to solely focus on learning the language model better.

### Ablation Study

We assess the value of each model element, by removing it from BoSsNet. Table 6 reports the per-response accuracy scores for various configurations of BoSsNet on bAbI dialog tasks. It also reports the BLEU and entity F1 metric of various configurations on CamRest.

**Without BosS Memory:** This configuration uses the Bag-of-Bags (BoB) Memory rather than BoSs memory. The BoB memory is a simplified representation, similar to the one in the original Memory Networks. Here the token representation is the vector embedding of the token with no influence from the surrounding words and the memory cell representation is the sum of all its token embeddings. As a result, each word \(w\) representation is influenced equally by all words in a memory cell, irrespective of its distance from \(w\). This makes capturing context in the immediate neighbourhood harder. Inability to capture the correct context prevents the configuartion from generalizing to OOV test sets.

**Without Disentangled Loss:** Disentangled Loss (\(\mathcal{L}_{d}\)) plays an important role in enforcing that KB words be copied and other language be generated. By removing this loss component, it achieves better BLEU score in CamRest, but with a drop in Entity F1. Without the disentangled loss, the model sometimes learns to generate KB words. This severely affects OOV performance. As described earlier, an error in bAbI dataset construction tasks 3 and 4 effectively injects the validation set with a lot of OOVs. This anomaly in conjunction with the dropout (DLD), helps the configuration in achieving an acceptable performance for those tasks.

**Without Disentangled Label Dropout:** BoSsNet learns to generate language and copy KB words. Without DLD, the model learns to memorize words to be copied rather than learning the context under which a word should be copied. Hence, the performance on OOV test sets is much inferior compared to the non-OOV setting.

Overall, we notice that combining all three model elements is necessary in obtaining the best performance across all tasks.

### Qualitative Evaluation

We qualitatively compare the performance of BoSsNet with other baselines using examples.

Table 7, demonstrates the ability of BoSsNet to copy entities (restaurant name and address) in its response. The other baselines either generate unwanted or irrelevant entities in their response, or fail to copy altogether. BoSsNet also best captures the language model effectively with a slight paraphrasing of the gold response.

Table 8 contains only unseen entities. This example highlights the shortcomings of the Seq2Seq model as it ends up predicting a restaurant encountered during training. Mem2Seq copies a restaurant name without learning to sort the restaurants based on rating. BoSsNet, with its efficient memory addressing, is seen to be able to solve both issues.

## 5 Related Work

Compared to the traditional slot-filling based dialog Williams and Young (2007); Wen et al. (2017); Williams et al. (2017), end-to-end training methods (e.g., Bordes and Weston (2017), this work) do not require handcrafted state representations and their corresponding annotations in each dialog. Thus, they can easily be adapted to a new domain. We discuss end-to-end approaches along two verticals: 1) decoder: whether the response is retrieved or generated and 2) encoder: how the dialog history and KB tuples are encoded.

\begin{table}
\begin{tabular}{l|c c|c c} \hline \hline  & \multicolumn{2}{c|}{**CamRest**} & \multicolumn{2}{c}{**SMD**} \\ \cline{2-5}  & **Info** & **Grammar** & **Info** & **Grammar** \\ \hline Seq2Seq & 26 & 2.28 & 22 & **2.44** \\ Seq2Seq+Copy & 22 & 1.22 & 16 & 1.04 \\ Mem2Seq & 35 & 2.06 & 26 & 1.9 \\ \hline BoSsNet & **80** & **2.44** & **51** & 2.28 \\ \hline \hline \end{tabular}
\end{table}
Table 5: AMT Evaluations on CamRest and SMD (50% unseen) KA datasets 

Most of the existing end-to-end approaches _retrieve_ a response from a pre-defined set Bordes and Weston (2017); Liu and Perez (2017); Seo et al. (2017). These methods are generally successful when they have to provide boilerplate responses - they cannot construct responses by using words in KB not seen during training. Alternatively, generative approaches are used where the response is _generated_ one word at a time Eric and Manning (2017); Madotto et al. (2018). These approaches mitigate the unseen entity problem by incorporating the ability to copy words from the input Vinyals et al. (2015); Gu et al. (2016). The copy mechanism has also found success in summarization Nallapati et al. (2016); See et al. (2017) and machine translation Gulcehre et al. (2016). BoSsNet is also a copy incorporated generative approach.

For encoding, some approaches represent the dialog history as a sequence Eric and Manning (2017); Gulcehre et al. (2016). Unfortunately, using a single long sequence for encoding also enforces an order over the set of KB tuples making it harder to perform inferencing over them. Other approaches represent the dialog context as a bag. Original Memory Networks Bordes and Weston (2017) and its extensions encode each memory element (utterance) as an average of all constituent words - this cannot point to individual words, and hence cannot be used with a copy mechanism. Mem2Seq encodes each word individually in a flat memory. Unfortunately, this loses the contextual information around a word, which is needed to decipher an unseen word. In contrast, BoSsNet uses a bag of sequences encoding, where KB tuples are a set for easier inference, and also each utterance is a sequence for effectively learning when to copy.

 



## 6 Conclusions

We propose BoSsNet for training task-oriented dialog systems in an end-to-end fashion. BoSsNet combines a novel bag of sequences memory for storing a dialog history and KB tuples, with a copy-augmented generative decoder to construct dialog responses. It augments standard cross entropy loss of a sequence decoder with an additional term to encourage the model to copy KB words. BoSs memory and new loss term, in conjunction with a disentangle label dropout, enables the decoder to disentangle its language and knowledge models.

BoSsNet achieves the state of the art results on bAbI dialog dataset, outperforming existing models by 10 points or more in its OOV conditions. In the knowledge adaptability test, we find that BoSsNet is highly robust to increasing the percentage of unseen entities at test time, suggesting a good language-knowledge disentanglement. Human evaluations show that BoSsNet responses are highly informative and slightly more grammatical compared to baselines. We will release our code and all curated datasets for further research.

## Acknowledgments

We thank Danish Contractor, Gaurav Pandey and Sachindra Joshi for their comments on an earlier version of this work. This work is supported by IBM AI Horizons Network grant, an IBM SUR award, grants by Google, Bloomberg and 1MG, and a Visvesvaraya faculty award by Govt. of India. We thank Microsoft Azure sponsorships, and the IIT Delhi HPC facility for computational resources.

\begin{table}
\begin{tabular}{c|l} \hline \hline  & **KB (restaurant—cuisine—address—phone)** \\ _pizza\_hut\_fen\_ditton—italian—cambridge\_retail\_park\_newmarket\_road\_fen\_ditton —01223\_323737_ \\ \hline \hline
**usr-1** & may i have information for an _italian_ restaurant in the _east_ part of town ? \\
**sys-1** & yes sure . there is only on _italian_ restaurant called _pizza\_hut\_fen\_ditton_ in the _east_ . what else do you want to know ? \\
**usr-2** & what is their address please ? \\ \hline
**Gold** & the address for _pizza\_hut\_fen\_ditnton_ is cambridge\_retail\_park\_newmarket\_road\_fen\_ditton_ . can i help you with anything else ? \\ \hline \hline
**Seq2Seq+Copy** & their address is at \\ \hline
**Seq2Seq** & _curry\_prime_ is at _451\_newwarket\_road\_fen\_ditton_ \\ \hline
**Mem2Seq** & _pizza\_hut\_fen\_ditton_ is located at _7\_barmwell\_road\_fen\_ditton_ is _7\_barmwell\_road\_fen\_ditton_ and the phone number is _01223\_24955_ \\ \hline BoSNet & _pizza\_hut\_fen\_ditton_ is located at _cambridge\_retail\_park\_newmarket\_road\_fen\_ditton_ is there anything \\  & else i can help you with \\ \hline \hline \end{tabular}
\end{table}
Table 7: An example of responses generated by BoSSNet and baselines on the CamRest test set. Thia example has no unseen entities.

\begin{table}
\begin{tabular}{r|l} \hline \hline  & **KB (restaurant—cuisine–protein\_range—location—rating)** \\ _retso\_tokyo\_affordable\_vietnamese\_Sstars –vietnamese\_affordable—tokyo--8_ \\ _retso\_tokyo\_affordable\_vietnamese\_7\_stars –vietnamese—affordable—tokyo--7_ \\ _retso\_tokyo\_affordable\_vietnamese\_6stars –vietnamese\_affordable—tokyo—6_ \\ _retso\_tokyo\_affordable\_vietnamese\_5stars –vietnamese—affordable—tokyo—5_ \\ \hline \hline
**usr-1** & can you book a table in _tokyo_with _vietnamese_ food in a _offidable_ price range ? \\
**sy-1** & what do you think of this option : _ret\_tookyo\_affordable\_vietnamese\_8stars_ ? \\
**usv-2** & no this does not work for me : \\
**sys-2** & what do you think of this option : _retso\_tokyo\_affordable\_vietnamese\_7stars_ ? \\
**usr-3** & do you have something else ? \\ \hline
**Gold** & what do you think of this option : _retso\_tokyo\_affordable\_vietnamese\_6stars_ \\ \hline \hline
**Seq2Seq+Copy** & what do you think of this option : what ? \\ \hline
**Seq2Seq** & what do you think of this option : _retso\_london\_moderate\_british\_2stars_ ? \\ \hline
**Mem2Seq** & what do you think of this option : _retso\_tokyo\_affordable\_vietnamese\_5stars_ ? \\ \hline BoSsNet & what do you think of this option : _retso\_tokyo\_affordable\_vietnamese\_6stars_ ? \\ \hline \hline \end{tabular}
\end{table}
Table 8: An example of responses generated by BoSsNet and baselines on bAbI dialog Task-5. This example is from the KA test set with 100% unseen entities.





## References

* Bordes and Weston (2017) Antoine Bordes and Jason Weston. 2017. Learning end-to-end goal-oriented dialog. In _International Conference on Learning Representations_.
* Eric et al. (2017) Mihail Eric, Lakshmi Krishnan, Francois Charette, and Christopher D. Manning. 2017. Key-value retrieval networks for task-oriented dialogue. In _Diadog System Technology Challenges, Saarbrucken, Germany, August 15-17, 2017_, pages 37-49.
* Eric and Manning (2017) Mihail Eric and Christopher Manning. 2017. A copy-augmented sequence-to-sequence architecture gives good performance on task-oriented dialogue. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 468-473. Association for Computational Linguistics.
* Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1631-1640. Association for Computational Linguistics.
* Gulcehre et al. (2016) Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 140-149. Association for Computational Linguistics.
* Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with re- current neural networks. In _In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_, pages 292-299.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Lei Ba. 2014. Adam: Am method for stochastic optimization. In _Proc. 3rd Int. Conf. Learn. Representations_.
* Liu and Perez (2017) Fei Liu and Julien Perez. 2017. Gated end-to-end memory networks. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_, volume 1, pages 1-10.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421. Association for Computational Linguistics.
* Madotto et al. (2018) A. Madotto, CS. Wu, and P. Fung. 2018. Mem2seq: Effectively incorporating knowledge bases into end-to-end task-oriented dialog systems. In _Proceedings of 56th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics.
* Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_, pages 280-290. Association for Computational Linguistics.
* Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting on association for computational linguistics_, pages 311-318. Association for Computational Linguistics.
* See et al. (2017)bigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083. Association for Computational Linguistics.
* Seo et al. (2017) Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishrizi. 2017. Query-reduction networks for question answering. In _International Conference on Learning Representations_.
* Sukhbaatar et al. (2015)Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In _Advances in neural information processing systems_, pages 2440-2448.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In _Advances in neural information processing systems_, pages 3104-3112.
* Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In _Advances in Neural Information Processing Systems_, pages 2692-2700.
* Wen et al. (2017) TH Wen, D Vandyke, N Mrkscic, M Gasic, LM Rojas-Barahona, PH Su, S Ultes, and S Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In _15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017-Proceedings of Conference_, volume 1, pages 438-449.
* Wen et al. (2016) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M. Rojas Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. Conditional generation and snapshot learning in neural dialogue systems. In _EMNLP_, pages 2153-2162, Austin, Texas. ACL.
* Williams et al. (2017) Jason D Williams, Katvosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, volume 1, pages 665-677.
* Williams and Young (2007) Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. volume 21, pages 393-422. Elsevier.



Two-Level attention on BoSs Memory

To visualize the benefit of two-level attention used on Boss memory by the decoder, we compare attention weights for two models: our proposed _two-level attention_ and a variant with just _one-level attention_ (over all the words in the memory). In the example of a sample dialog from bAbI Task 3, shown in Figure 7, the decoder is aimed at predicting the second best restaurant _3 stars_, given that the restaurant with rating _8 stars_ has already been suggested and rejected. We show attention only on the KB entries for brevity.

The models share some similarities in their distribution of attention. First, the attention weights are localized over the restaurant names, indicating the preference of the system to point to a specific restaurant. This is supported by the \(g_{s}\) values, \(3.14\) x \(10^{-5}\) and \(1.15\) x \(10^{-4}\) for two-level attention and one-level attention respectively, i.e., both models prefer to copy rather than generate. Moreover, entries with the same restaurant name have similar attention weights, reflecting the robustness of the distribution.

We also observe that two-level attention is able to perform the difficult task of _sorting_ the restaurant entries based on decreasing order of rating (number of stars). It gives more weight to entries with a high rating (_3 stars > 2 stars > 1 star_) and suppresses the weights of any previously suggested restaurant.

The attention over memory cells provides BoSsNet with the ability to infer over multiple sets of tuples. The ability to sort the restaurants and reject a previously seen restaurant can be observed by the attention heat map of Memory cells. Attention over tokens on the other hand can push the attention weights towards either the subject or object in the KB tuple, based on the query's request. Thus using both in conjunction helps BosSnet perform significantly better than the baselines and illustrates the importance of the BoSS memory in comparison to a flat memory layout.

## Appendix B Reproducibility

We list out the complete set of hyperparameters used to train BoSsNet for the various datasets in Table 9. Our code will be made publicably accessible for future research purposes. Our trained models and evaluation scripts will also be provided. We will also make our end-to-end reconstructed Camrest dataset along with our whole batch of knowledge adaptability test sets available.

## Appendix C Example Predictions of BoSsNet and Baselines

Examples from SMD is shown in Table 12 respectively. Examples from KA test set with percentage of unseen entities set to 50 from CamRest and SMD are shown in Table 11 and Table 13 respectively. Examples from KA test set with percentage of unseen entities set to 100 from bAbI dialog Task 1 is shown in Table 10.

### Mem2Seq Preprocessing

Mem2Seq paper used the following pre-processing on the data:

1. The subject (restaurant name) and object (rating) positions of the rating KB tuples in bAbI dialogs are flipped, while the order remains the same for other tuples remains the same. This pre-processing is illustrated in Figure 8
2. an extra fact was added to the navigation tasks in In-Car Assistant with all the properties (such as distance, address) combined together as the subject and _poi_ as the object. This preprocessing is illustrated in Figure 9

The pre-processing has major impact on the performance of Mem2Seq, as it can only copy objects of a KB tuple, while the subject and relation can never be copied.

### bAbI Dataset Faults

The KB entities present in validation and non-OOV test sets for task 3 and 4 do not overlap with those in the train set. This effectively means that non-OOV and OOV test conditions are the same for tasks 3 and 4. This explains the low performance of baseline models on task 3 and 4 non-OOV test sets.

## Appendix E AMT Setup

**Response Relevance Test** We show a sample of an Human Intelligence Task (HIT) on Amazon Mechanical Turk in Figure 9(a). We randomize the responses generated by the three baseline models and BoSsNet on the same dialog and ask the user to tick all those response options that seem to capture the relevant information of the given sample response. A total of 200 such annotations were collected for Camrest and SMD each 



**Response Grammar Test** We show a sample of an Human Intelligence Task (HIT) on Amazon Mechanical Turk in Figure 10. We randomize the responses generated by the three baseline models and BoSsNet on the same dialog and ask the user to rate each response based on the grammatical correctness and natural flow of the sentence. The rating ranges from 0-3 where 0 being the worst and 3 being the best. Note) the sentences were not asked to be rated with respect to each other, but instead as individual occurrences. A total of 200 such annotations were collected for Camrest and SMD each.

## Appendix F Multi-Hop vs 1-Hop Encoders

Table 14 shows the performance of bAbI tasks and CamRest on two BoSsNet encoder settings. Multi-hops in encoder helps in bAbI task 3 and 5, as they require inferencing over the KB tuples (sorting restaurants by rating) to recommend a restaurant. We also see substantial improvements on CamRest in both BLEU and entity F1 metric.





\begin{table}
\begin{tabular}{c|l} \hline \hline
**usr-1** & good morning \\
**sys-1** & hello what can i help you with today \\
**usr-2** & can you book a table with _korean_ cuisine for _five_ in _beijing_ \\
**sys-2** & i ’ m on it . which price range are looking for ? \\
**usr-3** & i am looking for a _overpriced_ restaurant \\ \hline
**Gold** & api\_call _korean Beijing five overpriced_ \\ \hline \hline
**Seq2Seq+Copy** & api,call _korean beijing five overpriced_ \\ \hline
**Seq2Seq** & api,call _french london four moderate_ \\ \hline
**Mem2Seq** & api,call _overpriced korean xorean overpriced_ \\ \hline BoSsNet & api,call _korean beijing five overpriced_ \\ \hline \end{tabular}
\end{table}
Table 10: Example from bAbI dialog Task 1 with 100% OOV.

\begin{table}
\begin{tabular}{r||l} \hline \hline  & **KB (restaurant—cuisine--address—phone—postcode)** \\ _prezzo—ethiopian—22.atlantis\_road—01799\_521260—cb30ad_ \\ \hline \hline
**usr-1** & i want a _moderately\_expensive ethiopian_ restauruart in the _south_ side of town \\
**sys-1** & i ’ m sorry . i can not find a _moderately\_expensive ethiopian_ restaurant in the _south_ side of town \\
**usr-2** & i would like to find a _moderately_ priced restaurant in the _west_ part of town \\
**sys-2** & _precizo_ is a moderately priced _ehiropian_ restaurant in the _west_ part of town . there is also _indian_ and \\
**burst-3** & yes can i get their address and postcode ? \\ \hline
**Gold** & of course , _prezzo_ ’ s address is _22.atlantis\_road_ and postcode is _cb30ad_ \\ \hline \hline
**Seq2Seq+Copy** & postcode a is is _01223350688_ UNK there get i for for today \\ \hline
**Seq2Seq** & the address is _83.regent\_street_ and their postcode is _cb58aq_ \\ \hline
**Mem2Seq** & the phone address is _cb21uj_ \\ \hline BoSsNet & the address is _22.altantis\_road at 22.antalins\_road_ the postcode is _cb30ad_ is there anything else i can \\  & help you with \\ \hline \hline \end{tabular}
\end{table}
Table 11: Example from Camrest with 50% OOV. The OOV entities present in the dialog are {_ethiopian_, _22.atlantis\_road_}

\begin{table}
\begin{tabular}{r|l} \hline \hline
**USr-1** & good morning \\
**sys-1** & hello what can i help you with today \\
**usr-2** & can you book a table with _korean_ cuisine for _five_ in _beijing_ \\
**sys-2** & i ’ m on it . which price range are looking for ? \\
**usr-3** & i am looking for a _overpriced_ restaurant \\ \hline
**Gold** & api\_call _korean beijing five overpriced_ \\ \hline \hline
**Seq2Seq+Copy** & api,call _korean beijing five overpriced_ \\ \hline
**Seq2Seq** & api,call _french london four moderate_ \\ \hline
**Mem2Seq** & api,call _overpriced korean overpriced_ \\ \hline BoSsNet & api,call _korean beijing five overpriced_ \\ \hline \hline \end{tabular}
\end{table}
Table 10: Example from bAbI dialog Task 1 with 100% OOV.





\begin{table}
\begin{tabular}{c|l}
**KB (location—day—weather)** \\ _mountain\_view—standay—foggy_ \\ _mouthain\_view—monday—overcast_ \\ _pittsburgh—sunday—overcast_ \\ _pittsburgh—monday—warm_ \\ _downtown\_chicago—sandary–foggy_ \\ _downtown\_chicago—monday—warm_ \\ \hline \end{tabular} \begin{tabular}{r|l} _toky________-_monday_ \\ \end{tabular} 
\begin{tabular}{r|l} _usr_**-1** & is the weather _cloudy_ right now ? \\ _sys-1_ & what city do you want the weather for ? \\ _usr-2_ & _pittsburgh_ \\ \hline
**Gold** & there are no clouds in _pittsburgh_ right now \\ \hline \hline
**Seq2Seq+Copy** & which is no \\ \hline
**Seq2Seq** & it is currently raining in _sen\_fracisco_ right now \\ \hline
**Mem2Seq** & it will not be cloudy in in \\ \hline BoSsNet & it will be _cloudy_ on _sunday_ in _pittsburgh_ \\ \hline \end{tabular}
\end{table}
Table 13: Example from SMD with 50% OOV. The OOV entity present in the dialog is \(\left\{\textit{pittsburgh}\right\}\)


* [10] M. C. D'Auria, R. J. H. Lee, and A. S. K. Sweede, _The Open-source Object 


* [10] M. C. D'Auria, A. G. K. R. and S. J. L. B. (2009) The 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 5th ACM SIGKDD international conference on Multimedia Computing_, pages 467-480. IEEE, 2009.
* [11] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [12] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [13] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [14] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [15] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [16] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [17] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [18] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [19] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [20] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [21] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [22] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [23] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [24] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [25] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [26] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [27] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [28] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [29] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [30] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [31] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [32] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [33] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [34] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [35] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [36] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [37] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [38] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [39] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [40] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [41] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [42] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [43] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [44] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [45] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [46] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [47] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone systems: a survey on the internet. In _Proceedings of the 1st ACM SIGKDD international conference on Multimedia computing_, pages 1-10, 2002. ACM, 2002.
* [48] P. F. H. W. and T. E. Y. (2002) On the 3-D array of mobile phone phone systems: a survey on the internet.

 

* [10] A. K. M. C. R. D. J. van der Meij, and S. G. H. T. W. (2013) The role of the agent's social network in a human world: how to use the social network? In _Proceedings of the 2nd international conference on Social information theory_, pages 45-68. ACM, 2013.
* [11] P. L. B. N. V. O. F. E. M.