End-to-End Memory Network	Since we construct our model based on the MemN2N by  , we first briefly recall its structure to facilitate the delivery of our models. The MemN2N consists of two components: context memory and next response prediction. As the model conducts a conversation with the user, utterance (from the user) and response (from the model) are in turn appended to the memory. At any given time step INLINEFORM0 there are INLINEFORM1 user utterances and INLINEFORM2 model responses. The aim at time INLINEFORM3 is to retrieve the next response INLINEFORM4 . Memory Representation Following  , we represent each utterance as a bag-of-words using the embedding matrix INLINEFORM0 , and the context memory INLINEFORM1 is represented as a vector of utterances as: DISPLAYFORM0  where INLINEFORM0 maps the utterance to a bag of dimension INLINEFORM1 (the vocabulary size), and INLINEFORM2 is a INLINEFORM3 matrix in which INLINEFORM4 is the embedding dimension. So far, information of which speaker spoke an utterance, and at what time during the conversation, are not included in the contents of memory.

Dataset	The personalized bAbI dialog dataset  is a multi-turn dialog corpus extended from the bAbI dialog dataset  . It introduces an additional user profile associated with each dialog and updates the utterances and KB entities to integrate personalized style. Five separate tasks in a restaurant reservation scenario are introduced along with the dataset. Here we briefly introduce them for better understanding of our experiments. More details on the dataset can be found in the work by  . Task 1: Issuing API Calls Users make queries that contain several blanks to fill in. The bot must ask proper questions to fill the missing fields and make the correct API calls. Task 2: Updating API Calls Users may update their request and the bot must change the API call accordingly. Task 3: Displaying Options Given a user request, the KB is queried and the returning facts are added to the dialog history. The bot is supposed to sort the options based on how much users like the restaurant. The bot must be conscious of the user profile and change the sorting strategy accordingly to accomplish this task.

Since there is usually no explicit personalized information in conversation context, existing models  ,  often require extra procedures to incorporate personalization in training.  add intermediate supervision to learn when to employ the user profile.  pre-train the user profile with external service. This work, in contrast, is totally end-to-end. A common approach to leveraging personality in these works is using a conditional language model as the response decoder  ,  . This can help assign personality or language style to chit-chat bots, but it is useless in goal-oriented dialog systems. Instead of assigning personality to agents  ,  ,  , our model pays more attention to the user persona and aims to make agents more adaptive to different kinds of interlocutors. Personalized Goal-Oriented Dialog As most previous works  ,  ,  focus on chit-chat, the combination of personalization and goal-oriented dialog remains unexplored. Recently a new dataset has been released that enriches research resources for personalization in chit-chat  . However, no open dataset allows researchers to train goal-oriented dialog with personalized information, until the personalized bAbI dialog corpus released by  .

Profile Model	Our first model is the Profile Model, which aims to integrate personalized information into the query and ranking part of the MemN2N. The model consists of two different components: profile embedding and global memory. Profile Embedding In the MemN2N, the query INLINEFORM0 plays a key role in both reading memory and choosing the response, while it contains no information about the user. We expect to add a personalized information term to INLINEFORM1 at each iteration of the query. Then, the model can be aware of the user profile in the steps of searching relevant utterances in the memory and selecting the final response from the candidates. We thus obtain a distributed profile representation INLINEFORM2 by applying a linear transformation with the one-hot user profile: INLINEFORM3 , where INLINEFORM4 . Note that this distributed profile representation shares the same embedding dimension INLINEFORM5 with the bag-of-words. The query update equation can be changed as: DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 are the query and output at the INLINEFORM2 -th hop, respectively. Also, the likelihood of a candidate being selected should be affected directly by the user profile, no matter what the query is.

Results	Following  , we report per-response accuracy across all models and tasks on the personalized bAbI dataset in Table TABREF18 . The per-response accuracy counts the percentage of correctly chosen candidates. Rows 4 to 6 of Table TABREF18 show the evaluation results of the Profile Model. As reported in  , their personalized dialogs model might be too complex for some simple tasks (such as Tasks 1 and 2, which do not rely on KB facts) and tends to overfit the training data. It is reflected in the failure of the split memory model on Tasks 1 and 2. Although it outperforms the standard MemN2N in some complicated tasks, the latter one is good enough to capture the profile information given in a simple raw text format, and defeats the split memory model in simpler tasks. To overcome such a challenge, we avoid using excessively complex structures to model the personality. Instead, we only represent the profile as an embedding vector or implicitly. As expected, both profile embedding and global memory approach accomplish Tasks 1 and 2 with a very high accuracy and also notably outperform the baselines in Task 3, which requires utilizing KB facts along with the profile information.

Task 4: Providing Information Users ask for some information about a restaurant, and more than one answer may meet the requirement (i.e., contact with-respect-to social media account and phone number). The bot must infer which answer the user prefers based on the user profile. Task 5: Full Dialog This task conducts full dialog combining all the aspects of Tasks 1 to 4. The difficulties of personalization in these tasks are not incremental. In Tasks 1 and 2, the bot is only required to select responses with appropriate meaning and language style. In Tasks 3 and 4, the knowledge base is supposed to be searched, which makes personalization harder. In these two tasks, apart from capturing shallow personalized features in the utterances such as language style, the bot also has to learn different searching or sorting strategies for different user profiles. In Task 5 we expect an average performance (utterance-wise) since it combines the other four tasks. There are two variations of dataset provided for each task: a full set with around 6000 dialogs and a small set with only 1000 dialogs to create realistic learning conditions. We get the dataset released on ParlAI.

Conclusion and Future Work	We introduce a novel end-to-end model for personalization in goal-oriented dialog. Experiment results on open datasets and further analysis show that the model is capable of overcoming some existing issues in dialog systems. The model improves the effectiveness of the bot responses with personalized information, and thus greatly outperforms state-of-the-art methods. In future work, more representations of personalities apart from the profile attribute can be introduced into goal-oriented dialogs models. Besides, we may explore on learning profile representations for non-domain-specific tasks and consider KB with more complex format such as ontologies.

To address the problem of ambiguity, the Preference Model learns user preferences among ambiguous candidates by building a connection between the user profile and the knowledge base. Since these two models are both under the MemN2N framework and make contributions to personalization in different aspects, we combine them into the Personalized MemN2N. Our experiments on a goal-oriented dialog corpus, the personalized bAbI dialog dataset, show that leveraging personal information can significantly improve the performance of dialog systems. The Personalized MemN2N outperforms current state-of-the-art methods with over 7% improvement in terms of per-response accuracy. A test with real human users also illustrates that the proposed model leads to better outcomes, including higher task completion rate and user satisfaction.

Personalized Dialog System	We first propose two personalized models. The Profile Model introduces the personality of the interlocutor explicitly (using profile embedding) and implicitly (using global memory). The Preference Model models user preferences over knowledge base entities. The two models are independent to each other and we also explore their combination as the Personalized MemN2N. Figure FIGREF8 shows the structure of combined model. The different components are labeled with dashed boxes separately.

Therefore, the memory can be iteratively reread to look for additional pertinent information using the updated query INLINEFORM6 instead of INLINEFORM7 , and in general using INLINEFORM8 on iteration INLINEFORM9 , with a fixed number of iterations INLINEFORM10 (termed INLINEFORM11 hops). Let INLINEFORM0 , where INLINEFORM1 is another word embedding matrix, and INLINEFORM2 is a (large) set of candidate responses which includes all possible bot utterances and API calls. The final predicted response distribution is then defined as: DISPLAYFORM0  where there are INLINEFORM0 candidate responses in INLINEFORM1 .

Also, the performance of combining the two components together, as shown in row 6, is slightly better than using them independently. The result suggests that we can take advantages of using profile information in an explicit and implicit way in the meantime. Since the Profile Model does not build a clear connection between the user and the knowledge base, as discussed in Section SECREF4 , it may not solve ambiguities among the KB columns. The experiment results are consistent with this inference: the performance of the Profile Model on Task 4, which requires user request disambiguation, is particularly close to the baselines. Row 7 shows the evaluation results of the Preference Model, which is proposed to handle the above mentioned challenge. The model achieves significant improvements on Task 4 by introducing the bias term derived from the learned user preference. Besides, the restaurant sorting challenge in Task 3 depends on the properties of a restaurant to some extent. Intuitively, different properties of the restaurants are weighted differently, and the user preference over the KB columns can be considered as scoring weights which is useful for task-solving. As a result, the model also improves the performance in Task 3 compared to the standard MemN2N.

Our work is in the vein of the memory network models for goal-oriented dialog from  and  . We enrich these models by incorporating the profile vector and using conversation context from users with similar attributes as global memory.

Therefore, we obtain tendency weights by computing the inner product between INLINEFORM0 and candidates followed by a sigmoid, and revise the candidates accordingly: DISPLAYFORM0  where INLINEFORM0 is a sigmoid. The prediction INLINEFORM1 is then computed by Equation ( EQREF5 ) using INLINEFORM2 instead of INLINEFORM3 . Global Memory Users with similar profiles may expect the same or a similar response for a certain request. Therefore, instead of using the profile directly, we also implicitly integrate personalized information of an interlocutor by utilizing the conversation history from similar users as a global memory. The definition of similarity varies with task domains. In this paper, we regard those with the same profile as similar users. As shown in Figure FIGREF8 , the global memory component has an identical structure as the original MemN2N. The difference is that the contents in the memory are history utterances from other similar users, instead of the current conversation.

Similarly, we construct the attention weights, output vector, and iteration equation by DISPLAYFORM0   where INLINEFORM0 denotes the global memory, INLINEFORM1 is the attention weight over the global memory, INLINEFORM2 is a INLINEFORM3 square matrix, INLINEFORM4 is the intermediate output vector and INLINEFORM5 is the result at the INLINEFORM6 -th iteration. Lastly, we use INLINEFORM7 instead of INLINEFORM8 to make the following computation.

Related Work	End-to-end neural approaches to building dialog systems have attracted increasing research interest. It is well accepted that conversation agents include goal-oriented dialog systems and non goal-oriented (chit-chat) bots. Generative recurrent models like Seq2Seq have showed promising performance in non goal-oriented chit-chat  ,  ,  . More recently, retrieval-based models using a memory network framework have shown their potential in goal-oriented systems  ,  . Although steady progress has been made, there are still issues to be addressed: most existing models are content-based, which are not aware of the interlocutor profile, and thus are not capable of adapting to different kinds of users. Considerable research efforts have been devoted so far to make conversational agents smarter by incorporating user profile. Personalized Chit-Chat The first attempt to model persona is  , which proposes an approach to assign specific personality and conversation style to agents based on learned persona embeddings.  describe an interesting approach that uses multi-task learning with personalized text data. There are some researchers attempting to introduce personalized information to dialogs by transfer learning  ,  .

Analysis of Preference	Remember that we use a preference vector INLINEFORM0 to represent the user's preference over the columns in the knowledge base. Therefore, we investigate the learned arguments grouped by profile attributes. As seen in Figure FIGREF31 , the model successfully learns the fact that young people prefer social media as their contact information, while middle-aged and elderly people prefer phone number. The result shows great potential and advantage of end-to-end models. They are capable of learning meaningful intermediate arguments while being much simpler than existing reinforcement learning methods and pipeline models for the task of personalization in dialogs.

We therefore encode those pieces of information in the mapping INLINEFORM0 by extending the vocabulary to contain INLINEFORM1 extra “time features” which encode the index INLINEFORM2 of an utterance into the bag-of-words, and two more features (# INLINEFORM3 , # INLINEFORM4 ) encoding whether the speaker is the user or the bot. The last user utterance INLINEFORM0 is encoded into INLINEFORM1 , which also denotes the initial query at time INLINEFORM2 , using the same matrix INLINEFORM3 . Memory Operation The model first reads the memory to find relevant parts of the previous conversation for responses selection. The match between INLINEFORM0 and the memory slots is computed by taking the inner product followed by a softmax: INLINEFORM1 , which yields a vector of attention weights. Subsequently, the output vector is constructed by INLINEFORM2 where INLINEFORM3 is a INLINEFORM4 square matrix. In a multi-layer MemN2N framework, the query is then updated with INLINEFORM5 .

Analysis of Profile Embeddings	Since we use the learned profile embeddings to obtain tendency weights for candidates selection, as is illustrated in Equation ( EQREF10 ), we expect to observe larger weights on candidates that correctly match the profile. For instance, given a profile “Gender: Male, Age: Young”, we can generate a weight for each response candidate. Due to the fact that candidates are collected from dialogs with different users, they can be divided based on the user profile. Those candidates in the group of young male should have larger weights than others. We group the candidates by their corresponding user profile. For each profile, we generate tendency weights and collect the average value for each group. Figure FIGREF27 visualizes the results by a confusion matrix. The weights on the diagonal are significantly larger than others, which demonstrates the contribution of profile embeddings in candidate selection.

Third, the word “contact” can be interpreted into “phone” or “social media” contact information in the knowledge base. Instead of choosing one randomly, the personalized model handles this ambiguity based on the learned fact that young people prefer social media account while the elders prefer phone number. Psychologists have proven that during a dialog humans tend to adapt to their interlocutor to facilitate understanding, which enhances conversational efficiency  ,  ,  . To improve agent intelligence, we may polish our model to learn such human behaviors in conversations. A big challenge in building personalized dialog systems is how to utilize the user profile and generate personalized responses correspondingly. To overcome it, existing works  ,  often conduct extra procedures to incorporate personalization in training, such as intermediate supervision and pre-training of user profiles, which are complex and time-consuming. In contrast, our work is totally end-to-end. In this paper, we propose a Profile Model and a Preference Model to leverage user profiles and preferences. The Profile Model learns user personalities with distributed profile representation, and uses a global memory to store conversation context from other users with similar profiles. In this way, it can choose a proper language style and change recommendation policy based on the user profile.

Introduction	There has been growing research interest in training dialog systems with end-to-end models  ,  ,  in recent years. These models are directly trained on past dialogs, without assumptions on the domain or dialog state structure  . One of their limitations is that they select responses only according to the content of the conversation and are thus incapable of adapting to users with different personalities. Specifically, common issues with such content-based models include: (i) the inability to adjust language style flexibly  ; (ii) the lack of a dynamic conversation policy based on the interlocutor's profile  ; and (iii) the incapability of handling ambiguities in user requests. Figure FIGREF1 illustrates these problems with an example. The conversation happens in a restaurant reservation scenario. First, the responses from the content-based model are plain and boring, and not able to adjust appellations and language styles like the personalized model. Second, in the recommendation phase, the content-based model can only provide candidates in a random order, while a personalized model can change recommendation policy dynamically, and in this case, match the user dietary.

Acknowledgements	We thank all reviewers for providing the constructive suggestions. Also thanks to Danni Liu, Haoyan Liu and Yuanhao Xiong for the helpful discussion and proofreading. Xu Sun is the corresponding author of this paper.

Preference Model	The Profile Model has not yet solved the challenge of handling the ambiguity among KB entities, such as the choice between “phone” and “social media” in Figure FIGREF1 . The ambiguity refers to the user preference when more than one valid entities are available for a specific request. We propose inferring such preference by taking the relation between user profile and knowledge base into account. Assuming we have a knowledge base that describes the details of several items, where each row denotes an item and each column denotes one of their corresponding properties. The entity INLINEFORM0 at row INLINEFORM1 and column INLINEFORM2 is the value of the INLINEFORM3 -th property of item INLINEFORM4 . The Preference Model operates as follows. Given a user profile and a knowledge base with INLINEFORM0 columns, we predict the user's preference on different columns. We first model the user preference INLINEFORM1 as: DISPLAYFORM0  where INLINEFORM0 . Note that we assume the bot cannot provide more than one option in a single response, so a candidate can only contains one entity at most. The probability of choosing a candidate response should be affected by this preference if the response mentions one of the KB entities.

Notation	The user profile representation is defined as follows. Each interlocutor has a user profile represented by INLINEFORM0 attributes INLINEFORM1 , where INLINEFORM2 and INLINEFORM3 denote the key and value of the INLINEFORM4 -th attribute, respectively. Take the user in the first dialog in Figure FIGREF1 as an example, the representation should be INLINEFORM5 . The INLINEFORM6 -th profile attribute is represented as a one-hot vector INLINEFORM7 , where there are INLINEFORM8 possible values for key INLINEFORM9 . We define the user profile INLINEFORM10 as the concatenation of one-hot representations of attributes: INLINEFORM11 , where INLINEFORM12 . The notations of the memory network are the same as introduced in Section SECREF3 .

We add a bias term INLINEFORM0 to revise the logits in Equation ( EQREF5 ). The bias for INLINEFORM1 -th candidate INLINEFORM2 is constructed as the following steps. If the INLINEFORM3 -th candidate contains no entity, then INLINEFORM4 ; if the candidate contains an entity INLINEFORM5 , which belongs to item INLINEFORM6 , then INLINEFORM7 , where given the current conversation context INLINEFORM8 , DISPLAYFORM0  For example, the candidate “Here is the information: The_Place_Phone” contains a KB entity “The_Place_Phone” which belongs to restaurant “The_Place” and column “Phone”. If “The_Place” has been mentioned in the conversation, the bias term for this response should be INLINEFORM0 . We update the Equation ( EQREF5 ) to DISPLAYFORM0

Combined Model	As discussed previously, the Profile Model and the Preference Model make contributions to personalization in different aspects. The Profile Model enables the MemN2N to change the response policy based on the user profile, but fails to establish a clear connection between the user and the knowledge base. On the other hand, the Preference Model bridges this gap by learning the user preferences over the KB entities. To take advantages of both models, we construct a general Personalized MemN2N model by combining them together, as shown in Algorithm SECREF16 . All these models are trained to minimize a standard cross-entropy loss between INLINEFORM0 and the true label INLINEFORM1 . Response Prediction by Personalized MemN2N Input: User utterance INLINEFORM0 , Context memory INLINEFORM1 , global memory INLINEFORM2 , candidates INLINEFORM3 and user profile INLINEFORM4  Output: The index INLINEFORM0 of the next response [1] Predict INLINEFORM1 INLINEFORM2 Profile embedding INLINEFORM3   INLINEFORM0 hops INLINEFORM1 INLINEFORM2 INLINEFORM3   INLINEFORM0 INLINEFORM1 INLINEFORM2   INLINEFORM0 INLINEFORM1 Bias term INLINEFORM2 Final query INLINEFORM3 Revised candidates INLINEFORM4 INLINEFORM5

Baselines	We consider the following baselines: Supervised Embedding Model: a strong baseline for both chit-chat and goal-oriented dialog  ,  . Memory Network: the MemN2N by  , which has been described in detail in Section SECREF3 . We add the profile information as an utterance said by the user at the beginning of each dialog. In this way the standard MemN2N may capture the user persona to some extent. Split Memory Network: the model proposed by  that splits the memory into two parts: profile attributes and conversation history. The various attributes are stored as separate entries in the profile memory before the dialog starts, and the conversation memory operates the same as the MemN2N.

Experiment Settings	The parameters are updated by Nesterov accelerated gradient algorithm  and initialized by Xavier initializer. We try different combinations of hyperparameters and find the best settings as follows. The learning rate is INLINEFORM0 , and the parameter of momentum INLINEFORM1 is INLINEFORM2 . Gradients are clipped to avoid gradient explosion with a threshold of 10. We employ early-stopping as a regularization strategy. Models are trained in mini-batches with a batch size of 64. The dimensionality of word/profile embeddings is 128. We set the maximum context memory and global memory size (i.e. number of utterances) as 250 and 1000, separately. We pad zeros if the number of utterances in a memory is less than 250 or 1000, otherwise we keep the last 250 utterances for the context memory, or randomly choose 1000 valid utterances for the global memory.

We test the performance of the combined Personalized MemN2N as well. As we have analyzed in Section SECREF4 , the Profile Model and the Preference Model make contributions to personalization in different aspects and their combination has the potential to take advantages of both models. Experiment results confirm our hypothesis that the combined model achieves the best performance with over 7% (and 9% on small sets) improvement over the best baseline for the full dialog task (Task 5).

Analysis	As the proposed Personalized MemN2N achieves better performance than previous approaches, we conduct an analysis to gain further insight on how the integration of profile and preference helps the response retrieval.

Analysis of Global Memory	To better illustrate how much the global memory impacts the performance of the proposed model, we conduct a control experiment. Specifically, we build a model with the same global memory component as described in Section SECREF7 , but the utterances in the memory are from randomly chosen users rather than similar users. We report the results of the control experiment on Task 5 in Table TABREF29 . The numbers indicate that the global memory does help improve the performance.

Human Evaluation	To demonstrate the effectiveness of the personalization approach over standard models more convincingly, we build an interactive system based on the proposed model and baselines, and conduct a human evaluation. Since it is impractical to find testers with all profiles we need, we randomly build 20 profiles with different genders, ages and preferences, and ask three judges to act as the given roles. They talk to the system and score the conversations in terms of task completion rate and satisfaction. Task completion rate stands for how much the system accomplish the users' goal. Satisfaction refers to whether the responses are appropriate to the user profile. The scores are averaged and range from 0 to 1 (0 is the worst and 1 is perfect). We find that Personalized MemN2N wins the MemN2N baseline with INLINEFORM0 and INLINEFORM1 higher in terms of task completion rate and satisfaction, respectively, with INLINEFORM2 .

Learning Personalized End-to-End Goal-Oriented Dialog	Most existing works on dialog systems only consider conversation content while neglecting the personality of the user the bot is interacting with, which begets several unsolved issues. In this paper, we present a personalized end-to-end model in an attempt to leverage personalization in goal-oriented dialogs. We first introduce a Profile Model which encodes user profiles into distributed embeddings and refers to conversation history from other similar users. Then a Preference Model captures user preferences over knowledge base entities to handle the ambiguity in user requests. The two models are combined into the Personalized MemN2N. Experiments show that the proposed model achieves qualitative performance improvements over state-of-the-art methods. As for human evaluation, it also outperforms other approaches in terms of task completion rate and user satisfaction.

Question: What datasets did they use?
Answer: the personalized bAbI dialog dataset