Venue: EMNLP 2017

# Composite Task-Completion Dialogue Policy Learning via

Hierarchical Deep Reinforcement Learning

 Baolin Peng\({}^{*}\) Xiujun Li\({}^{\dagger}\) Lihong Li\({}^{\dagger}\) Jianfeng Gao\({}^{\dagger}\) Asli Celikyilmaz\({}^{\dagger}\) Sungjin Lee\({}^{\dagger}\) Kam-Fai Wong\({}^{\star}\)

\({}^{\dagger}\)Microsoft Research, Redmond, WA, USA \({}^{\star}\)The Chinese University of Hong Kong, Hong Kong {blpeng, kfwong}@se.cuhk.edu.hk {xiul,lihongli,jfgao,aslicel,sule}@microsoft.com

###### Abstract

Building a dialogue agent to fulfill complex tasks, such as travel planning, is challenging because the agent has to learn to _collectively_ complete multiple subtasks. For example, the agent needs to reserve a hotel and book a flight so that there leaves enough time for commute between arrival and hotel check-in. This paper addresses this challenge by formulating the task in the mathematical framework of _options_ over Markov Decision Processes (MDPs), and proposing a hierarchical deep reinforcement learning approach to learning a dialogue manager that operates at different temporal scales. The dialogue manager consists of: (1) a top-level dialogue policy that selects among subtasks or options, (2) a low-level dialogue policy that selects primitive actions to complete the subtask given by the top-level policy, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied. Experiments on a travel planning task with simulated and real users show that our approach leads to significant improvements over three baselines, two based on handcrafted rules and the other based on flat deep reinforcement learning.

## 1 Introduction

There is a growing demand for intelligent personal assistants, mainly in the form of dialogue agents, that can help users accomplish tasks ranging from meeting scheduling to vacation planning. However, most of the popular agents in today's market, such as Amazon Echo, Apple Siri, Google Home and Microsoft Cortana, can only handle very simple tasks, such as reporting weather and requesting songs. Building a dialogue agent to fulfill complex tasks remains one of the most fundamental challenges for the NLP community and AI in general.

In this paper, we consider an important type of complex tasks, termed _composite task_, which consists of a set of subtasks that need to be fulfilled collectively. For example, in order to make a travel plan, we need to book air tickets, reserve a hotel, rent a car, etc. in a collective way so as to satisfy a set of cross-subtask constraints, which we call _slot constraints_. Examples of slot constraints for travel planning are: hotel check-in time should be later than the flight's arrival time, hotel check-out time may be earlier than the return flight depart time, the number of flight tickets equals to that of hotel check-in people, and so on.

It is common to learn a task-completion dialogue agent using reinforcement learning (RL); see Su et al. (2016); Cuayahuitl (2017); Williams et al. (2017); Dhingra et al. (2017) and Li et al. (2017a) for a few recent examples. Compared to these dialogue agents developed for individual domains, the composite task presents additional challenges to commonly used, _flat_ RL approaches such as DQN Mnih et al. (2015). The first challenge is reward sparsity. Dialogue policy learning for composite tasks requires exploration in a much larger state-action space, and it often takes many more conversation turns between user and agent to fulfill a task, leading to a much longer trajectory. Thus, the reward signals (usually provided by users at the end of a conversation) are delayed and sparse. As we will show in this paper, typical flat RL methods such as DQN with naive \(\epsilon\)-greedy exploration is rather inefficient. The second challenge is to satisfy slot constraints across subtasks. This requirement makes most of the existing methods of learning _multimomain dialogue_ agents Cuayahuitl (2009); Gasic et al. (2015b) inapplicable: these methods train a

collection of policies, one for each domain, and there is no cross-domain constraints required to successfully complete a dialogue. The third challenge is improved user experience: we find in our experiments that a flat RL agent tends to switch between different subtasks frequently when conversing with users. Such incoherent conversations lead to poor user experience, and are one of the main reasons that cause a dialogue session to fail.

In this paper, we address the above mentioned challenges by formulating the task using the mathematical framework of _options over MDPs_(Sutton et al., 1999), and proposing a method that combines deep reinforcement learning and hierarchical task decomposition to train a composite task-completion dialogue agent. At the heart of the agent is a dialogue manager, which consists of (1) a top-level dialogue policy that selects subtasks (options), (2) a low-level dialogue policy that selects primitive actions to complete a given subtask, and (3) a global state tracker that helps ensure all cross-subtask constraints be satisfied.

Conceptually, our approach exploits the structural information of composite tasks for efficient exploration. Specifically, in order to mitigate the reward sparsity issue, we equip our agent with an evaluation module (internal critic) that gives intrinsic reward signals, indicating how likely a particular subtask is completed based on its current state generated by the global state tracker. Such intrinsic rewards can be viewed as heuristics that encourage the agent to focus on solving a subtask before moving on to another subtask. Our experiments show that such intrinsic rewards can be used inside a hierarchical RL agent to make exploration more efficient, yielding a significantly reduced state-action space for decision making. Furthermore, it leads to a better user experience, as the resulting conversations switch between subtasks less frequently.

To the best of our knowledge, this is the first work that strives to develop a composite task-completion dialogue agent. Our main contributions are three-fold:

* We formulate the problem in the mathematical framework of options over MDPs.
* We propose a hierarchical deep reinforcement learning approach to efficiently learning the dialogue manager that operates at different temporal scales.
* We validate the effectiveness of the proposed approach in a travel planning task on simulated as well as real users.

## 2 Related Work

Task-completion dialogue systems have attracted numerous research efforts. Reinforcement learning algorithms hold the promise for dialogue policy optimization over time with experience (Scheffler and Young, 2000; Levin et al., 2000; Young et al., 2013; Williams et al., 2017). Recent advances in deep learning have inspired many deep reinforcement learning based dialogue systems that eliminate the need for feature engineering (Su et al., 2016; Cuayahuitl, 2017; Williams et al., 2017; Dhingra et al., 2017; Li et al., 2017a).

All the work above focuses on single-domain problems. Extensions to composite-domain dialogue problems are non-trivial due to several reasons: the state and action spaces are much larger, the trajectories are much longer, and in turn reward signals are much more sparse. All these challenges can be addressed by hierarchical reinforcement learning (Sutton et al., 1999, 1998; Singh, 1992; Dietterich, 2000; Barto and Mahadevan, 2003), which decomposes a complicated task into simpler subtasks, possibly in a recursive way. Different frameworks have been proposed, such as Hierarchies of Machines (Parr and Russell, 1997) and MAXQ decomposition (Dietterich, 2000). In this paper, we choose the _options framework_ for its conceptual simplicity and generality (Sutton et al., 1998); more details are found in the next section. Our work is also motivated by hierarchical-DQN (Kulkarni et al., 2016) which integrates hierarchical value functions to operate at different temporal scales. The model achieved superior performance on a complicated ATARI game "Montezuma's Revenge" with a hierarchical structure.

A related but different extension to single-domain dialogues is multi-domain dialogues, where each domain is handled by a separate agent (Lison, 2011; Gasic et al., 2015a,b; Cuayahuitl et al., 2016). In contrast to composite-domain dialogues studied in this paper, a conversation in a multi-domain dialogue normally involves one domain, so completion of a task does _not_ require solving sub-tasks in different domains. Consequently, work on multi-domain dialogues focuses on different technical challenges such as transfer learning across different domains (Gasic 

et al., 2015) and domain selection Cuayahuitl et al. (2016).

## 3 Dialogue Policy Learning

Our composite task-completion dialogue agent consists of four components: (1) an LSTM-based language understanding module HakkaniTur et al. (2016); Yao et al. (2014) for identifying user intents and extracting associated slots; (2) a state tracker for tracking the dialogue state; (3) a dialogue policy which selects the next action based on the current state; and (4) a model-based natural language generator Wen et al. (2015) for converting agent actions to natural language responses. Typically, a dialogue manager contains a state tracker and a dialogue policy. In our implementation, we use a _global_ state tracker to maintain the dialogue state by accumulating information across all subtasks, thus helping ensure all inter-subtask constraints be satisfied. In the rest of this section, we will describe the dialogue policy in details.

### Options over MDPs

Consider the following process of completing a composite task (e.g., travel planning). An agent first selects a subtask (e.g., book-flight-ticket), then takes a sequence of actions to gather related information (e.g., departure time, number of tickets, destination, etc.) until all users' requirements are met and the subtask is completed, and finally chooses the next subtask (e.g., reserve-hotel) to complete. The composite task is fulfilled after all its subtasks are completed collectively. The above process has a natural hierarchy: a top-level process selects which subtasks to complete, and a low-level process chooses primitive actions to complete the selected subtask. Such hierarchical decision making processes can be formulated in the _options_ framework Sutton et al. (1999), where options generalize primitive actions to higher-level actions. Different from the traditional MDP setting where an agent can only choose a primitive action at each time step, with options the agent can choose a "multi-step" action which for example could be a sequence of primitive actions for completing a subtask. As pointed out by Sutton et al. (1999), options are closely related to actions in a family of decision problems known as semi-Markov decision processes.

Following Sutton et al. (1999), an option consists of three components: a set of states where the option can be initiated, an intra-option policy that selects primitive actions while the option is in control, and a termination condition that specifies when the option is completed. For a composite task such as travel planning, subtasks like _book-flight-ticket_ and _reserve-hotel_ can be modeled as options. Consider, for example, the option _book-flight-ticket_: its initiation state set contains states in which the tickets have not been issued or the destination of the trip is long away enough that a flight is needed; it has an intra-option policy for requesting or confirming information regarding departure date and the number of seats, etc.; it also has a termination condition for confirming that all information is gathered and correct so that it is ready to issue the tickets.

### Hierarchical Policy Learning

The intra-option is a conventional policy over primitive actions, we can consider an inter-option

Figure 1: Overview of a composite task-completion dialogue agent.

Figure 2: Illustration of a two-level hierarchical dialogue policy learner.

 

policy over sequences of options in much the same way as we consider the intra-option policy over sequences of actions. We propose a method that combines deep reinforcement learning and hierarchical value functions to learn a composite task-completion dialogue agent as shown in Figure 1. It is a two-level hierarchical reinforcement learning agent that consists of a top-level dialogue policy \(\pi_{g}\) and a low-level dialogue policy \(\pi_{a,g}\), as shown in Figure 2. The top-level policy \(\pi_{g}\) perceives state \(s\) from the environment and selects a subtask \(g\in\mathcal{G}\), where \(\mathcal{G}\) is the set of all possible subtasks. The low-level policy \(\pi_{a,g}\) is shared by all options. It takes as input a state \(s\) and a subtask \(g\), and outputs a primitive action \(a\in\mathcal{A}\), where \(\mathcal{A}\) is the set of primitive actions of all subtasks. The subtask \(g\) remains a constant input to \(\pi_{a,g}\), until a terminal state is reached to terminate \(g\). The internal critic in the dialogue manager provides intrinsic reward \(r^{i}_{t}(g_{t})\), indicating whether the subtask \(g_{t}\) at hand has been solved; this signal is used to optimize \(\pi_{a,g}\). Note that the state \(s\) contains global information, in that it keeps track of information for all subtasks.

Naturally, we aim to optimize the low-level policy \(\pi_{a,g}\) so that it maximizes the following cumulative intrinsic reward at every step \(t\):

where \(r^{i}_{t+k}\) denotes the reward provided by the internal critic at step \(t+k\). Similarly, we want the top-level policy \(\pi_{g}\) to optimize the cumulative extrinsic reward at every step \(t\):

\[\max_{\pi_{g}}\mathbb{E}\Big{[}\sum_{k\geq 0}\gamma^{k}r^{e}_{t+k}\Big{|} s_{t}=s,a_{t+k}=\pi_{g}(s_{t+k})\Big{]}\,,\]

where \(r^{e}_{t+k}\) is the reward received from the environment at step \(t+k\) when a new subtask starts.

Both the top-level and low-level policies can be learned with deep Q-learning methods, like DQN. Specifically, the top-level dialogue policy estimates the optimal Q-function that satisfies the following:

\[Q^{*}_{1}(s,g)=\mathbb{E}\Big{[}\sum_{k=0}^{N-1}\gamma^{k}r^{e}_ {t+k}+\\ \gamma^{N}\cdot\max_{g^{\prime}}Q^{*}_{1}(s_{t+N},g^{\prime})|s_{ t}=s,g_{t}=g\Big{]}, \tag{1}\]

where \(N\) is the number of steps that the low-level dialogue policy (intra-option policy) needs to accomplish the subtask. \(g^{\prime}\) is the agent's next subtask in state \(s_{t+N}\). Similarly, the low-level dialogue policy estimates the Q-function that satisfies the following:

\[Q^{*}_{2}(s,a,g)=\mathbb{E}\Big{[}r^{i}_{t}+\\ \gamma\cdot\max_{a_{t+1}}Q^{*}_{2}(s_{t+1},a_{t+1},g)|s_{t}=s,g_{ t}=g\Big{]}\,.\]

Both \(Q^{*}_{1}(s,g)\) and \(Q^{*}_{2}(s,a,g)\) are represented by neural networks, \(Q_{1}(s,g;\theta_{1})\) and \(Q_{2}(s,a,g;\theta_{2})\), parameterized by \(\theta_{1}\) and \(\theta_{2}\), respectively.

The top-level dialogue policy tries to minimize the following loss function at each iteration \(i\):

\[\mathcal{L}_{1}(\theta_{1,i})=\mathbb{E}_{(s,g,r^{e},s^{\prime}) \sim\mathcal{D}_{1}}[(y_{i}-Q_{1}(s,g;\theta_{1,i}))^{2}]\\ y_{i}=r^{e}+\gamma^{N}\max_{g^{\prime}}Q_{1}(s^{\prime},g^{ \prime},\theta_{1,i-1})\,,\]

where, as in Equation (1), \(r^{e}=\sum_{k=0}^{N-1}\gamma^{k}r^{e}_{t+k}\) is the discounted sum of reward collected when subgoal \(g\) is being completed, and \(N\) is the number of steps \(g\) is completed.

The low-level dialogue policy minimizes the following loss at each iteration \(i\) using:

\[\mathcal{L}_{2}(\theta_{2,i}) = \mathbb{E}_{(s,g,a,r^{i},s^{\prime})\sim\mathcal{D}_{2}}[(y_{i}- \\ Q_{2}(s,g,a;\theta_{2,i}))^{2}]\] \[y_{i} = r^{i}+\gamma\max_{a^{\prime}}Q_{2}(s^{\prime},g,a^{\prime}, \theta_{2,i-1})\,.\]

We use SGD to minimize the above loss functions. The gradient for the top-level dialogue policy yields:

\[\nabla_{\theta_{1,i}} L_{1}(\theta_{1,i})=\mathbb{E}_{(s,g,r^{e},s^{\prime})\sim \mathcal{D}_{1}}[(r^{e}+\\ \gamma^{N}\max_{g^{\prime}}&Q_{2}(s^{\prime},g^{\prime},\theta_{ 1,i-1})-Q_{1}(s,g,\theta_{1,i}))\\ \nabla_{\theta_{1,i}}Q_{1}(s,g,\theta_{1,i})] \tag{2}\]

The gradient for the low-level dialogue policy yields:

\[\nabla_{\theta_{2,i}} L_{2}(\theta_{2,i})=\mathbb{E}_{(s,g,a,r^{i},s^{\prime})\sim \mathcal{D}_{2}}[(r^{i}+\\ \gamma\max_{a^{\prime}}Q_{2}(s^{\prime},g,a^{\prime},\theta_{2, i-1})-Q_{2}(s,g,a,\theta_{2,i}))\] \[\nabla_{\theta_{2,i}} Q_{2}(s,g,a,\theta_{2,i})] \tag{3}\]

Following previous studies, we apply two most commonly used performance boosting methods: target networks and experience replay. Experience replay tuples \((s,g,r^{e},s^{\prime})\) and \((s,g,a,r^{i},s^{\prime})\), are sampled from the experience replay buffers \(\mathcal{D}_{1}\) and \(\mathcal{D}_{2}\) respectively. A detailed summary of the learning algorithm for the hierarchical dialogue policy is provided in Appendix B 

Experiments and Results

To evaluate the proposed method, we conduct experiments on the composite task-completion dialogue task of travel planning.

### Dataset

In the study, we made use of a human-human conversation data derived from a publicly available multi-domain dialogue corpus1 El Asri et al. (2017), which was collected using the Wizard-of-Oz approach. We made a few changes to the schema of the data set for the composite task-completion dialogue setting. Specifically, we added inter-subtask constraints as well as user preferences (soft constraints). The data was mainly used to create simulated users, as will be explained below shortly.

Footnote 1: [https://datasets.maluuba.com/Frames](https://datasets.maluuba.com/Frames)

### Baseline Agents

We benchmark the proposed _HRL agent_ against three baseline agents:

* A _Rule Agent_ uses a sophisticated handcrafted dialogue policy, which requests and informs a hand-picked subset of necessary slots, and then confirms with the user about the reserved tickets.
* A _Rule+ Agent_ requests and informs all the slots in a pre-defined order exhaustedly, and then confirms with the user about the reserved tickets. The average turn of this agent is longer than that of the _Rule_ agent.
* A _flat RL Agent_ is trained with a standard flat deep reinforcement learning method (DQN) which learns a flat dialogue policy using extrinsic rewards only.

### User Simulator

Training reinforcement learners is challenging because they need an environment to interact with. In the dialogue research community, it is common to use simulated users as shown in Figure 3 for this purpose Schatzmann et al. (2007); Asri et al. (2016). In this work, we adapted the publicly-available user simulator, developed by Li et al. (2016), to the composite task-completion dialogue setting using the human-human conversation data described in Section 4.1.2 During training, the simulator provides the agent with an (extrinsic) reward signal at the end of the dialogue. A dialogue is considered to be successful only when a travel plan is made successfully, and the information provided by the agent satisfies user's constraints. At the end of each dialogue, the agent receives a positive reward of \(2*max\_turn\) (\(max\_turn=60\) in our experiments) for success, or a negative reward of \(-max\_turn\) for failure. Furthermore, at each turn, the agent receives a reward of \(-1\) so that shorter dialogue sessions are encouraged.

Footnote 2: A detailed description of the user simulator is presented in Appendix A.

User GoalA user goal is represented by a set of slots, indicating the user's request, requirement and preference. For example, an _inform slot_, such as dst_city="Honolulu", indicates a user requirement, and a _request slot_, such as price="?", indicates a user asking the agent for the information.

In our experiment, we compiled a list of user goals using the slots collected from the human-human conversation data set described in Section 4.1, as follows. We first extracted all the slots that appear in dialogue sessions. If a slot has multiple values, like "or_city=[San Francisco, San Jose]", we consider it as a user preference (soft constraint) which the user may later revise its value to explore different options in the course of the dialogue. If a slot has only one value, we treat it as a user requirement (hard constraint), which is unlikely negotiable. If a slot is with value "?", we treat it as a user request. We removed those slots from user goals if their values do not exist in our database. The compiled set of user goals contains \(759\) entries, each containing slots from at least two subtasks: _book-flight-ticket_ and _reserve-hotel_.

User TypeTo compare different agents' ability to adapt to user preferences, we also constructed three additional user goal sets, representing three different types of (simulated) users, respectively:

* _Type A_: All the informed slots in a user goal have a single value. These users have hard constraints for both the flight and hotel, and have no preference on which subtask to accomplish first.
* _Type B_: At least one of informed slots in the _book-flight-ticket_ subtask can have multiple values, and the user (simulator) prefers to start with the _book-flight-ticket_ subtask. If the user receives "no ticket available" from the agent during the conversation, she is willing to explore alternative slot values 

* _Type C_: Similar to _Type B_, at least one of informed slots of the _reserve-hotel_ subtask in a user goal can have multiple values. The user prefers to start with the _reserve-hotel_ subtask. If the user receives a "no room available" response from the agent, she is willing to explore alternative slot values.

### Implementation

For the RL agent, we set the size of hidden layer to \(80\). For the HRL agent, both top-level and low-level dialogue policies had a hidden layer size of \(80\). RMSprop was applied to optimize the parameters. We set batch size to \(16\). During training, we used the \(\epsilon\)-greedy strategy for exploration. For each simulation epoch, we simulated \(100\) dialogues and stored these state transition tuples in an experience replay buffer. At the end of each simulation epoch, the model was updated with all the transition tuples in the buffer in a batch manner.

The experience replay strategy is critical to the success of deep reinforcement learning. In our experiments, at the beginning, we used a rule-based agent to run \(N\) (\(N=100\)) dialogues to populate the experience replay buffer, which was an implicit way of imitation learning to initialize the RL agent. Then, the RL agent accumulated all the state transition tuples and flushes the replay buffer only when the current RL agent reached a success rate threshold no worse than that of the _Rule_ agent.

This strategy was motivated by the following observation. The initial performance of an RL agent was often not strong enough to result in dialogue sessions with a reasonable success rate. With such data, it was easy for the agent to learn the locally optimal policy that "failed fast"; that is, the policy would finish the dialogue immediately, so that the agent could suffer the least amount of per-turn penalty. Therefore, we provided some rule-based examples that succeeded reasonably often, and did not flush the buffer until the performance of the RL agent reached an acceptable level. Generally, one can set the threshold to be the success rate of the _Rule_ agent. To make a fair comparison, for the same type of users, we used the same _Rule_ agent to initialize both the RL agent and the HRL agent.

### Simulated User Evaluation

On the composite task-completion dialogue task, we compared the HRL agent with the baseline agents in terms of three metrics: success rate3, average rewards, and the average number of turns per dialogue session.

Footnote 3: Success rate is the fraction of dialogues where the tasks are successfully accomplished within the maximum turns.

Figure 4 shows the learning curves of all four agents trained on different types of users. Each learning curve was averaged over \(10\) runs. Table 1 shows the performance on test data. For all types of users, the HRL-based agent yielded more robust dialogue policies outperforming the hand-crafted rule-based agents and flat RL-based agent measured on success rate. It also needed fewer turns per dialogue session to accomplish a task than the rule-based agents and flat RL agent. The results

Figure 3: Illustration of the Composite Task-Completion dialogue System



across all three types of simulated users suggest the following conclusions.

First, he HRL agent significantly outperformed the RL agent. This, to a large degree, was attributed to the use of the hierarchical structure of the proposed agent. Specifically, the top-level dialogue policy selected a subtask for the agent to focus on, one at a time, thus dividing a complex task into a sequence of simpler subtasks. The selected subtasks, combined with the use of intrinsic rewards, alleviated the sparse reward and long-horizon issues, and helped the agent explore more efficiently in the state-action space. As a result, as shown in Figure 4 and Table 1, the performance of the HRL agent on types B and C users (who may need to go back to revise some slots during the dialogue) does not drop much compared to type A users, despite the increased search space in the former. Additionally, we observed a large drop in the performance of the RL Agent due to the increased complexity of the task, which required more dialogue turns and posed a challenge for temporal credit assignment.

Second, the HRL agent learned much faster than the RL agent. The HRL agent could reach the same level of performance with a smaller number of simulation examples than the RL agent,

\begin{table}
\begin{tabular}{c c c c c c c c c} \hline  & \multicolumn{3}{c}{Type A} & \multicolumn{3}{c}{Type B} & Type C \\ \hline Agent & Succ. & Turn & Reward & Succ. & Turn & Reward & Succ. & Turn & Reward \\ \hline \hline Rule & .322 & 46.2 & -24.0 & .240 & 54.2 & -42.9 & .205 & 54.3 & -49.3 \\ _Rule+_ & _.535_ & _82.0_ & _-3.7_ & _.385_ & _110.5_ & _-44.95_ & _.340_ & _108.1_ & _-51.85_ \\ RL & .437 & 45.6 & -3.3 & .340 & 52.2 & -23.8 & .348 & 49.5 & -21.1 \\ HRL & **.632** & **43.0** & **33.2** & **.600** & **44.5** & **26.7** & **.622** & **42.7** & **31.7** \\ \hline \end{tabular}
\end{table}
Table 1: Performance of three agents on different User Types. Tested on 2000 dialogues using the best model during training. Succ.: success rate, Turn: average turns, Reward: average reward.

Figure 4: Learning curves of dialogue policies for different User Types under simulation

Figure 5: Performance of HRL agent versus RL agent tested with real users: success rate, number of tested dialogues and p-values are indicated on each bar; the rightmost green ones are for total (difference in mean is significant with \(p<\) 0.01).

Figure 6: Distribution of user ratings for HRL agent versus RL agent, and total.

 

demonstrating that the hierarchical dialogue policies were more sample-efficient than flat RL policy and could significantly reduce the sample complexity on complex tasks.

Finally, we also found that the _Rule+_ and _flat_ RL agents had comparable success rates, as shown in Figure 4. However, a closer look at the correlation between success rate and the average number of turns in Table 1 suggests that the _Rule+_ agent required more turns which adversely affects its success, whilst the _flat_ RL agent achieves similar success with much less number of turns in all the user types. It suffices to say that our hierarchical RL agent outperforms all in terms of success rate as depicted in Figure 4.

### Human Evaluation

We further evaluated the agents, which were trained on simulated users, against real human users, recruited from the authors' affiliation. We conducted the study using the HRL and RL agents, each tested against two types of users: _Type A_ users who had no preference for subtask, and _Type B_ users who preferred to complete the _book-flight-ticket_ subtask first. Note that _Type C_ users were symmetric to Type B ones, so were not included in the study. We compared two (agent, user type) pairs: {RL A, HRL A} and {RL B, HRL B}; in other words, four agents were trained against their specific user types. In each dialogue session, one of the agents was randomly picked to converse with a user. The user was presented with a user

\begin{table}
\begin{tabular}{l l|l} \hline \multicolumn{3}{c}{User Goal} \\ _reserve-hotel_ subtask: & & _book-flight-ticket_ subtask: \\ { \{ & “request\_slots”: \{ & ”request\_slots”: \{ & ”inform\_slots”: \{ \\ “hotel\_price": ‘my”: & **“hotel\_city”: “Cancum”**, & “price”: \textbackslash*?’: & “or\_city”: “Campinas”, \\ “hotel\_date\_checkcut”: “my” & **“hotel\_numberofpeople”: “y”**, “hotel\_name”: “?”** & “return\_time\_dep”: “my” & “**dt\_city”: “Cancum”, \\ “hotel\_name”: “?” & **“hotel\_date\_checkchin”: “09/20”** & “return\_date\_deep”: “my” & “**numberofpeople\^{}; “x3”, \\ \} & \} & \multirow{2}{*}{} & \multirow{2}{*}{} \\  & & & \\ \cline{5-6} \end{tabular}
\end{table}
Table 2: Sample dialogue by RL and HRL agents with real user: Left column shows the dialogue with the RL agent; Right column shows the dialogue with the HRL agent; bolded slots are the joint constraints between two subtasks.

 

goal sampled from our corpus, and was instructed to converse with the agent to complete the task. If one of the slots in the goal had multiple values, the user had multiple choices for this slot and might revise the slot value when the agent replied with a message like "No ticket is available" during the conversation. At the end of each session, the user was asked to give a rating on a scale from \(1\) to \(5\) based on the naturalness and coherence of the dialogue. (\(1\) is the worst rating, and \(5\) the best). We collected a total of \(225\) dialogue sessions from \(12\) human users.

Figure 5 presents the performance of these agents against real users in terms of success rate. Figure 6 shows the comparison in user rating. For all the cases, the HRL agent was consistently better than the RL agent in terms of success rate and user rating. Table 2 shows a sample dialogue session. We see that the HRL agent produced a more coherent conversation, as it switched among subtasks much less frequently than the _flat_ RL agent.

## 5 Discussion and Conclusions

This paper considers composite task-completion dialogues, where a set of subtasks need to be fulfilled collectively for the entire dialogue to be successful. We formulate the policy learning problem using the options framework, and take a hierarchical deep RL approach to optimizing the policy. Our experiments, both on simulated and real users, show that the hierarchical RL agent significantly outperforms a flat RL agent and rule-based agents. The hierarchical structure of the agent also improves the coherence of the dialogue flow.

The promising results suggest several directions for future research. First, the hierarchical RL approach demonstrates strong adaptation ability to tailor the dialogue policy to different types of users. This motivates us to systematically investigate its use for dialogue _personalization_. Second, our hierarchical RL agent is implemented using a two-level dialogue policy. But more complex tasks might require multiple levels of hierarchy. Thus, it is valuable to extend our approach to handle such deep hierarchies, where a subtask can invoke another subtask and so on, taking full advantage of the options framework. Finally, designing task hierarchies requires substantial domain knowledge and is time-consuming. This challenge calls for future work on automatic learning of hierarchies for complex dialogue tasks.

## Acknowledgments

Baolin Peng was in part supported by General Research Fund of Hong Kong (14232816). We would like thank anonymous reviewers for their insightful comments.

## References

* El Asri et al. (2016) Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. In _Interspeech 2016_, pages 1151-1155. ISCA. Barto and Mahadevan (2003) Andrew G. Barto and Sridhar Mahadevan. 2003. Recent advances in hierarchical reinforcement learning. _Discrete Event Dynamic Systems_, 13(1-2):41-77.
* Cuayahuitl (2009) Heriberto Cuayahuitl. 2009. Hierarchical reinforcement learning for spoken dialogue systems.
* Cuayakuitl (2017) Heriberto Cuayahuitl. 2017. SimpleDS: A simple deep reinforcement learning dialogue system. In _Dialogues with Social Robots_, pages 109-118. Springer.
* Cuayahuitl et al. (2016) Heriberto Cuayahuitl, Seunghak Yu, Ashley Williamson, and Jacob Carse. 2016. Deep reinforcement learning for multi-domain dialogue systems. _arXiv preprint arXiv:1611.08675_.
* Dhingra et al. (2017) Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2017. End-to-end reinforcement learning of dialogue agents for information access. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Dietterich (2000) Thomas G. Dietterich. 2000. Hierarchical reinforcement learning with the MAXQ value function decomposition. _Journal of Artificial Intelligence Research_, 13:227-303.
* EAsri et al. (2017) Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaeber Suleman. 2017. Frames: A corpus for adding memory to goal-oriented dialogue systems. _arXiv preprint arXiv:1704.00057_.
* Gasic et al. (2015a) Milica Gasic, Dongho Kim, Pirros Tsiakoulis, and Steve J. Young. 2015a. Distributed dialogue policies for multi-domain statistical dialogue management. In _2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19-24, 2015_, pages 5371-5375. IEEE.
* Gasic et al. (2015b) Milica Gasic, Nikola Mrksic, Pei-hao Su, David Vandyke, Tsung-Hsien Wen, and Steve J. Young. 2015b. Policy committee for adaptation in multi-domain spoken dialogue systems. In _2015 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2015, Scottsdale, AZ, USA, December 13-17, 2015_, pages 806-812. IEEE 

Dilek Hakkani-Tur, Gokhan Tur, Asli Celikyilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Yei Wang. 2016. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM. In _Proceedings of The 17th Annual Meeting of the International Speech Communication Association_.
* Kulkarni et al. (2016) Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. 2016. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In _Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain_, pages 3675-3683.
* Levin et al. (2000) Esther Levin, Roberto Pieraccini, and Wieland Eckert. 2000. A stochastic model of human-machine interaction for learning dialog strategies. _IEEE Trans. Speech and Audio Processing_, 8(1):11-23.
* Li et al. (2017a) Xiujun Li, Yun-Nung Chen, Lihong Li, and Jianfeng Gao. 2017a. End-to-end task-completion neural dialogue systems. _arXiv preprint arXiv:1703.01008_.
* Li et al. (2017b) Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017b. Investigation of language understanding impact for reinforcement learning based dialogue systems. _arXiv preprint arXiv:1703.07055_.
* Li et al. (2016) Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016. A user simulator for task-completion dialogues. _arXiv preprint arXiv:1612.05688_.
* Lison (2011) Pierre Lison. 2011. Multi-policy dialogue management. In _Proceedings of the SIGDIAL 2011_, pages 294-300. The Association for Computer Linguistics.
* Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level control through deep reinforcement learning. _Nature_, 518(7540):529-533.
* Parr and Russell (1997) Ronald Parr and Stuart J. Russell. 1997. Reinforcement learning with hierarchies of machines. In _Advances in Neural Information Processing Systems 10, [NIPS Conference, Denver, Colorado, USA, 1997]_, pages 1043-1049. The MIT Press.
* Schatzmann et al. (2007) Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve J. Young. 2007. Agenda-based user simulation for bootstrapping a POMDP dialogue system. In _Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, April 22-27, 2007, Rochester, New York, USA_, pages 149-152. The Association for Computational Linguistics.
* Shatzmann and Young (2009) Jost Shatzmann and Steve Young. 2009. The hidden agenda user simulation model. _IEEE Transactions on Audio, Speech, and Language Processing_, 17(4):733-747.
* Scheffler and Young (2000) Konrad Scheffler and Steve J. Young. 2000. Probabilistic simulation of human-machine dialogues. In _IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2000, 5-9 June, 2000, Hilton Hotel and Convention Center, Istanbul, Turkey_, pages 1217-1220. IEEE.
* Singh (1992) Satinder P. Singh. 1992. Reinforcement learning with a hierarchy of abstract models. In _Proceedings of the 10th National Conference on Artificial Intelligence. San Jose, CA, July 12-16, 1992._, pages 202-207. AAAI Press / The MIT Press.
* Su et al. (2016) Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsungs-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. _arXiv preprint arXiv:1606.02689_.
* Sutton et al. (1998) Richard S. Sutton, Doina Precup, and Satinder P. Singh. 1998. Intra-option learning about temporally abstract actions. In _Proceedings of the Fifteenth International Conference on Machine Learning (ICML 1998), Madison, Wisconsin, USA, July 24-27, 1998_, pages 556-564.
* Sutton et al. (1999) Richard S. Sutton, Doina Precup, and Satinder P. Singh. 1999. Between MDPs and Semi-MDPs: A framework for temporal abstraction in reinforcement learning. _Artificial Intelligence_, 112(1-2):181-211.
* Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Petihao Su, David Vandyke, and Steve J. Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 1711-1721.
* Williams et al. (2017) Jason D Williams, Katvosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: Practical and efficient end-to-end dialog control with supervised and reinforcement learning. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_.
* Yao et al. (2014) Kaishep Yao, Baolin Peng, Yu Zhang, Dong Yu, Geoffrey Zweig, and Yangyang Shi. 2014. Spoken language understanding using long short-term memory neural networks. In _2014 IEEE Spoken Language Technology Workshop, SLT 2014, South Lake Tahoe, NV, USA, December 7-10, 2014_, pages 189-194.
* Young et al. (2013) Steve J. Young, Milica Gasic, Blaise Thomson, and Jason D. Williams. 2013. POMDP-based statistical spoken dialog systems: A review. _Proceedings of the IEEE_, 101(5):1160-1179