SlowGuess/ACL_2020_OCR huggingface dataset:
Ryuichi Takanobu, Runze Liang, Minlie Huang*
Institute for AI, BNRist, DCST, Tsinghua University, Beijing, China
{gxly19, liangrz15}@mails.tsinghua.edu.cn, aihuang@tsinghua.edu.cn
Many studies have applied reinforcement learning to train a dialog policy and show great promise these years. One common approach is to employ a user simulator to obtain a large number of simulated user experiences for reinforcement learning algorithms. However, modeling a realistic user simulator is challenging. A rule-based simulator requires heavy domain expertise for complex tasks, and a data-driven simulator requires considerable data and it is even unclear how to evaluate a simulator. To avoid explicitly building a user simulator beforehand, we propose Multi-Agent Dialog Policy Learning, which regards both the system and the user as the dialog agents. Two agents interact with each other and are jointly learned simultaneously. The method uses the actor-critic framework to facilitate pretraining and improve scalability. We also propose Hybrid Value Network for the role-aware reward decomposition to integrate role-specific domain knowledge of each agent in task-oriented dialog. Results show that our method can successfully build a system policy and a user policy simultaneously, and two agents can achieve a high task success rate through conversational interaction.
Dialog policy, which decides the next action that the dialog agent should take, plays a vital role in a task-oriented dialog system. More recently, dialog policy learning has been widely formulated as a Reinforcement Learning (RL) problem (Su et al., 2016; Peng et al., 2017; He et al., 2018; Zhao et al., 2019; Zhang et al., 2019; Takanobu et al., 2019), which models users as the interactive environment. Since RL requires much interaction for training, it is too time-consuming and costly to interact with real users directly. The most common way is first
to develop a dialog agent with a user simulator that mimics human behaviors in an offline scenario.
Designing a reliable user simulator, however, is not trivial and often challenging as it is equivalent to building a good dialog agent. With the growing needs for the dialog system to handle more complex tasks, it will be much challenging and laborious to build a fully rule-based user simulator, which requires heavy domain expertise. Data-driven user simulators have been proposed in recent studies (Kreyssig et al., 2018; Shi et al., 2019), but they require a considerable quantity of manually labeled data, most of which regard the simulator as a stationary environment. Furthermore, there is no standard automatic metric for evaluating these user simulators, as it is unclear to define how closely the simulator resembles real user behaviors.
In this paper, we propose Multi-Agent Dialog Policy Learning (MADPL), where the user is regarded as another dialog agent rather than a user simulator. The conversation between the user and the system is modeled as a cooperative interactive process where the system agent and the user agent are trained simultaneously. Two dialog agents interact with each other and collaborate to achieve the goal so that they require no explicit domain expertise, which helps develop a dialog system without the need of a well-built user simulator. Different from existing methods (Georgila et al., 2014; Papangelis et al., 2019), our approach is based on actor-critic framework (Barto et al., 1983) in order to facilitate pretraining and bootstrap the RL training. Following the paradigm of centralized training with decentralized execution (CTDE) (Bernstein et al., 2002) in multi-agent RL (MARL), the actor selects its action conditioned only on its local state-action history, while the critic is trained with the actions of all agents.
It should be noted that the roles of two agents are different though they interact with each other
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 625-638 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: The user has his/her own goal to be accomplished and the system is provided with an interface to access an external database. Both agents can only obtain information from the other side via communication.
in a cooperative setting. As shown in Fig. 1, only the user agent knows the user goal, while only the system agent can access the backend database. The user agent should express the requirements completely in an organized way, and the system should respond with useful information accurately and immediately. So it is inappropriate to apply simple self-play RL (Silver et al., 2017; Lewis et al., 2017) that views two agents as the same agent in this task. To address this issue, the system and the user are viewed as two asymmetric agents in MADPL. We introduce Hybrid Value Network (HVN) for role-aware reward decomposition. It decomposes the reward into two parts: one is the role-specific reward that focuses on its local target, and the other is the global reward that represents the shared goal.
To evaluate the proposed approach, we conduct our experiments on a multi-domain, multi-intent task-oriented dialog corpus, MultiWOZ (Budzianowski et al., 2018). The corpus involves high dimensional state and action spaces, multiple decision making in one turn, which makes it more difficult to get a good system policy as well as a good user policy. The experiments demonstrate that MADPL can successfully build a system policy as well as a user policy with the aid of HVN, and two agents can achieve high task success rate in complex tasks by interacting with each other as well as with benchmark policies.
To summarize, our contributions are in three folds:
- We apply actor-critic based multi-agent reinforcement learning to learn the task-oriented dialog policy to facilitate pretraining and
avoid explicitly building a user simulator.
- We propose Hybrid Value Network for reward decomposition to deal with the asymmetric role issue between the system agent and the user agent in the task-oriented dialog.
- We conduct in-depth experiments on the multi-domain, multi-intent task-oriented dialog corpus to show the effectiveness, reasonableness and scalability of our algorithm.
2.1 Multi-Agent Reinforcement Learning
The goal of RL is to discover the optimal strategy  $\pi^{*}(a|s)$  of the Markov Decision Process, which can be extended into the  $N$ -agent setting, where each agent has its own set of states  $S_{i}$  and actions  $\mathcal{A}_i$ . In MARL, the state transition  $s = (s_1,\dots ,s_N)\to$ $s^{\prime} = (s_1^{\prime},\ldots ,s_N^{\prime})$  depends on the actions taken by all agents  $(a_{1},\ldots ,a_{N})$  according to each agent's policy  $\pi_i(a_i|s_i)$  where  $s_i\in S_i$ ,  $a_{i}\in \mathcal{A}_{i}$ , and similar to single RL, each agent aims to maximize its local total discounted return  $R_{i} = \sum_{t}\gamma^{t}r_{i,t}$ .
Since two or more agents learn simultaneously, the agents continuously change as the training proceeds, therefore the environment is no longer stationary. Many MARL algorithms (Lowe et al., 2017; Foerster et al., 2018; Rashid et al., 2018) have been proposed to solve challenging problems. Most of them use the CTDE framework to address the non-stationarity of co-adapting agents. It allows the policies to use extra information to ease training, but the learned policies can only use local information (i.e. their own observations) at execution time.
Several studies have demonstrated that applying MARL delivers promising results in NLP tasks these years. While some methods use identical rewards for all agents (Das et al., 2017; Kottur et al., 2017; Feng et al., 2018), other studies use completely separate rewards (Georgila et al., 2014; Papangelis et al., 2019). MADPL integrates two types of rewards by role-aware reward decomposition to train a better dialog policy in task-oriented dialog.
2.2 User Modeling in Task-Oriented Dialog
User modeling is essential for training RL-based dialog models, because a large amount of dialog samples are required for RL policy learning, mak
ing it impractical to learn with real users directly from the beginning.
There are three main approaches for user modeling. The first approach is to build a rule-based user simulator. Among these methods, the most popular one is agenda-based simulator (Schatzmann et al., 2007; Shah et al., 2018), which is built on hand-crafted rules with a stack-like agenda based on the user goal. The second approach is to build a user simulator from the dialog data (Keizer et al., 2010; El Asri et al., 2016; Kreyssig et al., 2018). Recently, Gur et al. (2018) uses a variational hierarchical seq2seq framework to encode user goal and system turns, and then generate the user response. Shi et al. (2019) uses two decoders with a copy and attention mechanism to predict a belief span first and then decode user utterance. The third approach is to use model-based policy optimization that incorporates a differentiable model of the world dynamics and assumptions about the interactions between users and systems (Su et al., 2018; Zhang et al., 2019), but this approach still requires real users or a user simulator for world model learning.
Instead of employing a user simulator, a few methods jointly learn two agents directly from the corpus. Liu and Lane (2017) models the system and the user by iteratively training two policies. Papangelis et al. (2019) make the first attempt to apply MARL into the task-oriented dialog policy, whose algorithm is based on Q-learning for mixed policies. However, it is not well scalable to complex tasks such as multi-domain dialog. Therefore, MADPL uses the actor-critic framework instead to deal with the large discrete action space in dialog.
3 Multi-Agent Dialog Policy Learning
We first formally describe the task, and then present the overview of our proposed model. Specifically, given a user goal  $\mathsf{G} = (\mathsf{C},\mathsf{R})$  composed of the user constraints  $\mathsf{C}$  (e.g. a Japanese restaurant in the center of the city) and requests  $\mathsf{R}$  (e.g. inquiry for address, phone number of a hotel), and given an external database DB containing all candidate entities and corresponding information, the user agent and system agent interact with each other in a dialog session to fulfill the user goal. There can be multiple domains in  $\mathsf{G}$ , and two agents have to accomplish all the subtasks in each domain. Both agents can partially observe the environment, i.e. only the user agent knows  $\mathsf{G}$ , while only the sys
Figure 2: Architecture of MADPL. HVN consists of three critics. Each critic estimates its return based on role-aware reward decomposition, and each actor uses the estimated value to optimize itself.
tem agent can access DB, and the only way to know each other's information is through conversational interaction. Different from ordinary multiagent task setting, two agents in dialog are executed asynchronously. In a single dialog turn, the user agent posts an inquiry first, then the system agent returns a response, and the two communicate alternately. Therefore, each dialog session  $\tau$  can be seen as a trajectory of state-action pairs  $\{(s_0^U,a_0^U,s_0^S,a_0^S);(s_1^U,a_1^U,s_1^S,a_1^S);\ldots \}$ , where the user agent and the system agent make decisions according to each dialog policy  $\mu (a^{U}|s^{U}),\pi (a^{S}|s^{S})$  respectively.
Here we present a novel algorithm, Multi-Agent Dialog Policy Learning (MADPL), as shown in Fig. 2, which can be naturally formulated as a MARL problem. Two agents interact through dialog acts following (Georgila et al., 2014). We choose the actor-critic framework in order to learn an explicitly stochastic dialog policy (actor) for high scalability along with an estimated value function (critic) to bootstrap RL training. Besides, this can facilitate imitation learning to pretrain the dialog policy using human-human dialogs. Since two agents cooperate to reach success, yet their roles are asymmetric in the dialog, we propose Hybrid Value Network (HVN) to decompose the task reward into different parts for better policy learning. Note that our approach is fully data-driven without building a user simulator beforehand, and does not need any other human supervision during training.
In the subsequent subsections, we will first explain the state and action used in two dialog policies. Then we describe how we decompose the reward and the proposed HVN. At last, we present model optimization.
3.1 Dialog Policy
System Policy The system policy  $\pi$  decides the system action  $a^S$  according to the system dialog state  $s^S$  to give the appropriate response to user agent. Each system action  $a^S$  is a subset of dialog act set  $\mathcal{A}$  as there may be multiple intents in one dialog turn. A dialog act is an abstract representation of an intention (Stolcke et al., 2000), which can be represented in a quadruple composed of domain, intent, slot type and slot value (e.g. [restaurant, inform, food, Italian]). In practice, dialog acts are delexicalized in the dialog policy. We replace the slot value with a count placeholder and refill it with the true value according to the entity selected from the external database DB, which allows the system to operate on unseen values. The system dialog state  $s_t^S$  at dialog turn  $t$  is the concatenation of (I) user action at current turn  $a_t^U$ ; (II) system action at the last turn  $a_{t-1}^U$ ; (III) the belief state  $b_t$  (Williams et al., 2016) that keeps track of constraint slots and request slots supplied by the user agent; and (IV) embedding vectors of the number of query results  $q_t$  from DB.
User Policy The user policy  $\mu$  decides the user action  $a^U$  according to the user dialog state  $s^U$  to express its constraint and request to the system agent. Similar to the system policy, the user policy uses delexicalized dialog acts as actions, and the value is refilled according to the user goal  $\mathbb{G}$ . User dialog state  $s_t^U$  is the concatenation of (I) last system action  $a_{t - 1}^{S}$ ; (II) last user action  $a_{t - 1}^U$ ; (III) the goal state  $g_{t}$  that represents the remained constraint and request that need to send; (IV) inconsistency vector  $c_{t}$  (Kreyssig et al., 2018) that indicates the inconsistency between the systems response and user constraint  $\mathbb{C}$ . In addition to predicting dialog acts, the user policy outputs terminal signal  $T$  at the same time, i.e.  $\mu = \mu (a^{U},T|s^{U})$ .
3.2 Reward Decomposition
On the one hand, the roles between the user agent and the system agent are different. The user agent actively initiates a task and may change it during conversation, but the system agent passively responds to the user agent and returns the proper information, so the reward should be considered separately for each agent. On the other hand, two agents communicate and collaborate to accomplish the same task cooperatively, so the reward also involves a global target for both agents. Therefore, we decompose the mixed reward into three parts
according to the characteristic of each component. The reward of each part is explained as follows:
System Reward  $r_t^S$  consists of (I) empty dialog act penalty  $a_t^S = \varnothing$ ; (II) late answer penalty if there is a request slot triggered but the system agent does not reply the information immediately; and (III) task success reward based on the user agent's description.
User Reward  $r_t^U$  consists of (I) empty dialog act penalty  $a_t^U = \varnothing$ ; (II) early request penalty if the user agent requests for information when there is still a constraint slot remained to inform; and (III) user goal reward whether the user agents have expressed all the constraints C and requests R.
Global Reward  $r_t^G$  consists of (I) efficiency penalty that a small negative value will be given at each dialog turn; (II) sub-goal completion reward once the subtask of G in a particular domain is accomplished; and (III) task success reward based on user goal G.
Obviously, each agent should obtain its local reward, and both agents should receive the global reward during the training process. Note that the task success and the user goal reward are only computed at the end of the dialog, and the task success computed in the system reward differs from the one in the global reward.
3.3 Hybrid Value Network
The value function aims to estimate the expected return given the current state  $V(s_{t}) = \mathbb{E}[R_{t}] = \mathbb{E}[\sum_{t^{\prime}\geq t}\gamma^{t^{\prime} - t}r_{t^{\prime}}]$  so that the policy can directly use the estimated cumulative reward for optimization, without sampling the trajectories to obtain rewards which may cause high variance. Another advantage by applying actor-critic approaches in MARL is that it can integrate with the CTDE framework: the actor of each agent benefits from a critic that is augmented with additional information about the policies of other agents during training. However, a simple centralized critic conditioned on the global state and joint actions cannot well exploit the domain knowledge mentioned above since each part of the overall rewards only depends on a subset of features, e.g. the system reward only depends on the system agent's behaviors.
Inspired by Hybrid Reward Architecture (Van Seijen et al., 2017) that learns a separate Q function, we propose Hybrid Value Network to improve an estimate of the optimal role-aware
value function. It first encodes the dialog state of each agent to learn a state representation
$$
h _ {s} ^ {S} = \tanh  (f _ {s} ^ {S} (s ^ {S})),
$$
$$
h _ {s} ^ {U} = \tanh (f _ {s} ^ {U} (s ^ {U})),
$$
where  $f(\cdot)$  can be any neural network unit. The value network  $V$  is separated into three branches  $V^S, V^U$  and  $V^G$  for the value of system rewards, user rewards and global rewards, respectively.
$$
V ^ {S} (s ^ {S}) = f _ {S} (h _ {s} ^ {S}),
$$
$$
V ^ {U} (s ^ {U}) = f _ {U} (h _ {s} ^ {U}),
$$
$$
V ^ {G} (s) = f _ {G} ([ h _ {s} ^ {S}; h _ {s} ^ {U} ]).
$$
The action space for the policies can be very large since we deal with multi-domain, complex dialog tasks, which makes it almost impossible for the RL policies to explore and learn from scratch. So the training process can be split into two stages (Fatemi et al., 2016; Takanobu et al., 2019): pretraining the dialog policy with the conversational corpus first and then using RL to improve the pretrained policies. We use  $\beta$ -weighted logistic regression for policy pretraining here to alleviate data bias because each agent only generates several dialog acts in one dialog turn
$$
\begin{array}{l} L (X, Y; \beta) = - [ \beta \cdot Y ^ {T} \log \sigma (X) \tag {1} \\ + \left(I - Y\right) ^ {T} \log (I - \sigma (X)) ], \\ \end{array}
$$
where  $X$  is the state and  $Y$  is the action from the corpus in this task.
As for critic optimization, it aims to minimize the squared error between the temporal difference (TD) target  $r_t + \gamma V(s_{t+1})$  and the estimated value  $V(s_t) = \mathbb{E}[r_t + \gamma V(s_{t+1})]$ . Actor-critic algorithms have high variance since the critic is updated too frequently, which has contributed to severe changes in the estimated value, particularly in multi-agent tasks. So we introduce a target network (Mnih et al., 2015) to make the training process more stable. In the context of HVN, it aims to minimize the following loss functions:
$$
\begin{array}{l} L _ {V} ^ {S} (\theta) = \left(r ^ {S} + \gamma V _ {\theta^ {-}} ^ {S} \left(s ^ {\prime S}\right) - V _ {\theta} ^ {S} \left(s ^ {S}\right)\right) ^ {2}, \\ L _ {V} ^ {U} (\theta) = \left(r ^ {U} + \gamma V _ {\theta^ {-}} ^ {U} \left(s ^ {\prime U}\right) - V _ {\theta} ^ {U} \left(s ^ {U}\right)\right) ^ {2}, \\ L _ {V} ^ {G} (\theta) = (r ^ {G} + \gamma V _ {\theta^ {-}} ^ {G} (s ^ {\prime}) - V _ {\theta} ^ {G} (s)) ^ {2}, \\ L _ {V} = L _ {V} ^ {S} + L _ {V} ^ {U} + L _ {V} ^ {G}, \tag {2} \\ \end{array}
$$
Algorithm 1: Multi-Agent Dialog Policy Learning
Require:Dialog corpus  $\mathcal{D}$  with annotations of dialog acts  $\{a\}$
1 Initialize weights  $\phi, \omega$  for system policy  $\pi$  and user policy  $\mu$  respectively
2 Pretrain policies  $\pi, \mu$  on human conversational data  $\mathcal{D}$  using Eq. 1
3 Initialize weights  $\theta$  for hybrid value network  $V = (V^{S},V^{U},V^{G})$  and target network  $\theta^{-}\gets \theta$
4 foreach training iteration do
where  $\mathrm{HVN}V_{\theta}$  is parameterized by  $\theta$ , and  $\theta^{-}$  is the weight of target network, and the overall loss  $L_{V}$  is the sum of value estimation loss on each component reward.
Each dialog policy aims to maximize all the related returns, e.g. the system policy  $\pi$  aims to maximize the cumulative system rewards and global rewards  $\mathbb{E}[\sum_t\gamma^t (r_t^S +r_t^G)]$ . The advantage  $A(s) = r + \gamma V(s^{\prime}) - V(s)$  estimated by the critic can evaluate the new state  $s^\prime$  and current state  $s$  to determine whether the dialog has become better or worse than expected. With the aid of HVN, the sum of the related component advantages can be used to update different agents. By using the log-likelihood ratio trick, the gradients for the system policy and the user policy yield:
$$
\nabla_ {\phi} J _ {\pi} (\phi) = \nabla_ {\phi} \log \pi_ {\phi} (a ^ {S} | s ^ {S}) [ A ^ {S} (s ^ {S}) + A ^ {G} (s) ], \tag {3}
$$
$$
\nabla_ {\omega} J _ {\mu} (\omega) = \nabla_ {\omega} \log \mu_ {\omega} (a ^ {U} | s ^ {U}) [ A ^ {U} (s ^ {U}) + A ^ {G} (s) ],
$$
where the system policy  $\pi_{\phi}$  is parameterized by  $\phi$  and the user policy  $\mu_{\omega}$  by  $\omega$ .
In summary, a brief script for MADPL is shown in Algorithm 1.
MultiWOZ (Budzianowski et al., 2018) is a multi-domain, multi-intent task-oriented dialog corpus that contains 7 domains, 13 intents, 25 slot types, 10,483 dialog sessions, and 71,544 dialog turns. During the data collection process, a user is asked to follow a pre-specified user goal, and is allowed to change the goal during the session if necessary, so the collected dialogs are much closer to real-world conversations. The corpus also provides the domain knowledge that defines all the entities and attributes as the external database.
Evaluation of a task-oriented dialog system mainly consists of the cost and task success. We count the number of dialog turns to reflect the dialog cost. A user utterance and a subsequent system utterance are regarded as one dialog turn. We utilize two other metrics: inform F1 and match rate to estimate the task success. Both metrics are calculated at the dialog act level. Inform F1 evaluates whether all the requested information has been informed, and match rate checks whether the booked entities match all the indicated constraints given by the user. The overall task success is reached if and only if both inform recall and match rate are 1.
We compare MADPL with a series of baselines that involve both system policy learning and user policy learning. Note that we do not consider any other approaches that use a user simulator for policy training because our motivation is to avoid explicitly modeling a simulator.
SL Supervised Imitation Learning directly uses the dialog act annotations and trains the agents simply by behavior cloning using Eq. 1, which is the same as the pretraining phase in MADPL.
The following three baselines are all RL algorithms that start from the pretrained policy:
RL Independent Reinforcement Learning learns only one dialog policy by fixing another agent following the single RL setting, and the reward for
Class	Attraction	Hospital	Hotel
Count	320	22	389
Police	Restaurant	Taxi	Train
22	457	164	421
Num.	Single	Two	Three
Count	328	549	123
Table 1: Domain distribution of user goals used in the automatic evaluation. A user goal with multiple domains is counted repeatedly for each domain.
the agent is the sum of role-specific reward and global reward. For example, the user policy uses the reward  $r = r^{U} + r^{G}$  at each dialog turn.
CRL Centralized Reinforcement Learning is a MARL approach that uses a single centralized critic on the sum of reward  $r = r^{U} + r^{S} + r^{G}$  to train two agents simultaneously, which also serves for the ablation test of MADPL.