SlowGuess/ACL_2020_OCR huggingface dataset:
Huimin Wang\* Baolin Peng\* Kam-Fai Wong
\*The Chinese University of Hong Kong
bapeng@microsoft.com
{hmwang,kfwong}@se.cuhk.edu.hk
Training a task-oriented dialogue agent with reinforcement learning is prohibitively expensive since it requires a large volume of interactions with users. Human demonstrations can be used to accelerate learning progress. However, how to effectively leverage demonstrations to learn dialogue policy remains less explored. In this paper, we present that efficiently learns dialogue policy from demonstrations through policy shaping and reward shaping. We use an imitation model to distill knowledge from demonstrations, based on which policy shaping estimates feedback on how the agent should act in policy space. Reward shaping is then incorporated to bonus state-action similarities to demonstrations explicitly in value space encouraging better exploration. The effectiveness of the proposed  $S^2$ Agent is demonstrated in three dialogue domains and a challenging domain adaptation task with both user simulator evaluation and human evaluation.
With the flourishment of conversational assistants in daily life (like Google Assistant, Amazon Alexa, Apple Siri, and Microsoft Cortana), task-oriented dialogues that are able to serve users on certain tasks have increasingly attracted research efforts. Dialogue policy optimization is one of the most critical tasks of dialogue modeling. One of the most straightforward approaches is the rule-based method, which contains a set of expert-defined rules for dialogue modeling. Though rule-based dialogue systems have a reasonable performance in some scenarios, handcrafting such kinds of rules is time-consuming and not scalable.
Recently, dialogue policy learning is formulated as a reinforcement learning (RL) problem and tackled with deep RL models (Li et al., 2017; Lipton
et al., 2018; Peng et al., 2017). It has shown great potentials of using the RL-based method for building robust dialogue systems automatically. However, due to its interactive nature, RL-based agents demand of an environment to operate in. As illustrated in Figure 1, RL-based dialogue agents need to interact with human users and update its policy in an online fashion requiring that the agents have a good online performance from the start of training. In addition, one of the biggest challenges of RL approaches is reward sparsity issue, which leads to exploration in large action space inefficient. As a consequence, training RL-based agents expects a prohibitively large number of interactions to achieve acceptable performance, which may incur a significant amount of expense (Pietquin et al., 2011; Lipton et al., 2016; Peng et al., 2018b). Several attempts are made to improve learning efficiency and tackle reward sparsity issues. Different types of heuristics has been proposed in the form of intrinsic rewards to guide exploration more efficiently (Lipton et al., 2016; Mohamed and Rezende, 2015; Peng et al., 2017, 2018a; Takanobu et al., 2019).
When building a dialogue system, it is typically affordable to recruit experts to gather some demonstrations about the expected agent behaviors. We therefore aim to address the aforementioned challenges from a different perspective and assume having access to human-provided demonstrations. In this paper, we investigate how to efficiently leverage these demonstrations to alleviate reward sparsity and improve policy learning quality. Previous work (Lipton et al., 2016) used a simple technique termed as Replay Buffer Spiking (RBS) to pre-fill experience replay buffer with human demonstrations, which yields good performance, especially in the beginning. (Hester et al., 2018) proposed Deep Q-learning from Demonstrations (DQfD) that combines temporal difference updates with a supervised classification loss of actions in demonstrations to
* Corresponding author † Equal Contribution
6355
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6355-6365 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
improve learning efficiency in gaming domains. However, whether it is feasible and how to effectively leverage human demonstration in dialogue scenarios are less explored.
Hence, in this paper, we propose a new strategy of leveraging human demonstrations to learn dialogue policy efficiently. Our dialogue agent, termed as  $S^2 \text{Agent}^1$ , learns dialogue policy from demonstrations trough policy shaping and reward shaping. Policy shaping (Griffith et al., 2013) is an approach to incorporating human feedback to advise how policy should behave like experts. It estimates feedback of a state-action pair from human demonstrations and then utilizes the feedback to reconcile the policy from any RL-based agents. This method speeds up learning progress in gaming domains but has not yet been studied in dialogue. However, directly applying policy shaping to dialogue faces several challenges. The original policy shaping uses a tabular analogous method to estimate feedback. This method limits its feasibility for complex problems like dialogue that has large state action representations. To deal with this issue, we propose to use deep neural networks, which represent state-action space with function approximation and distill knowledge from human demonstrations, to estimate feedback. In addition, policy shaping calibrates agents' behavior in policy space, and it is inherently not designed to tackle reward sparsity issues. Considering this, we further introduce reward shaping to bonus these state-action pairs that are similar to demonstrations. It can be viewed as a shaping mechanism explicitly in value space to guide policy exploration towards actions which human experts likely conduct. Our contributions in this work are two-fold:
- We propose a novel  $S^2$ Agent that can effectively leverage human demonstrations to improve learning efficiency and quality through policy shaping and reward shaping.
- We experimentally show that  $S^2$  Agent can efficiently learn good policy with limited demonstrations on three single domain dialogue tasks and a challenging domain adaptation task using both simulator and human evaluations.
Dialogue policy learning Deep reinforcement learning (RL) methods have shown great potential in building a robust dialog system automatically (Young et al., 2013; Su et al., 2016; Williams et al., 2017; Peng et al., 2017, 2018a,b; Lipton et al., 2018; Li et al., 2020; Lee et al., 2019). However, RL-based approaches are rarely used in real-world applications, for these algorithms often require (too) many experiences for learning due to the sparse and uninformative rewards. A lot of progress is being made towards mitigating this sample complexity problem by incorporating prior knowledge. (Su et al., 2017) utilizes a corpus of demonstration to pre-train the RL-based models for accelerating learning from scratch. (Chen et al., 2017b) attempts to accelerate RL-based agents by introducing extra rewards from a virtual rule-based teacher. However, the method requires extra efforts to design a rule-based dialogue manager. (Hester et al., 2018) improve RL learning by utilizing a combination of demonstration, temporal difference (TD), supervised, and regularization losses. (Chen et al., 2017a) introduced a similar approach called companion teaching to incorporate human teacher feedback into policy learning. Nevertheless, companion teaching assumes that there is a human teacher to directly give a correct action during policy learning process and meanwhile train an action prediction model for reward shaping based on human feedback.
Policy shaping Policy Shaping is an algorithm that enables introducing prior knowledge into policy learning. (Griffith et al., 2013) formulates human feedback on the actions from an agent policy as policy feedback and proposes Advise algorithm to estimate humans Bayes feedback policy and combine it with the policy from the agent. It shows significant improvement in two gaming environment. (Misra et al., 2018) uses policy shaping to bias the search procedure towards semantic parses that are more compatible with the text and achieve excellent performance.
Reward shaping Reward shaping leverages prior knowledge to provide a learning agent with an extra intermediate reward  $F$  in addition to environmental reward  $r$ , making the system learn from a composite signal  $R + F$  (Ng et al., 1999). However, it is not guaranteed that with reward shaping, an MDP can still have an optimal policy that is
<sup>1</sup>Agent with policy Shaping and reward Shaping
6356
Figure 1: Illustration of the  $S^2$  Agent for dialogue policy learning.
identical to the original problem unless the shaping is potential-based reward shaping(Ng et al., 1999; Marthi, 2007). (Su et al., 2015) proposes to use RNNs to predict turn-level rewards and use the predicted reward as informative reward shaping potentials. (Peng et al., 2018a; Takanobu et al., 2019) use inverse reinforcement learning to recover reward functions from demonstrations for reward shaping. However, the estimated reward using these methods inevitably contains noise and failed to conform to potential-based reward function to guarantee the optimal policy. Inspired by (Brys et al., 2015), we directly estimate potential-based reward function from demonstrations.
Our  $S^2$  Agent is illustrated in Figure 1, consisting of four modules. 1) Dialogue policy model which selects the best next action based on the current dialogue state.; 2) Imitation Model is formulated as a classification task that takes dialogue states as input and predicts associated dialogue action, aiming to distill behaviors from human demonstrations.; 3) Policy Shaping provides feedback on how policy should behave like demonstrations. It then reconciles a final action based on actions from the policy model and imitation model attempting to generate more reliable exploration trajectories; 4) Followed by a reward shaping module that encourages demonstration similar state-action by providing extra intrinsic reward signals.
3.1 Policy Model
We consider dialogue policy learning as a Markov Decision Process (MDP) problem and improve the policy with Deep Q-network (DQN) (Mnih
et al., 2015). In each turn, the agent observes the dialogue state  $s$ , and then execute the action  $a$  with  $\epsilon$ -greedy exploration that selects a random action with probability  $\epsilon$  or adopts a greedy policy  $a = \operatorname{argmax}_{a'} Q(s, a'; \theta)$ , where  $Q(s, a'; \theta)$  approximates the value function, implemented as a multi-layer perceptron (MLP) parameterized by  $\theta$ . The agent then receives the reward  $r$ , perceives the next user response to  $a^u$ , and updates the state to  $s'$ . The tuple  $(s, a, r, s')$  is stored in the experience replay  $D^a$ . This loop continues until the dialogue terminates. The parameters of  $Q(s, a'; \theta)$  are updated by minimizing the following square loss with stochastic gradient descent:
$$
\begin{array}{r l} \mathcal {L} (\theta) = & \mathbb {E} _ {(s, a, r, s ^ {\prime}) \sim D ^ {a}} [ (y _ {i} - Q (s, a; \theta)) ^ {2} ] \\ y _ {i} = & r + \gamma \max  _ {a ^ {\prime}} Q ^ {\prime} (s ^ {\prime}, a ^ {\prime}; \theta^ {\prime}) \end{array} \tag {1}
$$
where  $\gamma \in [0,1]$  is a discount factor, and  $Q(.)$  is the target value function that is only periodically updated (line 26 in Algorithm 1). By differentiating the loss function with regard to  $\theta$ , we derive the following gradient:
$$
\begin{array}{l} \nabla_ {\theta} \mathcal {L} (\theta) = E _ {(s, a, r, s ^ {\prime}) \sim D ^ {a}} [ (r + \\ \gamma \max  _ {a ^ {\prime}} Q ^ {\prime} \left(s ^ {\prime}, a ^ {\prime}; \theta^ {\prime}\right) - \tag {2} \\ Q (s, a; \theta)) \nabla_ {\theta} Q (s, a; \theta) ] \\ \end{array}
$$
As shown in lines 25-26 in Algorithm 1, in each iteration, we update  $Q(.)$  using minibatch Deep Q-learning.
3.2 Imitation Model
We assume having access to a corpus of human-human dialogues either from a log file or provided by recruited experts, which in this paper are termed as human demonstrations  $D^{e}$ .  $D^{e}$  usually consists of a set of state-action pairs  $[(s_{1},a_{1}),(s_{2},a_{2}),\dots,(s_{n},a_{n})]$ . Theoretically, if  $D^{e}$  is large enough to cover all the possible states, then the agent can respond perfectly by looking up the corresponding action from  $D^{e}$ .
However, in practice,  $D^{e}$  is usually limited and can not cover all the states. Hence, we propose to use a supervised learning model (denoted as Imitation Model) to parameterize the relation of the
$^{2}$ Our shaping methods are compatible with any policy optimizer. In this paper, we employ DQN due to its simplicity and robustness in training. However, replacing with other methods like Actor-Critic is straightforward.
6357
states and actions expecting it to generalize to unseen state. We formulate the task as a classification problem. It takes dialogue  $s_i$  as input and is trained with cross-entropy to minimize loss between action  $a_i$  and predicted action  $a$ . There are multiple models like RNN, CNN can be used for this purpose, but for simplicity, we choose to use MLP.
3.3 Policy Shaping
Incorporating human feedback into RL can accelerate its learning progress (Griffith et al., 2013; Cederborg et al., 2015). Policy shaping is a representative that estimates human's Bayes optimal feedback policy and then combine the feedback policy with the policy of an underlying RL model. The feedback policy is computed with the following equation:
$$
\pi_ {e} (a | s) = \frac {\mathcal {C} ^ {\Delta_ {s , a}}}{\mathcal {C} ^ {\Delta_ {s , a}} + (1 - \mathcal {C}) ^ {\Delta_ {s , a}}} \tag {3}
$$
where  $\Delta_{s,a}$  is the difference between the number of positive feedback and negative feedback, i.e. the number of occurrence of  $(s,a)$  in human demonstrations.  $\mathcal{C}$  here means the probability of consistency feedback from demonstrations  $^3$ . For example,  $\mathcal{C} = 0.7$  means with 0.7 probability the feedback from the demonstrations is considered reliable. Otherwise, if  $\mathcal{C} = 0.5$ , then policy shaping is meaningless since it treats every action equally.
However,  $\Delta_{s,a}$  is difficult to estimate from the demonstrations in dialogue scenarios since the state and action are large and sparse. To deal with this issue, we propose to use the aforementioned imitation Model to estimate feedback from demonstrations. Specifically, we samples  $N$  times from imitation model policy  $\pi_e(a|s)$  to form a committee  $a_1, a_2, \ldots, a_N$  denoting  $N$  votes. Then we count for each action to generate  $c_a$  as positive feedback from human demonstrations. We use the expectation of binomial distribution  $N * (1 - \mathcal{C})$  as the number of negative feedback. Such that, in dialogue, we use:
$$
\Delta_ {s, a} = c _ {a} - N * (1 - \mathcal {C}) \qquad (4)
$$
Finally, the policy is reconciled from the policy model and the imitation model by multiplying them together:
$$
\pi (a | s) = \frac {\pi_ {a} (a | s) \times \pi_ {e} (a | s)}{\sum_ {a} \pi_ {a} (a | s) \times \pi_ {e} (a | s)} \tag {5}
$$
Policy shaping operates in the policy space and can be viewed as a mechanism of biasing the agent learning towards the policy distilled from the demonstrations to improve learning efficiency. The reconciled policy in equ. 5 allows the underlying RL model surpass the imitation model  $\pi_{e}$
3.4 Reward Shaping
Most of the reward functions in dialogue scenarios are usually manually defined. Typically, a -1 for each turn and a significant positive or negative reward indicating the status of the dialogue at the end of a session. Such sparse reward is one of the reasons that RL agents have poor learning efficiency. Initially, the agents are fain to explore state-action uniformly at random. To this end, we propose to use reward shaping to integrate priors into RL learning to alleviate reward sparsity.
Reward shaping is a popular method to integrate prior knowledge into reward function to improve policy exploration (Brys et al., 2015). It provides the learning agent with an extra intermediate and
It is a parameter to control noise in the demonstrations.
6358
task-related reward that enriches the original reward signal:
$$
r ^ {\prime} (s, a) = r (s, a) + F _ {D} (\cdot) \tag {6}
$$
Where  $F_{D}$  denotes rewards from demonstrations. However, modifying the reward function may change the original MDPs and make the agent converge to a suboptimal point. (Wiewiora et al., 2003) proved that the MDP keeps unchanged and maintains convergence property if  $F_{D}(\cdot)$  is defined as:
$$
F _ {D} (s, a, s ^ {\prime}, a ^ {\prime}) = \gamma \phi_ {D} (s ^ {\prime}, a ^ {\prime}) - \phi_ {D} (s, a) \quad (7)
$$
where  $\phi_D(s,a)$  is a potential function of state-action pair. Its definition is intuitive. We bonus these policy paths that were consistent with the demonstrations. As such, the value of  $\phi_D(s,a)$  is expected to be high when action  $a$  is demonstrated in a state  $s^d$  similar to  $s$ , and if  $s$  is completely different from  $s_a^d$ ,  $\phi_D(s,a)$  should be close to 0. To achieve this, multi-variate Gaussian is used to compute the similarity between state-action pairs.
$$
G (s, a, s ^ {d}, a ^ {d}) = \left\{ \begin{array}{l l} e ^ {\left(- \frac {1}{2} \left(s - s _ {a} ^ {d}\right) ^ {T} \Sigma^ {- 1} \left(s - s _ {a} ^ {d}\right)\right)}, a = a ^ {d} \\ 0 \quad o t h e r w i s e \end{array} \right. \tag {8}
$$
We search through the demonstrations to obtain the sample with highest similarity:
$$
\phi_ {D} (s, a) = \max  _ {s _ {a} ^ {d}} G (s, s _ {a} ^ {d}) \tag {9}
$$
Using reward shaping to learn policy has several advantages. It leverages demonstrations to bonus these state-action's that are similar to demonstrations. The reward calculated from reward shaping is more informative and demonstration guided than the human-defined reward, which mitigates the reward sparsity issue to some degree.
We evaluate the proposed  $S^2$ Agent with a user simulator on several public task-oriented datasets, including movie ticket booking, restaurant reservation, and taxi reservation. Additionally, to assess the generalization capability of shaping mechanism, We conduct domain adaptation experiments. Finally, human evaluation results are reported.
The raw conversation data in the movie ticket booking task are collected through Amazon Mechanical
Turk, and the data for the restaurant reservation and taxi calling scenario is provided by Microsoft Dialogue Challenge  $^{4}$ . The three datasets have been manually labeled based on a schema defined by domain experts. We extend and annotated movie booking task with a payment scenario to simulate the situation of extending the dialogue system with new slots and values. All datasets contain 11 intents. The movie dataset contains 13 slots, and the other three contain 29 slots. Detailed information about the intents and slots is provided in Appendix A table 3.
To benchmark the performance of the shaping mechanism, we have developed different versions of task-completion dialogue agents for comparison as follows:
- Imitation Model (IM) agent is implemented with Multi-Layer Perception and trained with the human demonstrations data to predict actions given dialogue states.
- DQN agent is learned with Deep Q-Network.
- EAPC Teaching via Example Action with Predicted Critique (EAPC) introduced in (Chen et al., 2017a) leverages real-time human demonstrations to improve policy learning. EAPC assumes the existence of human teachers during the learning process. It receives example actions from human teachers and, in the meantime, trains an action prediction model with the example actions as a critic for turn-level reward shaping. Since human teachers are not available in our case, we implement EAPC in the absence of teachers but use the same amount of human demonstrations to train a weak action prediction model. If the predicted action is identical to the action given by the policy model, the agent receives an extra positive reward otherwise an extra negative reward. This method can be viewed as a variant of  $S^2$  Agent with only reward shaping using noise reward estimations from the imitation model.
- DQfD (Hester et al., 2018) agent also leverages human demonstrations to improve policy learning. It adds additional classification
4https://github.com/xiul-msr/e2e dialog_challenge
6359
(a) Movie
(b) Restaurant
Figure 2: Learning curves of all the agents in Movie, Restaurant and Taxi domains. All the agents use the same amount of human demonstrations.
(c) Taxi
Table 1: The performance of the average turn and average reward of different agents in different domains.  $w / o$  rs denotes  ${S}^{2}$  Agent without reward shaping;  $w/{ops}$  denotes  ${S}^{2}$  Agent without policy shaping;* denotes significant level  $p < {0.05}$  with other baselines except DQfD in movie domain. Succ. denotes success rate.
Agent	Movie	Restaurant	Taxi	Movie-Ext
Succ.↑	Turn↓	Reward↑	Succ.↑	Turn↓	Reward↑	Succ.↑	Turn↓	Reward↑	Succ.↑	Turn↓	Reward↑
IM	0.33	32.62	-11.47	0.16	37.56	-52.03	0.22	15.07	-27.33	0.37	35.38	-8.84
DQN	0.82	37.13	21.57	0.51	50.84	-31.65	0.84	45.69	-10.08	0.66	57.01	-40.37
EAPC	0.82	30.66	42.22	0.53	48.07	-24.86	0.88	38.71	19.10	0.65	55.34	-31.50
DQfD	0.81	27.53	50.57	0.52	43.90	-5.10	0.86	34.88	32.33	0.61	53.64	-19.34
\( S^2 \)Agent*	0.82	21.25	72.68	0.57	38.35	16.40	0.87	27.25	61.50	0.70	49.97	3.39
w/o rs	0.82	23.30	69.20	0.57	39.66	11.85	0.88	28.14	55.47	0.67	51.03	-4.93
w/o ps	0.80	31.68	40.28	0.57	45.40	-9.45	0.85	35.39	28.20	0.65	53.68	-19.27
loss from human demonstrations to DQN to ensure that the agent predicts correct actions on human demonstrated states. In the early learning phase, DQfD is trained only with the demonstrations to obtain a policy that mimics the human. Then, accumulated experiences mixed with the demonstration are used to train DQfD.
-  $S^2$  Agent is our proposed agent that is trained with both policy shaping and reward shaping, as described in Algorithm 1.
-  $S^2$  Agent w/o rs is a variant of  $S^2$  Agent which learns policy with only policy shaping to reconcile the final action.
-  $S^2$  Agent w/o ps is a variant of  $S^2$  Agent but only has reward shaping to bonus state-actions similar to demonstrations.
Implementation Details Imitation model agents for all domains are single layer MLPs with 50 hidden dimensions and tanh as the activation function. The IM agent is also used in policy shaping to reconcile the policy. All RL-based agents (DQN, DQfD,  $S^2$  Agent) are MLPs with tanh activations. Each policy network Q(.) has one hidden layer with
60 hidden nodes. All the agents are trained with the same set of hyper-parameters.  $\epsilon$ -greedy is utilized for policy exploration. We set the discount factor as  $\gamma = 0.9$ . The target network is updated at the end of each epoch. To mitigate warm-up issues, We build a naive but occasionally successful rule-based agent to provide experiences in the beginning. For a fair comparison, we pre-fill the experience replay buffer  $D^{a}$  with human demonstrations for all the variants of agents (Lipton et al., 2016). Confidence factor  $C$  used in policy shaping is set 0.7. As for the reward shaping,  $\gamma$  in eq.7 is set as 1.
Training RL-based dialogue agents require an environment to interact with, and it usually needs a large volume of interactions to achieve good performance, which is not affordable in reality. It is commonly acceptable to employ a user simulator to train RL-based agents (Jain et al., 2018; Li et al., 2016; Schatzmann et al., 2007).
We adopt a public available agenda-based user simulator (Li et al., 2016) for our experiment setup. During training, the simulator provides the agent with responses and rewards. The reward is defined as -1 for each turn to encourage short turns and a
(b) Restaurant
Figure 3: The effect of number of human demonstration on the performance. The moving averaged success rate is calculated within 120 epochs for Movie, 200 epochs for Restaurant, and 200 epochs for Taxi.