SlowGuess/ACL_2020_OCR huggingface dataset:
large positive reward  $(2L)$  for successful dialogue or a negative reward of  $L$  for failed one, where  $L$  (set as 70) is the maximum number of turns in each dialogue. A dialogue is considered successful only if the agent helps the user simulator accomplish the goal and satisfies all the user's search constraints. In addition, the average number of turns and the average reward are also reported to evaluate each model.
4.4 Simulator Evaluation
Main Results. The main simulation results are shown in Table 1 and Figure.2, 3, 4. The results show that with shaping mechanisms,  $S^2$  Agent learns much faster and performs consistently better than DQN and DQfD in all the domains with a statistically significant margin.
Figure 2 shows the learning curve of different agents in different domains. Firstly, the DQN agent performs better than the IM agent, which is not surprising since it interacts with the simulator and is optimized to solve user goals. DQfD and EAPC agents leverage human demonstrations to mitigate the reward sparsity issues. Their performances are consistently better than DQN. Besides,  $S^2$  Agent w/o ps uses reward shaping to alleviate reward sparsity by bonusing additional rewards for states that are consistent with demonstrations. As a consequence, it performs better than DQN in all the domains. Though EAPC has a similar reward shaping mechanism, its reward estimation relies heavily on the qualify of the action prediction model. As such, EAPC performs slightly worse than  $S^2$  Agent w/o ps. In addition, policy shaping reconciles the agent action with knowledge learned from human demonstrations. It biases the agent to explore these actions which human expert does. As shown in figure 2,  $S^2$  Agent w/o rs learn the dialogue policy much
faster than all the baselines. In the Movie domain, it achieves nearly a  $60\%$  success rate using only 20 epochs. By contrast, the second-best agent DQfD only achieves a  $20\%$  successful rate at epoch 20. Similar results are also observed in Restaurant and Taxi domains. When integrating both policy shaping and reward shaping to DQN,  $S^2$  Agent achieves the best performance and is more data-efficient. For example,  $S^2$  Agent in the Taxi domain achieves approximately  $60\%$  successful rate at 50 epoch while the following competitor only has around  $40\%$  successful rate. The above observation also confirms that policy shaping and reward shaping operate in different dimensions, which means policy shaping improves the learning by directly calibrating in the action space and reward shaping in the value function space, and are mutual-complementary. Noted that the improvement of combining policy shaping and reward shaping in the Movie domain is not as significant as that in Restaurant and Taxi. This is too large degree attributed to the increased complexity of Restaurant and Taxi dataset, which have two times more slots than the Movie dataset, meaning that the state-action space is much larger than the movie domain and posing more challenges in exploration. Under this situation, policy shaping and reward shaping benefit the  $S^2$  Agent to a large extent.
Results of training with varying number of demonstrations. Intuitively, the number of human demonstrations has a large impact on policy learning. The imitation model agent might be able to summarize a good expert policy when a large volume of human demonstrations is available. However, we hope the shaping mechanism is capable of improving learning efficiency with limited human demonstrations for RL-base agents. As such, we
6361
Figure 4: Learning curves of different agents in Movie-Ext domain, all the agents are adapted from trained agents in Movie domain.
experiment with different sizes of demonstrations between 25 and 125 to assess the effect of different numbers of human demonstration on learning efficiency and quality. Figure 3 shows the average performance of each agent during learning, which indicates the learning speed and quality. Our proposed shaping mechanisms improve policy learning speed and quality and are robust to the number of demonstrations. Even with the small number of human demonstrations as 25,  $S^2$  Agent achieves a  $5\%$  higher success rate than DQfD and EAPC in the Movie domain and  $10\%$  in the Taxi domain. As the number of demonstrations increases, the gap between DQfD and  $S^2$  Agent becomes larger, showing that policy mechanisms can still benefit from more human demonstrations available.
Results of domain extension Typically, RL-based agents are built with a fixed ontology. However, a dialogue system should be able to evolve as being used to handle new intents, slots, unanticipated actions from users. To asses the ability of quickly adapting to the new environment, we extend existing movie user simulator, denoted as Movie-Ext, to simulate domain adaption scenario. Movie-Ext has an additional payment task requiring the agent to converse with users to firstly book a ticket and then finish the payment. Details about the extended intent/slots can be found in the in appendix Table.3. All the agents are continually optimized from the previously trained agents for the movie ticket booking task. Meanwhile, we additionally collect a small number of human demonstrations to update the IM agent. Figure 4 shows the learning curves of different agents on the extended task. As we can see, both  $S^2$  Agent and  $S^2$  Agent
$w / o$  rs can quickly adapt to the new environment and outperform the IM agent, with only 150 epochs it achieves around  $50\%$  success rate. Though DQfD explicitly leverages human demonstrations, it still lags behind  $w / o$  rs, showing that shaping in the policy space is more effective than solely adding supervised learning loss for Q-learning. Reward shaping also benefits DQN to explore better policy. These observations confirm that  $S^2 A g e n t$  with shaping mechanism is capable of quickly adapting to the new environment.
User simulators are not necessary to reflect the complexity of human users (Dhingra et al., 2017). To further evaluate the feasibility of  $S^2$ Agent in real scenarios, We deploy the agents in Table 1 to interact with real human users in Movie and Movie-Ext domains.
Table 2: Human evaluation results on Movie and Movie-Ext domains. We use models at epoch 50 and epoch 200 for Movie domain and Movie-Ext, respectively.  $w / o r s$  denotes  $S^2 A g e n t$  without reward shaping;  $w / o p s$  denotes  $S^2 A g e n t$  without policy shaping; * denotes significant level  $p < 0.05$  with other agents. Succ. denotes success rate.
Model	Movie	Movie-Ext
Succ.↑	Rating↑	Succ.↑	Rating↑
IM	0.42	3.92	0.40	1.96
DQN	0.56	3.36	0.26	2.68
EAPC	0.68	3.96	0.34	3.12
DQfD	0.72	3.92	0.50	3.24
\( S^2 Agent^* \)	0.74	4.36	0.62	3.56
w/o rs	0.72	4.26	0.46	2.94
w/o ps	0.70	4.12	0.52	3.20
All evaluated agents are trained with 50 epochs and 200 epochs for Movie and Movie-Ext respectively. In each dialogue session, one of the agents is randomly selected to converse with a human user. Each user is assigned with a goal sampled from the corpus and is instructed to converse with the agent to complete the task. Users have the choice of terminating the task and ending the session at any time if users believe that the dialogue is unlikely to succeed or simply because the agent repeats for several turns. In such a case, the session is considered as a failure. Finally, at the end of each session, users are required to give explicit feedback on whether the dialogue succeeded (i.e., whether
For the time and cost consideration, we only conduct experiments on Movie and Movie-Ext domains.
6362
the movie tickets were booked (and paid) with all the user constraints satisfied). Additionally, users are requested to rate the session on a scale from 1 to 5 about the quality/naturalness (5 is the best, 1 is the worst). We collect 50 dialogue sessions for each agent. The results are listed in Table 2.  $S^2$  Agent and  $S^2$  Agent w/o rs perform consistently better than DQN and DQfD, which is consistent with what we have observed in simulation evaluation. In addition,  $S^2$  Agent achieves the best performance in terms of success rate and user rating.
In this paper, we present a new strategy for learning dialogue policy with human demonstrations. Compared with previous work, our proposed  $S^2$  Agent is capable of learning in a more efficient manner. By using policy shaping and reward shaping,  $S^2$  Agent can leverage knowledge distilled from the demonstrations to calibrate actions from underlying RL agents for better trajectories, and obtains extra rewards for these state-action similar to demonstrations alleviating reward sparsity for better exploration. The results of simulation and human evaluation show that our proposed agent is efficient and effective in both single domain and a challenging domain adaptation setting.
We appreciate the efforts from the anonymous reviewers; they have helped us improve this paper a lot. The research described in this paper is partially supported by Hong Kong RGC-GRF grant 14204118.
Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E Taylor, and Ann Nowé. 2015. Reinforcement learning from demonstration through shaping. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Thomas Cederborg, Ishaan Grover, Charles L Isbell, and Andrea L Thomaz. 2015. Policy shaping with human teachers. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
Lu Chen, Runzhe Yang, Cheng Chang, Zihao Ye, Xi-ang Zhou, and Kai Yu. 2017a. On-line dialogue policy learning with companion teaching. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 198-204.
Lu Chen, Xiang Zhou, Cheng Chang, Runzhe Yang, and Kai Yu. 2017b. Agent-aware dropout dqn for safe and efficient on-line dialogue policy learning. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2454-2464.
Bhuwan Dhingra, Lihong Li, Xiujun Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2017. Towards end-to-end reinforcement learning of dialogue agents for information access. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 484-495.
Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. 2013. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in neural information processing systems, pages 2625-2633.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. 2018. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence.
Alankar Jain, Florian Pecune, Yoichi Matsuyama, and Justine Cassell. 2018. A user simulator architecture for socially-aware conversational agents. In Proceedings of the 18th International Conference on Intelligent Virtual Agents, pages 133-140. ACM.
Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Zheng Zhang, Yaoqin Zhang, Xiang Li, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, and Jianfeng Gao. 2019. ConvLab: Multi-domain end-to-end dialog system platform. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 64-69, Florence, Italy. Association for Computational Linguistics.
Jinchao Li, Baolin Peng, Sungjin Lee, Jianfeng Gao, Ryuichi Takanobu, Qi Zhu, Minlie Huang, Hannes Schulz, Adam Atkinson, and Mahmoud Adada. 2020. Results of the multi-domain task-completion dialog challenge. In Proceedings of the 34th AAAI Conference on Artificial Intelligence, Eighth Dialog System Technology Challenge Workshop.
Xiujun Li, Yun-Nung Chen, Lihong Li, Jianfeng Gao, and Asli Celikyilmaz. 2017. End-to-end task-completion neural dialogue systems. arXiv preprint arXiv:1703.01008.
Xiujun Li, Zachary C Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016. A user simulator for task-completion dialogues. arXiv preprint arXiv:1612.05688.
Zachary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng. 2018. BBQ-networks:
6363
Efficient exploration in deep reinforcement learning for task-oriented dialogue systems. In Thirty-Second AAAI Conference on Artificial Intelligence.
Zachary C Lipton, Jianfeng Gao, Lihong Li, Xiu-jun Li, Faisal Ahmed, and Li Deng. 2016. Efficient exploration for dialog policy learning with deep BBQ networks & replay buffer spiking. CoRR abs/1608.05081.
Bhaskara Marthi. 2007. Automatic shaping and decomposition of reward functions. In Proceedings of the 24th International Conference on Machine learning, pages 601-608. ACM.
Dipendra Misra, Ming-Wei Chang, Xiaodong He, and Wen-tau Yih. 2018. Policy shaping and generalized update equations for semantic parsing from denotations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2442-2452, Brussels, Belgium. Association for Computational Linguistics.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529.
Shakir Mohamed and Danilo Jimenez Rezende. 2015. Variational information maximisation for intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 2125-2133.
Andrew Y Ng, Daishi Harada, and Stuart Russell. 1999. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278-287.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Yun-Nung Chen, and Kam-Fai Wong. 2018a. Adversarial advantage actor-critic model for task-completion dialogue policy learning. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6149-6153. IEEE.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. 2018b. Deep dyna-q: Integrating planning for task-completion dialogue policy learning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2182-2192.
Baolin Peng, Xiujun Li, Lihong Li, Jianfeng Gao, Asli Celikyilmaz, Sungjin Lee, and Kam-Fai Wong. 2017. Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. arXiv preprint arXiv:1704.03084.
Olivier Pietquin, Matthieu Geist, Senthilkumar Chandramohan, and Hervé Frezza-Buet. 2011. Sample-efficient batch reinforcement learning for dialogue management optimization. ACM Transactions on Speech and Language Processing (TSLP), 7(3):7.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149-152. Association for Computational Linguistics.
Pei-Hao Su, Pawel Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017. Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management. arXiv preprint arXiv:1707.00130.
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. arXiv preprint arXiv:1606.02689.
Pei-Hao Su, David Vandyke, Milica Gasic, Nikola Mrkusic, Tsung-Hsien Wen, and Steve Young. 2015. Reward shaping with recurrent neural networks for speeding up on-line policy learning in spoken dialogue systems. arXiv preprint arXiv:1508.03391.
Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 100-110, Hong Kong, China. Association for Computational Linguistics.
Eric Wiewiora, Garrison W Cottrell, and Charles Elkan. 2003. Principled methods for advising reinforcement learning agents. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 792-799.
Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. arXiv preprint arXiv:1702.03274.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
6364
Table 3: The data annotation schema.
Movie	Restaurant	Taxi	Movie-Ext
Slots	city, numberof-people, theater,zip, distancecon-straints, theater chain, video format,state, starttime, date,moviename, ticket,taskcomplete	city, closing, date,distanceconstraints,cuisine, greeting,restaurantname,numberofpeople,numberofkids,taskcomplete, other,pricing, starttime,state,zip, address,reservation, theater,atmosphere, rating,dress code, food,mealtype, choice, seating, occasion,personfullname,phonenumber,restauranttype	car type, city, clos-ing, car level, date,distanceconstraints,dropoff location,greeting, name,driver id, numberof-people, other, pickuplocation, dropoff location city, budget,pickup location city,pickup time, speed,state, cost, taxi company, mc list,taskcomplete, taxi,zip, result, driver level, numberofkids,emergency degree	city, numberofpeo-ple, theater,zip, distanceconstraints, theater chain, video format, state,starttime, date,moviename, ticket,taskcomplete, bill,cost, tax, bill num-
ber, bank, service fee, pay type, discount, consump-
tion point, credit card point
Intent	request, inform ,confirm question, confirm answer, greeting, closing, multiple choice, thanks,welcome, deny, not sure
Table 4: The performance of Imitation Model on different dataset.
Domain	#Pair	Precision	Recall	F1-score
Movie	50	0.76	0.86	0.81
Restaurant	50	0.73	0.80	0.76
Taxi	50	0.83	0.90	0.86
Movie-Ext	100	0.84	0.83	0.82
Table 3 lists all annotated dialogue acts and slots in details. Table 4 lists the training results of Imitation Model on all dataset.
6365