SlowGuess/NAACL_2021_OCR huggingface dataset:
Training Settings We experiment on two training algorithms: PPO and LCPO. The hyperparameters of PPO and LCPO are the same as the simulated user experiment. A human user interacts with each agent for 100 dialogues. At the end of each dialogue, the user gives 20 scores to the agent for a successful dialogue and gives 0 scores for a failed one. A penalty of  $-1$  is also applied in each turn.
A successful dialogue means the restaurant given by the agent must fulfil all the constraints and the requested information like phone number or address must be provided. In other words, the agents only receive feedback on the aspect of task completion.
Evaluation Settings Each human user interacts with each agent for 5 dialogue and gives his/her feedback on four aspects:
- Task completion: The agent finds a restaurant that meets the constraints. The requested information is also given.
- Conciseness: The agent is to the point and does not ask/provide the same information repetitively.
- Fluency: The agent does not interrupt the dialogue flow and answer the questions logically.
- Overall score: The overall score for chatting with this agent.
Each agent is evaluated on 100 dialogues, the mean and variance of each score are reported in Table 3. The scores are range from 0 to 5. We also evaluate the agents by a simulated user via 500 dialogues for each agent.
Results In table 3, we can see that LCPO significantly outperform PPO in all aspects. The task completion is close to the success rate evaluated by the simulated user. Conciseness is the feature of this work, and the improvement is also the most considerable. Regarding fluency, the difference between PPO and LCPO is smaller. Sometimes a fluent conversation takes more turns. Sometimes a non-logical response can complete the task as well (e.g. inform a restaurant name in the beginning). However, LCPO is still better than PPO in terms of fluency since a non-logical response usually accompanies with no information gain.
PPO	LCPO
Task Completion	2.0 ± 1.7	3.2 ± 1.5
Conciseness	1.8 ± 0.8	3.9 ± 1.1
Fluency	2.6 ± 0.5	3.6 ± 0.9
Overall score	2.1 ± 0.4	3.7 ± 0.9
Success rate (SimUser)	41.7%	66.8%
Table 3: Human-in-the-loop experiment. Human users evaluate each agent in four aspects. Each agent is trained by interacting with a human for 100 dialogues. The highest success rate in each row is highlighted. The last row is the success rates over 500 dialogues evaluated by a simulated user.
- We propose LCPO to improve sample efficiency for dialogue policy optimisation. LCPO has two critical components: loop clipping and advantage clipping. Both of them are strongly effective in low resource scenario and easy to implement. LCPO also demonstrates strong robustness to hyperparameters.
- We train and evaluate dialogue agents with real users on the Cambridge Restaurant domain. We also demonstrate that human-in-the-loop training is feasible within 100 dialogues. The evaluation has four aspects to clarify what has been learnt by each agent. LCPO outperforms PPO in all aspects.
In this paper, LCPO integrates with PPO. In the future, we will generalise loop clipping method to other off-policy reinforcement learning approaches with episodic memory since off-policy approaches are considered more sample-efficient.
Inigo Casanueva, Paweł Budzianowski, Pei-Hao Su, Nikola Mrksić, Tsung-Hsien Wen, Stefan Ultes, Lina Rojas-Barahona, Steve Young, and Milica Gašić. 2017. A benchmarking environment for reinforcement learning based task oriented dialogue management. arXiv preprint arXiv:1711.11023.
Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational ai. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1371-1374.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Esther Levin, Roberto Pieraccini, and Wieland Eckert. 1997. Learning dialogue strategies within the markov decision process framework. In 1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings, pages 72-79. IEEE.
Jiwei Li, Alexander H Miller, Sumit Chopra, Marc'Aurelio Ranzato, and Jason Weston. 2016. Dialogue learning with human-in-the-loop. arXiv preprint arXiv:1611.09823.
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. arXiv preprint arXiv:1804.06512.
Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Kam-Fai Wong. 2018. Integrating planning for task-completion dialogue policy learning. arXiv preprint arXiv:1801.06176.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-based user simulation for bootstrapping a pomdp dialogue system. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Companion Volume, Short Papers, pages 149-152. Association for Computational Linguistics.
Jost Schatzmann and Steve Young. 2009. The hidden agenda user simulation model. IEEE transactions on audio, speech, and language processing, 17(4):733-747.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. On-line active reward learning for policy optimisation in spoken dialogue systems. arXiv preprint arXiv:1605.07669.
Shang-Yu Su, Xiujun Li, Jianfeng Gao, Jingjing Liu, and Yun-Nung Chen. 2018. Discriminative deep dyna-q: Robust planning for dialogue policy learning. arXiv preprint arXiv:1808.09442.
Stefan Ultes, Lina M Rojas Barahona, Pei-Hao Su, David Vandyke, Dongho Kim, Inigo Casanueva, Pawe Budzianowski, Nikola Mrkšić, Tsung-Hsien
Wen, Milica Gasic, et al. 2017. Pydial: A multidomain statistical dialogue system toolkit. Proceedings of ACL 2017, System Demonstrations, pages 73-78.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, PeiHao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:1508.01745.
Jason D Williams. 2008. The best of both worlds: Unifying conventional dialog systems and pomdpps. In Ninth Annual Conference of the International Speech Communication Association.
Yen-chen Wu, Bo-Hsiang Tseng, and Milica Gasic. 2020a. Actor-double-critic: Incorporating model-based critic for task-oriented dialogue systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 854-863.
Yen-Chen Wu, Bo-Hsiang Tseng, and Carl Edward Rasmussen. 2020b. Improving sample-efficiency in reinforcement learning for dialogue systems by using trainable-action-mask. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8024-8028. IEEE.
Yuexin Wu, Xiujun Li, Jingjing Liu, Jianfeng Gao, and Yiming Yang. 2019. Switch-based active deep dynaq: Efficient adaptive planning for task-completion dialogue policy learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7289-7296.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Zhirui Zhang, Xiujun Li, Jianfeng Gao, and Enhong Chen. 2019. Budgeted policy learning for task-oriented dialogue systems. arXiv preprint arXiv:1906.00499.