SlowGuess/ACL_2020_OCR huggingface dataset:
Table 1 shows evaluation results over 1,000 dialogs for baseline and DQfD dialog managers using our three proposed experts inside ConvLab's evaluation environment. The Rule baseline is a rule-based DM included in ConvLab. FFN is a supervised learning baseline DM that directly uses the same in-domain classifier introduced in Section 4 to predict the next action. It is trained on MultiWOZ, and achieves  $21.53\%$  accuracy on the test set. Deep Q-network (DQN) is an RL agent which uses the hyperparameters described
<sup>1</sup>We use MultiWOZ2.0 with ConvLab user annotations
Turns	Inform	Match	Success
Rule	5.25	94.00	100	100
FFN	11.67	81.00	52.63	61.00
DQN	18.79	28.50	11.07	11.85
PPO	5.79	65.67	72.51	63.27
RE	5.33	92.33	97.07	98.33
FLE	6.81	89.67	94.12	91.67
RLE	7.64	81.33	89.34	85.03
NLE	7.20	84.67	85.31	86.83
FFN-ft	9.62	83.00	90.79	76.00
FLE+R	6.75	90.00	94.57	92.47
RLE+R	6.38	88.67	90.62	92.93
NLE+R	6.89	89.00	92.68	91.00
Table 1: Evaluation results of baseline systems (top) as well as DQfD with rule-based and our weak expert approaches trained in-domain. The middle section denotes DQfD agents trained without RoFL; the bottom section shows results for agents trained with RoFL. Evaluation is conducted using an agenda-based user-simulator for 1000 dialogs. Reported scores are average number of Turns, Inform F1, Match Rate, and Success Rate. Best performing weak expert agents are in bold.
in Section 5 except that it does not use demonstrations. We also compare against an agent trained with Proximal Policy Optimization (PPO; Schulman et al. 2017), an actor-critic based RL algorithm widely used across domains. We use the PPO hyperparameters laid out in Takanobu et al. (2019). The middle third of Table 1 summarizes results for DQfD agents trained with rule-based (RE), Full Label (FLE), Reduced Label (RLE), and No Label (NLE) experts. The bottom third shows results for our weak expert methods trained with RoFL (+R). We follow Takanobu et al. (2019) and report evaluation results in terms of average dialog length (Turns), F1-Score of the information provided that was requested by the user, Match Rate of user-goals, and Success Rate – the percentage of dialogs in which all information has been provided and all booking information is correct.
As expected, the Rule agent – written specifically for ConvLab – almost perfectly satisfies user goals. FFN is considerably worse, with a  $40\%$  lower Success Rate, and half the Match Rate of the rule-based agent. For standard DQN, the environment's large state and action spaces pose a serious challenge, and it barely exceeds  $11\%$  Success and Match Rates. PPO achieves a respectable
$63\%$  success rate, outperforming the FFN baseline. Crucially, all DQfD agents significantly outperform the FFN, DQN, and PPO baselines, with the RE and FLE approaches coming within  $3\%$  and  $6\%$  respectively of the Rule agent's performance.
In the remainder of this section we will further analyze and compare the performances of DQfD agents with progressively weak demonstrations using in-domain and out-of-domain experts, as well as those trained with and without RoFL.
In-Domain Weak Expert DQfD We train in-domain reduced and no label experts on the MultiWOZ dataset. The RLE scores 77 F1 on the reduced label test set, while the NLE manages 71 F1 of predicting whether an agent response belongs to a user utterance on the unannotated test set. As shown in Table 1 (middle), the scores of DQfD agents with in-domain experts follow a clear trend corresponding to the type of demonstration data. After 2.5 million training steps, the FLE - with the most informative demonstrations - clearly outperforms both RLE and NLE methods, while the latter two perform similarly.
Figure 4 shows graphs of the average Success Rates of DQN, PPO, and our proposed DQfD agents over the course of training. DQN struggles to find successful dialog strategies, although its Success Rate slowly inclines and seems to gain some traction towards the end of the maximum training steps. To begin with PPO learns rapidly, faster than RLE and NLE, but its Success Rate plateaus in the  $60\%$  range; it seems to learn to end dialogues too early. Both RE and FLE start with performance advantages, due to their high quality expert demonstrations. Over time, RE even approaches the Success Rate of its rule-based expert demonstrator. The FLE consistently outperforms approaches with weaker demonstrations, quickly exceeding the Success Rate of the underlying FFN after an early dip when the agent's exploration parameter  $\epsilon$  is relatively high.
The NLE comfortably outperforms the Reduced Label Expert throughout training, with the RLE only overtaking it at the end. We believe that this strong relative performance makes sense if we consider that, during pre-training, the NLE acts according to a more fine-grained action set than the RLE. While the RLE partitions the actions according to their reduced label, these sets are broad and contain many irrelevant responses, whereas
Figure 4: Average Success Rates of our methods trained on in-domain data over the course of 2.5 million training steps.
the NLE acts randomly according to a smaller, potentially higher-quality, set of actions which have high correspondence scores.
Finally, the graphs in Figure 4 indicate that none of the agents fully converge after the training step limit, although RE and FLE plateau. It is possible that after significantly more steps even DQN would converge to the ceiling performance of the Rule DM – but all our methods are considerably more sample efficient.
RoFL Training Table 1 (bottom) shows evaluation results of DQfD agents trained with RoFL fine-tuning. All weak experts improve with RoFL, especially the RLE which records an  $8\%$  jump in Success Rate. We also include the performance of the final fine-tuned FFN classifier, whose improvement over its original incarnation (15% higher Success Rate) demonstrates that fine-tuning helps narrow the domain gap between data and the RL environment.
In addition to Table 1, Figure 5 shows DM performance over the course of training. RoFL dramatically improves both the performance and convergence rate of the RLE, indicating a domain gap between the reduced label data and the sets of environment actions. RoFL improves the FLE early in training, but this gain tails off after 1 million steps - possibly due to the relative strength of the expert. The trend for NLE-R is more ambiguous, falling behind its standard DQfD counterpart before catching up to its performance. RoFL seems
Figure 5: Average Success Rate of RL agents over the course of 2.5 million training steps, with and without RoFL fine-tuning. Agents were evaluated every 2000 steps on 100 evaluation dialogs. Experts were trained on in-domain data.
to lead to the greatest gains when the expert initially struggles.
Out-of-Domain Weak Experts The weakest experts that we evaluate were trained on out-of-domain data. The OOD RLE, trained on Microsoft E2E and Frames, scores 53 F1 on a reduced label MultiWOZ test set, while the OOD NLE, trained on the same datasets, unannotated, only manages 41 F1 on the test set. Results for OOD approaches trained with and without RoFL are shown in Table 2, with training graphs in Figure 6.
Even without RoFL, the OOD RLE guides the DQfD agent to performance rates comparable to its in-domain counterpart. This indicates that even reduced labels learned on the OOD data provide the agent with enough clues to correctly satisfy some user goals. With RoFL, the OOD RLE surpasses the Success Rate of the in-domain system, and is only marginally worse than the fine-tuned in-domain expert. This shows that with RoFL we can learn a competitive DM in a challenging multi-domain environment while only using unannotated data from other dialog tasks.
RoFL leads to the greatest gain with the OOD NLE. Without fine-tuning, it scores a measly  $26\%$  Success Rate (although it should be noted that this is still higher than DQN), compared to  $86\%$  when the expert is trained on in-domain sentences. This illustrates the clear difference between the lan
Figure 6: Average Success Rates of Reduced and No Label experts trained on out-of-domain data over the course of 2.5 million training steps, with and without RoFL fine-tuning.
Turns	Inform	Match	Success
RLE	7.60	80.33	87.30	85.00
NLE	16.27	40.00	27.15	26.55
RLE+R	6.64	85.00	89.03	91.00
NLE+R	9.94	70.00	62.64	68.90
Table 2: Results of out-of-domain weak experts with and without RoFL training, using an agenda-based user-simulator for 1000 evaluation dialogs. Reported scores are average number of Turns, Inform F1, Match Rate, and Success Rate.
guage in the in- and out-of-domain data. With RoFL, OOD NLE is able to update its weights to adapt to the language of the environment, outperforming the unaltered expert's Success Rate by  $35\%$ . This improvement holds true throughout training, as shown in Figure 6. The graph also shows that OOD NLE+R has not started to converge after 2.5 million training steps; it is likely that with more training it would perform similarly to the in-domain NLE DM.
In this paper, we have shown that weak demonstrations can be leveraged to learn an accurate dialog manager with Deep Q-Learning from Demonstrations in a challenging multi-domain environment. We established that expert demonstrators can be trained on labeled, reduced-labeled, and unlabeled data and still guide the RL agent by means of their
respective auxiliary losses. Evaluation has shown that all experts exceeded the performance of reinforcement and supervised learning baselines, and in some cases even approached the results of a hand-crafted rule-based dialog manager.
Furthermore, we introduced Reinforced Finetune Learning (RoFL) a DAgger-inspired extension to DQfD which allows a pre-trained expert to adapt to an RL environment on-the-fly, bridging the domain-gap. Our experiments show that RoFL training is beneficial across different sources of demonstration data, boosting both the rate of convergence and final system performance. It even enables an expert trained on unannotated out-of-domain data to guide an RL dialog manager in a challenging environment.
In future, we want to continue to investigate the possibility of using even weaker demonstrations. Since our No Label Expert is trained on unannotated data, it would be interesting to leverage large and noisy conversational datasets drawn from message boards or movie subtitles, and to see how RoFL training fares with such a significant domain gap between the data and the RL environment.
We thank Ignacio Iacobacci and Gerasimos Lampouras for their valuable suggestions.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. 2018. Playing Hard Exploration Games by Watching YouTube. In Advances in Neural Information Processing Systems, pages 2930-2941.
Johan Bos, Ewan Klein, Oliver Lemon, and Tetsushi Oka. 2003. DIPPER: Description and Formalisation of an Information-State Update Dialogue System Architecture. In Proceedings of the Fourth SIG-dial Workshop of Discourse and Dialogue, pages 115-124.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. arXiv preprint arXiv:1508.05326.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gašić. 2018. MultiWOZ - A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling. In Proc. of EMNLP 2018, pages 5016-5026.
Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Hal Daumé III. 2015. Learning to Search Better Than Your Teacher. arXiv preprint arXiv:1502.02206v2.
Kyunghyun Cho, Bart Van Merrienboer, Dzmitry Bah-danau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder-Decoder Approaches. arXiv preprint arXiv:1409.1259.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805.
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: A Corpus for Adding Memory to Goal-oriented Dialogue Systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 207-219, Saarbrücken, Germany. Association for Computational Linguistics.
Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. 2016. Policy Networks with Two-stage Training for Dialogue Systems. arXiv preprint arXiv:1606.03152.
Jianfeng Gao, Michel Galley, Lihong Li, et al. 2019. Neural Approaches to Conversational AI. Foundations and Trends in Information Retrieval, 13(2-3):127-298.
Gabriel Gordon-Hall, Philip John Gorinski, Gerasimos Lampouras, and Ignacion Iacobacci. 2020. Show Us the Way: Learning to Manage Dialog from Demonstrations. In *The Eight Dialog System Technology Challenge* (DSTC-8) at AAAI 2020.
Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. 2017. Learning from Demonstrations for Real World Reinforcement Learning. CoRR, abs/1704.03732v4.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. 2017. Self-Normalizing Neural Networks. In Advances in neural information processing systems, pages 971-980.
Sungjin Lee, Qi Zhu, Ryuichi Takanobu, Xiang Li, Yaoqin Zhang, Zheng Zhang, Jinchao Li, Baolin Peng, Xiujun Li, Minlie Huang, and Jianfeng Gao. 2019. ConvLab: Multi-Domain End-to-End Dialog System Platform. In Proc. of ACL 2019.
Xiujun Li, Yun-Nung Chen, Lihong Li, and Jianfeng Gao. 2017. End-to-End Task-Completion Neural Dialogue Systems. CoRR, abs/1703.01008.
Xiujun Li, Sarah Panda, Jingjing Liu, and Jianfeng Gao. 2018. Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems. arXiv preprint arXiv:1807.11125.
Zachary Lipton, Xiujun Li, Jianfeng Gao, Lihong Li, Faisal Ahmed, and Li Deng. 2018. BBQ-Networks: Efficient Exploration in Deep Reinforcement Learning for Task-oriented Dialogue Systems. In Proc. of AAAI 2018.
Diane J. Litman and James F. Allen. 1987. A Plan Recognition Model for Subdialogues in Conversations. Cognitive Science, 11(2):163-200.
Bing Liu and Ian Lane. 2017. Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 482-489. IEEE.
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018. Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems. arXiv preprint arXiv:1804.06512.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2019. On the Variance of the Adaptive Learning Rate and Beyond. arXiv preprint arXiv:1908.03265.
Keting Lu, Shiqi Zhang, and Xiaoping Chen. 2019. Goal-Oriented Dialogue Policy Learning from Failures. arXiv preprint arXiv:1808.06497v2.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. 2015. Human-level Control Through Deep Reinforcement Learning. Nature, 518(7540):529-533.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Proc. of EMNLP 2014, pages 1532-1543.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards Scalable Multi-domain Conversational Agents: The Schema-guided Dialogue Dataset. arXiv preprint arXiv:1909.05855.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. 2011. A Reduction of Imitation Learning and Structured Prediction to No-regret Online Learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635.
Tim Salimans and Richard Chen. 2018. Learning Montezuma's Revenge from a Single Demonstration. arXiv preprint arXiv:1812.03381.
Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve Young. 2007. Agenda-Based
User Simulation for Bootstrapping a POMDP Dialogue System. In Proc. NAACL (Short Papers), pages 149-152.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015. Prioritized Experience Replay. arXiv preprint arXiv:1511.05952.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Pararth Shah, Dilek Hakkani-Tür, Gokhan Tur, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a Conversational Agent Overnight with Dialogue Self-Play. arXiv preprint arXiv:1801.04871.
Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously Learning Neural Dialogue Management. arXiv preprint arXiv:1606.02689.
Richard S Sutton and Andrew G Barto. 2018. Reinforcement Learning: An Introduction. MIT press.
Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. Guided Dialog Policy Learning: Reward Estimation for Multi-Domain Task-Oriented Dialog. arXiv preprint arXiv:1908.10719.
Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep Reinforcement Learning with Double Q-Learning. In Proc. of AAAI 2016.
Marilyn A Walker. 2000. An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email. JAIR, 12:387-416.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Freitas. 2015. Dueling Network Architectures for Deep Reinforcement Learning. CoRR, abs/1511.06581.
Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017. Hybrid Code Networks: Practical and Efficient End-to-End Dialog Control with Supervised and Reinforcement Learning. arXiv preprint arXiv:1702.03274.
Steve J. Young, Milica Gasic, Blaise Thomson, and Jason D. Williams. 2013. POMDP-Based Statistical Spoken Dialog Systems: A Review. Proc. of IEEE, 101(5):1160-1179.
A Model Hyperperameters
Below we list the hyperparameters used for our reinforcement learning agents and the expert models used to generate demonstrations.
For No Label Expert RoFL we treat dialogs with a final reward  $r > = th$  as positive, and those with reward  $r < th$  as negative examples, and treat the individual user-agent utterance pairs accordingly.
General Hyperparameters Unless otherwise stated in specific expert sections, all of our agents use below hyperparameters, where applicable:
Steps	2,500,000
Pre-training steps d	2,000 dialogs
ε start value	0.1
ε end value	0.01
ε decay rate	Linear over 500,000 steps
Discount factor γ	0.9
Policy net π	1x100d hidden layer, ReLU activation
Learning rate	0.01
Target network update period τ	10,000 steps
L2 reg. weight	10-5
Max replay size	100,000
Prioritized replay α	0.6
Prioritized replay εp	0.001
Prioritized replay εd	1.0
Prioritized replay β0	0.4
Full Label Expert
Input	392d binary state
Expert network	1x150d hidden layer,ReLU, dropout = 0.1
Output	Single-label softmax over 300 actions
Penalty c	0.8
Reward threshold th	70
Finetune interval k	2,000 steps
Reduced Label Expert
Input	300d pre-trained GloVe embeddings
Expert network	1x128d GRU layer, in-put dropout = 0.1
Output	Multi-label sigmoid of 3 reduced actions, with threshold = 0.5
Pre-training steps d	3,000 dialogs
ε start value	0.2
Penalty c	1.0
Reward threshold th	70
Finetune interval k	15,000 steps
No Label Expert
Input	BERTbase-uncased em-beddings for user and agent utterances, concatenated 1536d
Expert network	6 linear layers, 512, 256, 128, 64, 32, 16 dims, SELU activations
Output	Binary output of response appropriateness with threshold ρ = 0.9
Penalty c	0.8
Reward threshold th	70
Finetune interval k	15,000 steps