SlowGuess/ACL_2020_OCR huggingface dataset:
the attraction domain surpasses the accuracy with training MERET from scratch using  $1\%$  (30 dialogues) of the attraction domain data. This demonstrates our model's capability to achieve good performance with a fraction of the target data.
We analyze the wrong predictions and draw a heat map of distributions for the slot classifier considering the importance of its determining to the final output. From the map in Figure 4, we can see the main cause of the error-maker is the classifier's inertia of omit-prediction from ptr to none, which stands up to  $47.3\%$  proportion. The over-prediction cause comes in the next, with a  $27.3\%$  rate. Value on the diagonal of the lower-left corner shows the mis-prediction rate of the generator. Combined with the comparison of the two pictures, we can get the point that our proposed model has a higher generative ability over state value.
An overview correct-error analysis of multidomain for slots is shown in Figure 5. The number-related slots book stay in hotel domain and book day in restaurant domain have the highest correct rates,  $98.97\%$  and  $98.94\%$ , respectively. The name-related slots in the restaurant, attraction, and hotel domains have the highest error rates,  $8.94\%$ ,  $7.36\%$ , and  $7.21\%$ , respectively. It is because these slots usually have a large number of possible values set and high annotation errors. The type slot of hotel domain also keeps a high error rate in different experiments, even if it is an easy task with only two possible values in the ontology. The reason is that labels of the (hotel, type) pair are usually missing in the dataset. We further show the performance of our model over different dialogue turn in Figure 6. As the number of dialogue turn increases,
7115
User:	I'm looking for a jamaican restaurant in the east.
System:	There are no jamaican restaurants in the east. Would you like to try another food type or area?
User:	I'm looking for a place that serves jamaican food in the east. If not, italian will do.
System:	There is one Italian place in the east, Pizza Hut Fen Ditton.
TRADE prediction:	{(restaurant, area, east), (restaurant, food, jamaican)}
MERET prediction:	{(restaurant, area, east), (restaurant, food, italian)}
Table 3: Case study for state Generator. We can find that with the same context, MERET outperforms TRADE in terms of state generation for DST.
the influence of context gradually appears for the final results due to the abilities of different models. We can see that MERET outperforms TRADE gradually. This is especially true when the context length is long. Our model can carry information over multiple turns which will be used for state generator with the help of RL maximizing rewards expectations in a better way. We sample one typical dialogue from MultiWOZ to demonstrate the effectiveness of MERET in the case study. Due to limited space, we present the same key parts derived from two models and the details are shown in Table 3. We observe that the constraint for food slot is dynamic and MERET is sensitive to capture this context information with the advantage of RL-based fine-tune state Generator, which reinforces in greater exploration for DST and maximizes reward expectation in a better way.
Mrkšić et al. (2017) propose neural belief tracking (NBT) framework without relying on hand-crafted semantic lexicons. The model uses Convolutional Neural Networks (CNN) or Deep Neural Networks (DNN) as dialogue context encoder and makes a binary decision for (slot, value) pairs. Zhong et al. (2018) propose global-local modules to learn representations of the user utterance and system actions and calculate similarity between the contextualized representation and the (slot, value) pair. Xu and Hu (2018) utilize pointer network to track dialogue state, which proposes a conception of unseen states and unknown states earlier. Chao and Lane (2019) use BERT as dialogue context encoder and get contextualized representation, which is passed to the classification module and get three classes: none, dontcare, span. When the class is span, the start and end positions of slot values are obtained in the dialogue context. However, Both Xu and Hu
(2018) and Chao and Lane (2019) suffers from the fact that they can not get correct answers when the value does not exist in the input. Wu et al. (2019) propose an approach that the model generates a sequence of values from utterances by copy mechanism, which can avoid the case that the value is not in the input. It also uses a three-way classifier to get a probability distribution over none, dontcare, ptr classes. Ren et al. (2019) achieve state-of-the-art performance on the MultiWOZ dataset by applying a hierarchical encoder-decoder structure for generating a sequence of belief states. The model shares parameters and has a constant inference time complexity.
Reinforcement learning is a way of training an agent during interaction with the environment by maximizing expected reward. The idea of policy gradient algorithm has been applied in training of sequence to sequence model. Ranzato et al. (2016) propose MIXER algorithm, which is the first application of REINFORCE algorithm (Williams, 1992) in training sequence to sequence model. However, an additional model, which is used to predict expected reward, is required in MIXER. Rennie et al. (2017) proposed a self-critical method for sequence training (SCST). It directly optimizes the true, sequence-level, evaluation metric, and avoids the training of expected future rewards estimating model. Paulus et al. (2018) applied SCST in summary generation, which improved the rouge value of generated result. SCST algorithm was also used by Zhao et al. (2018) for improving story ending generation. Keneshloo et al. (2018) present some of the most recent frameworks that combine concepts from RL and deep neural networks and explain how these two areas could benefit from each other in solving complex seq2seq tasks.
Meta-learning aims at learning target tasks with little data based on source tasks. This algorithm is
7116
compatible with any model optimized with gradient descent so that it has a wide range of applicability. Meta-learning has been applied in various fields such as image classification (Santoro et al., 2016; Finn et al., 2017) and robot manipulation (Duan et al., 2016; Wang et al., 2016), etc. In the field of natural language processing, some exploratory work (Gu et al., 2018; Huang et al., 2018; Qian and Yu, 2019; Madotto et al., 2019) have been proposed in recent years. Most of them are focused on the generation-related tasks and machine translation. To our knowledge, few related work in dialogue state tracking (DST) was found till now. We propose to apply model-agnostic meta-learning (MAML) (Finn et al., 2017) algorithm for training a DST meta-learning model with a few domains as the training domains and a new domain as the testing domain to achieve multi-domain adaptation.
We introduce an end-to-end generative framework with pre-trained language model and copy-mechanism, using RL-based generator to encourage higher semantic relevance in greater exploration space for DST. Experiments on multi-domain dataset show that our proposed model achieves state-of-the-art performance on the DST task, exceeding current best result by over  $2\%$ . In addition, we train the dialogue state tracker using multiple single-domain dialogue data with rich-resource by using the MAML. The model is capable of learning a competitive and scalable DST on a new domain with only a few training examples in an efficient manner. Empirical results on MultiWOZ datasets indicate that our solution outperforms non-meta-learning baselines training from scratch, adapting to new few-shot domains with less data and faster convergence rate.
In future work, we intend to explore more with the combination of RL and DST on the basis of reward designing, trying to explore more in the internal mechanism. In the long run, we are interested in combing many tasks into one learning process with meta-learning.
We thank Zhiqiang Yang and Chao Deng for their insightful discussion and great support. We also thank all anonymous reviewers for their constructive comments.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - a large-scale multi-domain wizard-of-Oz dataset for task-oriented dialogue modelling. pages 5016-5026.
Guan-Lin Chao and Ian Lane. 2019. Bert-dst: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer. arXiv:1907.03040.
Junyoung Chung, Caglar Gülcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171-4186.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. 2016. Rl<sup>2</sup>: Fast reinforcement learning via slow reinforcement learning. CoRR, abs/1611.02779.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1126-1135.
Rahul Goel, Shachi Paul, and Dilek Hakkani-Tur. 2019. Hyst: A hybrid approach for flexible and accurate dialogue state tracking. CoRR, abs/1907.00883.
Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. Meta-learning for low-resource neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622-3631.
Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292-299, Philadelphia, PA, U.S.A. Association for Computational Linguistics.
Matthew Henderson, Blaise Thomson, and Steve J. Young. 2013. Deep neural network approach for the dialog state tracking challenge. In Proceedings of the 14th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 467-471.
Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wen-tau Yih, and Xiaodong He. 2018. Natural language to structured query generation via meta-learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics, pages 732-738.
7117
Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, pages 448-456.
Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K. Reddy. 2018. Deep reinforcement learning for sequence to sequence models. CoRR, abs/1805.09461.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.
Mandy Korpusik and James R. Glass. 2018. Convolutional neural networks for dialogue state tracking without pre-trained word vectors or semantic dictionaries. In 2018 IEEE Spoken Language Technology Workshop, pages 884-891.
Qingbin Liu, Shizhu He, Kang Liu, Shengping Liu, and Jun Zhao. 2019. Copy-enhanced heterogeneous information learning for dialogue state tracking. CoR-R, abs/1908.07705.
Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019. Personalizing dialogue agents via meta-learning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 5454-5459.
Nikola Mrkšić, Diarmuid Žáeghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1777-1788, Vancouver, Canada. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In 6th International Conference on Learning Representations.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543.
Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 2639-2649.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In 4th International Conference on Learning Representations.
Abhinav Rastogi, Dilek Hakkani-Tur, and Larry Heck. 2017. Scalable multi-domain dialogue state tracking. In IEEE Automatic Speech Recognition and Understanding Workshop.
Liliang Ren, Jianmo Ni, and Julian J. McAuley. 2019. Scalable and accurate dialogue state tracking via a hierarchical sequence generation. CoRR, abs/1909.00754.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, pages 1179-1195.
Adam Santoro, Sergey Bartunov, Matthew Botvinick, Daan Wierstra, and Timothy P. Lillicrap. 2016. Meta-learning with memory-augmented neural networks. In Proceedings of the 33nd International Conference on Machine Learning, pages 1842-1850.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. 2016. Learning to reinforcement learn. arXiv:1611.05763.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808–819, Florence, Italy. Association for Computational Linguistics.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 1448-1457, Melbourne, Australia. Association for Computational Linguistics.
Steve Young, Milica Gasic, Blaise Thomson, and Jason D. Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proceedings of the IEEE, 101(5):1160-1179.
Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014. Recurrent neural network regularization. CoRR, abs/1409.2329.
Yan Zhao, Lu Liu, Chunhua Liu, Ruoyao Yang, and Dong Yu. 2018. From plots to endings: A reinforced pointer generator for story ending generation. In Natural Language Processing and Chinese Computing - 7th International Conference, pages 51-63.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 1458-1467, Melbourne, Australia. Association for Computational Linguistics.
7118