SlowGuess/ACL_2020_OCR huggingface dataset:
We also created domain templates for each domain in MultiWOZ. The number of templates and other characteristics of our synthesis are shown in Table 2. To simulate a zero-shot environment in which training data is not available, we derived the templates from only the validation data of that domain. We did not look at in-domain training data to design the templates, nor did we look at any test data until the results reported here were obtained. In the table, we also include the domain we chose to perform domain adaptation (Section 3.5) and the number of slots from the adapted domain that are applicable to the new domain.
Note that the validation and test sets are the same datasets as the MultiWOZ 2.1 release.
5.2 Evaluation On All Domains
Our first experiment evaluates how our synthesized data affects the accuracy of TRADE and SUMBT on the full MultiWOZ 2.1 dataset. As in previous work (Wu et al., 2019), we evaluate the Joint Accuracy and the Slot Accuracy. Joint Accuracy measures the number of turns in which all slots are predicted correctly at once, whereas Slot Accuracy measures the accuracy of predicting each slot individually, then averages across slots. Slot Accuracy is significantly higher than Joint Accuracy because, at any turn, most slots do not appear, hence predicting an empty slot yields high accuracy for each slot. Previous results were reported on the MultiWOZ 2.0 dataset, so we reran all models on MultiWOZ 2.1.
Results are shown in Table 3. We observe that our synthesis technique, which is derived from the MultiWOZ dataset, adds no value to this set. We obtain almost identical slot accuracy, and our joint accuracy is within the usual margin of error compared to training with the original dataset. This is a sanity-check to make sure our augmentation method generates compatible data and training on it does not worsen the results.
5.3 Zero-Shot Transfer Learning
Before we evaluate zero-shot learning on new domains, we first measure the accuracy obtained for each domain when trained on the full dataset. For each domain, we consider only the subset of dialogues that include that particular domain and only consider the slots for that domain when calculating the accuracy. In other words, suppose we have a dialogue involving an attraction and a restaurant: a prediction that gets the attraction correct but not the restaurant will count as joint-accurate for the attraction domain. This is why the joint accuracy of individual domains is uniformly higher than the joint accuracy of all the domains. Table 4 shows that the joint accuracy for TRADE varies from domain to domain, from  $50.5\%$  for "Hotel" to  $74.0\%$  for "Train". The domain accuracy with the SUMBT model is better than that of TRADE by between  $1\%$  and  $4\%$  for all domains, except for "Taxi" where it drops by about  $4.5\%$ .
In our zero-shot learning experiment, we withhold all dialogues that refer to the domain of interest from the training set, and then evaluate the joint and slot accuracies in the same way as before. The joint accuracy with the TRADE model is poor throughout except for  $59.2\%$  for "Taxi". The rest of the domains have a joint accuracy ranging from  $16.4\%$  for "Restaurant" to  $22.9\%$  for "Train". Upon closer examination, we found that simply predicting "empty" for all slots would yield the same joint accuracy. The zero-shot results for SUMBT are almost identical to that of TRADE.
A different evaluation methodology is used by Wu et al. (2019) in their zero-shot experiment. The model for each domain is trained with the full dataset, except that all the slots involving the domain of interest are removed from the dialogue state. The slots for the new domain are present in the validation and test data, however. The method they use, which we reproduce here², has consistently higher slot accuracy, but slightly worse joint accuracy than our baseline, by 1.9% to 5.8%, except for "Taxi" which improves by 1% to 60.2%.
To evaluate our proposed technique, we add our synthesized data for the domain of interest to the training data in the zero-shot experiment. Besides synthesizing from templates, we also apply domain adaptation. The pairs of domain chosen for
$^{2}$ Wu et al. (2019) reported results on MultiWOZ 2.0, while we report MultiWOZ 2.1. The results on the two datasets are all within  $3\%$  of each other.
Model	Training	Attraction	Hotel	Restaurant	Taxi	Train
Joint	Slot	Joint	Slot	Joint	Slot	Joint	Slot	Joint	Slot
TRADE	Full dataset	67.3	87.6	50.5	91.4	61.8	92.7	72.7	88.9	74.0	94.0
Zero-shot	22.8	50.0	19.5	62.6	16.4	51.5	59.2	72.0	22.9	48.0
Zero-shot (Wu)	20.5	55.5	13.7	65.6	13.4	54.5	60.2	73.5	21.0	48.9
Zero-shot (DM)	34.9	62.2	28.3	74.5	35.9	75.6	65.0	79.9	37.4	74.5
Ratio of DM over full (%)	51.9	71.0	56.0	81.5	58.1	81.6	89.4	89.9	50.5	79.3
SUMBT	Full dataset	71.1	89.1	51.8	92.2	64.2	93.1	68.2	86.0	77.0	95.0
Zero-shot	22.6	51.5	19.8	63.3	16.5	52.1	59.5	74.9	22.5	49.2
Zero-shot (DM)	52.8	78.9	36.3	83.7	45.3	82.8	62.6	79.4	46.7	84.2
Ratio of DM over full (%)	74.3	88.6	70.1	90.8	70.6	88.9	91.8	92.3	60.6	88.6
Table 4: Accuracy on the zero-shot MultiWOZ experiment (test set), with and without data augmentation. TRADE refers to Wu et al. (2019), SUMBT to Lee et al. (2019). "Zero-shot" results are trained by withholding in-domain data. "Zero-shot (Wu)" results are obtained with the unmodified TRADE zero-shot methodology, trained on MultiWOZ 2.1. "Zero-shot (DM)" refers to zero-shot learning using our Dialogue-Model based data synthesis. The last line of each model compares DM with full training, by calculating the  $\%$  of the accuracy of the former to the latter.
adaptation are shown in Table 2, together with the number of slot names that are common to both domains. "Taxi" uses a subset of the slot names as "Train" but with different values. "Attraction", "Restaurant" and "Hotel" share the "name" and "area" slot; "Restaurant" and "Hotel" also share the "price range", "book day", "book time" and "book people" slots. For slots that are not shared, the model must learn both the slot names and slot values exclusively from synthesized data.
Our dialogue-model based zero-shot result, reported as "Zero-shot (DM)" in Table 4, shows that our synthesized data improves zero-shot accuracy on all domains. For TRADE, the joint accuracy improves between  $6\%$  on "Taxi" and  $19\%$  on "Restaurant", whereas for SUMBT, joint accuracy improves between  $3\%$  on "Taxi" and  $30\%$  on "Attraction". With synthesis, SUMBT outperforms TRADE by a large margin. Except for "Taxi" which has uncharacteristically high joint accuracy of  $65\%$ , SUMBT outperforms TRADE from  $8\%$  to  $18\%$ . This suggests SUMBT can make better use of synthesized data.
To compare synthesized with real training data, we calculate how close the accuracy obtained with the synthetic data gets to full training. We divide the accuracy of the former with that of the latter, as shown in the last row for each model in Table 4.
Overall, training with synthesized data is about half as good as full training for TRADE, but is 2/3 as good as for SUMBT (the ratio is  $61\%$  to  $74\%$ , ignoring "Taxi" as an outlier). This suggests that our synthesis algorithm is generating a reasonable variety in the dialogue flows; the pretrained BERT model, which imbues the model with general knowledge of the English language, is better
Figure 3: Breakdown of accuracy by turn number and number of slots of the TRADE model on the "Restaurant" domain. "Zero-shot" results are trained by withholding in-domain data, and "Zero-shot (DM)" is our data synthesis based on the Dialogue Model. "Full dataset" refers to training with all domains.
at compensating for the lack of language variety in synthesized data. Thus, the model only needs to learn the ontology and domain vocabulary from the synthesized data. Conversely, TRADE has no contextual pretraining and must learn the language from the limited dialogue data. This suggests that the combination of unsupervised pretraining and training on synthesized data can be effective to bootstrap new domains.
To analyze the errors, we break down the result according to the turn number and number of slots in the dialogues in the test set, as shown in Fig. 3. We perform this analysis using the TRADE model on the "Restaurant" domain, which is the largest domain in MultiWOZ. We observe that the baseline model achieves  $100\%$  accuracy for turns with no slots, and  $0\%$  accuracy otherwise. The baseline results in the turn-number plot thus indicate
Figure 4: Accuracy plots for the few-shot MultiWOZ experiments. X axis indicates the percentage of real target domain data included in training. Y axis indicates joint accuracy.
the percentage of dialogues with all empty slots at each turn. It is possible for 5-turn dialogues to have all empty slots because a multi-domain dialogue may not have filled any slot in one domain.
By and large, the accuracy degrades for both the "full dataset" model and the "zero-shot (DM)" model, with the latter losing more accuracy than the former when there are 3 or 4 slots. The accuracy drops almost linearly with increasing turn numbers for the full model. This is expected because a turn is considered correct only if the full dialogue state is correct, and the state accumulates all slots mentioned up to that point. The results for the full and the zero-shot (DM) models look similar, but the zero-shot model has a larger drop in later turns. Modeling the first few turns in the dialogue is easier, as the user is exclusively providing information, whereas in later turns more interactions are possible, some of which are not captured well by our dialogue model.
5.5 Few-Shot Transfer Learning
Following Wu et al. (2019), we also evaluate the effect of mixing a small percentage of real training data in our augmented training sets. We use a naive few-shot training strategy, where we directly add a portion of the original training data in the domain of interest to the training set.
Fig. 4 plots the joint accuracy achieved on the new domain with the addition of different percentages of real training data. The results for  $0\%$  are the same as the zero-shot experiment. The advantage of the synthesized training data decreases as the percent of real data increases, because real data is more varied, informative, and more representative of the distribution in the test set. The impact of synthesized data is more pronounced for SUMBT than TRADE for all domains even with  $5\%$  real data, and it is significant for the "Attraction" domain with  $10\%$  real data. This suggests that SUMBT needs more data to train, due to
having more parameters, but can utilize additional synthesized data better to improve its training.
We propose a method to synthesize dialogues for a new domain using an abstract dialogue model, combined with a small number of domain templates derived from observing a small dataset. For transaction dialogues, our technique can bootstrap new domains with less than 100 templates per domain, which can be built in a few person-hours. With this little effort, it is already possible to achieve about  $2/3$  of the accuracy obtained with a large-scale human annotated dataset. Furthermore, this method is general and can be extended to dialogue state tracking beyond transactions, by building new dialogue models.
We show improvements in joint accuracy in zero-shot and few-shot transfer learning for both the TRADE and BERT-based SUMBT models. Our technique using the SUMBT model improves the zero-shot state of the art by  $21\%$  on average across the different domains. This suggests that pretraining complements the use of synthesized data to learn the domain, and can be a general technique to bootstrap new dialogue systems.
We have released our algorithm and dialogue model as part of the open-source Genie toolkit, which is available on GitHub<sup>3</sup>.
This work is supported in part by the National Science Foundation under Grant No. 1900638; Mehrad Moradshahi is supported by a Stanford Graduate Fellowship.
$^{3}$ https://github.com/stanford-oval/genie-toolkit
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Ultes Stefan, Ramadan Osman, and Milica Gasic. 2018. MultiWOZ - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Harry Bunt, Volha Petukhova, and Alex Chengyu Fang. 2017. Revisiting the ISO standard for dialogue act annotation. In Proceedings of the 13th Joint ISO-ACL Workshop on Interoperable Semantic Annotation (ISA-13).
Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam. 2019. Genie: A generator of natural language semantic parsers for virtual assistant commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI 2019, pages 394-410, New York, NY, USA. ACM.
Jianpeng Cheng, Siva Reddy, and Mirella Lapata. 2018. Building a neural semantic parser from a domain ontology. arXiv preprint arXiv:1812.10037.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyag Gao, and Dilek Hakkani-Tur. 2019. MultiWOZ 2.1: Multi-domain dialogue state corrections and state tracking baselines. arXiv preprint arXiv:1907.01669.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. 2018. Semantic parsing for task oriented dialog using hierarchical representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2787-2792.
Charles T Hemphill, John J Godfrey, and George R Doddington. 1990. The ATIS spoken language systems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Robust dialog state tracking using delexicalised recurrent neural networks and unsupervised adaptation. In 2014 IEEE Spoken Language Technology Workshop (SLT), pages 360-365. IEEE.
Drew A. Hudson and Christopher D. Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In The IEEE
Conference on Computer Vision and Pattern Recognition (CVPR).
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. 2017a. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017b. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Daniel Jurafsky, Elizabeth Shriberg, and Debra Bíasca. 1997. Switchboard SWBD-DAMSL shallow-discourse-function annotation coders manual. Technical report, University of Colorado, Institute of Cognitive Science.
John F Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS), 2(1):26-41.
Brenden Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International Conference on Machine Learning, pages 2879-2888.
Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5478-5483.
Nikola Mrkšić, Diarmuid Řeaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788.
Elnaz Nouri and Ehsan Hosseini-Asl. 2018. Toward scalable neural dialogue state tracking model. arXiv preprint arXiv:1812.00899.
Pararth Shah, Dilek Hakkani-Tür, Gokhan Tur, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871.
Yu Su, Ahmed Hassan Awadallah, Madian Khabsa, Patrick Pantel, Michael Gamon, and Mark Encarnacion. 2017. Building natural language interfaces to web APIs. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management - CIKM '17. ACM Press.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332-1342. Association for Computational Linguistics.
Zhuoran Wang and Oliver Lemon. 2013. A simple and generic belief tracking mechanism for the dialog state tracking challenge: On the believability of observed information. In Proceedings of the SIG-DIAL 2013 Conference, pages 423-432.
Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 438-449.
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merrienboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698.
Jason Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse, 7(3):4-33.
Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachandran. 2014. The dialog state tracking challenge series. AI Magazine, 35(4):121-124.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808-819.
Puyang Xu and Qi Hu. 2018. An end-to-end approach for handling unknown slot values in dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1448-1457.
Silei Xu, Giovanni Campagna, Jian Li, and Monica S Lam. 2020. Schema2qa: Answering complex queries on the structured web with a neural model. arXiv preprint arXiv:2001.05609.
Dian Yu and Zhou Yu. 2019. MIDAS: A dialog act annotation scheme for open domain human machine spoken conversations. arXiv preprint arXiv:1908.10023.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, et al. 2019. CoSQL: A conversational Text-to-SQL challenge towards cross-domain natural language interfaces to databases. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1962-1979.
Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, pages 1-10.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103.
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Global-locally self-attentive encoder for dialogue state tracking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1458-1467.
Li Zhou and Kevin Small. 2019. Multi-domain dialogue state tracking as dynamic knowledge graph enhanced question answering. arXiv preprint arXiv:1911.06192.