SlowGuess/NAACL_2021_OCR huggingface dataset:
encoding for dialog and schema are fully separated. A dialog only needed to be encoded for once during the inference of a dialog state example while the schema are precomputed once. However, for Cross-Encoder, to predict a dialog state for a single turn, it need to encode more than 300 sentence pairs in a batch, thus only processes 4.75 examples per GPU second. Fusion-Encoder performs one time encoding on dialog history, but it needs to jointly encode the same amount of dialog-schema pair ws Cross-Encoder, instead, however, with a two-layer transformer encoder. Overall it achieves 10.54 examples per GPU second, which is  $2.2\mathrm{x}$  faster than Cross-Encoder. With regarding to the accuracy in Table 3, Fusion-Encoder performs much better than Dual-Encoder, especially on unseen services.
6 Supplementary Training (Q2)
Besides the pretrain-fintune framework used in §5, Phang et al. (2018) propose to add a supplementary training phase on an intermediate task after the pretraining, but before finetuning on target task. It shows significant improvement on the target tasks. Moreover, large amount pretrained and finetuned transformer-based models are publicly accessible, and well-organized in model hubs for sharing, training and testing<sup>6</sup>. Given the new task of schemaguided dialog state tracking, in this section, we study our four subtasks with different intermediate tasks for supplementary training.
6.1 Intermediate Tasks
As described in § 5.2, all our 4 subtasks take a pair of dialog and schema description as input, and predict with the summerized sentence-pair CLS representation. While NonCat also requires span-based detection such as question answering. Hence, they share the similar problem structure with the following sentence-pair encoding tasks.
Natural Language Inference. Given a hypothesis/premise sentence pair, natural language inference is a task to determine whether a hypothesis is entailed, contradicted or neutral given that premise. Question Answering. Given a passage/question pairs, the task is to extract the span-based answer in the passage.
Hence, when finetuning BERT on our subtaks, instead of directly using the originally pretrained BERT, we use the BERT finetuned on the above
$^{6}$ e.g., Huggingface(https://huggingface.co/models) and ParlAL(https://parl.ai/docs/zoo.html), etc.
5 Appendix A.1 shows the detailed experiment setup
SG-DST	MULTIWOZ 2.2
intent	req	cat	noncat	cat	noncat
all seen	unseen	all seen	unseen	all seen	unseen	all seen	unseen	all seen	unseen	all seen	unseen	all seen
ΔSNLI	+0.51	+0.02	+0.68	-0.19	+0.38	-0.38	-1.63	-2.87	-1.23	-4.7	-0.1	-6.25	+2.05
ΔSQuAD	-1.81	-0.17	-1.32	-0.25	-0.01	-0.33	-2.87	-3.02	-5.17	+1.99	-1.79	+3.25	+0.04
Table 4: Relative performance improvement of different supplementary training on SG-DST and MULTIWOZ 2.2 dataset
two tasks for further finetuning. Due to better performance of Cross-Encoder in §5, we directly use the finetuned Cross-Encoder version of BERT models on SNLI and SQuAD2.0 dataset from Huggingface model hub. We add extra speaker tokens [user:] and [system:] into the vocabulary for encoding the multi-turn dialog histories.
6.2 Results on Supplementary Training
Table 4 shows the performances gain when finetuning 4 subtasks based on models with the above SNLI and SQuAD2.0 supplementary training.
We mainly find that SNLI helps on Intent task, SQuAD2 mainly helps on NonCat task, while neither of them helps much on Cat task. Recently, Namazifar et al. (2020) also found that when modeling dialog understanding as question answering task, it can benefit from a supplementary training on SQuAD2 dataset, especially on few-shot scenarios, which is a similar findings as our NonCat task. Result difference on Req task is minor, because it is a relatively easy task, adding any supplementary training did n't help much. Moreover, for Cat task, the sequence 2 of the input pair is the slot description with a categorical slot value, thus the meaning overlapping between the full dialog history and the slot/value is much smaller than SNLI tasks. On the other side, CLS token in SQuAD BERT is finetuned for null predictions via start and end token classifiers, which is different from the single CLS classifier in Cat task.
7 Impact of Description Styles (Q3)
Previous work on schema-guided dialog (Rastogi et al., 2020) are only based on the provided descriptions in SG-DST dataset. Recent work on modeling dialog state tracking as reading comprehension (Gao et al., 2019) only formulate the descriptions as simple question format with existing intent/slot names, it is unknown how it performs when compared to other description styles. Moreover, they only conduct homogeneous evaluation where training and test data share the same descrip
tion style. In this section, We also investigate how a model trained on one description style will perform on other different styles, especially in a scenario where chat bot developers may design their own descriptions. We first introduce different styles of descriptions in our study, and then we train models on each description style and evaluate on tests with corresponding homogeneous and heterogeneous styles of descriptions. Given the best performance of Cross-Encoder shown in the previous section and its popularity in DSTC8 challenges, we adopt it as our model architecture in this section.
7.1 Benchmarking Styles
For each intent/slot, we describe their functionalities by the following different descriptions styles:
Identifier. This is the least informative case of name-based description: we only use meaningless intent/slot identifiers, e.g. Intent_1, Slot_2. It means we don't use description from any schema component. We want to investigate how a simple identifier-based description performs in schemaguided dialog modeling, and the performance lower-bound on transferring to unseen services.
NameOnly. Using the original intent/slot names in SG-DST and MULTIWOZ 2.2 dataset as descriptions, to show whether name is enough for schema-guided dialog modeling.
Q-Name. This is corresponding to previous work by Gao et al. (2019). For each intent/slot, it generate a question to inquiry about the intent and slot value of the dialog. For each slot, it simply follows the template 'What is the value for slot i?' Besides that, our work also extend the intent description by following the template 'Is the user intending to intent j'.
Orig. The original descriptions in SG-DST and MULTIWOZ 2.2 dataset.
Q-Orig. Different from the Q-Name, firstly it is based on the original descriptions; secondly, rather than always use the "what is" template to inquiry the intent/slot value, We add "what", "which", "how many" or "when" depending on the entity type required for the slot. Same as Q-Name, we just add
prefixes as "Is the user intending to... " in front of the original description. In a sum, this description is just adding question format to original description. The motivation of this description is to see whether the question format is helpful or not for schema-guided dialog modeling.
To test the model robustness, we also create two paraphrased versions Name-Para and Orig-Para for NameOnly and Orig respectively. We first use nematus (Sennrich et al., 2017) to automatically paraphrase the description with back translation, from English to Chinese and then translate back, then we manually check the paraphrase to retain the main meaning. Appendix A.5.1 shows examples for different styles of schema descriptions.
7.2 Results on Description Styles
Unlike the composition used in Table 2, we don't use the service description to avoid its impact. For each style, we train separate models on 4 subtasks, then we evaluate them on different target styles. First, Table 5 summarizes the performance for homogeneous evaluation, while Table 6 shows how the question style description can benefit from SQuAD2 finetuning. Then we also conduct heterogeneous evaluation on the other styles<sup>7</sup> as shown in Table 7.
Style\Task	SG-DST	MULTIWOZ 2.2
Intent	Req	Cat	NonCat	Cat	NonCat
Identifier	61.16	91.48	62.47	30.19	34.25	52.28
NameOnly	94.24	98.84	74.01	75.63	53.72	56.18
Q-Name	93.31	98.86	74.36	74.86	54.19	56.17
Orig	93.01	98.55	74.51	75.76	52.19	57.20
Q-Orig	93.42	98.51	76.64	76.60	53.61	57.80
Table 5: Homogeneous evaluation results of different description style on SG-DST dataset and MULTIWOZ 2.2 datasets. The middle horizontal line separates the two name-based descriptions and two rich descriptions in our settings. All numbers in the table are mixed performance including both seen and unseen services.
7.2.1 Homogeneous Evaluation
Is name-based description enough? As shown in Table 5, Identifier is the worst case of using name description, its extremely bad performance indicates name-based description can be very unstable. However, we found that simple meaningful name-based description actually can perform the best in Intent and Req task, and they perform
worse on Cat and NonCat tasks comparing to the bottom two rich descriptions. After careful analysis on the intents in SG-DST datasets, we found that most services only contains two kinds of intents, an information retrieval intent with a name prefix "Find-," "Get-," "Search-"; another transaction intent like "Add-," "Reserve- or "Buy-". Interestingly, we found that all the intent names in the original schema-guided dataset strictly follows an action-object template with a composition of words without abbreviation, such as "FindEvents", "BuyEventTickets". This simple name template is good enough to describe the core functionality of an intent in SG-DST dataset. Additionally, Req is a relatively simper task, requesting information are related to specifial attributes, such as "has_live_music", "has_wifi", where keywords co-occurred in the slot name and in the user utterance, hence rich explanation cannot help further. On the other side, rich descriptions are more necessary for Cat and NonCat task. Because in many cases, slot names are too simple to represent the functionalities behind it, for example, slot name "passengers" cannot fully represent the meaning "number of passengers in the ticket booking".
Does question format help? As shown in Table 5, when comparing row Q-Orig v.s. Orig, we found extra question format can improve the performance on Cat and NonCat task on both SG-DST and MULTIWOZ 2.2 datasets, but not for Intent and Req tasks. We believe that question format helps the model to focus more on specific entities in the dialog history. However, when adding a simple question pattern to NameOnly, comparing row Q-Name and NameOnly, there is no consistent improvement on both of the two datasets. Further more, we are curious about whether BERT finetuned on SQuAD2 (SQuAD2-BERT) can further help on the question format. Because NonCat are similar with span-based question answering, we focus on NonCat here. Table 6 shows that, after applying the supplementary training on SQuAD2 (\$6), almost all models get improved on unseen splits however slightly dropped on seen services. Moreover, comparing to Q-Name, Q
<sup>7</sup>We don't consider the meaningless Identifier style due to its bad performance
Only exception happens in Cat on MULTIWOZ 2.2. When creating MULTIWOZ 2.2 (Zang et al., 2020), the slots with less than 50 different slot values are classified as categorical slots, which leads to inconsistencies. We put detailed discuss about MULTIWOZ 2.2 in the supplementary material
This action-object template has also been found efficient for open domain intent induction task(e.g., Vedula et al., 2020, OPINE).
Style/Dataset	SG-DST	MULTIWOZ 2.2
all	seen	unseen	all	seen	unseen
Orig	+1.99	-1.79	+3.25	+1.93	-2.21	+4.27
Q-Orig	+6.13	-2.01	+8.84	+1.06	-1.28	+3.06
NameOnly	-0.45	-1.49	-0.11	+1.75	+0.58	+1.77
Q-Name	+0.05	-2.98	+1.04	-0.04	-0.32	+1.25
Orig is more similar to the natural questions in the SQuAD2, we obverse that Q-Orig gains more than Q-Name from pretrained model on SQuAD2.
7.2.2 Heterogeneous
In this subsection, we first simulate a scenario when there is no recommended description style for the future unseen services. Hence, unseen services can follow any description style in our case. We average the evaluation performance on three other descriptions and summarized in Table 7. The  $\Delta$  column shows the performance change compared to the homogeneous performance. It is not surprising that almost all models perform worse on heterogeneous styles than on homogeneous styles due to different distribution between training and evaluation. The bold number shows the best average performance on heterogeneous evaluation for each subtask. The trends are similar with the analysis in homogeneous evaluation 7.2.1, the name-based descriptions perform better than other rich descriptions on intent classification tasks. While on other tasks, the Orig description performs more robust, especially on NonCat task.
Furthermore, we consider another scenario where fixed description convention such as Name-Only and Orig are suggested to developers, they must obey the basic style convention but still can freely use their own words, such as abbreviation, synonyms, adding extra modifiers. We train each model on NameOnly and Orig, then evaluate on the corresponding paraphrased version respectively. In the last two rows of Table 7, the column 'para' shows performance on paraphrased schema, while  $\Delta$  shows the performance change compared to the homogeneous evaluation. Orig still performs more robust than NameOnly when schema descriptions get paraphrased on unseen services.
Table 6: Performance changes when using BERT finetuned on SQuAD2 dataset to further finetuning on our NonCat task.
Style\Task	SG-DST
Intent(Acc)	Req(F1)	Cat(Joint Acc)	NonCat(Joint Acc)
mean	Δ	mean	Δ	mean	Δ	mean	Δ
NameOnly	82.47	-11.47	96.92	-1.64	61.37	-5.54	56.53
Q-Name	93.27	+0.58	97.88	-0.76	68.55	+2.63	62.92
Orig	79.47	-12.70	97.42	-0.74	68.58	-0.3	66.72
Q-Orig	84.57	-8.24	96.70	-1.45	68.40	-2.89	56.17
para	Δ	para	Δ	para	Δ	para
NameOnly	92.22	-1.74	97.69	-0.87	67.39	-0.7	67.17
Orig	91.54	-0.63	98.42	+0.26	71.74	+2.86	67.68
Table 7: Results on unseen service with heterogeneous description styles on SG-DST dataset. More results and qualitative analysis are in the appendix A.5
In this paper, we studied three questions on schemaguided dialog state tracking: encoder architectures, impact of supplementary training, and effective schema description styles. The main findings are as follows:
By caching the token embedding instead of the single CLS embedding, a simple partial-attention Fusion-Encoder can achieve much better performance than Dual-Encoder, while still infers two times faster than Cross-Encoder. We quantified the gain via supplementary training on two intermediate tasks. By carefully choosing representative description styles according to recent works, we are the first of doing both homogeneous/heterogeneous evaluations for different description style in schema-guided dialog. The results show that simple name-based description performs well on Intent and Req tasks, while NonCat tasks benefits from richer styles of descriptions. All tasks suffer from inconsistencies in description style between training and test, though to varying degrees.
Our study are mainly conducted on two datasets: SG-DST and MULTIWOZ 2.2, while the speed-accuracy balance of encoder architectures and the findings in supplementary training are expected to be dataset-agnostic, because they depend more on the nature of the subtasks than the datasets. Based on our proposed benchmarking descriptions suite, the homogeneous and heterogeneous evaluation has shed the light on the robustness of cross-style schema-guided dialog modeling, we believe our study will provide useful insights for future research.
The authors wish to thank the anonymous reviewers and members of the Amazon LEX team for their valuable feedback.
Daniel G Bobrow, Ronald M Kaplan, Martin Kay, Donald A Norman, Henry Thompson, and Terry Winograd. 1977. Gus, a frame-driven dialog system. Artificial intelligence, 8(2):155-173.
Antoine Bordes, Jason Weston, and Nicolas Usunier. 2014. Open question answering with weakly supervised embedding models. In Joint European conference on machine learning and knowledge discovery in databases, pages 165-180. Springer.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. 2018. Multiwoz-a largescale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5016-5026.
Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kyu-Young Kim. 2019. Taskmaster-1: Toward a realistic and diverse dialog dataset. arXiv preprint arXiv:1909.05358.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Goyal, Peter Ku, and Dilek Hakkani-Tur. 2020. Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 422-428.
Shuyang Gao, Sanchit Agarwal, Tagyoung Chung, Di Jin, and Dilek Hakkani-Tur. 2020. From machine reading comprehension to dialogue state tracking: Bridging the gap. arXiv preprint arXiv:2004.05827.
Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, Dilek Hakkani-Tur, and Amazon Alexa AI. 2019. Dialog state tracking: A neural reading comprehension approach. In 20th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 264.
Ting Han, Ximing Liu, Ryuichi Takanobu, Yixin Lian, Chongxuan Huang, Wei Peng, and Minlie Huang. 2020. Multiwoz 2.3: A multi-domain task-oriented dataset enhanced with annotation corrections and coreference annotation. arXiv e-prints, pages arXiv-2010.
Matthew Henderson, Blaise Thomson, and Steve Young. 2014. Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pages 292-299.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00796.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2019. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In ICLR.
John F Kelley. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS), 2(1):26-41.
Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang-Woo Lee. 2019. Efficient dialogue state tracking by selectively overwriting memory. arXiv preprint arXiv:1911.03906.
Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim. 2019. Sumbt: Slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5478-5483.
Ryan Lowe, Nissan Pow, Iulian Vlad Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 285-294.
Nikola Mrkšić, Diarmuid Žáeghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2017. Neural belief tracker: Data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1777-1788.
Chetan Naik, Arpit Gupta, Hancheng Ge, Mathias Lambert, and Ruhi Sarikaya. 2018. Contextual slot carry-over for disparate schemas. Proc. Interspeech 2018, pages 596-600.
Mahdi Namazifar, Alexandros Papangelis, Gokhan Tur, and Dilek Hakkani-Tur. 2020. Language model is all you need: Natural language understanding as question answering. arXiv preprint arXiv:2011.03023.
Vahid Noroozi, Yang Zhang, Evelina Bakhturina, and Tomasz Kornuta. 2020. A fast and robust bert-based dialogue state tracker for schema-guided dialogue dataset. Workshop on Conversational Systems Towards Mainstream Adoption at KDD 2020.
Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayan- deh, Lars Liden, and Jianfeng Gao. 2020. Soloist: Few-shot task-oriented dialog with a single pretrained auto-regressive model. arXiv, pages arXiv- 2005.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. AAAI 2019.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Schemaguided dialogue state tracking task at dstc8. Workshop on DSTC8, AAAI 2020.
Rico Sennrich, Orhan First, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Läubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nădejde. 2017. Nematus: a toolkit for neural machine translation. In Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 65-68, Valencia, Spain. Association for Computational Linguistics.
Pararth Shah, Dilek Hakkani-Tür, Bing Liu, and Gokhan Tur. 2018a. Bootstrapping a neural conversational agent with dialogue self-play, crowdsourcing and on-line reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 41-51, New Orleans - Louisiana. Association for Computational Linguistics.
Pararth Shah, Dilek Hakkani-Tür, Gokhan Tur, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018b. Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:1801.04871.
Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2020. Open intent extraction from natural language interactions. In Proceedings of The Web Conference 2020, pages 2009-2020.
Jason D Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. Dialogue & Discourse, 7(3):4-33.
Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 808-819.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 496-505.
Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W Bruce Croft, Jun Huang, and Haiqing Chen. 2018. Response ranking with deep matching networks and external knowledge in information-seeking conversation systems. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 245-254.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 109-117, Online. Association for Computational Linguistics.
Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. 2019. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. arXiv, pages arXiv-1910.