Venue: NAACL 2021

# A Comparative Study on Schema-Guided Dialogue State Tracking

Jie Cao &Yi Zhang

School of Computing, University of Utah jcao@cs.utah.edu &Work done when Jie Cao was an intern at Amazon

###### Abstract

Frame-based state representation is widely used in modern task-oriented dialog systems to model user intentions and slot values. However, a fixed design of domain ontology makes it difficult to extend to new services and APIs. Recent work proposed to use natural language descriptions to define the domain ontology instead of tag names for each intent or slot, thus offering a dynamic set of schema. In this paper, we conduct in-depth comparative studies to understand the use of natural language description for schema in dialog state tracking. Our discussion mainly covers three aspects: encoder architectures, impact of supplementary training, and effective schema description styles. We introduce a set of newly designed bench-marking descriptions and reveal the model robustness on both homogeneous and heterogeneous description styles in training and evaluation.

## 1 Introduction

From early frame-driven dialog system GUS [2] to virtual assistants Alexa, Siri, and Google Assistant _et al._), frame-based dialog state tracking has long been studied to meet various challenges. In particular, how to support an ever-increasing number of services and APIs spanning multiple domains has been a focal point in recent years, evidenced by multi-domain dialog modeling [3, 4, 5] and transferable dialog state tracking to unseen intent/slots [8, 9, 7, 6].

Recently, Rastogi et al. rastogi proposed a new paradigm called schema-guided dialog for transferable dialog state tracking by using natural language description to define a dynamic set of service schemata. As shown in Figure 1, the primary motivation is that these descriptions can offer effective knowledge sharing across different services, e.g., connecting semantically similar concepts across heterogeneous APIs, thus allowing a unified model to handle unseen services and APIs. With the publicly available schema-guided dialog dataset (SGDst henceforward) as a testbed, they organized a state tracking shared task composed of four subtasks: intent classfication (Intent), requested slot identification (Req), categorical slot labeling (Cat), and noncategorical slot labeling (NonCat) [10]. Many participants achieved promising performance by exploiting the schema description for dialog modeling, especially on unseen services.

Despite the novel approach and promising results, current schema-guided dialog state tracking

Figure 1: An example dialog from Restaurant_1 service, along with its service/intent/slot descriptions and dialog state representation.



task only evaluates on a single dataset with limited variation in schema definition. It is unknown how this paradigm generalizes to other datasets and other different styles of descriptions. In this paper, we focus our investigation on the study of three aspects in schema-guided dialog state tracking: (1) schema encoding model architectures (2) supplementary training on intermediate tasks (3) various styles for schema description. To make a more general discussion on the schema-guided dialog state tracking, we perform extensive empirical studies on both SG-dst and MultiWOZ 2.2 datasets. In summary, our contributions include:

* A comparative study on schema encoding architectures, suggesting a partial-attention encoder for good balance between inference speed and accuracy.
* An experimental study of supplementary training on schema-guided dialog state tracking, via intermediate tasks including natural language inference and question answering.
* An in-depth analysis of different schema description styles on a new suite of benchmarking datasets with variations in schema description for both SG-dst and MultiWOZ 2.2.

## 2 Schema-Guided Dialog State Tracking

A classic dialog state tracker predicts a dialog state frame at each user turn given the dialog history and predefined domain ontology. As shown in Figure 1, the key difference between schema-guided dialog state tracking and the classic paradigm is the newly added natural language descriptions. In this section, we first introduce the four subtasks and schema components in schema-guided dialog state tracking, then we outline the research questions in our paper.

**Subtasks.** As shown in Figure 1, the dialog state for each service consists of 3 parts: _active intent, requested slots, user goals (slot values)_. Without loss of generality, for both SG-dst and MultiWOZ 2.2 datasets, we divide their slots into categorical and non-categorical slots by following previous study on dual-strategies Zhang et al. (2019). Thus to fill the dialog state frame for each user turn, we solve four subtasks: intent classification (Intent), requested slot identification (Req), categorical slot labeling (Cat), and non-categorical slot labeling (NonCat). All subtasks require matching the current dialog history with candidate schema descriptions for multiple times.

**Schema Components.** Figure 1 shows three main schema components: service, intent, slot. For each intent, the schema also describes _optional_ or _required_ slots for it. For each slot, there are flags indicating whether it is categorical or not. _Categorical_ means there is a set of predefined candidate values (Boolean, numeric or text). For instance, _has_live_music_ in Figure 1 is a categorical slot with Boolean values. _Non-categorical_, on the other hand, means the slot values are filled from the string spans in the dialog history.

**New Questions.** These added schema descriptions pose the following three new questions. We discuss each of them in the following sections.

[leftmargin=*]

Q1. How should dialogue and schema be encoded? SS5

Q2. How do different supplementary trainings impact each subtask? SS6

Q3. How do different description styles impact the state tracking performance? SS7

## 3 Related Work

Our work is related to three lines of research: multi-sentence encoding, multi-domain and transferable dialog state tracking. However, our focus is on the comparative study of different encoder architectures, supplementary training, and schema description style variation. Thus we adopt existing strategies from multi-domain dialog state tracking.

**Multi-Sentence Encoder Strategies.** Similar to the recent study on encoders for response selection and article search tasks Humeau et al. (2019), we also conduct our comparative study on the two typical architectures _Cross-Encoder_Bordes et al. (2014); Lowe et al. (2015) and _Dual-Encoder_Wu et al. (2017); Yang et al. (2018). However, they only focus on sentence-level matching tasks. All subtasks in our case require sentence-level matching between dialog context and each schema, while the non-categorical slot filling task also needs to produce a sequence of token-level representation for span detection. Hence, we study multi-sentence encoding for both sentence-level and token-level tasks. Moreover, to share the schema encoding across subtasks and turns, we also introduce a simple _Fusion-Encoder_ by caching schema token embeddings in SS5.1, which improves efficiency without sacrificing much accuracy.

**Multi-domain Dialog State Tracking.** Recent research on multi-domain dialog system have been largely driven by the release of large-scale 

multi-domain dialog datasets, such as MultiWOZ (Budzianowski et al., 2018), M2M (Shah et al., 2018a), accompanied by studies on key issues such as in/cross-domain carry-over (Kim et al., 2019). In this paper, our goal is to understanding the design choice for schema descriptions in dialog state tracking. Thus we simply follow the inhomain cross-over strategies used in **TRADE**Wu et al. (2019). Additionally, explicit cross-domain carryover Naik et al. (2018) is difficult to generalize to new services and unknown carryover links. We use longer dialog history to inform the model on the dialog in the previous service. This simplified strategy does impact our model performance negatively in comparison to a well-designed dialog state tracking model on seen domains. However, it helps reduce the complexity of matching extra slot descriptions for cross-service carryover. We leave the further discussion for future work.

Transferable Dialog State TrackingAnother line of research focuses on how to build a transferable dialog system that is easily scalable to newly added intents and slots. This covers diverse topics including e.g., resolving lexical/morphological variabilities by symbolic de-lexicalization-based methods Henderson et al. (2014); Williams et al. (2016), neural belief tracking Mrksic et al. (2017), generative dialog state tracking Peng et al. (2020); Hosseini-Asl et al. (2020), modeling DST as a question answering task Zhang et al. (2019); Lee et al. (2019); Gao et al. (2020, 2019). Our work is similar with the last class. However, we further investigate whether the DST can benefit from NLP tasks other than question answering. Furthermore, without rich description for the service/intent/slot in the schema, previous works mainly focus on simple format on question answering scenarios, such as domain-slot-type compounded names (e.g., "_restaurant-food_"), or simple question template "_What is the value for slot i?_". We incorporate different description styles into a comparative discussion on SS7.1.

## 4 Datasets

To the best of our knowledge, at the time of our study, SG-dst and MultiWOZ 2.2 are the only two publicly available corpus for schema-guided dialog study. We choose both of them for our study. In this section, we first introduce these two representative datasets, then we discuss the generalizability in domain diversity, function overlapping, data collecting methods.

Schema-Guided Dialog DatasetSG-dst dataset1 is especially designed as a test-bed for schema-guided dialog, which contains well-designed heterogeneous APIs with overlapping functionalities between services Rastogi et al. (2019). In DSTC8Rastogi et al. (2020), SG-dst was introduced as the standard benchmark dataset for schema-guided dialog research. SG-dst covers 20 domains, 88 intents, 365 slots.2 However, previous research are mainly conducted based on this single dataset and the provided single description style. In this paper, we further extended this dataset with other benchmarking description styles as shown in SS7, and then we perform both homogenous and heterogenous evalution on it.

Footnote 1: [https://github.com/google-research-datasets/](https://github.com/google-research-datasets/) dstc8-schema-guided-dialogue

Footnote 2: Please refer to the original paper for more details.

Remixed MultiWOZ 2.2 DatasetTo eliminate potential bias from the above single SG-dst dataset, we further add MultiWOZ 2.2 Zang et al. (2020) to our study. Among various extended versions for MultiWOZ dataset (2.0-2.3, Budzianowski et al., 2018; Eric et al., 2020; Zang et al., 2020; Han et al., 2020) , besides rectifying the annotation errors, MultiWOZ 2.2 also introduced the schema-guided annotations, which covers 8 domains, 19 intents, 36 slots. To evaluate performance on seen/unseen services with MultiWOZ, we remix the MultiWOZ 2.2 dataset to include as seen services dialogs related to _restaurant_, _attraction_ and _train_ during training, and eliminate slots from other domains/services from training split. For dev, we add two new domains _hotel_ and _taxi_ as unseen services. For test, we add all remaining domains as unseen, including those that have minimum overlap with seen services, such as _hospital_, _police_, _bus_. The statistics of data splits are shown in Appendix A.2. Note that this data split is different from the previous work on zero-shot MultiWOZ DST which takes a leave-one-out approach in Wu et al. (2019). By remixing the data in the way described above, we can evaluate the zero-shot performance on MultiWOZ in a way largely compatible with SG-dst.

DiscussionFirst, the two datasets cover diverse domains. MultiWOZ 2.2 covers various possible dialogue scenarios ranging from requesting basic information about attractions through booking a hotel room or travelling between cities. While SG-dst covers more domains, such as 'Payments', 'Calender', 'DoctorServices' and so on.

 

Second, they include different levels of overlapping functionalities. SG-dst allows frequent function overlapping between multiple services, within the same domain (e.g. BookOneWayTicket v.s. BookRoundTriPricket), or across different domains (BusTicket v.s. TrainTicket). However, the overlapping in MultiWOZ 2.2 only exists across different domains, e.g., 'destination', 'leaveat' slots for Taxi and Bus services, 'pircerange', 'bookday' for Restaurant and Hotel services.

Third, they are collected by two different approaches which are commonly used in dialog collecting. SG-dst is firstly collected by machine-to-machine self-play [19] with dialog flows as seeds, then paraphrased by crowdworkers. While MultiWOZ 2.2 are human-to-human dialogs [16], which are collected with the Wizard-of-Oz approach.

We summarize the above discussion in Table 1. We believe that results derived from these two representative datasets can guide future research in schema guided dialog.

## 5 Dialog & Schema Representation and Inference (Q1)

In this section, we focus on the model architecture for matching dialog history with schema descriptions using pretrained BERT [40]3. To support four subtasks, we first extend _Dual-Encoder_ and _Cross-Encoder_ to support both sentence-level matching and token-level prediction. Then we propose an additional _Fusion-Encoder_ strategy to get faster inference without sacrificing much accuracy. We summarize different architectures in Figure 2. Then we show the classification head and results for each subtask.

Footnote 3: We use BERT-base-cased for all main experiments. Other pretrained language models can be easily adapted to our study

### Encoder Architectures

**Dual-Encoder.** It consists of two separate BERTs to encode dialog history and schema description respectively, as Figure 2 (a). We follow the setting in the official baseline provided by DSTC8 Track4 [7]. We first use a fixed BERT to encode the schema description once and cached the encoded schema \({\bf CLS}_{S}\). Then for sentence-level representation, we concatenate dialog history representation \({\bf CLS}_{D}\) and candidate schema representation \({\bf CLS}_{S}\) as the whole sentence-level representation for the pair, denoted as \({\bf CLS}^{DE}\). For token-level representation, we concatenate the candidate schema \({\bf CLS}_{S}\) with each token embedding in the dialog history, denoted as \({\bf TOK}^{DE}\).4 Because the candidate schema embeddings are encoded independently from the di

Figure 2: Dual-Encoder, Cross-Encoder and Fusion Encoder, shaded block will be cached during training



alog context, they can be pre-computed once and cached for fast inference.

**Cross-Encoder.** Another popular architecture as Figure 2 (b) is _Cross-Encoder_, which concatenates the dialog and schema as a single input, and encodes jointly with a single self-attentive encoder spanning over the two segments. When using BERT to encode the concatenated sentence pair, it performs full (cross) self-attention in every transformer layers, thus offer rich interaction between the dialog and schema. BERT naturally produces a summarized representation with [CLS] embedding \({\bf CLS}^{CE}\) and each schema-attended dialog token embeddings \({\bf TOK}^{CE}\). Since the dialog and schema encoding always depend on each other, it requires recomputing dialog and schema encoding for multiple times, thus much slower in inference.

**Fusion-Encoder.** In Figure 2 (c), similar to _Dual-Encoder_, _Fusion-Encoder_ also encodes the schema independently with a fixed BERT and finetuning another BERT for dialog encoding. However, instead of caching a single [CLS] vector for schema representation, it caches **all token representation** for the schema including the [CLS] token. What's more, to integrate the sequences dialog token representation with schema token representation, an extra stack of transformer layers are added on top to allow token-level fusion via self-attention, similar to _Cross-Encoder_. The top transformer layers will produce embeddings for each token \({\bf TOK}^{FE}\) including a schema-attended \({\bf CLS}^{FE}\) of the input [CLS] from the dialog history. With cached schema token-level representations, it can efficiently produce schema-aware sentence- and token-level representation for each dialog-schema pairs.

### Model Overview

All the above 3 encoders will produce both sentence- and token-level representations for a given sentence pair. In this section, we abstract them as two representations **CLS** and **TOK**, and present the universal classification heads to make decisions for each subtask.

**Active Intent.** To decide the intent for current dialog turn, we match current dialog history \(D\) with each intent descriptions \(I_{0}...I_{k}\). For each dialog-intent pair \((D,I_{k})\), we project the final sentence-level **CLS** representation to a single number \(P_{I_{k}}^{active}\) with a linear layer follows a sigmoid function. We predict _"NONE"_ if the \(P_{I_{k}}^{active}\) of all intents are less than a threshold 0.5, which means no intent is active. Otherwise, we predict the intent with largest \(P_{I_{k}}^{active}\). We predict the intent for each turn independently without considering the prediction on previous turns.

**Requested Slot.** As in Figure 1, mulitple requested slots can exist in a single turn. We use the same strategy as in active intent prediction to predict a number \(P_{req}^{active}\). However, to support the multiple requested slots prediction. We predict all the requested slots with \(P_{req}^{active}>0.5\).

**Categorical Slot.** Categorical slots have a set of candidate values. We cannot predict unseen values via n-way classification. Instead, we do binary classification on each candidate value. Besides, rather than directly matching with values, we also need to check that whether the corresponding slot has been activated. For _Cross-Encoder_ and _Fusion-Encoder_, we use typical two-stage state tracking to incrementally build the state: **Step** 1. Using \({\bf CLS}\) to predict the slot status as _none_, _dontcare_ or _active_. When the status is _active_, we use the predicted slot value; Otherwise, it will be assigned to _dontcare_ meaning no user preference for this slot, or _none_ meaning no value update for the slot in current turn; **Step** 2. If Step 1 is _active_, we match the dialog history with each value and select the most related value by ranking. We train on cross entropy loss. Two-stage strategy is efficient for _Dual-Encoder_ and _Fusion-Encoder_, where cached schema can be reused, and get efficiently ranked globally in a single batch. However, it is not scalable for _Cross-Encoder_, especially for large number of candidate values in MultiWOZ dataset. Hence, during training, we only use a binary cross-entropy for each single value and postpone the ranking only to the inference time.

**Noncategorical Slot.** The slot status prediction for noncategorical slot use the same two-stage strategy. Besides that, we use the token representation of dialog history \({\bf TOK}\) to compute two softmax scores \(f_{start}^{i}\) and \(f_{end}^{i}\) for each token \(i\), to represent the score of predicting the token as start and end position respectively. Finally, we find the valid span with maximum sum of the start and end scores.

### Experiments on Encoder Comparison

To fairly compare all three models, we follow the same schema input setting as in Table 2. We trained separate models for SG-dst and the remixed MultiWOZ datasets for all the experiments in our pa 

pers5. Because there are very few intent and requested slots in MultiWOZ 2.2 dataset, we ignore the intent and requested slots tasks for MultiWOZ 2.2 in our paper.

Footnote 5: Appendix A.1 shows the detailed experiment setup

**Results.** As shown in Table 3, _Cross-Encoder_ performs the best over all subtasks. Our _Fusion-Encoder_ with partial attention outperforms the _Dual-Encoder_ by a large margin, epsecially on categorical and noncategorical slots predictions. Additionally, on seen services, we found that _Dual-Encoder_ and _Fusion-Encoder_ can perform as good as _Cross-Encoder_ on Intent and Req tasks. However, they cannot generalize well on unseen services as _Cross-Encoder_.

**Inference Speed.** To test the inference speed, we conduct all the experiments with a maximum affordable batch size to fully exploit 2 V100 GPUs (with 16GB GPU RAM each). During training, we log the inference time of each evaluation on dev set. Both _Dual-Encoder_ and _Fusion-Encoder_ can do joint inference across 4 subtasks to obtain an integral dialog state for a dialog turn example. _Dual-Encoder_ achieves the highest inference speed of **603.35** examples per GPU second, because the encoding for dialog and schema are fully separated. A dialog only needed to be encoded for once during the inference of a dialog state example while the schema are precomputed once. However, for _Cross-Encoder_, to predict a dialog state for a single turn, it need to encode more than 300 sentence pairs in a batch, thus only processes **4.75** examples per GPU second. _Fusion-Encoder_ performs one time encoding on dialog history, but it needs to jointly encode the same amount of dialog-schema pair ws _Cross-Encoder_, instead, however, with a two-layer transformer encoder. Overall it achieves **10.54** examples per GPU second, which is **2.2x** faster than _Cross-Encoder_. With regarding to the accuracy in Table 3, _Fusion-Encoder_ performs much better than _Dual-Encoder_, especially on unseen services.

## 6 Supplementary Training (Q2)

Besides the pretrain-fintune framework used in SS5, Phang et al. (2018) propose to add a supplementary training phase on an intermediate task after the pretraining, but before finetuning on target task. It shows significant improvement on the target tasks. Moreover, large amount pretrained and finetuned transformer-based models are publicly accessible, and well-organized in model hubs for sharing, training and testing6. Given the new task of schema-guided dialog state tracking, in this section, we study our four subtasks with different intermediate tasks for supplementary training.

Footnote 6: e.g., Huggingface([https://huggingface.co/models](https://huggingface.co/models)) and ParlAL([https://parl.ai/docs/zoo.html](https://parl.ai/docs/zoo.html)), etc.

### Intermediate Tasks

As described in SS 5.2, all our 4 subtasks take a pair of dialog and schema description as input, and predict with the summerized sentence-pair **CLS** representation. While NonCat also requires span-based detection such as question answering. Hence, they share the similar problem structure with the following sentence-pair encoding tasks.

**Natural Language Inference.** Given a hypothesis/premise sentence pair, natural language inference is a task to determine whether a hypothesis is entailed, contradicted or neutral given that premise.

**Question Answering.** Given a passage/question pairs, the task is to extract the span-based answer in the passage.

Hence, when finetuning BERT on our subtaks, instead of directly using the originally pretrained BERT, we use the BERT finetuned on the above

\begin{table}
\begin{tabular}{l|c} \hline \hline  & service description, intent description \\ Req & service description, slot description \\ Cat & slot description, cat value \\ NonCat & service description, slot description \\ \hline \hline \end{tabular}
\end{table}
Table 2: Schema description input used for different tasks to compare _Dual-Encoder_, _Cross-Encoder_, and _Fusion-Encoder_. In the appendix A.3, we also studies other compositions of description input. We found that service description will not help for Intent, Req and Cat tasks, while the impact on NonCat task also varies from SG-dst and MultiWOZ 2.2 dataset.

\begin{table}
\begin{tabular}{l|c c c c|c c c c} \hline \hline  & \multicolumn{4}{c|}{SG-dst} & \multicolumn{4}{c}{MultiWOZ 2.2} \\ \cline{2-9} MethodTask & Acc & F1 & Joint Acc & \multicolumn{2}{c}{Joint Acc} \\  & Intent & Req & Cat & NonCat & All & Cat & NonCat & All \\ \hline \multicolumn{9}{l}{_Scan Series_} \\ \hline Dual-Encoder & 94.51 & 99.62 & 87.92 & 47.77 & 43.20 & 79.20 & 79.34 & 65.64 \\ Fusion-Encoder & 94.90 & **99.69** & 88.94 & 48.78 & 58.52 & 81.37 & 80.58 & 67.43 \\ Cross-Encoder & **95.55** & 99.59 & **93.68** & **91.85** & **87.58** & **85.99** & **81.02** & **71.93** \\ \hline \multicolumn{9}{l}{_Dense Services_} \\ \hline Dual-Encoder & 89.73 & 95.20 & 42.44 & 31.62 & 19.51 & 56.92 & 50.82 & 31.85 \\ Fusion-Encoder & 90.47 & 95.95 & 48.79 & 35.91 & 22.85 & 57.01 & 52.23 & 33.64 \\ Cross-Encoder & **93.84** & **98.26** & **71.55** & **74.13** & **54.54** & **99.85** & **59.62** & **38.46** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Test set results on SG-dst and MultiWOZ 2.2. The _Dual-Encoder_ model is a re-implementation of official DSTC8 baseline from Rastogi et al. (2019). Other models are trained with the architecture described in our paper.

 

two tasks for further finetuning. Due to better performance of _Cross-Encoder_ in SS5, we directly use the finetuned _Cross-Encoder_ version of BERT models on SNLI and SQuAD2.0 dataset from Huggingface model hub. We add extra speaker tokens [user:] and [system:] into the vocabulary for encoding the multi-turn dialog histories.

### Results on Supplementary Training

Table 4 shows the performances gain when fine-tuning 4 subtasks based on models with the above SNLI and SQuAD2.0 supplementary training.

We mainly find that SNLI helps on Intent task, SQuAD2 mainly helps on NonCat task, while neither of them helps much on Cat task. Recently, Namazifar et al. (2020) also found that when modeling dialog understanding as question answering task, it can benefit from a supplementary training on SQuAD2 dataset, especially on few-shot scenarios, which is a similar findings as our NonCat task. Result difference on Req task is minor, because it is a relatively easy task, adding any supplementary training did n't help much. Moreover, for Cat task, the sequence 2 of the input pair is the slot description with a categorical slot value, thus the meaning overlapping between the full dialog history and the slot/value is much smaller than SNLI tasks. On the other side, **CLS** token in SQuADBERT is finetuned for null predictions via start and end token classifers, which is different from the the single CLS classifer in Cat task.

## 7 Impact of Description Styles (Q3)

Previous work on schema-guided dialog Rastogi et al. (2020) are only based on the provided descriptions in SG-dst dataset. Recent work on modeling dialog state tracking as reading comprehension Gao et al. (2019) only formulate the descriptions as simple question format with existing intent/slot names, it is unknown how it performs when compared to other description styles. Moreover, they only conduct homogeneous evaluation where training and test data share the same description style. In this section, We also investigate how a model trained on one description style will perform on other different styles, especially in a scenario where chat-bot developers may design their own descriptions. We first introduce different styles of descriptions in our study, and then we train models on each description style and evaluate on tests with corresponding homogeneous and heterogeneous styles of descriptions. Given the best performance of _Cross-Encoder_ shown in the previous section and its popularity in DSTC8 challenges, we adopt it as our model architecture in this section.

### Benchmarking Styles

For each intent/slot, we describe their functionalities by the following different descriptions styles:

**Identifier.** This is the least informative case of name-based description: we only use meaningless intent/slot identifiers, e.g. Intent_1, Slot_2. It means we don't use description from any schema component. We want to investigate how a simple identifier-based description performs in schema-guided dialog modeling, and the performance lower-bound on transferring to unseen services.

**NameOnly.** Using the original intent/slot names in SG-dst and MultiWOZ 2.2 dataset as descriptions, to show whether name is enough for schema-guided dialog modeling.

**Q-Name.** This is corresponding to previous work by Gao et al. (2019). For each intent/slot, it generate a question to inquiry about the intent and slot value of the dialog. For each slot, it simply follows the template _'What is the value for_**slot i?**'. Besides that, our work also extend the intent description by following the template _"Is the user intending to_**intent j**".

**Orig.** The original descriptions in SG-dst and MultiWOZ 2.2 dataset.

**Q-Orig.** Different from the Q-Name, firstly it is based on the original descriptions; secondly, rather than always use the "what is" template to inquiry the intent/slot value, We add "what", "which", "how many" or "when" depending on the entity type required for the slot. Same as Q-Name, we just add

\begin{table}
\begin{tabular}{l|c c|c c|c c|c c|c c|c c|c c|c c} \hline \hline  & \multicolumn{6}{c|}{SG-DST} & \multicolumn{6}{c|}{MultiWOZ 2.2} \\ \cline{2-13}  & \multicolumn{2}{c|}{iment} & \multicolumn{2}{c|}{req} & \multicolumn{2}{c|}{cat} & \multicolumn{2}{c|}{noncat} & \multicolumn{2}{c|}{cat} & \multicolumn{2}{c|}{concer} & \multicolumn{2}{c}{noncat} & \multicolumn{2}{c}{noncat} \\ \cline{2-13}  & all & seen & unseen & all & seen & unseen & all & seen & unseen & all & seen & unseen & all & seen & unseen & all & seen & unseen \\ \hline \(\Delta\)**SNLI** +**0.51** & +0.02 & **+0.68** & -0.19 & +0.38 & -0.38 & -1.63 & -2.87 & -1.23 & -4.7 & -0.1 & -6.25 & +2.05 & +0.6 & -0.7 & **+3.64** & +1.05 & **+4.84** \\ \hline \(\Delta\)**SQQM** & -1.81 & -0.17 & -1.32 & -0.25 & -0.01 & -0.33 & -2.87 & -3.02 & -5.17 & **+1.99** & -1.79 & **+3.25** & +0.04 & -0.71 & +0.41 & **+1.93** & -2.21 & **+4.27** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Relative performance improvement of different supplementary training on SG-dst and MultiWOZ 2.2 dataset 

prefixes as "Is the user intending to..." in front of the original description. In a sum, this description is just adding question format to original description. The motivation of this description is to see whether the question format is helpful or not for schema-guided dialog modeling.

To test the model robustness, we also create two paraphrased versions Name-Para and Orig-Para for NameOnly and Orig respectively. We first use nematus Sennrich et al. (2017) to automatically paraphrase the description with back translation, from English to Chinese and then translate back, then we manually check the paraphrase to retain the main meaning. Appendix A.5.1 shows examples for different styles of schema descriptions.

### Results on Description Styles

Unlike the composition used in Table 2, we don't use the service description to avoid its impact. For each style, we train separate models on 4 sub-tasks, then we evaluate them on different target styles. First, Table 5 summarizes the performance for homogeneous evaluation, while Table 6 shows how the question style description can benefit from SQuAD2 finetuning. Then we also conduct heterogeneous evaluation on the other styles7 as shown in Table 7.

Footnote 7: We do’t consider the meaningless Identifier style due to its bad performance

#### 7.2.1 Homogeneous Evaluation

**Is name-based description enough?** As shown in Table 5, Identifier is the worst case of using name description, its extremely bad performance indicates name-based description can be very unstable. However, we found that simple meaningful name-based description actually can perform the best in Intent and Req task, and they perform worse on Cat and NonCat tasks comparing to the bottom two _rich_ descriptions. 8 After careful analysis on the intents in SG-dst datasets, we found that most services only contains two kinds of intents, an information retrieval intent with a name prefix "Find-", "Get-", "Search-"; another transaction intent like "Add-", "Reserve-" or "Buy-". Interestingly, we found that all the intent names in the original schema-guided dataset strictly follows an action-object template with a composition of words without abbreviation, such as "FindEvents", "BuyEventTickets". This simple name template is good enough to describe the core functionality of an intent in SG-dst dataset. 9 Additionally, Req is a relatively simper task, requesting information are related to specifial attributes, such as "has_live_music", "has_wifi", where keywords co-occured in the slot name and in the user utterance, hence rich explanation cannot help further. On the other side, _rich_ descriptions are more necessary for Cat and NonCat task. Because in many cases, slot names are too simple to represent the functionalities behind it, for example, slot name "passengers" cannot fully represent the meaning "number of passengers in the ticket booking".

Footnote 8: Only exception happens in Cat on MultiWOZ 2.2. When creating MultiWOZ 2.2 Zang et al. (2020), the slots with less than 50 different slot values are classified as categorical slots, which leads to inconsistencies. We put detailed discuss about MultiWOZ 2.2 in the supplementary material

**Does question format help?** As shown in Table 5, when comparing row Q-Orig v.s. Orig, we found extra question format can improve the performance on Cat and NonCat task on both SG-dst and MultiWOZ 2.2 datasets, but not for Intent and Req tasks. We believe that question format helps the model to focus more on specific entities in the dialog history. However, when adding a simple question pattern to NameOnly, comparing row Q-Name and NameOnly, there is no consistent improvement on both of the two datasets. Further more, we are curious about whether BERT finetuned on SQuAD2 (SQuAD2-BERT) can further help on the question format. Because NonCat are similar with span-based question answering, we focus on NonCat here. Table 6 shows that, after applying the supplementary training on SQuAD2 (SS6), almost all models get improved on unseen splits however slightly dropped on seen services. Moreover, compa

\begin{table}
\begin{tabular}{c|c c c c|c c} \hline \hline \multirow{2}{*}{StyleTask} & \multicolumn{3}{c|}{SG-dst} & \multicolumn{3}{c}{MultiWOZ 2.2} \\  & Intent & Req & Cat & NonCat & Cat & NonCat \\ \hline Identifier & 61.16 & 91.48 & 62.47 & 30.19 & 34.25 & 52.28 \\ \hline NameOnly & **94.24** & 98.84 & 74.01 & 75.63 & 53.72 & 56.18 \\ Q-Name & 93.31 & **98.86** & 74.36 & 74.86 & **54.19** & 56.17 \\ \hline Orig & 93.01 & 98.55 & 74.51 & 57.76 & 52.19 & 57.20 \\ O-Orig & 93.42 & 98.51 & **76.64** & **76.60** & 53.61 & **57.80** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Homogeneous evaluation results of different description style on SG-dst dataset and MultiWOZ 2.2 datasets. The middle horizontal line separate the two name-based descriptions and two rich descriptions in our settings. All numbers in the table are mixed performance including both seen and unseen services.

 

Orig is more similar to the natural questions in the SQuAD2, we obverse that _Q-Orig_ gains more than _Q-Name_ from pretrained model on SQuAD2.

#### 7.2.2 Heterogeneous

In this subsection, we first simulate a scenario when there is no recommended description style for the future unseen services. Hence, unseen services can follow any description style in our case. We average the evaluation performance on three other descriptions and summarized in Table 7. The \(\Delta\) column shows the performance change compared to the homogeneous performance. It is not surprising that almost all models perform worse on heterogeneous styles than on homogeneous styles due to different distribution between training and evaluation. The bold number shows the best average performance on heterogeneous evaluation for each subtask. The trends are similar with the analysis in homogeneous evaluation 7.2.1, the name-based descriptions perform better than other rich descriptions on intent classification tasks. While on other tasks, the Orig description performs more robust, especially on NonCat task.

Furthermore, we consider another scenario where fixed description convention such as NameOnly and Orig are suggested to developers, they must obey the basic style convention but still can freely use their own words, such as abbreviation, synonyms, adding extra modifiers. We train each model on NameOnly and Orig, then evaluate on the corresponding paraphrased version respectively. In the last two rows of Table 7, the column 'para' shows performance on paraphrased schema, while \(\Delta\) shows the performance change compared to the homogeneous evaluation. Orig still performs more robust than NameOnly when schema descriptions get paraphrased on unseen services.

## 8 Conclusion

In this paper, we studied three questions on schema-guided dialog state tracking: encoder architectures, impact of supplementary training, and effective schema description styles. The main findings are as follows:

By caching the token embedding instead of the single cls embedding, a simple partial-attention _Fusion-Encoder_ can achieve much better performance than Dual-Encoder, while still infers two times faster than Cross-Encoder. We quantified the gain via supplementary training on two intermediate tasks. By carefully choosing representative description styles according to recent works, we are the first of doing both homogeneous/heterogeneous evaluations for different description style in schema-guided dialog. The results show that simple name-based description performs well on Intent and Req tasks, while NonCat tasks benefits from richer styles of descriptions. All tasks suffer from inconsistencies in description style between training and test, though to varying degrees.

Our study are mainly conducted on two datasets: SG-dst and MultiWOZ 2.2, while the speed-accuracy balance of encoder architectures and the findings in supplementary training are expected to be dataset-agnostic, because they depend more on the nature of the subtasks than the datasets. Based on our proposed benchmarking descriptions suite, the homogeneous and heterogeneous evaluation has shed the light on the robustness of cross-style schema-guided dialog modeling, we believe our study will provide useful insights for future research.

## Acknowledgments

The authors wish to thank the anonymous reviewers and members of the Amazon LEX team for their valuable feedback.

\begin{table}
\begin{tabular}{c|l l l l l l} \hline \hline \multirow{2}{*}{Style/Dataset} & \multicolumn{3}{c}{SG-dst} & \multicolumn{3}{c}{MultiWOZ 2.2} \\ \cline{2-9}  & all & seen & unseen & all & seen & unseen \\ \hline Orig & +1.99 & -1.79 & **+3.25** & +1.93 & -2.21 & **+4.27** \\ Q-Orig & +6.13 & -2.01 & **+8.84** & +1.06 & -1.28 & **+3.06** \\ \hline NameOnly & -0.45 & -1.49 & -0.11 & +1.75 & +0.58 & **+1.77** \\ Q-Name & +0.05 & -2.98 & **+1.04** & -0.04 & -0.32 & **+1.25** \\ \hline \hline \end{tabular}
\end{table}
Table 6: Performance changes when using BERT fine-tuned on SQuAD2 dataset to further finetuning on our NonCat task.

\begin{table}
\begin{tabular}{c|l l l l l l l} \hline \hline \multirow{2}{*}{StyleTask} & \multicolumn{3}{c}{SG-dst} \\ \cline{2-9}  & latent (Acc) & \multicolumn{3}{c}{Req(T)} & \multicolumn{3}{c}{LEq(T)} & \multicolumn{3}{c}{LEq(T)} & \multicolumn{3}{c}{NonCat(Joint Acc)} \\ \cline{2-9}  & mean & \(\Delta\) & mean & \(\Delta\) & mean & \(\Delta\) & mean & \(\Delta\) \\ \hline NameOnly & 82.47 & -11.47 & 96.92 & -1.64 & 61.37 & -5.54 & 56.53 & -14.68 \\ Q-Name & **93.27** & +0.58 & **97.88** & -0.68 & **-5.65** & -2.63 & 62.92 & -6.30 \\ Orig & 79.47 & -12.79 & **07.42** & -0.74 & **68.58** & -0.3 & 66.72 & -3.11 \\ Q-Orig & 84.57 & -8.24 & 96.70 & -1.45 & 68.40 & -2.89 & 56.17 & -15.00 \\ \hline \hline \multirow{2}{*}{
\begin{tabular}{c} NameOnly \\ Origin \\ \end{tabular} } & Para & \(\Delta\) & para & para & \(\Delta\) & para & \(\Delta\) \\ \hline NameOnly & **92.22** & -1.74 & 97.69 & -0.87 & 73.90 & -0.7 & 6.17 & -4.04 \\ Orig & 91.54 & **-0.63** & **98.42** & **+0.26** & **71.74** & **+2.86** & **67.68** & **-2.16** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Results on unseen service with heterogeneous description styles on SG-dst dataset. More results and qualitative analysis are in the appendix A.5 



## References

* B. G. Bobrow, R. M. Kaplan, M. Kay, D. A. Norman, H. Thompson, and T. Winograd (1977)Gus, a frame-driven dialog system. Artificial intelligence82, pp. 155-173. Cited by: SS4.1.
* A. Bordes, J. Weston, and N. Usunier (2014)Open question answering with weakly supervised embedding models. In Joint European conference on machine learning and knowledge discovery in databases, pp. 165-180. Cited by: SS1.
* P. Budzianowski, T. Wen, B. Tseng, I. Casanueva, S. Ultes, O. Ramadan, and M. Gasic (2018)MultiwoZ-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026. Cited by: SS1.
* B. Byrne, K. Krishnanamoorthi, C. Sankar, A. Neelakantan, D. Duckworth, S. Yavuz, B. Goodrich, A. Dubey, A. Cedilnik, and K. Kim (2019)Taskmaster-1: toward a realistic and diverse dialog dataset. arXiv preprint arXiv:1909.05358. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186. External Links: Link, Document Cited by: SS1.
* M. Eric, R. Goel, S. Paul, A. Sethi, S. Agarwal, S. Gao, A. Kumar, A. Goyal, P. Ku, and D. Hakkani-Tur (2020)MultiWOZ 2.1: a consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 422-428. Cited by: SS1.
* S. Gao, S. Agarwal, T. Chung, D. Jin, and D. Hakkani-Tur (2020)From machine reading comprehension to dialogue state tracking: bridging the gap. arXiv preprint arXiv:2004.05827. Cited by: SS1.
* S. Gao, A. Sethi, S. Agarwal, T. Chung, D. Hakkani-Tur, and A. Alexa (2019)Dialog state tracking: a neural reading comprehension approach. In 20th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 264. Cited by: SS1.
* T. Han, X. Liu, R. Takanobu, Y. Lian, C. Huang, W. Peng, and M. Huang (2020)MultiWOZ 2.3: a multi-domain task-oriented dataset enhanced with annotation corrections and coreference annotation. arXiv e-prints, pp. arXiv-2010. Cited by: SS1.
* M. Henderson, B. Thomson, and S. Young (2014)Word-based dialog state tracking with recurrent neural networks. In Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 292-299. Cited by: SS1.
* E. Hosseini-Asl, B. McCann, C. Wu, S. Yavuz, and R. Socher (2020)A simple language model for task-oriented dialogue. arXiv preprint arXiv:2005.00796. Cited by: SS1.
* S. Humeau, K. Shuster, M. Lachaux, and J. Weston (2019)Poly-encoders: architectures and pre-training strategies for fast and accurate multi-sentence scoring. In ICLR, Cited by: SS1.
* J. F. Kelley (1984)An iterative design methodology for user-friendly natural language office information applications. ACM Transactions on Information Systems (TOIS)2 (1), pp. 26-41. Cited by: SS1.
* S. Kim, S. Yang, G. Kim, and S. Lee (2019)Efficient dialogue state tracking by selectively overwriting memory. arXiv preprint arXiv:1911.03906. Cited by: SS1.
* H. Lee, J. Lee, and T. Kim (2019)SumbRT: slot-utterance matching for universal and scalable belief tracking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5478-5483. Cited by: SS1.
* R. Lowe, N. Pow, I. V. Serban, and J. Pineau (2015)The ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 285-294. Cited by: SS1.
* N. Mrksic, D. O. Seaghdha, T. Wen, B. Thomson, and S. Young (2017)Neural belief tracker: data-driven dialogue state tracking. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1777-1788. Cited by: SS1.
* C. Naik, A. Gupta, H. Ge, M. Lambert, and R. Sarikaya (2018)Contextual slot carryover for disparate schemas. Proc. Interspeech 2018, pp. 596-600. Cited by: SS1.
* M. Namazifar, A. Papangelis, G. Tur, and D. Hakkani-Tur (2020)Language model is all you need: natural language understanding as question answering. arXiv preprint arXiv:2011.03023. Cited by: SS1.
* V. Noroozi, Y. Zhang, E. Bakhturina, and T. Kornuta (2020)A fast and robust bert-based dialogue state tracker for schema-guided dialogue dataset. Workshop on Conversational Systems Towards Mainstream Adoption at KDD 2020

* Peng et al. (2020) Baolin Peng, Chunyuan Li, Jinchao Li, Shahin Shayandeh, Lars Liden, and Jianfeng Gao. 2020. Soloist: Few-shot task-oriented dialog with a single pre-trained auto-regressive model. _arXiv_, pages arXiv-1905.
* Phang et al. (2018) Jason Phang, Thibault Fevry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. _arXiv preprint arXiv:1811.01088_.
* Ratogi et al. (2019) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghaw Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. _AAAI 2019_.
* Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghaw Gupta, and Pranav Khaitan. 2020. Schema-guided dialogue state tracking task at dstc8. _Workshop on DSTC8, AAAI 2020_.
* Sennrich et al. (2017) Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hirschler, Marcin Junczys-Dowmunt, Samuel Laubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a toolkit for neural machine translation. In _Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics_, pages 65-68, Valencia, Spain. Association for Computational Linguistics.
* Louisiana. Association for Computational Linguistics.
* Shah et al. (2018b) Pararth Shah, Dilek Hakkani-Tur, Gokhan Tur, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry Heck. 2018b. Building a conversational agent overnight with dialogue self-play. _arXiv preprint arXiv:1801.04871_.
* Vedula et al. (2020) Nikhita Vedula, Nedim Lipka, Pranav Maneriker, and Srinivasan Parthasarathy. 2020. Open intent extraction from natural language interactions. In _Proceedings of The Web Conference 2020_, pages 2009-2020.
* Williams et al. (2016) Jason D Williams, Antoine Raux, and Matthew Henderson. 2016. The dialog state tracking challenge series: A review. _Dialogue & Discourse_, 7(3):4-33.
* Wu et al. (2019) Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019. Transferable multi-domain state generator for task-oriented dialogue systems. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 808-819.
* Wu et al. (2017) Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatopts. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 496-505.
* Yang et al. (2018) Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W Bruce Croft, Jun Huang, and Haiqing Chen. 2018. Response ranking with deep matching networks and external knowledge in information-seeking conversation systems. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 245-254.
* Zang et al. (2020) Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In _Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI_, pages 109-117, Online. Association for Computational Linguistics.
* Zhang et al. (2019) Jian-Guo Zhang, Kazuma Hashimoto, Chien-Sheng Wu, Yao Wan, Philip S Yu, Richard Socher, and Caiming Xiong. 2019. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. _arXiv_, pages arXiv-1910.

## Appendix A Appendices

### Experiment Setup

All models are based on BERT-base-cased model with 2 V100 GPUs (with 16GB GPU RAM each). We train each models for maximum 10 epoch, by using AdamW to schedule the learning rate with a warm-up portion of 0.1. During training, we evaluate checkpoints per 3000 steps on dev splits, and select the model with best performance on dev split on all seen and unseen services. In our experiments, our model achieves the best performance on around 2-4 epochs on Intent, Req. and Cat, while NonCat needs 5-8 epochs to get the best performance. For all subtasks, as we model all of them as sentence pair encoding during training, we use batch size as 16 for each GPU, and gradient accumulate for 8 steps, in total 256 batch size on 2 GPUs.

### Statistic on MultiWOZ 2.2 Remix

To evaluate performance on seen/unseen services with MultiWOZ, we remix the MultiWOZ 2.2 dataset to include as seen services dialogs related to _restaurant_, _attraction_ and _train_ during training, and eliminate slots from other domains/services from training split. For dev, we add two new domains _hotel_ and _taxi_ as unseen services. For test, we add all 

remaining domains as unseen, including those that have minimum overlap with seen services, such as _hospital_, _police_, _bus_. The statistics are as shown in Table 8

### Composition of Descriptions

#### a.3.1 Composition Settings

For each subtask, the key description element must be included, e.g., intent description for intent task, and value for categorical slot tasks. To show how each component helps schema-guided dialog state tracking, we incrementally add richer schema component one by one.

**ID.** This is the least informative case: we only use meaningless intent/slot identifiers, e.g. Intent_4, Slot_2. It means we don't use description from any schema component. We want to investigate how a simple identifier-based description performs in schema-guided dialog modeling, and the performance lower-bound on transferring to unseen services.

**I/S Desc.** Only using the original intent/slot description of intent/slot in SG-dst and MultiWOZ 2.2 dataset for corresponding tasks.

**Service + I/S Desc.** Adding a service description to the above original description. Service description summarize the functionalities of the whole service, hence may offer extra background information for intent and slots. For categorical slot value detection, we simply add the value after each of the above composition.

#### a.3.2 Results on Description Compositions

Table 9 shows the results of using different description compositions. First, there are consistent findings across datasets and subtasks: (1) using meaningless identifier as intent/slot description shows the worse performance on all tasks of both datasets, and can not generalize well to unseen services. (2) using intent/slot descriptions can largely boost the performance, especially on unseen services.

However, the impact of service description varies by tasks. For example, it largely hurts performance on intent classification task, but does not impact requested slot and categorical slot tasks. According to manual analysis of SG-dst and MultiWOZ 2.2 dataset, we found that service description consists of the main functions of the service, especially the meaning of the supported intents. Hence, using service description for intent causes confusion between the intent description information and other supported intents. Moreover, in categorical slot value prediction task, the most important information is the slot description and value. When adding extra information from service description, it improves marginally on seen service while not generalizing well on unseen services, which indicates the model learns artifacts that are not general useful for unseen services.

Finally, on non-categorical slot tasks, the impact of service description may also varies on datasets. On SG-dst, there are 16 domains and more than 30 services, the rich background context from service description contains both domain and service-specific information, which seems to help both seen and unseen services. However, on MultiWOZ 2.2, it hurts the performance on seen service _restaurant_ the most, while improving the performance on the unseen service _hotel_ by 4 points. In this case, it works like a regularizer rather than a definitive clues. Because in MultiWOZ 2.2, there are only 8 domains, and one service per domain, thus service descriptions just contain domain related information without much extra information, it will not help the model to detect the span for the slot.

### More Results of Supplementary Training

Table 10 shows the detailed performance when using different intermediate tasks as supplementary training. For SNLI tasks, as the pretrained model is uncased model (textattack/ bert-base-uncased-snli),

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline \multirow{2}{*}{Domain} & \multicolumn{5}{c}{\#dialogs/Hurms} \\  & train & dev & test \\ \hline restaurant & 3900 & 37593 & 486 & 6979 & 451 & Total04 \\ attraction & 2716 & 28623 & 405 & 6198 & 400 & 6290 \\ train & 3001 & 29646 & 481 & 5897 & 491 & 6150 \\ hotel & 0 & 0 & 737 & 8590 & 718 & 7911 \\ taxi & 0 & 0 & 0 & 347 & 2692 & 364 & 2659 \\ hospital & 0 & 0 & 0 & 0 & 287 & 766 \\ police & 0 & 0 & 0 & 0 & 252 & 475 \\ bus & 0 & 0 & 0 & 0 & 6 & 132 \\ \hline \hline \end{tabular}
\end{table}
Table 8: The total number of dialogs and turns related to each domain in train, dev and test split of MultiWOZ 

hence, we first train different models with BERT-base-uncased, then compare the performance with SNLI pretrained model. For SQuAD2, we use deepest/bert-base-cased-SQuAD2 model, hence, we compare it all cased model. To fairly compare with our original _Cross-Encoder_, we add extra speaker tokens [user:] and [system:] for encoding the multi-turn dialog histories.

### Homogeneous and Hetergenuous Evaluation on Different Styles

#### a.5.1 Examples for Different Description Styles

Table 11 shows examples for different styles of schema descriptions.

#### a.5.2 More details on SQuAD2 Results on Different Styles

For homogeneous evaluation, Table 12 shows the detailed performance when we apply SQuAD2-finetuned BERT on our models.

#### a.5.3 More Results On Homogeneous and Heterogeneous Evalution

We list the detailed results for our evaluation across different styles. We use _italic_ to show the homogeneous evaluation, where the results are shown in the diagonal of each table, and we underline the best homogeneous results in the diagonal. We use **bold** to show the best heterogeneous performance and the best performance gap in the last two columns

**Intent.** The results on SG-dst dataset are shown in Table 13. Because there are very few intents in MultiWOZ 2.2 dataset, we don't conduct intent classification on MultiWOZ 2.2. All performance get dropped when evaluating on heterogeneous descriptions styles. For both heterogeneous and homogeneous evaluation, adding rich description on intent classification tasks seems not bring much benefits than simply using the named-based description. As the discussion in SS7.2.1, we believe the name template is good enough to describe the core functionality of an intent in SG-dst dataset.

**Requested Slot.** Table 14 shows the results on SG-dst dataset for the requested slots subtask. We ignore the requested slots in MultiWOZ 2.2 dataset due to its sparsity. Overall, the requested slot subtask are relatively easy, performances on heterogeneous styles still drops but not much. For both heterogeneous and homogeneous evaluation, the performance are not sensible to rich description.

**Categorical Slot.** The results on SG-dst and MultiWOZ 2.2 dataset are shown in Table 15.

\begin{table}
\begin{tabular}{l|c|c 

When creating MultiWOZ 2.2 (Zang et al., 2020), the slots with less than 50 different slot values are classified as categorical slots. We noticed that this leads inconsistent results with SG-dst dataset. It is hard to draw a consistent conclusion on the two datasets. According to the definition, we believe SG-dst are more suitable for categorical slot sub-tasks, we can further verify our guess when more datasets are created for the research of schema-guided dialog in the future.

**Non-categorical Slot.** We conduct non-categorical slot identification sub-tasks on both SG-dst and MultiWOZ 2.2 dataset. The results are shown in Table 16. Overall, the rich description performs better on both homogeneous and heterogeneous evaluations.

#### a.5.4 Qualitative Analysis On Heterogeneous Evaluation

We conduct qualitative analysis on heterogeneous evaluation on named-based description. Table 17 shows how paraphrasing the named-based description impact on the categorical and non-categorical slot prediction tasks.

The first 3 rows at the top are showing the cases of adding modifiers to the name. When the added extra modifiers are keywords in other slots, e.g. "attraction" are the keywords also used in "attraction_name". The first shows "attraction_location" may wrongly predicted as "attraction_name". It seems the model does not understand the compound nouns well, and they seems just pay attention to each key words "attraction" and "movie" here.

The 3 rows in the middle are showing the cases of using synonyms. Changing "to" to "target", and changing "movie" to "film" will cause extra confusion, which shows the model may fail to the synonyms.

The last 4 rows at the bottom is showing using abbreviations. Changing "number" to "num" will not impact the model, while changing "subtitle" to "sub" may let the model miss the key meaning of subtitle. The performance drop in the later case may be due to the misuse of the "sub" prefix, in En

\begin{table}
\begin{tabular}{c|c c c c|c c} \hline \hline Style & NameOnly & Q-Name & Orig & Q-Orig & mean & \(\Delta\) \\ \hline NameOnly & 98.56 & 96.01 & 97.2 & 97.54 & 96.92 - 1.64 \\ O-Name & 98.37 & 98.64 & 97.88 & 97.48 & **97.88** - 0.76 \\ Orig & 97.95 & 95.78 & 98.16 & 98.52 & **97.24** - **0.74** \\ O-Orig & 97.24 & 95.85 & 97.00 & 98.15 & 96.70 - 1.45 \\ \hline \hline \end{tabular}
\end{table}
Table 14: F1 Score of requested slot classification subtask with different description styles on unseen services. We train the model on SG-dst dataset for the description style in each row, then evaluate on 4 different descriptions styles. The mean are average performance of the remaining 3 descriptions styles. The \(\Delta\) means the performance gap between the mean and the homogeneous performance

\begin{table}
\begin{tabular}{c|c c c c|c c} \hline \hline Style & NameOnly & Q-Name & Orig & Q-Orig & mean & \(\Delta\) \\ \hline NameOnly & 98.56 & 96.01 & 97.2 & 97.54 & 96.92 - 1.64 \\ O-Name & 98.37 & 98.64 & 97.88 & 97.48 & **97.88** - 0.76 \\ Orig & 97.95 & 95.78 & 98.16 & 98.52 & **97.24** - **0.74** \\ O-Orig & 97.24 & 95.85 & 97.00 & 98.15 & 96.70 - 1.45 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Accuracy of intent classification subtask with different description styles on unseen services. Train the model on SG-dst dataset for each description in each row, then evaluating on 4 different descriptions styles. The mean are average performance of the remaining 3 descriptions styles. The \(\Delta\) means the performance gap between the mean and the homogeneous performance

\begin{table}
\begin{tabular}{c|c c c c|c c} \hline \hline Style & NameOnly & Q-Name & Orig & Q-Orig & mean & \(\Delta\) \\ \hline NameOnly & 68.09 & 58.41 & 63.49 & 62.21 & 61.37 - 6.72 \\ \(Q\)-Name & 69.01 & 68.29 & 68.53 & 68.12 & 68.55 + **0.26** \\ Orig & 70.19 & 65.91 & 68.88 & 69.64 & **68.58** - 0.30 \\ O-Orig & 69.98 & 65.97 & 69.26 & 71.29 & 68.40 - 2.89 \\ \hline \multicolumn{7}{c}{MultiWOZ 2.2} \\ \hline NameOnly & 59.24 & 59.32 & 59.12 & 59.29 & 59.24 & 0.00 \\ O-Name & 85.64 & 59.74 & 58.49 & 59.43 & 58.85 - 0.89 \\ Orig & 59.26 & 59.91 & 56.49 & 58.97 & **59.38** **+ **2.89** \\ O-Orig & 60.00 & 60.70 & 51.18 & 58.95 & 57.29 - 1.66 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Joint accuracy of categorical slot Subtask with different description styles on unseen services. Train the model on SG-dst and MultiWOZ 2.2 datasets respectively for each description style in each row, then evaluate on all 4 descriptions styles. The mean are the average performance of the remaining 3 descriptions styles. The \(\Delta\) means the performance gap between the mean and the homogeneous performance 

glish, it usually means "secondary, less important, parts". We also found the "orig" and "dest" abbreviations may also understand well by the model. The above abbreviations seems reasonable paraphrases people will use for naming, while the are not understood well in the given context. Hence, in the design of schema-guided dialog, if using named-based description, we should be careful for about abbreviations used in the naming.

\begin{table}
\begin{tabular}{l l c|l} \hline \hline Service Name & Original Name & Paraphraed Name & Extra impact by the paraphrased name \\ \hline Travel\_1 & location & attraction\_location & Confixed with ether “attraction” prefixed slots, e.g. attraction \_name \\ Movies\_1 & genre & movie\_genet & Confixed with movie\_name \\ Movies\_1 & price & ticket\_price\_total\_price\_No impact & No impact \\ \hline Bueses\_3 & t\_city & target\_t\_time\_time\_The synonyms ”target” is not understood well by model, confused with from city \\ Movies\_1 & movie\_name & film\_name & The synonyms ”film” is not understood well, getting wrong with theatre\_name \\ Hotels\_2 & where\_0, & house\_loc & Improved by specific “house” keywords \\ \hline Flights\_4 & origin\_ airport & orig\_ city\_ airport & More frequently predicted to slot ”destination\_airport” \\ Flights\_4 & destination\_ airport & dest\_city\_ airport & More frequently predicted to slot ”origin\_airport” \\ Media\_3 & subtle\_language & sub\_mag & Missing keyword ’subtitle’ make the slot inactive \\ Flights\_4 & number\_of\_tickets & num\_of\_tickets & No impact \\ \hline \hline \end{tabular}
\end{table}
Table 17: We analyze the confusion matrix of above slots before and after using the paraphrased name. We summarize the extra impact for using each paraphrased name.