SlowGuess/NAACL_2021_OCR huggingface dataset:
Jie Cao *
jcao@cs.utah.edu
yizhngn@amazon.com
Frame-based state representation is widely used in modern task-oriented dialog systems to model user intentions and slot values. However, a fixed design of domain ontology makes it difficult to extend to new services and APIs. Recent work proposed to use natural language descriptions to define the domain ontology instead of tag names for each intent or slot, thus offering a dynamic set of schema. In this paper, we conduct in-depth comparative studies to understand the use of natural language description for schema in dialog state tracking. Our discussion mainly covers three aspects: encoder architectures, impact of supplementary training, and effective schema description styles. We introduce a set of newly designed bench-marking descriptions and reveal the model robustness on both homogeneous and heterogeneous description styles in training and evaluation.
From early frame-driven dialog system GUS (Bobrow et al., 1977) to virtual assistants (Alexa, Siri, and Google Assistant et al.), frame-based dialog state tracking has long been studied to meet various challenges. In particular, how to support an ever-increasing number of services and APIs spanning multiple domains has been a focal point in recent years, evidenced by multi-domain dialog modeling (Budzianowski et al., 2018; Byrne et al., 2019; Shah et al., 2018a) and transferable dialog state tracking to unseen intent/slots (Mrkšić et al., 2017; Wu et al., 2019; Hosseini-Asl et al., 2020).
Recently, Rastogi et al. (2019) proposed a new paradigm called schema-guided dialog for transferable dialog state tracking by using natural language description to define a dynamic set of service schemata. As shown in Figure 1, the primary motivation is that these descriptions can offer effective
"service_name": "Restaurants_1"
"description": "A leading provider for restaurant search and reservations"
"name": "FindRestaurants"
"description": "Find a restaurant of a
particular cuisine in a city"
required_slot": ["cuisine","city"],
"optionalslots": {
"has_live_music": "dontcare",
"serves_alcohol": "dontcare"
Figure 1: An example dialog from Restaurant_1 service, along with its service/intent/slot descriptions and dialog state representation.
"name": "ReserveRestaurant"
"description": "Reserve a table at a
"restaurant"
"required_slot":
["restaurant_name","city","time"],
"name": "culicine"
"description": "Cuisine of food served in the
"is_categorical": false
"possible_values":
[“Mexican”,“Chinese”,“Indian”,
"American", "Italian"]
User: I am looking for Chinese food in SFO
System: I found 10 restaurants in SFO.
User: Any popular one? By the way, I also want to buy a drink.
2 Use: Any popular one? By the way, I also want to buy a drink. System: There is a nice restaurant called Butterfly Restaurant.
System: There is a free restaurant called BuHrynBuR
3 User: Do they have live music? Where are they located?
System: They do not have live music. They are at 33 The Embarcadero.
Dialogue State Tracking
User Turn 1	User Turn 2	User Turn 3
intent	FindRestanurants	intent	FindRestanurants	intent	FindRestanurants
reqSlots	N/A	req Slots	N/A	req Slots	has live music, street_address
slot_values	slot_values
city	SFO	city	SFO
cuisine	Chinese	cuisine	Chinese	slot_values
alcohol	TRUE	city	SFO
cuisine	Chinese
alcohol	TRUE
knowledge sharing across different services, e.g., connecting semantically similar concepts across heterogeneous APIs, thus allowing a unified model to handle unseen services and APIs. With the publicly available schema-guided dialog dataset (SG-DST henceforward) as a testbed, they organized a state tracking shared task composed of four subtasks: intent classification (Intent), requested slot identification (Req), categorical slot labeling (Cat), and noncategorical slot labeling (NonCat) (Rastogi et al., 2020). Many participants achieved promising performance by exploiting the schema description for dialog modeling, especially on unseen services.
Despite the novel approach and promising results, current schema-guided dialog state tracking
*Work done when Jie Cao was an intern at Amazon
Proceedings of the 2021 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, pages 782-796
June 6-11, 2021. ©2021 Association for Computational Linguistics
task only evaluates on a single dataset with limited variation in schema definition. It is unknown how this paradigm generalizes to other datasets and other different styles of descriptions. In this paper, we focus our investigation on the study of three aspects in schema-guided dialog state tracking: (1) schema encoding model architectures (2) supplementary training on intermediate tasks (3) various styles for schema description. To make a more general discussion on the schema-guided dialog state tracking, we perform extensive empirical studies on both SG-DST and MULTIWOZ 2.2 datasets. In summary, our contributions include:
- A comparative study on schema encoding architectures, suggesting a partial-attention encoder for good balance between inference speed and accuracy.
- An experimental study of supplementary training on schema-guided dialog state tracking, via intermediate tasks including natural language inference and question answering.
- An in-depth analysis of different schema description styles on a new suite of benchmarking datasets with variations in schema description for both SG-DST and MULTIWOZ 2.2.
2 Schema-Guided Dialog State Tracking
A classic dialog state tracker predicts a dialog state frame at each user turn given the dialog history and predefined domain ontology. As shown in Figure 1, the key difference between schema-guided dialog state tracking and the classic paradigm is the newly added natural language descriptions. In this section, we first introduce the four subtasks and schema components in schema-guided dialog state tracking, then we outline the research questions in our paper. Subtasks. As shown in Figure 1, the dialog state for each service consists of 3 parts: active intent, requested slots, user goals (slot values). Without loss of generality, for both SG-DST and MULTIWOZ 2.2 datasets, we divide their slots into categorical and non-categorical slots by following previous study on dual-strategies (Zhang et al., 2019). Thus to fill the dialog state frame for each user turn, we solve four subtasks: intent classification (Intent), requested slot identification (Req), categorical slot labeling (Cat), and non-categorical slot labeling (NonCat). All subtasks require matching the current dialog history with candidate schema descriptions for multiple times.
Schema Components. Figure 1 shows three main schema components: service, intent, slot. For each intent, the schema also describes optional or required slots for it. For each slot, there are flags indicating whether it is categorical or not. Categorical means there is a set of predefined candidate values (Boolean, numeric or text). For instance, has_live_music in Figure 1 is a categorical slot with Boolean values. Non-categorical, on the other hand, means the slot values are filled from the string spans in the dialog history.
New Questions. These added schema descriptions pose the following three new questions. We discuss each of them in the following sections.
Q1. How should dialogue and schema be encoded? §5
Q2. How do different supplementary trainings impact each subtask? §6
Q3. How do different description styles impact the state tracking performance? §7
Our work is related to three lines of research: multisentence encoding, multi-domain and transferable dialog state tracking. However, our focus is on the comparative study of different encoder architectures, supplementary training, and schema description style variation. Thus we adopt existing strategies from multi-domain dialog state tracking. Multi-Sentence Encoder Strategies. Similar to the recent study on encoders for response selection and article search tasks Humeau et al. (2019), we also conduct our comparative study on the two typical architectures Cross-Encoder (Bordes et al., 2014; Lowe et al., 2015) and Dual-Encoder (Wu et al., 2017; Yang et al., 2018). However, they only focus on sentence-level matching tasks. All subtasks in our case require sentence-level matching between dialog context and each schema, while the non-categorical slot filling task also needs to produce a sequence of token-level representation for span detection. Hence, we study multi-sentence encoding for both sentence-level and token-level tasks. Moreover, to share the schema encoding across subtasks and turns, we also introduce a simple Fusion-Encoder by caching schema token embeddings in §5.1, which improves efficiency without sacrificing much accuracy.
Multi-domain Dialog State Tracking. Recent research on multi-domain dialog system have been largely driven by the release of large-scale
multi-domain dialog datasets, such as MultiWOZ (Budzianowski et al., 2018), M2M (Shah et al., 2018a), accompanied by studies on key issues such as in/cross-domain carry-over (Kim et al., 2019). In this paper, our goal is to understanding the design choice for schema descriptions in dialog state tracking. Thus we simply follow the in-domain cross-over strategies used in TRADE (Wu et al., 2019). Additionally, explicit cross-domain carryover (Naik et al., 2018) is difficult to generalize to new services and unknown carryover links. We use longer dialog history to inform the model on the dialog in the previous service. This simplified strategy does impact our model performance negatively in comparison to a well-designed dialog state tracking model on seen domains. However, it helps reduce the complexity of matching extra slot descriptions for cross-service carryover. We leave the further discussion for future work.
Transferable Dialog State Tracking. Another line of research focuses on how to build a transferable dialog system that is easily scalable to newly added intents and slots. This covers diverse topics including e.g., resolving lexical/morphological variabilities by symbolic de-lexicalization-based methods (Henderson et al., 2014; Williams et al., 2016), neural belief tracking (Mrkšić et al., 2017), generative dialog state tracking (Peng et al., 2020; Hosseini-Asl et al., 2020), modeling DST as a question answering task (Zhang et al., 2019; Lee et al., 2019; Gao et al., 2020, 2019). Our work is similar with the last class. However, we further investigate whether the DST can benefit from NLP tasks other than question answering. Furthermore, without rich description for the service/intent/slot in the schema, previous works mainly focus on simple format on question answering scenarios, such as domain-slot-type compounded names (e.g., "restaurant-food"), or simple question template "What is the value for slot i?". We incorporate different description styles into a comparative discussion on §7.1.
To the best of our knowledge, at the time of our study, SG-DST and MULTIWOZ 2.2 are the only two publicly available corpus for schema-guided dialog study. We choose both of them for our study. In this section, we first introduce these two representative datasets, then we discuss the generalizability in domain diversity, function overlapping, data collecting methods.
Schema-Guided Dialog Dataset. SG-DST dataset is especially designed as a test-bed for schema-guided dialog, which contains well-designed heterogeneous APIs with overlapping functionalities between services (Rastogi et al., 2019). In DSTC8 (Rastogi et al., 2020), SG-DST was introduced as the standard benchmark dataset for schema-guided dialog research. SG-DST covers 20 domains, 88 intents, 365 slots. However, previous research are mainly conducted based on this single dataset and the provided single description style. In this paper, we further extended this dataset with other benchmarking description styles as shown in §7, and then we perform both homogenous and heterogenous evaluation on it.
Remixed MultiWOZ 2.2 Dataset. To eliminate potential bias from the above single SG-DST dataset, we further add MULTIWOZ 2.2 (Zang et al., 2020) to our study. Among various extended versions for MultiWOZ dataset (2.0-2.3, Budzianowski et al., 2018; Eric et al., 2020; Zang et al., 2020; Han et al., 2020), besides rectifying the annotation errors, MULTIWOZ 2.2 also introduced the schema-guided annotations, which covers 8 domains, 19 intents, 36 slots. To evaluate performance on seen/unseen services with MultiWOZ, we remix the MULTIWOZ 2.2 dataset to include as seen services dialogs related to restaurant, attraction and train during training, and eliminate slots from other domains/services from training split. For dev, we add two new domains hotel and taxi as unseen services. For test, we add all remaining domains as unseen, including those that have minimum overlap with seen services, such as hospital, police, bus. The statistics of data splits are shown in Appendix A.2. Note that this data split is different from the previous work on zero-shot MultiWOZ DST which takes a leave-one-out approach in Wu et al. (2019). By remixing the data in the way described above, we can evaluate the zero-shot performance on MultiWOZ in a way largely compatible with SG-DST.
Discussion. First, the two datasets cover diverse domains. MULTIWOZ 2.2 covers various possible dialogue scenarios ranging from requesting basic information about attractions through booking a hotel room or travelling between cities. While SG-DST covers more domains, such as 'Payments', 'Calender', 'DoctorServices' and so on.
$^{1}$ https://github.com/google-research-datasets/dstc8-schema-guided-dialogue
2Please refer to the original paper for more details.
Datasets	Splits	Dialog	Domains	Services	Zero-shot Domains	Zero-shot Services	Function Overlap	Collecting Method
SG-DST	Train	16142	16	26	-	-	Across-domain Within-domain	M2M
Dev	2482	16	17	1	8
Test	4201	18	21	3	11
MULTIWOZ 2.2	Train	9617	3	3	-	-	Across-domain	H2H
Dev	2455	5	5	2	2
Test	2969	8	8	5	5
Table 1: Summary of characteristics of SG-DST MULTIWOZ 2.2 datasets, in domain diversity, function overlap, data collecting methods
Second, they include different levels of overlapping functionalities. SG-DST allows frequent function overlapping between multiple services, within the same domain (e.g. BookOneWayTicket v.s. BookRoundTripTicket), or across different domains (BusTicket v.s. TrainTicket). However, the overlapping in MULTIWOZ 2.2 only exists across different domains, e.g., 'destination', 'leaveat' slots for Taxi and Bus services, 'pricerange', 'bookday' for Restaurant and Hotel services.
Third, they are collected by two different approaches which are commonly used in dialog collecting. SG-DST is firstly collected by machine-to-machine self-play (M2M, Shah et al., 2018b) with dialog flows as seeds, then paraphrased by crowd-workers. While MULTIWOZ 2.2 are human-to-human dialogs (H2H, Kelley, 1984), which are collected with the Wizard-of-Oz approach.
We summarize the above discussion in Table 1. We believe that results derived from these two representative datasets can guide future research in schema guided dialog.
5 Dialog & Schema Representation and Inference (Q1)
In this section, we focus on the model architecture for matching dialog history with schema descriptions using pretrained BERT (Devlin et al., 2019) $^3$ . To support four subtasks, we first extend Dual-Encoder and Cross-Encoder to support both sentence-level matching and token-level prediction. Then we propose an additional Fusion-Encoder strategy to get faster inference without sacrificing much accuracy. We summarize different architectures in Figure 2. Then we show the classification head and results for each subtask.
Figure 2: Dual-Encoder, Cross-Encoder and Fusion Encoder, shaded block will be cached during training
5.1 Encoder Architectures
Dual-Encoder. It consists of two separate BERTs to encode dialog history and schema description respectively, as Figure 2 (a). We follow the setting in the official baseline provided by DSTC8 Track4 (Rastogi et al., 2020). We first use a fixed BERT to encode the schema description once and cached the encoded schema  $\mathbf{CLS}_S$ . Then for sentence-level representation, we concatenate dialog history representation  $\mathbf{CLS}_D$  and candidate schema representation  $\mathbf{CLS}_S$  as the whole sentence-level representation for the pair, denoted as  $\mathbf{CLS}^{DE}$ . For token-level representation, we concatenate the candidate schema  $\mathbf{CLS}_S$  with each token embedding in the dialog history, denoted as  $\mathbf{TOK}^{DE}$ . Because the candidate schema embeddings are encoded independently from the di
<sup>4</sup>A schema-aware dialog token embedding can also be computed by attention or other method for span-based detection tasks (Humeau et al., 2019; Noroozi et al., 2020)
<sup>3</sup>We use BERT-base-cased for all main experiments. Other pretrained language models can be easily adapted to our study
alog context, they can be pre-computed once and cached for fast inference.
Cross-Encoder. Another popular architecture as Figure 2 (b) is Cross-Encoder, which concatenates the dialog and schema as a single input, and encodes jointly with a single self-attentive encoder spanning over the two segments. When using BERT to encode the concatenated sentence pair, it performs full (cross) self-attention in every transformer layers, thus offer rich interaction between the dialog and schema. BERT naturally produces a summarized representation with [CLS] embedding  $\mathbf{CLS}^{CE}$  and each schema-attended dialog token embeddings  $\mathbf{TOK}^{CE}$ . Since the dialog and schema encoding always depend on each other, it requires recomputing dialog and schema encoding for multiple times, thus much slower in inference.
Fusion-Encoder. In Figure 2 (c), similar to DualEncoder, Fusion-Encoder also encodes the schema independently with a fixed BERT and finetuning another BERT for dialog encoding. However, instead of caching a single [CLS] vector for schema representation, it caches all token representation for the schema including the [CLS] token. What's more, to integrate the sequences dialog token representation with schema token representation, an extra stack of transformer layers are added on top to allow token-level fusion via self-attention, similar to Cross-Encoder. The top transformer layers will produce embeddings for each token  $\mathbf{TOK}^{FE}$  including a schema-attended  $\mathbf{CLS}^{FE}$  of the input [CLS] from the dialog history. With cached schema token-level representations, it can efficiently produce schema-aware sentence- and token-level representation for each dialog-schema pairs.
5.2 Model Overview
All the above 3 encoders will produce both sentence- and token-level representations for a given sentence pair. In this section, we abstract them as two representations CLS and TOK, and present the universal classification heads to make decisions for each subtask.
Active Intent. To decide the intent for current dialog turn, we match current dialog history  $D$  with each intent descriptions  $I_0\dots I_k$ . For each dialog-intent pair  $(D,I_k)$ , we project the final sentence-level CLS representation to a single number  $P_{I_k}^{\text{active}}$  with a linear layer follows a sigmoid function. We predict "NONE" if the  $P_{I_k}^{\text{active}}$  of all intents are less than a threshold 0.5, which means
no intent is active. Otherwise, we predict the intent with largest  $P_{I_k}^{\text{active}}$ . We predict the intent for each turn independently without considering the prediction on previous turns.
Requested Slot. As in Figure 1, multiple requested slots can exist in a single turn. We use the same strategy as in active intent prediction to predict a number  $P_{req}^{active}$ . However, to support the multiple requested slots prediction. We predict all the requested slots with  $P_{req}^{active} > 0.5$
Categorical Slot. Categorical slots have a set of candidate values. We cannot predict unseen values via n-way classification. Instead, we do binary classification on each candidate value. Besides, rather than directly matching with values, we also need to check that whether the corresponding slot has been activated. For Cross-Encoder and Fusion-Encoder, we use typical two-stage state tracking to incrementally build the state: Step 1. Using CLS to predict the slot status as none, dontcare or active. When the status is active, we use the predicted slot value; Otherwise, it will be assigned to dontcare meaning no user preference for this slot, or none meaning no value update for the slot in current turn; Step 2. If Step 1 is active, we match the dialog history with each value and select the most related value by ranking. We train on cross entropy loss. Two-stage strategy is efficient for Dual-Encoder and Fusion-Encoder, where cached schema can be reused, and get efficiently ranked globally in a single batch. However, it is not scalable for Cross-Encoder, especially for large number of candidate values in MultiWOZ dataset. Hence, during training, we only use a binary cross-entropy for each single value and postpone the ranking only to the inference time.
Noncategorical Slot. The slot status prediction for noncategorical slot use the same two-stage strategy. Besides that, we use the token representation of dialog history TOK to compute two softmax scores  $f_{start}^{i}$  and  $f_{end}^{i}$  for each token  $i$ , to represent the score of predicting the token as start and end position respectively. Finally, we find the valid span with maximum sum of the start and end scores.
5.3 Experiments on Encoder Comparison
To fairly compare all three models, we follow the same schema input setting as in Table 2. We trained separate models for SG-DST and the remixed MultiWOZ datasets for all the experiments in our pa
Intent
Req
Cat
NonCat	service description, intent description
service description, slot description
slot description, cat value
service description, slot description
Table 2: Schema description input used for different tasks to compare Dual-Encoder, Cross-Encoder, and Fusion-Encoder. In the appendix A.3, we also studies other compositions of description input. We found that service description will not help for Intent, Req and Cat tasks, while the impact on NonCat task also varies from SG-DST and MULTIWOZ 2.2 dataset.
SG-DST	MULTIWOZ 2.2
Method/Task	Acc Intent	F1 Req	Joint Acc	Joint Acc
Cat	NonCat	All	Cat	NonCat	All
Seen Services
Dual-Encoder	94.51	99.62	87.92	47.77	43.20	79.20	79.34	65.64
Fusion-Encoder	94.90	99.69	88.94	48.78	58.52	81.37	80.58	67.43
Cross-Encoder	95.55	99.59	93.68	91.85	87.58	85.99	81.02	71.93
Unseen Services
Dual-Encoder	89.73	95.20	42.44	31.62	19.51	56.92	50.82	31.83
Fusion-Encoder	90.47	95.95	48.79	35.91	22.85	57.01	52.23	33.64
Cross-Encoder	93.84	98.26	71.55	74.13	54.54	59.85	59.62	38.46
Table 3: Test set results on SG-DST and MULTIWOZ 2.2. The Dual-Encoder model is a re-implementation of official DSTC8 baseline from Rastogi et al. (2019). Other models are trained with the architecture described in our paper.
pers<sup>5</sup>. Because there are very few intent and requested slots in MULTIWOZ 2.2 dataset, we ignore the intent and requested slots tasks for MULTIWOZ 2.2 in our paper.
Results. As shown in Table 3, Cross-Encoder performs the best over all subtasks. Our FusionEncoder with partial attention outperforms the Dual-Encoder by a large margin, epsecially on categorical and noncategorical slots predictions. Additionally, on seen services, we found that DualEncoder and Fusion-Encoder can perform as good as Cross-Encoder on Intent and Req tasks. However, they cannot generalize well on unseen services as Cross-Encoder.
Inference Speed. To test the inference speed, we conduct all the experiments with a maximum affordable batch size to fully exploit 2 V100 GPUs (with 16GB GPU RAM each). During training, we log the inference time of each evaluation on dev set. Both Dual-Encoder and Fusion-Encoder can do joint inference across 4 subtasks to obtain an integral dialog state for a dialog turn example. Dual-Encoder achieves the highest inference speed of 603.35 examples per GPU second, because the