SlowGuess/NAACL_2021_OCR huggingface dataset:
All models are based on BERT-base-cased model with 2 V100 GPUs (with 16GB GPU RAM each). We train each models for maximum 10 epoch, by using AdamW to schedule the learning rate with a warm-up portion of 0.1. During training, we evaluate checkpoints per 3000 steps on dev splits, and select the model with best performance on dev split on all seen and unseen services. In our experiments, our model achieves the best performance on around 2-4 epochs on Intent, Req. and Cat, while NonCat needs 5-8 epochs to get the best performance. For all subtasks, as we model all of them as sentence pair encoding during training, we use batch size as 16 for each GPU, and gradient accumulate for 8 steps, in total 256 batch size on 2 GPUs.
A.2 Statistic on MultiWOZ 2.2 Remix
To evaluate performance on seen/unseen services with MultiWOZ, we remix the MULTIWOZ 2.2 dataset to include as seen services dialogs related to restaurant, attraction and train during training, and eliminate slots from other domains/services from training split. For dev, we add two new domains hotel and taxi as unseen services. For test, we add all
remaining domains as unseen, including those that have minimum overlap with seen services, such as hospital, police, bus. The statistics are as shown in Table 8
Domain	#dialogs/#turns
train	dev	test
restaurant	3900	37953	458	6979	451	7104
attraction	2716	28632	405	6198	400	6290
train	3001	29646	481	5897	491	6150
hotel	0	0	737	8509	718	7911
taxi	0	0	374	2692	364	2659
hospital	0	0	0	0	287	766
police	0	0	0	0	252	475
bus	0	0	0	0	6	132
A.3 Composition of Descriptions
A.3.1 Composition Settings
For each subtask, the key description element must be included, e.g., intent description for intent task, and value for categorical slot tasks. To show how each component helps schema-guided dialog state tracking, we incrementally add richer schema component one by one.
ID. This is the least informative case: we only use meaningless intent/slot identifiers, e.g. Intent_4, Slot_2. It means we don't use description from any schema component. We want to investigate how a simple identifier-based description performs in schema-guided dialog modeling, and the performance lower-bound on transferring to unseen services.
I/S Desc. Only using the original intent/slot description of intent/slot in SG-DST and MULTI-WOZ 2.2 dataset for corresponding tasks.
Service + I/S Desc. Adding a service description to the above original description. Service description summarize the functionalities of the whole service, hence may offer extra background information for intent and slots. For categorical slot value detection, we simply add the value after each of the above composition.
A.3.2 Results on Description Compositions
Table 9 shows the results of using different description compositions. First, there are consistent findings across datasets and subtasks: (1) using meaningless identifier as intent/slot description shows the worse performance on all tasks of both datasets, and can not generalize well to unseen services. (2) using intent/slot descriptions can largely boost the performance, especially on unseen services.
Table 8: The total number of dialogs and turns related to each domain in train, dev and test split of MultiWOZ
Model\Task	SG-DST	MultiWOZ
Intent	Req	Cat	NonCat	Cat	NonCat
Seen Service
Identifier	92.76	99.70	87.86	88.38	58.46	77.29
I/S Desc	95.35	99.74	92.10	93.52	85.84	83.67
Service + I/S Desc	95.28	99.74	93.19	92.34	85.07	80.56
Unseen Service
Identifier	50.63	88.74	54.34	10.77	53.05	56.18
I/S Desc	92.17	98.16	68.88	69.84	56.49	61.39
Service + I/S Desc	86.95	97.99	67.08	71.30	60.58	59.63
Table 9: Models using different composition of schema, results on test set of SG-DST and our remixed MULTIWOZ 2.2
However, the impact of service description varies by tasks. For example, it largely hurts performance on intent classification task, but does not impact requested slot and categorical slot tasks. According to manual analysis of SG-DST and MultiWOZ 2.2 dataset, we found that service description consists of the main functions of the service, especially the meaning of the supported intents. Hence, using service description for intent causes confusion between the intent description information and other supported intents. Moreover, in categorical slot value prediction task, the most important information is the slot description and value. When adding extra information from service description, it improves marginally on seen service while not generalizing well on unseen services, which indicates the model learns artifacts that are not general useful for unseen services.
Finally, on non-categorical slot tasks, the impact of service description may also varies on datasets. On SG-DST, there are 16 domains and more than 30 services, the rich background context from service description contains both domain and service-specific information, which seems to help both seen and unseen services. However, on MULTIWOZ 2.2, it hurts the performance on seen service restaurant the most, while improving the performance on the unseen service hotel by 4 points. In this case, it works like a regularizer rather than a definitive clues. Because in MULTIWOZ 2.2, there are only 8 domains, and one service per domain, thus service descriptions just contain domain related information without much extra information, it will not help the model to detect the span for the slot.
A.4 More Results of Supplementary Training
Table 10 shows the detailed performance when using different intermediate tasks as supplementary training. For SNLI tasks, as the pretrained model is uncased model (textattack/ bert-base-uncased-snli),
sgd	multiwoz
intent	req	cat	noncat	cat	noncat
snli	uncased	93.31	95.4	92.62	98.62	99.34	98.37	75.66	93.39	69.98	80.38	90.93	76.87	51.77	84.93	59.40	56.47	82.39	61.62
snli	93.82	95.42	93.3	98.43	99.72	97.99	74.03	90.52	68.75	75.68	90.83	70.62	53.82	85.53	58.70	60.11	83.44	66.46
ΔSNLI	+0.51	+0.02	+0.68	-0.19	+0.38	-0.38	-1.63	-2.87	-1.23	-4.7	-0.1	-6.25	+2.05	+0.6	-0.7	3.64	+1.05	+4.84
squad	cased	93.01	95.51	92.2	98.59	99.59	98.26	74.51	92.1	71.23	75.76	93.52	69.84	52.19	85.74	56.49	57.2	83.67	61.39
squad	91.2	95.34	90.88	98.34	99.58	97.93	71.64	89.08	66.06	77.75	91.73	73.09	52.23	85.03	56.90	59.13	81.46	65.66
ΔSQuAD	-1.81	-0.17	-1.32	-0.25	-0.01	-0.33	-2.87	-3.02	-5.17	1.99	-1.79	3.25	+0.04	-0.71	+0.41	1.93	-2.21	+4.27
Table 10: Results of different supplementary training on SG-DST and MULTIWOZ 2.2 dataset
style	Intent Description	Slot Description
Identifier	intent_1	slot_4
NameOnly	CheckBalance	account_type
Q-Name	Is the user intending to CheckBalance?	What is the value of acct count_type ?
Orig	Check the amount of money in a user's bank account	The account type of the user
Q-Origin	Does the user want to check the amount of money in the bank account ?	What is the account type of the user ?
Name-Para	CheckAccountBalance	user_account_type
Orig-Para	Check the balance of the user's bank account	Type of the user account
hence, we first train different models with BERT-base-uncased, then compare the performance with SNLI pretrained model. For SQuAD2, we use deepset/bert-base-cased-SQuAD2 model, hence, we compare it all cased model. To fairly compare with our original Cross-Encoder, we add extra speaker tokens [user:] and [system:] for encoding the multi-turn dialog histories.
A.5 Homogeneous and Heterogenous Evaluation on Different Styles
A.5.1 Examples for Different Description Styles
Table 11 shows examples for different styles of schema descriptions.
A.5.2 More details on SQuAD2 Results on Different Styles
For homogeneous evaluation, Table 12 shows the detailed performance when we apply SQuAD2-finetuned BERT on our models.
A.5.3 More Results On Homogeneous and Heterogeneous Evaluation
We list the detailed results for our evaluation across different styles. We use italic to show the homogeneous evaluation, where the results are shown in the diagonal of each table, and we underline the best homogeneous results in the diagonal. We use bold to show the best heterogeneous performance and the best performance gap in the last two columns Intent. The results on SG-DST dataset are shown in Table 13. Because there are very few intents in MULTIWOZ 2.2 dataset, we don't conduct intent classification on MULTIWOZ 2.2. All perfor
Table 11: Different extensions of schema descriptions
Style/Dataset	SG-DST	MULTIWOZ 2.2
all	seen	unseen	all	seen	unseen
Orig	75.76	93.52	69.84	57.2	83.67	61.39
77.75	91.73	73.09	59.13	81.46	65.66
+1.99	-1.79	+3.25	+1.93	-2.21	+4.27
Q-Orig	76.60	92.86	71.18	57.80	82.45	62.45
82.73	90.85	80.02	58.86	81.17	65.51
+6.13	-2.01	+8.84	+1.06	-1.28	+3.06
NameOnly	75.63	88.90	71.21	56.18	81.68	61.30
75.18	87.41	71.10	57.93	82.26	63.07
-0.45	-1.49	-0.11	+1.75	+0.58	+1.77
Q-Name	74.86	91.78	69.22	56.17	81.19	60.47
74.91	88.8	70.26	56.13	80.87	61.72
+0.05	-2.98	+1.04	-0.04	-0.32	+1.25
Table 12: Results on different description style on SG-DST and MULTIWOZ 2.2 dataset, when performing SQuAD2 supplementary training
mance get dropped when evaluating on heterogeneous descriptions styles. For both heterogeneous and homogeneous evaluation, adding rich description on intent classification tasks seems not bring much benefits than simply using the named-based description. As the discussion in §7.2.1, we believe the name template is good enough to describe the core functionality of an intent in SG-DST dataset.
Requested Slot. Table 14 shows the results on SG-DST dataset for the requested slots subtask. We ignore the requested slots in MULTIWOZ 2.2 dataset due to its sparsity. Overall, the requested slot subtask are relatively easy, performances on heterogeneous styles still drops but not much. For both heterogeneous and homogeneous evaluation, the performance are not sensible to rich description.
Categorical Slot. The results on SG-DST and MULTIWOZ 2.2 dataset are shown in Table 15.
Style	NameOnly	Q-Name	Orig	Q-Orig	mean	Δ
NameOnly	93.94	78.27	93.18	75.95	82.47	-11.47
Q-Name	93.18	92.69	93.26	93.36	93.27	+0.58
Orig	81.57	66.42	92.17	90.43	79.47	-12.70
Q-Orig	81.48	79.04	93.19	92.81	84.57	-8.24
Table 13: Accuracy of intent classification subtask with different description styles on unseen services. Train the model on SG-DST dataset for each description in each row, then evaluating on 4 different descriptions styles. The mean are average performance of the remaining 3 descriptions styles. The  $\Delta$  means the performance gap between the mean and the homogeneous performance
Style	NameOnly	Q-Name	Orig	Q-Orig	mean	Δ
NameOnly	98.56	96.01	97.2	97.54	96.92	-1.64
Q-Name	98.37	98.64	97.8	97.48	97.88	-0.76
Orig	97.95	95.78	98.16	98.52	97.42	-0.74
Q-Orig	97.24	95.85	97.00	98.15	96.70	-1.45
When creating MULTIWOZ 2.2 (Zang et al., 2020), the slots with less than 50 different slot values are classified as categorical slots. We noticed that this leads inconsistent results with SG-DST dataset. It is hard to draw a consistent conclusion on the two datasets. According to the definition, we believe SG-DST are more suitable for categorical slot subtasks, we can further verify our guess when more datasets are created for the research of schemaguided dialog in the future.
Non-categorical Slot. We conduct noncategorical slot identification sub-tasks on both SG-DST and MULTIWOZ 2.2 dataset. The results are shown in Table 16. Overall, the rich description performs better on both homogeneous and heterogeneous evaluations.
A.5.4 Qualitative Analysis On Heterogeneous Evaluation
We conduct qualitative analysis on heterogeneous evaluation on named-based description. Table 17 shows how paraphrasing the named-based description impact on the categorical and non-categorical slot prediction tasks.
The first 3 rows at the top are showing the cases of adding modifiers to the name. When the added
Table 14: F1 Score of requested slot classification subtask with different description styles on unseen services. We train the model on SG-DST dataset for the description style in each row, then evaluate on 4 different descriptions styles. The mean are average performance of the remaining 3 descriptions styles. The  $\Delta$  means the performance gap between the mean and the homogeneous performance
Style	NameOnly	Q-Name	Orig	Q-Orig	mean	Δ
SG-DST
NameOnly	68.09	58.41	63.49	62.21	61.37	-6.72
Q-Name	69.01	68.29	68.53	68.12	68.55	+0.26
Orig	70.19	65.91	68.88	69.64	68.58	-0.30
Q-Orig	69.98	65.97	69.26	71.29	68.40	-2.89
MULTIWOZ 2.2
NameOnly	59.24	59.32	59.12	59.29	59.24	0.00
Q-Name	58.64	59.74	58.49	59.43	58.85	-0.89
Orig	59.26	59.91	56.49	58.97	59.38	+2.89
Q-Orig	60.00	60.70	51.18	58.95	57.29	-1.66
Table 15: Joint accuracy of categorical slot Subtask with different description styles on unseen services. Train the model on SG-DST and MULTIWOZ 2.2 datasets respectively for each description style in each row, then evaluate on all 4 descriptions styles. The mean are the average performance of the remaining 3 descriptions styles. The  $\Delta$  means the performance gap between the mean and the homogeneous performance
Style	NameOnly	Q-Name	Orig	Q-Orig	mean	Δ
SG-DST
NameOnly	71.21	49.85	59.8	59.95	56.53	-14.68
Q-Name	66.32	69.22	61.67	60.77	62.92	-6.30
Orig	78.73	51.57	69.84	69.87	66.72	-3.12
Q-Orig	62.6	36.44	69.49	71.18	56.18	-15.00
MULTIWOZ 2.2
NameOnly	61.30	57.88	61.51	64.05	61.15	-0.15
Q-Name	60.62	60.47	60.6	62.58	61.27	+0.80
Orig	61.77	65.4	61.39	62.4	63.19	+1.80
Q-Orig	61.29	60.6	62.46	62.45	61.45	-1.00
Table 16: Joint accuracy of non-categorical slot Subtask with different description styles on unseen services. We train the model on SG-DST and MULTI-WOZ 2.2 datasets respectively for the description style in each row, then evaluate on all 4 different descriptions styles. The mean are the average performance of the remaining 3 descriptions styles. The  $\Delta$  means the performance gap between the mean and the homogeneous performance
extra modifiers are keywords in other slots, e.g. "attraction" are the keywords also used in "attraction_name". The first shows "attraction_location" may wrongly predicted as "attraction_name". It seems the model does not understand the compound nouns well, and they seems just pay attention to each key words "attraction" and "movie" here.
The 3 rows in the middle are showing the cases of using synonyms. Changing "to" to "target", and changing "movie" to "film" will cause extra confusion, which shows the model may fail to the synonyms.
The last 4 rows at the bottom is showing using abbreviations. Changing "number" to "num" will not impact the model, while changing "subtitle" to "sub" may let the model miss the key meaning of subtitle. The performance drop in the later case may be due to the misuse of the "sub" prefix, in En
Service Name	Original Name	Paraphrased Name	Extra impact by the paraphrased name
Travel_1	location	attraction_location	Confused with other "attraction" prefixed slots, e.g. attraction_name
Movies_1	genre	movie_genre	Confused with movie_name
Movies_1	price	ticket_price, total_price	No impact
Bueses_3	to_city	target_city	The synonyms "target" is not understood well by model, confused with from_city
Movies_1	movie_name	film_name	The synonyms "film" is not understood well, getting wrong with theather_name
Hotels_2	where_to	house_location	Improved by specific "house" keywords
Flights_4	origin_airport	orig_city_airport	More frequently predicted to slot "destination_airport"
Flights_4	destination_airport	dest_city_airport	More frequently predicted to slot "origin_airport"
Media_3	subtitle_language	sub-lang	Missing keyword "subtitle" make the slot inactive
Flights_4	number_ofTickets	num_ofTickets	No impact
Table 17: We analyze the confusion matrix of above slots before and after using the paraphrased name. We summarize the extra impact for using each paraphrased name.
glish, it usually means "secondary, less important, parts". We also found the "orig" and "dest" abbreviations may also understand well by the model. The above abbreviations seems reasonable paraphrases people will use for naming, while the are not understood well in the given context. Hence, in the design of schema-guided dialog, if using named-based description, we should be careful for about abbreviations used in the naming.