SlowGuess/ACL_2020_OCR huggingface dataset:
Anuj Kumar Goyal, Angeliki Metallinou, and Spyros Matsoukas. 2018. Fast and scalable expansion of natural language understanding functionality for intelligent agents. In Proceedings of NAACL-HLT, pages 145-152.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS Spoken Language Systems Pilot Corpus. In Proceedings of the Workshop on Speech and Natural Language, HLT '90, pages 96-101.
Matthew Henderson, Inigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2019a. ConveRT: Efficient and accurate conversational representations from transformers. CoRR, abs/1911.03688.
Matthew Henderson, Ivan Vulic, Daniela Gerz, Inigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019b. Training neural response selection for task-oriented dialogue systems. In Proceedings of ACL, pages 5392-5404.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. In Proceedings of ICLR.
Aaron Jaech, Larry P. Heck, and Mari Ostendorf. 2016. Domain adaptation of recurrent neural networks for natural language understanding. In Proceedings of INTERSPEECH, pages 690-694.
Rahul Jha, Alex Marin, Suvamsh Shivaprasad, and Imed Zitouni. 2018. Bag of experts architectures for model reuse in conversational language understanding. In Proceedings of NAACL-HLT, pages 153-161.
Kunho Kim, Rahul Jha, Kyle Williams, Alex Marin, and Imed Zitouni. 2019. Slot tagging for task oriented spoken language understanding in human-to-human conversation scenarios. In Proceedings of CoNLL, pages 757-767.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, pages 260-270.
Liliana Laranjo, Adam G. Dunn, Huong Ly Tong, Ahmet Baki Kocaballi, Jessica Chen, Rabia Bashir, Didi Surian, Blanca Gallego, Farah Magrabi, Annie Y.S. Lau, and Enrico Coiera. 2018. Conversational agents in healthcare: A systematic review. Journal of the American Medical Informatics Association, 25(9):1248-1258.
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry P. Heck. 2018. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In Proceedings of NAACL-HLT, pages 2060-2069.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of ACL, pages 1064-1074.
Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. In Proceedings of ACL, pages 3836-3845.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of ICML, pages 1310-1318.
Prajit Ramachandran, Barret Zoph, and Quoc V. Le. 2017. Searching for activation functions. CoRR, abs/1710.05941.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855.
Antoine Raux, Brian Langner, Alan W. Black, and Maxine Eskenazi. 2003. LET's GO: Improving spoken dialog systems for the elderly and non-natives. In Proceedings of EUROSPEECH.
Gökhan Tür, Dilek Z. Hakkani-Tür, and Larry P. Heck. 2010. What is left to be understood in atis? In Proceedings of SLT, pages 19-24.
Tsung-Hsien Wen, Yishu Miao, Phil Blunsom, and Steve J. Young. 2017a. Latent intention dialogue models. In Proceedings of ICML, pages 3732-3741.
Tsung-Hsien Wen, David Vandyke, Nikola Mrkšić, Milica Gašić, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017b. A network-based end-to-end trainable task-oriented dialogue system. In Proceedings of EACL, pages 438-449.
Jason Williams. 2012. A critical analysis of two statistical spoken dialog systems in public use. In Proceedings of SLT.
Jason D. Williams. 2014. Web-style ranking and SLU combination for dialog state tracking. In Proceedings of SIGDIAL, pages 282-291.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation. CoRR, abs/1904.12848.
Steve Young. 2010. Still talking to machines (cognitively speaking). In Proceedings of INTERSPEECH, pages 1-10.
Steve J. Young. 2002. Talking to machines (statistically speaking). In Proceedings of INTERSPEECH.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of NAACL-HLT, pages 1208-1218.
A DSTC8 Datasets: Data Extraction and Statistics
As discussed in §3, we extract span annotated data sets from the Schema Guided Dialog Dataset (SGDD) in four different domains. SGDD is a multi-domain data set with each domain consisting of several sub-domains. As the data set has been built for transfer learning from one domain to another, many sub-domains only exist in either the training or development data sets. We are interested in single-domain dialog, and therefore chose datasets from four different domains of the original dataset: (1) bus and coach booking, (2) buying tickets for events, (3) property viewing and renting cars. We select these domains due to their high number of conversations and their large variety of slots (e.g. area of city to view an apartment, type of event to attend, time/date of coach to book). For each of these domains, we chose their first sub-domain<sup>6</sup>, and took all turns from conversations that stay within this sub-domain. For the requested slots feature, we check for when the system action of the turn prior contains a REQUEST action. The training and development split is kept the same for all extracted turns. Table 5 shows the resulting data set sizes for each sub-domain. We are releasing these filtered single-domain data sets, along with the code to create them from the original SGDD data.
<sup>6</sup>We refer to them by their corresponding ID in the original data set: Buses_1, Events_1, Homes_1, RentalCars_1
Sub-domain	Train Size	Dev Size	Slots
Buses_1	1133	377	from_location (169/54), leaving_date (165/57), to_location (166/52)
Events_1	1498	521	city_of_event (253/82), date (151/33), subcate-gory (56/26)
Homes_1	2064	587	area (288/86), visit_date (237/62)
RentalCars_1	874	328	dropoff_date (112/42), pickup_city (116/48), pickup_date (120/43), pickup_time (119/43)
Table 5: Statistics of the used data sets extracted from the DSTC8 schema-guided dialog dataset. We also report the number of examples in the train and development sets for each slot in parentheses.
Slot	Fraction	Span-ConveRT	V-CNN-CRF	Span-BERT
date	1	0.96	0.95	0.92
1/2	0.95	0.94	0.90
1/4	0.93	0.93	0.86
1/8	0.91	0.88	0.84
1/16	0.86	0.82	0.76
1/32	0.83	0.70	0.62
1/64	0.76	0.64	0.21
1/128	0.58	0.43	0.20
first_name	1	0.97	0.93	0.92
1/2	0.95	0.92	0.92
1/4	0.93	0.88	0.85
1/8	0.93	0.85	0.82
1/16	0.81	0.65	0.53
1/32	0.54	0.30	0.19
1/64	0.45	0.23	0.02
1/128	0.19	0.09	0.00
last_name	1	0.97	0.92	0.93
1/2	0.96	0.88	0.92
1/4	0.94	0.83	0.89
1/8	0.90	0.78	0.72
1/16	0.80	0.67	0.71
1/32	0.51	0.45	0.30
1/64	0.33	0.07	0.01
1/128	0.24	0.04	0.00
people	1	0.96	0.95	0.91
1/2	0.94	0.93	0.90
1/4	0.91	0.92	0.87
1/8	0.88	0.87	0.80
1/16	0.83	0.79	0.79
1/32	0.73	0.63	0.58
1/64	0.68	0.49	0.43
1/128	0.60	0.39	0.29
time	1	0.95	0.95	0.91
1/2	0.93	0.94	0.89
1/4	0.91	0.91	0.86
1/8	0.88	0.89	0.82
1/16	0.76	0.85	0.76
1/32	0.62	0.76	0.67
1/64	0.53	0.52	0.46
1/128	0.43	0.36	0.37
Table 6: F1 scores for each slot in the Restaurants8k dataset.
B Experimental Results on RESTAURANTS-8K and DSTC8:  $F_{1}$  Scores for Each Slot
Dataset	Slot	Fraction	ConveRT Reps	Vanilla Reps	BERT Reps.
Buses_1	from_location	1	0.93	0.94	0.87
1/2	0.78	0.80	0.75
1/4	0.82	0.77	0.72
1/8	0.71	0.67	0.52
1/16	0.53	0.54	0.35
leaving_date	1	0.96	0.95	0.96
1/2	1.00	0.88	0.95
1/4	0.96	0.88	0.89
1/8	0.91	0.81	0.72
1/16	0.79	0.61	0.57
to_location	1	0.87	0.89	0.84
1/2	0.82	0.81	0.81
1/4	0.82	0.65	0.79
1/8	0.75	0.64	0.61
1/16	0.49	0.44	0.38
Events_1	city_of_event	1	0.94	0.94	0.90
1/2	0.92	0.91	0.85
1/4	0.90	0.80	0.81
1/8	0.74	0.68	0.51
1/16	0.80	0.72	0.58
date	1	0.90	0.88	0.89
1/2	0.88	0.91	0.91
1/4	0.84	0.83	0.79
1/8	0.74	0.62	0.57
1/16	0.77	0.53	0.68
subcategory	1	0.90	0.94	0.58
1/2	0.78	0.71	0.42
1/4	0.68	0.70	0.50
1/8	0.46	0.30	0.00
1/16	0.40	0.31	0.00
Homes_1	area	1	0.97	0.98	0.94
1/2	0.93	0.90	0.90
1/4	0.93	0.87	0.86
1/8	0.87	0.76	0.72
1/16	0.81	0.64	0.56
visit_date	1	0.98	0.93	0.99
1/2	0.98	0.89	0.98
1/4	0.98	0.88	0.89
1/8	0.96	0.87	0.88
1/16	0.95	0.73	0.83
RentalCars_1	dropoff_date	1	0.93	0.89	0.88
1/2	0.89	0.87	0.72
1/4	0.73	0.58	0.70
1/8	0.64	0.71	0.46
1/16	0.62	0.48	0.33
pickup_city	1	0.88	0.84	0.86
1/2	0.86	0.75	0.85
1/4	0.83	0.65	0.71
1/8	0.74	0.60	0.49
1/16	0.53	0.15	0.10
pickup_date	1	0.86	0.87	0.87
1/2	0.76	0.74	0.81
1/4	0.74	0.70	0.72
1/8	0.71	0.53	0.58
1/16	0.47	0.26	0.42
pickup_time	1	0.98	0.95	0.95
1/2	0.98	0.96	0.91
1/4	0.95	0.81	0.84
1/8	0.91	0.50	0.69
1/16	0.85	0.33	0.68
Table 7: F1 scores for all of the slots in the DSTC8 single-domain experiments
C Quantitative Error Analysis of Span-ConveRT and V-CNN-CRF on RESTAURANTS-8K
We divide the errors into four categories:
1. The model predicted no span when there was a span present.
2. The model predicted a span when no span was present.
3. The model predicted a span which does not overlap the label span.
4. The model predicted a span which overlaps label span.
When training on the full training set (Figure 3), there is little difference in error breakdown between Span-ConveRT and V-CNN-CRF. This suggests the behavior of these models is similar when trained in a high-data setting, but improvements made by Span-ConveRT are on all fronts.
When trained on a 16th of the dataset (Figure 4), the difference between the models becomes more pronounced. Most notably, the Span-ConveRT model produces a greater proportion of type 4 errors compared to the V-CNN-CRF model on every slot. This suggests that the errors Span-ConveRT makes, although not precisely correct with its span prediction, are more likely to yield a span that could parse to a correct value. For example, consider the sentence "a table for 8pm this evening". The correct span for the slot time is "8pm", but if a model erroneously predicts "8pm this evening" (a span which overlaps the label span) it will still parse to the same time as the label span.
Figure 3: Breakdown of errors made on the test set of RESTAURANTS-8K after training on the entire train set.
Figure 4: Breakdown of errors made on the test set of RESTAURANTS-8K after training on a 16th of the train set.
D Qualitative Error Analysis of Span-ConveRT and V-CNN-CRF on RESTAURANTS-8K
As an accompaniment to the quantitative results, we provide a brief qualitative analysis of errors in the best performing models. Considering only the first name slot, we collect the errors made on the test set that are exclusive to each model. That left 10 errors for Span-ConveRT and 50 for V-CNN-CRF. Along with our analysis based on the full set of 60 errors, we provide a random sample of 5 errors from each model in Tables 8 and 9.
A large portion of the errors exclusively made by V-CNN-CRF were predictions of spans where no name was mentioned. Many words that are not standard to the domain of restaurant booking were, often confidently, wrongly predicted as names. For example, in Table 9 we show that the words "bloody", "web", "animal" and "spread" were all predicted as first names by the baseline model. Employing transferred conversational representations evidently lessens the likelihood of these forms of errors occurring. Also included in the table is an example where the baseline model fails to recognize a name which, when corroborated with similar occurrences in the wider set of errors, suggests that it is less likely to predict spans for out-of-vocabulary names than Span-ConveRT.
As well as backing up the conclusions formed by our numerical results, we were also interested in what ways using pretrained representations might hinder performance. With only 10 errors exclusively made by Span-ConveRT it was not possible to form any sweeping conclusions but a handful of errors suggest that the model might employ its background knowledge to reject unfamiliar first names or accept familiar ones in spite of the sentence structure suggesting otherwise. For example, in the first row of Table 8 we find that the model rejects the name "Wen" despite it being part of a fairly common exchange for this domain and in a natural place for a first name. The other examples demonstrate that the model can sometimes predict last names as first names and in spite of contextual cues suggesting otherwise, can do so over-confidently.
Probability	Text/Spans
N/A	Wen Books, for 7:15PM, I made a reservation yesterday for a party of 8
0.4447	Saul
0.9685	Adragna
0.9247	last name Prader
0.9553	Verjan
Table 8: Random sample of errors exclusively made by Span-ConveRT for the slot first name. Red text denotes incorrectly predicted spans and orange denotes true spans that were not predicted.
Probability	Text/Spans
0.8872	bloody useless
0.3939	What is their web URL?
0.3319	ok are you guys animal friendly
0.8604	My 7 friends and I can spread ourselves over two tables if necessary
N/A	Gertrudis Hayslett
Table 9: Random sample of errors exclusively made by V-CNN-CRF for the slot first name. Red text denotes incorrectly predicted spans and orange denotes true spans that were not predicted.