SlowGuess/NAACL_2021_OCR huggingface dataset:
The SNIPS evaluation task slightly differs from RESTAURANTS-8K and DSTC8: we thus provide additional details related to fine-tuning and evaluation procedure on SNIPS, replicating the setup of Hou et al. (2020). Each of the 7 domains in turn acts as a held-out test domain, and the other 6 can be used for training. From the held-out test domain, episodes are generated that contain around 5 examples, covering all the slots in the domain. For each domain, we first further pretrain the ConvEx decoder layers (the ones that get fine-tuned) on the other 6 domains: we append the slot name to the template sentence, which allows training on all the slots. This gives a single updated fine-tuned ConvEx decoder model, trained on all slots of all other domains. For each episode, for each slot in the target domain we fine-tune 3 ConvEx decoders. The predictions are ensembled by averaging probabilities to give final predictions. This helps reduce variability and improves prediction quality.
Baseline Models. For  $\text{RESTAURANTS-8K}$  and DSTC8, we compare ConVEx to the current best-performing approaches from Coope et al. (2020): Span-BERT and Span-ConveRT. Both models rely on the same CNN+CRF architecture applied on top of the subword representations transferred from a pretrained BERT(-Base/Large) model (Devlin et al., 2019) (Span-BERT), or from a pretrained
ConveRT model (Henderson et al., 2020). Similar to Coope et al. (2020), for each baseline we run hyper-parameter optimization via grid search, evaluating on the dev set of RESTAURANTS-8K.
For SNIPS, we compare ConVEx to a wide spectrum of different few-shot learning models proposed and compared by Hou et al. (2020).<sup>7</sup>
One crucial difference between our approach and the methods evaluated by Hou et al. (2020) is as follows: we treat each slot independently, using separate ConVEx decoders for each, while the their methods train a single CRF decoder that models all slots jointly. One model per slot is simpler, easier for practical use (e.g., it is possible to keep and manage data sets for each slot independently), and makes pretraining conceptually easier.[8]
Evaluation Measure. Following previous work (Coucke et al., 2018; Rastogi et al., 2019; Coope et al., 2020), we report the average  $F_{1}$  scores for extracting the correct span per user utterance. If the models extract part of the span or a longer span, this is treated as an incorrect span prediction.
Intrinsic (Reddit) Evaluation. ConVEx reaches a precision of  $84.8\%$  and a recall of  $85.3\%$  on the held-out Reddit test set (see Table 2 again), using  $25\%$  random negatives as during pretraining. The ConVEx variant without the auxiliary loss (termed no-aux henceforth) reaches a precision of  $82.7\%$  and a recall of  $83.9\%$ , already indicating the usefulness of the auxiliary loss.[9] These preliminary results serve mostly as a sanity check, suggesting ConVEx's ability to generalize over unseen Reddit data; we now evaluate its downstream task efficacy.
$^{6}$ Coope et al. (2020) also evaluated an approach based on the same CNN+CRF architecture as Span-  $\{\mathrm{BERT},$  ConveRT} which does not rely on any pretrained sentence encoder, and learns task-specific subword representations from scratch. However, that approach is consistently outperformed by Span-ConveRT, and we therefore do not report it for brevity.
7A full description of each baseline model is beyond the scope of this work, and we refer to (Hou et al., 2020) for further details. For completeness, short summaries of each baseline model on SNIPS are provided in the appendix.
Moreover, the methods of Hou et al. (2020) are arguably more computationally complex: at inference, their strongest models (i.e., TapNet and WPZ, see the appendix, run BERT for every sentence in the fine-tuning set (TapNet), or run classification for every pair of test words and words from the fine-tuning set (WPZ). The computational complexity of the ConVEx approach does not scale with the fine-tuning set, only with the number of words in the query sequence.
9While we evaluate the two ConVEx variants also in the slot-labeling tasks later, unless noted otherwise, in all experi
See (Coope et al., 2020) for further technical details.
Figure 2: Average  $F_{1}$  across all slots for (a) RESTAURANTS-8K, and (b) DSTC8, with varying training set sizes.
Weather	Music	Playlist	Book	Search	Restaurant	Creative	Average
Hou et al. (2020)
TransferBERT	59.4	42.0	46.1	20.7	28.2	67.8	58.6	46.1
SimBERT	53.5	54.1	42.8	75.5	57.1	55.3	32.4	52.9
WPZ+BERT	67.8	56.0	46.0	72.2	73.6	60.2	66.9	63.2
TapNet	53.0	49.8	54.9	83.4	63.1	59.8	67.0	61.6
TapNet+CDT	66.5	66.4	68.2	85.8	73.6	64.2	68.5	70.4
L-WPZ+CDT	74.7	56.7	52.2	78.8	80.6	69.6	67.5	68.6
L-TapNet+CDT	71.6	67.2	75.9	84.4	82.6	70.1	73.4	75.0
This work
ConVEx (with aux)	71.5	77.6	79.0	84.5	84.0	73.8	67.4	76.8
Table 4:  $F_{1}$  scores on SNIPS 5-shot evaluation, following the exact setup of Hou et al. (2020). For an overview of the baseline models from Hou et al. (2020), see the original work and short summaries available in the appendix.
Fraction	ConVEx	ConVEx ensemble
1 (8198)	96.0	95.8
1/2 (4099)	94.1	94.2
1/4 (2049)	92.5	92.5
1/8 (1024)	90.6	90.7
1/16 (512)	86.4	88.2
1/32 (256)	81.8	83.9
1/64 (128)	76.0	78.2
1/128 (64)	71.7	73.5
Table 5: Average  $F_{1}$  scores across all slots for RESTAURANTS-8K for ConVEx with and without ensembling. The ConVEx ensemble model fine-tunes 3 decoders per slot, and then averages their output scores.
Evaluation on RESTAURANTS-8K and DSTC8. The main respective results are summarized in Figure 2a and Figure 2b, with additional results available in the appendix. In full-data scenarios all models in our comparison, including the baselines from Coope et al. (2020), yield strong performance reaching  $\geq 90\%$  or even  $\geq 95\%$  average  $F_{1}$  across the board. $^{10}$  However, it is encouraging that Con
VEx is able to surpass the baseline models on average even in the full-data regimes.
Figure 2a and Figure 2b also suggest true benefits of the proposed ConVEx approach: the ability of ConVEx to handle few-shot scenarios well. The gap between ConVEx and the baseline models becomes more and more pronounced as we continue to reduce the number of annotated examples for the labeling task. On RESTAURANTS-8K the gain is still small when dealing with 1,024 annotated examples  $(+2.1F_{1}$  points over the strongest baseline), but it increases to  $+18.4F_{1}$  points when 128 annotated examples are available, and further to  $+31.2F_{1}$  points when only 64 annotated examples are available. We can trace a similar behavior on DSTC8, with gains reported for all the DSTC8 single-domain subsets in few-shot setups.
These results point to the following key conclusion. While pretrained representations are clearly useful for slot-labeling dialog tasks, and the importance of pretraining becomes increasingly important when we deal with few-shot scenarios, the
ments we assume the use of the variant with the aux loss.
10The only exception is Span-BERT's lower performance on the DSTC8 Homes_1 evaluation, see the appendix. In general, as shown previously by Coope et al. (2020) and
revalidated here, conversational pretraining based on response selection (ConveRT) seems more useful for conversational applications than regular LM-based pretraining (BERT).
chosen pretraining paradigm has a profound impact on the final performance. The pairwise cloze pretraining task, tailored for slot-labeling tasks in particular, is more robust and better adapted to few-shot slot-labeling tasks. This also verifies our hypothesis that it is possible to learn effective domain-specific slot-labeling systems by simply fine-tuning a pretrained general-purpose slot labeler relying only on a handful of domain-specific examples.
SNIPS Evaluation (5-Shot). The versatility of ConVEx is further verified in the 5-shot labeling task on SNIPS following Hou et al. (2020)'s setup. The results are provided in Table 4. We report the highest average  $F_{1}$  scores with ConVEx; ConVEx also surpasses all the baselines in 4/7 domains, while the highest scores in the remaining three domains are achieved by three different models from Hou et al. (2020). This again hints at the robustness of ConVEx, especially in few-shot setups, and shows that a single pretrained model can be adapted to a spectrum of slot-labeling tasks and domains.
These results also stand in contrast with the previous findings of Hou et al. (2020) where they claimed "...that fine-tuning on extremely limited examples leads to poor generalization ability". On the contrary, our results validate that it is possible to fine-tune a pretrained slot-labeling model directly with a limited number of annotated examples for various domains, without hurting the generalization ability of ConVEx. In other words, we demonstrate that the mainstream "pretrain then fine-tune" paradigm is a viable solution to sequence-labeling tasks in few-shot scenarios, but with the condition that the pretraining task must be structurally well-aligned with the intended downstream tasks.
Next, we analyze the benefits of model ensembling, as done in the 5-shot SNIPS task, also on RESTAURANTS-8K. The results across different training data sizes are shown in Table 5. While there is no performance difference when a sufficient number of annotated examples is available, the scores suggest that the model ensembling strategy does yield small but consistent improvements in few-shot scenarios, as it mitigates the increased variance that is typically met in these setups.
Pretraining on CC100. We also test the robustness of ConVEx by pretraining it on another large Web-scale dataset: CC100 (Wenzek et al., 2020; Conneau et al., 2020) is a large CommonCrawl corpus available for English and more than 100 other
Figure 3:  $F_{1}$  scores on RESTAURANTS-8K (averaged over all slots) with varying training data sizes when ConVEx is pretrained on Reddit versus CC100.
Figure 4: Average  $F_{1}$  scores for each slot for the Span-ConveRT and ConVEx models trained on  $1/128$  of the RESTAURANTS-8K training set, i.e., 64 examples.
languages. We use the English CC100 portion to pretrain ConVEx relying on exactly the same procedure described in §2, and then fine-tune it as before. First, its intrinsic evaluation on the held-out test set already hints that the CC100-based ConVEx is also a powerful slot labeller: we reach a precision of  $85.9\%$  and recall of  $86.3\%$ . More importantly, the results on RESTAURANTS8K, provided in Figure 3, confirm that another general-purpose corpus can be successfully used to pretrain the ConVEx model. We even observe slight gains on average over the Reddit-based model.
Inductive Bias of ConVEx. In sum, ConVEx outperforms current state-of-the-art slot-labeling models such as Span-ConveRT, especially in low-data settings, where the performance difference is particularly large. The model architectures of Span{BERT, ConveRT} and ConVEx are very similar: the difference in performance thus arises mainly from the pretraining task, and the fact that ConVEx's sequence-decoding layers are pretrained, rather than learned from scratch. We now analyse the inductive biases of ConVEx, that is, how
Figure 5: Performance of the ConVEx decoder across all slots in RESTAURANTS-8K without any fine-tuning.
the pretraining regime and the main assumptions affect its behavior before and after fine-tuning.
First, we analyze per-slot performance on RESTAURANTS-8K, comparing ConVEx (with aux) with Span-BERT and Span-ConveRT. The scores in a few-shot scenario with 64 examples are provided in Figure 4, and we observe similar patterns in other few-shot scenarios. The results indicate the largest performance gap for the slots first name and last name. This is expected, given that by the ConVEx design the keyphrases extracted from Reddit consist of rare words, and are thus likely to cover plenty of names without sufficient coverage in small domain-specific data sets. Nonetheless, we also mark prominent gains over the baselines achieved also for the other slots with narrower semantic fields, where less lexical variability is expected (date and people).
We can also expose ConVEx's built-in biases by applying it with no fine-tuning. Figure 5 shows the results with no slot-specific fine-tuning on RESTAURANTS-8K, feeding the user input as both the template and input sentence. We extract at most one value from each sentence, where the model predicted a value for  $96\%$  of all the test examples,  $16\%$  of which corresponded to an actual labeled slot, and  $86\%$  did not. The highest recalls were for the name slots, and the time slot, which correlates with the slot-level breakdown results from Figure 4.[11]
We have introduced ConVEx (Conversational Value Extractor), a light-weight pretraining and finetuning neural approach to slot-labeling dialog tasks. We have demonstrated that it is possible to learn domain-specific slot labelers even in low-data regimes by simply fine-tuning decoder layers of the pretrained general-purpose ConVEx model. The ConVEx framework has achieved a new leap in performance on standard dialog slot-labeling tasks, most notably in few-shot setups, by aligning the pretraining phase with the downstream fine-tuning phase for slot-labeling tasks.
In future work, we plan to investigate the limits of data-efficient slot labeling, focusing on one-shot and zero-shot setups. We will also apply ConVEx to related tasks such as named entity recognition and conversational question answering.
To the best of our knowledge, the conducted work does not imply any undesirable ethical ramifications. By design and its uncontrollable nature, the Reddit data does encode a variety of societal, gender, and other biases; however, the models pretrained on the Reddit data are always fine-tuned for specific tasks using controlled data, and the Reddit-pretrained models are not used for any text generation nor full-fledged dialogue applications directly. The evaluation data used in this work have been collected in previous work following standard crowdsourcing and data annotation practices.
We would like to thank Yutai Hou for sharing the data and evaluation episodes for the SNIPS evaluation. Thanks to our colleagues at PolyAI for fruitful discussions and critical examinations of this work. We would also like to thank Sam Coope and Tyler Farghly for their help with rerunning and validating Span-BERT and Span-ConveRT.
Rami Al-Rfou, Marc Pickett, Javier Snader, Yun-Hsuan Sung, Brian Strobe, and Ray Kurzweil. 2016. Conversational contextual cues: The case of personalization and history for response ranking. CoRR, abs/1606.00372.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
11The most frequent predictions from non-finetuned ConVEx that do not correspond to a labeled slot on RESTAURANTS-8K give further insight into its inductive biases. The top 10 extracted non-labeled values are in descending order: booking, book, reservation, a reservation, a table, indoors, restaurant, cuisine, outside table, and outdoors. Some of these could be modeled as slot values with an extended ontology, such as indoors or outdoors/outside table.
Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of NeurIPS.
Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of EMNLP, pages 5016-5026.
Tanja Bunk, Daksh Varshneya, Vladimir Vlasov, and Alan Nichol. 2020. DIET: Lightweight language understanding for dialogue systems. CoRR, abs/2004.09936.
Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of EMNLP, pages 169-174.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL, pages 8440-8451.
Sam Coope, Tyler Farghly, Daniela Gerz, Ivan Vulic, and Matthew Henderson. 2020. Span-ConveRT: Few-shot span extraction for dialog with pretrained conversational representations. In Proceedings of ACL, pages 107-121.
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips Voice Platform: An embedded spoken language understanding system for private-by-design voice interfaces. arXiv preprint arXiv:1805.10190, pages 12-16.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named entity recognition task. In Proceedings of the 34th
ACM/SIGAPP Symposium on Applied Computing, page 993-1000.
Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, GP Shrivatsa Bhargav, Dinesh Garg, and Avirup Sil. 2020. Span selection pretraining for question answering. In Proceedings of ACL, pages 2773-2782.
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. CoRR, abs/1705.00652.
Matthew Henderson, Pawel Budzianowski, Inigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, and Tsung-Hsien Wen. 2019a. A repository of conversational datasets. In Proceedings of the 1st Workshop on Natural Language Processing for Conversational AI, pages 1-10.
Matthew Henderson, Inigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2020. ConveRT: Efficient and accurate conversational representations from transformers. In Findings of EMNLP 2020, pages 2161-2174.
Matthew Henderson, Blaise Thomson, and Jason D. Wiliams. 2014. The Second Dialog State Tracking Challenge. In Proceedings of SIGDIAL, pages 263-272.
Matthew Henderson, Ivan Vulic, Daniela Gerz, Inigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019b. Training neural response selection for task-oriented dialogue systems. In Proceedings of ACL, pages 5392-5404.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (GELUs). arXiv preprint arXiv:1606.08415.
Yutai Hou, Wanxiang Che, Yongkui Lai, Zhihan Zhou, Yijia Liu, Han Liu, and Ting Liu. 2020. Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network. In Proceedings of ACL, pages 1381-1393.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. In Proceedings of ICLR.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR (Conference Track).
Jason Krone, Yi Zhang, and Mona Diab. 2020. Learning to classify intents and slot labels given a handful of examples. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 96-108.
Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. 2020. Pre-training via paraphrasing. CoRR, abs/2006.15020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2017. SGDR: stochastic gradient descent with warm restarts. In Proceedings of ICLR.
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. 2020. *DiaoGLUE: A natural language understanding benchmark for task-oriented dialogue*. CoRR, abs/2009.13570.
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. CoRR, abs/2101.00438.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. arXiv preprint arXiv:1909.05855.
Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019. Transfer learning in natural language processing. In Proceedings of NAACL-HLT: Tutorials, pages 15-18.
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. 2019. Green AI. Communications of the ACM.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In Proceedings of NeurIPS, pages 4077-4087.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of ACL, pages 3645-3650.
Charles Sutton and Andrew McCallum. 2012. An introduction to Conditional Random Fields. Foundations and Trends in Machine Learning, 4(4):267-373.
Gökhan Tür and Renato De Mori. 2011. Spoken Language Understanding: Systems for Extracting Semantic Information from Speech. John Wiley & Sons.
Ashish Vaswani, Samy Bengio, Eugene Brevdo, Francois Chollet, Aidan Gomez, Stephan Gouws, Llion Jones, Lukasz Kaiser, Nal Kalchbrenner, Niki Parmar, Ryan Sepassi, Noam Shazeer, and Jakob Uszkoreit. 2018. Tensor2Tensor for neural machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas, pages 193-199.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS, pages 6000-6010.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. 2020. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of LREC, pages 4003-4012.
Jason D. Williams. 2014. Web-style ranking and SLU combination for dialog state tracking. In Proceedings of SIGDIAL, pages 282-291.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of NeurIPS, pages 5754-5764.
When Sung Yoon, Jun Seo, and Jaekyun Moon. 2019. TapNet: neural network augmented with task-adaptive projection for few-shot learning. In Proceedings of ICML, pages 282-291.
Steve Young. 2010. Still talking to machines (cognitively speaking). In Proceedings of INTERSPEECH, pages 1-10.
Steve J. Young. 2002. Talking to machines (statistically speaking). In Proceedings of INTERSPEECH.
Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701.
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. In Proceedings of NAACL-HLT, pages 1208-1218.
A Evaluation Data Statistics
For completeness, we provide the summary stats of the evaluation data used in our work:
Table 6 shows the statistics of the RESTAURANTS-8K data set. The data set is available at:
github.com/PolyAI-LDN/
task-specific-datasets.
Table 7 shows the statistics of the DSTC8 data set. The data set is available at: