SlowGuess/NAACL_2022_OCR huggingface dataset:
$\mathbf{N L U} + + : \mathbf{A}$  Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue
Inigo Casanueva,\* Ivan Vulic,\* Georgios P. Spithourakis, and Pawe Budzianowski
{inigo,ivan,georgios,pawel}@poly.ai
We present NLU++, a novel dataset for natural language understanding (NLU) in task-oriented dialogue (ToD) systems, with the aim to provide a much more challenging evaluation environment for dialogue NLU models, up to date with the current application and industry requirements. NLU++ is divided into two domains (BANKING and HOTELS) and brings several crucial improvements over current commonly used NLU datasets. 1) NLU++ provides fine-grained domain ontologies with a large set of challenging multi-intent sentences, introducing and validating the idea of intent modules that can be combined into complex intents that convey complex user goals, combined with finer-grained and thus more challenging slot sets. 2) The ontology is divided into domain-specific and generic (i.e., domain-universal) intent modules that overlap across domains, promoting cross-domain reusability of annotated examples. 3) The dataset design has been inspired by the problems observed in industrial ToD systems, and 4) it has been collected, filtered and carefully annotated by dialogue NLU experts, yielding high-quality annotated data. Finally, we benchmark a series of current state-of-the-art NLU models on NLU++. the results demonstrate the challenging nature of the dataset, especially in low-data regimes, the validity of 'intent modularisation', and call for further research on ToD NLU.
Research on task-oriented dialogue (ToD) systems (Levin and Pieraccini, 1995; Young et al., 2002) has become a key aspect in industry: e.g., ToD is used to automate telephone customer service tasks ranging from hospitality over healthcare to banking (Raux et al., 2003; Young, 2010; El Asri et al., 2017). Typical ToD systems still rely on a modular design: (i) the natural language understanding
Figure 1: Multi-intent examples from the two domains of the NLU++ dataset: BANKING (top) and HOTELS (middle, bottom), illustrating the two core NLU subtasks of intent detection (ID) and slot labeling (SL) in ToD systems. The extracted information is structured into intents and slots, the latter having associated values.
(NLU) module maps user utterances into a domain-specific set of intent labels and values (Rastogi et al., 2019; Heck et al., 2020; Dai et al., 2021), followed by (ii) the policy module, which makes decisions based on the information extracted by the NLU (Gašić et al., 2012; Casanueva et al., 2017; Lubis et al., 2020; Wang et al., 2020a)
The NLU module is a critical part of any ToD system, as it must extract the relevant information from the user's utterances. The information relevance is denoted by the structured dialogue domain ontology, which enables the policy module to make decisions about next system actions. The domain ontology covers the information on 1) intents and 2) slots, see Figure 1. The former is aimed at extracting general conversational ideas (i.e., the user's intents) and corresponds to the standard NLU task of intent detection (ID); the latter extracts specific slot values and corresponds to the NLU task of slot labeling (SL) (Gupta et al., 2019).
In order to make the policy operational and tractable, NLU should extract only the minimal information required by the policy. Therefore, the ontologies differ for each domain of ToD application and are typically built from scratch for each domain.
1Slot labeling is also known under other names such as slot filling or value extraction.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1998 - 2013
Example	Traditional Intent	Intent Modules
I need to change my restaurant reservation	change_restaurantbooking	change, restaurant, booking
When is my booking for the spa?	when_spaBooking	when, spa, booking
TV is not showing any image	tv_NOT_working	tv, not-working
Why can’t I cancel this standing order?	why Cancel Standing_order_not_Working	why, cancel, standing_order, not_WORKing
Table 1: Comparison of "traditional" intent annotations vs intent module-based multi-label annotations.
Consequently, this makes domain-relevant NLU data extremely expensive to collect and annotate, and prevents its reusability (Budzianowski et al., 2018). Due to this, NLU research in recent years has heavily focused on very data-efficient models that can effectively operate in low-data regimes. Current state-of-the-art (SotA) NLU models leverage large pretrained language models (PLMs) (Devlin et al., 2019; Liu et al., 2019c; Henderson et al., 2020) and fine-tune them with small task-specific datasets (Larson et al., 2019b; Casanueva et al., 2020; Coucke et al., 2018)
At the same time, the progress in creation of NLU datasets has not kept up with the impressive pace of NLU methodology development. However, designing domain ontologies and NLU datasets is also critical for steering further progress in NLU, both from methodology and application perspective. Put simply, current publicly available NLU datasets do not keep up to date with current industry/application requirements for many reasons. 1) They are usually crowdsourced by untrained annotators (thus typically optimised for quantity rather than quality), yielding examples with low lexical diversity and prone to annotation errors. 2) They typically assume one intent per example, and thus enable only much simpler single-label ID experiments; such setups are not realistic in more complex industry settings (see Figure 1 again) and lead to unnecessarily large intent sets. 3) Their ontologies are tied to specific domains, making it difficult to reuse already available annotated data in other domains. 4) The complexity of the defined tasks and ontologies is limited; the undesired artefact is that current NLU datasets might overestimate the NLU models' abilities, and are not able to separate models any more performance-wise.
In order to address all these gaps, we introduce NLU++, a novel NLU dataset which provides high-quality NLU data annotated by dialogue experts. NLU++ provides multi-intent, slot-rich and semantically varied NLU data, and is inspired by a number of NLU challenges which ToD systems typically face in production environments. Unlike previous ID datasets, examples are annotated with multiple labels, named intent modules (see Table 1), with some examples naturally obtaining even up to 6-7 labels. These labels can be seen as sub-intent annotations, where their combinations yield full intents equivalent to "traditional" intents (Table 1). In addition, NLU++ defines a rich set of slots which are combined with the multi-intent sentences. NLU++ is divided into two domains (BANKING and HOTELS) where the two domain ontologies blend a set of domain-specific intents and slots with a set of generic (i.e., domain-universal) intents and slots. This design makes a crucial step towards generalisation and data reusability in NLU.
Finally, we run a series of experiments on NLU++ with current SotA ID and SL models, demonstrating the challenging nature of NLU++ and ample room for future improvement, especially in low-data setups. Our benchmark comparisons also demonstrate strong performance and shed new light on the (ability of) recently emerging QA-based NLU models (Namazifar et al., 2021; Fuisz et al., 2022), and warrant further research on ToD NLU. The NLU++ dataset is available at: github.com/PolyAI-LDN/task-specific-datasets.
A Brief History of NLU Datasets. As a core module of ToD systems, NLU has been researched since the early 1990s, when the Airline Travel Information System (ATIS) project was started (Hemphill
Henceforth, whenever intents are mentioned in the context of NLU++, we will be referring to intent modules.
For instance, for some standard and commonly used NLU datasets such as ATIS (Hemphill et al., 1990; Xu et al., 2020) and SNIPS (Coucke et al., 2018), the results of SotA models are all in the region of  $97 - 98F_{1}$ , with new models getting statistically insignificant gains which might be due to overfitting to the test set or even some remaining annotation errors.
Domain	Number of examples	INTENTS	SLOTS
Total	Generic	Avg. per example	Total	Generic	Avg. per example
BANKING	2,071	48	26	2.25	13	10	0.46
HOTELS	1,009	40	26	1.52	14	10	1.03
ALL	3,080	62	26	2.01	17	10	0.65
Table 2: Key statistics of the NLU++ dataset.
et al., 1990), consisting of spoken queries on flight-related information. Over the next two decades, very few NLU resources were released.
The lack of ToD NLU resources ended in 2013, with the beginning of the 'dialogue state tracking (DST) era' (Williams et al., 2013; Henderson et al., 2014; Kim et al., 2016). Instead of just classifying each turn of the user, DST deals with keeping track of the user's goal over the entire dialogue history, i.e., all the previous user and system turns. Several datasets where released during the DST challenges, all of them comprising simple intent sets (usually tagged as dialogue acts).
In order to adapt to the increasing data requirements of deep learning models, increasingly larger dialogue datasets have been released in recent years (Budzianowski et al., 2018; Wei et al., 2018; Rastogi et al., 2019; Peskov et al., 2019). However, the design of ToD datasets comes with some profound differences to datasets for e.g. machine translation or speech recognition, which affect current ToD datasets. 1) The domain-specific nature of ToD datasets made the data tied to its ontologies, not allowing data reusability across different domains. 2) The domain-specific ontologies required a lot of expertise for annotation, therefore many annotation mistakes were made (Eric et al., 2019; Zang et al., 2020). 3) Collecting datasets of that size is unfeasible for development cycles in production, where new domains and models for them need to be very quickly developed and deployed.
Current NLU Trends, inspired by such production requirements, thus deviate from previous DST-oriented NLU research in two main aspects. First, the models went back to focusing on single-turn utterances, which 1) simplifies the NLU design and
2) renders the NLU tasks more tractable. The requirement of fast development cycles also instigated more research on NLU (i.e., ID and SL tasks) in low-data scenarios. This way, systems can be developed and maintained faster by reducing the data collection and annotation effort. In addition, the NLU focus shifted from ontologies with only a handful of simple intents and slots (Coucke et al., 2018) to complex ontologies with much larger intent sets (Larson et al., 2019b; Liu et al., 2019b; Casanueva et al., 2020, inter alia).
Inspired by these NLU datasets and empowered by transfer learning with PLMs and sentence encoders (Devlin et al., 2019; Liu et al., 2019a; Henderson et al., 2020), there have been great improvements in single-turn NLU systems recently, especially in low-data scenarios (Coope et al., 2020; Mehri and Eric, 2021; Wu et al., 2020b,a; Krone et al., 2020; Henderson and Vulic, 2021; Namazifar et al., 2021; Dopierre et al., 2021; Zhang et al., 2021a,b).
Current Gaps in NLU Datasets. However, existing NLU datasets are still not up to the current industry requirements. 1) They use crowdworkers for data collection and annotation, often through simple rephrasings; they thus suffer from low lexical diversity and annotation errors (Larson et al., 2019a). 2) ID datasets always assume a single intent per sentence, which does not support modern production requirements. 3) The ontologies of these datasets are very domain-specific (i.e., they thus do not allow data reusability) and narrow (i.e., they tend to overestimate abilities of the current SotA NLU models). 4) Current NLU datasets do not combine a large set of fine-grained intents (again, with multi-intent examples) and a large set of fine-grained slots, which prevents proper and more insightful evaluations of joint NLU models (Chen et al., 2019; Gangadharajaiah and Narayanaswamy, 2019).
While DST is theoretically more accurate, it requires amounts of data that grow exponentially with the number of turns; moreover, rule-based trackers have proven to be on par with the learned/statistical ones and require no data (Wang and Lemon, 2013).
Remarkably, ATIS is still considered at present as one of the main go-to datasets in NLU research. This is also reflected in the fact that the recent most popular dataset for multilingual dialogue NLU was obtained by simply translating English ATIS to 8 more languages (Xu et al., 2020, MultiATIS++)
<sup>5</sup>We note that some Question Classification (Hovy et al., 2001), Paraphrasing (Dolan and Brockett, 2005) and Semantic Text Similarity(Agirre et al., 2012) datasets could be seen as the seed of modern ID datasets, but were not initially built for that purpose.
Example	Intents	Domain
I want to change my room reservation	change, booking, room	HOTELS
I want to cancel a booking	cancel, booking	HOTELS
Why can't I amend my restaurant booking?	why, change, restaurant, booking, not-working	HOTELS
I am trying to make a transfer but it doesn't let me	make, transfer-payment, not-working	BANKING
I need to increase my overdraft	change, overdraft, higher	BANKING
Please close my savings account	cancel, account, savings	BANKING
The savings one	savings	BANKING
Make it higher	change, higher	GENERAL
Cancel it	cancel	GENERAL
Don't cancel it	deny, cancel	GENERAL
Table 3: NLU++ examples showing the combinatorial expressiveness of intent modules in the multi-intent setting.
We note that there has been some work on multi-label ID on ATIS, MultiWOZ and DSTC4 as multi-intent datasets; however, their multi-label examples remain very limited, simple, and span a small number of intents (Gangadharaiah and Narayanaswamy, 2019). Further, synthetic multi-intent datasets have been created by concatenating single-intent sentences, but such datasets also do not capture the complexity of true and natural multi-intent sentences (Qin et al., 2020).
3 NLU++ Dataset
The NLU++ dataset has been designed with the aim of addressing some of the major shortcomings of the current NLU datasets. In what follows, we describe the main improvements and new evaluation opportunities offered by NLU++.
3.1 Ontology
$\mathrm{NLU}++$  comprises two domains: BANKING and HOTELS. The former represents a banking services task (e.g., making transfers, depositing cheques, reporting lost cards, requesting mortgage information) and the latter is a hotel 'bell desk' reception task (e.g., booking rooms, asking about pools or gyms, requesting room service). Both domains combine a large set of intents with a rich set of slots, with the ontologies inspired by requirements in production. A large number of intents and slots is shared between the two domains, in an attempt to increase data reusability/transferability. Table 2 provides the main statistics of the  $\mathrm{NLU}++$  dataset, while the full ontology is presented in Appendix A.
3.2 Multi-Intent Examples
One of the main contributions of this work is the novel design of the intent space, defined in a highly modular manner that natively supports intent re
combinations and multi-intent annotations<sup>7</sup>. For instance, Table 3 shows several multi-intent examples based on the intent sets (termed intent modules) from Table 9 in Appendix A.
This design brings several benefits. 1) The modular nature of the ontology allows for expressing a much more complex set of ideas through different combinations of intent modules (see Table 3), while reducing the overall size of the intent set compared to previous ID datasets<sup>8</sup> (see Table 1 and Table 5). 2) It allows for the definition of partial intents (e.g., "The savings one"). This is crucial in multi-turn interactions, where the user often has to answer disambiguation questions (e.g., "Which account would you like to close?"). 3) The modular approach allows the models to generalise to unseen combinations of intent modules. For instance, if (i) examples with the intents change and booking, and (ii) examples with the intents cancel and account exist in the training data, (iii) an unseen example with the intents cancel and booking could be properly predicted, as all the single intents/modules have already been seen by the ID model<sup>9</sup>. 4) The design also allows us to distinguish between domain-specific versus generic intent modules. For example, the module overdraft is clearly related to BANKING, but the module change is much more generic, likely to occur in several different domains.
Finally, the modular design also allows us to
$^{7}$ Zhang et al. (2020) proposed a similar way of annotating existing intent detection datasets, showing performance improvements. However, this approach forced categorising the sub-intents in four predefined factors.
Similar to how sub-word tokenization reduced the size of language model vocabularies while covering a larger set of words (Vaswani et al., 2018)
Note that in single-label ID setups, all possible intent module combinations (i.e. "traditional" intents) must be covered (Bi and Kwok, 2013; Hou et al., 2021), which leads to unnecessarily large intent sets and larger data requirements.
study semantic variation of intent modules. Some intents (e.g., especially the domain-specific ones) can only be expressed in a few ways (e.g. overdraft, direct_debit, swimming_pool), while others can have much more varied surface semantic realisations, (e.g. make, not_working). Table 9 in Appendix A provides an estimation of the semantic variability of each intent (module).
3.3 Slots
$\mathrm{NLU}++$  further includes a rich set of 17 slots, defined in Table 10 in Appendix A. Table 4 displays several  $\mathrm{NLU}++$  examples where complex combinations of intents and slots occur, showcasing how  $\mathrm{NLU}++$  might provide a much more challenging environment for the evaluation of joint ID and SL models in future research.
Following the design of previous standard SL datasets (Hemphill et al., 1990; Coucke et al., 2018; Coope et al., 2020), we provide span annotations for slots. On top of this, to also support training and evaluation of SL models which are not span-based, we also provide value annotations (or canonical values as named by Rastogi et al. (2019)) for times, dates, and numeric values.
Similarly to intent modules, slots can also be divided into the generic ones (e.g. time, date) and the domain-specific ones (e.g. company_name, rooms, kids), see Table 10. Again, this distinction allows for the cross-domain reusability of annotated data.
3.4 Data Collection and Annotation
Previous NLU datasets have usually relied on crowdworkers, aiming to collect a large number of examples, and typically optimising for quantity over quality. However, even with much simpler ontologies, workers are prone to make annotation mistakes, leading to very noisy datasets (Eric et al., 2019). In addition, when workers are asked to rephrase a sentence, they often change its semantic meaning or tend to provide rephrasings with extremely low lexical variability (Kang et al., 2018).
$\mathrm{NLU}++$  reflects true production requirements and focuses on data quality. Instead of relying on crowdworkers, 4 highly skilled annotators with dialogue and NLP expertise, also familiar with production environments, collected, annotated, and corrected the data. The process started by defining the ontology for BANKING and HOTELS. Then, real user examples were fully anonymised and reannotated following the defined ontology. Finally, new examples were created in order to cover less
frequent intents and slots, aiming at creating realistic and semantically varied sentences with new combinations of intents and slots.
3.5 Comparison with Other NLU Datasets
Aiming to reflect the differences between NLU++ and the most popular ToD NLU datasets, Table 5 compares their general statistics. Since the focus of NLU++ is on curated high-quality data, NLU++ covers a fewer number of examples than the other datasets, but it is evident that NLU++ is the only real multi-intent dataset: it averages 2.01 intents per example with a high standard deviation. In addition, NLU++ is the only dataset that combines a large set of intents with a large set of slots.
In order to assess the quality and diversity of the NLU data, we include two additional metrics: 1) Type-Token Ratio (TTR) (Jurafsky and Martin, 2000) which measures lexical diversity) and semantic diversity. Both metrics are computed for the set of examples sharing an intent, weighted by the frequency of that intent $^{10}$  and finally averaged over intents. The semantic diversity per intent is computed as follows: (i) sentence encodings, obtained by the ConveRT sentence encoder (Henderson et al., 2020), $^{11}$  are computed for the set of sentences sharing the same intent; (ii) the centroid of these encodings is then computed; (iii) finally, the average cosine distance from each encoding to the centroid is computed. The overall scores clearly indicate that  $\mathrm{NLU}++$  offers a much higher lexical and semantic diversity than previous datasets, which should also render it more challenging for current SotA NLU models. $^{12}$
In hope to establish NLU++ as a more challenging production-oriented testbed for dialogue NLU, especially in low-data scenarios, we evaluate a series of current cutting-edge models for both NLU tasks: intent detection (§4.1) and slot labeling (§4.2). Our aim is to assess and analyse their performance across different setups, and provide solid baseline reference points for future evaluations on NLU++.
Data Setups. Unless noted otherwise, for both tasks we adopt the standard  $K$ -fold cross-validation
<sup>10</sup>Note that ATIS has some intents with a single example: for these intents the TTR score would be 1. Weighting by the intent frequency avoids these intents dominating the metric.
See Appendix B for a short description of ConveRT.
$^{12}$ SNIPS also shows high semantic diversity, but this is mostly due to the high frequency of named entities.
Example	Intents	Slots (Values)
How much less did I spend on Amazon during the current year?	how_much, less, transfer-payment	date_period (current year), company_name (Amazon)
Show me all the transactions from Sunday to Monday please	request_info transfer-payment	date_from (Sunday), date_to (Monday)
Hi there, what I want is setting up a 50£ direct debit with Eon for the next 2 months	greet, make, direct_debit	amount_of-money (50£), company_name (Eon), date_period (next 2 months)
Can I make a reservation for 4 adults in 2 rooms, from the 1st of June to the 7th?	make, booking	adults (4), rooms (2), date_from (1st of June), date_to (7th)
Table 4: NLU++ examples combining several intents and slots.
Dataset	Number of examples	Number of intents	Number of slots	Avg. intents per example	Avg. slots per example	Type-token ratio (TTR)	Semantic diversity
ATIS	5,871	18	47	1±0.08	3.3±1.61	0.043	0.202
SNIPS	14,484	7	39	1	2.6±1.05	0.154	0.336
OOS	23,700	151	0	1	0	0.148	0.254
BANKING77	13,083	77	0	1	0	0.125	0.209
NLU++	3,080	62	17	2.01±1.25	0.65±0.95	0.268	0.367
Table 5: Comparison of NLU++ with other popular NLU datasets; ATIS (Hemphill et al., 1990), SNIPS (Coucke et al., 2018), OOS (Larson et al., 2019b) and BANKING77 (Casanueva et al., 2020)
as done e.g. by Liu et al. (2019b). Through such folding evaluation, (i) we avoid overfitting to any particular test set and (ii) we ensure more stable results with smaller training and test data (i.e., when simulating low-data regimes typically met in production) through averaging over different folds.[13]