SlowGuess/NAACL_2022_OCR huggingface dataset:
The experiments are run with  $K = 20$  (20-Fold) and  $K = 10$  (10-Fold), where we train on 1 fold and evaluate on the remaining  $K - 1$  folds. These setups simulate different degrees of data scarcity: e.g., the average training fold comprises  $\approx 100$  examples for BANKING and  $\approx 50$  for HOTELS for 20-Fold experiments, and twice as much for 10-Fold experiments. Besides these low-data training setups, we also run experiments in a Large-data setup, where we train the models on merged 9 folds, and evaluate on the single held-out fold.[14] The key questions we aim to answer with these data setups are: Which NLU models are better adapted to low-data scenarios? How much does NLU performance improve with the increase of annotated NLU data? How challenging is NLU++ in low-data versus large-data scenarios?
Domain Setups. Further, experiments are run in the following domain setups: (i) single-domain experiments where we only use the BANKING or
the HOTELS portion of the entire dataset; (ii) both-domain experiments (termed ALL) where we use the entire dataset and combine the two domain ontologies (see Table 2); (iii) cross-domain experiments where we train on the examples associated with one domain and test on the examples from the other domain, keeping only shared intents and slots for evaluation. The key questions we aim to answer are: Are there major performance differences between the two domains and can they be merged into a single (and more complex) domain? Is it possible to use examples labeled with generic intents from one domain to boost another domain, effectively increasing reusability of data annotations and reducing data scarcity?
$F_{1}$  (micro) is the main evaluation measure in all ID and SL experiments.
4.1 Intent Detection: Experimental Setup
We evaluate two groups of SotA intent detection models: (i) MLP-Based, and (ii) QA-Based ones.
MLP-Based ID Baselines. Casanueva et al. (2020) and Gerz et al. (2021) have recently shown that, for the ID task, full and expensive fine-tuning of large pretrained models such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019a) is not needed to reach strong ID performance. As an alternative, they propose a much more efficient MLP-based approach to intent detection which works on par or
$^{13}$ Due to folding, variations in results with different random seeds were negligible, even in lowest-data setups.  $^{14}$  Effectively, Large-data experiments can be seen as 10-Fold experiments with swapped training and test data.
even outperforms full fine-tuning on the ID task. $^{15}$  In a nutshell, the idea is to use fixed/frozen “off-the-shelf” universal sentence encoders such as ConveRT (Henderson et al., 2020) or Sentence-BERT (Reimers and Gurevych, 2019) models to encode input sentences. A standard multi-layer perceptron (MLP) classifier is then learnt on top of the sentence encodings.
Two core differences to the previous work stem from the fact that we now deal with the multi-label ID task: 1) to this end, we replace the output softmax layer with the sigmoid layer; and 2) we define a threshold  $\theta$  which determines the final classification: only intents with probability scores  $\geq \theta$  are taken as positives. This way, the hyper-parameter  $\theta$  effectively controls the trade-off between precision and recall of the multi-label classifier.
We comparatively evaluate several widely used state-of-the-art (SotA) sentence encoders, but remind the reader that this decoupling of the MLP classification layers from the fixed encoder allows for a much wider empirical comparison of sentence encoders in future work. The evaluated sentence encoders are: 1) CONVERT (Henderson et al., 2020), which produces 1,024-dimensional sentence encodings; 2) LABSE (Feng et al., 2020) (768-dim); 3) ROBL-1B (1,024-dim) and 4) LM12-1B (384-dim) (Reimers and Gurevych, 2019; Thakur et al., 2021). For completeness, we provide brief descriptions of each encoder in our evaluation, along with their public URLs, in Appendix B, and refer the reader to the original work for more details about each sentence encoder.
QA-Based ID Baselines. Another group of SotA ID baselines reformulates the ID task into the (extractive) question-answering (QA) problem (Namazifar et al., 2021; Fuisz et al., 2022). This QA-oriented reformatting then allows for additional specialised QA-tuning of large PLMs. In a nutshell, the idea is to (i) fine-tune the original PLM such as BERT/RoBERTa on readily available large general-purpose QA data such as SQuAD (Rajpurkar et al., 2016), and then (ii) further fine-tune this general QA model with in-domain ID data. This strategy has recently shown very strong performance on single-label ATIS data (Namazifar et al., 2021).
The main 'trick' is to reformat the input ID examples into the following format: "yes. no. [SEN-
TENCE]” and pose a question such as: “is the intent to ask about [INTENT]?” (see Appendix A for the actual questions associated with each intent, also shared with the dataset). Here, [SENTENCE] is the placeholder for the actual input sentence, and [INTENT] is the placeholder for a short manually defined text (akin to language modeling prompts (Liu et al., 2021), see again Appendix A) which briefly describes the intent. The QA formulation lends itself naturally to the multi-label ID setup as each ‘intent-related’ question is posed separately. In other words, for each input example and for each of the  $L$  intents in the ontology the QA model must extract yes or no as the answer, where correct intent labels are the ones for which the answer is yes.[16] We note that our work is the first to apply and evaluate the QA approach on multi-label ID.
We experiment with two pretrained language models, both fine-tuned on the SQuAD2.0 dataset (Rajpurkar et al., 2018) before additional QA-tuning on NLU++ examples converted to the aforementioned QA format: ROBB-QA uses RoBERTa-Base as the underlying LM, while ALB-QA relies on the more compact ALBERT (Lan et al., 2020).
ID: Training and Evaluation. All MLP-based baselines rely on the same training protocol and hyper-parameters in all data and domain setups. The MLP classifier consists of 1 hidden layer of size 512, and is trained via binary cross-entropy loss for 500 epochs with the batch size of 32 and the dropout rate is 0.6. We use the standard AdamW optimizer (Loshchilov and Hutter, 2018) with the learning rate of 0.003 and linear decay; weight decay is 0.02. The threshold  $\theta$  is set to 0.4. $^{17}$
For QA models, we largely follow Namazifar et al. (2021) and fine-tune all models for 5 epochs, using AdamW; the learning rate of  $2e - 5$  with linear decay; weight decay is 0; batch size is 32.
$^{15}$ Our preliminary results on the NLU++ dataset corroborated these findings from prior work; due to a large number of experiments, we thus opt for this more efficient yet also very effective approach to ID.
16 For instance, for the input sentence "I need to increase my overdraft" from the BANKING domain, we would pose all 48 questions associated with each of the  $L = 48$  intents in BANKING, where the QA model should extract yes as the answer for intents change, overdraft and more_higher_after, and extract no for the remaining 45 intents in BANKING.
17These hyper-parameters were selected based on preliminary experiments with a single (most efficient) sentence encoder LM12-1B and training only on Fold 0 of the 10-Fold BANKING setup; they were then propagated without change to all other MLP-based experiments with other encoders and in other setups. We repeated the similar hyper-parameter search procedure for QA-based models, using ALB-QA..
Setup→	BANKING	HOTELS	ALL
20-Fold	10-Fold	Large	20-Fold	10-Fold	Large	20-Fold	10-Fold	Large
Sentence Encoder↓	MLP-Based Baselines
CONVERT	58.6	70.2	90.3	52.3	63.1	82.8	58.6	70.2	88.9
LABSE*	54.8	66.6	88.7	48.9	58.9	82.3	55.4	66.1	87.0
ROBL-1B*	56.8	68.4	87.4	55.2	64.2	81.8	57.3	67.7	86.2
LM12-1B*	59.1	69.0	87.8	53.5	62.8	79.5	58.4	68.2	86.0
QA-Pretrained Model↓	QA-Based Baselines
ROBB-QA*	80.3	85.6	93.1	67.4	73.3	86.7	79.5	84	91.8
ALBB-QA*	76.6	82.1	92.0	60.7	67.2	85.1	75.5	80.8	90.6
Table 6:  $F_{1}$  scores  $(\times 100\%)$  of benchmarked state-of-the-art intent detection models on NLU++ in three data setups (see §4.1). We also refer to §4 for the brief descriptions of each sentence encoder (for MLP-based baselines) and the two QA-pretrained models. *All models were retrieved from the HuggingFace model repository (Wolf et al., 2020), with exact model URLs available in Appendix §B and Appendix §C. The overall best-performing model per column is in bold, while the best-performing MLP-based model per column is underlined.
CONVEX	20-Fold	10-Fold	Large
BANKING	30.1	40.0	68.1
HOTELS	29.7	40.0	64.5
ALL	34.0	45.2	71.4
QA-Based: ROBB-QA	20-Fold	10-Fold	Large
BANKING	50.5	56.7	70.2
HOTELS	48.1	52.4	70.4
ALL	55.5	53.6	72.1
Table 7:  $F_{1}$  scores  $(\times 100\%)$  on the NLU++ SL task for CONVEX (Henderson and Vulic, 2021) and a QA-Based approach (Namazifar et al., 2021) across different domains and data setups.
BANKING →HOTELS	HOTELS →BANKING
MLP-Based
CONVERT	75.4	65.2
LM12-1B	67.3	49.2
QA-Based
ALB-QA	76.7	72.7
ROBB-QA	79.3	74.2
Table 8:  $F_{1}$  scores of cross-domain intent detection experiments, evaluating performance on the set of 26 intents shared by the two domains. Large-data setup.
4.2 Slot Labeling: Experimental Setup
For slot labeling, we benchmark two current SotA models: (i) ConvEx (Henderson and Vulić, 2021), as a SotA span-extraction SL model and (ii) the QA-based SL model (Namazifar et al., 2021) based on ROBB-QA, which operates similarly to QA-based ID baselines discussed in §4.1, and relies on the same fine-tuning regime as our QA-based ID baselines. Again, we refer the reader to the original work for further details, and provide brief descriptions in Appendix D.
Main results with all the evaluated baselines are summarised in Table 6 (for ID) and Table 7 (SL).
ID: MLP versus QA Models. First, the comparisons among only MLP-based models reveal that 1) all sentence encoders offer ID performance in similar, reasonably narrow score intervals (e.g., the variations in  $F_{1}$  scores between all sentence encoders are typically below 4-6  $F_{1}$  points in all setups), and 2) that CONVERT is the best-performing sentence encoder on average, which corroborates findings from prior work on other ID datasets (Casanueva et al., 2020; Wu and Xiong, 2020).
One very apparent and important indication in the reported results is the superiority of QA-based ID models over their MLP-based competitors. QA-based models largely outperform MLP-Based baselines in all domain setups, as well as in all data setups. The gains are visible even in Large-data setups, but the benefits of QA-based ID are immense in the lowest-data 20-Fold setups: e.g., 12  $F_{1}$  points over the strongest MLP ID model on HOTELS and 20  $F_{1}$  points on BANKING.
Moreover, the use of larger underlying LMs might push the scores with QA even further: using SQuAD-tuned Roberta-Large (ROBL-QA) instead of Base (ROBB) yields further gains – e.g.,  $F_{1}$  rises from 85.6 to 87.8 on 10-Fold BANKING, and similar trends are observed in other low-data setups.
Slot Labeling. In the SL task, the QA-based model also demonstrates its superiority, again with huge gains in low-data 20-Fold and 10-Fold setups, confirming that such QA-based or prompt-based methods (Liu et al., 2021; Gao et al., 2021) are especially well suited for low-data setups. The use of manually defined questions/prompts, which are typically easy to write by humans, combined with the expressive power of QA-based task formatting yields immense gains on low-resource dialogue NLU.
Given these very promising ID and SL results on NLU++, our work also calls for further and more intensive future research on QA-based models for dialogue NLU. However, we note that QA-based ID and SL methods do come with efficiency detriments, especially with larger intent and slot sets: the model must copy the input utterance and run a separate answer extraction for each intent/slot from the set, which is by several orders of magnitudes more costly at both training and inference than MLP-based models. A promising future research avenue is thus to investigate combined approaches that could combine and trade off the performance benefits of QA-based models and the efficiency advantages of, e.g., MLP-based ID.
Low-Data vs. Large-Data. We also note that scores on both tasks, as reported in Tables 6-7, leave ample room for improvement in NLU methodology in future work, especially on SL (even in Large-data setups), and in low-data setups.
Cross-Domain Experiments. We also verify potential reusability of annotated data across domains with a simple ID experiment, where we train ID models on BANKING and evaluate on HOTELS, and vice versa. The results are summarised in Table 8. Besides (again) indicating that QA-based models outscore MLP-based ID, the results also suggest that for some generic intents it is possible to meet high ID performance without any in-domain annotations. For instance, we observe particularly high scores for highly generic and reusable intent modules such as change, how, how_much, thank, when, and affirm, all with per-intent  $F_{1}$  scores of  $\geq 90$ . We hope that these preliminary results might inspire similar ontology (re)designs in future work.
We have presented NLU++, a novel dataset for task-oriented dialogue (ToD) NLU that overcomes the shortcomings of previous NLU evaluation sets. NLU++ presents a multi-intent and slot-rich ontology, defines generic and domain-specific intents and slots to promote data reusability, and it focuses on the creation of high-quality complex examples and annotations collected by dialogue experts. Experimental results show that NLU++ raises the bar with respect to current NLU benchmarks, helping better discriminate and compare the performance of current state-of-the-art NLU models, particularly in low-data setups. We hope that NLU++ will be valuable in guiding future modeling efforts for ToD
NLU, both in academia and in industry.
Limitations and Future Work. This work has shown that a better design of the intent set can improve data reusability. However, the current ontology does not cover generic sets of intents exhaustively, and we acknowledge a (sometimes) fine line between truly generic intents versus intents 'anecdotally' shared by two domains (e.g., refund). Further, the boundaries of some generic intents can sometimes be unclear and difficult to annotate, even for expert annotators.[18] Future work should try to ground the set of generic intents.
Further, we believe that span-based annotation might be sub-optimal for canonical values such as times and dates, where small differences in the span would lead to evaluation errors but would not suppose a problem for the value to be parsed. In addition, separating time and date intervals in different slots increases the difficulty of the annotations and models need to learn a more conflicting set of slots. Further, NLU++ currently provides fine-grained slots such as date_from, date_to and date to enable more complex scenarios, but such a design might slow down annotation process and make it cumbersome. Future work includes rethinking the SL task for these slots.
Finally, while single-turn NLU is more data-efficient and easier to model, some user utterances only make sense in the presence of context from the previous system utterance. While some previous datasets (Coope et al., 2020) deal with this issue with the help of extra annotations indicating if a slot has been requested, in this work we opt for using non-contextualised slots such as number and time and let the policy handle the contextualisation. However, future work should start looking into NLU datasets composed by system + user turns.
We are grateful to our colleagues in PolyAI for many fruitful discussions and suggestions. We also thank the anonymous reviewers for their helpful feedback and comments on the presentation.
PolyAI Limited is ISO27k-certified and fully GDPR-compliant.
<sup>18</sup>For example, boundaries for intents like greet, why and change are clear, while others such as make or not_working are more prone to ambiguity and different interpretation.
Before data collection: all the data has been collected by workers of PolyAI Limited and all the annotators are also employees of PolyAI Limited.
During data collection: we did not include any personal information (e.g. personal names or addresses) and all the examples that included any had been fully anonymised or removed from the dataset. All the names in the dataset are created by randomly concatenating names and surnames from the list of the top 10K names from the US registry. Upon collection, the dataset has undergone an additional check by the internal Ethics committee of the company. NLU++ is licensed under CC-BY-4.0.
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 Task 6: A pilot on semantic textual similarity. In Proceedings of *SEM, pages 385-393.
Rami Al-Rfou, Marc Pickett, Javier Snaider, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2016. Conversational contextual cues: The case of personalization and history for response ranking. CoRR, abs/1606.00372.
Wei Bi and James Tin-Yau Kwok. 2013. Efficient multi-label classification with many labels. In Proceedings of ICML 2013, pages 405-413.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018. MultiWOZ - A large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling. In Proceedings of EMNLP 2018, pages 5016-5026.
Inigo Casanueva, Paweł Budzianowski, Pei-Hao Su, Nikola Mrksić, Tsung-Hsien Wen, Stefan Ultes, Lina Rojas-Barahona, Steve Young, and Milica Gašić. 2017. A benchmarking environment for reinforcement learning based task oriented dialogue management. CoRR, abs/1711.11023.
Inigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45.
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. BERT for joint intent classification and slot filling. CoRR, abs/1902.10909.
Sam Coope, Tyler Farghly, Daniela Gerz, Ivan Vulic, and Matthew Henderson. 2020. Span-ConveRT: Few-shot span extraction for dialog with pretrained conversational representations. In Proceedings of ACL 2020, pages 107-121.
Alice Coucke, Alaa Saade, Adrien Ball, Theodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, et al. 2018. Snips Voice Platform: An embedded spoken language understanding system for private-by-design voice interfaces. CoRR, abs/1805.10190:12-16.
Yinpei Dai, Hangyu Li, Yongbin Li, Jian Sun, Fei Huang, Luo Si, and Xiaodan Zhu. 2021. Preview, attend and review: Schema-aware curriculum learning for multi-domain dialogue state tracking. In Proceedings of ACL-IJCNLP 2021, pages 879-885.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171-4186.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021. *ProtAugment: Intent detection meta-learning through unsupervised diverse paraphrasing*. In *Proceedings of ACL-IJCNLP* 2021, pages 2454-2466.
Layla El Asri, Hannes Schulz, Shikhar Sharma, Jeremie Zumer, Justin Harris, Emery Fine, Rahul Mehrotra, and Kaheer Suleman. 2017. Frames: A corpus for adding memory to goal-oriented dialogue systems. In Proceedings of SIGDIAL 2017, pages 207-219.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, and Dilek Hakkani-Tür. 2019. MultiWOZ 2.1: Multi-domain dialogue state corrections and state tracking baselines. CoRR, abs/1907.01669.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. CoRR, abs/2007.01852.
Gabor Fuisz, Ivan Vulić, Samuel Gibbons, Inigo Casanueva, and Paweł Budzianowski. 2022. Improved and efficient conversational slot labeling through question answering. CoRR, abs/2204.02123.
Rashmi Gangadharaiiah and Balakrishnan Narayanaswamy. 2019. Joint multiple intent detection and slot labeling for goal-oriented dialog. In Proceedings of NAACL-HLT 2019, pages 564-569.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of ACL-IJCNLP 2021, pages 3816-3830.
Milica Gašić, Matthew Henderson, Blaise Thomson, Pirros Tsiakoulis, and Steve J. Young. 2012. Policy optimisation of POMDP-based dialogue systems
without state space compression. In Proceedings of SLT 2021, pages 31-36.
Daniela Gerz, Pei-Hao Su, Razvan Kusztos, Avishek Mondal, Michal Lis, Eshan Singhal, Nikola Mrkšić, Tsung-Hsien Wen, and Ivan Vulić. 2021. Multilingual and cross-lingual intent detection from spoken data. In Proceedings of EMNLP 2021.
Arshit Gupta, John Hewitt, and Katrin Kirchhoff. 2019. Simple, fast, accurate intent classification and slot labeling for goal-oriented dialogue systems. In Proceedings of SIGDIAL 2019, pages 46-55.
Michael Heck, Carel van Niekerk, Nurul Lubis, Christian Geishauser, Hsien-Chin Lin, Marco Moresi, and Milica Gašić. 2020. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of SIGDIAL 2020, pages 35-44.
Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS Spoken Language Systems Pilot Corpus. In Proceedings of the Workshop on Speech and Natural Language, HLT '90, pages 96-101.
Matthew Henderson, Pawel Budzianowski, Inigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrksic, Georgios Spithourakis, Pei-Hao Su, Ivan Vulic, and Tsung-Hsien Wen. 2019a. A repository of conversational datasets. In Proceedings of the 1st Workshop on Natural Language Processing for Conversational AI, pages 1-10.
Matthew Henderson, Inigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulic. 2020. ConveRT: Efficient and accurate conversational representations from transformers. In Findings of EMNLP 2020, pages 2161-2174.
Matthew Henderson, Blaise Thomson, and Jason D. Wiliams. 2014. The Second Dialog State Tracking Challenge. In Proceedings of SIGDIAL 2014, pages 263-272.
Matthew Henderson and Ivan Vulic. 2021. ConVEx: Data-efficient and few-shot slot labeling. In Proceedings of NAACL-HLT 2021, pages 3375-3389.
Matthew Henderson, Ivan Vulic, Daniela Gerz, Inigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019b. Training neural response selection for task-oriented dialogue systems. In Proceedings of ACL 2019, pages 5392-5404.
Yutai Hou, Yongkui Lai, Yushan Wu, Wanxiang Che, and Ting Liu. 2021. Few-shot learning for multi-label intent detection. In Proceedings of AAAI 2021, pages 13036-13044.
Eduard Hovy, Ulf Hermjakob, Chin-Yew Lin, et al. 2001. The use of external knowledge in factoid QA. In Proceedings of TREC 2001, pages 644-652.