Venue: NAACL 2024

# SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU

Evgeniia Razumovskaia1, Goran Glavas2, Anna Korhonen1, Ivan Vulic1,3

###### Abstract

Task-oriented dialogue (ToD) systems help users execute well-defined tasks across a variety of domains (e.g., _flight booking_ or _food ordering_), with their Natural Language Understanding (NLU) components being dedicated to the analysis of user utterances, predicting users' intents (_Intent Detection_, ID) and extracting values for informational slots (_Value Extraction_, VE). In most domains, labelled NLU data is scarce, making sample-efficient learning - enabled with effective transfer paradigms - paramount. In this work, we introduce SQATIN, a new framework for dialog NLU based on (i) instruction tuning and (ii) question-answering-based formulation of ID and VE tasks. According to the evaluation on established NLU benchmarks, SQATIN sets the new state of the art in dialogue NLU, substantially surpassing the performance of current models based on standard fine-tuning objectives in both in-domain training and cross-domain transfer, and it also surpasses off-the-shelf large language models for the same task, both in terms of performance and inference efficiency. Furthermore, SQATIN yields particularly large performance gains in cross-domain transfer, owing to the fact that our QA-based instruction tuning leverages similarities between natural language descriptions of classes (i.e., slots and intents) across domains.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8195–8211

June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Task-oriented dialogue (ToD) systems support users in execution of specific, well-defined tasks through natural language interaction (e.g., ordering food or purchasing tickets) Young (2002); Budzianowski et al. (2018). Fine-grained understanding of user's utterances, commonly referred to as (dialogue) natural language understanding (NLU) is necessary for successful ToD Larson et al. (2019); Casanueva et al. (2022). NLU modules of ToD systems typically solve two complementary tasks: (1) _Intent detection_ (ID) aims to recognise the purpose (i.e., intent) of the user's utterance, classifying utterances into a set of pre-defined classes (e.g., the intent lost_luggage in _flight booking_); (2) _Value extraction_ (VE) aims to extract spans that express values for any of the pre-defined informational slots (e.g., a dialog system for _booking flights_ would have slots such as origin, destination, time, maximal_price). Realistic ToD setups for both ID and VE typically involve a relatively large number of labels (e.g., >100 different intent classes), commonly with a limited number of labelled instances per class. Successfully addressing these tasks thus amounts to enabling sample-efficient learning by means of transferring knowledge from other tasks Gao et al. (2019), languages Hung et al. (2022b); Moghe et al. (2023), or domains Hung et al. (2022a); Moghe et al. (2023).

In recent years - in line with general NLP trends - most NLU models Budzianowski and Vulic (2019); Hosseini-Asl et al. (2020); Henderson and Vulic (2021), inter alia) were obtained via standard, task-specific fine-tuning of pretrained Transformer-based language models (PLMs) Devlin et al. (2019); Radford et al. (2019). Standard fine-tuning comes with task-specific (discriminative) objectives - different from LM-ing as the pretraining objective - which in principle impedes both knowledge transfer (1) from pretraining to downstream tasks and (2) between different downstream tasks. Prompting in contrast Liu et al. (2023b) recasts downstream tasks into language modelling, making them more aligned with the models' pretraining. Finally, instruction-tuning Sanh et al. (2022); Chung et al. (2022) - supervised training in which prompts created from instances are prepended with natural language descriptions of the tasks - facilitate the transfer between arbitrary tasks, leveraging the generalisation over task descriptions for zero-shot inference (i.e., inference for tasks unseen in training). Despite the impressive zero-shot and in-context few-shot inference abilities of the more recent Large

LMs (LLMs) Brown et al. (2020); Chowdhery et al. (2023); Touvron et al. (2023), supervised fine-tuning still brings substantial performance gains for dialog NLU Hudecek and Dusek (2023) and standard NLU tasks, even with low-resource supervision Gao et al. (2021).

As generalisation to new domains (with limited in-domain annotation effort) is one of the main desiderata of ToD, some recent work on dialog NLU Fuisz et al. (2022); Casanueva et al. (2022) has recognised that ID and VE can be cast as question answering (QA) tasks: this facilitates transfer from models trained on large QA datasets Rajpurkar et al. (2016); Lee et al. (2020), allowing also to capitalise on other large datasets previously recast as QA McCann et al. (2018); Wang et al. (2022b). These efforts, however, amount to sequential transfer with standard fine-tuning for QA and thus (i) do not align their fine-tuning with the models' pretraining objective; and without an LM-based objective they (ii) cannot benefit from cross-task transfer via natural language task formulations.

* this is highly desirable in most ToD domains, where one typically deals with only a handful of labelled utterances; (2) while small-scale ID/VE instruction-tuning specialises the model for a particular ToD domain (e.g., _restaurant booking_), the negligible size of in-domain training (compared to model's massive instruction-"pretraining") should prevent overfitting to the ToD training domain and allow for effective cross-domain transfer.

Our results strongly support both of the above assumptions: SQATIN yields state-of-the-art performance on two prominent dialogue NLU benchmarks both in in-domain and cross-domain evaluations. SQATIN brings particularly large gains in transfer between close ToD domains: classes in these domains have similar prompt descriptions, unlike the existing approaches based on standard fine-tuning. The code is openly available at [https://github.com/cambridgeltl/sqatin/](https://github.com/cambridgeltl/sqatin/).

## 2 SQATIN: Methodology

**Standard Classification vs. Instruction Tuning for Dialog NLU.** ID and VE are two tasks that comprise most Dialogue NLU modules. Both tasks are commonly cast as classification tasks: ID as a sequence classification task (i.e., one or more intent labels assigned for the whole utterance) and VE as a span extraction task, i.e., token-level classification.

In standard classification with pretrained LMs, a task-specific classifier \(c_{t}:\mathbf{X}\in\mathbb{R}^{h}\mapsto\mathcal{P}(C_{t})\) converts \(h\)-dimensional sequence or token representations (output by the LM) into a multinomial probability distribution over the set of task classes \(C_{t}\). This means that a classifier \(c_{t}\), trained for task \(t\) with classes \(C_{t}\), cannot be used to make predictions for any other classification task \(t^{\prime}\) with a different set of classes \(C_{t^{\prime}}\): thus, transfer between tasks can only occur indirectly through the parameters of the LM. This is particularly unfortunate for domain transfer in dialog NLU, where different domains often have semantically overlapping ID and VE classes (e.g., intent confirm_order is essentially the same intent in _flight booking_ and in _food ordering_). In contrast, instruction-tuning recasts classification as a language modelling (i.e., generation) task \(LM:\mathbf{x}\in\mathbb{R}^{h}\mapsto\mathcal{P}(V_{t})\), with \(V_{t}\) as the subset of the LM's vocabulary where each token \(v_{t}\in V_{t}\) represents one class \(c_{t}\). This removes the need for a task-specific classifier (on top of the LM) and facilitates transfer between tasks, especially those with semantically overlapping class tokens.

**QA-Based Instruction Tuning in SQATIN.** For the above reasons, we adopt an instruction tuning approach to ID and VE. We start from models that have been instruction-tuned at scale Wang et al. (2022a); Chung et al. (2022), since these models come with a strong inductive bias to complete any new task expressed as an instruction, exhibiting impressive generalisation abilities (i.e., good performance on new tasks).

As illustrated in Figure 1, we formulate both ID and VE as text-to-text tasks, with our instruction input consisting of (i) _context_, (ii) _instance_, and (iii) _prompt_. _Context_ (e.g., _"The user says:"_) is the additional natural language description that is added (in our case, prepended) to the _instance_, a user's utterance; _Prompt_ is the text that follows 

the _instance_ and describes the actual task, that is, what is to be predicted from the instance. We formulate _prompts_ as _questions_ for both tasks. The motivation for this is the fact that the instruction-tuned model from which we start Chung et al. (2022) has been pretrained on QA formulations of various tasks and thus comes with an inductive bias for answering questions. For each training utterance, we create one instruction-based training example for each of the intent and slot classes: (1) for ID, the question incorporates a natural language description of the intent class (e.g., _did the user intend to talk about some booking? corresponds to the intent class_booking) and requires a binary answer (yes or no); (2) for VE, the question incorporates a natural language description of an informational slot (e.g., _what is the number of people mentioned?_ corresponds to the slot num_guests) - the expected answer is the value for that slot, as expressed in the instance or unanswerable if the instance does not contain a value for the slot.

A possible alternative to this "one instruction per instance and class" approach would be the more common prompt-based classification approach in which we create only one instruction per instance (e.g., with the question prompt _"what is the intent of this sentence?"_) and the model is expected to generate the token of the correct intent, choosing between tokens of all intent classes. This, however, comes with two major drawbacks: (i) ID tasks commonly come with a large number of classes (e.g., more than 50) - incorporating descriptions of all intent classes into a single prompt might thus surpass the input size of most models or they might struggle with memorizing all the options Liu et al. (2023); (ii) ID is, in principle, a multi-label, rather than multi-class problem, which means that utterances can express more than just one intent - this would require the model to output the text that somehow combines the tokens of more than one class, which is not something that instruction-based models have been pretrained for.

We experimented with two different instruction formulations: (1) without context (_None_), in which the instruction consists only of the instance and prompt; and (2) with descriptive context (_Desc._, where we prepend the utterance with _"The user says:"_ and the question prompt with _"Question:"_, as illustrated Figure 2. We selected these two particular instruction formulations (_None_ and _Desc._) based on their performance in a pilot study, which we describe in detail in the Appendix (A).

## 3 Experimental Setup

We rely on the Flan-T5 instruction-pretrained models Chung et al. (2022). Unless stated otherwise, the main model is the Base variant. Training hyperparameters are described in detail in Appendix D.

* _change_, _room_, and _booking_

Figure 1: Instruction examples for ID and VE: for each we show one example where the class matches the utterance (i.e., for ID: correct intent class; for VE: a value for the slot class present) and one where it does not.

Figure 2: An annotated utterance from NLU++ transformed into corresponding SQATIN instruction instances. For brevity, we display the transformation for only two intents (withi and housekeeping), but the same transformation was applied for all intents.

 

than one complex intent change_room_boking). CLINC-150 encompasses over 20K utterances from 10 versatile domains (e.g., _travel_, _small talk_). Each domain has 15 intent labels, resulting in 150 intents in total. CLINC also contains utterances that do not belong to any of the 150 intents (labelled as out_of_scope). The fact that all CLINC domains have 15 intents, with the same number of instances per intent, allows for direct performance comparison across domains.1 With few-shot fine-tuning in focus, we evaluate the models in a folded cross-validation setup. NLU++ already comes with predefined splits for 10-fold and 20-fold cross-validation.2 Analogously, we split data from each CLINC domain in 10 folds, resulting in 150 training examples per fold.

Footnote 1: Prior work has mostly used CLINC-150 as a single-domain dataset with 150 intents, rather than multi-domain with domain-specific intents. In contrast, we are interested in cross-domain dialogue NLU performance and thus split the examples by domains. To ensure the replicability of results, we will make public the exact dataset splits that we used.

**Baselines.** We compare SQATIN against two types of state-of-the-art models for dialogue NLU. For brevity, we provide training and model selection details for both baselines in the appendix.

_Classification from Sentence Embeddings (CL-SE)._ Recent work on ID Gerz et al. (2021); Casanueva et al. (2022) resorts to classifying - with a shallow feed-forward classifier - fixed sentence embeddings produced by of-the-shelf sentence encoders (SE). This avoids expensive fine-tuning of base LMs (e.g., RoBERTa) and yields comparable (or better) performance. We use LaBSE Feng et al. (2022) as a state-of-the-art (SotA) SE.

_Standard QA Fine-Tuning (QA-FT)._ Similar to us, these models adopt a QA-based formulation of dialogue NLU but exclude the instruction component Namazifar et al. (2021); Casanueva et al. (2022); Fuisz et al. (2022). The key aspect is that the QA-based fine-tuning for ID and VE starts from the model that has previously been fine-tuned on large-scale QA datasets (e.g., SQUAD, Rajpurkar et al. (2016, 2018)). To maximise comparability (given that SQATIN is based on Flan-T5), we obtain our QA-FT baseline by fine-tuning the T5 model Raffel et al. (2020) previously trained on SQUAD 2.03

Footnote 2: In the 20-fold setup, one fold contains \(\approx 100\) utterances in the _banking_ domain and \(\approx 50\) in the _hotels_ domain.

Footnote 3: We use the checkpoint at [https://huggingface.co/mrn8488/t5-base-finetuned-squadv2](https://huggingface.co/mrn8488/t5-base-finetuned-squadv2).

We report the standard micro-F1 scores. VE predictions are considered correct only if they exactly match the gold value span.

## 4 Main Evaluation

**Preliminary Study: Zero-Shot ID & VE.** The key hypothesis behind SQATIN is that instruction-tuned models have stronger inductive bias for dialogue NLU than models fine-tuned in the standard manner, including those trained for QA Namazifar et al. (2021); Fuisz et al. (2022). We thus preliminarily compare zero-shot ID/VE performance of (1) the instruction-trained Flan-T5 and (2) T5 finetuned for QA on SQUAD2.0 (denoted QA-T5) on NLU++. The results in Table 1 show that Flan-T5 is much more robust "out of the box". While QA-T5 has better VE performance in the _banking_ domain, it yields near-zero performance in all other setups. This validates our selection of the instruction-tuned Flan-T5 as the starting point for SQATIN.

**In-Domain Results.** We next compare the supervised in-domain performance (i.e., training and test instances from the same domain) of SQATIN against the CL-SE and QA-FT baselines. Tables 2 and 3 display the results on NLU++ and CLINC-150, respectively. On NLU++, we additionally provide QA-FT results with t

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{2}{c}{**ID**} & \multicolumn{2}{c}{**VE**} \\ \cline{2-5}  & _20-Fold_ & _10-Fold_ & _20-Fold_ & _10-Fold_ \\ \hline \multicolumn{5}{|c|}{**Ranking**} \\ \hline QA-T5 & 0.6 & 0.6 & 12.5 & 12.5 \\ Flan-T5 & 21.9 & 21.9 & 3.2 & 3.2 \\ \hline \hline \multicolumn{5}{|c|}{**HOTELS**} \\ \hline QA-T5 & 0.4 & 0.4 & 0.0 & 0.0 \\ Flan-T5 & 20.9 & 21.9 & 5.9 & 5.8 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Zero-shot results for ID and VE on NLU++.

\begin{table}
\begin{tabular}{l r r r r} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{_Templ._} & \multicolumn{2}{c}{**ID**} & \multicolumn{2}{c}{**VE**} \\ \cline{3-5}  & & _20-F_ & _10-F_ & _20-F_ & _10-F_ \\ \hline \multicolumn{5}{|c|}{**BankING**} \\ \hline CL-SE & & 58.1 & 68.8 & N/A & N/A \\ QA-FT: RoBERTa & 80.3 & 85.6 & 50.5 & 56.7 \\ QA-FT: ImDeBERta & 80.8 & 85.0 & 59.7 & 66.5 \\ QA-FT: T5 & 82.7 & 86.8 & 61.5 & 73.5 \\ \hline \multirow{2}{*}{SQATIN} & \multirow{2}{*}{\begin{tabular}{r} _None_ \\ _Desc._ \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{r} _None_ \\ _Resc.

 

RoBERTa Liu et al. (2019) and mDeBERTa He et al. (2022), copied directly from Casanueva et al. (2022) and Moghe et al. (2023), respectively.

SQATIN consistently and considerably outperforms the baseline models, on both tasks and on both datasets. These results confirm that instruction-based models have stronger inductive biases than QA-fine-tuned models: these biases are propagated in task-specific instruction-based fine-tuning, resulting in SotA performance. The gains seem more pronounced in setups with less training data (i.e., 20-Fold in Table 2) rendering instruction-tuning more sample efficient than (QA-based) fine-tuning. Overall, SQATIN seems to work slightly better with descriptive context prompts added to the instruction (compare _Desc._ vs. _None_).

**Domain Transfer Results.** We next train SQATIN in one (source) domain and apply it in another (target) domain. Table 4 and Figure 3 summarize the domain transfer results for NLU++ and CLINC-150 (all domain pairs), respectively.

Much like in in-domain training, SQATIN consistently outperforms the SoTA baseline QA-FT in domain transfer (the only exception is banking\(\rightarrow\)hotels transfer for ID in the 10-Fold setup), only now by much wider margins for VE (e.g., by over 10 points in hotels\(\rightarrow\)banking transfer in the 20-Fold setup). On CLINC-150, the results reveal not only that SQATIN consistently outperforms QA-FT (consistently lighter heatmap cells for SQATIN variants than for QA-T5) but that it is also able to better exploit label similarity between domains: e.g., for credit card as the target domain, SQATIN obtains best performance when transferring from the banking domain, whereas QA-FT, in this case, finds auto as the best source.

**Similarity of Intent Class Descriptions.** Observing that SQATIN yields best transfer performance between intuitively related domains, we now investigate more closely what type of similarity between domains drives the transfer: (i) similarity of examples (sim-E) or (ii) similarity of intent class descriptions, incorporated in SQUTIN's prompts (sim-C). We quantify sim-E as the average similarity across all pairs of utterances between the domains: with similarity of two utterances computed as cosine between their sentence embeddings, obtained with mpnet Song et al. (2020) as the sentence encoder. Analogously, sim-C is computed as the average similarity of pairs of class prompts between the two domains. We then measure the correlation Pearson's \(\rho\)) between the transfer performance and Sim-E or sim-C. Table 5 shows these correlations for each CLINC-150 domain as transfer target. Correlations are largest for domains that do have related domains in the dataset (e.g., banking and credit card) and lowest for domains that are quite different from all other (e.g., auto or utility). Importantly, sim-C shows higher average correlation with transfer performance than sim-E: this suggests that SQATIN's instruction-based tuning with class descriptions in prompts truly captures similarities sets of intents and, consequently, especially improves transfer between related domains.

## 5 Further Analyses and Discussion

**Cross-Task Generalisation.** We next hypothesise that SQATIN facilitates transfer between the two dialogue NLU tasks, given that SQATIN's QA formulation conceptually allows for such cross-task transfer and presents both tasks to the model in the same format. Table 6 compares the zero-shot ID performance of the off-the-shelf Flan-T5 (_Nontuned_) against the variant we SQATIN-fine-tune

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{1}{c}{\multirow{2}{*}{_Templ._}} & \multicolumn{2}{c}{**ID**} & \multicolumn{2}{c}{**VE**} \\ \cline{3-7}  & & _20-F_ & _10-F_ & _20-F_ & _10-F_ \\ \hline \multicolumn{7}{c}{**Banking**} & **HOTEIS** \\ \hline QA-FT: T5 & \multirow{3}{*}{_None_} & 66.70 & **69.68** & 30.86 & 38.09 \\ \cline{3-7}  & & 66.68 & 68.18 & **33.24** & **39.48** \\ \cline{3-7}  & & **G7.04** & 68.48 & **33.24** & 37.41 \\ \hline \multicolumn{7}{c}{**NOTELS**} & **RANKING** \\ \hline QA-FT: T5 & \multirow{3}{*}{_None_} & 59.76 & 66.12 & 35.08 & 44.60 \\ \cline{3-7}  & & 65.35 & 67.34 & 44.72 & **52.05** \\ \cline{3-7}  & & **ECS.44** & **68.56** & **45.69** & 51.87 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Domain transfer results for SQATIN and the QA-FT (T5) baseline on NLU++ (between banking and hotels). **Bold:** best score in each column.

\begin{table}
\begin{tabular}{l l r r r r r r r r r r} \hline \hline Model & Template & auto & bankin & credit card & home & kt/chINE & meta & small & travel & Utility & work & AVG \\ \hline CL-SE & & 92.74 & 92.30 & 90.48 & 88.58 & 91.19 & 90.19 & 90.90 & 95.29 & 94.53 & 91.93 & 91.81 \\ QA-FT: T5 & & 90.42 & 94.38 & 94.42 & 89.23 & 93.22 & 90.10 & 81.36 & 97.67 & 94.66 & 89.99 & 91.54 \\ \hline \multicolumn{7}{c}{_None_} & **94.47** & 96.04 & 95.64 & 91.92 & 95.01 & 90.55 & 93.10 & **97.77** & 95.72 & 91.56 & 94.18 \\ \multicolumn{7}{c}{_Desc._} & **94.47** & **96.11** & **95.85** & **92.66** & **95.36** & **91.52** & **93.12** & 96.97 & **96.07** & **92.01** & **94.42** \\ \hline \hline \end{tabular}
\end{table}
Table 3: In-domain ID results on CLINC-150 for SQATIN and the baselines (CL-SE and QA-FT).

 

for VE. We observe substantial improvements in ID after instruction-tuning for VE (around 5% in the banking domain and over 10% in the hotels domain), proving effective cross-task generalisation of SQATIN in dialogue NLU.

We then fine-tune the models _jointly_ on ID and VE. Table 7 compares single-task training vs. multi-task training on both tasks. While multi-task training yields no clear gains for ID (as the easier of the two tasks), it gives consistent gains for VE (0.5-1.5 F1 points). This again indicates that SQATIN facilitates transfer between the dialog NLU tasks.

**Model Size.** To analyse the impact of the underlying instruction-tuned model's size on performance, we also train SQATIN on top of the following Flan-T5 models: small (80M parameters), base (250M) and large (780M), with the scores provided in Appendix E. SQATIN yields strong in-domain performance even on top of the small Flan-T5. The margin between large and base is substantially smaller than that between base and small; for in-domain ID, the gap between large and base is negligible. The small models performs notably worse than its larger siblings only in cross-domain transfer, especially for VE. Cross-domain performance of large almost reaches the in-domain performance of small, which is in line with observations that generalisation abilities of instruction-tuned models generally improve with their size Chung et al. (2022).

**Sample Efficiency.** Due to large-scale instruction pretraining, we expect SQATIN to be more sample efficient than QA-FT and CL-SE. To test this,

\begin{table}
\begin{tabular}{l c r r r} \hline \hline \multirow{3}{*}{**Model**} & \multicolumn{2}{c}{banking} & \multicolumn{2}{c}{hotels} \\ \cline{2-4}  & _20-Fold_ & _10-Fold_ & _20-Fold_ & _10-Fold_ \\ \hline Non-tuned & 21.91 & 21.93 & 20.85 & 21.94 \\ Tuned for VE & 26.28 & 26.85 & 30.77 & 33.39 \\ \hline \hline \end{tabular}
\end{table}
Table 6: SQATIN’s (_Desc._ cross-task transfer performance on NLU++; VE\(\rightarrow\)ID.

Figure 3: Cross-domain transfer results for ID on CLINC-150 for SQATIN and the SotA QA-FT baseline. Full results in the tabular format are in Appendix B. Diagonal values correspond to in-domain results. Source domains shown along the vertical axis and target domains along the horizontal axis.

\begin{table}
\begin{tabular}{l l r r r r r r r r r} \hline \hline Template & auto & banking & credit & home & Kitchen & small & talk & travel & utility & work & avg \\ \hline \multicolumn{11}{r}{**In-Domain Training Examples**} \\ \hline None & -0.1443 & 0.5476 & 0.4268 & 0.1318 & 0.0204 & 0.0970 & 0.3279 & 0.0890 & -0.2613 & 0.5451 & 0.2591 \\ Desc. & -0.1069 & 0.5710 & 0.4695 & -0.1121 & 0.1649 & 0.0929 & 0.1304 & -0.3360 & -0.35 & 0.6086 & 0.2942 \\ \hline \multicolumn{11}{r}{**Intent Descriptions**} \\ \hline None & -0.2660 & 0.5620 & 0.5076 & 0.3059 & 0.1208 & 0.2454 & 0.6019 & 0.1633 & 0.1388 & 0.3830 & 0.3353 \\ Desc. & -0.3376 & 0.5533 & 0.5327 & 0.2319 & -0.1091 & 0.3165 & 0.4884 & 0.1076 & 0.0449 & 0.4860 & 0.3208 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Correlation (Pearson’s \(\rho\)) between domain transfer performance and domain similarity, measured in terms (i) of examples (sim-E) and (ii) class prompts (sim-C): shown for every CLINC-150 domain as the target.

 

we train the models on training data of different sizes. The process is as follows: i) first, 1000 examples are randomly chosen for the test set; ii) from the rest we sample a random subset of \(N\) training examples; iii) models are then trained on training set from step ii) and evaluated on test set from step i). This ensures that models trained on sets of different sizes are evaluated on the same test set, making the performances comparable. We use the same hyperparameter configuration from SS3 for all training sizes. Results in Figure 4 demonstrate that the scarcer the resources are, the more benefits SQATIN brings over the baselines (QA-FT and especially CL-SE). Another observation is that both QA-based approaches, QA-FT as well as SQATIN drastically outperform CL-SE in few-shot scenarios (cf. results for 32 and 64 training examples): this result justifies QA formulation for intent detection and value extraction in low-data setups.

**Base Instruction-Tuned Model.** To evaluate the impact of the underlying instruction-tuned model, we apply SQATIN on top of Flan-T5-base and Tk-Instruct-base Wang et al. (2022c). The models are of the same size (250M), but pretrained on a different set of tasks (1,800+ tasks for Flan and 1,600+ tasks for Tk-Instruct). The experimental setup and training hyperparameters were kept the same across the models (see SS3 and Appendix D for the exact values). We run experiments on the intent detection task in the _20-Fold_ setup. The results presented in Table 8 demonstrate improvements of SQATIN over QA-FT irrespective of the underlying instruction-tuned model. At the same time, the absolute scores do depend on the underlying model's quality with Flan-T5 achieving better absolute scores.

### Independent QA versus Multiple-Choice.

By design SQATIN involves asking an independent question about every intent (for ID) and every slot (VE) from the ontology for each user utterance: this decomposition might impact inference efficiency. A more efficient alternative might be a common multiple-choice prompt-based approach, where we create one instruction per utterance and provide the model with all possible intent classes or slots. The model is then expected to generate all intents or slot values that apply to the given utterance in a single response. We use the same instruction formulations to ensure comparability and represent possible intent classes with natural language descriptions (e.g., "to deny something", "to greet someone"); see an input example in Appendix F. Similarly to SQATIN, we finetune an instruction-tuned model, namely, Flan-T5 (base), on the MC-style input. Training hyperparameters are provided in Appendix D.

While offering potential benefits with inference speed, there are known deficiencies of this multilechoic formulation (MC), as previously discussed in SS2. For instance, the average length (in tokens) of input of the independent, binary SQATIN romualation for NLU++ ID and the MC formulation is 29.85 and 310.13, respectively. The difference might become even more salient with larger ontologies. The results for NLU++ in Table 9 demonstrate that the MC approach is considerably behind the independent-QA SQATIN both in in-domain and cross-domain setups, regardless of the training data size or template formulation. This indicates that the per-intent or per-slot independent question formulation is necessary for sample-efficient generalisation of SQATIN. We hypothesise that this is due to the data augmentation effects achieved this way.

#### Sqatn versus In-Context Learning with ChatGPT.

One alternative to supervised tuning of smaller models is in-context learning (ICL) with much larger instruction-tuned language models. ICL could be more computationally efficient at training time as it does not require fine-tuning the model while being more demanding at inference

Figure 4: Comparison of ID models on banking domain on NLU++ for different training data sizes. The results are averages over 3 random seeds.

 

time, as the model size is considerably larger. To compare the performance of ICL with SQATIN, we evaluate ChatGPT in two standard scenarios: (i) _zero-shot (ZS)_, when the provided instruction includes task description with all possible options (intent descriptions in our case); and (ii) _ICL_, when in addition to the above, the instruction also includes training examples which were used for supervised training in the models in every respective setting.4 We evaluate GPT-3.5-turbo-instruct as the underlying model due to its strong ICL capabilities [21].

Footnote 4: For the _10-Fold_ setup including all examples was impossible due to the context length limit. In this case, we fitted as many examples as possible by the context length.

Results in Table 9 demonstrate that SQATIN performs consistently better than ChatGPT in both ZS and ICL scenarios. This suggests that even large models with ICL (and higher inference demands and cost) cannot surpass smaller highly specialised SQATIN models for the fine-grained dialogue NLU tasks such as the NLU++ ones.

**Parameter Efficiency.** Next, we also investigate whether the performance benefits of SQATIN extend when we replace full-model fine-tuning with the standard parameter-efficient fine-tuning (PEFT) methods [86] such as _adapters_[7, 14]. In our case, relying on the standard bottleneck adapters with the reduction factor of 16 [14], for Flan-T5 Base, the number of tunable parameters is \(\approx 250\times\) smaller than the size of the original model. The hyperparameters and training procedure are the same (see SS3), except for the learning rate which was increased to \(5e\text{-}4\).5 Figure 5 displays the performance of adapter-based fine-tuning on NLU++. The results render adapters extremely effective and comparable to full model fine-tuning, indicating that the benefits of SQATIN are not tied only to full-model fine-tuning.

Footnote 5: Grid search over the set {\(5e\text{-}5\), \(5e\text{-}4\), \(5e\text{-}3\)} was run.

## 6 Related Work

**Pretraining for ToD Dialogue.** LLMs, trained on large web-scale corpora, revolutionised NLP, bringing massive performance gains to most NLP tasks. Besides general corpora, the most successful pretrained LMs for dialogue have have been additionally trained on more specialised, conversation-like data (e.g., from Reddit or Twitter). These models have been increasingly successful in both open-domain [1, 1, 19, 18, 17], instead a task-oriented dialogue [19, 18, 17, 16]. Compared to general-purpose LM pretraining (e.g., BERT), dialogic pretraining has been shown to lead to higher performance in cross-domain transfer for dialogue NLU tasks [18, 17, 16] due to the versatility of texts used in pretraining. Another stand of work investigated multi-task learning setups for dialogue NLU [18, 19, 17]. In this work, in contrast, we resorted to models _pretrained_ on multiple tasks with instruction-based objectives, resulting with stronger inductive biases for cross-domain and cross-task settings. To the best of our knowledge, this work is the first to propose a unified (QA- and instruction-based) framework for both dialogue NLU tasks (ID and VE).

**Instruction Tuning for Dialogue NLU.** Instruction tuning is an emergent framework in NLP

Figure 5: Full-model fine-tuning (\(\approx\!248\)M tunable parameters) versus PEFT with Adapters (\(\approx\!1.8\)M tunable parameters) in in-domain ID and VE.

 

where a generative model completes a task by following natural language _instructions_, possibly including few labelled instances following the instruction to make the whole prompt. These models generalise particularly well to tasks unseen during training Chung et al. (2022); Chowdhery et al. (2023) due to their ability to leverage the information about a task during inference Liu et al. (2023b). The performance, especially in zero-shot setup, is highly dependent on task definitions Liu et al. (2023b) or providing several training examples Min et al. (2022) in the instruction text (commonly known as in-context learning).6 Dialogue follows the same trend: recent work Gupta et al. (2022) demonstrated the zero-shot effectiveness of instruction-tuned models on dialogue tasks. Instruction engineering Gupta et al. (2022); Ruder et al. (2023) and increasing the number of in-context instances can further improve the models' performance Madotto et al. (2021); Mi et al. (2022). The input (context) size of the models, however, puts a limit on the number of (1) training examples (2) classes (i.e., their descriptions) one can include in the prompt. SQATIN deals with the issue in two ways: a) by recasting the dialogue NLU tasks as independent QA, at inference time we remove the need for the model to see all class descriptions at once; and b) we allow the model to learn from training examples in supervised fashion (versus incontext) thus not being limited by the base model's input length. We empirically validate that both have strong positive impact on task performance.

Footnote 6: We note for completeness that prior work proposed to utilise masked language modelling abilities of pretrained models on standard, non-dialogue NLU tasks via cloze-style phrases Schick and Schütze (2021) and demonstrated the potential of using few labelled examples for parameter updates Schick and Schütz (2021). Gao et al. (2021) show the effectiveness of prompt-based fine-tuning of masked language models when several task demonstrations are included into the input context. This is in contrast to SQATIN which **a)** focuses on dialogue NLU tasks; **b)** utilises instruction-tuned generative language models and relies on their instruction-following, rather than mask-filling capabilities.

## 7 Conclusion

We have introduced a novel framework for dialogue NLU, SQATIN, which combined (i) supervised instruction tuning and (ii) question-answering formulation of intent detection and value extraction. We evaluated SQATIN on two established dialogue NLU benchmarks, demonstrating that SQATIN brings substantial and consistent improvements over the existing SoTA approaches. The performance gains are especially pronounced in cross-domain transfer, as SQATIN can leverage similarities between classes across domains via their descriptions. SQATIN also performs well in cross-task transfer, enabling the two dialogue NLU tasks to benefit from one another. We also show that SQATIN supports parameter-efficient fine-tuning and that it largely outperforms ICL with much larger (and more expensive) language models.

### Limitations

The majority of our experiments are based on the Flan collection of models as they were pretrained on a wide collection of tasks. However, we note that there are other instruction-based models Ouyang et al. (2022); Sanh et al. (2022); Zhang et al. (2022) inter alia), with more getting published almost on a daily basis, which could be used with the proposed method and the choice of the instruction-based model is orthogonal to the proposed methodology. We leave wider exploration in this direction as future work.

Additionally, we have focused on a single-source transfer across domains, i.e., a model trained on one domain was expected to be able to transfer to a multitude of others. Future work will also explore the multi-source cross-domain transfer where the model would be finetuned on combined data from several domains and tested on data from domains not included in training.

In the evaluation, we rely on available standard dialogue NLU benchmarks built specifically to test few-shot in-domain and cross-domain generalisation abilities of the models. It is important to note that the benchmarks are only for English dialogue NLU. We opt to confirm the effectiveness of SQATIN in multilingual settings in future work. Exploration of SQATIN in multilingual settings would be also dependent on the availability of strong multilingually pretrained instruction-based models.

NLU++ includes the descriptions of intent and slot classes which means that they did not have to be created specifically for SQATIN. It is important to note that when applying it to other datasets, such descriptions would need to be created manually, and there is no guarantee that the descriptions in NLU++ are optimal for fine-tuning.

Lastly, due to the computational cost of finenting instruction-based models we largely rely on instruction wordings and training hyperparameters 

from prior work. We hope to perform a more detailed hyperparameter search in both wording of the instructions and training hyperparameters in the future, which might yield even higer absolute scores with SQATIN.

## Acknowledgments

We thank the anonymous reviewers for their helpful feedback. The work has been in part supported by a Huawei research donation to the Language Technology Lab at the University of Cambridge. It has also been supported by the UK Research and Innovation (UKRI) Frontier Research Grant EP/Y031350/1 EQUATE (the UK government's funding guarantee for ERC Advanced Grants) awarded to Anna Korhonen at the University of Cambridge. The work of Ivan Vulic has been supported in part by a Royal Society University Research Fellowship _'Inclusive and Sustainable Language Technology for a Truly Multilingual World'_ (no 221137; 2022-).

## References

* Akiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Towards a human-like open-domain chatbot. _ArXiv preprint_, abs/2001.09977.
* Bao et al. (2021) Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. 2021. PLATO-2: Towards building an open-domain chatbot via curriculum learning. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 2513-2525, Online. Association for Computational Linguistics.
* Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* how can I help you? towards the use of pre-trained language models for task-oriented dialogue systems. In _Proceedings of the 3rd Workshop on Neural Generation and Translation_, pages 15-22, Hong Kong. Association for Computational Linguistics.
* a large-scale multi-domain Wizard-of-Qr dataset for task-oriented dialogue modelling. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 5016-5026, Brussels, Belgium. Association for Computational Linguistics.
* Casanueva et al. (2022) Inigo Casanueva, Ivan Vulic, Georgios Spithourakis, and Pawel Budzianowski. 2022. NLU++: A multi-label, slot-rich, generalisable dataset for natural language understanding in task-oriented dialogue. In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 1998-2013, Seattle, United States. Association for Computational Linguistics.
* Chowdhery et al. (2022) Akanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2023. Palm: Scaling language modeling with pathways. _Journal of Machine Learning Research_, 24(240):1-113.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barter Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Debbani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _ArXiv preprint_, abs/2210.11416.
* Dettmers et al. (2023) Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized lms. _Advances in Neural Information Processing Systems 2023 (to appear)_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivarazhagan, and Wei Wang. 2022. Language-agnostic BERT sentence embedding. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 878-891, Dublin, Ireland. Association for Computational Linguistics.
* Fuisz et al. (2022) Gabor Fuisz, Ivan Vulic, Samuel Gibbons, Inigo Casanueva, and Pawel Budzianowski. 2022. Improved and efficient conversational slot labeling through question answering. _ArXiv preprint_, abs/2204.02123.
* Gao et al. (2019) Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilele Hakkani-Tur. 2019. Dialog state tracking: A neural reading comprehension approach. In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_, pages 264-273, Stockholm, Sweden. Association for Computational Linguistics 

Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3816-3830, Online. Association for Computational Linguistics.
* Gerz et al. (2021) Daniela Gerz, Pei-Hao Su, Razvan Kusztos, Avishek Mondal, Michal Lis, Eshan Singhal, Nikola Mrksic, Tsung-Hsien Wen, and Ivan Vulic. 2021. Multilingual and cross-lingual intent detection from spoken data. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7468-7475, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Gupta et al. (2022) Prakhar Gupta, Cathy Jiao, Yi-Ting Yeh, Shikib Mehri, Maxine Eskenazi, and Jeffrey Bigham. 2022. IntructDial: Improving zero and few-shot generalization in dialogue through instruction tuning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 505-525. Association for Computational Linguistics.
* Ham et al. (2020) Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 583-592, Online. Association for Computational Linguistics.
* He et al. (2022) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2022. Debertav3: Improving deberta using electra-style pretraining with gradient-disentangled embedding sharing.
* Henderson and Vulic (2021) Matthew Henderson and Ivan Vulic. 2021. ConVEx: Data-efficient and few-shot slot labeling. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3375-3389, Online. Association for Computational Linguistics.
* Hosseini-Asl et al. (2020) Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzewski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In _Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA_, volume 97 of _Proceedings of Machine Learning Research_, pages 2790-2799. PMLR.
* Hudecek and Dusek (2023) Vojtech Hudecek and Ondrej Dusek. 2023. Are large language models all you need for task-oriented dialogue? In _Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue_, pages 216-228, Prague, Czechia. Association for Computational Linguistics.
* Hung et al. (2022a) Chia-Chien Hung, Anne Lauscher, Simone Ponzetto, and Goran Glavas. 2022a. DS-TOD: Efficient domain specialization for task-oriented dialog. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 891-904, Dublin, Ireland. Association for Computational Linguistics.
* Hung et al. (2022b) Chia-Chien Hung, Anne Lauscher, Ivan Vulic, Simone Ponzetto, and Goran Glavas. 2022b. Multi2WOZ: A robust multilingual dataset and conversational pretraining for task-oriented dialog. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3687-3703, Seattle, United States. Association for Computational Linguistics.
* Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1311-1316, Hong Kong, China. Association for Computational Linguistics.
* Lee et al. (2020) Gyeongbek Lee, Seung-won Hwang, and Hyunsouk Cho. 2020. SQuAD2-CR: Semi-supervised annotation for cause and rationales for unanswerability in SQuAD 2.0. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_, pages 5425-5432, Marseille, France. European Language Resources Association.
* Lin et al. (2021) Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, and Pascale Fung. 2021. Zero-shot dialogue state tracking via cross-task transfer. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 7890-7900, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Lin et al. (2020) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, and Pascale Fung. 2020. MinTL: Minimalist transfer learning for task-oriented dialogue systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, 

pages 3391-3405, Online. Association for Computational Linguistics.
* Liu et al. (2023a) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023a. Lost in the middle: How language models use long contexts. _ArXiv preprint_, abs/2307.03172.
* Liu et al. (2023b) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. _ACM Computing Surveys_, 55(9):1-35.
* Liu et al. (2021) Qi Liu, Lei Yu, Laura Rimell, and Phil Blunsom. 2021. Pretraining the noisy channel model for task-oriented dialogue. _Transactions of the Association for Computational Linguistics_, 9:657-674.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _ArXiv preprint_, abs/1907.11692.
* Madotto et al. (2021) Andrea Madotto, Zhaojiang Lin, Genta Indra Winata, and Pascale Fung. 2021. Few-shot bot: Prompt-based learning for dialogue systems. _ArXiv preprint_, abs/2110.08118.
* McCann et al. (2018) Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. _ArXiv preprint_, abs/1806.08730.
* March 1, 2022_, pages 11076-11084. AAAI Press.
* Mi et al. (2021) Fei Mi, Wanhao Zhou, Lingjing Kong, Fengyu Cai, Minlie Huang, and Boi Faltings. 2021. Self-training improves pre-training for few-shot learning in task-oriented dialog systems. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1887-1898, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Min et al. (2022) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirizi. 2022. MetaICL: Learning to learn in context. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2791-2809, Seattle, United States. Association for Computational Linguistics.
* Moghe et al. (2023) Nikita Moghe, Evgeniiia Razumovskaaie, Liane Guillou, Ivan Vulic, Anna Korhonen, and Alexandra Birch. 2023. Multi3NLU++: A multilingual, multi-intent, multi-domain dataset for natural language understanding in task-oriented dialogue. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 3732-3755.
* Namazifar et al. (2021) Mahdi Namazifar, Alexandros Papangelis, Gokhan Tur, and Dile Hakkani-Tur. 2021. Language model is all you need: Natural language understanding as question answering. In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_, pages 7803-7807. IEEE.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwarya Kamath, Andreas Ruckle, Kyunghyun Cho, and Iryna Gurevych. 2021. AdapterFusion: Non-destructive task composition for transfer learning. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 487-503, Online. Association for Computational Linguistics.
* Poth et al. (2023) Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purakayatha, Leon Englander, Timo Imhof, Ivan Vulic, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. 2023. Adapters: A unified library for parameter-efficient and modular transfer learning. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 149-160, Singapore. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _J. Mach. Learn. Res._, 21:140:1-140:67.
* Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
* Rajpurkar et al. (2016a) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Rajpurkar et al. (2016b) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. SQuAD: 100,000+ questions 

for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 2383-2392. External Links: Link Cited by: SS4.
* S. Ruder, J. H. Clark, A. Gutkin, M. Kale, M. Ma, M. Nicosia, B. Rujwani, P. Riley, J. A. Sarr, X. Wang, et al. (2023)Xtreme-up: a user-centric scarce-data benchmark for under-represented languages. ArXiv preprintabs/2305.11938. External Links: Link, 2305.11938 Cited by: SS4.
* S. Ruder and F. Vulic (2022)Modular and parameter-efficient fine-tuning for NLP models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, Abu Dubai, UAE, pp. 23-29. External Links: Link, Document Cited by: SS4.
* V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, A. Raja, M. Dey, M. SaifulBari, C. Xu, U. Thakker, S. Sharma Sharma, E. Szczechla, T. Kim, G. Chhablani, N. V. Nyayk, D. Datta, J. Chang, M. Tian-Jian Jiang, H. Wang, M. Manica, S. Shen, Z. Yin Yong, H. Pandey, R. Basden, T. Wang, I. Neeraj, J. Rozen, A. Sharma, A. Santilli, T. Fevry, J. Alan Fries, R. Teehan, T. Le Scao, S. Biderman, L. Gao, T. Wolf, and A. M. Rush (2022)Multi-task prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022, External Links: Link Cited by: SS4.
* T. Schick and H. Schutze (2021)Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 255-269. External Links: Link, Document Cited by: SS4.
* T. Schick and H. Schutze (2021)HCT's not just size that matters: small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 2339-2352. External Links: Link, Document Cited by: SS4.
* K. Song, X. Tan, T. Qin, J. Lu, and Y. Liu (2020)Mpnet: masked and permuted pretraining for language understanding. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, External Links: Link Cited by: SS4.
* Y. Su, L. Shu, E. Mansimov, A. Gupta, D. Cai, Y. Lai, and Y. Zhang (2022)Multi-task pre-training for plug-and-play task-oriented dialogue system. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 4661-4676. External Links: Link, ISBN 978-3-319-31 

Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Ye et al. (2023) Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zakai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. _ArXiv preprint_, abs/2303.10420.
* Young (2002) Steve J Young. 2002. Talking to machines (statistically speaking). In _INTERSPEECH_.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shouhui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _ArXiv preprint_, abs/2205.01068.
* Zhao et al. (2020) Yufan Zhao, Can Xu, and Wei Wu. 2020. Learning a simple and effective model for multi-turn response generation with auxiliary tasks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3472-3483, Online. Association for Computational Linguistics.

 



## Appendix A Different Instruction Formulations

Choosing the right instruction formulation is often crucial (or at least important) to obtain strong performance from the instruction-based models. Thus, we conducted a pilot study for picking an optimal one. We experiment with 4 _context_ options, 4 options of text preceding a question and 3 _prompt_ options. The options (shown in Table 10) were adapted from the templates used to train the Flan models Chung et al. (2022). We use Fold-0 of 10-Fold in-domain setting for intent detection to determine the best instruction formulation.

The results of the preliminary study are shown in Table 11. Although the range of results is not that large, we focus on two instruction formulations in further experiments: none-none-none and usersaid-question-none. The former is picked for similarity with the simple question answering formulation, although it leads to a lower performance. This enables direct comparison to QA-based models. As this formulation contains only the input sentence and the questions (no description of the task or its context), we denote it as _None_. The former instruction formulation (usersaid-question-none) is used as it contains the description of the context of the task and it led to the highest performance in the pilot study. As it contains a short description of the task, we denote it as _Descriptive (Desc.)_.

## Appendix B Full Cross-Domain Results on CLINC-150 for Different Base Models

The cross-domain results on CLINC-150 for QA-FT and different versions of SQATIN are provided in Tables 12, 13 and 14.

## Appendix C Comparison of Single-Task and Multi-Task Models for Cross-Domain Setups

Comparison of cross-domain results of models trained with SQATIN in single-task and multi-task settings is shown in Table 15.

## Appendix D Fine-tuning and Hyperparameters

The classifier of the CL-SE baseline is a feed-forward network with a single hidden layer of dimensionality \(512\) and \(tanh\) as the non-linear acti

\begin{table}
\begin{tabular}{l l r r r r} Context & Pre-question & Prompt & Banking & Hotels & AVG \\ \hline none & none & none & 77.2 & 67.3 & 72.25 \\ sent & none & none & 81.31 & 76.45 & 78.88 \\ none & none & answer & 80.96 & 77.14 & 79.05 \\ given & none & none & 81.4 & 76.96 & 79.18 \\ none & none & answer-options & 81.22 & 77.26 & 79.24 \\ none & based-above & answer & 82.65 & 75.9 & 79.28 \\ useraid & none & none & 81.72 & 77.35 & 79.54 \\ given & none & answer & 81.49 & 77.69 & 79.59 \\ sent & none &answer & 81.36 & 78.06 & 79.71 \\ none & based & answer & 82.17 & 77.33 & 79.72 \\ none & based & answer-options & 82.17 & 77.37 & 79.74 \\ sent & based & none & 82.13 & 77.38 & 79.76 \\ sent & based-above & none & 82.68 & 77 & 79.84 \\ sent & based-above & answer & 82.73 & 77.06 & 79.90 \\ sent & based-above & answer & 82.15 & 77.74 & 79.95 \\ none & based-above & answer-options & 82.67 & 77.24 & 79.96 \\ sent & none & answer-options & 81.4 & 78.63 & 80.02 \\ none & based & none & 82.08 & 78.1 & 80.09 \\ usersaid & none & 82.34 & 77.92 & 80.13 \\ usersaid & none & answer-options & 82.05 & 78.28 & 80.17 \\ given & none & answers-options & 81.7 & 78.63 & 80.17 \\ given & question & answer & 83.49 & 76.94 & 80.22 \\ sent & based-above & answer-options & 82.78 & 77.65 & 80.23 \\ none & based-above & none & 82.57 & 77.93 & 80.25 \\ none & question & answer & 83.17 & 77.35 & 80.26 \\ sent & question & none & 83.25 & 77.27 & 80.26 \\ usersaid & based & answer & 82.39 & 78.15 & 80.27 \\ senti & question & answer & 83.39 & 77.29 & 80.34 \\ usersaid & based & none & 82.99 & 77.72 & 80.36 \\ useraid & based & answer & 83.05 & 77.68 & 80.37 \\ none & question & answer-options & 83.22 & 77.61 & 80.42 \\ given & question & answer-options & 83.76 & 77.39 & 80.50 \\ userali & none & answer & 81.83 & 79.17 & 80.5 \\ sent & based & answer-options & 82.29 & 78.78 & 80.56 \\ given & question & none & 83.42 & 77.66 & 80.54 \\ usersaid & based & answer-options & 82.42 & 78.67 & 80.55 \\ sent & question & answer-options & 83.4 & 77.7 & 80.55 \\ sensr & question & answer-options & 83.08 & 78.44 & 80.76 \\ none & question & none & 83.08 & 78.5 & 80.79 \\ usersaid & question & answer & 83.88 & 77.74 & 80.81 \\ usersaid & question & answer-options & 84.2 & 77.43 & 80.82 \\ usersaid & question & none & 83.85 & 78.07 & 80.96 \\ \end{tabular}
\end{table}
Table 11: Performance of SQATIN with different instruction wordings. The options are ordered in ascending average order.





\begin{table}
\begin{tabular}{l r r r r r r r r r r} \hline \hline  & & \multicolumn{6}{c}{SQATIN: _None_} \\  & auto & banking & credit & home & k and d & meta & small & travel & utility & work \\ \hline auto & 94.47 & 75.69 & 70.47 & 41.68 & 56.88 & 50.47 & 59.61 & 82.45 & 68.54 & 67.51 \\ banking & 72.43 & 96.11 & 75.91 & 46.77 & 59.13 & 51.44 & 55.68 & 81.96 & 65.14 & 69.08 \\ credit & 73.62 & 80.39 & 95.85 & 49.55 & 61.13 & 54.34 & 60.59 & 80.81 & 66.01 & 70.23 \\ bome & 65.04 & 76.7 & 66.99 & 92.66 & 62.81 & 49.83 & 54.21 & 88.98 & 66.03 & 72.07 \\ k and d & 66.79 & 73.88 & 66.92 & 47.91 & 95.36 & 57.31 & 65.57 & 87.18 & 72.71 & 69.37 \\ meta & 66.73 & 73.66 & 67.55 & 47.56 & 59.12 & 91.52 & 68.59 & 86.31 & 67.01 & 63.85 \\ small & 67.08 & 69.89 & 61.95 & 41.26 & 55.93 & 51.33 & 93.12 & 84.28 & 62.62 & 62.97 \\ travel & 64.5 & 73.05 & 63.56 & 46 & 54.73 & 48.81 & 59.14 & 96.97 & 68.92 & 66.66 \\ utility & 65.39 & 73.03 & 64.25 & 45.66 & 55.26 & 59.82 & 68.29 & 87.59 & 96.07 & 67.09 \\ work & 67.8 & 79.15 & 71.26 & 50.41 & 58.86 & 47.48 & 53.41 & 82.07 & 67.15 & 92.01 \\ \hline \end{tabular}
\end{table}
Table 14: _Cross-domain_ intent detection using SQATIN on CLINC-150 (Larson et al., 2019) with _Descriptive_ templates. K and D stands for Kitchen and Mining domain. The rows are source domains while columns show target domains.

\begin{table}
\begin{tabular}{l l c c c c c c c c c} \hline \hline  & & & \multicolumn{6}{c}{QA-FT pretrained on SQUAD 2.0} \\  & auto & banking & credit & home & k and d & meta & small & traxel & utility & work \\ \hline Auto & 90.42 & 71.08 & 65.22 & 42.03 & 61.23 & 61.78 & 65.64 & 77.04 & 66.7 & 60.5 \\ banking & 34.67 & 94.38 & 62.16 & 43.35 & 62.51 & 49.43 & 50.35 & 74.33 & 58.96 & 61.45 \\ credit & 35.19 & 66.94 & 94.42 & 41.28 & 64.05 & 55.86 & 61.13 & 76.54 & 64.14 & 66.92 \\ tomo & 26.68 & 60.4 & 46.07 & 89.23 & 55.95 & 48.64 & 43.35 & 76.05 & 56.65 & 68.08 \\ k and d & 35.96 & 66.85 & 67.75 & 46.98 & 93.22 & 54.52 & 68.6 & 80.95 & 71.08 & 65.5 \\ meta & 32.51 & 58.92 & 45.94 & 41.11 & 51.25 & 90.1 & 61.68 & 74.11 & 67.33 & 58.19 \\ small & 27.2 & 49.17 & 39.61 & 30.69 & 49.17 & 52.4 & 81.36 & 64.59 & 58.16 & 51.62 \\ taxel & 32.96 & 58.54 & 38.89 & 39.71 & 50.6 & 46.53 & 39.46 & 97.67 & 61.13 & 59.72 \\ utility & 32.61 & 63.12 & 42.76 & 35.91 & 46.87 & 52.67 & 65.77 & 73.62 & 94.65 & 60.08 \\ work & 36.32 & 62.9 & 55.93 & 41.05 & 58.24 & 53.14 & 58.62 & 81.83 & 69.13 & 89.99 \\ \hline \hline \end{tabular}
\end{table}
Table 12: _Cross-domain_ intent detection using QA-based model on CLINC-150 (Larson et al., 2019). K and D stands for Kitchen and Mining domain. The rows are source domains while columns show target domains.



vation function. With multi-label formulations of classification tasks (because instances in NLU++ can have multiple labels and those in CLINC-150 none), we apply _sigmoid_ as an output activation and train with the binary cross-entropy loss. At inference, we consider an intent class to be predicted if its probability, output of the sigmoid activation, is above the threshold \(\theta=0.3\).

The models are implemented using Transformers library Wolf et al. (2020). The models are loaded with sequence-to-sequence language modeling head. Baseline QA-based models and SQATIN are fine-tuned with the same protocol and hyperparameters as in prior work Casanueva et al. (2022); Fuisz et al. (2022); Moghe et al. (2023). They are trained for 10 epochs with the batch size of 8, with Adam optimizer Kingma and Ba (2015) and the learning rate of 5e-5. Unless stated differently, we report the average cross-validation performance across all 10 or 20 folds the results are averages of 10 and 20 runs for 10- and 20-Fold setups, respectively.7

Footnote 7: We focus on the pre-defined few-shot 10-Fold and 20-Fold setups, as the baselines already demonstrate saturated performance on Large training data setups Casanueva et al. (2022).

## Appendix E Results for Different Model Sizes

The results for different model sizes for the two domains of NLU++ are plotted in Figure 6 and Figure 7.

## Appendix F Instructions with the Multiple Choice Formulation

Figure 8 shows an example of the multiple choice formulation for the ID task, including the instruction text, user query example and all possible options for the answers.