From Discrimination to Generation: Low-Resource Intent Detection with Language Model Instruction Tuning (2024)

Abstract: Intent detection aims to identify user goals from utterances, and is a ubiquitous step towards the satisfaction of user desired needs in many interaction systems. As dynamic and varied intents arise, models that are capable of identifying new intents promptly are required. However, existing studies usually fine-tune discriminative models on the specific defined intent classes, precluding them from being directly adopted to new intent domains. In this paper, we introduce a generative pre-trained intent model that can recognize new intents from different domains in low-resource scenarios. We reformulate intent detection into a generation task and design descriptive and regularized instructions to guide the model effectively to detect new intents in open domains with no parameter updates. To validate the proposed method, we introduce a new intent detection benchmark, including the Meta-Intent Dataset and three types of representative evaluation settings. We conduct extensive experiments which demonstrate that our method outperforms a range of strong baselines that needs further fine-tuning or domain-specific samples.

heading: 1 Introduction
text: Intent detection aims to identify user intentions or main topic, playing the first and foremost role in task-oriented dialogue systems (Gupta et al., 2019), recommend systems (Qian et al., 2023) and other ubiquitous systems. Many existing studies have conducted considerable exploration (Goo et al., 2018; Wang et al., 2021), where pre-trained language models are fine-tuned under full supervision. However, due to the dynamic change of intents over
∗ Corresponding author.
time, gathering a large number of manually annotated examples and pre-defining a comprehensive intent set are quite challenging. Consequently, lowresource intent detection is proposed to recover new emerging intents with few or even zero labeled samples for relieving the need of substantial domain-specific labeled data.
Recent studies have been proposed to utilize meta-learning based methods, which transfer knowledge from seen intents to target new intents (Dopierre et al., 2021b; Chen et al., 2022a; Zhang et al., 2023a). Also, Zhang et al. (2023b) propose to directly fine-tune the model on the new intents with context augmentation and self-distillation. As illustrated in Figure 1a, these methods aim to fine-tune a customized model for each dataset. However, they require domain-specific labeled data for each intent detection task, which restricts the model generalization and leads to poor performance when facing large gap domains. Another type of method is designed to learn a unified intent-aware encoder, as shown in Figure 1b. Zhang et al. (2021) propose IntentBERT, an intent utterance encoder which firstly is trained on public source corpora and then is adapted to target domains via non-parametric
10167
matching or tuning additional classifiers. Sung et al. (2023) propose PIE, which is designed to align embeddings of utterances and pseudo-labels. However, they still need additional training for new tasks according to given intent classes and can not generate meaningful intent classes.
In this paper, we reformulate the intent detection as a language generation paradigm and propose a generative pre-trained intent model (Gen-PINT). Instead of customized networks in closed-world scenarios, Gen-PINT, as a unified model, can recognize new intents by the guidance of easily crafted prompts in open domains, as shown in Figure 1c. Specifically, Gen-PINT is tuned on a large set of general corpora to learn how to detect intents with instructions, and then is evaluated on new and diverse unseen target tasks. For the instruction format, we present four descriptive components to direct it to understand commonalities in intent detection. Task definition is the driver to enable GenPINT to comprehend the whole task paradigm, and the output space consists of all intent candidate options as restriction of the answer space. And demonstrations built on several sampled training samples guide the model to recover semantics and increase the diversity of training set. Through instruction tuning on general corpora, Gen-PINT acquires the ability to uncover intents in unlabeled utterances with provided context. Then even for distant target domains, it can generate correct and novel intents in the form of natural language. To evaluate the performance of our method, we introduce a new intent detection benchmark including the Meta-Intent Dataset and three types of evalu-
ation settings. Meta-Intent Dataset, a dataset of intent datasets, consists of 12 intent datasets and covers diverse domains. We conduct comprehensive experiments under three types of settings including any-way any-shot intent detection, episodic intent detection and new intent discovery.
The main contributions of this paper are: (1) We reformulate intent detection into the language generation paradigm and propose a unified framework for intent detection, Gen-PINT, which generates correct intents by following specialized natural language instructions. (2) We present an intent detection benchmark, including the Meta-Intent Dataset which contains 12 datasets from different domains and three types of evaluation settings in low-resource intent detection. (3) We conduct extensive evaluations on various experimental settings. Experimental results show that our method consistently outperforms strong baselines.

heading: 2 Related Work

heading: 2.1 Intent Detection
text: Intent detection aims to identify user intentions and is an important part in ubiquitous systems, such as accomplishing a helpful conversation in chatbots (Ouyang et al., 2022; Song et al., 2022), building commercial recommend systems (Qian et al., 2023; Wang et al., 2023a) and collecting information from search engines (Chen et al., 2023). Existing studies (Gong et al., 2021, 2022) have achieved promising performance by training a supervised model on large amounts of labeled examples. However, collecting sufficient labeled samples and re-training
models for new intents are challenging. Recent studies (Krone et al., 2020; Dopierre et al., 2021a; Chen et al., 2022a; Liu et al., 2023) have adopted few-shot learning methods to recognize novel intents. Mi et al. (2022) introduce a comprehensive instruction that exploits pre-trained language models with extra task-specific instructions. Parikh et al. (2023) leverage label description and intent filtering in zero-shot setting, and tune instructiontuned language models with parameter-efficient fine-tuning in the few-shot setting. Zhang et al. (2023b) propose to directly fine-tune pre-trained language models with a few labeled examples. However, these methods train a new model for each dataset, which is not only costly in deployment but also inevitably hard to scale to new intents from different domains.

heading: 2.2 Instruction Tuning
text: Instruction tuning (Wang et al., 2022; Iyer et al., 2022; Longpre et al., 2023), is becoming a novel paradigm that unlock the generalized knowledge of large language models by leveraging natural language instructions. During inference, it only conditions a language model with domain-specific instructions and several optional demonstrations. Recently, some existing studies have explored question answering (Min et al., 2022; Chen et al., 2022b), information extraction (Wang et al., 2023b; Sainz et al., 2023), and so on (Chung et al., 2022; Muennighoff et al., 2023). However, specialized intent language model with instruction tuning is still under exploration. Some studies (Song et al., 2023; Raedt et al., 2023) conduct intent discovery evaluation on ChatGPT (OpenAI, 2023), while they suffer two limitations, the risk of test data contamination (Li and Flanigan, 2023) and data privacy restrictions.

heading: 2.3 Pre-trained Intent Language Models
text: Recent studies (Wu et al., 2020; Henderson et al., 2020) have explored continual pre-training on discriminative models with dialogue corpora to improve generalization on new intent detection tasks. Zhang et al. (2021) propose IntentBERT, a BERT model tuned on public intent datasets. Sung et al. (2023) introduce a pre-trained intent-aware encoder by using contrastive learning to align embeddings of utterances with their pseudo labels. They focus on tuning discriminative language models while neglecting powerful generative language models with instructions. One of the shining points of gen-
erative models is that generated text may provide supervising view for the novel target task. Here, we design detailed instructions and propose a unified generative intent model to generalize beyond the tuning data and recognize new intents with specialized instructions.

heading: 3 Approach
text: We introduce Gen-PINT: a Generative Pre-trained INTent detection model via instruction tuning. As illustrated in Figure 2, Gen-PINT is a generative model with instructions. Different from previous methods, Gen-PINT is trained on a collection of general intent detection corpora and is directly evaluated on unseen target datasets without parameter updates. At test phase, leveraging several demonstrated samples and task instructions, it learns to recognize semantic information in utterances and then generates the corresponding intent label from given options as the predicted result.

heading: 3.1 Input-Output Schema
text: Different from typical discriminative classification methods, we reformulate the intent detection into text-to-text format and solve it through in-context learning on generative language models. Figure 2 shows four main properties of the input: task definition, output space, demonstrations and query. Output is the intent label. Task Definition describes the intent detection task. This section contains task explanation, and target format. Task explanation shows the main topic of the task and target format indicates the definition of generated output. By aligning to the pre-trained tasks, the detailed guidelines lead the language model to understand how to perform intent detection tasks and how to output the expected content, which more properly utilizes the knowledge stored in language models. To ensure the fine-tuned model follows the provided guidelines rather than overfitting and memorizing specific inputs, we paraphrase the task instructions. By doing this, the tuned model is more robust to different task definition inputs and remains better generalization ability. Table 9 lists the used task definition. Output Space is the label options for each task. The output space adds constraints for output content and guides the model to conduct multiple choice questions. Output space is flexible and varies for each task. It is easy to refine output space by just replacing different label options, rather than
re-train a classification model. In training and inference phase, we shuffle the label option order to obtain more stable performance. Demonstrations are concatenated query-output pairs. Following previous literature (Brown et al., 2020), we randomly sample k training samples and concatenate them together to build up inputs. The demonstrations are optional and we also conduct experiments without demonstrations. We observe empirically that training with context brings more improvement due to the increased diversity of the training corpora. Query is the test utterance. For each test sample, it is appended by task definition, output space and demonstrations as an single input to the model.

heading: 3.2 Training and Inference
text: The model is trained on a collection of general intent detection corpora. Formally, given a training sample (xi, yi) ∈ Dtrain, where xi is the query and yi is the corresponding label, we first randomly sample k examples, and then combine them to construct demonstrations:
Di = π(x1, y1)⊕π(x2, y2)⊕ ...⊕π(xk, yk), (1) where π is the template of transforming the querylabel pair and ⊕ denotes the concatenation operation. Then we fine-tune a pre-trained language model and force the model to learn from task definition, demonstrations and generate predictions from output space. The in-context fine-tuning objective L is calculated as:
L= ∑
(xi,yi)∈Dtrain −log p(yi|Ii, ci, Di, xi,θ), (2)
where Ii and ci is the corresponding task definition and output space, and θ is parameters of the model.
At test phase, for each test sample x∗ ∈ Dtest from a unseen target task, we directly concatenate its task definition I∗, output space c∗, demonstrations and query to form the input and then obtain generated predictions. Note that there are not any updating of parameters in test phase. The finetuned model Gen-PINT can be used to any intent detection tasks by simply specializing the output space and demonstrations.

heading: 3.3 Intent Detection Benchmark
text: We introduce a benchmark of intent detection in low-resource scenarios, consisting of the MetaIntent Dataset, 12 representative and diverse intent datasets, and three types of evaluation settings.
When collecting Meta-Intent Dataset, diversity is carefully considered. We include corpora from various domains, such as the common intent domains, e.g., travel, flight, banking, weather, and alarm etc., and also include some specific domains that have a large gap compared to common domains, like privacy policies, covid-19 and software etc. These diverse corpora from different domains can be used to evaluate model generalization. Table 8 in Appendix A reports detailed dataset statistics. All task instances follow the same text-to-text schema described in Section 3.1. For each dataset split, we follow the original split if it has other we divide the datasets into training/validation/test set with the ratio of 8:1:1.
The benchmark also offers comprehensive and standardized evaluation rules for the performance of models in both zero-shot and few-shot situations for intent detection tasks. Specifically, we conduct evaluations from three perspectives, consisting of any-way any-shot intent detection, episodic intent detection and new intent discovery. In any-way any-shot intent detection, strict zero/few-shot experiments are conducted where the number of unseen classes is larger and there is no validation set. For episodic evaluation, we employ meta-learning strategy to simulate few-shot scenarios, i.e., the popular N -way K-shot setting. In new intent discovery, we test the model ability of discovering new intents. The three types of evaluation make it easier to compare different approaches more comprehensively and will help create stronger and more efficient models.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: We introduce Meta-Intent Dataset to evaluate our proposed method in zero/few-shot settings. We select the most common intent datasets, Clinc (Larson et al., 2019), HWU64 (Liu et al., 2019a), Facebook (Schuster et al., 2019) and MTOP (Li et al., 2021) to make up the training set. The remaining eight datasets, Bank77 (Casanueva et al., 2020), MCID (Arora et al., 2020a), HINT3 (Arora et al., 2020b), SNIPS (Coucke et al., 2018), ATIS (Hemphill et al., 1990), StackOverflow (Xu et al., 2015), PolicyIE (Ahmad et al., 2021) and StackExchange (Braun et al., 2017), are used as test set in zero/few-shot settings. Detailed split and statistics of datasets can be found in Appendix A.

heading: 4.2 Baselines
text: We compare Gen-PINT with a series of baselines under three scenarios, i.e., strict any-way any-shot intent detection, episodic intent detection and new intent discovery. In any-way any-shot intent detection, baselines contains discriminative models and generative language models. For both baselines and our method, we test them under zero-shot and fewshot setting respectively. (1) IntentBERT: Zhang et al. (2021) propose to continually train BERT on public intent datasets and then detect novel intents by adding specific classifiers. (2) PIE: It is a pretrained intent-aware encoder through key phrases identification and contrastive learning on public intent datasets (Sung et al., 2023). (3) BERT-A: Comi et al. (2023) first train the model for natural language inference and then perform zero-shot intent recognition. (4) 0-shot: We run zero-shot inference on raw pre-trained generative language models (Mi et al., 2022). (5) ICL 1-shot: We follow the in-context learning by conditioning on several training samples (Brown et al., 2020). (6) Prompt Tuning (PT): We train the language model on the same training tasks without demonstrations in context.
In episodic intent detection, we compare with three strong baselines: (1) PN (Snell et al., 2017) averages the corresponding support samples to calculate the prototype for each class, and uses the negative Euclidean distance between queries and prototypes to perform classification. (2) CNet (Chen et al., 2022a) aims to generate better sample repre-
sentations by designing a task-level and an instancelevel unsupervised contrastive losses. (3) DE (Liu et al., 2023) proposes to estimate the novel class distribution by leveraging support samples and the nearest unlabeled query samples.
In new intent discovery, we compare with recent promising baselines: (1) MTP (Zhang et al., 2022) leverages the multi-task pre-training strategy to learn semantic utterance representations. MTPCLNN is a variant that integrates contrastive loss. (2) IDAS (Raedt et al., 2023) formulates the intent discovery task into abstractive summarization to retain core elements of user utterances.

heading: 4.3 Evaluation Metrics
text: In any-way any-shot intent detection, we use accuracy and macro-F1 as evaluation metric to evaluate balanced and imbalanced classification tasks. For both zero-shot and few-shot settings, we report results on all test samples in test set as described in Table 8. In few-shot setting, i.e., k-shot, we random sample k samples from training set for each class to construct training samples. We repeat this procedure 5 times using five different seeds and report average results. And we follow strict few-shot setting which means that there is no validation set to be used. In episodic evaluation, we follow Liu et al. (2023) and report accuracy in 5-way 1/5-shot settings. For new intent discovery, following Zhang et al. (2022), we report the Normalized Mutual Information (NMI) and Cluster Accuracy (ACC) according to the Hungarian algorithm (Kuhn, 2010).

heading: 4.4 Parameter Settings
text: We use FlanT5-Large (Chung et al., 2022) as our backbone model. In training phase, we append 8 samples for each sample as context. We optimize the model parameters using AdamW (Loshchilov and Hutter, 2019) with learning rate of 1 × 10−5. For fair comparison, we leverage the same training set, as described in Table 8, to re-train baselines. Experiments are conducted on RTX 3090 with 24G GPU memory.

heading: 5 Experimental Results

heading: 5.1 Any-Way Any-Shot Intent Detection
text: Tables 1 and 2 report accuracy and macro-F1 scores on eight unseen intent datasets both in zero-shot and few-shot settings. We can make the following observations. For all methods, increasing the number of labeled samples in target domains brings further improvements, especially for in-context learning of raw LLMs, where labeled samples give direct signals to guide models to generate labels. However, the improvements of discriminative mod-
els are relative small because they rely on the generalized features of utterances obtained from intent encoder while the limited labeled utterance features may cause biased class decision boundaries. Generative methods that model the semantic relationship between utterances and intent labels achieve better improvement.
Gen-PINT consistently outperforms a series of strong baselines on most datasets and achieves the best results on averaged scores. Specifically, in zero-shot setting, Gen-PINT achieves competitive performance comparing with discriminative and generative models. In few-shot setting, Gen-PINT performs better by conditioning on few target labeled samples, where brings about 11.84% and 9.44% on improvements accuracy and F1 respectively than raw LLMs. Comparing with discriminative mothods, gains are particularly significant in StackOverflow, PolicyIE and MCID datasets, where the test domains are very different from the training corpus. This demonstrates that Gen-PINT is able to infer the semantics of new intents even when the test corpus is from novel domains.

heading: 5.2 Episodic Intent Detection
text: Recently, many papers (Liu et al., 2019b; Chen et al., 2022a; Liu et al., 2023) detect new intents in the few-shot setting via meta-learning paradigm, i.e., episodic strategy. Specifically, these methods split each dataset into training/validation/test set respectively and there is no class overlap among these splits. Then they meta-train models on the training set and select the best one with validation set to evaluate test set, achieving promising performance due to the same distribution of training and test set. We compare Gen-PINT with these models on the 5-way 1/5-shot settings and more detailed experimental settings can be found at Appendix B. Note that baselines train a specific model for each dataset using the training data from the same domain with test data, while our method directly evaluate on the test samples without seeing any training data. Table 3 presents average accuracy from five different dataset splits on Bank77, MCID, HINT3 and StackOverflow. From Table 3, we can observe that Gen-PINT without assessing to any samples in these target datasets, still outperforms the strong baselines trained on each dataset, which further demonstrates the superiority and generalization of our proposed method.

heading: 5.3 New Intent Discovery
text: New intent discovery aims to identify novel unknown intents from a set of unlabeled utterances without any predefined knowledge, which is challenging and practical. Current studies (Zhang et al., 2022; Kumar et al., 2022) usually leverage twostep methods. First, they learn semantic representations of utterances and then cluster them to uncover novel intents. However, these methods just align a serial number to each cluster and they can not
generate a meaningful intent name for each cluster, which lacks interpretability and practicality.
On the contrary, benefiting from the nature of generative models, our proposed method can directly generate novel descriptive intents for utterances. IDAS (Raedt et al., 2023) is the latest work that formulates the intent discovery task into abstractive summarization to retain core elements. It prompts text-davinci-003 model (Ouyang et al., 2022) to generate intent labels. To evaluate the ability to discover unknown intents, we conduct experiments on MCID and StackOverflow datasets in unsupervised setting. Specifically, we replace the GPT-3 in IDAS with xl-sized Gen-PINT. We conduct comprehensive comparison with both clusterbased and generation-based methods. From Table 4, we can observe that our model achieves competitive results. Note that in StackOverflow dataset, our model performs better than MTP-CLNN and even achieves similar performance with IDAS that leverages GPT-3 (175B). Experimental details can be found in Appendix D.

heading: 5.4 Ablation Studies
text: Instruction regularization We conduct ablation studies to examine the contribution of several com-
ponents in instructions. We use w/o Shuffle when we remove label random shuffle regularization, and w/o Paraphrase when we remove task definition paraphrase. w/o Demons. means that we remove appended demonstrations, which is the prompt tuning baseline. As presented in Table 6, label shuffle and task instruction paraphrase boosts better performance in zero-shot and few-shot evaluation. Also, demonstrations brings more improvements, where all three strategies help to construct diverse training samples and demonstrations may give more clear signals to guide the model.
Number of training samples In the training phase, we conduct experiments with varying number of appended samples, i.e., k ∈ {0, 4, 8, 16, 32}. In the test phase, we conduct both zero-shot (0-shot) and few-shot (1-shot) evaluation. Figure 3 shows accuracy and F1 scores on all datasets. We can observe that increasing the number of appended samples brings promising improvements for most datasets especially in few-shot setting. Also, we can see that increasing k may bring light negative influence to zero-shot inference for some datasets due to the gap between training and test input format. Considering both situations, we find that the performance reaches to a saturate value when k equals 8.
Comparison to stronger LLMs In Table 5, we compare our model with FlanT5-XL and FlanT5XXL raw baselines which consist of 3B and 11B parameters respectively. We can observe that scaling up language model size yields superior performance, while it inevitably brings significant increased computational costs and resources. Comparing with these baselines, our large-sized GenPINT achieves competitive or better performance and is more efficient on computing and hardware.
Model scaling Recent studies has reported that more parameters yields better generalization (Brown et al., 2020). We scale Gen-PINT with Base (250M), Large (780M) and XL (3B) versions. Figure 4 reports the average results on all datasets and more details can be found at Figure 5 in Appendix C. We can see that for average performance, more parameters brings superior improvements while this is accompanied by increasing computational costs and deployed resources. Also, we find that scaling model size brings more improvements for zero-shot evaluation than the few-shot setting. In general, large-sized Gen-PINT has exhibited strong generalization ability and simultaneously has relative low computational load in zero-shot
and few-shot intent detection.

heading: 6 Conclusion
text: In this paper, we reformulate intent detection into a generation task and introduce Gen-PINT, a new generative pre-trained model for low-resource intent detection. By conditioning on several training samples and instructions, Gen-PINT, as a unified and end-to-end model, is able to recover semantic of inputs and generate intents for utterances from different domains. We further build a new benchmark for intent detection with a diverse collections of datasets and conduct comprehensive evaluation under three settings. Gen-PINT outperforms a series of strong baselines including discriminative methods, in-context learning and prompt tuning. Also, Gen-PINT performs better than recent dominated meta-learning methods and it can generate new descriptive intents that are beyond the reach of discriminative models.
Limitations
We propose a well-generalized generative pretrained intent model (Gen-PINT) for low-resource intent detection, and introduce a benchmark to conduct comprehensive evaluations. For a new intent task, Gen-PINT can recognize new emerging intents from different domains with easily crafted instructions rather than any further fine-tuning. Instructions used in this paper consist of task definition, output space, demonstrations and query, which have achieved promising performance. However, we do not discuss the target domain knowledge due to space limits, like the description and definition of intent labels. Integrating domain knowledge can provide more information to excite model performance and we leave this for future exploration.

heading: Acknowledgements
text: This research is supported by the National Science and Technology Major Project (No. 2021ZD0111202).

heading: A Dataset Details
text: We elaborate the details of Meta-Intent Dataset, a dataset consisting of 12 intent datasets.
• Clinc (Larson et al., 2019) is composed of 22,500 utterances covering 150 intents over 10 general domains. The distribution of utterances is uniform across each intent.
• HWU64 (Liu et al., 2019a) covers 25,716 original examples. We clean the examples and remain 25,606 examples.
• MTOP (Li et al., 2021) is a multilingual taskoriented semantic parsing dataset consisting of 100k annotated utterances in 6 languages across 11 domains. Here we use 22,288 English utterances.
• Facebook (Schuster et al., 2019) contains utterances in English from the weather, alarm, and reminder domains. For the training set, we remove duplicated samples.
• Bank77 (Casanueva et al., 2020) is a singledomain intent dataset consisting of 13,083 annotated customer queries over 77 fine-grained intents, where there may have partially overlap among some intent categories.
• MCID(Arora et al., 2020a) consists of 1,745 queries which covers 16 covid-19 specific intents.
• HINT3 (Arora et al., 2020b) contains 2,011 samples across 51 intents from three domains.
• SNIPS (Coucke et al., 2018) is composed of 14,484 utterances collected from Snips personal voice assistant.
• ATIS (Hemphill et al., 1990) (Airline Travel Information Systems) is a dataset consisting of 5,871 samples about making flight reservations over 21 intents.
• StackOverflow (Xu et al., 2015) contains 20,000 samples collected from the stackoverflow platform about computer technology.
• PolicyIE (Ahmad et al., 2021) is a challenging dataset about privacy policies of websites and mobile applications. Here we use 1,989 examples over four privacy policies.
• StackExchange (Braun et al., 2017) is composed of 251 samples, where 89 from ask ubuntu and 162 from Web Applications platforms.
Table 8 reports dataset statistics, where # denotes the number of samples in the specific subset. From Table 8, we can see that our benchmark covers diverse domains, including travel, bank, transport, software, privacy policies and so on, which provides a more comprehensive comparison of various methods to evaluate their effectiveness and robustness. For the dataset with official split, we directly follow it. And for other datasets, we split them into training set, validation set and test set respectively based on the ratio of 8:1:1.

heading: B Episodic Evaluation Details
text: Meta-learning methods have attracted increasing attentions predominantly to solve few-shot learning problems and they usually adapt the episodic strategy. Each episode usually is 5-way 1-shot or 5-way 5-shot, which means that there are 5 new intents and each intent contains 1 or 5 labeled samples. The baseline models firstly meta-train on the training set and then meta-test on the test set, where training classes and test classes do not have overlaps. We compare Gen-PINT with three strong baselines on Bank77, MCID, HINT3 and StackOverflow in the episodic setting.
Following Liu et al. (2023), we use accuracy to evaluate the performance. All reported results are averaged from five different dataset splits. Table 7 shows detailed dataset splits. At test phase, we present the average accuracy on 600 episodes samples from test set, where each class has five query samples to predict in each episode. Note that our method does not assess any training samples, which is different from other baselines. From Table 3, we can observe that our model achieves better performance in most cases without any domain-specific training samples.

heading: C Scale Model Size
text: We scale Gen-PINT to Base (250M), Large (780M) and XL (3B) sized models. Figure 5 shows the accuracy and macro-F1 performance on 8 datasets under the zero-shot (0-shot) and few-shot (1-shot) settings. We can make the following observations. Firstly, from the model size perspective, the improvements from base to large version is more significant than the improvements obtained from large to xl version. Taking both deploy efficiency and performance improvements into consideration, large-sized Gen-PINT has achieved a good balanced point. Secondly, the enhancement brought by increasing model size is more notable on zeroshot scenario than few-shot setting. In other words, conditioning a few samples can further reduce the performance gap caused by model size.
During the experiments, we find that format mismatch is an important cause of generation errors in the base-sized model, such as "top up by card" for "topping up by card" intent. Selecting the most similar intent with the generated answer significantly enhance performance. By using SimCSE (Gao et al., 2021), the average F1 of our basesized model is 61.51/66.48 (Gen-PINT/Gen-PINT 1-shot). We can observe that Gen-PINT continues to outperform strong baselines. Our focus lies more on its generalization and universal application ability, with post-processing methods left for future exploration.

heading: D New Intent Discovery
text: New intent discovery is a useful and challenging method in practical open domain scenario, where it is hard for experts to pre-define all intent classes especially in the fast-emerging new domains. Most existing studies (Zhang et al., 2022; Kumar et al., 2022) mainly focus on cluster methods to uncover new intents, while they cannot produce descriptive intent labels. In contrast, generative models are able to produce novel intent names. IDAS (Raedt et al., 2023) is proposed to formulate the intent discovery task into abstractive summarization to retain core elements. To evaluate the ability to discover unknown intents, we replace the GPT-3 in IDAS with xl-sized Gen-PINT.
During the process of prototypical label generation, we change the direct format in IDAS and construct set-level similar query context to generate the common intent behind them. As shown in Table 10, the top instructions are used in the direct
intent generation, called instance-level instructions and the down instructions are used in the similar context called set-level instructions. Specifically, we choose top-k sentences that are closest to the prototypical center as candidates and then prompt our model to discover the same latent intent behind them. This is inspired by the fact that humans struggle to recognize the core intent from a single case, yet it is natural for people to summarize the common intent from multiple similar examples.
Following Zhang et al. (2022), we report the Normalized Mutual Information (NMI) and Cluster Accuracy (ACC) according to the Hungarian algorithm (Kuhn, 2010). From Table 4, we can observe that our model achieves competitive results. Note that in StackOverflow dataset, our model performs better than MTP-CLNN and even achieves similar performance with IDAS that leverages GPT3 (175B). Table 11 shows several examples in the MCID dataset. We can observe that Gen-PINT generates abstractive intents. Also, we believe that there is still ample room to design different methods to better leverage our proposed model, which provides a promising starting point for further exploration.

heading: Future Work
text: Our focus lies more on its generalization and universal application ability, with post-processing methods left for future exploration