Towards Open Environment Intent Prediction
Year: 2023

Abstract:
Out-of-Domain (OOD) Intent Classification and New Intent Discovery are as two basic and critical tasks in the Task-Oriented Dialogue System, which are typically treated as two independent tasks. Classification focuses on identifying intents beyond the predefined set of the dialog system, but it will not further differentiate detected OOD intents in fine granularity. Discovery focuses on how to cluster unlabeled samples according to their semantic representation, which relies heavily on prior knowledge and can not provide label information for the formed clusters. To be closer to the real userfacing scenarios, we strengthen a combined generative task paradigm to extend Classification with Discovery referred to as Open Environment Intent Prediction, which is to make a further fine-grained discovery of OOD based on OOD Intent Classification. Using various widely-used generative models as an archetype, we propose a general scheme for Open Environment Intent Prediction. In a nutshell, we first perform intent detection to identify the Indomain (IND) samples and then generate labels for those identified as OOD. With these generated labels, we can discover new general intents and provide label information for them. We develop a suite of benchmarks on the existing intent datasets and present a simple yet effective implementation. Extensive experiments demonstrate that our method establishes substantial improvement compared to the baselines. Codes is publicly available.1

heading: 1 Introduction
text: OOD Intent Classification, also known as OOD Intent Detection (OID), and New Intent Discovery (NID), as two basic tasks of the Task-Oriented Dialogue System, have been two areas of active research. The purpose of OOD Intent Classification (Zhang et al., 2021b; Zhan et al., 2021; Zhou
∗Corresponding author. 1https://github.com/zyh190507/Open-Enviroment-Intent-
Prediction
et al., 2022a) is to identify utterances with not supported intents to prevent them from being wrongly post-processed. However, in the setting of OID, all OOD samples, which contain a lot of valuable corpus with different meaningful intents, are just grouped into one rejected class and are not distinguished in a fine-grained way. At the same time, how to effectively identify intents under the generative paradigm has been underdeveloped.
New Intent Discovery (Zhang et al., 2021c; Zhou et al., 2022b) focuses on how to cluster unlabeled data according to their learned semantic representation. However, existing research on New Intent Discovery needs strong prior knowledge (Zhang et al., 2022) to learn the semantic representation that can adapt to subsequent clustering, which also depends on unacceptable assumptions in real scenarios, such as knowing the number of categories of OOD intents in advance. In addition, its processing procedure is usually cumbersome with multiple dependent processing stages, which often suffers the dilemma that the knowledge learned in the previous is often forgotten in the follow-up as demonstrated in Zhou et al. (2022b) and the generated clusters usually lack semantic label information. Further, since unlabeled data usually contains a large number of samples with known intents, a closer look at
2226
the process of NID will reveal that it pays a lot of costs, but in many cases, it is just gathering a large number of samples with definite intents into clusters but cannot provide labels and not fully commit to discovering new intents.
To be closer to the realistic scenarios, we first strengthen a combined generative task paradigm based on the characteristics of the above two tasks– Open Environment Intent Prediction, which is to make a further fine-grained discovery of OOD based on OOD Intent Classification and not only gives the specific categories of IND samples but also further gives the label information of OOD. This paradigm can reduce the “burden” of existing NID tasks by avoiding clustering a large number of known intent samples and focusing on discovering new intents while giving specific label information. Compared with OID and NID, our proposed task paradigm is more general and practical, whose whole process is shown in Figure 1. Then we offer a general implementation based on the generative models. Specifically, with a generative model in hand, we carry out OID according to the learned semantic representation and give the corresponding predefined label for IND. At the same time, labels are generated for the samples identified as OOD, which also can help to discover more general intents in fine granularity.
For more general and practical, we expect to not rely on any assumptions or prior about OOD and directly provide high-quality OOD labels, which also becomes more challenging. Especially for the label generation, since only IND samples are available in the training set, fine-tuning the model directly (Model-tuning) will cause the generated labels to overfit training labels, making it a poor choice for the Open Environment Intent Prediction. Therefore, we adopt prefix-tuning (Li and Liang, 2021) to retain the general knowledge learned during pretraining to avoid shifting toward the training labels to generate more diverse labels. On this basis, to discover more general intents, we reformulate the intent discovery as a minimum cost Multi-Cut problem, which can automatically divide samples belonging to a general intent into a cluster according to the similarity of labels. Further, to mitigate the impact of Inherent Label Uncertainty (Wang et al., 2022) on the Open Environment Intent Prediction, with the help of large pre-trained models such as GPT-3 (Brown et al., 2020) or ChatGPT2, we intro-
2https://openai.com/blog/chatgpt/
duce a simple yet effective method of enriching the expression of intents and generate multiple related labels for each intent in the training set.
The contributions can be summarized as follows: Firstly, this paper strengthens a combined generative task paradigm, which can not only give the specific category of IND but also give the label information of OOD and can further discover more general intents. Secondly, this paper offers an effective implementation for such a paradigm without relying on any prior about OOD and provides a novel solution for enriching the expression of intents and a general method for intent discovery. Thirdly, to evaluate the effectiveness and generality of our method, we establish a suite of benchmarks across widely-used generative models and datasets. The experimental results demonstrate our method not only performs better classification but also makes an effective discovery.

heading: 2 Related Work
text: OOD Intent Detection (OID) OID is a field of concern recently, and many excellent related pieces of research have emerged. According to whether there are additional OOD samples involved in the training process, these works can be broadly categorized into two main groups, namely supervised and unsupervised. The supervised approaches (Zheng et al., 2020; Zhan et al., 2021; Lang et al., 2022) focus on how to help distinguish IND and OOD by using additional collected or synthesized OOD samples. The unsupervised methods usually constrain decision boundaries through specific training paradigms (Zeng et al., 2021; Zhou et al., 2022a) or post-processing methods (Zhang et al., 2021b). The existing work usually groups all OOD samples into one rejected class without further fine-grained distinction. At the same time, there is less research on generative models for OOD Intent classification. This work explores how to carry out OOD classification on the generative models and expand the OOD Intent Classification. New Intent Discovery (NID) This name may be a bit misleading (it is called Generalized Category Discovery (Vaze et al., 2022) in the field of computer vision). In the field of natural language processing, unlabeled corpus in the setting of NID include samples with known intents in addition to OOD samples. Zhang et al. (2021c, 2022) learn the clustering friendly-representation by generalizing prior knowledge to the representation of unlabeled
samples so that samples with similar representations can be divided into the same cluster. Gao et al. (2021b) discover new intents by a variant of PageRank and Intent rank algorithm and Zhou et al. (2022b) introduce a principled probabilistic framework for this task. Zhang et al. (2021a) provide a tool platform to integrate various existing methods about OID and NID. Vedula et al. (2020); Zheng et al. (2022) can be approximated as two specific implementations of the paradigm proposed in this work. However, they either need to rely on the prior knowledge of OOD or need to make complex category estimations. Further, they need to rely on all samples during discovery and cannot directly provide label information, which is not general. Different from the previous work, we use a model to implement the Open Environment Intent Prediction, and our method does not rely on any prior knowledge or assumptions about OOD while providing effective label information. Parameter-Efficient Tuning (PET) PET aims to optimize as few parameters as possible while achieving the effect as optimizing all parameters (He et al., 2022). To this end, Lester et al. (2021) inject tunable prompts into the input layer. Li and Liang (2021); Liu et al. (2022) go a step further and put tunable prompts on each internal layer of the model to achieve better results.

heading: 3 Proposed Method
text: A natural solution to solve the Open Environment Intent Prediction is to carry out full model tuning, i.e., fine-tune all the parameters of the generative models, by taking generating labels for IND sam-
ples as the downstream task. However, model tuning could lead to a certain degree of “degradation” of the vocabulary generated by the fine-tuned generative model, which means that the generated labels overfit the labels in the training set.
Specifically, as shown in Figure 2(a), almost all the words generated by the fine-tuned model fall in the vocabulary composed of the labels in the training set (solid red line in Figure 2(a)), and few words beyond the vocabulary (dotted red line in Figure 2(a)) can be generated, which will fail to generate correct labels for OOD samples. Prompt-based prefix tuning To retain the general knowledge (avoid shifting towards training labels) obtained by pre-training in large-scale corpus while adapting the model to the downstream task that generates diverse labels, we achieve it by prompting the model with tunable instructions to retain the main parameters of the model unchanged. Specifically, we adopt the prefix-tuning (Li and Liang, 2021; Liu et al., 2022) training paradigm to prepend continuous tunable tokens pl ∈ Rn×d (termed as prefix) to the l-th internal layer of the model, denoting P = [p1, p2, ..., pl] as the whole prefixes in all layers. In addition, to steer generative models to generate labels according to the content of samples, we formulate the input X to model with natural language prompts (such as “It was [Mask]”, which a crafted template of prompt) into T (X) = {x.It was [Mask].|x ∈ X} to prompt the model to generate appropriate labels for [Mask] during decoding as suggested in Gao et al. (2021a). The optimization objective is formulated as follows:
P = argmin P∈P
Lobj(F(T (X), P ; θ), Y ), (1)
where F is the generative model, θ is main parameters, P is the prefix space, Lobj is the tuning loss Eq.(6) and Y is the label space. The whole process of tuning is shown in Figure 3 and the advantages of this proposed method are shown in subsequent experiments. Label Extension with Pre-trained Models Both the OID and NID tasks face a real dilemma. Because of the inherent defects of annotation and diversity of intent expression, only one label given in the dataset usually can not accurately reflect the true intents behind the samples or even is wrong for some samples, which can be called Inherent Label Uncertainty (ILU). ILU not only affects the defi-
nition of decision boundary with IND intents but also weakens the ability of the model to generate correct labels.
To alleviate the Inherent Label Uncertainty, we extend the label space Y in the dataset and provide multiple candidate labels for each intent. We propose utilizing the emerging generative capacity of large generative language models such as GPT-3 or ChatGPT (used in this paper) to expand labels. For a specific label y ∈ Y in the training set, we use crafted template T followed by a certain amount of randomly selected samples x1:n in this category to prompt model F to expand the label. The extended label space can be denoted as Y = F(T (y), x1:n). The process of extension is shown in Figure 2(b).
Unlike the previous work of generating training samples using large models, compared with the number of samples required for training, the number of labels to be expended can be almost negligible. Therefore, our method is extremely efficient and can obtain labels of higher quality than human annotations with the help of general knowledge of large models. The Loss Function of OID Considering the existence of Inherent Label Uncertainty and the waste of generating labels for a large number of IND samples, it is not the best choice to directly use the generated labels for OID (See Appendix B for more discussion). We adopt the previous OID paradigm to detect through the learned discriminative representation of samples. For the semantic representation z of the input x, it can be obtained by averaging the hidden vectors outputted by the last layer of the model (decoder-only PTMs, GPT-2 (Rad-
ford et al., 2019)) or averaging the hidden vectors outputted by the encoder (encoder-decoder PTMs, BART (Lewis et al., 2020), T5 (Raffel et al., 2020)), which is shown in Figure 3. With the original label space Y , a head for the OID task can be trained by cross-entropy loss:
Lce = − 1
N
N∑
i=1
log exp(φyi(zi))∑
k∈[K] exp(φk(zi)) , (2)
where yi is the gold label for input xi, φ is a linear classifier and K is the number of IND classes. For each sample, there is also extended label space Y , which can help learn the discriminative representation. Inspired by the Multi-label research, we introduce an additional loss suggested in (Su, 2020) for a specific input x with Y:
Lex(x) = log(1 + ∑
i∈Ω,j∈Ω exp(φi(z)− φj(z))),
(3) where z is the representation of input x, Ω is the extended set of the label of x, Ω = Y −Ω is the set of remaining classes and φj(z) denotes the logit score of the j-th class. Intuitively, the purpose of Eq.(3) is to make the score of each extended class no less than that of each other class, so that the learned representation can be more discriminative. So far, we can train the OID-specific head by the following loss:
LOID = (1− α) · Lce + α · Lex, (4)
where α is a hyper-parameter and Lex is calculated by 1|X| ∑ x∈X Lex(x).
After obtaining the representation, in order not to rely on any assumptions or prior knowledge, we perform detection following Zhang et al. (2021c). First, determine a decision boundary in the representation space for each known intent. Those samples falling into the boundary are considered as the intent, and those not within any decision boundary are OOD. The Loss Funciton of NID The whole NID consists of two parts. First, generate labels for the samples identified as OOD, and cluster according to the label similarity to discover general intents. For label generation, we adopt the standard language modeling objective to decode:
LNID = −α(x) ∑
(x,u∈π(y))
|u|∑
j=1
p(uj |ui<j , T (x)),
(5) where D is the training data, (x, y) is a pair in D, π(y) is a set just containing extended labels (not original labels), T is the template of prompts and p is the conditional probability calculated by softmax function, whose input is the hidden vector output at the corresponding position of the last layer of the decoder and output is the probability of token uj . The α(x) is set as 1/Nπ(y).
Combined with Eq.(4) and Eq.(5), the finetune optimization objective are as:
LOBJ = (1− λ) · LOID + λ · LNID, (6)
where λ is a hyper-parameter to balance the loss of two tasks.
Since multiple similar labels can be generated for the same intent, to discover a more general new intent, samples with labels belonging to the same
intent should be divided into one group as an intent set. To this end, we establish a weighted association network (graph) with nodes as samples and the weights of edges as the similarity (ROUGE (Lin, 2004) adopted in this paper, see Appendix A for details and more discussion) between labels of the linked samples. We reformulate the new intent discovery as a minimum cost Multi-cut problem on a graph. Samples belonging to the same intent will be automatically divided into the same cluster due to the high label similarity (see Figure 4), and thus do not rely on any prior about OOD.
For a specific weighted association graph G = (V,E,W ), a multi-cut refers to a subset of edges dividing the graph into distinct clusters, which satisfies the following constraints:
P := {p(V1, . . . , Vn)| n⋃
i
Vi = V ;Vi ∩ Vj = ∅},
(7) where Vi is a node set, P is the space of all multicut and p is a specific cut.
The minimum cost multi-cut problem takes the weight of the edge W ∈ RE×E into account. Intuitively, a greater weight of an edge (u, v) ∈ E suggests a higher likelihood that u and v are in the same cluster and more cost is needed to remove the edge. The minimum cost muli-cut is to find a cut with the lowest cost, which can be defined as minp∈P <W, p> suggested in Abbas and Swoboda (2022). In this paper, we find the minimum cost muli-cut by the implementation in Abbas and Swoboda (2022), which is an algorithm that can be run in GPU. See Appendix A for more discussion.

heading: 4 Experiments

heading: 4.1 Evaluation Datasets and Backbones
text: We conduct extensive experiments across two challenging real-world datasets and three widely used generative models. CLINC (Larson et al., 2019) This is a widely studied intent dataset, which covers a wide range of intent categories. Specifically, This dataset includes 150 classes distributed across 10 different domains, consisting of 22500 utterances totally. BANKING (Casanueva et al., 2020) This is a dataset related to the banking business, which is notable for its imbalanced distribution of samples across different categories. The dataset includes 77 intents, consisting of 9003 training samples and 3080 test samples. Appendix C summarizes de-
tailed statistics of each dataset. To verify the generality and effectiveness of our proposed method, we set up benchmarks on the widely used generative models across various architectures, i.e., autoregressive language model (decoder-only, GPT-2 (Radford et al., 2019)), and encoder-decoder architecture (BART (Lewis et al., 2020), T5 (Raffel et al., 2020)), and make a comprehensive comparison with our proposed method.

heading: 4.2 Evaluation Protocol and Baselines
text: We follow the generally accepted metrics used in the previous work of OID and NID tasks. In the task of OID, as suggested in Zhang et al. (2021b); Zhou et al. (2022a), we calculate macro F1-score for IND and OOD classes donated as F1-IND and F1-OOD respectively. Calculate accuracy score (ACC-ALL) and F1-score (F1-ALL) on all classes meanwhile. For the task of NID, following Zhang et al. (2021c, 2022), we adopt the two metrics: Adjusted Mutual Information (AMI) and Adjusted Rand Index (ARI), to measure the quality of clustering (new intents found). In particular, we use the Hungarian algorithm (consistent with the previous methods) to align predicted classes and gold classes to calculate Accuracy (ACC). Finally, we calculate the macro average (AVG.) of these metrics to comprehensively measure the performance of different methods. Based on the above evaluation metrics of different tasks, we use two baseline methods (Model Tuning and Prefix Tuning) to establish comparable benchmarks on the above datasets. Model Tuning refers to fine-tuning the full parameters of models. Prefix-tuning is a variation based on Li and Liang (2021) and our method is introduced in Section 3. In particular, the cluster-based method is a common method in the NID field, so we also made a comparison with it. We perform K-means with the representations identified as OOD to discover new intents as Zhang et al. (2021c, 2022) do.

heading: 4.3 Experimental Setting
text: Following the general setting in OID and NID tasks, we randomly select 75% of the intent classes given in the dataset as known intents (IND intents), and the rest are regarded as unknown intents (OOD intents). The OOD samples in training and validation sets are discarded. In the OID task, the disposed of classes in the test set are grouped into one rejected class (remarked as OOD), while in the NID task,
the disposed of labels are retained in the test set to evaluate the quality of the predicted new class.
The details about the used models and hyperparameters are listed in Appendix D. Baselines and our method use the same experimental settings. Whether it is the main experiment or the analysis experiments, we use multiple different random seeds to conduct at least three rounds of experiments and report the average results.

heading: 4.4 Main Results
text: The comparison results of our methods and baselines across different generative models and datasets are shown in Table 1 (See Appendix D for the statistics of experimental parameters and the standard deviation). On the whole, our method obtain substantial improvements across various metrics in different datasets compared with baselines, which shows that our method can not only distinguish IND and OOD better but also better further distinguish OOD in fine granularity.
A closer look at Table 1, for the OOD Intent Detection, it can be observed from the table that BART and T5 are better than GPT-2 on the whole, and T5 performs better than BART on the CLINC dataset, but the opposite is true on the BANKING dataset. Interestingly, we observe that the effect of Prefixtuning is better than that of Model-tuning, especially in the BANKING dataset, which shows that overfitting not only affects the generation of labels but also affects the learning of representations. Furthermore, our method is better than Prefix-tuning, which shows that expended labels and prompts can help to learn discriminative representations.
Further observation of the comparison results on the New Intent Discovery task shows that the results of intent discovery based on label similarity are better than those based on cluster-based (Kmeans), reflecting the advantages of our proposed method. The comparison between different models shows that T5 performs better than other models in different datasets (across different training methods), which relies on the excellent generation ability of T5. The Prefix-based training methods are better than the Model-tuning, which shows that the Prefix-based training method can well alleviate the generated labels overfitting to the labels in the training set and is also in line with our expectations. At the same time, by comparing our method with Prefix-tuning, we can further show that prompts and extended labels can help the model generate
better labels (Appendix A).

heading: 5 Analysis

heading: 5.1 Impact of Prefix Length
text: In this section, we explore the specific impact of the length of the prefix. From Figure 5, we can observe that both tasks seem to be sensitive to the length of the prefix. A small prefix will not play its advantages. Further, the performance of Detection may decline with the increase of prefix length (especially for GPT-2). For the Discovery, similar phenomena will be observed, but the downward trend will be postponed. The above phenomenon may be attributed to the fact that the increase of the prefix length brings more fine-tuned parameters, which results in the model shifting toward the limited IND data, which not only weakens the ability to generate labels but also affects the learning of discriminative representations. Under various prefix lengths (other parameters remain the same), Our method is better than Prefix-tuning.

heading: 5.2 Towards a Win-win Training
text: We adopt hyper-parameter λ to balance the losses of two tasks in Eq.(6) during training. In this sec-
tion, we evaluate the benefits of λ in the training process. Specifically, we vary the value to obtain the changing trend of the performance of two tasks, and the results are shown in Figure 6. When the λ at around 0.5, the two tasks can achieve a win-win situation across the different models and different training methods, which demonstrates the rationality of extending Classification with Discovery. In addition, by varying the value of the λ while keeping other parameters unchanged, our method is always better than the baseline method. See Appendix B for more related discussion.

heading: 5.3 Effect of Extended Labels
text: In this section, we explore the effect of extended labels in the Open Environment Intent Prediction. The extended labels affect the Detection in the form of Lex in Eq.(3). By varying α in Eq.(4), we can observe the effect of extended labels. The results are shown in Figure 7, which shows that with the increase of weight α, the model can learn better
discriminative representations (F1-ALL rises), but if the α continues to increase, the recognition accuracy may be affected due to the influence of uncertainty between labels. More experiments in Appendix B can demonstrate its effect is general. For the Discovery, it has been proved that extended labels can alleviate the degradation of the generated vocabulary(Section 3) and help to discover new intents(Section 4.4). We also evaluate the quality of labels generated with the help of extended labels in Appendix A.

heading: 5.4 Necessity of Prompts
text: To steer the model to generate high-quality labels, we task the model with natural language prompts in the input. In this section, we explore the specific effect of prompts. The experimental results in BANKING are listed in Table 2, where the input in the first row is without prompt and the inputs in the following three rows are with templates generated in different ways. From Table 2, it can be observed that the existence of prompts can not only help with detection (Dete.) but also has an obvious effect on new intent discovery (Disc.). At the same time, the help of prompts is general. In addition to manual design, we also try to automatically generate templates based on Gao et al. (2021a) (Appendix E). Compared with only inputting utterances to the model, formulating input with these generated templates T (X) shows a certain degree of help.

heading: 6 Conclusion
text: In this paper, we strengthen a combined generative task paradigm to expand the two basic tasks of the Task-Oriented Dialogue system, which is more general and practical. Further, without relying on prior knowledge about OOD, we provide an effective and efficient implementation based on the generative model. At the same time, we introduce an effective method of intent expansion to
alleviate Inherent Label Uncertainty and provide a method for constructing multi-label intent datasets to inspire further research. Extensive experiments across different models and different datasets have verified effectiveness and generality.
Limitations
To better enlighten the follow-up research, we conclude the limitations of our method as follows: 1) Although the method we proposed can help improve the quality of generated labels, there is still room for further improvement; 2) Because our detection is not perfect, it will lead to inaccurate labels of some samples. We look forward to better methods to improve detection in the future; 3) This work has verified that the extended labels can effectively help the performance of models and proposed a method of label extension, but has not tried other extension methods or whether it is helpful to extend more labels. 4) This work focuses on solving Open Environment Intent Prediction with different generative models, without exploring other types of models.

heading: Acknowledgements
text: We thank Dr.Guo qipeng for his patient and valuable feedback in this work. This work was supported by the National Key Research and Development Program of China (No.2022CSJGG0801), National Natural Science Foundation of China (No.62022027) and CAAI-Huawei MindSpore Open Fund.

heading: A More Discussion on Generated Labels and New Intent Discovery
text: In this section, we evaluate the quality of generated labels. Because we discover general intents based on generated labels (Section 3), the better effect of intent discovery suggests the better quality of generated labels. In this paper, For efficiency and effect, we use ROGUE (Lin, 2004) to measure the similarity between two labels. Specifically, we calculate the average of the ROGUE-1, ROGUE2, and ROGUE-L3 F1-scores of two labels as the similarity score. In addition, for the sake of generality, we try two additional widely-used similarity measures: GloVe (Pennington et al., 2014) and BERTScore (Zhang et al., 2020). We use labels generated in different ways to discover intents and compare the effects in Table 3. Under different similarity measures, our methods have achieved better results, which shows that our methods can generate better labels.
At the same time, it should be emphasized that the scheme we proposed for new intent discovery based on generated labels in Section 3 is a general framework that can be flexibly implemented. In addition to the way to establish graphs described in Section 3, we can also build a weighted association graph with labels as nodes, whose edges
3https://pypi.org/project/rouge/
are the similarity between linked labels, then perform minimum cost multi-cut on this graph, where the segmentations (composed by similar labels) divided are also regarded as more general intents, and the whole process also does not depend on any prior or assumptions about OOD. We leave more and broader exploration for future research.
For the label for discovered general intents, you can either pick the label with the highest frequency in the corresponding segmentation as the label of the intent, or the labels with the top k highest frequency, which depends on the purpose of using the data.

heading: B More Comprehensive Comparison of Detection
text: As mentioned in Section 3, considering the existence of Inherent Label Uncertainty and the waste of generating labels (or clusters) for a large number of IND samples, we conduct the OID task based on the learned representation. To learn the discriminative representations, we enrich the expression of intent by multi labels and train together with the loss of generation Eq.(6) during training (The effectiveness is proved in Section 5.2).
In this section, in order to further verify the effectiveness of our method, we make a comprehensive comparison with various paradigms. Clusterbased refers to the paradigm adopted by previous work in the NID (Zhang et al., 2021c, 2022), all samples are directly clustered by K-means for intent discovery after learning representation, Detection-based means that only the OID loss LOID (λ = 0.0 in Eq.(6)) is used for training to obtain representations of samples, which is a paradigm in the OID task (Zhang et al., 2021b), and Label-based means that only the NID loss LNID (λ = 1.0 in Eq.(6)) is used for training then
discovery intents based on labels (same as that we used in 3). The experimental parameters of all methods are consistent.
We show the comparison results of different methods in Table 4, which demonstrates our method is superior to other methods. In addition, many meaningful observations can be obtained from the table. The introduction of extended labels can improve the effect of detection under different paradigms, which reflects the generality. The effect of the Cluster-based method is significantly lower than that of representation-based detection, which also shows that the previous paradigms in the NID not only waste a lot of costs to cluster IND samples but also may have very limited effect. The above comparison results can fully demonstrate the rationality and effectiveness of our method.

heading: C Statistics of Datasets
text: The detailed statistics of the datasets described in the Section 4.1 are summarized in Table 5.

heading: D Details of the Models and Hyper-parameters
text: In this paper, experiments are conducted on models with different architectures, i.e., decoder-only (GPT-2 (Radford et al., 2019)), and encoderdecoder architecture (BART (Lewis et al., 2020), T5 (Raffel et al., 2020)), whose details are shown in 6. The implementations of GPT-24, BART5 and T56 are based on the Huggingface Transformer models. We tried learning rate in {1e-4, 2e-4, 3e-4, 4e-4}, training batch size in {64,128}, the length of tunable prefix in {64,128,256} and trained 100 epochs with an AdamW optimizer. We utilized four extended labels for each intent during the experiment (In fact, for certain intents, the number of labels was expanded to five.). In the K-means setting, we set k to three times the ground truth number of intent categories. Baselines and our method use the same experimental settings. Whether it is the main experiment or the analysis experiments, we use multiple different random seeds to conduct multiple rounds of experiments and report the average results. We list the standard deviation of the main experiment results (Table 1) in Table 7. Our experiments are conducted on a single NVIDIA
4https://huggingface.co/gpt2 5https://huggingface.co/facebook/bart-base 6https://huggingface.co/t5-base
A100 Tensor Core GPU. We also have tried to conduct experiments on a single NVIDIA GTX 3090 with small batchsizes.

heading: E Automatic Generation of Templates
text: To verify the generality of the benefits of the prompts, in addition to manually designing templates, we use T5 to automatically generate models based on Gao et al. (2021a). The difference is that to maintain the semantics of labels, we have not pruned the generated vocabulary set. To generate templates, we formalize the input (x, y) ∈ Dtrain to T5 as x.<s1>y<s2> (The <s1> and <s2> are the mask tokens) and let T5 automatically fill in <s1> and <s2> (i.e., templates) during decoding. We select the templates with higher beam search scores as the candidates then use Ddev to pick templates with better performance. See Gao et al. (2021a) for more details.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section "Limitations" (7th Section)
3 A2. Did you discuss any potential risks of your work? Section "Limitations" (7th Section)
3 A3. Do the abstract and introduction summarize the paper’s main claims? "Abstract" and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.1 and Appendix C
3 B1. Did you cite the creators of artifacts you used? Section 4.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? These datasets are available for all researchers in the NLP community.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? These datasets are only for scientific research and are available for all members of the NLP research community. We have adhered to the typical method of utilizing these resources.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? These datasets are only for scientific research and are available for all members of the NLP research community. We have adhered to the typical method of utilizing these resources.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix C
C 3 Did you run computational experiments? Section 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4.3 and Appendix D
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.3 and Appendix D
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix A and Appendix D
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave more and broader exploration for future research. For the label for discovered general intents, you can either pick the label with the highest frequency in the corresponding segmentation as the label of the intent, or the labels with the top k highest frequency, which depends on the purpose of using the data