ClusterPrompt: Cluster Semantic Enhanced Prompt Learning for New Intent Discovery
Authors: Jinggui Liang
Lizi Liao
Venue: EMNLP 2023 Findings (2023)
Keywords: dialogue systems
intent discovery
prompt learning
contrastive learning

Abstract: The discovery of new intent categories from user utterances is a crucial task in expanding agent skills. The key lies in how to efficiently solicit semantic evidence from utterances and properly transfer knowledge from existing intents to new intents. However, previous methods laid too much emphasis on relations among utterances or clusters for transfer learning, while paying less attention to the usage of semantics. As a result, these methods suffer from in-domain over-fitting and often generate meaningless new intent clusters due to data distortion. In this paper, we present a novel approach called Cluster Semantic Enhanced Prompt Learning (CsePL) for discovering new intents. Our method leverages two-level contrastive learning with label semantic alignment to learn meaningful representations of intent clusters. These learned intent representations are then utilized as soft prompt initializations for discriminating new intents, reducing the dominance of existing intents. Extensive experiments conducted on three public datasets demonstrate the superiority of our proposed method. It not only outperforms existing methods but also suggests meaningful intent labels and enables early detection of new intents.

ClusterPrompt: Cluster Semantic Enhanced Prompt Learning for New Intent Discovery
Jinggui Liang Singapore Management University jg ww..iang..aa 2023@phdcs. .mu.edu. sg
Lizi Liao
Singapore Management University 1zl azliao@smu.ede edu.
Abstract
Eeprosmnaio Specr
Konenition nict
Ukdan hane
Clock
Fak haip
The discovery of new intent categories from user utterances is crucial task in expand- ing agent skills. The key lies in how to ef- ficiently solicit semantic evidence from utter- ances and properly transfer knowledge from isting intents to new intents. However, previous methods laid too much emphasis on relations among utterances clusters for transfer learn- ing, while paying less attention to the usage of semantics. As result, these methods suffer from in-domain over-fitting and often gener- ate meaningless new intent clusters due to data distortion. In this paper, present novel approach called Cluster Semantic Enhanced Prompt Learning (CsePL.) for discovering new intents. Our method leverages two-level con- trastive learning with label semantic alignment to learn meaningful representations of intent clusters. These learned intent representations are then utilized as soft prompt initializations for discriminating new intents, reducing the dominance of existing intents. Extensive ex- periments conducted on three public datasets demonstrate the superiority of our proposed method. It not only outperforms existing meth- ods also suggests meaningful intent labels and enables early detection of new intents.
Figure 1: The overview of the in in-domain over-fitting problem NID and our label semantic alignment.
ing intents to discover new intents and efficiently solicit semantic evidence from user utterances.
Existing NID methods can be divided into two categories: unsupervised and semi-supervised. In unsupervised NID, researchers mainly focus on how to extract better utterance features to assist the clustering (Padmasundari and Bangalore, 2018; Shi et al., 2018). But they tend to ignore the prior knowledge contained in the labeled data. Thus, in semi-supervised NID, various methods train repre- sentation learning models to facilitate knowledge transfer between labeled and unlabeled data, then perform clustering on utterance representations for NID in a two-stage fashion (Zhang et al., 2023; Zhou et al., 2022). In view of knowledge transfer, there works that first pre-train an in-domain intent classifier, and then gradually update it with clustered pseudo-labels (Lin al., 2020; Zhang et al., 2021c). There are also works that formulate contrastive learning objectives to optimize model parameters (Mou et al., 2022b; Zhang et al., 2022).
Introduction
New Intent Discovery (NID) aims to automatically identify novel intent categories that are not defined or observed beforehand. It plays critical role in task-oriented dialogue systems with the ability to discern newly emerging user preferences (Liao
The essence these two-stage methods lies in learning discriminative semantic features for utter- ances via minimizing intra-class variance and maxi- mizing inter-class variance in the first stage. Hence, the similarity or dis-similarity relations between utterances or clusters emphasized. For instance, wair-wise similaritics are used pscudo supervi- sion in (Lin et al., 2020) and various contrastive
et al., 20)3a,b), thereby providing high-quality ser- vices (Lin et al., 2020; Zhang et al., 2021c, 2023). Different from the traditional intent classification (E et al., 2019; Chen et al., 2019; Wang et al.,
2021), the key challenges of NID lie in how to properly transfer the prior knowledge from exist-

---

learning objectives naturally enforce such relations (Wei et al.. 2022; Mou et al., 2022a). However, the clustering process in the second stage can be easily distorted in favor of labeled data and domi- nant intent categories, resulting in the in-domain over-fitting problem shown in Figure 1. Another risk is that such relation distortion would obscure the semantic meaning of intent clusters, leading to less meaningful new intents.
Related Work
New Intent Discovery. Identifying new intents are key to adaptable conversational agents for better dialogue state understanding (Zhang et al., 2019; Liao et al., 2021). Previous research on NID can be predominantly categorized into two types: unsupervised and semi-supervised. For the former, early approaches (Cheung and Li, 2012; 22 Li et al., 2013) primarily relied on statistical features of the unlabeled data to cluster similar queries for discov- cring new user intents. Subsequently, some studies (Xie et al., 2016; Yang et al., 2017; Shi et al., 2018 88; Hadifar et al., 2019) have endeavored to leverage deep neural networks to learn robust representa- tions conducive to new intent clustering. However, none of these fully leveraged supervised signals, such as existing intent labels.
In this paper, we thus propose Cluster semantic enhanced Prompt Learning (CsePL) method for NID as two stages. Specifically, we leverage the semantic knowledge to regulate both of the two stages, which are formulated as Intent Cluster Rep- resentation Learning (ICRL.,, and Prompting for Intent Diserimination (PID). In the ICRL stage, besides using two-level contrastive learning ob- jectives to learn compact and closely connected regions for intents in feature space, align the intent cluster representations with their correspond- ing label semantics. It enables the model learn stable semantic features and semantic-aware intent cluster representations. In the following PID stage, we employ the learned intent cluster representa- tions for soft prompt initializations and integrate them into input utterances to facilitate new intent discrimination. Given that the new inputs encom- pass the semantics from all intents, the prompting mechanism will encourage the model to focus on matching the utterances with their inherent seman- tic meaning, thus reducing the dominance exist- ing intents. We evaluate the proposed CsePL on three widely-used datasets. It outperforms state-of- the-art (SOTA) methods in various aspects.
To address this, recent studies (Lin et al., 2020; Zhang et al., 2021 Mou et al., 2022b; Zhang et al., 2022) have extended NID to a semi- supervised setting to achieve prior knowledge transfer, in which the labeled data is incorporated into the training process assist new intent clus- tering. For example. Wei et al. (2022) and Zhang et al. (2023) first pre-trained a backbone model with the supervision of the limited labeled data. Then, they employed the pre-trained backbone to generate pscudo labels for the unlabeled data, di- recting the model to discern novel intents. Ad- ditionally, different from those pseudo-labeling based semi-supervised methods, Mou et al. (2022a) and Zhang al. (2022) sought directly optimize utterance representations with the aid of supervised data. They formulated distinct contrastive learning objectives to learn discriminative utterance repre- sentations, facilitating the similar utterance clus- tering and establishing distinct boundaries for new intent clusters.
To summarize, our contributions are three-fold:
In view of the in-domain over-fitting and meaningless intent problems, propose reiterate the importance of semantic knowl- edge in utterances and intent clusters for NID,
However, all these methods overemphasize on relations such as similarity or dissimilarity among utterances for better clustering effects, while be- little the usage of semantics inside utterances and intents. Similar to the trivial solution clustering (Yang et al., 2017; Caron et al., 2018; 8; Ji et al., 2019; Zhang et al., 2021a; Zheng et al., 2023), it brings the problem of in-domain over- wet--fiting and meaning- less new intents due to the data distortion. In our work, enhance model with semantic knowledge and soft prompts detect new intents.
We propose two-level contrastive learning ob- jectives with label semantic alignment for learning semantic-aware intent cluster repre- sentations and leverage the soft prompting mechanism to enhance the usage of semantic knowledge in intent discrimination.
Experiments show that CsePL not only gains significant improvements over SOTA meth- ods, but also suggests meaningful intent labels and enables early detection of new intents.
2

---

ICRL Loss
Represosaation Space
Encoder
Encoder
Label Sementio
Untemaee
8
Uniabeled Uiteranes
Claster Controids
Chatar-canitoii Repecsemations
Poll Togriter
(CLS) The ineat [MASK] in: [nmoroved] (SEP]
Posh Away
4y (CLS) [untrowee]] [SEP]
Figure 2: The overall architecture our proposed CsePL. framework new intent discovery. The left part denotes the ICRL stage and the right part is the PID stage, where [utterance] the original utterance, and (2,, Cs: are the soft prompts initialized by all the learned intent cluster representations.
Prompt Learning. It new NLP paradigm of leveraging pre-trained language models (PLMs) (Brown et al., 2020; Sanh et al., 2022; Deng et al.. 2023), which reformulates downstream tasks by inserting task-specific instructions into the input to align them with pre-training tasks. Early works (Jiang et al., 2020; Shin al., 2020; Yuan al., 2021; Ben-David al., 2022) mainly utilized discrete hand-crafted or automatically searched prompts to acquire knowledge from PLMs. How- ever, since discrete prompts are hard to optimize, recent works (Li and Liang, 2021; Lester et al., 2021; Liu et al., 2021; Gu et al., 2022; Hou et al., 2022) in prompt learning made efforts to optimize soft prompts in the continuous embedding space. is more flexible and performs well in various downstream tasks. Here we explore utilize both discrete prompts and soft prompts to leverage se- mantic knowledge for NID.
The CsePL Approach
3.1 Problem Formulation
Let Cs and C. denote the known intent set and un- known intent set respectively, where C₁ nC, Ø. In general, given a set of labeled data Diabled { 3:)ly: 1y
C+} and a set of unlabeled data Duniabeled {a:|yy E CR C,,, the goal of NID to identify potential unknown intents in C. from Duniabeled and classify the input I; into its corre- sponding intent y: where y: E Cj U... Here we focus on the semi-supervised NID setting.
3.2 Model Overview
The proposed CsePL model is illustrated in Fig- ure 2, which consists of two stages for discovering new intents. The first stage is Intent Cluster Repre- sentation Learning (ICRL) while the second stage is Prompting for Intent Discrimination (PID). We introduce these two stages in the following subsec- tions one by one. In general, the ICRL stage is designed to solicit meaningful intent cluster rep- resentations. To achieve this, we reorganize the user utterances using unified hand-crafted dis- crete prompt, which then provided inputs to BERT- backbone for feature extraction. Then the two-level contrastive learning with label semantic alignment objectives are applied to opti- mize model parameters. Given the learned intent cluster representations, the PID stage targets dis- criminating intents for all utterances under the all intents- aware situation. Hence, we construct soft prompts with these learned semantic-awar intent cluster representations initializations and per- form further fine-tuning guide the pre-trained
Contrastive Learning. Contrastive learning is a popular and effective approach to learn discrimi- native representations in both computer vision and NLP tasks (Chen et al., 2020; He et al., 2020; Fang and Xie, 2020; Carlsson et al., 2021; Giorgi et al., 2021; Gao al., 2021; Wu et al., b022a,b; Ye al., 2022: Guo al., 2022). The primary in- tuition of contrastive learning is to pull together positive pairs in feature space, while push away negative pairs. Motivated by its superior perfor- mance, contrastive learning has also been adopted to intent recognition in recent works (Zhang et al., 2021d; Mou et al., 2022a; Wei et al., 2022; Zhang ct al., 2022, 2023). We leverage it to help learn better utterance and intent cluster representations.
3

---

backbone to discriminate intents. During inference, we send all test utterances to the well-trained PID model to extract utterances' intents-aware repre- sentations, and then conduct K-means to predict the intent categories for them.
utterances sharing the same intent label as 2;, the utterance-level supervised contrastive loss is:
copls; (olf:.f3//)) (I
C
Σ log jC)(()
Eipre ff/T))
3.3 Intent Cluster Representation Learning
where f, = o(z),, 7i denotes the temperature in contrastive loss, and Φ is a normalizing projection. Beyond the label supervision, we also conduct unsupervised contrastive learning to help learning utterance representations similar to Equation 1. For the utterance n4, its positive sample is derived from its cropout-augmented view 24. Mcanwhile, the negative samples for æ; are taken from all other utterances and their corresponding augmentations within the same mini-batch.
Different from previous approaches that emphasis on relations among utterances or clusters, in this stage, we aim to enhance our model with semantic knowledge inside utterances and intents to learn meaningful intent cluster representations.
To achieve this, we first employ prompt learn- ing method to extract representations for the in- put utterances in both Diabeled and Dunlabeled, As aforementioned, it is an effective approach to lever- age PLMs to extract semantic information inside utterances. Given input utterance 74, we convert I; to z. z.pormpt by inserting unified discrete prompt into it. The prompted utterance prompt given as:
Morcover, since intent labels are high-quality supervision signals, we also adopt cross-entropy loss for labeled utterances to optimize the model.
Cluster-level Contrastive Loss. In the cluster level, we optimize contrastive objectives to pull each cluster representation close its utterance members while push away from other intent clus- ter utterances. We conduct K-means to obtain intent clusters from Duntabeled at the beginning of each training epoch. Suppose cluster S; directly define the cluster
prompt I
[CLS) The intent [MASK] is in: (SEP),
where "[CLS] The intent [MASK] is in:" are hand- crafted discrete prompt tokens. We tried differ- ent token designs and empirically chose these for best performance. We sent x. aepoopp to PLMs, and regard the extracted representation 2; position "[MASK]" as the representation of input mi- Then, we conduct two-level contrastive learn- ing with label semantic alignment to optimize the obtained utterance representations for learning meaningful intent cluster representations. In the utterance-level, conduct both supervised and unsupervised contrastive learning to learn more accurate utterance representations. In the cluster level, we enforce the cluster center to be far away from other cluster utterances and close its own cluster members. Beyond these, we further use label semantics to regulate the intent cluster center representations via contrastive learning objective.
{r1, .2, ... ). center c; for the cluster
S:
f
C;
S
j=1
Therefore, the cluster-level contrastive learning ob- jective is defined as:
cxplf. cf /T2) Extperple of /T2)
log
(2)
where C; is the corresponding intent cluster center representation of the utterance Ij, while сy repre- sents any intent cluster center that is not of x; and T2 denotes the temperature in this contrastive loss.
Utterance-level Contrastive Loss. Inspired by the contrastive learning scheme (Khosla et al., 2020) under supervised setting, we optimize the model to bring utterances under the same intent together while disperse utterances from different intents apart. Let y denotes the ground-truth in- tent label for utterance x; and ((ii) denote all
Semantic Alignment. For the labeled training data, we investigate to solicit semantic information from both utterances and intent labels and optimize the model with label semantic alignment. The intu- ition behind this is that, the label semantic features more stable in the fcature space and will not be influenced by the distribution of the training data.
4

---

Aligning the known intent cluster representations to such stable label semantic features can protect the model from in-domain over-fitting problem and enable the model to give suggestions about intent labels. To achieve this, we also adopt contrastive learning objective for label semantic alignment. Specifically, for each known intent yi Ck. we first use the BERT-based model to get embeddings for tokens in intent label y4- As each intent label may contain multiple tokens, we directly apply mean pooling operation to obtain intent label rep- resentation as:
a range of downstream tasks. Specifically, we uti- lize the learned intent cluster representations as soft prompt initializations. Given an input utter- ance Ii, we first use the pre-trained backbone to convert 2; into sequence of token embeddings
Fij {e|CLs), e1, [[SEP]] Then, insert sequence of soft prompt vectors Eg construct the formally input as:
Pprrmpt = {c1, C(CLS), 61.'1 ""(""))))
where C₁ is the intent cluster representation learned in the ICRL stage. The prompted input Eprompt is send to the pre-trained backbone to extract rep- resentation for I4. In this stage, we regard the extracted representation at position "[CLS]" after normalizing projection as the utterance represen- tation h;- It's noteworthy that each derived intent cluster representation acts as an independent soft prompt token in the prompted imput, which reveals all the intent candidate semantics to the model. To update the model in this stage, following the work of Zhang et al. (2022), we optimize a contrastive learning objective, which mines neighboring utter- ance representations and pulls them together and pushes away distant ones in the feature space. The contrastive learning loss is calculated as:
3el
BmbbEmb (ty),
(3)
==1
where [891 is the number of tokens in the intent la- bel yi and BertEmbb..) is the embedding projection in BERT-based model. 1 denotes the j-th token in the yt- Then, the label semantic alignment loss is defined as follows:
[Cul
Σ log exple /T3 Eits exple: 8/ /T8)
Cel allggn
(4)
where |C&| is the number of known intents and T33 denotes the temperature. Note that cluster indices generated by K K-means permuted randomly in different training epoch. To tackle this pseudo label inconsistency problem and provide high-quality su- pervised signals, we also conduct cluster alignment following the work of Zhang et al. (2021c).
crp(h; h \/T))a)) PCN(1) Σizs Eitee exp(h: h3) \/Ta)
LPID Σ log
where (i) is the neighbor utterance set of x; and T4 denotes the temperature. Similar to Zhang et al. (2022), we select the most similar 50 utterances to 2; in the feature space as its neighbors. Here is the mini-batch size and each utterance is accompa- nied with augmented version. During training, we update the neighbor utterance set ((i) every few epochs to guide the model to form clear cluster boundaries for new intent discovery.
As a result, the overall loss function in the ICRL stage can be formulated as follows:
LIGRI. = Loe + aLd BLud XLd du align
where Zyv is the cross-entropy loss. far X, represent paprr-paraeeters that modulate the re-
spective contributions of distinct losses.
3.4 Prompting for Intent Discrimination
4 Experiments
In the former stage, we have trained the model with different contrastive learning objectives to learn meaningful intent cluster representations. In the PID stage, we apply soft prompt learning to exploit these learned representations for efficient intent discrimination. It is an effective and flexible approach to leverage the semantic knowledge of PLMs and demonstrates strong performance across
4.1 Datasets
We conduct experiments to evaluate the perfor- mance of our CsePL on three widely-used NID datasets. Banking77 (Casanueva et al., 2020) is a fine-grained banking domain dataset and comprises 13,083 customer service queries over 77 intents. Cline150 (Larson et al., 2019) is multi-domaii
5

---

Banking77
Clinc150
StackOvrrllw
Known Intent Rate
Methods
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
ACC
DTC
55.59
31.75
79.35
$6.90
17.51
29.54
CDAC+
66.39
33.74
48.00
84.68
50.02
66.24
46.16
30.99
51.61
DeepAligned
70.50
37.62
49.08
88.97
64.63
74.07
50.86
37,96
54,50
25%
DCSC
78.18
60.15
91.70
72.68
79.89
MTP-CLNN
80.04
52.91
65,06
93.17
76.20
83.26
73.35
54,80
74.70
USNID
81.94
56,53
65.85
94.17
77.95
83.12
74,91
65.45
75,76
CsePI.
83.32'
60.36'
71.06'
94.07
79.6S1
86,16'
74.88
64.92
79.47
DTC
69.46
37.05
49.85
83.01
50.44
64.39
49.80
37.38
52,92
CDAC+
67.30
34.97
48.55
86.00
54.87
68.01
46.21
30.88
51.79
DeepAligned
76.67
47.95
59.38
51.
72.56
80.70
68.28
57.62
74.52
50%
DCSC
81.19
56.94
68,30
93.75
78.82
84.57
MTP-CLNN
83.42
60.17
70.97
94.30
80.17
86.18
76.66
62.24
80.36
USNID
85.05
63.77
73.27
95.45
82.87
87.22
78.77
71.63
82,06
CsePI.
85.65
66.66
76.94'
95.09
83.14
88.66
80,28
71.99
85.68
DTC
74.44
44.68
57.16
89.19
67.15
77.65
63.05
53.83
71.04
CDAC+
69:54
37.78
51.07
85.96
55.17
67.77
58.23
40.95
64.57
DeepAligned
79.39
53.09
64.63
93.92
79.94
86.79
73.28
60.09
77.97
75%
DCSC
84.65
89.70
MTP-CLNN
86.19
66,98
77.22
95.45
84.30
89.46
77.12
69.36
82.90
USNID
87.41
69.54
78.36
96.42
86.77
90.36
80.13
74,90
85.66
CscPL
87.70
71.36"
81.93'
96.58
88.881
93.46'
82.81
75.99"
87.80'
Table I: Main performance results on the new intent discovery across three public datasets. denotes p-value<0.01,, * denotes p-v0lue<00005 and denotes p-value:>0.05 under 1-test.
datasct and contains 22,500 samples with 150 in- tents across 10 domains. StackOverflow (Xu et al.. 2015) dataset is collected from Kaggle.com, which includes 20,000 samples over 20 intents.
4.4 Main Results
4.4.1 NID Performance Comparison
We present the main performance comparison re- sults in Table where the best results are high- lighted in bold. Generally speaking, our proposed CsePL. achieves significant improvements com- pared with the previous baselines. Here, we present the result analyses from the following aspects:
In the experiments, we retain the same division of Banking77, Cline150, and StackOverflow as de- lineated in Zhang et al. (2023). More experimental details can be found in Appendix A.2.
Our proposed CsePL learns cluster-friendly semanties for discovering new intents: It can be seen that the proposed CsePL outperforms the baselines such as USNID and MTP-CLNN sig- nificantly and achieves new SOTA performances on three NID datasets. For example, compared with the USNID, our CsePL improves the ACC by 3.57%, ARI by 1.82%, and NMI by 0.29% on the Banking77 dataset with 75% known intent rate. It is worth mentioning that the Banking77 dataset is fine- fine-grained dataset collected from banking dia- logues, whose utterances and intent labels contain rich semantic knowledge. This demonstrates that the CsePL. can mine valuable and cluster-friend]] semantic knowledge from the training data to en- hance new intent clustering.
4.2 Evaluation Metrics
We adopt three commonly used metrics evaluate the clustering performance: Normalized Mutual Information (NMI), Adjusted Rand Index (ART) and Accuracy (ACC). To evaluate ACC, we use the Hungarian algorithm (Kuhn, 1955) to construct the mapping between predicted clusters and ground- truth intent categories. Note that ACC is the most important evaluation metric in experiments.
4.3 Baselines
We extensively compare the proposed CsePL method with several SOTA baselines including DTC (Han et al., 2019), CDAC+ (Lin et al., 2020), DeepAligned (Zhang et al., 2021c), DCSC (Wei al., 2022), MTP-CLNN (Zhang et al., 2022) and the most recent USNID (Zhang et al., 2023). We leave the details of the baselines in Appendix A.I.
Interestingly, we find that although the proposed CsePL tends to perform worse than the strongest bascline USNID the NMI metric for the 25% StackOverflow setting, the results are generally
6

---

Intent Type
Intent Label
Related Tokens
lost or stolen phone
or on lost charge statement extra stolen supported phone supported and cards currencies cute surfer schooner globe failed extra on charge statement up top tornadoes hundreds cash withdrawal declined money metropolitan why pickup desk
atm support
Ck
top up failed supported cards and currencies
card working
order physical receiving money withdrawal card clay exchange via app club cash crank pilot reset hit top up failed limits not sometimes entire get upper receiving transfer order physical losing lily money lost clay
exchange charge
C
top up reverted wrong amount of cash received
Table 2: Intent label suggestions: the related tokens that appear in the intent label are marked in red, while the related tokens that have similar meaning to the intent label tokens are highlighted in bold.
less stable as evidenced by the large p-value. We find that given different random seeds, CsePL would perform better or worse than the USNID on the 15%tSStackOvrllle This might due to the difference the of known intents selected.
4.4.2 Suggesting New Intent Labels
In order demonstrate that our proposed CsePL can solicit semantic knowledge from existing utter- ances and intent labels to suggest new intent cluster labels, we select four known intents and unknown intents from C₁ and C, respectively in the Bank- ing77 dataset with the 25% known intent rate, and utilize their intent cluster representations learned in the ICRL stage search for the most related tokens from the whole BERT-based vocabulary. We use the cosine similarity for the ranking and only present the most relevant tokens after filtering special tokens such as "[MASK]" and "[CLS]".
Our proposed CsePL reduces in-domain over- fitting for NID: We can observe that our CsePL. maintains its superior performance when con- fronting the predominance of a larger scale labeled data. For example, for the Clinc150 dataset with 75% known intent rate setting, the CsePL surpasses the best-performing baseline USNID by margins of 3. 1% ACC, 2.1 2.11% in ARI, and 0.1 1.16 in NMI. is worth noting that the C5inc150 dataset encom- passes 150 distinct intents and possesses a greater number of labeled training utterances for known intents. This shows the efficacy of the CsePL. in effectively mitigating in-domain over-fitting.
As reported in Table 2, for the known intents, the intent cluster representations learned by our CscPL can exactly retrieve the label tokens or semantic similar tokens from the BERT-based vocabulary. For example, given an intent top up failed, all to- kens appearing in the intent label are retrieved by the intent cluster representation as the most related tokens. For the intent supported cards and cur- rencies, proposed CsePL search for and distinguish the semantic similar tokens such as cash and money as the relevant tokens. This sug- gests that the intent cluster representations derived by our CsePL can accurately capture the semantics associated with their respective intent labels.
Effect of different known intent rates: From the Table 1, we can observe that the performances of all NID methods gradually decrease with the known intent rate going down. As the lower the known intent rate is, the less labeled data is available for guiding the model training, which leads to more difficult prior knowledge transfer for discovering new intents. However, with the decrease of the known intent rale, our proposed CsePL achieves more substantial improvements. For example, on the Banking77 dataset with a 25% known intent rate, the CsePL. achieves 5.21% ACC improvement compared with the USNID, while the ACC improvements are 3.67% and 3.57% with known intent rates of 50% and 75%, respectively, These results further indicates that our proposed CsePL is more generalized in NID.
It is noteworthy that the CsePL is also capa- ble of providing meaningful label suggestions for unknown intents. For instance, our CsePL picks up the token exchange from the vocabulary as the most relevant token for the intent exchange char in Cu. Even for the unknown intent wrong amount of cash received with more complex semanties, the semantic similar tokens receiving and money are retrieved as the related tokens by the CsePL. This shows the ability of the CsePL in suggesting
7

---

Banking77
Banking77
Methods
Methods
NM1
ARI
ACC
NMI
ARI
ACC
MTP-CLNN
74.12
43.87
56.67
CsePi.
83,32
60.36
71.06
25%
USNID
74.52
44,46
56.12
25%
w/o PID
77.88
50.21
61.38
CsePI.
76.31
47.39
59.17
w/o Semantic-liga
82.12
58.09
68.75
CsePL
85,65
66.66
76.94
Table 3: Results of early detection of new intents. For each unknown intent, only 20 utterances are available.
50%
w/o PID
83,91
62.63
73.46
w/o SemanricAlign
85.38
65.88
76.38
CscPL
87.70
71.36
81,93
75%
w/o PID
86.91
69. 23
79.40
w/o SemanticAliga
87.39
70.84
81.24
meaningful unknown intent labels.
Table 4: Ablation results on Banking77 dataset.
4.4.3 Early Detection of New Intents
Early detection is critical requirement for new intent discovery methods. To demonstrate this, we compare the performances of different methods when only a few utterances are available for each unknown intent. The results are reported in Table 3. We can observe that when the utterances for each unknown intent are limited, all methods per- form worse than before, but the proposed CscPL significantly outperforms other two methods. It indicates the ability of the CsePL discriminating new intents in the early stage. This also signals the importance of leveraging semantic knowledge.
Banking77
Cluster Num
Methods
NMI
ARI
ACC
USNID
81.94
56.53
65.85
K = 77 (gold)
CsePL
83.32
60.36
71.06
USNID
78.11
49.18
60.72
K 74 (predicted)
CscPL
81.30
56.70
69.75
Table 5: Effect estimating cluster number K.
that investigate various soft prompt initialization techniques during the PID stage. We leave the de- tails of the experimental results in Appendix A.3.
4.5 Detailed Analysis
4.5.2 Efffet of Semantic Alignment
In this subsection, we conduct detailed analysis to explore the impact of each key component in the CsePL: 1) CsePL w/o PID: we entirely remove the PID stage and only use the model trained in the ICRL stage for NID. 2) CsePL w/o SemanticAlign: we remove the label semantic alignment (Equation 4) in the ICRL stage during training. 3) We also analyze the effect of predicted cluster number K. Note that present results exclusively for the Banking77. While other datasets exhibit similar patterns but we omit them due space limitation.
We also compare the model performance of remov- ing semantic alignment process in the ICRL stage with the standard CsePL to explore the contribution of the semantic alignment. We find that eliminating the semantic alignment for intent cluster represen- tation learning degrades the performance for new intent discovery. For example, the ACC drops from 71.06% to 68.75% on the Banking77 dataset with the 25% known intent rate without utilizing the semantics. This demonstrates the importance of the semantic alignment process.
4.5.1 Effect of Prompt Discrimination
4.5.3 Effect of Estimating Cluster Number K We have been assuming the cluster number K as given hyper-parameter in the same fashion baselines. However, in the practical dialogue sys- tems, the number of clusters unknown and important to predict K for new intent discov- ery. Following the work of Zhang et al. (2021c), we predict the cluster number K via an estima- tion algorithm. More details can be found in Ap- pendix A.4. We present the model performances of different cluster number K in Table 5. We can observe that although the performances of both
As shown in Table 4, formance of the CsePL substantially diminishes across all the known intent rates after removing the PID stage. Especially, when the known intent rate set to 25% the ACC of the CsePL considerably drops 9%+ while the NMI and ARI drop 5%+ and 10%+ respectively. It indicates that prompting the input with all the intent cluster representations learned during the ICRL stage can assist discrimi- nating new intents. To delve deeper into the effec- tiveness of the PID, also conduct experiments
observe that the per-
8

---

Youd
Methods
UPK
KPE
USNID
0.091
0.046
25%
CscPI.
0.065 (-28.57%)
0.041 (-70.87%)
Table 6: Ratio of wrongly predicted intents. UPK de- notes utterances that belong to Unknown intents but are inaccurately Predicted Known intents. KPE repre- sents utterances that originate from Known intents but are Predicted Erroneously.
so.w Aetod yac whtdoon
pooting prpser
Figure 3: T-SNE visualization. The prefix "UK_" " and "K_" " denote unknown intents (hollow circles) and known intents (solid circles) respectively.
from known intents but were inaccurately pre- dicted. We can observe that the leading baseline USNID incorrectly classifies 9.1% of utterances with unknown intents known intents. This mis- classification rate is nearly twice that of known intents being predicted inaccurately, which stands at 4. 4.6%. This implies that the presence of known intent data could excessively sway the clustering procedure, leading to the in-domain over-fitting problem. Compared with the USNID, the cluster- ing outcomes derived from the CsePL demonstrate diminished influence of the known intent data, leading to notable reduction in both ratios.
our CsePL and the SOTA baseline USNID decline with an inaccurate cluster number K, the proposed CsePL. still achieves significant performance im- provements over the USNID method. It shows that the proposed CsePL is more robust regarding the estimated cluster number K.
4.6 NID Representation Visualisation
In order to more intuitively analyze the effect of the proposed CsePL in representation learning, present t-SNE visualizations comparing the lead- ing baseline USNID and our CsePL approach, as illustrated in Figure 3. The USNID visualization reveals that data points of the unknown intent are distorted and dispersed within two known intent clusters wnable to verify identity and verify my iden- lity. This results in the in-domain over-fitting prob- lem. Furthermore, this dispersion undermines the meaning of the newly learned intent cluster, as encompasses instances from three distinct in- tents. Conversely, the visualization for the CsePL demonstrates how the label semantic alignment ef- fectively aligns the intent cluster representations with the semantics of their corresponding labels. This process renders the unknown intent cluster, why verify identity, more coherent and less dis- torted. Additionally, with reduced noise in this cluster, its meaning becomes more discernible.
Conclusion
In this paper, we reemphasized the importance of semantic knowledge in new intent discovery and proposed Cluster semantic enhanced Prompt Learning (CsePL) method. Specifically, we de- signed two-level contrastive learning with label se- mantic alignment for intent cluster representation learning, and soft prompting method to lever- age the learned intent cluster representation for NID. Experimental results on three public datasets demonstrate the effectiveness the CsePL. Ex- tensive analyses further show that the CsePL not only significantly outperforms the existing base- lines, but also implies new intent labels and detects the appearance of the new intents at an early stage.
Acknowledgement
4.7 Error Analysis
In this subsection, we conduct an error analysis to delve into the problem of in-domain over-fitte over-fitting and to evaluate the effectiveness of our proposed CsePL method. In Table 6, we present the ratio of unknown intent samples that the model wrongly classified as known intents. Additionally, high- light the percentage of utterances that originate
This research is supported by the Ministry of Edu- cation, Singapore, under its AcRF Tier 2 Funding (Proposal ID: T2EP20123-0052). Any opinions, findings and conclusions or recommendations ex- pressed in this material are those of the author(s) and do not reflect the views of the Ministry of Edu- cation, Singapore.
9

---

Limitations
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. BERT for joint intent classification and slot filling. CoRR, abs/1902.10900
We discuss the limitations from the following per- spectives: (1) Usage of LLMs. Recently, large lan- guage models (LLMs) such as ChatGPT or GPT-4 have exhibited their outstanding performance various NLP task, showing its abundance in seman- tic knowledge. Though BERT has advantage in relatively low resource consumption, we will look into how to leverage LLMs's knowledge for better NID. (2) New intent labels. Although our method has show the potential in suggesting new intent labels, we plan to further investigate the possibility of generating the whole label directly, which will be more useful. (3) Early detection. It is critical in deployed systems. We plan to further look into this aspect and conduct comprehensive experiments to test the limit on how early it can work.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In ICMI., pages 1597-1607.
Jackic Chi Kit Cheung and Xiao Li. 2012. Sequence clustering and labeling for unsupervised query intent discovery. In WSDM, pages 383-392.
Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. 2023. Prompt- ing and evaluating large language models for proac- tive dialogues: Clarification, target- target-guided, and non- collaboration. arXiv preprint 23X.::2305.1666..
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre vre-training of deep bidirectional transformers for language under- standing. In NAACL-HLT, pages 4171-4186.
Haihong E, Peiqing Niu, Zhongfu Chen, and Meina Song. 2019. A novel bi-directional interrelated model for joint intent detection and slot filling, In ACL, pages 5467-5471.
References
Eyal Ben-David, Nadav Oved, and Roi Reichart. 2022.
PADA: example-base based prompt learning for ou-the-fly fly adaptation to unseen domains. TACL, 10:414-430 433.
Hongchao Fang and Pengtao Xie. 2020. CERT: con- trastive self-supervised learning for language under- standing. CoRR, abs/2005. 12766.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Πlya Sutskever, and Dario Amodei. 2020. Language models are few-shot learn- ers. NeurlPS, pages 1877-1901.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In EMNLP, pages 6894-6910.
John M. Giorgi, Osvald Nitski, Bo Wang, and Gary D. Bader. 2021. Declutr: Deep contrastive learning for unsupervised textual representations. In ACL- IJCNLP, pages 879-895.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. 2022. PPT: pre-trained prompt tuming for few-shot lcarning, In ACL, pages 8410-8423,
Shasha Guo, Jing Zhang, Yanling Wang, Qianyi Zhang, Cuiping Li, and Hong Chen. 2022. DSM: ques- tion generation over knowledge base via modeling diverse subgraphs with meta-learner. EMNLP, pages 4194-4077.
Fredrik Carlsson, Amaru Cuba Gyllensten, Evan- gelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2021. Semantic re-tuning with contrastive tension. In ICLR.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsuper- vised learning of visual features. In Computer Vision ECCV 2018- 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Science pages 139-1166.
Amir Hadifar, Lueas Sterekx, Thomas Demeester, and Chris Develder. 2019. A self-training approach for short text clustering. In RepLANLP@ACL. pages
Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2019. Learning discover novel visual categories via deep transfer clustering. In ICCV, pages 2400-8408.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Effi- cient intent detection with dual sentence encoders. In nip4commai, pages 38-45.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for un- supervised visual representation learning. In CVPR, pages 9726-9735.
10

---

Yutai Hou, Hongyuan Dong, Xinghao Wang, Bohan Li, and Wanxiang Che. 2022. Metaprompting: Learning to learn better prompts. In COLING, pages 3251- 3262.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021. P-tuning v2: Prompt tuning can be comparable fine- tuning universally across scales and tasks. CoRR, abs/21 10.07602.
Xu Ji, Andrea Vedaldi, and João Henriques. 2019, Invariant information clustering for unsupervised im- age classification and segmentation. In ICCV, pages 9864-9873. IEEE.
Yutao Mou, Keqing He, Pei Wang. Yanan Wu, Jingang Wang. Wei Wu, and Weiran Xu. 2022a. Watch the neighbors: A unified k-nearest neighbor contrastive learning framework for OOD intent discovery. In EMNLP, pages 1517-1529.
Zhengbao Jiang. Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know. TACL, 8:423-438.
Yutao Mou, Keqing He, Yanan Wu, Zhiyuan Zeng. Hong Xu, Huixing Jiang, Wei Wu, and Weiran Xu. 2022b. Disentangled knowledge transfer for OOD intent discovery with unified contrastive learning, In ACL, pages 46-53.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarn,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. NeurIPS, 33: 18:16 18673.
Padmasundari and Srinivas Bangalore. 2018. Intent discovery through unsupervised semantic text clus- tering. In Interspeech, pages 606-610.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly. 211-2)883-77.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj. Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. ICLR.
Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In EMNLP-IJCNLPP pages 131 1311-1316.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale parameter-efficien prompt tuning, In EMNLP, pages 3045-3059.
Xiang Lisa and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP, pages 4582 4597.
Chen Shi. Qi Chen, Lei Sha. Sujian Li, Xu Sun, Houfeng Wang, and Lintao Zhang. 2018. Auto- dialabel: Labeling dialogue data with unsupervised learning, In EMNLP. pages 684-689.
Yanen Li, Bo-June Paul Hsu, and ChengXiang Zhai. 2013. Unsupervised identification of synonymous query intent templates for attribute intents. CIKM, pages 2029-2038.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with au- tomatically generated prompts. In EMNLP, pages 4235..
Lizi Liao, Le Hong Long, Yunshan Ma, Wenqiang Lei, and Tat-Seng Chua. 2021. Dialogue state tracking with incremental reasoning. TACL, 9:557-569.
Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023a. Proactive conversational agents. In WSDM, pages 1244-1247.
Jixuan Wang, Kai Wei, Martin Radfar, Weiwei Zhang, and Clement Chung. 2021. Encoding syntactic knowledge in transformer encoder for intent detec- tion and slot filling. In AAAI, pages 13943-13951.
Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023b. Proactive conversational agents the post-chatgpt world. In SIGIR, pages 3452-3455.
Feng Wei. Zhenbo Chen, Zhenghong Hao, Fengxin Yang, Hua Wei, Bing Han, and Sheng Guo. 2022. Sumi-supe clustering with contrastive learning for discovering new intents. CORR, abs/2201.07604.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Dis- covering new intents via constrained deep adaptive clustering with cluster refinement. In AAAI, pages 8360-8367.
Xing Wu, Chaochen Gao, Zijia Lin, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022a. Infocse:

---

Information-aggregaaeed contrastive learning of sen- tence embeddings. In Findings of EMNLP, pages 3060-3070.
Zheng Zhang, Lizi Liao, Minlie Huang, Xiaoyan Zhu, and Tat-Seng Chua. 2019. Neural multimodal belief tracker with adaptive attention for dialogue systems. In WwW, pages
Yuxia Wu, Lizi Liao, Xueming Qian, and Tat-Seng Chua. 2022b. Semi-supervised new slot discovery with incremental clustering. In Findings of EMNLP. pages 6207-6218.
Xiaolin Zheng, Mengling Hu, Weiming Liu, Chaochao Chen, and Xinting Liao. 2023. Robust representation learning with reliable pseudo-labels generation via self-adaptive optimal transport for short text cluster- ing. In ACI. pages 10493-10507.
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analy- sis. In ICML, volume 48, pages 478-487.
Yunhua Zhou, Peiju Liu, and Xipeng Qiu. 2022. Knn- contrastive learning for out-of- intent classifi- cation. In ACL, pages 5129-5141.
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural net- works. In VS@NAACL-HLTT, pages 62-69.
Yang. Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. 2017. Towards k-means- k-means-friendly spaces: Simultaneous deep learning and clustering. In ICML, volume 70, pages 3861-3870.
Chenchen Ye, Lizi I. Liao, Suyu Liu, and Tat-Seng Chua. 2022. Reflecting on experiences response gener- ation. In ACM Multimedia, pages 5265-5273.
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text genera- tion. In NeurlPS, pages
Dejiao Zhang, Feng Nan. Xiaokai Wei, Shang- Wen Li, Henghui Zhu, Kathleen R. McKeown, Ramesh Nal- lapati, Andrew Arnold, Bing Xiang. 202 Supporting clustering with contrastive learning. In NAACL-HLT, pages 5419-5430.
Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. 2021b. TEXTOIR: An in- tegrated and visualized platform for text open intent recognition. In ACL-IJCNLP, pages 167-174.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021c. Discovering new intents with deep aligned clustering. In AAAI, pages 14365-14373.
Hanlei Zhang, Hua Xu, Xin Wang, Fei Long, and Kai Gao. 2023. USNID: A framework for unsupervised and sewi-eemsupeeeee supervised new intent discovery. CoRR, abs/2304.07699.
Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 202 Few-shot intent detection via contrastive pre-training and fine-tuning, In EMNLP. pages 1906-1912.
Yuwei Zhang, Haode Zhang. Li-Ming Zhan, Xiao-Ming Wu, and Albert Y. S. Lam. 2022. New intent discov- ery with pre-training and contrastive learning. In ACL, pages 256-269.
12

---

Appendix
A.1
Additional Details of Baseline Methods
In this work, we compare the proposed CsePL framework against several representative baselines including:
320 Aa Magaie
Lo
DTC (Han et al., 2019): semi-supervised deep embedding clustering method with mechanism to estimate new intent numbers. CDAC+ (Lin al., 2020): a pseudo-labeling method the uses pairwise similarities to guide the clustering process.
Figure 4: NID performance of distinct loss weights on the Banking77 dataset with 25% known intent rate.
DeenAligned (Zhang et al., 2021c): an im- proved DeepClustering (Caron et al., 2018) that uses an aligmment strategy to alleviate the label inconsistency problem.
contrastive loss, we employ a projection layer to transform the output representations of the back- bone into (68-dimensions. vector space. The contrastive learning temperatures {TI, 72, Ta are uniformly set to 0.07. The values of (a, (3 X, n) are 1.0 to balance distinet losses in CICLL We also conduct extensive hyper-parameter explo- ration experiments to determine appropriate values for {a, 3, A, 77- The detailed experimental results are reported in Figure 4. It's worth mentioning that our approach exhibits relative stability concerning these hyper-parameters. Thus, we select 1.0 as our definitive setting in experiments.
DCSC (Wei et al., 2022): two-stage NID method with the dual-task process, which re- gards the soft pseudo cluster assignments as the supervision signals train the model. MTP-CLNN (Zhang et al., 2022); a method that applics multi-task pre-training and near- est neighbors contrastive learning for NID. USNID (Zhang et al., 2023): a two-stage framework for both unsupervised and semi- supervised NID with efficient centroid- guided clustering mechanism.
For fair comparison, the BERT-based model is used as the backbone in all aforementioned baselines. The external labeled dataset used in MTP-CLNN is also removed.
the PID stage, we train the model for 50 epochs with the learning
x 10 5 The output representations in this stage are mapped to a 128- dimensional vector space for contrastive learning.
The temperature 74 is set to 0.07 similarly. Note that all experimental results averaged over 4 different random seeds.
A.2 Experimental Details
In the experiments, we keep the same split of Bank- ing77, Cline150 and StackOverflow as (Zhang et al., 2023). For each dataset, we randomly se- lect a certain ratio (25%, 50%, 75%) of intents as known intents, and sample 10% labeled instances from known intent categories to form labeled subset, while the remaining instances are treated the unlabeled data.
A.3 Results of Different Soft Prompt Initialization Methods
To further study the effect of the soft prompts in the PID stage, we compare the NID performance of different CsePL variants across various datasets, illustrated in Table 7. Specifically, we incorporate two types prompts: soft prompts initialized ran- domly and hand-crafted discrete prompts, and iden- tically integrate them with input utterances in the PID stage for comprehensive experiments. We can observe that, when utilizing hand-crafted discrete prompts in the PID stage, the NID performance consistently declines across all datasets in compar- ison the standard CsePL. The CsePL-random
For the proposed CsePL, we adopt the BERT- model (Devlin et al., 2019) (bert-buse-uncased with 12 transformer layers) as the backbone model. The AdamW optimizer is used to update the model parameters. In the ICRL stage, we pre-train the model 100 epochs with 20-epochs patience to validate carly-stopping the developments. The learning is set 5 10-5, To calculate
13

---

Banking77
Clinc150
StackOverflow
Known Intent Rate
Methods
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
ACC
CsePL
83.32
60.36
71.06
94.07
79.65
86.16
64.92
79.47
25%
CsePl-random
82.13
58.22
69.45
93.36
77.23
84.74
74.27
63.57
77.25
CsePI-manual
81.24
56.89
69.11
93.36
77.06
83.71
73.43
61.48
76.30
CscPL
85,65
66.66
76.94
95.09
83,14
88.66
80.28
71.99
85,68
50%
CsePl rrandom
85,45
65.50
75,91
94.63
81.41
87.85
78.60
70.31
84,20
CsePl
84.26
63.76
75.47
94.32
80.07
86.98
77.60
69.03
83.95
CscPL
87.70
71.36
81.93
96.58
88.88
93.46
82.81
75.99
87.80
75%
CscPL-randon
86.80
69.57
79.97
95.87
85,98
91.02
79.96
72.16
85.55
CscPL-manual
68.94
79.76
95.61
90.54
80.39
71.82
84.75
Table 7: NID performance comparison across different soft prompt initialization methods. CsePL-random and CsePL-manual represent the CsePL variants wherein the prompts in the PID stage are randomly initialized soft prompts and hand-crafted discrete prompts respectively.
employs soft prompts with random initialization in the PID stage and exhibits a slight performance uplift compared with the CsePL-manual. Yet, both of CsePL-random and CsePL-manual consistently lag behind the CsePL in the discovering new in- tents. This indicates that utilizing soft prompts, which are initialized based on the learned intent cluster representations, can provide the model with comprehensive array of all intent options and effectively directs the model in matching input ut- terances to their respective intents.
A.4 Estimate Cluster Number K
In practical dialogue systems, intents emerge constantly and we cannot know the exact number of the intent clusters. In this paper, following the work of Zhang et al. (2021c), we take the fully usage the well well-initialized intent features to au- tomatically estimate the intent cluster number K. Specifically, we first assign big K' as the initial intent cluster number. Then we directly use the well-pretrained model to extract the feature rep- resentations for the training data and perform the K-means algorithm group these feature represen- tations into different clusters. From these clusters, we can distinguish the dense and boundary- clusters as the real intent clusters, while the remain- ing low size clusters are filtered out. The filtering function can be formulated follows:
K'
K=
Σ 5(S > 1)
(5)
where |S:] is the size the in grouped cluster, is the threshold of filtering. 8(-) is the indicator function, whose output is if the condition is satisfied.
14


Reviews:
[{"review_id": "3IoQSQRgAF", "reviewer": "Reviewer_QMDg", "summary": "", "questions": "1. The core innovation appears to lie in the application of label semantic alignment in ICRL and the use of soft prompts in PID. These aspects should be clearly highlighted to better delineate the unique contributions of this work.\n2. About \"in-domain over-fitting\" and \"meaningless new intent problems”, detailed evidence, possibly involving case studies, model comparisons or visual representations of the model performance, could strengthen these claims.\n3. The total loss function in stage 1/2 needs to be summarized, including how to balance the losses.\n4. The \"meaningless new intent\" problem is actually the \"trivial solution\" in traditional clustering, and related works should be investigated.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your time and valuable comments. We answer the questions as below.\n\n**Q1: The core innovation appears to lie in the application of label semantic alignment in ICRL and the use of soft prompts in PID. These aspects should be clearly highlighted to better delineate the unique contributions of this work.**\n\n**A1:** We sincerely appreciate your suggestion. We will revise the manuscript as below to highlight our contribution on label semantic alignment and soft prompt in PID:\n\n- **Introduction**: Instead of showing an illustration figure (Figure 1) for the new intent discovery task, we will provide an illustrative figure to emphasize the effect of label semantic alignment.\n- **Methodology**: We will modify our model figure (Figure 2) to highlight the label semantic alignment of our method. For the detailed methodology section, we will specifically update Section 3.3 to emphasize the Semantic Alignment.\n- **Experiment**: We will add data visualization results and error analysis results to showcase the effect of label semantic alignment and soft prompts in PID. More details in Q2&A2.\n\n**Q2: About \"in-domain over-fitting\" and \"meaningless new intent problems\", detailed evidence, possibly involving case studies, model comparisons or visual representations of the model performance, could strengthen these claims.**\n\n**A2:** Thank you for your suggestion. To strength our claims of \"in-domain over-fitting\" and \"meaningless new intents\" problems, we add **(1) an error analysis** and **(2) a data visualization**, which will be incorporated to the Experiment Section.\n\n**(1) an error analysis**\n\nWe report the ratio of unknown intent samples that the model wrongly predicted as known intents and the ratio of utterances belong to known intents but are predicted wrongly. In the table below, we notice that 9.1\\% of unknown intent utterances are erroneously classified as known intents by our strongest baseline USNID. It almost doubles the ratio of known intents predicted wrongly. This observation suggests that the known intent data might overly influence the clustering process, thereby giving rise to the issue of \"in-domain over-fitting.\" Compared with the baseline, the clustering results achieved by the CsePL exihbit a reduced dominance of the known intent data, resulting in a significant decrease in both ratios.\n\n```other\nTable 1. Ratio of intents predicted wrongly.\n```\n\n| **method** | **label unknown, predicted as known** | **label known, predicted error** |\n| ---------- | ------------------------------------- | -------------------------------- |\n| USNID      | 0.091                                 | 0.046                            |\n| CsePL      | 0.065 (**-28.57\\%**)                   | 0.041 (**-10.87\\%**)              |\n\n**(2) a data visualization**\n\nWe display the data representation visualization for USNID and our CsePL. As depicted in the following figure, the prefix \"UK_\" and \"K_\" denote unknown intents (hollow circles) and known intents (solid circles) respectively. The solid stars are the locations of label representations for label semantic alignment. The visualization is shown as below:\n\n![representation_visualization_graph]( https://anonymous.4open.science/r/emnlp2023-3858/representation_visualization.png )\n\nFrom the USNID baseline visualisation (left hand side), we observe that the data points of the unknown intent \"why_verify_identity\" are distorted and dispersed within two known intent clusters: \"unable_to_verify_identity\" and \"verify_my_identity\". This observation indicates that the labeled data exert a dominant influence on unknown intent data, leading to the \"in-domain over-fitting\" problem. It will also cause the meaning of the learned new cluster less meaningful, as it contains instances from three intents.\n\nFor our CsePL method (right hand side), we observe that the label semantics represented as stars will attract their corresponding utterance representations. It helps to leave the unknown intent \"why_verify_identity\" cluster less distorted. Also, since less noise appears in this cluster, the meaning of the cluster would be more clear.\n\n**Q3: The total loss function in stage 1/2 needs to be summarized, including how to balance the losses.**\n\n**A3:** We apologize for the initial omission of the total loss functions and the weight analysis for these losses. The total loss functions are formulated as follows:\n\n***In stage 1:***\n\n$$\nL_{stage1} = \\alpha \\cdot L_{ce} + \\beta \\cdot L_{utt}^{cl} + \\gamma \\cdot L_{utt}^{ucl} + \\lambda \\cdot L_{clu}^{cl} + \\eta \\cdot L_{align}^{cl}\n$$\n\nwhere $L_{utt}^{cl}$, $L_{clu}^{cl}$ and $L_{align}^{cl}$ are defined in Eq(1), Eq(2) and Eq(4) respectively. $L_{ce}$ is the cross-entropy loss and $L_{utt}^{ucl}$ is the unsupervised contrastive learning loss. In reality, the hyper-parameters $\\{\\alpha, \\beta, \\gamma, \\lambda, \\eta\\}$ are set to 1.0. We determined the optimal hyper-parameters by conducting experiments on the development set. The table below illustrates the fluctuation in overall performance with variations in $\\eta$ for the Banking77 dataset with 25\\% known intent rate. We carefully considered a range of values for $\\eta$, ranging from 0.75 to 1.25. We notice that any subtle adjustment to $\\eta$ does not appear to yield significant impacts on the overall performance. This observation also demonstrates the robustness and stability of our method.\n\n```other\nTable 2. Performance of the model with varying loss weights.\n```\n\n| $\\eta$ | **MNI**   | **ARI**   | **ACC**   |\n| -------- | --------- | --------- | --------- |\n| 0.75     | 82.86     | 59.60     | 70.58     |\n| 0.80     | 83.28     | 60.37     | 71.17     |\n| 0.85     | 82.69     | 59.50     | 70.62     |\n| 0.90     | 83.33     | 60.56     | 70.81     |\n| 0.95     | 83.10     | 60.42     | 71.20     |\n| **1.00** | **83.32** | **60.36** | **71.06** |\n| 1.05     | 83.13     | 60.40     | 71.01     |\n| 1.10     | 83.02     | 59.91     | 70.88     |\n| 1.15     | 82.46     | 59.50     | 71.07     |\n| 1.20     | 83.52     | 60.69     | 71.10     |\n| 1.25     | 83.18     | 60.35     | 70.84     |\n\nWe observed similar trends with other hyper-parameters for the $L_{stage1}$, and we plan to supplement this analysis with other hyper-parameters in the revised manuscript.\n\n***In stage 2:*** The loss function in this stage is composed of a single contrastive learning loss as below:\n\n$$\nL_{stage2} = L_{PID}\n$$\n\nwhere $L_{PID}$ is defined in line 372.\n\n**Q4: The \"meaningless new intent\" problem is actually the \"trivial solution\" in traditional clustering, and related works should be investigated.**\n\n**A4:** We are grateful for your suggestion. We will add a new section to explore the \"trivial solution\" problem in clustering and discuss how it connects to and differs from the \"meaningless new intent\" problem. Thanks for providing the related works. We will add discussion about these and other related works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method reiterates the importance of semantics in new intent discovery, overcoming the limitations of previous methods that suffered from over-fitting and data distortion. \n2. Extensive experiments on three public datasets proves the superiority of their method over existing ones but also the capability of suggesting meaningful intent labels.\n3. The author conducted a commendable comparison with the latest state-of-the-art (SOTA) method, USNID, and carried out significant tests. This is highly praiseworthy.", "reasons_to_reject": "1. The novelty of the paper is somewhat vague and needs to be more concise. The primary components seem to integrate effective elements of previous works, including SCL, Aligned, and CLNN, among others. \n2. The authors' claims of solving \"in-domain over-fitting\" and \"meaningless new intent problems\" require more in-depth analysis and visualization.", "questions_to_authors": "1. The core innovation appears to lie in the application of label semantic alignment in ICRL and the use of soft prompts in PID. These aspects should be clearly highlighted to better delineate the unique contributions of this work.\n2. About \"in-domain over-fitting\" and \"meaningless new intent problems”, detailed evidence, possibly involving case studies, model comparisons or visual representations of the model performance, could strengthen these claims.\n3. The total loss function in stage 1/2 needs to be summarized, including how to balance the losses.\n4. The \"meaningless new intent\" problem is actually the \"trivial solution\" in traditional clustering, and related works should be investigated.", "paper_topic_and_main_contributions": "This paper focuses on the discovery of new intent categories from user utterances, emphasizing the importance of utilizing semantic evidence in the process. \nThe primary contributions include the introduction of Cluster Semantic Enhanced Prompt Learning (CsePL), which employs a two-level contrastive learning mechanism combined with label semantic alignment for effective intent cluster representation. The paper highlights how the CsePL method can suggest pertinent intent labels and enables early identification of new intents (when utterances of unknown intents are few).", "excitement": 4, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The \"related work\" section overlaps somewhat with the introduction. It would be advisable to streamline and simplify to avoid redundancy.\n2. Line 009: \"while paid” —> “while paying”"}, "extra_text": {"Missing_References": "1. Caron, Mathilde, et al. \"Deep clustering for unsupervised learning of visual features.\" Proceedings of the European conference on computer vision (ECCV). 2018.\n2. Zhang, Hanlei, et al. \"TEXTOIR: An Integrated and Visualized Platform for Text Open Intent Recognition.\" Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations. 2021."}}, {"review_id": "JdeysMM0HH", "reviewer": "Reviewer_wx3r", "summary": "", "questions": "See weakness section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your recognition of our work, and we will improve our paper further.\n\n**Q1: The proposed method underperforms on NMI metric in comparison with USNID under some settings.**\n\n**A1:** We appreciate your observation. NMI is a metric used to measure the agreement between the ground truth clusters and the predicted clusters. In our experiments, ACC is the most important metric to evaluate the overall model performance. Our method outperforms the strongest baseline USNID by 1%~5% on ACC metric across three datasets. This indicates the effectiveness of the proposed method. \n\nRegarding the NMI metric, we find that due to the dataset characteristics, it is possible for the semantics of some short intent labels to overlap. For example, in the Clinc150 dataset, the semantics of the labels \"order_status\" and \"order\" overlap heavily. This occasionally leads to misclassifications of utterances between these two intents. We will further explore this limitation. One possible way is to enhance the short labels with domain knowledge such as using LLMs to rewrite the label with more details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This papers tackles and important and relevant problem of discovering new intents from the existing ones. In order the enhance the capabilities of a dialogue system, new intent discovery is necessary.\n\n2. A good number of baselines are studied and evaluated in order to establish the superiority of the proposed method.\n\n3. Contrastive learning has been employed for capturing the semantics.\n\n4. Several baselines has been employed for comparison.\n\n5. Use of label semantic alignment is interesting.\n\n6. Significant improvements are reported with insightful discussion.", "reasons_to_reject": "1. The proposed method underperforms on NMI metric in comparison with USNID under some settings.", "questions_to_authors": "See weakness section.", "paper_topic_and_main_contributions": "This work is related with problem of discovering new intent categories from user utterances. Building upon the observation that existing approaches emphasize too much on relation among utterances and pay less attention to employing the semantics, this work presents Cluster Semantic Enhanced Prompt Learning (CsePL) for discovering new intents. The proposed approach leverages two-level contrastive learning with label semantic alignment to learn meaningful representations of intent clusters which are then utilized as soft prompt initializations for discriminating new intents, reducing the dominance of existing intents. Experimentations on three datasets (BANKING77, CLINC150 and StackOverflow) demonstrate that the proposed method achieves superior results compared to baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}, {"review_id": "KkDPAgDBBc", "reviewer": "Reviewer_ASNm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely thank you for your time and detailed comments on this work. Here are our answers to the questions.\n\n**Q1: The motivation of this paper is not clear. The authors claim that previous studies paid less attention to the usage of semantics and the limitations described from line 71 to line 76 are vague and less intuitive. It's also unclear the relation between the proposed methods and these limitations.**\n\n**A1:** We apologize for not making the motivation clear enough. To concisely describe our motivation, we add a data visualization for the strongest baseline USNID and our method.\n\nIn the figure, the prefix \"UK_\" and \"K_\" refer to unknown intents (hollow circles) and known intents (solid circles) respectively. The solid stars indicate the positions of label representations for label semantic alignment.\n\n![representation_visualization_graph]( https://anonymous.4open.science/r/emnlp2023-3858/representation_visualization.png )\n\n\nThe limitations of existing works include two main aspects: \"in-domain over-fitting\" problem and \"meaningless new intents\" problem. For example, as illustrated in the data visualization of the strongest baseline USNID (left hand side), some data points of the unknown intent \"why_verify_identity\" have become distorted and scattered within other two known intent clusters: \"unable_to_verify_identity\" and \"verify_my_identity\". This suggests that the known intent data can significantly impact the distribution of unknown intent data, leading to the emergence of the \"in-domain over-fitting\" issue. It will also result in the learned new cluster having less meaningful semantics, as it comprises data points from three different intents.\n\nTo alleviate these limitations, we proposed to train the model using label semantic alignment and the PID process. As the label semantics is more stable, we can observe from the data visualization of the CsePL (right hand side) that the label representations (solid stars) can attract their associated data points, forming compact and well-defined cluster regions. Hence, the \"in-domain over-fitting\" problem can be mitigated. Also, the meaning of the newly learned cluster would be more clear due to fewer noisy data points in this cluster.\n\nWe will modify our illustration figure (Figure 1) in Introduction, modify the Method section accordingly and add these experiments to highlight our motivation.\n\n**Q2: Method in this paper contains multiple components. It's hard to determine the effectiveness of these proposed parts.**\n\n**A2:** Most of the components used in our method come from existing baselines, which have been already validated as effective, such as \"utterance-level contrastive loss\", \"Cluster level contrastive loss\" etc. To alleviate the \"in-domain over-fitting\" and \"meaningless new intents\" problems, we specifically propose two primary parts: label semantic alignment and the PID process. The label semantic alignment aims to align intent cluster representations to their corresponding label in the semantic space. For example, removing this part from our model can cause a performance drop in ACC from 71.06% to 68.75% in the Banking77 dataset with 25% known intent rate. The goal of the PID process is to provide the model with a comprehensive set of all intent options, assisting the model to match input utterances to their corresponding intents. For example, without the PID process, the ACC performance of the model drops from 71.06% to 61.38% under the setting of Banking77 dataset with 25% known intent rate. Detailed ablation results can be found in Table 4 in our manuscript.\n\n**Q3: It's unclear what the augmented version is described in line 283.**\n\n**A3:** Similar to [1], we use dropout to construct augmented examples for contrastive learning. Here, the augmented version is the dropout-augmented utterance representation of the input utterance.\n\\\n[1]. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. ArXiv, abs/2104.08821.\n\n**Q4: The effectiveness of the sequence of soft prompts is unclear. My understanding is that all cluster representations are inserted into all data instances, so it's identical to all examples. Why can this help distinguish different clusters?**\n\n**A4:** \nThanks for the insightful comment. Within our method, intent cluster representations are trained to align their corresponding label representations via label semantic alignment. Thus, they can reflect the global intent distribution in the feature space. Utilizing them as soft prompt initializations provides the model with a comprehensive array of all intent options, effectively guiding the model to match input utterances to their respective intents.\n\nOn the contrary, we randomly initialize the soft prompts and identically integrate them with all input utterances to conduct additional experiments. The experimental results are reported in Table 1. We can notice that the model has a performance decline on all three datasets without initializing the soft prompts with the aligned intent cluster representations. We will include these findings in the revised version of the manuscript.\n\n```\nTable 1. Comparison of model performance using various soft prompt initializations.\n```\n| Banking77         |              |         |        |         |\n| :-----:           | :-----:      | :-----: | :----: | :----:  |\n| Known Intent Rate | Method       | MNI     | ARI    | ACC     |\n| 0.25              | CsePL        | 83.32   | 60.36  | 71.06   |\n| 0.25              | CsePL-random | 82.13   | 58.22  | 69.45   |\n| 0.5               | CsePL        | 85.65   | 66.66  | 76.94   |\n| 0.5               | CsePL-random | 85.45   | 65.50  | 75.91   |\n| 0.75              | CsePL        | 87.70   | 71.36  | 81.93   |\n| 0.75              | CsePL-random | 86.80   | 69.57  | 79.97   |\n\n| Clinc150          |              |         |        |         |\n| :-----:           | :-----:      | :-----: | :----: | :----:  |\n| Known Intent Rate | Method       | MNI     | ARI    | ACC     |\n| 0.25              | CsePL        | 94.07   | 79.65  | 86.16   |\n| 0.25              | CsePL-random | 93.36   | 77.23  | 84.74   |\n| 0.5               | CsePL        | 95.09   | 83.14  | 88.66   |\n| 0.5               | CsePL-random | 94.63   | 81.41  | 87.85   |\n| 0.75              | CsePL        | 96.58   | 88.88  | 93.46   |\n| 0.75              | CsePL-random | 95.87   | 85.98  | 91.02   |\n\n| StackOverflow     |              |         |        |         |\n| :-----:           | :-----:      | :-----: | :----: | :----:  |\n| Known Intent Rate | Method       | MNI     | ARI    | ACC     |\n| 0.25              | CsePL        | 74.88   | 64.92  | 79.47   |\n| 0.25              | CsePL-random | 74.27   | 63.57  | 77.25   |\n| 0.5               | CsePL        | 80.28   | 71.99  | 85.68   |\n| 0.5               | CsePL-random | 78.60   | 70.31  | 84.20   |\n| 0.75              | CsePL        | 82.81   | 75.99  | 87.80   |\n| 0.75              | CsePL-random | 79.96   | 72.16  | 85.55   |\n\n**Q5: In PID, why do you use a pre-trained backbone to get token representations? The backbone is trained with the manual prompt but in the PID stage, utterances do not use prompts. I think this inconsistency would cause suboptimal representations for tokens.**\n\n**A5:** Thank you for your comment. The primary goal of our method is to first learn a global distribution of all intents, and then refine the model to better associate input utterances with their corresponding intents. Therefore, we believe that utilizing the backbone pre-trained in stage 1 as the feature extractor in PID would not cause the inconsistent problem. We conduct experiments on employing a new BERT-model as the feature extractor in PID, the model cluster performance drops dramatically. For example, under the setting of Banking77 dataset with 25% known intent rate, the ACC metric decreased from 71.06% to 41.4%. \n\nIt should be noted that we intentionally employ different prompting methods in stage 1 and stage 2. In the first stage, we train the model with a manually constructed prompt to learn aligned intent cluster representations. In the second stage, we initialize the soft prompts with all the learned intent representations, integrating them with input utterances to further improve the model's performance. It provides the model with a comprehensive array of all intent options, effectively guiding the model to match input utterances to their respective intents.\n\nTo verify whether different prompting methods in two stages cause the suboptimal issue, we conduct experiments by employing identical manual prompt in both two stages. As reported in Table 2, when using the identical manual prompt in both two stages, the model performance declines on all three datasets compared with the CsePL. We will include this analysis in the revised version to further support our arguments.\n\n```\nTable 2. Comparison of model performance using soft prompts and identical manual prompt.\n```\n| Banking77         |              |         |        |         |\n| :-----:           | :-----:      | :-----: | :----: | :----:  |\n| Known Intent Rate | Method       | MNI     | ARI    | ACC     |\n| 0.25              | CsePL        | 83.32   | 60.36  | 71.06   |\n| 0.25              | CsePL-manual | 81.24   | 56.89  | 69.11   |\n| 0.5               | CsePL        | 85.65   | 66.66  | 76.94   |\n| 0.5               | CsePL-manual | 84.26   | 63.76  | 75.47   |\n| 0.75              | CsePL        | 87.70   | 71.36  | 81.93   |\n| 0.75              | CsePL-manual | 86.49   | 68.94  | 79.76   |\n\n| Clinc150          |              |         |        |         |\n| :-----:           | :-----:      | :-----: | :----: | :----:  |\n| Known Intent Rate | Method       | MNI     | ARI    | ACC     |\n| 0.25              | CsePL        | 94.07   | 79.65  | 86.16   |\n| 0.25              | CsePL-manual | 93.36   | 77.06  | 83.71   |\n| 0.5               | CsePL        | 95.09   | 83.14  | 88.66   |\n| 0.5               | CsePL-manual | 94.32   | 80.07  | 86.98   |\n| 0.75              | CsePL        | 96.58   | 88.88  | 93.46   |\n| 0.75              | CsePL-manual | 95.61   | 85.26  | 90.54   |\n\n| StackOverflow     |              |         |        |         |\n| :-----:           | :-----:      | :-----: | :----: | :----:  |\n| Known Intent Rate | Method       | MNI     | ARI    | ACC     |\n| 0.25              | CsePL        | 74.88   | 64.92  | 79.47   |\n| 0.25              | CsePL-manual | 73.43   | 61.48  | 76.30   |\n| 0.5               | CsePL        | 80.28   | 71.99  | 85.68   |\n| 0.5               | CsePL-manual | 77.60   | 69.03  | 83.95   |\n| 0.75              | CsePL        | 82.81   | 75.99  | 87.80   |\n| 0.75              | CsePL-manual | 80.39   | 71.82  | 84.75   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Authors conducted extensive experiments to show the effectiveness of the proposed methods.\n2. The improvements compared to baselines are not trivial.\n3. The paper presents a detailed introduction to methods that makes readers easy to follow their ideas.", "reasons_to_reject": "1. The motivation of this paper is not clear. The authors claim that previous studies paid less attention to the usage of semantics and the limitations described from line 71 to line 76 are vague and less intuitive. It's also unclear the relation between the proposed methods and these limitations.\n2. Method in this paper contains multiple components. It's hard to determine the effectiveness of these proposed parts.\n3. It's unclear what the augmented version is described in line 283.\n4. The effectiveness of the sequence of soft prompts is unclear. My understanding is that all cluster representations are inserted into all data instances, so it's identical to all examples. Why can this help distinguish different clusters?\n5. In PID, why do you use a pre-trained backbone to get token representations? The backbone is trained with the manual prompt but in the PID stage, utterances do not use prompts. I think this inconsistency would cause suboptimal representations for tokens.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a semi-supervised method for new intent discovery in user utterances.\nThe authors of this paper recognize that previous works suffer from in-domain over-fitting and generate low-quality new intent clusters.\nHence, the paper present CsePL, which leverages two-level contrastive learning with label semantic alignment to learn representations of intent clusters.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work is concerned with the challenge of discovering new intent categories from user utterances, and introduces Cluster Semantic Enhanced Prompt Learning (CsePL) as a means to discover new intents. The proposed approach employs a two-level contrastive learning framework, incorporating label semantic alignment to acquire meaningful representations of intent clusters. These representations are subsequently employed as soft prompt initializations for distinguishing new intents, thereby reducing the influence of pre-existing intent categories. Experimental evaluations demonstrate that the proposed method outperforms baseline approaches. While the novelty of the paper may require further clarification and conciseness, the primary components of the proposed approach appear to integrate elements from previous works.
Decision: Accept-Findings