A Diffusion Weighted Graph Framework for New Intent Discovery
Authors: Wenkai Shi
Wenbin An
Feng Tian
Qinghua Zheng
QianYing Wang
Ping Chen
Venue: EMNLP 2023 Main (2023)
Keywords: New Intent Discovery
Contrastive Learning
Graph Structure Learning
TL;DR: We propose a novel Diffusion Weighted Graph Framework to discover new user intents from unlabeled dialogues.

Abstract: New Intent Discovery (NID) aims to recognize both new and known intents from unlabeled data with the aid of limited labeled data containing only known intents. Without considering structure relationships between samples, previous methods generate noisy supervisory signals which cannot strike a balance between quantity and quality, hindering the formation of new intent clusters and effective transfer of the pre-training knowledge. To mitigate this limitation, we propose a novel $\textit{Diffusion Weighted Graph Framework}$ (DWGF) to capture both semantic similarities and structure relationships inherent in data, enabling more sufficient and reliable supervisory signals. 
Specifically, for each sample, we diffuse neighborhood relationships along semantic paths guided by the nearest neighbors for multiple hops to characterize its local structure discriminately. Then, we sample its positive keys and weigh them based on semantic similarities and local structures for contrastive learning. During inference, we further propose $\textit{Graph Smoothing Filter}$ (GSF) to explicitly utilize the structure relationships to filter high-frequency noise embodied in semantically ambiguous samples on the cluster boundary. Extensive experiments show that our method outperforms state-of-the-art models on all evaluation metrics across multiple benchmark datasets. Code and data will be made public.

A Diffusion Weighted Graph Framework for New Intent Discovery
Wenkai Shi', Wenbin An', Feng Tian"*, Yan Chen', Qinghua Zheng
QianYing Wang", Ping Chen
School of Automation Science and Engineering, Xi'an an Jiaotong University
2 School of Computer Science and Technology, MOEKLNNS Lab, Xi an Jiaotong University
3 Lenovo Research Department of Engineering, University of Massachusetts Boston com, (fengtian, chenyan)@maill xutu. edu. wenbinanestu. xjtu. edu. cn, wanggya@lenovo. com, ping. chen@umb. edu
shiyibai178@
Abstract
New Intent Discovery (NID) aims to recognize both new and known intents from unlabeled data with the aid of limited labeled data con- taining only known intents. Without consid- cring structure relationships between samples, previous methods generate noisy supervisory signals which cannot strike a balance between quantity and quality, hindering the formation of new intent clusters and effective transfer of the pre-training knowledge. To mitigate this limi- tation, we propose a novel Diffusion Weighted Graph Framework (DWGF) to capture both se- mantic similarities and structure relationships inherent in data, enabling more sufficient and reliable supervisory signals. Specifically, for each sample, we diffuse neighborhood relation- ships along semantic paths guided by the near- est neighbors for multiple hops to characterize local structure discriminately. Then, we sam- ple its positive keys and weigh them based on semantic similarities and local structures for contrastive learning. During inference, we fur- ther propose Craph Smoothing Filter (GSF) to explicitly utilize the structure relationships filter high-frequency noise embodied in mantically ambiguous samples on the cluster boundary. Extensive experiments show that our method outperforms mate-of-theeart models all evaluation metrics across multiple bench- mark datasets. Code and data are available at https:/ / / thub. www.cooaa/teeeeeee
Figure 1: Iilustration the transformation of supervi- sory signal generation method. Bottom Left: generat- ing supervisory signals indiscriminately along all direc- tions the hypersphere, which is sensitive to threshold changing. Top: an example of selecting samples with semantic paths. Bottom Right: generating supervisory signals directionally with structure relationships com- posed of multiple semantic paths in relaxed feature hypersphere.
New Intent Discovery (NID), which aims to dis- cover new intents from unlabeled data automati- cally, has attracted much attention recently.
Most existing NID methods (Lin et al., 2020; Zhang et al., 2021; Wei al., 2022; Zhang et al., 2022; An et al., 2023) adopt a two-stage training strategy: pre-training on labeled data, then learning clustering-friendee representation with pscudo su- pervisory signals. However, previous methods only rely semantic similarities generate supervi- sory signals based the assumption that samples within the feature hypersphere belong to the same category as the hypersphere anchor, e.g. cluster centroids (Zhang et al., 2021), class prototypes (An et al., 2022b), or query samples (Zhang et al., 2022).
1 Introduction
Even though current machine learning methods have achieved superior performance on many NLP tasks, they often fail to meet application require- ments in an open-world environment. For instance, general intent classification models trained on pre- defined intents cannot recognize new intents from unlabeled dialogues, which is clear obstacle for real-world applications. Therefore, research on
Even though these methods can learn some dis- criminative features, they still face limitations generating both adequate and reliable supervisory
"Corresponding Authors.

---

signals, which we call the Quantity and Quality Dilemma. Specifically, as shown in Fig.1 Bot- tom Left, these methods rely on a fixed threshold to determine the search radius of the hypersphere. Shrinking the threshold (blue solid line) helps re- trieve more accurate positive keys, but it loses infor- mation from positive keys out of the hypersphere. resulting in low recall. However, simply relaxing the threshold (red dashed line) will introduce much noise and lead to low accuracy.
adopt the idea of Xie al. (2016) to help learn Blustering-rriende representations from the global view through self-training.
During the inference stage, in order to filter high- frequency noise embodied in the semantically am- biguous samples on the cluster boundary, we pro- pose a novel inference improvement Graph Smooth- ing Filter (GSF), which utilizes normalized graph Laplacian to aggregate neighborhood information revealed by structure relationships of testing sam- ples. Smoothed testing features help to obtain bet- ter clustering results.
Quantity and Quality Dilemma is caused by the fact that the previous methods searched positive keys indiscriminately along all directions of the hypersphere with fixed search radius. In order to selectively sample both adequate and reliable positive keys to ensure the formation of new intent clusters, we propose to model and utilize structure relationships inherent in data, which reflect the semantic correlations between samples from the perspective of connectivity. As shown in Fig. Top. for each sample, we first initialize its k-nearest neighbors with a tightened threshold. Then we connect any two samples if they have at least one shared neighbor since the semantics of the shared neighbor are highly correlated with the samples on both sides. According to this rule, we identify two samples (with brown borders in Fig.1 Top) that can be used as bridges and diffuse the anchor along them to search positive keys near the boundary of the hypersphere, forming the final semantic path. the case of the same semantic similarity, we additionally require the positive keys to appear on the semantic paths diffused from the anchor.
Our main contributions can be summarized as follows:
We propose Diffusion Weighted Graph Framework (DWGF) for NID, which can cap- ture both semantic similarities and structure relationships inherent in data generate ade- quate and reliable supervisory signals.
We improve inference through Graph Smooth- ing Filter (GSF), which exploits structure re- lationships to correct semantically ambiguous samples explicitly.
We conduct extensive experiments on multiple benchmark datasets to verify the effectiveness.
Related Work
2.1 New Intent Discovery
Semi-supervised NID aims to discover novel in- tents by utilizing the prior knowledge of known intents. First, it is assumed that the labeled data and the unlabeled data are disjoint in terms of cate- gories. To tackle the NID challenge under this set- ting, Mou et al. (2022a) proposed a unified neigh- bor contrastive learning framework to bridge the transfer gap, while Mou al. (2022b) suggested one-stage framework simultancously classify novel and known intent classes. However, more common setting in practice is that the unlabeled data are mixed with both known and new intents. Compared to the previous setting, the latter is more challenging because the above methods have dif- ficulty distinguishing mixture of two kinds of intents and are prone to overfit the known intent classes. To this end, Lin al. (2020) conducted pair-wise similarity prediction to discover novel in- tents, and Zhang et al. (2021) used aligned pseudo- labels to help the model learn clustering-friend representations. Recently, contrastive learning has
In this paper, we propose a novel Diffusion Weighted Graph Framework to model and utilize structure relationships. Specifically, from any an- chor, we diffuse neighborhood relationships along the nearest nrighbor-gudded semantic paths for mul- tiple hops to construct the final DWG. As shown in Fig.1 Bottom Right, then we sample positive keys along the semantic paths (arrow lines) in DWG within the relaxed feature hypersphere. Moreover, sampled keys are assigned to different contrastive weights according to their frequency of being sam- pled on different semantic paths, where keys that are diffuuee repeatedly from different outsets will accumulate larger values and vice versa. We con- duct contrastive learning with sampled positive keys and corresponding weights in the embedding space. Apart from considering the sample-sample structure relationships from the local view, we

---

become an important part of NID research. For example, An al. (2022a) proposed hierarchical weighted self-contrasting to better control intra- class and inter-class distance. Wei et al. (2022) exploited supervised contrastive learning (Khosla et al., 2020) to pull samples with the same pseudo- label closer. An et al. (2022b) achieved trade-off between generality and discriminability in NID by contrasting samples and corresponding class prototypes. Zhang et al. (2022) acquired compact clusters with the method of neighbor contrastive learning. However, these methods don't fully ex- plore the structure relationships inherent in data, causing the generated supervisory signals to fall into a Quantity and Quality Dilemma.
on the testing set D {wily) E 3 U J.").
3.2 Approach Overview
Fig.2 illustrates the overall architecture of our pro- posed Diffusion Weighted Graph Framework. The framework includes two parts: training with Diffu- sion Weighted Graph (DWG) and inference with Graph Smoothing Filter (GSF). Firstly, we con- duct pre- tre-training detailed Sec.3.3. Secondly, as shown in Fig.2's I, we extract intent representations to simultancously conduct self-training from the global view and contrastive learning with DWG from the local view. More training details are pro- vided in Sec.3.4. Finally, shown in Fig.2's II, we construct GSF to smooth testing features and adopt KMeans clustering to complete the inference. More inference details are provided in Scc.3.5.
2.2 Contrastive Learning
Contrastive learning pulls similar samples closer, pushes dissimilar samples far away, and has gained promising results in computer vision (Chen et al., 2020; He et al., 2020; Khosla et al., 2020) and nat- ural language processing (Gao et al., 2021; Kim al., 2021). Inspired by the success of contrastive learning, large number of works extend the defi- nition of positive and negative keys in it to adapt to more research fields. For example, Li et al. (2021) conducted cluster-level contrastive learning in the column space of logits, Li et al. (2020) proposed to use cluster centroids as positive keys in contrastive learning, and Dwibedi et al. (2021) treated nearest neighbors in feature space as positive keys. These works all help model learn cluster-friendly rep- resentations that benefit NID. However, they solely rely on semantic similarities to search positive keys, which inevitably generate noisy pseudo supervisory signals.
In summary, combined with structure relation- ships, proposed DWGF can 1) break through the limitation of tightened threshold and achieve higher sampling accuracy and recall simultane- ously; 2) suppress sampling noise while retaining rich semantics through soft weighting: 3) consider the local comple-sampl supervision and the global sample-cluster supervision simultancously; 4) fil- ter high-freee frequency noise embodied in semantically ambiguous samples on the cluster boundary during inference.
3.3 Model Pre-training
We use BERT (Devlin et al., 2019) to encode input sentences and take all token embeddings from the last hidden layer. Then apply average pooling to acquire the final intent representations.
2j = mean-pooingg(ERRT
(1)
3 Methods
where and refer to i-th input sentence and cor- responding representation. Motivated by (Zhang et al., 2021), use Cross-Entropy loss on labeled data acquire prior knowledge from known in- tents. Furthermore, we follow (Zhang al., 2022) to use Masked Language Modeling (MLM) loss on all training data to learn domain-specific semantics. We pre-train the model with the above two kinds of loss simultaneously:
3.1 Problem Formulation
Traditional intent classification task follows closed-world setting, i.e., the model is only devel- oped based on labeled dataset D' = [ yo Yau E y*3 where y* refers to the set of known intent classes. New Intent Discovery follows an open- world setting, which aims to recognize all intents with the aid of limited labeled known intent data and unlabeled data containing all classes. There- fore, in addition to the above D'. D {2ily) e yk U ym) from both known intents y and new in- tents n will be utilized to train the model together. Finally, the model performance will be evaluated
E Loo + Lyntm (D")
where D and D are labeled and unlabeled dataset, respectively.

---

oooooo
Cluster
Self-training Loss
shappen
Centroids
Pooling Layer
KL loss
Embedding
Encoder
Layer
Weighting Stralegy
Sampling Strategy
DWG Contrastive Loss
(wngredd)
a0t4. anilleed) distc
Ae E N
Negative samppees
BERT
Selected podiiv seppla
Smooth Hiltering
Chastering
Selected begative sumples
Threshald
Diflinio deection
EE MXD
2 E 8
Figure 2: Overall architecture of our proposed Diffusion Weighted Graph Framework (DWGF). illustration of the model training. II: illustration of the inference. III: illustration of self-training, IV: illustration of contrastive learning based on DWG.
3.4 Representation Learning with DWG
where r refers to the diffusion rounds, 0 is the magnitude of diffusion, which is set it to 1 for simplicity. Combined with relaxed semantic sim- ilarity threshold y, further filter keys in DWG with similarity below the threshold, i.e. A. 0 2 2j <
After pre-training, we extract all training samples' 12- 12-normalized intent representations and initialize the instance graph Ao with monomial kernel (Is- et al., 2017) the similarity metric.
Weighting Strategy. Apart from semantic simi- larity, DWG Á also reflects the confidence of sam pled keys to the anchor from the perspective of the frequency that the key is repeatedly diffused. How- ever, the numerical scale of each in Á varies significantly due the different diffusion process. To ensure consistency subsequent contrastive learning, we normalize them to [o, 1] interval with the degree D, = E, Mi of a; and modulation factor X.
max( (ff Zj 0)P, (4jAjEN((z)
49
0,
otherwise
(3)
Here N2)2)) saves the indices of k-nearest neigh- bors of 24, and p controls the weights of similarity. We set p for simplicity and generality.
Different from Zhang et al. (2022), we reduce the neighborhood size and retain the similarities with anchor instead of 0-1 assignment. We aim to model the structure relationships through KNN rather than directly sample positive keys. With the initial high- cogghhconieeece neighbors, we perform subsequent diffusion to complete the DWG and implement sampling and weighting.
Wij min(1, Au) D
A. 0
(5)
Aij
As shown in Fig.2's IV, such soft weighting strategy has two advantages. Firstly, the blue line shows that homogeneous samples will prevail in contrastive learning because of the cumulative influ- ence from the multiple diffusion of different keys. Secondly, the red line indicates that even hetero- geneous samples are selected, they will be assigned smaller weights because of infrequent sampling and lower similarity to the diffusion outset.
Sampling Strategy. As shown in Fig.2's IV, the smaller neighborhood size first ensures that se- mantically unrelated heterogeneous samples are not used as outsets for diffusion. Then start with the anchor and diffuse its neighborhood relationships along semantic paths guided by tigh-confidence neighbors, which would be included as new an- chors for the next diffusion. We define DWG as the accumulation of multiple self-mmuttplicationn of Ap.
To fully utilize sampled positive keys, we main- tain momentum encoder and dynamic queue following (He et al., 2020), which can help the model benefit from contrasting large amounts
Å =
0¹
A
(4)

---

consistent keys at once. At the end of each itera- tion, the dynamic queue will be updated by adding current samples and removing the oldest samples. We denote the final DWG contrastive learning loss as:
Dataset
129
[
D
DD4]
DD'
BANKING
58
19
673
8330
3080
StackOverflow
15
5
1350
16650
1000
CLINC
113
37
1344
16656
2250
Table Statistics of datasets. [3**) [D"), [D', |D"]
N
e chiihg/t hgghT Waj
and [D'] represent the number of known categories, categories, labeled data, unlabeled data and testing data.
Liocol
log
BN
18 2j==1 higlt
(6) where B and N refer to the size of the batch and dy- namic queue, respectively. h; is the embedding of Tt. h is the embedding of 2j stored in the dynamic queue. T is the temperature hyperparameter.
encoder and utilized to improve inference in explicit way.
Specifically, we extract features of the testing set and construct corresponding instance graph A, as Sec.3.4. Then, with the renormalization trick Å = 1 A, (Kipf and Welling, 2016), we compute the symmetric normalized graph Laplacian:
Apart from considering the sample-sample neighborhood structure from the local view, we adopt the idea of Xie al. (2016) to add the sample- cluster supervision from the global view. Firstly, we initialize the cluster centroids with the KMeans result on pre-training features and use t-distribution to estimate the distance between intent representa- tions 2; and cluster centroid jı::
Lasym --1
(11)
where D.c - E, A. and L - À are degree matrix and Laplacian matrix corresponding to Å. According to (Wang et al., 2019), we denote the Graph Smoothing Filter (GSF) as:
Qik (1+ ||
1²ll /n) 9t2
(1+ 2 1| /n) n=1
H = (1 0.5 Lsgm)*
(12)
Here we set 7) = for simplicity. Then we generate the auxiliary distribution with both instance-wise and wiuster-wise normalization:
where refers to the number stacking layers. We apply the filter to the extracted features and acquire the smoothed feature matrix Ž = Z for KMeans clustering. To the best of our knowledge, this is the first attempt to apply the structure-based filter to inference in NID.
PuelI
Pa: Ex /
(8)
Qik are soft cluster frequencies. Finally, the cluster assignment distribution Q is optimized by minimizing KL-divergence with the corresponding auxiliary distribution P:
where f& = Qie
Experiments
4.1 Datasets
We evaluate our method on three benchmark datasets. BANKING (Casanueva al., 2020) is fine-grained intent classification dataset. Stack- Overflow (Xu al., 2015) is a question classifica- tion dataset collected from technical queries online. CLINC released by (Larson et al., 2019) a multi- domain intent classification dataset. More details of these datasets are summarized in Table 1.
1
y Palog
1
Pik Qik
Cgloball =
(9)
B
1=1
Overall, the training objective of our model can be formulated as follows:
C = Liocal + Eglobal
(10)
4.2 Comparison Methods
where is the relative weight of self- self-training loss.
We compare our method with various baselines and state-of-the-un art methods.
3.5 Inference with GSF
During the training phase, model and utilize the structure relationships help the encoder learn representations that are aware of the local struc- tures. Therefore, the structure relationships inher- ent in the testing set can be captured by the trained
Unsupervised Methods. GloVe-KM: KMeans with GloVe embeddings (Pennington al., 2014);
AAE-KM: KMeans with embeddings learned by stacked auto-encoder; DEC: Deep Embedded Clus- tering (Xie et al., 2016); DCN: Deep Clustering

---

BANKING
StackOverflow
CLINC
Method
NMI
ARI
ACC
NMI
ARI
ACC
NMI
ARI
ACC
DeepCluster
39.72
7.78
18.93
17.52
3.09
18.64
53.82
12.27
28.46
GloVe-KM
48.75
12.74
27.92
21.79
4.54
24.26
54.57
12.18
29.55
SAE-KM
60.12
24.00
37.38
48.72
23.36
37.16
73.13
29.95
46.75
DEC
62.92
25.68
39.35
61.32
21.17
57.09
74.83
27.46
46.89
DCN
62.94
25.69
39.36
61.34
24.98
57.09
75.66
31.15
49.29
DTC
74.51
44.57
57.34
67.02
55.14
71.14
90.54
65.02
74.15
CDAC+
71.76
40.68
53.36
76.68
43.97
75.34
86.65
54.33
69.89
DAC
79.56
53.64
64.90
75.24
60.09
78.74
93.89
79.75
86.49
DSSCC
81.24
58.09
69.82
77.08
68.67
82.65
93.87
81.09
87.91
PTJN
81.69
59.20
71.77
75.43
61.90
74.18
94.41
81.07
87.35
DPN
82.58
61.21
72.96
78.39
68.59
84.23
95.11
86.72
89.06
DCSC
84.65
64.55
75.18
95.28
84.41
89.70
CLNN
85.77
67.6
76.82
81.62
74.74
86.6
96.08
86.97
91.24
Ours
86.41
68.16
79.38
81.73
75.30
87.6
96.89
90.05
94.49
Table 2: Evaluation (%) on testing sets. Average results over runs are reported. set the known class ratio Dil//D% noal to 0.75, and the labeled ratio of known intent classes to 0.1 to conduct experiments.
80
Methods
NMI
ARI
ACC
weo ditfnsionte-O) diCluiiu(e-1)
Ours
86.41
68.16
79.38
diéfusiont- 2)
GSF 85.82
66.96
78.21
-Self-training
85.78
66.77
77.73
Clustering Ace
78
-DWG
53.89
19.30
33.05
Table 3: Ablation study on the effectiveness of different components. '.' means that we remove the correspond- ing component.
76
75
Network (Yang et al., 2017); DeepCluster: Deep Clustering (Caron et al., 2018).
74
0
25
100
Neighborhood Size
Semi-supervised Methods. DTC: Deep Transfer Clustering (Han et al., 2019); CDAC+: Constrained Adaptive Clustering (Lin et al., 2020); DAC: Deep Aligned Clustering (Zhang al., 2021); DSSCC: Deep Semi-Supervised Contrastive Clustering (Ku- mar et al., 2022); DCSC: Deep Contrastive Semi- supervised Clustering (Wei al., 2022); DPN: De- coupled Prototypical Network (An et al., 2022b); CLNN: Contrastive Learning with Nearest Neigh- bors (Zhang et al., 2022); PTJN: Robust Pseudo Label Training and Source Domain Joint-training Network (An al., 2023). Notably, for fair com- parison, the external dataset is not used in CLNN as other methods.
Figure 3: Evaluation (%) under non-diffusion and diffusion-bret conditions.
racy (ACC) based on the Hungarian algorithm.
4.4 Implementation Details
We use the pre-trained BERT model (bert-bsae- uncased) as our backbone and AdamW optimizer with 0.01 weight decay and 1.0 gradient clipping for parameter update. During pre-training, we set the learning rate to 55 and adopt the early- stopping strategy with patience of 20 epochs. Dur- ing representation learning with DWG, set the first-order neighborhood size/number of diffusion rounds 15/r for BANKING and CLINC, and & 50/r for StackOverflow construct DWG, which is updated per 50 epochs. Relaxed threshold 7, modulation factor X, loss weight a
4.3 Evaluation Metrics
We adopt three metrics for evaluating clustering results: Normalized Mutual Information (NMI), Adjusted Rand Index (ART), and clustering Accu-

---

0.50
3.74
Sumpligg Acturacy
3.55 0/9
Samplng
Samplis
3.5
620
class2 DWG
wio
DWG
w/o
(a) Top-50 accuracy
Top-50 accuracy at T =
(c) Top-100 accuracy at 0
Figure 4: Accuracy comparison of sampling w/ DWG and w/o DWG, Here, class 1-4 represent {card about to expire, apple pay or google pay, terminate account and verify source of funds], respectively.
training and contrastive learning based on DWG in sequence to conduct experiments BANKING again. As shown in Table 3, removing them impairs model performance consistently, indicating GSF ally alleviates the negative effect of tiguency noise, and both local and global supervision pro- vided by Eq. 10 benefit new intent discovery, espe- cially DWG contrastive learning.
190g ulf. neight
0.4
Average welght
5.2 Analysis of DWG
Training Epochs
To validate the effectiveness of DWG contrastive learning, we compare the model performance under diffusion and non-diffusion conditions. Moreover, we also explore the sensitivity of our method to hyperparameter changes, including the first-order neighborhood size k and the number of diffusion rounds r. As shown in Fig.3, DWG generally helps the model outperform the original non-diffusion method adopted by (Zhang et al., 2022) and dra- matically reduces the search scope of indicating our method is both effective and robust.
Figure 5: Average weight changing of sampled keys.
and temperature 7 are set to 0.3, 1.1, 0.3 and 0.2, respectively. We adopt the data augmentation of random token replacement as (Zhang et al., 2022). We set the learning rate to le-5 and train until con- vergence without early-stopping. During inference with GSF, we set the number of stacking layers t 2 and neighborhood size to one-third of the av- erage size of the testing for each class. All the experiments are conducted on single RTX-3090 and averaged over runs.
To further illustrate the positive effect brought by structure relationships, we separately analyze the sampling strategy and weighting strategy based on DWG.
4.5
Main Results
Sampling Strategy. Taking the BANKING datasct as an example, we choose representative classes from according to the sampling difficulty, Fig.4a 4a and Fig.4b show the Top-50 positive keys sampling accuracy at epoch 50 and 0, respectively, indicating the connectivity required by structure relationships can effectively improve sampling ac- curacy, especially 1) on categories with high sam- pling difficulty; 2) at the beginning of the training that samples haven't form compact clusters. Fig.4c shows the Top-100 sampling accuracy at epoch 0. which indicates our method more robust to trieve positive keys selectively when relaxing the threshold.
The main results are shown Table Our method outperforms various comparison methods consis- tently and achieves clustering accuracy improve- ments of 2.56%. 0.90% and 3.25% on three bench- mark datasets compared with previous state-of-the- art CLNN, respectively. It demonstrates the effec- tiveness of our method discover new intents with limited known intent data.
5 Discussion
5.1 Ablation Study
To investigate the contributions of different com- ponents our method, we remove GSF, self-

---

DAC
DCSC
85.0
30.
30
NMI
ART
77
65.0
70-
910
Known Class Ratio
Known Class Ratio
Known Class Ratio
Figure 6: Influence of known class ratio on the CLINC dataset.
ARI
ACC
SC
w/o GSF
88.48
92.84
0.64
k=5
88.57
92.89
0.70
[=1
k=10
90.34
94.49
0.76
k=15
90.05
94.36
0.74
k=5
89.95
94.31
0.72
t=2
k=10
89.47
94.04
0.81
k=15
87.71
92.71
0.79
Table 4: Ablation study hyperparameters of GSF.
Weighting Strategy. The average weight of sampled positive and negative keys without seman- tic similarity threshold is presented in Fig.5 per 10 epochs. It clearly shows that positive keys domi- nate model training consistently, while semantic- unrelated negative keys are suppressed, and the semantic-related negative keys provide rich seman- tics for training through soft weighting.
Figure 7: Visualization of embeddings on CLINC. Left: w/o GSF Right: w/ GSF,
GSF corrects some semantically ambiguous sam- ples on the boundary by bringing them closer to the side with stronger connectivity.
5.4 Inftuence of Known Class Ratio
5.3 Analysis of GSF
To investigate the influence of the known class ratio on model performance, we vary it in the range of 0.25, 0.50 and 0.75. As shown in Fīg.6, our method achieves comparable best performance under different settings all evaluation metrics, which fully demonstrates the effectiveness and robustness of our method.
To verify the effectiveness of GSF under different stacking layers and neighborhood sizes, we freeze the trained model and perform KMeans clustering with representations smoothed to varying degrees. Table 4 shows the results of ARI, ACC and Silhou- ette Coefficient (SC) on CLINC. The performance on different evaluation metrics is mostly superior to direct clustering and robust to hyperparameter changes. In particular, the SC value shows a signif- icant improvement, indicating a reduction in clus- tering uncertainty.
6 Conclusion
In this paper, we propose a novel Diffusion Weighted Graph Framework (DWGF) for new in- tent discovery, which models structure relation- ships inherent in data through nearest neighbor- guided diffusion. Combined with structure relation- ships, we improve both the sampling and weighting strategy in contrastive learning and adopt super- vision from local and global views. We further
To further illustrate how GSF improves infer- ence, we randomly sample 15 classes from CLINC and t-SNE visualize them. Fig.7 clearly shows the more compact cluster distributions after smoothing, and the partially zoomed-in illustrations show that

---

propose Graph Smoothing Filter (GSF) explore the potential of structure relationships in inference, which effectively filters noise embodied in seman- tically ambiguous samples on the cluster bound- ary. Extensive experiments on all three clustering metrics across multiple benchmark datasets fully validate the effectiveness and robustness of our method.
lñigo Casanueva, Tadas Temčinas. Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- lernational conference on machine learning, pages 1597-1607. PMLR.
Limitations
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguisties: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.
Even though the proposed Diffusion Weighted Graph framework achieves superior performance on the NID task, it still faces the following limita- tions. Firstly, the construction of DWG and GSF needs extra hyperparameters, and their changes will slightly impact the model's performance. Sec- ondly, it is time-consuming to do nearest neighbor retrieval the entire dataset.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. 2021. With little help from my friends: Nearest-neigghor con- trastive learning of visual representations. In Pro- ceedings of the IEEF/CVE International Conference on Computer Vision pages 9588-9597.
Acknowledgments
This work was supported by National Key Research and Development Program of China 3022ZD0117102 National Natural Science Foun- dation of China (62293551, 62177038, 62277042, 62137002, 61721002, 61937001, 62377038). In- novation Research Team of Ministry of Educa- tion (IRT_17886). Project of China Knowledge Centre for Engineering Science and Technology, "LENOVO-XITU" Intelligent Industry Joint Labo- ratory Project.
Tianyu Gao, Xingcheng Yao, Danqi Chen. 2021, Simese: Simple contrastive learning of sentence em- beddings. arXiv preprint arXiv:2104.0888
Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2019. Learning discover novel visual categories via deep transfer clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8401-8409.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738.
References
Wenbin An, Feng Tian, Ping Chen, Siliang Tang. Qinghua Zheng, and QianYing Wang. 2022a. Fine- grained category discovery under coarse-grained su- pervision with hierarchical weighted self-contrastive learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Teddy Furon, and Ondrej Chum. 2017. Efficient diffusion region manifolds: Recovering small objects with compact cnn representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2077-2086.
Wenbin An, Feng Tian, Ping Chen, Qinghua Zheng, and Wei Ding. 2023. New user intent discovery with robust pseudo label training and source domain joint- training. IEEE Intelligent Systems.
Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in neural information processing systems, 13:18661-18675
Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, Qian Ying Wang, and Ping Chen. 2022b. General- ized category discovery with decoupled prototypical network. arXiv preprint arXiv:2211.15//5.
Tacuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for bert sentence representations. arXiv preprint arXiv:2106.0334
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the Eurapean conference on computer vision (ECCV), pages 132-149.
Thomas N Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks. arXiv preprint arXiv: 169

---

Rajat Kumar, Mayur Patidar, Vaibhav Varshney, Lovekesh Vig, and Gautam Shroff. 2022. Intent de- tection and discovery from user logs via deep semi- supervised contrastive clustering. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistis: Human Language Technologies, pages 1836-1853.
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analy- sis. In International conference machine learning. pages 478-487. PMLR.
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural net- works. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing. pages 62-69.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, al. 2019. An evaluation dataset for intent classification and out-af-scope pre- diction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLA) pages 1311-1316.
Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos. and Mingyi Hong. 2017. Towards mmens-ffreendl spaces: Simultaneous deep learning and clustering. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3861-3870. PMLR.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. 2020. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966
Hanlei Zhang, Hua Xu, Ting En Lin, and Rui Lyu. 2021. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14365-14373,
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng- Joey Tianyi Zhou, and Xi Peng. 2021. Contrastive clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8547- 8555.
Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao- Wu, and Albert Lam. 2022. New intent discovery with pre- pre-training and contrastive learning. arXiv preprint arXiv:2205.12911 12914.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Dis- covering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference Artificial Intelligence. volume 34, pages 8360 8367.
Yutao Mou, Keqing He, Pei Wang. Yanan Wu, Jingang Wang, Wei Wu, and Weiran Xu. 2022a. Watch the neighhors: A unified k-nearest neighbor contrastive learning framework for ood intent discovery. arXiv preprint arXhv:2210.0800
Yutao Mou, Keqing He, Yanan Wu, Pei Wang. Jingang Wang, Wei Wu. Yi Huang, Junlan Feng, and Weiran Xu. 2022b. Generalized intent discovery: Learning from open world dialogue system. arXiv preprint arXiv:2209.06330.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Chun Wang, Shirui Pan, Ruiqi Hu, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019. Attributed graph clustering: deep attentional embedding approach. In Proceedings of the 28th International Joint Con- ference on Artificial Intelligence, pages 3670-3676.
Feng Wei, Zhenbo Chen, Zhenghong Hao, Fengxin Yang, Hua Wei, Bing Han, and Sheng Guo. 2022. Semi-supervised clustering with contrastive learn- ing for discovering new intents. arXiv preprint urXiv:2201.06004


Reviews:
[{"review_id": "TMkGIyYW9h", "reviewer": "Reviewer_zbJ8", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your comments and will thoroughly revise our paper. Responses to the comments are listed below.\n\n[Q1]: When dividing the dataset, the number of known intent classes significantly outweighs the unknown intent classes. However, in an open-world scenario, the situation may reverse, where unknown classes could outnumber the known intent classes. The experiments do not thoroughly capture the models' performance in such open-world settings.\n\n[R1]: We divide datasets by Known Class Ratio (KCR), which refers to the proportion of known classes against all classes. First, following previous works, we set KCR=0.75 to conduct main experiments to make a fair comparison. Second, in Sec. 5.4, we vary KCR in the range of {0.25, 0.50, 0.75} to simulate different open-world settings, which include your proposed scenario where novel classes outnumber known classes. The experimental results shown in the table below show our method outperforms previous methods in all settings. Notably, in the case of extremely low KCR (e.g. 0.05), the model training almost degenerates into unsupervised learning, resulting in insufficient known class knowledge to be utilized and losing the research value of NID.\n\n|          | KCR=0.75  |           |           | KCR=0.50  |           |           | KCR=0.25 |           |           |\n| -------- | --------- | --------- | --------- | --------- | --------- | --------- | -------- | --------- | :-------: |\n|          | NMI       | ARI       | ACC       | NMI       | ARI       | ACC       | NMI      | ARI       |    ACC    |\n| DTC      | 89.19     | 67.15     | 77.65     | 83.01     | 50.44     | 64.39     | 79.35    | 41.92     |   55.90   |\n| CDAC+    | 85.96     | 55.17     | 68.23     | 86.02     | 54.87     | 68.01     | 84.68    | 50.02     |   66.24   |\n| DAC      | 93.92     | 79.94     | 86.79     | 91.59     | 72.56     | 80.70     | 88.97    | 64.63     |   74.07   |\n| DCSC     | 95.28     | 84.41     | 89.70     | 93.75     | 78.82     | 84.57     | 91.7     | 72.68     |   79.89   |\n| DPN      | 95.12     | 85.47     | 89.06     | 94.35     | 80.09     | 85.1      | 92.3     | 75.32     |   80.37   |\n| CLNN     | 96.08     | 86.97     | 91.24     | 94.80     | 81.17     | 87.18     | 92.97    | 77.21     |   84.09   |\n| **Ours** | **96.89** | **90.05** | **94.49** | **95.53** | **85.73** | **91.12** | **93.3** | **78.91** | **86.81** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed method is novel.\n2. The experiments are comprehensive.\n3. The performances are very competitive compared with recent baselines.\n4. The paper is well-written.\n5. Ablation studies are convincing.", "reasons_to_reject": "When dividing the dataset, the number of known intent classes significantly outweighs the unknown intent classes. However, in an open-world scenario, the situation may reverse, where unknown classes could outnumber the known intent classes. The experiments do not thoroughly capture the models' performance in such open-world settings.", "questions_to_authors": "", "paper_topic_and_main_contributions": "New Intent Discovery (NID) seeks to identify both familiar and novel intents in unlabeled data, using limited labeled data that exclusively includes known intents. This paper introduces the novel Diffusion Weighted Graph Framework (DWGF) for discovering new intents. DWGF captures inherent data structure relationships via guided diffusion. This enhances sampling and weighting in contrastive learning and the supervision from local and global perspectives is adopted. Graph Smoothing Filter (GSF) is proposed to leverage structural relationships during inference. This filters noise in semantically ambiguous samples at cluster boundaries effectively.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rJfgFce6QI", "reviewer": "Reviewer_CQvg", "summary": "", "questions": "A) Do you have any significance testing results for the tables, particularly Table 2?\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your comments and will thoroughly revise our paper. Responses to the comments are listed below.\n\n[Q1]: Do you have any significance testing results for the tables, particularly Table 2?\n\n[R1]: We conducted significance tests of the main results (Table 2) across all benchmark datasets compared to other methods. As shown in the table below, the t-test results ($p$-value) fully demonstrate that our method is significantly better than other methods.\n\n|       | BANKING |       |       | StackOverflow |       |       | CLINC |       |       |\n| ----- | ------- | ----- | ----- | ------------- | ----- | ----- | ----- | ----- | :---: |\n|       | NMI     | ARI   | ACC   | NMI           | ARI   | ACC   | NMI   | ARI   |  ACC  |\n| DTC   | 1e-6    | 5e-6  | 5e-6  | 1e-5          | 2e-5  | 8e-5  | 2e-8  | 5e-9  | 2e-8  |\n| CDAC+ | 6e-7    | 3e-6  | 2e-6  | 8e-5          | 8e-6  | 1e-4  | 2e-9  | 9e-10 | 1e-8  |\n| DAC   | 1e-5    | 3e-5  | 3e-5  | 5e-5          | 4e-5  | 3e-4  | 4e-7  | 6e-7  | 1e-6  |\n| DSSCC | 4e-5    | 1e-4  | 1e-4  | 1e-3          | 2e-3  | 0.001 | 4e-7  | 1e-6  | 3e-6  |\n| DPN   | 1e-3    | 5e-3  | 7e-3  | 2e-3          | 2e-3  | 0.002 | 3e-6  | 3e-4  | 6e-6  |\n| DCSC  | 0.002   | 0.006 | 0.003 | -             | -     | -     | 5e-6  | 2e-5  | 1e-5  |\n| CLNN  | 0.043   | 0.036 | 0.025 | 0.082         | 0.034 | 0.045 | 0.011 | 0.015 | 0.008 |\n\nMoreover, we conducted significance tests of the ablation study on BANKING (Table 3). As shown in the table below, the t-test results ($p$-value) fully demonstrate that our proposed DWG and GSF significantly contribute to performance improvement.\n\n|               | NMI   | ARI   | ACC   |\n| ------------- | ----- | ----- | ----- |\n| GSF           | 0.043 | 0.004 | 0.037 |\n| Self-training | 0.003 | 0.002 | 0.006 |\n| DWG           | 3e-8  | 2e-9  | 1e-8  |\n\n[Q2]: Missing references and presentation improvements.\n\n[R2]: Thank you for your suggestions on writing. We will add citations and provide a brief description for the evaluation metrics, adjust the citation style, specify random seeds to improve reproducibility, bold the best-performing numbers in Table 2, correct typos, and thoroughly improve presentation in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors introduce a novel method for new intent discovery that outperforms the baselines (including the previous state-of-the-art model). \nThe authors provide extensive evaluation of their method, including a detailed ablation study.\nThe paper is well-written and clear.", "reasons_to_reject": "The authors do not test for significance for these results, so it is unclear to what extent their results are significant when compared to the SOTA model.\n", "questions_to_authors": "A) Do you have any significance testing results for the tables, particularly Table 2?\n", "paper_topic_and_main_contributions": "In this work, the authors propose a method for new intent discovery that contains two novel components: a Diffusion Weighted Graph Framework (DWGF) which captures information about semantic similarity and structures within the data, and a Graph Smoothing Filter (GSF) that utilizes this graph to filter high-frequency noise. The authors test their method against existing unsupervised and semi-supervised baselines on three different datasets: BANKING, Stack-Overflow, and CLINC. The results show performance improvements against all of the other baselines, though some of these improvements are more substantial than others (the authors' model exhibited the highest improvements in clustering accuracy, of 2.56%, 0.90% and 3.25%). The authors also conduct an extensive ablation study to examine the effects of different variables on DWGF and GSF, and provide detailed analyses and takeaways for the reader.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Section 4.3 - should add citations for the evaluation metrics used"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 12 and 125: Here, the acronym DWGF is used, while in the experiments section, DWG is typically used\nLine 110 and 259: should be \\citet\nLine 369: Include the domain of the BANKING dataset.\nLine 399: Provide brief description of (and justification for) the evaluation metrics used for readers not as familiar with the literature \nLine 425: Did the three runs use specific seeds? If so, for reproducibility purposes it would be good to specify what those were.\nTable 2: Should bold the best-performing numbers to draw more attention to the fact that your model outperforms the other baselines\nTable 3: Typo for \"methods\"\nLine 526: \"Extensive experiments on all evaluation metrics across multiple benchmark datasets fully validate the effectiveness and robustness of our method.\" - should clarify \"all evaluation metrics\" more, e.g. \"all three clustering metrics\".\n"}}, {"review_id": "q3GT3NhwNf", "reviewer": "Reviewer_DFjL", "summary": "", "questions": "Please refer to the \"Reasons To Reject\".", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your comments and will thoroughly revise our paper. Responses to the comments are listed below.\n\n[Q1]: Model Robustness Issue: The proposed Diffusion Weighted Graph Framework (DWGF) and Graph Smoothing Filter (GSF) demonstrate notable sensitivity to hyperparameters such as neighborhood size, diffusion rounds, and modulation factors. This sensitivity gives rise to concerns regarding the robustness of the model.\n\n[R1]: We totally agree that sensitivity to hyperparameters can be a concern for the robustness of the model. We conducted a comprehensive set of experiments to assess such sensitivity, and the results show our model’s robustness. \n\n1. neighborhood size $k$: Too large a neighborhood size will introduce much noise, while too small a neighborhood size will lead to insufficient supervisory signals. We vary neighborhood size $k$ with smaller steps in the range of [10, 20] and keep other hyperparameters unchanged to conduct experiments on BANKING. The experimental results shown in the table below demonstrate our model is robust to the first-order neighborhood size change, which can be attributed to the adaptive sampling through DWG.\n\n   |        | NMI   | ARI   | ACC   |\n   | ------ | ----- | ----- | ----- |\n   | $k$=10 | 85.98 | 66.99 | 77.93 |\n   | $k$=12 | 86.21 | 67.51 | 79.06 |\n   | $k$=14 | 86.74 | 68.31 | 79.39 |\n   | $k$=16 | 86.08 | 67.04 | 78.98 |\n   | $k$=18 | 86.21 | 67.94 | 79.18 |\n   | $k$=20 | 86.23 | 67.35 | 78.54 |\n\n2. diffusion rounds $r$: As shown in Figure 3, no matter the diffusion rounds $r$ is set to 1 or 2, the equivalent performance can be achieved by slightly adjusting the neighborhood size. \n\n3. semantic similarity threshold $\\gamma$: We vary $\\gamma$ in the range of {0.1, 0.2, 0.3, 0.4, 0.5} (we set it to 0.3 in Sec. 4.4) and keep other hyperparameters unchanged to conduct experiments on BANKING. The experimental results shown in the table below demonstrate our model is robust to the threshold changing around the optimal value.\n\n   |              | NMI   | ARI   | ACC   |\n   | ------------ | ----- | ----- | ----- |\n   | $\\gamma$=0.1 | 86.17 | 67.38 | 78.83 |\n   | $\\gamma$=0.2 | 86.23 | 67.94 | 79.12 |\n   | $\\gamma$=0.3 | 86.41 | 68.16 | 79.38 |\n   | $\\gamma$=0.4 | 85.95 | 67.11 | 79.29 |\n   | $\\gamma$=0.5 | 86.11 | 67.35 | 78.38 |\n\n4. modulation factor $\\lambda$: We vary $\\lambda$ in the range of {0.9, 1.0, 1.1, 1.2, 1.3} (we set it to 1.1 in Sec. 4.4) and keep other hyperparameters unchanged to conduct experiments on BANKING. The experimental results shown in the table below demonstrate our model is robust to the factor changing around the optimal value.\n\n   |               | NMI   | ARI   | ACC   |\n   | ------------- | ----- | ----- | ----- |\n   | $\\lambda$=0.9 | 86.92 | 68.52 | 79.35 |\n   | $\\lambda$=1.0 | 86.33 | 67.99 | 79.38 |\n   | $\\lambda$=1.1 | 86.41 | 68.16 | 79.38 |\n   | $\\lambda$=1.2 | 86.86 | 68.58 | 79.22 |\n   | $\\lambda$=1.3 | 86.57 | 67.72 | 79.03 |\n\n[Q2]: Lack of Model Complexity Discussion: The intricacies involved in constructing graphs and executing diffusion within the DWGF could result in significant computational requirements. However, the paper lacks a comprehensive discourse on the complexity of the model.\n\n[R2]: The extra time complexity caused by constructing graphs and executing diffusion is neglectable since we only initialize DWG once and update it once or twice during the whole training process. In contrast, other pseudo-labeling methods such as DAC and CLNN require frequent and time-consuming label alignment (per epoch) and neighbor searching (every 5 epochs), respectively. We calculate the average training time per epoch for each method, and the experimental results shown in the table below support our claims. We will add the complexity discussion in the revised version. Thanks.\n\n|          | BANKING (sec.) | Stackoverflow (sec.) | CLINC (sec.) |\n| -------- | -------------- | -------------------- | ------------ |\n| DAC      | 44.24          | 54.23                | 116.78       |\n| CLNN     | 61.97          | 98.67                | 81.32        |\n| **DWGF** | **29.63**      | **46.04**            | **37.63**    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Performance Improvement: The proposed DWGF consistently outperforms various baselines and state-of-the-art methods across multiple benchmark datasets. The demonstrated improvements in clustering accuracy, normalized mutual information, and adjusted Rand index highlight the effectiveness of DWGF in discovering new intents with limited known intent data.\n\nGeneral Applicability: The versatility of the proposed framework is evident in its successful application to various benchmark datasets, including fine-grained intent classification, question classification, and multi-domain intent classification. This showcases its potential for addressing diverse NLP tasks related to intent discovery.", "reasons_to_reject": "Model Robustness Issue: The proposed Diffusion Weighted Graph Framework (DWGF) and Graph Smoothing Filter (GSF) demonstrate notable sensitivity to hyperparameters such as neighborhood size, diffusion rounds, and modulation factors. This sensitivity gives rise to concerns regarding the robustness of the model.\n\nLack of Model Complexity Discussion: The intricacies involved in constructing graphs and executing diffusion within the DWGF could result in significant computational requirements. However, the paper lacks a comprehensive discourse on the complexity of the model.", "questions_to_authors": "Please refer to the \"Reasons To Reject\".", "paper_topic_and_main_contributions": "The paper addresses the problem of discovering new intents in text data, which is crucial for improving the performance of intent recognition systems. The main contributions of this paper are twofold. Firstly, it introduces a novel Diffusion Weighted Graph (DWG) framework, which leverages structural relationships among intents to enhance intent discovery. This framework incorporates semantic similarity and leverages supervised signals for effective representation learning. Secondly, the paper presents comprehensive experimental evaluations on real-world datasets, demonstrating the superiority of the proposed DWG framework over existing methods in terms of performance metrics such as Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Accuracy (ACC). ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NfFcTriDLV", "reviewer": "Reviewer_pSNe", "summary": "", "questions": "1. the novelty of this work is not high, this work combines a lot of existing techniques: Bert, MLM, diffusion, neural KMenans, GNN etc.  The authors does provide ablation study. However, I feel like the proposed algorithm is a little heavy and complex.\n", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate your comments and will thoroughly revise our paper. Responses to the comments are listed below.\n\n[Q1]: The novelty of this work is not high, this work combines a lot of existing techniques: Bert, MLM, diffusion, neural KMenans, GNN etc.\n\n[R1]: We agree that BERT, MLM, and KMeans are commonly used as backbone, model initialization strategy, and evaluation method in a series of IND works (CDAC+ / DAC / DCSC / DPN / CLNN), and these components are not our main contributions. For the novelty of our work, there are three aspects to be clarified:\n\n1.  We point out the common limitation of current NID methods: Quantity and Quality Dilemma of supervisory signals. Instead of directly applying any existing methods about GNN to train the model, we diffuse neighborhood relationships along semantic paths to fully utilize structure relationships in data and improve sampling and weighting in contrastive learning, aiming to handle the quantity and quality dilemma of supervisory signals, which is totally different from GNN.\n\n2.  As our core novelty, we diffuse neighborhood relationships along semantic paths to model structure relationships (DWG) and improve the sampling and weighting based on DWG for the balance between quantity and quality, which helps better representation learning. (Note: “diffusion” in our paper is different from the 'diffusion model' used for image and text generation. We will highlight the difference in the revised version.)\n\n3.  As our secondary novelty, we are the first to introduce graph filter to further explore the potential of structure relationships in inference for NID.\n\n[Q2]: I feel like the proposed algorithm is a little heavy and complex. The authors can further discuss the complexity of the proposed algorithm.\n\n[R2]: The extra time complexity caused by constructing graphs and executing diffusion is neglectable since we only initialize DWG once and update it once or twice during the whole training process. In contrast, other pseudo-labeling methods such as DAC and CLNN require frequent and time-consuming label alignment (per epoch) and neighbor searching (every 5 epochs), respectively. We calculate the average training time per epoch for each method,  and the experimental results shown in the table below support our claims. We will add the complexity discussion in the revised version.\n\n|      | BANKING (sec.) | Stackoverflow (sec.) | CLINC (sec.) |\n| ---- | -------------- | -------------------- | ------------ |\n| DAC  | 44.24          | 54.23                | 116.78       |\n| CLNN | 61.97          | 98.67                | 81.32        |\n| DWGF | 29.63          | 46.04                | 37.63        |\n\n[Q3]: Some explanation are not very clear like Figure 7.\n\n[R3]: We feel very sorry for the caption error in Figure 7, where 'bottom' should be corrected to 'right'. The left half of the figure is the visualization of embeddings w/o GSF, the right half is the visualization with GSF, and the two subfigures at the bottom represent the corresponding zoomed-in views in both cases, respectively. The visualizations illustrate GSF can 1) help form more compact clusters, 2) correct some semantically ambiguous samples on the boundary by bringing them closer to the cluster with the same ground-truth class.\n\n[Q4]: Theoretical Analysis in A.3 of Graph Smoothing filter is not new, which is a common fact in GNN.\n\n[R4]: In Sec. 3.5, we emphasize that our contribution is to first introduce the structure-based filter into NID (Line 362-364) and verify its effectiveness in inference. The theoretical analysis in A.3 is solely to help readers unfamiliar with related works better understand the working principle of the filter since we have specified corresponding references in Sec. 3.5 for both the renormalization trick and the graph filter. Last but not least, the introduction of graph filter for inference is the second contribution. Our core contribution is to diffuse neighborhood relationships along semantic paths to model structure relationships (DWG) and cope with the quantity and quality dilemma through enhanced sampling and weighting based on DWG.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposed a diffusion method to capture both semantic similarities and structure relationships.\n\n2. The source code is available.\n\n3. The performance of the proposed work is good as shown in Table 2.\n", "reasons_to_reject": "1. the novelty of this work is not high, this work combines a lot of existing techniques: Bert, MLM, diffusion, neural KMenans, GNN etc.  The authors does provide ablation study. However, I feel like the proposed algorithm is a little heavy and complex.\n\n2. The authors can further discuss the complexity of the proposed algorithm.\n\n3. Some explanation are not very clear like Figure 7.\n\n4. A.3 Theoretical Analysis of Graph Smoothing filter is not new, which is a common fact in GNN.", "questions_to_authors": "1. the novelty of this work is not high, this work combines a lot of existing techniques: Bert, MLM, diffusion, neural KMenans, GNN etc.  The authors does provide ablation study. However, I feel like the proposed algorithm is a little heavy and complex.\n", "paper_topic_and_main_contributions": "The authors propose a Diffusion Weighted Graph Framework (DWGF) for NID, which can capture both semantic similarities and structure relationships inherent in data.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work considers discovery of new intents from limited labeled known intent data using a graph theoretic framework to capture structural similarities and perform inference.

**Pros**: Most reviewers agree the approach is novel, the paper is well-written, and the experiments convincingly argue the utility of the method. Versatile benchmarks are considered with competitive results and ablation study is convincing. 

**Cons**: Reviewers concerns are generally addressed during the rebuttal period: authors provide results which ease concerns on the statistical significance of the results, potential real-world scenarios wherein the method may fail, and sensitivity to hyper parameters. Concerns from one reviewer do remain about the novelty of the work, representing the method as a combination of known approaches. 

Given this disagreement on excitement, I looked into the work myself (considering the authors rebuttal and verifying in the paper's details). The backbone model, techniques, and use of GNN across *ACL venues is common. The graph-theoretic techniques are arguably known, as pointed out by reviewer pSNe, but the application of such techniques to self-training for new intent discovery is new. At the very least, the application specific results (which are comprehensive according to most reviewers) provide some new insight to the community.
Decision: Accept-Main