Learning Geometry-Aware Representations for New Intent Discovery (2024)

Abstract: New intent discovery (NID) is an important problem for deploying practical dialogue systems, which trains intent classifiers on a semisupervised corpus where unlabeled user utterances contain both known and novel intents. Most existing NID algorithms place hope on the sample similarity to cluster unlabeled corpus to known or new samples. Lacking supervision on new intents, we experimentally find the intent classifier fails to fully distinguish new intents since they tend to assemble into intertwined centers. To address this problem, we propose a novel GeoID framework that learns geometry-aware representations to maximally separate all intents. Specifically, we are motivated by the recent findings on Neural Collapse (NC) in classification tasks to derive optimal intent center structure. Meanwhile, we devise a dual pseudo-labeling strategy based on optimal transport assignments and semi-supervised clustering, ensuring proper utterances-to-center arrangement. Extensive results show that our GeoID method establishes a new state-of-the-art performance, achieving a +3.49% average accuracy improvement on three standardized benchmarking datasets. We also verify its usefulness in assisting large language models for improved in-context performance. The code is available at https: //github.com/zjutangk/GeoID.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5641–5654 August 11-16, 2024 ©2024 Association for Computational Linguistics
New intent discovery (NID) is an important problem for deploying practical dialogue systems, which trains intent classifiers on a semisupervised corpus where unlabeled user utterances contain both known and novel intents. Most existing NID algorithms place hope on the sample similarity to cluster unlabeled corpus to known or new samples. Lacking supervision on new intents, we experimentally find the intent classifier fails to fully distinguish new intents since they tend to assemble into intertwined centers. To address this problem, we propose a novel GeoID framework that learns geometry-aware representations to maximally separate all intents. Specifically, we are motivated by the recent findings on Neural Collapse (NC) in classification tasks to derive optimal intent center structure. Meanwhile, we devise a dual pseudo-labeling strategy based on optimal transport assignments and semi-supervised clustering, ensuring proper utterances-to-center arrangement. Extensive results show that our GeoID method establishes a new state-of-the-art performance, achieving a +3.49% average accuracy improvement on three standardized benchmarking datasets. We also verify its usefulness in assisting large language models for improved in-context performance. The code is available at https: //github.com/zjutangk/GeoID.

heading: 1 Introduction
text: Intent detection, which aims at recognizing the intention of the user query, is an important component in industrial-grade dialogue systems (Qin et al., 2021; Li et al., 2022). Though there has been significant progress in recent years (Perkins and Yang, 2019; Min et al., 2020; Vedula et al., 2020), the effectiveness of these models greatly depends on the predefined intent labels, which often fall short of fulfilling the application requirements in
* Corresponding author.
an open-world environment. To address this problem, new intent discovery (NID) has been proposed that learns from a semi-supervised corpus containing: (i)-a limited amount of utterance annotated with true intents; (ii)-a vast quantity of unlabeled data containing both known intents and unknown new intents. The goal of NID is to simultaneously classify known intents while discovering those unknown intent clusters.
Emerging attempts have been made to tackle the NID task (Zhou et al., 2023a; Mou et al., 2023; Wang et al., 2023). Amongst them, the most popular methods (Zhou et al., 2023b,c) rely on the similarity-based assumption, i.e., samples sharing the same intent typically located at a neighbor region. Then, they design specific objectives to spontaneously group different samples into separated clusters, e.g. pairwise constraints (Lin et al., 2020), contrastive learning (Zhang et al., 2022), and graph diffusion (Shi et al., 2023). However, without new intent supervision, the learning procedure can be dominated by the labeled samples, resulting in biased feature representations. Thus, the unknown
5641
samples are prone to collapse into intertwined centers, thus being inseparable, as empirically verified in Figure 5. This gives rise to a representation dilemma—how to obtain the optimal hypersphere distribution for identifying new intents?
To answer this question, we draw inspiration from a recent study on the neural collapse (NC) phenomenon (Papyan et al., 2020). It implies that, under ideal conditions, as a classifier undergoes training towards convergence, the final-layer characteristics of a specific class gradually converge towards their average value within that class. These averages then tend to position themselves on a hypersphere with the utmost angular distance, known as the simplex equiangular tight frame (simplex ETF), as referenced in (Martinez and Kak, 2001). The NC phenomenon provides an optimal frame of hypersphere centers that aligns with our expectations for the final intent centers to be maximally separated. Nevertheless, it is non-trivial to achieve NC for NID with largely missing supervision.
In this paper, we propose a novel Geometryaware representation learning framework for NID (dubbed GeoID), which encourages both known/unknown intents to be equiangular separative and maximally discriminative. Specifically, GeoID replaces the learnable weights of the intent classifier (implicitly the class centers) with fixed ETF weights. That is, it pulls the sample features to predefined intent centers instead of merely clustering by sample similarity themselves. Moreover, we develop a dual pseudo-labeling strategy to ensure proper sample allocation to the centers: (i)optimal transport-based objective to refine pseudolabels from classifier predictions, which is reliable with pattern fitting; (ii)-semi-supervised clusteringdriven pseudo-labeling that is more robust in the early stages. Finally, We train the NID classifier to fit both pseudo-labels for improved robustness.
We comprehensively evaluate GeoID on three NID benchmark datasets. Specifically, our visualized analysis shows that GeoID does indeed learn highly distinguishable intent clusters (Figure 5(c)), and its mean intent centers almost achieve neural collapsed distribution (Figure 3(c)). Thanks to this excellent geometry property, our GeoID establishes new state-of-the-art NID performance, e.g., it improves the accuracy by +2.90%, +3.88%, and +3.69% accuracy on the BANKING, StackOverflow, and CLINC datasets, respectively. We hope our work will inspire future NID studies to tackle this important problem of cluster separability.

heading: 2 Related Work
text: New Intent Discovery (NID). NID is a developing research area aiming to identify unlabeled utterances from known and new intents (Wu et al.; Liang and Liao, 2023; Kumar et al., 2022; An et al., 2023a). To address this issue, initial efforts were made from the perspective of constrained clustering. Hsu et al. (2018a,b) utilized pairwise similarity-constrained clustering and refinement module to discover new intents and Zhang et al. (2021) employed pretraining techniques and aligned clustering labels to learn clustering-friendly representations. Subsequent work aims to learn more discriminative feature representations from the perspective of contrastive learning. Wei et al. (2022) emphasized intra-class compactness by employing contrastive learning following (Kim et al., 2021; Yan et al., 2021; Giorgi et al., 2021) to bring samples with the same pseudo-label closer in feature space. An et al. (2023b) adopted a decoupled training approach to construct prototype learning, aiming to obtain more discriminative features. Zhang et al. (2022) harnessed neighbor contrastive learning and more challenging pre-training tasks to achieve outstanding performance. In addition, Zhang et al. (2023a) integrated techniques from previous works including aligned clustering and contrastive learning. These works primarily tackle the intra-class compactness issue while neglecting the influence of bias on inter-class separation.
Neural Collapse (NC). The NC phenomenon was initially discovered by Papyan et al. (2020). They observed that towards the end of training, a classification model would exhibit the collapse of last-layer features towards their respective withinclass centers. These within-class centers, along with the classifiers, will converge to form a simplex equiangular tight frame (ETF). Subsequent studies have sought to provide theoretical insights into this elegant phenomenon. It has been proven that neural collapse corresponds to the global optimality of simplified models under various conditions, including regularization (Tirer and Bruna, 2022; Zhou et al., 2022; Zhu et al., 2021)and constraints (Fang et al., 2021; Graf et al., 2021). These findings hold for both cross-entropy (CE) (Ji et al., 2021; Graf et al., 2021) and mean squared error (MSE) loss functions (Han et al., 2021; Zhou et al., 2022). Phenomenon of neural collapse (NC) has also been explored in specific scenarios, such as imbalanced learning (Xie et al., 2023; Yang et al., 2022; Zhong
et al., 2023), learning with noisy labels (Nguyen et al., 2022), and transfer learning (Galanti et al., 2021; Xiao et al., 2024), but they have never tapped in language domain. We are the first to investigate the NC phenomenon in the NID task.

heading: 3 Method
text: Problem Statement. Confronted with a new intent discovery task, we are usually given an intent dataset Dtrain encompassing two subsets: a labeled set Dl = {(xi, yi) | yi ∈ Cknown}mi=1, and an unlabelled set Du = {xi | yi ∈ Cknown ∪ Cnovel}ni=1, where xi refers to the input utterance,Cknown is the set of known intent labels and Cnovel is the set of new intent labels. We set |Cknown ∪ Cnovel| to L. NID can be viewed as a direct extension of general category discovery and the objective comprises two main goals: firstly, to identify new intents from unlabelled data, and secondly, to accurately classify inputs into their respective intents.
Intent Representation. Similar to the majority of mainstream tasks, we use BERT (Devlin et al., 2019) to extract intent representations. First, we input utterance xi to BERT and get all its token embeddings [CLS, T1, . . . , TM ]. Next, we utilize mean-pooling and dense layer to obtain intent feature representation zi. For contrastive learning, we utilize Random Token Replacement (RTR) following (Zhang et al., 2022) as data augmentation strategy to generate zi′.
Overview of Our Approach. Figure 2 illustrates the overall architecture of our method. We learn geometry-aware representations to maximally separate all intents by pushing utterances close to the corresponding vertex of the ETF structure inspired by the NC phenomenon. To achieve this, we devise pseudo-labeling a dual strategy based on optimal transport and clustering to assign samples to the correct centers.

heading: 3.1 Neural Collapse for Separation
text: In this section, we introduce the neural collapse (NC) phenomenon (Papyan et al., 2020) to demonstrate the characteristics of an ideal intent classifier for NID. In concrete, NC says the features learned from deep neural networks, in which last-layer features have the following appearances:(1) For each class, features collapse to the class mean. (2) The within-class means of all classes are located on a hypersphere and form a simplex equiangular tight frame (Simplex ETF).
Simplex ETF. Suppose the vector space is ddimensional. When d ≥ L − 1, we can always derive a collection of L equal-length and maximally-equiangular d-dimensional embedding vectors E = [e∗1, . . . , e ∗ L] ∈ Rd×L to construct a simplex equiangular tight frame (ETF).
E =
√ L
L− 1U(IL − 1 L 1L1 ⊤ L ) (1)
where IL is the identity matrix, 1L is a vector of all ones and U ∈ Rd×L is a rotation matrix.
As mentioned above, an optimal ETF structure for the arrangements of features facilitates the minimization of within-class variance and the maximization of between-class variance for the features. This is very appealing for the NID task to distinguish all known/unknown intents.
In our GeoID framework, we calculate a set of pre-assigned centers E = [e∗1, . . . , e ∗ l ] ∈ Rd×L, each of which is a vertex of a random simple ETF structure according to Eq. (1). Then we optimize the following cross-entropy loss to pull these samples close to these centers:
Letf (xi, ŷi) = −log exp(z⊤i · e∗ŷi)∑L l=1 exp(z ⊤ i · e∗l )
(2)
where zi is feature representation of xi gotten by deep neural model. The key difference here from traditional CE loss is that we keep the classifier weights E fixed.
There remains one crucial problem — how to assign a sample xi to its corresponding ETF vertex e∗ŷi . For labeled utterances (xi, yi), we can directly set ground-truth label yi as ŷi. For the rest of unlabeled utterances, we resort to a pseudo-labeling algorithm to group them to proper centers, as described in the sequel.

heading: 3.2 Dual Pseudo-Labeling
text: To promote the formation of the novel class NC structure, we design two complementary pseudolabel generation strategies to guide the samples moving toward their corresponding ETF vertices.
Optimal Transport-based Strategy. Having gotten the softmax-based classifier prediction P = {pi | pi = softmax(z⊤i ·E)}, one straightforward technique to obtain pseudo-labels is to adopt the maximum index of prediction (Sohn et al., 2020). However, due to the lack of supervision information, the classifier can be largely dominated by
known classes, making the representation of unknown classes collapse. To mitigate this problem, we introduce equipartition constraint into label assignment and transform pseudo-labeling into an optimal transport problem. Formally, given a batch of b unlabelled samples, we search for optimal label assignment Q = [q1, . . . , qb] close to the current prediction P while subject to some constraints:
Q = max Q∈∆
Tr(Q⊤P ) = i=b∑
i=1
j=L∑
j=1
q⊤ijpij
s.t. ∆ = {[qij ]L×b | Q1b = r,Q⊤1L = c} (3)
where Tr(·) is the trace function. qij is the j-th element of pseudo-label assignment for the i-th sample. r = 1L1L is an L-dimensional uniform probability distribution indicating the prior class distribution. We set c = 1b1b indicating that b samples in a batch are sampled uniformly. The solution of Eq.(3) can be obtained by the Sinkhorn-Knopp algorithm (Cuturi, 2013); see Appendix A.3.
Such a strong assignment inevitably introduces a significant amount of noise. To alleviate the accumulation of label errors, we filter pseudolabels by high-confidence selection. For each class j ∈ [1, L], we select samplesDjsel:
Djsel = {(xi, yoti ) | yot = j, qij > τj} (4)
where yoti = argj max qij is the predicted category label of the ETF classifier on xi. τj is the confidence threshold that enables selecting the top R% ranked samples in the data subset whose prediction is class j. Lastly, we only calculate LETF on Dl and Dsel = ∪Lj=1Djsel. In other words, we select the most reliable samples within each cluster and employ them as pivots to guide the whole cluster being moved toward the ETF centers.
Clustering-based Strategy. Even with optimal transport-based allocation, the classifier can be unreliable at the early stage of training. To this end, we supplement a data-driven strategy by using feature-based clustering to regularize the learning procedure. It should be noted such a data-driven strategy does not suffer from the bias caused by the absence of supervision and thus largely enhances the robustness of GeoID during the early stages.
Specifically, we harness the rationality of clustering algorithms to generate pseudo-labels once per epoch. Considering that we have labeled data available, we adopt semi-supervised k-means following (Vaze et al., 2022) as the clustering algorithm. The indices of clustering labels obtained by k-means are randomly permuted in each training epoch which makes it challenging for us to leverage the supervision information provided by the clustering results fully. To address this issue, we devise
a clustering label alignment strategy that combines with simplex ETF structure. To be specific, we align the cluster centers of different epochs with the pre-assigned centers E provided by NC to ensure consistent cluster assignment in semi-supervised k-means employing Hungarian algorithm (Kuhn, 2010) and obtain the aligned cluster label yalign which can be found in Appendix A.4. Finally, we allow the model to fit both pseudolabels in a trade-off manner :
Lcls = αLetf (x,yot) + (1− α)Letf (x,yalign) (5) where yot and yalign respectively correspond to the set of pseudo-labels generated by the two strategies. α is the trade-off hyperparameter ramped up from 0 to 1 during the training process.
In the beginning, our ETF classifier tends to trust the clustering pseudo-labels to avoid representation collapse. As the training proceeds, the filtered OT labels can gradually become more trustworthy, guiding the data clusters to move toward predefined ETF frames. In the meantime, the data-driven clustering pseudo-label still serves a proper regularization term that aligns the remaining data to its filtered neighbors, ensuring full data utilization. Finally, all data samples are tightly clustered to their corresponding centers, achieving the optimal geometry distribution we desire.

heading: 3.3 Contrastive Representation Enhancement
text: Finally, we also involve contrastive learning for improved representation. To explore semantically neighboring information and further enhance intraclass compactness and inter-class variance, we propose a contrastive loss following (Khosla et al., 2020) which brings similar samples close to improve clustering performance. Specifically, the positive sample set Pi for sample xi from two aspects: (1) samples sharing the same ground-truth labels or pseudo labels and (2) k-nearest neighboring samples using dot product as a distance metric.
Pi = {j | zj ∈ k-NN(zi) ∨ yj = yi} (6) where yi is respectively grand-truth label for labeled sample and yot for unlabeled sample in Dsel. Different from (Zhang et al., 2022) which conducts contrastive learning within a mini-batch, we mine neighbor samples from the entire dataset and utilize self-supervision signals in addition.
Lcon = − 1
|Pi| log
∑ j∈Pi exp(z
⊤ i zj)∑n
k=1 exp(z ⊤ i zk)
(7)
We also involve a conventional consistency regularization term Lreg for improved semi-supervised training, please refer to Appendix A.2 for more details. The overall training loss is given by,
Lall = Lcls + Lcon + Lreg (8)

heading: 4 Experiment

heading: 4.1 Datasets
text: We first evaluate our method on four benchmark intent datasets. Banking (Casanueva et al., 2020) is a fine-grained imbalanced intent classification dataset collected from banking dialogues. StackOverflow (Xu et al., 2015) is a large-scale question classification dataset collected from online queries. CLINC (Larson et al., 2019) is an intent classification dataset from multiple domains. For each dataset, we randomly select 25% of the classes as Cknown, while the remaining classes are treated as Cnovel. Detailed statistics of datasets are summarized in Table 1.

heading: 4.2 Baselines
text: We compare our method with various new intent discovery baselines, including CDAC+ (Lin et al., 2020), DeepAligned (Zhang et al., 2021), MTPCLNN (Zhang et al., 2022), and UNISD (Zhang et al., 2023a). We also include comparisons with constrained clustering and novel class discovery methods: KCL (Hsu et al., 2018a), MCL (Hsu et al., 2018b), GCD (Vaze et al., 2022), and DTC (Han et al., 2019). For clustering and novel class discovery methods used for computer vision tasks, we adapt them for our task by leveraging the BERT backbone. For performances of baselines, we directly adopt reported results from receptive papers or UNISD. For a fair comparison, we remove the use of the extern dataset in MTP-CLNN.

heading: 4.3 Evaluation Metrics
text: Following (Zhang et al., 2021, 2023a), We employ three commonly used metrics, namely Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and Accuracy (ACC), to evaluate the quality of the clustering results. To calculate accuracy, we
utilize the Hungarian algorithm (Kuhn, 2010) to derive the optimal assignments between clustering classes and ground-truth classes.

heading: 4.4 Implementation Details
text: We use the pre-trained bert-base-uncased BERT model from (Wolf et al., 2019) as our backbone. In the pre-training phase, we employ the same settings following (Zhang et al., 2022). Regarding model optimization, We employ AdamW optimizer (Wolf et al., 2019) with a warm-up schedule and 0.01 weight decay. The learning rate is set to 1e−4 for all benchmark datasets. For Sinkhorn-Knopp, we set hyperparameters following (Fini et al., 2021) that ϵ = 0.05 and run for 3 iterations. For mining k-nearest neighbors, we set k = 64 for BANKING and CLINC, k = 256 for StackOverflow. For pseudo-labeling, we set R of high-confidence selection as 30%, 10% in begin respectively for Cknown, Cnovel and linearly increases to 80% in 70 epochs. Additionally, to enhance training efficiency with the BERT backbone, we freeze all transformer layer parameters except the last layer following (Lin et al., 2020). Following most of the baselines, we run cluster-based evaluation when testing. Details of consistency regularization can be found in Appendix A.2. Although we involved several hyperparameters, most of them followed the default settings, which had minimal impact on the experiments.

heading: 4.5 Main Results
text: Table 2 shows the main experimental results on the BANKING, StackOverflow, and CLINC datasets. Our method consistently outperforms all competing
methods by a considerable margin on all metrics across various datasets. Specifically, without an extern dataset in pre-training, our method achieves improvements on three benchmark datasets with +2.90%, +3.88%, and +3.69% compared with best baselines. Moreover, in the scenario where the CLINC dataset is utilized as an external dataset, our model can maintain advantages on the Banking and StackOverflow respectively and details of this experiment are shown in Appendix B.1.

heading: 4.6 Discussion
text: Ablation Study. To further analyze the contributions of different components in our method, we conduct three ablation studies here. First, to analyze the effect of our dual pseudo-labeling strategy, we separately removed the optimal transport and clustering components during the pseudo-labeling process. Then we exclude the use of the ETF structure by replacing the ETF structure with a linear classification head while keeping the form of crossentropy loss unchanged. These experiments are all conducted on the BANKING dataset to evaluate their effectiveness. As shown in Table 3, removing
these components respectively brings negative effects of−3.54%,−2.65%,−5.16% on accuracy, indicating that all these modules are beneficial to NID task. More details of the ablation study can be found in Appendix B.2.
Analysis on Neural Collapse. Considering the features of simplex ETF struct in Eq.(1), we can summarize the characteristics of the NC phenomenon from two perspectives: (1) uniform distribution on hypersphere (equiangularity) and (2) cluster centers are maximally separated. To further investigate the impact of NC on feature distribution, we visualize the cosine similarity between cluster centers that intuitively demonstrate the equiangularity of NC, and the results are shown in Figure 3(a)-3(c). To quantitatively analyze the second character of NC phenomenon, we calculate the average value Avgl ̸=l′ (cos(ĉ l, ĉl ′ )), where ĉl is the means of features from class 1 ≤ l ≤ L. For the maximal separation character of NC, the optimal pair-wise angle − 1L−1 is derived from Eq.(1). The trends of this metric on the BANKING dataset with different methods are shown in Figure 3(d). According to the visualization and quantitative results
shown in Figure 3, we have demonstrated that the distribution of features in GeoID can closely approximates the ideal scenario of the neural collapse phenomenon (NC).
Analysis on Representation Learning. To investigate the effectiveness of our method in geometryaware representation learning, we visualize the representation learned by GeoID and two strong baselines on StackOverflow using t-SNE (Van der Maaten and Hinton, 2008) in Figure 5. It can be shown that MTP-CLNN and DWGF produce intertwined cluster centers. In contrast, our method is demonstrated to effectively avoid representation collapse and learn clearly separable clusters. These results demonstrate that GeoID is indeed able to learn optimal geometry distributions for NID.
Influence of Known Class Ratio. Here we further evaluate the performance of our method on the BANKING dataset with different known class ratios from 0.25 to 0.75. As shown in Figure 4, our method consistently outperforms other baselines under varying known class ratios.
Unknown Number of Novel Classes. The number of all intents is an important hyperparameter in the NID task. The previous experiments were conducted based on the assumption that we already knew the number of novel classes which is often impractical in real-world applications. Now some methods to estimate class number have been proposed, such as (Han et al., 2019; Zhang et al., 2021). These methods can reliably maintain the error ratio below 10%, for example, methods in (Zhang et al., 2021) can estimate L in BANKING to be 71. So we conducted simulations on the BANKING dataset
using different prediction results of L to observe the impact of class number on model performance. In Figure 6, we show the performance variation of GeoID across different total numbers of predicted intent categories L. It can be observed that GeoID exhibits overall robustness across different total numbers of predicted intent categories.
Discussion of LLM. Recently, there have been explorations (Zhang et al., 2023b; Song et al., 2023) of applying large language models to the task of New Intent Discovery. In this section, we further investigate the feasibility of collaboration between LLMs and GeoID. Especially, we use GeoID to select corpora close to centers. This allows LLMs to perform in-context learning and subsequently perform direct clustering tasks. We design prompts following (Song et al., 2023) to implement two baselines ChatGPT-ZSD and ChatGPT-FSD for comparison. The results in Table 4 demonstrate that the supplement of novel class samples significantly improves the clustering performance of LLMs, particularly in balancing the performance gap between known classes and novel classes. Additionally, due to the limitations of LLMs in unsupervised cluster-
ing performance on corpora, their performance still falls short of fine-tuning SLMs. One may design other LLMs-based techniques specifically for clustering tasks for improved performance on NID. But, it is beyond the scope of our paper. More details and discussion can be found in appendix A.5.

heading: 5 Conclusion
text: In this paper, we propose a new method GeoID which revisits new intent discovery from the perspective of geometric-aware representation learning. Inspired by the neural collapse phenomenon, we use a fixed ETF structure to facilitate the learning of a feature distribution that aligns with the predefined optimal structure. Additionally, we design a dual pseudo-labeling strategy using optimal transport theory and semi-supervised clustering to generate pseudo-labels respectively. This strategy enhances the quality of pseudo-labels while maintaining sample utilization. Experiments show GeoID’s significant improvements over baselines. We hope our work can inspire future research to further improve the NID problem from the perspective of geometry-aware representations.

heading: Limitations
text: To inspire future research, we summarize the limitations of our method. Firstly, we do not design a more accurate method for category estimation, leaving ample scope for further enhancement. Secondly, we did not delve deeply into the collaboration between large and small models on NID tasks. Thirdly, we have conducted preliminary research on basic intent classification tasks within open-world scenarios, but there exist numerous other intricate data formats (like continual learning) that merit further exploration.

heading: Acknowledgements
text: This work is supported by the Pioneer R&D Program of Zhejiang (No. 2024C01035), CCFNetEase ThunderFire Innovation Research Funding (NO. 202305) and the Fundamental Research Funds for the Central Universities (No. 226-2024- 00049).

heading: A More Details of Implement

heading: A.1 Details of ETF Structure Initialization
text: We employ random initialization, a widely adopted and effective methodology in recent neural collapse research. As described in Eq.(1), we guarantee the randomness of the initialization by using a random rotation matrix U. The procedure involves only two hyperparameters: intent number L and feature dimension d. So there are no extra hyperparameters in this process.

heading: A.2 Details of Consistency Regularization
text: In practice, we use Random Token Replacement (RTR) following (Zhang et al., 2022) as our data augmentation strategy. We denote the augmented view of x as x′. Then we employ the r-drop method to obtain two representations, z′1 and z ′ 2 for x ′ through the backbone network. Finally, we calculate the Kullback-Leibler divergence between z′1 and z′2 as Lreg.

heading: A.3 Sinkhorn-Knopp Algorithm
text: Formally, we define a cost matrix M = exp(Pε ). The Sinkhorn-Knopp algorithm is conducted by:
Q = M ⊙ (µ · υT ) (9)
where u ∈ RL and v ∈ Rb are scaling coefficients vectors and are updated iteratively by,
µ← c./(Mυ),υ ← r./(M⊤µ) (10)
where ./ denotes element-wise division. In practice, we utilize a small iteration number of 3.

heading: A.4 Details of Alignment in Clustering
text: Having gotten pseudo-label assignment ycluster and class centers C = c1, c2, . . . , cL from semisupervised k-means. We obtain the optimal projection matrix M with Hungarian algorithm:
C = M ·E (11)
Then we get aligned pseudo labels:
yalign = M−1 · ycluster (12)

heading: A.5 Details of LLMs Experiment
text: The main challenge of LLMs in NID task is the absence of novel classes prior which seriously affects the incontext learning of large models. To solve this problem, we use GeoID to select corpora close to the center to provide novel classes prior.
In practice, for each novel class, we select 10 % samples nearest to the class center. We design the prompt in the form of <Prior: labeled samples and selected samples><Cluster Instruction><Response Format><Dtest>.

heading: B Additional Experimental Results

heading: B.1 Pretraining with External Dataset
text: In the experimental section of the main text, to ensure a fair comparison, we excluded the external dataset used in the pretraining process of the MTP-CLNN (Zhang et al., 2022). In this section, we investigate the model’s performance, which incorporates CLINC as the external dataset during the pretraining phase. The results shown in table 7 demonstrate that our method is equally capable of benefiting from the external dataset and maintaining a performance advantage on the benchmark.

heading: B.2 More Ablation Study
text: In this section, we conducted additional ablation experiments regarding sample selection and contrastive learning. Regarding sample selection, we design two different schemes: without filtering and fixing the selection ratio as 50%. We also remove Lcon during the training process to investigate the effect of contrastive learning. As shown in Table 6, removing them consistently impairs the model’s performance, indicating that these designs indeed contribute to new intent discovery. We also add the ablation experiments on regularization.
B.3 Influence of Labeled Ratio
In addition to the known class ratio, we also analyze the influence of label ratio in model training. In the experiment, we vary the labeled ratio in the range of 25%, 50%, 75%. As shown in Table 7,
our method achieves the best results with different labeled sample ratios.

heading: B.4 More Fine-grained Metrics
text: In this section, we further analyze both the known class accuracy and unknown class accuracy to validate the effectiveness of the model on novel intents. As shown in Table 8, tt can be seen that the performance improvement of our algorithm mainly comes from the improved accuracy of novel intent samples. This is precisely realized based on the optimal distribution of features we obtained through the ETF structure.

heading: B.5 Training Complexity
text: We compare the time required for the model to complete one training epoch with other works. As demonstrated in the table 5, GeoID is as fast as current state-of-the-art NID algorithms.

heading: C Overall Algorithm
text: We summarize the pseudo-code of our proposed GeoID in Algorithm 1.
Algorithm 1: Pseudo-code of GeoID. 1 Input: Training dataset D, model f , filter ratio R,
num of intents L; 2 Initialize ETF strcuture E 3 for epoch = 1, 2, . . . , do 4 for xi ∈ D do 5 Induce representation zi = f(xi) // optimal transport 6 pi = zi · E 7 qi = Sinkhorn(pi) as Eq.(3) 8 end // high-confidence selection 9 Dsel = filter(q) as Eq.(4)
10 yot = argmax(qij) for xi ∈ Dsel // cluster for pseudo-labeling 11 obtain ycluster with k-means clustering(z) 12 yalign = Hungarian(ycluster,E) 13 calculate Lcls with ETF structure as Eq.(5) // overall training objectives 14 minimize loss Lall = Lcls + Lcon + Lreg 15 end

heading: Future Work
text: To inspire future research, we summarize the limitations of our method. Firstly, we do not design a more accurate method for category estimation, leaving ample scope for further enhancement. Secondly, we did not delve deeply into the collaboration between large and small models on NID tasks. Thirdly, we have conducted preliminary research on basic intent classification tasks within open-world scenarios, but there exist numerous other intricate data formats (like continual learning) that merit further exploration