Venue: NAACL 2024

# Actively Learn from LLMs with Uncertainty Propagation for Generalized Category Discovery

Jinggui Liang\({}^{1}\), Lizi Liao\({}^{1}\), Hao Fei\({}^{2}\), Bobo Li\({}^{3}\), Jing Jiang\({}^{1}\)

\({}^{1}\)Singapore Management University, \({}^{2}\)National University of Singapore, \({}^{3}\)Wuhan University

jg.liang.2023@phdcs.smu.edu.sg lzliao@smu.edu.sg haofei37@nus.edu.sg boboli@whu.edu.cn jingjiang@smu.edu.sg

###### Abstract

Generalized category discovery faces a key issue: the lack of supervision for new and unseen data categories. Traditional methods typically combine supervised pretraining with self-supervised learning to create models, and then employ clustering for category identification. However, these approaches tend to become overly tailored to known categories, failing to fully resolve the core issue. Hence, we propose to integrate the feedback from LLMs into an active learning paradigm. Specifically, our method innovatively employs uncertainty propagation to select data samples from high-uncertainty regions, which are then labeled using LLMs through a comparison-based prompting scheme. This not only cases the labeling task but also enhances accuracy in identifying new categories. Additionally, a soft feedback propagation mechanism is introduced to minimize the spread of inaccurate feedback. Experiments on various datasets demonstrate our framework's efficacy and generalizability, significantly improving baseline models at a nominal average cost. 1

Footnote 1: [https://github.com/liangjinggui/ALUP](https://github.com/liangjinggui/ALUP)

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 7845–7858 June 16-21, 2024 0204 Association for Computational Linguistics

## 1 Introduction

Generalized Category Discovery (GCD) is a crucial task in open-world computing Lin et al. (2020); Zhang et al. (2021b), where the goal is to automate the classification of partially labeled data. It uniquely challenges systems to not only recognize predefined categories but also to discover entirely new categories from a mix of labeled and unlabeled data Yang et al. (2021); Zeng et al. (2022). This task mirrors the dynamic and evolving nature of real-world data, where new categories frequently emerge, necessitating models that can adapt and learn continually.

In traditional GCD methods, the initial step often involves supervised pretraining on a labeled dataset to establish a foundational understanding of known categories Zhong et al. (2021); Vaze et al. (2022). This is followed by self-supervised learning on unlabeled data or even contrastive learning, allowing the model to extract and learn patterns without explicit category labels An et al. (2023). The final stage typically employs clustering techniques, like K-Means MacQueen et al. (1967), to group similar data points, aiming to identify categories. However, this sequential process tends to imprint a bias towards the initially learned, known categories, thus limiting the model's ability to generalize to new, unseen categories Mou et al. (2022). Such overfitting to familiar data restricts the scope of GCD, preventing it from fully embracing the open-world setting it is intended for.

Recently, Large Language Models (LLMs) such as GPT-4 OpenAI (2023), PaLM Chowdhery et al. (2023), and LLaMA Touvron et al. (2023) have shown extraordinary versatility across a broad range of NLP tasks, providing good quality super

Figure 1: The active learning loop with propagated LLM feedback for model training.



vision signals for summarization Liu et al. (2023), clustering Zhang et al. (2023c), etc. Their ability to understand and generate nuanced language patterns makes them promising for supplementing the supervision of new categories in GCD. However, the direct application of LLMs in GCD, which typically involves processing and clustering thousands of samples, raises substantial challenges. The intensive computational demands of LLMs could lead to issues with data privacy, high latency, and increased costs, which are particularly problematic in large-scale GCD scenarios.

To circumvent the above challenges, integrating LLMs into an active learning framework presents a practical and efficient solution. This approach entails selectively using LLMs to provide supervision signals, especially in cases where the data is most uncertain or the categories are novel. However, this integration brings forth new challenges: optimizing the use of LLMs to ensure cost and time efficiency, and critically, ensuring the reliability of the feedback provided by LLMs. Effective strategies are needed to mitigate the risk of propagating incorrect feedback from LLMs.

Addressing these challenges, we propose a novel framework for GCD to _A_ctively _L_earn from LLMs with _U_ncertainty _P_ropagation, termed as **ALUP**. As shown in Figure 1, we begin by employing an uncertainty propagation strategy, which systematically identifies data samples in regions of high uncertainty - these are the areas where the model is least confident and, therefore, where LLM input could be most beneficial. The selected samples are then labeled using LLMs through a sophisticated comparison-based prompting technique. This method leverages the comparative strength of LLMs, making it easier for them to provide accurate feedback, especially for new and complex categories. To further enhance our approach, we incorporate a soft label propagation mechanism. This mechanism carefully extends the LLMs-generated feedback to similar, neighboring samples, effectively amplifying the value of each LLM query while minimizing the risk of propagating errors. Rigorous testing on diverse datasets has shown that our method not only significantly improves upon existing baseline models but also does so with a nominal increase in cost, offering a scalable, efficient, and effective solution for the intricate problem of GCD.

The main contributions of this work can be summarized as follows:

* We developed an innovative active learning framework integrating LLMs' feedback for GCD, addressing the challenge of limited supervision for new data categories.
* We combined uncertainty-region based data selection and comparison-based LLMs prompting, significantly enhancing GCD accuracy and efficiency with soft propagation.
* Experiments demonstrated marked improvements over traditional GCD methods across diverse datasets, affirming the ALUP's effectiveness and resource efficiency.

## 2 Related Work

### Generalized Category Discovery

Unsupervised Methods:The realm of GCD has been fundamentally shaped by unsupervised methods, focusing on learning cluster-friendly representations. These early methods Xie et al. (2016); Yang et al. (2017); Padmasundari and Bangalore (2018); Caron et al. (2018); Hadifar et al. (2019) laid the groundwork by using unsupervised clustering algorithms to group samples based on inherent similarities. Recent advancements, particularly with the emergence of LLMs, have brought a paradigm shift. The integration of LLMs in unsupervised GCD De Raedt et al. (2023); Zhang et al. (2023c); Viswanathan et al. (2023) represents a novel direction, pushing the boundaries of category identification beyond traditional clustering techniques.

Semi-Supervised Methods:In contrast, semi-supervised GCD methods blend limited labeled data with possibly larger unlabeled data to enhance category discovery Hsu et al. (2018, 2019); Han et al. (2019). Methods like CDAC+ Lin et al. (2020) utilize labeled data to guide clustering, creating a synergy between supervised knowledge and unsupervised discovery. The two-stage scheme, involving base model pretraining and iterative optimization Zhang et al. (2021a, b); Wu et al. (2022); Wei et al. (2022); Zhang et al. (2023a); Zhou et al. (2023); Mou et al. (2023), has gained popularity. It benefits from pseudo label signals generated by the pretrained model, although it often struggles with 

the quality of pseudo labels and sample representations. Efforts to refine learning objectives, such as contrastive learning (Mou et al., 2022; Zhang et al., 2022), aim to directly learn discriminative representations for new categories. Yet, the challenge remains in effectively decoupling pseudo label generation from representation learning (Wu et al., 2024), a gap our work addresses by introducing LLMs into the GCD.

### Active Learning in the Era of LLMs

Traditional Active Learning (AL):AL has traditionally been a solution to the data scarcity problem in NLP (Ren et al., 2022; Zhang et al., 2022), focusing on identifying and annotating informative samples. Various acquisition strategies have been employed, including uncertainty-based (Wang and Shang, 2014; Schroder et al., 2022; Yu et al., 2023), diversity-based (Sener and Savarese, 2018; Gissin and Shalev-Shwartz, 2019; Citovsky et al., 2021), and hybrid methods (Liu et al., 2018; Zhan et al., 2022). While effective, these methods still rely on expensive human expertise for annotation.

LLMs as a Game-Changer in AL:With the advent of LLMs, a new frontier in AL has been explored. LLMs are now being considered as cost-effective alternatives to human experts (Zhang et al., 2023; Cheng et al., 2023; Zhang et al., 2023; Margatina et al., 2023; Liao et al., 2023). For instance, Xiao et al. (2023) demonstrated the use of LLMs as active annotators, harnessing their ability to distill task-specific knowledge interactively. In our work, we further this exploration by applying AL with LLMs to GCD. Our unique contribution not only lies in the implementation of an uncertainty-driven propagation strategy to maximize the utility of LLMs in a cost-effective manner, but also in the design of a soft feedback propagation scheme to minimize the spread of inaccurate feedback.

## 3 Methodology

### Problem Formulation

We study the GCD problem defined as follows: Assuming we have a known category set \({\mathcal{C}_{k}}\) and an unknown category set \({\mathcal{C}_{u}}\), where \(\{{\mathcal{C}_{k}}\cap{\mathcal{C}_{u}}\}=\varnothing\) and \(|{\mathcal{C}_{k}}|+|{\mathcal{C}_{u}}|=K\). Here \(K\) is the total number of categories. Under the semi-supervised GCD setting, given a labeled data set \({\mathcal{D}_{l}}=\{(x_{i},y_{i})|y_{i}\in{\mathcal{C}_{k}}\}^{L}_{i=1}\), and an unlabeled data set \({\mathcal{D}_{u}}=\{x_{j}\}^{U}_{j=1}\) where the category of each \(x_{j}\) belongs to \(\{{\mathcal{C}_{k}}\cup{\mathcal{C}_{u}}\}\), the task is to learn a representation extractor \({\mathcal{M}}\) to identify all unknown categories from \({\mathcal{D}_{u}}\) and perform accurate clustering to classify each \(x_{i}\) in \(\{{\mathcal{D}_{l}}\cup{\mathcal{D}_{u}}\}\) into its corresponding category.

### Approach Overview

General GCD methods typically first extract representations \({\mathcal{Z}}=\{{\boldsymbol{z}}_{i}\}_{i=1}^{|{\mathcal{D}_{l}}\cup{ \mathcal{D}_{u}}|}\) via model \({\mathcal{M}}\) for each sample \(x_{i}\) and then perform K-Means to locate cluster centers \(\{{\boldsymbol{\mu}}_{i}\}_{i=1}^{K}\) for doing GCD. Our proposed ALUP builds upon existing GCD models and effectively incorporates LLMs' feedback in an active learning scheme.

Figure 2 depicts an overview of our ALUP framework for GCD. It encompasses three key designs: **Uncertainty Propagation** for sample selection, **Comparison-based Prompting** for soliciting LLMs' feedback, and **Soft Feedback Propagation** for wisely spreading the feedback. In what follows, we will detail these designs separately.

### Uncertainty Propagation (UP)

Within the ALUP framework, we design the uncertainty propagation to select the most informative unlabeled samples that are representative of high-uncertainty regions. Note that given a general GCD model \({\mathcal{M}}\), we can extract representations \({\boldsymbol{z}}_{i}\) for each \(x_{i}\) in the dataset and perform \(K\)-means to locate cluster centers \(\{{\boldsymbol{\mu}}_{k}\}_{k=1}^{K}\). To estimate the model predictive uncertainty, following Xie et al. (2016), we use the Student's \(t\)-distribution to compute the probability of assigning the sample \(x_{i}\) to each cluster \(k\):

\[q_{ik}=\frac{(1+\|{\boldsymbol{z}}_{i}-{\boldsymbol{\mu}}_{k}\|^{2}/\alpha)^{- \frac{\alpha+1}{2}}}{\sum_{k^{\prime}}(1+\|{\boldsymbol{z}}_{i}-{\boldsymbol{ \mu}}_{k^{\prime}}\|^{2}/\alpha)^{-\frac{\alpha+1}{2}}}, \tag{1}\]

where \(\alpha\) represents the degrees of freedom in the Student's \(t\)-distribution. After obtaining the model predictive probabilities, we use the entropy (Lewis and Gale, 1994) to measure the uncertainty for each sample \(x_{i}\):

\[u(x_{i})=-\sum_{k=1}^{K}q_{ik}\text{log}q_{ik}. \tag{2}\]

Here, a higher \(u(x_{i})\) can indicate a higher likelihood of the model \({\mathcal{M}}\) incorrectly assigning \(x_{i}\) to a 

wrong cluster. However, directly adopting this individual uncertainty score for selecting samples can lead to suboptimal outcomes as it can be sensitive to outliers (Karamcheti et al., 2021). To address this issue, following Yu et al. (2023), we further measure the similarities between each sample and its neighbors and propagate the individual uncertainty score to neighbors. Specifically, for each data point \(x_{i}\), we first find its \(k\)-nearest neighbors based on the Euclidean distance as:

\[\mathcal{N}(x_{i})=\underset{top-k}{\text{KNN}}(\mathbf{z}_{i},\mathcal{Z}^{u}), \tag{3}\]

where \(\mathcal{Z}^{u}\) denotes the representations of unlabeled samples and \(\mathcal{N}(x_{i})\) represents the set of nearest neighbors of \(x_{i}\). Then, we calculate the similarities between \(x_{i}\) and its neighbors based on the radial basis function (RBF) (Scholkopf et al., 1997):

\[sim(\mathbf{z}_{i},\mathbf{z}_{j})=\text{exp}(-\rho\|\mathbf{z}_{i}-\mathbf{z}_{j}\|_{2}^{2}), \tag{4}\]

where \(x_{j}\in\mathcal{N}(x_{i})\) and \(\rho\) is a hyper-parameter that regulates the extent of uncertainty propagation. After measuring the similarities, we refine the uncertainty score of sample \(x_{i}\) as:

\[u(x_{i})=u(x_{i})+\frac{\sum_{x_{j}\in\mathcal{N}(x_{i})}sim(\mathbf{z}_{i},\mathbf{ z}_{j})\cdot u(x_{j})}{|\mathcal{N}(x_{i})|}\,. \tag{5}\]

After several rounds of uncertainty score propagation, we obtain the final uncertainty score \(u(x_{i})\). Based on this score, we greedily select one sample \(x_{i}^{q}\) from each cluster \(c_{i}\) to form the sample set \(\mathcal{Q}\):

\[x_{i}^{q}=\underset{x_{j}\in c_{i}}{\text{argmax}}(u(x_{j})). \tag{6}\]

We emphasize that a sample will exhibit higher propagated uncertainty only when it and its neighboring samples both possess high uncertainty levels. Hence, we are selecting samples from uncertain regions. By actively obtaining feedback from LLMs for such samples in \(\mathcal{Q}\), we can significantly improve the model performance in GCD.

### Comparison-based Prompting (CP)

Upon identifying the most informative unlabeled samples through the UP strategy, we need to query LLMs to obtain pseudo category labels for these samples. However, since the category labels of newly emerged categories remain unknown, it is infeasible to request LLMs to directly generate possibly a brand new label for each selected sample. To overcome this, we design a comparison-based prompting method from the clustering perspective, which prompts LLMs to classify a sample by comparing it with other samples representing distinct categories.

This CP method requires the selection of a representative sample for each category cluster. To this end, we first compute the distances of various samples within the cluster to its center \(\mathbf{\mu}_{i}\), and then

Figure 2: The overall ALUP framework. It consists of three main designs: _Uncertainty Propagation_ for region-based sample selection, _Comparison-based Prompting_ for soliciting more accurate LLM’s feedback, and _Soft Feedback Propagation_ for wisely spreading the feedback to boost both efficiency and effectiveness.

 

select the sample closest to \({\mathbf{\mu}_{i}}\) to represent this cluster. We denote this close-to-center sample as \(\mu_{i}\). With these close-to-center samples \(\mathcal{S}=\left\{\mu_{i}\right\}^{K}_{i=1}\), we construct the prompt to query LLMs as:

Cluster [\(c_{1}\)]: Sample [\(\mu_{1}\)]; Cluster [\(c_{2}\)]: Sample [\(\mu_{2}\)]; ... ; Cluster [\(c_{p}\)]: Sample [\(\mu_{p}\)]. Above is a list of samples representing distinct categories. Please identify one sample that shares the same or similar underlying category as the input sample from the provided list.

Here, \(p\) is the number of representative samples used for the comparison. In our experiments, for each \(x^{q}_{i}\) in \(\mathcal{Q}\), we empirically incorporate \(p=|\mathcal{Q}|/2\) representative samples that are closest to \(x^{q}_{i}\) into the prompt. With this design, we can effectively utilize LLMs to classify the selected samples into their corresponding categories, denoted as \(\mathcal{Q}=\{x^{q}_{i},y^{LLM}_{i}\}^{K}_{i=1}\), thus bypassing the requirement for explicit labels of unknown categories.

### Soft Feedback Propagation (SFP)

By querying LLMs using the CP method, we can endow the selected unlabeled samples with their respective pseudo labels to augment the GCD models for discerning new categories. However, a performance gap persists between the partially and fully LLM-augmented GCD models. Given that the selection of the unlabeled samples is based on their model predictive uncertainty and neighboring uncertainty, and samples distributed close to each other are more likely to share the same category, we thus propose a soft feedback propagation mechanism to propagate the pseudo labels generated by LLMs across their similar neighbors, amplifying the utility of the feedback from LLMs without any additional cost. Specifically, for each \(x^{q}_{i}\) in \(\mathcal{Q}\), we refine the model prediction \({\mathbf{q}_{j}}\) of its uncertain neighbor \(x_{j}\in\mathcal{N}(x^{q}_{i})\) in Equation (1) to propagate the LLM-generated pseudo label \(y^{LLM}_{i}\):

\[{\mathbf{q}_{j}}=(1-sim({\mathbf{z}_{j}},{\mathbf{z}^{q}_{i}}))\cdot{\mathbf{q}_{ j}}+sim({\mathbf{z}_{j}},{\mathbf{z}^{q}_{i}})\cdot{\mathbf{y}^{LLM}}, \tag{7}\]

\[y^{prop}_{j}=\begin{cases}y^{LLM}_{i},&\text{if argmax}({\mathbf{q}_{j}})=y^{LLM}_{ i}\\ -1,&\text{otherwise}\end{cases}, \tag{8}\]

where \(sim(\cdot,\cdot)\) denotes the similarity function defined in Equation (4). \({\mathbf{y}^{LLM}}\) is a one-hot vector where the value of position \(y^{LLM}_{i}\) is set to 1. To interpret the Equation (8), we argue that when the uncertain neighbor \(x_{j}\in\mathcal{N}(x^{q}_{i})\) is assigned to the same cluster as the LLM-labeled sample \(x^{q}_{i}\) according to the refined \({\mathbf{q}_{j}}\), the pseudo label \(y^{LLM}_{i}\) will be propagated to the \(x_{j}\). Otherwise, the \(x_{j}\) will reject the pseudo label \(y^{LLM}_{i}\) and remain as an unlabeled sample.

### Model Optimization

After obtaining pseudo labels for the selected unlabeled samples in \(\mathcal{Q}\) from LLMs and propagating these labels via SFP, we update the model using a supervised contrastive learning loss Gao et al. (2021); Guo et al. (2022) as follows:

\[\small\mathcal{L}=\sum_{i=1}^{L^{\prime}}-\frac{1}{|\mathcal{N}^{ \prime}(x_{i})|}\underset{x_{j}\in\mathcal{N}^{\prime}(x_{i})}{\sum}\log\!\frac {e^{sim(x_{i}x_{j})/\tau}}{\sum_{k\neq i}e^{sim(x_{i}x_{k})/\tau}}~{}, \tag{9}\]

where \(L^{\prime}\) denotes the total number of labeled samples, including both the original labeled samples and the newly labeled samples obtained via the CP and SFP. \(\mathcal{N}^{\prime}(x_{i})\) is the set of samples sharing the same category label with \(x_{i}\). \(\tau\) is the temperature.

## 4 Experiments

### Datasets

We conduct experiments on three GCD datasets: **BANKING**Casanueva et al. (2020), **CLINC**Larson et al. (2019), and **StackOverflow**Xu et al. (2015). The detailed statistics are reported in Appendix A.1. In our experiments, we keep the same train, development, and test splits as previous work Liang and Liao (2023). More experimental details are provided in the Appendix A.2.

### Evaluation Metrics

Following Zhang et al. (2022); Liang and Liao (2023), we adopt the three metrics for evaluating the GCD performance: Accuracy (**ACC**) based on the Hungarian algorithm, Adjusted Rand Index (**ARI**), and Normalized Mutual Information (**NMI**). The specific definitions are presented in Appendix A.3. It is worth noting that **ACC** is regarded as the primary metric for evaluation, with higher values indicating better GCD performance.

### Baselines

We compare with the following SOTA GCD methods: **DTC**Han et al. (2019), **CDC+**Lin et al. ( 

2020), **DeepAligned**(Zhang et al., 2021b), **Prob-NID**(Zhou et al., 2023), **DCSC**(Wei et al., 2022), **MTP-CLNN**(Zhang et al., 2022a), **US-NID**(Zhang et al., 2023a), and the best-performing method CsePL**(Liang and Liao, 2023). We leave the details of these baselines in Appendix A.4.

### Main Results

#### 4.4.1 GCD Performance Comparison

Table 1 presents the main GCD results of our proposed ALUP against existing baselines, where the peak performance is highlighted in **bold**. Generally speaking, our ALUP consistently outperforms all existing baselines across three datasets by large margins. We analyze the results as follows:

Comparison of different methods in GCD:Table 1 reveals that ALUP significantly outperforms the existing leading baselines, such as CePL and USMID. For example, the proposed ALUP surpasses previous SOTA CsePL by margins of 2.51% in ACC, 2.12% in ARI, and 1.14% in NMI on BANKING-50%. Notably, the performance gains are more pronounced when a larger number of categories remain unknown. For example, ALUP's ACC improves by 3.55% on BANKING-25%. This proves that the ALUP can acquire effective supervision signals from LLMs, enhancing the model performance in discovering new categories.

Comparison of different datasets:We evaluate the performance of the ALUP framework on different datasets, including the single-domain, fine-grained BANKING dataset, and the multi-domain CLINC dataset. From Table 1, we can notice that all existing methods exhibit significantly lower performance on BANKING compared to CLINC, indicating that the single-domain fine-grained scenario is more challenging for GCD. However, ALUP achieves a more significant improvement of 1%\(\sim\)3% on BANKING-50% compared with the CsePL, while only 0.8%\(\sim\)2% on CLINC-50%. This observation further strengthens the benefits of our ALUP in providing effective supervision signals to cope with the challenges in fine-grained category discovery.

### In-depth Analyses

In this subsection, we conduct further detailed analyses to explore the impact of each key component within the proposed ALUP framework.

#### 4.5.1 Effect of Uncertainty Propagation

Table 2 presents the experimental results of removing the UP strategy in Equation (5) from ALUP on the BANKING dataset. It observes a significant reduction in GCD performance across various known category ratios upon removal. In particular, the ACC of the ALUP decreases by 1.20% while the ARI and NMI drop 1.34% and 0.64% on BANKING-25%, respectively. This indicates that the UP strategy can accurately identify the most informative samples for querying LLMs to boost the GCD model performance. It notably avoids selecting outliers with high model uncertainty but which are less beneficial for model learning.

#### 4.5.2 Effect of Soft Feedback Propagation

We also explore the contribution of the SFP mechanism by comparing the model performance when omitting the feedback propagation from LLMs in Equation (8) with the standard ALUP. Table 2 illustrates a notable decline in model performance in the absence of SFP, with a decrease of 1.88% in ACC, 1.67% in ARI, and 0.38% in NMI. Nevertheless, ALUP _w/o_ SFP still slightly outperforms the best-performing baseline CsePL. We suggest that this observation can be explained by two main points: (1) The acquisition of supervision signals from LLMs for the informative samples is beneficial for enhancing the model's capacity to discover new categories. (2) The SFP strategy can effectively propagate the accurate supervision signals from LLMs, amplifying the utility of LLM's feedback while concurrently minimizing the risk of propagating errors.

In contrast to the SFP strategy, we also investigate the Hard Propagation strategy within the proposed ALUP (ALUP \(w\) HP), where LLMs' feedback is directly extended to the neighboring samples without any control. As presented in Table 2, we can observe that the model performance significantly decreases using the hard propagation, descending even below the levels achieved by CsePL. This is probably due to the propagation of inaccurate supervision signals from LLMs, which introduces considerable noise into the model learning 



#### 4.5.3 Number of Propagated Neighbors

To delve deeper into the effectiveness of the UP strategy within the ALUP framework, we conduct further experiments on the BANKING dataset to explore the effect of varying the number of propagated neighbors in unlabeled sample selection on the model performance. Figure 3 illustrates performance trends across different counts of propagated neighbors. Notably, as the number of propagated neighbors in Equation (3) increases, ALUP's performance improves, reaching an optimum with 25 propagated neighbors. Beyond this point, the model performance begins to decline. We hypothesize that this decrease might be attributed to the inclusion of samples with lower uncertainty, which potentially introduces significant noise into the process of unlabeled sample selection.

#### 4.5.4 Effect of Representative Samples

To assess the effectiveness of the CP method, we examine the impact of varying the number \(p\) of representative samples integrated into the prompt for querying LLMs on the model performance. Experiments are conducted with \(p\) values set at [19, 38, 57, 77], where 19 denotes about a quarter of the total cluster count. As detailed in Table 3, the optimal

\begin{table}
\begin{tabular}{c|l c c c c c c c c} \hline \hline \multirow{2}{*}{KCR} & \multirow{2}{*}{Methods} & \multicolumn{6}{c}{**BANKING**} \\ \cline{3-10}  & & **ACC** & ARI & NMI & **ACC** & ARI & NMI & **ACC** & ARI & NMI \\ \hline \multirow{4}{*}{25\%} & DTC & 31.75 & 19.09 & 55.59 & 56.90 & 41.92 & 79.35 & 29.54 & 17.51 & 29.96 \\  & CDAC+ & 48.00 & 33.74 & 66.39 & 66.24 & 50.02 & 84.68 & 51.61 & 30.99 & 46.16 \\  & DeepAligned & 49.08 & 37.62 & 70.50 & 74.07 & 64.63 & 88.97 & 54.50 & 37.96 & 50.86 \\  & ProbNID & 55.75 & 44.25 & 74.37 & 71.56 & 63.25 & 89.21 & 54.10 & 38.10 & 53.70 \\  & DCSC & 60.15 & 49.75 & 78.18 & 79.89 & 72.68 & 91.70 & - & - & - \\  & MTP-CLNN & 65.06 & 52.91 & 80.04 & 83.26 & 76.20 & 93.17 & 74.70 & 54.80 & 73.35 \\  & USINID & 65.85 & 56.53 & 81.94 & 83.12 & 77.95 & 94.17 & 75.76 & **65.45** & 74.91 \\  & CsePL & 71.06 & 60.36 & 83.32 & 86.16 & 79.65 & 94.07 & 79.47 & 64.92 & 74.88 \\  & ALUP & **74.61** & **62.64** & **84.06** & **88.40** & **82.44** & **94.84** & **82.20** & 64.54 & **76.58** \\ \hline \multirow{4}{*}{50\%} & DTC & 49.85 & 37.05 & 69.46 & 64.39 & 50.44 & 83.01 & 52.92 & 37.38 & 49.80 \\  & CDAC+ & 48.55 & 34.97 & 67.30 & 68.01 & 54.87 & 86.00 & 51.79 & 30.88 & 46.21 \\  & DeepAligned & 59.38 & 47.95 & 76.67 & 80.70 & 72.56 & 91.59 & 74.52 & 57.62 & 68.28 \\  & ProNID & 63.02 & 50.42 & 77.95 & 82.62 & 75.27 & 92.72 & 73.20 & 62.46 & 74.54 \\  & DCSC & 68.30 & 56.94 & 81.19 & 84.57 & 78.82 & 93.75 & - & - & - \\  & MTP-CLNN & 70.97 & 60.17 & 83.42 & 86.18 & 80.17 & 94.30 & 80.36 & 62.24 & 76.66 \\  & USNID & 73.27 & 63.77 & 85.05 & 87.22 & 82.87 & 95.45 & 82.06 & 71.63 & 78.77 \\  & CsePL & 76.94 & 66.66 & 85.65 & 88.66 & 83.14 & 95.09 & 85.68 & 71.99 & 80.28 \\  & ALUP & **79.45** & **68.78** & **86.79** & **90.53** & **84.84** & **95.97** & **86.70** & **73.85** & **81.45** \\ \hline \hline \end{tabular}
\end{table}
Table 1: Main performance results on the generalized category discovery across three public datasets. KCR denotes the known category rate.

Figure 3: Effect of the number of propagated neighbors.



GCD performance is achieved by integrating 38 representative samples into the prompt to acquire supervision signals from LLMs for the unlabeled samples. We suggest that the main reasons for this observation come from two aspects: (1) A smaller \(p\) may potentially omit the representative samples sharing the same underlying category as the selected samples, possibly limiting LLMs' ability to offer the requisite supervision signals during comparisons with the integrated representative samples. (2) Conversely, incorporating a larger number of representative samples for the CP method results in an extended prompt length. This could lead LLMs to misclassify the chosen unlabeled samples into inaccurate categories, thereby negatively affecting the model's performance.

In our standard approach to the CP method, we select the single closest-to-center sample within each cluster as the representative for constructing prompts to query LLMs. Expanding our investigation into the CP method, we experiment with an alternative strategy involving a close-to-center set--specifically, the top 3 samples nearest to the cluster center--to represent distinct clusters for prompting LLMs to determine pseudo category labels. As illustrated in Table 4, the experimental results on BANKING-25% demonstrate marginal gains with this strategy, achieving an increase of no more than 0.5% across all three metrics. Nonetheless, it necessitates an increased querying cost with LLMs. Balancing the slight improvement in performance against the rise in costs, we thus opt for the more straightforward and cost-effective strategy of utilizing single closest-to-center samples within the CP method.

### Impact of Different Base GCD Models

In our experiments, we select the most informative unlabeled samples based on the existing GCD models. To validate the effectiveness of the proposed ALUP, we also examine how its performance varies when different GCD models are integrated within ALUP on the BANKING-50% dataset. As depicted in Figure 4, we can observe consistent and significant improvements with the proposed ALUP. This demonstrates that the proposed ALUP framework is effective in acquiring supervision signals from LLMs to enhance the model performance of discovering new categories and is adaptable to other GCD models.

### Influence of Query Sample Number

We study the effect of varying the number of selected unlabeled samples for querying LLMs in Figure 5. It is observed that there is an increase in model performance corresponding to the rise in the number of samples selected for querying LLMs. Yet, this growth rate progressively diminishes as the LLMs' feedback is propagated, and selecting informative samples becomes more challenging with the increasing number of selected samples.

\begin{table}
\begin{tabular}{l|c c c} \hline \hline \multirow{2}{*}{Methods} & \multicolumn{3}{c}{**BANKING**} \\ \cline{2-4}  & **ACC** & ARI & NMI \\ \hline ALUP-_standard_ & 74.61 & 62.64 & 84.06 \\ ALUP-_close-to-center set_ & **74.87** & **63.07** & **84.39** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance of representative sample selection strategies within the CP method on BANKING-25%.

Figure 4: Performances of various base GCD models in ALUP on the BANKING-50%.

 



### Effect of Different LLMs

We also examine the performance impact of utilizing different LLMs within our ALUP. Specifically, we conduct experiments on BANKING-25%, comparing the performance of the closed-source _gpt-3.5-turbo_ against the open-sourced _FlanT5-XXL_ in deriving supervision signals. As shown in Table 5, the experimental results illustrate a marginal performance decrease when employing _FlanT5-XXL_ compared to _gpt-3.5-turbo_. Despite this, the use of _FlanT5-XXL_ still markedly outperforms the best-performing baseline CsePL, highlighting the adaptability of our ALUP to various LLMs.

Furthering our exploration into the mechanisms of LLMs' utilization in GCD, we evaluate the efficacy of our CP method against an alternative approach based on embedding similarity scores. For this comparison, we leverage the embedding model _text-embedding-3-small_ from OpenAI to generate embeddings for both uncertain samples and cluster-representative samples, calculating their similarity scores to determine pseudo category labels. As reported in Table 5, the results demonstrate a drop in performance metrics using the embedding score method, underscoring the rationale of our CP method and its proficiency in capturing the nuanced semantic relationships essential for GCD.

## 5 Conclusion

In summary, our ALUP framework innovatively integrates Large Language Models with uncertainty propagation in generalized category discovery, marking a significant leap in the field. By employing comparison-based LLM prompting and a novel soft feedback propagation mechanism, ALUP adeptly identifies and categorizes new data with enhanced accuracy and efficiency. This approach not only surpasses traditional GCD methods but also minimizes the risk of error propagation, a critical advancement in handling real-world, dynamic datasets with LLMs. Future endeavors will focus on refining LLM integration, extending our methods to multi-modal data, and enhancing scalability and data privacy measures, furthering ALUP's potential in diverse and evolving open-world computing.

This research is supported by the Ministry of Education, Singapore, under its AcRF Tier 2 Funding (Proposal ID: T2EP20123-0052). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the Ministry of Education, Singapore.

## Limitations

While our ALUP framework marks a significant advance in Generalized Category Discovery using LLMs, it does have some limitations. The reliance on LLMs can introduce biases and inaccuracies, particularly in areas where these models have limited training data or exposure. Although our propagation method effectively reduces overall costs, the initial computational demands of LLMs may still pose scalability challenges, especially for resource-limited environments. Additionally, the framework currently focuses on textual data, which could limit its applicability in multi-modal data scenarios. Moreover, while our soft feedback propagation mechanism aims to minimize error spread, it is not immune to the risk of amplifying initial inaccuracies from LLM feedback. Finally, data privacy and security remain critical concerns in the use of external LLMs, necessitating ongoing vigilance and adaptation.

Figure 5: Effect of the number of query samples on BANKING-50%.

\begin{table}
\begin{tabular}{l|c c c} \hline \multirow{2}{*}{Methods} & \multicolumn{4}{c}{**BANKING**} \\ \cline{2-4}  & **ACC** & ARI & NMI \\ \hline ALUP-_gpt-3.5-turbo_ & **74.61** & **62.64** & **84.06** \\ ALUP-_FlanT5-XXL_ & 73.38 & 62.29 & 83.76 \\ ALUP-_text-embedding_ & 71.95 & 61.48 & 83.7 \\ \hline \end{tabular}
\end{table}
Table 5: Effect of different LLMs.





## References

* An et al. (2023) Wenbin An, Feng Tian, Qinghua Zheng, Wei Ding, Qianying Wang, and Ping Chen. 2023. Generalized category discovery with decoupled prototypical network. In _AAAI_, pages 12527-12535.
* Caron et al. (2018) Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsupervised learning of visual features. In _ECCV_, pages 139-156.
* Casanueva et al. (2020) IInigo Casanueva, Tadas Temcinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Efficient intent detection with dual sentence encoders. In _NLP4ComAI@ACL_, pages 38-45.
* Cheng et al. (2023) Qinyuan Cheng, Xiaogui Yang, Tianxiang Sun, Linyang Li, and Xipeng Qiu. 2023. Improving contrastive learning of sentence embeddings from AI feedback. In _Findings of ACL_, pages 11122-11138.
* Chowdhery et al. (2019) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhinsek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodokumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Lian Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalyayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katnerine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Castata, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023. Palm: Scaling language modeling with pathways. _J. Mach. Learn. Res._, pages 240:1-240:113.
* Citovsky et al. (2021) Gui Citovsky, Giulia DeSalvo, Claudio Gentile, Lazeros Karydas, Anand Rajagopalan, Afshin Rostamizadeh, and Sanjiv Kumar. 2021. Batch active learning at scale. In _NeurlPS_, pages 11933-11944.
* Cuturi (2013) Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. In _NeurIPS_, pages 2292-2300.
* de Raedt et al. (2023) Maarten De Raedt, Frederic Godin, Thomas Demeester, and Chris Develder. 2023. IDAS: Intent discovery with abstractive summarization. In _NLP4ConvAI@ACL_, pages 71-88.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimceS: Simple contrastive learning of sentence embeddings. In _EMNLP_, pages 6894-6910.
* Gissin and Shalev-Shwartz (2019) Daniel Gissin and Shaiev-Shwartz. 2019. Discriminative active learning. _CoRR_, abs/1907.06347.
* Guo et al. (2022) Shasha Guo, Jing Zhang, Yanling Wang, Qianyi Zhang, Cuiping Li, and Hong Chen. 2022. DSM: Question generation over knowledge base via modeling diverse subgraphs with meta-learner. In _EMNLP_, pages 4194-4207.
* Hadifar et al. (2019) Amir Hadifar, Lucas Sterckx, Thomas Demeester, and Chris Develder. 2019. A self-training approach for short text clustering. In _RepL4NLP@ACL_, pages 194-199.
* Han et al. (2019) Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2019. Learning to discover novel visual categories via deep transfer clustering. In _ICCV_, pages 8400-8408.
* Hsu et al. (2018) Yen-Chang Hsu, Zhaoyang Lv, and Zsolt Kira. 2018. Learning to cluster in order to transfer across domains and tasks. In _ICLR_.
* Hsu et al. (2019) Yen-Chang Hsu, Zhaoyang Lv, Joel Schlosser, Phillip Odom, and Zsolt Kira. 2019. Multi-class classification without multi-class labels. In _ICLR_.
* Karamechi et al. (2021) Siddharth Karamechi, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. 2021. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. In _ACL-IJCNLP_, pages 7265-7281.
* Larson et al. (2019) Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of-scope prediction. In _EMNLP-IJCNLP_, pages 1311-1316.
* Lewis and Gale (1994) David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In _SIGIR_, pages 3-12.
* Liang and Liao (2023) Jinggui Liang and Lizi Liao. 2023. ClusterPrompt: Cluster semantic enhanced prompt learning for new intent discovery. In _Findings of EMNLP_, pages 10468-10481.
* Liao et al. (2023) Lizi Liao, Grace Hui Yang, and Chirag Shah. 2023. Proactive conversational agents in the post-chatgpt world. In _SIGIR_, pages 3452-3455.
* Lin et al. (2020) Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Discovering new intents via constrained deep adaptive clustering with cluster refinement. In _AAAI_, pages 8360-8367.
* Liu et al. (2018) Ming Liu, Wray L. Buntine, and Gholamreza Haffari. 2018. Learning how to actively learn: A deep imitation learning approach. In _ACL_, pages 1874-1883

Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. 2023. On learning to summarize with large language models as references. _CoRR_, abs/2305.14239.
* MacQueen et al. (1967) James MacQueen et al. 1967. Some methods for classification and analysis of multivariate observations. In _Proceedings of the fifth Berkeley symposium on mathematical statistics and probability_, volume 1, pages 281-297.
* Margatina et al. (2023) Katerina Margatina, Timo Schick, Nikolaos Aletras, and Jane Dwivedi-Yu. 2023. Active learning principles for in-context learning with large language models. In _Findings of EMNLP_, pages 5011-5034.
* Mou et al. (2022) Yutao Mou, Keqing He, Pei Wang, Yanan Wu, Jinggang Wang, Wei Wu, and Weiran Xu. 2022. Watch the neighbors: A unified k-nearest neighbor contrastive learning framework for OOD intent discovery. In _EMNLP_, pages 1517-1529.
* Mou et al. (2023) Yutao Mou, Xiaoshuai Song, Keqing He, Chen Zeng, Pei Wang, Jingang Wang, Yunsen Xian, and Weiran Xu. 2023. Decoupling pseudo label disambiguation and representation learning for generalized intent discovery. In _ACL_, pages 9661-9675.
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _CoRR_, abs/2303.08774.
* Pandrasundari and Bangalore (2018) Padmasundari and Srinivas Banglore. 2018. Intent discovery through unsupervised semantic text clustering. In _INTERSPEECH_, pages 606-610.
* Ren et al. (2022) Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. 2022. A survey of deep active learning. _ACM Comput. Surv._, pages 180:1-180:40.
* Scholkopf et al. (1997) Bernhard Scholkopf, Kah Kay Sung, Christopher J. C. Burges, Federico Girosi, Partha Niyogi, Tomasso A. Poggio, and Vladimir Vapnik. 1997. Comparing support vector machines with gaussian kernels to radial basis function classifiers. _IEEE Trans. Signal Process._, pages 2758-2765.
* Schroder et al. (2022) Christopher Schroder, Andreas Niekler, and Martin Potthast. 2022. Revisiting uncertainty-based query strategies for active learning with transformers. In _Findings of ACL_, pages 2194-2203.
* Sener and Savarese (2018) Ozan Sener and Silvio Savarese. 2018. Active learning for convolutional neural networks: A core-set approach. In _ICLR_.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Vaze et al. (2022) Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. 2022. Generalized category discovery. In _CVPR_, pages 7482-7491.
* Viswanathan et al. (2023) Vijay Viswanathan, Kiril Gastheovski, Carolin Lawrence, Tongshuang Wu, and Graham Neubig. 2023. Large language models enable few-shot clustering.
* Wang and Shang (2014) Dan Wang and Yi Shang. 2014. A new active labeling method for deep learning. In _IJCNN_, pages 112-119.
* Wei et al. (2022) Feng Wei, Zhenbo Chen, Zhenghong Hao, Fengxin Yang, Hua Wei, Bing Han, and Sheng Guo. 2022. Semi-supervised clustering with contrastive learning for discovering new intents. _arXiv preprint arXiv:2201.07604_.
* Wu et al. (2024) Yukia Wu, Tianhao Dai, Zhedong Zheng, and Lizi Liao. 2024. Active discovering new slots for task-oriented conversation. _TASLP_, pages 1-11.
* Wu et al. (2022) Yukia Wu, Lizi Liao, Xueming Qian, and Tat-Seng Chua. 2022. Semi-supervised new slot discovery with incremental clustering. In _EMNLP Findings_, pages 6207-6218.
* Xiao et al. (2023) Ruixuan Xiao, Yiwen Dong, Junbo Zhao, Runze Wu, Minmin Lin, Gang Chen, and Haobo Wang. 2023. FreeAL: Towards human-free active learning in the era of large language models. In _EMNLP_, pages 14520-14535.
* Xie et al. (2016) Junyuan Xie, Ross B. Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In _ICML_, pages 478-487.
* Xu et al. (2015) Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural networks. In _VS@HLT-NAACL_, pages 62-69.
* Yang et al. (2017) Bo Yang, Xiao Fu, Nicholas D. Sidiropoulos, and Mingyi Hong. 2017. Towards k-means-friendly spaces: Simultaneous deep learning and clustering. In _ICML_, pages 3861-3870.
* Yang et al. (2021) Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2021. Generalized out-of-distribution detection: A survey. _CoRR_, abs/2110.11334.
* Yu et al. (2023) Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang. 2023. Cold-start data selection for better few-shot language model fine-tuning: A prompt-based uncertainty propagation approach. In _ACL_, pages 2499-2521.
* Zeng et al. (2022)Weihao Zeng, Keqing He, Zechen Wang, Dayuan Fu, Guanting Dong, Ruotong Geng, Pei Wang, Jinggang Wang, Chaobo Sun, Wei Wu, and Weiran Xu. 2022. Semi-supervised knowledge-grounded pre-training for task-oriented dialog systems. In _SeretOD_, pages 39-47.

 

Xueying Zhan, Qingzhong Wang, Kuan-Hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B. Chan. 2022. A comparative survey of deep active learning. _CoRR_, abs/2203.13450.
* Zhang et al. (2021a) Hanlei Zhang, Xiaoteng Li, Hua Xu, Panpan Zhang, Kang Zhao, and Kai Gao. 2021a. TEXTOIR: An integrated and visualized platform for text open intent recognition. In _ACL-IJCNLP_, pages 167-174.
* Zhang et al. (2021b) Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021b. Discovering new intents with deep aligned clustering. In _AAAI_, pages 14365-14373.
* Zhang et al. (2023a) Hanlei Zhang, Hua Xu, Xin Wang, Fei Long, and Kai Gao. 2023a. A clustering framework for unsupervised and semi-supervised new intent discovery. _IEEE TKDE_, page 1-14.
* Zhang et al. (2023b) Ruoyu Zhang, Yanzeng Li, Yongliang Ma, Ming Zhou, and Lei Zou. 2023b. Llmaaa: Making large language models as active annotators. In _Findings of EMNLP_, pages 13088-13103.
* Zhang et al. (2023c) Yuwei Zhang, Zihan Wang, and Jingbo Shang. 2023c. Clusterllm: Large language models as a guide for text clustering. In _EMNLP_, pages 13903-13920.
* Zhang et al. (2022a) Yuwei Zhang, Haode Zhang, Li-Ming Zhan, Xiao-Ming Wu, and Albert Lam. 2022a. New intent discovery with pre-training and contrastive learning. In _ACL_, pages 256-269.
* Zhang et al. (2022b) Zhisong Zhang, Emma Strubell, and Eduard H. Hovy. 2022b. A survey of active learning for natural language processing. In _EMNLP_, pages 6166-6190.
* Zhong et al. (2021) Zhuhong, Enrico Fini, Subhankar Roy, Zhiming Luo, Elisa Ricci, and Nicu Sebe. 2021. Neighborhood contrastive learning for novel class discovery. In _CVPR_, pages 10867-10875.
* Zhou et al. (2023) Yunhua Zhou, Guofeng Quan, and Xipeng Qiu. 2023. A probabilistic framework for discovering new intents. In _ACL_, pages 3771-3784.

 

Appendix

### Dataset Statistics

We show the detailed statistics of BANKING, CLINC and StackOverflow datasets in Table 6. Specifically, BANKING is a fine-grained category discovery dataset collected from user dialogues in the banking domain. It contains over 13K user utterances that span over 77 distinct categories. CLINC is a multi-domain dataset, which encompasses 150 distinct categories and 22,500 utterances across 10 domains. StackOverflow is a technical question dataset collected from Kaggle.com, which includes 20K questions with 20 categories.

### Implementation Details

For the dataset setup, following Zhang et al. (2023), we randomly select a specified ratio {25%, 50%} of categories, denoted as known category rate (KCR), to serve as known categories. For each known category, 10% of labeled samples are selected to constitute a labeled dataset \(\mathcal{D}_{l}\), while the remaining samples are deemed as unlabeled data, forming the unlabeled dataset \(\mathcal{D}_{u}\).

For the Uncertainty Propagation, we set the freedom \(\alpha\) in Equation (1) to 1.0. The number of propagated neighbors is specifically set to 25 for all datasets. The \(\rho\) for calculating similarities in Equation (4) is set to 1.0.

For the Comparison-based Prompting, we employ the _gpt-3.5-turbo_ as the basic LLM in our experiments. While acquiring supervision signals, the temperature is set to 0 for deterministic outputs, and the maximum tokens are constrained to 256. The default values are retained for the rest of the parameters. The number of representative samples is specifically set to 38 for the BANKING dataset, 75 for the CLINC dataset, and 20 for the StackOverflow dataset.

### Evaluation Metrics

In the experiments, we employ three standard evaluation metrics: ACC, ARI, and NMI to evaluate the GCD performance. Specifically, ACC measures the performance of GCD by comparing the predicted labels with the ground-truth labels. The definition of ACC is as follows:

\[ACC=\frac{\sum_{i=1}^{N}\mathbbm{1}_{y_{i}=map(\hat{y_{i}})}}{N}\]

where \(\{\hat{y_{i}},y_{i}\}\) denote the predicted label and the ground-truth label for a given sample \(x_{i}\) respectively. \(map(\cdot)\) is a mapping function that maps each predicted label \(\hat{y_{i}}\) to its corresponding ground-truth label \(y_{i}\) by Hungarian algorithm.

ARI calculates the similarity between the predicted and ground-truth clusters, assessing the accuracy of clustering on a pairwise basis. ARI is defined as:

\[ARI=\frac{\sum_{i,j}\binom{n_{ij}}{2}-[\sum_{i}\binom{u_{i}}{2}\sum_{i}\binom{ v}{2}]/\binom{N}{2}}{\frac{1}{|\sum_{i}\binom{u_{i}}{2}+\sum_{j}\binom{v}{2}|-[ \sum_{i}\binom{u_{i}}{2}\sum_{j}\binom{v_{j}}{2}]/\binom{N}{2}}}\]

where \(u_{i}=\sum_{j}n_{i,j}\), and \(v_{j}=\sum_{i}n_{i,j}\). \(N\) denotes the number of all samples. \(n_{i,j}\) is the number of sample pairs that are both assigned to \(i^{th}\) predicted cluster and \(j^{th}\) ground-truth cluster.

NMI computes the normalized mutual information to quantify the agreement between the predicted and ground-truth clusters, providing a measure of clustering consistency. It can be calculated as follows:

\[NMI(\hat{\mathbf{y}},\mathbf{y})=\frac{2\cdot I(\hat{\mathbf{y}},\mathbf{y})}{H(\hat{\mathbf{y}}) +H(\mathbf{y})}\]

where \(\{\hat{\mathbf{y}},\mathbf{y}\}\) denote the predicted labels and the ground-truth labels respectively. \(I(\hat{\mathbf{y}},\mathbf{y})\) is the mutual information between \(\hat{\mathbf{y}}\) and \(\mathbf{y}\). \(H(\cdot)\) represents the entropy function.

### Baselines

In this work, we compare the proposed ALUP with the following representative baselines:

* **DTC**Han et al. (2019): A semi-supervised deep clustering approach with a novel mechanism for estimating the number of intents based on labeled data.
* **CDAC+**Lin et al. (2020): A pseudo-labeling approach that employs pairwise constraints and a target distribution as guiding factors in the learning of new categories.
* **DeepAligned**Zhang et al. (2021): A semi-supervised approach that addresses the clustering inconsistency problem by using an alignment strategy for learning utterance embeddings.

\begin{table}
\begin{tabular}{c c c c} \hline \hline Dataset & Domain & Categories & Utterances \\ \hline BANKING & banking & 77 & 13,083 \\ CLINC & multi-domain & 150 & 22,500 \\ StackOverflow & question & 20 & 20,000 \\ \hline \hline  & & & \\ \end{tabular}
\end{table}
Table 6: Statistics of datasets used in the experiments.

 

* **ProbNID**(Zhou et al., 2023): A probabilistic framework that capitalizes on the expectation-maximization algorithm, conceptualizing intent assignments as probable latent variables.
* **DCSC**(Wei et al., 2022): A pseudo-labeling method involving the dual-task, which uses the SwAV algorithm and Sinkhorn-Knopp Cuturi (2013) to assign soft clusters.
* **MTP-CLNN**(Zhang et al., 2022a): A two-stage method that enhances representation learning via a multi-task pre-training and a nearest neighbor contrastive learning for identifying new categories.
* **USNID**(Zhang et al., 2023a): A framework supports both unsupervised and semi-supervised new intent discovery, incorporating an effective centroid initialization strategy designed to learn cluster representations by utilizing historical clustering information.
* **CsePL**(Liang and Liao, 2023): A method that utilizes two-level contrastive learning with label semantic alignment to enhance the cluster semantics and a soft prompting strategy for discovering new intents.

We re-run the released code of ProbNID to get its results. The other baselines' results are retrieved from Zhang et al. (2023a).

## Appendix B Estimate the Category Number K

In the complex task of generalized category discovery in real-world scenarios, accurately predicting the total number of categories, represented as \(K\), remains a significant challenge. Drawing from the methodologies proposed by Zhang et al. (2021b), our research leverages pre-initialized intent features to determine \(K\) autonomously. We begin by assigning an initially large number of clusters, \(K^{\prime}\), and then utilize a refined model to extract feature representations from our training dataset. These representations are grouped into distinct clusters using the K-means algorithm. Clusters that are densely populated and demonstrate well-defined boundaries are recognized as valid category clusters. Conversely, smaller, less distinct clusters are considered less relevant and subsequently discarded. The selection criteria for this process can be outlined as follows.

\[K=\sum_{i=1}^{K^{\prime}}\delta(|S_{i}|>\rho),\]

where \(|S_{i}|\) is the \(i\)-th grouped cluster size, \(\rho\) is the filtering threshold. \(\delta(\cdot)\) denotes the indicator function, whose output is 1 if the condition is satisfied.

Experimental results are reported in Table 7. The comparative results show that the proposed ALUP incurs only a minor performance decline with the predicted category number. This indicates that our ALUP exhibits robustness in handling inaccurately predicted category number.

\begin{table}
\begin{tabular}{c|l c c c} \hline \hline \multirow{2}{*}{Cluster Num} & \multirow{2}{*}{Methods} & \multicolumn{4}{c}{Banking77} \\ \cline{3-5}  & & ACC & ARI & NMI \\ \hline \multirow{2}{*}{\(K=77\) (gold)} & USNID & 65.85 & 56.53 & 81.94 \\  & CsePL & 71.06 & 60.36 & 83.32 \\  & ALUP & **74.61** & **62.64** & **84.06** \\ \hline \multirow{2}{*}{\(K=74\) (predicted)} & USNID & 60.72 & 49.18 & 78.11 \\  & CsePL & 69.75 & 56.70 & 81.30 \\ \cline{1-1}  & ALUP & **72.55** & **61.04** & **82.78** \\ \hline \hline \end{tabular}
\end{table}
Table 7: Effect of estimating cluster number \(K\).