Towards Real-world Scenario: Imbalanced New Intent Discovery (2024)

Abstract: New Intent Discovery (NID) aims at detecting known and previously undefined categories of user intent by utilizing limited labeled and massive unlabeled data. Most prior works often operate under the unrealistic assumption that the distribution of both familiar and new intent classes is uniform, overlooking the skewed and long-tailed distributions frequently encountered in real-world scenarios. To bridge the gap, our work introduces the imbalanced new intent discovery (i-NID) task, which seeks to identify familiar and novel intent categories within long-tailed distributions. A new benchmark (ImbaNID-Bench) comprised of three datasets is created to simulate the real-world long-tail distributions. ImbaNID-Bench ranges from broad cross-domain to specific single-domain intent categories, providing a thorough representation of practical use cases. Besides, a robust baseline model ImbaNID is proposed to achieve cluster-friendly intent representations. It includes three stages: model pre-training, generation of reliable pseudo-labels, and robust representation learning that strengthens the model performance to handle the intricacies of real-world data distributions. Our extensive experiments on previous benchmarks and the newly established benchmark demonstrate the superior performance of ImbaNID in addressing the i-NID task, highlighting its potential as a powerful baseline for uncovering and categorizing user intents in imbalanced and long-tailed distributions1.

heading: 1 Introduction
text: New intent discovery (NID) has captured increasing attention due to its adaptability to the evolving user needs in open-world scenarios (Mou et al., 2022; Siddique et al., 2021; Yang et al., 2020; Chrabrowa et al., 2023; Raedt et al., 2023). NID
*Corresponding author. 1https://github.com/Zkdc/i-NID
methods generally follow a two-stage training process, including a knowledge transfer and a discovery stage. The prior knowledge is injected into the model via pre-training and then the discriminative representation is learned for known and novel intent categories (Zhang et al., 2021a, 2022; Zhou et al., 2023; Zhang et al., 2023b; Shi et al., 2023).
Despite the considerable advancements in NID, there remain two salient challenges impeding adoption in practical scenarios. In Fig. 1, most NID approaches predominantly address the issue of intent discovery within the framework of balanced datasets. But the distribution of intents often follows a long-tailed pattern (Mou et al., 2022), particularly in dialogue systems, wherein a small number of intents are highly represented and a wide variety of intents (unknown intents) are sparsely exemplified. Secondly, NID methods suffer from severe clustering degradation, where lack of improved methods for unbalanced data distributions and leading to poor performance in unbalanced scenarios. Therefore, we explore the new methods under the Imbalanced New Intent Discovery (i-NID) task to bridge the gap between the NID and real-world applications.
To break out the aforementioned limitations, we propose a novel framework ImbaNID, which includes three key components: model pre-training,
3949
reliable pseudo-labeling (RPL), and robust representation learning (RRL). Specifically, the multitask pre-training incorporates the generalized prior knowledge into the mode for establishing a robust representational foundation conducive to clustering known and novel intents. The RPL component formulates the pseudo-label generation as a relaxed optimal transport problem, applying adaptive constraints to recalibrate the class distribution for enhanced uniformity. The model bias issues can be mitigated in long-tail settings while furnishing reliable supervisory cues for downstream representation learning. Then, a novel distribution-aware and quality-aware noise regularization technique is introduced in RRL to effectively distinguish between clean and noisy samples. A contrastive loss function is subsequently used to facilitate the formation of distinct and well-separated clusters of representations for known and novel intent categories. The collaborative synergy between RPL and RRL fosters an iterative training process to create a symbiotic relationship. This iterative approach cultivates intent representations conducive to clustering, significantly aiding the i-NID task. For better evaluation of unbalanced distribution, we introduce a comprehensive benchmark ImbaNIDBench for i-NID evaluation.
Extensive experiments of ImbaNID are evaluated on the previous common benchmarks and our proposed benchmark ImbaNID-Bench. The results demonstrate that ImbaNID consistently achieves state-of-the-art performance across all clusters, notably surpassing standard NID models by an average margin of 2.7% in long-tailed scenarios. The contributions are summarized as follows:
• We introduce the imbalanced new intent discovery (i-NID) task, which first encapsulates the challenges of clustering known and novel intent classes within long-tailed distributions. Different model performances under unbalanced distribution are sufficiently explored.
• We construct three comprehensive i-NID datasets to facilitate further advancements in i-NID research. Our extensive experiments on these datasets validate the superiority of the proposed method ImbaNID.
• For i-NID, we develop a novel ImbaNID approach that iteratively enhances pseudo-label generation and representation learning to ensure cluster-adaptive intent representations.

heading: 2 Datasets
text: We introduce a new benchmark (called ImbaNIDBench) for NID evaluation tailored to long-tail distribution scenarios, which comprises three datasets named CLINC150-LT, BANKING77-LT, and StackOverflow20-LT, derived from CLINC (Larson et al., 2019), BANKING (Casanueva et al., 2020) and StackOverflow (Xu et al., 2015). Comprehensive statistics for each dataset are documented in Appendix B. Here, we describe the details of the ImbaNID-Bench datasets.
Data Construction The first step is to simulate the long-tail distribution frequently encountered in real-world scenarios (Cui et al., 2019). Each class is assigned an index i (1 ≤ k ≤ K), where K denotes the total number of intent categories. γ = nmaxnmin denotes the imbalance ratio, where nk denotes the data size of class k, nmax = max1≤k≤K(nk), and nmin = min1≤k≤K(nk). We sample from each class based on nk = nmaxγ(j−1)/K . To explore the impact of data imbalance in NID, we construct ImbaNIDBench by sampling with diverse imbalance ratios γ ∈ {3, 5, 10}. Fig. 2 shows the datasets created for CLINC150-LT with different imbalance factors (More details can be found in Appendix B). To simulate an open-world NID setting. We randomly select 75% of intents as known intents, and sample only 10% instances from known intent categories to form a labeled subset, while the remaining in-
stances are treated as unlabeled data.
Data Statistics Since different proportions of imbalance ratios γ have different statistics, here we only display the results of γ = 10 for brevity. Table 1 shows the statistics of CLINC150-LT, BANKING77-LT and StackOverflow20-LT. We will release these datasets for future research.

heading: 3 Methodology

heading: 3.1 i-NID
text: Supposing we have a set of labeled intent data Dl = {(xi, yi)|yi ∈ Yk} only comprised of known intent categories Yk, the deployed model in the wild may encounter inputs from unlabeled data Du = {xi|yi ∈ {Yk,Yn}}.The unlabeled data Du contains both known intent categories Yk and novel intent categories Yn, where Yk and Yn denote the data with the Known and Novel intents data, respectively. Both Dl and Du present a long-tail distribution with imbalance ratio γ > 1. The goal of i-NID is to classify known classes and cluster novel intent classes in Du by leveraging Dl. Finally, model performance will be evaluated on a balanced testing set Dt = {(xi, yi)|yi ∈ {Yk,Yn}}.

heading: 3.2 Overall Framework
text: To achieve the learning objective of i-NID, we propose an iterative method to bootstrap model performance on reliable pseudo-labeling and robust representation learning. As shown in Fig. 3, our model mainly consists of three stages. Firstly, we pre-train a feature extractor on both labeled and unlabeled data to optimize better knowledge trans-
fer (Sec. 3.3). Secondly, we obtain more accurate pseudo-labels by solving a relaxed optimal transport problem (Sec. 3.4). Thirdly, we propose two noise regularization techniques to divide pseudo-labels and employ contrastive loss to generate well-separated clusters of representations for both known and novel intent categories(Sec. 3.5).

heading: 3.3 Model Pre-training
text: Intent Representation Extraction To trigger the power of pre-trained language models in NID, we use BERT (Devlin et al., 2019; Yang et al., 2023) as the intent encoder (Eθ : X → RH). Firstly, we feed the ith input sentence xi to BERT, and take all token embeddings [t0, . . . , tM ] ∈R(M+1)×H from the last hidden layer (t0 is the embedding of the [CLS] token). The mean pooling is applied to get the averaged sentence representation zi ∈ RH :
zi = 1
M + 1
M∑
i=0
ti (1)
where [CLS] is the vector for text classification, M is the sequence length, and H is the hidden size.
Knowledge Sharing To effectively generalize prior knowledge through pre-training to unlabeled data, we fine-tuned BERT on labeled data (Dl) using the cross-entropy (CE) loss and on all available data (Da = Dl ∪ Du) using the masked language modeling (MLM) loss. The training objective of the fine-tuning can be formulated as follows:
Lp = −Ex∈Dl logP (y|x)−Ex∈Da logP (x̂|x\m(x)) (2)
where Dl and Du are labeled and unlabeled intent corpus, respectively. P (x̂|x\m(x))predicts masked
tokens x̂ based on the masked sentence x\m(x), where m(x) denotes the masked tokens. The model is trained on the whole corpus Da = Dl ∪ Du.

heading: 3.4 Reliable Pseudo-labeling
text: Optimal Transport Here we briefly recap the well-known formulation of optimal transport (OT). Given two probability simplex vectors α and β indicating two distributions, as well as a cost matrix C ∈ R|α|×|β| , where |α| denotes the dimension of α, OT aims to seek the optimal coupling matrix Q by minimizing the following objective:
min Q∈Π(α,β)
⟨Q,C⟩ (3)
where ⟨·, ·⟩ denotes frobenius dot-product. The coupling matrix Q satisfies the polytope Π(α,β) ={ Q ∈ R|α|×|β|+ | Q1|β| = α,Q⊤1|α| = β } , where α and β are essentially marginal probability vectors. Intuitively speaking, these two marginal probability vectors can be interpreted as coupling budgets, which control the mapping intensity of each row and column in Q.
Relaxed Optimal Transport for Pseudo-labeling The variables Q ∈ RN×K+ and P ∈ RN×K+ represent pseudo-labels matrix and classifier predictions, respectively, where N is the number of samples, and K 2 is the number of classes. The OT-based PL considers mapping samples to class and the cost matrix C can be formulated as − logP. So, we can rewrite the objective for OT-based PL based on the problem (3) as follows:
min Q,b
⟨Q,− logP⟩+ λH(Q)
s.t. Q1 = α,QT1 = β,Q ≥ 0 (4)
where the function H is the entropy regularization, λ is a scalar factor, α = 1N 1 is the sample distribution and β is class distribution. So the pseudolabels matrix Ua can be obtained by normalization: NQ. However, in the i-NID setup, the class distribution is often long-tailed and unknown, and the model optimized based on the problem (4) tends to learn degenerate solutions. This mismatched class distribution will lead to unreliable pseudo-labels. To mitigate this issue, we impose a soft constraint (ROT) on the problem (4). Instead of the traditional equality constraint (Asano et al., 2020; Caron et al.,
2We estimate the number of classes K based on previous works (Zhang et al., 2021a) to ensure a fair comparison. We provide a detailed discussion on estimating K in Appendix E.
2020a), we employ a Kullback-Leibler (KL) divergence constraint to encourage a uniform class distribution. This constraint is crucial for preventing degenerate solutions in long-tailed scenarios while allowing for the generation of imbalanced pseudo-labels due to its more relaxed nature compared to an equality constraint. The formulation of ROT is articulated as follows:
min Q,β
⟨Q,− logP⟩+ λ1H(Q) + λ2DKL( 1
K 1,β)
s.t. Q1 = α,QT1 = β,Q ≥ 0,βT1 = 1 (5)
where λ2 is a hyper-parameter and DKL is the Kullback-Leibler divergence. The optimization problem (5) can be tractably solved using the Sinkhorn-Knopp algorithm (Cuturi, 2013) and we detail the optimization process in Appendix A.

heading: 3.5 Robust Representation Learning
text: Directly using generated pseudo-labels for representational learning is risky due to significant noise in early-stage pseudo-labeling. Consequently, we categorize pseudo-labels as clean or noisy based on their distribution and quality, applying contrastive loss to achieve cluster-friendly representations. Noise Regularization We initially introduce a distribution-aware regularization (DR) to align the sample selection ratio with the class prior distribution, effectively mitigating selection bias in i-NID setup. This regularization combines small-loss instances with class distributions, ensuring inclusive representation of all classes, particularly Tail categories, during training. Specifically, the final set of selected samples S′ is represented as follows:
S′ = K⋃
j=1
s′j (6)
where K is total classes, s′j is the set of samples selected from the j-th category slice sj , defined as:
s′j = {h | (h ∈ sj) ∧ (sort(l(h)) ≤ kj)} (7) where l(h) is the instance-level loss of h, ρ is threshold hyper-parameter, rj is the class distribution, kj = min(|sj | , ⌈Nρrj⌉).
In addition, to select high-confidence pseudolabels that closely align with the predicted labels, we propose a quality-aware regularization (QR). Specifically, we calculate confidence scores for each pseudo-label and then select the clean samples, denoted as h, whose confidence scores exceed a certain threshold τg:
A′ = {h | (h ∈ Ua) ∧ (max (p) > τg)} (8)
where p is the probability vector for h and τg ∈ [0, 1] is a confidence threshold hyper-parameter. Then the overall pseudo-labels Ua can filter out the clean pseudo-labels Uclean as follows:
Uclean = { h | ( h ∈ S′ ) ∨ ( h ∈ A′ )} (9)
Contrastive Clustering Following the extraction of clean pseudo-labels, we extend the traditional contrastive loss (Khosla et al., 2020) to utilize label information, forming positive pairs from sameclass samples within Uclean. Additionally, to enhance the model’s emphasis on clean samples, we introduce a method for encoding soft positive correlation among pseudo-positive pairs, enabling adaptive contribution. Specifically, for an intent sample xi, we first acquire its L2-normalized embedding zi. By multiplying the confidence scores q of two samples, we obtain an adaptive weight wij = qi ·qj . The class-wise contrastive loss (CWCL) is then defined as follows:
Lc(i) = ∑
p∈P (i) wip · log exp (zi · zp/τ)∑ j 1i ̸=j exp ( zi · zj/τ )
P (i) = {p | (p ∈ Uclean) ∧ (ci = cp)} (10)
where P (i) represents the indices of instances sharing the same label as xi, and τ is a hyper-parameter. Fundamentally, CWCL loss brings intents of the same class closer together while distancing clusters of different classes, effectively creating a clustering effect. To enhance the generalization of intent representation, we incorporate instance-wise contrastive learning (Chen et al., 2020). The augmented views of instances in Ua are used as positive examples. The instance-wise contrastive loss (IWCL) is defined as follows:
Li(i) = − log exp (zi · z̄i/τ)∑
j 1i̸=j exp ( zi · zj/τ ) (11)
where zi, z̄i regard an anchor and its augmented sample, respectively, and z̄i denotes the random token replacement augmented view of zi.
Joint Training To mitigate the risk of catastrophic forgetting of knowledge, we incorporate cross-entropy loss on Uclean into the training process. Overall, the optimization of ImbaNID is to minimize the combined training objective:
Lall = ω · ( ∑
i∈N
1
1 + |P (i)| (Lc(i)+Li(i)))+(1−ω) ·Lce
(12)
where ω is a hyper-parameter and | · | is the cardinality computation. When xi is a noisy example, Lc(i) = 0 and |P (i)| = 0. During inference, we only utilize the cluster-level head and compute the argmax to get the cluster results.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: Baseline Methods We compare our method with various baselines and state-of-the-art methods, including DeepAligned (Zhang et al., 2021a), GCD (Vaze et al., 2022), CLNN (Zhang et al., 2022), DPN (An et al., 2023), LatentEM (Zhou et al., 2023), and USNID (Zhang et al., 2023b). Please see Appendix C for more comprehensive comparison and implementation details.
Evaluation Metrics We adopt three metrics for evaluating clustering results: Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and clustering Accuracy (ACC) based on the Hungarian algorithm. Furthermore, to more easily assess the impact of long tail distribution on performance, we divide Yk and Yn into three distinct groups {Head, Medium, Tail} with the proportions |Head| : |Medium| : |Tail| = 3 : 4 : 3.
Implementation Details To ensure a fair comparison for ImbaNID and all baselines, we adopt the pre-trained 12-layer bert-uncased BERT model3 (Devlin et al., 2019) as the backbone encoder in all experiments and only fine-tune the last transformer layer parameters to expedite the training process (Zhang et al., 2021a). We adopt the AdamW optimizer with the weight decay of 0.01 and gradient clipping of 1.0 for parameter updating. For CLNN (Zhang et al., 2022), the external dataset is not used as in other baselines, the parameter of top-k nearest neighbors is set to {100, 50, 500} for CLINC, BANKING, and StackOverflow, respectively, as utilized in Zhang et al. (2022). For all experiments, we set the batch size as 512 and the temperature scale as τ = 0.07 in Eq. (10) and Eq. (11). We set the parameter ρ = 0.7 in Eq. (7) and the confidence threshold τg = 0.9 in Eq. (8). We adopt the data augmentation of random token replacement as Zhang et al. (2022). All experiments are conducted on 4 Tesla V100 GPUs and averaged over 3 runs.
3https://huggingface.co/bert-base-uncased

heading: 4.2 Main Results
text: ImbaNID achieves SOTA results in both balanced and imbalanced settings. In Table 2, we present a comprehensive comparison of ImbaNID with prior start-of-the-art baselines in both balanced and multiple imbalanced settings. We observe that ImbaNID significantly outperforms prior rivals by a notable margin of 3.9% under various settings of imbalance ratio. Specifically, on the broad cross-domain CLINC150-LT dataset, ImbaNID beats the previous state-of-the-art with an increase of 3.5% in ACC, 0.7% in NMI, and 3.9% in ARI on average. On the StackOverflow20-LT with fewer categories, ImbaNID demonstrates its effectiveness with significant improvements of 2.6% in ACC, 0.6% in NMI, and 2.4% in ARI on average, consistently delivering substantial performance gains across each imbalanced subset. When applied to the specific single-domain BANKING77-LT datasets, ImbaNID reliably achieves significant performance improvements, underscoring its effectiveness in narrow-domain scenarios with indis-
tinguishable intents. These results show the conventional NID models with naive pseudo-labeling and representation learning methods encounter a great challenge in handling the i-NID task. Our method efficiently produces accurate pseudo-labels under imbalanced conditions by employing soft constraints and utilizes these pseudo-labels to construct cluster-friendly representations.
Effectiveness on Long-tailed Distribution We also provide a detailed analysis of the results for the Head, Medium, and Tail classes, offering a more comprehensive understanding of our method’s performance across three i-NID datasets. Fig. 4 presents the comparative accuracy among various groups under the condition γ = 3. It is noteworthy that in Tail classes, the gaps between ImbaNID and the best baseline are 4.2%, 3.5% and 3.7% across three datasets. In contrast, most baselines exhibit degenerated performance, particularly on CLINC150-LT and BANKING77-LT. Moreover, ImbaNID retains a competitive performance on Head classes. These results highlight the effective-
Head Medium Tail0
20
40
60
80
A C
C
CLINC150-LT
Head Medium Tail0
20
40
60
80 BANKING77-LT
Head Medium Tail0
20
40
60
80
StackOverflow20-LT
DPN LtentEM USNID GCD DeepAligned CLNN ImbaNID
Figure 4: Head, Medium, and Tail comparison on the ImbaNID-Bench datasets.
Methods CLINC150-LT BANKING77-LT StackOverflow20-LT
Head Medium Tail Head Medium Tail Head Medium Tail
ImbaNID 82.52 90.67 71.26 68.26 66.05 65.87 90.67 87.25 81.67
① w/ COT 72.74 87.44 58.67 62.72 63.11 48.70 86.63 85.75 79.67 ② w/ EOT 81.41 83.00 65.33 66.59 65.40 57.61 90.00 86.11 81.60 ③ w/ MOT 69.33 57.67 30.52 62.07 57.34 26.20 88.97 66.00 64.33
④ w/o DR 80.74 88.57 71.21 67.17 65.08 49.67 88.33 86.75 81.33 ⑤ w/o QR 82.50 88.94 70.52 63.91 65.42 59.02 87.67 86.00 81.57 ⑥ w/o DR and QR 81.19 87.19 71.05 67.50 64.88 50.00 88.33 86.51 80.33
⑦ w/o Adaptive Weight 82.37 90.22 71.11 68.18 65.81 64.57 90.30 87.00 79.67 ⑧ w/o CWCL 81.93 90.11 70.81 67.83 66.03 58.70 90.33 85.22 78.00 ⑨ w/o IWCL 81.78 86.44 71.23 65.54 64.22 65.20 90.51 76.75 80.33
Table 3: Experimental results of the ablation study on the ImbaNID-Bench datasets at imbalance ratios γ = 10.
ness of ImbaNID in i-NID setup, making it particularly advantageous for Head and Tail classes.

heading: 4.3 Effect of Pseudo-label Assignment
text: To evaluate ROT in reliable pseudo-labels generation of the i-NID setup, we compare three OTbased optimizations for pseudo-labels generation, including COT (Caron et al., 2020a), EOT (Asano et al., 2020), and MOT (Li et al.). (1) COT denotes the removal of the KL term from our optimization problem (5). (2) EOT signifies the replacement of the KL term in our optimization problem (5) with a typical entropy regularization KL(β∥β̂). (3) MOT operates without any assumption on the class distribution β, allowing β to be updated by the model prediction using a moving-average mechanism. Specifically, β = µβ̂ + (1 − µ)v, where µ is the moving-average parameter, β̂ is the last updated β and vj = 1N ∑N i=1 1 (j = argmaxPi). From Table 3, we can observe that ImbaNID outperforms the model ①, which indicates the necessity of imposing constraints on the class distribution. Compared to the model ②, ImbaNID achieves the most gains for Head and Tail classes, indicating it better constrains the class distribution towards uniformity. Finally, when compared to the above strategies, the performance of the model ② in the Tail classes
is notably inferior. The results stem from inadequate constraints on the category distribution, leading to a decline in cluster quality. The comparisons underscore that ImbaNID demonstrates strong proficiency in generating accurate pseudo-labels within the i-NID setup.

heading: 4.4 Effect of Noise Regularization
text: To investigate the effectiveness of noise regularization (NR) in filtering noisy pseudo-labels, we conduct ablation experiments to analyze its contributions. In Table 3, eliminating DR diminishes intent discovery performance, particularly in Tail classes. This occurs because a higher proportion of Head classes in pseudolabels inevitably results in model bias. Furthermore, removing QR results in decreased performance, primarily because fewer examples are initially selected due to the classifier’s low confidence, leading to degenerate solutions. Notably, considering all pseudo-labels as clean leads to significant performance drops across all datasets, indicating that numerous noisy pseudo-labels may cause model overfitting and reduced generalization. The results indicate that NR is indispensable to ImbaNID in handling i-NID setup.

heading: 4.5 Effect of Contrastive Clustering
text: To assess the impact of contrastive clustering in representation learning, we carry out ablation experiments to analyze its individual effects in Table 3. When the adaptive weight strategy is removed from Eq. (10), the model disregards probability distribution information and becomes more susceptible to noisy pseudo-labels. Then, removing CWCL or IWCL from Eq. (12) results in performance degradation, suggesting that class-wise and instancewise contrastive learning respectively aid in developing compact cluster representations and enhancing representation generalization. In Fig. 5, we use t-SNE to illustrate embeddings learned on the StackOverflow20-LT dataset, where ImbaNID visibly forms more distinct clusters than comparative methods, underscoring the effectiveness of our model.

heading: 4.6 Effect of Known Class Ratio
text: To investigate the impact of varying numbers of known intents, we vary the ratio of known intents ranging in {25%, 50%, 75%} during training. Fig. 6 illustrates the comparative accuracy among various ratio of known intents under the condition γ = 3. We observe that even when only a few known intents are available, our method still performs better than other strong baselines. This demonstrates its strength in learning from labeled data and discovering inherent patterns from unlabeled data. Meanwhile, we note a rise in performance as the volume of labeled data incorporated increases, aligning with anticipated outcomes. In short, our proposed method exhibits strong robustness and generalization capability.

heading: 4.7 Effect of Exploration and Utilization
text: The weight of the multitask learning ω in Eq. 12 adjusts the contribution of two objectives. Intuitively, the first term aims to explore cluster-friendly intent representations across all samples, while the second term focuses on mitigating the risk of catastrophic forgetting, ensuring the effective utilization of knowledge derived from clean samples. We vary the value of ω and conduct experiments on ImbaNIDBench (γ = 10) to explore the effect of ω, which also reflects the inference of exploration and utilization. In Fig. 7, only utilizing clean samples (ω = 0.0) or only exploring(ω = 1.0) the intent representation will not achieve the best results. Interestingly, the effect of ω shows a similar trend (increase first and then decrease) on all metrics and datasets, which indicates that we can adjust the value of ω to give full play to the role of both so that the model can make better use of known knowledge to discover intents accurately.

heading: 4.8 Comparison of Time Complexity
text: The majority of existing methods (Zhang et al., 2022; An et al., 2023; Zhou et al., 2023) are mostly based on k-means for pseudo-labeling, while we propose a novel ROT approach for
pseudo-labeling. We discuss the comparison and selection of time complexity between pseudo-labeling methods based on k-means and ROT. Specifically, the k-means method is a clustering-based approach that iteratively computes distances between data points and assigns them to k cluster centers. Its time complexity, typically around O(nkt), depends on the dataset size (n), the number of cluster centers (k), and the convergence speed (t). While the k-means method has lower time complexity, it is sensitive to the selection of initial cluster centers and convergence, leading to potentially unstable outcomes. On the other hand, ROT involves iteratively optimizing the distance or similarity between two data distributions to find the best mapping. Although the time complexity of ROT methods, such as those based on the Sinkhorn algorithm, is typically polynomial (e.g., O(n2m) where n is the number of source domain data points and m is the number of target domain data points), they generally provide high-quality pseudo-labels.

heading: 5 Related Work

heading: 5.1 New Intent Discovery
text: New Intent Discovery (NID) is similar to generalized category discovery (GCD) (Vaze et al., 2022), which originates from computer vision and aims to discover novel intents by utilizing the prior knowledge of known intents. Lin et al. (2020) conducts pair-wise similarity prediction to discover novel intents, and Zhang et al. (2021a) uses aligned pseudo-labels to help the model learn discriminative intent representations. Recent works further advance NID by incorporating contrastive learning (Shen et al., 2021; Kumar et al., 2022; Zhang et al., 2022, 2023b), knowledge transfer (An et al., 2023), probabilistic frameworks (Zhou et al., 2023), pseudo-label learning (Zhang et al., 2024a) or prototype attracting and dispersing (Zhang et al., 2024b) to capture cluster-friendly intent representation. However, those methods operate under the unrealistic assumption that the distribution of both known and new intent classes is uniform, overlooking the long-tailed distributions frequently encountered in realworld scenarios. In this work, we explore the imbalanced NID scenario.

heading: 5.2 Optimal Transport
text: Optimal Transport (OT) aims to find the most efficient transportation plan while adhering to marginal distribution constraints. It has been used in a broad spectrum of various tasks, including generative model (Gulrajani et al., 2017), semi-supervised learning (Taherkhani et al., 2020; Tai et al., 2021), clustering (Caron et al., 2020a; Zhang et al., 2023a) and new intent discovery (Zhang et al., 2024a). However, these methods typically impose an equality constraint when solving the OT problem. In contrast, we explore generating pseudo-labels by solving a relaxed OT problem. This approach encourages a uniform class distribution and addresses class degeneration in long-tailed

heading: 5.3 Contrastive Learning
text: Contrastive Learning (CL) has been widely adopted to generate discriminative sentence representations for various scenarios (Chen et al., 2020; Khosla et al., 2020; Li et al., 2021), such as out-of-domain detection (Zhang et al., 2023c,d), machine translation (Yang et al., 2020, 2021a,b, 2022b, 2024), and named entity recognition (Yang et al., 2022a; Mo et al., 2024). In essence, the primary intuition behind CL is to pull together positive pairs in the feature space while pushing away negative pairs. Motivated by its superior performance, contrastive learning has also been leveraged for intent recognition where it is used for NID. In this work, we design both class-wise and instance-wise contrastive learning objectives to learn cluster-friendly intent representations.

heading: 6 Conclusion
text: In this work, we first propose the i-NID task to identify known and infer novel intents within these long-tailed distributions. Then, we develop an effective ImbaNID baseline method for the i-NID task, where pseudo-label generation and representation learning mutually iterate to achieve cluster-friendly representations. Comprehensive experimental results on our ImbaNID-Bench benchmark datasets demonstrate the effectiveness of our ImbaNID method for i-NID. We hope our work will draw more attention from the community toward a broader view of tackling the i-NID problem.
Limitations
To better enlighten the follow-up research, we conclude the limitations of our method as follows: (1) Enhancing interpretability. Our ImbaNID automatically assigns labels to unlabeled utterances in real-world long-tail data distributions, yet it does not generate interpretable intent names for each cluster. (2) Integration with LLMs. Large-scale language models (LLMs) have shown an impressive ability in a variety of NLP tasks, we plan to explore the integration of ImbaNID with LLMs to boost performance in practical scenarios. (3) Reducing time complexity. The time complexity of relaxed optimal transport (ROT) is O (n2), we plan to further develop a fast matrix scaling algorithm to reduce the complexity.

heading: Acknowledgements
text: This work was supported in part by the National Natural Science Foundation of China (Grant Nos. U1636211, U2333205, 61672081, 62302025, 62276017), a fund project: State Grid Co., Ltd. Technology R&D Project (ProjectName: Research on Key Technologies of Data Scenario-based Security Governance and Emergency Blocking in Power Monitoring System, Proiect No.: 5108-202303439A-3-2ZN), the 2022 CCF-NSFOCUS Kun-Peng Scientific Research Fund and the Opening Project of Shanghai Trusted Industrial Control Platform and the State Key Laboratory of Complex & Critical Software Environment (Grant No. SKLSDE-2021ZX-18).

heading: A ROT
text: In this section, we provide a comprehensive optimization process for the ROT problem (5), the ROT objective is:
min Q,β
⟨Q,− logP⟩+ λ1H(Q) + λ2DKL( 1
K 1,β)
s.t. Q1 = α,QT1 = β,Q ≥ 0,βT1 = 1 (13)
where λ1 and λ2 are hyper-parameters, and DKL(A,B) denotes the Kullback-Leibler Divergence. We utilize the Lagrangian multiplier algorithm for optimization:
L(Q,β,f , g, h) = ⟨Q,− logP⟩+ λ1H(Q)
+ λ2DKL( 1
K 1,β)− fT (Q1−α)
− gT (QT1− β)− h(βT1− 1) (14)
where f , g, and h are Lagrangian multipliers. Differentiating Eq. (14) yields the following result:
∂L
∂Qij = λ1log(Qij)− log(Pij)− fi − gj (15)
∂L ∂fi = −(
K∑
j
Qij) + αi (16)
∂L ∂gj = −(
N∑
i
Qij) + βj (17)
∂L ∂βj = − λ2 Kβj + gj − h (18)
∂L ∂h = −(
K∑
j
βj) + 1 (19)
Initially, we fix β and h, and then update Q, f , and g. By setting ∂L
∂Qij , ∂L ∂fi , and ∂L ∂gj to zero, we obtain the following results:
Qij = exp( fi + log(Pij) + gj
λ1 )
= exp( fi λ1 ) · exp( log(Pij) λ1 ) · exp( gj λ1 )
(20)
K∑
j
Qij = αi,
N∑
i
Qij = βj (21)
Based on Eq. (20), we derive the following:
Q = diag(exp( f
λ1 )) exp(
logP
λ1 )diag(exp(
g
λ1 )) (22)
Considering the constraints (21) and the conditions βT1 = αT1 = 1, we solve Eq. (22) to determine the values of Q, f , and g using the Sinkhorn algorithm (Cuturi, 2013). Subsequently, with f , g, and Q fixed, we update β and h. Setting Eq. (18) to zero yields the following solution:
βj = λ2
K(gj − h) (23)
Take Eq. (23) into the Eq. (19) and let Eq. (19) equal to 0, we can obtain:
(
K∑
j
βj(h))− 1 = 0 (24)
Algorithm 1 The optimization of ROT Input: The cost matrix: − logP. Output: The transport matrix: Q, The class distribution: β. Procedure:
1: Initialize β as uniform distribution; 2: for i = 1 to T do 3: Fix β and h, calculate Q, f and g with Sinkhorn algorithm. 4: Fix Q, f and g, update β and h with Eq. (23) and (24). 5: end for 6: Return Q and β.
We obtain h from Eq.(24) using the bisection method and subsequently determine the corresponding β. In the final step, we iteratively update f , g, Q, and β, h. The iterative optimization process for ROT is outlined in Algorithm1.

heading: B Statistics of Datasets
text: We present detailed statistics of the CLINC (Larson et al., 2019), BANKING (Casanueva et al., 2020) and StackOverflow (Xu et al., 2015) datasets in Table 4. In addition, we display the number of samples per class for BANKING77-LT and StackOverflow20-LT under various imbalance factors, as shown in Fig. 8. We also provide dataset statistics for the ImbaNID-Bench datasets with imbalance factors of 3 and 5, as shown in Table 5.

heading: C Comparison Methods
text: In this work, we compare the proposed ImbaNID method against several representative baselines including: GCD (Vaze et al., 2022) introduces a combination of supervised and self-supervised contrastive learning to learn distinctive representations, which are then clustered using k-means. DeepAligned (Zhang et al., 2021a) is an improved DeepClustering (Caron et al., 2018) that uses an alignment strategy to alleviate the label inconsistency problem. MTP-CLNN (Zhang et al., 2022) is a method that applies multi-task pre-training and nearest
neighbors contrastive learning for NID. DPN (An et al., 2023) proposes a decoupled prototypical network that, by framing a bipartite matching problem for category prototypes, separates known and novel categories to meet their distinct training objectives and transfers category-specific knowledge for capturing high-level semantics. LatentEM (Zhou et al., 2023) introduces a principled probabilistic framework optimized with the EM algorithm. In the E-step, it assigns pseudo-labels, and in the M-step, it learns cluster-friendly representations and updates parameters through contrastive learning. USNID (Zhang et al., 2023b) is a two-stage framework for both unsupervised and semisupervised NID with an efficient centroidguided clustering mechanism.
D Implementation Details
To ensure a fair comparison for ImbaNID and all baselines, we consistently adopt the pretrained 12-layer bert-uncased BERT model4 (Devlin et al., 2019) as the backbone encoder in all experiments and only fine-tune the last transformer layer parameters to expedite the training process as suggested in (Zhang et al., 2021a). We adopt the AdamW optimizer with 0.01 weight decay and 1.0 gradient clipping for parameter update. During pre-training, we set the learning rate to 5e-5 and adopt the early stopping strategy with a patience of 20 epochs. For CLNN (Zhang et al., 2022), the external dataset is not used as in other baselines, the parameter of top-k nearest neighbors is set to {100, 50, 500} for CLINC, BANKING, and StackOverflow, respectively, as utilized in Zhang et al. (2022). For all experiments, we set the batch size as 512 and the temperature scale as τ = 0.1 in Eq. (10) and Eq. (11). We set the parameter ρ = 0.65 in Eq. (7), the confidence threshold τg = 0.9 in Eq. (8). We adopt the data augmentation of random token replacement as Zhang et al. (2022). All experiments are conducted on 4 Tesla V100 GPUs and averaged over 3 runs. we split the datasets into train, valid, and test sets, and randomly select 25% of categories as unknown and only
4https://huggingface.co/bert-base-uncased
10% of training data as labeled. The number of intent categories is set as ground truth.
E Estimate the Number of Intents (K)
In practical dialogue systems, new intents emerge constantly and we cannot know the exact number of the intent clusters. In this paper, following the work of (Zhang et al., 2021b), we take the full usage of the well-initialized intent features to automatically estimate the intent cluster number K. Specifically, we first assign a big K ′ as the initial intent cluster number. Then we directly use the pre-trained model to extract the feature representations for the training data and perform the K-means algorithm to group these feature representations into different clusters. From these clusters, we can distinguish the dense and boundary-clear clusters as the real intent clusters, while the remaining low-size clusters are filtered out. The filtering function can be formulated as follows:
K =
K′∑
i=1
δ (|Ti| ≥ t) (25)
where |Ti| is the size the ith grouped cluster, t is the threshold of filtering. δ(·) is the indicator function, whose output is 1 if the condition is satisfied.

heading: F Hyper-Parameter Analyses
text: To investigate the sensitiveness of the hyperparameters in Eq. 5, we first referred to the experience from previous studies (Asano et al., 2020; Caron et al., 2020b) and identified λ1 = 0.05 on the all datasets. Then we examine the impact of λ2 on model performance by varying the value of λ2 to observe the performance changes. The results are reported in Fig. 9. Specifically, Fig. 9(a) shows the impact of λ2 variation on the performance of balanced
datasets, while Fig. 9(b) demonstrates the effect of λ2 on the performance of imbalanced datasets. Empirically, we choose λ2 = 7 on the balanced datasets, and λ2 = 2 on the imbalanced ImbaNID-Bench datasets.

heading: Future Work
text: We will release these datasets for future research