TK-KNN: A Balanced Distance-Based Pseudo Labeling Approach for Semi-Supervised Intent Classification
Authors: Nicholas Botzer
David Vazquez
Tim Weninger
Issam H. Laradji
Venue: EMNLP 2023 Findings (2023)
Keywords: semi-supervised learning
intent classification
contrastive learning
TL;DR: We propose a new method for semi-supervised learning applied to intent classification that demonstrates improved performance in resource constrained settings.

Abstract: The ability to detect intent in dialogue systems has become increasingly important in modern technology.
These systems often generate a large amount of unlabeled data, and manually labeling this data requires substantial human effort.
Semi-supervised methods attempt to remedy this cost by using a model trained on a few labeled examples and then by assigning pseudo-labels to further a subset of unlabeled examples that has a model prediction confidence higher than a certain threshold. However, one particularly perilous consequence of these methods is the risk of picking an imbalanced set of examples across classes, which could lead to poor labels. In the present work, we describe Top-K K-Nearest Neighbor (TK-KNN), which uses a more robust pseudo-labeling approach based on distance in the embedding space while maintaining a balanced set of pseudo-labeled examples across classes through a ranking-based approach. Experiments on several datasets show that TK-KNN outperforms existing models, particularly when labeled data is scarce on popular datasets such as CLINC150 and Banking77.

TK-KNN: A Balanced Distance- Based Pseudo Labeling Approach for Semi-Supervised Intent Classification
Nicholas B David Vasquez*, Tim Weninger* Issam Laradji*
University of Notre Dame, ServiceNow Research
nbotzer@nd. *issam. laradj @gmail com
Abstract
The ability to detect intent in dialogue systems has become increasingly important in modern technology. These systems often generate large amount unlabeled data, and manually labeling this data requires substantial human ef- fort. Semi- supervised methods attempt to rem- edy this cost by using model trained on few labeled examples and then by assigning pseudo- labels to further a subset of unlaheled examples that has a model prediction confidence higher than certain threshold. However, particu- larly perilous consequence of these methods is the risk of picking an imbalanced of exam- ples across classes, which could lead to poor labels. In the present work, we describe Top-K K-Nearest Neighbor (TK-KNN), which more robust useudo-labeling approach based on distance embedding space while main- taining balanced set of pseudo-labeled ex- amples acToss classes through ranking-based approach. Experiments on several datasets show that TK-KNN outperforms existing mod- els. particularly when labeled data is scarce on popular datasets such as CLINC150 and Bank- ing77. Code is available at httpp::// t/ubb thub. com/ServiceNow// kkkknn
Exeating trrehols relection statngy
Iniaalenced cecsioon boundary
Our TK-OHN solection ststngg
Balanced codision houndary
Figure 1: Example of pseudo label selection when using a threshold (top) versus the top-k sampling strategy (bottom). this toy scenario, we chose = 2, where each class is represented by unique shape. As the threshold selection strategy pseudo-labels data elements (shown as yellow) that exceed the confidence level, the model tends to become biased towards classes that are easier to predict. This bias causes a cascade of mis- labels that leads even more bias towards the majority class.
vious solution because they can significantly re- duce the amount of human effort required to train these kinds of models [Laradji et al., 2021, Zhu and Goldberg, 2009] especially in image classification tasks [Zhai et al., 2019, Ouali et al., 2020b, Yalniz et al., 2019]. However, well shall see, applica- tions of these models difficult for NLU and intent classification because of the label distribution.
1 Introduction.
Large language models like BERT [Devlin al., 2018] have significantly pushed the boundaries of Natural Language Understanding (NLU) and created interesting applications such as automatic ticket resolution [Marcuzzo et al., 2022]. key component of such systems is a virtual agent's ability to understand a user's intent to respond appropriately. Successful implementation and de- ployment of models for these systems require large amount of labeled data to be effective. Al- though deployment of these systems often gener- ate a large amount of data that could be used for fine-tume tinettunigg the cost of labeling this data is high. Semi-sun sumi-supervieed learning methodologies are an ob-
Indeed, research most closely realted to the present work is the Slot-List model by Basu et al. [2021], which focuses on the meta-learning as- pect of semi-supervised learning rather than us- ing unlabeled data. In a similar vein the GAN- BERT [Croce et al., 2020] model shows that using an adversarial learning regime can be devised ensure that the extracted BERT features are similar amongst the unlabeled and the labeled data sets and substantially boost classification performance. Other methods have investigated how data augmen-

---

tation can be applied to the NLP domain enforce consistency in the models [Chen et al., 2020], and several other methods have been proposed from the computer vision community. However, a recent empirical study found that many of these methods do not provide the same benefit to NLU tasks as they provide to computer vision tasks [Chen et al.. 2021] and can even hinder performance.
2 Related Work
The task of intent classifi- cation has attracted much attention in recent years due to the increasing use of virtual customer ser- vice agents. Recent research into intent classi- fication systems has mainly focused on learning out of distribution data [Zhan et al., 2021, Zhang ct al., 2021b, Cheng al., 2022, Zhou al., 2022]. These techniques configure their experiments learn from a reduced number of the classes and treat the remaining classes as out-of-distribution during testing. Although this research is indeed important in its own regard, it deviates from the present work's focus semi-supervised learning.
Intent Classification
Intent classification remains a challenging prob- lem for multiple reasons. Generally, the number of intents system must consider is relatively large, with sixty classes or more. On top of that, most queries consists of only a short sentence or two. This forces models to need many examples in order to learn nuance between different intents within the same domain. In the semi-supervised setting, many methods set confidence threshold for the model and assign pseudo-labels to the unlabeled data if their confidence is above the threshold [Sohn et al., 2020]. This strategy permits high-confidence pseudo-labeled data elements to included in the training set, which typically results in performance gains. Unfortunately, this approach also causes the model to become overconfident for classes that are easier to predict.
Pseudo Labeling Pscudo labeling is mainstay in semi-supervised learning [Lee et al., 2013, Rizve et al., 2021, Cascante-Bonilly et al., 2021J. In sim- plc terms, pscudo labelling uses the model itself to acquire hard labels for each of the unlabeled data elements. This achieved by taking the argmax of the models' output and treating the resulting label as the example's label. In this lcarning regime, the hard labels are assigned to the unlabeled examples without considering the confidence of the model's ss predictions. These pseudo-labeled examples are then combined with the labeled data to train the model iteratively. The model is then expected to iteratively improve until convergence. The main drawback of this method is that mislabeled data elements early in training can severely degrade the performance of the system.
In the present work, we describe the Top-K K- Nearest Neighbor (TK-KNN) method for training semi-supervised models. The main idea of this method is illustrated in Figure 1. TK-KNN makes two improvements over other pseudo-labeling ap- proaches. First, address the model overcon- fidence problem, we use a top-k sampling strat- egy when assigning pseudo-labels to enforce a balanced set of classes by taking the top-k pre- dictions per class, not simply the predictions that exceed a confidence threshold overall predictions. Furthermore, when selecting the top-k examples the sampling strategy does not simply rely on the model's predictions, which tend be noisy. In- stead we leverage the embedding space of the la-
A common practice to help alleviate mislabeled samples is to use threshold т to ensure that only high-quality (i.e., confident) labels are re- tained [Sohn et al., 2020J. The addition of con- fidence restrictions into the training process [Sohn et al., 2020] has shown improvements but also re- stricts the data used at inference time and intro- duces the confidence threshold value yet another hyperparameter that needs to be tuned.
beled and unlabeled examples to find those with similar embeddings and combine them with the models' predictions. Experiments using standard performance metrics of intent classification are per- formed on three datasets: CLINC150 [Larson et al., 2019], Banking77 [Casanueva et al., 2020], and Hwu64 [Liu al., 2019]. We find that the TK- KNN method outperforms existing methods in all scenarios and performs exceptionally well in the low-data scenarios.
Another major drawback of this selection method is that the model can become very biased towards the easy classes in the early iterations of learning [Arazo et al., 2020]. Recent methods, such as Flex [Zhang et al., 2021a], have discussed this problem length and attempted to address this issue with a curriculum learning paradigm that allows each class to have its own threshold. These thresholds tend to be higher for majority classes lower for less-common classes. However, this only

---

serves exacerbate the problem because the less- common classes will have less-confident labels. A previous work by Zou et al. [2018] proposes a similar class balancing parameter to be learned per class, but is applied to the task of unsuper- vised domain adaptation. A close previous work to ours is co-training [Nigam and Ghani, 2000] that iteratively adds a single example from each class throughout the self-training- Another more recent work [Gera al., 2022] also proposes bal- anced sampling mechanism for self-training, but starts from a zero-shot perspective and limits to two cycles of self trainng..
tance to match clusters of unlabeled examples labeled clusters for pseudo-labcling.
Second, the aptl--amedd feature affinity based pseudo-labeling [Ding et al., 2019] method uses the cosine similarity between unlabeled examples and cluster centers that have been discovered for each class. The selected pseudo label determined based on the highest similarity score calculated for the unlabeled example.
Results from both of these works demonstrate that distance-baset pscudo-labeling strategies yield significant improvements over previous methods. However, both of these methods depend on clusters formed from the labeled data. In the intent clas- sification task considered in the current study, the datasets sometimes have an extremely limited num- ber of labeled examples per class, with instances where there is only one labeled example per class. This scarcity of labeled data makes forming reliable clusters quite challenging. Therefore, the TK-KNN model described in the present work adapted the K-Nearest Neighbors search strategy to help guide our pseudo-labeling process.
Another pertinent work is by Chen et al. [2022], who introduce ContrastNet, framework that lever- ages contrastive learning for few-shot text classi- fication. This is particularly relevant to our study considering the challenges posed by datasets with a searce number of labeled examples per class. A notable work by Wang et al. [2022] employs a con- trastive learning-enhanced nearest neighbor mecha- nism for multi-label text classification, which bears some resemblance to the KNN strategies discussed in our work.
The TK-KNN strategy described the present work addresses these issues by learning the deci- sion boundaries for all classes in a balanced way while still giving preference to accurate labels by considering the proximity between the labeled and the unlabeled examples in the embedding space.
3 Top-K KNN Semi-Supervised Learning
3.1 Problem Definition
We formulate the problem of semi-supervised in- tent classification as follows:
Given a set of labeled intents X (In,, yn)) :
E (1,..,,N) where In represents the intent ex-
Distance-based Pseudo labeling Another direc- tion explored in recent work is to consider the smoothness and clustering assumptions found in semi-supervised learning [Ouali et al., 2020a] for pscudo labeling. The smoothness assumption states that if two points lie in a high-density region their outputs should be the same. The clustering assump- tion similarly states that if points are in the same cluster, they are likely from the same class. Recent work by Zhu al. [2022] propose a training-free approach detect corrupted labels. They use k- style approach to detect corrupted labels that share similar features. The results of this work show that the smoothness and clustering assumptions are also applicable in a latent embedding space and there- fore data elements that are close in the latent space are likely to share the same clean label.
ample and the corresponding intent class E C and set of unlabeled intents U um m C
(1, ..., M), where each instance um is intent ex- ample lacking label. Intents are fed to the model as input and the model outputs a predicted intent class, denoted as Pmodel (cl x. 0), where represents some pre-trained model parameters. Our goal is to learn the optimal parameters for 8.
3.2 Method Overview
As described above, we first employ pseudo label- ing to iteratively train (and re-train) a model based on its most confident-paat predictions. In the first training cycle, the model is trained on only the small portion of labeled data X. In the subsequent cycles, the model is trained on the union of X and a subset of the unlabeled data U that has been pseudo-labeles by the model in the previous cy- clc. Figure 2 illustrates an example of this training regime with the KNN method.
Two other recent works have made use of these assumptions in semi-supervised learning to improve their pseudo-labeling process. First, Taherkhani et al. [2021] use the Wasserstein dis-

---

Labeled Examples
Unlabeled Examples
1. Train
2. Predict Unlabeled
evrry gin
Caryy
Model
How you selup direct deposk?
Direct Depost (Preb 0.78)
BERT
clrect
chark?
Dised Deposit
What
(Phob 624
1 Adddd to the labeled
Intent: Diract
Compule Cosine Similrtties
Frw do setup dinct
0.80
Nearest Neighbors
Selected Examples
wecld lon sel.p direct deposk.
1181
Pseudo Label
Select too-k
Setup direct depoost for me..
0.66
4. Rank examples
Intent: Deposi
Intenti
Direct Deposil
Text
Cosine Boore
direct
Direct Deposil
What am the carry sules?
0.89
What are the carry on rules?
Carry On
Tell the carry welriciiona for Linied
0.80
Tell me the carry on restrictonal for United
Carry On
Labeled Examplee
Selected
Carry mosticcions for Emiroies.
0.74
Clocast
Examples
Noighbar
Figure 2: TK-KNN overview. The model is (1) trained on the small portion of labeled data. Then, this model is used to predict (2) pseudo labels on the unlabeled data. Then the cosine similarity (3) is calculated for each unlabeled data point with respect to the labeled data points in each class. Yellow shapes represent unlabeled data and green represent labeled data. Similarities are computed and unlabeled examples ranked (4) based combination of their predicted probabilities and cosine similarities. Then, the top-k (k 2) examples are selected (5) for each class. These examples are finally added (6) the labeled dataset continue the iterative learning process.
We use the BERT-base [Devlin et al., 2018] model with an added classification head to the top. The classification head consists of a dropout layer followed by a linear layer with dropout and ends with output layer that represents the dataset's class set C. We select the BERT-base model for fair comparison with other methods.
score(um)) = Ponoded = clum,
(1)
After each training cycle, the number of pseudo labels in the dataset will have increased by l times the number of classes. This process continues un- til all examples are labeled or some number of pre-defined cycles has been reached. We employ standard early stopping criteria [Prechelt, 1998] during each training cycle to determine whether or not to stop training.
3.3 Top-K Sampling
When applying pseudo-labeling, it is often ob- served that some classes easier predict than others. In practice, this causes the model to be- come biased towards the easier classes [Arazo et al., 2020] and perform poorly on the more difficult ones. The Top-K sampling process within the TK- KNN system seeks to alleviate this issue by grow- ing the pseudo-label set across all labels together.
3,4 KNN-Alignment
Although our top-k selection strategy helps allevi- the model's bias, it still relies entirely on the model predictions. To enhance our top-k selection strategy, we utilize a KNN search to modify the scoring function that used to rank which pseudo- labeled examples should be included in the next training iteration. The intuition for the use of the KNN search comes from the findings in [Zhu et al., 2022] where "closer" instances are share the same label based on the neighborhood information when some labels are corrupted, which often occurs in semi cemi-supervised learning from the pseudo-labelle strategy.
When we perform pseudo labeling, we select the top-k predictions per class from the unlabeled data. This selection neither uses nor requires any threshold: instead, limits each class choose the predictions with the highest confidence. We rank each predicted data element with a score based on the models predicted probability.
likely

---

When adopting the contrastive loss previous works (EI-Nouby et al., 2021] have discussed how result Specifically, we extract a latent representation from each example in our training dataset, both the labeled and unlabeled examples. We formulate this latent representation in the same way as Sentence- BERT [Reimers and Gurevych, 2019] to construct a robust sentence representation. This representation is defined as the mean-pooles representation of the final BERT layer that we formally define as:
which our pseudo labeling relies on. To address this issue, we supplemented the cross-entropy loss with a supervised contrastive loss [Khosla et al., 2020] and a differential entropy regularization loss [Sablayrolles et al., 2019) 9], and trained the model using all three losses jointly.
LCE =
Σ yi log((:) 1=1
(4)
2 meann( Tı. T2, .... Tas)
(2)
We select the supervised contrastive loss shown in [Khosla et al., 2020]. This ensures that our model with learn good discriminative features in the latent space that separate examples belong- ing to different classes. The supervised contrastive loss relies on augmentations of the original exam- ples. To get this augmentation we simply apply a dropout layer of 0.2 to the representations that we extract from the model. As is standard for the su- pervised contrastive loss we add a separate projec- tion layer to model to align the representations. The representations fed to the projection layer is the mean-pooled BERT representation as shown in Eq. 2.
Where CLS is the class token, T is each token in the sequence, M the sequence length, and is the extracted latent representation. When we perform our pseudo labeling process we extract the latent representation for all of our labeled data X as well as our unlabeled data U.
For each unlabeled example, we calculate the cosine similarity between its latent representation and the latent representations of the labeled coun- terparts belonging to the predicted class.
The highest cosine similarity score between the unlabeled example and its labeled neighbors is used to calculate the score of an unlabeled example. Let 3m and on be the latent representations of the un- labeled data point thm and a labeled data point Zn. respectively. An additional hyperparameter, 3, per- mits the weighing of the model's prediction and the cosine similarity for the final scoring function.
sim(2., 2p)/T EeEA sim( 2/)/T
LSCL
log PEP(
P())
lET
(5)
score(um) = (1 - B) x Pmoodl (1) Um +)+ B x sim(2ns 3m)
the model can collapse in dimensions of the loss. We follow this work in adopting a dif- ferential entropy regularizer in order to spread the representations our more uniformly. The method we use based on the Kozachenko and Leonenko [1987] differential entropy estimator:
In this equation sim(Zn, 2 computes the co- sine similarity between the latent representations of tm and its closest labeled counterpart in the predicted class. With these scores then follow the previously discussed top-k selection strategy ensure balanced classes. The addition of the K-nearest neighbor search helps us to select more accurate labels early in the learning process. We provide pseudo code for our pseudo-labeling strat- egy in Appendix Algorithm 1.
L Σ log(p:)
(6)
Where pt (I f(xz)1l- This regularization helps maximize the distance be- tween each point and its neighbors. By doing helps to alleviate the rank collapse issue. We combine this term with the cross-entropy and con- trastive objectives, weighting it using coefficient %
3.5
Loss Function
As we use the cosine similarity to help our ranking method want to ensure that similar examples are grouped together in the latent space. While the cross entropy loss is an ideal choice for classification, as it incentivizes the model to produce accurate predictions, it does not guarantce that discrimina- tive features will be learned [Elsayed et al., 201 8),
LALL = LoE + LSCL + TKKaLe
(7)
The joint training of these individual compo- nents leads our model to have better discriminative

---

features that are more robust, that results in im- proved generalization and prediction accuracy.
data examples. These examples then com- bined with the labeled data and used re-train the model an iterative manner.
4 Experiments
Pseudo Labeling with Threshold (PL-T) [Sohn al., 2020]: This process follows the pseudo labeling strategy but only seleets unla- beled data elements which are predicted above a threshold т. We use a 7 of 0.95 based on the findings from previous work.
4.1 Experimental Settings
Datasets We use three well-known benchmark datasets to test and compare the TK-KNN model against other models on the intent classifica- tion task. Our intent classification datasets are CLINC150 [Larson et al., 2019] that contains 150 in-domain intents classes from ten different domains and one out-of-domain class. BANK- ING77 [Casanueva et al., 2020] that contains 77 intents, all related to the banking domain. HWU64 [Liu et al., 2019] which includes 64 in- tents coming from 21 different domains. Bank- ing77 and Hwu64 do not provide validation sets, so we created our own from the original training sets. All datasets are in English. A breakdown of each dataset is shown in Table 1.
Pseudo Labeling with Flexmatch (PI.-Flex) [Zhang et al., 2021a]: Rather than using a static threshold across all classes, dynamic threshold is used for each class based a curriculum learning framework.
GAN-BERT [Croce et al., 2020J: This method applies generative adversarial net- works [Goodfellow et al., 2020] to pre- trained BERT model. The generator is MLP that takes in a noise vector. The output head added to the BERT model acts as the discriminator and includes an extra class for predicting whether a given data element is real or not.
Dataset
Intents
Domain
Train
CLINCISO
15,250
5,580
Banking77
77
9,002
1.011
3,080
Hwu64
8,884
1,076
Table 1: Breakdown of the intent classification datasets. Note that BANKING77 and HWU64 do not provide validation sets, so we generated validation set from the original training set.
MixText [Chen et al., 2020]: This method ex- tends the MixUp [Zhang et al., 2017] frame- work to NLP and uses the hidden representa- tion of BERT to mix together. The method also takes advantage of consistency regulariza- tion in the form of back translated examples.
We conducted our experiments with varying amounts of labeled data for each dataset. All meth- ods are run with five random seeds and the mean average accuracy of their results are reported along with their 95% confidence intervals [Dror et al., 2018].
TK-KNN
The method described in the present work using top-k sampling with weighted selection based on model predictions and cosine similarity to the labeled samples.
4.2 Baselines
To perform a proper and thorough comparison of TK-KNN with existing methods, we implemented and repeated the experiments on the following mod- els and strategies.
4.3 Implementation Details
Each method uses the BERT base model with a clas- sification head attached. We use the base BERT im- plementation provided by Huggingface Wolf et al., 2019, that contains a total of 1 110M parameters. All models are trained for 30 cycles of self-training The models are optimized with the Adam opti- mizer with learning rate of 5e-5. Each model is trained until convergence by early stopping applied according the validation set. We use batch size of 256 across experiments and limit the sequence length to 64 tokens. For TK-KNN, we set I: 6. = 0.75, and y = 0.1 and report the results for
Supervised: Use only labeled portion of dataset train the model without any semi- supervised training. This model constitutes a competitive lower bound performance be- cause of the limits in the amount of labeled data.
Pseudo Labeling (PL) [Lee et al., 2013]: This strategy trains the model to convergence then makes predictions on all of the unlabeled

---

Percent Labeled
Method
198
2%%
598
10%8
0.5
CLINCI50
Test Accuracy
Supervised
27.38 1.771
49.15
11542
1.90
67
10.85
75.05 1.557
24.51 13.92
48.58
1.770
10.54
1:000
0.4
PL-T
39.05 +3.26
56.65
+1.53
71.25 +0.5
79.29
+1.62
11.42
GAN-BERT
18.18 20.0
23.29
11.42
44.89
124.30
63.02
1:25.
0.3
MixText
12.86 +0.39
37.93
+16.8
61.39
+0.77
74.29
+00.7
TK-KNN
53.73 1.172
65.87
11.18
74.31
10099
79.45
111001
BANKING72
0.2
Supervised
34.73
47:51
+2.89
70.27
+1.08
80.82 +0.41
30
29,09 13.83
45.16 12271
09:69
12:18
10.49
PL-T
35.12 #3.86
51.67
+3.14
71.16 11.98
81.88 +0.43
Cycle
GAN-BERT
5.4 20.16
16.98
121
54.09
1:9956
79
1:33
PL-T PL-Flex TK-KNN
MixText
32.73 +0.02
54.75
+3.16
76.59
+1.06
82.34 ++0.44
TK-KNN
54.16 +44.56
62.71
+2.30
76.73
+:0..6
84.45
+00.22
Figure 3: Convergence analysis of pseudo-labelling
HWU064
strategies on CLINC150 at 1% labeled data. TK-KNN clearly outperforms the other pseudo-labelling strategies
Supervised
48.87 +1.56
63.88
74.67
+1.91
82:21 +1.72
1..88
1:666
75.76 1:6.6
82.49 1.0.9
PL-T
56.9 =1.64
68.29
1.79
76.9 =1.1
82.96 +.6.9
by balancing class pseudo labels after each training cycle.
10.93
GAN-BERT
33.36 416.55
12,9
-29.07
72.32
+1.41
81.78 +1.64
MixText
33.3
66.65
79:72
TK-KNN
65.33 +2229
73.03
11.31
79.63
+0.56
84.59 +00.58
Table 2: Mean test accuracy results and their 95% con- fidence intervals across 5 repetitions with different dif- ferent random seeds. All experiments used & 6 and 3 = 0.75. TK-KNN outperformed existing state of the models, especially when the label set small. The confidence interval also shows that the TK-KNN results were stable across repetitions.
labeled data is limited. However, GAN-BERT does improve the proportion of labeled data increases. This is consistent with previous findings training GANs from computer vision [Salimans al., 2016, Arjovsky and Bottou, 2017]. We also find that while the MixText method shows improvements the benefits of consistency regularization are not as strong compared to works from the computer vision domain.
these settings. An ablation study of these two hy- perparameters presented later. For details on the settings used for MixText please see Appendix B.
These results demonstrate the benefits of TK- KNN's balanced sampling strategy and its use of the distances in the latent space.
Computational Use. In total we estimate that we used around 18,000 GPU hours for this project. For the final experiments and ablation studies we estimate that the TK-KNN model used 4400 GPU hours. Experiments were carried out Nvidia Tesla P100 GPUs that each had 12GB of memory and 16GB of memory.
5.1 Overconfidence in Pseudo-Labelling Regimes
A key observation we found throughout self- training was that the performance of existing pseudo-lahelling methods tended to degrade as the number of cycles increased. An example of this is illustrated in Figure 3. Here pre-defined threshold used, the model tends improve performance for the first few training cy- cles. After that point, the pseudo-labeling becomes heavily biased towards the easier classes. This causes the model to become overconfident in pre- dictions for those classes and neglect more difficult classes. PL-Flex corrects this issue but converges much carlier in the learning process. TK-KNN achieves the best performance thanks the slower balanced pseudo-labeling approach. This process helps the model learn clearer decision boundaries for all classes simultancously and prevent overcon- fidence in the model in some classes.
5 Results
that when
Results from these experiments shown in Ta- ble 2. These quantitative results demonstrate that TK-KNN yielded the best performance on the benchmark datasets. We observed the most signifi- cant performance gains for CLINC150 and BANK- ING77, where these datasets have more classes. For instance, on the CLINC150 dataset with 1 1% labeled data, our method performs 10.92% better than the second best strategy, Flexh As the portion of labeled data used increases, notice that the effectiveness of TK-KNN diminishes.
Another observation from these results is that the GAN-BERT model tends to be unstable when the

---

0.65
Percent Labeled
Method
1%
2%8
CLINC150
Test Accuracy
0.6
Top-k U
38.37
1.
15.011.44
Top-k
$1.36:2:1
10.81
Top-k KNN U
41.24 +0.97
55.01 +1.49
Top-k KNN
BANKING77
Top-k U
41,56 14473
54.78 13.51
0.5
Top-lc B
50.45 4:4.58
63.19
Top-k KNNU
44,12 13.14
55.9 22.65
Top-k KNN B
54.16 +4.00
62.71 +2.30
0
20
ITWU64
Cvcle
Top-k U
54.87 1..04
64.85 1.1.4
+
=
0.25 e
0.5
0.75 *
1.0
Top-k B
$4,13 16.0
65.12 10.35
Top-k
+22.25
Top-k KNN B
65.33 +2.29
73.03 +1.31
Figure 4: A comparison TK-NKNN on UWU64 with 1% labeled data as / varies.
Table 3: Ablation study of top-k sampling. U stands for unbalanced sampling, where classes are not balanced. B is for balanced sampling, and classes are balanced with the top-k per class.
5.2 Ablation Study
Because TK-KNN different from existing meth- ods in two distinct ways: (1) top-k balanced sam- pling and (2) KNN ranking, we perform a set of ablation experiments to better understand how each test
Test Accuracy
0.6
0.55
of these affects performance. Specifically, TK-KNN under three scenarios, top-k sampling without balancing the classes, top-k sampling with balanced classes, and top-k KNN without balanc- ing for classes. When we perform top-k sampling in an unbalanced manner, we ensure that the total data sampled is still equal to k C, where C' is the number of classes. We report the results for these experiments in Table 3. We also conduct experiments the addition of the different loss objectives detailed in Appendix C
10
30
Cvcle
+
*
Figure 5: A comparison of TK-KNN on HWU64 with 1% labeled data varies.
5.3 Parameter Search
TK-KNN relies on hyperparameters k and 8 that can affect performance based on how they are configured. We explore experiments to gauge their effect on learning by testing & E (4, 6, 8) and B E (0.0, 0.25, 0.50, 0.75, 1.00). When varying k we hold 8 at 0.75. For 3 experiments we keep k = 6. When 3 = 0.0, this is equivalent to just top- k sampling based on Equation 3. Alternatively, when 3 1.0, this is equivalent to only using the KNN similarity for ranking. Results from our experiments are shown Figures 4 for and for k.
The results from the ablation study demonstrate both the effectiveness of top-k sampling and KNN ranking. A comparison between our unbalanced sampling top-k sampling and balanced versions show drastic difference in performance across all datasets. We highlight again that the performance difference is greatest in the lowest resource setting, with 12.47% % increase in accuracy for CLINC150 in the setting.
Results from the TK-KNN method with unbal- anced sampling also show an improvement over unbalanced sampling alone. This increase in per- formance is smaller than the difference between balanced and balanced sampling but still highlights the benefits of leveraging the geometry for selec- tive pseudo-labeling We also present an ablation the Top-k sampling methodology when passing correctly labeled examples the Appendix A.
As we varied the 3 parameter, we noticed that all configurations tended to have similar training pat- terns. After we trained the model for the first five cycles, the model tended move in small jumps between subsequent cycles. From the illustration, we can see that no single method was always the best, but the model tended to perform worse when

---

3 0.0, highlighting the benefits of including our KNN similarity for ranking. The model reached the best performance when 3 = 0.75, which oc- curs about a third of the way through the training process.
8 Ethics Statement
This work can be used to help improve current virtual assistant systems for many of the businesses that may be seeking to use one. As the goal of these systems is to understand persons request, failures can lead wrong actions being taken that potentially impact an individual.
Comparison of values for k show that TK-KNN is robust to adjustments in this hyperparameter. We notice slight performance benefits from selecting a higher l of 6 and in comparison to 4. When a higher value of k is used the model will see an increase in performance carlier in the self-training process, as it has more examples to train from. This is only acheivable though when high quality cor- rect samples are selected across the entire class distribution. If a & value was selected that is too large, more bad examples will be included early in the training process and may result in poor model performance.
References
E. Arazo, D. Ortego, P. Albert, N. E. O'Connor, and K. McGuinness. Pscudo-labeling and confirmation bias in deep semi-supervised learning. In IJCN,, 2020.
M. Arjovsky and L. Bottou. Towards principled methods for training generative adversarial networks, arXiv preprint arxiv: 7777000822 2017.
S. Basu, A. Sharaf, A. Fischer, V. Rohra, M. Amoake, H. El- Hammamy, E. Nosakhare. V. Ramani, B. Han, et al. Semi- supervised few-shot intent classification and slot filling. arXiv preprint 10Xi::2109.08754 2021.
6 Conclusions
I. Casanueva, T. Temčinas, D. Gerz, M. Henderson, and I. Vulić. Efficient intent detection with dual sentence coders. In Workshop Natural Language Processing for Conversational AI, 2020.
This paper introduces TK-KNN, a balanced distance-based pseudo-labeling approach for semi- supervised intent classification. TK-KNN devi- ates from previous pseudo-labeling methods as it does not rely threshold to select the sam- ples. Instead, we show that balanced approach that takes the model prediction and K-Nearest Neighbor similarity measure allows for more ro- bust decision boundaries to be learned. Experi- ments on three popular intent classification datasets, 50, Banking77, and Hwu64, demonstrate that our method improved performance in all sce-
Cascante-Ronile Tan, Y. Qi, and V. Ordonez. Cur- riculum labeling: Revisiting pseudo-labeling semi- supervised learning. In AAAI, 2021.
Chen, Yang, and D. Yang. Mixtext: Linguistically- informed interpolation of hidden space for semi-supervised text classification. In ACI, 2020.
J. Chen, D. Tam, C. Raffel, M. Bansal, and D. Yang. An empirical survey of data augmentation for limited data learning in nip. arXiv preprint 10Xiv:206..07:900 2021.
J. Chen, R. Zhang. Y. Mao, and J. Xu. Contrastnet: A con- trastive learning framework for few-shot text classification. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pages 10492-10500, 2022.
narios.
7 Limitations
7. Cheng, 7. Jiang, Y. Yin, C. Wang, and Q Gu. Learning to classify open intent via soft labeling and manifold mixup. IFFF/ACM Transactions on Audio, Speech, and Language Processing, 30:635-644 645, 2022.
While our method shows noticeable improvements, it is not without limitations. Our method does not require searching for good threshold but instead requires two different hyperparameters, & and 1, that must be found. We offer a reasonable method and findings for selecting both of these but others may want to search for other combinations depend- ing on the dataset. A noticeable drawback from our self-training method is that more cycles of training will need to be done, especially if the value of i is small. This requires much more GPU usage to converge good point. Further, we did not ex- plore any heavily imbalanced datasets, so we are unaware of how TK-KNN KNN would perform under those scenarios.
D. Croce, G. Castellucei, and R. Basili. Gan-bert: Generative adversarial learning for robust text classification with bunch labeled examples. In ACI. 2020.
Devlin, M. W. Chang, K. Lee, and K. Toutanova. Bert: Pre- training of deep bidirectional transformers language understanding. arXiv preprint arXiv:1810.04803 2018.
G. Ding, S. Zhang, S. Khan, 7. Tang, J. Zhang, and F. Porikli. Feature affinity-based based pseudo labeling for semi superviseed person re-identification. IEF.F Transactions on Multinedia, 21(11))2891 29022 2019.
R. Dror, G. Baumer, S. Shlomov, and R. Reichart. The hitch- hiker's guide to testing statistical significance in natural language processing. In ACI., 2018

---

El-Nouby, Neverova, I. Laptev, and H. Jégou. Training vision transformers for image retrieval. arXiv preprint arXiv:2102.5564 2021.
M. Rizve, K. Duarte, Y. S. Rawat, and M. Shah. In defense of pseudo-labeling: An uncertainty-awarr pseudo- selection framework for semi-supervised learning. arXiv preprint (rXiv:2101.06222 2021.
G. Elsayed, D. Krishnan, H. Mobahi, K. Regan, and S. Bengio. Large margin deep networks for classification, Advances neural information processing systems, 31, 2018.
A. Sablayrolles. M. Douze, C. Schmid, and H. Jégou. Spread- ing vectors for similarity search. In ICLR 2019-7th Inter- national Conference on Learning Representations, pages 1-13, 2019.
A. Gera, A. Halfon, F. Shnarch, Y. Perlitz, I. Gin-Dor, and N. Slonim. shot text classification with self- training. arXiv preprint arXiv:2210.17544 2022.
Salimans, Goodfellow. W. Zaremba, V. Cheung, A. Rad- ford, and X. Chen. Improved techniques for training gans. Advances neural information processing systems, 29, 2016.
I. Goodfellow, J. Pouget-Abadie M. Mirza, B. Xu, D. Warde- Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. Communications of the ACM, 63 (11):139-144,22) 2020.
K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang. C. A. Raffel, F. Cubuk, Kurakin, and C.-I.. Li. Fixmatch: Simplifying supe-ssper learning with consistency and confidence. Newr|PS, 2020.
P. Khosla, P. Teterwak, Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and Krishnan. Supervised con- trastive learning. Advances in neural information process- ing systems, 33:1 1-111661122 8661-1867, 2020.
F. Taherkhani. A. Dabouei, S. Soleymani, J. Dawson, and N. M. Nasrabadi, Self-supervised wasserstein pseudo- labeling for supi-super image classification. In CVPR, 2021.
L. F. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii 23 1987.
R. Wang, X. Dai, al. Contrastive learning-ehhanced nearest neighbor mechanism for multi-label text classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 672 679, 2022.
1. Laradji, P. Rodríguez, D. Vazquez, and D. Nowrouzezahrai. Ssr: Semi-supervised soft rasterizer for siegle-view 2d to 3d reconstruction. erXiv preprint 2XXv:2108.00592 2021.
S. Larson, A. Mahendran, J. Peper, C. Clarke, A. Lee, Hill, J. K. Kummerfeld, K. Leach, M. A. Laurenzano, L. Tang, et al. An evaluation dataset for intent classification and out-of-scope prediction. In EMNLP-IJCNLP, 2019.
Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's transformers: State-of-the-ae natural lan- guage processing. arXiv preprint 170.07191, 2019.
D.-H. Lee et al. Pseudo-lahel: The simple and efficient semi- supervised learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, 2013.
I. Z. Yalniz, H. Jégou, K. Chen, M. Paluri, and D. Mahajan. Billion-scale semi-supervised learning for image classifica- tion. arXiv preprint arXiv: 1905. 00546, 2019.
X. Liu, A. Eshghi, P. Swietojanski, and V. Rieser. Benchmark- ing natural language understanding services for building conversational agents. arXiv preprint arXiv:1903.00 19555566...6 2019.
X. Zhai A. Oliver. A. Kolesnikov, and L. Beyer. S41: Self- supervised supervied learning. In Proceedings of the IEEE/CVP International Conference on Computer Vision. 2019.
M. Marcuzzo, A. Zangari, M. Schiavinato, L. Giudice, A. Gas- paretto, and A. Albarelli. A multi-level approach for hier- archical ticket classification. In W-NUT, 2022.
L. Zhan, H. Liang, B. Liu, I., Fan, X.-M. Wu, and A. Lam. Out-of-scope intent detection with supersipn and discriminative training. arXiv preprint 2rXiv:2106.08616 2021.
K. Nigam and R. Ghani. Analyzing the effectiveness and applicability of co-truining. In Proceedings of the ninth international conference on Information and knowledge management, pages 86-93, 2000.
B. Zhang, Y. Wang. W. Hou, H. Wu, J. Wang. M. Okumura, and T. Shinozaki. Flexmatch: Boosting supe-supe learning with curriculum pseudo labeling. In NeurIPS. 2021a.
Y. Ouali, C. Hudelot, and M. Tami. An overview of deep semi- supervised lcarning. arXiv preprint arXiv:2006.08 2020a.
H. Zhang, M. Cisse, N. Dauphin, and D. Loper-Paz. Paz. mixup: Beyond empirical risk minimization. arXiv preprint 1710v:1710.09442. 2017.
Y. Ouali. C. Hudelot, and M. Tami, Semi--upervised semantic segmentation with cross-consistency training. In Proceed- ings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, 2020b.
H. Zhang, H. Xu. and T.-E. Lin. Deep open intent classifica- tion with adaptive decision boundary. In AAAI, 2021b.
L. Prechelt. Early stopping-bu but when? In Neurul Networks: Tricks of the trade, pages 55-69. Springer, 1998.
Y. Zhou, P. Liu. and X. Qiu. Knn-contrastive learning for out-of-domain intent classification. In ACL, 2022.
N. Reimers and I. Gurevych. Sentence-bert. Sentence embed- dings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Confer- ence on Natural Language Processing FEMNI.P-IJCNI.. pages 3982-3992, 2019.
X. Zhu and A. B. Goldherg. Introduction to semi-supervised learning. Synthesis lectures on artificial intelligence and muchine learning, 31):1-130, 2009.
7. 7ha, 7. Dong, and Y. Liu. Detecting corrupted labels without training a model to predict. In ICML, 2022.

---

Y. Zou, Kumar, and Wang. Unsupervised domain adaptation semantic segmentation via class balanced self-training. Proceedings of European conference on computer vision (ECCV), pages 289-305, 2018.
C Ablation Table
We present a detailed breakdown our full abla- tion results in Table 4. These results include the performance when using only the cross-entropy loss, as well as various combinations of the super- vised contrastive loss and differential entropy reg- ularizer. The results in this table demonstrate that the inclusion additional loss objectives improves performance. This is particularly evident when we add our KNN objective, as we observe an increase of approximately 1-4%. Although the addition of the differential entropy regularizer yields smaller performance improvements, it remains beneficial to our method overall.
Algorithm TK-KNN Sampling For a Cycle
Input: Data of X En. yn E (I,...N).
Um m E (1...., M),,
1: Predict pscudo-labels for all U
2: Calculate cosine similarity via. Eq. (2) for all
U per class
3: Calculate score via Eq.(3)
4: Combine X and top-k per class from U
A Upper Bound Analysis
We further ran experiments to gauge the perfor- mance of top-k sampling when ground truth labels are fed to the model instead of predicted pseudo labels. This experiment gives us an indicator to how performance should increase throughout the self-training process in an ideal pseudo-labeling scenario. We present the results of this in Figure. As expected, the model tends to converge towards fully supervised performance as the cycle increases and more data is )aabudd,)labbedd. Another point of interest is that the method's upper bound can continue learning with proper labels, while TK- KNN method tends to converge earlier. The upper bound method also takes significant increase in the first few cycles as well. This highlights need investigate methods for accurate pseudo label selection further, so that the model can continue to improve.
B MixText Experiments
The Mix'eex method requires number of hyper- parameters be selected order to acheive good performance. The method also relies on data aug- mentation the original text. For our experiments we used the Helsinki -NLP models available on the huggingface repository to perform back trans- lation. Following the original paper we performed back translation into German (de) and Russian (ru). Our experiments used labeled batch size of 4 and unlabeled batch size of 8, the same as the original paper. The Temperature parameter was set 0.5.

---

CLINC150
BANKING77
HWU64
0.8
Test Accuracy
0.6
Cycle
Cycle
Cycle
TK-KNN + Top K Upper Bound
Figure 6: Ablation results for each dataset using 1% labeled data.

---

Percent Labeled 2%
Method
CLINC150
Top-k CE
37.06
50.8
+1.79
Top-k CE CON
38.67 +1.47
55.09 +0.85
Top-k U CE CON DER
38.37 11.08
55.0 11.44
Top-k B CE
49.53 +:111
62.45 1:221
Top-k CE CON
52.33
64.52
Top-k CON DER
51.36 12.1
64.99 10.64
Top-k KNN CE
40.58 +2.54
53.45 +1.05
Top-k KNN U CE CON
42.3 12.86
56.07 11.26
Top-k KNN U CE CON DER
41.24
55,01
Top-k KNN CE
+3.00
63,99 +0.85
Top-k KNN CE CON
53.23 +3.17
65.37 +1.17
Top-k KNN B CE CON DER
53.73 +1.72
65.87 + +1.18
BANKING77
Top-k U CE
36.19 #2.33
$1.69 +3.54
Top-k U CE CON
42.16 +4.61
54.77 #2.48
Top-k U CE CON DER
41.56 14.73
54.78
13.51
Top-k CE
44.35
57.19
Top-k CON
13.75
62.34 +1.57
Top-k B CE CON DER
50.45 14.53
63.19 11.78
Top-k KNN U CE
36.96 +1.78
52.42 +2.73
Top-k KNN CE CON
44.86
56.49
Top-k KNN CE CON DER
44.12
55.9
Top-k KNN CE
49.16 +3.02
59.81 +1.09
Top-k KNN B CE CON
52.4 #3.91
62.35 42.18
Top-k KNN B CF CON DER
54.16 +4406
62.71 +2.30
HWU64
Top-k U CE
51.45 +3.8
62.79 +..00
Top-k U CE CON
54.8 12.1
66.58 11.17
Top-k CE CON DER
54.87
64.85
Top-k
61.69 13.15
70.41 10.66
Top-k B CE CON
63.05 126
72.55 11.41
Top-k B CE CON DER
54.13 2600
65.12 20.33
Top-k KNN CE
$2.97
64.41
Top-k KNN CE
59.26 +2.99
69.98
Top-k KNN U CE CON DER
57.86 +2.25
69.33 20.96
Top-k KNN B CE
62,43 +2.78
71.04 +0.93
Top-k KNN CE CON
64.41 +2.26
72.71 1.007
Top-k KNN CE CON DER
65.33
73.03
Table Ablation study of top-k sampling. U stands for
unbalanced sampling, where classes are not balanced. B is for balanced sampling, and classes are balanced with the top-k per class. CE stands for cross-entropy loss, CON for contrastive loss, and DER differential entropy regularizer.


Reviews:
[{"review_id": "ngIeIjo5r9", "reviewer": "Reviewer_Z4is", "summary": "", "questions": "Some questions above.\nWhat is the \"Top-k upper\" setting? \"select the correct pseudo label\" - what does this mean? you never know whether the pseudo label is correct or not.\nIs \"k\" fixed across cycles? Did you experiment with varying k? Perhaps this might help the issue of \"more cycles of training will need to be done, especially if the value of k is small.\"\nEquation 1.. what is u_m?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We are encouraged to know that the paper addresses an important problem in deep learning related to data efficiency, that we propose  \"sound ideas to address the bias related challenges in a pseudo labelling approaches\", and that the evaluation supports our claims and consistently show improved performance over multiple baselines.\n\nWe now respond to your comments in detail:\n\n>  The authors hypothesise that two points close in the latent space have the same output label. Am not sure if this always holds true in the context of the intent classification problem considered here. Data elements close in the latent space are 'semantically related' but not necessarily in the same-label sense.\n\nIt is true that points that are semantically close might not be close in the latent space for intent classification in the case of using pretrained LLMs without finetuning. \n\nHowever,  we are finetuning these LLMs with the contrastive loss function to ensure that the label representations for the same class are pulled together and different ones will be forced apart, encouraging points with the same output label  to be close in the latent space.\n\n> For instance, GAN-BERT is unstable initially but improves with increase in the proportion of labeled data - Why?\n\nPrevious studies have shown that training GANs with too little data can cause overfitting in the discriminator causing training to diverge [A, B], which aligns with our GAN-BERT experiments.\n\n- [A] Arjovsky, Martin, and Léon Bottou. \"Towards principled methods for training generative adversarial networks.\" arXiv preprint arXiv:1701.04862 (2017).\n\n- [B] Salimans, Tim, et al. \"Improved techniques for training gans.\" Advances in neural information processing systems 29 (2016).\n\n> What is the top-k upper setting?\n\ntop-k upper is the upper bound performance that would be achieved if all selected examples have the correct pseudo label. It gives us the highest potential improvement we could make in our setting and identify how close our proposed method is to the upper bound.\n\n\n> “Select the correct pseudo label” -What does this mean? You never know whether the pseudo label is correct or not.\n\nThis is an experiment we conducted to see the best performance possible if you give the model the ground truth result at each phase of pseudo labeling (it can also be considered as the oracle performance). We have clarified what we mean by “correct pseudo label” in Section 4.2.\n \n> Is “k” fixed across cycles?\n\nThe value of k is fixed across all cycles.\n \n> Did you experiment with varying “k”? \n\nWe present in the paper an ablation study that we conducted with varying the value of k for our experiments (please see Figure 5). Our findings showed that our method is generally robust to the value of k.  \n\n> Equation 1 what is u_m?\n\nU_m is a typo and is meant to be x_m which refers to an unlabeled example that we are pseudo labeling. We have fixed the typo in the paper\n\n> Typos Grammar Style And Presentation Improvements\n\nThank you for pointing out fixes to several typos we had. We have addressed them in the paper.\n\n> \"it does not guarantee that discriminative features will be learned, ...\" - needs citation\n\nWe thank you for pointing this out and have added the following citation [C] to our paper in this location.\n\n[C] Elsayed, Gamaleldin, et al. \"Large margin deep networks for classification.\" Advances in neural information processing systems 31 (2018).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Data efficient deep learning is important from the perspective of cost and resource requirements as well as far reaching benefits to the environment. Further, conversational AI, the use case considered in this paper, has potential to save the high costs typically involved in customer support. This paper aims to address these and extends the state of art in these areas.\nThe authors propose sound ideas to address the bias related challenges in a pseudo labelling approaches. The evaluation supports their claims and consistently shows improved performance over multiple baselines. ", "reasons_to_reject": "The authors hypothesise that two points close in the latent space have the same output label. Am not sure if this always holds true in the context of the intent classification problem considered here. Data elements close in the latent space are 'semantically related' but not\nnecessarily in the same-label sense. They could have subsuming labels. “book_hotel” vs “change_hotel_booking” for instance. I would have liked to see more supporting evidence of this in the evaluation.\n\nWhile the authors make a lot of interesting observations through their evaluation, the intuitions and reasoning are missing. For instance, GAN-BERT is unstable initially but improves with increase in the proportion of labeled data - Why?", "questions_to_authors": "Some questions above.\nWhat is the \"Top-k upper\" setting? \"select the correct pseudo label\" - what does this mean? you never know whether the pseudo label is correct or not.\nIs \"k\" fixed across cycles? Did you experiment with varying k? Perhaps this might help the issue of \"more cycles of training will need to be done, especially if the value of k is small.\"\nEquation 1.. what is u_m?", "paper_topic_and_main_contributions": "In this paper, the authors focus on the problem of intent detection - an important first step in dialogue systems to understand the customer query intent and route the query accordingly. Typically posed as a multiclass classification problem, this requires access to large amounts of labelled data across the intent labels. This can be costly to get and, therefore, researchers have turned to semi-supervised techniques to alleviate this challenge. Specifically, one of the strategies is to generate high confidence pseudo labels for the unlabelled data using the classification model and use these to iteratively train the model. Here, the current research often resorts to using a fixed or flexible threshold across classes. The authors argue that this leads to biased sampling from over confident (easy) labels or samples with error labels due to low confidence that might adversely impact the model training.\nThey then propose an approach that they call TK-KNN that attempt to address these challenges. In their proposed approach, they dispense from using a threshold and instead sample top-k pseudo labelled instances from across intent labels irrespective of their prediction scores. While the model predictions can be noisy, the authors propose to address this by sampling top-k instances for a label that are also closer to their manually labelled counterparts.\nThe authors demonstrate the efficacy of their approach through evaluation on three recent intent classification datasets on varying sizes of labeled data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\"it does not guarantee that discriminative features will be learned, ...\" - needs citation\n"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "\"However, as well shall... \" --> \"However, as we shall ...\"\n\"closely realted\" --> \"closely related\"\n\"spread the representations our ...\" --> \"spread the representations out ...\"\n\"ensures that our model with...\" --> \"ensures that our model will ...\""}}, {"review_id": "osAYWIWxBH", "reviewer": "Reviewer_Rv94", "summary": "", "questions": "1. A major design choice of the scoring function is to use a distance of closest \"predicted\" instance to adjust the scoring. What is the motivation of that choice (why not using the prototype or label embedding)? How to justify a single instance is robust?\n\n2. In the baselines, which ones employ self-training techniques?\n\n3. [Optionally] How many iterations for self-training and what is the stop criteria?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We are encouraged to know that the paper proposes methods that show improvement in intent classification datasets, especially in low resource settings, and that we address an important topic related to improving the stability of self-training which is a hot topic in the NLP community.\n\nWe now respond to your comments in detail:\n\n> The references discussed above. It is not comprehensive or even representative, but it marks some of the missing fields in the related work.\n\nWe would like to thank the reviewer for pointing out these recent relevant works. We will be adding all of these papers to our related work section and do a more thorough literature review. We have discussed these papers (and additional papers that are relevant to our task) in the related work section.\n\n> balanced topk selection [1] is already a known trick. kNN-based classification [2], prototypical network, and contrastive learning methods [3,4] are well established in the community\n\nIn regards to [1], we address a different task than the zero-shot classification that the authors address. Further, our methods are different because we do not perform token masking, use NLI entailment methods, nor have to use strategies to select negative/positive examples. \n\nFurther, we incorporate K-nearest neighbor in the embedding space and contrastive learning, thus our method is more finetuned towards the semi-supervised setup that we have. We also provided an ablation study in our work to show that our method achieves better results than if we used top-k alone (also shown in the table below).\n\n\n\nPaper [2] addresses a different task of multi-label classification rather than semi-supervised. Further, paper [2] uses contrastive learning for a very different objective than ours. While we use it to predict the right pseudo label using the embedding space, they use it to identify missing labels from similar examples. They also use this method for the full dataset whereas we use a variation of this method in the low data setting where we have to address stability issues related to that setup by including a balanced version of top-k example selection mechanism.\n\n[3] addresses a significantly different task of meta-learning in a few-shot setup, which also does not include a balanced way of performing self-training. However, both [2, 3] inspire the idea of using KNN + Contrastive learning, and we show in the table below that our method significantly outperforms this strategy.\n\n**Table 1**\n| Method | Top-k | KNN | Self-training | Contrastive Learning | CLINC Results 1% (Our Method) |\n|--------|-------|-----|---------------|----------------------|-------------------------------|\n| Inspired by [1]    | X     |     | X             |                      | 49.53 ±3.1                    |\n| Inspired by [2, 3]    |       | X   |  X $^\\dagger$          | X                    | 42.3 ±2.86                    |\n| Ours   | X     | X   | X             | X                    | 53.73 ± 1.72                  |\n\n\n$^\\dagger$ While the methods in [2,3] do not use self-training, we used it for our experiments to make them comparable.\n\n\n> A major design choice of the scoring function is to use a distance of closest “predicted” instance to adjust the scoring. What is the motivation of that choice? (why not use the prototype or the label embedding? How to justify a single instance is robust. \n\nOur justification that a single instance is robust relies on the fact that we only compare our pseudo labeled examples with the known instances that we have. Further, our justification for not picking the closest to the mean is that it will lead to lower variance than just choosing a single instance and could hurt diversity in the examples learned in subsequent self-training cycles. \n\nOur approach is shown to be robust as we ran experiments across 5 different random seeds for datasets to show that the performance does not generate a high degree of variance in results. \n\nWe have also ran an experiment using a prototype showing that single instance outperforms the Prototype method (see Table below).\n\n| CLINC Experiments          | 1%            | 2%            | 5%            | 10%           |\n|--------------------------|---------------|---------------|---------------|---------------|\n| Prototype       | 51.9 ± 2.15   | 64.88 ± 0.86  | 74.29 ± 0.82  | 79.27 ± 1.09  |\n| Single Instance (Ours)   | 53.73 ± 1.72  | 65.87 ± 1.18  | 74.31 ± 0.96  | 79.45 ± 1.01  |\n\n> In the baselines, which ones employ self-training techniques?\n\nAll baseline methods utilize self-training.\n\n> How many iterations for self-training and what is the stop criteria?\n\nAs discussed in Section 4.3 of the paper, we used 30 cycles of self-training for our experiments and used early stopping on the validation set to determine when to stop the training procedure.\n\n> Typos Grammar Style And Presentation Improvements\n\nThank you for suggesting grammar improvements. We have addressed the typos, improved the clarity of equation 1 by explaining what p(y|x) in the context mentioned in the paper and equation 3 by providing a clear definition for Z_n in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "\n1. Improving the stability of self-training is an important and hot topic in the NLP community.\n2. The proposed methods show improvement in intent classification datasets, especially in low resource settings.\n", "reasons_to_reject": "1. One of the concern is on the major contribution of the paper, as a balanced topk selection [1] is already a known trick to improve stability of self-training (including up-sampling and down-sampling label with different frequencies). \n2. kNN-based classification [2],  prototypical network, and contrastive learning methods [3,4] are well established in the community. The related work should be discussed in order to claim novel contribution.\n\n[1] Gera, Ariel, et al. \"Zero-shot text classification with self-training.\" arXiv preprint arXiv:2210.17541 (2022).\n\n[2] Wang, Ran, and Xinyu Dai. \"Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022.\n\n[3] Chen, Junfan, et al. \"Contrastnet: A contrastive learning framework for few-shot text classification.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 10. 2022.\n\n\n", "questions_to_authors": "1. A major design choice of the scoring function is to use a distance of closest \"predicted\" instance to adjust the scoring. What is the motivation of that choice (why not using the prototype or label embedding)? How to justify a single instance is robust?\n\n2. In the baselines, which ones employ self-training techniques?\n\n3. [Optionally] How many iterations for self-training and what is the stop criteria?\n", "paper_topic_and_main_contributions": "This paper improves self-training of classifiers with the topk-kNN selection for the pseudo-labeled data points. Specifically, the authors propose  a scoring function that uses a weighted sum of predicted probability of pseudo-label of unlabeled data, and its cosine similarity to the nearest labeled instance in the predicted class. Then the topk scored instances in each class is used as labeled data for supervised training.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Missing_References": "The references discussed above. It is not comprehensive or even representative, but it marks some of the missing fields in the related work.\n\n[1] Gera, Ariel, et al. \"Zero-shot text classification with self-training.\" arXiv preprint arXiv:2210.17541 (2022).\n\n[2] Wang, Ran, and Xinyu Dai. \"Contrastive learning-enhanced nearest neighbor mechanism for multi-label text classification.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 2022.\n\n[3] Chen, Junfan, et al. \"Contrastnet: A contrastive learning framework for few-shot text classification.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 10. 2022.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "For equation (1), and (3), the notation is not very clear. Normally, p(y|x) in equation (1) means a probability distribution over label space, but in the context, it is the probability of the most confident prediction y (and y is the pseudo-label). For equation (3), Z_n is not clearly defined."}}, {"review_id": "rOFeB4kGI5", "reviewer": "Reviewer_67QQ", "summary": "", "questions": "1) You write on line 455ff:\n\"For TK-KNN, we set k = 6, 455, β = 0.75, and γ = 0.1 and report the results for these settings. An ablation study of these two hyperparameters is presented later.\"\nAs far as I see, only k and  β habe been used in the ablation study. Please comment on this.\n\n2) 4400 GPU for the final experiment is a lot. Please comment for which steps you think you needed the most computing time.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. We are encouraged to know that the paper proposes a well-designed new method that is not overly complex, and that the results look strong compared to previous approaches given the confidence intervals.\n\nWe now respond to your comments in detail:\n\n> You write on line 455ff that use you 3 hyper-parameters for the ablation but I only see two used in the ablation study.\n\nThanks for pointing out that we are missing an ablation study for the Gamma hyperparameter, we have included it in the paper. Our selection of 0.1 for Gamma in our work was based on previous literature cited in our paper [A] that uses this value. Below is an ablation table on the HWU dataset at 1% data for different values of gamma.  Our findings align with using the value 0.1 as a good default for Gamma and also show that the method is robust to variations in this hyper-parameter.\n\n\n| HWU Gamma | ACC ± CI    |\n|-----------|-------------|\n| 0         | 64.42 ± 2.08|\n| 0.05      | 64.74 ± 2.73|\n| 0.1       | 65.33 ± 2.29|\n| 0.15      | 64.44 ± 2.33|\n| 0.2       | 64.68 ± 2.24|\n| 1         | 63.1 ± 2.65 |\n\n\n[A] Sablayrolles, Alexandre, et al. \"Spreading vectors for similarity search.\" arXiv preprint arXiv:1806.03198 (2018).\n\n> 4400 GPU for the final experiment is a lot. Please comment for which steps you think you needed the most computing time.\n\nThe number of self-training cycles we selected likely contributed the most to this amount of GPU time, followed by the amount of ablation studies we conducted and the fact that we ran each experiment 5 times to get the mean and confidence interval.  After seeing our results and how the method tends to flatline after ~15 cycles we could have selected a lower number for the number of cycles, such as 20 or 25, and significantly reduced the total amount of compute needed.\n\nFortunately, when we run an experiment on a single dataset and a single set of hyperparameters then the number of GPU hours would be around 3-4 hours, which is practical.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main strength of the paper is the proposition of a well-designed new method for learning from a small of number of labeled examples and by applying it to problems with a relatively large number of classes (e.g. one dataset has 150+1 classes). \nThe new approach is compared to other proposed methods including an lower bound and upper bound model with respect to the performance.\nThe approach seems to be not overly complex in the design - although very time consuming -, so that ideas may be easily translated to other supervised learning problems.\nConfidence intervals are given for the results.\nThe results look strong compared to previous approaches (e.g. PL-Flex or MixText), especially if the percentage of already labeled data is small.", "reasons_to_reject": "The ablation study seems to be incomplete (see the sectoin \"Questions for the authors\").\nWhen the percentage of labeled data reaches 10%, the improvement to other methods is much less impressive.\nAs mentioned in the limitation section, the approach was not tested on highly imbalanced data.", "questions_to_authors": "1) You write on line 455ff:\n\"For TK-KNN, we set k = 6, 455, β = 0.75, and γ = 0.1 and report the results for these settings. An ablation study of these two hyperparameters is presented later.\"\nAs far as I see, only k and  β habe been used in the ablation study. Please comment on this.\n\n2) 4400 GPU for the final experiment is a lot. Please comment for which steps you think you needed the most computing time.", "paper_topic_and_main_contributions": "The paper is a contribution to the problem of generating pseudo-labels of data in a supervised learning context when the amount of labeled data is small, a lot of unlabeled data is available or continuously generated and additional (manual, i.e. human) labeling should be avoided. It is well-known that a naive creation of pseudo-labels by using predictions from a model fit to a small training data set (usually with a threshold, fix or class dependent) leads into serious problems as the problem becomes biased towards labels which are easy to predict. This leads to unbalance in the classes, becomes increasingly worse if the procedure is iterated and leads finally to bad labels.\nThe paper proposes a new idea to avoid this problem. \nAdditionally, the number of classes in intent classification is rather high which is a challenge for the method. The new idea combines pseudo-labeling ideas (called top-K sampling) with distance based K nearest neighbor ideas from unsupervised learning (KNN).", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new method, TK-KNN, for acquiring semi-supervised data for the task of intent detection. The reviewers agree that the problem itself is quite important, that the proposed method is a meritorious contribution towards solving the problem of acquiring labeled data for intent classification, and that the experimental results appear to be sound. Each reviewer has some different individual concerns; R1 about the theoretical motivation behind the method, which the authors try to clarify; R2 about novelty, as top-k KNN + contrastive learning is already a well established technique on other tasks; and R3 about the experimental setup, particularly that more ablations could be useful. Of these, the most salient concern is about novelty: the authors note that the particular combination of top-k KNN + contrastive learning has not been studied for semi-supervised learning on the task of intent classification, and is therefore still a good contribution, while R2 holds that these are well established techniques and it already makes sense that they would perform well in this setting. The study appears to be overall well executed, and though the method itself may not be particularly exciting from a novelty perspective, its application to the task of intent classification with extreme number of labels is still a potentially useful contribution.
Decision: Accept-Findings