APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection
Authors: Pei Wang
Keqing He
Yutao Mou
Xiaoshuai Song
Yanan Wu
Jingang Wang
Yunsen Xian
Xunliang Cai
Weiran Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: OOD
Intent Detection
Few-shot
Prototype

Abstract: Detecting out-of-domain (OOD) intents from user queries is essential for a task-oriented dialogue system. Previous OOD detection studies generally work on the assumption that plenty of labeled IND intents exist. In this paper, we focus on a more practical few-shot OOD setting where there are only a few labeled IND data and massive unlabeled mixed data that may belong to IND or OOD. The new scenario carries two key challenges: learning discriminative representations using limited IND data and leveraging unlabeled mixed data. Therefore, we propose an adaptive prototypical pseudo-labeling(APP) method for few-shot OOD detection, including a prototypical OOD detection framework (ProtoOOD) to facilitate low-resourceOOD detection using limited IND data, and an adaptive pseudo-labeling method to produce high-quality pseudo OOD and IND labels. Extensive experiments and analysis demonstrate the effectiveness of our method for few-shot OOD detection.

APP: Adaptive Prototypical Pseudo-Labeling for Few-shot OOD Detection
Pei Wang",, Keqing He?,, Yutao Mou',, Xiaoshuai Song', Yanan Wu' Jingang Wang", Yunsen Xian", Xunliang Caï", WWiran Xul 'Beijing University of Posts and Telecommunications Beijing, China "Meituan, Beijing, China
[wangpei, myt, songxiaoshuai, yanan wu, xuwei (uweiiaan@bbppp edu. cn (hekeqing, wang ingang, xianyunsen, cai iang Gmeei tuan. com
Abstract
to learn intent representations and then use scor- ing functions to estimate the confidence score of a test query belonging to OOD. For example, Hendrycks and Gimpel (2016) proposes Maximum Softmax Probability (MSP) to use maximum soft- max probability as the confidence score and re- gard a query OOD if the score is below fixed threshold. Xu et al. (2020) further introduces other distance-based method, Gaussian discrimi- nant analysis (GDA), which uses the maximum Mahalanobis distance (Mahalanobis, 1936) to all in-domain classes centroids as the confidence score. Although these models achieve satisfying perfor- mance, they all rely on sufficient labeled IND data, which limits their ability to practical scenarios.
Detecting out-of-domain (OOD) intents from user queries is essential for task-oriented dia- logue system. Previous OOD detection studies generally work on the assumption that plenty of labeled IND intents exist. In this paper, we focus on more practical few-shot OOD setting where there only few labeled IND data and massive unlabeled mixed data that may belong to IND or OOD. The new scenario carries two key challenges: learning discriminative repre- sentations using limited IND data and leverag- ing unlabeled mixed data. Therefore, we pro- pose an adaptive prototypical pseudo-labeling (APP) method for few-shot OOD detection, in- cluding a prototypical OOD detection frame- work (ProtoOOD) to facilitate low-resource OOD detection using limited IND data, and adaptive pseudo-labeling method to produce high-qualip pseudo IND labels. Exten- sive experiments and analysis demonstrate effectiveness of our method for few-shot OOD detection.'
In this paper, we focus on more practical set- ting of OOD detection: there are only few labeled IND data and massive unlabeled mixed data that may belong IND OOD. We call it as Few-Shot OOD Detection. Note that the mixed data don't t have labeled IND or OOD annotations. Consid- cring that unlabeled data is easily accessible, we believe this setting is more valuable to be explored. However, few-shot OOD detection carries two key challenges. (1) Learning discriminative repre- sentations using limited IND data: OOD detcc- tion requires discriminative intent representations to separate IND&IND intents and IND&OOD in- tents. However, traditional models based on cross- entropy or supervised contrastive Icarning (Zeng et al., 2021a) fail to distinguish intent types under the few-shot setting. Limited labeled IND data makes it hard to learn discriminative intent rep- resentations. (2) Leveraging unlabeled mixed data: Unlabeled data contains IND and OOD in- tents which both benefit in-domain intent recog- nition and OOD detection. But it's nontrivial leverage the mixed data because we don't know prior knowledge of OOD data.
1 Introduction
Out-of-domain (OOD) intent detection learns whether a user query falls outside the range of pre-defined supported intents. It helps reject ab- normal queries and provide potential directions of future development tem (Akasaki and Kaji, 2017; Tulshan and Dhage, 2018; Lin and Xu, 2019; Xu et al., 2020; Zeng et al., a,b; Wu et al., b022a,b; Mou et al., 2022). Since OOD data is hard to label, we need to rely on labeled in-domain samples to facilitate detecting OOD intents.
task-oriented dialogue sys-
Previous OOD detection studies generally work on the assumption that plenty of labeled IND in- tents exist. They require labeled in-domain data
"The three authors contribute equally. Weiran Xu the corresponding author. 'Wo release our code at http://githab.oo//upp.. Wang/App-OOD.
To solve the issues, we propose an Adaptive Prototypical Pseudo labelig (APP) for Few-shot

---

Stagel: Prototypical Classifier Training
Stage 2: Adaptive Pseudo-labeling
Instance- prototype instance Loss
Instance
OOD Boundary Loss
IND Boundary Loss
Prolutype
sample
Pacudo IND sample
Loss
sumple
C Unlabeled sample
Boundary
POOOOOOO
Self-
training
BERT
Stagel:
In-domain labeled samples
Stage2:
Unlabeled samples
Figure I: The overall architecture of our adaptive prototypical pscudo-labeling (APP) for few-shot OOD detection.
OOD Detection. To learn discriminative repre- sentations, we propose a prototypical OOD detec- tion framework (ProtoOOD) using limited IND data. Inspired by the idea of PCL. (Li et al., 2020), we introduce gether samples of the same class and instance-
IND and OOD data by adjusting distances between
IND or OOD samples and prototypes (see Section 4.2).
Our contributions are: (1) We propose an adap- tive prototypical pseudo-labeling (APP) method for few-shot OOD detection. (2) We introduce a proto- typical OOD detection framework (ProtoOOD) to learn discriminative representations and facilitate low-resource OOD detection using limited IND data, and adaptive useudo-labeling method produce high-quality pseudo OOD&IND labels leverage unlabeled mixed data. (3) Experiments and analysis demonstrate effectiveness of our method for few-shot OOD detection.
instance-instanee loss to pull to-
prototype loss to enforce the prototypes to be cen- ter points of classes. After training prototypical in-domain classifier using few-shot IND data, we compute the maximum cosine distance of an in- put query to all in-domain prototypes as the con- fidence score. If the score is above fixed thresh- old, we believe it's an OOD intent. Compared existing OOD detection methods (Hendrycks and Gimpel, 2016; Lin and Xu, 2019; Xu al., 2020; Wu et al., 2022c; Mou et al., 2022), our pro- totypical OOD detection framework models rich class-level semantics expressed implicitly by train- ing instances. Empirical experiments in Section 3.3 demonstrate our framework achieves superior per- formance both on IND and OOD metrics. To lever- age unlabeled mixed data, propose adaptive secudo-labeling method to iteratively label mixed data and update the prototypical IND classifier. We find typical pseudo-labeling methods (Lee, 2013; Cascante-Bonilla al., 2020; Rizve al., 2021) work poorly because the model can't produce high- quality pseudo IND&OOD labels, and get even worse performance. Therefore, we introduce two restance-prototype margin objectives to adaptively pull together pscudo IND samples and prototypes and push apart pseudo OOD samples. We aim to distinguish the confidence score distributions of
Methodology
2.1 Problem Formulation
In the few-shot OOD detection setting, assume that a limited labeled in-domain (IND) dataset Dj {(x;, 9:) })){ consists n samples drawn from IND, and an unlabeled dataset D, = {(x,) m consists m unlabeled samples drawn both ID and OOD. Note that we don't 't know whether an un- labeled sample in D,, belongs to IND or OOD. Our goal is to distinguish whether unknown test sample is drawn from ID not using Dj and D... Compared traditional OOD detection setting, few-shot OOD detection carries two key challenges: learning discriminative representations using lim- ited IND data and leveraging unlabeled mixed data.

---

2.2 Adaptive Prototypical Pseudo-Labeling
instance-prototype loss:
exp (s; ci)
Overall Architecture Fig shows the overall ar- chitecture of our proposed adaptive prototypical pseudo-labeling (APP) for few-shot OOD detection. APP includes two training stages. We first use the limited labeled IND data Dj train a prototypical in-domain classifier to learn discriminative intent representations. Then, we use an adaptive pseudo- labeling method to iteratively label mixed data Du and update the prototypical classifier. In the in- ference stage, we compute the maximum cosine similarity of an input test query to all in-domain prototypes as the confidence score. If the score is below fixed threshold, we believe it's OOD.
Cpotto
log
(2)
Σ |c| j==1 (s: cj)
where c; is the corresponding prototype of the sample 8; and [C] the the total number of proto- types. This objective forces each prototype to lie at the center point of its instances. Our final train- ing objective Cpe = Cine + proto combines the rustance-instanee loss and instance-prototype loss. A daptive Pseudo-Labeling Few-shot OOD de- tection has large corpus of unlabeled data that contains mixed IND and OOD intents. How to ploit the data is vital to both benefiting in-domain intent recognition and OOD detection. However, it's nontrivial to apply existing semi-supervised methods (Lee, 2013; Cascante-Boninn et al., 2020; Sohn et al., 2020) to leveraging the unlabeled data. Because prior knowledge of OOD data is unknown, making it hard to distinguish unlabeled IND and OOD intents simultancously (sce Section 4.2). Therefore, we propose an adaptive pseudo- labeling method to iteratively label mixed data and update the prototypical classifier.
Prototypical OOD Detection Previous OOD de- tection models (Hendrycks and Gimpel, 2016; Xu et al., 2020; Zeng et al., 2021a; Wu et al., 2022e; Mou et al., 2022) are customized for the setting with sufficient labeled IND data and lack general- ization capability to few-shot OOD detection. We find these models suffer from the overconfidence is- sue (Liang et al., 20, where an OOD test sam- ple even gets abnormally high confidence score and is wrongly classified into in-domain types (see Section 4. 1) Therefore, inspired by recent proto- type learning work (Li et al., 2020; Cui et al., 2022), we propose a prototypical OOD detection frame- work (ProtoOOD) to learn discriminative intent representations in the few-shot setting.
Specifically, we design two thresholds 5 and L (S < L): if the maximum cosine similarity of an in- put query to all in-domain prototypes is higer than the larger threshold L, we believe it belongs to IND. And if the similarity is smaller than the smaller threshold S, we believe it belongs to OOD 2, Then the pseudo IND samples can be directly used by optimizing Cyet. To use the pscudo OOD samples x 200D we propose an OOD iistance-prototyy mar- gin objective to push away pseudo OOD samples from in-domain prototypes:
As shown in Fig 1, we first get the hidden state of [CLS] represent an input IND sample, then project it to another embedding space for prototype learning. The prototypes are used as class centroids. Denote = .c..-.. . } as the set of prototype vectors, which are randomly initialized. We intro- duce the following objectives. The first one is the
Lund = max 0, max (cos (a 00D ci) Moon))
(3)
loss: Lins
where Cj is the I-th prototype and MOOD is a margin hyperparameter. However, we find simply applying Lood can't produce correct pseudo IND samples. Because noisy pseudo OOD labels in Lood may also have IND samples and will make the distances of all the samples including IND and OOD to prototypes larger. Therefore, we further propose IND instance-protoyye margin objec-
1 1=
1/4
1=1
(I)
exp (s: 83)
log
Ex=n
1igh exp (st 8%)
where 8;, 85 are projected features from [CLS] hidden states. Ny. is the total number of examples in the batch that have the same label as y4 and 1 is an indicator function. This loss aims pull together samples of the same class and push apart samples from different classes, which helps learn discriminative representations. The second is the
"In the implementation. we use the pre-trained prototype classifier in the first stage to compute the maximum cosine similarity of all the unlabeled data, and select the scores in the first 5th and last 5th positions after sorting in ascending order as the thresholds S and J. These two thresholds are fixed in each macudo-labcling process.

---

10:shot
Mathod
ALL
IND
OOD
ALL
IND
ACC
8-1
ACC
Recall
F1
ACC
Kecall
1-1
ACC
Recall
1-1
27.55
16
36.18
49. 38.38
70.39
58.26
81.35
GIMA
33.47
26.46
46.62
25.64
42.06
56.84
40.75
56.06
74.18
60.4
59.26
75.58
81.97
UniNL
40.64
27.51
44.25
26.17
39.45
53.03
61.68
43.67
58.63
42.14
62.67
72.72
76,43
60.02
70.39
58.74
78.41
83,9
MSP
55.66
37.99
53.51
36.42
56.37
67.74
62,57
46.34
61.45
44.95
62.7
72.73
73.52
59.41
70.39
58.26
74.54
81.35
Energy
58.55
39:57
53.55
37.92
60.19
70.97
66.06
46.87
60.24
47.38
68.76
77.19
77.80
02.52
68, 16
61.52
30.94
85.06
ProtOOD((unn)
70.23
51.14
60.97
49.64
73.26
79.83
75.27
58.75
66.89
57.47
78,02
83.1
79.98
63.61
67.63
62.38
$4.03
87.1
Epd
62.59
72.95
74.55
76.55
83.03
Spt Cood
43.05
70.44
69.31
59.87
58.64
78.15
83.45
fued
54.05
57.45
68.57
67.53
66.35
89.44
89.92
APPlours)
54.78
63.95
76.03
82.51
83.21
71.58
87.03
70.76
71.52
89.11
Table I: Performance comparison on Banking. Here we report the results of 25% IND ratio under 5-shot, 10-shot and 20-shot. Results averaged over three random runs.
tive to pull together pseudo IND samples IND and prototypes:
Cind max 0, max (MIND cos (1. IND ci))
Algorithm 1
Adaptive Prototypical Pseudo-
Labeling
Require: training dataset Dz = { 91 )) and Du = (2:000) ) training steps S, epoch E
where MIND is a margin hyperparameter and cos is the cosine similarity. We show the num- ber of correct predicted pseudo-labeled IND and OOD samples in Fig 6 and find Lood and Cind sys- tematically work as a whole and reciprocate each other. We aim to distinguish the confidence score distributions of IND and OOD data by adjusting distances between IND or OOD samples and pro- totypes. The final training loss in the second stage is L = Cpd + 0.05Lind + 0.05Lood- We show he overall training process in Figure 1 and summarize the pseudo-code of our method in Algorithm 1.
Ensure:
OOD intent detection model, which can classify input query to either one IND class or OOD class. y = (1...., NN, U (OOD). randomly initialize prototype embedding Hj.j = 1,2,
2: for step = 1 to S do
3: sample a mini-batch B from D₁
4: get the embedding of sample through
the projection layer
compute Lins and Lyroto > prototypical contrastive representation learning
3 Experiments
update the network parameters and proto- type vectors
3.1 Datasets
We evaluate our method on two commonly used OOD intent detection datasets, Banking (Casanueva et al., 2020) and Stackoverflow (Xu et al., 2015). Following previous work (Mou et al., 2022), we randomly sample 25%, 50%, and 75% intents as the IND intents, and regard all remaining intents as OOD intents. To verify the effectiveness of our method on few-shot OOD detection with mixed unlabeled data, we divide the original train- ing sets of the two datasets into two parts. We ran- domly sample 5, 10, 20 instances in each IND intent from the original training set construct few-shot labeled IND dataset, while the remaining training samples are treated as the unlabeled mixed data including both IND and OOD class. We only
7: end for
8: for epoch = to E do
9: align sample from Dy with prototypes
10:
compute Linss Cprato and LIND for pseudo-IND samples
11:
compute LOOD for pseudo-OOD samples add Line: Eproto- LIND and LOOD to- gether, and jointly optimize them prototypi- cal contrastive representation learning and adaptive margin learning
12:
13: update the network parameters and proto-
type vectors. 14: end for
WW leave out these hyperparameters to Implementation Detuils in Section A.3.

---

use the few-shot labeled IND dataset train the prototypical in-domain classifier. And we adopt the adaptive pseudo-labeling method to label unlabeled mixed IND and OOD data. Besides, since we focus on few-shot OOD detection setting, we also reduce the number of IND samples in the development set. Test set is the same as orginal data. More detailed information of dataset is shown in Appendix A.I.
RD 0O0 Eredeet
ND OOD
Scare
Scom
(a) GDA
(b) MSP
Donsle
Brroshol
3.2 Baselines
To verify the effectiveness of our prototypical OOD Detection (ProtoOOD) method under the few-shot OOD detection setting. We compare ProtoOOD with OOD detetcion baselines using recent base- lines. For the feature extractor, we use the same BERT (Devlin et al., 2019) as backbone. We com- pare our method with the training objective cross- entropy(CE) and the scoring functions including MSP (Hendrycks and Gimpel, 2017), LOF (Lin and Xu, 2019), GDA (Xu et al., 2020) and En- engy (Wu al., 2022c). Besides, we also com- pare our method with the state-of-the-art baselines Unou et al., 2022). We supplement the de- tails of relevant baselines the appendix A.2.
Srore
Scom
(c) Energy
(d) Prototypical
Figure 2:
Score distribution curves of IND and OOD data using different scoring functions.
IND data, the more obvious the advantages of it. For example, on Banking dataset, ProtoOOD out- performs previous state of wnf-the-an art baseline Energy by 2.04% F1), 2ALL ACC) on 20-shot setting, 5.91%(OOD F1), 8.61 %(ALL ACC) on 10-shot setting and 8. (OOD F1), 11 (ALL ACC) on 5-shot setting. We also observed that the performance of previous methods under the 5-shot setting decreased significantly. We think that this is because previous methods rely on large number of labeled IND samples to learn the generalized intent representations. LOF and KNN-based detec- tion methods are easily affected by outliers. GDA will be affected by the inaccurate estimation of the covariance of the IND cluster distribution under the few-shot setting. MSP and Energy will encounter overconfidence problems due to the over-fitting of the neural network under the few-shot setting, sulting in the wrong detection of OOD as IND. contrast, Proto0OD is insensitive to outlier sam- ples and has good generalization ability. (2) Comparison of different self-training methods. We introduce two mustance-prototypn margin objectives for prototype-based self-training on unlabeled data. To understand the importance of the two margin objectives on our adaptive prototyp- ical pseudo-laheling method, we perform an abla- tion study. The experimental results show that the joint optimization of L Lind and Lood achieves the best performance. We think that since there are not only IND samples but also a large number of OOD samples in the unlabeled data, chal- lenging to directly use the model pre-trained on
For the self- self-training unlabeled mixed data, we compare our adaptive prototypical pseudo-labelligg (APP) method with three different training objec- tives the baselines of our unlabeled data training. (1) Epet: For pseudo IND samples, we only use instance-instanee loss and instance-prototyie loss for training. (2) Lpd + Lind: For pseudo IND samples, in addition to instance-instance loss and instance-prototype loss training, the IND margin objective is added. (3) Epel + Lood: For pscudo IND samples, we use instance-instance loss and instance prottoyye loss, and OOD margin objec- tive training for pseudo OOD samples.
3.3 Main Results
Table and show the performance comparison of different methods on Banking and Stackoverflow dataset respectively. In general, our proposed pro- totypical OOD detection framework (ProtoOOD) and adaptive prototypical pseudo-labeling method (APP) consistently outperform all the baselines with large margin. Next, we analyze the results from two aspects:
(1) Advantage of prototypical OOD detection. We compare our proposed ProtoOOD with previ- ous OOD detection baselines. Experimental results show that ProtoOOD is superior to other methods in both IND and OOD metrics, and the less labeled

---

5-shot
10:shot
20-shot
Mahhol
ALL
IND
OOD
ALL
IND
ACC
8-1
ACC
Recall
F1
ACC:
Kocall
ACC
1-1
ACC
Recall
F-
18.37
54.47
30.29
29.14
$4.50
20.60
46.51
96.87
87.49
GIMA
28.03
24.56
39.47
22.09
24.22
36.92
35.04
27.66
51.74
24.66
29.48
56.57
48.5
70:27
52.00
63.05
UniNL
32,05
20.55
26.53
15.24
33,80
47.09
39.44
34.66
51.00
31.76
35.39
49.16
61.58
51.39
68.3
47.76
59.34
70.69
MSP
36,45
27.32
41.2
23.1
34.87
48.42
42,78
36.94
55.47
33.85
38.56
52,47
58.08
49.27
69.27
45.77
54.36
66.81
Energy
37.63
27.38
44,07
22,95
35,49
49.52
45.75
79.44
57.53
36.20
41,82
55.64
62.3
51.5
67.73
47.5
60.49
71,45
ProtOOD(ounn)
41.75
34.47
53.47
31,02
37.84
51.69
49.22
43.07
63.16
39,89
44.58
58.48
62.63
52,06
67.87
48.11
60.89
71.79
36,95
34.84
45.27
47.59
51.27
43.33
48.98
55.95
70.04
78.67
Spt Cood
33.93
44.56
47,40
$4.05
42.76
45.2
55.55
69.2
53.35
54.33
66.65
Sa fied
40.24
54.14
67.53
72,08
54.92
75.62
62.72
48.67
57.52
94.84
89.72
APP(ours)
58.35
44.87
51.53
60.62
70.81
75.22
65.06
61.67
61.24
79.73
84.18
85.17
73.45
65.47
70.02
91.73
90.61
Table 2: Performance comparison on Stackoverflow. Here we report the results 25% IND ratio under 5-shot, 10-shot and 20-shot. Results are averaged over three random runs.
few-shot labeled IND samples for pseudo-labeling, which will limit the performance of prototype- based self-training. After introducing two instance- prototype margin objectives, which adaptively ad- just the distance between the IND/OOD samples and prototypes, can obtain more reliable pscudo labels stably. We also find that when the two margin objectives are used alone, they will hin- der the prototype-
ping area of IND and OOD intents means that it is more beneficial to distinguish IND and OOD. We can see that the prototypical score can better distinguish IND and OOD under the few-shot set- ting, while other scores suffer serious IND and OOD overlapping. This also shows that compared with the previous OOD detection methods, our Pro- toOOD method can learn more generalized rep- resentations under the few-shot setting, which is beneficial to distinguish IND and OOD.
self-training. This shows that when we use instance-proton prototype margin objec- tives for prototype-based self- teff-training, we need to constrain the distance from IND and OOD to the prototypes at the same time. Besides, in order to explore the adaptability of adaptive prototypical pseudo-labeling on other OOD detection methods, we discuss Appendix C. The conclusion is that our APP method has strong generality and is com- patible with other OOD detection methods well.
4.2 Effect of Adaptive Pseudo-Labeling
Change of IND and OOD score distribution In order to further explore the advantages of adaptive pseudo-I Labeling (APP), we show the change of IND and OOD score distribution curves during the self- trainiiiinng process in Fig 4. We can clearly see that the training process goes on, IND and OOD intents are gradually separated, and there is no case that the OOD samples are too close to the proto- types or the IND samples are too far away from the prototypes. We also show the change of score distribution curves of other teaf-train variants in Appendix A.2. It can be seen that when we remove Lood, a large number of OOD samples will be iden- tified as IND; When we remove Lind, a large num- ber of IND samples will be classified as OOD. We think that this is because the two prstance- margin objectives constrain the distance between IND/OOD and class prototypes respectively. It is necessary to constrain the distances between IND and OOD to the class prototypes at the same time. This also shows that the two instance-prototype margin objectives is beneficial to obtain reliable pseudo labels stably.
4 Qualitative Analysis
4.1 Effect of Prototypical OOD Detection
few-shot OOD detection, how to use limited IND data to learn discriminative intent representa- tions is a key challenge. In order to compare the performance of different representation learning ob- jectives under few-shot settings, we perform intent visualization of CE, KNCL (Mou et al., 2022) and our prototypical contrastive learning objective, as shown in Fig The results show that the prototype- based method can learn more compact IND clus- ters, and the distance between OOD and IND is farther, which facilitates OOD detection.
To analyze the performance of different scoring functions in the few-shot setting, we compare the GDA, MSP, Energy and our proposed Prototype- based score distribution curves for IND and OOD data, as shown in Fig 2. The smaller the overlap-
Changes of class prototypes In addition, we also observe the changes of class prototypes dur- ing adaptive pseudo-labeling in Fig 5. We can

---

122
Can UEEERAA Lourg
44
-72
-42
<25
00
25
16
11.
(a) CE
(b) KNCL
(c)
ProtoOOD
Figure 3: Visualization of IND and OOD intents using different IND pre-training losses.
Denaity
Deneily
Demelly
IND
000
OOD
OOD
OOD
Score
Szore
Score
Score
(a) Epoch = 10
(b) Epoch 20
(c) Epoch = 30
(d) Epoch 40
Figure 4: Change of score distribution curves IND and OOD data during adaptive pseudo-labeling.
see that the class prototypes are gradually close to the center of IND clusters, and away from OOD data. This intuitively reflects that our APP method can adaptively adjust the distance between IND OOD samples and class prototypes by adding two instance-prote pestooye-p margin objectives, which facilitates aseudo-labeling
Threshold
ALL
IND
OOD
ACC
Fl
ACC
Fl
Recall
Fl
End + Lase
72.08
59.30
61.47
54.92
75.62
81.25
T=1
75.32
60.94
61.67
56.38
79,87
83.72
T=5
75.22
65.06
61.67
61.24
79.73
84.18
T=10
72.75
60.71
56.87
56.35
78.04
82.47
Table 3: Effect of threshold on the training performance of adaptive pseudo-labeling on stackhoverflow 25% IND ratio under 10-shot. Here compare the results with Lpet + Lood to prove the robustness of APP.
Changes of the number of correct pseudo la- bels We also count the number of correct pseudo labels in the self-training process. Fig 6 shows the results. The horizontal axis is the epoch self- training, and the vertical axis is the correct number of pseudo labels. It can be seen that when only using Lpat, more and more IND data can be cor- rectly detected, but OOD data cannot be effectively detected. We believe that this is because we only pre-train the model on few-shot labeled IND data, and did not encounter OOD samples during the model pre-training, so simply using fpe for self- training cannot get reliable OOD pseudo labels. When we use Lpd + Lood, we get the opposite result, we can only detect more and more OOD samples. We think that it is because the OOD mar- gin objective only pushes the prototype far away from OOD data, but cannot pull it close more IND data, resulting in the model can not correctly detect more IND data. In contrast, when we use three objectives for joint optimization, we get more
and more correct pseudo IND samples and pseudo OOD samples. This is why APP achieves the best effeet compared with the other two self-training methods. can promote prototype optimization towards the direction of being close to IND and far from OOD. Since using Lpet + Lind gets the similar results as using only Eyel, so we only show the results of Lpd for brevity.
4.3 Hyper-parameter Analysis
The effect of the threshold for pseudo-labeling. We use T to represent the position of the threshold selected when pscudo-labeling. For example, T=5 means that we choose the fifth highest score the upper threshold L and the fifth lowest score as the lower threshold S. We compare the effect of different on the OOD detection performance as shown in Table 3. It can be seen that when

---

amppo
prstaapp
180
10.0
12.0
7.5
7.5
5.0
2.5
0.0
255
-25
-6.
-5.0
-7.5
-7.
(a) Epach= 0
(b) Epoch 28
(c) Epoch=50
Figure 5: Visualization IND and OOD intents during adaptive pseudo-labeling
AL..
IND
OND
Number
2300
Led
Number
3.00
Margin
ACC
61.47
ACC
54.02
F1
Recall
61.25
F1
MIND = 300 Moon L.1
72.5
60.41
36.32
36.54
78.14
$2.27
35300
4000
Mixo 29
$1.94
MMin= Moon 1.I
72.75
60.71
56.87
56.35
78.01
$2.47
=ixo=28
2300
Minn 2aND=28.0 Mooz L.1 72.75
60.71
36.87
56.35
78,01
$2.47
Mixn 28 Monn 1.0
73,32
63.67
59.47
59.82
77.93
$2.90
130
Table 5: Effect of Margin Values on the training perfor-
10
61
1.
20
mance of adaptive pseudo-labeling on StackOverflow 25% IND ratio under 10-shot. Here we compare the results with Lpa + Food to prove the robustness of APP.
Epoch
Epoch
(a) IND
(b) OOD
Figure 6: Numbers of correct pseudo-labeled IND and OOD samples.
efficients. As shown Table 5, it can be seen that coefficient has little effect on the OOD detection performance, which illustrates the robustness of our method.
Model
ALL
ADC
IND
00D
Cur 0.88C... 0.8....
72.11
$3.21
68.19
71.88
67.07
C- Ext C.3 C.dd
82.85
67.09
70,99
65.97
56.70
98.23
4.4
Effect of Differentt Ratios of OOD Data
Table 4: Effect of cofficients on the training perfor- mance of adaptive Iseudo-Jabeling on banking 25%
We compared the effect of different ratios of OOD Data. The results are shown in Table 9. We can see our proposed prototypical OOD Detection out- performs Energy on all IND ratios. shows the effectiveness protoOOD few-shot OOD detec- tion. After the self-training of unlabeled data, the effect of both IND classification and OOD detec- tion are improved. As the ratio of OOD decreases, find that F1-OOD is decreasing and F1-IND is increasing, some OOD samples are more likely be confused with one of the IND intents. The num- ber of IND classes increases the prior knowledge available for IND learning, enabling the model to learn better representations of IND and distinguish it from OOD.
IND ratio under 10-shot.
T takes smaller value, it can often bring better OOD detection performance. This is because the stricter threshold brings higher accuracy of pseudo- labeling, thus improving the detection performance. Moreover, under different T, our method APP all exceeds the performance of other methods, which proves APP is robust for different T.
The effect of the Margin Value for IND and OOD margin loss. We select six different sets of IND margin and OOD margin values to illustrate the impact of Margin Values on detection perfor- mance as shown in Table 5. It can be seen that margin value has little effect on the OOD detection performance, which illustrates the robustness of proposed margin loss.
4.5 Few-shot OOD detection of ChatGPT
Language Learning Models (LLMs) such as Chat- G 4 have demonstrated strong capabilities across a variety of tasks. In order to compare the perfor-
The effect of coefficients for IND and OOD margin loss. We select three different coefficient combination of loss to illustrate the impact of co-
chtte///opemaiccommbb//Caaaaa

---

Model
ALL
IND
OOD
where the (N+ class represents the OOD in- tents. We focus on the unsupervised OOD detec- tion setting where labeled OOD samples are not available for training. Unsupervised OOD detec- tion first learns discriminative representations only using labeled IND data and then employs scoring functions, such as Maximum Softmax Probabil- ity (MSP) (Hendrycks and Gimpel, 2017), Local Outlier Factor (LOFF (Lin and Xu, 2019), Gaussian Discriminant Analysis (GDA) (Xu et al., 2020), En- ergy (Wu et al., 2022c) to estimate the confidence score of a test query. Inspired by recent proto- type models (Li et al., 2020; Cui et al., 2022) for few-shot learning, we propose a prototypical OOD detection framework (ProtoOOD) to facilitate low- resource OOD detection using limited IND data. Besides, (Zhan et al., 2022) is about few-shot OOD detection utilizing generation model create more IND and OOD.
ACC
F1
ACC
F1
Recail
F1
Chilgpt (0-shot)
$1.4
45.8
77.94
45.11
42.72
58.84
Chaigtt (1-shol)
48.96
47.11
82.46
46.72
51.45
Chilgpt (3-hbo))
47,68
49.56
86.2
49.47
55,05
51.35
Chilgpt (5-shol)
44.39
47.11
89,78
47.31
29.57
45.32
APP(5-hh0)
58.35
44.87
51.35
39.68
60.62
70.81
Table 6: LLM's OOD detection ability on banking 50IND ratio.
mance of ChatGPT in OOD detection with our method, we conduct comparative experiments.
The prompt we use is: <Task description> You are an out-of-domain intent detector, and your task is to detect whether the intents of users' queries be- long to the intents supported by the system. If they do, return the corresponding intent label, otherwise return unknown. The supported intents include: [Intent 1J ([Example]] [Example 2]...). ... The text in parentheses is the example of the correspond- ing intent. <Response format> Please respond to me with the format of "Intent: XX" or "Intent: un- known". <Utterance for test> Please tell me the intent of this text: [Here is the utterance for text.] Results are shown in Figure 6. It shows that ChatGPT's effectiveness in identifying OOD sam- ples notably low, as shown by the significantly low Recall-OOD of 29.57 for 5-shot. Through case studies, we discover that it often misclassi- fies OOD samples as IND. This might be due to a clash between its broad general knowledge and specific domain knowledge. It further suggests that it struggles to filter out unusual samples, which is usually the first step in practical uses of the OOD task and constitutes the main goal of this task. On a positive note, its broad general knowl- edge has given it strong abilities to classify IND samples. We hope to explore strategies to merge the strengths of both models to improve overall performance in our future work.
Sell-Super Learning is an active re- search area, including pseudo-labeling (Lee, 2013; Cascante-Bonilly et al., 2020), consistency regular- ization (Verma al., 2019; Sohn al., 2020) and calibration (Yu et al., 2019; Xia al., 2018). Since we focus on the few-shot OOD detection, we use the simple pseudo-labeling method and leave other methods to future work. Different from existing self-supervised learning work based on the assump- tion that all the data is drawn from the same distri- bution, few-shot OOD detection faces the challenge of lack of prior OOD knowledge, making it hard to distinguish unlabeled IND and OOD intents si- multaneously. Therefore, we propose an adaptive pseudo-labeling method to iteratively label mixed data and update the prototypical classifier.
Conclusion
In this paper, we establish a practical OOD detec- tion scenario: there are only a few labeled IND data and massive unlabeled mixed data that may belong to IND or OOD, We find existing OOD work can't effectively recognize OOD queries using limited IND data. Therefore, we propose an adaptive proto- typical pseudo-labeling (APP) method for few-shot OOD detection. Two key components are the pro- totyicall OOD detection framework (ProtoOOD) and adaptive pseudo-labeling strategy. We perform comprehensive experiments and analysis to show the effectiveness of APP. We hope to provide new insight of OOD detection and explore more self- supervised learning methods for future work.
5 Related Work
OOD Detection Previous OOD detection works can be generally classified into two types: super- vised (Fei and Liu, 2016; Kim and Kim, 2018; Larson et al., 2019; Zheng et al., 2020) and un- supervised (Bendale and Boult, 2016; Hendrycks and Gimpel, 2017; Shu al., 2017; Lee et al., 2018; Ren al., 2019; Lin and Xu, 2019; Xu et al., 2020) OOD detection. The former indicates that there are extensive labeled OOD samples in the training data. Fei and Liu (2016); Larson et al. (2019), form a (N+ / cllas classification problem

---

Limitations
Dan Hendrycks and Kevin Gimpel. 2017. A baseline for detecting misclassified and out- -istr--iistiitttion ex- amples in neural networks. ArXiv, abs/ 16 13..02136.
In this paper, we propose an adaptive prototypical pseudo-labeling (APP) method for few-shot OOD detection, including a prototypical OOD detec- tion framework (ProtoOOD) and adaptive pscudo- labeling method. Although our model achieves ex- cellent performance, some directions are still to be improved. (1) We consider a basic self-supervised learning (SSL) method, pseudo-labeling. More other SSL methods should be considered. (2) Al- though our model achieves superior performance than the baselines, there is still large gap to be improved compared to the full-data OOD detection. (3) Apart from SSL, unsupervised representation learning methods (Li and Xiangling. 2022; Zeng et al., 2021c) also make an effect, which are orthog- onal to our pseudo-labeling method.
Joo- Kyung Kim and Young- Young-Bum Kim. 2018. Joint learning of domain classification and out-of-domain detection with dynamic class weighting for satisficing false acceptance rates. ArXiv, abs/ 807.00072.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980
Stefan Larson, Anish Mahendran, Joseph Peper, Christo- pher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael Laurenzano, Lingja Tang, and Jason Mars. 2019. An evalua- tion dataset for intent classification and out-of-scope prediction. In EMNLP/JCNNP..
Dong-Hyun Lee, 2013. Pseudo-label The simple and efficient semi-supervised learning method for deep neural networks.
Kimin Lec, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribbtiin samples and adversarial attacks. ArXiv, abs/ 187.838888 003888.
References
Satoshi Akasaki and Nobuhiro Kaji. 2017. Chat de- tection in an intelligent assistant: Combining task- oriented and con-task-orientee spoken dialogue sys- tems. ArXiv, abs/1 701770500000 00746.
Jinghe Li and Wu Xiangling. 2022. Simelr- based data- driven tight frame for seismic denoising and interpo- lation. GEOPHYSICS.
Abhijit Bendale and Terrance F. Boult. 2016. Towards open set deep networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1563-1572.
Junnan Li, Pan Zhou, Caiming Xiong. Richard Socher, and Steven C. H. Hoi. 2020. Prototypical contrastive learning of unsupervised representations. ArXiv, 0bs/2005.04966..
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulié. 2020. Efficient intent detection with dual sentence encoders. arXiv preprint 200.00077
Shiyu Liang, Yixuan Li, and R. Srikant. 2017a. Prin- cipled detection of out-of-diittibutiinn examples in neural networks. ArXiv, abs/1706.0266)
Paola Cascante- Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. 2020. Curriculum labeling: Revis- iting pseudo- labeling for semi supersupe learning. In AAAI Conference on Artificial Intelligence.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. 2017b. Enhancing the reliability of out-of- distribution image detection in neural networks. arXiv: Learning.
Ganqu Cui, Shengding Hu, Ning Ding, Longtao Huang, and Zhiyuan Liu. 2022. Prototypical verbalizer for prompt- few-shot tuning. In Annual Meeting of the Association for Computational Linguistics.
Ting- Lin and Hua Xu. 2019. Deep unknown intent
detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pages 5491-5496.
J. Devlin, Ming- wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NATCL-HLT.
Prasanta Chandra Mahalanobis. 1936. On the general- ized distance in statistics.
Yutao Mou, Pei Wang. Keqing He, Yanan Wu, Jin- gang Wang, Wei Wu, and Weiran Xu. 2022. Uninl: Aligning representation learning with scoring func- tion for ood detection via unified neighborhood leam- ing, ArXiv, abs/2210.10722.
Geli Fei and Bing Liu. 2016. Breaking the closed world assumption in text classification. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 506-514.
Jie Ren, Peter Liu, Emily Fertig, Jasper Snoek. Ryan Poplin, Mark A. DePristo, Joshua V. Dil- lon, and Balaji Lakshminarayanan. 2019. Likeli- hood ratios for distr---ist detection. ArXiv. 0e//196600244.
Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution ex- amples in neural networks. ArXiv, abs/16 610.02136.

---

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Singh Rawat, and Mubarak Shah. 2021. In defense of pseudo-labeling: An uncertainty-wware pseudo-lahel selection framework for semi-supervised learning. ArXiv, abs/2101.0632
Jiaming Xu, Peng Wang, Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao. 2015. Short text clustering via convolutional neural net- works. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 62-69.
Lei Shu, Hu Xu, and Bing Liu. 2017. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2911-2916
Lequan Yu, Shujun Wang. X. Li, Chi-Wing Fu, and Pheng-Ann Heng. 2019. Uncertainty-awe self- ensembling model for semi-supervised 3d left atrium segmentation. ArXiv, 0bs//197..0704..
Kihyuk Sohn, David Berthelot, Chun- Liang Li, Zizhao Zhang. Nicholas Carlini, Ekin Dogus Cubuk, Alexey Kurakin, Han Zhang, and Colin Raffel. 2020. Fix- match: Simplifying semi-supervised learning with consistency and confidence. ArXiv, abs/2001.0700 007655.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021a. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the Iith International Joint Conference on Nutu- ral Language Processing (Volume Short Papers), pages 570-878, Online. Association for Computa- tional Linguistics.
Amrita S Tulshan and Sudhir Namdeorao Dhage. 2018. Survey on virtual assistant: Google assistant, siri, cortana, alexa. In International symposium on signal processing and intelligent recognition systems, pages 190-201.
Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Hong Xu, and Weiran Xu. 2021b. Adversarial self-suuervised learning for out-of-domain detection. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 5631-5639 Online, Association for Computational Linguistics.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Ben- gio, and David Lopez-Paz. 2019. Interpolation con- sistency training for semi-supervised learning. Neu- ral networks: the official journal of the International Neural Network Society, 145:90-106.
Yanan Wu, Keqing Hc, Yuanmeng Yan, QiXiang Gao. Zhiyuan Zeng, Fujia Zheng. Lulu Zhao, Huixing Jiang. Wei Wu, and Weiran Xu. 2022a. Revisit over- confidence for ood detection: Reassigned contrastive learning with adaptive class-dependent threshold. In NAACL.
Zhiyuan Zeng. Keqing He, Yuanmeng Yan. Hong Xu, and Weiran Xu. 2021c. Adversarial self-supervised learning for out-of-domain detection. In North Amer- ican Chapter of the Association for Computational Linguistics.
Li-Ming Zhan, Haowen Liang, Lu Fan, Xiao-Ming Wu, and Albert YS Lam. 2022. A closer look at few-shot -iitt--ddittibtttinn intent detection. Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 451-460
Yanan Wu, Zhiyuan Zeng. Keqing He, Yutao Mou, Pei Wang, and Weiran Xu. 2022b. Distribution calibra- tion for out-of domain detection with bayesian ap- proximation. In International Conference on Compu- lational Linguistics.
Yinhe Zheng, Guanyi Chen, and Minlie Huang. 2020. Out-of-dommin detection for natural language un- derstanding in dialog systems. IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, 28:1 11119981200 1209.
Yanan Wu, Zhiyuan Zeng, Keqing He, Yutao Mou, Pei Wang. Yuanmeng Yan, and Weiran Xu. 2022c. Dis- entangling confidence score distribution for out-of- domain intent detection with energy-hased learning. ArXiv, abs/22 10.08830.
A Experiment Setups
Yingda Xia, Fengze Liu, D. Yang, Jinzheng Cai, Lequan Yu. Zhuotun Zhu, Daguang Xu, Alan Loddon Yuille, and Holger R. Roth. 2018. 3d semi-supervised learn- ing with uncertainty-aware multi-view co-training. 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 3635-3644.
A.1 Dataset
Since the setting we focus on is few-shot OOD detection with massive mixed unlabeled data, we adjust the original data set. Table 7 shows the sta- tistical information of the adjusted dataset of 25% IND ratio under 10-shot. Taking Stackoverflow an example, it contains 20 intents. First, we di- vide all intents, randomly select intents IND intents, and the other 15 intents as OOD intents. Then we randomly select 10 instances for each IND intent as labeled IND training data to participate in
Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zi- jun Liu, and Weiran Xu. 2020. A deep generative cestance-bascd classifier for out-of-domain detection with mahalanobis space. Proceedings of the 28th International Conference on Computational Linguis- rics, pages 1452-1460, Barcelona. Spain (Online). International Committee on Computational Linguis- tics.

---

Statistic
Avg utterince length
Banking
12
Stackoverrlow
10
Methol
ACC
ALL.
F1
ACC
END
Recall
GOD
FL
Intents
77
20
SCL+GDA
68. 38
49.29
57.37
47.72
71.58
79.02
75.27
58.75
66.89
57.47
78,02
83.1
IND Intents
19
Paculi-lbhdiigg (CT))
73.6
56.25
67.11
75.73
OOD Intents
APPluas)
K3.21
68.19
71.58
67.00
87.03
89.51
Labeled training set size
190
50
Labeled training somples per class
10
10
Table 8: Performance comparison on Banking 25% IND ratio. We report the performance of SCL+GDA and prototypical OOD detection under 10-shot and the performance of the two methods after using the same pseudo-labeling strategy.
Unlabeled Mixed trinnngg size Unlabeled IND training samples Unlaheled OOD training samples
8813
11950
2099
2950
6721
9000
Unlaheled IND training samples per class Unlabeled OOD training samples per Development size
590
190
50
Development samples per class
10
10
Testing Set Size
3080
6000
Testing samples per class
dence score. However, the covariance matrix re- quired to calculate Mahalanobis distance is diffi- cult to accurately estimate with limited ID sam- ples, which will seriously affect the performance of OOD detection.
Table 7: Statistics of the few-shot OOD detection datasets of 25% IND ratio under 10-shot.
the few-shot training in the first stage. Therefore, each intent has 10 labeled training samples, and we have total of 50 labeled training data. The remaining data in the original training set is treated as unlabeled data to participate in the second stage self- trafnnraining. Therefore, there are 590 unlabeled samples left for each IND intent and 600 unlabeled samples left for OOD intent. A total of 1 11950 pieces of data is the training set of the second self- training stage, and their labels are unknown during training. Besides, each IND intent in the development set and the test set is consistent with the original dataset.
Energy (Wu et al., 2022c) maps a sample x to a single scalar called the energy. It uses the threshold on the energy score to consider whether a test query belongs to OOD.
UninL(Mou et al., 2022) uses a unified neighbor- hood learning framework detect OOD intents, in which a KNCL objective is employed for IND pre- training and a based score function is used for OOD detection.Because KNN detection needs to calculate the distance between the test sample and the training sample, the limited training set makes the KNN detection result easy to be dis- turbed by abnormal points.
only retain 10 samples for
A.2 Baselines
A.3 Implementation Details
MSP (Maximum Softmax (oobabilitt))Hendryye and Gimpel, 2017) uses maximum softmax proba- bility the confidence score. If the score is lower than a fixed threshold, the query is regarded as OOD. MSP can easily lead to overconfidence in OOD in low-resource scenarios.
For the training of prototypical OOD detection, we use BERT embed tokens. We initialize the pro- totype embeddings randomly with dimension 256 and optimize the loss function with Adam opti- mizer (Kingma and Ba, 2014). We set the learning rate to le-04 for BERT and le-03 to prototype. We train the model for 20 epochs. The batch size is 20. Then we use the pretrained prototypical OOD clas- sifier to predict unlabeled data. We select the scores in the 5th and last 5th positions after sorting as the upper and lower thresholds. These two thresholds are fixed each pseudo-labeling process. Those with scores higher than the threshold are classified as prediction, and those with scores lower than the lower threshold are classified as OOD. The remain- ing samples are still regarded as unlabeled samples and do participate in training. The coefficients of the three losses are 1.0(Cpet), 0.05(Cina) and O.05(Lood). For two margin losses, the IND mar- gin value MIND is 2.8 and the OOD margin value MooD is 1.1. During inference, we calculate the
LOF (Local Outlier Factor)(Lin and Xu, 2019) uses the local outlier factor to detect unknown in- tents. It detects OOD by comparing the local den- sity of a test query with its k-nearest neighbor's. s. If a query's s local density significantly lower than its k-nearest neighbor's s, is more likely to be re- garded as OOD. LOF needs to check the local den- sity of the sample, which is more vulnerable to the interference of few-shot, leading to the result that is not robust.
GDA (Gaussian Discriminant Analysis)(Xu et al., 2020) is generative distance-based classifier detect OOD samples. estimates the class- conditional distribution on feature spaces of DNNs via Gaussian discriminant analysis, and then ap- plies Mahalamobis distance to measure the confi-

---

Dendry
Density
Denety
Denelyy
ooD
OCD
OCB
OOD
(a) Epoch = 10
(b) Epoch 20
(c) Epoch = 30
(d) Epoch 40
Figure 7: Change of score distribution curve of IND OOD during aseudo-labeling with Eptt. Since the change trend is the same when using Cpd and ppt Lind for self-training only show one of them here.
Denaly
Density
Dereity
Dereily
IND
IND
IND
OOO
OOD
OOD
OOD
Score
Score
Soore
Score
(a)
Epoch = 10
(b) Epoch = 20
(c) Epoch = 30
(d) Epoch = 40
Figure Change of score distribution curve IND and OOD data during pseudo-labeling with Lpet + Lood-
confidence score of IND data in the validation set, sort it from large to small, and select the score at 75% the threshold. We conducted a total of 50 epochs pscudo-labelin To avoid randomness, random runs. The training of prototypical OOD detection lasts about 1.5 minutes and 20 minutes for pseudo-labeling of the second stage both on a single Tesla T4 GPU(16 GB of memory). The average value of the trainable model parameters is 110.18M.
cantly better than SCL+GDA under the few-shot OOD detection setting. In addition, we use our pro- posed APP method for self-training unlabeled data, and find that SCL+GDA also achieve perfor- mance improvement, which shows that our APP method has strong generality, and can be compati- ble with other OOD detection methods well.
and pseudo-data training.
average results over
B Distribution Changes in Adaptive Pseudo-Labeling
We also show the change of score distribution curves of other self-training methods Fig 7 and It can seen that when we remove Loods large number of OOD samples will be identified as IND; When we remove Lind, a large number of IND samples will be classified as OOD.
C Generality of adaptive prototypical pseudo-labeling
Our adaptive prototypical pseudo-labeling (APP) method can also be combined with other OOD de- tection methods. In Table 8, combine APP with SCL+GDA and compare with our ProtoOOD. The experimental results show that the OOD detec- tion method ProtoOOD proposed by us is signifi-

---

25%
75%
Malhod
ALL
INT)
0OD
ALL.
IND
0C0
AIJ.
IND
ODD
ACC
F1
ACC
F-1
Recall
P1
ACC
P1
ACC
Kecall
F1
ACC
F1
ACC:
F1
Recall
F1
Banking
Energy
66.66
48.87
60.24
47.38
68.76
77.19
60.13
57.2
61.34
56.39
00.53
65.36
58.33
61.1
59,49
61.31
55.04
49.22
(oor)
75.27
58.75
66.89
57.47
78.02
83.1
65.61
63.32
67.06
63.17
64.18
69.11
64.62
69.17
65.37
69.45
62.50
53.20
APP(our))
83.21
68.19
71.58
67.07
87.03
N9.51
71.9
68.72
67.99
68.54
75.73
75.53
67.74
73.72
68.47
74.06
65.66
54.11
Stackoverflow
Fnergy
45.75
39.44
57.53
56.20
41.82
55.64
49.38
48.28
53.73
47.85
45.02
52.53
49.56
$2.74
51.82
53.65
42.78
39.16
ProteODD(oun)
49.22
43.07
63.16
39
44.58
5%
4M
$7.33
55.83
60.53
55.45
54.13
59.71
99.25
62.65
60,89
63.62
$4.33
48.17
APP(curs)
75.22
68.06
61.67
61.24
79.73
$4.18
7488
72.27
69.4
71.73
$0.2
77.66
77.95
80.73
73.09
81.4
94,05
70.36
Table 9: Performance comparison on different IND ratios under 10-shot.


Reviews:
[{"review_id": "Rg3OxML3Z2", "reviewer": "Reviewer_LnTL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback on our manuscript. We are encouraged by your recognition of our contributions. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns. \n\n*Q1:  In few-shot learning, in-context examples and prompting strategies are also proven to be effective and play an important role in LLMs. How does the proposed prototypical method compare to those methods? How does the proposed method compare to LLMs in zero-shot setting or few-shot setting ?*      \nA1: Currently, LLM and the techniques like in-context learning are so popular. We agree with the issues you mentioned and we need to consider the effectiveness of LLM in OOD detection in our research. Therefore, we add the experimental results of ChatGPT in zero-shot and few-shot OOD detection, and the data set division is consistent with the previous experiments (Banking-25%). Due to the input length limitation of ChatGPT, we tested OOD detection under 0-shot, 1-shot, 3-shot, and 5-shot settings. More samples will exceed the input length of ChatGPT. The experimental results are as follows.  \n\n| Model | ACC-ALL | F1-ALL | ACC-IND | F1-IND | Recall-OOD | F1-OOD |\n| --- | --- | --- | --- | --- | --- | --- |\n| APP（5-shot） | 58.35 | 44.87 | 51.53 | 39.68 | 60.62 | 70.81 |\n| APP (10-shot) | 83.21 | 68.19 | 71.58 | 67.07 | 87.03 | 89.51 |\n| Chatgpt (0-shot) | 51.4 | 45.8 | 77.94 | 45.11 | 42.72 | 58.84 |\n| Chatgpt (1-shot) | 48.96 | 47.11 | 82.46 | 46.72 | 38.26 | 54.45 |\n| Chatgpt (3-shot) | 47.68 | 49.56 | 86.2 | 49.47 | 35.03 | 51.35 |\n| Chatgpt (5-shot) | 44.39 | 47.11 | 89.78 | 47.21 | 29.57 | 45.32 |\n\nWe find that ChatGPT's ability to detect OOD samples is quite poor resulting in a very low Recall-OOD (29.57 for 5-shot). Through case analysis, we discover that it tends to classify OOD samples as IND samples. This may be due to a conflict between its vast world knowledge and domain-specific knowledge. **However, the weak OOD indicator of ChatGPT means that it cannot filter out abnormal samples, which is usually the first step in practical applications of the OOD task and also the purpose of OOD task.** Certainly, we also find that its vast world knowledge has brought it excellent IND sample classification capabilities. In our future work, we will attempt to combine the strengths of both models to achieve better overall performance.\n\nNote: We tried various prompts, mainly considering three factors: 1) telling ChatGPT whether it is an intent detector or an OOD intent detector; 2) the order of each part of the prompt; 3) whether to include few-shot samples or intent descriptions. In the end, we chose the optimal prompt as follows:\n\nzero-shot prompt：  \n<Task description>\nYou are an out-of-domain intent detector, and\nyour task is to detect whether the intents of\nusers' queries belong to the intents supported\nby the system. If they do, return the\ncorresponding intent label, otherwise return\nunknown. The supported intents include:\n[Intent 1], [Intent 2] [Intent N]\n<Response format>\nPlease respond to me with the format of\n\"Intent: XX\" or \"Intent: unknown\".\n<Utterance for test>\nPlease tell me the intent of this text: [Here is\nthe utterance for text.]\n\nfew-shot prompt :  \n<Task description>\nYou are an out-of-domain intent detector, and\nyour task is to detect whether the intents of\nusers' queries belong to the intents supported by\nthe system. If they do, return the corresponding\nintent label, otherwise return unknown. The\nsupported intents include:\n[Intent 1] ([Example1] [Example 2]...),\n[Intent 2] ([Example1] [Example 2]...), ...\nThe text in parentheses is the example of the\ncorresponding intent.\n<Response format>\nPlease respond to me with the format of\n\"Intent: XX\" or \"Intent: unknown\".\n<Utterance for test>\nPlease tell me the intent of this text: [Here is the\nutterance for text.]\n\n*Q2：The current OOD detection setting has limitations as it only detects out-of-domain instances without specifying the exact intent. A more challenging and practical scenario is to generate new intents in a natural language format. Unlike the existing formulation, a generation framework is not restricted to predefined labels, enabling it to create human-understandable new intents based on given examples.*  \nA2：As shown in the table above, the generative model (ChatGPT) performs well on the IND classification task, but poorly on OOD samples, which may be due to its lack of domain knowledge. However, the universality of large models makes it difficult to resolve this conflict, so traditional OOD detection methods are difficult to abandon. Inspired by you, we attempted to complete the OOD task by prompting the large model to discover the new intent. Below are our experimental results.\n\n| Model | ACC-ALL | F1-ALL | ACC-IND | F1-IND | Recall-OOD | F1-OOD |\n| --- | --- | --- | --- | --- | --- | --- |\n| APP（5-shot） | 58.35 | 44.87 | 51.53 | 39.68 | 60.62 | 70.81 |\n| APP (10-shot) | 83.21 | 68.19 | 71.58 | 67.07 | 87.03 | 89.51 |\n| Chatgpt (OOD detection 3-shot) | 47.68 | 49.56 | 86.2 | 49.47 | 35.03 | 51.35 |\n| Chatgpt (OOD detection- 5-shot) | 44.39 | 47.11 | 89.78 | 47.21 | 29.57 | 45.32 |\n| Chatgpt (new intent generation-3-shot) | 42.27 | 46.78 | 85.7 | 46.96 | 28.02 | 43.3 |\n| Chatgpt (new intent generation-5-shot) | 38.97 | 43.95 | 85.29 | 44.28 | 23.77 | 37.76 |\n\nFrom the results, it seems that the framework for generating new intents has a lower ability to detect OOD compared to our APP and ChatGPT. However, I believe the question you raised is a very promising research direction, and we hope to conduct further research in the future.\n\nThe used prompt:  \n<Task description>\nYou are an intent detector, and\nyour task is to detect whether the intents of\nusers' queries belong to the intents supported\nby the system. If they do, return the\ncorresponding intent label, otherwise return\na new intent label. The supported intents include:\n[Intent 1], [Intent 2] [Intent N]\n<Response format>\nPlease respond to me with the format of\n\"Intent: XX\".\n<Utterance for test>\nPlease tell me the intent of this text: [Here is\nthe utterance for text.]", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper studies an important problem to detect out-of-domain intent. The paper proposes an adaptive prototypical pseudo-labeling (APP) method, which combines a prototypical OOD detection framework (ProtoOOD) with an adaptive pseudo-labeling strategy.  The proposed method achieves promising results and conducted comprehensive results. ", "reasons_to_reject": "1. In few-shot leanring,  in-context examples and prompting strategies are also proven to be effective and play an important role in LLMs. How does the proposed prototypical method compare to those methods?  How does the proposed method compare to LLMs in zero-shot setting or few-shot setting ?\n2. The current OOD detection setting has limitations as it only detects out-of-domain instances without specifying the exact intent. A more challenging and practical scenario is to generate new intents in a natural language format. Unlike the existing formulation, a generation framework is not restricted to predefined labels, enabling it to create human-understandable new intents based on given examples. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addresses the problem of few-shot out-of-domain (OOD) intent detection in task-oriented dialogue systems. Unlike previous studies that assume plenty of labeled in-domain (IND) intents, this paper focuses on a more practical scenario with limited labeled IND data and massive unlabeled mixed data that may belong to IND or OOD. The paper proposes an adaptive prototypical pseudo-labeling (APP) method, which includes a prototypical OOD detection framework (ProtoOOD) for low-resource OOD detection using limited IND data and an adaptive pseudo-labeling method to produce high-quality pseudo OOD and IND labels from unlabeled mixed data. The experiments and analysis demonstrate the effectiveness of the proposed method for few-shot OOD detection in this challenging setting.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ifkmQkVFD", "reviewer": "Reviewer_tHSS", "summary": "", "questions": "1. In the Limitations, the authors point out that the proposed method still has significant gaps compared to full-data OOD detection. In contrast to semi-supervised OOD detection[1], does the APP method have any advantage?\n2. Can the effectiveness of $L_{ins}$ and $L_{proto}$ be demonstrated through ablation studying?\n3. How is the hyperparameter 0.05 for the final loss $L$ determined at line 240? What is the robustness of the model for different hyperparameter choices?\n4. Why does the F1 score of IND improve but the ACC decrease when adding the Adaptive Pseudo-Labeling method to the Stackoverflow dataset, compared to ProtoOOD in Table 2?\n5. Where are the pseudo IND samples in Figure 1?\n", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback on our manuscript. We are encouraged by your recognition of the practical setting, clear motivation, simple and intuitive framework, extensive experimental results, and readability of our paper. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\n*Q1: The methodology is not novel enough. Similar techniques are commonly applied in representation learning, such as InfoNCE loss, triplet loss, etc.*  \nA1: Although InfoNCE loss and triplet loss are commonly used techniques in representation learning, our proposed method is significantly different from these two losses. First of all, our method is based on prototype for few-shot OOD detection. In the first stage, we propose to learn IND representation using prototype contrastive learning. In the second stage, we design two prototype-based boundary losses to learn more diverse intent representations during self-training. Our representation learning method is more suitable for OOD detection and few-shot learning scenarios. In addition, our method has achieved very good results in experiments, demonstrating its effectiveness and superiority in practical applications.\n\n*Q2: In Table 1 & 2, the latest LLM, e.g., ChatGPT is not compared. Moreover, some results in Table 2 are mistakenly bolded, e.g., ACC for 20-shot IND, recall & F1 for 20-shot OOD*   \nA2: We agree with the issues you mentioned and we need to consider the effectiveness of LLM in OOD detection in our research. We add the experimental results of ChatGPT in zero-shot and few-shot OOD detection, and the data set division is consistent with the previous experiments (Banking-25%). Due to the input length limitation of ChatGPT, we tested OOD detection under 0-shot, 1-shot, 3-shot, and 5-shot settings. The experimental results are as follows.\n\n| Model | ACC-ALL | F1-ALL | ACC-IND | F1-IND | Recall-OOD | F1-OOD |\n| --- | --- | --- | --- | --- | --- | --- |\n| APP（5-shot） | 58.35 | 44.87 | 51.53 | 39.68 | 60.62 | 70.81 |\n| APP (20-shot) | 83.21 | 68.19 | 71.58 | 67.07 | 87.03 | 89.51 |\n| Chatgpt (0-shot) | 51.4 | 45.8 | 77.94 | 45.11 | 42.72 | 58.84 |\n| Chatgpt (1-shot) | 48.96 | 47.11 | 82.46 | 46.72 | 38.26 | 54.45 |\n| Chatgpt (3-shot) | 47.68 | 49.56 | 86.2 | 49.47 | 35.03 | 51.35 |\n| Chatgpt (5-shot) | 44.39 | 47.11 | 89.78 | 47.21 | 29.57 | 45.32 |\n\nWe find that ChatGPT's ability to detect OOD samples is quite poor resulting in a very low Recall-OOD (29.57 for 5-shot). Through case analysis, we discover that it tends to classify OOD samples as IND samples. This may be due to a conflict between its vast world knowledge and domain-specific knowledge. **However, the weak OOD indicator of ChatGPT means that it cannot filter out abnormal samples, which is usually the first step in practical applications of the OOD task and also the purpose of OOD task.** Certainly, we also find that its vast world knowledge has brought it excellent IND sample classification capabilities. In our future work, we will attempt to combine the strengths of both models to achieve better overall performance.\n\nNote: We tried various prompts, mainly considering three factors: 1) telling ChatGPT whether it is an intent detector or an OOD intent detector; 2) the order of each part of the prompt; 3) whether to include few-shot samples or intent descriptions. In the end, we chose the optimal prompt as follows:\n\nzero-shot prompt：   \n<Task description>\nYou are an out-of-domain intent detector, and\nyour task is to detect whether the intents of\nusers' queries belong to the intents supported\nby the system. If they do, return the\ncorresponding intent label, otherwise return\nunknown. The supported intents include:\n[Intent 1], [Intent 2] [Intent N]\n<Response format>\nPlease respond to me with the format of\n\"Intent: XX\" or \"Intent: unknown\".\n<Utterance for test>\nPlease tell me the intent of this text: [Here is\nthe utterance for text.]\n\nfew-shot prompt:    \n<Task description>\nYou are an out-of-domain intent detector, and\nyour task is to detect whether the intents of\nusers' queries belong to the intents supported by\nthe system. If they do, return the corresponding\nintent label, otherwise return unknown. The\nsupported intents include:\n[Intent 1] ([Example1] [Example 2]...),\n[Intent 2] ([Example1] [Example 2]...), ...\nThe text in parentheses is the example of the\ncorresponding intent.\n<Response format>\nPlease respond to me with the format of\n\"Intent: XX\" or \"Intent: unknown\".\n<Utterance for test>\nPlease tell me the intent of this text: [Here is the\nutterance for text.]\n\nQ3.The robustness of coefficients before and is not discussed in the experiments, which I think would have a crucial impact on the generalization of the proposed method.   \nA3: Thank you for your correction. We verify through experimental results that this parameter value is an optimal value. The parameter requires a value between 0.01 and 0.001 to obtain a stable result. Here are the results of our ablation experiments in the following table:\n\n| parameter（Banking -25% 10-shot） | ACC-ALL | F1-ALL | ACC-IND | F1-IND | Recall-OOD | F1-OOD |\n| --- | --- | --- | --- | --- | --- | --- |\n| 0.03 | 81.10 | 66.06 | 72.11 | 66.21 | 85.0 | 88.79 |\n| 0.05 | 83.21  | 68.19  | 71.58 |  67.07  | 87.03 |  89.51 |\n| 0.07 | 82.83 | 67.09 | 70.89 | 65.97 | 86.70 | 90.23 |\n\n**The results demonstrate that our parameters have a certain degree of robustness and can provide a stable outcome.**\n\n*Q4: The data set's division lacks clarity, which I think would affect reproducibility.*  \nA4: we adopted three partition methods, which are 25%, 50%, and 75%, respectively. The experiments were conducted and compared under the same partition for each method. We will add the specific partition details in the appendix. Thanks for your correction.\n\n*Q5：In the Limitations, the authors point out that the proposed method still has significant gaps compared to full-data OOD detection. In contrast to semi-supervised OOD detection[1], does the APP method have any advantage?*   \nA5: Thank you for your question. The method [1] you mentioned is a few-shot semi-supervised learning classification algorithm, which extracts features through Prototypical Network and then clusters the features. Because our task involves a large number of OOD samples, which may come from many different intents, we cannot simply transfer the classification method to the OOD detection task. We think this can be a direction for us to explore in future research.\n\n*Q6：Can the effectiveness of and be demonstrated through ablation studying?*  \nA6: Our ablation experiments mainly demonstrate the differences between Lpcl, Lpcl+Loos, and APP in terms of changes in IND and OOD scores and the number of correctly labeled samples. We did not include the results of Lpcl+Lins in the ablation experiments, as we have mentioned the reason for this at line 448. The reason is using Lpcl+Lins gets similar results as using only Lpcl, so we only show the results of Cc for brevity.\n\n*Q7:How is the hyperparameter 0.05 for the final loss determined at line 240? What is the robustness of the model for different hyperparameter choices?*   \nA7: We have added the results of the ablation experiments and the analysis in the answer of Q3\n\n*Q8. Why does the F1 score of IND improve but the ACC decrease when adding the Adaptive Pseudo-Labeling method to the Stackoverflow dataset, compared to ProtoOOD in Table 2?*    \nA8: Thanks for your question. We also notice this experimental result. First, the decrease in IND-ACC and the increase in IND-Precision leads to an increase in IND-F1, as can be seen from the data in the following table. Second, we analyze the reason for this, which may be due to the learning of the representation of IND intent through the few-shot learning in the first stage. After using the adaptive learning algorithm, a large number of OOD samples can be prevented from being misclassified as IND intent, leading to an increase in precision. However, due to the lack of diversity in the features learned through 5-shot, self-learning may introduce noise to the detection of IND intent, leading to a decrease in ACC.\n\n*Q9: Where are the pseudo IND samples in Figure 1?*  \nA9: The true IND samples are in deep orange color, while the fake IND samples are in light orange color. Thank you very much for pointing out the issue with the images. We will immediately correct it in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The setting in this paper is practical, where there are only a few labeled IND data and massive unlabeled mixed data that may belong to IND or OOD.\n2. The motivation is clear and the proposed framework is simple and intuitive.\n3. The extensive experimental results demonstrate the effectiveness of the proposed framework to some extent.\n4. The paper is well written for readability.\n", "reasons_to_reject": "1. The methodology is not novel enough. Similar techniques are commonly applied in representation learning, such as InfoNCE loss, triplet loss, etc.\n2. In Table 1 & 2, the latest LLM, e.g., ChatGPT is not compared. Moreover, some results in Table 2 are mistakenly bolded, e.g., ACC for 20-shot IND, recall & F1 for 20-shot OOD.\n3. The robustness of coefficients before $L_{ind}$ and $L_{ood}$ is not discussed in the experiments, which I think would have a crucial impact on the generalization of the proposed method.\n4. The data set's division lacks clarity, which I think would affect reproducibility.\n", "questions_to_authors": "1. In the Limitations, the authors point out that the proposed method still has significant gaps compared to full-data OOD detection. In contrast to semi-supervised OOD detection[1], does the APP method have any advantage?\n2. Can the effectiveness of $L_{ins}$ and $L_{proto}$ be demonstrated through ablation studying?\n3. How is the hyperparameter 0.05 for the final loss $L$ determined at line 240? What is the robustness of the model for different hyperparameter choices?\n4. Why does the F1 score of IND improve but the ACC decrease when adding the Adaptive Pseudo-Labeling method to the Stackoverflow dataset, compared to ProtoOOD in Table 2?\n5. Where are the pseudo IND samples in Figure 1?\n", "paper_topic_and_main_contributions": "This paper presents a ProtoOOD framework for out-of-domain intent detection in few-shot scenarios. Additionally, a novel adaptive pseudo-labeling method (APP) is proposed to generate high-quality pseudo out-of-domain (OOD) and in-domain (IND) labels, facilitating OOD detection using limited IND data. Extensive experiments demonstrate the effectiveness of this framework.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Ren, Mengye, et al. \"Meta-learning for semi-supervised few-shot classification.\" 6th International Conference on Learning Representations, ICLR 2018. 2018.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure 3, the legend and other sample points are mixed together."}, "extra_text": {}}, {"review_id": "E6Az0lpZ6h", "reviewer": "Reviewer_2RaW", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback on our manuscript. We are encouraged by your recognition of our contributions. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\n*Q1. The problem setting of few-shot OOD detection is new in this work. Previous work such as \"Few-shot Out-of-Distribution Detection\" and \"A Closer Look at Few-Shot Out-of-Distribution Intent Detection\" should properly cited and compared.*   \nA1：Although this paper and the two papers you mentioned are all about the problem of few-shot OOD detection, the setting of this paper is fundamentally different from those methods. **The OOD intent detection in this paper not only involves a small number of few-shot IND samples, but also a large amount of unlabeled mixed data from IND and OOD.** Therefore, we cannot directly compare this paper with the previous articles using their methods. Thank you very much for your attention to the issue of insufficient references in our paper. We will add these references to the paper.\n\n*Q2.The proposed method a combination of existing methods.*   \nA2: Instead of copying the previous work, **we creatively apply PCL and self-training to new problems and add targeted methodological innovations like margin loss and adaptive pseudo-labeling process.** We conducted a detailed analysis of the dynamic process of pseudo-labeling in the second stage, and the experimental results demonstrated the effectiveness of our method. **Besides, the few-shot OOD detection with amount of unlabeled data is the first proposed by us, which is a more practical setting than before.**\n\n*Q3.Baselines should include previous few-shot OOD detection methods and OOD detector using unlabeled data like \"Adversarial self-supervised learning for out-of-domain detection\"*   \nA3: About previous few-shot OOD detection methods, we have expressed our opinion in A1. As for the OOD detector using unlabeled data like \"Adversarial self-supervised learning for out-of-domain detection\". Its motivation is to combine the advantages of unsupervised and supervised methods to extract distinguishable features of OOD intents from unlabeled data. It is different from our research perspective. As we mentioned in the paper, our proposed method is to address a new few-shot scenario, where we face the following problems: **1) how to model IND intents with a small number of IND samples; 2) how to use the preliminary IND intent representation learned in the first stage (which may be inaccurate and unstable) to supplement the diversity loss caused by the small number of IND samples using unlabeled data, avoid noise interference in unlabeled data, and learn the features of OOD samples.** Therefore, the other methods are not suitable for solving this specific setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. An adaptive prototypical pseudo-labeling method is developed for few-shot OOD detection.\n\n2. Experiments and analysis show that limited IND data and mixture data can be properly used to enhance OOD detection performance.", "reasons_to_reject": "1. The problem setting of few-shot OOD detection is new in this work. Previous work such as \"Few-shot Out-of-Distribution Detection\" and \"A Closer Look at Few-Shot Out-of-Distribution Intent Detection\" should properly cited and compared.\n\n2. The proposed method a combination of existing methods.\n\n3. Baselines should include previous few-shot OOD detection methods and OOD detector using unlabeled data like \"Adversarial self-supervised learning for out-of-domain detection\"", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an adaptive prototypical pseudo-labeling method for few-shot OOD detection, where limited IND data and mixture of IND and OOD data exist. And experiments and analysis show the effectiveness of the proposed method.", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of few-shot out-of-domain (OOD) intent detection in the specific setting of very few labelled in-domain (IND) samples and a large set of unlabelled (mixed OOD/IND) samples.  Two of the reviewers appreciate that the presented work addresses this practical setting and that the experimental results demonstrate the effectiveness of the proposed prototypical pseudo-labelling method.  The same reviewers also asked for a comparison with LLMs, which was addressed by the authors in their rebuttal by providing results for chatGPT, along with an analysis and an explanation how it was used.  The other reviewer expressed a more critical opinion that the paper lacked novelty and originality, which the authors addressed in the rebuttal by discussing the differences between their approach and the methods mentioned by this reviewer.|meta review
Decision: Accept-Findings