Continual Generalized Intent Discovery: Marching Towards Dynamic and Open-world Intent Recognition
Authors: Xiaoshuai Song
Yutao Mou
Keqing He
Yueyan Qiu
Jinxu Zhao
Pei Wang
Weiran Xu
Venue: EMNLP 2023 Findings (2023)
Keywords: Out of Domain
Intent classification
Continual Learning

Abstract: In a practical dialogue system, users may input out-of-domain (OOD) queries. The Generalized Intent Discovery (GID) task aims to discover OOD intents from OOD queries and extend them to the in-domain (IND) classifier. However, GID only considers one stage of OOD learning, and needs to utilize the data in all previous stages for joint training, which limits its wide application in reality. In this paper, we introduce a new task, Continual Generalized Intent Discovery (CGID), which aims to continuously and automatically discover OOD intents from dynamic OOD data streams and then incrementally add them to the classifier with almost no previous data, thus moving towards dynamic intent recognition in an open world. Next, we propose a method called Prototype-guided Learning with Replay and Distillation (PLRD) for CGID, which bootstraps new intent discovery through class prototypes and balances new and old intents through data replay and feature distillation. Finally, we conduct detailed experiments and analysis to verify the effectiveness of PLRD and understand the key challenges of CGID for future research.

Continual Generalized Intent Discovery: Marching Towards Dynamic and Open-world Intent Recognition
Xiaoshuai Song", Yutao Mou'r Keqing He Yueyan Qiu' ,Pei Wang', Weiran Xul
*Beijing University of Posts and Telecommunications Beijing, China
Meituan, Beijing, China
(songxiaoshuai, myt, yuanquu, wangee,, (uweiran)(bbppt hwa.nneuputteu..... hekeqingême tuan com
Abstract
IC Thua'a Das
DC 2U
practical dialogue system, users may in- put out-of-domain (OOD) queries. The Gen- eralized Intent Discovery (GID) task aims to discover OOD intents from OOD queries and extend them the in-domain (IND) classi- fier. However, GID only considers one stage of OOD learning, and needs to utilize the data in all previous stages for joint training. which lim- its its wide application in reality. In this paper, we introduce new task, Continual Generalized Intent Discovery (CGID), which aims to con- tinuously and automatically discover OOD in- tents from dynamic OOD data streams and then incrementally add them to the classifier with almost no previous data, thus moving towards dynamic intent recognition in open world. Next, we propose a method called Prototype- guided Learning with Replay and Distillation (PLRD) for CGID, which bootstraps new in- tent discovery through class prototypes and bal- ances new and old intents through data replay and feature distillation. Finally, we conduct detailed experiments and analysis to verify the effectiveness of PLRD and understand the key challenges of CGID for future research.'
Use Usiace
ViA
Syane
Spsket
can
......
G20
Figure 1: Illustration of the defects of GID and the ad- vantages of CGID. GID only perform single stage of OOD learning and requires all IND data for joint train- ing. In contrast, CGID timely updates the system from dynamic OOD data streams through continual OOD learning stage and almost does not rely on previous data.
avoid wrong operations. It can safely reject OOD intents, but also ignores OOD concepts that are valuable for future development. OOD intent dis- covery (Lin et al., 2020; Zhang et al., 2021; Mou et al., 2022c,a) helps determine potential develop- ment directions by grouping unlabelled OOD data into different clusters, but still cannot incremen- tally expand the recognition scope of existing IND classifiers. Generalized Intent Discovery (GID) (Mou et al., 2022b) further trains a network that can classify a set of labelled IND intent classes and simultancously discover new classes from an unlabelled OOD set and incrementally add them to the classifier.
1 Introduction
The traditional intent classification (IC) in a task- oriented dialogue system (TOD) is based on closed set assumption (Chen et al., 2019; Yang et al., 2021; Zeng et al., 2022) and can only handle queries within limited scope of in-domain (IND) intents. However, users may input out-of-domain (OOD) queries in the real open world, Recently, the research community has paid more attention to OOD problems. OOD detection (Lin and Xu, 2019; Zeng et al., 2021; Wu et al., 2022; Mou et al., 2022d) aims to identify whether a user's query is outside the range of the predefined intent set to
A Although GID realizes the incremental expan- sion of the recognition scope of the intent classifier without any new intents labels, two major problems limit the widespread application of GID in reality as shown in Fig I: (1) GID only considers single- stage of OOD discovery and classifier expansion. In real scenarios, OOD data is gradually collected over time. Even if the current intent classifier is incrementally expanded, new OOD queries and intents will continue to emerge. Besides, the time- liness of OOD discovery needs to be considered:
"The three authors contribute equally. Weiran Xu the corresponding author.
'Wo release our code at songriaoshuai/CGID
com/

---

known knowledge well as the interference and propagation of OOD noise (Wu et al., 2021). To ad- dress the issues, we propose the Prototype-guides Learning with Replay and Distillation (PLRD) for CGID. Specifically, PLRD consists of a main mod- ule composed of an encoder and a joint classifier, as well as three sub-modules: (1) class prototype guides pseudo-labels for new OOD samples and alleviate the OOD noise; (2) feature distillation reduces catastrophic forgetting; (3) a memory bal- ances new class learning and old class classifica- tion by replaying old class samples (Section 3) 2. Furthermore, to verify the effectiveness of PLRD, we construct two public datasets and three base- line methods for CGID. Extensive experiments prove that PLRD has significant performance im- provement and the least forgetting compared to the baselines, and achieves a good balance among old classes classification, new class discovery and in- cremental learning (Section 4). To further shed light on the unique challenges faced by the CGID task, we conduct detailed qualitative analysis (Sec- tion 5). We find that the main challenges of CGID are conflicts between different sub-tasks, OOD noise propagation, fine-grained OOD classes and strategies for replayed samples (Section 6), which provide profound guidance for future work.
CIL
CHL
CIL
O O *
40
CGID
Figure 2: The comparison of CGID with GID and CIL.
timely discovery of new intents and expansion to the system can help improve the subsequent user experience. (2) GID require data in all previous stages for joint training to maintain the classi- fication ability for known intents. Since OOD samples are collected from users' real queries, stor- ing past data may bring serious privacy issues. In addition, unlike Class Incremental Learning (CIL) that require new classes with real labels, hard obtain a large amount of dynamic labeled data in reality, and the label set for OOD queries is not predefined and needs to be mined from query logs. Inspired by the above problems, in this paper, introduce a new task, Continual Generalized Intent Discovery (CGID), which aims to continually and automatically discover OOD intents from OOD data streams and expand them to the existing IND classifier. In addition, CGID requires the system to maintain the ability to classify known intents with almost no need to store previous data, which makes existing GID methods fails to be applied to CGID. Through CGID, the IC system can continu- ally enhance the ability of intent recognition from unlabeled OOD data streams, thus moving towards dynamic intent recognition in an open world. We show the difference between CGID and GID, as well as the CIL task in Fig 2 and then leave the definition and evaluation protocol in Section 2.
Our contributions are three-fold: (1) We intro- duce a new task, Continual Generalized Intent Dis- covery (CGID), to achieve the dynamic and open- world intent recognition and then construct datasets and baselines for evaluating CGID. (2) We propose practical method PLRD for CGID, which guides new samples through class prototypes and balances new and old tasks through data replay and feature distillation. (3) We conduct comprehensive experi- ments and in-depth analysis to verify the effective- ness of PLRD and understand the key challenges of CGID for future work.
2 Problem Definition
In this section, first briefly introduce the Gen- eralized Intent Discovery(GID) task, then delve into the details of the Continual Generalized Intent Discovery (CGID) task we proposed.
2PLRD's memory mechanism stores only a tiny fraction of samples. offering significant privacy advantage compared to GID, which stores all past data. PLRD serves not only to provide a privacy- privacy-conscoou mode of learning but also takes into account the long-term stability of task performance. As for the methods that completely eliminate the need for prior data storage, we leave them for further exploration.
As CGID needs to continuously learn from unla- beled OOD data, it is foreseeable that the system will inevitably suffer from the catastrophic forget- ting (Biesialska et al., 2020; Masana et al., 2022) of

---

Perudo Label
Am
Replayed Labels
Protalype
Joint Clacsifier
Ej
Now Prooepe
Capy
BERT
Frosen
New Samples
Replay Samples
Dow
Dyno
D
Memory
IND Training Stage
Continual 0OD Training Stages
Figure 3: The Overall architecture of our PLRD method. During the IND training stage, we only use the cross- entropy loss. In the OOD training stages, multiple modules and learning objectives jointly optimize the model.
2.1 GID
2.3 Evaluation Protocol
Given set of labeled in-domain data DIND (x V DND and unlabeled OOD data (<x000) )= where y/ NDD E GID aims to train a joint classifier to classify an input query into the total label set Y (I...., N, + +,....... l........ N + M), where the first N elements represent the la- beled IND classes and the last M elements repre- sent newly discovered unlabeled OOD classes.
For CGID, we mainly focus the classification performance along the training phase. Following (Mehta al., 2021), let at,i" denote the accu- racy on class set Y, after training on stage t. When we calculate the accuracy A₃ as follows:
DOOD
YIND ,YIND
A AND 4t
DOOD
Σ Yilon
H(ssist)
1 ALL
Yilar,
2.2
CGID
Yall
contrast, CGID provides data and expands the classifier in a sequential manner, which is more in line with real scenarios.
(1)
Moreover, to measure catastrophic forgetting CGID, we introduce the forgetting F; as follows:
First, define current learning stage of CGID and T denotes the maximum number of learning stages of CGID. In the IND learning stage (t 0), given a labeled in-domain dataset DIND = {xx: y }. the model needs to classify IND classes to a predefined set Yo 21, 11, NN} and learn representations that are also helpful for subsequent stages.
[0, 7], which denotes the
FIND a0,0 - 40,0
FOOD
Yil(as, ai,))
HYYssisH
(2)
PALL
Yallars- - a1,s)
|Yail|
2=0
On the whole, ANND and FUND measure the extent of maintaining the IND knowledge, while 4O0D and FOOD denote the ability to learn the OOD classes. AALL and FALL are comprehensive met- rics for CGID.
Then, series of unlabeled out-of-domain datasets (
DOOD = [x:0bb 11=1 are given in sequence, where (OD.). In the OOD learning stage E [1, T). the model expected discover new OOD classes Y from DOOD and incrementally extend Y, to the classifier while maintaining the ability to classify known classes {Y.gr). The ul- timate goal of CGID is to obtain joint classi- fier that can classify queries into the total label set y:ll Yo UYUYU.... Yr-
3 Method
Overall Architecture
As shown in Fig 3, our proposed PLRD frame- work consists of a main module which composed
"Estimating YY| is out of the scope of this paper. In the following ex experiment, we assume that Y is ground-truth and provide an analysis in Section 5.5.
'Following (Zhang et al., 2021), we use the Hungarian algorithm (Kuhn, 1955) to obtain the mapping between the predicted OOD classes and ground-truth classes in the test set.

---

of encoder and a joint classifier and three sub- modules: (1) Memory module is responsible for replaying known class samples to balance the learn- ing of new classes and maintain known classes; (2) Class prototype module is responsible for gener- ating pscudo-labels for new OOD samples; (3) Fea- ture distillation is responsible for alleviating catas- trophic forgetting of old classes. The joint classifier h consists of an old class classification head hold and a new class classification head heew, outputting logit [rold; p. After stage t ends, nnew will be merged into good i.c., pold + [Iold; [ne"). Then, when stage + 1 starts, a new head gres with the dimension Y will be created.
Banking OOD Ratio
CLINC OOD Ratio
Stage
40%
60%
80%
40%
60%
80%
0
47
32
17
90
60
30
10
15
20
20
30
40
15
20
30
40
15
40
Table I: The number of new classes at each stage.
to all prototypes:
onehot
W; {wold) ai E {xuew)
qi =
0xxxy
1 1r
(3)
where yi is the ground-truth or pseudo label of 2; in M and 0yye4 is |ymll| Idiimesiionnl zero vector. Then we introduce prototypical contrastive learning (PCL) (Li al., 2020) as follows:
3.2 Memory for Data Replay
We equip memory module M for PLRD. After each learning stage, M stores very small number of training samples and replays old class samples in the next learning stage to prevent catastrophic forgetting and encourage positive transfer. Specifi- cally, in the IND learning stage, we randomly se- lect samples for each IND class according to the ground-truth labels; in each OOD learning stage, since the ground-truth labels unknown, we ran- domly select n samples for each new class ac- cording to the pscudo-labels and store them in M together with the pscudo-labels. In the new learning stage, for each batch, we randomly select old class samples {arodd] with the same number new class samples [==ew] from M and input them into the BERT encoder f(-) together with new class samples new i.e., [rnew]] [zold)) {x} (anew) } (zold).
exp(sim (is/ (4)
Lpel =
Σ of log 8
exp(sim(( (Zi, Ht)//T
where denotes temperature, of is the j-th ele- ment of qi and = g(f()))). By pulling similar samples into the same prototype. PCL can learn clear intent representations for new classes and maintain representations for old classes. To further improve the generalization representation, we also introduce the instance-level contrastive loss (Chen et al., 2020) to x; as follows:
exp(sim (2, &)/T)
Line = Σ log Σ, [و exp(sim (2 2/)/T)
(5)
where & denotes the dropout-augmented view 2j. Next, we update new and old prototypes a sample-wise moving average manner to reduce the computational complexity following (Wang et al., 2022). For sample Z;, prototype μj is updated
3.3 Prototype-guide Learning
Previous semantic learning studies (Yu et al., 2020; Wang et al., 2022: Ma et al., 2022; Dong et al.. 2023a,b) have shown that learned representations help to disambiguate the noisy sample labels and mitigate forgetting. Therefore, we build pro- totypes through linear projection layer g(-) after the encoder. In stage > O, we first randomly ini- tialize new class prototype [ E Y,, For sample E {x), we use an |Y!"l|| -dimeeniooall vector qi representing the probabilities of being assigned
follows:
M4 = 7 (1-7)2;
(6)
where the moving average coefficient y an adjustable hyperparameter and j is the index of the maximum element q:-
Finally, for the new sample x; E [arew), its pseudo label is assigned as the index of the nearest new class prototype to its representation 2;- We optimize the joint classifier using cross-entropy fye over both the new and replayed samples.
'In following experiment, we set n= analyze the effects of different the Section 5.3. "When = 1, we additionally initialize the prototypes of IND classes.

---

OODRoti 10M
Ratio 60S
OODRatio 80%
Method
IND
00D
ALL
IND
0OD
ALL
IND
OOD
ALL
F4
Ay
Fr
Ar
Ar
F4
Ar
Fj
Av
F2
F2
Kmeans
27.94
13.04
72.00
14.08
73.03
13.05
22.55
22.13
02.55
12.67
66.71
16.00
64.61
31.42
49.64
15.10
52.94
18.71
DeqAligned
27.91
15.67
73.39
72.16
22.52
68.02
12.50
67.98
16.67
68.04
27.99
58.78
60.81
15.67
R2F
74.20
19.38
74.91
14.92
74.48
17.66
71.28
23.41
69:44
13.06
70.20
17.36
65.59
30.44
62.81
15.48
63.41
18.78
PLRD
83.94
9.64
76.70
10.40
70.30
10.69
74.77
76.23
66.06
13,09
Table 2:
Performance comparison on Banking after the final stage T=3. 1 indicates higher is better, | indicates
lower is better. We bold the best results and underline the second-beee results. Results averaged over three random run (p < 0.01 under t-test).
Rationilis
RationédSs
Methou
IND
OOD
ALL
IND
OOD
ALL
IND
OOD
ALL
Ar t
Fr 4
Ay
Fr
Ay
Fr 4
Ay 1
Fr
Ay 1
F4
Ay
Fy 4
Ar
Fy.
Ay
Fly
Ap
Fy
Kincans
K5.57
88
3.67
1.23
6.30
74.74
8.51
DeepAligned
91.98
6.22
89.45
4.61
90.96
5.58
88.04
10.82
86.15
5.04
86.91
7.35
84.30
14.24
80.59
6.25
81.33
7.95
E2D
90.74
$4.46
10:30
PLRD (Ours)
93.90
4.29
91.22
5.00
92.83
4.58
92.15
6.70
89.09
3.78
90.31
4.96
89.56
9.48
84.76
5.33
85.72
6,16
Table 3: Performance comparison CLINC.
3.4 Feature Distillation
respectively) as OOD types, with the rest being IND types. Furthermore, we assign the maximum stage 7=3, so we divide the OOD data into three equal parts for each OOD training stage. We show the number of classes at cach stage in Table and leave the detailed statistics in Appendix A.
It can be expected that the encoder features may change significantly when updating the network pa- rameters in the new learning stage. This means that the network tends forget the knowledge learned from the old classes before and suffers from catas- trophic forgetting. To further remember the knowl- edge in the non-forgt features, we integrate the feature distillation into PLRD. Specifically, at the beginning of stage f. we copy and freeze the encoder, denoted as (.). Then given replayed samples x; E {x= in batch, we constrain the feature output (xx) of the current encoder with the feature finit (x;)). Formally, the feature distillation
4.2 Baselines
Since this is the first study on CGID, there are no existing methods that solve exactly the same task. We adopt three prevalent methods in OOD discov- ery and GID, and extend them the CGID setting to develop the following competitive basclines?.
K-means a pipeline baseline which first use the clustering algorithm K-means (MacQueen, 1965) to cluster the new samples to obtain pscudo labels and then combine these samples and replayed samples in the memory to train the joint classifier each OOD training stage.
loss is as follows:
[[2oid]]
f =
(f(x;) finit fintt (Ix))²
3.5 Overall Training
DeepAligned is another pipeline bascline that leverages the iterative clustering algorithm DeepAligned (Zhang et al., 2021). each OOD training phase, DeepAligned iteratively clusters the new data and then utilizes them along with the replayed samples for classification training.
The total loss is defined as follows:
L = Lre + fapel + Lins + Ldd
(8)
4
Experiment
4.1
Datasets
E2E is an end-to-end baseline. At each OOD training stage, E2E (Mou et al., 2022b) amalga- mates the new instances and replayed samples and then obtain the logits through the encoder and joint classifier. The model is optimized with a unified classification loss, where the new OOD pseudo-labels are obtained by swapping predic- tions (Caron et al., 2020).
We construct the CGID datasets based on two widely used intent classification datasets, Banking (Casanueva et al., 2020) and CLINC (Larson et al., 2019). Banking covers only single domain, con- taining 13,083 user queries and 77 intents, while CLINC contains 22,500 queries covering 150 in- tents across 10 domains. For the CLINC and Bank- ing datasets, we randomly select specified propor- tion of all intent classes (about 40%, 60%, and 80%
"e leave the detailed implementation of these haselines in Appendix B.

---

(4) IND Leaniigg Stage
(b) GOD Leaming Stage
(6) DQD Leariing Stage
(4) ODD Learning Stage
Figure 4: Intents visualization of different learning stage under Banking OOD ratio 40% (index 0-2 belongs Yo.
index 3-5 belongs to Y1, index 6-8 belongs to Y/ and index 9-11 belongs to Y2).
4.3 Main Results
100
We conduct experiments on Banking and CLINC with three different OOD ratios, as shown in Ta- bles 2 and 3. general, our proposed PLRD con- sistently outperform all the baselines with a large margin. Next, we analyze the results from three aspects:
Clatee Compactness
3.0
90
Acturayy (5]
(1) Comparison of different methods We ob- serve that DeepAligned roughly achieves the best IND performance while E2E has the best OOD performance among all baselines. However, our proposed PLRD consistently outperforms base- lines significantly in both IND and OOD, achieving best performance and new-old task balance. Specif- ically, under the average of three ratios, PLRD is better than the optimal baseline by 7.57% (), 1.09% (ADOD), and 4.06% (AALL) Banking, and by 3.35% (A 0.04% (ADOD), and 2.13% (AALL) on CLINC. As for forgetting, E2E experi- ences a substantial performance drop on old classes when learning new classes, while PLRD is lower than the optimal baseline by 3.72%, 1.69% (FAL.)) on Banking and CLINC respectively. This indicates PLRD does not sacrifice too much performance on old classes when learning new classes and has the least forgetting among all methods.
3.5
Stage
Stage
Stage
Stage
Figure 5: The compactness and accuracy of classes different stages under Banking OOD ratio = 40%.
are easily confused, which leads to serious new-old task conflicts and high forgetting. However, PLRD achieves larger improvements than baselines on Banking, indicating that PLRD can better cope with challenges in fine-graines intent scenarios. (3) Effect of different OOD ratios We observe that as the OOD ratio increases, the forgetting of IND classes increases and accuracy of OOD classes decreases significantly for all methods. For PLRD, when the OOD ratio increases from 40% to 80% on Banking, FIND rises from 9. 9.64% to 19.80%, and AOOD drops from 76.70% to 63. 19%. Intuitively, more OOD classes make it challenging distin- guish samples from different distributions, leading to noisy pseudo-labels. Moreover, the incre- mental process, more OOD classes will update the model to a greater extent, resulting in more IND knowledge forgetting.
(2) Comparison of different datasets We val- idate the effectiveness of our method on different datasets, where CLINC is multi-domain and coarse- grained while Banking contains more fine-grained intent types within a single domain. We see that the performance of all methods on CLINC is signif- icantly better than that on Banking. For example, PLRD is 11. .7% (AA4-) higher CLINC than Banking at an OOD ratio of 60%. In addition, the same OOD ratio, PLRD shows average increase of 7.53% (F/ND), 6.45% (FOOD) and 6.57% (FALL) on Banking over CLINC. We believe this could be because grained new and old classes
Qualitative Analysis
5.1 Representation Learning
In order to better understand the evolution of CGID along different training stages, we visualize the intent representations after each training stage for PLRD in Fig 4. can be seen that in the IND learn

---

Strategy
Replay
4
Ag
DompAlgred
Acc
Var
PERD
random
77.78
0.31
80.39
70.00
74.32
icarl
84.45
0.10
80.55
69.78
74.25
Gen
lcarl controry
48.44
0.62
78.44
61.00
68.25
Deepklypd
Table 4:
Comparison of different selection strategies
RUDD
E2
under Banking OOD ratio 60%.
S0age
Figure 6: The Loss and Gain of the classifier different stage under Banking, where the maximal stage T=6, the number of IND classes is 17, and the number of new classes in each OOD stage is 10.
80
70
60
ALLACC
50
ing stage, the IND classes form compact clusters, while the OOD samples are scattered in space. As the stage progresses, the gray points are gradually colored and move from dispersion to aggregation, indicating that new OOD classes continue to be discovered and learned good representations. In addition, the already aggregated clusters are grad- ually dispersed (see "red" points), indicating that the representations of old classes deteriorating.
40
30
Kmeans
DsopAligned
20
E2E
PLRD
10
ALL
Number replayed samples per class
Figure 7: The effect of different number of replayed samples under Banking OOD ratio = 60%, where ALL means storing all training data in the memory.
Next, to quantitatively measure the quality of representations, we calculate the intra-class and inter-class distances and use the ratio of inter-class distance to intra-class distance the compactness following (Islam al., 2021). We report the com- pactness and accuracy in Fig 5. It can see that the compactness of OOD classes is much lower than that of IND classes, indicating that the repre- sentation learning with labeled IND samples out- performs that with unlabeled OOD samples. As the stage increases, the compactness of the IND classes gradually decreases. And the compactness of the Yi > 0) classes increases significantly when equals i, and then gradually decreases. This demonstrates the learning and forgetting effects in CGID from representation perspective. Further- more, we observe that the maximal compactness Y; decreases as i increases, showing that the learning ability of new classes gradually declines. We attribute this to the noise in the OOD pseudo- labeled data and the greater need to suppress for- getting of more old classes. Finally, the trend of accuracy and compactness remains consistent, sug- gesting that representation is closely related the classification performance of CGID.
the number of supported OOD classes continually expanded. In order to quantify the change the classifier, we define the Loss and the Gain in stage t for CGID as follows:
-FIND Loss: = ANND
Y.all A,LL |oolAND
Gain
(9)
We illustrate the variations in Loss and Gain of all methods over stages in Fig 6. The results show that as the training progresses, the Loss of all methods decreases overall and the Gain increases contin- uously. After finishing the training, although the Loss decrease by 20% roughly, the increase in Gain of PLRD is greater, reaching over 200%. This in- dicates that the Gain generated by CGID much higher than the Loss and brings positive effects to the classifier as a whole. Compared with other methods, PLRD has the lowest Loss and highest Gain at each stage, and its advantage continuous amplifies over stages. These further consolidate the conclusion that PLRD outperforms the basclines.
5.3 Effect of Replaying Samples in Memory
5.2 Loss and Gain of CGID
In this section, we explore the effect of replayed samples from both selection strategies and quantity. Selection Strategy In the CIL task, since the
During the CGID process, the performance of the classifier on IND classes gradually declined, while

---

Model
A
A7 AGOD
AqEL
Method
Num of Classes
Af
Ag
Agul
[=l
t=2
1=3
E2E
71,28
69.44
70,20
DeepAligned
15
15
15
72.16
65 .2.02
67.98
PLRD
80.39
70.00
74.32
Giunnd
Truth
E2E
15
15
15
71.28
69.44
70.20
-w/o Lfá
75.94
66.78
70.58
PLRD
15
15
15
81.07
70.30
74.77
-wlo Lins
79.77
68.17
72.99
DeepAligned
13
14
7
71.25
49.56
58.57
Estimate model isselt
67.03
60.19
-wlo Cya
80.16
69.50
73.93
PLRD
13
14
12
79.22
66,28
71.66
Loe
73.36
63.67
67.69
DeepAlimed
13
12
11
72.42
56.94
63.38
Estimane by IND Model
E2E
13
12
11
67.03
55.33
60.19
Table 5: Ablation study of different learning objective for PLRD under Banking OOD ratio = 60%.
PLRD
13 12
11
79.22
60.72
68.41
Table 6: Estimate the number of new classes at each
OOD learning stage under Banking OOD ratio = 60%.
samples are labeled, we only need to consider the diversity of replayed samples. However, in CGID, we need take the quality pseudo-labels into account additionally. We explore three se- lection strategies for replaying samples: random (randomly sampling from training set), icarl (se- lecting these closest to their prototypes, following (select
on the PLRD framework. When removing L the performance declines significantly in both IND and OOD classes. This suggests that forgetting is one of the main challenges faced by CGID, and mitigat- ing forgetting can bring positive effects to continual OOD learning stage by retaining prior knowledge. In addition, removing Lins and Epet respectively leads to a certain degree of performance decline, indicating that prototype and instance-level con- trastive learning are helpful for OOD discovery and relieving OOD noise. Finally, only retaining the Lee of PLRD will result in the largest accu- racy decline, proving the importance of multiple optimization objectives in PLRD.
Rebuffi et al. (2017)), and the samples farthest from their prototypes). As shown in Table 4, report the pseudo-label ac- curacy (Acc) and average feature variance (Var) of replayed samples, as well as the final classifier accuracy of PLRD. We can see that icarl has the highest pseudo-label accuracy while icarl_ icarl_contrary has the largest sample variance and is inclined to diversity. However, PLRD under the random strat- egy has the highest OOD and ALL accuracy. This diversity
demonstrates that neither accuracy alone leads to better performance. CGID needs to strike a balance between diversity and accuracy of the replayed samples.
5.5 Estimate the Number of OOD intents
In the previous experiments, we assumed that the number of new OOD classes at each stage is pre- defined and is ground-truth. However, in practical applications, the number of new classes usually needs to be estimated automatically. We adopt the same estimation algorithm Zhang et al. (2021); Mou et al. (2022b). 8, Since the estimation algo- rithm is based on sample features, we use the model itself as the feature extractor at the beginning of each OOD learning stage. As shown in Table 6, when the estimated number of classes is inaccurate, the performance of all methods declines to some extent. However, PLRD can estimate the number most accurately and achieve the best performance. Then, in order to align different methods, we con- sistently use the frozen model after finishing the IND training stage as the feature extractor for sub- sequent stages. With the same estimation quality, PLRD still significantly outperforms each baseline, demonstrating that PLRD is robust.
Quantity of Replayed Samples Fig 7 illus- trates the effect of replaying different numbers of previous examples. It is evident that replay- ing more previous examples leads higher ac- curacy. Compared with replaying no examples (n=0), storing just one example for each old class (n=1) significantly improves accuracy, demonstrat- ing that replaying old samples is crucial. In addi- tion, PLRD outperforms the basclines significantly when n & 10, proving PLRD's 's effectiveness with few-shot samples replay. However, when all pre- vious examples replayed (n=AL.L), PLRD per- forms slightly worse than E2E. We believe this is because PLRD's anti-forgetting mechanism limits learning new classes lightly, and replaying all pre- vious examples deviates from the setting of CGID,
5.4 Ablation Study
As reported in Table 5, we perform ablation study to investigate the effect of each learning objective
WWe leave the details of the algorithm in Appendix C.

---

6 Challenges
that OOD intent discovery primarily focus on un- veiling new intents, overlooking the integration of these newfound, unknown intents with the existing, well-defined intent categories.
Based on the above experiments and analysis, we comprehensively summarize the unique challenges faced by CGID:
Conflicts between different sub-tasks In CGID, the discovery and classification of new OOD classes tend towards different features, and learn- ing new OOD classes interfere with existing knowl- edge about old classes inevitably. However, pre- venting forgetting will lead to model rigidity and is not conducive to the learning of new classes.
Generalized Intent Discovery To overcome the limitation of OOD intent discovery that cannot pand the existing classifier, Mou et al. (2022b) pro- posed the Generalized Intent Discovery (GID) task, GID takes both labeled IND data and unlabeled OOD data as input and performs joint classification over IND and OOD intents. As such, GID needs to discover semantic concepts from unlabeled OOD data and learn joint classification. However, GID can only perform one-off OOD learning stage and requires full data of known classes, severely lim- iting its practical use. Therefore, we introduce Continual Generalized Intent Discovery (CGID) address the challenges of dynamic and continual open-world intent classification.
OOD noise accumulation and propagation In the continual OOD learning stage, using pseudo- labeled OOD samples with noise to fine-tune the model well replaying samples with noise will cause the noise accumulate and spread to the learning of new OOD samples in future stages. This will potentially affect the model's ability to learn effectively from new OOD samples in subse- quent stages of learning.
Class Incremental Learning The primary goal of class-incremental learning (CIL) is acquire knowledge about new classes while preserving the information related to the previously learned ones, thereby constructing a unified classifier. Earlier studies (Ke et al., 2021; Geng et al., 2021; Li et al., 2022) mainly focused on preventing catas- trophic forgetting and efficient replay in CIL. How- ever, these studies assumed labeled data streams, whereas in reality large amounts of continuously annotated data are hard to obtain and the label space undefined. Unlike CIL, CGID charts a distinct course by continuously identifying and assimilat- ing new classes from unlabeled OOD data streams. This task presents a set of formidable challenges compared to conventional CIL.
Fine-grained OOD classes Section 4.3 indicate that fine-grained data leads to high forgetting and poor performance. We believe this is because fine- grained new classes and old classes are easily con- fused, which brings serious conflicts between new and old tasks.
Strategy for replayed samples The experiment in Section 5.3 shows that CGID needs to consider the trade-off off between replay sample diversity and accuracy as well as the trade-off between quantity of replayed samples and user privacy.
Continual quantity estimation of new classes Section 5.5 shows that even minor estimation errors for each stage can accumulate over stages, leading to severely biased estimation and deteriorated per- formance.
7 Related Work
Conclusion
OOD Intent Discovery OOD Intent Discovery aims to discover new intent concepts from un- labeled OOD data. Unlike simple text cluster- ing tasks, it considers how to leverage IND prior to enhance the discovery of unknown OOD in- tents. Lin et al. (2020) use OOD representations to compute similarities as weak supervision sig- nals. Zhang et al. (2021) propose iterative method, DeepAligned, that performs representa- tion learning and clustering assignment iteratively while Mou al. (2022c) perform contrastive clus- tering to jointly learn representations and cluster- ing assignments. Nevertheless, it's essential to note
In this paper, we introduce more challenging yet practical task as Continuous Generalized Intent Dis- covery (CGID), which aims at continuously and automatically discovering new intents from OOD data streams and incrementally extending the classi- fier, thereby enabling dynamic intent recognition in an open- world setting. To address this task, pro- pose a new method called Prototype-guided Learn- ing with Replay and Distillation (PLRD). Extensive experiments and qualitative analyses validate the effectiveness of PLRD and provide insights into the key challenges of CGID.

---

Limitations
Guanting Dong, Zechen Wang. Liwen Wang. Daichi Guo, Dayuan Fu, Yuxiang Wu, Chen Zeng. Xuefeng Li, Tingfeng Hui, Keging He, Xinyue Cui, Qixiang Gao, and Weiran Xu. 2023a. prototypical semantic decoupling method via joint contrastive learning for few-shot named entity recognition. In ICASSP 2023 2023 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5.
This paper proposes a new task as Continual Gen- eral Intent Discovery (CGID) aimed at continually and automatically discovering new intents from unlabeled out-of-domain (OOD) data and incre- mentally expand them to the existing classifier. Furthermore, a practical method Prototype-guided Learning with Replay and Distillation (PLRD) is proposed for CGID. However, there are still sev- eral directions be improved: (1) Although PL.D achieves better performance than cach baseline, the performance still has a large gap to improve compared with the theoretical upper bound of a model without forgetting previous knowledge. (2) In addition, all baselines and PLRD use small number of previous samples for replay. The CGID method without utilizing any previous samples is not explored in this paper and can be a direction for future work. (3) Although PLRD does not generate additional overhead during inference, requires maintaining prototypes and a frozen copy of the encoder during training, resulting in additional re- source occupancy. This can be further optimized in future work.
Guanting Dong. Zechen Wang, Jinxu Zhao, Gang Zhao, Daichi Guo, Dayuan Fu, Tingfeng Hui, Chen Zeng, Keging He. Xuefeng Li, Liwen Wang. Xinyue Cui, and Weiran Xu. 2023b. A multi-tast semantic decom- position framework with task-specific pre-training for few-shot ner.
Binzong Geng, Min Yang, Fajie Yuan, Shupeng Wang. Xiang Ao, and Ruifeng Xu. 2021. Iterative network pruning with uncertainty regularization for lifelong sentiment classification. In Proceedings of the 44th International ACM SIGIR conference on Research and Development in Information Retrieval, pages 1229-1238.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Roge- rio Feris. 2021. A broad study on the transferability of visual representations with contrastive learning. In Proceedings of the IEEE/CVF International Confer- on Computer Vision, pages 8845-858 8855.
Zixuan Ke, Hu Xu, and Bing Liu. 2021. Adapting BERT for continual learning of sequence of aspect senti- ment classification tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4746-4755, Online. Association for Computational Linguisties.
References
Magdalena Bicsialska, Katarzyna Biesialska, and Marta R Costa-Jussa. 2020. Continual lifelong learn- ing in natural language processing: A survey, arXiv preprint arXiv:2012.09922
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):83997.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020. Unsupervised learning of visual features by contrast- ing cluster assignments. Advances in neural informa- tion processing systems, 12-9944.
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lec, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLE pages 131 1-1316, Hong Kong, China. Association for Computational Linguistics.
lñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45, On- line. Association for Computational Linguistics.
Qian Chen, Zhu Zhuo and Wen Wang. 2019. Bert for joint intent classification and slot filling. arXiv preprint 1902. 10909.
Guodun Li, Yuchen Zhai Qianglong Chen, Xing Gao, Ji Zhang, and Yin Zhang. 2022. Continual few-shot intent detection. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 333-343, Gyeongju, Republic of Korea. Inter- national Committee on Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR
Junnan Li, Pan Zhou, Caiming Xiong, and Steven CH Hoi. 2020. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04966.
Marco Cuturi. 2013. Sinkhom distances: Lightspeed computation of optimal transport. Advances neu- ral information processing systems, 26.

---

Ting-En Lin and Hua Xu. 2019. Deep unknown intent detection with margin loss. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 5491-5496, Florence, Italy, Association for Computational Linguistics.
Aligning representation learning with scoring fune- tion for OOD detection via unified neighborhood learning. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 7317-7325, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Ting-En Lin, Hua Xu, and Hanlei Zhang. 2020. Dis- covering new intents via constrained deep adaptive clustering with cluster refinement. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 3367-
John Platt et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized like- lihood methods. Advances in large margin classifiers, 10(3):61-74.
Tingting Ma, Huiqiang Jiang, Qianhui Wu, Tiejun Zhao, and Chin-Yew Lin. 2022. Decomposed meta- learning for few-shot named entity recognition. In Findings of the Association for Computational Lin- guistics: ACI. 2022, pages 1584-1596, Dublin, Ire- land. Association for Computational Linguistics.
Sylvestre-Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017. icarl: In- cremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2001-2015
Haobo Wang. Ruixuan Xiao, Yixuan Li, Lei Feng. Gang Niu, Gang Chen, and Junbo Zhao. 2022. Pico: Con- trastive label disambiguation for partial label learning. arXiv preprint 2rXi 220.0840..
MacQueen. 1965. Some methods for classification and analysis of multivariate observations. In Proc. Sth Berkeley Symposium on Math., Stat., and Prob, page 281.
Yanan Wu, Zhiyuan Zeng, Keqing He, Yutao Mou, Pei Wang, and Weiran Xu. 2022. Distribution cali- bration for out-of-domain detection with Bayesian approximation. In Proceedings of the 29th Inter- national Conference Computational Linguistics. pages 608-615, Gyeongju. Republic of Korea. Inter- national Committee on Computational Linguistics.
Mare Masana, Xialci Liu, Barttomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. 2022. Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao. Mingqian Tang, and Yu-Feng Li. 2021. Nge: A uni- fied framework for learning with open-world noisy data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 62-71.
Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2021. An empirical investigation of the role of pre-training in lifelong learning. arXiv preprint urXiv:2112.0913
Yutao Mou, Keging He, Pei Wang, Yanan Wu, Jingang Wang. Wei Wu, and Weiran Xu. 2022a. Watch the neighbors: A unified k-nearest neighbor contrastive learning framework for OOD intent discovery. In Proceedings of the 2022 Conference Empirical Methods Natural Language Processing, pages 1517-1529, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2021. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:21 10. 11334.
Lu Yu, Bartlomiej Twardowski, Xialei Liu, Luis Her- ranz, Kai Wang, Yongmei Cheng, Shangling Jui, and Joost van de Weijer. 2020. Semantic drift compensa- tion for inseemnncrm learning. Proceedings of the IEFF/CVV conference on computer vision and pattern recognition, pages 6982-6991.
Yutao Mou, Keqing He, Yanan Wu, Pei Wang, Jingang Wang. Wei Wu, Yi Huang. Junlan Feng, and Weiran Xu. 2022b. Generalized intent discovery: Learning from open world dialogue system. In Proceedings of the 29th International Conference on Computational Linguistics, pages 707-720, Gyeongju, Republic of' Korea. International Committee Computational Linguistics.
Weihao Zeng, Keqing He, Zechen Wang, Dayuan Fu. Guanting Dong. Ruotong Geng. Pei Wang, Jingang Wang, Chaobo Sun, Wei Wu, and Weiran Xu. 2022. Semi-supervised knowledge-groundee pre- pre-tiain for task-oriented dialog systems. In Proceedings of the Towards Semi- Suprr-iuee and Reinforced Task- Oriented Dialog Systems (SereTOD), pages 39-47, Abu Dhabi, Beijing (Hybrid). Association for Com- putational Linguistics.
Yutao Mou, Keqing He, Yanan Wu, Zhiyuan Zeng. Hong Xu, Huixing Jiang, Wei Wu, and Weiran Xu. 2022c. Disentangled knowledge transfer for OOD intent discovery with unified contrastive learning. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 2: Short Papers), pages 46-53, Dublin, Ireland. Associ- ation for Computational Linguistics.
Zhiyuan Zeng. Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. 2021. Modeling discriminative representations for -om-ann detection with supervised contrastive learning. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing ( (Volume 2: Short Papers),
Yutao Mou, Pei Wang, Keqing He, Yanan Wu, Jingang Wang, Wei Wu, and Weiran Xu. 2022d. UniNL:

---

pages 870-878, Online. Association for Computa- tional Linguistics.
and multi-head clustering (number of heads is is used to improve performance for E2E.
For all experiments of PLRD, we also use SGD with momentum as the optimizer (warm-up ratio is 10%, momentum is 0.9, maximum learning rate is 0.01, and weight decay of 1.5e-4). Class prototype embedding (dimension=118) is obtained by linear projection layer through the output feature (dimen- sion=768) of the encoder. We set the temperature т of prototype and instance-Level contrastive learning to 0.5. For the construction of the augmented ex- amples, we set the dropout value is 0.5. Following (Mou et al., 2022b), we also calibrate the logit yoo by the SK algorithm (Cuturi, 2013) when assigning prototypes. We set the corresponding hyperparam- eters such number of iteration is and e=0.05 same as E2E. When OOD ratio=40% or 60%, the moving average coefficient -=0.7, and when OOD ratio=80%, 7=0.9. We believe that high OOD ratio will lead to high forgetting, while larger moving average coefficient can mitigate this by helping the model remember learned prototypes.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. 2021. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35. pages 14365-14373.
A Datasets
We present detailed statistics for the original dataset Banking and CLINC in Table 7. Then, we show statistics of the CGID datasets that are constructed based on Banking and CLINC in Table 8. Since we conduct three random partitions of classes under each OOD ratio and Banking is dass-imbalanccd, we report the the average number of samples.
B Implementation Details
To ensure a fair comparison for PLRD and all base- lines, we consistently use the pre-trained BERT model (BERT-base uncased ?, with layer trans- former) as the network backbone and add a pooling layer to obtain the intent representation (dimen- sion = 768). In addition, we freeze all but the last transformer layer parameters to achieve bet- ter performance with BERT backbone, and speed up the training process as suggested in Lin et al. (2020). We use the same IND training phase and Memory mechanism for all methods. In the IND training stage, following Zhang et al. (2021), we adopt Adam with linear warm-up as the optimizer, with a batch size of 128 and a learning rate of 5e- 5. and select the best checkpoint according to the accuracy of validation set.
For all methods, we train 200 epochs for each OOD learning stage to achieve sufficient conver- genee. The trainable model parameters of PLRD are almost consistent with the baseline (approxi- mately 9. 9.1M). However, PLRD adopts prototype learning and a frozen BERT for knowledge distil- lation, which leads to additional memory occupa- tion and training computation. It should be noted that PLRD only needs the classifier branch in the inference stage, so there no additional computa- tional and spatial overhead. All experiments use a single Nvidia RTX 3090 GPU (24 GB memory).
During the continual OOD training stages, for the CGID bascline, follow the hyperparameter settings of the original method as much as possible. Specifically, for DeepAligned and K-means, fol- lowing Zhang et al. (2021), we adopt Adam with linear warm- up as the optimizer and set the training batch size to 128, the learning rate to 5e-4. In addi- tion, we set the weight coefficient X =3 of the cross- entropy loss corresponding to the replay samples, which is obtained by grid search À 21, 5,5). For E2E, following Mou et al. (2022b), we use SGD with momentum as the optimizer with lincar warm-up and cosine annealing (warm-up ratio of 10%, momentum of 0.9, maximum learning rate of 0.1, and weight decay of 1.5c-4). In addition, the temperature coefficient of cross-entropy is 0.1
C
The algorithm of estimating the number of new classes
We follow the estimation algorithm (Zhang et al., 2021) to estimate the number of new intents for each OOD stage. Specifically, We assign big K' as the number of clusters (In this paper, is twice the number of ground-truth classes) at the beginning of each OOD stage. As a good feature initialization is helpful for partition aartition-based meth- ods (e.g., K-means) (Platt et al., 1999), we use the encoder f(-) to extract intent features of all new training examples. Then, perform K-means with the extracted features. We suppose that real clusters tend be dense even with K', and the size of more confident clusters is larger than some threshold t. Therefore, we drop the low confidence

---

where |8a] is the size of the th produced cluster, and 8(-) is an indicator function. It outputs 1 if the condition is satisfied, and outputs 0 if not. Notably, we assign the threshold as the expected cluster Length (max mean)
Dataset
Classes
Training
Validation
Test
Vocabulary
Banking
77
9003
1000
3080
5028
79/11.91 28/8.31
CLINC
150
18000
2250
2250
7283
Table 7: Detailed statistics of original Banking and CLINC datasets.
Dataset
OOD Raho
New Classes
Classes
Domains
Triin
Val Samples
Test sumples
40%
47/10/10/10
47/57/67/77
1/1/1/1
5445/1 136/12221 200)
604/127/16/133
188V2280226003000
Banking
60%
52/15/15/15
32/47/62/77
1/1/1/1
3612/11191848/1722
401/202/210/191
1280V B80N248033000
80%
17/20/20/00
17/37/52/77
11/I11
191902432332/23233
214/27 1/258/257
680/ 28001480/228//0000
40%
90/2020/20
30/330000
10/10/10/10
0000022.400224000/2.2
1350/300/3003300
135001550119002222
CLINC
3200036000360000666
9004404450440
9001
80%
30/40/40/4
30/10/
10/10/10/10
3600-4800480044000
450/6006004000
330/1050/165022255
Table Detailed statistics of the CGID datasets, where x/x/x/x respectively represent data under the stage 0/1/2/3.
cluster which size smaller than i, and calculate K with:
K'
K
8(1S)) 2 t)
(10)
mean size N in this formula.
K'


Reviews:
[{"review_id": "kipr3vuVCh", "reviewer": "Reviewer_fQPy", "summary": "", "questions": "- How does the memory module alleviate the privacy concern issues if it stores prior training samples for both IND and OOD stages?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback. We are encouraged by your recognition of  PLRD's superior performance, comprehensive multi-stage analysis, and new dataset creation. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\nQ1: Insufficiently justified arguments and unclear necessity of task. \n\nA1: We apologize for any inconvenience caused by unclear expressions.  Due to the open-ended nature of user queries, intent recognition systems need to continually adapt and learn new intents over time. As outlined in lines 52 to 74 of the paper, while progress has been made in discovering and learning new intents, the existing GID task has limitations in handling OOD data in a single step and requiring all past data. To address this, we introduce the CGID task to overcome these constraints. As discussed in lines 67-69, since OOD samples arise from authentic user queries that may contain sensitive information, storing and accessing prior-stage data could raise user privacy concerns. Our proposed PLRD method allows the model to maintain the classification capability of known intents with minimal storage of past data, significantly mitigating privacy issues. Regarding the timeliness of classifier expansion, as depicted in Figure 1, traditional GID methods severely constrain the flexibility of classifier expansion due to their single-stage learning approach. In contrast, PLRD can incrementally learn new knowledge from unlabeled OOD data in a multi-stage manner, thereby requiring only a small accumulation of OOD data to learn new categories and expand the classifier. This facilitates timely expansion.\n\nQ2: How does the memory module alleviate the privacy concern issues if it stores prior training samples? \n\nA2: As illustrated in Section 3.2, we introduce a memory mechanism that enables PLRD to balance the learning of new and old tasks by replaying previous samples. This replay process necessitates the retention of an exceedingly small subset of samples (in our experimentation, only 5 samples are stored per class). Consequently, in contrast to GID methods that require the storage of all prior data, PLRD significantly reduces the volume of stored data, thereby exhibiting notable advantages in terms of privacy. However, it is imperative to candidly acknowledge that privacy concerns are not entirely resolved despite our diligent efforts. As indicated in Section 5.3 and Figure 7, both PLRD and the baselines encounter a sharp decline in performance when prior data is completely unused. We delve into a thorough consideration of the trade-off between privacy concerns and performance aspects, striving to identify an equilibrium point. The introduction of the PLRD method serves not only to provide a privacy-conscious mode of learning but also takes into account the long-term stability of task performance. As for methods that completely eliminate the need for prior data storage, we intend to leave them for further exploration.\n\nQ3: Typos Grammar Style And Presentation Improvements.\n\nA3: We will enhance our grammar and expression in the revised version based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - PLRD consistently outperforms baselines\n- Detailed analysis examining performance across multiple stages and datasets\n- Construction of new datasets for evaluating dynamic OOD detection.", "reasons_to_reject": "- Arguments are not sufficiently justified or convincing. After reading the introduction several times it was unclear as to why the task of Continual Generalized Intent Discovery is needed, making this work seem very incremental in nature. The authors claim that accessing data from the prior stage raises privacy concerns but do not expound or justify this. Also, the argument for the timeliness of classifier expansion mentioned isn't actually explored in the method proposed.", "questions_to_authors": "- How does the memory module alleviate the privacy concern issues if it stores prior training samples for both IND and OOD stages?", "paper_topic_and_main_contributions": "This paper focuses on the task of Out-of-Domain Discovery for Intent classification systems. Generalized Intent Discovery systems aim to discover new intents from OOD queries and integrate them into the system over time. The authors argue that existing systems are atomic in nature i.e the classifier is expanded in a single stage as opposed to being continual and expanding naturally over time. Additionally, existing generalized intent discovery systems require access to data from all prior stages in order to maintain coverage of known intents when expanding. As such the authors introduce Continual Generalized Intent Discovery (CGID) task which aims to continually and automatically discover OOD intents from OOD data and expand it into the in-domain taxonomy. They propose a method called Prototype guided Learning with Replay and Distillation (PLRD) for solving this task and construct a series of baselines and datasets for evaluating the CGID task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Line 56 - 57: \"GID only considers single stage of OOD discovery and classifier expansion.\": Grammar is incorrect. Also, this is very unclear when reading it. Line 56 - 65 needs rewriting to be more crisp and concise. Given that this is the main argument in your paper, having the reader guess what this means is less than ideal.\n- The mixture of color schemes and shapes across diagrams is confusing. Choose a consistent color scheme and legend, the inconsistency makes it harder the follow.\n"}}, {"review_id": "nmaYIiqxDg", "reviewer": "Reviewer_Ldu4", "summary": "", "questions": "What if, instead of using previous known dataset(which has privacy concerns), system passes randomly generated data to get the features for  knowledge distillation. i.e: to capture the previous learning, before adding new training data, we pass random inputs and capture the projections, and then use this pseudo dataset to maintain previous distribution.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback. We are encouraged by your recognition of  the novel problem definition and comprehensive experimentation. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\nQ1: Limited Novelty.\n\nA1: We aim to expound upon the innovative aspects and contributions of our paper in response to your inquiry regarding the concept of \"Limited Novelty.\" Firstly, we introduce the Continual Generalized Intent Discovery (CGID) task, distinct from GID and traditional continual learning. CGID focuses on discovering new intents and enhancing classifiers within evolving unlabeled OOD data streams. Subsequently, we propose the PLRD method to tackle challenges within the CGID task, encompassing new intent discovery, forgetting, and noise propagation. This method is intricately designed with submodules such as prototype guidance, feature distillation, and data replay, culminating in an effective resolution. Lastly, through dataset creation and baseline establishment, we conduct comparative experiments that highlight significant performance improvements achieved by PLRD. This substantiates PLRD's innovative value in practical applications. In summation, while our paper builds upon existing GID and continual learning concepts to some extent, our innovation notably lies in the amalgamation of these concepts with the real-world challenges inherent to the CGID task. The proposed solution has exhibited significant improvement, further underscoring the advancement brought forth by our work.\n\nQ2: Essentially the same system as GID, with limiting data recall.\n\nA2: Regarding the matter you have raised, we seek to elucidate certain misconceptions. While CGID has been devised as an extension upon the foundation of GID, it presents distinct challenges in various facets, such as continual learning from unlabeled OOD data, catastrophic forgetting, and the accumulation and propagation of OOD noise (refer to Section 6). These challenges drove us to propose the PLRD method as a solution. The perspective you mentioned as \"Essentially the same system as traditional GIDs with limiting recall\" actually corresponds to the baselines in this paper (refer to Section 4.2). Our proposed PLRD, incorporating key components including prototype-guided learning and feature distillation, effectively addresses these challenges, allowing for the discovery of new categories while significantly mitigating forgetting. Thorough experimentation confirms the substantial improvements of PLRD over the baselines (refer to Section 4.3, 5.2, 5.3).\n\nQ3: Still has privacy concerns as it also stores a subset of previous data.\n\nA3: As illustrated in Section 3.2, we introduce a memory mechanism that enables PLRD to balance the learning of new and old tasks by replaying previous samples. This replay process necessitates the retention of an exceedingly small subset of samples (in our experimentation, only 5 samples are stored per class). Consequently, in contrast to GID methods which require the storage of all prior data, PLRD significantly reduces the volume of stored data, thereby exhibiting notable advantages in terms of privacy. However, it is imperative to candidly acknowledge that privacy concerns are not entirely resolved despite our efforts. As indicated in Section 5.3 and Figure 7, both PLRD and all baselines encounter a sharp decline in performance when prior data is completely unused. We delve into a thorough consideration of the trade-off between privacy concerns and performance aspects, striving to identify an equilibrium point. The introduction of the PLRD method serves not only to provide a privacy-conscious mode of learning but also takes into account the long-term stability of task performance. For methods that completely eliminate the need for prior data storage, we hope to keep it for further exploration.\n\nQ4: Privacy-safe approach: replace known data with random inputs for knowledge distillation.\n\nA4: We greatly appreciate your attention and suggestions regarding our paper. We regard the utilization of randomly generated data for feature acquisition followed by knowledge distillation as an intriguing approach, potentially offering a comprehensive solution to privacy concerns arising from the retention of prior data.  However, this method may encounter the following challenges: Firstly, relying on randomly generated data might not accurately capture the distribution of real-world data, leading to disparities between synthetic and actual data that could affect the success of knowledge distillation. Secondly, using synthetic data for knowledge distillation could result in model overfitting and introduce unknown biases. Therefore, it's crucial to ensure a reasonable similarity between the generated data and actual data in the feature space. For example, focusing on feature points near class prototypes might yield data more representative of actual instances. Despite these obstacles, we believe this idea deserves further investigation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-Novel Problem definition. \n\n-Comprehensive Experimentation. ", "reasons_to_reject": "-Limited Novelty. \n\n-Essentially the same system as traditional GIDs with limiting the size of the recall dataset.\n\n-Still has the same privacy concerns as it also stores a subset of previous data. ", "questions_to_authors": "What if, instead of using previous known dataset(which has privacy concerns), system passes randomly generated data to get the features for  knowledge distillation. i.e: to capture the previous learning, before adding new training data, we pass random inputs and capture the projections, and then use this pseudo dataset to maintain previous distribution.", "paper_topic_and_main_contributions": "This work first formulates the problem of Continual Generalized Intent Discovery (CGID), in contrast to traditional Generalized Intent Discovery(GID). The authors argue that Out of Domain Intent Discovery(OOD) in the real world is a continuous process, where new OODs continuously emerge, but traditional GID systems only do intent discovery in one phase only. \n\nTraditional GIDs also need to rely on retraining the system with all of In Domain(ID) data along with the new OOD data, to mitigate the problem of forgetting previous learning. This shortcoming of GIDs has privacy concerns, as the data used by these systems is highly personal, and storing it for later recall can have privacy breaches. \n\nThe authors provide a system to do Continual Generalized Intent Discovery. They use three modules in the system 1) Memory module: stores a subset of previous data for recall. 2)Class prototype module is responsible for generating pseudo-labels for new OOD samples 3) Feature distillation is responsible for balancing between new and old data for alleviating catastrophic forgetting of old classes.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA", "Justification_for_Ethical_Concerns": "NA"}, "extra_text": {}}, {"review_id": "9A0RiByk3L", "reviewer": "Reviewer_3pu5", "summary": "", "questions": "1. Why apply the continual learning paradigm to the GID task?\n2. Why is the main experiment not compared with others, and does using stage=3 fail to demonstrate the superiority of PLRD?\n3. Is the majority of the paper's focus on addressing knowledge forgetting and procedural learning brought about by the continual learning paradigm, deviating somewhat from the core problem of the GID task?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review and valuable feedback. We are encouraged by your recognition of  the new CGID task paradigm, a valuable method, and comprehensive experiments. We sincerely apologize for any unclear presentation and hope this response can resolve your concerns.\n\nQ1: A new GID task with the concept of continual learning and lacks substantial novelty.\n\nA1: We aim to expound upon the innovative aspects and contributions of our paper in response to your inquiry regarding the concept of \"lacks substantial novelty.\" Firstly, we introduce the Continual Generalized Intent Discovery (CGID) task, distinct from GID and traditional continual learning. CGID focuses on discovering new intents and enhancing classifiers within evolving unlabeled OOD data streams. Subsequently, we propose the PLRD method to tackle challenges within the CGID task, encompassing new intent discovery, forgetting, and noise propagation. This method is intricately designed with submodules such as prototype guidance, feature distillation, and data replay, culminating in an effective resolution. Lastly, through dataset creation and baseline establishment, we conduct comparative experiments that highlight significant performance improvements achieved by PLRD. This substantiates PLRD's innovative value in practical applications. In summation, while our paper builds upon existing GID and continual learning concepts to some extent, our innovation notably lies in the amalgamation of these concepts with the real-world challenges inherent to the CGID task. The proposed solution has exhibited significant improvement, further underscoring the advancement brought forth by our work.\n\nQ2: Lacks result comparison, relies on self-constructed baselines, limited stage validation.\n\nA2: In response to the points you raised, we'd like to clarify some misunderstandings to better explain our experimentation and validation process. As stated in lines 300 to 305 of the paper, this is the first work in the field of CGID, lacking direct comparisons with existing work. To address this, we thoroughly examined current GID SOTA methods and continual learning approaches, integrating them into three baseline models. Regarding your concern about the effectiveness validation at \"stage=3,\" we have exhaustively presented the performance metrics of both PLRD and the baselines across stages 0 to 6 in Section 5.2 and Figure 6. These data conspicuously demonstrate that irrespective of the specific stage, the PLRD approach consistently outperforms all baselines. Thus, we remain steadfast in our conviction that our method is poised to manifest its efficacy and potential across the spectrum of learning stages.\n\nQ3: The limitations of the CGID paradigm may make it unsuitable for dynamic OOD intentions in real-world scenarios.\n\nA3: The core design objective of the CGID is to facilitate the ongoing discovery and learning of new intents within a dynamic and open environment, thereby enhancing its adaptability to the continually evolving user intents in the real world. Although the integration of continual learning into the GID framework may introduce certain constraining factors, it is our assertion that the presence of such limiting factors does not constitute an inherent failure of the CGID Paradigm itself; rather, it presents a significant challenge necessitating in-depth investigation and resolution. We firmly believe this to be a domain of continuous evolution. Future research endeavors can delve further into the exploration of strategies to more effectively address these issues to adeptly tackle the challenges of dynamic intent recognition within an open-world context.\n\nQ4: Why apply the continual learning paradigm to the GID task?\n\nA4: In summary, due to the open-ended nature of user queries, intent recognition systems need to continuously discover and learn new intents over time. As discussed in lines 52 to 74 of the paper, although the GID task has made progress in discovering and learning new intents from unlabeled data, it faces two main limitations. Firstly, GID only handles OOD queries and extends the classifier within a single phase. However, in reality, OOD data accumulates over time and requires continual learning for adaptation. Additionally, to preserve recognition of known intents while learning new ones, GID requires using all known intent data for joint training. Nevertheless, storing all previous data raises privacy concerns, as OOD samples are collected from real user queries. To address these limitations and challenges, we incorporate the concept of continual learning into GID, introducing the CGID task.\n\nQ5: The main experiment lacks comparison, stage=3 fails to show PLRD superiority.\n\nA5: Regarding your concern about the limited comparisons in our experiment, we aim to clarify some misconceptions. Firstly, as demonstrated in Table 2 and 3, we extensively compare PLRD with three strong baselines across diverse dataset partitions. Secondly, while we focused on stage=3 in the main experiment due to space constraints, we present the performance metrics of PLRD and all baselines across stages 0 to 6 in Section 5.6 and Figure 6. The results indicate that PLRD outperforms the baselines across all stages, thereby emphasizing its global effectiveness.\n\nQ6: Does the paper primarily address knowledge forgetting and dev from the GID task?\n\nA6: Although our paper introduces the CGID task addressing knowledge forgetting and procedural learning, our main focus remains on addressing the core issue of GID within continual and dynamic OOD data contexts. Firstly, given the dynamic nature of the task, addressing knowledge forgetting and procedural learning within CGID is a necessary step. Although these aspects may appear to deviate from the core problem of GID, they emerge as inherent challenges when attempting to extend the GID framework to handle real-world scenarios. By addressing these challenges, we enhance the applicability of the GID framework, rendering it more effective in managing evolving user intents. Secondly, distinct from traditional continual learning, our work centers on solving continuous learning with unlabeled OOD data. This brings about unique challenges, such as balancing diversity in OOD replay samples and pseudo-label accuracy, as well as the accumulation and propagation of OOD noise. As a result, specific approaches are needed to address these challenges. We hope these explanations can provide clearer insight into the focal points of our work.\n\nQ7: Typos Grammar Style And Presentation Improvements.\n\nA7: We will enhance our grammar and expression in the revised version based on your feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introduction of a New Task Paradigm: This paper introduces a novel task paradigm called CGID, aiming to address the dynamic intent recognition problem in practical dialogue systems. It extends the current Generalized Intent Discovery (GID) task and has significant implications for advancing research in this field\n2. Proposal of PLRD Method: The paper presents the PLRD (Prototype-guided Learning with Replay and Distillation) method, which combines class prototypes, data replay, and feature distillation to offer a novel and effective solution for the CGID task. The comprehensive application of this method in dynamic intent discovery holds technical value.\n3. Comprehensive Experimental Design: This paper includes a well-designed set of experiments to validate the effectiveness of the PLRD method and provides in-depth analysis of the challenges faced by CGID.", "reasons_to_reject": "1. Lack of Significant Innovation: Although this paper introduces a new GID task paradigm and the PRLD method, it mainly incorporates the concept of continual learning and a solution paradigm into GID without presenting substantial innovation.\n2. Insufficient Comparative Analysis: The main experiments do not compare the results with existing works but rely on self-constructed baselines, which lacks persuasiveness. Moreover, the use of a small stage=3 may not adequately validate the effectiveness of the PLRD method.\n3. Limited Capability of CGID Paradigm: While the paper discusses the primary challenges faced by the CGID task, these challenges appear to be a result of incorporating continual learning into GID. It seems that the CGID paradigm itself may not be well-suited to handle the dynamic OOD intent streams in real-world scenarios due to its inherent limitations.", "questions_to_authors": "1. Why apply the continual learning paradigm to the GID task?\n2. Why is the main experiment not compared with others, and does using stage=3 fail to demonstrate the superiority of PLRD?\n3. Is the majority of the paper's focus on addressing knowledge forgetting and procedural learning brought about by the continual learning paradigm, deviating somewhat from the core problem of the GID task?", "paper_topic_and_main_contributions": "Due to the existing limitations of the Generalized Intent Discovery (GID) task, which involves a single-stage learning process, specifically considering only one stage of out-of-domain (OOD) learning, and requires joint training with all previous data, it fails to address the continuous flow of OOD intents in an open-world scenario. To tackle this issue, this paper proposes a novel task paradigm called Continual Generalized Intent Discovery (CGID), which dynamically recognizes OOD intent data streams and extends them to the classifier without relying on previous data. The paper also introduces the PLRD (Prototype-guided Learning with Replay and Distillation) method to address this task.\nThe main contributions of this paper include:\n1. Introduction of CGID: The paper proposes a new paradigm, CGID, to solve the problem of dynamic intent recognition in an open-world scenario. It also introduces evaluation criteria and baselines to assess the effectiveness of the proposed approach.\n2. PLRD Method: To address the CGID paradigm, the paper presents an effective PLRD method that leverages class prototypes to learn new samples and balances the gap between new and old tasks through data replay and feature distillation.\n3. Comprehensive Experimentation and Analysis: The paper conducts thorough experiments and in-depth analysis to validate the effectiveness of the PLRD method and discusses the challenges and prospects of the CGID task.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There some typos in Eq.(4)."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the task of Continual Generalized Intent Discovery (CGID), tackling existing limitations of Generalized Intent Discovery (GID).  Specifically, the paper proposes the Prototype-guided Learning with Replay and Distillation (PLRD) to address this task.  The reviewers appreciate the introduction of a new task that adds continual learning to the GID task, the proposed PLRD model for solving this task, and the experiments that demonstrate the model's effectiveness.  However, the work is considered to be somewhat lacking in novelty.|meta review
Decision: Accept-Findings